SmartBench/cmvr-es
6
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
b4ec07c851
cmvr-es/protos/cmvr/quic_edge/v1/README.md

12 KiB
Raw Blame History

QUIC edge v1 wire format

Scope and topology

QUIC edge v1 carries two functions over one authenticated QUIC connection:

  • a reliable bidirectional control stream for node registration, heartbeat, current local IP addresses, the advertised gRPC endpoint and media metadata;
  • QUIC DATAGRAM payloads for lossy, real-time audio and video.

Robot arm, AGV and other device-control APIs remain on the existing cmvr-es gRPC server. The QUIC node descriptor advertises that server's current endpoint; it does not move robot-control RPCs onto QUIC.

cmvr-es -- custom QUIC edge v1 --> Java receiver / edge gateway
                                               |
                                               +-- WebTransport/HTTP3 --> browser

A browser cannot connect to this custom QUIC ALPN directly. The gateway must terminate v1, register the node, reassemble media fragments and expose WebTransport (or another browser-supported media API). Browser sessions do not participate in edge-node registration or heartbeat.

Build and host-safe defaults

cmvr-es pins MsQuic v2.5.9 and builds it from source into the repository dependency tree. It is not installed system-wide:

script/build_msquic.sh \
  --arch x86 \
  --version 2.5.9 \
  --jobs 2 \
  --clean

cmake -S . -B build-quic \
  -DCMVR_ARCH=x86 \
  -DBUILD_TESTING=ON \
  -DCMVR_INSTALL_DEFAULT_RUNTIME_ASSETS=OFF
cmake --build build-quic -j2
ctest --test-dir build-quic --output-on-failure
cmake --install build-quic

The source helper clones the official v2.5.9 tag, verifies its reviewed source commit, and initializes its QuicTLS submodule into build/third_party/, then installs the relocatable public result under dependency/x86/third_party/msquic/v2.5.9. Its first run therefore requires network access, but neither root privileges nor a system MsQuic package. The result uses a statically linked QuicTLS backend; libmsquic.so is copied to output/lib when cmvr-es is installed.

The repository dependency is declared in request.txt, which adds its include and library directories and installs libmsquic.so* with the other bundled runtime libraries. MsQuic is therefore a required build dependency for cmvr-es. The repository runtime configuration keeps quic_edge disabled, and the existing inbound gRPC server remains enabled independently.

CMVR_INSTALL_DEFAULT_RUNTIME_ASSETS=OFF preserves an existing output/bin/config and output/bin/model while updating the binary and runtime libraries. Leave its default value ON when a clean copy of the repository defaults is desired.

Local real-QUIC verification

The development-only server in test/quic_gateway listens through real MsQuic/TLS/UDP and implements the v1 control and DATAGRAM receiver. With the strict build above, CTest registers:

  • cmvr_quic_msquic_e2e_test, which connects the production edge service and feeds synthetic H.264 video plus AAC audio through MediaSourceHub, then verifies registration, heartbeat, descriptors, DATAGRAMs and frame reassembly;
  • cmvr_es_quic_process_smoke_test, which starts the actual cmvr_es executable with a temporary no-device configuration and verifies registration and heartbeat ACKs against the local gateway.

Both tests use a short-lived loopback certificate generated under the build tree. They do not modify output/bin/config or require physical devices. See test/e2e/README.md and test/quic_gateway/README.md for their exact scope and manual gateway options.

To enable node presence without media:

  1. Configure the gateway address, ALPN cmvr-quic-edge/1, TLS trust, node ID, advertised gRPC endpoint and heartbeat values in cmvr-es/config/tasks/quic_edge_task/quic_edge_task.pb.txt. Certificate paths are relative to the cmvr-es configuration root, for example certs/quic_gateway_ca.pem.
  2. Leave tracks empty, set quic_edge.enable: true, and enable the quic_edge TaskManager entry.

To add a media uplink, also enable the camera/microphone in DeviceManager and add a matching tracks entry. A zero-track configuration is valid: unavailable media must not suppress node registration, IP reporting or heartbeat. maximum_frame_bytes must fit in one atomically admitted DATAGRAM batch; reduce it or increase datagram_send_queue_depth when changing the DATAGRAM size.

Reliable control stream

The edge opens one bidirectional reliable stream after the QUIC handshake. Both directions carry a sequence of length-prefixed protobuf messages:

uint32_be protobuf_length
protobuf EdgeControlEnvelope

The length excludes the four-byte prefix. Each receiver must reject a frame larger than its configured control-frame limit. protocol_version is 1. message_sequence increases independently in each direction; gaps are allowed, but duplicates and reordering are rejected. The edge resets its outbound control sequence on every connection, so its first NodeRegisterRequest currently has message_sequence=0; gateways must not assume the sequence starts at 1. Heartbeat uses a separate sequence so its acknowledgement remains explicit.

The legal session order is:

  1. The edge sends NodeRegisterRequest as its first application message after every QUIC connect or reconnect. It includes node_id, boot_id, software version, the current IPv4/IPv6 interface snapshot and advertised gRPC endpoint.
  2. The gateway replies with NodeRegisterResponse. Media and heartbeat must not start until accepted=true and a non-empty session_id are received.
  3. The edge sends NodeHeartbeat at the negotiated interval. The gateway returns NodeHeartbeatAck with the same session_id and exact acknowledged_sequence. A missing, rejected or mismatched ACK causes the edge to reconnect and register again after its configured backoff. Every heartbeat also carries a fresh DeviceManagerSnapshot; its devices list contains only enabled devices, including enabled devices whose construction, initialization or start failed.
  4. If media tracks are available, the edge sends MediaSessionOpen, binding its session_epoch to the accepted session_id, followed by track descriptors. The edge calls reliable descriptor send before DATAGRAM send for a new generation. QUIC does not guarantee arrival order across a reliable stream and DATAGRAMs, so a gateway must boundedly buffer or drop media whose session or generation metadata has not arrived yet.
  5. The gateway may send ProtocolError; a fatal error closes the connection. The current edge receive path otherwise accepts only NodeRegisterResponse and NodeHeartbeatAck. Although MediaSessionClose is present in the v1 schema, the current edge neither sends it nor accepts it inbound. Gateways must not depend on that message until both sides implement and test it.

The edge sends a fresh local-interface snapshot in every heartbeat, so an IP change is reported without a second protocol. These addresses are edge claims. observed_source_ip is authoritative for the public/NAT-facing address and must be derived by the gateway from the authenticated QUIC peer, never copied from a client field.

The heartbeat's device_manager.devices contains only entries with enabled=true. Disabled configuration entries remain available in the edge's local DeviceManager snapshot but are not transmitted. An enabled entry that fails creation, initialization or start remains visible with MANAGED_DEVICE_STATE_ERROR, has_error=true and an error detail. Its independent health value may still be UNSPECIFIED when no trustworthy device probe exists. kind is the stable category used for machine decisions, while type_name is a concrete implementation name when the object exists and otherwise a category label; it is intended only for display and diagnostics. Senders order entries by device_id to make snapshots deterministic. sampled_at_unix_ms is the snapshot time, whereas status_updated_at_unix_ms records when DeviceManager last changed that row's lifecycle/error record. The enclosing snapshot time is the freshness timestamp for the health observation.

Lifecycle and health are deliberately separate. In particular, DEVICE_HEALTH_STATUS_UNSPECIFIED means that no trustworthy health observation was available; it is not equivalent to DEVICE_HEALTH_STATUS_HEALTHY. Similarly, has_error=false only means that no error is currently confirmed and must not be used to turn unknown health into healthy health. A health probe failure must degrade that row to an unknown or fault result without suppressing the rest of the heartbeat.

The edge caps each diagnostic string at 512 bytes without splitting a UTF-8 code point. Device identifiers, implementation names and manager metadata are not silently truncated because doing so would change identity. The deployment must therefore size maximum_control_frame_bytes for its enabled inventory; the sender and receiver both reject an oversized control frame. Very large inventories require a future explicit pagination/truncation extension rather than silently dropping rows from this enabled-device snapshot.

The locally configured QuicEdgeConfig.heartbeat_interval_ms controls the reporting interval before registration. A gateway may negotiate a different interval through NodeRegisterResponse.heartbeat_interval_ms: zero keeps the locally configured value, while a non-zero value overrides it for the current registered QUIC connection. The edge clamps the negotiated value to its supported safety range and returns to the local value on reconnect until a new registration response is accepted.

device_manager = 9 is an additive protobuf field in NodeHeartbeat, so this extension remains QUIC edge protocol v1. Existing gateways ignore the unknown field. Updated gateways must continue accepting older v1 heartbeats where device_manager is absent and must not treat an absent snapshot as an empty, healthy DeviceManager. Enum values may only be appended; existing numeric meanings must never be renumbered or reused.

DATAGRAM negotiation is required only when at least one media track is enabled. The reliable registration and heartbeat path remains valid for a zero-track node or while all media sources are unavailable.

Media source registration, device start/keyframe callbacks and DATAGRAM packetization run on a dedicated media worker. Reliable registration and heartbeat therefore continue while a media source is slow or unavailable; the reliable send path serializes heartbeat and media metadata so envelope sequence order remains strict.

MediaSourceHub is protocol-neutral and gives each adapter an independent, single-consumer subscription cursor. Its source-start callback receives a cancellation predicate and must check it around potentially blocking device startup. QUIC reconnect/stop and gRPC client cancellation can therefore abandon startup without blocking presence or process shutdown. Ring overflow keeps the newest bounded set of immutable frames, advances a slow cursor to the oldest retained frame, and reports the exact dropped count so protocol adapters can mark a discontinuity and request a new video keyframe.

Media DATAGRAM wire format

Every media DATAGRAM starts with this fixed 64-byte, network-byte-order header:

Offset Size Field
0 4 magic CMQD (0x434d5144)
4 1 protocol version (1)
5 1 media kind (1 video, 2 audio)
6 2 flags
8 2 header size (64)
10 2 fragment index
12 2 fragment count
14 2 payload size
16 4 track ID
20 4 codec generation token
24 8 media session epoch
32 8 DATAGRAM packet sequence
40 8 media frame sequence
48 8 monotonic capture timestamp in microseconds
56 4 complete frame size
60 4 fragment byte offset in the complete frame

Flag bit 0 marks a video keyframe, bit 1 is reserved and must currently be zero, and bit 2 marks a discontinuity. Codec initialization bytes are carried reliably in MediaTrackDescriptor. QUIC already authenticates each DATAGRAM, so the application header has no redundant checksum.

Receivers must key reassembly by (session_epoch, track_id, frame_sequence), drop incomplete frames at their playback deadline, and reject fragments whose offset plus payload exceeds frame_size. A new session epoch invalidates all fragments retained from a previous connection. frame_sequence is generated by the QUIC edge service and remains strictly increasing per track throughout the media epoch, including after a device source restarts.