254 lines
12 KiB
Markdown
254 lines
12 KiB
Markdown
# QUIC edge v1 wire format
|
|
|
|
## Scope and topology
|
|
|
|
QUIC edge v1 carries two functions over one authenticated QUIC connection:
|
|
|
|
- a reliable bidirectional control stream for node registration, heartbeat,
|
|
current local IP addresses, the advertised gRPC endpoint and media metadata;
|
|
- QUIC DATAGRAM payloads for lossy, real-time audio and video.
|
|
|
|
Robot arm, AGV and other device-control APIs remain on the existing cmvr-es
|
|
gRPC server. The QUIC node descriptor advertises that server's current endpoint;
|
|
it does not move robot-control RPCs onto QUIC.
|
|
|
|
```text
|
|
cmvr-es -- custom QUIC edge v1 --> Java receiver / edge gateway
|
|
|
|
|
+-- WebTransport/HTTP3 --> browser
|
|
```
|
|
|
|
A browser cannot connect to this custom QUIC ALPN directly. The gateway must
|
|
terminate v1, register the node, reassemble media fragments and expose
|
|
WebTransport (or another browser-supported media API). Browser sessions do not
|
|
participate in edge-node registration or heartbeat.
|
|
|
|
## Build and host-safe defaults
|
|
|
|
cmvr-es pins MsQuic v2.5.9 and builds it from source into the repository
|
|
dependency tree. It is not installed system-wide:
|
|
|
|
```bash
|
|
script/build_msquic.sh \
|
|
--arch x86 \
|
|
--version 2.5.9 \
|
|
--jobs 2 \
|
|
--clean
|
|
|
|
cmake -S . -B build-quic \
|
|
-DCMVR_ARCH=x86 \
|
|
-DBUILD_TESTING=ON \
|
|
-DCMVR_INSTALL_DEFAULT_RUNTIME_ASSETS=OFF
|
|
cmake --build build-quic -j2
|
|
ctest --test-dir build-quic --output-on-failure
|
|
cmake --install build-quic
|
|
```
|
|
|
|
The source helper clones the official `v2.5.9` tag, verifies its reviewed
|
|
source commit, and initializes its QuicTLS submodule
|
|
into `build/third_party/`, then installs the relocatable public result under
|
|
`dependency/x86/third_party/msquic/v2.5.9`. Its first run therefore requires
|
|
network access, but neither root privileges nor a system MsQuic package. The
|
|
result uses a statically linked QuicTLS backend; `libmsquic.so` is copied to
|
|
`output/lib` when cmvr-es is installed.
|
|
|
|
The repository dependency is declared in `request.txt`, which adds its include
|
|
and library directories and installs `libmsquic.so*` with the other bundled
|
|
runtime libraries. MsQuic is therefore a required build dependency for
|
|
cmvr-es. The repository runtime configuration keeps `quic_edge` disabled, and
|
|
the existing inbound gRPC server remains enabled independently.
|
|
|
|
`CMVR_INSTALL_DEFAULT_RUNTIME_ASSETS=OFF` preserves an existing
|
|
`output/bin/config` and `output/bin/model` while updating the binary and
|
|
runtime libraries. Leave its default value `ON` when a clean copy of the
|
|
repository defaults is desired.
|
|
|
|
## Local real-QUIC verification
|
|
|
|
The development-only server in `test/quic_gateway` listens through real
|
|
MsQuic/TLS/UDP and implements the v1 control and DATAGRAM receiver. With the
|
|
strict build above, CTest registers:
|
|
|
|
- `cmvr_quic_msquic_e2e_test`, which connects the production edge service and
|
|
feeds synthetic H.264 video plus AAC audio through `MediaSourceHub`, then
|
|
verifies registration, heartbeat, descriptors, DATAGRAMs and frame
|
|
reassembly;
|
|
- `cmvr_es_quic_process_smoke_test`, which starts the actual `cmvr_es`
|
|
executable with a temporary no-device configuration and verifies
|
|
registration and heartbeat ACKs against the local gateway.
|
|
|
|
Both tests use a short-lived loopback certificate generated under the build
|
|
tree. They do not modify `output/bin/config` or require physical devices. See
|
|
[`test/e2e/README.md`](../../../../test/e2e/README.md) and
|
|
[`test/quic_gateway/README.md`](../../../../test/quic_gateway/README.md) for
|
|
their exact scope and manual gateway options.
|
|
|
|
To enable node presence without media:
|
|
|
|
1. Configure the gateway address, ALPN `cmvr-quic-edge/1`, TLS trust, node ID,
|
|
advertised gRPC endpoint and heartbeat values in
|
|
`cmvr-es/config/tasks/quic_edge_task/quic_edge_task.pb.txt`.
|
|
Certificate paths are relative to the cmvr-es configuration root, for
|
|
example `certs/quic_gateway_ca.pem`.
|
|
2. Leave `tracks` empty, set `quic_edge.enable: true`, and enable the
|
|
`quic_edge` TaskManager entry.
|
|
|
|
To add a media uplink, also enable the camera/microphone in DeviceManager and
|
|
add a matching `tracks` entry. A zero-track configuration is valid: unavailable
|
|
media must not suppress node registration, IP reporting or heartbeat.
|
|
`maximum_frame_bytes` must fit in one atomically admitted DATAGRAM batch; reduce
|
|
it or increase `datagram_send_queue_depth` when changing the DATAGRAM size.
|
|
|
|
## Reliable control stream
|
|
|
|
The edge opens one bidirectional reliable stream after the QUIC handshake. Both
|
|
directions carry a sequence of length-prefixed protobuf messages:
|
|
|
|
```text
|
|
uint32_be protobuf_length
|
|
protobuf EdgeControlEnvelope
|
|
```
|
|
|
|
The length excludes the four-byte prefix. Each receiver must reject a frame
|
|
larger than its configured control-frame limit. `protocol_version` is `1`.
|
|
`message_sequence` increases independently in each direction; gaps are allowed,
|
|
but duplicates and reordering are rejected. The edge resets its outbound
|
|
control sequence on every connection, so its first `NodeRegisterRequest`
|
|
currently has `message_sequence=0`; gateways must not assume the sequence starts
|
|
at 1. Heartbeat uses a separate sequence so its acknowledgement remains
|
|
explicit.
|
|
|
|
The legal session order is:
|
|
|
|
1. The edge sends `NodeRegisterRequest` as its first application message after
|
|
every QUIC connect or reconnect. It includes `node_id`, the immutable
|
|
`robot_id`, `boot_id`, software version, the current IPv4/IPv6 interface
|
|
snapshot and advertised gRPC endpoint.
|
|
2. The gateway replies with `NodeRegisterResponse`. Media and heartbeat must not
|
|
start until `accepted=true` and a non-empty `session_id` are received.
|
|
3. The edge sends `NodeHeartbeat` at the negotiated interval. The gateway returns
|
|
`NodeHeartbeatAck` with the same `session_id` and exact
|
|
`acknowledged_sequence`. A missing, rejected or mismatched ACK causes the edge
|
|
to reconnect and register again after its configured backoff. Every heartbeat
|
|
also carries a fresh `DeviceManagerSnapshot`; its `devices` list contains
|
|
only enabled devices, including enabled devices whose construction,
|
|
initialization or start failed.
|
|
4. If media tracks are available, the edge sends `MediaSessionOpen`, binding its
|
|
`session_epoch` to the accepted `session_id`, followed by track descriptors.
|
|
The edge calls reliable descriptor send before DATAGRAM send for a new
|
|
generation. QUIC does not guarantee arrival order across a reliable stream
|
|
and DATAGRAMs, so a gateway must boundedly buffer or drop media whose session
|
|
or generation metadata has not arrived yet.
|
|
5. The gateway may send `ProtocolError`; a fatal error closes the connection.
|
|
The current edge receive path otherwise accepts only `NodeRegisterResponse`
|
|
and `NodeHeartbeatAck`. Although `MediaSessionClose` is present in the v1
|
|
schema, the current edge neither sends it nor accepts it inbound. Gateways
|
|
must not depend on that message until both sides implement and test it.
|
|
|
|
The edge sends a fresh local-interface snapshot in every heartbeat, so an IP
|
|
change is reported without a second protocol. These addresses are edge claims.
|
|
`observed_source_ip` is authoritative for the public/NAT-facing address and must
|
|
be derived by the gateway from the authenticated QUIC peer, never copied from a
|
|
client field.
|
|
|
|
The heartbeat's `device_manager.devices` contains only entries with
|
|
`enabled=true`. Disabled configuration entries remain available in the edge's
|
|
local DeviceManager snapshot but are not transmitted. An enabled entry that
|
|
fails creation, initialization or start remains visible with
|
|
`MANAGED_DEVICE_STATE_ERROR`, `has_error=true` and an error detail. Its
|
|
independent health value may still be `UNSPECIFIED` when no trustworthy device
|
|
probe exists. `kind` is the stable category used for machine decisions, while
|
|
`type_name` is a concrete implementation name when the object exists and
|
|
otherwise a category label; it is intended only for display and diagnostics.
|
|
Senders order entries by `device_id` to make
|
|
snapshots deterministic. `sampled_at_unix_ms` is the snapshot time, whereas
|
|
`status_updated_at_unix_ms` records when DeviceManager last changed that row's
|
|
lifecycle/error record. The enclosing snapshot time is the freshness timestamp
|
|
for the health observation.
|
|
|
|
Lifecycle and health are deliberately separate. In particular,
|
|
`DEVICE_HEALTH_STATUS_UNSPECIFIED` means that no trustworthy health observation
|
|
was available; it is not equivalent to `DEVICE_HEALTH_STATUS_HEALTHY`.
|
|
Similarly, `has_error=false` only means that no error is currently confirmed
|
|
and must not be used to turn unknown health into healthy health. A health probe
|
|
failure must degrade that row to an unknown or fault result without suppressing
|
|
the rest of the heartbeat.
|
|
|
|
The edge caps each diagnostic string at 512 bytes without splitting a UTF-8
|
|
code point. Device identifiers, implementation names and manager metadata are
|
|
not silently truncated because doing so would change identity. The deployment
|
|
must therefore size `maximum_control_frame_bytes` for its enabled inventory;
|
|
the sender and receiver both reject an oversized control frame. Very large
|
|
inventories require a future explicit pagination/truncation extension rather
|
|
than silently dropping rows from this enabled-device snapshot.
|
|
|
|
The locally configured `QuicEdgeConfig.heartbeat_interval_ms` controls the
|
|
reporting interval before registration. A gateway may negotiate a different
|
|
interval through `NodeRegisterResponse.heartbeat_interval_ms`: zero keeps the
|
|
locally configured value, while a non-zero value overrides it for the current
|
|
registered QUIC connection. The edge clamps the negotiated value to its
|
|
supported safety range and returns to the local value on reconnect until a new
|
|
registration response is accepted.
|
|
|
|
`NodeDescriptor.robot_id = 6`, `NodeHeartbeat.device_manager = 9` and
|
|
`NodeHeartbeat.robot_id = 10` are additive protobuf fields, so these extensions
|
|
remain QUIC edge protocol v1. Existing gateways ignore unknown fields. Updated
|
|
gateways must continue accepting older v1 messages where these fields are absent
|
|
and must not treat an absent device snapshot as an empty, healthy DeviceManager.
|
|
Enum values may only be appended; existing numeric meanings must never be
|
|
renumbered or reused.
|
|
|
|
DATAGRAM negotiation is required only when at least one media track is enabled.
|
|
The reliable registration and heartbeat path remains valid for a zero-track
|
|
node or while all media sources are unavailable.
|
|
|
|
Media source registration, device start/keyframe callbacks and DATAGRAM
|
|
packetization run on a dedicated media worker. Reliable registration and
|
|
heartbeat therefore continue while a media source is slow or unavailable;
|
|
the reliable send path serializes heartbeat and media metadata so envelope
|
|
sequence order remains strict.
|
|
|
|
`MediaSourceHub` is protocol-neutral and gives each adapter an independent,
|
|
single-consumer subscription cursor. Its source-start callback receives a
|
|
cancellation predicate and must check it around potentially blocking device
|
|
startup. QUIC reconnect/stop and gRPC client cancellation can therefore abandon
|
|
startup without blocking presence or process shutdown. Ring overflow keeps the
|
|
newest bounded set of immutable frames, advances a slow cursor to the oldest
|
|
retained frame, and reports the exact dropped count so protocol adapters can
|
|
mark a discontinuity and request a new video keyframe.
|
|
|
|
## Media DATAGRAM wire format
|
|
|
|
Every media DATAGRAM starts with this fixed 64-byte, network-byte-order header:
|
|
|
|
| Offset | Size | Field |
|
|
|---:|---:|---|
|
|
| 0 | 4 | magic `CMQD` (`0x434d5144`) |
|
|
| 4 | 1 | protocol version (`1`) |
|
|
| 5 | 1 | media kind (`1` video, `2` audio) |
|
|
| 6 | 2 | flags |
|
|
| 8 | 2 | header size (`64`) |
|
|
| 10 | 2 | fragment index |
|
|
| 12 | 2 | fragment count |
|
|
| 14 | 2 | payload size |
|
|
| 16 | 4 | track ID |
|
|
| 20 | 4 | codec generation token |
|
|
| 24 | 8 | media session epoch |
|
|
| 32 | 8 | DATAGRAM packet sequence |
|
|
| 40 | 8 | media frame sequence |
|
|
| 48 | 8 | monotonic capture timestamp in microseconds |
|
|
| 56 | 4 | complete frame size |
|
|
| 60 | 4 | fragment byte offset in the complete frame |
|
|
|
|
Flag bit 0 marks a video keyframe, bit 1 is reserved and must currently be
|
|
zero, and bit 2 marks a discontinuity. Codec initialization bytes are carried
|
|
reliably in `MediaTrackDescriptor`. QUIC already authenticates each DATAGRAM,
|
|
so the application header has no redundant checksum.
|
|
|
|
Receivers must key reassembly by `(session_epoch, track_id, frame_sequence)`,
|
|
drop incomplete frames at their playback deadline, and reject fragments whose
|
|
offset plus payload exceeds `frame_size`. A new session epoch invalidates all
|
|
fragments retained from a previous connection. `frame_sequence` is generated
|
|
by the QUIC edge service and remains strictly increasing per track throughout
|
|
the media epoch, including after a device source restarts.
|