161 lines
7.6 KiB
Markdown
161 lines
7.6 KiB
Markdown
|
|
# QUIC edge v1 wire format
|
||
|
|
|
||
|
|
## Scope and topology
|
||
|
|
|
||
|
|
QUIC edge v1 carries two functions over one authenticated QUIC connection:
|
||
|
|
|
||
|
|
- a reliable bidirectional control stream for node registration, heartbeat,
|
||
|
|
current local IP addresses, the advertised gRPC endpoint and media metadata;
|
||
|
|
- QUIC DATAGRAM payloads for lossy, real-time audio and video.
|
||
|
|
|
||
|
|
Robot arm, AGV and other device-control APIs remain on the existing cmvr-es
|
||
|
|
gRPC server. The QUIC node descriptor advertises that server's current endpoint;
|
||
|
|
it does not move robot-control RPCs onto QUIC.
|
||
|
|
|
||
|
|
```text
|
||
|
|
cmvr-es -- custom QUIC edge v1 --> Java receiver / edge gateway
|
||
|
|
|
|
||
|
|
+-- WebTransport/HTTP3 --> browser
|
||
|
|
```
|
||
|
|
|
||
|
|
A browser cannot connect to this custom QUIC ALPN directly. The gateway must
|
||
|
|
terminate v1, register the node, reassemble media fragments and expose
|
||
|
|
WebTransport (or another browser-supported media API). Browser sessions do not
|
||
|
|
participate in edge-node registration or heartbeat.
|
||
|
|
|
||
|
|
## Build and host-safe defaults
|
||
|
|
|
||
|
|
MsQuic is optional at configure time. It can be installed under
|
||
|
|
`dependency/<arch>/third_party/msquic/<version>` or supplied explicitly:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cmake -S . -B build \
|
||
|
|
-DCMVR_ENABLE_MSQUIC_BACKEND=ON \
|
||
|
|
-DCMVR_MSQUIC_ROOT=/absolute/path/to/msquic
|
||
|
|
cmake --build build -j2
|
||
|
|
cmake --install build
|
||
|
|
```
|
||
|
|
|
||
|
|
Without MsQuic the project still builds, but enabling the QUIC task fails with
|
||
|
|
an explicit error. The repository default keeps `quic_edge` disabled. The
|
||
|
|
existing inbound gRPC server remains enabled independently.
|
||
|
|
|
||
|
|
To enable node presence without media:
|
||
|
|
|
||
|
|
1. Configure the gateway address, ALPN `cmvr-quic-edge/1`, TLS trust, node ID,
|
||
|
|
advertised gRPC endpoint and heartbeat values in
|
||
|
|
`cmvr-es/config/tasks/quic_edge_task/quic_edge_task.pb.txt`.
|
||
|
|
Certificate paths are relative to the cmvr-es configuration root, for
|
||
|
|
example `certs/quic_gateway_ca.pem`.
|
||
|
|
2. Leave `tracks` empty, set `quic_edge.enable: true`, and enable the
|
||
|
|
`quic_edge` TaskManager entry.
|
||
|
|
|
||
|
|
To add a media uplink, also enable the camera/microphone in DeviceManager and
|
||
|
|
add a matching `tracks` entry. A zero-track configuration is valid: unavailable
|
||
|
|
media must not suppress node registration, IP reporting or heartbeat.
|
||
|
|
`maximum_frame_bytes` must fit in one atomically admitted DATAGRAM batch; reduce
|
||
|
|
it or increase `datagram_send_queue_depth` when changing the DATAGRAM size.
|
||
|
|
|
||
|
|
## Reliable control stream
|
||
|
|
|
||
|
|
The edge opens one bidirectional reliable stream after the QUIC handshake. Both
|
||
|
|
directions carry a sequence of length-prefixed protobuf messages:
|
||
|
|
|
||
|
|
```text
|
||
|
|
uint32_be protobuf_length
|
||
|
|
protobuf EdgeControlEnvelope
|
||
|
|
```
|
||
|
|
|
||
|
|
The length excludes the four-byte prefix. Each receiver must reject a frame
|
||
|
|
larger than its configured control-frame limit. `protocol_version` is `1`.
|
||
|
|
`message_sequence` increases independently in each direction; gaps are allowed,
|
||
|
|
but duplicates and reordering are rejected. The edge resets its outbound
|
||
|
|
control sequence on every connection, so its first `NodeRegisterRequest`
|
||
|
|
currently has `message_sequence=0`; gateways must not assume the sequence starts
|
||
|
|
at 1. Heartbeat uses a separate sequence so its acknowledgement remains
|
||
|
|
explicit.
|
||
|
|
|
||
|
|
The legal session order is:
|
||
|
|
|
||
|
|
1. The edge sends `NodeRegisterRequest` as its first application message after
|
||
|
|
every QUIC connect or reconnect. It includes `node_id`, `boot_id`, software
|
||
|
|
version, the current IPv4/IPv6 interface snapshot and advertised gRPC endpoint.
|
||
|
|
2. The gateway replies with `NodeRegisterResponse`. Media and heartbeat must not
|
||
|
|
start until `accepted=true` and a non-empty `session_id` are received.
|
||
|
|
3. The edge sends `NodeHeartbeat` at the negotiated interval. The gateway returns
|
||
|
|
`NodeHeartbeatAck` with the same `session_id` and exact
|
||
|
|
`acknowledged_sequence`. A missing, rejected or mismatched ACK causes the edge
|
||
|
|
to reconnect and register again after its configured backoff.
|
||
|
|
4. If media tracks are available, the edge sends `MediaSessionOpen`, binding its
|
||
|
|
`session_epoch` to the accepted `session_id`, followed by track descriptors.
|
||
|
|
The edge calls reliable descriptor send before DATAGRAM send for a new
|
||
|
|
generation. QUIC does not guarantee arrival order across a reliable stream
|
||
|
|
and DATAGRAMs, so a gateway must boundedly buffer or drop media whose session
|
||
|
|
or generation metadata has not arrived yet.
|
||
|
|
5. The gateway may send `ProtocolError`; a fatal error closes the connection.
|
||
|
|
The current edge receive path otherwise accepts only `NodeRegisterResponse`
|
||
|
|
and `NodeHeartbeatAck`. Although `MediaSessionClose` is present in the v1
|
||
|
|
schema, the current edge neither sends it nor accepts it inbound. Gateways
|
||
|
|
must not depend on that message until both sides implement and test it.
|
||
|
|
|
||
|
|
The edge sends a fresh local-interface snapshot in every heartbeat, so an IP
|
||
|
|
change is reported without a second protocol. These addresses are edge claims.
|
||
|
|
`observed_source_ip` is authoritative for the public/NAT-facing address and must
|
||
|
|
be derived by the gateway from the authenticated QUIC peer, never copied from a
|
||
|
|
client field. `heartbeat_interval_ms=0` in the registration response means the
|
||
|
|
edge retains its configured interval.
|
||
|
|
|
||
|
|
DATAGRAM negotiation is required only when at least one media track is enabled.
|
||
|
|
The reliable registration and heartbeat path remains valid for a zero-track
|
||
|
|
node or while all media sources are unavailable.
|
||
|
|
|
||
|
|
Media source registration, device start/keyframe callbacks and DATAGRAM
|
||
|
|
packetization run on a dedicated media worker. Reliable registration and
|
||
|
|
heartbeat therefore continue while a media source is slow or unavailable;
|
||
|
|
the reliable send path serializes heartbeat and media metadata so envelope
|
||
|
|
sequence order remains strict.
|
||
|
|
|
||
|
|
`MediaSourceHub` is protocol-neutral and gives each adapter an independent,
|
||
|
|
single-consumer subscription cursor. Its source-start callback receives a
|
||
|
|
cancellation predicate and must check it around potentially blocking device
|
||
|
|
startup. QUIC reconnect/stop and gRPC client cancellation can therefore abandon
|
||
|
|
startup without blocking presence or process shutdown. Ring overflow keeps the
|
||
|
|
newest bounded set of immutable frames, advances a slow cursor to the oldest
|
||
|
|
retained frame, and reports the exact dropped count so protocol adapters can
|
||
|
|
mark a discontinuity and request a new video keyframe.
|
||
|
|
|
||
|
|
## Media DATAGRAM wire format
|
||
|
|
|
||
|
|
Every media DATAGRAM starts with this fixed 64-byte, network-byte-order header:
|
||
|
|
|
||
|
|
| Offset | Size | Field |
|
||
|
|
|---:|---:|---|
|
||
|
|
| 0 | 4 | magic `CMQD` (`0x434d5144`) |
|
||
|
|
| 4 | 1 | protocol version (`1`) |
|
||
|
|
| 5 | 1 | media kind (`1` video, `2` audio) |
|
||
|
|
| 6 | 2 | flags |
|
||
|
|
| 8 | 2 | header size (`64`) |
|
||
|
|
| 10 | 2 | fragment index |
|
||
|
|
| 12 | 2 | fragment count |
|
||
|
|
| 14 | 2 | payload size |
|
||
|
|
| 16 | 4 | track ID |
|
||
|
|
| 20 | 4 | codec generation token |
|
||
|
|
| 24 | 8 | media session epoch |
|
||
|
|
| 32 | 8 | DATAGRAM packet sequence |
|
||
|
|
| 40 | 8 | media frame sequence |
|
||
|
|
| 48 | 8 | monotonic capture timestamp in microseconds |
|
||
|
|
| 56 | 4 | complete frame size |
|
||
|
|
| 60 | 4 | fragment byte offset in the complete frame |
|
||
|
|
|
||
|
|
Flag bit 0 marks a video keyframe, bit 1 is reserved and must currently be
|
||
|
|
zero, and bit 2 marks a discontinuity. Codec initialization bytes are carried
|
||
|
|
reliably in `MediaTrackDescriptor`. QUIC already authenticates each DATAGRAM,
|
||
|
|
so the application header has no redundant checksum.
|
||
|
|
|
||
|
|
Receivers must key reassembly by `(session_epoch, track_id, frame_sequence)`,
|
||
|
|
drop incomplete frames at their playback deadline, and reject fragments whose
|
||
|
|
offset plus payload exceeds `frame_size`. A new session epoch invalidates all
|
||
|
|
fragments retained from a previous connection. `frame_sequence` is generated
|
||
|
|
by the QUIC edge service and remains strictly increasing per track throughout
|
||
|
|
the media epoch, including after a device source restarts.
|