Wire Protocol

All peer-to-peer communication in mosaik flows over QUIC streams, using a consistent framing layer built on Link<P>. This chapter explains the protocol stack from the transport up to application messages.

Protocol stack

┌─────────────────────────────────┐
│  Application Messages           │  BondMessage, Datum, CatalogSync, ...
├─────────────────────────────────┤
│  postcard serialization         │  compact binary encoding (no_std, varint)
├─────────────────────────────────┤
│  LengthDelimitedCodec framing   │  4-byte big-endian length prefix
├─────────────────────────────────┤
│  QUIC bidirectional stream      │  SendStream + RecvStream
├─────────────────────────────────┤
│  iroh / quinn transport         │  QUIC with TLS 1.3, hole-punching
└─────────────────────────────────┘

Link<P>

The Link<P> type is the core abstraction for typed, bidirectional communication over a QUIC stream:

struct Link<P: Protocol> {
    connection: Connection,
    cancel: CancellationToken,
    writer: FramedWrite<SendStream, LengthDelimitedCodec>,
    reader: FramedRead<RecvStream, LengthDelimitedCodec>,
}

The Protocol trait

Every protocol declares its ALPN (Application-Layer Protocol Negotiation) identifier:

trait Protocol: Serialize + DeserializeOwned + Send + 'static {
    const ALPN: &'static [u8];
}

ALPN identifiers are exchanged during the TLS handshake, so peers agree on the protocol before any application data flows. Mosaik uses these ALPNs:

Creating a link

Links are created by either connecting to a peer or accepting an incoming connection:

// Outgoing
let link = Link::<BondMessage>::connect(&endpoint, node_addr).await?;

// Incoming (in a ProtocolHandler)
let link = Link::<BondMessage>::accept(connecting).await?;

Sending and receiving

// Send a message (serialized with postcard, length-prefixed)
link.send(BondMessage::Ping).await?;

// Receive a message (read length prefix, deserialize with postcard)
let msg: BondMessage = link.recv().await?;

Under the hood:

1. send() serializes the message with postcard::to_allocvec().
2. The resulting bytes are written through FramedWrite which prepends a 4-byte big-endian length prefix.
3. recv() reads the length prefix from FramedRead, reads exactly that many bytes, and deserializes with postcard::from_bytes().

Splitting a link

For concurrent send/receive, a link can be split:

let (writer, reader) = link.split();

// In one task:
writer.send(msg).await?;

// In another task:
let msg = reader.recv().await?;

// Rejoin if needed:
let link = Link::join(writer, reader);

Cancellation

Every link carries a CancellationToken. When cancelled, both send and receive operations return immediately. This is used for graceful shutdown:

link.cancel(); // signals both sides to stop

Wire format

postcard encoding

Postcard is a #[no_std]-compatible binary serialization format based on variable-length integers (varints). It produces very compact output:

• u8 → 1 byte
• Small u32 → 1 byte (varint)
• Enum variant → 1 byte discriminant + payload
• Strings → varint length + UTF-8 bytes
• Vec<T> → varint length + elements

This keeps message sizes minimal, which matters for high-frequency heartbeats and Raft messages.

Framing

Each message on the wire looks like:

┌──────────────┬───────────────────────────────┐
│ Length (4B)   │ postcard-encoded payload       │
│ big-endian    │                               │
└──────────────┴───────────────────────────────┘

The LengthDelimitedCodec from the tokio-util crate handles this automatically. It supports messages up to 2³² - 1 bytes (4 GiB), though in practice mosaik messages are typically under a few kilobytes.

QUIC transport

Mosaik uses iroh (built on quinn) for QUIC transport. Key features:

Bidirectional streams

Each Link<P> uses a single QUIC bidirectional stream. This means:

• One stream per bond connection.
• One stream per catalog sync session.
• One stream per producer-consumer pair.

QUIC’s multiplexing means these streams don’t interfere with each other even when sharing the same underlying UDP connection.

Error handling

Link operations return std::io::Error. Common failure modes: