feat(webrtc): negotiate data channel message limits

[yexiyue]

Description

Add a connection-level WebRTC data-channel message limit and negotiate an effective limit after the existing Noise authentication handshake.

Both native libp2p-webrtc and browser libp2p-webrtc-websys expose with_max_message_size. The selected effective value is used consistently by the libp2p frame codec, write high-water mark, browser buffered-amount accounting, and the native WebRTC read buffer.

Fixes #6557.

Why

The WebRTC DataChannel stack has a message-size limit below the libp2p stream abstraction. The current fixed 16 KiB libp2p frame limit follows RFC 8831's guidance for SCTP without message interleaving, but deployments can have a smaller effective limit. In a browser-to-browser relay path, this led to an underlying SCTP implementation rejecting a frame and closing the data channel.

W3C WebRTC specifies that the maximum size accepted by RTCDataChannel.send() is derived from the remote max-message-size SDP attribute and local sending capability, using the smaller applicable value. See WebRTC §6.1.1.2 and the RTCSctpTransport.maxMessageSize definition. RFC 8831 also recommends a 16 KiB maximum when SCTP message interleaving is unavailable: RFC 8831 §6.7.

WebRTC Direct does not exchange arbitrary application SDP between libp2p peers: each endpoint synthesizes the remote SDP needed to establish the direct connection. Consequently, putting a local configuration value in that SDP would not negotiate the remote capability and could make the two framing layers disagree.

Design

1. The Noise handshake data channel continues to use the historical 16 KiB compatible limit.
2. Once both peers are authenticated, they exchange a fixed-width advertised limit on that protected channel.
3. New peers use min(local, remote) as the connection's effective limit.
4. A peer that closes the channel immediately after Noise is treated as a legacy peer and falls back to the historical 16 KiB limit.
5. SDP remains protocol-fixed; the negotiated value is only carried by StreamConfig into the framing and transport implementations.

This avoids a global conservative limit while preserving compatibility with existing WebRTC Direct peers. The capability value is exchanged only after Noise has authenticated the peer and bound the DTLS fingerprints into its prologue.

Validation

• cargo test -p libp2p-webrtc-utils — 26 tests, including smaller-limit selection and legacy/invalid-peer fallback.
• cargo test -p libp2p-webrtc --features tokio --test smoke smoke — real two-node UDP WebRTC Direct connection.
• cargo check -p libp2p-webrtc --features tokio
• cargo check -p libp2p-webrtc-websys --target wasm32-unknown-unknown
• git diff --check

The existing multi-node WebRTC smoke test was also run. It completes successfully, but its detached background tasks log cleanup-time NoListeners / SendError(Disconnected) panics, so I do not present it as a clean regression signal in this PR.

AI Assistance Disclosure

Tools used (required — write none if no AI was used): Codex 5.6

Attestation (required):

• I have read every line of this diff, understand what it does, and can explain it in review.

Notes & open questions

The post-Noise exchange deliberately uses the existing reserved handshake data channel rather than introducing a new SDP extension or a second negotiated data channel. It adds no new public wire format to application substreams.

Change checklist

• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• A changelog entry has been made in the appropriate crates

[yexiyue]

Follow-up from browser-to-browser relay transfer testing: the negotiated max_message_size must not also cap the browser callback's aggregate receive queue.

With an 8 KiB negotiated frame limit, multiple individually valid RTCDataChannel.onmessage events can be delivered before the deferred Rust waker is polled. Using that 8 KiB value as the aggregate read_buffer limit incorrectly reports Remote is overloading us with messages and resets the stream, even though no individual message exceeds the negotiated limit.

This update adds webrtc_websys::Config::with_max_read_buffer_size(NonZeroUsize). It is deliberately local to the browser transport and is not negotiated. The default is 256 KiB; the effective value is max(configured_read_buffer_size, negotiated_message_size), so one valid message always fits while the queue remains bounded.

I added focused tests for consecutive valid 8 KiB messages, the configured bound, and a negotiated message larger than the default. Validation also passed with:

• cargo test -p libp2p-webrtc-websys
• cargo check -p libp2p-webrtc-websys --target wasm32-unknown-unknown
• browser-to-browser WebRTC relay transfer in SwarmDrop after rebuilding the wasm package

This keeps the message-size API and the local resource-limit API separate, rather than making an application-layer workaround (such as skipping flush) responsible for a transport-level queueing policy.