Jorijn van der Graaf
9ea07c54f1
~ClientQUIC shut the connection down first and drained its streams after. msquic aborts every stream with the connection, and with them whatever it had not sent yet, so a server that replied with SendAsync(..., finish = true), let the stream go and deleted the connection lost the reply whenever the shutdown, queued right behind the send, reached the worker before the flush that would have sent it. The interface promised the opposite: that the destructor waits for every stream "to finish closing first". Found with Crafter.Debug, whose session answers every request still waiting when it ends and then deletes its connections: the follower it told "shutting down" missed the reply in 4 runs of 30. Streams now count as finishing from QUICStream::Stop to their SHUTDOWN_COMPLETE, in the registry beside the open count, and the destructor waits for that count to reach zero before the shutdown. Only streams that can send are counted, since a stream that cannot has nothing on its way. A stopped stream finishes once the peer has acknowledged its FIN, about a round trip; the wait is bounded at 1 s so a peer that stopped answering, or one that never reads, holds the destructor no longer than that, and a peer closing meanwhile aborts the streams, which ends the wait too. Streams their owner still holds are not waited for: they are aborted as before. Measured with ShouldFinishStoppedStreamsQUIC: before, 0 of 20 replies of 64 KiB arrived; after, 20 of 20, with the slowest delete at 1 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
393 lines
21 KiB
C++
393 lines
21 KiB
C++
//SPDX-License-Identifier: LGPL-3.0-only
|
|
//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
|
|
|
|
module;
|
|
#ifndef CRAFTER_NETWORK_BROWSER
|
|
#include <msquic.h>
|
|
#endif
|
|
export module Crafter.Network:ClientQUIC;
|
|
import std;
|
|
|
|
namespace Crafter {
|
|
export class ListenerQUIC;
|
|
|
|
export class QUICException : public std::runtime_error {
|
|
public:
|
|
using std::runtime_error::runtime_error;
|
|
};
|
|
|
|
export class QUICClosedException : public std::exception {
|
|
public:
|
|
const char* what() const noexcept override { return "QUIC connection closed"; }
|
|
};
|
|
|
|
// Transport knobs applied to the msquic Configuration. The defaults
|
|
// reproduce the values these were hardcoded to before they were made
|
|
// configurable, so leaving this alone keeps the previous behaviour.
|
|
//
|
|
// peerUnidiStreamCount / peerBidiStreamCount cap how many streams the
|
|
// *peer* may open toward us. msquic defaults both to 0; a server's
|
|
// HTTP/3 control + QPACK streams alone need 3, and a protocol that
|
|
// fans data out over many concurrent streams needs correspondingly
|
|
// more. Raising these costs per-stream flow-control state, so raise
|
|
// them to what a protocol actually needs rather than to a large
|
|
// round number.
|
|
//
|
|
// keepAliveIntervalMs is client-only (the listener does not set it).
|
|
// Keep it below idleTimeoutMs or the peer's idle timer will expire.
|
|
// handshakeIdleTimeoutMs bounds how long a connect attempt to an
|
|
// unreachable peer takes to fail. This one is set explicitly where it
|
|
// previously was not: ClientQUIC's constructor waits on a condition
|
|
// variable with no timeout of its own, so without a bound here a client
|
|
// dialling a dead port blocks for minutes rather than failing and
|
|
// retrying. 10s matches msquic's own documented default.
|
|
export struct QUICSettings {
|
|
std::uint32_t idleTimeoutMs = 120'000;
|
|
std::uint32_t handshakeIdleTimeoutMs = 10'000;
|
|
std::uint32_t keepAliveIntervalMs = 10'000;
|
|
std::uint16_t peerUnidiStreamCount = 16;
|
|
std::uint16_t peerBidiStreamCount = 16;
|
|
bool datagramReceiveEnabled = true;
|
|
};
|
|
|
|
// Server certificate sources. Pick one of (filePaths) or (selfSigned).
|
|
// selfSigned serves a development certificate made in-process and kept
|
|
// per user under $XDG_CACHE_HOME/crafter-network/quic-cert/, reused
|
|
// while valid (see GetSelfSignedCertificatePath) — fine for dev/LAN.
|
|
export struct QUICServerCredentials {
|
|
std::string certPath;
|
|
std::string keyPath;
|
|
bool selfSigned = false;
|
|
QUICSettings settings;
|
|
};
|
|
|
|
// Client-side credential validation. By default we require a real cert.
|
|
// insecureNoServerValidation disables peer cert checks — only for dev,
|
|
// and silently ignored in the browser build (browsers enforce their own
|
|
// certificate policy). For browser dev against a self-signed listener,
|
|
// populate serverCertificateHash with the SHA-256 of the server's DER
|
|
// certificate; on the browser it is forwarded to WebTransport's
|
|
// serverCertificateHashes option. A zeroed array means "unused" — the
|
|
// browser will then require a publicly trusted cert.
|
|
export struct QUICClientCredentials {
|
|
bool insecureNoServerValidation = false;
|
|
std::array<std::uint8_t, 32> serverCertificateHash{};
|
|
QUICSettings settings;
|
|
};
|
|
|
|
export class ClientQUIC;
|
|
|
|
// A reliable, ordered stream within a QUIC connection. May be
|
|
// bidirectional or unidirectional; for unidi streams either canSend or
|
|
// canReceive will be false depending on which side initiated. Owned by
|
|
// ClientQUIC; do not destroy directly. Obtain via ClientQUIC::OpenStream
|
|
// (optionally with unidirectional=true) or via the on-stream callback
|
|
// for inbound streams initiated by the peer.
|
|
export class QUICStream {
|
|
public:
|
|
#ifndef CRAFTER_NETWORK_BROWSER
|
|
// Underlying msquic HQUIC handle. Treated as opaque by callers.
|
|
HQUIC handle = nullptr;
|
|
#endif
|
|
|
|
// The connection that owns this stream (non-owning).
|
|
ClientQUIC* connection = nullptr;
|
|
|
|
// Direction flags. Bidi streams have both true; outgoing unidi sets
|
|
// canReceive=false; incoming unidi (peer-initiated) sets canSend=false.
|
|
bool canSend = true;
|
|
bool canReceive = true;
|
|
|
|
QUICStream();
|
|
#ifndef CRAFTER_NETWORK_BROWSER
|
|
QUICStream(HQUIC handle, ClientQUIC* connection);
|
|
#else
|
|
// Browser-only constructor: wraps a JS-side WebTransport stream
|
|
// identified by its integer handle. Used by ClientQUIC::OpenStream
|
|
// and by the incoming-stream dispatcher in the JS bridge — not
|
|
// intended for direct use.
|
|
QUICStream(std::int32_t handle, ClientQUIC* connection,
|
|
bool canSend, bool canReceive);
|
|
#endif
|
|
~QUICStream();
|
|
QUICStream(const QUICStream&) = delete;
|
|
QUICStream(QUICStream&&) noexcept;
|
|
QUICStream& operator=(QUICStream&&) noexcept;
|
|
|
|
#ifndef CRAFTER_NETWORK_BROWSER
|
|
// Send a buffer. If finish=true, the send-side of the stream is closed
|
|
// after the buffer is delivered (peer will see graceful shutdown).
|
|
// Blocks until msquic accepts the buffer; throws QUICClosedException
|
|
// when the stream or connection closed before it did. SendAsync is
|
|
// the same without the thread.
|
|
void SendSync(const void* buffer, std::uint32_t size, bool finish = false);
|
|
|
|
// Block until at least one byte is received; returns the received
|
|
// chunk. Throws QUICClosedException once the peer has closed the
|
|
// send-side and the buffer is drained.
|
|
std::vector<char> RecieveSync();
|
|
|
|
// Read until the peer closes the send-side, accumulating all chunks.
|
|
std::vector<char> RecieveUntilCloseSync();
|
|
|
|
// Read exactly bufferSize bytes. Throws if the peer closes early,
|
|
// leaving what did arrive for the next receive.
|
|
std::vector<char> RecieveUntilFullSync(std::uint32_t bufferSize);
|
|
#endif
|
|
|
|
// Send a buffer. If finish=true, the send-side is closed after the
|
|
// buffer is delivered. onSent fires with true once the transport has
|
|
// accepted the buffer (native) or the WritableStream writer has
|
|
// resolved (browser), and with false when the stream was closed or
|
|
// the send failed. Available on both native and browser builds.
|
|
//
|
|
// Native: holds no thread. It copies the buffer, hands the copy to
|
|
// msquic and returns; msquic's SEND_COMPLETE for it frees the copy and
|
|
// fires onSent, exactly once: true when msquic took the whole buffer
|
|
// (it is in msquic's send buffer or acknowledged, what SendSync's
|
|
// return means; not a receipt from the peer's application), false
|
|
// when the stream or the connection closed first. Callable from any
|
|
// thread, inside this stream's own callbacks included, and as often
|
|
// as you like: the sends queue in msquic in call order, none waits for
|
|
// another, and BytesInFlight says how far behind they are.
|
|
// - onSent runs on the connection's msquic worker thread, never at
|
|
// the same time as another callback of this stream (see
|
|
// OnRecieve). A send that fails before msquic takes it (a closed
|
|
// stream, a moved-from wrapper) reports from a pool task, or
|
|
// after the callback that made it returns, never from inside
|
|
// SendAsync, so the caller may hold a lock onSent takes.
|
|
// - In onSent: SendAsync and the other non-blocking calls are fine;
|
|
// nothing that blocks, no *Sync call (it waits for this very
|
|
// worker), no ~ClientQUIC; it must not throw.
|
|
// - It fires even after the wrapper is gone: destroying it lets the
|
|
// sends already made go out, then the FIN.
|
|
void SendAsync(const void* buffer, std::uint32_t size, bool finish,
|
|
std::function<void(bool sent)> onSent);
|
|
|
|
// Async receive variants. Dispatched on Crafter.Thread's ThreadPool
|
|
// (native) or driven by a per-stream JS reader loop (browser). A
|
|
// stream that has closed gives the callback an empty chunk, and
|
|
// RecieveUntilFullAsync on one that closes early gives what arrived:
|
|
// the browser build has no exceptions to report it with, and on the
|
|
// ThreadPool an exception would end the program.
|
|
void RecieveAsync(std::function<void(std::vector<char>)> recieveCallback);
|
|
void RecieveUntilCloseAsync(std::function<void(std::vector<char>)> recieveCallback);
|
|
void RecieveUntilFullAsync(std::uint32_t bufferSize, std::function<void(std::vector<char>)> recieveCallback);
|
|
|
|
#ifndef CRAFTER_NETWORK_BROWSER
|
|
// Advanced: re-inject already-consumed bytes at the front of the
|
|
// receive queue so the next Recieve* call sees them. Used by
|
|
// protocol demuxers (e.g. the WebTransport stream router in
|
|
// ListenerHTTP) that need to peek a prefix off the wire, then hand
|
|
// the stream to user code as if the prefix had never been read.
|
|
void PrependReceived(std::vector<char> bytes);
|
|
|
|
// Underlying QUIC stream id. Stable for the stream's lifetime.
|
|
// Browsers identify WT streams by the session's CONNECT stream id,
|
|
// so the server has to query and remember it at session creation
|
|
// time. Throws if the stream is not yet started.
|
|
std::uint64_t GetStreamId() const;
|
|
|
|
// Bytes, and sends, handed to SendAsync or SendSync whose completion
|
|
// has not come yet: how far behind a slow peer is. A server streaming
|
|
// to a follower can drop one whose backlog passes a cap. Any thread.
|
|
std::uint64_t BytesInFlight() const;
|
|
std::size_t SendsInFlight() const;
|
|
|
|
// Receive by callback, holding no thread while nothing arrives.
|
|
// onData gets each run of bytes as it arrives, as a span valid only
|
|
// for the call (copy what you keep); onEnd fires once, after the last
|
|
// onData: clean=true when the peer finished its side (FIN), false
|
|
// when either side aborted the stream (Abort(), the peer's reset) or
|
|
// the connection went first.
|
|
// - What arrived before the call is delivered first, on the calling
|
|
// thread before OnRecieve returns, and so is onEnd if the stream
|
|
// had already ended; from then on the callbacks run on the
|
|
// connection's msquic worker thread. No two callbacks of one
|
|
// stream (onData, onEnd, SendAsync's onSent) ever run at once.
|
|
// - Inside them: SendAsync, Abort, BytesInFlight, SendsInFlight,
|
|
// and destroying or moving this wrapper are fine. Nothing that
|
|
// blocks: no *Sync call on any stream (it waits for this very
|
|
// worker), no ~ClientQUIC (it waits for its streams to drain), no
|
|
// waiting on a thread that may be waiting on this one. A callback
|
|
// that throws ends the program.
|
|
// - Destroying the wrapper, or Stop(), unregisters both: once that
|
|
// returns neither runs again, a callback running on another
|
|
// thread has finished (as with std::stop_callback; one on this
|
|
// thread, destroying its own wrapper, is not waited for), and an
|
|
// onEnd not delivered by then never is. So do not destroy it
|
|
// while holding a lock its callbacks take. Sends still complete.
|
|
// - Once per stream: a second call throws QUICException. Do not mix
|
|
// with the Recieve* calls, which then get nothing. On a stream
|
|
// that cannot receive, or a closed or moved-from wrapper, onEnd
|
|
// (false) fires at once.
|
|
void OnRecieve(std::function<void(std::span<const char> data)> onData,
|
|
std::function<void(bool clean)> onEnd);
|
|
|
|
// Reset both directions at once with a QUIC application error code
|
|
// (62 bits; larger values are clamped): the peer's receive side sees
|
|
// a reset and its send side a stop-sending. Returns at once. Sends
|
|
// msquic has not taken yet report false (one already in its send
|
|
// buffer reported true, and the reset discards it); no onData
|
|
// follows (not even the rest of a run being delivered when called
|
|
// from inside one; from another thread, one already under way
|
|
// finishes); and onEnd(false) follows unless the end was delivered
|
|
// already or the wrapper is destroyed first. Callable from any thread
|
|
// and from inside this stream's callbacks, which is how a server
|
|
// enforces a byte cap. A no-op on a closed or moved-from wrapper.
|
|
void Abort(std::uint64_t errorCode = 0);
|
|
#endif
|
|
|
|
// Shut the stream down in both directions: our send side gracefully
|
|
// (the peer sees a FIN), the receive side abortively. Aborting the
|
|
// receive side matters — a bidirectional stream whose peer keeps its
|
|
// send side open otherwise never completes its shutdown, and so never
|
|
// releases its transport handle. Returns immediately; completion is
|
|
// asynchronous. Called by the destructor, so an explicit call is only
|
|
// needed to shut a stream down ahead of its wrapper going away.
|
|
// Natively it also unregisters OnRecieve's callbacks (see there);
|
|
// sends already made still go out, ahead of the FIN, and
|
|
// ~ClientQUIC waits (bounded) for them before closing the connection.
|
|
void Stop();
|
|
|
|
private:
|
|
struct Impl;
|
|
// shared_ptr (not unique_ptr): the msquic stream callback holds its own
|
|
// ref so it can never run on a freed Impl if the wrapper is destroyed
|
|
// before SHUTDOWN_COMPLETE fires. See Crafter.Network-ClientQUIC.cpp.
|
|
std::shared_ptr<Impl> impl;
|
|
friend class ClientQUIC;
|
|
};
|
|
|
|
// A QUIC connection. On the client side, constructing one initiates the
|
|
// handshake and (on native) blocks until it succeeds, or throws on
|
|
// failure. On the server side, ListenerQUIC instantiates these for
|
|
// accepted peers.
|
|
//
|
|
// A connection multiplexes:
|
|
// - Reliable, ordered streams (open via OpenStream() / observe inbound
|
|
// via OnStream()).
|
|
// - Unreliable, unordered datagrams (SendDatagram() / OnDatagram()).
|
|
//
|
|
// Lifetime: ~ClientQUIC first lets the streams already stopped finish
|
|
// (bounded — 1s, about a round trip when the peer answers): what was
|
|
// sent on them, the FIN last, reaches the peer, so a server may reply
|
|
// with SendAsync(..., finish = true), let the stream go and delete the
|
|
// connection at once. Then it shuts the connection down, which aborts
|
|
// any stream still open, and closes it, waiting (bounded — 5s) for every
|
|
// stream on it to finish closing first, since QUICStream::Stop only
|
|
// initiates a shutdown that completes asynchronously, and then (bounded
|
|
// — 1s, about a round trip when the peer answers) for the shutdown to
|
|
// complete, so the peer is sent the close rather than left to time the
|
|
// connection out.
|
|
// Streams outliving the connection are therefore safe to hold, though they
|
|
// are dead once it goes; prefer destroying or moving them out first.
|
|
//
|
|
// Browser build: the only QUIC-shaped API the browser exposes is
|
|
// WebTransport, which is HTTP/3-based and reached at a fixed URL. Here:
|
|
// - The constructor returns immediately; the connection is opened in
|
|
// the background. Operations issued before the connection is ready
|
|
// are queued JS-side until WebTransport's "ready" promise resolves
|
|
// (or fail with QUICClosedException if the connection rejects).
|
|
// - `alpn` is mapped to the URL path: new WebTransport(
|
|
// `https://${host}:${port}/${alpn}`). The QUIC-layer ALPN itself
|
|
// is fixed to "h3" by the browser and cannot be customised.
|
|
// - The server side must accept WebTransport sessions (HTTP/3 extended
|
|
// CONNECT) on the path equal to `alpn`. Plain QUIC with a custom
|
|
// ALPN — what ListenerQUIC offers today — is not reachable from a
|
|
// browser.
|
|
// - Synchronous send/receive methods are not compiled. Use the *Async
|
|
// variants instead.
|
|
export class ClientQUIC {
|
|
public:
|
|
// ALPN identifier exchanged in the handshake. Both peers must agree.
|
|
// For 3DForts use e.g. "f3d/1" or similar — a short stable token.
|
|
// On the browser build, this is the WebTransport URL path instead
|
|
// of an ALPN token; see the class comment above.
|
|
std::string alpn;
|
|
|
|
// Client constructor: connects to host:port using QUIC. ALPN must
|
|
// match the listener. Throws QUICException on connect failure.
|
|
// Blocks for at most the handshake idle timeout (10s), so a host that
|
|
// never answers fails in seconds rather than at the idle timeout.
|
|
ClientQUIC(const char* host, std::uint16_t port, std::string alpn,
|
|
QUICClientCredentials creds = {});
|
|
ClientQUIC(std::string host, std::uint16_t port, std::string alpn,
|
|
QUICClientCredentials creds = {});
|
|
|
|
#ifndef CRAFTER_NETWORK_BROWSER
|
|
// Server-side constructor used by ListenerQUIC for accepted peers.
|
|
// Takes ownership of an already-accepted msquic connection handle
|
|
// and the server configuration handle. Not intended for direct use.
|
|
ClientQUIC(HQUIC connectionHandle, HQUIC serverConfiguration, std::string alpn);
|
|
#endif
|
|
|
|
~ClientQUIC();
|
|
ClientQUIC(const ClientQUIC&) = delete;
|
|
ClientQUIC(ClientQUIC&&) noexcept;
|
|
|
|
// Open a new stream initiated by this side. Defaults to bidirectional;
|
|
// pass unidirectional=true to open a one-way send stream (used for
|
|
// HTTP/3's control + QPACK encoder/decoder streams).
|
|
// Blocks until the stream is started; throws on failure.
|
|
QUICStream OpenStream(bool unidirectional = false);
|
|
|
|
// Send a datagram. Best-effort: may be silently dropped under loss
|
|
// or congestion. Size must fit within the path MTU (msquic surfaces
|
|
// the maximum via QUIC_PARAM_CONN_DATAGRAM_SEND_ENABLED — typically
|
|
// ~1200 bytes safely on the open internet).
|
|
void SendDatagram(const void* buffer, std::uint32_t size);
|
|
|
|
// Register a handler for streams the peer initiates toward us.
|
|
// Called on the msquic worker; offload heavy work to ThreadPool.
|
|
void OnStream(std::function<void(QUICStream)> callback);
|
|
|
|
// Register a handler for datagrams from the peer. Called on the
|
|
// msquic worker; copy/queue and return promptly.
|
|
void OnDatagram(std::function<void(std::vector<char>)> callback);
|
|
|
|
#ifndef CRAFTER_NETWORK_BROWSER
|
|
// Block the caller until the next datagram arrives; returns it.
|
|
// Throws QUICClosedException if the connection closes first.
|
|
std::vector<char> RecieveDatagramSync();
|
|
|
|
// Told once when the connection has shut down for any reason other
|
|
// than this object being destroyed: the peer closed it, the idle
|
|
// timeout or a transport error ended it, or Stop() was called. This
|
|
// is how the owner of a connection a ListenerQUIC accepted learns
|
|
// that it is time to delete it.
|
|
//
|
|
// - It runs on the connection's msquic worker thread, once the
|
|
// connection's handle is closed; if the connection had already
|
|
// closed when OnClosed was called, it runs as a pool task
|
|
// instead, never inside OnClosed itself.
|
|
// - Deleting this ClientQUIC from inside it is safe, and the usual
|
|
// thing to do. Do nothing else that blocks: deleting a
|
|
// *different* ClientQUIC, or any *Sync call, waits on msquic
|
|
// workers, this one perhaps among them. It must not throw; an
|
|
// exception leaving it ends the program.
|
|
// - Once ~ClientQUIC has begun it never starts, and a run already
|
|
// under way on another thread finishes before ~ClientQUIC goes
|
|
// on, as with std::stop_callback (one on this thread, deleting
|
|
// its own connection, is not waited for). So do not destroy the
|
|
// ClientQUIC while holding a lock the callback takes.
|
|
// - Calling OnClosed again replaces the callback, and an empty one
|
|
// unregisters it; once it has fired, later calls do nothing.
|
|
void OnClosed(std::function<void()> callback);
|
|
#endif
|
|
|
|
// Cleanly shut down the connection.
|
|
void Stop();
|
|
|
|
#ifndef CRAFTER_NETWORK_BROWSER
|
|
// Underlying handle for advanced use (parameter queries, etc.).
|
|
HQUIC GetHandle() const;
|
|
#endif
|
|
|
|
private:
|
|
struct Impl;
|
|
std::unique_ptr<Impl> impl;
|
|
friend class ListenerQUIC;
|
|
friend class QUICStream;
|
|
};
|
|
}
|