Native SendAsync was SendSync run as a pool task. That held a worker of
Crafter.Thread's pool, which the host shares and which runs each
worker's queue in order, for as long as msquic took to accept the
buffer, and a peer that reads slowly stretches that to its own pace: a
server streaming to a slow follower stalled the tasks queued behind it.
The task also reached back through `this`, so moving or destroying the
wrapper before it ran was a use-after-free, and two sends could reach
msquic out of order from two workers.
Now SendAsync copies the buffer into a SendRequest, hands it to msquic
and returns. SEND_COMPLETE, which msquic delivers exactly once for every
send it took and before the stream's SHUTDOWN_COMPLETE, frees the copy
and calls onSent on the msquic worker: true when msquic took the whole
buffer (in its send buffer or acknowledged, which is what SendSync's
return has always meant), false when it was canceled because the stream
or the connection ended first. Sends queue in msquic in call order and
none waits for another. A send msquic never took (a closed stream, a
moved-from wrapper, StreamSend refusing it) reports false from a pool
task, never from inside SendAsync, so a caller may hold a lock its own
callback takes. Only `impl` is read, not `handle`, so SendAsync is
callable from any thread, a callback included; on a stopped stream
msquic cancels the send.
BytesInFlight() and SendsInFlight() count what SendAsync and SendSync
handed msquic whose SEND_COMPLETE has not come, so a server can drop a
follower that falls too far behind.
Rewritten rather than added beside it: nothing in this repository calls
the native SendAsync (ListenerHTTP, WebTransport and ClientHTTP use
SendSync; the example's SendAsync is the browser build's), nor does any
consumer found here (3DForts uses SendSync natively); its contract is
unchanged, copy now and onSent once with whether it went; and the old
one's pool thread, `this` capture and reordering were defects, not
behaviour to keep. What changes is onSent's thread, msquic's worker
rather than a pool task, so a callback that blocks, a *Sync call above
all, now stalls its connection rather than a pool worker. The browser
build is untouched.
Breaking: native onSent runs on msquic's worker thread, not as a pool
task, and must not block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A ListenerQUIC hands each accepted connection to its callback as a
heap-allocated ClientQUIC the callback owns, and nothing said when to
delete it: the SHUTDOWN_COMPLETE handler closed the msquic handle of a
connection the peer or the idle timeout ended, but the object stayed, so
a server either leaked its peers or deleted them blind, and one still
alive at exit holds msquic's registration open until its idle timeout.
OnClosed(callback) is told once when the connection has shut down for
any reason but this object's destruction: the peer closed it, the idle
timeout or a transport error ended it, or Stop() was called. It runs on
the msquic worker, in the SHUTDOWN_COMPLETE handler, after that handler
has claimed and closed the handle, so deleting the ClientQUIC from
inside it is safe: ~ClientQUIC then finds no handle to claim and waits
for nothing. Delivering it there, rather than as a pool task, keeps it
independent of a pool the host shares and possibly saturates; the cost
is the usual worker rule, which the interface spells out (no blocking,
no *Sync call, no deleting another ClientQUIC from inside it). A
callback set after the close, which happens when the peer goes before a
ListenerQUIC's connectCallback runs, fires as a pool task instead, never
from inside OnClosed, where the caller is still using the object.
Its state is a shared CloseNotice rather than part of the Impl, since
the handler fires it after giving up the handle, when ~ClientQUIC on
another thread no longer waits for the worker and may free the Impl.
The rules are std::stop_callback's: ~ClientQUIC marks the owner gone
first, after which the callback never starts; one already running on
another thread is waited for, so the owner may free what it uses as
soon as the destructor returns; one on the destructor's own thread, a
callback deleting its connection, is not. The handler takes the handle
and the notice under one lock, so the destructor cannot slip between.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SendSync waited on one sendInFlight flag per stream and, once woken,
threw QUICClosedException whenever the stream had finished shutting
down by then, even though its own send had completed. A peer that
answers at once and finishes the stream (echo, FIN, drop it, its Stop()
also stopping our sending side) completes the stream within a round
trip, which on loopback is shorter than a busy machine can take to wake
the sending thread, so SendSync threw for a send the peer had already
answered. The shared flag also let two SendSync calls on one stream wake
on each other's completion.
Each send now carries its own request: the copy msquic reads, the
QUIC_BUFFER describing it (msquic keeps a pointer to that, not a copy,
which the old code satisfied only because the waiter's stack outlived
the send), and an outcome shared with the waiter. SEND_COMPLETE, which
msquic delivers exactly once for every send it took and always before
the stream's SHUTDOWN_COMPLETE, records whether it was Canceled and
frees the request; SendSync returns unless its own send was canceled or
never completed.
Measured with ShouldReturnFromSendSyncQUIC, 2000 exchanges on fresh
connections four at a time: before, 1 or 2 spurious throws in 6 of 7
runs, the echo of each failed send arriving intact; after, none in 9
runs. It surfaced as one lost echo in a hundred in the OnClosed test
being written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
~ClientQUIC asked msquic to shut the connection down, waited for its
streams to drain, and closed the handle. With no stream open the drain
returns at once, so ConnectionClose followed ConnectionShutdown within
microseconds. msquic sends the CONNECTION_CLOSE frame from a FLUSH_SEND
operation it queues while it processes the shutdown; the close the
destructor had queued meanwhile reached the worker first, forced the
shutdown to complete silently (QuicConnTryClose: "silent close forced
after we already started the close process"), and the flush, which is
ignored once ShutdownComplete is set, sent nothing. The peer learned of
the close only at its idle timeout, two minutes by default: a server
whose clients each live for one request, such as a debug server driven
by a CLI that connects once per command, kept every one of them that
long.
The destructor now also waits, after the drain, for the connection's
SHUTDOWN_COMPLETE before closing the handle. That comes about one round
trip later once the peer answers the close, or after msquic's closing
period (three PTOs) when it does not; the wait is bounded at 1 s, since
the frame went out at the start either way and a peer that is gone must
not hold the destructor for the whole closing period. The handler sets a
flag for it before it looks at the handle, which the destructor holds,
so the close stays the destructor's.
Measured with ShouldTellPeerOfCloseQUIC: before, the server was not told
of the first of 20 client closes within 2 s (and, counted another way,
8 of 100 peer closes reached the server within 10 s in one run, 0 of
100 in the next); after, all 20 within a millisecond.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- --no-quic leaves QUIC, HTTP/3 and WebTransport out of the build, and with
them msquic: a consumer that only needs TCP builds no msquic. The flag
isn't passed on to dependencies.
- --local takes Crafter.Thread from the sibling checkout, as the other
Crafter projects do.
- ClientTCP gains SetNoDelay, SetReceiveTimeout, a connect timeout and
SocketTimeoutException.
- TCP receives return the bytes that arrived:
- RecieveSync(n) returned the whole buffer when fewer had come;
- RecieveUntilCloseSync truncated what it had read;
- a signal (EINTR) during a receive threw where it should wait on;
- RecieveSync(size, buffer) returns the count and throws like the others;
- a receive of no bytes returns at once: recv's 0 for it isn't a close.
- Every TCP *Async receive takes an onError(std::exception_ptr): an
exception escaping a ThreadPool task ended the program. An empty onError
drops the error.
- QUIC streams:
- async receives deliver an empty chunk on close, as the browser build does;
- RecieveUntilFullAsync gives what arrived on a stream that closes early;
- RecieveUntilFullSync takes all it asked for or nothing, under one hold
of the lock, so what did arrive stays for the next receive and no
other receive can take bytes from among it;
- SendAsync's callback hears whether the send went out (bool sent): native,
browser, the JS bridge and the example.
- msquic is pinned to v2.6.3 and built against the system OpenSSL
(QUIC_TLS_LIB=openssl, 3.5 or newer). Its bundled quictls read the system's
openssl.cnf, and the Flatpak runtime's (a PKCS#11 engine, with
config_diagnostics on) made it fail to initialise. The README says to use
--no-quic where the distro's OpenSSL is older.
- Tests now register on any Linux triple: they never ran under the Flatpak's
x86_64-unknown-linux-gnu. New: ShouldBuildWithoutQUIC, ShouldConfigureTCP,
ShouldRecieveTCP, ShouldReportClosedQUICStream.
Breaking: SendAsync's callback takes a bool, the TCP *Async receives take
an onError, and RecieveSync(size, buffer) returns the count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A client that sends on a stream while its connection is being torn down —
the shape any long-lived client with periodic acks or keepalives has —
would hang at exit inside MsQuicRegistrationClose, and occasionally
bugcheck on an msquic worker thread instead.
Root cause: StreamStart is asynchronous. It returns QUIC_STATUS_PENDING
and queues the start onto the connection, so it can still fail later, and
it fails with QUIC_STATUS_INVALID_STATE whenever the connection is shut
down before the queued start runs. Passed QUIC_STREAM_START_FLAG_NONE,
msquic leaves such a stream neither started nor shut down, so
SHUTDOWN_COMPLETE never arrives. The stream close is driven entirely off
that event, so the handle stays open: ~ClientQUIC's bounded drain expires,
the connection is closed with a stream msquic still considers open, and
the registration's rundown at exit() waits forever for the handle.
Instrumenting the handle lifecycle over the reproducer shows one leaked
locally-opened stream per failing run, correlating 1:1 with a
START_COMPLETE carrying 0x1 (INVALID_STATE).
Fixes:
- OpenStream passes QUIC_STREAM_START_FLAG_SHUTDOWN_ON_FAIL, so a start
that loses the race is followed by SHUTDOWN_COMPLETE like any other
stream and its handle is closed.
- QUICStream::Stop also aborts the receive direction. GRACEFUL closes
only the send side and msquic rejects it combined with any other flag,
so a bidirectional stream whose peer keeps its send side open never
reached SHUTDOWN_COMPLETE either.
- The msquic stream handle is now refcounted, held by the callback plus
every app thread inside an msquic call on it, and closed by whoever
lets go last. StreamSend/StreamShutdown/GetParam could previously run
against a handle a worker thread had just closed — a use-after-free
msquic reports as a bugcheck. A refcount rather than an exclusion lock
because GetParam blocks on the connection's worker, which must never
be the thread waiting.
- ~ClientQUIC drops undispatched peer streams before closing the
connection rather than with the rest of Impl afterwards, so the drain
covers them.
The reproducer (tests/ShouldSurviveConnectionChurn) grows a client-opened
bidirectional control stream with a thread sending across the teardown,
and no longer ends in std::_Exit — returning from main runs the static
MsQuicRuntime destructor, and its RegistrationClose is where a leaked
handle shows up. Before: 5 of 15 runs clean, 10 hung. After: 90 of 90
clean, with every opened handle observed closed and the drain never
expiring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
QUIC_SETTINGS never set HandshakeIdleTimeoutMs, so it inherited the
120s idle timeout. The constructor blocks until msquic reports the
connection either up or shut down, which meant dialling a host that does
not answer parked the caller for minutes. Bound it to 10s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MsQuicConnectionClose was reachable twice for one handle: ~ClientQUIC
called it unconditionally, and the SHUTDOWN_COMPLETE callback also called
it whenever AppCloseInProgress was clear -- which it is for the whole
window between the destructor's ConnectionShutdown and its ConnectionClose.
The second call trips CXPLAT_TEL_ASSERT(!Connection->State.HandleClosed)
and aborts the process. Clearing impl->connection from the callback did not
help: the destructor had already loaded it.
`connection` is now a claim token, taken under the mutex, and only whoever
wins the claim closes. The callback keeps closing peer-dropped connections
so they do not leak while their ClientQUIC lives on.
Separately, the destructor now waits (bounded, 5s) for the connection's
streams to reach StreamClose before closing it. QUICStream::Stop only
initiates a shutdown whose async completion does the close, so a
connection could otherwise be closed with streams msquic still considers
open. The count lives in a shared StreamRegistry rather than behind the
ClientQUIC*, so a stream finalising after the wait gave up decrements a
live object.
A failed handshake also closes its connection and configuration now.
Throwing from the constructor body means ~ClientQUIC never runs, so both
handles leaked -- and the connection callback holds the impl that was
about to be destroyed with the half-built object.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
