fix(quic): close stream handles whose async StreamStart lost the teardown race

A client that sends on a stream while its connection is being torn down —
the shape any long-lived client with periodic acks or keepalives has —
would hang at exit inside MsQuicRegistrationClose, and occasionally
bugcheck on an msquic worker thread instead.

Root cause: StreamStart is asynchronous. It returns QUIC_STATUS_PENDING
and queues the start onto the connection, so it can still fail later, and
it fails with QUIC_STATUS_INVALID_STATE whenever the connection is shut
down before the queued start runs. Passed QUIC_STREAM_START_FLAG_NONE,
msquic leaves such a stream neither started nor shut down, so
SHUTDOWN_COMPLETE never arrives. The stream close is driven entirely off
that event, so the handle stays open: ~ClientQUIC's bounded drain expires,
the connection is closed with a stream msquic still considers open, and
the registration's rundown at exit() waits forever for the handle.

Instrumenting the handle lifecycle over the reproducer shows one leaked
locally-opened stream per failing run, correlating 1:1 with a
START_COMPLETE carrying 0x1 (INVALID_STATE).

Fixes:

  - OpenStream passes QUIC_STREAM_START_FLAG_SHUTDOWN_ON_FAIL, so a start
    that loses the race is followed by SHUTDOWN_COMPLETE like any other
    stream and its handle is closed.

  - QUICStream::Stop also aborts the receive direction. GRACEFUL closes
    only the send side and msquic rejects it combined with any other flag,
    so a bidirectional stream whose peer keeps its send side open never
    reached SHUTDOWN_COMPLETE either.

  - The msquic stream handle is now refcounted, held by the callback plus
    every app thread inside an msquic call on it, and closed by whoever
    lets go last. StreamSend/StreamShutdown/GetParam could previously run
    against a handle a worker thread had just closed — a use-after-free
    msquic reports as a bugcheck. A refcount rather than an exclusion lock
    because GetParam blocks on the connection's worker, which must never
    be the thread waiting.

  - ~ClientQUIC drops undispatched peer streams before closing the
    connection rather than with the rest of Impl afterwards, so the drain
    covers them.

The reproducer (tests/ShouldSurviveConnectionChurn) grows a client-opened
bidirectional control stream with a thread sending across the teardown,
and no longer ends in std::_Exit — returning from main runs the static
MsQuicRuntime destructor, and its RegistrationClose is where a leaked
handle shows up. Before: 5 of 15 runs clean, 10 hung. After: 90 of 90
clean, with every opened handle observed closed and the drain never
expiring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
catbot 2026-08-25 23:18:01 +00:00
commit ffc118f825
3 changed files with 231 additions and 76 deletions

View file

@ -1,25 +1,38 @@
//SPDX-License-Identifier: LGPL-3.0-only
//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// Regression test for the connection-teardown race in ~ClientQUIC.
// Regression test for connection teardown on the client side.
//
// QUICStream::Stop only *initiates* a graceful shutdown; the actual
// MsQuicStreamClose happens later, from the terminal SHUTDOWN_COMPLETE
// callback on an msquic worker thread. ~ClientQUIC used to call
// MsQuicConnectionClose straight away, so a connection could be closed while
// streams belonging to it were still open on the msquic side — which trips an
// msquic bugcheck and aborts the process.
// QUICStream::Stop only *initiates* a shutdown; the msquic handle is closed
// later, off the terminal SHUTDOWN_COMPLETE callback on a worker thread. Two
// things used to go wrong with that, and both need a reconnect loop to show
// up, which is what this test is:
//
// Nothing else in the suite exercises this: every other QUIC test ends in
// std::_Exit(0) and so never runs a ClientQUIC destructor. The shape that
// breaks is a reconnect loop — build a connection, tear it down, build
// another — which is exactly what this test does, `cycles` times.
// 1. ~ClientQUIC closed the connection straight away, so a connection could
// be closed while streams belonging to it were still open on the msquic
// side — an msquic bugcheck, which aborts the process.
//
// The blocked readers matter. Each inbound stream gets a thread parked inside
// RecieveSync for the life of the connection (the shape a long-lived
// streaming client has), so teardown happens with readers still inside the
// receive path. An otherwise identical loop whose handler reads once and
// returns does not reproduce.
// 2. StreamStart is asynchronous. If the connection was shut down before the
// queued start ran, msquic failed the start and left the stream neither
// started nor shut down, so SHUTDOWN_COMPLETE never arrived and the handle
// was never closed. The registration's rundown at exit() then blocked
// forever waiting for it.
//
// The test therefore ends by falling off the end of main() rather than with
// std::_Exit — the static MsQuicRuntime destructor's RegistrationClose is
// where a leaked handle shows up, so skipping it would skip half the point.
//
// Two things about the shape matter:
//
// - Blocked readers. Each inbound stream gets a thread parked inside
// RecieveSync for the life of the connection (the shape a long-lived
// streaming client has), so teardown happens with readers still inside the
// receive path.
//
// - A concurrent sender. A client-opened bidirectional control stream with a
// thread sending on it across the teardown, which is the shape a client
// with periodic acks or keepalives has. It is the send racing the shutdown
// that reaches (2); a loop without it passes on the broken build.
import Crafter.Network;
import Crafter.Thread;
@ -57,6 +70,20 @@ int main() {
auto listener = std::make_unique<ListenerQUIC>(
port, alpn, serverCreds, [&](ClientQUIC* peer) {
// Drain whatever the client opens toward us. Without this the
// client's control-stream sends would block on flow control
// and the test would wedge on its own account rather than on
// anything the library did.
peer->OnStream([](QUICStream inbound) {
auto shared = std::make_shared<QUICStream>(std::move(inbound));
std::thread([shared] {
try {
while (true) (void)shared->RecieveSync();
} catch (...) {
// Connection closed.
}
}).detach();
});
std::lock_guard lock(serverMutex);
accepted = peer;
for (int s = 0; s < streamsPerConnection; ++s) {
@ -106,22 +133,40 @@ int main() {
}
}
// The sends keep going while the connection is torn down
// underneath them.
QUICStream control = client.OpenStream(/*unidirectional=*/false);
std::atomic<bool> sending{ true };
std::thread sender([&] {
while (sending) {
try {
const char tick[] = "ack";
control.SendSync(tick, sizeof(tick) - 1, /*finish=*/false);
} catch (...) {
return; // connection went away mid-send
}
std::this_thread::sleep_for(std::chrono::milliseconds(2));
}
});
client.Stop();
sending = false;
if (sender.joinable()) sender.join();
for (std::thread& reader : readers) {
if (reader.joinable()) reader.join();
}
// `client` is destroyed here, once the readers have unwound.
// `control` and then `client` are destroyed here, once the sender
// and readers have unwound.
}
{
std::lock_guard lock(serverMutex);
// Deliberately the other order from the client side: destroy the
// connection while the app still holds its stream wrappers, so
// ~ClientQUIC has to cope with streams that are open as far as
// Deliberately the other order from the client side: shut the
// connection down while the app still holds its stream wrappers,
// so teardown has to cope with streams that are open as far as
// msquic is concerned.
delete accepted;
accepted = nullptr;
serverStreams.clear();
if (accepted) accepted->Stop();
}
listener->Stop();
listener.reset();
@ -129,8 +174,9 @@ int main() {
std::println("survived {} connection teardowns", cycles);
std::cout.flush();
// Skip the static-dtor cleanup: msquic's RegistrationClose blocks until
// every connection it ever opened is fully drained, which the suite does
// not need to wait on. Everything this test asserts has already happened.
std::_Exit(0);
// Deliberately no std::_Exit: returning from main runs the static
// MsQuicRuntime destructor, whose RegistrationClose blocks until every
// handle opened under the registration has been closed. That wait is the
// assertion for (2) above.
return 0;
}