fix(quic): close stream handles whose async StreamStart lost the teardown race
A client that sends on a stream while its connection is being torn down —
the shape any long-lived client with periodic acks or keepalives has —
would hang at exit inside MsQuicRegistrationClose, and occasionally
bugcheck on an msquic worker thread instead.
Root cause: StreamStart is asynchronous. It returns QUIC_STATUS_PENDING
and queues the start onto the connection, so it can still fail later, and
it fails with QUIC_STATUS_INVALID_STATE whenever the connection is shut
down before the queued start runs. Passed QUIC_STREAM_START_FLAG_NONE,
msquic leaves such a stream neither started nor shut down, so
SHUTDOWN_COMPLETE never arrives. The stream close is driven entirely off
that event, so the handle stays open: ~ClientQUIC's bounded drain expires,
the connection is closed with a stream msquic still considers open, and
the registration's rundown at exit() waits forever for the handle.
Instrumenting the handle lifecycle over the reproducer shows one leaked
locally-opened stream per failing run, correlating 1:1 with a
START_COMPLETE carrying 0x1 (INVALID_STATE).
Fixes:
- OpenStream passes QUIC_STREAM_START_FLAG_SHUTDOWN_ON_FAIL, so a start
that loses the race is followed by SHUTDOWN_COMPLETE like any other
stream and its handle is closed.
- QUICStream::Stop also aborts the receive direction. GRACEFUL closes
only the send side and msquic rejects it combined with any other flag,
so a bidirectional stream whose peer keeps its send side open never
reached SHUTDOWN_COMPLETE either.
- The msquic stream handle is now refcounted, held by the callback plus
every app thread inside an msquic call on it, and closed by whoever
lets go last. StreamSend/StreamShutdown/GetParam could previously run
against a handle a worker thread had just closed — a use-after-free
msquic reports as a bugcheck. A refcount rather than an exclusion lock
because GetParam blocks on the connection's worker, which must never
be the thread waiting.
- ~ClientQUIC drops undispatched peer streams before closing the
connection rather than with the rest of Impl afterwards, so the drain
covers them.
The reproducer (tests/ShouldSurviveConnectionChurn) grows a client-opened
bidirectional control stream with a thread sending across the teardown,
and no longer ends in std::_Exit — returning from main runs the static
MsQuicRuntime destructor, and its RegistrationClose is where a leaked
handle shows up. Before: 5 of 15 runs clean, 10 hung. After: 90 of 90
clean, with every opened handle observed closed and the drain never
expiring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
9a5cab88eb
commit
ffc118f825
3 changed files with 231 additions and 76 deletions
|
|
@ -1,25 +1,38 @@
|
|||
//SPDX-License-Identifier: LGPL-3.0-only
|
||||
//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
|
||||
|
||||
// Regression test for the connection-teardown race in ~ClientQUIC.
|
||||
// Regression test for connection teardown on the client side.
|
||||
//
|
||||
// QUICStream::Stop only *initiates* a graceful shutdown; the actual
|
||||
// MsQuicStreamClose happens later, from the terminal SHUTDOWN_COMPLETE
|
||||
// callback on an msquic worker thread. ~ClientQUIC used to call
|
||||
// MsQuicConnectionClose straight away, so a connection could be closed while
|
||||
// streams belonging to it were still open on the msquic side — which trips an
|
||||
// msquic bugcheck and aborts the process.
|
||||
// QUICStream::Stop only *initiates* a shutdown; the msquic handle is closed
|
||||
// later, off the terminal SHUTDOWN_COMPLETE callback on a worker thread. Two
|
||||
// things used to go wrong with that, and both need a reconnect loop to show
|
||||
// up, which is what this test is:
|
||||
//
|
||||
// Nothing else in the suite exercises this: every other QUIC test ends in
|
||||
// std::_Exit(0) and so never runs a ClientQUIC destructor. The shape that
|
||||
// breaks is a reconnect loop — build a connection, tear it down, build
|
||||
// another — which is exactly what this test does, `cycles` times.
|
||||
// 1. ~ClientQUIC closed the connection straight away, so a connection could
|
||||
// be closed while streams belonging to it were still open on the msquic
|
||||
// side — an msquic bugcheck, which aborts the process.
|
||||
//
|
||||
// The blocked readers matter. Each inbound stream gets a thread parked inside
|
||||
// RecieveSync for the life of the connection (the shape a long-lived
|
||||
// streaming client has), so teardown happens with readers still inside the
|
||||
// receive path. An otherwise identical loop whose handler reads once and
|
||||
// returns does not reproduce.
|
||||
// 2. StreamStart is asynchronous. If the connection was shut down before the
|
||||
// queued start ran, msquic failed the start and left the stream neither
|
||||
// started nor shut down, so SHUTDOWN_COMPLETE never arrived and the handle
|
||||
// was never closed. The registration's rundown at exit() then blocked
|
||||
// forever waiting for it.
|
||||
//
|
||||
// The test therefore ends by falling off the end of main() rather than with
|
||||
// std::_Exit — the static MsQuicRuntime destructor's RegistrationClose is
|
||||
// where a leaked handle shows up, so skipping it would skip half the point.
|
||||
//
|
||||
// Two things about the shape matter:
|
||||
//
|
||||
// - Blocked readers. Each inbound stream gets a thread parked inside
|
||||
// RecieveSync for the life of the connection (the shape a long-lived
|
||||
// streaming client has), so teardown happens with readers still inside the
|
||||
// receive path.
|
||||
//
|
||||
// - A concurrent sender. A client-opened bidirectional control stream with a
|
||||
// thread sending on it across the teardown, which is the shape a client
|
||||
// with periodic acks or keepalives has. It is the send racing the shutdown
|
||||
// that reaches (2); a loop without it passes on the broken build.
|
||||
|
||||
import Crafter.Network;
|
||||
import Crafter.Thread;
|
||||
|
|
@ -57,6 +70,20 @@ int main() {
|
|||
|
||||
auto listener = std::make_unique<ListenerQUIC>(
|
||||
port, alpn, serverCreds, [&](ClientQUIC* peer) {
|
||||
// Drain whatever the client opens toward us. Without this the
|
||||
// client's control-stream sends would block on flow control
|
||||
// and the test would wedge on its own account rather than on
|
||||
// anything the library did.
|
||||
peer->OnStream([](QUICStream inbound) {
|
||||
auto shared = std::make_shared<QUICStream>(std::move(inbound));
|
||||
std::thread([shared] {
|
||||
try {
|
||||
while (true) (void)shared->RecieveSync();
|
||||
} catch (...) {
|
||||
// Connection closed.
|
||||
}
|
||||
}).detach();
|
||||
});
|
||||
std::lock_guard lock(serverMutex);
|
||||
accepted = peer;
|
||||
for (int s = 0; s < streamsPerConnection; ++s) {
|
||||
|
|
@ -106,22 +133,40 @@ int main() {
|
|||
}
|
||||
}
|
||||
|
||||
// The sends keep going while the connection is torn down
|
||||
// underneath them.
|
||||
QUICStream control = client.OpenStream(/*unidirectional=*/false);
|
||||
std::atomic<bool> sending{ true };
|
||||
std::thread sender([&] {
|
||||
while (sending) {
|
||||
try {
|
||||
const char tick[] = "ack";
|
||||
control.SendSync(tick, sizeof(tick) - 1, /*finish=*/false);
|
||||
} catch (...) {
|
||||
return; // connection went away mid-send
|
||||
}
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(2));
|
||||
}
|
||||
});
|
||||
|
||||
client.Stop();
|
||||
sending = false;
|
||||
if (sender.joinable()) sender.join();
|
||||
for (std::thread& reader : readers) {
|
||||
if (reader.joinable()) reader.join();
|
||||
}
|
||||
// `client` is destroyed here, once the readers have unwound.
|
||||
// `control` and then `client` are destroyed here, once the sender
|
||||
// and readers have unwound.
|
||||
}
|
||||
|
||||
{
|
||||
std::lock_guard lock(serverMutex);
|
||||
// Deliberately the other order from the client side: destroy the
|
||||
// connection while the app still holds its stream wrappers, so
|
||||
// ~ClientQUIC has to cope with streams that are open as far as
|
||||
// Deliberately the other order from the client side: shut the
|
||||
// connection down while the app still holds its stream wrappers,
|
||||
// so teardown has to cope with streams that are open as far as
|
||||
// msquic is concerned.
|
||||
delete accepted;
|
||||
accepted = nullptr;
|
||||
serverStreams.clear();
|
||||
if (accepted) accepted->Stop();
|
||||
}
|
||||
listener->Stop();
|
||||
listener.reset();
|
||||
|
|
@ -129,8 +174,9 @@ int main() {
|
|||
|
||||
std::println("survived {} connection teardowns", cycles);
|
||||
std::cout.flush();
|
||||
// Skip the static-dtor cleanup: msquic's RegistrationClose blocks until
|
||||
// every connection it ever opened is fully drained, which the suite does
|
||||
// not need to wait on. Everything this test asserts has already happened.
|
||||
std::_Exit(0);
|
||||
// Deliberately no std::_Exit: returning from main runs the static
|
||||
// MsQuicRuntime destructor, whose RegistrationClose blocks until every
|
||||
// handle opened under the registration has been closed. That wait is the
|
||||
// assertion for (2) above.
|
||||
return 0;
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue