feat(http1): ClientHTTP1::Fetch follows redirects by the RFC, on a URL type

One request to an http(s) URL with redirects followed by RFC 9110 §15.4:
301, 302, 303, 307 and 308 for every method; a 303 makes the request a
GET without its body and content headers, a 301 or 302 does so for a POST
alone, a 307 or 308 keeps the method and the body. The Location is
resolved against the sending URL by RFC 3986, a hop between http and
https is allowed, Authorization, Proxy-Authorization and Cookie are
dropped on a change of origin, and each hop is a client of its own, since
one ClientHTTP1 is one origin. A redirect without a usable Location, to
another scheme, back to a URL already visited or past maxRedirects comes
back as the response, with `stop` saying why.

The new :URL partition is the URL type behind it, exported: Parse,
Resolve (RFC 3986 §5.2; its §5.4 examples are the test), ToString, and
the pieces a request is built from.

This replaces the FetchHTTP1 free function, which followed redirects the
way Python's urllib did (its repeat limit, its quoting of the Location, a
POST turned back on a 307) for a port that no longer needs the fidelity.

Breaking: FetchHTTP1, FetchOptionsHTTP1 and FetchResultHTTP1 are gone;
use ClientHTTP1::Fetch, FetchOptions and FetchResult, whose `url` is a
URL and whose `limited` became `stop`.
This commit is contained in:
Jorijn van der Graaf 2026-10-31 20:39:43 +01:00
commit 7ee938c98e
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
10 changed files with 596 additions and 372 deletions

View file

@ -13,7 +13,7 @@ The Crafter engine's networking library: a C++ module, `import Crafter.Network;`
|---|---|---|---|
| `HTTPRequest`, `HTTPResponse`, `CreateRequestHTTP`, `CreateResponseHTTP`, `PathWithoutQueryHTTP` | yes | yes | yes |
| `ClientTCP`, `ListenerTCP` | yes | yes | no |
| `ByteStream`, `PlainStream`, `TLSContext`, `TLSStream`, `HTTP1::*`, `ClientHTTP1`, `ListenerHTTP1` | yes | yes | no |
| `ByteStream`, `PlainStream`, `TLSContext`, `TLSStream`, `HTTP1::*`, `URL`, `ClientHTTP1`, `ListenerHTTP1` | yes | yes | no |
| `ClientQUIC`, `QUICStream` | yes | no | `*Async` only, over `WebTransport` |
| `ClientHTTP` | yes | no | `SendAsync` only, over `fetch()` |
| `ListenerQUIC`, `ListenerHTTP`, `WebTransportSession`, `HTTP3::*` | yes | no | no |
@ -43,6 +43,8 @@ A route map is `std::unordered_map<std::string, std::function<HTTPResponse(const
**`ClientHTTP1(host, port)`** is plaintext, **`ClientHTTP1(host, port, TLSClientCredentials)`** is `https://`. `Send(request)` returns the `HTTPResponse` on the calling thread, no pool; `SendAsync(request, onSuccess, onError)` runs it as a pool task with the error as a string. The connection is persistent: the first `Send` dials, later ones reuse the socket until the peer asks to close; a pooled connection the peer closed meanwhile is redialled once and the request replayed, never once a response byte has arrived and never on a fresh connection, so a broken server is an exception, not a loop. An empty `authority` means `host:port`, with the scheme's default port left off (443 under TLS, 80 without); `scheme` is ignored (origin-form targets; the constructor decided the transport). Set `limits` (`HTTP1::MessageLimits`: start line 8 KiB, header section 64 KiB, body 512 MiB), `connectTimeout` (15 s, per address the host resolves to), `handshakeTimeout` (15 s) and `timeout` (30 s for the next piece of a response) before the first `Send`. `Connected()`, `Secure()`, `Protocol()` (the ALPN the last connection agreed, `"http/1.1"` under TLS), `Disconnect()`. One instance serves one caller at a time; instances are independent. Throws `HTTP1::HTTP1ProtocolError` for a response it refuses to parse, `TLSException` for a certificate it rejects, `SocketErrorException` for a socket that cannot resolve or connect or breaks under the request, `SocketTimeoutException` when the connect runs out, and `std::runtime_error` when a response stops arriving or the connection closes before one is complete.
**`ClientHTTP1::Fetch(url, options = {})`** is one request to an `http://` or `https://` URL with redirects followed by RFC 9110 §15.4 (301, 302, 303, 307, 308 for every method; a 303 makes the request a GET without its body and content headers, a 301 or 302 does so for a POST alone, a 307 or 308 keeps method and body), the `Location` resolved against the sending URL, a hop between the schemes allowed, `Authorization`, `Proxy-Authorization` and `Cookie` dropped on a change of origin, and a client per hop. `FetchOptions { method, headers, body, connectTimeout, handshakeTimeout, timeout, limits, credentials, maxRedirects = 10 }`; `FetchResult { response, url, redirects, stop }`, where `url` is the `Crafter::URL` that answered and `stop` (`FetchStop::None`, `NoLocation`, `OtherScheme`, `Loop`, `Limit`) says why a redirect came back unfollowed. Throws `std::invalid_argument` for a URL that is not http(s) or that `URL::Parse` refuses, else what `Send` throws. **`URL::Parse(text)`**, `url.Resolve(reference)`, `url.ToString()`, `Authority()`, `Target()`, `PortOrDefault()`, `Secure()`, `SameOrigin(other)` are the URL type behind it (`scheme`, `userinfo`, `host`, `port`, `path`, `query`, `fragment`; scheme and host lowercased, an IPv6 host without its brackets).
**`ListenerHTTP1(port, routes[, fallback][, TLSServerCredentials])`** is a `ListenerTCP` underneath (every IPv4 address, `SO_REUSEADDR`) and throws `std::runtime_error` when the port cannot be bound, or `TLSException` when the credentials yield no usable certificate. `Listen()` runs the accept loop on the calling thread until `Stop()`, which stops accepting, closes every live connection and joins their threads. **`ListenerAsyncHTTP1`**, same constructors, runs that loop on its `thread` and exposes the `listener`; `Stop()` joins and the destructor stops. Each accepted connection gets its own thread, served one request at a time until the peer closes, sends `connection: close`, or a timeout passes: `keepAliveTimeout` 15 s between requests, `requestTimeout` 30 s between pieces of one request (each read, not the whole request), `handshakeTimeout` 15 s for TLS; `limits` as on the client. `ConnectionCount()`, `AcceptedCount()`, `HandshakeFailureCount()` (connections dropped in the TLS handshake: a scanner, a client with no protocol in common, an untrusted client certificate; ordinary on a public port), `Secure()`.
- Implemented: keep-alive and pipelining, `content-length` and `chunked` request bodies with trailers (merged into `headers`), `Expect: 100-continue`, absolute-form targets, obs-fold, HTTP/1.0 peers (reuse only when they ask), HEAD (the route's handler runs with method `HEAD` and its body is dropped, `content-length` kept), an automatic `date` unless the handler set one, a handler ending the connection with a `connection: close` header. Requests reach handlers with `scheme` `"http"` or `"https"` as the transport decided. Not implemented: CONNECT, `Upgrade`, chunked responses (a handler returns the whole body; responses are `content-length` framed, and 1xx/204/304 carry none).
@ -115,6 +117,7 @@ On a `wasm32-*` target the same `ClientHTTP` and `ClientQUIC` compile against th
| `TLSContext::Server`/`Client`, `TLSStream::Connect`/`Accept` | `TLSException`, naming the certificate error |
| `ByteStream::ReadSome` / `Write` | `std::runtime_error`; a read that times out is `StreamStatus::TimedOut`, a write that does throws |
| `ClientHTTP1::Send` | `HTTP1::HTTP1ProtocolError`, `TLSException`, `SocketErrorException` (the socket's), `SocketTimeoutException` (the connect), `std::runtime_error` (`timed out waiting for the response`, `connection closed before a complete response arrived`) |
| `ClientHTTP1::Fetch`, `URL::Parse`, `URL::Resolve` | `std::invalid_argument` for a URL that is not http(s), has no host or a bad port; `Fetch` otherwise what `Send` throws |
| `ClientHTTP1::SendAsync`, `ClientHTTP::SendAsync` | `what()` as a string to `onError` |
| `HTTP1::MessageParser`, `SerializeRequest`/`SerializeResponse` | `HTTP1::HTTP1ProtocolError` |
| `ClientQUIC` constructor, `OpenStream`, `SendDatagram`, `GetStreamId`, `ListenerQUIC` constructor | `QUICException` (native; the browser build's `ClientQUIC` never throws, see "Browser build") |