feat(http1): ClientHTTP1::Fetch follows redirects by the RFC, on a URL type

One request to an http(s) URL with redirects followed by RFC 9110 §15.4:
301, 302, 303, 307 and 308 for every method; a 303 makes the request a
GET without its body and content headers, a 301 or 302 does so for a POST
alone, a 307 or 308 keeps the method and the body. The Location is
resolved against the sending URL by RFC 3986, a hop between http and
https is allowed, Authorization, Proxy-Authorization and Cookie are
dropped on a change of origin, and each hop is a client of its own, since
one ClientHTTP1 is one origin. A redirect without a usable Location, to
another scheme, back to a URL already visited or past maxRedirects comes
back as the response, with `stop` saying why.

The new :URL partition is the URL type behind it, exported: Parse,
Resolve (RFC 3986 §5.2; its §5.4 examples are the test), ToString, and
the pieces a request is built from.

This replaces the FetchHTTP1 free function, which followed redirects the
way Python's urllib did (its repeat limit, its quoting of the Location, a
POST turned back on a 307) for a port that no longer needs the fidelity.

Breaking: FetchHTTP1, FetchOptionsHTTP1 and FetchResultHTTP1 are gone;
use ClientHTTP1::Fetch, FetchOptions and FetchResult, whose `url` is a
URL and whose `limited` became `stop`.
This commit is contained in:
Jorijn van der Graaf 2026-10-31 20:39:43 +01:00
commit 7ee938c98e
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
10 changed files with 596 additions and 372 deletions

View file

@ -134,20 +134,22 @@ The connection is persistent: the first `Send()` dials, and later calls reuse th
Three timeouts bound a request: `connectTimeout` (default 15 s) for the TCP connect, per address the host resolves to; `handshakeTimeout` (default 15 s) for the TLS handshake; and `timeout` (default 30 s) for each wait on the next piece of a response.
#### FetchHTTP1
#### ClientHTTP1::Fetch
`FetchHTTP1(url, options)` is one request to an `http://` or `https://` URL, following redirects the way Python's urllib does, so code ported from urllib keeps its behaviour:
`ClientHTTP1::Fetch(url, options)` is one request to an `http://` or `https://` URL, following redirects by RFC 9110 §15.4:
```cpp
Crafter::FetchOptionsHTTP1 options;
Crafter::FetchOptions options;
options.timeout = std::chrono::seconds(10);
options.headers["accept-encoding"] = "identity";
Crafter::FetchResultHTTP1 result = Crafter::FetchHTTP1("http://localhost:8080/", options);
// result.response: the last response; result.url: the URL that sent it;
// result.redirects: how many were followed; result.limited: the limits stopped the chain.
Crafter::FetchResult result = Crafter::ClientHTTP1::Fetch("http://localhost:8080/", options);
// result.response: the last response; result.url: the URL that sent it (a Crafter::URL);
// result.redirects: how many were followed; result.stop: why a redirect was not followed, or None.
```
301, 302, 303, 307 and 308 are followed for a GET or HEAD, and 301/302/303 also for a POST; the redirected request is a GET (a HEAD stays a HEAD) without the body or its content headers. A relative `Location` is resolved against the URL that sent it, as `urljoin` resolves it, and is percent-encoded the way urllib quotes it; a redirect may move between `http://` and `https://` (`options.credentials` applies to every TLS hop). Each redirect's body is read before the next request goes out. A redirect is returned unfollowed when it has no `Location`, points at another scheme, or comes for a method urllib does not redirect. The chain stops, returning the redirect with `limited` set, at the fifth redirect to one URL (`maxRepeats`, 4) or once ten different URLs have been visited (`maxRedirects`, 10). Proxies are not consulted. A URL that is not `http(s)://`, has no host or a bad port throws `std::invalid_argument`.
301, 302, 303, 307 and 308 are followed for every method: a 303 turns the request into a GET (a HEAD stays a HEAD) without its body and content headers, a 301 or 302 does the same for a POST alone, and a 307 or 308 keeps the method and the body. A relative `Location` is resolved against the URL that sent it (RFC 3986, through `Crafter::URL::Resolve`), and a redirect may move between `http://` and `https://` (`options.credentials` applies to every TLS hop). `Authorization`, `Proxy-Authorization` and `Cookie` are not carried to another origin. Each hop is a client of its own, so a redirect's body is read and its connection closed before the next request goes out. A redirect comes back as the response, with `result.stop` saying why, when it has no usable `Location`, points at another scheme, points back at a URL the chain already visited, or would be the eleventh (`maxRedirects`, 10). Proxies are not consulted. A URL that is not `http(s)://`, has no host or has a bad port throws `std::invalid_argument`.
`Crafter::URL` is the URL type behind it, exported for callers that build or resolve URLs themselves: `URL::Parse(text)`, `url.Resolve(reference)`, `url.ToString()`, and the pieces a request is built from (`Authority()`, `Target()`, `PortOrDefault()`, `Secure()`, `SameOrigin(other)`).
#### ListenerHTTP1
```cpp
@ -331,7 +333,7 @@ The library includes tests covering:
- Send and receive errors (`ShouldReportIoErrors`) — a connection the peer resets fails the next receive, and a send into it, with `SocketErrorException` carrying ECONNRESET or EPIPE under the plain strerror text a receive and a send always threw
- Loopback listeners (`ShouldBindLoopbackEphemeral`) — a listener on `127.0.0.1` at port 0 reports its port, takes loopback clients and refuses them through the host's other address; `::1` works too, a name is refused, and the port-only constructor still listens on every interface
- HTTP/1.1 timeouts (`ShouldTimeoutSilentHTTP1`) — a server that accepts and never answers fails after `timeout`, and a connect nobody answers after `connectTimeout`
- Redirects (`ShouldFollowRedirectsHTTP1`) — relative and `..` Locations, an http-to-https hop, the 4-repeat and 10-URL limits, a redirect without Location, POST after 303 and 307, percent-encoding a Location, and the URLs `FetchHTTP1` refuses
- Redirects (`ShouldFollowRedirectsHTTP1`) — relative and `..` Locations, an http-to-https hop, a loop and the redirect limit, a redirect without Location or to another scheme, a POST after 303, 302 and 307 and a PUT after 302, credentials dropped across origins, and the URLs `Fetch` refuses; `ShouldParseURLs` runs RFC 3986 §5.4's resolution examples and the parse rules
The external-interop test requires outbound UDP/443; if your network blocks it the test will fail. `ShouldInteropCurlHTTP1` and `ShouldInteropCurlHTTPS1` skip whichever half is unavailable when `curl` or `python3` is not installed, and the mutual-TLS half of `ShouldRequireClientCertificateHTTPS1` skips without the `openssl` CLI (a *client* certificate needs the `clientAuth` extended key usage, which the built-in development certificate does not carry) — so all three pass on a bare machine. Install `curl`, `python3` and `openssl` to actually exercise them.