feat(http1): bound the connect, and FetchHTTP1 for a URL with redirects

- ClientHTTP1 gains connectTimeout (default 15 s), applied to each
  address the host resolves to. `timeout` only ever bounded the waits for
  a response, so a host that drops the SYNs held Send() for the kernel's
  retries, about two minutes, whatever the caller had asked for.
- The default Host header puts an IPv6 literal in brackets.
- FetchHTTP1(url, options) makes one request to an http:// or https://
  URL and follows redirects as Python's urllib does, so urllib code ported
  over keeps its behaviour: 301/302/303/307/308 for GET and HEAD,
  301/302/303 for POST (re-sent as a GET without its body), a relative
  Location resolved as urljoin resolves it and percent-encoded as urllib
  quotes it, http and https hops mixed, each redirect's body read before
  the next request, and the chain stopped by urllib's limits (4 repeats
  of one URL, 10 different URLs) with the last redirect returned and
  `limited` set. Proxies are not consulted.

New: ShouldTimeoutSilentHTTP1, ShouldFollowRedirectsHTTP1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-31 05:06:41 +01:00
commit f9a45c143e
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
6 changed files with 649 additions and 6 deletions

View file

@ -87,10 +87,66 @@ namespace Crafter {
// from `timeout` because it covers a multi-round-trip exchange before
// any request has been written.
std::chrono::milliseconds handshakeTimeout{15000};
// How long the TCP connect may take, for each address the host
// resolves to (see ClientTCP). Without it a host that drops the SYNs
// held Send() for the kernel's retries, about two minutes, whatever
// `timeout` said.
std::chrono::milliseconds connectTimeout{15000};
private:
struct Impl;
std::unique_ptr<Impl> impl;
};
// What FetchHTTP1 needs beyond the URL.
export struct FetchOptionsHTTP1 {
std::string method = "GET";
// Lowercase names, as on HTTPRequest. Host comes from the URL.
std::unordered_map<std::string, std::string> headers;
std::string body;
// As on ClientHTTP1, for every request of the chain.
std::chrono::milliseconds connectTimeout{15000};
std::chrono::milliseconds handshakeTimeout{15000};
std::chrono::milliseconds timeout{30000};
HTTP1::MessageLimits limits;
// For every https:// request of the chain.
TLSClientCredentials credentials;
// Python urllib's limits: a chain stops at a redirect to a URL it has
// already been sent to `maxRepeats` times, or once it has been sent
// to `maxRedirects` different URLs.
std::uint32_t maxRedirects = 10;
std::uint32_t maxRepeats = 4;
};
export struct FetchResultHTTP1 {
// The last response of the chain. A redirect only when it had nothing
// to follow — no Location, a method urllib does not redirect, a
// scheme other than http(s) — or when the limits stopped the chain.
HTTPResponse response;
// The URL that answered with `response`.
std::string url;
// Redirects followed to get there.
std::uint32_t redirects = 0;
// The limits stopped the chain; `response` is the redirect that would
// have gone past them.
bool limited = false;
};
// One request to an http:// or https:// URL, following redirects the way
// Python's urllib does, so a port of urllib code keeps its behaviour:
// 301, 302, 303, 307 and 308 are followed for GET and HEAD, the first
// three also for POST; the redirected request is a GET (a HEAD stays a
// HEAD) without the body and its content headers. A relative Location is
// resolved against the URL that sent it, as urljoin resolves it; a
// redirect may switch between http:// and https://. Each redirect's body
// is read before the next request goes out. Proxies are not consulted.
//
// The URL is `scheme://[user@]host[:port][/path][?query][#fragment]`,
// with an IPv6 host in brackets; the fragment is not sent and the user
// part is dropped. Throws std::invalid_argument for a URL that is not
// http(s) or has no host or a bad port, and whatever a request throws
// (SocketErrorException, SocketTimeoutException, TLSException,
// std::runtime_error) otherwise.
export FetchResultHTTP1 FetchHTTP1(std::string_view url, const FetchOptionsHTTP1& options = {});
}
#endif