feat(http1): bound the connect, and FetchHTTP1 for a URL with redirects
- ClientHTTP1 gains connectTimeout (default 15 s), applied to each address the host resolves to. `timeout` only ever bounded the waits for a response, so a host that drops the SYNs held Send() for the kernel's retries, about two minutes, whatever the caller had asked for. - The default Host header puts an IPv6 literal in brackets. - FetchHTTP1(url, options) makes one request to an http:// or https:// URL and follows redirects as Python's urllib does, so urllib code ported over keeps its behaviour: 301/302/303/307/308 for GET and HEAD, 301/302/303 for POST (re-sent as a GET without its body), a relative Location resolved as urljoin resolves it and percent-encoded as urllib quotes it, http and https hops mixed, each redirect's body read before the next request, and the chain stopped by urllib's limits (4 repeats of one URL, 10 different URLs) with the last redirect returned and `limited` set. Proxies are not consulted. New: ShouldTimeoutSilentHTTP1, ShouldFollowRedirectsHTTP1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
d6f8f4edcb
commit
f9a45c143e
GPG key ID:
2937E59CDCC1BCFB
6 changed files with 649 additions and 6 deletions
21
README.md
21
README.md
|
|
@ -129,7 +129,24 @@ Crafter::HTTPResponse response = client.Send(
|
||||||
);
|
);
|
||||||
```
|
```
|
||||||
|
|
||||||
The connection is persistent: the first `Send()` dials, and later calls reuse the socket unless the peer asked for it to be closed. A pooled connection that the peer closed while it looked idle — the race HTTP/1.1 keep-alive cannot avoid — is redialled once and the request replayed; nothing is replayed once a response byte has arrived, and a freshly dialled connection is never retried on, so a genuinely broken server surfaces as an exception rather than a retry loop. `authority` defaults to the host:port the client was constructed with (the port is elided when it is the scheme default — 443 under TLS, 80 without).
|
The connection is persistent: the first `Send()` dials, and later calls reuse the socket unless the peer asked for it to be closed. A pooled connection that the peer closed while it looked idle — the race HTTP/1.1 keep-alive cannot avoid — is redialled once and the request replayed; nothing is replayed once a response byte has arrived, and a freshly dialled connection is never retried on, so a genuinely broken server surfaces as an exception rather than a retry loop. `authority` defaults to the host:port the client was constructed with (the port is elided when it is the scheme default — 443 under TLS, 80 without; an IPv6 literal goes in brackets).
|
||||||
|
|
||||||
|
Three timeouts bound a request: `connectTimeout` (default 15 s) for the TCP connect, per address the host resolves to; `handshakeTimeout` (default 15 s) for the TLS handshake; and `timeout` (default 30 s) for each wait on the next piece of a response.
|
||||||
|
|
||||||
|
#### FetchHTTP1
|
||||||
|
|
||||||
|
`FetchHTTP1(url, options)` is one request to an `http://` or `https://` URL, following redirects the way Python's urllib does, so code ported from urllib keeps its behaviour:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
Crafter::FetchOptionsHTTP1 options;
|
||||||
|
options.timeout = std::chrono::seconds(10);
|
||||||
|
options.headers["accept-encoding"] = "identity";
|
||||||
|
Crafter::FetchResultHTTP1 result = Crafter::FetchHTTP1("http://localhost:8080/", options);
|
||||||
|
// result.response: the last response; result.url: the URL that sent it;
|
||||||
|
// result.redirects: how many were followed; result.limited: the limits stopped the chain.
|
||||||
|
```
|
||||||
|
|
||||||
|
301, 302, 303, 307 and 308 are followed for a GET or HEAD, and 301/302/303 also for a POST; the redirected request is a GET (a HEAD stays a HEAD) without the body or its content headers. A relative `Location` is resolved against the URL that sent it, as `urljoin` resolves it, and is percent-encoded the way urllib quotes it; a redirect may move between `http://` and `https://` (`options.credentials` applies to every TLS hop). Each redirect's body is read before the next request goes out. A redirect is returned unfollowed when it has no `Location`, points at another scheme, or comes for a method urllib does not redirect. The chain stops, returning the redirect with `limited` set, at the fifth redirect to one URL (`maxRepeats`, 4) or once ten different URLs have been visited (`maxRedirects`, 10). Proxies are not consulted. A URL that is not `http(s)://`, has no host or a bad port throws `std::invalid_argument`.
|
||||||
|
|
||||||
#### ListenerHTTP1
|
#### ListenerHTTP1
|
||||||
```cpp
|
```cpp
|
||||||
|
|
@ -311,6 +328,8 @@ The library includes tests covering:
|
||||||
- Dual-stack connect (`ShouldConnectDualStack`) — `localhost` reaches a server on `::1` only and one on `127.0.0.1` only, an IPv6 literal connects over IPv6, and the last failure is what is thrown when no address answers; skips on a host without IPv6 loopback
|
- Dual-stack connect (`ShouldConnectDualStack`) — `localhost` reaches a server on `::1` only and one on `127.0.0.1` only, an IPv6 literal connects over IPv6, and the last failure is what is thrown when no address answers; skips on a host without IPv6 loopback
|
||||||
- Connect errors (`ShouldReportConnectErrors`) — a refused connect and an unresolvable name throw `SocketErrorException` with their errno or getaddrinfo code and the message ClientTCP always threw, and still reach a `std::runtime_error` handler
|
- Connect errors (`ShouldReportConnectErrors`) — a refused connect and an unresolvable name throw `SocketErrorException` with their errno or getaddrinfo code and the message ClientTCP always threw, and still reach a `std::runtime_error` handler
|
||||||
- Loopback listeners (`ShouldBindLoopbackEphemeral`) — a listener on `127.0.0.1` at port 0 reports its port, takes loopback clients and refuses them through the host's other address; `::1` works too, a name is refused, and the port-only constructor still listens on every interface
|
- Loopback listeners (`ShouldBindLoopbackEphemeral`) — a listener on `127.0.0.1` at port 0 reports its port, takes loopback clients and refuses them through the host's other address; `::1` works too, a name is refused, and the port-only constructor still listens on every interface
|
||||||
|
- HTTP/1.1 timeouts (`ShouldTimeoutSilentHTTP1`) — a server that accepts and never answers fails after `timeout`, and a connect nobody answers after `connectTimeout`
|
||||||
|
- Redirects (`ShouldFollowRedirectsHTTP1`) — relative and `..` Locations, an http-to-https hop, the 4-repeat and 10-URL limits, a redirect without Location, POST after 303 and 307, percent-encoding a Location, and the URLs `FetchHTTP1` refuses
|
||||||
|
|
||||||
The external-interop test requires outbound UDP/443; if your network blocks it the test will fail. `ShouldInteropCurlHTTP1` and `ShouldInteropCurlHTTPS1` skip whichever half is unavailable when `curl` or `python3` is not installed, and the mutual-TLS half of `ShouldRequireClientCertificateHTTPS1` skips without the `openssl` CLI (a *client* certificate needs the `clientAuth` extended key usage, which the built-in development certificate does not carry) — so all three pass on a bare machine. Install `curl`, `python3` and `openssl` to actually exercise them.
|
The external-interop test requires outbound UDP/443; if your network blocks it the test will fail. `ShouldInteropCurlHTTP1` and `ShouldInteropCurlHTTPS1` skip whichever half is unavailable when `curl` or `python3` is not installed, and the mutual-TLS half of `ShouldRequireClientCertificateHTTPS1` skips without the `openssl` CLI (a *client* certificate needs the `clientAuth` extended key usage, which the built-in development certificate does not carry) — so all three pass on a bare machine. Install `curl`, `python3` and `openssl` to actually exercise them.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -21,8 +21,10 @@ namespace {
|
||||||
// which is what every other HTTP/1.1 client on the wire does and what
|
// which is what every other HTTP/1.1 client on the wire does and what
|
||||||
// virtual-host matching on the far side tends to expect.
|
// virtual-host matching on the far side tends to expect.
|
||||||
std::string DefaultAuthority(const std::string& host, std::uint16_t port, bool secure) {
|
std::string DefaultAuthority(const std::string& host, std::uint16_t port, bool secure) {
|
||||||
if (port == (secure ? 443 : 80)) return host;
|
// An IPv6 literal goes in brackets, or its colons read as the port's.
|
||||||
return host + ":" + std::to_string(port);
|
const std::string name = host.contains(':') ? "[" + host + "]" : host;
|
||||||
|
if (port == (secure ? 443 : 80)) return name;
|
||||||
|
return name + ":" + std::to_string(port);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -37,8 +39,8 @@ struct ClientHTTP1::Impl {
|
||||||
std::unique_ptr<ByteStream> stream;
|
std::unique_ptr<ByteStream> stream;
|
||||||
std::string protocol;
|
std::string protocol;
|
||||||
|
|
||||||
void Connect(std::chrono::milliseconds handshakeTimeout) {
|
void Connect(std::chrono::milliseconds connectTimeout, std::chrono::milliseconds handshakeTimeout) {
|
||||||
tcp = std::make_unique<ClientTCP>(host, port);
|
tcp = std::make_unique<ClientTCP>(host, port, connectTimeout);
|
||||||
try {
|
try {
|
||||||
if (tls) {
|
if (tls) {
|
||||||
stream = TLSStream::Connect(tcp->socketid, tls, host, handshakeTimeout);
|
stream = TLSStream::Connect(tcp->socketid, tls, host, handshakeTimeout);
|
||||||
|
|
@ -146,7 +148,7 @@ HTTPResponse ClientHTTP1::Send(const HTTPRequest& request) {
|
||||||
// read fails. Nothing is replayed once a response byte has arrived.
|
// read fails. Nothing is replayed once a response byte has arrived.
|
||||||
for (int attempt = 0;; ++attempt) {
|
for (int attempt = 0;; ++attempt) {
|
||||||
const bool reused = impl->tcp != nullptr;
|
const bool reused = impl->tcp != nullptr;
|
||||||
if (!reused) impl->Connect(handshakeTimeout);
|
if (!reused) impl->Connect(connectTimeout, handshakeTimeout);
|
||||||
|
|
||||||
bool received = false;
|
bool received = false;
|
||||||
try {
|
try {
|
||||||
|
|
@ -176,3 +178,305 @@ void ClientHTTP1::SendAsync(const HTTPRequest& request,
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
// The five parts urllib.parse.urlsplit cuts a URL into. FetchHTTP1 needs
|
||||||
|
// the splitting, the joining (urljoin) and the reassembly to resolve a
|
||||||
|
// Location the way urllib does, and nothing else of urllib.parse.
|
||||||
|
struct SplitURL {
|
||||||
|
std::string scheme;
|
||||||
|
std::string netloc;
|
||||||
|
std::string path;
|
||||||
|
std::string query;
|
||||||
|
std::string fragment;
|
||||||
|
};
|
||||||
|
|
||||||
|
bool IsAsciiAlpha(char c) {
|
||||||
|
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z');
|
||||||
|
}
|
||||||
|
|
||||||
|
bool IsSchemeChar(char c) {
|
||||||
|
return IsAsciiAlpha(c) || (c >= '0' && c <= '9') || c == '+' || c == '-' || c == '.';
|
||||||
|
}
|
||||||
|
|
||||||
|
std::string ToLowerAsciiCopy(std::string_view text) {
|
||||||
|
std::string lower(text);
|
||||||
|
for (char& c : lower) {
|
||||||
|
if (c >= 'A' && c <= 'Z') c = static_cast<char>(c - 'A' + 'a');
|
||||||
|
}
|
||||||
|
return lower;
|
||||||
|
}
|
||||||
|
|
||||||
|
// str.split('/'): every piece, empty ones included, and [""] for "".
|
||||||
|
std::vector<std::string> SplitSlashes(std::string_view text) {
|
||||||
|
std::vector<std::string> pieces;
|
||||||
|
std::size_t start = 0;
|
||||||
|
while (true) {
|
||||||
|
const std::size_t slash = text.find('/', start);
|
||||||
|
if (slash == std::string_view::npos) {
|
||||||
|
pieces.emplace_back(text.substr(start));
|
||||||
|
return pieces;
|
||||||
|
}
|
||||||
|
pieces.emplace_back(text.substr(start, slash - start));
|
||||||
|
start = slash + 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// urlsplit: tabs and newlines dropped anywhere, leading controls and
|
||||||
|
// spaces stripped, the scheme lowercased.
|
||||||
|
SplitURL Split(std::string_view input) {
|
||||||
|
std::string url;
|
||||||
|
url.reserve(input.size());
|
||||||
|
for (const char c : input) {
|
||||||
|
if (c != '\t' && c != '\r' && c != '\n') url.push_back(c);
|
||||||
|
}
|
||||||
|
std::size_t start = 0;
|
||||||
|
while (start < url.size() && static_cast<unsigned char>(url[start]) <= 0x20) ++start;
|
||||||
|
url.erase(0, start);
|
||||||
|
|
||||||
|
SplitURL parts;
|
||||||
|
const std::size_t colon = url.find(':');
|
||||||
|
if (colon != std::string::npos && colon > 0 && IsAsciiAlpha(url[0])
|
||||||
|
&& std::ranges::all_of(std::string_view(url).substr(0, colon), IsSchemeChar)) {
|
||||||
|
parts.scheme = ToLowerAsciiCopy(std::string_view(url).substr(0, colon));
|
||||||
|
url.erase(0, colon + 1);
|
||||||
|
}
|
||||||
|
if (url.starts_with("//")) {
|
||||||
|
std::size_t end = url.find_first_of("/?#", 2);
|
||||||
|
if (end == std::string::npos) end = url.size();
|
||||||
|
parts.netloc = url.substr(2, end - 2);
|
||||||
|
url.erase(0, end);
|
||||||
|
if (parts.netloc.contains('[') != parts.netloc.contains(']')) {
|
||||||
|
throw std::invalid_argument("FetchHTTP1: unbalanced brackets in the host of " + std::string(input));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (const std::size_t hash = url.find('#'); hash != std::string::npos) {
|
||||||
|
parts.fragment = url.substr(hash + 1);
|
||||||
|
url.erase(hash);
|
||||||
|
}
|
||||||
|
if (const std::size_t question = url.find('?'); question != std::string::npos) {
|
||||||
|
parts.query = url.substr(question + 1);
|
||||||
|
url.erase(question);
|
||||||
|
}
|
||||||
|
parts.path = std::move(url);
|
||||||
|
return parts;
|
||||||
|
}
|
||||||
|
|
||||||
|
// urlunsplit, for the http(s) URLs FetchHTTP1 puts back together.
|
||||||
|
std::string Unsplit(const SplitURL& parts) {
|
||||||
|
std::string url = parts.path;
|
||||||
|
if (!parts.netloc.empty()) {
|
||||||
|
if (!url.empty() && url.front() != '/') url.insert(url.begin(), '/');
|
||||||
|
url = "//" + parts.netloc + url;
|
||||||
|
} else if (url.starts_with("//")) {
|
||||||
|
url = "//" + url;
|
||||||
|
} else if ((parts.scheme == "http" || parts.scheme == "https") && (url.empty() || url.front() == '/')) {
|
||||||
|
url = "//" + url;
|
||||||
|
}
|
||||||
|
if (!parts.scheme.empty()) url = parts.scheme + ":" + url;
|
||||||
|
if (!parts.query.empty()) url += "?" + parts.query;
|
||||||
|
if (!parts.fragment.empty()) url += "#" + parts.fragment;
|
||||||
|
return url;
|
||||||
|
}
|
||||||
|
|
||||||
|
// urljoin, for a reference found in a redirect: RFC 3986 resolution the
|
||||||
|
// way Python does it, a `..` that would climb above the root dropped.
|
||||||
|
std::string Join(std::string_view base, std::string_view reference) {
|
||||||
|
if (base.empty()) return std::string(reference);
|
||||||
|
if (reference.empty()) return std::string(base);
|
||||||
|
const SplitURL from = Split(base);
|
||||||
|
SplitURL to = Split(reference);
|
||||||
|
if (to.scheme.empty()) to.scheme = from.scheme;
|
||||||
|
const bool relative = to.scheme == "http" || to.scheme == "https";
|
||||||
|
if (to.scheme != from.scheme || !relative) return std::string(reference);
|
||||||
|
if (!to.netloc.empty()) return Unsplit(to);
|
||||||
|
to.netloc = from.netloc;
|
||||||
|
if (to.path.empty()) {
|
||||||
|
to.path = from.path;
|
||||||
|
if (to.query.empty()) to.query = from.query;
|
||||||
|
return Unsplit(to);
|
||||||
|
}
|
||||||
|
|
||||||
|
std::vector<std::string> baseParts = SplitSlashes(from.path);
|
||||||
|
// The base's last segment is a file, not a directory: it does not
|
||||||
|
// take part in resolving.
|
||||||
|
if (!baseParts.back().empty()) baseParts.pop_back();
|
||||||
|
std::vector<std::string> segments;
|
||||||
|
if (to.path.starts_with('/')) {
|
||||||
|
segments = SplitSlashes(to.path);
|
||||||
|
} else {
|
||||||
|
segments = std::move(baseParts);
|
||||||
|
std::vector<std::string> more = SplitSlashes(to.path);
|
||||||
|
segments.insert(segments.end(), more.begin(), more.end());
|
||||||
|
// Empty segments between the first and the last would come back
|
||||||
|
// as doubled slashes.
|
||||||
|
if (segments.size() > 2) {
|
||||||
|
std::vector<std::string> kept { segments.front() };
|
||||||
|
for (std::size_t i = 1; i + 1 < segments.size(); ++i) {
|
||||||
|
if (!segments[i].empty()) kept.push_back(segments[i]);
|
||||||
|
}
|
||||||
|
kept.push_back(segments.back());
|
||||||
|
segments = std::move(kept);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
std::vector<std::string> resolved;
|
||||||
|
for (const std::string& segment : segments) {
|
||||||
|
if (segment == "..") {
|
||||||
|
if (!resolved.empty()) resolved.pop_back();
|
||||||
|
} else if (segment != ".") {
|
||||||
|
resolved.push_back(segment);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// A reference ending in a directory step keeps its trailing slash.
|
||||||
|
if (segments.back() == "." || segments.back() == "..") resolved.emplace_back();
|
||||||
|
std::string path;
|
||||||
|
for (std::size_t i = 0; i < resolved.size(); ++i) {
|
||||||
|
if (i > 0) path += '/';
|
||||||
|
path += resolved[i];
|
||||||
|
}
|
||||||
|
to.path = path.empty() ? std::string("/") : path;
|
||||||
|
return Unsplit(to);
|
||||||
|
}
|
||||||
|
|
||||||
|
// urllib quotes a Location before it follows it, keeping every ASCII
|
||||||
|
// letter, digit and punctuation mark: what it escapes are spaces,
|
||||||
|
// controls and bytes outside ASCII.
|
||||||
|
std::string QuoteLocation(std::string_view location) {
|
||||||
|
std::string quoted;
|
||||||
|
for (const char c : location) {
|
||||||
|
const unsigned char byte = static_cast<unsigned char>(c);
|
||||||
|
if (byte > 0x20 && byte < 0x7f) {
|
||||||
|
quoted.push_back(c);
|
||||||
|
} else {
|
||||||
|
quoted += std::format("%{:02X}", byte);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return quoted;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FetchTarget {
|
||||||
|
bool secure = false;
|
||||||
|
// Without brackets: what the socket and TLS verification are given.
|
||||||
|
std::string host;
|
||||||
|
std::uint16_t port = 0;
|
||||||
|
// The URL's host[:port] as written, for the Host header.
|
||||||
|
std::string authority;
|
||||||
|
// Path and query: the request target.
|
||||||
|
std::string target;
|
||||||
|
};
|
||||||
|
|
||||||
|
FetchTarget ParseTarget(std::string_view url) {
|
||||||
|
const SplitURL parts = Split(url);
|
||||||
|
if (parts.scheme != "http" && parts.scheme != "https") {
|
||||||
|
throw std::invalid_argument("FetchHTTP1: not an http:// or https:// URL: " + std::string(url));
|
||||||
|
}
|
||||||
|
FetchTarget target;
|
||||||
|
target.secure = parts.scheme == "https";
|
||||||
|
std::string hostinfo = parts.netloc;
|
||||||
|
if (const std::size_t at = hostinfo.rfind('@'); at != std::string::npos) hostinfo.erase(0, at + 1);
|
||||||
|
std::string port;
|
||||||
|
if (hostinfo.starts_with('[')) {
|
||||||
|
const std::size_t close = hostinfo.find(']');
|
||||||
|
target.host = hostinfo.substr(1, close - 1);
|
||||||
|
const std::string rest = hostinfo.substr(close + 1);
|
||||||
|
if (rest.starts_with(':')) {
|
||||||
|
port = rest.substr(1);
|
||||||
|
} else if (!rest.empty()) {
|
||||||
|
throw std::invalid_argument("FetchHTTP1: unexpected text after the IPv6 host in " + std::string(url));
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const std::size_t colon = hostinfo.find(':');
|
||||||
|
target.host = hostinfo.substr(0, colon);
|
||||||
|
if (colon != std::string::npos) port = hostinfo.substr(colon + 1);
|
||||||
|
}
|
||||||
|
if (target.host.empty()) {
|
||||||
|
throw std::invalid_argument("FetchHTTP1: no host in " + std::string(url));
|
||||||
|
}
|
||||||
|
if (port.empty()) {
|
||||||
|
target.port = target.secure ? 443 : 80;
|
||||||
|
} else {
|
||||||
|
std::uint32_t value = 0;
|
||||||
|
const auto [end, error] = std::from_chars(port.data(), port.data() + port.size(), value);
|
||||||
|
if (error != std::errc() || end != port.data() + port.size() || value > 65535) {
|
||||||
|
throw std::invalid_argument("FetchHTTP1: bad port '" + port + "' in " + std::string(url));
|
||||||
|
}
|
||||||
|
target.port = static_cast<std::uint16_t>(value);
|
||||||
|
}
|
||||||
|
target.authority = hostinfo;
|
||||||
|
target.target = parts.path.empty() ? std::string("/") : parts.path;
|
||||||
|
if (!target.target.starts_with('/')) target.target.insert(target.target.begin(), '/');
|
||||||
|
if (!parts.query.empty()) target.target += "?" + parts.query;
|
||||||
|
return target;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::int32_t StatusCode(const HTTPResponse& response) {
|
||||||
|
std::int32_t code = 0;
|
||||||
|
std::from_chars(response.status.data(), response.status.data() + response.status.size(), code);
|
||||||
|
return code;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
FetchResultHTTP1 Crafter::FetchHTTP1(std::string_view url, const FetchOptionsHTTP1& options) {
|
||||||
|
std::string current(url);
|
||||||
|
std::string method = options.method.empty() ? std::string("GET") : options.method;
|
||||||
|
std::unordered_map<std::string, std::string> headers = options.headers;
|
||||||
|
std::string body = options.body;
|
||||||
|
// How often the chain has been sent to each URL: urllib's loop check.
|
||||||
|
std::map<std::string, std::uint32_t> visited;
|
||||||
|
FetchResultHTTP1 result;
|
||||||
|
while (true) {
|
||||||
|
const FetchTarget target = ParseTarget(current);
|
||||||
|
ClientHTTP1 client = target.secure ? ClientHTTP1(target.host, target.port, options.credentials)
|
||||||
|
: ClientHTTP1(target.host, target.port);
|
||||||
|
client.connectTimeout = options.connectTimeout;
|
||||||
|
client.handshakeTimeout = options.handshakeTimeout;
|
||||||
|
client.timeout = options.timeout;
|
||||||
|
client.limits = options.limits;
|
||||||
|
|
||||||
|
HTTPRequest request;
|
||||||
|
request.method = method;
|
||||||
|
request.scheme = target.secure ? "https" : "http";
|
||||||
|
request.authority = target.authority;
|
||||||
|
request.path = target.target;
|
||||||
|
request.headers = headers;
|
||||||
|
request.body = body;
|
||||||
|
result.response = client.Send(request);
|
||||||
|
result.url = current;
|
||||||
|
|
||||||
|
const std::int32_t code = StatusCode(result.response);
|
||||||
|
const bool redirect = code == 301 || code == 302 || code == 303 || code == 307 || code == 308;
|
||||||
|
if (!redirect) return result;
|
||||||
|
// The methods urllib redirects; any other comes back as the redirect.
|
||||||
|
const bool followed = method == "GET" || method == "HEAD" || (method == "POST" && code <= 303);
|
||||||
|
if (!followed) return result;
|
||||||
|
auto location = result.response.headers.find("location");
|
||||||
|
if (location == result.response.headers.end()) location = result.response.headers.find("uri");
|
||||||
|
if (location == result.response.headers.end()) return result;
|
||||||
|
|
||||||
|
std::string next = QuoteLocation(location->second);
|
||||||
|
SplitURL parts = Split(next);
|
||||||
|
if (!parts.scheme.empty() && parts.scheme != "http" && parts.scheme != "https") return result;
|
||||||
|
if (parts.path.empty() && !parts.netloc.empty()) {
|
||||||
|
parts.path = "/";
|
||||||
|
next = Unsplit(parts);
|
||||||
|
}
|
||||||
|
next = Join(current, next);
|
||||||
|
|
||||||
|
const auto seen = visited.find(next);
|
||||||
|
const std::uint32_t times = seen == visited.end() ? 0 : seen->second;
|
||||||
|
if (times >= options.maxRepeats || visited.size() >= options.maxRedirects) {
|
||||||
|
result.limited = true;
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
visited[next] = times + 1;
|
||||||
|
++result.redirects;
|
||||||
|
|
||||||
|
// What urllib sends next: a GET, or a HEAD for a HEAD, without the
|
||||||
|
// body or the headers that described it.
|
||||||
|
if (method != "HEAD") method = "GET";
|
||||||
|
body.clear();
|
||||||
|
headers.erase("content-length");
|
||||||
|
headers.erase("content-type");
|
||||||
|
current = std::move(next);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -87,10 +87,66 @@ namespace Crafter {
|
||||||
// from `timeout` because it covers a multi-round-trip exchange before
|
// from `timeout` because it covers a multi-round-trip exchange before
|
||||||
// any request has been written.
|
// any request has been written.
|
||||||
std::chrono::milliseconds handshakeTimeout{15000};
|
std::chrono::milliseconds handshakeTimeout{15000};
|
||||||
|
// How long the TCP connect may take, for each address the host
|
||||||
|
// resolves to (see ClientTCP). Without it a host that drops the SYNs
|
||||||
|
// held Send() for the kernel's retries, about two minutes, whatever
|
||||||
|
// `timeout` said.
|
||||||
|
std::chrono::milliseconds connectTimeout{15000};
|
||||||
|
|
||||||
private:
|
private:
|
||||||
struct Impl;
|
struct Impl;
|
||||||
std::unique_ptr<Impl> impl;
|
std::unique_ptr<Impl> impl;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// What FetchHTTP1 needs beyond the URL.
|
||||||
|
export struct FetchOptionsHTTP1 {
|
||||||
|
std::string method = "GET";
|
||||||
|
// Lowercase names, as on HTTPRequest. Host comes from the URL.
|
||||||
|
std::unordered_map<std::string, std::string> headers;
|
||||||
|
std::string body;
|
||||||
|
// As on ClientHTTP1, for every request of the chain.
|
||||||
|
std::chrono::milliseconds connectTimeout{15000};
|
||||||
|
std::chrono::milliseconds handshakeTimeout{15000};
|
||||||
|
std::chrono::milliseconds timeout{30000};
|
||||||
|
HTTP1::MessageLimits limits;
|
||||||
|
// For every https:// request of the chain.
|
||||||
|
TLSClientCredentials credentials;
|
||||||
|
// Python urllib's limits: a chain stops at a redirect to a URL it has
|
||||||
|
// already been sent to `maxRepeats` times, or once it has been sent
|
||||||
|
// to `maxRedirects` different URLs.
|
||||||
|
std::uint32_t maxRedirects = 10;
|
||||||
|
std::uint32_t maxRepeats = 4;
|
||||||
|
};
|
||||||
|
|
||||||
|
export struct FetchResultHTTP1 {
|
||||||
|
// The last response of the chain. A redirect only when it had nothing
|
||||||
|
// to follow — no Location, a method urllib does not redirect, a
|
||||||
|
// scheme other than http(s) — or when the limits stopped the chain.
|
||||||
|
HTTPResponse response;
|
||||||
|
// The URL that answered with `response`.
|
||||||
|
std::string url;
|
||||||
|
// Redirects followed to get there.
|
||||||
|
std::uint32_t redirects = 0;
|
||||||
|
// The limits stopped the chain; `response` is the redirect that would
|
||||||
|
// have gone past them.
|
||||||
|
bool limited = false;
|
||||||
|
};
|
||||||
|
|
||||||
|
// One request to an http:// or https:// URL, following redirects the way
|
||||||
|
// Python's urllib does, so a port of urllib code keeps its behaviour:
|
||||||
|
// 301, 302, 303, 307 and 308 are followed for GET and HEAD, the first
|
||||||
|
// three also for POST; the redirected request is a GET (a HEAD stays a
|
||||||
|
// HEAD) without the body and its content headers. A relative Location is
|
||||||
|
// resolved against the URL that sent it, as urljoin resolves it; a
|
||||||
|
// redirect may switch between http:// and https://. Each redirect's body
|
||||||
|
// is read before the next request goes out. Proxies are not consulted.
|
||||||
|
//
|
||||||
|
// The URL is `scheme://[user@]host[:port][/path][?query][#fragment]`,
|
||||||
|
// with an IPv6 host in brackets; the fragment is not sent and the user
|
||||||
|
// part is dropped. Throws std::invalid_argument for a URL that is not
|
||||||
|
// http(s) or has no host or a bad port, and whatever a request throws
|
||||||
|
// (SocketErrorException, SocketTimeoutException, TLSException,
|
||||||
|
// std::runtime_error) otherwise.
|
||||||
|
export FetchResultHTTP1 FetchHTTP1(std::string_view url, const FetchOptionsHTTP1& options = {});
|
||||||
}
|
}
|
||||||
#endif
|
#endif
|
||||||
|
|
|
||||||
|
|
@ -177,6 +177,8 @@ extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> a
|
||||||
cfg.AddTest("ShouldConnectDualStack").Dependencies({ &cfg });
|
cfg.AddTest("ShouldConnectDualStack").Dependencies({ &cfg });
|
||||||
cfg.AddTest("ShouldReportConnectErrors").Dependencies({ &cfg });
|
cfg.AddTest("ShouldReportConnectErrors").Dependencies({ &cfg });
|
||||||
cfg.AddTest("ShouldBindLoopbackEphemeral").Dependencies({ &cfg });
|
cfg.AddTest("ShouldBindLoopbackEphemeral").Dependencies({ &cfg });
|
||||||
|
cfg.AddTest("ShouldTimeoutSilentHTTP1").Dependencies({ &cfg });
|
||||||
|
cfg.AddTest("ShouldFollowRedirectsHTTP1").Dependencies({ &cfg });
|
||||||
cfg.AddTest("ShouldInteropCurlHTTP1").Dependencies({ &cfg });
|
cfg.AddTest("ShouldInteropCurlHTTP1").Dependencies({ &cfg });
|
||||||
cfg.AddTest("ShouldInteropCurlHTTPS1").Dependencies({ &cfg });
|
cfg.AddTest("ShouldInteropCurlHTTPS1").Dependencies({ &cfg });
|
||||||
cfg.AddTest("ShouldParseHTTP1").Dependencies({ &cfg });
|
cfg.AddTest("ShouldParseHTTP1").Dependencies({ &cfg });
|
||||||
|
|
|
||||||
153
tests/ShouldFollowRedirectsHTTP1/main.cpp
Normal file
153
tests/ShouldFollowRedirectsHTTP1/main.cpp
Normal file
|
|
@ -0,0 +1,153 @@
|
||||||
|
//SPDX-License-Identifier: LGPL-3.0-only
|
||||||
|
//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
|
||||||
|
|
||||||
|
// FetchHTTP1 follows redirects the way Python's urllib does, so code ported
|
||||||
|
// from urllib keeps its behaviour:
|
||||||
|
// - a relative Location is resolved against the URL that sent it (urljoin,
|
||||||
|
// `..` included), and a redirect may move from http:// to https://;
|
||||||
|
// - 301/302/303/307/308 are followed for a GET; a POST follows 301/302/303
|
||||||
|
// as a GET without its body, and comes back with a 307 or 308 unfollowed;
|
||||||
|
// - a redirect without a Location comes back as the response;
|
||||||
|
// - the chain stops at the 5th redirect to one URL (4 repeats) or after 10
|
||||||
|
// different URLs, returning that redirect with `limited` set;
|
||||||
|
// - a Location with a space is percent-encoded, as urllib quotes it;
|
||||||
|
// - a URL that is not http(s), has no host or has a bad port is refused.
|
||||||
|
//
|
||||||
|
// Two listeners: plaintext on 8130, TLS on 8131 with a self-signed certificate
|
||||||
|
// the client is handed as its trust anchor.
|
||||||
|
|
||||||
|
#include <cstdlib>
|
||||||
|
|
||||||
|
import Crafter.Network;
|
||||||
|
import std;
|
||||||
|
using namespace Crafter;
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
|
||||||
|
int failures = 0;
|
||||||
|
|
||||||
|
void Check(bool ok, std::string_view what) {
|
||||||
|
std::println("{} {}", ok ? "PASS" : "FAIL", what);
|
||||||
|
if (!ok) ++failures;
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTPResponse Redirect(std::string status, std::string location) {
|
||||||
|
return CreateResponseHTTP(std::move(status), {{"location", std::move(location)}}, std::string());
|
||||||
|
}
|
||||||
|
|
||||||
|
std::unordered_map<std::string, std::function<HTTPResponse(const HTTPRequest&)>> Routes() {
|
||||||
|
std::unordered_map<std::string, std::function<HTTPResponse(const HTTPRequest&)>> routes;
|
||||||
|
routes["/start"] = [](const HTTPRequest&) { return Redirect("302", "next"); };
|
||||||
|
routes["/next"] = [](const HTTPRequest&) { return Redirect("301", "/final?x=1"); };
|
||||||
|
routes["/a/b/start"] = [](const HTTPRequest&) { return Redirect("302", "../c/d"); };
|
||||||
|
routes["/to-https"] = [](const HTTPRequest&) { return Redirect("307", "https://localhost:8131/secure"); };
|
||||||
|
routes["/loop"] = [](const HTTPRequest&) { return Redirect("302", "/loop"); };
|
||||||
|
routes["/nowhere"] = [](const HTTPRequest&) { return CreateResponseHTTP("302", std::string("no location")); };
|
||||||
|
routes["/form"] = [](const HTTPRequest&) { return Redirect("303", "/method"); };
|
||||||
|
routes["/keep"] = [](const HTTPRequest&) { return Redirect("307", "/method"); };
|
||||||
|
routes["/method"] = [](const HTTPRequest& request) {
|
||||||
|
return CreateResponseHTTP("200", request.method + " body=" + request.body);
|
||||||
|
};
|
||||||
|
routes["/spaced"] = [](const HTTPRequest&) { return Redirect("302", "/with space"); };
|
||||||
|
return routes;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Everything else echoes its target, and /hop/<n> redirects to /hop/<n+1>.
|
||||||
|
HTTPResponse Fallback(const HTTPRequest& request) {
|
||||||
|
if (request.path.starts_with("/hop/")) {
|
||||||
|
const std::int32_t next = std::stoi(request.path.substr(5)) + 1;
|
||||||
|
return Redirect("302", std::format("/hop/{}", next));
|
||||||
|
}
|
||||||
|
return CreateResponseHTTP("200", request.path);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool Refused(std::string_view url) {
|
||||||
|
try {
|
||||||
|
FetchHTTP1(url);
|
||||||
|
} catch (const std::invalid_argument&) {
|
||||||
|
return true;
|
||||||
|
} catch (const std::exception&) {
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace
|
||||||
|
|
||||||
|
int main() {
|
||||||
|
try {
|
||||||
|
ListenerAsyncHTTP1 plain(8130, Routes(), Fallback);
|
||||||
|
std::unordered_map<std::string, std::function<HTTPResponse(const HTTPRequest&)>> secureRoutes;
|
||||||
|
secureRoutes["/secure"] = [](const HTTPRequest& request) {
|
||||||
|
return CreateResponseHTTP("200", "secure over " + request.scheme);
|
||||||
|
};
|
||||||
|
ListenerAsyncHTTP1 secure(8131, secureRoutes, TLSServerCredentials { .selfSigned = true });
|
||||||
|
|
||||||
|
{
|
||||||
|
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/start");
|
||||||
|
Check(result.response.status == "200" && result.response.body == "/final?x=1",
|
||||||
|
std::format("a relative chain arrives at /final?x=1 (got {} '{}')", result.response.status, result.response.body));
|
||||||
|
Check(result.url == "http://127.0.0.1:8130/final?x=1", std::format("the URL that answered is reported ({})", result.url));
|
||||||
|
Check(result.redirects == 2 && !result.limited, std::format("two redirects, not limited (got {})", result.redirects));
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/a/b/start");
|
||||||
|
Check(result.response.body == "/a/c/d", std::format("../c/d from /a/b/start resolves to /a/c/d (got '{}')", result.response.body));
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
FetchOptionsHTTP1 options;
|
||||||
|
options.credentials.caPem = GetSelfSignedCertificatePem().certificate;
|
||||||
|
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/to-https", options);
|
||||||
|
Check(result.response.status == "200" && result.response.body == "secure over https",
|
||||||
|
std::format("a redirect from http:// to https:// is followed (got {} '{}')", result.response.status, result.response.body));
|
||||||
|
Check(result.url == "https://localhost:8131/secure", std::format("and lands on the https:// URL ({})", result.url));
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/loop");
|
||||||
|
Check(result.limited && result.response.status == "302" && result.redirects == 4,
|
||||||
|
std::format("a redirect loop stops after 4 repeats (limited {}, {} redirects)", result.limited, result.redirects));
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/hop/0");
|
||||||
|
Check(result.limited && result.redirects == 10 && result.url == "http://127.0.0.1:8130/hop/10",
|
||||||
|
std::format("an endless chain stops after 10 different URLs ({} redirects, at {})", result.redirects, result.url));
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/nowhere");
|
||||||
|
Check(result.response.status == "302" && result.redirects == 0 && !result.limited,
|
||||||
|
"a redirect without a Location comes back as the response");
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
FetchOptionsHTTP1 post;
|
||||||
|
post.method = "POST";
|
||||||
|
post.body = "payload";
|
||||||
|
post.headers["content-type"] = "text/plain";
|
||||||
|
const FetchResultHTTP1 seeOther = FetchHTTP1("http://127.0.0.1:8130/form", post);
|
||||||
|
Check(seeOther.response.body == "GET body=",
|
||||||
|
std::format("a POST answered with 303 is followed as a GET without its body (got '{}')", seeOther.response.body));
|
||||||
|
const FetchResultHTTP1 kept = FetchHTTP1("http://127.0.0.1:8130/keep", post);
|
||||||
|
Check(kept.response.status == "307" && kept.redirects == 0, "a POST answered with 307 comes back unfollowed");
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/spaced");
|
||||||
|
Check(result.response.body == "/with%20space",
|
||||||
|
std::format("a Location with a space is percent-encoded before it is followed (got '{}')", result.response.body));
|
||||||
|
}
|
||||||
|
|
||||||
|
Check(Refused("ftp://127.0.0.1/file"), "an ftp:// URL is refused");
|
||||||
|
Check(Refused("http:///path"), "a URL without a host is refused");
|
||||||
|
Check(Refused("http://127.0.0.1:99999/"), "a port above 65535 is refused");
|
||||||
|
Check(Refused("http://127.0.0.1:8x/"), "a port that is not a number is refused");
|
||||||
|
} catch (const std::exception& e) {
|
||||||
|
Check(false, std::format("unexpected exception: {}", e.what()));
|
||||||
|
}
|
||||||
|
|
||||||
|
std::println("{}", failures == 0 ? "all checks passed" : "check(s) failed");
|
||||||
|
return failures == 0 ? EXIT_SUCCESS : EXIT_FAILURE;
|
||||||
|
}
|
||||||
109
tests/ShouldTimeoutSilentHTTP1/main.cpp
Normal file
109
tests/ShouldTimeoutSilentHTTP1/main.cpp
Normal file
|
|
@ -0,0 +1,109 @@
|
||||||
|
//SPDX-License-Identifier: LGPL-3.0-only
|
||||||
|
//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
|
||||||
|
|
||||||
|
// ClientHTTP1 bounds the TCP connect (connectTimeout) as well as every wait
|
||||||
|
// for a response (timeout). Before, the connect had no bound of its own: a
|
||||||
|
// host that drops the SYNs held Send() for the kernel's retries, about two
|
||||||
|
// minutes, whatever `timeout` said — a readiness probe with a ten-second
|
||||||
|
// budget could hang its caller for two minutes.
|
||||||
|
//
|
||||||
|
// Two servers on loopback. One accepts — the kernel completes the handshake
|
||||||
|
// from its backlog — and never answers: the read timeout. The other has a
|
||||||
|
// full accept queue, so Linux drops the SYNs of further connects: the connect
|
||||||
|
// timeout.
|
||||||
|
|
||||||
|
#include <arpa/inet.h>
|
||||||
|
#include <netinet/in.h>
|
||||||
|
#include <sys/socket.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
#include <cstdlib>
|
||||||
|
|
||||||
|
import Crafter.Network;
|
||||||
|
import std;
|
||||||
|
using namespace Crafter;
|
||||||
|
using namespace std::chrono_literals;
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
|
||||||
|
int failures = 0;
|
||||||
|
|
||||||
|
void Check(bool ok, std::string_view what) {
|
||||||
|
std::println("{} {}", ok ? "PASS" : "FAIL", what);
|
||||||
|
if (!ok) ++failures;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A socket listening on 127.0.0.1, on a port the kernel picks.
|
||||||
|
struct LoopbackListener {
|
||||||
|
int fd = -1;
|
||||||
|
std::uint16_t port = 0;
|
||||||
|
|
||||||
|
explicit LoopbackListener(int backlog) {
|
||||||
|
fd = socket(AF_INET, SOCK_STREAM, 0);
|
||||||
|
sockaddr_in address {};
|
||||||
|
address.sin_family = AF_INET;
|
||||||
|
address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
|
||||||
|
bind(fd, reinterpret_cast<sockaddr*>(&address), sizeof(address));
|
||||||
|
listen(fd, backlog);
|
||||||
|
socklen_t length = sizeof(address);
|
||||||
|
getsockname(fd, reinterpret_cast<sockaddr*>(&address), &length);
|
||||||
|
port = ntohs(address.sin_port);
|
||||||
|
}
|
||||||
|
~LoopbackListener() { close(fd); }
|
||||||
|
};
|
||||||
|
|
||||||
|
long long Milliseconds(std::chrono::steady_clock::duration duration) {
|
||||||
|
return std::chrono::duration_cast<std::chrono::milliseconds>(duration).count();
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace
|
||||||
|
|
||||||
|
int main() {
|
||||||
|
{
|
||||||
|
LoopbackListener silent(8);
|
||||||
|
ClientHTTP1 client("127.0.0.1", silent.port);
|
||||||
|
client.timeout = 300ms;
|
||||||
|
const auto start = std::chrono::steady_clock::now();
|
||||||
|
bool failed = false;
|
||||||
|
try {
|
||||||
|
client.Send(CreateRequestHTTP("GET", "/", "127.0.0.1"));
|
||||||
|
} catch (const std::exception&) {
|
||||||
|
failed = true;
|
||||||
|
}
|
||||||
|
const auto waited = std::chrono::steady_clock::now() - start;
|
||||||
|
Check(failed, "a server that accepts and never answers fails the request");
|
||||||
|
Check(waited >= 280ms && waited < 5s, std::format("after about `timeout` ({} ms)", Milliseconds(waited)));
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
// Connects queue up until the accept queue is full, which with a
|
||||||
|
// backlog of 0 is after one or two; the next one goes unanswered.
|
||||||
|
LoopbackListener full(0);
|
||||||
|
std::vector<ClientTCP> queued;
|
||||||
|
bool queueFull = false;
|
||||||
|
for (int i = 0; i < 8 && !queueFull; ++i) {
|
||||||
|
try {
|
||||||
|
queued.emplace_back("127.0.0.1", full.port, 200ms);
|
||||||
|
} catch (const SocketTimeoutException&) {
|
||||||
|
queueFull = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Check(queueFull, std::format("the accept queue fills up (after {} connects)", queued.size()));
|
||||||
|
|
||||||
|
ClientHTTP1 client("127.0.0.1", full.port);
|
||||||
|
client.connectTimeout = 300ms;
|
||||||
|
const auto start = std::chrono::steady_clock::now();
|
||||||
|
bool timedOut = false;
|
||||||
|
try {
|
||||||
|
client.Send(CreateRequestHTTP("GET", "/", "127.0.0.1"));
|
||||||
|
} catch (const SocketTimeoutException&) {
|
||||||
|
timedOut = true;
|
||||||
|
} catch (const std::exception&) {
|
||||||
|
}
|
||||||
|
const auto waited = std::chrono::steady_clock::now() - start;
|
||||||
|
Check(timedOut, "a connect nobody answers throws SocketTimeoutException");
|
||||||
|
Check(waited >= 280ms && waited < 5s, std::format("after about `connectTimeout` ({} ms)", Milliseconds(waited)));
|
||||||
|
}
|
||||||
|
|
||||||
|
std::println("{}", failures == 0 ? "all checks passed" : "check(s) failed");
|
||||||
|
return failures == 0 ? EXIT_SUCCESS : EXIT_FAILURE;
|
||||||
|
}
|
||||||
Loading…
Reference in a new issue