diff --git a/AGENTS.consumer.md b/AGENTS.consumer.md index 0e70b41..ae8486e 100644 --- a/AGENTS.consumer.md +++ b/AGENTS.consumer.md @@ -13,7 +13,7 @@ The Crafter engine's networking library: a C++ module, `import Crafter.Network;` |---|---|---|---| | `HTTPRequest`, `HTTPResponse`, `CreateRequestHTTP`, `CreateResponseHTTP`, `PathWithoutQueryHTTP` | yes | yes | yes | | `ClientTCP`, `ListenerTCP` | yes | yes | no | -| `ByteStream`, `PlainStream`, `TLSContext`, `TLSStream`, `HTTP1::*`, `ClientHTTP1`, `ListenerHTTP1` | yes | yes | no | +| `ByteStream`, `PlainStream`, `TLSContext`, `TLSStream`, `HTTP1::*`, `URL`, `ClientHTTP1`, `ListenerHTTP1` | yes | yes | no | | `ClientQUIC`, `QUICStream` | yes | no | `*Async` only, over `WebTransport` | | `ClientHTTP` | yes | no | `SendAsync` only, over `fetch()` | | `ListenerQUIC`, `ListenerHTTP`, `WebTransportSession`, `HTTP3::*` | yes | no | no | @@ -43,6 +43,8 @@ A route map is `std::unordered_map.cppm` | One partition per file. The native-only ones wrap their whole body in `#ifndef CRAFTER_NETWORK_BROWSER`, so the browser build compiles them as empty modules. | -| `implementations/Crafter.Network-.cpp` | The native implementation units, `Crafter.Network:_impl`. `HTTP`, `HTTP1` and `HTTP3` have none: they are inline. | +| `implementations/Crafter.Network-.cpp` | The native implementation units, `Crafter.Network:_impl`. `HTTP`, `HTTP1`, `HTTP3` and `URL` have none: they are inline. | | `implementations/Crafter.Network-ClientHTTP-Browser.cpp`, `-ClientQUIC-Browser.cpp` | The browser implementations of those two partitions, over the JS bridge. | | `additional/network-env.js` | The JS bridge: `fetch()` and `WebTransport` behind the `env` imports the wasm declares, shipped next to a consumer's `.wasm` through `cfg.files`. | | `project.cpp` | The build description: one static library, `crafter-network`, in its three variants, the msquic external, the OpenSSL link flags and the test declarations. | @@ -25,8 +25,9 @@ Partitions and what they own: - `:Stream`: `ByteStream` (`ReadSome` and `Write` with a timeout per call, `Shutdown`, `Descriptor`, `Protocol`, `Secure`) and `PlainStream`, so one HTTP/1.1 implementation serves `http://` and `https://`. `SetNonBlocking`, `PollDescriptor`. - `:TLS`: `TLSContext`, `TLSStream`, both credential structs, `GetSelfSignedCertificatePem`. OpenSSL stays behind the `Impl` pointers; no OpenSSL header reaches an importer. - `:HTTP`: `HTTPRequest`, `HTTPResponse`, the `Create*` helpers, `PathWithoutQueryHTTP`. Shared by both HTTP versions and by every build. +- `:URL`: `URL`, an absolute URL as RFC 3986 reads it (`Parse`), the resolution of a reference against it (`Resolve`, RFC 3986 §5.2, dot segments removed), the recomposition, and the pieces a request is built from. Inline, native only; what `ClientHTTP1::Fetch` follows a Location with. - `:HTTP1`: the RFC 9112 codec, inline: `SerializeRequest`, `SerializeResponse`, `SerializeContinue`, `MessageParser`, `MessageLimits`, `HTTP1ProtocolError`, and the framing it refuses to guess at. -- `:ClientHTTP1`, `:ListenerHTTP1`: HTTP/1.1 over a `ByteStream&`. The listener is a `ListenerTCP` with a thread per accepted connection; the client keeps one pooled connection with a replay-once rule. +- `:ClientHTTP1`, `:ListenerHTTP1`: HTTP/1.1 over a `ByteStream&`. The listener is a `ListenerTCP` with a thread per accepted connection; the client keeps one pooled connection with a replay-once rule, and `ClientHTTP1::Fetch` is the URL fetch that follows redirects by RFC 9110 §15.4 with a client per hop (`FetchOptions`, `FetchResult`, `FetchStop`). - `:ClientQUIC`: `ClientQUIC` and `QUICStream` for both ends of a connection, `QUICSettings`, both QUIC credential structs, `QUICException`, `QUICClosedException`. The native unit holds the process-wide `MsQuicRuntime` (API table and registration, closed by a static destructor at exit), the stream registry with its refcounted handles, and the drain in `~ClientQUIC`. - `:ListenerQUIC`: the msquic listener, its four `Listen*` variants, the self-signed certificate (made by the `openssl` CLI and cached under `/tmp/crafter-network-quic-cert/`), `ComputeCertificateHashSHA256`, `GetSelfSignedCertificatePath`. - `:HTTP3`: QUIC varints, HTTP/3 frames, the QPACK static table, Huffman decoding, the control-stream preludes and the WebTransport stream prefixes. Native only: it throws for protocol errors. @@ -50,9 +51,9 @@ crafter-build # native: TCP, TLS, HTTP/1.1, QUI crafter-build --no-quic # native without QUIC, HTTP/3, WebTransport and msquic crafter-build --local # Crafter.Thread from ../Crafter.Thread crafter-build --target=wasm32-wasip1 # the browser library -crafter-build test # the suite, native: 23 tests +crafter-build test # the suite, native: 30 tests crafter-build test 'Should*HTTP1' # a name glob -crafter-build test --no-quic # the TCP, TLS and HTTP/1.1 tests plus ShouldBuildWithoutQUIC: 12 +crafter-build test --no-quic # the TCP, URL, TLS and HTTP/1.1 tests plus ShouldBuildWithoutQUIC: 19 crafter-build test --list crafter-build test --jobs=1 --timeout=120 crafter-build clean diff --git a/README.md b/README.md index 274f1c1..ece27d0 100644 --- a/README.md +++ b/README.md @@ -134,20 +134,22 @@ The connection is persistent: the first `Send()` dials, and later calls reuse th Three timeouts bound a request: `connectTimeout` (default 15 s) for the TCP connect, per address the host resolves to; `handshakeTimeout` (default 15 s) for the TLS handshake; and `timeout` (default 30 s) for each wait on the next piece of a response. -#### FetchHTTP1 +#### ClientHTTP1::Fetch -`FetchHTTP1(url, options)` is one request to an `http://` or `https://` URL, following redirects the way Python's urllib does, so code ported from urllib keeps its behaviour: +`ClientHTTP1::Fetch(url, options)` is one request to an `http://` or `https://` URL, following redirects by RFC 9110 §15.4: ```cpp -Crafter::FetchOptionsHTTP1 options; +Crafter::FetchOptions options; options.timeout = std::chrono::seconds(10); options.headers["accept-encoding"] = "identity"; -Crafter::FetchResultHTTP1 result = Crafter::FetchHTTP1("http://localhost:8080/", options); -// result.response: the last response; result.url: the URL that sent it; -// result.redirects: how many were followed; result.limited: the limits stopped the chain. +Crafter::FetchResult result = Crafter::ClientHTTP1::Fetch("http://localhost:8080/", options); +// result.response: the last response; result.url: the URL that sent it (a Crafter::URL); +// result.redirects: how many were followed; result.stop: why a redirect was not followed, or None. ``` -301, 302, 303, 307 and 308 are followed for a GET or HEAD, and 301/302/303 also for a POST; the redirected request is a GET (a HEAD stays a HEAD) without the body or its content headers. A relative `Location` is resolved against the URL that sent it, as `urljoin` resolves it, and is percent-encoded the way urllib quotes it; a redirect may move between `http://` and `https://` (`options.credentials` applies to every TLS hop). Each redirect's body is read before the next request goes out. A redirect is returned unfollowed when it has no `Location`, points at another scheme, or comes for a method urllib does not redirect. The chain stops, returning the redirect with `limited` set, at the fifth redirect to one URL (`maxRepeats`, 4) or once ten different URLs have been visited (`maxRedirects`, 10). Proxies are not consulted. A URL that is not `http(s)://`, has no host or a bad port throws `std::invalid_argument`. +301, 302, 303, 307 and 308 are followed for every method: a 303 turns the request into a GET (a HEAD stays a HEAD) without its body and content headers, a 301 or 302 does the same for a POST alone, and a 307 or 308 keeps the method and the body. A relative `Location` is resolved against the URL that sent it (RFC 3986, through `Crafter::URL::Resolve`), and a redirect may move between `http://` and `https://` (`options.credentials` applies to every TLS hop). `Authorization`, `Proxy-Authorization` and `Cookie` are not carried to another origin. Each hop is a client of its own, so a redirect's body is read and its connection closed before the next request goes out. A redirect comes back as the response, with `result.stop` saying why, when it has no usable `Location`, points at another scheme, points back at a URL the chain already visited, or would be the eleventh (`maxRedirects`, 10). Proxies are not consulted. A URL that is not `http(s)://`, has no host or has a bad port throws `std::invalid_argument`. + +`Crafter::URL` is the URL type behind it, exported for callers that build or resolve URLs themselves: `URL::Parse(text)`, `url.Resolve(reference)`, `url.ToString()`, and the pieces a request is built from (`Authority()`, `Target()`, `PortOrDefault()`, `Secure()`, `SameOrigin(other)`). #### ListenerHTTP1 ```cpp @@ -331,7 +333,7 @@ The library includes tests covering: - Send and receive errors (`ShouldReportIoErrors`) — a connection the peer resets fails the next receive, and a send into it, with `SocketErrorException` carrying ECONNRESET or EPIPE under the plain strerror text a receive and a send always threw - Loopback listeners (`ShouldBindLoopbackEphemeral`) — a listener on `127.0.0.1` at port 0 reports its port, takes loopback clients and refuses them through the host's other address; `::1` works too, a name is refused, and the port-only constructor still listens on every interface - HTTP/1.1 timeouts (`ShouldTimeoutSilentHTTP1`) — a server that accepts and never answers fails after `timeout`, and a connect nobody answers after `connectTimeout` -- Redirects (`ShouldFollowRedirectsHTTP1`) — relative and `..` Locations, an http-to-https hop, the 4-repeat and 10-URL limits, a redirect without Location, POST after 303 and 307, percent-encoding a Location, and the URLs `FetchHTTP1` refuses +- Redirects (`ShouldFollowRedirectsHTTP1`) — relative and `..` Locations, an http-to-https hop, a loop and the redirect limit, a redirect without Location or to another scheme, a POST after 303, 302 and 307 and a PUT after 302, credentials dropped across origins, and the URLs `Fetch` refuses; `ShouldParseURLs` runs RFC 3986 §5.4's resolution examples and the parse rules The external-interop test requires outbound UDP/443; if your network blocks it the test will fail. `ShouldInteropCurlHTTP1` and `ShouldInteropCurlHTTPS1` skip whichever half is unavailable when `curl` or `python3` is not installed, and the mutual-TLS half of `ShouldRequireClientCertificateHTTPS1` skips without the `openssl` CLI (a *client* certificate needs the `clientAuth` extended key usage, which the built-in development certificate does not carry) — so all three pass on a bare machine. Install `curl`, `python3` and `openssl` to actually exercise them. diff --git a/implementations/Crafter.Network-ClientHTTP1.cpp b/implementations/Crafter.Network-ClientHTTP1.cpp index 5e43633..2ecb6bb 100644 --- a/implementations/Crafter.Network-ClientHTTP1.cpp +++ b/implementations/Crafter.Network-ClientHTTP1.cpp @@ -11,6 +11,7 @@ import :HTTP; import :HTTP1; import :Stream; import :TLS; +import :URL; import Crafter.Thread; import std; @@ -180,254 +181,35 @@ void ClientHTTP1::SendAsync(const HTTPRequest& request, } namespace { - // The five parts urllib.parse.urlsplit cuts a URL into. FetchHTTP1 needs - // the splitting, the joining (urljoin) and the reassembly to resolve a - // Location the way urllib does, and nothing else of urllib.parse. - struct SplitURL { - std::string scheme; - std::string netloc; - std::string path; - std::string query; - std::string fragment; - }; - - bool IsAsciiAlpha(char c) { - return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); - } - - bool IsSchemeChar(char c) { - return IsAsciiAlpha(c) || (c >= '0' && c <= '9') || c == '+' || c == '-' || c == '.'; - } - - std::string ToLowerAsciiCopy(std::string_view text) { - std::string lower(text); - for (char& c : lower) { - if (c >= 'A' && c <= 'Z') c = static_cast(c - 'A' + 'a'); - } - return lower; - } - - // str.split('/'): every piece, empty ones included, and [""] for "". - std::vector SplitSlashes(std::string_view text) { - std::vector pieces; - std::size_t start = 0; - while (true) { - const std::size_t slash = text.find('/', start); - if (slash == std::string_view::npos) { - pieces.emplace_back(text.substr(start)); - return pieces; - } - pieces.emplace_back(text.substr(start, slash - start)); - start = slash + 1; - } - } - - // urlsplit: tabs and newlines dropped anywhere, leading controls and - // spaces stripped, the scheme lowercased. - SplitURL Split(std::string_view input) { - std::string url; - url.reserve(input.size()); - for (const char c : input) { - if (c != '\t' && c != '\r' && c != '\n') url.push_back(c); - } - std::size_t start = 0; - while (start < url.size() && static_cast(url[start]) <= 0x20) ++start; - url.erase(0, start); - - SplitURL parts; - const std::size_t colon = url.find(':'); - if (colon != std::string::npos && colon > 0 && IsAsciiAlpha(url[0]) - && std::ranges::all_of(std::string_view(url).substr(0, colon), IsSchemeChar)) { - parts.scheme = ToLowerAsciiCopy(std::string_view(url).substr(0, colon)); - url.erase(0, colon + 1); - } - if (url.starts_with("//")) { - std::size_t end = url.find_first_of("/?#", 2); - if (end == std::string::npos) end = url.size(); - parts.netloc = url.substr(2, end - 2); - url.erase(0, end); - if (parts.netloc.contains('[') != parts.netloc.contains(']')) { - throw std::invalid_argument("FetchHTTP1: unbalanced brackets in the host of " + std::string(input)); - } - } - if (const std::size_t hash = url.find('#'); hash != std::string::npos) { - parts.fragment = url.substr(hash + 1); - url.erase(hash); - } - if (const std::size_t question = url.find('?'); question != std::string::npos) { - parts.query = url.substr(question + 1); - url.erase(question); - } - parts.path = std::move(url); - return parts; - } - - // urlunsplit, for the http(s) URLs FetchHTTP1 puts back together. - std::string Unsplit(const SplitURL& parts) { - std::string url = parts.path; - if (!parts.netloc.empty()) { - if (!url.empty() && url.front() != '/') url.insert(url.begin(), '/'); - url = "//" + parts.netloc + url; - } else if (url.starts_with("//")) { - url = "//" + url; - } else if ((parts.scheme == "http" || parts.scheme == "https") && (url.empty() || url.front() == '/')) { - url = "//" + url; - } - if (!parts.scheme.empty()) url = parts.scheme + ":" + url; - if (!parts.query.empty()) url += "?" + parts.query; - if (!parts.fragment.empty()) url += "#" + parts.fragment; - return url; - } - - // urljoin, for a reference found in a redirect: RFC 3986 resolution the - // way Python does it, a `..` that would climb above the root dropped. - std::string Join(std::string_view base, std::string_view reference) { - if (base.empty()) return std::string(reference); - if (reference.empty()) return std::string(base); - const SplitURL from = Split(base); - SplitURL to = Split(reference); - if (to.scheme.empty()) to.scheme = from.scheme; - const bool relative = to.scheme == "http" || to.scheme == "https"; - if (to.scheme != from.scheme || !relative) return std::string(reference); - if (!to.netloc.empty()) return Unsplit(to); - to.netloc = from.netloc; - if (to.path.empty()) { - to.path = from.path; - if (to.query.empty()) to.query = from.query; - return Unsplit(to); - } - - std::vector baseParts = SplitSlashes(from.path); - // The base's last segment is a file, not a directory: it does not - // take part in resolving. - if (!baseParts.back().empty()) baseParts.pop_back(); - std::vector segments; - if (to.path.starts_with('/')) { - segments = SplitSlashes(to.path); - } else { - segments = std::move(baseParts); - std::vector more = SplitSlashes(to.path); - segments.insert(segments.end(), more.begin(), more.end()); - // Empty segments between the first and the last would come back - // as doubled slashes. - if (segments.size() > 2) { - std::vector kept { segments.front() }; - for (std::size_t i = 1; i + 1 < segments.size(); ++i) { - if (!segments[i].empty()) kept.push_back(segments[i]); - } - kept.push_back(segments.back()); - segments = std::move(kept); - } - } - std::vector resolved; - for (const std::string& segment : segments) { - if (segment == "..") { - if (!resolved.empty()) resolved.pop_back(); - } else if (segment != ".") { - resolved.push_back(segment); - } - } - // A reference ending in a directory step keeps its trailing slash. - if (segments.back() == "." || segments.back() == "..") resolved.emplace_back(); - std::string path; - for (std::size_t i = 0; i < resolved.size(); ++i) { - if (i > 0) path += '/'; - path += resolved[i]; - } - to.path = path.empty() ? std::string("/") : path; - return Unsplit(to); - } - - // urllib quotes a Location before it follows it, keeping every ASCII - // letter, digit and punctuation mark: what it escapes are spaces, - // controls and bytes outside ASCII. - std::string QuoteLocation(std::string_view location) { - std::string quoted; - for (const char c : location) { - const unsigned char byte = static_cast(c); - if (byte > 0x20 && byte < 0x7f) { - quoted.push_back(c); - } else { - quoted += std::format("%{:02X}", byte); - } - } - return quoted; - } - - struct FetchTarget { - bool secure = false; - // Without brackets: what the socket and TLS verification are given. - std::string host; - std::uint16_t port = 0; - // The URL's host[:port] as written, for the Host header. - std::string authority; - // Path and query: the request target. - std::string target; - }; - - FetchTarget ParseTarget(std::string_view url) { - const SplitURL parts = Split(url); - if (parts.scheme != "http" && parts.scheme != "https") { - throw std::invalid_argument("FetchHTTP1: not an http:// or https:// URL: " + std::string(url)); - } - FetchTarget target; - target.secure = parts.scheme == "https"; - std::string hostinfo = parts.netloc; - if (const std::size_t at = hostinfo.rfind('@'); at != std::string::npos) hostinfo.erase(0, at + 1); - std::string port; - if (hostinfo.starts_with('[')) { - const std::size_t close = hostinfo.find(']'); - target.host = hostinfo.substr(1, close - 1); - const std::string rest = hostinfo.substr(close + 1); - if (rest.starts_with(':')) { - port = rest.substr(1); - } else if (!rest.empty()) { - throw std::invalid_argument("FetchHTTP1: unexpected text after the IPv6 host in " + std::string(url)); - } - } else { - const std::size_t colon = hostinfo.find(':'); - target.host = hostinfo.substr(0, colon); - if (colon != std::string::npos) port = hostinfo.substr(colon + 1); - } - if (target.host.empty()) { - throw std::invalid_argument("FetchHTTP1: no host in " + std::string(url)); - } - if (port.empty()) { - target.port = target.secure ? 443 : 80; - } else { - std::uint32_t value = 0; - const auto [end, error] = std::from_chars(port.data(), port.data() + port.size(), value); - if (error != std::errc() || end != port.data() + port.size() || value > 65535) { - throw std::invalid_argument("FetchHTTP1: bad port '" + port + "' in " + std::string(url)); - } - target.port = static_cast(value); - } - target.authority = hostinfo; - target.target = parts.path.empty() ? std::string("/") : parts.path; - if (!target.target.starts_with('/')) target.target.insert(target.target.begin(), '/'); - if (!parts.query.empty()) target.target += "?" + parts.query; - return target; - } - std::int32_t StatusCode(const HTTPResponse& response) { std::int32_t code = 0; std::from_chars(response.status.data(), response.status.data() + response.status.size(), code); return code; } + + bool IsRedirect(std::int32_t code) { + return code == 301 || code == 302 || code == 303 || code == 307 || code == 308; + } } -FetchResultHTTP1 Crafter::FetchHTTP1(std::string_view url, const FetchOptionsHTTP1& options) { - std::string current(url); +FetchResult ClientHTTP1::Fetch(std::string_view url, const FetchOptions& options) { + URL current = URL::Parse(url); + if (current.scheme != "http" && current.scheme != "https") { + throw std::invalid_argument("ClientHTTP1::Fetch: not an http:// or https:// URL: " + std::string(url)); + } std::string method = options.method.empty() ? std::string("GET") : options.method; std::unordered_map headers = options.headers; std::string body = options.body; - // How often the chain has been sent to each URL: urllib's loop check. - std::map visited; - FetchResultHTTP1 result; + // Every URL the chain has been sent to: a redirect back to one of them + // would loop. + std::set visited { current.ToString() }; + FetchResult result; while (true) { - const FetchTarget target = ParseTarget(current); - ClientHTTP1 client = target.secure ? ClientHTTP1(target.host, target.port, options.credentials) - : ClientHTTP1(target.host, target.port); + // A client per hop: one ClientHTTP1 is one origin, and a redirect + // may leave it. Destroying the client closes the hop's connection + // once its response, body included, has been read. + ClientHTTP1 client = current.Secure() ? ClientHTTP1(current.host, current.PortOrDefault(), options.credentials) + : ClientHTTP1(current.host, current.PortOrDefault()); client.connectTimeout = options.connectTimeout; client.handshakeTimeout = options.handshakeTimeout; client.timeout = options.timeout; @@ -435,48 +217,59 @@ FetchResultHTTP1 Crafter::FetchHTTP1(std::string_view url, const FetchOptionsHTT HTTPRequest request; request.method = method; - request.scheme = target.secure ? "https" : "http"; - request.authority = target.authority; - request.path = target.target; + request.scheme = current.scheme; + request.authority = current.Authority(); + request.path = current.Target(); request.headers = headers; request.body = body; result.response = client.Send(request); result.url = current; const std::int32_t code = StatusCode(result.response); - const bool redirect = code == 301 || code == 302 || code == 303 || code == 307 || code == 308; - if (!redirect) return result; - // The methods urllib redirects; any other comes back as the redirect. - const bool followed = method == "GET" || method == "HEAD" || (method == "POST" && code <= 303); - if (!followed) return result; - auto location = result.response.headers.find("location"); - if (location == result.response.headers.end()) location = result.response.headers.find("uri"); - if (location == result.response.headers.end()) return result; - - std::string next = QuoteLocation(location->second); - SplitURL parts = Split(next); - if (!parts.scheme.empty() && parts.scheme != "http" && parts.scheme != "https") return result; - if (parts.path.empty() && !parts.netloc.empty()) { - parts.path = "/"; - next = Unsplit(parts); - } - next = Join(current, next); - - const auto seen = visited.find(next); - const std::uint32_t times = seen == visited.end() ? 0 : seen->second; - if (times >= options.maxRepeats || visited.size() >= options.maxRedirects) { - result.limited = true; + if (!IsRedirect(code)) return result; + const auto location = result.response.headers.find("location"); + if (location == result.response.headers.end()) { + result.stop = FetchStop::NoLocation; + return result; + } + URL next; + try { + next = current.Resolve(location->second); + } catch (const std::invalid_argument&) { + result.stop = FetchStop::NoLocation; + return result; + } + if (next.scheme != "http" && next.scheme != "https") { + result.stop = FetchStop::OtherScheme; + return result; + } + if (result.redirects >= options.maxRedirects) { + result.stop = FetchStop::Limit; + return result; + } + if (!visited.insert(next.ToString()).second) { + result.stop = FetchStop::Loop; return result; } - visited[next] = times + 1; ++result.redirects; - // What urllib sends next: a GET, or a HEAD for a HEAD, without the - // body or the headers that described it. - if (method != "HEAD") method = "GET"; - body.clear(); - headers.erase("content-length"); - headers.erase("content-type"); + // RFC 9110 §15.4: a 303 asks for a GET of the Location; a 301 or 302 + // keeps the method, but a POST becomes a GET, as the user agents the + // sections describe have always done; a 307 or 308 keeps the method + // and the body. A request that turned into a GET loses the body and + // the headers that described it. + const bool toGet = (code == 303 && method != "HEAD") || ((code == 301 || code == 302) && method == "POST"); + if (toGet) { + method = "GET"; + body.clear(); + std::erase_if(headers, [](const auto& header) { return header.first.starts_with("content-"); }); + } + // Credentials stay with the origin they were meant for. + if (!next.SameOrigin(current)) { + headers.erase("authorization"); + headers.erase("proxy-authorization"); + headers.erase("cookie"); + } current = std::move(next); } } diff --git a/interfaces/Crafter.Network-ClientHTTP1.cppm b/interfaces/Crafter.Network-ClientHTTP1.cppm index b5cb7b8..2f73783 100644 --- a/interfaces/Crafter.Network-ClientHTTP1.cppm +++ b/interfaces/Crafter.Network-ClientHTTP1.cppm @@ -6,9 +6,53 @@ import std; import :HTTP; import :HTTP1; import :TLS; +import :URL; #ifndef CRAFTER_NETWORK_BROWSER namespace Crafter { + // What ClientHTTP1::Fetch needs beyond the URL. + export struct FetchOptions { + std::string method = "GET"; + // Lowercase names, as on HTTPRequest. Host comes from the URL. + std::unordered_map headers; + std::string body; + // As on ClientHTTP1, for every request of the chain. + std::chrono::milliseconds connectTimeout{15000}; + std::chrono::milliseconds handshakeTimeout{15000}; + std::chrono::milliseconds timeout{30000}; + HTTP1::MessageLimits limits; + // For every https:// request of the chain. + TLSClientCredentials credentials; + // How many redirects are followed at most; the one after comes back + // unfollowed. + std::uint32_t maxRedirects = 10; + }; + + // Why a Fetch ended on a redirect it did not follow. + export enum class FetchStop { + // It did not: the chain ended on a final response. + None, + // The redirect carried no Location, or one that is no URL. + NoLocation, + // The Location names a scheme other than http or https. + OtherScheme, + // The Location is a URL the chain has already been sent to. + Loop, + // maxRedirects redirects had been followed already. + Limit, + }; + + export struct FetchResult { + // The last response of the chain: the final response, or the + // redirect that was not followed, with `stop` saying why. + HTTPResponse response; + // The URL that answered with `response`. + URL url; + // Redirects followed to get there. + std::uint32_t redirects = 0; + FetchStop stop = FetchStop::None; + }; + // HTTP/1.1 client over TCP, with or without TLS, for peers that cannot // speak HTTP/3. The request/response types are the ones the HTTP/3 client // uses, so swapping ClientHTTP for ClientHTTP1 is a one-line change at the @@ -78,6 +122,27 @@ namespace Crafter { // Drop the pooled connection; the next Send() dials again. void Disconnect(); + // One request to an http:// or https:// URL, following redirects by + // RFC 9110 §15.4: 301, 302, 303, 307 and 308, for every method. A + // 303 turns the request into a GET (a HEAD stays a HEAD) without its + // body and content headers; 301 and 302 do the same for a POST + // alone; 307 and 308 keep the method and the body. The Location is + // resolved against the URL that sent it (URL::Resolve), and a + // redirect may move between http:// and https://, `credentials` + // serving every https:// hop. Authorization, Proxy-Authorization and + // Cookie are not carried to another origin. Each hop is a client of + // its own, since one ClientHTTP1 is one origin: the redirect's body + // is read and its connection closed before the next request goes + // out. A redirect without a usable Location, to another scheme, to + // a URL already visited, or past maxRedirects comes back as the + // response with `stop` set. Proxies are not consulted. + // + // Throws std::invalid_argument for a URL that is not http(s) or that + // URL::Parse refuses, and whatever a request throws otherwise + // (SocketErrorException, SocketTimeoutException, TLSException, + // HTTP1::HTTP1ProtocolError, std::runtime_error). + static FetchResult Fetch(std::string_view url, const FetchOptions& options = {}); + // Limits applied to responses. Set before the first Send(). HTTP1::MessageLimits limits; // How long to wait for the next piece of a response before giving @@ -97,56 +162,5 @@ namespace Crafter { struct Impl; std::unique_ptr impl; }; - - // What FetchHTTP1 needs beyond the URL. - export struct FetchOptionsHTTP1 { - std::string method = "GET"; - // Lowercase names, as on HTTPRequest. Host comes from the URL. - std::unordered_map headers; - std::string body; - // As on ClientHTTP1, for every request of the chain. - std::chrono::milliseconds connectTimeout{15000}; - std::chrono::milliseconds handshakeTimeout{15000}; - std::chrono::milliseconds timeout{30000}; - HTTP1::MessageLimits limits; - // For every https:// request of the chain. - TLSClientCredentials credentials; - // Python urllib's limits: a chain stops at a redirect to a URL it has - // already been sent to `maxRepeats` times, or once it has been sent - // to `maxRedirects` different URLs. - std::uint32_t maxRedirects = 10; - std::uint32_t maxRepeats = 4; - }; - - export struct FetchResultHTTP1 { - // The last response of the chain. A redirect only when it had nothing - // to follow — no Location, a method urllib does not redirect, a - // scheme other than http(s) — or when the limits stopped the chain. - HTTPResponse response; - // The URL that answered with `response`. - std::string url; - // Redirects followed to get there. - std::uint32_t redirects = 0; - // The limits stopped the chain; `response` is the redirect that would - // have gone past them. - bool limited = false; - }; - - // One request to an http:// or https:// URL, following redirects the way - // Python's urllib does, so a port of urllib code keeps its behaviour: - // 301, 302, 303, 307 and 308 are followed for GET and HEAD, the first - // three also for POST; the redirected request is a GET (a HEAD stays a - // HEAD) without the body and its content headers. A relative Location is - // resolved against the URL that sent it, as urljoin resolves it; a - // redirect may switch between http:// and https://. Each redirect's body - // is read before the next request goes out. Proxies are not consulted. - // - // The URL is `scheme://[user@]host[:port][/path][?query][#fragment]`, - // with an IPv6 host in brackets; the fragment is not sent and the user - // part is dropped. Throws std::invalid_argument for a URL that is not - // http(s) or has no host or a bad port, and whatever a request throws - // (SocketErrorException, SocketTimeoutException, TLSException, - // std::runtime_error) otherwise. - export FetchResultHTTP1 FetchHTTP1(std::string_view url, const FetchOptionsHTTP1& options = {}); } #endif diff --git a/interfaces/Crafter.Network-URL.cppm b/interfaces/Crafter.Network-URL.cppm new file mode 100644 index 0000000..59f6808 --- /dev/null +++ b/interfaces/Crafter.Network-URL.cppm @@ -0,0 +1,271 @@ +//SPDX-License-Identifier: LGPL-3.0-only +//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +export module Crafter.Network:URL; +import std; + +#ifndef CRAFTER_NETWORK_BROWSER +namespace Crafter { + namespace URLDetail { + // The components RFC 3986 appendix B cuts a URI reference into; a + // reference may lack the scheme and the authority, a URL may not. + struct Parts { + std::optional scheme; + std::optional authority; + std::string path; + std::optional query; + std::optional fragment; + }; + + inline bool IsAsciiAlpha(char c) { + return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); + } + + inline bool IsSchemeChar(char c) { + return IsAsciiAlpha(c) || (c >= '0' && c <= '9') || c == '+' || c == '-' || c == '.'; + } + + inline std::string Lowered(std::string_view text) { + std::string lower(text); + for (char& c : lower) { + if (c >= 'A' && c <= 'Z') c = static_cast(c - 'A' + 'a'); + } + return lower; + } + + // ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?, with the + // scheme taken only when it is one by RFC 3986 §3.1. + inline Parts Split(std::string_view text) { + Parts parts; + const std::size_t mark = text.find_first_of(":/?#"); + if (mark != std::string_view::npos && mark > 0 && text[mark] == ':' && IsAsciiAlpha(text[0]) + && std::ranges::all_of(text.substr(0, mark), IsSchemeChar)) { + parts.scheme = Lowered(text.substr(0, mark)); + text.remove_prefix(mark + 1); + } + if (text.starts_with("//")) { + text.remove_prefix(2); + std::size_t end = text.find_first_of("/?#"); + if (end == std::string_view::npos) end = text.size(); + parts.authority = std::string(text.substr(0, end)); + text.remove_prefix(end); + } + std::size_t end = text.find_first_of("?#"); + if (end == std::string_view::npos) end = text.size(); + parts.path = std::string(text.substr(0, end)); + text.remove_prefix(end); + if (text.starts_with('?')) { + text.remove_prefix(1); + std::size_t hash = text.find('#'); + if (hash == std::string_view::npos) hash = text.size(); + parts.query = std::string(text.substr(0, hash)); + text.remove_prefix(hash); + } + if (text.starts_with('#')) parts.fragment = std::string(text.substr(1)); + return parts; + } + + inline void RemoveLastSegment(std::string& path) { + const std::size_t slash = path.rfind('/'); + path.erase(slash == std::string::npos ? 0 : slash); + } + + // RFC 3986 §5.2.4. + inline std::string RemoveDotSegments(std::string_view path) { + std::string in(path); + std::string out; + while (!in.empty()) { + if (in.starts_with("../")) { + in.erase(0, 3); + } else if (in.starts_with("./")) { + in.erase(0, 2); + } else if (in.starts_with("/./")) { + in.erase(0, 2); + } else if (in == "/.") { + in = "/"; + } else if (in.starts_with("/../")) { + in.erase(0, 3); + RemoveLastSegment(out); + } else if (in == "/..") { + in = "/"; + RemoveLastSegment(out); + } else if (in == "." || in == "..") { + in.clear(); + } else { + std::size_t next = in.find('/', in.starts_with('/') ? 1 : 0); + if (next == std::string::npos) next = in.size(); + out += in.substr(0, next); + in.erase(0, next); + } + } + return out; + } + + // RFC 3986 §5.2.3. + inline std::string Merge(bool baseHasAuthority, std::string_view basePath, std::string_view referencePath) { + if (baseHasAuthority && basePath.empty()) return "/" + std::string(referencePath); + const std::size_t slash = basePath.rfind('/'); + if (slash == std::string_view::npos) return std::string(referencePath); + return std::string(basePath.substr(0, slash + 1)) + std::string(referencePath); + } + } + + // An absolute URL, as RFC 3986 reads it: + // `scheme://[userinfo@]host[:port][/path][?query][#fragment]`. What an + // HTTP client needs of URLs: the parse, the resolution of the reference + // a redirect carries against the URL that sent it, the recomposition, + // and the pieces a request is built from. + export struct URL { + // Lowercased. + std::string scheme; + std::string userinfo; + // Lowercased; an IPv6 literal without its brackets. + std::string host; + // None when the URL names no port: the scheme's default. + std::optional port; + // Empty, or starting with '/'. + std::string path; + // Without the '?'; none when the URL has no query (an empty query, + // `?` and nothing after it, is ""). + std::optional query; + // Without the '#'. + std::optional fragment; + + // Throws std::invalid_argument for text without a scheme or a host, + // a port that is not a number or is above 65535, and unbalanced + // brackets around an IPv6 host. + static URL Parse(std::string_view text); + + // RFC 3986 §5.2 resolution of a reference against this URL as the + // base: an absolute reference stands as it is, a relative one takes + // the parts it lacks from the base, and dot segments are removed. + // Throws std::invalid_argument when the result is no URL (a + // reference such as `mailto:x` has no host). + URL Resolve(std::string_view reference) const; + + std::string ToString() const; + + bool Secure() const { + return scheme == "https"; + } + + // The port, or the scheme's default: 443 for https, 80 for http, 0 + // for any other scheme. + std::uint16_t PortOrDefault() const { + if (port) return *port; + return scheme == "https" ? 443 : scheme == "http" ? 80 : 0; + } + + // host[:port] as the Host header carries it: brackets around an IPv6 + // literal, the port only when the URL names one. + std::string Authority() const { + std::string authority = host.contains(':') ? "[" + host + "]" : host; + if (port) authority += ":" + std::to_string(*port); + return authority; + } + + // The request target: the path, "/" when there is none, and the + // query. + std::string Target() const { + std::string target = path.empty() ? std::string("/") : path; + if (query) target += "?" + *query; + return target; + } + + // The same scheme, host and effective port. + bool SameOrigin(const URL& other) const { + return scheme == other.scheme && host == other.host && PortOrDefault() == other.PortOrDefault(); + } + }; + + namespace URLDetail { + inline URL FromParts(const Parts& parts, std::string_view shown) { + if (!parts.scheme) throw std::invalid_argument("URL: no scheme in " + std::string(shown)); + if (!parts.authority) throw std::invalid_argument("URL: no host in " + std::string(shown)); + URL url; + url.scheme = *parts.scheme; + std::string hostport = *parts.authority; + if (const std::size_t at = hostport.rfind('@'); at != std::string::npos) { + url.userinfo = hostport.substr(0, at); + hostport.erase(0, at + 1); + } + std::string port; + if (hostport.starts_with('[')) { + const std::size_t close = hostport.find(']'); + if (close == std::string::npos) throw std::invalid_argument("URL: unbalanced brackets in the host of " + std::string(shown)); + url.host = Lowered(std::string_view(hostport).substr(1, close - 1)); + const std::string rest = hostport.substr(close + 1); + if (rest.starts_with(':')) { + port = rest.substr(1); + } else if (!rest.empty()) { + throw std::invalid_argument("URL: unexpected text after the IPv6 host in " + std::string(shown)); + } + } else { + if (hostport.contains(']')) throw std::invalid_argument("URL: unbalanced brackets in the host of " + std::string(shown)); + const std::size_t colon = hostport.rfind(':'); + url.host = Lowered(std::string_view(hostport).substr(0, colon)); + if (colon != std::string::npos) port = hostport.substr(colon + 1); + } + if (url.host.empty()) throw std::invalid_argument("URL: no host in " + std::string(shown)); + if (!port.empty()) { + std::uint32_t value = 0; + const auto [end, error] = std::from_chars(port.data(), port.data() + port.size(), value); + if (error != std::errc() || end != port.data() + port.size() || value > 65535) { + throw std::invalid_argument("URL: bad port '" + port + "' in " + std::string(shown)); + } + url.port = static_cast(value); + } + url.path = parts.path; + url.query = parts.query; + url.fragment = parts.fragment; + return url; + } + } + + inline URL URL::Parse(std::string_view text) { + return URLDetail::FromParts(URLDetail::Split(text), text); + } + + inline URL URL::Resolve(std::string_view reference) const { + using namespace URLDetail; + const Parts ref = Split(reference); + Parts target; + if (ref.scheme) { + target = ref; + target.path = RemoveDotSegments(ref.path); + } else { + if (ref.authority) { + target.authority = ref.authority; + target.path = RemoveDotSegments(ref.path); + target.query = ref.query; + } else { + if (ref.path.empty()) { + target.path = path; + target.query = ref.query ? ref.query : query; + } else { + target.path = RemoveDotSegments(ref.path.starts_with('/') ? ref.path : Merge(true, path, ref.path)); + target.query = ref.query; + } + // The base's authority, as it was written. + std::string authority; + if (!userinfo.empty()) authority += userinfo + "@"; + authority += Authority(); + target.authority = authority; + } + target.scheme = scheme; + target.fragment = ref.fragment; + } + return FromParts(target, reference); + } + + inline std::string URL::ToString() const { + std::string text = scheme + "://"; + if (!userinfo.empty()) text += userinfo + "@"; + text += Authority(); + text += path; + if (query) text += "?" + *query; + if (fragment) text += "#" + *fragment; + return text; + } +} +#endif diff --git a/interfaces/Crafter.Network.cppm b/interfaces/Crafter.Network.cppm index 66ea843..8eabf09 100755 --- a/interfaces/Crafter.Network.cppm +++ b/interfaces/Crafter.Network.cppm @@ -33,6 +33,9 @@ export import :HTTP1; // same way. export import :Stream; export import :TLS; +// The URL type ClientHTTP1::Fetch parses and resolves, for callers that +// build or resolve URLs themselves. +export import :URL; export import :ClientHTTP1; export import :ListenerHTTP1; #endif \ No newline at end of file diff --git a/project.cpp b/project.cpp index 1030b81..76a3c66 100644 --- a/project.cpp +++ b/project.cpp @@ -8,7 +8,7 @@ using namespace Crafter; extern "C" Configuration CrafterBuildProject(std::span args) { // TCP, HTTP/1.1 and TLS: what every native build has. - constexpr std::array networkInterfaces = { + constexpr std::array networkInterfaces = { "interfaces/Crafter.Network", "interfaces/Crafter.Network-ClientTCP", "interfaces/Crafter.Network-ListenerTCP", @@ -16,6 +16,7 @@ extern "C" Configuration CrafterBuildProject(std::span a "interfaces/Crafter.Network-HTTP1", "interfaces/Crafter.Network-Stream", "interfaces/Crafter.Network-TLS", + "interfaces/Crafter.Network-URL", "interfaces/Crafter.Network-ClientHTTP1", "interfaces/Crafter.Network-ListenerHTTP1", }; @@ -180,6 +181,7 @@ extern "C" Configuration CrafterBuildProject(std::span a cfg.AddTest("ShouldBindLoopbackEphemeral").Dependencies({ &cfg }); cfg.AddTest("ShouldTimeoutSilentHTTP1").Dependencies({ &cfg }); cfg.AddTest("ShouldFollowRedirectsHTTP1").Dependencies({ &cfg }); + cfg.AddTest("ShouldParseURLs").Dependencies({ &cfg }); cfg.AddTest("ShouldInteropCurlHTTP1").Dependencies({ &cfg }); cfg.AddTest("ShouldInteropCurlHTTPS1").Dependencies({ &cfg }); cfg.AddTest("ShouldParseHTTP1").Dependencies({ &cfg }); diff --git a/tests/ShouldFollowRedirectsHTTP1/main.cpp b/tests/ShouldFollowRedirectsHTTP1/main.cpp index c99d718..0ba456b 100644 --- a/tests/ShouldFollowRedirectsHTTP1/main.cpp +++ b/tests/ShouldFollowRedirectsHTTP1/main.cpp @@ -1,20 +1,22 @@ //SPDX-License-Identifier: LGPL-3.0-only //SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® -// FetchHTTP1 follows redirects the way Python's urllib does, so code ported -// from urllib keeps its behaviour: -// - a relative Location is resolved against the URL that sent it (urljoin, -// `..` included), and a redirect may move from http:// to https://; -// - 301/302/303/307/308 are followed for a GET; a POST follows 301/302/303 -// as a GET without its body, and comes back with a 307 or 308 unfollowed; -// - a redirect without a Location comes back as the response; -// - the chain stops at the 5th redirect to one URL (4 repeats) or after 10 -// different URLs, returning that redirect with `limited` set; -// - a Location with a space is percent-encoded, as urllib quotes it; +// ClientHTTP1::Fetch follows redirects by RFC 9110 §15.4: +// - a relative Location is resolved against the URL that sent it (RFC +// 3986, `..` included), and a redirect may move from http:// to https://; +// - 301, 302, 303, 307 and 308 are followed for every method: 303 turns +// the request into a GET without its body, 301 and 302 do so for a POST +// alone, 307 and 308 keep the method and the body; +// - a redirect without a Location, or to another scheme, comes back as the +// response, with `stop` saying why; +// - a chain stops at a URL it has already been sent to, or after +// maxRedirects, returning that redirect; +// - Authorization and Cookie are not carried to another origin; // - a URL that is not http(s), has no host or has a bad port is refused. // // Two listeners: plaintext on 8130, TLS on 8131 with a self-signed certificate -// the client is handed as its trust anchor. +// the client is handed as its trust anchor. The cross-origin case goes from +// 127.0.0.1 to localhost, the same listener under another name. #include @@ -43,12 +45,20 @@ std::unordered_map> routes["/to-https"] = [](const HTTPRequest&) { return Redirect("307", "https://localhost:8131/secure"); }; routes["/loop"] = [](const HTTPRequest&) { return Redirect("302", "/loop"); }; routes["/nowhere"] = [](const HTTPRequest&) { return CreateResponseHTTP("302", std::string("no location")); }; + routes["/elsewhere"] = [](const HTTPRequest&) { return Redirect("302", "ftp://127.0.0.1/file"); }; routes["/form"] = [](const HTTPRequest&) { return Redirect("303", "/method"); }; + routes["/found"] = [](const HTTPRequest&) { return Redirect("302", "/method"); }; routes["/keep"] = [](const HTTPRequest&) { return Redirect("307", "/method"); }; routes["/method"] = [](const HTTPRequest& request) { - return CreateResponseHTTP("200", request.method + " body=" + request.body); + return CreateResponseHTTP("200", request.method + " body=" + request.body + " type=" + (request.headers.contains("content-type") ? request.headers.at("content-type") : "none")); + }; + routes["/auth-away"] = [](const HTTPRequest&) { return Redirect("302", "http://localhost:8130/show-auth"); }; + routes["/auth-home"] = [](const HTTPRequest&) { return Redirect("302", "/show-auth"); }; + routes["/show-auth"] = [](const HTTPRequest& request) { + return CreateResponseHTTP("200", std::string("authorization=") + (request.headers.contains("authorization") ? "yes" : "no") + + " cookie=" + (request.headers.contains("cookie") ? "yes" : "no") + + " accept=" + (request.headers.contains("accept") ? "yes" : "no")); }; - routes["/spaced"] = [](const HTTPRequest&) { return Redirect("302", "/with space"); }; return routes; } @@ -63,7 +73,7 @@ HTTPResponse Fallback(const HTTPRequest& request) { bool Refused(std::string_view url) { try { - FetchHTTP1(url); + ClientHTTP1::Fetch(url); } catch (const std::invalid_argument&) { return true; } catch (const std::exception&) { @@ -83,61 +93,81 @@ int main() { ListenerAsyncHTTP1 secure(8131, secureRoutes, TLSServerCredentials { .selfSigned = true }); { - const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/start"); + const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/start"); Check(result.response.status == "200" && result.response.body == "/final?x=1", std::format("a relative chain arrives at /final?x=1 (got {} '{}')", result.response.status, result.response.body)); - Check(result.url == "http://127.0.0.1:8130/final?x=1", std::format("the URL that answered is reported ({})", result.url)); - Check(result.redirects == 2 && !result.limited, std::format("two redirects, not limited (got {})", result.redirects)); + Check(result.url.ToString() == "http://127.0.0.1:8130/final?x=1", std::format("the URL that answered is reported ({})", result.url.ToString())); + Check(result.redirects == 2 && result.stop == FetchStop::None, std::format("two redirects, the chain ended on a final response (got {})", result.redirects)); } { - const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/a/b/start"); + const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/a/b/start"); Check(result.response.body == "/a/c/d", std::format("../c/d from /a/b/start resolves to /a/c/d (got '{}')", result.response.body)); } { - FetchOptionsHTTP1 options; + FetchOptions options; options.credentials.caPem = GetSelfSignedCertificatePem().certificate; - const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/to-https", options); + const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/to-https", options); Check(result.response.status == "200" && result.response.body == "secure over https", std::format("a redirect from http:// to https:// is followed (got {} '{}')", result.response.status, result.response.body)); - Check(result.url == "https://localhost:8131/secure", std::format("and lands on the https:// URL ({})", result.url)); + Check(result.url.ToString() == "https://localhost:8131/secure" && result.url.Secure(), std::format("and lands on the https:// URL ({})", result.url.ToString())); } { - const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/loop"); - Check(result.limited && result.response.status == "302" && result.redirects == 4, - std::format("a redirect loop stops after 4 repeats (limited {}, {} redirects)", result.limited, result.redirects)); + const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/loop"); + Check(result.stop == FetchStop::Loop && result.response.status == "302" && result.redirects == 0, + std::format("a redirect back to a URL already visited stops the chain at once (stop {}, {} redirects)", static_cast(result.stop), result.redirects)); } { - const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/hop/0"); - Check(result.limited && result.redirects == 10 && result.url == "http://127.0.0.1:8130/hop/10", - std::format("an endless chain stops after 10 different URLs ({} redirects, at {})", result.redirects, result.url)); + const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/hop/0"); + Check(result.stop == FetchStop::Limit && result.redirects == 10 && result.url.ToString() == "http://127.0.0.1:8130/hop/10", + std::format("an endless chain stops after maxRedirects, 10 ({} redirects, at {})", result.redirects, result.url.ToString())); + FetchOptions two; + two.maxRedirects = 2; + const FetchResult shorter = ClientHTTP1::Fetch("http://127.0.0.1:8130/hop/0", two); + Check(shorter.stop == FetchStop::Limit && shorter.redirects == 2, "maxRedirects is the option's"); } { - const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/nowhere"); - Check(result.response.status == "302" && result.redirects == 0 && !result.limited, + const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/nowhere"); + Check(result.stop == FetchStop::NoLocation && result.response.status == "302" && result.redirects == 0, "a redirect without a Location comes back as the response"); + const FetchResult other = ClientHTTP1::Fetch("http://127.0.0.1:8130/elsewhere"); + Check(other.stop == FetchStop::OtherScheme && other.response.status == "302", "a redirect to another scheme comes back as the response"); } { - FetchOptionsHTTP1 post; + FetchOptions post; post.method = "POST"; post.body = "payload"; post.headers["content-type"] = "text/plain"; - const FetchResultHTTP1 seeOther = FetchHTTP1("http://127.0.0.1:8130/form", post); - Check(seeOther.response.body == "GET body=", - std::format("a POST answered with 303 is followed as a GET without its body (got '{}')", seeOther.response.body)); - const FetchResultHTTP1 kept = FetchHTTP1("http://127.0.0.1:8130/keep", post); - Check(kept.response.status == "307" && kept.redirects == 0, "a POST answered with 307 comes back unfollowed"); + const FetchResult seeOther = ClientHTTP1::Fetch("http://127.0.0.1:8130/form", post); + Check(seeOther.response.body == "GET body= type=none", + std::format("a POST answered with 303 is followed as a GET without its body or content headers (got '{}')", seeOther.response.body)); + const FetchResult found = ClientHTTP1::Fetch("http://127.0.0.1:8130/found", post); + Check(found.response.body == "GET body= type=none", std::format("a POST answered with 302 is followed as a GET (got '{}')", found.response.body)); + const FetchResult kept = ClientHTTP1::Fetch("http://127.0.0.1:8130/keep", post); + Check(kept.response.body == "POST body=payload type=text/plain" && kept.redirects == 1, + std::format("a POST answered with 307 is followed with its method, body and content headers (got '{}')", kept.response.body)); + FetchOptions put = post; + put.method = "PUT"; + const FetchResult putFound = ClientHTTP1::Fetch("http://127.0.0.1:8130/found", put); + Check(putFound.response.body == "PUT body=payload type=text/plain", std::format("a PUT answered with 302 keeps its method and body (got '{}')", putFound.response.body)); } { - const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/spaced"); - Check(result.response.body == "/with%20space", - std::format("a Location with a space is percent-encoded before it is followed (got '{}')", result.response.body)); + FetchOptions options; + options.headers["authorization"] = "Bearer secret"; + options.headers["cookie"] = "a=b"; + options.headers["accept"] = "text/plain"; + const FetchResult away = ClientHTTP1::Fetch("http://127.0.0.1:8130/auth-away", options); + Check(away.response.body == "authorization=no cookie=no accept=yes", + std::format("Authorization and Cookie are dropped on a redirect to another origin, other headers kept (got '{}')", away.response.body)); + const FetchResult home = ClientHTTP1::Fetch("http://127.0.0.1:8130/auth-home", options); + Check(home.response.body == "authorization=yes cookie=yes accept=yes", + std::format("and kept on a redirect within the origin (got '{}')", home.response.body)); } Check(Refused("ftp://127.0.0.1/file"), "an ftp:// URL is refused"); diff --git a/tests/ShouldParseURLs/main.cpp b/tests/ShouldParseURLs/main.cpp new file mode 100644 index 0000000..3cbaf73 --- /dev/null +++ b/tests/ShouldParseURLs/main.cpp @@ -0,0 +1,105 @@ +//SPDX-License-Identifier: LGPL-3.0-only +//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +// URL reads an absolute URL as RFC 3986 does and resolves a reference +// against it as §5.2 says, which is what ClientHTTP1::Fetch leans on to +// follow a Location. The resolution cases are RFC 3986 §5.4's own examples, +// normal and abnormal, against its base `http://a/b/c/d;p?q`; the parse +// cases guard the pieces a request is built from (the default port, the +// IPv6 brackets, the userinfo that must not reach the host) and what is +// refused. + +#include + +import Crafter.Network; +import std; +using namespace Crafter; + +namespace { + +int failures = 0; + +void Check(bool ok, std::string_view what) { + std::println("{} {}", ok ? "PASS" : "FAIL", what); + if (!ok) ++failures; +} + +bool Refused(std::string_view text) { + try { + URL::Parse(text); + } catch (const std::invalid_argument&) { + return true; + } + return false; +} + +} // namespace + +int main() { + { + const URL url = URL::Parse("HTTP://User:pw@Example.COM:8080/a/b?x=1&y#frag"); + Check(url.scheme == "http" && url.host == "example.com", "the scheme and the host are lowercased"); + Check(url.userinfo == "User:pw", "the userinfo is kept apart from the host"); + Check(url.port && *url.port == 8080 && url.PortOrDefault() == 8080, "an explicit port"); + Check(url.path == "/a/b" && url.query && *url.query == "x=1&y" && url.fragment && *url.fragment == "frag", "path, query and fragment"); + Check(url.Authority() == "example.com:8080" && url.Target() == "/a/b?x=1&y", "the Host value and the request target"); + Check(url.ToString() == "http://User:pw@example.com:8080/a/b?x=1&y#frag", std::format("recomposed ({})", url.ToString())); + } + { + const URL plain = URL::Parse("http://example.com"); + const URL secure = URL::Parse("https://example.com/"); + Check(!plain.port && plain.PortOrDefault() == 80 && secure.PortOrDefault() == 443, "the scheme's default port when none is named"); + Check(plain.path.empty() && plain.Target() == "/" && plain.Authority() == "example.com", "no path is the target `/`, and the Host value carries no port"); + Check(!plain.query && URL::Parse("http://example.com/?").query && URL::Parse("http://example.com/?").query->empty(), + "no query is none; `?` alone is an empty one"); + Check(!plain.SameOrigin(secure) && plain.SameOrigin(URL::Parse("http://EXAMPLE.com:80/other")) && !plain.SameOrigin(URL::Parse("http://example.com:8080/")), + "the origin is the scheme, the host and the effective port"); + } + { + const URL v6 = URL::Parse("http://[::1]:8130/x"); + Check(v6.host == "::1" && v6.port && *v6.port == 8130 && v6.Authority() == "[::1]:8130" && v6.ToString() == "http://[::1]:8130/x", + std::format("an IPv6 host loses its brackets in `host` and gets them back in the Host value ({})", v6.ToString())); + } + Check(Refused("example.com/path"), "text without a scheme is refused"); + Check(Refused("http:///path"), "a URL without a host is refused"); + Check(Refused("http://example.com:99999/"), "a port above 65535 is refused"); + Check(Refused("http://example.com:8x/"), "a port that is not a number is refused"); + Check(Refused("http://[::1/"), "unbalanced brackets are refused"); + Check(URL::Parse("http://example.com:/").PortOrDefault() == 80, "an empty port is the default"); + + // RFC 3986 §5.4. + const URL base = URL::Parse("http://a/b/c/d;p?q"); + const std::vector> cases = { + {"g", "http://a/b/c/g"}, {"./g", "http://a/b/c/g"}, {"g/", "http://a/b/c/g/"}, {"/g", "http://a/g"}, {"//g", "http://g"}, + {"?y", "http://a/b/c/d;p?y"}, {"g?y", "http://a/b/c/g?y"}, {"#s", "http://a/b/c/d;p?q#s"}, {"g#s", "http://a/b/c/g#s"}, + {"g?y#s", "http://a/b/c/g?y#s"}, {";x", "http://a/b/c/;x"}, {"g;x", "http://a/b/c/g;x"}, {"g;x?y#s", "http://a/b/c/g;x?y#s"}, + {"", "http://a/b/c/d;p?q"}, {".", "http://a/b/c/"}, {"./", "http://a/b/c/"}, {"..", "http://a/b/"}, {"../", "http://a/b/"}, + {"../g", "http://a/b/g"}, {"../..", "http://a/"}, {"../../", "http://a/"}, {"../../g", "http://a/g"}, + // §5.4.2, the abnormal ones. + {"../../../g", "http://a/g"}, {"../../../../g", "http://a/g"}, {"/./g", "http://a/g"}, {"/../g", "http://a/g"}, + {"g.", "http://a/b/c/g."}, {".g", "http://a/b/c/.g"}, {"g..", "http://a/b/c/g.."}, {"..g", "http://a/b/c/..g"}, + {"./../g", "http://a/b/g"}, {"./g/.", "http://a/b/c/g/"}, {"g/./h", "http://a/b/c/g/h"}, {"g/../h", "http://a/b/c/h"}, + {"g;x=1/./y", "http://a/b/c/g;x=1/y"}, {"g;x=1/../y", "http://a/b/c/y"}, {"g?y/./x", "http://a/b/c/g?y/./x"}, + {"g?y/../x", "http://a/b/c/g?y/../x"}, {"g#s/./x", "http://a/b/c/g#s/./x"}, {"g#s/../x", "http://a/b/c/g#s/../x"}, + {"https://b/x", "https://b/x"}, + }; + for (const auto& [reference, expected] : cases) { + const std::string resolved = base.Resolve(reference).ToString(); + Check(resolved == expected, std::format("'{}' against the base resolves to {} (got {})", reference, expected, resolved)); + } + { + const URL from = URL::Parse("http://127.0.0.1:8130/a/b/start"); + Check(from.Resolve("../c/d").ToString() == "http://127.0.0.1:8130/a/c/d", "a port and a userinfo-free base keep their authority through a resolve"); + Check(URL::Parse("http://u@h:81/p").Resolve("x").ToString() == "http://u@h:81/x", "the base's userinfo stays"); + bool refused = false; + try { + from.Resolve("mailto:someone@example.com"); + } catch (const std::invalid_argument&) { + refused = true; + } + Check(refused, "a reference that resolves to no URL (mailto:) is refused"); + } + + std::println("{}", failures == 0 ? "all checks passed" : "check(s) failed"); + return failures == 0 ? EXIT_SUCCESS : EXIT_FAILURE; +}