Catcrafts/Crafter.Network
Watch
2
0
Fork
You've already forked Crafter.Network
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(http1): ClientHTTP1::Fetch follows redirects by the RFC, on a URL type

Browse source 
One request to an http(s) URL with redirects followed by RFC 9110 §15.4:
301, 302, 303, 307 and 308 for every method; a 303 makes the request a
GET without its body and content headers, a 301 or 302 does so for a POST
alone, a 307 or 308 keeps the method and the body. The Location is
resolved against the sending URL by RFC 3986, a hop between http and
https is allowed, Authorization, Proxy-Authorization and Cookie are
dropped on a change of origin, and each hop is a client of its own, since
one ClientHTTP1 is one origin. A redirect without a usable Location, to
another scheme, back to a URL already visited or past maxRedirects comes
back as the response, with `stop` saying why.

The new :URL partition is the URL type behind it, exported: Parse,
Resolve (RFC 3986 §5.2; its §5.4 examples are the test), ToString, and
the pieces a request is built from.

This replaces the FetchHTTP1 free function, which followed redirects the
way Python's urllib did (its repeat limit, its quoting of the Location, a
POST turned back on a 307) for a port that no longer needs the fidelity.

Breaking: FetchHTTP1, FetchOptionsHTTP1 and FetchResultHTTP1 are gone;
use ClientHTTP1::Fetch, FetchOptions and FetchResult, whose `url` is a
URL and whose `limited` became `stop`.
This commit is contained in:
Jorijn van der Graaf 2026-10-31 20:39:43 +01:00
commit 7ee938c98e
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
10 changed files with 596 additions and 372 deletions
Download patch file Download diff file Expand all files Collapse all files

106
tests/ShouldFollowRedirectsHTTP1/main.cpp
View file

@ -1,20 +1,22 @@
//SPDX-License-Identifier: LGPL-3.0-only
//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// FetchHTTP1 follows redirects the way Python's urllib does, so code ported
// from urllib keeps its behaviour:
// - a relative Location is resolved against the URL that sent it (urljoin,
// `..` included), and a redirect may move from http:// to https://;
// - 301/302/303/307/308 are followed for a GET; a POST follows 301/302/303
// as a GET without its body, and comes back with a 307 or 308 unfollowed;
// - a redirect without a Location comes back as the response;
// - the chain stops at the 5th redirect to one URL (4 repeats) or after 10
// different URLs, returning that redirect with `limited` set;
// - a Location with a space is percent-encoded, as urllib quotes it;
// ClientHTTP1::Fetch follows redirects by RFC 9110 §15.4:
// - a relative Location is resolved against the URL that sent it (RFC
// 3986, `..` included), and a redirect may move from http:// to https://;
// - 301, 302, 303, 307 and 308 are followed for every method: 303 turns
// the request into a GET without its body, 301 and 302 do so for a POST
// alone, 307 and 308 keep the method and the body;
// - a redirect without a Location, or to another scheme, comes back as the
// response, with `stop` saying why;
// - a chain stops at a URL it has already been sent to, or after
// maxRedirects, returning that redirect;
// - Authorization and Cookie are not carried to another origin;
// - a URL that is not http(s), has no host or has a bad port is refused.
//
// Two listeners: plaintext on 8130, TLS on 8131 with a self-signed certificate
// the client is handed as its trust anchor.
// the client is handed as its trust anchor. The cross-origin case goes from
// 127.0.0.1 to localhost, the same listener under another name.
#include <cstdlib>
@ -43,12 +45,20 @@ std::unordered_map<std::string, std::function<HTTPResponse(const HTTPRequest&)>>
routes["/to-https"] = [](const HTTPRequest&) { return Redirect("307", "https://localhost:8131/secure"); };
routes["/loop"] = [](const HTTPRequest&) { return Redirect("302", "/loop"); };
routes["/nowhere"] = [](const HTTPRequest&) { return CreateResponseHTTP("302", std::string("no location")); };
routes["/elsewhere"] = [](const HTTPRequest&) { return Redirect("302", "ftp://127.0.0.1/file"); };
routes["/form"] = [](const HTTPRequest&) { return Redirect("303", "/method"); };
routes["/found"] = [](const HTTPRequest&) { return Redirect("302", "/method"); };
routes["/keep"] = [](const HTTPRequest&) { return Redirect("307", "/method"); };
routes["/method"] = [](const HTTPRequest& request) {
return CreateResponseHTTP("200", request.method + " body=" + request.body);
return CreateResponseHTTP("200", request.method + " body=" + request.body + " type=" + (request.headers.contains("content-type") ? request.headers.at("content-type") : "none"));
};
routes["/auth-away"] = [](const HTTPRequest&) { return Redirect("302", "http://localhost:8130/show-auth"); };
routes["/auth-home"] = [](const HTTPRequest&) { return Redirect("302", "/show-auth"); };
routes["/show-auth"] = [](const HTTPRequest& request) {
return CreateResponseHTTP("200", std::string("authorization=") + (request.headers.contains("authorization") ? "yes" : "no")
+ " cookie=" + (request.headers.contains("cookie") ? "yes" : "no")
+ " accept=" + (request.headers.contains("accept") ? "yes" : "no"));
};
routes["/spaced"] = [](const HTTPRequest&) { return Redirect("302", "/with space"); };
return routes;
}
@ -63,7 +73,7 @@ HTTPResponse Fallback(const HTTPRequest& request) {
bool Refused(std::string_view url) {
try {
FetchHTTP1(url);
ClientHTTP1::Fetch(url);
} catch (const std::invalid_argument&) {
return true;
} catch (const std::exception&) {
@ -83,61 +93,81 @@ int main() {
ListenerAsyncHTTP1 secure(8131, secureRoutes, TLSServerCredentials { .selfSigned = true });
{
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/start");
const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/start");
Check(result.response.status == "200" && result.response.body == "/final?x=1",
std::format("a relative chain arrives at /final?x=1 (got {} '{}')", result.response.status, result.response.body));
Check(result.url == "http://127.0.0.1:8130/final?x=1", std::format("the URL that answered is reported ({})", result.url));
Check(result.redirects == 2 && !result.limited, std::format("two redirects, not limited (got {})", result.redirects));
Check(result.url.ToString() == "http://127.0.0.1:8130/final?x=1", std::format("the URL that answered is reported ({})", result.url.ToString()));
Check(result.redirects == 2 && result.stop == FetchStop::None, std::format("two redirects, the chain ended on a final response (got {})", result.redirects));
}
{
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/a/b/start");
const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/a/b/start");
Check(result.response.body == "/a/c/d", std::format("../c/d from /a/b/start resolves to /a/c/d (got '{}')", result.response.body));
}
{
FetchOptionsHTTP1 options;
FetchOptions options;
options.credentials.caPem = GetSelfSignedCertificatePem().certificate;
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/to-https", options);
const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/to-https", options);
Check(result.response.status == "200" && result.response.body == "secure over https",
std::format("a redirect from http:// to https:// is followed (got {} '{}')", result.response.status, result.response.body));
Check(result.url == "https://localhost:8131/secure", std::format("and lands on the https:// URL ({})", result.url));
Check(result.url.ToString() == "https://localhost:8131/secure" && result.url.Secure(), std::format("and lands on the https:// URL ({})", result.url.ToString()));
}
{
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/loop");
Check(result.limited && result.response.status == "302" && result.redirects == 4,
std::format("a redirect loop stops after 4 repeats (limited {}, {} redirects)", result.limited, result.redirects));
const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/loop");
Check(result.stop == FetchStop::Loop && result.response.status == "302" && result.redirects == 0,
std::format("a redirect back to a URL already visited stops the chain at once (stop {}, {} redirects)", static_cast<int>(result.stop), result.redirects));
}
{
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/hop/0");
Check(result.limited && result.redirects == 10 && result.url == "http://127.0.0.1:8130/hop/10",
std::format("an endless chain stops after 10 different URLs ({} redirects, at {})", result.redirects, result.url));
const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/hop/0");
Check(result.stop == FetchStop::Limit && result.redirects == 10 && result.url.ToString() == "http://127.0.0.1:8130/hop/10",
std::format("an endless chain stops after maxRedirects, 10 ({} redirects, at {})", result.redirects, result.url.ToString()));
FetchOptions two;
two.maxRedirects = 2;
const FetchResult shorter = ClientHTTP1::Fetch("http://127.0.0.1:8130/hop/0", two);
Check(shorter.stop == FetchStop::Limit && shorter.redirects == 2, "maxRedirects is the option's");
}
{
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/nowhere");
Check(result.response.status == "302" && result.redirects == 0 && !result.limited,
const FetchResult result = ClientHTTP1::Fetch("http://127.0.0.1:8130/nowhere");
Check(result.stop == FetchStop::NoLocation && result.response.status == "302" && result.redirects == 0,
"a redirect without a Location comes back as the response");
const FetchResult other = ClientHTTP1::Fetch("http://127.0.0.1:8130/elsewhere");
Check(other.stop == FetchStop::OtherScheme && other.response.status == "302", "a redirect to another scheme comes back as the response");
}
{
FetchOptionsHTTP1 post;
FetchOptions post;
post.method = "POST";
post.body = "payload";
post.headers["content-type"] = "text/plain";
const FetchResultHTTP1 seeOther = FetchHTTP1("http://127.0.0.1:8130/form", post);
Check(seeOther.response.body == "GET body=",
std::format("a POST answered with 303 is followed as a GET without its body (got '{}')", seeOther.response.body));
const FetchResultHTTP1 kept = FetchHTTP1("http://127.0.0.1:8130/keep", post);
Check(kept.response.status == "307" && kept.redirects == 0, "a POST answered with 307 comes back unfollowed");
const FetchResult seeOther = ClientHTTP1::Fetch("http://127.0.0.1:8130/form", post);
Check(seeOther.response.body == "GET body= type=none",
std::format("a POST answered with 303 is followed as a GET without its body or content headers (got '{}')", seeOther.response.body));
const FetchResult found = ClientHTTP1::Fetch("http://127.0.0.1:8130/found", post);
Check(found.response.body == "GET body= type=none", std::format("a POST answered with 302 is followed as a GET (got '{}')", found.response.body));
const FetchResult kept = ClientHTTP1::Fetch("http://127.0.0.1:8130/keep", post);
Check(kept.response.body == "POST body=payload type=text/plain" && kept.redirects == 1,
std::format("a POST answered with 307 is followed with its method, body and content headers (got '{}')", kept.response.body));
FetchOptions put = post;
put.method = "PUT";
const FetchResult putFound = ClientHTTP1::Fetch("http://127.0.0.1:8130/found", put);
Check(putFound.response.body == "PUT body=payload type=text/plain", std::format("a PUT answered with 302 keeps its method and body (got '{}')", putFound.response.body));
}
{
const FetchResultHTTP1 result = FetchHTTP1("http://127.0.0.1:8130/spaced");
Check(result.response.body == "/with%20space",
std::format("a Location with a space is percent-encoded before it is followed (got '{}')", result.response.body));
FetchOptions options;
options.headers["authorization"] = "Bearer secret";
options.headers["cookie"] = "a=b";
options.headers["accept"] = "text/plain";
const FetchResult away = ClientHTTP1::Fetch("http://127.0.0.1:8130/auth-away", options);
Check(away.response.body == "authorization=no cookie=no accept=yes",
std::format("Authorization and Cookie are dropped on a redirect to another origin, other headers kept (got '{}')", away.response.body));
const FetchResult home = ClientHTTP1::Fetch("http://127.0.0.1:8130/auth-home", options);
Check(home.response.body == "authorization=yes cookie=yes accept=yes",
std::format("and kept on a redirect within the origin (got '{}')", home.response.body));
}
Check(Refused("ftp://127.0.0.1/file"), "an ftp:// URL is refused");

105
tests/ShouldParseURLs/main.cpp Normal file
View file

@ -0,0 +1,105 @@
//SPDX-License-Identifier: LGPL-3.0-only
//SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
// URL reads an absolute URL as RFC 3986 does and resolves a reference
// against it as §5.2 says, which is what ClientHTTP1::Fetch leans on to
// follow a Location. The resolution cases are RFC 3986 §5.4's own examples,
// normal and abnormal, against its base `http://a/b/c/d;p?q`; the parse
// cases guard the pieces a request is built from (the default port, the
// IPv6 brackets, the userinfo that must not reach the host) and what is
// refused.
#include <cstdlib>
import Crafter.Network;
import std;
using namespace Crafter;
namespace {
int failures = 0;
void Check(bool ok, std::string_view what) {
std::println("{} {}", ok ? "PASS" : "FAIL", what);
if (!ok) ++failures;
}
bool Refused(std::string_view text) {
try {
URL::Parse(text);
} catch (const std::invalid_argument&) {
return true;
}
return false;
}
} // namespace
int main() {
{
const URL url = URL::Parse("HTTP://User:pw@Example.COM:8080/a/b?x=1&y#frag");
Check(url.scheme == "http" && url.host == "example.com", "the scheme and the host are lowercased");
Check(url.userinfo == "User:pw", "the userinfo is kept apart from the host");
Check(url.port && *url.port == 8080 && url.PortOrDefault() == 8080, "an explicit port");
Check(url.path == "/a/b" && url.query && *url.query == "x=1&y" && url.fragment && *url.fragment == "frag", "path, query and fragment");
Check(url.Authority() == "example.com:8080" && url.Target() == "/a/b?x=1&y", "the Host value and the request target");
Check(url.ToString() == "http://User:pw@example.com:8080/a/b?x=1&y#frag", std::format("recomposed ({})", url.ToString()));
}
{
const URL plain = URL::Parse("http://example.com");
const URL secure = URL::Parse("https://example.com/");
Check(!plain.port && plain.PortOrDefault() == 80 && secure.PortOrDefault() == 443, "the scheme's default port when none is named");
Check(plain.path.empty() && plain.Target() == "/" && plain.Authority() == "example.com", "no path is the target `/`, and the Host value carries no port");
Check(!plain.query && URL::Parse("http://example.com/?").query && URL::Parse("http://example.com/?").query->empty(),
"no query is none; `?` alone is an empty one");
Check(!plain.SameOrigin(secure) && plain.SameOrigin(URL::Parse("http://EXAMPLE.com:80/other")) && !plain.SameOrigin(URL::Parse("http://example.com:8080/")),
"the origin is the scheme, the host and the effective port");
}
{
const URL v6 = URL::Parse("http://[::1]:8130/x");
Check(v6.host == "::1" && v6.port && *v6.port == 8130 && v6.Authority() == "[::1]:8130" && v6.ToString() == "http://[::1]:8130/x",
std::format("an IPv6 host loses its brackets in `host` and gets them back in the Host value ({})", v6.ToString()));
}
Check(Refused("example.com/path"), "text without a scheme is refused");
Check(Refused("http:///path"), "a URL without a host is refused");
Check(Refused("http://example.com:99999/"), "a port above 65535 is refused");
Check(Refused("http://example.com:8x/"), "a port that is not a number is refused");
Check(Refused("http://[::1/"), "unbalanced brackets are refused");
Check(URL::Parse("http://example.com:/").PortOrDefault() == 80, "an empty port is the default");
// RFC 3986 §5.4.
const URL base = URL::Parse("http://a/b/c/d;p?q");
const std::vector<std::pair<std::string_view, std::string_view>> cases = {
{"g", "http://a/b/c/g"}, {"./g", "http://a/b/c/g"}, {"g/", "http://a/b/c/g/"}, {"/g", "http://a/g"}, {"//g", "http://g"},
{"?y", "http://a/b/c/d;p?y"}, {"g?y", "http://a/b/c/g?y"}, {"#s", "http://a/b/c/d;p?q#s"}, {"g#s", "http://a/b/c/g#s"},
{"g?y#s", "http://a/b/c/g?y#s"}, {";x", "http://a/b/c/;x"}, {"g;x", "http://a/b/c/g;x"}, {"g;x?y#s", "http://a/b/c/g;x?y#s"},
{"", "http://a/b/c/d;p?q"}, {".", "http://a/b/c/"}, {"./", "http://a/b/c/"}, {"..", "http://a/b/"}, {"../", "http://a/b/"},
{"../g", "http://a/b/g"}, {"../..", "http://a/"}, {"../../", "http://a/"}, {"../../g", "http://a/g"},
// §5.4.2, the abnormal ones.
{"../../../g", "http://a/g"}, {"../../../../g", "http://a/g"}, {"/./g", "http://a/g"}, {"/../g", "http://a/g"},
{"g.", "http://a/b/c/g."}, {".g", "http://a/b/c/.g"}, {"g..", "http://a/b/c/g.."}, {"..g", "http://a/b/c/..g"},
{"./../g", "http://a/b/g"}, {"./g/.", "http://a/b/c/g/"}, {"g/./h", "http://a/b/c/g/h"}, {"g/../h", "http://a/b/c/h"},
{"g;x=1/./y", "http://a/b/c/g;x=1/y"}, {"g;x=1/../y", "http://a/b/c/y"}, {"g?y/./x", "http://a/b/c/g?y/./x"},
{"g?y/../x", "http://a/b/c/g?y/../x"}, {"g#s/./x", "http://a/b/c/g#s/./x"}, {"g#s/../x", "http://a/b/c/g#s/../x"},
{"https://b/x", "https://b/x"},
};
for (const auto& [reference, expected] : cases) {
const std::string resolved = base.Resolve(reference).ToString();
Check(resolved == expected, std::format("'{}' against the base resolves to {} (got {})", reference, expected, resolved));
}
{
const URL from = URL::Parse("http://127.0.0.1:8130/a/b/start");
Check(from.Resolve("../c/d").ToString() == "http://127.0.0.1:8130/a/c/d", "a port and a userinfo-free base keep their authority through a resolve");
Check(URL::Parse("http://u@h:81/p").Resolve("x").ToString() == "http://u@h:81/x", "the base's userinfo stays");
bool refused = false;
try {
from.Resolve("mailto:someone@example.com");
} catch (const std::invalid_argument&) {
refused = true;
}
Check(refused, "a reference that resolves to no URL (mailto:) is refused");
}
std::println("{}", failures == 0 ? "all checks passed" : "check(s) failed");
return failures == 0 ? EXIT_SUCCESS : EXIT_FAILURE;
}