Catcrafts/catcrafts.net
Watch
2
0
Fork
You've already forked catcrafts.net
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

posts and avif
Some checks failed
Deploy / build-deploy (push) Failing after 5m49s
Details

Browse source
This commit is contained in:
Jorijn van der Graaf 2026-08-10 01:37:26 +02:00
commit 6841623e23
17 changed files with 2306 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

30
.forgejo/workflows/deploy.yaml
View file

@ -27,12 +27,13 @@ jobs:
# run inside this archlinux container — the runner execs them with
# node. This shell step needs no node, so installing it here (before
# Checkout) is enough.
# ffmpeg is for ffprobe, which tools/fetch-media.sh uses to read the
# pixel dimensions of each mirrored file. Those become the width/height
# attributes that stop the posts page reflowing as 5 MB recordings
# arrive, and tools/e2e.sh asserts they are present — so without this
# package the deploy fails at the e2e gate rather than shipping a
# janky page.
# ffmpeg/ffprobe are both used by tools/fetch-media.sh. ffprobe reads
# the pixel dimensions of each mirrored file, which become the
# width/height attributes that stop the posts page reflowing as 5 MB
# recordings arrive. ffmpeg transcodes each still image into the AVIF
# and PNG renditions the page serves it between. tools/e2e.sh asserts
# both are present — so without this package the deploy fails at the
# e2e gate rather than shipping a janky page.
pacman -Syu --noconfirm --needed \
nodejs \
clang lld libc++ \
@ -84,18 +85,19 @@ jobs:
run: tools/fetch-rates.sh
- name: Fetch fediverse posts
# Build-time, not run-time: the site embeds the owner's own posts and
# links out for discussion, so there is no sync service and no runtime
# dependency on the instance being up. The script leaves the committed
# content/posts.json untouched and exits 0 on any failure, so a
# Build-time, not run-time: the site hosts the owner's own posts in full
# and links out for the discussion, so there is no sync service and no
# runtime dependency on the instance being up. The script leaves the
# committed content/posts.json untouched and exits 0 on any failure, so a
# fediverse outage cannot fail a deploy.
run: tools/fetch-posts.sh
- name: Mirror post media
# Downloads the images and screen recordings the posts carry and rewrites
# content/posts.json to point at our own copies, so nothing the browser
# loads is third-party — which is what keeps the privacy notice's
# "everything comes from catcrafts.net" true.
# Downloads the images and screen recordings the posts carry — both the
# headline file and everything embedded inside the body — and rewrites
# content/posts.json (including the body text itself) to point at our own
# copies, so nothing the browser loads is third-party. That is what keeps
# the privacy notice's "everything comes from catcrafts.net" true.
#
# Content-addressed: the name is the hash of the bytes. Note what that
# does NOT mean — a third-party file is re-fetched on every build, because

231
content/posts.json
View file

File diff suppressed because one or more lines are too long

56
deploy/README.md
View file

@ -128,6 +128,62 @@ build. Neither fails the build on a network error — a fediverse outage leaves
previous `posts.json` in place, and a single failed download leaves that one entry
pointing at its original URL rather than losing the post.
### The full body, and its inline media
Each post is hosted whole at `/posts/<slug>`, with the card on `/posts` linking
to it — the writing is what the site is about, and a page this site can name as
canonical is the only version a search engine can be pointed at. The comments
are *not* mirrored: every post page links out to the thread, which is where the
discussion belongs.
The body stays Markdown in `posts.json` and is rendered by
`Catcrafts.Shared:Markdown` at page-render time, never converted to HTML by the
shell. That is deliberate: the renderer is inside the escaping guarantee, and
text fetched from someone else's server must not be able to become markup
anywhere else. Raw HTML in a body is always shown as text.
`fetch-media.sh` mirrors what the body embeds as well as the headline file, and
rewrites the URLs **inside the Markdown**, so a post page loads nothing
third-party either. It also writes a `body_media` list per post — dimensions,
poster frame and format renditions for each inline file, which Markdown syntax
has nowhere to carry. Slugs come from the title; a duplicate title takes the
post's numeric id as a suffix, so an old post's URL is never renumbered by a new
one. Re-running the script is a no-op: local paths are adopted from the mount
rather than re-fetched.
### The image format ladder
Every mirrored still image is transcoded to two siblings named after its content
hash, and `Catcrafts.Shared:Media` serves all three as one `<picture>` — so the
browser fetches **exactly one**:
| tier | file | size vs. WebP | who gets it |
|---|---|---|---|
| `<source type="image/avif">` | `<hash>.avif` | **76%** | almost everyone |
| `<source type="image/webp">` | `<hash>.webp` (the mirrored original) | 100% | Safari 14–16 |
| `<img src>` | `<hash>.png` | **875%** | neither of the above |
The middle tier is why the PNG being ~9× the WebP does not matter: it is free
(the mirror already downloaded that file) and it is what the small number of
non-AVIF browsers actually land on. The PNG is the floor nothing can refuse.
AVIF is encoded at `crf 26, cpu-used 6, yuv444p` — measured at SSIM 0.997
against the source and still smaller than it. Full chroma is deliberate: these
are screenshots of text, and re-subsampling chroma that pict-rs already
subsampled once fringes coloured text visibly, for about 3% more bytes.
Encoding is skipped when the sibling is already on the mount, so only genuinely
new images cost encoder time (~0.5 s each). Animated sources are left alone
entirely — one moving GIF beats three copies of its first frame. Video posters
are skipped too: `poster` takes exactly one URL, so a `<video>` cannot negotiate
a format the way `<picture>` can and the renditions would be unreachable.
Both encodes pin `-c:v` and then **verify the codec that actually came out**.
That check earned its place immediately: `-f image2 out.png` without an explicit
codec makes ffmpeg fall back to the muxer default, which is MJPEG — it silently
produced a full set of lossy JPEGs under `.png` names, served to browsers as
`image/png`. A rendition that fails the check is discarded and its tier dropped.
### Publish the media first, then post it
**The recommended flow is to put a recording on catcrafts.net before writing the

3
implementations/Catcrafts-Root.cpp
View file

@ -177,8 +177,7 @@ namespace Catcrafts {
// restarting at cc-link-0 and colliding with the content links —
// getElementById would then return whichever appeared first in the
// document and half the links would navigate to the wrong place.
const RouteKind navKind =
route.kind == RouteKind::LegacyBlog ? RouteKind::Posts : route.kind;
const RouteKind navKind = NavKindFor(route.kind);
Dom::HtmlElementPtr header("cc-header");
if (header.ptr != 0) {
header.SetInnerHTML(TagLinks(Views::RenderNav(navKind).View(), linkTargets));

4
project.cpp
View file

@ -45,12 +45,14 @@ static Configuration* SharedLibrary(std::span<const std::string_view> args) {
ApplyStandardArgs(*shared, args); // inherits --target / --debug from the parent
shared->type = ConfigurationType::LibraryStatic;
std::array<fs::path, 9> ifaces = {
std::array<fs::path, 11> ifaces = {
"shared/interfaces/Catcrafts.Shared",
"shared/interfaces/Catcrafts.Shared-Html",
"shared/interfaces/Catcrafts.Shared-Form",
"shared/interfaces/Catcrafts.Shared-Json",
"shared/interfaces/Catcrafts.Shared-Model",
"shared/interfaces/Catcrafts.Shared-Media",
"shared/interfaces/Catcrafts.Shared-Markdown",
"shared/interfaces/Catcrafts.Shared-Content",
"shared/interfaces/Catcrafts.Shared-Money",
"shared/interfaces/Catcrafts.Shared-Route",

12
server/implementations/Catcrafts.Server-Http.cpp
View file

@ -287,8 +287,7 @@ HTTPResponse RenderPage(std::string_view target) {
if (const Demo* d = gContent.FindDemo(route.slug)) wantsWasm = d->needsWasm;
}
res.body = Views::RenderDocument(page,
Views::RenderNav(route.kind == RouteKind::LegacyBlog
? RouteKind::Posts : route.kind),
Views::RenderNav(NavKindFor(route.kind)),
Views::RenderFooter(),
wantsWasm ? gBootScripts : std::string_view{},
gCssHref);
@ -310,6 +309,15 @@ HTTPResponse ServeSitemap() {
out += Html::Escape(pr.slug).Str();
out += "</loc></url>\n";
}
// Post pages, from the same HasPage() test the "read more" links use — a
// sitemap that advertised a post without a body would be pointing crawlers
// at the 404 the dispatcher correctly returns for it.
for (const Post& po : gContent.posts) {
if (!po.HasPage()) continue;
out += " <url><loc>https://catcrafts.net/posts/";
out += Html::Escape(po.slug).Str();
out += "</loc></url>\n";
}
out += "</urlset>\n";
ApplyPageHeaders(res, "application/xml; charset=utf-8", true, false);
res.body = std::move(out);

403
server/implementations/main.cpp
View file

@ -109,6 +109,394 @@ void RunSelfTest() {
CheckEq(Escape("a") + Escape("<"), "a&lt;", "operator+: escapes preserved");
}
// The format ladder. One <picture>/<video> builder serves both the cards and
// the post bodies, so these assertions cover every image and video the site
// emits — and the ordering ones matter: a browser takes the FIRST source it
// understands, so a mis-ordered ladder silently serves the wrong tier to
// everyone rather than failing visibly.
void RunMediaSelfTest() {
auto img = [](std::string src, std::string avif, std::string png,
std::int64_t w = 0, std::int64_t h = 0) {
PostMedia m;
m.src = std::move(src);
m.kind = "image";
m.avif = std::move(avif);
m.fallback = std::move(png);
m.width = w;
m.height = h;
return m;
};
// The full ladder: AVIF, then the mirrored original, then the PNG the <img>
// itself points at. Exactly one of the three is ever fetched.
CheckEq(Media::Tag(img("/media/x.webp", "/media/x.avif", "/media/x.png", 800, 600)),
R"(<picture><source srcset="/media/x.avif" type="image/avif">)"
R"(<source srcset="/media/x.webp" type="image/webp">)"
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.png" width="800" height="600"></picture>)",
"media: image ladder is avif, original, png");
// Alt text reaches the <img>, not the <picture> — a screen reader reads the
// img, and an alt on the wrapper is invisible to it.
Check(Media::Tag(img("/media/x.webp", "/media/x.avif", "/media/x.png"), "a cat")
.View().find(R"(alt="a cat" src="/media/x.png")") != std::string_view::npos,
"media: alt lands on the img");
// Degradation, one tier at a time. Each of these is a real state: no
// encoder on the build host, a source that was already PNG, a body image
// whose download failed so there is nothing but the original URL.
CheckEq(Media::Tag(img("/media/x.webp", "", "/media/x.png")),
R"(<picture><source srcset="/media/x.webp" type="image/webp">)"
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.png"></picture>)",
"media: no avif still offers the original above the png");
CheckEq(Media::Tag(img("/media/x.webp", "/media/x.avif", "")),
R"(<picture><source srcset="/media/x.avif" type="image/avif">)"
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.webp"></picture>)",
"media: no png leaves the original as the base");
CheckEq(Media::Tag(img("/media/x.webp", "", "")),
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.webp">)",
"media: no renditions is a bare img, as before any of this existed");
// A source that is already PNG is its own fallback, and must not be
// offered twice — once as a <source> and once as the <img>.
CheckEq(Media::Tag(img("/media/x.png", "/media/x.avif", "/media/x.png")),
R"(<picture><source srcset="/media/x.avif" type="image/avif">)"
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.png"></picture>)",
"media: a png source is not also listed as a source");
// Likewise a source that is already AVIF.
Check(Media::Tag(img("/media/x.avif", "/media/x.avif", "/media/x.png"))
.View().find("image/avif\"><source") == std::string_view::npos,
"media: an avif source is not listed twice");
// A URL is still a URL: the scheme allowlist applies to srcset exactly as
// it does to src, or the ladder becomes a way around it.
Check(Media::Tag(img("/media/x.webp", "javascript:alert(1)", "/media/x.png"))
.View().find(R"(srcset="#")") != std::string_view::npos,
"media: a hostile srcset is neutralised");
// Video is unchanged by any of this and must stay so.
{
PostMedia v;
v.src = "/media/v.mp4";
v.kind = "video";
v.poster = "/media/v.webp";
v.fallback = "/media/v.h264.mp4";
const auto out = Media::Tag(v);
Check(out.View().starts_with("<video class=\"post-media__item\" controls preload=\"metadata\""),
"media: video is still a video", out.View());
Check(out.View().find("codecs=av01") != std::string_view::npos
&& out.View().find(R"(<source src="/media/v.h264.mp4" type="video/mp4">)")
!= std::string_view::npos,
"media: AV1 then H.264, in that order");
Check(out.View().find("<picture>") == std::string_view::npos,
"media: a video is not wrapped in a picture");
}
// A path with no record renders from the path alone — the mirror failed,
// and showing the picture beats dropping the paragraph's subject.
{
const std::array<PostMedia, 1> known{
img("/media/x.webp", "/media/x.avif", "/media/x.png") };
Check(Media::Find(known, "/media/x.webp") != nullptr, "media: found by src");
Check(Media::Find(known, "/media/nope.webp") == nullptr, "media: unknown src");
const PostMedia guessed = Media::Describe(known, "https://i.example/a.webp");
Check(guessed.kind == "image" && guessed.avif.empty(),
"media: an unmirrored image is described from its path");
Check(Media::Describe(known, "https://i.example/a.mp4").kind == "video",
"media: an unmirrored video is recognised as one");
}
}
// The Markdown renderer, which is the newest place untrusted text becomes
// markup — post bodies are fetched from someone else's server, so every one of
// these assertions is ultimately about the same thing: nothing in a body can
// escape into the document. The structural cases are here too, because a parser
// that silently drops a construct loses content invisibly.
void RunMarkdownSelfTest() {
auto md = [](std::string_view text,
std::span<const PostMedia> media = {}) {
return Markdown::Render(text, media);
};
// ── the guarantee ─────────────────────────────────────────────────
CheckEq(md("<script>alert(1)</script>"),
"<p>&lt;script&gt;alert(1)&lt;/script&gt;</p>", "md: html is text, never markup");
CheckEq(md("![x](javascript:alert(1))"),
R"(<div class="post-media"><img class="post-media__item" loading="lazy" )"
R"(decoding="async" alt="x" src="#"></div>)",
"md: javascript: image source neutralised");
CheckEq(md("[x](javascript:alert(1))"),
R"(<p><a href="#" rel="noopener">x</a></p>)",
"md: javascript: link neutralised");
CheckEq(md("a \" b & c"), "<p>a &quot; b &amp; c</p>", "md: quotes and ampersands escaped");
// A code span is verbatim text, and verbatim is exactly where an escaper
// is most often forgotten.
CheckEq(md("`<b>`"), "<p><code>&lt;b&gt;</code></p>", "md: code span escaped");
// ── blocks ────────────────────────────────────────────────────────
CheckEq(md(""), "", "md: empty body renders nothing");
CheckEq(md("plain text"), "<p>plain text</p>", "md: paragraph");
// Demotion by one: the page h1 is the post title, so a body's own top-level
// heading is a section within it.
CheckEq(md("# Heading"), "<h2>Heading</h2>", "md: h1 demoted to h2");
CheckEq(md("### Heading"), "<h4>Heading</h4>", "md: h3 demoted to h4");
CheckEq(md("#nothashtag"), "<p>#nothashtag</p>", "md: # without a space is not a heading");
CheckEq(md("> quoted"),
R"(<blockquote class="post-body__quote"><p>quoted</p></blockquote>)",
"md: blockquote");
// The quoted lines are re-parsed, so a multi-paragraph quote keeps its
// paragraphs instead of collapsing into one run-on line.
CheckEq(md("> one\n>\n> two"),
R"(<blockquote class="post-body__quote"><p>one</p><p>two</p></blockquote>)",
"md: blockquote keeps its paragraphs");
CheckEq(md("- a\n- b"),
R"(<ul class="post-body__list"><li>a</li><li>b</li></ul>)", "md: unordered list");
CheckEq(md("1. a\n2. b"),
R"(<ol class="post-body__list"><li>a</li><li>b</li></ol>)", "md: ordered list");
// A list resumed after an interrupting paragraph continues its numbering.
// Without the start attribute the mini-guide in one of these posts renders
// as steps 1-4 followed by steps 1, 2, 3.
CheckEq(md("5. e"),
R"(<ol class="post-body__list" start="5"><li>e</li></ol>)",
"md: ordered list keeps the number it announced");
// Blank lines between items are spacing, not seven one-item lists.
CheckEq(md("1. a\n\n2. b"),
R"(<ol class="post-body__list"><li>a</li><li>b</li></ol>)",
"md: blank line inside a list does not split it");
CheckEq(md("---"), "<hr>", "md: thematic break");
CheckEq(md("- - -"), "<hr>", "md: spaced rule is not a one-item list");
// Whitespace in pasted terminal output is the content.
CheckEq(md("```\n a\tb\n```"),
"<pre class=\"post-body__code\"><code> a\tb\n</code></pre>",
"md: fenced code is verbatim");
// An unterminated fence must not swallow the document into nothing.
Check(md("```\nx").View().find("<code>x") != std::string_view::npos,
"md: unterminated fence still renders its content");
// ── inline ────────────────────────────────────────────────────────
CheckEq(md("**bold**"), "<p><strong>bold</strong></p>", "md: strong");
CheckEq(md("*em*"), "<p><em>em</em></p>", "md: emphasis");
CheckEq(md("2 * 3 * 4"), "<p>2 * 3 * 4</p>", "md: spaced asterisks stay literal");
// Underscores are deliberately inert: these posts paste kernel symbol
// names into prose, and italicising half of one is worse than not
// italicising a word that used the underscore form.
CheckEq(md("kworker/u16:8-qc_ufs_qos_swq"),
"<p>kworker/u16:8-qc_ufs_qos_swq</p>", "md: underscores are not emphasis");
CheckEq(md("\\*literal\\*"), "<p>*literal*</p>", "md: backslash escape");
CheckEq(md("[label](https://x.example/y)"),
R"(<p><a href="https://x.example/y" rel="noopener">label</a></p>)", "md: link");
// Bare addresses are pasted constantly in these posts; leaving them inert
// would strip most of the outbound value out of the page.
CheckEq(md("see https://x.example/y"),
R"(<p>see <a href="https://x.example/y">https://x.example/y</a></p>)",
"md: bare URL autolinked");
// ── embedded media ────────────────────────────────────────────────
// A paragraph that is nothing but images becomes the same media block the
// cards use, rather than a <p> of pictures.
CheckEq(md("![a](/media/x.webp)"),
R"(<div class="post-media"><img class="post-media__item" loading="lazy" )"
R"(decoding="async" alt="a" src="/media/x.webp"></div>)",
"md: image-only paragraph is a media block");
Check(md("text ![a](/media/x.webp)").View().starts_with("<p>text <img"),
"md: an image inside a sentence stays inline");
// Dimensions come from the sidecar list, because Markdown syntax has
// nowhere to carry them — and without them the prose below every
// screenshot jumps as the file arrives.
{
std::vector<PostMedia> media;
PostMedia img;
img.src = "/media/x.webp";
img.kind = "image";
img.avif = "/media/x.avif";
img.fallback = "/media/x.png";
img.width = 800;
img.height = 600;
media.push_back(img);
PostMedia vid;
vid.src = "/media/v.mp4";
vid.kind = "video";
vid.poster = "/media/v.poster.webp";
vid.fallback = "/media/v.h264.mp4";
vid.width = 1080;
vid.height = 1920;
media.push_back(vid);
const auto out = md("![a](/media/x.webp)", media);
Check(out.View().find(R"(width="800" height="600")") != std::string_view::npos,
"md: inline image carries its dimensions", out.View());
// Routed through :Media, so a body image gets the same format ladder a
// card image does rather than a second, plainer implementation.
Check(out.View().find(R"(<source srcset="/media/x.avif" type="image/avif">)")
!= std::string_view::npos
&& out.View().find(R"(src="/media/x.png")") != std::string_view::npos,
"md: inline image gets the avif/png ladder", out.View());
// An inline video gets the same treatment a headline one does,
// fallback source and all.
const auto vout = md("![](/media/v.mp4)", media);
Check(vout.View().find(R"(poster="/media/v.poster.webp")") != std::string_view::npos
&& vout.View().find("codecs=av01") != std::string_view::npos
&& vout.View().find(R"(<source src="/media/v.h264.mp4")") != std::string_view::npos,
"md: inline video gets poster and H.264 fallback", vout.View());
}
// ── termination ───────────────────────────────────────────────────
// Unbalanced delimiters are the classic way to hang a hand-written
// parser, and a body is input from someone else's server.
Check(!md("**unclosed").View().empty(), "md: unclosed strong terminates");
Check(!md("[unclosed](").View().empty(), "md: unclosed link terminates");
Check(!md("![](").View().empty(), "md: unclosed image terminates");
Check(!md("`unclosed").View().empty(), "md: unclosed code span terminates");
Check(!md("> > > > > > > > deep").View().empty(), "md: over-deep nesting terminates");
}
// Post pages: the routing, the loader's guard on what becomes a URL, and the
// schema.org joins that keep every post attributed to the one Organization and
// the one Person the rest of the site describes.
void RunPostSelfTest() {
// ── routing ───────────────────────────────────────────────────────
Check(ParseRoute("/posts").kind == RouteKind::Posts, "route: /posts is the list");
Check(ParseRoute("/posts/hello-world").kind == RouteKind::Post, "route: /posts/<slug>");
Check(ParseRoute("/posts/hello-world").slug == "hello-world", "route: post slug captured");
Check(ParseRoute("/posts/hello-world/").kind == RouteKind::Post,
"route: trailing slash normalised");
Check(ParseRoute("/posts/Hello").kind == RouteKind::NotFound,
"route: uppercase slug is not a post URL");
Check(ParseRoute("/posts/../etc").kind == RouteKind::NotFound,
"route: traversal never reaches a lookup");
Check(NavKindFor(RouteKind::Post) == RouteKind::Posts,
"route: a post page highlights the Posts nav entry");
Check(NavKindFor(RouteKind::LegacyBlog) == RouteKind::Posts,
"route: the retired /blog URL highlights it too");
Check(NavKindFor(RouteKind::Shop) == RouteKind::Shop, "route: a nav route is its own entry");
// ── the loader ────────────────────────────────────────────────────
{
const auto posts = LoadPosts(R"([
{"title":"Good","slug":"good-post","permalink":"https://i.example/post/1",
"body":"Hello.","published":"2026-01-01T00:00:00Z",
"body_media":[{"src":"/media/a.webp","kind":"image","w":10,"h":20}]},
{"title":"Bad slug","slug":"NOT A SLUG","permalink":"https://i.example/post/2",
"body":"Hello."},
{"title":"No body","slug":"no-body","permalink":"https://i.example/post/3"}
])");
Check(posts.size() == 3, "posts: all three load");
if (posts.size() == 3) {
Check(posts[0].HasPage() && posts[0].slug == "good-post", "posts: valid slug kept");
Check(posts[0].body == "Hello.", "posts: body loaded");
Check(posts[0].bodyMedia.size() == 1 && posts[0].bodyMedia[0].width == 10,
"posts: body media loaded with dimensions");
// A slug that could never match a route would render a "read more"
// link to a 404 this site points at itself.
Check(posts[1].slug.empty() && !posts[1].HasPage(),
"posts: malformed slug is dropped, costing the page");
// A title and a link out is not a page worth minting a URL for.
Check(!posts[2].HasPage(), "posts: no body means no page");
}
Views::SiteContent content;
content.posts = posts;
Check(content.FindPost("good-post") != nullptr, "posts: found by slug");
Check(content.FindPost("no-body") == nullptr, "posts: a pageless post is not findable");
Check(content.FindPost("nope") == nullptr, "posts: unknown slug is not found");
// Which means the route 404s rather than rendering an empty article.
Check(Views::RenderRoute(ParseRoute("/posts/no-body"), content).status == 404,
"posts: a pageless slug is a real 404");
Check(Views::RenderRoute(ParseRoute("/posts/good-post"), content).status == 200,
"posts: a real post renders");
}
// ── the page ──────────────────────────────────────────────────────
{
Post p;
p.title = "Working GPS!";
p.slug = "working-gps";
p.permalink = "https://lemmy.example/post/42";
p.community = "linuxphones@lemmy.example";
p.published = "2026-06-26T23:16:25Z";
p.excerpt = "A short summary.";
p.body = "# How\n\nIt works.";
PostMedia m;
m.src = "/media/shot.webp";
m.kind = "image";
p.media.push_back(m);
const auto page = Views::RenderPost(p);
Check(page.meta.canonical == "/posts/working-gps",
"post page: canonical is this site, not the instance");
Check(page.meta.ogType == "article", "post page: og:type is article");
Check(page.meta.ogImage == "/media/shot.webp", "post page: og:image from the post media");
Check(page.meta.description == p.excerpt, "post page: description is the excerpt");
Check(page.main.View().find("<h2>How</h2>") != std::string_view::npos,
"post page: the body is rendered, not escaped away");
// The whole reason the body is hosted: the thread is still one click
// away, and the reader is told where the discussion is.
Check(page.main.View().find(p.permalink) != std::string_view::npos,
"post page: still links the thread");
// The identity graph, same joins every other page makes. A typo'd @id
// still renders and still validates — it just quietly splits this post
// away from the entity the rest of the site describes.
auto ld = Json::Parse(page.meta.jsonLd);
Check(ld && ld->IsObject() && ld->Str("@type") == "BlogPosting",
"post schema: parses as a BlogPosting");
if (ld && ld->IsObject()) {
Check(ld->Str("url") == "https://catcrafts.net/posts/working-gps",
"post schema: url is the on-site page");
Check(ld->Str("discussionUrl") == p.permalink,
"post schema: the thread is the discussion, not the content");
Check(ld->Str("datePublished") == p.published, "post schema: publication date");
const Json::Value* author = ld->Find("author");
const Json::Value* publisher = ld->Find("publisher");
Check(author && author->Str("@id") == "https://catcrafts.net/about#person",
"post schema: authored by the Person node on /about");
Check(publisher && publisher->Str("@id") == "https://catcrafts.net/#organization",
"post schema: published by the Organization node");
}
// A post with a page is advertised as one from the list and from home;
// one without keeps pointing at the thread, because there is nothing
// here to send the reader to.
const std::array<Post, 1> one{ p };
const auto list = Views::RenderPosts(one);
Check(list.main.View().find(R"(href="/posts/working-gps")") != std::string_view::npos,
"posts list: links the on-site page");
Check(list.main.View().find("Read the full post") != std::string_view::npos,
"posts list: offers the full post");
// Inside the excerpt paragraph, trailing the text — not a row of its
// own below the media, where it was the same offer made a screen
// further down.
Check(list.main.View().find(
R"(A short summary. <a class="link-more" href="/posts/working-gps">)"
R"(Read the full post</a></p>)") != std::string_view::npos,
"posts list: read-more trails the excerpt", list.main.View());
// With no excerpt there is no sentence to continue, so it falls back to
// a row rather than vanishing with the paragraph that would have held it.
Post unexcerpted = p;
unexcerpted.excerpt.clear();
const std::array<Post, 1> bare{ unexcerpted };
const auto listBare = Views::RenderPosts(bare);
Check(listBare.main.View().find("Read the full post") != std::string_view::npos,
"posts list: an excerptless post still offers the full post");
Post bodyless = p;
bodyless.body.clear();
const std::array<Post, 1> none{ bodyless };
const auto listNone = Views::RenderPosts(none);
Check(listNone.main.View().find("Read the full post") == std::string_view::npos,
"posts list: no read-more without a page to read");
Check(listNone.main.View().find(p.permalink) != std::string_view::npos,
"posts list: a pageless post still links its thread");
}
}
void RunJsonSelfTest() {
using namespace Catcrafts::Json;
@ -810,6 +1198,9 @@ int main(int argc, char** argv) {
RunJsonSelfTest();
RunFormSelfTest();
RunMoneySelfTest();
RunMediaSelfTest();
RunMarkdownSelfTest();
RunPostSelfTest();
if (failures == 0) {
std::println("Catcrafts.Shared self-test: all assertions passed");
return 0;
@ -829,7 +1220,7 @@ int main(int argc, char** argv) {
const Views::RenderedPage page = Views::RenderRoute(route, content);
std::print("{}", Views::RenderDocument(
page,
Views::RenderNav(route.kind == RouteKind::LegacyBlog ? RouteKind::Posts : route.kind),
Views::RenderNav(NavKindFor(route.kind)),
Views::RenderFooter(),
/*bootScripts=*/"", // no wasm on a plain server render
/*cssHref=*/"/styles.css"));
@ -859,6 +1250,14 @@ int main(int argc, char** argv) {
std::print(" <url><loc>https://catcrafts.net/shop/{}</loc></url>\n",
Html::Escape(pr.slug).Str());
}
// Same rule as the served sitemap: only posts that actually have a
// page. Both must agree, because this is the copy baked into the wasm
// bundle and that one is what a crawler fetches.
for (const Post& po : content.posts) {
if (!po.HasPage()) continue;
std::print(" <url><loc>https://catcrafts.net/posts/{}</loc></url>\n",
Html::Escape(po.slug).Str());
}
std::print("</urlset>\n");
return 0;
}
@ -877,7 +1276,7 @@ int main(int argc, char** argv) {
"/order/not-a-token",
"/legal/privacy", "/legal/imprint",
"/legal/terms", "/legal/nope",
"/projects", "/posts", "/demos",
"/projects", "/posts", "/posts/nope", "/demos",
"/demos/raytracer", "/demos/nope", "/demo",
"/projects/", "/blog", "/blog/hello-world", "/nope" }) {
const Route r = ParseRoute(p);

588
shared/interfaces/Catcrafts.Shared-Markdown.cppm Normal file
View file

@ -0,0 +1,588 @@
/*
catcrafts.net
Copyright (C) 2026 Catcrafts
The source code of this website is made available for viewing purposes only.
No permission is granted to copy, modify, distribute, or create derivative works.
*/
// A deliberately small Markdown renderer, for fediverse post bodies only.
//
// The site used to have no markdown pipeline at all, on the grounds that the
// posts page only ever showed a 280-character preview and the body stayed on
// the instance. Hosting the full body changes that calculation: a post IS
// prose with headings, quotes, code blocks and screenshots, and rendering it
// as one flat paragraph of literal `**asterisks**` would be worse than not
// hosting it. So: a parser, but only as much of one as these bodies use.
//
// WHAT IT SUPPORTS — everything observed in the real bodies, and nothing else:
//
// blocks ATX headings, fenced code, blockquotes (nested), ordered and
// unordered lists, thematic breaks, paragraphs
// inline links, images, code spans, ** strong **, * emphasis *, and bare
// URLs via Html::Autolink
//
// WHAT IT DELIBERATELY DOES NOT SUPPORT:
//
// * Raw HTML. Never. A post body is text fetched from someone else's server,
// so the ONE thing this renderer must guarantee is that no byte of it can
// become markup. Every character of body text leaves here through
// Html::Escape or Html::Autolink, and the only SafeHtml built from a raw
// string is the fixed structural markup written in this file. That is why
// `<` in a body renders as a less-than sign rather than opening a tag.
// * Underscore emphasis. `_` is common inside identifiers that appear in
// these posts unquoted (kworker/u16:8-qc_ufs_qos_swq), and mangling half a
// symbol name into italics is a worse failure than not italicising a word
// that used the underscore form. Asterisks are unambiguous here.
// * Setext headings, reference links, tables, footnotes, HTML entities.
// None appear; adding them speculatively is parser surface with no reader.
// * Trailing-double-space hard breaks. An invisible two-character difference
// is not something a reader can see in the source or a writer can rely on
// having typed; the lines of a paragraph join with a space, and a break
// that was meant is written as a blank line.
//
// Anything unrecognised degrades to text rather than being dropped, so a
// construct this parser does not know shows up as visibly odd prose instead of
// silently vanishing from the page.
export module Catcrafts.Shared:Markdown;
import std;
import :Html;
import :Media;
import :Model;
namespace Catcrafts::Markdown {
using Html::SafeHtml;
using Html::Escape;
using Html::Autolink;
using Html::Attr;
using Html::Format;
using Html::Join;
using Html::Raw;
using Html::Url;
// Blockquotes recurse, and a body is untrusted input, so the recursion needs a
// bound that does not depend on the input being sane. Four is past anything
// these posts do (a quote inside a list item inside a quote) and far short of
// anything that could trouble the stack.
constexpr int kMaxDepth = 4;
// ── small string helpers ──────────────────────────────────────────────
bool IsSpace(char c) { return c == ' ' || c == '\t'; }
std::string_view TrimRight(std::string_view s) {
while (!s.empty() && (IsSpace(s.back()) || s.back() == '\r')) s.remove_suffix(1);
return s;
}
std::string_view TrimLeft(std::string_view s) {
while (!s.empty() && IsSpace(s.front())) s.remove_prefix(1);
return s;
}
std::string_view Trim(std::string_view s) { return TrimLeft(TrimRight(s)); }
bool Blank(std::string_view line) { return Trim(line).empty(); }
// Up to three leading spaces are indentation a block marker is still allowed
// to carry; four or more would be a code block in real Markdown, which these
// bodies never use (they fence instead).
std::string_view Undent(std::string_view line) {
std::size_t n = 0;
while (n < line.size() && n < 3 && line[n] == ' ') ++n;
return line.substr(n);
}
// ── media ─────────────────────────────────────────────────────────────
// One embedded file, in the same shape (and CSS) the posts page uses for a
// post's headline media — an inline screenshot and a post's headline recording
// are the same kind of thing to a reader, so they should not look like two
// different components. :Media is what guarantees that: both go through it.
//
// A src with no mirrored record (its download failed, so the body still points
// at the original URL) still renders, just without dimensions or format tiers.
SafeHtml MediaTag(std::string_view src, std::string_view alt,
std::span<const PostMedia> media) {
return Media::Tag(Media::Describe(media, src), alt);
}
// ── inline ────────────────────────────────────────────────────────────
SafeHtml RenderInline(std::string_view text, std::span<const PostMedia> media, int depth);
// The span between `open` and its matching close delimiter, honouring nesting,
// or npos when it never closes. Used for [link text] and (link target), both of
// which can legitimately contain their own brackets — the Fairphone manual link
// in one of these posts has a parenthesised sentence as its text.
std::size_t MatchingDelimiter(std::string_view s, std::size_t from, char open, char close) {
int depth = 0;
for (std::size_t i = from; i < s.size(); ++i) {
if (s[i] == '\\') { ++i; continue; }
if (s[i] == open) { ++depth; continue; }
if (s[i] == close) {
if (depth == 0) return i;
--depth;
}
}
return std::string_view::npos;
}
// A run of ordinary prose, escaped and with bare URLs linked.
//
// Autolink rather than Escape because these bodies paste addresses constantly —
// mailing-list archives, merge requests, the shop — as bare text with no
// Markdown link syntax around them. Rendering those inert would strip most of
// the outbound value out of the post. Autolink escapes everything itself and
// puts each address through Html::Url, so this adds no new trusted path.
SafeHtml PlainRun(std::string_view text) {
return text.empty() ? SafeHtml{} : Autolink(text);
}
// True when the delimiter at `i` opens (rather than closes) an emphasis run:
// it must be followed by something that is not whitespace. Combined with the
// closing test below this is what keeps a lone asterisk — a multiplication
// sign, a footnote marker, a shell glob — from swallowing the rest of a
// paragraph into italics.
bool OpensEmphasis(std::string_view s, std::size_t after) {
return after < s.size() && !IsSpace(s[after]) && s[after] != '\n';
}
// The closing delimiter for an emphasis run opened at `from`, or npos. The
// character before it must not be whitespace, so "a * b * c" stays literal.
std::size_t FindEmphasisClose(std::string_view s, std::size_t from, std::string_view delim) {
std::size_t i = from;
while (i < s.size()) {
const std::size_t at = s.find(delim, i);
if (at == std::string_view::npos) return std::string_view::npos;
if (at > from && !IsSpace(s[at - 1])) return at;
i = at + delim.size();
}
return std::string_view::npos;
}
SafeHtml RenderInline(std::string_view text, std::span<const PostMedia> media, int depth) {
std::vector<SafeHtml> out;
std::size_t run = 0; // start of the pending plain-text run
auto flush = [&](std::size_t upto) {
if (upto > run) out.push_back(PlainRun(text.substr(run, upto - run)));
};
std::size_t i = 0;
while (i < text.size()) {
const char c = text[i];
// A backslash escape hides the next character from this parser. The
// pair is emitted as the second character alone, which is what lets a
// post write a literal asterisk.
if (c == '\\' && i + 1 < text.size()) {
flush(i);
out.push_back(Escape(text.substr(i + 1, 1)));
i += 2;
run = i;
continue;
}
// `code` — highest precedence, so an asterisk inside a code span is a
// literal asterisk and not an emphasis delimiter.
if (c == '`') {
const std::size_t close = text.find('`', i + 1);
if (close != std::string_view::npos) {
flush(i);
out.push_back(Format(R"(<code>{}</code>)",
Escape(text.substr(i + 1, close - i - 1))));
i = close + 1;
run = i;
continue;
}
}
// ![alt](src) — an image. Checked before the link case, since the
// bracket that follows would otherwise parse as one.
if (c == '!' && i + 1 < text.size() && text[i + 1] == '[' && depth < kMaxDepth) {
const std::size_t altEnd = MatchingDelimiter(text, i + 2, '[', ']');
if (altEnd != std::string_view::npos && altEnd + 1 < text.size()
&& text[altEnd + 1] == '(') {
const std::size_t srcEnd = MatchingDelimiter(text, altEnd + 2, '(', ')');
if (srcEnd != std::string_view::npos) {
flush(i);
out.push_back(MediaTag(Trim(text.substr(altEnd + 2, srcEnd - altEnd - 2)),
text.substr(i + 2, altEnd - i - 2),
media));
i = srcEnd + 1;
run = i;
continue;
}
}
}
// [text](href)
if (c == '[' && depth < kMaxDepth) {
const std::size_t textEnd = MatchingDelimiter(text, i + 1, '[', ']');
if (textEnd != std::string_view::npos && textEnd + 1 < text.size()
&& text[textEnd + 1] == '(') {
const std::size_t hrefEnd = MatchingDelimiter(text, textEnd + 2, '(', ')');
if (hrefEnd != std::string_view::npos) {
const std::string_view href =
Trim(text.substr(textEnd + 2, hrefEnd - textEnd - 2));
flush(i);
// The label is rendered rather than escaped flat, because
// these posts bold inside link text. Depth-guarded, so a
// link whose label contains a link cannot recurse forever.
out.push_back(Format(
R"(<a{} rel="noopener">{}</a>)",
Url("href", href),
RenderInline(text.substr(i + 1, textEnd - i - 1), media, depth + 1)));
i = hrefEnd + 1;
run = i;
continue;
}
}
}
// **strong** before *emphasis*: the longer delimiter has to win, or
// every bold run parses as an empty italic followed by loose text.
if (c == '*') {
const bool doubled = i + 1 < text.size() && text[i + 1] == '*';
const std::string_view delim = doubled ? "**" : "*";
const std::size_t inner = i + delim.size();
if (depth < kMaxDepth && OpensEmphasis(text, inner)) {
const std::size_t close = FindEmphasisClose(text, inner, delim);
if (close != std::string_view::npos) {
flush(i);
const SafeHtml body =
RenderInline(text.substr(inner, close - inner), media, depth + 1);
out.push_back(doubled ? Format(R"(<strong>{}</strong>)", body)
: Format(R"(<em>{}</em>)", body));
i = close + delim.size();
run = i;
continue;
}
}
}
++i;
}
flush(text.size());
return Join(out);
}
// ── blocks ────────────────────────────────────────────────────────────
// A line's leading run of '#', when it is an ATX heading marker.
// Returns 0 when the line is not a heading.
int HeadingLevel(std::string_view line) {
std::size_t n = 0;
while (n < line.size() && line[n] == '#') ++n;
if (n == 0 || n > 6) return 0;
// "#tag" is not a heading; a marker has to be followed by space or be the
// whole line.
if (n < line.size() && !IsSpace(line[n])) return 0;
return static_cast<int>(n);
}
bool IsFence(std::string_view line) { return TrimLeft(line).starts_with("```"); }
// `---`, `***`, `___`: three or more of one character, nothing else but spaces.
bool IsThematicBreak(std::string_view line) {
const std::string_view s = Trim(line);
if (s.size() < 3) return false;
const char c = s.front();
if (c != '-' && c != '*' && c != '_') return false;
int count = 0;
for (const char ch : s) {
if (ch == c) { ++count; continue; }
if (!IsSpace(ch)) return false;
}
return count >= 3;
}
struct ListMarker {
bool ok = false;
bool ordered = false;
std::int64_t start = 1; // the number an ordered item announced
std::size_t contentAt = 0; // offset of the item's text within the line
};
ListMarker ParseListMarker(std::string_view line) {
ListMarker m;
const std::string_view s = Undent(line);
const std::size_t indent = line.size() - s.size();
if (s.empty()) return m;
if ((s[0] == '-' || s[0] == '*' || s[0] == '+') && s.size() > 1 && IsSpace(s[1])) {
// A thematic break is also a run of dashes; it wins, because "- - -"
// is a rule everywhere and a three-item list nowhere.
if (IsThematicBreak(line)) return m;
m.ok = true;
m.contentAt = indent + 2;
return m;
}
std::size_t n = 0;
while (n < s.size() && s[n] >= '0' && s[n] <= '9') ++n;
// Bounded so a line starting with a long number is not mistaken for a list.
if (n == 0 || n > 9) return m;
if (n + 1 >= s.size() || (s[n] != '.' && s[n] != ')') || !IsSpace(s[n + 1])) return m;
std::int64_t value = 0;
std::from_chars(s.data(), s.data() + n, value);
m.ok = true;
m.ordered = true;
m.start = value;
m.contentAt = indent + n + 2;
return m;
}
bool StartsBlock(std::string_view line) {
return HeadingLevel(Undent(line)) > 0 || IsFence(line) || IsThematicBreak(line)
|| Undent(line).starts_with('>') || ParseListMarker(line).ok;
}
SafeHtml RenderBlocks(std::span<const std::string_view> lines,
std::span<const PostMedia> media, int depth);
// A paragraph's lines, joined and rendered.
//
// A paragraph whose entire content is embedded files becomes the same
// .post-media block the cards use rather than a <p> of images: that is what
// gives a run of screenshots the two-up grid instead of a column of full-width
// pictures with paragraph spacing between them.
SafeHtml RenderParagraph(std::span<const std::string_view> lines,
std::span<const PostMedia> media, int depth) {
// Lines join with a space: a body wraps its prose at whatever width the
// author's editor used, and those wraps are not meaningful.
std::string joined;
for (std::size_t i = 0; i < lines.size(); ++i) {
joined += Trim(lines[i]);
if (i + 1 < lines.size()) joined += ' ';
}
// Only-images test: strip every ![](...) span and see whether anything but
// whitespace is left.
bool onlyMedia = false;
{
std::string rest;
std::size_t i = 0;
std::size_t found = 0;
const std::string_view s = joined;
while (i < s.size()) {
if (s[i] == '!' && i + 1 < s.size() && s[i + 1] == '[') {
const std::size_t altEnd = MatchingDelimiter(s, i + 2, '[', ']');
if (altEnd != std::string_view::npos && altEnd + 1 < s.size()
&& s[altEnd + 1] == '(') {
const std::size_t srcEnd = MatchingDelimiter(s, altEnd + 2, '(', ')');
if (srcEnd != std::string_view::npos) {
++found;
i = srcEnd + 1;
continue;
}
}
}
rest += s[i];
++i;
}
onlyMedia = found > 0 && Trim(rest).empty();
}
const SafeHtml inner = RenderInline(joined, media, depth);
if (inner.Empty()) return SafeHtml{};
if (onlyMedia) return Format(R"(<div class="post-media">{}</div>)", inner);
return Format(R"(<p>{}</p>)", inner);
}
SafeHtml RenderBlocks(std::span<const std::string_view> lines,
std::span<const PostMedia> media, int depth) {
std::vector<SafeHtml> out;
std::size_t i = 0;
while (i < lines.size()) {
if (Blank(lines[i])) { ++i; continue; }
const std::string_view line = lines[i];
const std::string_view body = Undent(line);
// ── fenced code ───────────────────────────────────────────────
//
// Taken verbatim and escaped: the battery-measurement tables and the
// android top output in these posts are the one place where every
// space matters and nothing inside should be interpreted at all.
if (IsFence(line)) {
std::size_t j = i + 1;
std::string code;
while (j < lines.size() && !IsFence(lines[j])) {
code += TrimRight(lines[j]);
code += '\n';
++j;
}
out.push_back(Format(R"(<pre class="post-body__code"><code>{}</code></pre>)",
Escape(code)));
// Past the closing fence, or to the end when it never closed —
// an unterminated fence renders as code rather than swallowing
// the rest of the post into a parse failure.
i = (j < lines.size()) ? j + 1 : j;
continue;
}
// ── heading ───────────────────────────────────────────────────
//
// Demoted by one: the page's <h1> is the post title, so a body's own
// top-level heading is a section inside it. Without the shift every
// post would carry two h1s and the document outline would be wrong on
// exactly the pages this whole change exists to make indexable.
if (const int level = HeadingLevel(body); level > 0) {
std::string_view textPart = body.substr(static_cast<std::size_t>(level));
// A closing run of #s is decoration, not content.
textPart = TrimRight(textPart);
while (!textPart.empty() && textPart.back() == '#') textPart.remove_suffix(1);
const int tag = std::min(level + 1, 6);
out.push_back(Format("<h{}>{}</h{}>",
Html::Num(tag),
RenderInline(Trim(textPart), media, depth),
Html::Num(tag)));
++i;
continue;
}
if (IsThematicBreak(line)) {
out.push_back(Raw("<hr>"));
++i;
continue;
}
// ── blockquote ────────────────────────────────────────────────
//
// The quoted lines are re-parsed as blocks, so a quote keeps its own
// paragraphs and headings — which matters here, because the longest
// quotes in these posts are multi-paragraph company copy being taken
// apart line by line.
if (body.starts_with('>')) {
std::vector<std::string_view> quoted;
std::size_t j = i;
while (j < lines.size() && Undent(lines[j]).starts_with('>')) {
std::string_view q = Undent(lines[j]).substr(1);
if (!q.empty() && q.front() == ' ') q.remove_prefix(1);
quoted.push_back(q);
++j;
}
// At the depth limit the quote is still shown, just flattened —
// dropping it would lose content, which is the worse failure.
out.push_back(Format(
R"(<blockquote class="post-body__quote">{}</blockquote>)",
depth < kMaxDepth ? RenderBlocks(quoted, media, depth + 1)
: RenderParagraph(quoted, media, depth)));
i = j;
continue;
}
// ── list ──────────────────────────────────────────────────────
if (const ListMarker first = ParseListMarker(line); first.ok) {
std::vector<SafeHtml> items;
std::vector<std::string_view> current;
std::size_t j = i;
auto flushItem = [&] {
if (current.empty()) return;
items.push_back(Format(R"(<li>{}</li>)",
RenderInline([&] {
std::string joined;
for (std::size_t k = 0; k < current.size(); ++k) {
joined += Trim(current[k]);
if (k + 1 < current.size()) joined += ' ';
}
return joined;
}(), media, depth)));
current.clear();
};
while (j < lines.size()) {
if (Blank(lines[j])) {
// A blank line ends the list UNLESS the next non-blank line
// is another item of the same kind. These posts space their
// numbered steps apart, and treating that as seven separate
// one-item lists would restart the numbering at every gap.
std::size_t peek = j;
while (peek < lines.size() && Blank(lines[peek])) ++peek;
const ListMarker next =
peek < lines.size() ? ParseListMarker(lines[peek]) : ListMarker{};
if (!next.ok || next.ordered != first.ordered) break;
j = peek;
continue;
}
if (const ListMarker m = ParseListMarker(lines[j]); m.ok) {
if (m.ordered != first.ordered) break;
flushItem();
current.push_back(lines[j].substr(
std::min(m.contentAt, lines[j].size())));
++j;
continue;
}
// A non-marker line that would start some other block ends the
// list; anything else is this item's text continuing onto the
// next line.
if (StartsBlock(lines[j])) break;
current.push_back(lines[j]);
++j;
}
flushItem();
if (first.ordered) {
// start= only when it is not 1, so the common case stays clean
// markup — and so a list resumed after an interrupting
// paragraph continues its numbering instead of starting over.
out.push_back(Format(R"(<ol class="post-body__list"{}>{}</ol>)",
first.start == 1 ? SafeHtml{}
: Attr("start", std::to_string(first.start)),
Join(items)));
} else {
out.push_back(Format(R"(<ul class="post-body__list">{}</ul>)", Join(items)));
}
i = j;
continue;
}
// ── paragraph ─────────────────────────────────────────────────
{
std::size_t j = i;
while (j < lines.size() && !Blank(lines[j])) {
// A block marker on a later line interrupts the paragraph
// rather than being absorbed into it as text.
if (j > i && StartsBlock(lines[j])) break;
++j;
}
out.push_back(RenderParagraph(lines.subspan(i, j - i), media, depth));
i = j;
}
}
return Join(out);
}
// ── entry point ───────────────────────────────────────────────────────
// Render a post body to the inner markup of the article element.
//
// `media` is the post's mirrored inline files, used to attach dimensions (and
// a poster, and the H.264 fallback) to whatever the body embeds. Passing an
// empty span is fine: the markup then simply carries no dimensions, exactly
// like a file whose mirror failed.
export SafeHtml Render(std::string_view text, std::span<const PostMedia> media = {}) {
std::vector<std::string_view> lines;
std::size_t start = 0;
while (start <= text.size()) {
const std::size_t nl = text.find('\n', start);
if (nl == std::string_view::npos) {
lines.push_back(text.substr(start));
break;
}
lines.push_back(text.substr(start, nl - start));
start = nl + 1;
}
return RenderBlocks(lines, media, 0);
}
} // namespace Catcrafts::Markdown

192
shared/interfaces/Catcrafts.Shared-Media.cppm Normal file
View file

@ -0,0 +1,192 @@
/*
catcrafts.net
Copyright (C) 2026 Catcrafts
The source code of this website is made available for viewing purposes only.
No permission is granted to copy, modify, distribute, or create derivative works.
*/
// The markup for one mirrored file — image or video.
//
// This exists as its own partition because two callers need identical output:
// :Views renders a post's headline media on the card and at the top of the post
// page, and :Markdown renders whatever the body embeds. They were separate
// copies of the same element-building code, which is how the AV1 fallback logic
// came to be written twice; a format tier added to one and forgotten in the
// other shows up as the same picture served two different ways on two pages.
//
// FORMAT LADDER. Every file the mirror stores is served in the best form the
// browser will take, with a form every browser will take underneath it:
//
// video AV1 in MP4, falling back to H.264 in MP4. Two <source> elements;
// the codecs parameter on the first is what lets a browser without
// AV1 skip it rather than fail on a file it cannot decode.
// image AVIF, falling back to the mirrored original (usually WebP), falling
// back to PNG. A <picture>, so the browser fetches exactly one of
// them — the tiers cost nothing to the visitors who do not need them.
//
// The middle image tier is free: it is the file the mirror already downloaded,
// so it adds no encode and no disk. It matters because it is what stands
// between "no AVIF" and the PNG, and PNG is lossless — for a photograph that is
// an order of magnitude larger than the WebP beside it. With the middle tier
// the PNG is reached only by a browser that supports neither AVIF nor WebP.
//
// Any tier can be absent (the transcode was unavailable, or the file was never
// mirrored at all) and the markup degrades a step at a time, down to a bare
// <img src> — which is what the site emitted before any of this existed.
export module Catcrafts.Shared:Media;
import std;
import :Html;
import :Model;
namespace Catcrafts::Media {
using Html::SafeHtml;
using Html::Escape;
using Html::Attr;
using Html::Format;
using Html::Join;
using Html::Url;
bool EndsWithNoCase(std::string_view s, std::string_view suffix) {
if (s.size() < suffix.size()) return false;
const std::size_t off = s.size() - suffix.size();
for (std::size_t i = 0; i < suffix.size(); ++i) {
char a = s[off + i];
if (a >= 'A' && a <= 'Z') a = static_cast<char>(a - 'A' + 'a');
if (a != suffix[i]) return false;
}
return true;
}
// True when the path names something the <video> path should handle. Used only
// to guess for a file that was never mirrored, so there is no record to ask.
export bool LooksLikeVideo(std::string_view src) {
return EndsWithNoCase(src, ".mp4") || EndsWithNoCase(src, ".webm")
|| EndsWithNoCase(src, ".mov");
}
// The type= a <source> should advertise. Empty for anything not recognised, in
// which case the tier is dropped rather than guessed: a wrong type is worse
// than a missing source, because the browser believes it.
std::string_view MimeFor(std::string_view src) {
if (EndsWithNoCase(src, ".avif")) return "image/avif";
if (EndsWithNoCase(src, ".webp")) return "image/webp";
if (EndsWithNoCase(src, ".png")) return "image/png";
if (EndsWithNoCase(src, ".jpg") || EndsWithNoCase(src, ".jpeg")) return "image/jpeg";
if (EndsWithNoCase(src, ".gif")) return "image/gif";
return {};
}
// Width and height, or nothing when the mirror could not probe them. Both or
// neither: a lone dimension is worse than none, because the browser derives the
// other from it and gets the aspect ratio wrong.
SafeHtml Dimensions(const PostMedia& m) {
if (m.width <= 0 || m.height <= 0) return SafeHtml{};
return Format("{}{}", Attr("width", std::to_string(m.width)),
Attr("height", std::to_string(m.height)));
}
SafeHtml ImageTag(const PostMedia& m, std::string_view alt) {
// What every browser reads, and the only element that is always present.
// The PNG when there is one, because that is the tier nothing can refuse.
const std::string_view imgSrc = m.fallback.empty() ? m.src : m.fallback;
std::vector<SafeHtml> sources;
if (!m.avif.empty()) {
sources.push_back(Format(R"(<source{} type="image/avif">)", Url("srcset", m.avif)));
}
// The mirrored original, sitting between the AVIF and the PNG. Skipped when
// it IS what the <img> points at or what the AVIF tier already offered
// (nothing left to say), or when its type cannot be named.
if (m.src != imgSrc && m.src != m.avif) {
if (const std::string_view mime = MimeFor(m.src); !mime.empty()) {
sources.push_back(Format(R"(<source{}{}>)",
Url("srcset", m.src), Attr("type", mime)));
}
}
// alt is whatever the author wrote, which for a post's headline media is
// nothing: those are screenshots whose meaning is already in the title and
// the excerpt, and inventing descriptive alt text here would be making up
// what the picture shows. Emitted explicitly even when empty — alt="" is
// skipped cleanly by a screen reader, a missing alt makes it read the file
// name out loud.
const SafeHtml img = Format(
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="{}"{}{}>)",
Escape(alt), Url("src", imgSrc), Dimensions(m));
if (sources.empty()) return img;
return Format("<picture>{}{}</picture>", Join(sources), img);
}
SafeHtml VideoTag(const PostMedia& m) {
// preload="metadata", not "auto": a page with several 5 MB recordings must
// not pull them all on load.
const SafeHtml poster = m.poster.empty() ? SafeHtml{} : Url("poster", m.poster);
const SafeHtml dims = Dimensions(m);
if (m.fallback.empty() || m.fallback == m.src) {
return Format(
R"(<video class="post-media__item" controls preload="metadata" playsinline{}{}{}></video>)",
Url("src", m.src), poster, dims);
}
// An AV1 video with its H.264 rendition. Both are .mp4, so the container
// alone cannot tell them apart: the codecs parameter on the first <source>
// is what lets a browser without AV1 (Safari before 17, Apple hardware
// without the decoder) skip it and take the H.264 instead of failing on a
// file it cannot decode. The string is advisory and used only for
// selection — once a source is picked the browser reads the actual stream —
// so the canonical profile-0 8-bit form is right for anything
// publish-media.sh emits (yuv420p is pinned there).
return Format(
R"(<video class="post-media__item" controls preload="metadata" playsinline{}{}>)"
R"(<source{} type="video/mp4; codecs=av01.0.08M.08">)"
R"(<source{} type="video/mp4">)"
R"(</video>)",
poster, dims, Url("src", m.src), Url("src", m.fallback));
}
// The element for one mirrored file.
export SafeHtml Tag(const PostMedia& m, std::string_view alt = {}) {
return m.kind == "video" ? VideoTag(m) : ImageTag(m, alt);
}
// The mirrored record for `src`, or nullptr when this path was never mirrored.
//
// Matching is on the exact string because tools/fetch-media.sh rewrites the
// body text and the body_media list from the same mapping in the same pass —
// they cannot disagree about a path without the mirror step itself being wrong.
export const PostMedia* Find(std::span<const PostMedia> media, std::string_view src) {
for (const PostMedia& m : media) {
if (m.src == src) return &m;
}
return nullptr;
}
// The record for `src`, or one invented from the path alone.
//
// The invented case is a file whose download failed, so the body still points
// at its original URL: there are no dimensions and no renditions to offer, and
// the markup degrades to the bare element. Rendering something beats dropping
// the picture the paragraph is talking about.
export PostMedia Describe(std::span<const PostMedia> media, std::string_view src) {
if (const PostMedia* hit = Find(media, src)) return *hit;
PostMedia m;
m.src = std::string(src);
m.kind = LooksLikeVideo(src) ? "video" : "image";
return m;
}
// A post's media as one block. More than one file gets a two-up grid via CSS,
// so a post with four screenshots does not become a mile of scrolling.
export SafeHtml Block(std::span<const PostMedia> media) {
if (media.empty()) return SafeHtml{};
std::vector<SafeHtml> items;
items.reserve(media.size());
for (const PostMedia& m : media) items.push_back(Tag(m));
return Format(R"(<div class="post-media">{}</div>)", Join(items));
}
} // namespace Catcrafts::Media

108
shared/interfaces/Catcrafts.Shared-Model.cppm
View file

@ -30,6 +30,11 @@ No permission is granted to copy, modify, distribute, or create derivative works
export module Catcrafts.Shared:Model;
import std;
import :Json;
// For IsValidSlug. A post slug becomes a URL, so it is checked here on the way
// in rather than trusted because CI wrote it: an unparseable slug would render
// a "read more" link to a route that can never match, which is a 404 the site
// links to itself. Dropping it instead degrades to a post with no page.
import :Route;
namespace Catcrafts {
@ -50,34 +55,69 @@ export struct PostMedia {
// play — and these posts are their video, so the black box is the page.
// Empty for images, and empty when the instance generated no thumbnail.
std::string poster;
// H.264 rendition of an AV1 video, when tools/publish-media.sh uploaded one
// beside it. Non-empty means the renderer emits a <source> pair instead of
// a bare src, so a browser without AV1 (Safari before 17, Apple hardware
// without the decoder) gets a file it can play. Empty for images and for
// videos that need no fallback.
// The rendition every browser can take, and the bottom of the format
// ladder :Media builds. Empty means there is only `src`.
//
// video H.264 in MP4, uploaded beside the AV1 by
// tools/publish-media.sh. Without it a browser lacking AV1
// (Safari before 17, Apple hardware without the decoder) has an
// element it cannot play.
// image PNG, transcoded by tools/fetch-media.sh. It is what the <img>
// itself points at, so a browser that understands no <source>
// type at all still gets the picture.
std::string fallback;
// AVIF rendition of an image, transcoded by tools/fetch-media.sh. Offered
// above `src` — it is consistently smaller than the WebP the instances
// serve, at 4:4:4 chroma so coloured text in a screenshot does not fringe.
// Empty for videos, and for an image the transcode could not produce.
std::string avif;
std::int64_t width = 0;
std::int64_t height = 0;
};
// A post mirrored from the fediverse (Lemmy). Deliberately not the full post:
// the body stays on the community's instance, where the comments are. We show
// enough to be worth clicking and then hand off — no crawling, no comment
// mirroring, no markdown pipeline.
// A post mirrored from the fediverse (Lemmy).
//
// The body IS mirrored — the comments are not. That split is the whole policy:
// the writing is the work and belongs on the site that is about the work (and
// is the only version a search engine can be pointed at as canonical), while
// the discussion belongs to the community that is having it. So every post
// page carries the full text and then hands off to the thread. No crawling, no
// comment mirroring, no runtime dependency on the instance.
//
// `permalink` is resolved at fetch time to the COMMUNITY's instance, not the
// author's. Both host a federated copy; the community's is where the discussion
// actually is, and it is what the site should be pointing readers at.
export struct Post {
std::string title;
// URL-safe name of this post's own page at /posts/<slug>, derived from the
// title by tools/fetch-posts.sh. Empty only if the title reduced to nothing
// a slug can be made of, which is what HasPage() below tests for.
std::string slug;
std::string permalink; // canonical ap_id on the instance; where "discuss" goes
std::string linkUrl; // for link posts, the linked target; empty otherwise
std::string community; // e.g. "linux@lemmy.ml"
std::string published; // ISO-8601, as emitted by the API
std::string excerpt; // plain text, truncated by CI — never markdown
// The full post, still as Markdown. Rendered by :Markdown at page-render
// time rather than converted to HTML at fetch time, because the renderer is
// where this codebase's escaping guarantee lives — text that arrives from
// someone else's server must not become markup anywhere but there.
std::string body;
std::int64_t score = 0;
std::int64_t comments = 0;
// The post's headline media — `post.url` upstream, the recording or
// screenshot the post is *about*. Shown on the card and above the body.
std::vector<PostMedia> media;
// Files embedded inside the body, mirrored by the same pass. The body text
// already points at these local paths; this list exists so the renderer can
// attach dimensions (and a poster, and the H.264 fallback) to them, which
// Markdown syntax has nowhere to carry.
std::vector<PostMedia> bodyMedia;
// Whether this post gets its own page. A post with no body has nothing to
// put on one — a page consisting of a title and a link out is thin content
// that should not be minted as a URL, indexed, or linked to as "read more".
bool HasPage() const { return !slug.empty() && !body.empty(); }
};
// An entry on the projects page. Repo content, not a database — these change
@ -294,6 +334,32 @@ export struct LegalPage {
// ── loaders ───────────────────────────────────────────────────────────
// The media array shape, shared by a post's headline media and its body media —
// they are the same records produced by the same mirror pass, so they are read
// by one function rather than two that could drift.
std::vector<PostMedia> LoadPostMedia(const Json::Value* array) {
std::vector<PostMedia> out;
if (!array || !array->IsArray()) return out;
for (const Json::Value& mv : array->array) {
if (!mv.IsObject()) continue;
PostMedia pm;
pm.src = std::string(mv.Str("src"));
pm.kind = std::string(mv.Str("kind", "image"));
pm.poster = std::string(mv.Str("poster"));
pm.fallback = std::string(mv.Str("fallback"));
pm.avif = std::string(mv.Str("avif"));
pm.width = mv.Int("w");
pm.height = mv.Int("h");
// Only the two kinds the renderer knows how to emit. Anything else
// would fall through to an <img> for a file that is not an image, so
// treat it as image only when it says so.
if (pm.kind != "image" && pm.kind != "video") pm.kind = "image";
if (pm.src.empty()) continue;
out.push_back(std::move(pm));
}
return out;
}
export std::vector<Post> LoadPosts(std::string_view json) {
std::vector<Post> out;
auto doc = Json::Parse(json);
@ -303,31 +369,19 @@ export std::vector<Post> LoadPosts(std::string_view json) {
if (!item.IsObject()) continue;
Post p;
p.title = std::string(item.Str("title"));
if (const std::string_view slug = item.Str("slug"); IsValidSlug(slug)) {
p.slug = std::string(slug);
}
p.permalink = std::string(item.Str("permalink"));
p.linkUrl = std::string(item.Str("url"));
p.community = std::string(item.Str("community"));
p.published = std::string(item.Str("published"));
p.excerpt = std::string(item.Str("excerpt"));
p.body = std::string(item.Str("body"));
p.score = item.Int("score");
p.comments = item.Int("comments");
if (const Json::Value* m = item.Find("media"); m && m->IsArray()) {
for (const Json::Value& mv : m->array) {
if (!mv.IsObject()) continue;
PostMedia pm;
pm.src = std::string(mv.Str("src"));
pm.kind = std::string(mv.Str("kind", "image"));
pm.poster = std::string(mv.Str("poster"));
pm.fallback = std::string(mv.Str("fallback"));
pm.width = mv.Int("w");
pm.height = mv.Int("h");
// Only the two kinds the renderer knows how to emit. Anything
// else would fall through to an <img> for a file that is not an
// image, so treat it as image only when it says so.
if (pm.kind != "image" && pm.kind != "video") pm.kind = "image";
if (pm.src.empty()) continue;
p.media.push_back(std::move(pm));
}
}
p.media = LoadPostMedia(item.Find("media"));
p.bodyMedia = LoadPostMedia(item.Find("body_media"));
// A post with no title and nowhere to click is not renderable; drop it
// rather than emit an empty card.
if (p.title.empty() && p.permalink.empty()) continue;

32
shared/interfaces/Catcrafts.Shared-Route.cppm
View file

@ -23,6 +23,7 @@ export enum class RouteKind {
About, // /about — the person behind the company
Projects,
Posts,
Post, // /posts/<slug> — one post, in full
Demos, // /demos — the list
Demo, // /demos/<slug>
Shop, // /shop — the (currently single-item) product list
@ -45,7 +46,8 @@ export struct Route {
std::string query; // raw, including leading '?', or empty
// Non-empty when the request should be canonicalised to a different URL.
std::string canonicalRedirect;
// The <slug> of /shop/<slug>, empty for every other route.
// The <slug> of /shop/<slug>, /demos/<slug>, /legal/<slug> or /posts/<slug>
// (and the token of /order/<token>); empty for every other route.
std::string slug;
};
@ -134,6 +136,20 @@ export Route ParseRoute(std::string_view path, std::string_view query = {}) {
return r;
}
// /posts/<slug>. Whether a slug names a post the site actually has is the
// dispatcher's question, exactly as it is for /shop/<slug>; this only
// decides that the URL is shaped like a post page at all.
if (p.starts_with("/posts/")) {
const std::string_view slug = p.substr(7);
if (IsValidSlug(slug)) {
r.kind = RouteKind::Post;
r.slug = std::string(slug);
return r;
}
r.kind = RouteKind::NotFound;
return r;
}
if (p.starts_with("/demos/")) {
const std::string_view slug = p.substr(7);
if (IsValidSlug(slug)) {
@ -183,6 +199,20 @@ export struct NavItem {
RouteKind kind;
};
// Which nav entry a route highlights. A route that is not itself a nav entry
// borrows its section's: an individual post and the retired /blog URL both sit
// under Posts. Shared so the three call sites (the server, the CLI renderer and
// the wasm router) cannot disagree about where the reader is — they each used
// to spell the /blog case out, and the third one to be added would have been
// the third chance to forget it.
export RouteKind NavKindFor(RouteKind kind) {
switch (kind) {
case RouteKind::Post:
case RouteKind::LegacyBlog: return RouteKind::Posts;
default: return kind;
}
}
export std::span<const NavItem> NavItems() {
static constexpr std::array<NavItem, 6> items{{
{ "Home", "/", RouteKind::Home },

228
shared/interfaces/Catcrafts.Shared-Views.cppm
View file

@ -23,6 +23,8 @@ import std;
import :Content;
import :Html;
import :Form;
import :Markdown;
import :Media;
import :Model;
import :Money;
import :Route;
@ -189,7 +191,11 @@ export RenderedPage RenderHome(std::span<const Project> projects,
const Post& post = posts[i];
recent.push_back(Format(
R"(<li class="recent__item"><a{}>{}</a><span class="recent__date">{}</span></li>)",
Url("href", post.permalink),
// The on-site page when the post has one, the thread otherwise.
// Same rule as the cards on /posts: a link from the home page is
// worth more pointing at a page this site owns than at someone
// else's copy of it, and the post page links onward to the thread.
Url("href", post.HasPage() ? "/posts/" + post.slug : post.permalink),
Escape(post.title),
Escape(DateOnly(post.published))));
}
@ -341,65 +347,62 @@ export RenderedPage RenderProjects(std::span<const Project> projects) {
// keeps the privacy notice's "everything comes from catcrafts.net" true and
// stops every visitor's IP reaching whichever instance hosted the file.
//
// Video is preload="metadata", not "auto": a page with several 5 MB recordings
// must not pull them all on load. Dimensions come from the content file so the
// browser reserves the right box and nothing jumps as files arrive.
// The element itself — the format ladder, the dimensions, the poster — is
// :Media's job, shared with the body renderer so a post's headline screenshot
// and one embedded in its prose cannot be served differently.
SafeHtml RenderPostMedia(std::span<const PostMedia> media) {
if (media.empty()) return SafeHtml{};
std::vector<SafeHtml> items;
for (const PostMedia& m : media) {
SafeHtml dims = (m.width > 0 && m.height > 0)
? Format("{}{}", Attr("width", std::to_string(m.width)),
Attr("height", std::to_string(m.height)))
: SafeHtml{};
if (m.kind == "video") {
SafeHtml poster = m.poster.empty() ? SafeHtml{} : Url("poster", m.poster);
if (m.fallback.empty() || m.fallback == m.src) {
items.push_back(Format(
R"(<video class="post-media__item" controls preload="metadata" )"
R"(playsinline{}{}{}></video>)",
Url("src", m.src), poster, dims));
} else {
// An AV1 video with its H.264 rendition. Both are .mp4, so the
// container alone cannot tell them apart: the codecs parameter
// on the first <source> is what lets a browser without AV1
// (Safari before 17, Apple hardware without the decoder) skip
// it and take the H.264 instead of failing on a file it cannot
// decode. The string is advisory and used only for selection —
// once a source is picked the browser reads the actual stream —
// so the canonical profile-0 8-bit form is right for anything
// publish-media.sh emits (yuv420p is pinned there).
items.push_back(Format(
R"(<video class="post-media__item" controls preload="metadata" )"
R"(playsinline{}{}>)"
R"(<source{} type="video/mp4; codecs=av01.0.08M.08">)"
R"(<source{} type="video/mp4">)"
R"(</video>)",
poster, dims, Url("src", m.src), Url("src", m.fallback)));
return Media::Block(media);
}
} else {
// alt is empty and aria-hidden is absent on purpose: these are
// screenshots whose meaning is already in the post title and
// excerpt, and inventing descriptive alt text here would be making
// up what the picture shows.
items.push_back(Format(
R"(<img class="post-media__item" loading="lazy" decoding="async" )"
R"(alt=""{}{}>)",
Url("src", m.src), dims));
// A link post gets a second, clearly-labelled outbound link. Without the label
// the two links are indistinguishable and one of them silently leaves the site.
SafeHtml RenderPostLinkRow(const Post& p) {
if (p.linkUrl.empty()) return SafeHtml{};
return Format(
R"(<p class="post-card__link"><a{} rel="nofollow noopener">{}</a></p>)",
Url("href", p.linkUrl), Escape(p.linkUrl));
}
}
return Format(R"(<div class="post-media">{}</div>)", Join(items));
// Points and the thread link, identical on a card and at the foot of a post
// page — the reader is being offered the same two things in both places, and
// they should not look like two different components.
SafeHtml RenderPostStats(const Post& p) {
return Format(
R"(<footer class="post-card__footer">)"
R"(<span class="stat">{} points</span>)"
R"(<a class="stat stat--link"{} rel="noopener">Discuss on the fediverse ({} comments) &rarr;</a>)"
R"(</footer>)",
Num(p.score), Url("href", p.permalink), Num(p.comments));
}
export RenderedPage RenderPosts(std::span<const Post> posts) {
std::vector<SafeHtml> cards;
for (const Post& p : posts) {
// A link post gets a second, clearly-labelled outbound link. Without
// the label the two links are indistinguishable and one of them
// silently leaves the site.
SafeHtml linkRow = p.linkUrl.empty() ? SafeHtml{} : Format(
R"(<p class="post-card__link"><a{} rel="nofollow noopener">{}</a></p>)",
Url("href", p.linkUrl), Escape(p.linkUrl));
// The title goes to this site's copy when there is one. Before post
// pages existed it could only go to the thread, which meant the list
// of everything this site is about was a list of links off it.
const std::string titleHref = p.HasPage() ? "/posts/" + p.slug : p.permalink;
// No "read more" without a page to read more on — a post whose body
// never arrived would otherwise offer a link to its own 404.
//
// It goes inline at the end of the excerpt, immediately after the
// ellipsis the truncation left: that is where the sentence stops and
// where the reader is already looking for the rest of it. Below the
// media it was the same offer made a screen further down, after the
// reader had already decided.
const SafeHtml more = !p.HasPage() ? SafeHtml{} : Format(
R"( <a class="link-more"{}>Read the full post</a>)",
Url("href", "/posts/" + p.slug));
// With no excerpt there is no sentence to continue, so the link falls
// back to a row of its own rather than disappearing along with it.
const SafeHtml excerptRow =
!p.excerpt.empty()
? Format(R"(<p class="post-card__excerpt">{}{}</p>)", Escape(p.excerpt), more)
: more.Empty()
? SafeHtml{}
: Format(R"(<p class="post-card__more">{}</p>)", more);
cards.push_back(Format(
R"(<article class="post-card">)"
@ -412,32 +415,28 @@ export RenderedPage RenderPosts(std::span<const Post> posts) {
R"({})"
R"({})"
R"({})"
R"(<footer class="post-card__footer">)"
R"(<span class="stat">{} points</span>)"
R"(<a class="stat stat--link"{} rel="noopener">Discuss on the fediverse ({} comments) &rarr;</a>)"
R"(</footer>)"
R"({})"
R"(</article>)",
Url("href", p.permalink), Escape(p.title),
Url("href", titleHref), Escape(p.title),
Attr("datetime", p.published), Escape(DateOnly(p.published)),
Escape(p.community),
p.excerpt.empty() ? SafeHtml{}
: Format(R"(<p class="post-card__excerpt">{}</p>)", Escape(p.excerpt)),
excerptRow,
RenderPostMedia(p.media),
linkRow,
Num(p.score),
Url("href", p.permalink), Num(p.comments)));
RenderPostLinkRow(p),
RenderPostStats(p)));
}
RenderedPage page;
page.meta.title = "Posts — Catcrafts";
page.meta.description = "Posts from the fediverse; discussion happens there.";
page.meta.description = "Posts about mobile Linux, kernel work and open hardware, "
"written for the fediverse and archived here in full.";
page.meta.canonical = "/posts";
page.main = Format(
R"(<header class="page-header">)"
R"(<h1 class="page-header__title">Posts</h1>)"
R"(<p class="page-header__lede">Catcrafts posts on the fediverse rather than keeping a blog here. )"
R"(Each of these links to the original thread on whichever instance it lives on. )"
R"(Follow one to read it and join the discussion there.</p>)"
R"(Each one is kept here in full, and links to the original thread on whichever )"
R"(instance it lives on — that is where the discussion is.</p>)"
R"(</header>)"
R"(<div class="post-list">{}</div>)",
cards.empty()
@ -446,6 +445,88 @@ export RenderedPage RenderPosts(std::span<const Post> posts) {
return page;
}
// ── one post, in full ─────────────────────────────────────────────────
// The whole point of hosting the body: a page a search engine can index, a
// reader can link to, and this site can claim as canonical. The card on /posts
// is a summary of this; the thread on the instance is where the comments are.
//
// The body arrives as Markdown and is rendered HERE rather than converted at
// fetch time, because :Markdown is inside the escaping guarantee and a shell
// script writing HTML into a content file would not be. See that module for
// what it does and does not accept.
export RenderedPage RenderPost(const Post& p) {
// Whatever the post leads with, for the link-preview card. A video has no
// still of its own to offer, so its poster frame stands in. Only our own
// mirrored copies qualify: an og:image on someone else's instance is the
// same third-party fetch the mirror exists to avoid, just performed by a
// crawler instead of a reader.
std::string ogImage;
for (const PostMedia& m : p.media) {
const std::string& candidate = m.kind == "video" ? m.poster : m.src;
if (candidate.starts_with("/")) { ogImage = candidate; break; }
}
const std::string canonical = "/posts/" + p.slug;
RenderedPage page;
page.meta.title = p.title + " — Catcrafts";
page.meta.description = p.excerpt;
page.meta.canonical = canonical;
page.meta.ogType = "article";
page.meta.ogImage = ogImage;
// BlogPosting, joined to the same two nodes every other page names: the
// Person on /about is the author and the Organization is the publisher.
// Without those @ids each post would introduce a fourth unrelated
// "Catcrafts" to a search index that already has trouble telling this one
// from the name-twins — see RenderHome for the whole argument.
//
// discussionUrl is the honest way to say what the fediverse link is: the
// comments belong to the thread, and this page is not pretending to mirror
// them.
page.meta.jsonLd = std::format(
R"({{"@context":"https://schema.org","@type":"BlogPosting",)"
R"("headline":{},"datePublished":{},"url":{},)"
R"("mainEntityOfPage":{{"@type":"WebPage","@id":{}}},)"
R"("author":{{"@id":"https://catcrafts.net/about#person",)"
R"("@type":"Person","name":"Jorijn van der Graaf"}},)"
R"("publisher":{{"@id":"https://catcrafts.net/#organization",)"
R"("@type":"Organization","name":"Catcrafts"}},)"
R"("discussionUrl":{}{}{}}})",
JsonStr(p.title), JsonStr(p.published),
JsonStr("https://catcrafts.net" + canonical),
JsonStr("https://catcrafts.net" + canonical),
JsonStr(p.permalink),
p.excerpt.empty() ? std::string{} : ",\"description\":" + JsonStr(p.excerpt),
ogImage.empty() ? std::string{}
: ",\"image\":" + JsonStr("https://catcrafts.net" + ogImage));
page.main = Format(
R"(<article class="post">)"
R"(<header class="page-header">)"
R"(<h1 class="page-header__title">{}</h1>)"
R"(<p class="post-card__meta">)"
R"(<time{}>{}</time><span class="post-card__community">{}</span>)"
R"(</p>)"
R"(</header>)"
R"({})"
R"({})"
R"(<div class="post-body">{}</div>)"
R"({})"
R"(<p class="post__back"><a class="link-more"{}>All posts</a></p>)"
R"(</article>)",
Escape(p.title),
Attr("datetime", p.published), Escape(DateOnly(p.published)),
Escape(p.community),
RenderPostMedia(p.media),
RenderPostLinkRow(p),
Markdown::Render(p.body, p.bodyMedia),
RenderPostStats(p),
Url("href", "/posts"));
return page;
}
// ── shop ──────────────────────────────────────────────────────────────
// The product page's price block: the same single headline number as the shop
@ -1329,6 +1410,16 @@ export struct SiteContent {
}
return nullptr;
}
// Only posts that have a page. A post with no body has a slug but nothing
// to show, so finding it here would mint an indexable URL for a title and
// a link — see Post::HasPage.
const Post* FindPost(std::string_view slug) const {
for (const Post& p : posts) {
if (p.HasPage() && p.slug == slug) return &p;
}
return nullptr;
}
};
RenderedPage RenderRouteBody(const Route& route, const SiteContent& content);
@ -1357,6 +1448,13 @@ RenderedPage RenderRouteBody(const Route& route, const SiteContent& content) {
case RouteKind::About: return RenderAbout(Content::AboutPage());
case RouteKind::Projects: return RenderProjects(content.projects);
case RouteKind::Posts: return RenderPosts(content.posts);
case RouteKind::Post: {
// A slug that parsed but names no post is a 404, for the same
// reason an unknown product slug is: otherwise every typo and
// every retired post becomes an indexable empty page.
if (const Post* p = content.FindPost(route.slug)) return RenderPost(*p);
break;
}
case RouteKind::Demos: return RenderDemos(content.demos);
case RouteKind::Demo: {
if (const Demo* d = content.FindDemo(route.slug)) return RenderDemo(*d);

2
shared/interfaces/Catcrafts.Shared.cppm
View file

@ -35,6 +35,8 @@ export import :Html;
export import :Json;
export import :Form;
export import :Model;
export import :Media;
export import :Markdown;
export import :Content;
export import :Money;
export import :Route;

90
styles/styles.css
View file

@ -487,6 +487,96 @@ treatment it replaces, which read as the wrong category for the work.
overflow-wrap: anywhere;
}
/* Only reached by a post with no excerpt to trail — normally the read-more
link rides at the end of .post-card__excerpt instead. */
.post-card__more { margin-top: var(--s-1); }
/* Trailing an excerpt, the link is part of that paragraph and wraps with it,
but it must not break across lines itself: "Read the full / post →" reads
as two different things. */
.post-card__excerpt .link-more { white-space: nowrap; }
/* ── one post, in full ──────────────────────────────────────────────
The body is prose rendered from Markdown by Catcrafts.Shared:Markdown,
so the element set here is exactly what that module can emit and no
more. Everything is capped at --measure: a post page is reading
material, and full-viewport line lengths are not read, they are
skimmed. Media is the exception — it keeps the full column, because a
screenshot of a feature-support table is unreadable at text width. */
.post-body {
display: grid;
gap: var(--s0);
margin-top: var(--s1);
}
/* The measure goes on the prose, not on the container, so media can use the
full column: a post page is reading material and full-viewport line
lengths get skimmed rather than read, but a screenshot of a feature
support table is unreadable at text width. */
.post-body > :not(.post-media) { max-width: var(--measure); }
/* .post-media carries its own top margin for the cards, where it follows an
excerpt; inside this grid the gap already provides it. */
.post-body > .post-media { margin-top: 0; }
/* Grid items default to min-width:auto, which refuses to shrink below their
content. Without this the overflow-x on a code block below never engages —
the track widens instead and the whole PAGE scrolls sideways, which is the
one thing wide content must never cause. */
.post-body > *, .post-body__quote > * { min-width: 0; }
.post-body h2 { font-size: var(--step-2); letter-spacing: -0.01em; }
.post-body h3 { font-size: var(--step-1); }
.post-body h4, .post-body h5, .post-body h6 { font-size: var(--step-0); }
/* Extra air above a heading, none below: a heading belongs to what
follows it, and the grid gap alone reads as floating between two
equally distant paragraphs. */
.post-body :is(h2, h3, h4, h5, h6) { margin-top: var(--s1); }
.post-body :is(h2, h3, h4, h5, h6):first-child { margin-top: 0; }
.post-body a { overflow-wrap: anywhere; }
.post-body__list { padding-left: var(--s1); display: grid; gap: var(--s-2); }
.post-body__list li { padding-left: var(--s-3); }
.post-body__quote {
padding: var(--s-2) var(--s0);
border-left: 2px solid var(--border-strong);
background: var(--surface-1);
border-radius: 0 var(--radius-sm) var(--radius-sm) 0;
color: var(--text-muted);
display: grid;
gap: var(--s-1);
}
.post-body__code {
padding: var(--s-1) var(--s0);
background: var(--surface-0);
border: 1px solid var(--border);
border-radius: var(--radius-sm);
font-family: var(--font-mono);
font-size: var(--step--1);
/* Terminal output and measurement tables are tabular: wrapping them
destroys the alignment that is the whole reason they were pasted.
Scroll the block instead — never the page. */
overflow-x: auto;
}
.post-body :not(pre) > code {
font-family: var(--font-mono);
font-size: 0.9em;
padding: 0.1em 0.3em;
background: var(--surface-0);
border: 1px solid var(--border);
border-radius: var(--radius-sm);
}
/* An image inside a sentence still gets its own line — these are
screenshots being pointed at, not icons sitting in the text. The <picture>
is what sits in the paragraph now (the reset already makes it a block);
the bare .post-media__item covers an image with no format ladder, which
is what a mirror failure or a missing encoder leaves behind. */
.post-body p > picture,
.post-body p > .post-media__item { margin-top: var(--s-2); }
.post__back { margin-top: var(--s1); }
/* Post media — screenshots and screen recordings of the work, mirrored to our
own origin by tools/fetch-media.sh. */
.post-media {

126
tools/e2e.sh
View file

@ -140,6 +140,10 @@ status /legal/nope 404
# A slug that cannot be one of ours is rejected before any lookup.
status /shop/BAD--slug 404
status /demos/nope 404
# A post slug that parsed but names nothing must be a real 404, or every typo
# and every retired post becomes an indexable empty page.
status /posts/nope 404
status /posts/BAD--slug 404
# The retired blog URLs are still in the wild; they must redirect, not 404.
status /blog 301
status /blog/hello-world 301
@ -367,7 +371,7 @@ else
fi
# `poster` is in the list because a video poster is fetched on page load exactly
# like an <img> src is, so a third-party poster leaks the same visitor IP.
if curl -s "$BASE/posts" | grep -qE '(src|href|poster)="https?://[^"]*\.(mp4|webm|webp|png|jpe?g|gif)'; then
if curl -s "$BASE/posts" | grep -qE '(src|srcset|href|poster)="https?://[^"]*\.(mp4|webm|webp|avif|png|jpe?g|gif)'; then
bad "/posts media origin" "media loaded from a third party"
else
ok "/posts loads no media from a third party"
@ -417,6 +421,126 @@ else
bad "video preload" "expected preload=\"metadata\""
fi
echo "== post pages =="
# The post page is where the body lives, and the body is the reason the site
# has anything for a search engine to index beyond a list of links off it. Its
# slug is data, so take one from the page rather than hardcoding a title that
# will be wrong the week after it is written.
POST_PATH=$(curl -s "$BASE/posts" | grep -oE 'href="/posts/[a-z0-9-]+"' \
| head -n1 | sed 's/href="//; s/"$//')
if [ -z "$POST_PATH" ]; then
bad "post pages" "/posts links no post page; nothing carries a body"
else
ok "/posts links a post page ($POST_PATH)"
status "$POST_PATH" 200
body_has /posts 'Read the full post' "/posts offers the full post"
# And it trails the excerpt, immediately after the ellipsis the truncation
# left, rather than sitting as its own row below the media. The excerpt is
# escaped text, so nothing but the link can put a '<' between the two.
if curl -s "$BASE/posts" \
| grep -qE '<p class="post-card__excerpt">[^<]*<a class="link-more" href="/posts/'; then
ok "read-more trails the excerpt"
else
bad "read-more placement" "not inside the excerpt paragraph"
fi
body_has "$POST_PATH" '<div class="post-body">' "post page carries the rendered body"
# Rendered, not dumped: a body that reached the page as literal Markdown
# would show its own asterisks and hashes to the reader and to a crawler.
if curl -s "$BASE$POST_PATH" | grep -qE '<(p|h2|h3|h4|ul|ol|blockquote|pre)>'; then
ok "post body is real markup, not literal Markdown"
else
bad "post body markup" "no block elements found in the body"
fi
# The canonical points here, not at the instance. That is the entire SEO
# argument for hosting the body: two copies of the text exist, and this
# says which one is the original as far as this site is concerned.
body_has "$POST_PATH" 'rel="canonical" href="https://catcrafts.net/posts/' \
"post page is its own canonical"
body_has "$POST_PATH" '"@type":"BlogPosting"' "post page carries BlogPosting JSON-LD"
body_has "$POST_PATH" '"@id":"https://catcrafts.net/#organization"' \
"post JSON-LD joins the organization node"
body_has "$POST_PATH" '"@id":"https://catcrafts.net/about#person"' \
"post JSON-LD joins the founder node"
body_has "$POST_PATH" 'property="og:type" content="article"' "post page is an article to og:"
# Hosting the body does not mirror the discussion; the thread is still one
# click away and is still where the comments are.
if curl -s "$BASE$POST_PATH" | grep -qE 'href="https://[a-z0-9.-]+/post/[0-9]+"'; then
ok "post page still links its thread"
else
bad "post thread link" "no https://<instance>/post/<id> link on the post page"
fi
# The body is prose, not an application. Same rule as /projects.
body_lacks "$POST_PATH" '<script>' "post page ships no executable script"
body_lacks "$POST_PATH" '<base' "post page has no base tag"
# Every inline image and video the body embeds is mirrored, exactly like a
# card's media — the privacy notice's "everything comes from catcrafts.net"
# covers href as well as src, so a body linking a .webp on someone else's
# instance is the same leak as embedding one.
for pg in $(curl -s "$BASE/sitemap.xml" \
| grep -oE '/posts/[a-z0-9-]+' | head -n 20); do
if curl -s "$BASE$pg" \
| grep -qE '(src|srcset|href|poster)="https?://[^"]*\.(mp4|webm|webp|avif|png|jpe?g|gif)'; then
bad "$pg media origin" "body media loaded from a third party"
else
ok "$pg loads no media from a third party"
fi
done
# The image format ladder: AVIF first, the mirrored original next, and a
# PNG on the <img> underneath, so exactly one file is fetched and every
# browser can read one of them. Order is the whole point — a browser takes
# the first source it understands — so assert the sequence, not just that
# the pieces are present.
if curl -s "$BASE$POST_PATH" | grep -q '<picture>'; then
if curl -s "$BASE$POST_PATH" \
| grep -qE '<picture><source srcset="/media/[^"]+\.avif" type="image/avif">'; then
ok "inline images lead with an AVIF source"
else
bad "image ladder" "the first source is not the AVIF"
fi
if curl -s "$BASE$POST_PATH" \
| grep -qE '<img class="post-media__item"[^>]*src="/media/[^"]+\.png"'; then
ok "inline images fall back to a PNG the img itself points at"
else
bad "image fallback" "the <img> base is not a PNG"
fi
# Every tier has to be a file that exists, or the ladder serves a 404 to
# whichever browsers pick that rung — which is precisely the set of
# browsers nobody testing this site is using.
missing=0
for f in $(curl -s "$BASE/sitemap.xml" | grep -oE '/posts/[a-z0-9-]+' | head -n 20 \
| while read -r pg; do curl -s "$BASE$pg"; done \
| grep -oE '(src|srcset)="/media/[^"]+"' \
| sed 's/.*="//; s/"$//' | sort -u); do
[ -f "media${f#/media}" ] || { missing=$((missing + 1)); echo " missing: $f" >&2; }
done
if [ "$missing" -eq 0 ]; then
ok "every referenced media file is on the mount"
else
bad "media files" "$missing referenced file(s) not on the mount"
fi
else
skip "image format ladder" "no <picture> on this page — ffmpeg absent at mirror time?"
fi
# Inline screenshots get dimensions from the sidecar list fetch-media.sh
# writes, because Markdown syntax has nowhere to carry them — without it
# the prose below every image jumps as the file arrives. Conditional: a
# post whose body embeds nothing has nothing to check.
if curl -s "$BASE$POST_PATH" | grep -q '<img class="post-media__item"'; then
if curl -s "$BASE$POST_PATH" \
| grep -qE '<img class="post-media__item"[^>]*width="[0-9]+" height="[0-9]+"'; then
ok "inline body images carry width/height"
else
bad "inline image dimensions" "an embedded body image has no dimensions"
fi
fi
fi
# The sitemap has to advertise the pages, or hosting the bodies buys nothing.
if curl -s "$BASE/sitemap.xml" | grep -qE '<loc>https://catcrafts.net/posts/[a-z0-9-]+</loc>'; then
ok "sitemap lists the post pages"
else
bad "sitemap post pages" "no /posts/<slug> entry"
fi
echo "== headers =="
header_has / 'x-content-type-options: *nosniff' "nosniff on pages"
header_has / 'cache-control: *public' "pages are cacheable"

275
tools/fetch-media.sh
View file

@ -4,6 +4,21 @@
#
# Run AFTER tools/fetch-posts.sh, which records the original URLs.
#
# TWO KINDS OF MEDIA, one pipeline:
#
# * a post's headline file — `.media`, the recording or screenshot the post is
# about;
# * everything embedded inside the body — the screenshots a post argues with,
# which are just as much content and, until the body was hosted here, were
# never fetched at all.
#
# Both are content-addressed into the same directory, so a file used as one
# post's headline and quoted inside another's body is stored once. Body files
# are rewritten IN THE MARKDOWN TEXT (the body is still Markdown at this point)
# and additionally recorded in `.body_media`, which is where the renderer reads
# the dimensions, poster and H.264 fallback that Markdown syntax has nowhere to
# carry.
#
# WHY MIRROR rather than embed from the source:
#
# * Privacy. The privacy notice states that everything the browser loads comes
@ -32,6 +47,21 @@ MEDIA_DIR="${1:-media}"
POSTS="content/posts.json"
MAX_BYTES=$((64 * 1024 * 1024))
# What counts as a media reference inside a post body: an absolute URL or a path
# we have already rewritten, ending in a media extension.
#
# Local paths are in the pattern deliberately. Leaving them out looked right —
# nothing needs downloading twice — but it is what made a second run destructive
# rather than idempotent: the already-rewritten body references were not
# enumerated, so they never re-entered the mirror map, so the body_media sidecar
# came back with only the handful of entries that happened to still be absolute.
# Matching them means they are adopted from the mount and everything is rebuilt
# exactly as it was.
#
# One definition, passed to every jq that needs it, because three copies of a
# regex is three chances for one of them to drift.
MEDIA_REF_RE='(?:https?://|/media/)[^\s)\]"<>]+\.(?:mp4|webm|mov|webp|png|jpe?g|gif|avif)'
# Media we host ourselves, published by tools/publish-media.sh before the post
# that carries it exists. Such a URL is ALREADY the one the page should use, so
# there is nothing to fetch: the bytes are on the media mount, and downloading
@ -54,6 +84,12 @@ mkdir -p "$MEDIA_DIR"
HAVE_FFPROBE=0
command -v ffprobe >/dev/null 2>&1 && HAVE_FFPROBE=1
# ffmpeg does the still-image transcodes below. Optional in exactly the same
# way ffprobe is: without it every image is served as the single file the mirror
# downloaded, which is what this site did before the format ladder existed.
HAVE_FFMPEG=0
command -v ffmpeg >/dev/null 2>&1 && HAVE_FFMPEG=1
# Sets $w and $h for the file named in $1, or leaves both 0.
#
# One query per dimension. Asking for both at once and splitting the CSV looked
@ -95,24 +131,161 @@ probe_dims() {
fi
}
# Derive the two renditions a mirrored image is served between: AVIF above it
# and PNG below. Sets $avif_name / $png_name to the sibling file names, or
# leaves one empty when that rendition could not be produced — :Media then drops
# the tier rather than pointing at a file that is not on the mount.
#
# Siblings are named after the source file, which is itself the hash of its
# bytes, so a rendition already present is never re-encoded and a changed source
# gets new names. Only genuinely new images cost encoder time; a rebuild costs
# none, which is what keeps this off the critical path of every deploy.
#
# WHY BOTH TIERS. AVIF is smaller than the WebP the instances serve (~15% on
# these screenshots, far more on photographs) and is what almost every visitor
# actually receives. PNG is lossless and universally understood, which is what
# makes it a fallback worth having — but it is also several times the size of
# the WebP beside it, so the <picture> offers the mirrored original in between
# and the PNG is reached only by a browser that understands neither of the
# other two.
#
# The settings, measured against these files rather than guessed:
# crf 26, cpu-used 6 SSIM 0.997 against the source and still smaller than
# it, at roughly half a second per image.
# yuv444p these are screenshots of text. Re-subsampling chroma
# that pict-rs already subsampled once fringes coloured
# text visibly, and full chroma costs about 3% here.
transcode_image() {
avif_name=""
png_name=""
_file="$1"
_name="$2"
_base="${_name%.*}"
# Already that format: serve the mirrored file as the tier rather than
# re-encoding it into a second copy of itself.
case "$_name" in *.avif) avif_name="$_name" ;; esac
case "$_name" in *.png) png_name="$_name" ;; esac
[ "$HAVE_FFMPEG" = 1 ] || return 0
# An animated source is not a still, and -frames:v 1 would silently freeze
# it. Leave it entirely alone: one moving GIF is worth more than three
# copies of its first frame. nb_frames is N/A for WebP, so the frames have
# to actually be counted — ~75 ms on a 3 MP image, once per new file.
_frames=$(ffprobe -v error -select_streams v:0 -count_frames \
-show_entries stream=nb_read_frames \
-of default=nw=1:nk=1 "$_file" 2>/dev/null | head -n1)
case "$_frames" in
''|*[!0-9]*|1) ;; # unknown or a single frame: a still
*) echo "fetch-media: $_name is animated, serving it as one file" >&2
return 0 ;;
esac
# Alpha has to survive the transcode: an image with a transparent corner
# encoded into a format with no alpha plane gains an opaque black one.
_pixfmt=$(ffprobe -v error -select_streams v:0 -show_entries stream=pix_fmt \
-of default=nw=1:nk=1 "$_file" 2>/dev/null | head -n1)
case "$_pixfmt" in
yuva*|rgba*|bgra*|argb*|abgr*|gbrap*|ya8|ya16*|pal8) _avif_pix=yuva444p ;;
*) _avif_pix=yuv444p ;;
esac
# Encoded to a .part, checked, and only then renamed — so an interrupted or
# wrong-format encode cannot leave a file the next run adopts as finished.
if [ -z "$avif_name" ]; then
_cand="$_base.avif"
if [ -f "$MEDIA_DIR/$_cand" ]; then
avif_name="$_cand"
elif encode_rendition "$_file" "$_cand" av1 \
-c:v libaom-av1 -still-picture 1 -crf 26 -cpu-used 6 \
-pix_fmt "$_avif_pix" -f avif; then
avif_name="$_cand"
fi
fi
if [ -z "$png_name" ]; then
_cand="$_base.png"
if [ -f "$MEDIA_DIR/$_cand" ]; then
png_name="$_cand"
elif encode_rendition "$_file" "$_cand" png -c:v png -f image2; then
png_name="$_cand"
fi
fi
}
# encode_rendition SRC OUTNAME EXPECTED_CODEC ffmpeg-args...
#
# Runs the encode into a .part, verifies the result really is the codec asked
# for, and only then publishes it. Returns non-zero (leaving nothing behind) if
# either step fails, which drops that tier rather than shipping a broken one.
#
# The verification is not paranoia. ffmpeg picks an encoder from the MUXER when
# one is not named, and the image2 muxer defaults to MJPEG — so `-f image2
# out.png` silently produced a run of lossy JPEGs sitting under .png names, which
# the page then advertised to browsers as image/png. The codec is pinned by the
# callers above; this is the check that the pin held.
encode_rendition() {
_src="$1"; _out="$2"; _want="$3"
shift 3
if ! ffmpeg -y -v error -i "$_src" -frames:v 1 "$@" \
"$MEDIA_DIR/$_out.part" 2>/dev/null; then
rm -f "$MEDIA_DIR/$_out.part"
echo "fetch-media: could not encode $_out, serving without that tier" >&2
return 1
fi
_got=$(ffprobe -v error -select_streams v:0 -show_entries stream=codec_name \
-of default=nw=1:nk=1 "$MEDIA_DIR/$_out.part" 2>/dev/null | head -n1)
if [ "$_got" != "$_want" ]; then
rm -f "$MEDIA_DIR/$_out.part"
echo "fetch-media: $_out came out as '$_got', expected '$_want' — discarding it" >&2
return 1
fi
mv "$MEDIA_DIR/$_out.part" "$MEDIA_DIR/$_out"
chmod 0644 "$MEDIA_DIR/$_out"
encoded=$((encoded + 1))
return 0
}
MAP="$(mktemp)"
POSTERMAP="$(mktemp)"
FALLBACKMAP="$(mktemp)"
trap 'rm -f "$MAP" "$POSTERMAP" "$FALLBACKMAP"' EXIT
AVIFMAP="$(mktemp)"
POSTERONLY="$(mktemp)"
trap 'rm -f "$MAP" "$POSTERMAP" "$FALLBACKMAP" "$AVIFMAP" "$POSTERONLY"' EXIT
printf '[]' > "$MAP"
printf '[]' > "$POSTERMAP"
printf '[]' > "$FALLBACKMAP"
printf '[]' > "$AVIFMAP"
# URLs that are ONLY ever a video's poster frame. They are skipped by the
# transcode above, because `poster` takes exactly one URL: a <video> cannot
# negotiate a format the way <picture> can, so the renditions would be files
# nothing is able to ask for. A file that is a poster somewhere and an ordinary
# image somewhere else is not in this list and is transcoded normally.
jq -r --arg re "$MEDIA_REF_RE" \
'([.[].media[]? | .poster // empty] | map(select(. != "")) | unique) as $posters
| ([.[].media[]? | .src] + [.[] | .body // "" | scan($re)] | unique) as $srcs
| ($posters - $srcs) | .[]' "$POSTS" > "$POSTERONLY" 2>/dev/null || true
downloaded=0
reused=0
adopted=0
failed=0
encoded=0
# Every distinct media URL across all posts, so a file shared by two posts is
# fetched once. Video posters are in here too: a poster left pointing at the
# source instance would leak a visitor IP on page load exactly like an embedded
# image would, and it is the frame shown before anyone presses play.
#
# Body URLs are found by pattern rather than by parsing Markdown: anything that
# looks like an absolute URL ending in a media extension is mirrored, whether it
# was written as an embed, as a link, or bare. That is deliberately wider than
# "images the body displays" — the origin rule covers href as well as src, and a
# link whose target is a .webp on someone else's instance is still a third-party
# address on our page. Already-rewritten paths start with /media/ and so do not
# match, which is what makes re-running this a no-op.
#
# Fed by a here-document rather than a pipe so the counters below survive — in
# `jq | while`, the loop runs in a subshell and every increment is discarded.
while IFS= read -r src; do
@ -152,6 +325,26 @@ while IFS= read -r src; do
fi
adopted=$((adopted + 1))
;;
/media/*)
# Already rewritten by an earlier run of this script. Adopt the file on
# the mount rather than trying to fetch our own path as though it were a
# URL — which is what makes running this twice a no-op instead of a way
# to lose every rewrite it made the first time. The script is meant to
# follow fetch-posts.sh, but "meant to" is not a guarantee, and the
# failure was silent: the body_media list simply came back empty.
name=${src#/media/}
case "$name" in
''|*/*|*..*)
echo "fetch-media: refusing suspicious local path: $src" >&2
failed=$((failed + 1)); continue ;;
esac
dest="$MEDIA_DIR/$name"
if [ ! -f "$dest" ]; then
echo "fetch-media: $name not on the media mount, leaving it alone: $src" >&2
failed=$((failed + 1)); continue
fi
adopted=$((adopted + 1))
;;
*)
ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z')
case "$ext" in
@ -226,6 +419,26 @@ while IFS= read -r src; do
&& mv "$FALLBACKMAP.new" "$FALLBACKMAP"
fi
;;
*.webp|*.png|*.jpg|*.jpeg|*.gif|*.avif)
# The AVIF and PNG tiers this image is served between. Both maps are
# keyed by the LOCAL path, like the video ones above, so the rewrite
# below can look them up from the src it has just written.
if grep -qxF "$src" "$POSTERONLY" 2>/dev/null; then
: # poster-only; see POSTERONLY above
else
transcode_image "$dest" "$name"
if [ -n "$avif_name" ]; then
jq --arg k "/media/$name" --arg v "/media/$avif_name" \
'. + [{key: $k, value: $v}]' "$AVIFMAP" > "$AVIFMAP.new" \
&& mv "$AVIFMAP.new" "$AVIFMAP"
fi
if [ -n "$png_name" ]; then
jq --arg k "/media/$name" --arg v "/media/$png_name" \
'. + [{key: $k, value: $v}]' "$FALLBACKMAP" > "$FALLBACKMAP.new" \
&& mv "$FALLBACKMAP.new" "$FALLBACKMAP"
fi
fi
;;
esac
jq --arg src "$src" --arg path "/media/$name" \
@ -233,20 +446,25 @@ while IFS= read -r src; do
'. + [{src: $src, path: $path, w: $w, h: $h}]' "$MAP" > "$MAP.new" \
&& mv "$MAP.new" "$MAP"
done <<EOF
$(jq -r '[.[].media[]? | .src, (.poster // empty)] | map(select(. != "")) | unique[]' "$POSTS")
$(jq -r --arg re "$MEDIA_REF_RE" \
'[ (.[].media[]? | .src, (.poster // empty)),
(.[] | .body // "" | scan($re)) ]
| map(select(. != "")) | unique[]' "$POSTS")
EOF
echo "fetch-media: $downloaded new, $reused already present, $adopted self-hosted, $failed failed"
echo "fetch-media: $encoded image rendition(s) encoded this run"
# Rewrite each media entry to the local path. An entry with no mapping (download
# failed) keeps its original src, so the page still shows something rather than
# silently dropping the post's whole point.
TMP_POSTS="$(mktemp)"
if jq --slurpfile map "$MAP" --slurpfile posters "$POSTERMAP" \
--slurpfile fallbacks "$FALLBACKMAP" '
--slurpfile fallbacks "$FALLBACKMAP" --slurpfile avifs "$AVIFMAP" '
($map[0] | map({key: .src, value: .}) | from_entries) as $m
| ($posters[0] | from_entries) as $pm
| ($fallbacks[0] | from_entries) as $fm
| ($avifs[0] | from_entries) as $am
| map(.media = ((.media // []) | map(
. as $item
| ($m[$item.src] // null) as $hit
@ -271,7 +489,42 @@ if jq --slurpfile map "$MAP" --slurpfile posters "$POSTERMAP" \
# sibling to find.
| if (($fm[.src] // "") != "")
then . + { fallback: $fm[.src] }
else . end
# The AVIF tier, keyed by the LOCAL src like the two maps above. Only
# ever set for images, and only when the encode actually produced one.
| if (($am[.src] // "") != "")
then . + { avif: $am[.src] }
else . end)))
# ── the body ──────────────────────────────────────────────────────
#
# Substitution is literal (split/join, not gsub), because these URLs are
# full of regex metacharacters and a mirrored path must land in the text
# exactly as written. A URL with no mapping — its download failed — is left
# alone, so the post still shows the image rather than losing it; the
# accounting at the end of this script reports that as media still pointing
# at its source.
| ($m | to_entries) as $subs
| map(.body = (reduce $subs[] as $s ((.body // "");
split($s.key) | join($s.value.path))))
# Everything the rewritten body now points at, as records the renderer can
# read: Markdown has nowhere to put a width, a poster frame or a second
# source, so the sidecar list is how an inline video gets the same treatment
# as a headline one. Keyed by the LOCAL path, which is what the body says
# by this point.
| ($m | map({ key: .path, value: . }) | from_entries) as $byPath
| map(.body_media = ([ (.body // "")
| scan("/media/[A-Za-z0-9._-]+")
| . as $path
| select($byPath[$path] != null)
| { src: $path,
kind: (if ($path | test("\\.(?:mp4|webm|mov)$"))
then "video" else "image" end),
poster: ($pm[$path] // ""),
fallback: ($fm[$path] // ""),
avif: ($am[$path] // ""),
w: ($byPath[$path].w // 0),
h: ($byPath[$path].h // 0) } ]
| unique_by(.src)))
' "$POSTS" > "$TMP_POSTS" 2>/dev/null; then
# Same reason as the chmod on each mirrored file: mktemp is 0600 and the
# mode survives to production, where other users must read this.
@ -283,10 +536,18 @@ else
exit 1
fi
total=$(jq '[.[].media[]? | .src, (.poster // empty) | select(. != "")] | length' "$POSTS")
local_count=$(jq '[.[].media[]? | .src, (.poster // empty)
| select(startswith("/media/"))] | length' "$POSTS")
echo "fetch-media: $local_count of $total media entries served locally ($(du -sh "$MEDIA_DIR" | cut -f1) in $MEDIA_DIR)"
# Counted over the bodies too, because a body URL that never mirrored is the
# same privacy leak as a card one and must not be reported as success.
total=$(jq --arg re "$MEDIA_REF_RE" \
'[ (.[].media[]? | .src, (.poster // empty)),
(.[] | .body // "" | scan($re)),
(.[].body_media[]? | .src, (.poster // empty)) ]
| map(select(. != "")) | unique | length' "$POSTS")
local_count=$(jq '[ (.[].media[]? | .src, (.poster // empty)),
(.[].body_media[]? | .src, (.poster // empty)) ]
| map(select(startswith("/media/"))) | length' "$POSTS")
inline=$(jq '[.[].body_media[]?] | length' "$POSTS")
echo "fetch-media: $local_count of $total media entries served locally ($inline of them embedded in post bodies; $(du -sh "$MEDIA_DIR" | cut -f1) in $MEDIA_DIR)"
if [ "$local_count" -ne "$total" ]; then
echo "fetch-media: $((total - local_count)) still point at their source — see the failures above" >&2
fi

70
tools/fetch-posts.sh
View file

@ -28,6 +28,12 @@ CONFIG="${1:-content/posts-sources.json}"
OUT="content/posts.json"
LIMIT=50
EXCERPT_CHARS=280
# A ceiling, not a target: the longest of these bodies is around 6 KB and this
# file is fetched into memory before the wasm module starts, so an outlier must
# not be able to grow the bundle without bound. A truncated body renders as
# truncated prose (the Markdown renderer tolerates an unclosed fence), which is
# a better failure than a build that silently ships a megabyte.
MAX_BODY_CHARS=32768
command -v jq >/dev/null 2>&1 || { echo "fetch-posts: jq not found, keeping existing $OUT" >&2; exit 0; }
[ -f "$CONFIG" ] || { echo "fetch-posts: $CONFIG not found, keeping existing $OUT" >&2; exit 0; }
@ -84,13 +90,39 @@ fi
# Both are ORIGINAL urls here; tools/fetch-media.sh mirrors them
# and rewrites to local paths, so nothing the browser loads is
# third-party.
# excerpt : body flattened to one line and truncated. Markdown is NOT
# rendered — the site has no markdown pipeline by design, so any
# surviving syntax would show as literal characters. Strip the
# common inline markers and let the rest be plain text.
# excerpt : body flattened to one line and truncated, for the card and for
# the meta description. Markdown markers are stripped rather than
# rendered — this string lands in places that are plain text by
# definition (<meta name="description">, og:description), so any
# surviving syntax would show as literal characters.
# body : the post, whole, still as Markdown. Rendered by
# Catcrafts.Shared:Markdown at page-render time rather than
# converted here, because that module is inside the escaping
# guarantee and a shell script writing HTML into a content file
# would not be — text from someone else's server must not be able
# to become markup anywhere but there.
# slug : the post's own URL at /posts/<slug>, from the title. Collisions
# take the post's numeric id as a suffix rather than a positional
# one: a later post sharing a title would otherwise renumber an
# earlier post's URL out from under everyone who linked it.
# deleted / removed posts are dropped rather than rendered as empty cards.
if ! jq --argjson n "$EXCERPT_CHARS" \
--argjson maxbody "$MAX_BODY_CHARS" \
--slurpfile cfg "$CONFIG" '
# Catcrafts.Shared:Route::IsValidSlug is the contract: lowercase ASCII,
# digits and single hyphens, no leading or trailing hyphen, at most 64
# characters. A slug that fails it is dropped by the loader, which costs
# the post its page — so the shape is produced correctly here rather than
# sanitised on the way out.
def slugify:
ascii_downcase
| gsub("[^a-z0-9]+"; "-")
| sub("^-+"; "") | sub("-+$"; "");
# Truncate on a word boundary where there is one: a slug cut mid-word reads
# like a typo, and these titles are long enough to hit the limit.
def clamp($n):
(if (length > $n) then (.[0:$n] | sub("-[^-]*$"; "")) else . end)
| sub("-+$"; "");
($cfg[0].communities | map(ascii_downcase)) as $allow
| [ .posts[]
| select((.post.deleted // false) == false)
@ -101,6 +133,11 @@ if ! jq --argjson n "$EXCERPT_CHARS" \
| select(($comm | ascii_downcase) as $c | $allow | index($c))
| {
title: ($p.post.name // ""),
slug: (($p.post.name // "") | slugify | clamp(64)),
# Carried only as far as the de-duplication pass below, which strips
# it: it is a tie-breaker, not content.
uid: (($p.post.ap_id // "") | sub(".*/"; "") | ascii_downcase
| gsub("[^a-z0-9]"; "") | .[0:12]),
permalink: ($p.post.ap_id // ""),
# A link post whose target IS an image or video is a media post, not a
# link post: the file is captured in `media` and embedded, so keeping
@ -123,6 +160,10 @@ if ! jq --argjson n "$EXCERPT_CHARS" \
| gsub(" +"; " ")
| ltrimstr(" ") | rtrimstr(" ")
| if (. | length) > $n then (.[0:$n] | sub(" [^ ]*$"; "")) + "…" else . end),
# Verbatim apart from CR removal (a stray \r would render as a stray
# character inside a code block, where nothing is interpreted) and the
# size ceiling.
body: (($p.post.body // "") | gsub("\r"; "") | .[0:$maxbody]),
media: ([ ($p.post.url // "")
| select(test("\\.(?:mp4|webm|mov|webp|png|jpe?g|gif|avif)$"))
| (if test("\\.(mp4|webm|mov)$") then "video" else "image" end) as $kind
@ -139,7 +180,26 @@ if ! jq --argjson n "$EXCERPT_CHARS" \
score: ($p.counts.score // 0),
comments: ($p.counts.comments // 0)
}
]' "$RAW" > "$TMP" 2>/dev/null; then
]
# Make every slug unique, and keep it that way across builds.
#
# Only a duplicate is suffixed, so the ordinary post keeps the clean URL the
# title earned. The suffix is the numeric id of the post itself rather than
# a counter, because a counter is positional: a new post repeating an older
# title arrives at the front of this newest-first list, takes the bare slug,
# and silently renumbers the older one — breaking a URL that is already in
# search results and in whatever links people have shared.
# (Note for editors: this jq program is inside a single-quoted shell string,
# so no apostrophes anywhere in it.)
| reduce .[] as $e ({ seen: {}, out: [] };
(if ($e.slug | length) == 0 then "post" else $e.slug end) as $base
| (if (.seen[$base] // false)
then (($base | clamp(50)) + "-"
+ (if ($e.uid | length) == 0 then "x" else $e.uid end))
else $base end) as $slug
| { seen: (.seen + { ($base): true, ($slug): true }),
out: (.out + [ ($e | del(.uid)) + { slug: $slug } ]) })
| .out' "$RAW" > "$TMP" 2>/dev/null; then
echo "fetch-posts: response did not match the expected shape, keeping existing $OUT" >&2
exit 0
fi