posts and avif
Some checks failed
Deploy / build-deploy (push) Failing after 5m49s

This commit is contained in:
Jorijn van der Graaf 2026-08-10 01:37:26 +02:00
commit 6841623e23
17 changed files with 2306 additions and 148 deletions

View file

@ -27,12 +27,13 @@ jobs:
# run inside this archlinux container — the runner execs them with # run inside this archlinux container — the runner execs them with
# node. This shell step needs no node, so installing it here (before # node. This shell step needs no node, so installing it here (before
# Checkout) is enough. # Checkout) is enough.
# ffmpeg is for ffprobe, which tools/fetch-media.sh uses to read the # ffmpeg/ffprobe are both used by tools/fetch-media.sh. ffprobe reads
# pixel dimensions of each mirrored file. Those become the width/height # the pixel dimensions of each mirrored file, which become the
# attributes that stop the posts page reflowing as 5 MB recordings # width/height attributes that stop the posts page reflowing as 5 MB
# arrive, and tools/e2e.sh asserts they are present — so without this # recordings arrive. ffmpeg transcodes each still image into the AVIF
# package the deploy fails at the e2e gate rather than shipping a # and PNG renditions the page serves it between. tools/e2e.sh asserts
# janky page. # both are present — so without this package the deploy fails at the
# e2e gate rather than shipping a janky page.
pacman -Syu --noconfirm --needed \ pacman -Syu --noconfirm --needed \
nodejs \ nodejs \
clang lld libc++ \ clang lld libc++ \
@ -84,18 +85,19 @@ jobs:
run: tools/fetch-rates.sh run: tools/fetch-rates.sh
- name: Fetch fediverse posts - name: Fetch fediverse posts
# Build-time, not run-time: the site embeds the owner's own posts and # Build-time, not run-time: the site hosts the owner's own posts in full
# links out for discussion, so there is no sync service and no runtime # and links out for the discussion, so there is no sync service and no
# dependency on the instance being up. The script leaves the committed # runtime dependency on the instance being up. The script leaves the
# content/posts.json untouched and exits 0 on any failure, so a # committed content/posts.json untouched and exits 0 on any failure, so a
# fediverse outage cannot fail a deploy. # fediverse outage cannot fail a deploy.
run: tools/fetch-posts.sh run: tools/fetch-posts.sh
- name: Mirror post media - name: Mirror post media
# Downloads the images and screen recordings the posts carry and rewrites # Downloads the images and screen recordings the posts carry — both the
# content/posts.json to point at our own copies, so nothing the browser # headline file and everything embedded inside the body — and rewrites
# loads is third-party — which is what keeps the privacy notice's # content/posts.json (including the body text itself) to point at our own
# "everything comes from catcrafts.net" true. # copies, so nothing the browser loads is third-party. That is what keeps
# the privacy notice's "everything comes from catcrafts.net" true.
# #
# Content-addressed: the name is the hash of the bytes. Note what that # Content-addressed: the name is the hash of the bytes. Note what that
# does NOT mean — a third-party file is re-fetched on every build, because # does NOT mean — a third-party file is re-fetched on every build, because

File diff suppressed because one or more lines are too long

View file

@ -128,6 +128,62 @@ build. Neither fails the build on a network error — a fediverse outage leaves
previous `posts.json` in place, and a single failed download leaves that one entry previous `posts.json` in place, and a single failed download leaves that one entry
pointing at its original URL rather than losing the post. pointing at its original URL rather than losing the post.
### The full body, and its inline media
Each post is hosted whole at `/posts/<slug>`, with the card on `/posts` linking
to it — the writing is what the site is about, and a page this site can name as
canonical is the only version a search engine can be pointed at. The comments
are *not* mirrored: every post page links out to the thread, which is where the
discussion belongs.
The body stays Markdown in `posts.json` and is rendered by
`Catcrafts.Shared:Markdown` at page-render time, never converted to HTML by the
shell. That is deliberate: the renderer is inside the escaping guarantee, and
text fetched from someone else's server must not be able to become markup
anywhere else. Raw HTML in a body is always shown as text.
`fetch-media.sh` mirrors what the body embeds as well as the headline file, and
rewrites the URLs **inside the Markdown**, so a post page loads nothing
third-party either. It also writes a `body_media` list per post — dimensions,
poster frame and format renditions for each inline file, which Markdown syntax
has nowhere to carry. Slugs come from the title; a duplicate title takes the
post's numeric id as a suffix, so an old post's URL is never renumbered by a new
one. Re-running the script is a no-op: local paths are adopted from the mount
rather than re-fetched.
### The image format ladder
Every mirrored still image is transcoded to two siblings named after its content
hash, and `Catcrafts.Shared:Media` serves all three as one `<picture>` — so the
browser fetches **exactly one**:
| tier | file | size vs. WebP | who gets it |
|---|---|---|---|
| `<source type="image/avif">` | `<hash>.avif` | **76%** | almost everyone |
| `<source type="image/webp">` | `<hash>.webp` (the mirrored original) | 100% | Safari 1416 |
| `<img src>` | `<hash>.png` | **875%** | neither of the above |
The middle tier is why the PNG being ~9× the WebP does not matter: it is free
(the mirror already downloaded that file) and it is what the small number of
non-AVIF browsers actually land on. The PNG is the floor nothing can refuse.
AVIF is encoded at `crf 26, cpu-used 6, yuv444p` — measured at SSIM 0.997
against the source and still smaller than it. Full chroma is deliberate: these
are screenshots of text, and re-subsampling chroma that pict-rs already
subsampled once fringes coloured text visibly, for about 3% more bytes.
Encoding is skipped when the sibling is already on the mount, so only genuinely
new images cost encoder time (~0.5 s each). Animated sources are left alone
entirely — one moving GIF beats three copies of its first frame. Video posters
are skipped too: `poster` takes exactly one URL, so a `<video>` cannot negotiate
a format the way `<picture>` can and the renditions would be unreachable.
Both encodes pin `-c:v` and then **verify the codec that actually came out**.
That check earned its place immediately: `-f image2 out.png` without an explicit
codec makes ffmpeg fall back to the muxer default, which is MJPEG — it silently
produced a full set of lossy JPEGs under `.png` names, served to browsers as
`image/png`. A rendition that fails the check is discarded and its tier dropped.
### Publish the media first, then post it ### Publish the media first, then post it
**The recommended flow is to put a recording on catcrafts.net before writing the **The recommended flow is to put a recording on catcrafts.net before writing the

View file

@ -177,8 +177,7 @@ namespace Catcrafts {
// restarting at cc-link-0 and colliding with the content links — // restarting at cc-link-0 and colliding with the content links —
// getElementById would then return whichever appeared first in the // getElementById would then return whichever appeared first in the
// document and half the links would navigate to the wrong place. // document and half the links would navigate to the wrong place.
const RouteKind navKind = const RouteKind navKind = NavKindFor(route.kind);
route.kind == RouteKind::LegacyBlog ? RouteKind::Posts : route.kind;
Dom::HtmlElementPtr header("cc-header"); Dom::HtmlElementPtr header("cc-header");
if (header.ptr != 0) { if (header.ptr != 0) {
header.SetInnerHTML(TagLinks(Views::RenderNav(navKind).View(), linkTargets)); header.SetInnerHTML(TagLinks(Views::RenderNav(navKind).View(), linkTargets));

View file

@ -45,12 +45,14 @@ static Configuration* SharedLibrary(std::span<const std::string_view> args) {
ApplyStandardArgs(*shared, args); // inherits --target / --debug from the parent ApplyStandardArgs(*shared, args); // inherits --target / --debug from the parent
shared->type = ConfigurationType::LibraryStatic; shared->type = ConfigurationType::LibraryStatic;
std::array<fs::path, 9> ifaces = { std::array<fs::path, 11> ifaces = {
"shared/interfaces/Catcrafts.Shared", "shared/interfaces/Catcrafts.Shared",
"shared/interfaces/Catcrafts.Shared-Html", "shared/interfaces/Catcrafts.Shared-Html",
"shared/interfaces/Catcrafts.Shared-Form", "shared/interfaces/Catcrafts.Shared-Form",
"shared/interfaces/Catcrafts.Shared-Json", "shared/interfaces/Catcrafts.Shared-Json",
"shared/interfaces/Catcrafts.Shared-Model", "shared/interfaces/Catcrafts.Shared-Model",
"shared/interfaces/Catcrafts.Shared-Media",
"shared/interfaces/Catcrafts.Shared-Markdown",
"shared/interfaces/Catcrafts.Shared-Content", "shared/interfaces/Catcrafts.Shared-Content",
"shared/interfaces/Catcrafts.Shared-Money", "shared/interfaces/Catcrafts.Shared-Money",
"shared/interfaces/Catcrafts.Shared-Route", "shared/interfaces/Catcrafts.Shared-Route",

View file

@ -287,8 +287,7 @@ HTTPResponse RenderPage(std::string_view target) {
if (const Demo* d = gContent.FindDemo(route.slug)) wantsWasm = d->needsWasm; if (const Demo* d = gContent.FindDemo(route.slug)) wantsWasm = d->needsWasm;
} }
res.body = Views::RenderDocument(page, res.body = Views::RenderDocument(page,
Views::RenderNav(route.kind == RouteKind::LegacyBlog Views::RenderNav(NavKindFor(route.kind)),
? RouteKind::Posts : route.kind),
Views::RenderFooter(), Views::RenderFooter(),
wantsWasm ? gBootScripts : std::string_view{}, wantsWasm ? gBootScripts : std::string_view{},
gCssHref); gCssHref);
@ -310,6 +309,15 @@ HTTPResponse ServeSitemap() {
out += Html::Escape(pr.slug).Str(); out += Html::Escape(pr.slug).Str();
out += "</loc></url>\n"; out += "</loc></url>\n";
} }
// Post pages, from the same HasPage() test the "read more" links use — a
// sitemap that advertised a post without a body would be pointing crawlers
// at the 404 the dispatcher correctly returns for it.
for (const Post& po : gContent.posts) {
if (!po.HasPage()) continue;
out += " <url><loc>https://catcrafts.net/posts/";
out += Html::Escape(po.slug).Str();
out += "</loc></url>\n";
}
out += "</urlset>\n"; out += "</urlset>\n";
ApplyPageHeaders(res, "application/xml; charset=utf-8", true, false); ApplyPageHeaders(res, "application/xml; charset=utf-8", true, false);
res.body = std::move(out); res.body = std::move(out);

View file

@ -109,6 +109,394 @@ void RunSelfTest() {
CheckEq(Escape("a") + Escape("<"), "a&lt;", "operator+: escapes preserved"); CheckEq(Escape("a") + Escape("<"), "a&lt;", "operator+: escapes preserved");
} }
// The format ladder. One <picture>/<video> builder serves both the cards and
// the post bodies, so these assertions cover every image and video the site
// emits — and the ordering ones matter: a browser takes the FIRST source it
// understands, so a mis-ordered ladder silently serves the wrong tier to
// everyone rather than failing visibly.
void RunMediaSelfTest() {
auto img = [](std::string src, std::string avif, std::string png,
std::int64_t w = 0, std::int64_t h = 0) {
PostMedia m;
m.src = std::move(src);
m.kind = "image";
m.avif = std::move(avif);
m.fallback = std::move(png);
m.width = w;
m.height = h;
return m;
};
// The full ladder: AVIF, then the mirrored original, then the PNG the <img>
// itself points at. Exactly one of the three is ever fetched.
CheckEq(Media::Tag(img("/media/x.webp", "/media/x.avif", "/media/x.png", 800, 600)),
R"(<picture><source srcset="/media/x.avif" type="image/avif">)"
R"(<source srcset="/media/x.webp" type="image/webp">)"
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.png" width="800" height="600"></picture>)",
"media: image ladder is avif, original, png");
// Alt text reaches the <img>, not the <picture> — a screen reader reads the
// img, and an alt on the wrapper is invisible to it.
Check(Media::Tag(img("/media/x.webp", "/media/x.avif", "/media/x.png"), "a cat")
.View().find(R"(alt="a cat" src="/media/x.png")") != std::string_view::npos,
"media: alt lands on the img");
// Degradation, one tier at a time. Each of these is a real state: no
// encoder on the build host, a source that was already PNG, a body image
// whose download failed so there is nothing but the original URL.
CheckEq(Media::Tag(img("/media/x.webp", "", "/media/x.png")),
R"(<picture><source srcset="/media/x.webp" type="image/webp">)"
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.png"></picture>)",
"media: no avif still offers the original above the png");
CheckEq(Media::Tag(img("/media/x.webp", "/media/x.avif", "")),
R"(<picture><source srcset="/media/x.avif" type="image/avif">)"
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.webp"></picture>)",
"media: no png leaves the original as the base");
CheckEq(Media::Tag(img("/media/x.webp", "", "")),
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.webp">)",
"media: no renditions is a bare img, as before any of this existed");
// A source that is already PNG is its own fallback, and must not be
// offered twice — once as a <source> and once as the <img>.
CheckEq(Media::Tag(img("/media/x.png", "/media/x.avif", "/media/x.png")),
R"(<picture><source srcset="/media/x.avif" type="image/avif">)"
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="")"
R"( src="/media/x.png"></picture>)",
"media: a png source is not also listed as a source");
// Likewise a source that is already AVIF.
Check(Media::Tag(img("/media/x.avif", "/media/x.avif", "/media/x.png"))
.View().find("image/avif\"><source") == std::string_view::npos,
"media: an avif source is not listed twice");
// A URL is still a URL: the scheme allowlist applies to srcset exactly as
// it does to src, or the ladder becomes a way around it.
Check(Media::Tag(img("/media/x.webp", "javascript:alert(1)", "/media/x.png"))
.View().find(R"(srcset="#")") != std::string_view::npos,
"media: a hostile srcset is neutralised");
// Video is unchanged by any of this and must stay so.
{
PostMedia v;
v.src = "/media/v.mp4";
v.kind = "video";
v.poster = "/media/v.webp";
v.fallback = "/media/v.h264.mp4";
const auto out = Media::Tag(v);
Check(out.View().starts_with("<video class=\"post-media__item\" controls preload=\"metadata\""),
"media: video is still a video", out.View());
Check(out.View().find("codecs=av01") != std::string_view::npos
&& out.View().find(R"(<source src="/media/v.h264.mp4" type="video/mp4">)")
!= std::string_view::npos,
"media: AV1 then H.264, in that order");
Check(out.View().find("<picture>") == std::string_view::npos,
"media: a video is not wrapped in a picture");
}
// A path with no record renders from the path alone — the mirror failed,
// and showing the picture beats dropping the paragraph's subject.
{
const std::array<PostMedia, 1> known{
img("/media/x.webp", "/media/x.avif", "/media/x.png") };
Check(Media::Find(known, "/media/x.webp") != nullptr, "media: found by src");
Check(Media::Find(known, "/media/nope.webp") == nullptr, "media: unknown src");
const PostMedia guessed = Media::Describe(known, "https://i.example/a.webp");
Check(guessed.kind == "image" && guessed.avif.empty(),
"media: an unmirrored image is described from its path");
Check(Media::Describe(known, "https://i.example/a.mp4").kind == "video",
"media: an unmirrored video is recognised as one");
}
}
// The Markdown renderer, which is the newest place untrusted text becomes
// markup — post bodies are fetched from someone else's server, so every one of
// these assertions is ultimately about the same thing: nothing in a body can
// escape into the document. The structural cases are here too, because a parser
// that silently drops a construct loses content invisibly.
void RunMarkdownSelfTest() {
auto md = [](std::string_view text,
std::span<const PostMedia> media = {}) {
return Markdown::Render(text, media);
};
// ── the guarantee ─────────────────────────────────────────────────
CheckEq(md("<script>alert(1)</script>"),
"<p>&lt;script&gt;alert(1)&lt;/script&gt;</p>", "md: html is text, never markup");
CheckEq(md("![x](javascript:alert(1))"),
R"(<div class="post-media"><img class="post-media__item" loading="lazy" )"
R"(decoding="async" alt="x" src="#"></div>)",
"md: javascript: image source neutralised");
CheckEq(md("[x](javascript:alert(1))"),
R"(<p><a href="#" rel="noopener">x</a></p>)",
"md: javascript: link neutralised");
CheckEq(md("a \" b & c"), "<p>a &quot; b &amp; c</p>", "md: quotes and ampersands escaped");
// A code span is verbatim text, and verbatim is exactly where an escaper
// is most often forgotten.
CheckEq(md("`<b>`"), "<p><code>&lt;b&gt;</code></p>", "md: code span escaped");
// ── blocks ────────────────────────────────────────────────────────
CheckEq(md(""), "", "md: empty body renders nothing");
CheckEq(md("plain text"), "<p>plain text</p>", "md: paragraph");
// Demotion by one: the page h1 is the post title, so a body's own top-level
// heading is a section within it.
CheckEq(md("# Heading"), "<h2>Heading</h2>", "md: h1 demoted to h2");
CheckEq(md("### Heading"), "<h4>Heading</h4>", "md: h3 demoted to h4");
CheckEq(md("#nothashtag"), "<p>#nothashtag</p>", "md: # without a space is not a heading");
CheckEq(md("> quoted"),
R"(<blockquote class="post-body__quote"><p>quoted</p></blockquote>)",
"md: blockquote");
// The quoted lines are re-parsed, so a multi-paragraph quote keeps its
// paragraphs instead of collapsing into one run-on line.
CheckEq(md("> one\n>\n> two"),
R"(<blockquote class="post-body__quote"><p>one</p><p>two</p></blockquote>)",
"md: blockquote keeps its paragraphs");
CheckEq(md("- a\n- b"),
R"(<ul class="post-body__list"><li>a</li><li>b</li></ul>)", "md: unordered list");
CheckEq(md("1. a\n2. b"),
R"(<ol class="post-body__list"><li>a</li><li>b</li></ol>)", "md: ordered list");
// A list resumed after an interrupting paragraph continues its numbering.
// Without the start attribute the mini-guide in one of these posts renders
// as steps 1-4 followed by steps 1, 2, 3.
CheckEq(md("5. e"),
R"(<ol class="post-body__list" start="5"><li>e</li></ol>)",
"md: ordered list keeps the number it announced");
// Blank lines between items are spacing, not seven one-item lists.
CheckEq(md("1. a\n\n2. b"),
R"(<ol class="post-body__list"><li>a</li><li>b</li></ol>)",
"md: blank line inside a list does not split it");
CheckEq(md("---"), "<hr>", "md: thematic break");
CheckEq(md("- - -"), "<hr>", "md: spaced rule is not a one-item list");
// Whitespace in pasted terminal output is the content.
CheckEq(md("```\n a\tb\n```"),
"<pre class=\"post-body__code\"><code> a\tb\n</code></pre>",
"md: fenced code is verbatim");
// An unterminated fence must not swallow the document into nothing.
Check(md("```\nx").View().find("<code>x") != std::string_view::npos,
"md: unterminated fence still renders its content");
// ── inline ────────────────────────────────────────────────────────
CheckEq(md("**bold**"), "<p><strong>bold</strong></p>", "md: strong");
CheckEq(md("*em*"), "<p><em>em</em></p>", "md: emphasis");
CheckEq(md("2 * 3 * 4"), "<p>2 * 3 * 4</p>", "md: spaced asterisks stay literal");
// Underscores are deliberately inert: these posts paste kernel symbol
// names into prose, and italicising half of one is worse than not
// italicising a word that used the underscore form.
CheckEq(md("kworker/u16:8-qc_ufs_qos_swq"),
"<p>kworker/u16:8-qc_ufs_qos_swq</p>", "md: underscores are not emphasis");
CheckEq(md("\\*literal\\*"), "<p>*literal*</p>", "md: backslash escape");
CheckEq(md("[label](https://x.example/y)"),
R"(<p><a href="https://x.example/y" rel="noopener">label</a></p>)", "md: link");
// Bare addresses are pasted constantly in these posts; leaving them inert
// would strip most of the outbound value out of the page.
CheckEq(md("see https://x.example/y"),
R"(<p>see <a href="https://x.example/y">https://x.example/y</a></p>)",
"md: bare URL autolinked");
// ── embedded media ────────────────────────────────────────────────
// A paragraph that is nothing but images becomes the same media block the
// cards use, rather than a <p> of pictures.
CheckEq(md("![a](/media/x.webp)"),
R"(<div class="post-media"><img class="post-media__item" loading="lazy" )"
R"(decoding="async" alt="a" src="/media/x.webp"></div>)",
"md: image-only paragraph is a media block");
Check(md("text ![a](/media/x.webp)").View().starts_with("<p>text <img"),
"md: an image inside a sentence stays inline");
// Dimensions come from the sidecar list, because Markdown syntax has
// nowhere to carry them — and without them the prose below every
// screenshot jumps as the file arrives.
{
std::vector<PostMedia> media;
PostMedia img;
img.src = "/media/x.webp";
img.kind = "image";
img.avif = "/media/x.avif";
img.fallback = "/media/x.png";
img.width = 800;
img.height = 600;
media.push_back(img);
PostMedia vid;
vid.src = "/media/v.mp4";
vid.kind = "video";
vid.poster = "/media/v.poster.webp";
vid.fallback = "/media/v.h264.mp4";
vid.width = 1080;
vid.height = 1920;
media.push_back(vid);
const auto out = md("![a](/media/x.webp)", media);
Check(out.View().find(R"(width="800" height="600")") != std::string_view::npos,
"md: inline image carries its dimensions", out.View());
// Routed through :Media, so a body image gets the same format ladder a
// card image does rather than a second, plainer implementation.
Check(out.View().find(R"(<source srcset="/media/x.avif" type="image/avif">)")
!= std::string_view::npos
&& out.View().find(R"(src="/media/x.png")") != std::string_view::npos,
"md: inline image gets the avif/png ladder", out.View());
// An inline video gets the same treatment a headline one does,
// fallback source and all.
const auto vout = md("![](/media/v.mp4)", media);
Check(vout.View().find(R"(poster="/media/v.poster.webp")") != std::string_view::npos
&& vout.View().find("codecs=av01") != std::string_view::npos
&& vout.View().find(R"(<source src="/media/v.h264.mp4")") != std::string_view::npos,
"md: inline video gets poster and H.264 fallback", vout.View());
}
// ── termination ───────────────────────────────────────────────────
// Unbalanced delimiters are the classic way to hang a hand-written
// parser, and a body is input from someone else's server.
Check(!md("**unclosed").View().empty(), "md: unclosed strong terminates");
Check(!md("[unclosed](").View().empty(), "md: unclosed link terminates");
Check(!md("![](").View().empty(), "md: unclosed image terminates");
Check(!md("`unclosed").View().empty(), "md: unclosed code span terminates");
Check(!md("> > > > > > > > deep").View().empty(), "md: over-deep nesting terminates");
}
// Post pages: the routing, the loader's guard on what becomes a URL, and the
// schema.org joins that keep every post attributed to the one Organization and
// the one Person the rest of the site describes.
void RunPostSelfTest() {
// ── routing ───────────────────────────────────────────────────────
Check(ParseRoute("/posts").kind == RouteKind::Posts, "route: /posts is the list");
Check(ParseRoute("/posts/hello-world").kind == RouteKind::Post, "route: /posts/<slug>");
Check(ParseRoute("/posts/hello-world").slug == "hello-world", "route: post slug captured");
Check(ParseRoute("/posts/hello-world/").kind == RouteKind::Post,
"route: trailing slash normalised");
Check(ParseRoute("/posts/Hello").kind == RouteKind::NotFound,
"route: uppercase slug is not a post URL");
Check(ParseRoute("/posts/../etc").kind == RouteKind::NotFound,
"route: traversal never reaches a lookup");
Check(NavKindFor(RouteKind::Post) == RouteKind::Posts,
"route: a post page highlights the Posts nav entry");
Check(NavKindFor(RouteKind::LegacyBlog) == RouteKind::Posts,
"route: the retired /blog URL highlights it too");
Check(NavKindFor(RouteKind::Shop) == RouteKind::Shop, "route: a nav route is its own entry");
// ── the loader ────────────────────────────────────────────────────
{
const auto posts = LoadPosts(R"([
{"title":"Good","slug":"good-post","permalink":"https://i.example/post/1",
"body":"Hello.","published":"2026-01-01T00:00:00Z",
"body_media":[{"src":"/media/a.webp","kind":"image","w":10,"h":20}]},
{"title":"Bad slug","slug":"NOT A SLUG","permalink":"https://i.example/post/2",
"body":"Hello."},
{"title":"No body","slug":"no-body","permalink":"https://i.example/post/3"}
])");
Check(posts.size() == 3, "posts: all three load");
if (posts.size() == 3) {
Check(posts[0].HasPage() && posts[0].slug == "good-post", "posts: valid slug kept");
Check(posts[0].body == "Hello.", "posts: body loaded");
Check(posts[0].bodyMedia.size() == 1 && posts[0].bodyMedia[0].width == 10,
"posts: body media loaded with dimensions");
// A slug that could never match a route would render a "read more"
// link to a 404 this site points at itself.
Check(posts[1].slug.empty() && !posts[1].HasPage(),
"posts: malformed slug is dropped, costing the page");
// A title and a link out is not a page worth minting a URL for.
Check(!posts[2].HasPage(), "posts: no body means no page");
}
Views::SiteContent content;
content.posts = posts;
Check(content.FindPost("good-post") != nullptr, "posts: found by slug");
Check(content.FindPost("no-body") == nullptr, "posts: a pageless post is not findable");
Check(content.FindPost("nope") == nullptr, "posts: unknown slug is not found");
// Which means the route 404s rather than rendering an empty article.
Check(Views::RenderRoute(ParseRoute("/posts/no-body"), content).status == 404,
"posts: a pageless slug is a real 404");
Check(Views::RenderRoute(ParseRoute("/posts/good-post"), content).status == 200,
"posts: a real post renders");
}
// ── the page ──────────────────────────────────────────────────────
{
Post p;
p.title = "Working GPS!";
p.slug = "working-gps";
p.permalink = "https://lemmy.example/post/42";
p.community = "linuxphones@lemmy.example";
p.published = "2026-06-26T23:16:25Z";
p.excerpt = "A short summary.";
p.body = "# How\n\nIt works.";
PostMedia m;
m.src = "/media/shot.webp";
m.kind = "image";
p.media.push_back(m);
const auto page = Views::RenderPost(p);
Check(page.meta.canonical == "/posts/working-gps",
"post page: canonical is this site, not the instance");
Check(page.meta.ogType == "article", "post page: og:type is article");
Check(page.meta.ogImage == "/media/shot.webp", "post page: og:image from the post media");
Check(page.meta.description == p.excerpt, "post page: description is the excerpt");
Check(page.main.View().find("<h2>How</h2>") != std::string_view::npos,
"post page: the body is rendered, not escaped away");
// The whole reason the body is hosted: the thread is still one click
// away, and the reader is told where the discussion is.
Check(page.main.View().find(p.permalink) != std::string_view::npos,
"post page: still links the thread");
// The identity graph, same joins every other page makes. A typo'd @id
// still renders and still validates — it just quietly splits this post
// away from the entity the rest of the site describes.
auto ld = Json::Parse(page.meta.jsonLd);
Check(ld && ld->IsObject() && ld->Str("@type") == "BlogPosting",
"post schema: parses as a BlogPosting");
if (ld && ld->IsObject()) {
Check(ld->Str("url") == "https://catcrafts.net/posts/working-gps",
"post schema: url is the on-site page");
Check(ld->Str("discussionUrl") == p.permalink,
"post schema: the thread is the discussion, not the content");
Check(ld->Str("datePublished") == p.published, "post schema: publication date");
const Json::Value* author = ld->Find("author");
const Json::Value* publisher = ld->Find("publisher");
Check(author && author->Str("@id") == "https://catcrafts.net/about#person",
"post schema: authored by the Person node on /about");
Check(publisher && publisher->Str("@id") == "https://catcrafts.net/#organization",
"post schema: published by the Organization node");
}
// A post with a page is advertised as one from the list and from home;
// one without keeps pointing at the thread, because there is nothing
// here to send the reader to.
const std::array<Post, 1> one{ p };
const auto list = Views::RenderPosts(one);
Check(list.main.View().find(R"(href="/posts/working-gps")") != std::string_view::npos,
"posts list: links the on-site page");
Check(list.main.View().find("Read the full post") != std::string_view::npos,
"posts list: offers the full post");
// Inside the excerpt paragraph, trailing the text — not a row of its
// own below the media, where it was the same offer made a screen
// further down.
Check(list.main.View().find(
R"(A short summary. <a class="link-more" href="/posts/working-gps">)"
R"(Read the full post</a></p>)") != std::string_view::npos,
"posts list: read-more trails the excerpt", list.main.View());
// With no excerpt there is no sentence to continue, so it falls back to
// a row rather than vanishing with the paragraph that would have held it.
Post unexcerpted = p;
unexcerpted.excerpt.clear();
const std::array<Post, 1> bare{ unexcerpted };
const auto listBare = Views::RenderPosts(bare);
Check(listBare.main.View().find("Read the full post") != std::string_view::npos,
"posts list: an excerptless post still offers the full post");
Post bodyless = p;
bodyless.body.clear();
const std::array<Post, 1> none{ bodyless };
const auto listNone = Views::RenderPosts(none);
Check(listNone.main.View().find("Read the full post") == std::string_view::npos,
"posts list: no read-more without a page to read");
Check(listNone.main.View().find(p.permalink) != std::string_view::npos,
"posts list: a pageless post still links its thread");
}
}
void RunJsonSelfTest() { void RunJsonSelfTest() {
using namespace Catcrafts::Json; using namespace Catcrafts::Json;
@ -810,6 +1198,9 @@ int main(int argc, char** argv) {
RunJsonSelfTest(); RunJsonSelfTest();
RunFormSelfTest(); RunFormSelfTest();
RunMoneySelfTest(); RunMoneySelfTest();
RunMediaSelfTest();
RunMarkdownSelfTest();
RunPostSelfTest();
if (failures == 0) { if (failures == 0) {
std::println("Catcrafts.Shared self-test: all assertions passed"); std::println("Catcrafts.Shared self-test: all assertions passed");
return 0; return 0;
@ -829,7 +1220,7 @@ int main(int argc, char** argv) {
const Views::RenderedPage page = Views::RenderRoute(route, content); const Views::RenderedPage page = Views::RenderRoute(route, content);
std::print("{}", Views::RenderDocument( std::print("{}", Views::RenderDocument(
page, page,
Views::RenderNav(route.kind == RouteKind::LegacyBlog ? RouteKind::Posts : route.kind), Views::RenderNav(NavKindFor(route.kind)),
Views::RenderFooter(), Views::RenderFooter(),
/*bootScripts=*/"", // no wasm on a plain server render /*bootScripts=*/"", // no wasm on a plain server render
/*cssHref=*/"/styles.css")); /*cssHref=*/"/styles.css"));
@ -859,6 +1250,14 @@ int main(int argc, char** argv) {
std::print(" <url><loc>https://catcrafts.net/shop/{}</loc></url>\n", std::print(" <url><loc>https://catcrafts.net/shop/{}</loc></url>\n",
Html::Escape(pr.slug).Str()); Html::Escape(pr.slug).Str());
} }
// Same rule as the served sitemap: only posts that actually have a
// page. Both must agree, because this is the copy baked into the wasm
// bundle and that one is what a crawler fetches.
for (const Post& po : content.posts) {
if (!po.HasPage()) continue;
std::print(" <url><loc>https://catcrafts.net/posts/{}</loc></url>\n",
Html::Escape(po.slug).Str());
}
std::print("</urlset>\n"); std::print("</urlset>\n");
return 0; return 0;
} }
@ -877,7 +1276,7 @@ int main(int argc, char** argv) {
"/order/not-a-token", "/order/not-a-token",
"/legal/privacy", "/legal/imprint", "/legal/privacy", "/legal/imprint",
"/legal/terms", "/legal/nope", "/legal/terms", "/legal/nope",
"/projects", "/posts", "/demos", "/projects", "/posts", "/posts/nope", "/demos",
"/demos/raytracer", "/demos/nope", "/demo", "/demos/raytracer", "/demos/nope", "/demo",
"/projects/", "/blog", "/blog/hello-world", "/nope" }) { "/projects/", "/blog", "/blog/hello-world", "/nope" }) {
const Route r = ParseRoute(p); const Route r = ParseRoute(p);

View file

@ -0,0 +1,588 @@
/*
catcrafts.net
Copyright (C) 2026 Catcrafts
The source code of this website is made available for viewing purposes only.
No permission is granted to copy, modify, distribute, or create derivative works.
*/
// A deliberately small Markdown renderer, for fediverse post bodies only.
//
// The site used to have no markdown pipeline at all, on the grounds that the
// posts page only ever showed a 280-character preview and the body stayed on
// the instance. Hosting the full body changes that calculation: a post IS
// prose with headings, quotes, code blocks and screenshots, and rendering it
// as one flat paragraph of literal `**asterisks**` would be worse than not
// hosting it. So: a parser, but only as much of one as these bodies use.
//
// WHAT IT SUPPORTS — everything observed in the real bodies, and nothing else:
//
// blocks ATX headings, fenced code, blockquotes (nested), ordered and
// unordered lists, thematic breaks, paragraphs
// inline links, images, code spans, ** strong **, * emphasis *, and bare
// URLs via Html::Autolink
//
// WHAT IT DELIBERATELY DOES NOT SUPPORT:
//
// * Raw HTML. Never. A post body is text fetched from someone else's server,
// so the ONE thing this renderer must guarantee is that no byte of it can
// become markup. Every character of body text leaves here through
// Html::Escape or Html::Autolink, and the only SafeHtml built from a raw
// string is the fixed structural markup written in this file. That is why
// `<` in a body renders as a less-than sign rather than opening a tag.
// * Underscore emphasis. `_` is common inside identifiers that appear in
// these posts unquoted (kworker/u16:8-qc_ufs_qos_swq), and mangling half a
// symbol name into italics is a worse failure than not italicising a word
// that used the underscore form. Asterisks are unambiguous here.
// * Setext headings, reference links, tables, footnotes, HTML entities.
// None appear; adding them speculatively is parser surface with no reader.
// * Trailing-double-space hard breaks. An invisible two-character difference
// is not something a reader can see in the source or a writer can rely on
// having typed; the lines of a paragraph join with a space, and a break
// that was meant is written as a blank line.
//
// Anything unrecognised degrades to text rather than being dropped, so a
// construct this parser does not know shows up as visibly odd prose instead of
// silently vanishing from the page.
export module Catcrafts.Shared:Markdown;
import std;
import :Html;
import :Media;
import :Model;
namespace Catcrafts::Markdown {
using Html::SafeHtml;
using Html::Escape;
using Html::Autolink;
using Html::Attr;
using Html::Format;
using Html::Join;
using Html::Raw;
using Html::Url;
// Blockquotes recurse, and a body is untrusted input, so the recursion needs a
// bound that does not depend on the input being sane. Four is past anything
// these posts do (a quote inside a list item inside a quote) and far short of
// anything that could trouble the stack.
constexpr int kMaxDepth = 4;
// ── small string helpers ──────────────────────────────────────────────
bool IsSpace(char c) { return c == ' ' || c == '\t'; }
std::string_view TrimRight(std::string_view s) {
while (!s.empty() && (IsSpace(s.back()) || s.back() == '\r')) s.remove_suffix(1);
return s;
}
std::string_view TrimLeft(std::string_view s) {
while (!s.empty() && IsSpace(s.front())) s.remove_prefix(1);
return s;
}
std::string_view Trim(std::string_view s) { return TrimLeft(TrimRight(s)); }
bool Blank(std::string_view line) { return Trim(line).empty(); }
// Up to three leading spaces are indentation a block marker is still allowed
// to carry; four or more would be a code block in real Markdown, which these
// bodies never use (they fence instead).
std::string_view Undent(std::string_view line) {
std::size_t n = 0;
while (n < line.size() && n < 3 && line[n] == ' ') ++n;
return line.substr(n);
}
// ── media ─────────────────────────────────────────────────────────────
// One embedded file, in the same shape (and CSS) the posts page uses for a
// post's headline media — an inline screenshot and a post's headline recording
// are the same kind of thing to a reader, so they should not look like two
// different components. :Media is what guarantees that: both go through it.
//
// A src with no mirrored record (its download failed, so the body still points
// at the original URL) still renders, just without dimensions or format tiers.
SafeHtml MediaTag(std::string_view src, std::string_view alt,
std::span<const PostMedia> media) {
return Media::Tag(Media::Describe(media, src), alt);
}
// ── inline ────────────────────────────────────────────────────────────
SafeHtml RenderInline(std::string_view text, std::span<const PostMedia> media, int depth);
// The span between `open` and its matching close delimiter, honouring nesting,
// or npos when it never closes. Used for [link text] and (link target), both of
// which can legitimately contain their own brackets — the Fairphone manual link
// in one of these posts has a parenthesised sentence as its text.
std::size_t MatchingDelimiter(std::string_view s, std::size_t from, char open, char close) {
int depth = 0;
for (std::size_t i = from; i < s.size(); ++i) {
if (s[i] == '\\') { ++i; continue; }
if (s[i] == open) { ++depth; continue; }
if (s[i] == close) {
if (depth == 0) return i;
--depth;
}
}
return std::string_view::npos;
}
// A run of ordinary prose, escaped and with bare URLs linked.
//
// Autolink rather than Escape because these bodies paste addresses constantly —
// mailing-list archives, merge requests, the shop — as bare text with no
// Markdown link syntax around them. Rendering those inert would strip most of
// the outbound value out of the post. Autolink escapes everything itself and
// puts each address through Html::Url, so this adds no new trusted path.
SafeHtml PlainRun(std::string_view text) {
return text.empty() ? SafeHtml{} : Autolink(text);
}
// True when the delimiter at `i` opens (rather than closes) an emphasis run:
// it must be followed by something that is not whitespace. Combined with the
// closing test below this is what keeps a lone asterisk — a multiplication
// sign, a footnote marker, a shell glob — from swallowing the rest of a
// paragraph into italics.
bool OpensEmphasis(std::string_view s, std::size_t after) {
return after < s.size() && !IsSpace(s[after]) && s[after] != '\n';
}
// The closing delimiter for an emphasis run opened at `from`, or npos. The
// character before it must not be whitespace, so "a * b * c" stays literal.
std::size_t FindEmphasisClose(std::string_view s, std::size_t from, std::string_view delim) {
std::size_t i = from;
while (i < s.size()) {
const std::size_t at = s.find(delim, i);
if (at == std::string_view::npos) return std::string_view::npos;
if (at > from && !IsSpace(s[at - 1])) return at;
i = at + delim.size();
}
return std::string_view::npos;
}
SafeHtml RenderInline(std::string_view text, std::span<const PostMedia> media, int depth) {
std::vector<SafeHtml> out;
std::size_t run = 0; // start of the pending plain-text run
auto flush = [&](std::size_t upto) {
if (upto > run) out.push_back(PlainRun(text.substr(run, upto - run)));
};
std::size_t i = 0;
while (i < text.size()) {
const char c = text[i];
// A backslash escape hides the next character from this parser. The
// pair is emitted as the second character alone, which is what lets a
// post write a literal asterisk.
if (c == '\\' && i + 1 < text.size()) {
flush(i);
out.push_back(Escape(text.substr(i + 1, 1)));
i += 2;
run = i;
continue;
}
// `code` — highest precedence, so an asterisk inside a code span is a
// literal asterisk and not an emphasis delimiter.
if (c == '`') {
const std::size_t close = text.find('`', i + 1);
if (close != std::string_view::npos) {
flush(i);
out.push_back(Format(R"(<code>{}</code>)",
Escape(text.substr(i + 1, close - i - 1))));
i = close + 1;
run = i;
continue;
}
}
// ![alt](src) — an image. Checked before the link case, since the
// bracket that follows would otherwise parse as one.
if (c == '!' && i + 1 < text.size() && text[i + 1] == '[' && depth < kMaxDepth) {
const std::size_t altEnd = MatchingDelimiter(text, i + 2, '[', ']');
if (altEnd != std::string_view::npos && altEnd + 1 < text.size()
&& text[altEnd + 1] == '(') {
const std::size_t srcEnd = MatchingDelimiter(text, altEnd + 2, '(', ')');
if (srcEnd != std::string_view::npos) {
flush(i);
out.push_back(MediaTag(Trim(text.substr(altEnd + 2, srcEnd - altEnd - 2)),
text.substr(i + 2, altEnd - i - 2),
media));
i = srcEnd + 1;
run = i;
continue;
}
}
}
// [text](href)
if (c == '[' && depth < kMaxDepth) {
const std::size_t textEnd = MatchingDelimiter(text, i + 1, '[', ']');
if (textEnd != std::string_view::npos && textEnd + 1 < text.size()
&& text[textEnd + 1] == '(') {
const std::size_t hrefEnd = MatchingDelimiter(text, textEnd + 2, '(', ')');
if (hrefEnd != std::string_view::npos) {
const std::string_view href =
Trim(text.substr(textEnd + 2, hrefEnd - textEnd - 2));
flush(i);
// The label is rendered rather than escaped flat, because
// these posts bold inside link text. Depth-guarded, so a
// link whose label contains a link cannot recurse forever.
out.push_back(Format(
R"(<a{} rel="noopener">{}</a>)",
Url("href", href),
RenderInline(text.substr(i + 1, textEnd - i - 1), media, depth + 1)));
i = hrefEnd + 1;
run = i;
continue;
}
}
}
// **strong** before *emphasis*: the longer delimiter has to win, or
// every bold run parses as an empty italic followed by loose text.
if (c == '*') {
const bool doubled = i + 1 < text.size() && text[i + 1] == '*';
const std::string_view delim = doubled ? "**" : "*";
const std::size_t inner = i + delim.size();
if (depth < kMaxDepth && OpensEmphasis(text, inner)) {
const std::size_t close = FindEmphasisClose(text, inner, delim);
if (close != std::string_view::npos) {
flush(i);
const SafeHtml body =
RenderInline(text.substr(inner, close - inner), media, depth + 1);
out.push_back(doubled ? Format(R"(<strong>{}</strong>)", body)
: Format(R"(<em>{}</em>)", body));
i = close + delim.size();
run = i;
continue;
}
}
}
++i;
}
flush(text.size());
return Join(out);
}
// ── blocks ────────────────────────────────────────────────────────────
// A line's leading run of '#', when it is an ATX heading marker.
// Returns 0 when the line is not a heading.
int HeadingLevel(std::string_view line) {
std::size_t n = 0;
while (n < line.size() && line[n] == '#') ++n;
if (n == 0 || n > 6) return 0;
// "#tag" is not a heading; a marker has to be followed by space or be the
// whole line.
if (n < line.size() && !IsSpace(line[n])) return 0;
return static_cast<int>(n);
}
bool IsFence(std::string_view line) { return TrimLeft(line).starts_with("```"); }
// `---`, `***`, `___`: three or more of one character, nothing else but spaces.
bool IsThematicBreak(std::string_view line) {
const std::string_view s = Trim(line);
if (s.size() < 3) return false;
const char c = s.front();
if (c != '-' && c != '*' && c != '_') return false;
int count = 0;
for (const char ch : s) {
if (ch == c) { ++count; continue; }
if (!IsSpace(ch)) return false;
}
return count >= 3;
}
struct ListMarker {
bool ok = false;
bool ordered = false;
std::int64_t start = 1; // the number an ordered item announced
std::size_t contentAt = 0; // offset of the item's text within the line
};
ListMarker ParseListMarker(std::string_view line) {
ListMarker m;
const std::string_view s = Undent(line);
const std::size_t indent = line.size() - s.size();
if (s.empty()) return m;
if ((s[0] == '-' || s[0] == '*' || s[0] == '+') && s.size() > 1 && IsSpace(s[1])) {
// A thematic break is also a run of dashes; it wins, because "- - -"
// is a rule everywhere and a three-item list nowhere.
if (IsThematicBreak(line)) return m;
m.ok = true;
m.contentAt = indent + 2;
return m;
}
std::size_t n = 0;
while (n < s.size() && s[n] >= '0' && s[n] <= '9') ++n;
// Bounded so a line starting with a long number is not mistaken for a list.
if (n == 0 || n > 9) return m;
if (n + 1 >= s.size() || (s[n] != '.' && s[n] != ')') || !IsSpace(s[n + 1])) return m;
std::int64_t value = 0;
std::from_chars(s.data(), s.data() + n, value);
m.ok = true;
m.ordered = true;
m.start = value;
m.contentAt = indent + n + 2;
return m;
}
bool StartsBlock(std::string_view line) {
return HeadingLevel(Undent(line)) > 0 || IsFence(line) || IsThematicBreak(line)
|| Undent(line).starts_with('>') || ParseListMarker(line).ok;
}
SafeHtml RenderBlocks(std::span<const std::string_view> lines,
std::span<const PostMedia> media, int depth);
// A paragraph's lines, joined and rendered.
//
// A paragraph whose entire content is embedded files becomes the same
// .post-media block the cards use rather than a <p> of images: that is what
// gives a run of screenshots the two-up grid instead of a column of full-width
// pictures with paragraph spacing between them.
SafeHtml RenderParagraph(std::span<const std::string_view> lines,
std::span<const PostMedia> media, int depth) {
// Lines join with a space: a body wraps its prose at whatever width the
// author's editor used, and those wraps are not meaningful.
std::string joined;
for (std::size_t i = 0; i < lines.size(); ++i) {
joined += Trim(lines[i]);
if (i + 1 < lines.size()) joined += ' ';
}
// Only-images test: strip every ![](...) span and see whether anything but
// whitespace is left.
bool onlyMedia = false;
{
std::string rest;
std::size_t i = 0;
std::size_t found = 0;
const std::string_view s = joined;
while (i < s.size()) {
if (s[i] == '!' && i + 1 < s.size() && s[i + 1] == '[') {
const std::size_t altEnd = MatchingDelimiter(s, i + 2, '[', ']');
if (altEnd != std::string_view::npos && altEnd + 1 < s.size()
&& s[altEnd + 1] == '(') {
const std::size_t srcEnd = MatchingDelimiter(s, altEnd + 2, '(', ')');
if (srcEnd != std::string_view::npos) {
++found;
i = srcEnd + 1;
continue;
}
}
}
rest += s[i];
++i;
}
onlyMedia = found > 0 && Trim(rest).empty();
}
const SafeHtml inner = RenderInline(joined, media, depth);
if (inner.Empty()) return SafeHtml{};
if (onlyMedia) return Format(R"(<div class="post-media">{}</div>)", inner);
return Format(R"(<p>{}</p>)", inner);
}
SafeHtml RenderBlocks(std::span<const std::string_view> lines,
std::span<const PostMedia> media, int depth) {
std::vector<SafeHtml> out;
std::size_t i = 0;
while (i < lines.size()) {
if (Blank(lines[i])) { ++i; continue; }
const std::string_view line = lines[i];
const std::string_view body = Undent(line);
// ── fenced code ───────────────────────────────────────────────
//
// Taken verbatim and escaped: the battery-measurement tables and the
// android top output in these posts are the one place where every
// space matters and nothing inside should be interpreted at all.
if (IsFence(line)) {
std::size_t j = i + 1;
std::string code;
while (j < lines.size() && !IsFence(lines[j])) {
code += TrimRight(lines[j]);
code += '\n';
++j;
}
out.push_back(Format(R"(<pre class="post-body__code"><code>{}</code></pre>)",
Escape(code)));
// Past the closing fence, or to the end when it never closed —
// an unterminated fence renders as code rather than swallowing
// the rest of the post into a parse failure.
i = (j < lines.size()) ? j + 1 : j;
continue;
}
// ── heading ───────────────────────────────────────────────────
//
// Demoted by one: the page's <h1> is the post title, so a body's own
// top-level heading is a section inside it. Without the shift every
// post would carry two h1s and the document outline would be wrong on
// exactly the pages this whole change exists to make indexable.
if (const int level = HeadingLevel(body); level > 0) {
std::string_view textPart = body.substr(static_cast<std::size_t>(level));
// A closing run of #s is decoration, not content.
textPart = TrimRight(textPart);
while (!textPart.empty() && textPart.back() == '#') textPart.remove_suffix(1);
const int tag = std::min(level + 1, 6);
out.push_back(Format("<h{}>{}</h{}>",
Html::Num(tag),
RenderInline(Trim(textPart), media, depth),
Html::Num(tag)));
++i;
continue;
}
if (IsThematicBreak(line)) {
out.push_back(Raw("<hr>"));
++i;
continue;
}
// ── blockquote ────────────────────────────────────────────────
//
// The quoted lines are re-parsed as blocks, so a quote keeps its own
// paragraphs and headings — which matters here, because the longest
// quotes in these posts are multi-paragraph company copy being taken
// apart line by line.
if (body.starts_with('>')) {
std::vector<std::string_view> quoted;
std::size_t j = i;
while (j < lines.size() && Undent(lines[j]).starts_with('>')) {
std::string_view q = Undent(lines[j]).substr(1);
if (!q.empty() && q.front() == ' ') q.remove_prefix(1);
quoted.push_back(q);
++j;
}
// At the depth limit the quote is still shown, just flattened —
// dropping it would lose content, which is the worse failure.
out.push_back(Format(
R"(<blockquote class="post-body__quote">{}</blockquote>)",
depth < kMaxDepth ? RenderBlocks(quoted, media, depth + 1)
: RenderParagraph(quoted, media, depth)));
i = j;
continue;
}
// ── list ──────────────────────────────────────────────────────
if (const ListMarker first = ParseListMarker(line); first.ok) {
std::vector<SafeHtml> items;
std::vector<std::string_view> current;
std::size_t j = i;
auto flushItem = [&] {
if (current.empty()) return;
items.push_back(Format(R"(<li>{}</li>)",
RenderInline([&] {
std::string joined;
for (std::size_t k = 0; k < current.size(); ++k) {
joined += Trim(current[k]);
if (k + 1 < current.size()) joined += ' ';
}
return joined;
}(), media, depth)));
current.clear();
};
while (j < lines.size()) {
if (Blank(lines[j])) {
// A blank line ends the list UNLESS the next non-blank line
// is another item of the same kind. These posts space their
// numbered steps apart, and treating that as seven separate
// one-item lists would restart the numbering at every gap.
std::size_t peek = j;
while (peek < lines.size() && Blank(lines[peek])) ++peek;
const ListMarker next =
peek < lines.size() ? ParseListMarker(lines[peek]) : ListMarker{};
if (!next.ok || next.ordered != first.ordered) break;
j = peek;
continue;
}
if (const ListMarker m = ParseListMarker(lines[j]); m.ok) {
if (m.ordered != first.ordered) break;
flushItem();
current.push_back(lines[j].substr(
std::min(m.contentAt, lines[j].size())));
++j;
continue;
}
// A non-marker line that would start some other block ends the
// list; anything else is this item's text continuing onto the
// next line.
if (StartsBlock(lines[j])) break;
current.push_back(lines[j]);
++j;
}
flushItem();
if (first.ordered) {
// start= only when it is not 1, so the common case stays clean
// markup — and so a list resumed after an interrupting
// paragraph continues its numbering instead of starting over.
out.push_back(Format(R"(<ol class="post-body__list"{}>{}</ol>)",
first.start == 1 ? SafeHtml{}
: Attr("start", std::to_string(first.start)),
Join(items)));
} else {
out.push_back(Format(R"(<ul class="post-body__list">{}</ul>)", Join(items)));
}
i = j;
continue;
}
// ── paragraph ─────────────────────────────────────────────────
{
std::size_t j = i;
while (j < lines.size() && !Blank(lines[j])) {
// A block marker on a later line interrupts the paragraph
// rather than being absorbed into it as text.
if (j > i && StartsBlock(lines[j])) break;
++j;
}
out.push_back(RenderParagraph(lines.subspan(i, j - i), media, depth));
i = j;
}
}
return Join(out);
}
// ── entry point ───────────────────────────────────────────────────────
// Render a post body to the inner markup of the article element.
//
// `media` is the post's mirrored inline files, used to attach dimensions (and
// a poster, and the H.264 fallback) to whatever the body embeds. Passing an
// empty span is fine: the markup then simply carries no dimensions, exactly
// like a file whose mirror failed.
export SafeHtml Render(std::string_view text, std::span<const PostMedia> media = {}) {
std::vector<std::string_view> lines;
std::size_t start = 0;
while (start <= text.size()) {
const std::size_t nl = text.find('\n', start);
if (nl == std::string_view::npos) {
lines.push_back(text.substr(start));
break;
}
lines.push_back(text.substr(start, nl - start));
start = nl + 1;
}
return RenderBlocks(lines, media, 0);
}
} // namespace Catcrafts::Markdown

View file

@ -0,0 +1,192 @@
/*
catcrafts.net
Copyright (C) 2026 Catcrafts
The source code of this website is made available for viewing purposes only.
No permission is granted to copy, modify, distribute, or create derivative works.
*/
// The markup for one mirrored file — image or video.
//
// This exists as its own partition because two callers need identical output:
// :Views renders a post's headline media on the card and at the top of the post
// page, and :Markdown renders whatever the body embeds. They were separate
// copies of the same element-building code, which is how the AV1 fallback logic
// came to be written twice; a format tier added to one and forgotten in the
// other shows up as the same picture served two different ways on two pages.
//
// FORMAT LADDER. Every file the mirror stores is served in the best form the
// browser will take, with a form every browser will take underneath it:
//
// video AV1 in MP4, falling back to H.264 in MP4. Two <source> elements;
// the codecs parameter on the first is what lets a browser without
// AV1 skip it rather than fail on a file it cannot decode.
// image AVIF, falling back to the mirrored original (usually WebP), falling
// back to PNG. A <picture>, so the browser fetches exactly one of
// them — the tiers cost nothing to the visitors who do not need them.
//
// The middle image tier is free: it is the file the mirror already downloaded,
// so it adds no encode and no disk. It matters because it is what stands
// between "no AVIF" and the PNG, and PNG is lossless — for a photograph that is
// an order of magnitude larger than the WebP beside it. With the middle tier
// the PNG is reached only by a browser that supports neither AVIF nor WebP.
//
// Any tier can be absent (the transcode was unavailable, or the file was never
// mirrored at all) and the markup degrades a step at a time, down to a bare
// <img src> — which is what the site emitted before any of this existed.
export module Catcrafts.Shared:Media;
import std;
import :Html;
import :Model;
namespace Catcrafts::Media {
using Html::SafeHtml;
using Html::Escape;
using Html::Attr;
using Html::Format;
using Html::Join;
using Html::Url;
bool EndsWithNoCase(std::string_view s, std::string_view suffix) {
if (s.size() < suffix.size()) return false;
const std::size_t off = s.size() - suffix.size();
for (std::size_t i = 0; i < suffix.size(); ++i) {
char a = s[off + i];
if (a >= 'A' && a <= 'Z') a = static_cast<char>(a - 'A' + 'a');
if (a != suffix[i]) return false;
}
return true;
}
// True when the path names something the <video> path should handle. Used only
// to guess for a file that was never mirrored, so there is no record to ask.
export bool LooksLikeVideo(std::string_view src) {
return EndsWithNoCase(src, ".mp4") || EndsWithNoCase(src, ".webm")
|| EndsWithNoCase(src, ".mov");
}
// The type= a <source> should advertise. Empty for anything not recognised, in
// which case the tier is dropped rather than guessed: a wrong type is worse
// than a missing source, because the browser believes it.
std::string_view MimeFor(std::string_view src) {
if (EndsWithNoCase(src, ".avif")) return "image/avif";
if (EndsWithNoCase(src, ".webp")) return "image/webp";
if (EndsWithNoCase(src, ".png")) return "image/png";
if (EndsWithNoCase(src, ".jpg") || EndsWithNoCase(src, ".jpeg")) return "image/jpeg";
if (EndsWithNoCase(src, ".gif")) return "image/gif";
return {};
}
// Width and height, or nothing when the mirror could not probe them. Both or
// neither: a lone dimension is worse than none, because the browser derives the
// other from it and gets the aspect ratio wrong.
SafeHtml Dimensions(const PostMedia& m) {
if (m.width <= 0 || m.height <= 0) return SafeHtml{};
return Format("{}{}", Attr("width", std::to_string(m.width)),
Attr("height", std::to_string(m.height)));
}
SafeHtml ImageTag(const PostMedia& m, std::string_view alt) {
// What every browser reads, and the only element that is always present.
// The PNG when there is one, because that is the tier nothing can refuse.
const std::string_view imgSrc = m.fallback.empty() ? m.src : m.fallback;
std::vector<SafeHtml> sources;
if (!m.avif.empty()) {
sources.push_back(Format(R"(<source{} type="image/avif">)", Url("srcset", m.avif)));
}
// The mirrored original, sitting between the AVIF and the PNG. Skipped when
// it IS what the <img> points at or what the AVIF tier already offered
// (nothing left to say), or when its type cannot be named.
if (m.src != imgSrc && m.src != m.avif) {
if (const std::string_view mime = MimeFor(m.src); !mime.empty()) {
sources.push_back(Format(R"(<source{}{}>)",
Url("srcset", m.src), Attr("type", mime)));
}
}
// alt is whatever the author wrote, which for a post's headline media is
// nothing: those are screenshots whose meaning is already in the title and
// the excerpt, and inventing descriptive alt text here would be making up
// what the picture shows. Emitted explicitly even when empty — alt="" is
// skipped cleanly by a screen reader, a missing alt makes it read the file
// name out loud.
const SafeHtml img = Format(
R"(<img class="post-media__item" loading="lazy" decoding="async" alt="{}"{}{}>)",
Escape(alt), Url("src", imgSrc), Dimensions(m));
if (sources.empty()) return img;
return Format("<picture>{}{}</picture>", Join(sources), img);
}
SafeHtml VideoTag(const PostMedia& m) {
// preload="metadata", not "auto": a page with several 5 MB recordings must
// not pull them all on load.
const SafeHtml poster = m.poster.empty() ? SafeHtml{} : Url("poster", m.poster);
const SafeHtml dims = Dimensions(m);
if (m.fallback.empty() || m.fallback == m.src) {
return Format(
R"(<video class="post-media__item" controls preload="metadata" playsinline{}{}{}></video>)",
Url("src", m.src), poster, dims);
}
// An AV1 video with its H.264 rendition. Both are .mp4, so the container
// alone cannot tell them apart: the codecs parameter on the first <source>
// is what lets a browser without AV1 (Safari before 17, Apple hardware
// without the decoder) skip it and take the H.264 instead of failing on a
// file it cannot decode. The string is advisory and used only for
// selection — once a source is picked the browser reads the actual stream —
// so the canonical profile-0 8-bit form is right for anything
// publish-media.sh emits (yuv420p is pinned there).
return Format(
R"(<video class="post-media__item" controls preload="metadata" playsinline{}{}>)"
R"(<source{} type="video/mp4; codecs=av01.0.08M.08">)"
R"(<source{} type="video/mp4">)"
R"(</video>)",
poster, dims, Url("src", m.src), Url("src", m.fallback));
}
// The element for one mirrored file.
export SafeHtml Tag(const PostMedia& m, std::string_view alt = {}) {
return m.kind == "video" ? VideoTag(m) : ImageTag(m, alt);
}
// The mirrored record for `src`, or nullptr when this path was never mirrored.
//
// Matching is on the exact string because tools/fetch-media.sh rewrites the
// body text and the body_media list from the same mapping in the same pass —
// they cannot disagree about a path without the mirror step itself being wrong.
export const PostMedia* Find(std::span<const PostMedia> media, std::string_view src) {
for (const PostMedia& m : media) {
if (m.src == src) return &m;
}
return nullptr;
}
// The record for `src`, or one invented from the path alone.
//
// The invented case is a file whose download failed, so the body still points
// at its original URL: there are no dimensions and no renditions to offer, and
// the markup degrades to the bare element. Rendering something beats dropping
// the picture the paragraph is talking about.
export PostMedia Describe(std::span<const PostMedia> media, std::string_view src) {
if (const PostMedia* hit = Find(media, src)) return *hit;
PostMedia m;
m.src = std::string(src);
m.kind = LooksLikeVideo(src) ? "video" : "image";
return m;
}
// A post's media as one block. More than one file gets a two-up grid via CSS,
// so a post with four screenshots does not become a mile of scrolling.
export SafeHtml Block(std::span<const PostMedia> media) {
if (media.empty()) return SafeHtml{};
std::vector<SafeHtml> items;
items.reserve(media.size());
for (const PostMedia& m : media) items.push_back(Tag(m));
return Format(R"(<div class="post-media">{}</div>)", Join(items));
}
} // namespace Catcrafts::Media

View file

@ -30,6 +30,11 @@ No permission is granted to copy, modify, distribute, or create derivative works
export module Catcrafts.Shared:Model; export module Catcrafts.Shared:Model;
import std; import std;
import :Json; import :Json;
// For IsValidSlug. A post slug becomes a URL, so it is checked here on the way
// in rather than trusted because CI wrote it: an unparseable slug would render
// a "read more" link to a route that can never match, which is a 404 the site
// links to itself. Dropping it instead degrades to a post with no page.
import :Route;
namespace Catcrafts { namespace Catcrafts {
@ -50,34 +55,69 @@ export struct PostMedia {
// play — and these posts are their video, so the black box is the page. // play — and these posts are their video, so the black box is the page.
// Empty for images, and empty when the instance generated no thumbnail. // Empty for images, and empty when the instance generated no thumbnail.
std::string poster; std::string poster;
// H.264 rendition of an AV1 video, when tools/publish-media.sh uploaded one // The rendition every browser can take, and the bottom of the format
// beside it. Non-empty means the renderer emits a <source> pair instead of // ladder :Media builds. Empty means there is only `src`.
// a bare src, so a browser without AV1 (Safari before 17, Apple hardware //
// without the decoder) gets a file it can play. Empty for images and for // video H.264 in MP4, uploaded beside the AV1 by
// videos that need no fallback. // tools/publish-media.sh. Without it a browser lacking AV1
// (Safari before 17, Apple hardware without the decoder) has an
// element it cannot play.
// image PNG, transcoded by tools/fetch-media.sh. It is what the <img>
// itself points at, so a browser that understands no <source>
// type at all still gets the picture.
std::string fallback; std::string fallback;
// AVIF rendition of an image, transcoded by tools/fetch-media.sh. Offered
// above `src` — it is consistently smaller than the WebP the instances
// serve, at 4:4:4 chroma so coloured text in a screenshot does not fringe.
// Empty for videos, and for an image the transcode could not produce.
std::string avif;
std::int64_t width = 0; std::int64_t width = 0;
std::int64_t height = 0; std::int64_t height = 0;
}; };
// A post mirrored from the fediverse (Lemmy). Deliberately not the full post: // A post mirrored from the fediverse (Lemmy).
// the body stays on the community's instance, where the comments are. We show //
// enough to be worth clicking and then hand off — no crawling, no comment // The body IS mirrored — the comments are not. That split is the whole policy:
// mirroring, no markdown pipeline. // the writing is the work and belongs on the site that is about the work (and
// is the only version a search engine can be pointed at as canonical), while
// the discussion belongs to the community that is having it. So every post
// page carries the full text and then hands off to the thread. No crawling, no
// comment mirroring, no runtime dependency on the instance.
// //
// `permalink` is resolved at fetch time to the COMMUNITY's instance, not the // `permalink` is resolved at fetch time to the COMMUNITY's instance, not the
// author's. Both host a federated copy; the community's is where the discussion // author's. Both host a federated copy; the community's is where the discussion
// actually is, and it is what the site should be pointing readers at. // actually is, and it is what the site should be pointing readers at.
export struct Post { export struct Post {
std::string title; std::string title;
// URL-safe name of this post's own page at /posts/<slug>, derived from the
// title by tools/fetch-posts.sh. Empty only if the title reduced to nothing
// a slug can be made of, which is what HasPage() below tests for.
std::string slug;
std::string permalink; // canonical ap_id on the instance; where "discuss" goes std::string permalink; // canonical ap_id on the instance; where "discuss" goes
std::string linkUrl; // for link posts, the linked target; empty otherwise std::string linkUrl; // for link posts, the linked target; empty otherwise
std::string community; // e.g. "linux@lemmy.ml" std::string community; // e.g. "linux@lemmy.ml"
std::string published; // ISO-8601, as emitted by the API std::string published; // ISO-8601, as emitted by the API
std::string excerpt; // plain text, truncated by CI — never markdown std::string excerpt; // plain text, truncated by CI — never markdown
// The full post, still as Markdown. Rendered by :Markdown at page-render
// time rather than converted to HTML at fetch time, because the renderer is
// where this codebase's escaping guarantee lives — text that arrives from
// someone else's server must not become markup anywhere but there.
std::string body;
std::int64_t score = 0; std::int64_t score = 0;
std::int64_t comments = 0; std::int64_t comments = 0;
// The post's headline media — `post.url` upstream, the recording or
// screenshot the post is *about*. Shown on the card and above the body.
std::vector<PostMedia> media; std::vector<PostMedia> media;
// Files embedded inside the body, mirrored by the same pass. The body text
// already points at these local paths; this list exists so the renderer can
// attach dimensions (and a poster, and the H.264 fallback) to them, which
// Markdown syntax has nowhere to carry.
std::vector<PostMedia> bodyMedia;
// Whether this post gets its own page. A post with no body has nothing to
// put on one — a page consisting of a title and a link out is thin content
// that should not be minted as a URL, indexed, or linked to as "read more".
bool HasPage() const { return !slug.empty() && !body.empty(); }
}; };
// An entry on the projects page. Repo content, not a database — these change // An entry on the projects page. Repo content, not a database — these change
@ -294,6 +334,32 @@ export struct LegalPage {
// ── loaders ─────────────────────────────────────────────────────────── // ── loaders ───────────────────────────────────────────────────────────
// The media array shape, shared by a post's headline media and its body media —
// they are the same records produced by the same mirror pass, so they are read
// by one function rather than two that could drift.
std::vector<PostMedia> LoadPostMedia(const Json::Value* array) {
std::vector<PostMedia> out;
if (!array || !array->IsArray()) return out;
for (const Json::Value& mv : array->array) {
if (!mv.IsObject()) continue;
PostMedia pm;
pm.src = std::string(mv.Str("src"));
pm.kind = std::string(mv.Str("kind", "image"));
pm.poster = std::string(mv.Str("poster"));
pm.fallback = std::string(mv.Str("fallback"));
pm.avif = std::string(mv.Str("avif"));
pm.width = mv.Int("w");
pm.height = mv.Int("h");
// Only the two kinds the renderer knows how to emit. Anything else
// would fall through to an <img> for a file that is not an image, so
// treat it as image only when it says so.
if (pm.kind != "image" && pm.kind != "video") pm.kind = "image";
if (pm.src.empty()) continue;
out.push_back(std::move(pm));
}
return out;
}
export std::vector<Post> LoadPosts(std::string_view json) { export std::vector<Post> LoadPosts(std::string_view json) {
std::vector<Post> out; std::vector<Post> out;
auto doc = Json::Parse(json); auto doc = Json::Parse(json);
@ -303,31 +369,19 @@ export std::vector<Post> LoadPosts(std::string_view json) {
if (!item.IsObject()) continue; if (!item.IsObject()) continue;
Post p; Post p;
p.title = std::string(item.Str("title")); p.title = std::string(item.Str("title"));
if (const std::string_view slug = item.Str("slug"); IsValidSlug(slug)) {
p.slug = std::string(slug);
}
p.permalink = std::string(item.Str("permalink")); p.permalink = std::string(item.Str("permalink"));
p.linkUrl = std::string(item.Str("url")); p.linkUrl = std::string(item.Str("url"));
p.community = std::string(item.Str("community")); p.community = std::string(item.Str("community"));
p.published = std::string(item.Str("published")); p.published = std::string(item.Str("published"));
p.excerpt = std::string(item.Str("excerpt")); p.excerpt = std::string(item.Str("excerpt"));
p.body = std::string(item.Str("body"));
p.score = item.Int("score"); p.score = item.Int("score");
p.comments = item.Int("comments"); p.comments = item.Int("comments");
if (const Json::Value* m = item.Find("media"); m && m->IsArray()) { p.media = LoadPostMedia(item.Find("media"));
for (const Json::Value& mv : m->array) { p.bodyMedia = LoadPostMedia(item.Find("body_media"));
if (!mv.IsObject()) continue;
PostMedia pm;
pm.src = std::string(mv.Str("src"));
pm.kind = std::string(mv.Str("kind", "image"));
pm.poster = std::string(mv.Str("poster"));
pm.fallback = std::string(mv.Str("fallback"));
pm.width = mv.Int("w");
pm.height = mv.Int("h");
// Only the two kinds the renderer knows how to emit. Anything
// else would fall through to an <img> for a file that is not an
// image, so treat it as image only when it says so.
if (pm.kind != "image" && pm.kind != "video") pm.kind = "image";
if (pm.src.empty()) continue;
p.media.push_back(std::move(pm));
}
}
// A post with no title and nowhere to click is not renderable; drop it // A post with no title and nowhere to click is not renderable; drop it
// rather than emit an empty card. // rather than emit an empty card.
if (p.title.empty() && p.permalink.empty()) continue; if (p.title.empty() && p.permalink.empty()) continue;

View file

@ -23,6 +23,7 @@ export enum class RouteKind {
About, // /about — the person behind the company About, // /about — the person behind the company
Projects, Projects,
Posts, Posts,
Post, // /posts/<slug> — one post, in full
Demos, // /demos — the list Demos, // /demos — the list
Demo, // /demos/<slug> Demo, // /demos/<slug>
Shop, // /shop — the (currently single-item) product list Shop, // /shop — the (currently single-item) product list
@ -45,7 +46,8 @@ export struct Route {
std::string query; // raw, including leading '?', or empty std::string query; // raw, including leading '?', or empty
// Non-empty when the request should be canonicalised to a different URL. // Non-empty when the request should be canonicalised to a different URL.
std::string canonicalRedirect; std::string canonicalRedirect;
// The <slug> of /shop/<slug>, empty for every other route. // The <slug> of /shop/<slug>, /demos/<slug>, /legal/<slug> or /posts/<slug>
// (and the token of /order/<token>); empty for every other route.
std::string slug; std::string slug;
}; };
@ -134,6 +136,20 @@ export Route ParseRoute(std::string_view path, std::string_view query = {}) {
return r; return r;
} }
// /posts/<slug>. Whether a slug names a post the site actually has is the
// dispatcher's question, exactly as it is for /shop/<slug>; this only
// decides that the URL is shaped like a post page at all.
if (p.starts_with("/posts/")) {
const std::string_view slug = p.substr(7);
if (IsValidSlug(slug)) {
r.kind = RouteKind::Post;
r.slug = std::string(slug);
return r;
}
r.kind = RouteKind::NotFound;
return r;
}
if (p.starts_with("/demos/")) { if (p.starts_with("/demos/")) {
const std::string_view slug = p.substr(7); const std::string_view slug = p.substr(7);
if (IsValidSlug(slug)) { if (IsValidSlug(slug)) {
@ -183,6 +199,20 @@ export struct NavItem {
RouteKind kind; RouteKind kind;
}; };
// Which nav entry a route highlights. A route that is not itself a nav entry
// borrows its section's: an individual post and the retired /blog URL both sit
// under Posts. Shared so the three call sites (the server, the CLI renderer and
// the wasm router) cannot disagree about where the reader is — they each used
// to spell the /blog case out, and the third one to be added would have been
// the third chance to forget it.
export RouteKind NavKindFor(RouteKind kind) {
switch (kind) {
case RouteKind::Post:
case RouteKind::LegacyBlog: return RouteKind::Posts;
default: return kind;
}
}
export std::span<const NavItem> NavItems() { export std::span<const NavItem> NavItems() {
static constexpr std::array<NavItem, 6> items{{ static constexpr std::array<NavItem, 6> items{{
{ "Home", "/", RouteKind::Home }, { "Home", "/", RouteKind::Home },

View file

@ -23,6 +23,8 @@ import std;
import :Content; import :Content;
import :Html; import :Html;
import :Form; import :Form;
import :Markdown;
import :Media;
import :Model; import :Model;
import :Money; import :Money;
import :Route; import :Route;
@ -189,7 +191,11 @@ export RenderedPage RenderHome(std::span<const Project> projects,
const Post& post = posts[i]; const Post& post = posts[i];
recent.push_back(Format( recent.push_back(Format(
R"(<li class="recent__item"><a{}>{}</a><span class="recent__date">{}</span></li>)", R"(<li class="recent__item"><a{}>{}</a><span class="recent__date">{}</span></li>)",
Url("href", post.permalink), // The on-site page when the post has one, the thread otherwise.
// Same rule as the cards on /posts: a link from the home page is
// worth more pointing at a page this site owns than at someone
// else's copy of it, and the post page links onward to the thread.
Url("href", post.HasPage() ? "/posts/" + post.slug : post.permalink),
Escape(post.title), Escape(post.title),
Escape(DateOnly(post.published)))); Escape(DateOnly(post.published))));
} }
@ -341,65 +347,62 @@ export RenderedPage RenderProjects(std::span<const Project> projects) {
// keeps the privacy notice's "everything comes from catcrafts.net" true and // keeps the privacy notice's "everything comes from catcrafts.net" true and
// stops every visitor's IP reaching whichever instance hosted the file. // stops every visitor's IP reaching whichever instance hosted the file.
// //
// Video is preload="metadata", not "auto": a page with several 5 MB recordings // The element itself — the format ladder, the dimensions, the poster — is
// must not pull them all on load. Dimensions come from the content file so the // :Media's job, shared with the body renderer so a post's headline screenshot
// browser reserves the right box and nothing jumps as files arrive. // and one embedded in its prose cannot be served differently.
SafeHtml RenderPostMedia(std::span<const PostMedia> media) { SafeHtml RenderPostMedia(std::span<const PostMedia> media) {
if (media.empty()) return SafeHtml{}; return Media::Block(media);
std::vector<SafeHtml> items;
for (const PostMedia& m : media) {
SafeHtml dims = (m.width > 0 && m.height > 0)
? Format("{}{}", Attr("width", std::to_string(m.width)),
Attr("height", std::to_string(m.height)))
: SafeHtml{};
if (m.kind == "video") {
SafeHtml poster = m.poster.empty() ? SafeHtml{} : Url("poster", m.poster);
if (m.fallback.empty() || m.fallback == m.src) {
items.push_back(Format(
R"(<video class="post-media__item" controls preload="metadata" )"
R"(playsinline{}{}{}></video>)",
Url("src", m.src), poster, dims));
} else {
// An AV1 video with its H.264 rendition. Both are .mp4, so the
// container alone cannot tell them apart: the codecs parameter
// on the first <source> is what lets a browser without AV1
// (Safari before 17, Apple hardware without the decoder) skip
// it and take the H.264 instead of failing on a file it cannot
// decode. The string is advisory and used only for selection —
// once a source is picked the browser reads the actual stream —
// so the canonical profile-0 8-bit form is right for anything
// publish-media.sh emits (yuv420p is pinned there).
items.push_back(Format(
R"(<video class="post-media__item" controls preload="metadata" )"
R"(playsinline{}{}>)"
R"(<source{} type="video/mp4; codecs=av01.0.08M.08">)"
R"(<source{} type="video/mp4">)"
R"(</video>)",
poster, dims, Url("src", m.src), Url("src", m.fallback)));
} }
} else {
// alt is empty and aria-hidden is absent on purpose: these are // A link post gets a second, clearly-labelled outbound link. Without the label
// screenshots whose meaning is already in the post title and // the two links are indistinguishable and one of them silently leaves the site.
// excerpt, and inventing descriptive alt text here would be making SafeHtml RenderPostLinkRow(const Post& p) {
// up what the picture shows. if (p.linkUrl.empty()) return SafeHtml{};
items.push_back(Format( return Format(
R"(<img class="post-media__item" loading="lazy" decoding="async" )" R"(<p class="post-card__link"><a{} rel="nofollow noopener">{}</a></p>)",
R"(alt=""{}{}>)", Url("href", p.linkUrl), Escape(p.linkUrl));
Url("src", m.src), dims));
} }
}
return Format(R"(<div class="post-media">{}</div>)", Join(items)); // Points and the thread link, identical on a card and at the foot of a post
// page — the reader is being offered the same two things in both places, and
// they should not look like two different components.
SafeHtml RenderPostStats(const Post& p) {
return Format(
R"(<footer class="post-card__footer">)"
R"(<span class="stat">{} points</span>)"
R"(<a class="stat stat--link"{} rel="noopener">Discuss on the fediverse ({} comments) &rarr;</a>)"
R"(</footer>)",
Num(p.score), Url("href", p.permalink), Num(p.comments));
} }
export RenderedPage RenderPosts(std::span<const Post> posts) { export RenderedPage RenderPosts(std::span<const Post> posts) {
std::vector<SafeHtml> cards; std::vector<SafeHtml> cards;
for (const Post& p : posts) { for (const Post& p : posts) {
// A link post gets a second, clearly-labelled outbound link. Without // The title goes to this site's copy when there is one. Before post
// the label the two links are indistinguishable and one of them // pages existed it could only go to the thread, which meant the list
// silently leaves the site. // of everything this site is about was a list of links off it.
SafeHtml linkRow = p.linkUrl.empty() ? SafeHtml{} : Format( const std::string titleHref = p.HasPage() ? "/posts/" + p.slug : p.permalink;
R"(<p class="post-card__link"><a{} rel="nofollow noopener">{}</a></p>)",
Url("href", p.linkUrl), Escape(p.linkUrl)); // No "read more" without a page to read more on — a post whose body
// never arrived would otherwise offer a link to its own 404.
//
// It goes inline at the end of the excerpt, immediately after the
// ellipsis the truncation left: that is where the sentence stops and
// where the reader is already looking for the rest of it. Below the
// media it was the same offer made a screen further down, after the
// reader had already decided.
const SafeHtml more = !p.HasPage() ? SafeHtml{} : Format(
R"( <a class="link-more"{}>Read the full post</a>)",
Url("href", "/posts/" + p.slug));
// With no excerpt there is no sentence to continue, so the link falls
// back to a row of its own rather than disappearing along with it.
const SafeHtml excerptRow =
!p.excerpt.empty()
? Format(R"(<p class="post-card__excerpt">{}{}</p>)", Escape(p.excerpt), more)
: more.Empty()
? SafeHtml{}
: Format(R"(<p class="post-card__more">{}</p>)", more);
cards.push_back(Format( cards.push_back(Format(
R"(<article class="post-card">)" R"(<article class="post-card">)"
@ -412,32 +415,28 @@ export RenderedPage RenderPosts(std::span<const Post> posts) {
R"({})" R"({})"
R"({})" R"({})"
R"({})" R"({})"
R"(<footer class="post-card__footer">)" R"({})"
R"(<span class="stat">{} points</span>)"
R"(<a class="stat stat--link"{} rel="noopener">Discuss on the fediverse ({} comments) &rarr;</a>)"
R"(</footer>)"
R"(</article>)", R"(</article>)",
Url("href", p.permalink), Escape(p.title), Url("href", titleHref), Escape(p.title),
Attr("datetime", p.published), Escape(DateOnly(p.published)), Attr("datetime", p.published), Escape(DateOnly(p.published)),
Escape(p.community), Escape(p.community),
p.excerpt.empty() ? SafeHtml{} excerptRow,
: Format(R"(<p class="post-card__excerpt">{}</p>)", Escape(p.excerpt)),
RenderPostMedia(p.media), RenderPostMedia(p.media),
linkRow, RenderPostLinkRow(p),
Num(p.score), RenderPostStats(p)));
Url("href", p.permalink), Num(p.comments)));
} }
RenderedPage page; RenderedPage page;
page.meta.title = "Posts — Catcrafts"; page.meta.title = "Posts — Catcrafts";
page.meta.description = "Posts from the fediverse; discussion happens there."; page.meta.description = "Posts about mobile Linux, kernel work and open hardware, "
"written for the fediverse and archived here in full.";
page.meta.canonical = "/posts"; page.meta.canonical = "/posts";
page.main = Format( page.main = Format(
R"(<header class="page-header">)" R"(<header class="page-header">)"
R"(<h1 class="page-header__title">Posts</h1>)" R"(<h1 class="page-header__title">Posts</h1>)"
R"(<p class="page-header__lede">Catcrafts posts on the fediverse rather than keeping a blog here. )" R"(<p class="page-header__lede">Catcrafts posts on the fediverse rather than keeping a blog here. )"
R"(Each of these links to the original thread on whichever instance it lives on. )" R"(Each one is kept here in full, and links to the original thread on whichever )"
R"(Follow one to read it and join the discussion there.</p>)" R"(instance it lives on — that is where the discussion is.</p>)"
R"(</header>)" R"(</header>)"
R"(<div class="post-list">{}</div>)", R"(<div class="post-list">{}</div>)",
cards.empty() cards.empty()
@ -446,6 +445,88 @@ export RenderedPage RenderPosts(std::span<const Post> posts) {
return page; return page;
} }
// ── one post, in full ─────────────────────────────────────────────────
// The whole point of hosting the body: a page a search engine can index, a
// reader can link to, and this site can claim as canonical. The card on /posts
// is a summary of this; the thread on the instance is where the comments are.
//
// The body arrives as Markdown and is rendered HERE rather than converted at
// fetch time, because :Markdown is inside the escaping guarantee and a shell
// script writing HTML into a content file would not be. See that module for
// what it does and does not accept.
export RenderedPage RenderPost(const Post& p) {
// Whatever the post leads with, for the link-preview card. A video has no
// still of its own to offer, so its poster frame stands in. Only our own
// mirrored copies qualify: an og:image on someone else's instance is the
// same third-party fetch the mirror exists to avoid, just performed by a
// crawler instead of a reader.
std::string ogImage;
for (const PostMedia& m : p.media) {
const std::string& candidate = m.kind == "video" ? m.poster : m.src;
if (candidate.starts_with("/")) { ogImage = candidate; break; }
}
const std::string canonical = "/posts/" + p.slug;
RenderedPage page;
page.meta.title = p.title + " — Catcrafts";
page.meta.description = p.excerpt;
page.meta.canonical = canonical;
page.meta.ogType = "article";
page.meta.ogImage = ogImage;
// BlogPosting, joined to the same two nodes every other page names: the
// Person on /about is the author and the Organization is the publisher.
// Without those @ids each post would introduce a fourth unrelated
// "Catcrafts" to a search index that already has trouble telling this one
// from the name-twins — see RenderHome for the whole argument.
//
// discussionUrl is the honest way to say what the fediverse link is: the
// comments belong to the thread, and this page is not pretending to mirror
// them.
page.meta.jsonLd = std::format(
R"({{"@context":"https://schema.org","@type":"BlogPosting",)"
R"("headline":{},"datePublished":{},"url":{},)"
R"("mainEntityOfPage":{{"@type":"WebPage","@id":{}}},)"
R"("author":{{"@id":"https://catcrafts.net/about#person",)"
R"("@type":"Person","name":"Jorijn van der Graaf"}},)"
R"("publisher":{{"@id":"https://catcrafts.net/#organization",)"
R"("@type":"Organization","name":"Catcrafts"}},)"
R"("discussionUrl":{}{}{}}})",
JsonStr(p.title), JsonStr(p.published),
JsonStr("https://catcrafts.net" + canonical),
JsonStr("https://catcrafts.net" + canonical),
JsonStr(p.permalink),
p.excerpt.empty() ? std::string{} : ",\"description\":" + JsonStr(p.excerpt),
ogImage.empty() ? std::string{}
: ",\"image\":" + JsonStr("https://catcrafts.net" + ogImage));
page.main = Format(
R"(<article class="post">)"
R"(<header class="page-header">)"
R"(<h1 class="page-header__title">{}</h1>)"
R"(<p class="post-card__meta">)"
R"(<time{}>{}</time><span class="post-card__community">{}</span>)"
R"(</p>)"
R"(</header>)"
R"({})"
R"({})"
R"(<div class="post-body">{}</div>)"
R"({})"
R"(<p class="post__back"><a class="link-more"{}>All posts</a></p>)"
R"(</article>)",
Escape(p.title),
Attr("datetime", p.published), Escape(DateOnly(p.published)),
Escape(p.community),
RenderPostMedia(p.media),
RenderPostLinkRow(p),
Markdown::Render(p.body, p.bodyMedia),
RenderPostStats(p),
Url("href", "/posts"));
return page;
}
// ── shop ────────────────────────────────────────────────────────────── // ── shop ──────────────────────────────────────────────────────────────
// The product page's price block: the same single headline number as the shop // The product page's price block: the same single headline number as the shop
@ -1329,6 +1410,16 @@ export struct SiteContent {
} }
return nullptr; return nullptr;
} }
// Only posts that have a page. A post with no body has a slug but nothing
// to show, so finding it here would mint an indexable URL for a title and
// a link — see Post::HasPage.
const Post* FindPost(std::string_view slug) const {
for (const Post& p : posts) {
if (p.HasPage() && p.slug == slug) return &p;
}
return nullptr;
}
}; };
RenderedPage RenderRouteBody(const Route& route, const SiteContent& content); RenderedPage RenderRouteBody(const Route& route, const SiteContent& content);
@ -1357,6 +1448,13 @@ RenderedPage RenderRouteBody(const Route& route, const SiteContent& content) {
case RouteKind::About: return RenderAbout(Content::AboutPage()); case RouteKind::About: return RenderAbout(Content::AboutPage());
case RouteKind::Projects: return RenderProjects(content.projects); case RouteKind::Projects: return RenderProjects(content.projects);
case RouteKind::Posts: return RenderPosts(content.posts); case RouteKind::Posts: return RenderPosts(content.posts);
case RouteKind::Post: {
// A slug that parsed but names no post is a 404, for the same
// reason an unknown product slug is: otherwise every typo and
// every retired post becomes an indexable empty page.
if (const Post* p = content.FindPost(route.slug)) return RenderPost(*p);
break;
}
case RouteKind::Demos: return RenderDemos(content.demos); case RouteKind::Demos: return RenderDemos(content.demos);
case RouteKind::Demo: { case RouteKind::Demo: {
if (const Demo* d = content.FindDemo(route.slug)) return RenderDemo(*d); if (const Demo* d = content.FindDemo(route.slug)) return RenderDemo(*d);

View file

@ -35,6 +35,8 @@ export import :Html;
export import :Json; export import :Json;
export import :Form; export import :Form;
export import :Model; export import :Model;
export import :Media;
export import :Markdown;
export import :Content; export import :Content;
export import :Money; export import :Money;
export import :Route; export import :Route;

View file

@ -487,6 +487,96 @@ treatment it replaces, which read as the wrong category for the work.
overflow-wrap: anywhere; overflow-wrap: anywhere;
} }
/* Only reached by a post with no excerpt to trail normally the read-more
link rides at the end of .post-card__excerpt instead. */
.post-card__more { margin-top: var(--s-1); }
/* Trailing an excerpt, the link is part of that paragraph and wraps with it,
but it must not break across lines itself: "Read the full / post →" reads
as two different things. */
.post-card__excerpt .link-more { white-space: nowrap; }
/* one post, in full
The body is prose rendered from Markdown by Catcrafts.Shared:Markdown,
so the element set here is exactly what that module can emit and no
more. Everything is capped at --measure: a post page is reading
material, and full-viewport line lengths are not read, they are
skimmed. Media is the exception it keeps the full column, because a
screenshot of a feature-support table is unreadable at text width. */
.post-body {
display: grid;
gap: var(--s0);
margin-top: var(--s1);
}
/* The measure goes on the prose, not on the container, so media can use the
full column: a post page is reading material and full-viewport line
lengths get skimmed rather than read, but a screenshot of a feature
support table is unreadable at text width. */
.post-body > :not(.post-media) { max-width: var(--measure); }
/* .post-media carries its own top margin for the cards, where it follows an
excerpt; inside this grid the gap already provides it. */
.post-body > .post-media { margin-top: 0; }
/* Grid items default to min-width:auto, which refuses to shrink below their
content. Without this the overflow-x on a code block below never engages
the track widens instead and the whole PAGE scrolls sideways, which is the
one thing wide content must never cause. */
.post-body > *, .post-body__quote > * { min-width: 0; }
.post-body h2 { font-size: var(--step-2); letter-spacing: -0.01em; }
.post-body h3 { font-size: var(--step-1); }
.post-body h4, .post-body h5, .post-body h6 { font-size: var(--step-0); }
/* Extra air above a heading, none below: a heading belongs to what
follows it, and the grid gap alone reads as floating between two
equally distant paragraphs. */
.post-body :is(h2, h3, h4, h5, h6) { margin-top: var(--s1); }
.post-body :is(h2, h3, h4, h5, h6):first-child { margin-top: 0; }
.post-body a { overflow-wrap: anywhere; }
.post-body__list { padding-left: var(--s1); display: grid; gap: var(--s-2); }
.post-body__list li { padding-left: var(--s-3); }
.post-body__quote {
padding: var(--s-2) var(--s0);
border-left: 2px solid var(--border-strong);
background: var(--surface-1);
border-radius: 0 var(--radius-sm) var(--radius-sm) 0;
color: var(--text-muted);
display: grid;
gap: var(--s-1);
}
.post-body__code {
padding: var(--s-1) var(--s0);
background: var(--surface-0);
border: 1px solid var(--border);
border-radius: var(--radius-sm);
font-family: var(--font-mono);
font-size: var(--step--1);
/* Terminal output and measurement tables are tabular: wrapping them
destroys the alignment that is the whole reason they were pasted.
Scroll the block instead never the page. */
overflow-x: auto;
}
.post-body :not(pre) > code {
font-family: var(--font-mono);
font-size: 0.9em;
padding: 0.1em 0.3em;
background: var(--surface-0);
border: 1px solid var(--border);
border-radius: var(--radius-sm);
}
/* An image inside a sentence still gets its own line these are
screenshots being pointed at, not icons sitting in the text. The <picture>
is what sits in the paragraph now (the reset already makes it a block);
the bare .post-media__item covers an image with no format ladder, which
is what a mirror failure or a missing encoder leaves behind. */
.post-body p > picture,
.post-body p > .post-media__item { margin-top: var(--s-2); }
.post__back { margin-top: var(--s1); }
/* Post media screenshots and screen recordings of the work, mirrored to our /* Post media screenshots and screen recordings of the work, mirrored to our
own origin by tools/fetch-media.sh. */ own origin by tools/fetch-media.sh. */
.post-media { .post-media {

View file

@ -140,6 +140,10 @@ status /legal/nope 404
# A slug that cannot be one of ours is rejected before any lookup. # A slug that cannot be one of ours is rejected before any lookup.
status /shop/BAD--slug 404 status /shop/BAD--slug 404
status /demos/nope 404 status /demos/nope 404
# A post slug that parsed but names nothing must be a real 404, or every typo
# and every retired post becomes an indexable empty page.
status /posts/nope 404
status /posts/BAD--slug 404
# The retired blog URLs are still in the wild; they must redirect, not 404. # The retired blog URLs are still in the wild; they must redirect, not 404.
status /blog 301 status /blog 301
status /blog/hello-world 301 status /blog/hello-world 301
@ -367,7 +371,7 @@ else
fi fi
# `poster` is in the list because a video poster is fetched on page load exactly # `poster` is in the list because a video poster is fetched on page load exactly
# like an <img> src is, so a third-party poster leaks the same visitor IP. # like an <img> src is, so a third-party poster leaks the same visitor IP.
if curl -s "$BASE/posts" | grep -qE '(src|href|poster)="https?://[^"]*\.(mp4|webm|webp|png|jpe?g|gif)'; then if curl -s "$BASE/posts" | grep -qE '(src|srcset|href|poster)="https?://[^"]*\.(mp4|webm|webp|avif|png|jpe?g|gif)'; then
bad "/posts media origin" "media loaded from a third party" bad "/posts media origin" "media loaded from a third party"
else else
ok "/posts loads no media from a third party" ok "/posts loads no media from a third party"
@ -417,6 +421,126 @@ else
bad "video preload" "expected preload=\"metadata\"" bad "video preload" "expected preload=\"metadata\""
fi fi
echo "== post pages =="
# The post page is where the body lives, and the body is the reason the site
# has anything for a search engine to index beyond a list of links off it. Its
# slug is data, so take one from the page rather than hardcoding a title that
# will be wrong the week after it is written.
POST_PATH=$(curl -s "$BASE/posts" | grep -oE 'href="/posts/[a-z0-9-]+"' \
| head -n1 | sed 's/href="//; s/"$//')
if [ -z "$POST_PATH" ]; then
bad "post pages" "/posts links no post page; nothing carries a body"
else
ok "/posts links a post page ($POST_PATH)"
status "$POST_PATH" 200
body_has /posts 'Read the full post' "/posts offers the full post"
# And it trails the excerpt, immediately after the ellipsis the truncation
# left, rather than sitting as its own row below the media. The excerpt is
# escaped text, so nothing but the link can put a '<' between the two.
if curl -s "$BASE/posts" \
| grep -qE '<p class="post-card__excerpt">[^<]*<a class="link-more" href="/posts/'; then
ok "read-more trails the excerpt"
else
bad "read-more placement" "not inside the excerpt paragraph"
fi
body_has "$POST_PATH" '<div class="post-body">' "post page carries the rendered body"
# Rendered, not dumped: a body that reached the page as literal Markdown
# would show its own asterisks and hashes to the reader and to a crawler.
if curl -s "$BASE$POST_PATH" | grep -qE '<(p|h2|h3|h4|ul|ol|blockquote|pre)>'; then
ok "post body is real markup, not literal Markdown"
else
bad "post body markup" "no block elements found in the body"
fi
# The canonical points here, not at the instance. That is the entire SEO
# argument for hosting the body: two copies of the text exist, and this
# says which one is the original as far as this site is concerned.
body_has "$POST_PATH" 'rel="canonical" href="https://catcrafts.net/posts/' \
"post page is its own canonical"
body_has "$POST_PATH" '"@type":"BlogPosting"' "post page carries BlogPosting JSON-LD"
body_has "$POST_PATH" '"@id":"https://catcrafts.net/#organization"' \
"post JSON-LD joins the organization node"
body_has "$POST_PATH" '"@id":"https://catcrafts.net/about#person"' \
"post JSON-LD joins the founder node"
body_has "$POST_PATH" 'property="og:type" content="article"' "post page is an article to og:"
# Hosting the body does not mirror the discussion; the thread is still one
# click away and is still where the comments are.
if curl -s "$BASE$POST_PATH" | grep -qE 'href="https://[a-z0-9.-]+/post/[0-9]+"'; then
ok "post page still links its thread"
else
bad "post thread link" "no https://<instance>/post/<id> link on the post page"
fi
# The body is prose, not an application. Same rule as /projects.
body_lacks "$POST_PATH" '<script>' "post page ships no executable script"
body_lacks "$POST_PATH" '<base' "post page has no base tag"
# Every inline image and video the body embeds is mirrored, exactly like a
# card's media — the privacy notice's "everything comes from catcrafts.net"
# covers href as well as src, so a body linking a .webp on someone else's
# instance is the same leak as embedding one.
for pg in $(curl -s "$BASE/sitemap.xml" \
| grep -oE '/posts/[a-z0-9-]+' | head -n 20); do
if curl -s "$BASE$pg" \
| grep -qE '(src|srcset|href|poster)="https?://[^"]*\.(mp4|webm|webp|avif|png|jpe?g|gif)'; then
bad "$pg media origin" "body media loaded from a third party"
else
ok "$pg loads no media from a third party"
fi
done
# The image format ladder: AVIF first, the mirrored original next, and a
# PNG on the <img> underneath, so exactly one file is fetched and every
# browser can read one of them. Order is the whole point — a browser takes
# the first source it understands — so assert the sequence, not just that
# the pieces are present.
if curl -s "$BASE$POST_PATH" | grep -q '<picture>'; then
if curl -s "$BASE$POST_PATH" \
| grep -qE '<picture><source srcset="/media/[^"]+\.avif" type="image/avif">'; then
ok "inline images lead with an AVIF source"
else
bad "image ladder" "the first source is not the AVIF"
fi
if curl -s "$BASE$POST_PATH" \
| grep -qE '<img class="post-media__item"[^>]*src="/media/[^"]+\.png"'; then
ok "inline images fall back to a PNG the img itself points at"
else
bad "image fallback" "the <img> base is not a PNG"
fi
# Every tier has to be a file that exists, or the ladder serves a 404 to
# whichever browsers pick that rung — which is precisely the set of
# browsers nobody testing this site is using.
missing=0
for f in $(curl -s "$BASE/sitemap.xml" | grep -oE '/posts/[a-z0-9-]+' | head -n 20 \
| while read -r pg; do curl -s "$BASE$pg"; done \
| grep -oE '(src|srcset)="/media/[^"]+"' \
| sed 's/.*="//; s/"$//' | sort -u); do
[ -f "media${f#/media}" ] || { missing=$((missing + 1)); echo " missing: $f" >&2; }
done
if [ "$missing" -eq 0 ]; then
ok "every referenced media file is on the mount"
else
bad "media files" "$missing referenced file(s) not on the mount"
fi
else
skip "image format ladder" "no <picture> on this page — ffmpeg absent at mirror time?"
fi
# Inline screenshots get dimensions from the sidecar list fetch-media.sh
# writes, because Markdown syntax has nowhere to carry them — without it
# the prose below every image jumps as the file arrives. Conditional: a
# post whose body embeds nothing has nothing to check.
if curl -s "$BASE$POST_PATH" | grep -q '<img class="post-media__item"'; then
if curl -s "$BASE$POST_PATH" \
| grep -qE '<img class="post-media__item"[^>]*width="[0-9]+" height="[0-9]+"'; then
ok "inline body images carry width/height"
else
bad "inline image dimensions" "an embedded body image has no dimensions"
fi
fi
fi
# The sitemap has to advertise the pages, or hosting the bodies buys nothing.
if curl -s "$BASE/sitemap.xml" | grep -qE '<loc>https://catcrafts.net/posts/[a-z0-9-]+</loc>'; then
ok "sitemap lists the post pages"
else
bad "sitemap post pages" "no /posts/<slug> entry"
fi
echo "== headers ==" echo "== headers =="
header_has / 'x-content-type-options: *nosniff' "nosniff on pages" header_has / 'x-content-type-options: *nosniff' "nosniff on pages"
header_has / 'cache-control: *public' "pages are cacheable" header_has / 'cache-control: *public' "pages are cacheable"

View file

@ -4,6 +4,21 @@
# #
# Run AFTER tools/fetch-posts.sh, which records the original URLs. # Run AFTER tools/fetch-posts.sh, which records the original URLs.
# #
# TWO KINDS OF MEDIA, one pipeline:
#
# * a post's headline file — `.media`, the recording or screenshot the post is
# about;
# * everything embedded inside the body — the screenshots a post argues with,
# which are just as much content and, until the body was hosted here, were
# never fetched at all.
#
# Both are content-addressed into the same directory, so a file used as one
# post's headline and quoted inside another's body is stored once. Body files
# are rewritten IN THE MARKDOWN TEXT (the body is still Markdown at this point)
# and additionally recorded in `.body_media`, which is where the renderer reads
# the dimensions, poster and H.264 fallback that Markdown syntax has nowhere to
# carry.
#
# WHY MIRROR rather than embed from the source: # WHY MIRROR rather than embed from the source:
# #
# * Privacy. The privacy notice states that everything the browser loads comes # * Privacy. The privacy notice states that everything the browser loads comes
@ -32,6 +47,21 @@ MEDIA_DIR="${1:-media}"
POSTS="content/posts.json" POSTS="content/posts.json"
MAX_BYTES=$((64 * 1024 * 1024)) MAX_BYTES=$((64 * 1024 * 1024))
# What counts as a media reference inside a post body: an absolute URL or a path
# we have already rewritten, ending in a media extension.
#
# Local paths are in the pattern deliberately. Leaving them out looked right —
# nothing needs downloading twice — but it is what made a second run destructive
# rather than idempotent: the already-rewritten body references were not
# enumerated, so they never re-entered the mirror map, so the body_media sidecar
# came back with only the handful of entries that happened to still be absolute.
# Matching them means they are adopted from the mount and everything is rebuilt
# exactly as it was.
#
# One definition, passed to every jq that needs it, because three copies of a
# regex is three chances for one of them to drift.
MEDIA_REF_RE='(?:https?://|/media/)[^\s)\]"<>]+\.(?:mp4|webm|mov|webp|png|jpe?g|gif|avif)'
# Media we host ourselves, published by tools/publish-media.sh before the post # Media we host ourselves, published by tools/publish-media.sh before the post
# that carries it exists. Such a URL is ALREADY the one the page should use, so # that carries it exists. Such a URL is ALREADY the one the page should use, so
# there is nothing to fetch: the bytes are on the media mount, and downloading # there is nothing to fetch: the bytes are on the media mount, and downloading
@ -54,6 +84,12 @@ mkdir -p "$MEDIA_DIR"
HAVE_FFPROBE=0 HAVE_FFPROBE=0
command -v ffprobe >/dev/null 2>&1 && HAVE_FFPROBE=1 command -v ffprobe >/dev/null 2>&1 && HAVE_FFPROBE=1
# ffmpeg does the still-image transcodes below. Optional in exactly the same
# way ffprobe is: without it every image is served as the single file the mirror
# downloaded, which is what this site did before the format ladder existed.
HAVE_FFMPEG=0
command -v ffmpeg >/dev/null 2>&1 && HAVE_FFMPEG=1
# Sets $w and $h for the file named in $1, or leaves both 0. # Sets $w and $h for the file named in $1, or leaves both 0.
# #
# One query per dimension. Asking for both at once and splitting the CSV looked # One query per dimension. Asking for both at once and splitting the CSV looked
@ -95,24 +131,161 @@ probe_dims() {
fi fi
} }
# Derive the two renditions a mirrored image is served between: AVIF above it
# and PNG below. Sets $avif_name / $png_name to the sibling file names, or
# leaves one empty when that rendition could not be produced — :Media then drops
# the tier rather than pointing at a file that is not on the mount.
#
# Siblings are named after the source file, which is itself the hash of its
# bytes, so a rendition already present is never re-encoded and a changed source
# gets new names. Only genuinely new images cost encoder time; a rebuild costs
# none, which is what keeps this off the critical path of every deploy.
#
# WHY BOTH TIERS. AVIF is smaller than the WebP the instances serve (~15% on
# these screenshots, far more on photographs) and is what almost every visitor
# actually receives. PNG is lossless and universally understood, which is what
# makes it a fallback worth having — but it is also several times the size of
# the WebP beside it, so the <picture> offers the mirrored original in between
# and the PNG is reached only by a browser that understands neither of the
# other two.
#
# The settings, measured against these files rather than guessed:
# crf 26, cpu-used 6 SSIM 0.997 against the source and still smaller than
# it, at roughly half a second per image.
# yuv444p these are screenshots of text. Re-subsampling chroma
# that pict-rs already subsampled once fringes coloured
# text visibly, and full chroma costs about 3% here.
transcode_image() {
avif_name=""
png_name=""
_file="$1"
_name="$2"
_base="${_name%.*}"
# Already that format: serve the mirrored file as the tier rather than
# re-encoding it into a second copy of itself.
case "$_name" in *.avif) avif_name="$_name" ;; esac
case "$_name" in *.png) png_name="$_name" ;; esac
[ "$HAVE_FFMPEG" = 1 ] || return 0
# An animated source is not a still, and -frames:v 1 would silently freeze
# it. Leave it entirely alone: one moving GIF is worth more than three
# copies of its first frame. nb_frames is N/A for WebP, so the frames have
# to actually be counted — ~75 ms on a 3 MP image, once per new file.
_frames=$(ffprobe -v error -select_streams v:0 -count_frames \
-show_entries stream=nb_read_frames \
-of default=nw=1:nk=1 "$_file" 2>/dev/null | head -n1)
case "$_frames" in
''|*[!0-9]*|1) ;; # unknown or a single frame: a still
*) echo "fetch-media: $_name is animated, serving it as one file" >&2
return 0 ;;
esac
# Alpha has to survive the transcode: an image with a transparent corner
# encoded into a format with no alpha plane gains an opaque black one.
_pixfmt=$(ffprobe -v error -select_streams v:0 -show_entries stream=pix_fmt \
-of default=nw=1:nk=1 "$_file" 2>/dev/null | head -n1)
case "$_pixfmt" in
yuva*|rgba*|bgra*|argb*|abgr*|gbrap*|ya8|ya16*|pal8) _avif_pix=yuva444p ;;
*) _avif_pix=yuv444p ;;
esac
# Encoded to a .part, checked, and only then renamed — so an interrupted or
# wrong-format encode cannot leave a file the next run adopts as finished.
if [ -z "$avif_name" ]; then
_cand="$_base.avif"
if [ -f "$MEDIA_DIR/$_cand" ]; then
avif_name="$_cand"
elif encode_rendition "$_file" "$_cand" av1 \
-c:v libaom-av1 -still-picture 1 -crf 26 -cpu-used 6 \
-pix_fmt "$_avif_pix" -f avif; then
avif_name="$_cand"
fi
fi
if [ -z "$png_name" ]; then
_cand="$_base.png"
if [ -f "$MEDIA_DIR/$_cand" ]; then
png_name="$_cand"
elif encode_rendition "$_file" "$_cand" png -c:v png -f image2; then
png_name="$_cand"
fi
fi
}
# encode_rendition SRC OUTNAME EXPECTED_CODEC ffmpeg-args...
#
# Runs the encode into a .part, verifies the result really is the codec asked
# for, and only then publishes it. Returns non-zero (leaving nothing behind) if
# either step fails, which drops that tier rather than shipping a broken one.
#
# The verification is not paranoia. ffmpeg picks an encoder from the MUXER when
# one is not named, and the image2 muxer defaults to MJPEG — so `-f image2
# out.png` silently produced a run of lossy JPEGs sitting under .png names, which
# the page then advertised to browsers as image/png. The codec is pinned by the
# callers above; this is the check that the pin held.
encode_rendition() {
_src="$1"; _out="$2"; _want="$3"
shift 3
if ! ffmpeg -y -v error -i "$_src" -frames:v 1 "$@" \
"$MEDIA_DIR/$_out.part" 2>/dev/null; then
rm -f "$MEDIA_DIR/$_out.part"
echo "fetch-media: could not encode $_out, serving without that tier" >&2
return 1
fi
_got=$(ffprobe -v error -select_streams v:0 -show_entries stream=codec_name \
-of default=nw=1:nk=1 "$MEDIA_DIR/$_out.part" 2>/dev/null | head -n1)
if [ "$_got" != "$_want" ]; then
rm -f "$MEDIA_DIR/$_out.part"
echo "fetch-media: $_out came out as '$_got', expected '$_want' — discarding it" >&2
return 1
fi
mv "$MEDIA_DIR/$_out.part" "$MEDIA_DIR/$_out"
chmod 0644 "$MEDIA_DIR/$_out"
encoded=$((encoded + 1))
return 0
}
MAP="$(mktemp)" MAP="$(mktemp)"
POSTERMAP="$(mktemp)" POSTERMAP="$(mktemp)"
FALLBACKMAP="$(mktemp)" FALLBACKMAP="$(mktemp)"
trap 'rm -f "$MAP" "$POSTERMAP" "$FALLBACKMAP"' EXIT AVIFMAP="$(mktemp)"
POSTERONLY="$(mktemp)"
trap 'rm -f "$MAP" "$POSTERMAP" "$FALLBACKMAP" "$AVIFMAP" "$POSTERONLY"' EXIT
printf '[]' > "$MAP" printf '[]' > "$MAP"
printf '[]' > "$POSTERMAP" printf '[]' > "$POSTERMAP"
printf '[]' > "$FALLBACKMAP" printf '[]' > "$FALLBACKMAP"
printf '[]' > "$AVIFMAP"
# URLs that are ONLY ever a video's poster frame. They are skipped by the
# transcode above, because `poster` takes exactly one URL: a <video> cannot
# negotiate a format the way <picture> can, so the renditions would be files
# nothing is able to ask for. A file that is a poster somewhere and an ordinary
# image somewhere else is not in this list and is transcoded normally.
jq -r --arg re "$MEDIA_REF_RE" \
'([.[].media[]? | .poster // empty] | map(select(. != "")) | unique) as $posters
| ([.[].media[]? | .src] + [.[] | .body // "" | scan($re)] | unique) as $srcs
| ($posters - $srcs) | .[]' "$POSTS" > "$POSTERONLY" 2>/dev/null || true
downloaded=0 downloaded=0
reused=0 reused=0
adopted=0 adopted=0
failed=0 failed=0
encoded=0
# Every distinct media URL across all posts, so a file shared by two posts is # Every distinct media URL across all posts, so a file shared by two posts is
# fetched once. Video posters are in here too: a poster left pointing at the # fetched once. Video posters are in here too: a poster left pointing at the
# source instance would leak a visitor IP on page load exactly like an embedded # source instance would leak a visitor IP on page load exactly like an embedded
# image would, and it is the frame shown before anyone presses play. # image would, and it is the frame shown before anyone presses play.
# #
# Body URLs are found by pattern rather than by parsing Markdown: anything that
# looks like an absolute URL ending in a media extension is mirrored, whether it
# was written as an embed, as a link, or bare. That is deliberately wider than
# "images the body displays" — the origin rule covers href as well as src, and a
# link whose target is a .webp on someone else's instance is still a third-party
# address on our page. Already-rewritten paths start with /media/ and so do not
# match, which is what makes re-running this a no-op.
#
# Fed by a here-document rather than a pipe so the counters below survive — in # Fed by a here-document rather than a pipe so the counters below survive — in
# `jq | while`, the loop runs in a subshell and every increment is discarded. # `jq | while`, the loop runs in a subshell and every increment is discarded.
while IFS= read -r src; do while IFS= read -r src; do
@ -152,6 +325,26 @@ while IFS= read -r src; do
fi fi
adopted=$((adopted + 1)) adopted=$((adopted + 1))
;; ;;
/media/*)
# Already rewritten by an earlier run of this script. Adopt the file on
# the mount rather than trying to fetch our own path as though it were a
# URL — which is what makes running this twice a no-op instead of a way
# to lose every rewrite it made the first time. The script is meant to
# follow fetch-posts.sh, but "meant to" is not a guarantee, and the
# failure was silent: the body_media list simply came back empty.
name=${src#/media/}
case "$name" in
''|*/*|*..*)
echo "fetch-media: refusing suspicious local path: $src" >&2
failed=$((failed + 1)); continue ;;
esac
dest="$MEDIA_DIR/$name"
if [ ! -f "$dest" ]; then
echo "fetch-media: $name not on the media mount, leaving it alone: $src" >&2
failed=$((failed + 1)); continue
fi
adopted=$((adopted + 1))
;;
*) *)
ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z') ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z')
case "$ext" in case "$ext" in
@ -226,6 +419,26 @@ while IFS= read -r src; do
&& mv "$FALLBACKMAP.new" "$FALLBACKMAP" && mv "$FALLBACKMAP.new" "$FALLBACKMAP"
fi fi
;; ;;
*.webp|*.png|*.jpg|*.jpeg|*.gif|*.avif)
# The AVIF and PNG tiers this image is served between. Both maps are
# keyed by the LOCAL path, like the video ones above, so the rewrite
# below can look them up from the src it has just written.
if grep -qxF "$src" "$POSTERONLY" 2>/dev/null; then
: # poster-only; see POSTERONLY above
else
transcode_image "$dest" "$name"
if [ -n "$avif_name" ]; then
jq --arg k "/media/$name" --arg v "/media/$avif_name" \
'. + [{key: $k, value: $v}]' "$AVIFMAP" > "$AVIFMAP.new" \
&& mv "$AVIFMAP.new" "$AVIFMAP"
fi
if [ -n "$png_name" ]; then
jq --arg k "/media/$name" --arg v "/media/$png_name" \
'. + [{key: $k, value: $v}]' "$FALLBACKMAP" > "$FALLBACKMAP.new" \
&& mv "$FALLBACKMAP.new" "$FALLBACKMAP"
fi
fi
;;
esac esac
jq --arg src "$src" --arg path "/media/$name" \ jq --arg src "$src" --arg path "/media/$name" \
@ -233,20 +446,25 @@ while IFS= read -r src; do
'. + [{src: $src, path: $path, w: $w, h: $h}]' "$MAP" > "$MAP.new" \ '. + [{src: $src, path: $path, w: $w, h: $h}]' "$MAP" > "$MAP.new" \
&& mv "$MAP.new" "$MAP" && mv "$MAP.new" "$MAP"
done <<EOF done <<EOF
$(jq -r '[.[].media[]? | .src, (.poster // empty)] | map(select(. != "")) | unique[]' "$POSTS") $(jq -r --arg re "$MEDIA_REF_RE" \
'[ (.[].media[]? | .src, (.poster // empty)),
(.[] | .body // "" | scan($re)) ]
| map(select(. != "")) | unique[]' "$POSTS")
EOF EOF
echo "fetch-media: $downloaded new, $reused already present, $adopted self-hosted, $failed failed" echo "fetch-media: $downloaded new, $reused already present, $adopted self-hosted, $failed failed"
echo "fetch-media: $encoded image rendition(s) encoded this run"
# Rewrite each media entry to the local path. An entry with no mapping (download # Rewrite each media entry to the local path. An entry with no mapping (download
# failed) keeps its original src, so the page still shows something rather than # failed) keeps its original src, so the page still shows something rather than
# silently dropping the post's whole point. # silently dropping the post's whole point.
TMP_POSTS="$(mktemp)" TMP_POSTS="$(mktemp)"
if jq --slurpfile map "$MAP" --slurpfile posters "$POSTERMAP" \ if jq --slurpfile map "$MAP" --slurpfile posters "$POSTERMAP" \
--slurpfile fallbacks "$FALLBACKMAP" ' --slurpfile fallbacks "$FALLBACKMAP" --slurpfile avifs "$AVIFMAP" '
($map[0] | map({key: .src, value: .}) | from_entries) as $m ($map[0] | map({key: .src, value: .}) | from_entries) as $m
| ($posters[0] | from_entries) as $pm | ($posters[0] | from_entries) as $pm
| ($fallbacks[0] | from_entries) as $fm | ($fallbacks[0] | from_entries) as $fm
| ($avifs[0] | from_entries) as $am
| map(.media = ((.media // []) | map( | map(.media = ((.media // []) | map(
. as $item . as $item
| ($m[$item.src] // null) as $hit | ($m[$item.src] // null) as $hit
@ -271,7 +489,42 @@ if jq --slurpfile map "$MAP" --slurpfile posters "$POSTERMAP" \
# sibling to find. # sibling to find.
| if (($fm[.src] // "") != "") | if (($fm[.src] // "") != "")
then . + { fallback: $fm[.src] } then . + { fallback: $fm[.src] }
else . end
# The AVIF tier, keyed by the LOCAL src like the two maps above. Only
# ever set for images, and only when the encode actually produced one.
| if (($am[.src] // "") != "")
then . + { avif: $am[.src] }
else . end))) else . end)))
# ── the body ──────────────────────────────────────────────────────
#
# Substitution is literal (split/join, not gsub), because these URLs are
# full of regex metacharacters and a mirrored path must land in the text
# exactly as written. A URL with no mapping — its download failed — is left
# alone, so the post still shows the image rather than losing it; the
# accounting at the end of this script reports that as media still pointing
# at its source.
| ($m | to_entries) as $subs
| map(.body = (reduce $subs[] as $s ((.body // "");
split($s.key) | join($s.value.path))))
# Everything the rewritten body now points at, as records the renderer can
# read: Markdown has nowhere to put a width, a poster frame or a second
# source, so the sidecar list is how an inline video gets the same treatment
# as a headline one. Keyed by the LOCAL path, which is what the body says
# by this point.
| ($m | map({ key: .path, value: . }) | from_entries) as $byPath
| map(.body_media = ([ (.body // "")
| scan("/media/[A-Za-z0-9._-]+")
| . as $path
| select($byPath[$path] != null)
| { src: $path,
kind: (if ($path | test("\\.(?:mp4|webm|mov)$"))
then "video" else "image" end),
poster: ($pm[$path] // ""),
fallback: ($fm[$path] // ""),
avif: ($am[$path] // ""),
w: ($byPath[$path].w // 0),
h: ($byPath[$path].h // 0) } ]
| unique_by(.src)))
' "$POSTS" > "$TMP_POSTS" 2>/dev/null; then ' "$POSTS" > "$TMP_POSTS" 2>/dev/null; then
# Same reason as the chmod on each mirrored file: mktemp is 0600 and the # Same reason as the chmod on each mirrored file: mktemp is 0600 and the
# mode survives to production, where other users must read this. # mode survives to production, where other users must read this.
@ -283,10 +536,18 @@ else
exit 1 exit 1
fi fi
total=$(jq '[.[].media[]? | .src, (.poster // empty) | select(. != "")] | length' "$POSTS") # Counted over the bodies too, because a body URL that never mirrored is the
local_count=$(jq '[.[].media[]? | .src, (.poster // empty) # same privacy leak as a card one and must not be reported as success.
| select(startswith("/media/"))] | length' "$POSTS") total=$(jq --arg re "$MEDIA_REF_RE" \
echo "fetch-media: $local_count of $total media entries served locally ($(du -sh "$MEDIA_DIR" | cut -f1) in $MEDIA_DIR)" '[ (.[].media[]? | .src, (.poster // empty)),
(.[] | .body // "" | scan($re)),
(.[].body_media[]? | .src, (.poster // empty)) ]
| map(select(. != "")) | unique | length' "$POSTS")
local_count=$(jq '[ (.[].media[]? | .src, (.poster // empty)),
(.[].body_media[]? | .src, (.poster // empty)) ]
| map(select(startswith("/media/"))) | length' "$POSTS")
inline=$(jq '[.[].body_media[]?] | length' "$POSTS")
echo "fetch-media: $local_count of $total media entries served locally ($inline of them embedded in post bodies; $(du -sh "$MEDIA_DIR" | cut -f1) in $MEDIA_DIR)"
if [ "$local_count" -ne "$total" ]; then if [ "$local_count" -ne "$total" ]; then
echo "fetch-media: $((total - local_count)) still point at their source — see the failures above" >&2 echo "fetch-media: $((total - local_count)) still point at their source — see the failures above" >&2
fi fi

View file

@ -28,6 +28,12 @@ CONFIG="${1:-content/posts-sources.json}"
OUT="content/posts.json" OUT="content/posts.json"
LIMIT=50 LIMIT=50
EXCERPT_CHARS=280 EXCERPT_CHARS=280
# A ceiling, not a target: the longest of these bodies is around 6 KB and this
# file is fetched into memory before the wasm module starts, so an outlier must
# not be able to grow the bundle without bound. A truncated body renders as
# truncated prose (the Markdown renderer tolerates an unclosed fence), which is
# a better failure than a build that silently ships a megabyte.
MAX_BODY_CHARS=32768
command -v jq >/dev/null 2>&1 || { echo "fetch-posts: jq not found, keeping existing $OUT" >&2; exit 0; } command -v jq >/dev/null 2>&1 || { echo "fetch-posts: jq not found, keeping existing $OUT" >&2; exit 0; }
[ -f "$CONFIG" ] || { echo "fetch-posts: $CONFIG not found, keeping existing $OUT" >&2; exit 0; } [ -f "$CONFIG" ] || { echo "fetch-posts: $CONFIG not found, keeping existing $OUT" >&2; exit 0; }
@ -84,13 +90,39 @@ fi
# Both are ORIGINAL urls here; tools/fetch-media.sh mirrors them # Both are ORIGINAL urls here; tools/fetch-media.sh mirrors them
# and rewrites to local paths, so nothing the browser loads is # and rewrites to local paths, so nothing the browser loads is
# third-party. # third-party.
# excerpt : body flattened to one line and truncated. Markdown is NOT # excerpt : body flattened to one line and truncated, for the card and for
# rendered — the site has no markdown pipeline by design, so any # the meta description. Markdown markers are stripped rather than
# surviving syntax would show as literal characters. Strip the # rendered — this string lands in places that are plain text by
# common inline markers and let the rest be plain text. # definition (<meta name="description">, og:description), so any
# surviving syntax would show as literal characters.
# body : the post, whole, still as Markdown. Rendered by
# Catcrafts.Shared:Markdown at page-render time rather than
# converted here, because that module is inside the escaping
# guarantee and a shell script writing HTML into a content file
# would not be — text from someone else's server must not be able
# to become markup anywhere but there.
# slug : the post's own URL at /posts/<slug>, from the title. Collisions
# take the post's numeric id as a suffix rather than a positional
# one: a later post sharing a title would otherwise renumber an
# earlier post's URL out from under everyone who linked it.
# deleted / removed posts are dropped rather than rendered as empty cards. # deleted / removed posts are dropped rather than rendered as empty cards.
if ! jq --argjson n "$EXCERPT_CHARS" \ if ! jq --argjson n "$EXCERPT_CHARS" \
--argjson maxbody "$MAX_BODY_CHARS" \
--slurpfile cfg "$CONFIG" ' --slurpfile cfg "$CONFIG" '
# Catcrafts.Shared:Route::IsValidSlug is the contract: lowercase ASCII,
# digits and single hyphens, no leading or trailing hyphen, at most 64
# characters. A slug that fails it is dropped by the loader, which costs
# the post its page — so the shape is produced correctly here rather than
# sanitised on the way out.
def slugify:
ascii_downcase
| gsub("[^a-z0-9]+"; "-")
| sub("^-+"; "") | sub("-+$"; "");
# Truncate on a word boundary where there is one: a slug cut mid-word reads
# like a typo, and these titles are long enough to hit the limit.
def clamp($n):
(if (length > $n) then (.[0:$n] | sub("-[^-]*$"; "")) else . end)
| sub("-+$"; "");
($cfg[0].communities | map(ascii_downcase)) as $allow ($cfg[0].communities | map(ascii_downcase)) as $allow
| [ .posts[] | [ .posts[]
| select((.post.deleted // false) == false) | select((.post.deleted // false) == false)
@ -101,6 +133,11 @@ if ! jq --argjson n "$EXCERPT_CHARS" \
| select(($comm | ascii_downcase) as $c | $allow | index($c)) | select(($comm | ascii_downcase) as $c | $allow | index($c))
| { | {
title: ($p.post.name // ""), title: ($p.post.name // ""),
slug: (($p.post.name // "") | slugify | clamp(64)),
# Carried only as far as the de-duplication pass below, which strips
# it: it is a tie-breaker, not content.
uid: (($p.post.ap_id // "") | sub(".*/"; "") | ascii_downcase
| gsub("[^a-z0-9]"; "") | .[0:12]),
permalink: ($p.post.ap_id // ""), permalink: ($p.post.ap_id // ""),
# A link post whose target IS an image or video is a media post, not a # A link post whose target IS an image or video is a media post, not a
# link post: the file is captured in `media` and embedded, so keeping # link post: the file is captured in `media` and embedded, so keeping
@ -123,6 +160,10 @@ if ! jq --argjson n "$EXCERPT_CHARS" \
| gsub(" +"; " ") | gsub(" +"; " ")
| ltrimstr(" ") | rtrimstr(" ") | ltrimstr(" ") | rtrimstr(" ")
| if (. | length) > $n then (.[0:$n] | sub(" [^ ]*$"; "")) + "…" else . end), | if (. | length) > $n then (.[0:$n] | sub(" [^ ]*$"; "")) + "…" else . end),
# Verbatim apart from CR removal (a stray \r would render as a stray
# character inside a code block, where nothing is interpreted) and the
# size ceiling.
body: (($p.post.body // "") | gsub("\r"; "") | .[0:$maxbody]),
media: ([ ($p.post.url // "") media: ([ ($p.post.url // "")
| select(test("\\.(?:mp4|webm|mov|webp|png|jpe?g|gif|avif)$")) | select(test("\\.(?:mp4|webm|mov|webp|png|jpe?g|gif|avif)$"))
| (if test("\\.(mp4|webm|mov)$") then "video" else "image" end) as $kind | (if test("\\.(mp4|webm|mov)$") then "video" else "image" end) as $kind
@ -139,7 +180,26 @@ if ! jq --argjson n "$EXCERPT_CHARS" \
score: ($p.counts.score // 0), score: ($p.counts.score // 0),
comments: ($p.counts.comments // 0) comments: ($p.counts.comments // 0)
} }
]' "$RAW" > "$TMP" 2>/dev/null; then ]
# Make every slug unique, and keep it that way across builds.
#
# Only a duplicate is suffixed, so the ordinary post keeps the clean URL the
# title earned. The suffix is the numeric id of the post itself rather than
# a counter, because a counter is positional: a new post repeating an older
# title arrives at the front of this newest-first list, takes the bare slug,
# and silently renumbers the older one — breaking a URL that is already in
# search results and in whatever links people have shared.
# (Note for editors: this jq program is inside a single-quoted shell string,
# so no apostrophes anywhere in it.)
| reduce .[] as $e ({ seen: {}, out: [] };
(if ($e.slug | length) == 0 then "post" else $e.slug end) as $base
| (if (.seen[$base] // false)
then (($base | clamp(50)) + "-"
+ (if ($e.uid | length) == 0 then "x" else $e.uid end))
else $base end) as $slug
| { seen: (.seen + { ($base): true, ($slug): true }),
out: (.out + [ ($e | del(.uid)) + { slug: $slug } ]) })
| .out' "$RAW" > "$TMP" 2>/dev/null; then
echo "fetch-posts: response did not match the expected shape, keeping existing $OUT" >&2 echo "fetch-posts: response did not match the expected shape, keeping existing $OUT" >&2
exit 0 exit 0
fi fi