media: publish to our own origin instead of mirroring a file host
All checks were successful
Deploy / build-deploy (push) Successful in 7m34s

A 167 MB screen recording uploaded to catbox.moe hit fetch-media.sh's 64 MB
MAX_BYTES, so curl refused it, the entry kept its third-party URL, and e2e
failed "/posts media origin" on a file we had on disk the whole time. Raising
the cap would have papered over it: the mirror step still depends on someone
else's server being up, fast, and still holding the file.

Invert it. tools/publish-media.sh uploads a recording to the media mount under
its content hash BEFORE the post exists and prints the URL to post, and
fetch-media.sh adopts an own-origin URL by rewriting it to /media/<hash> with no
request at all — no size cap, no third party in the build.

Also here, because publishing exposed them:

  * Rotation. Phones record 1920x1080 and attach a display matrix rather than
    rotating pixels, so an untouched file reports landscape while playing
    portrait and width/height reserve exactly the wrong box. publish-media.sh
    bakes rotation into the frames; fetch-media.sh swaps the dimensions on a
    quarter-turn matrix for anything mirrored straight from a phone.
  * Posters. pict-rs will not thumbnail AV1, so a self-hosted video usually
    arrives with no poster. publish-media.sh uploads <hash>.poster.webp beside
    the video and fetch-media.sh falls back to it — a real thumbnail still wins.
  * The workflow comment claiming a file on the mount is never downloaded again
    was wrong: the name is the hash of the bytes, so third-party media is
    re-fetched every build and only the write is skipped.

Transcoding to AV1 is what makes self-hosting cheap: that clip was 78 s of a
dark room at 17 Mbps, and denoise + AV1 gives the same picture in 15 MB. Note
<video> carries a single src with no fallback, so AV1 excludes Safari < 17;
--raw skips the transcode when that matters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-08-08 01:16:17 +02:00
commit c1b0c29af6
4 changed files with 351 additions and 66 deletions

View file

@ -118,8 +118,9 @@ Only listed communities are fetched, so joining a new one does not silently
publish it to the site — add a line first.
```sh
tools/fetch-posts.sh # writes content/posts.json
tools/fetch-media.sh # mirrors the media, rewrites posts.json to /media/... paths
tools/publish-media.sh FILE # BEFORE posting: uploads to /media, prints the URL
tools/fetch-posts.sh # writes content/posts.json
tools/fetch-media.sh # mirrors the media, rewrites posts.json to /media/... paths
```
Order matters: the second reads what the first wrote. CI runs both before the
@ -127,6 +128,47 @@ build. Neither fails the build on a network error — a fediverse outage leaves
previous `posts.json` in place, and a single failed download leaves that one entry
pointing at its original URL rather than losing the post.
### Publish the media first, then post it
**The recommended flow is to put a recording on catcrafts.net before writing the
post, and use that URL as the post's link.** Run `tools/publish-media.sh
recording.mp4`; it transcodes, uploads to the media mount under its content hash,
and prints a `https://catcrafts.net/media/<hash>.mp4` URL to paste into the post.
`fetch-media.sh` recognises its own origin and **adopts** such a URL: it rewrites
it to `/media/<hash>` and probes the local file for dimensions, downloading
nothing. That is not just an optimisation — it removes the whole class of build
failure where the media step depends on a third party. A 167 MB recording on a
file host once hit `MAX_BYTES` (64 MB), so the entry kept its original URL, and
the deploy then failed the e2e "media origin" check on a file that was sitting on
our own disk the entire time.
The transcode matters as much as the hosting. Phone recordings are wildly
oversized for what they show — that same 167 MB clip was 78 s of a dark room at
17 Mbps, and denoising into AV1 gives the same picture in 15 MB. It also **bakes
in the rotation**: phones record landscape and attach a display matrix, so an
untouched file reports 1920x1080 while playing portrait, and the `width`/`height`
attributes then reserve exactly the wrong box. (`fetch-media.sh` swaps the
dimensions when it sees a quarter-turn matrix, so a straight-from-phone mirror is
correct too — but transcoding means nothing downstream has to know.)
Two things to know about publishing AV1:
* Nothing emits a fallback encoding and `<video>` carries a single `src`, so
browsers without AV1 (Safari before 17, Apple hardware older than A17/M3) get
an element that will not play. Use `publish-media.sh --raw`, or a H.264
re-encode, when that matters for a particular post.
* Lemmy's `pict-rs` will not generate a thumbnail from an AV1 file, so a
self-hosted video usually arrives with no `poster`. `publish-media.sh` uploads
a poster frame beside the video, named `<video-hash>.poster.webp`, and
`fetch-media.sh` falls back to that sibling when the instance supplied nothing.
A thumbnail the instance *did* provide always wins.
An own-origin URL naming a file that is **not** on the mount is deliberately left
pointing at its original URL rather than rewritten. That is a post published
without its media, and failing the e2e origin check loudly beats shipping a 404
inside a `<video>` tag.
**`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions,
which become the `width`/`height` attributes that stop the page reflowing as
several 5 MB recordings arrive. It degrades to no dimensions without it —