media: publish to our own origin instead of mirroring a file host

A 167 MB screen recording uploaded to catbox.moe hit fetch-media.sh's 64 MB
MAX_BYTES, so curl refused it, the entry kept its third-party URL, and e2e
failed "/posts media origin" on a file we had on disk the whole time. Raising
the cap would have papered over it: the mirror step still depends on someone
else's server being up, fast, and still holding the file.

Invert it. tools/publish-media.sh uploads a recording to the media mount under
its content hash BEFORE the post exists and prints the URL to post, and
fetch-media.sh adopts an own-origin URL by rewriting it to /media/<hash> with no
request at all — no size cap, no third party in the build.

Also here, because publishing exposed them:

  * Rotation. Phones record 1920x1080 and attach a display matrix rather than
    rotating pixels, so an untouched file reports landscape while playing
    portrait and width/height reserve exactly the wrong box. publish-media.sh
    bakes rotation into the frames; fetch-media.sh swaps the dimensions on a
    quarter-turn matrix for anything mirrored straight from a phone.
  * Posters. pict-rs will not thumbnail AV1, so a self-hosted video usually
    arrives with no poster. publish-media.sh uploads <hash>.poster.webp beside
    the video and fetch-media.sh falls back to it — a real thumbnail still wins.
  * The workflow comment claiming a file on the mount is never downloaded again
    was wrong: the name is the hash of the bytes, so third-party media is
    re-fetched every build and only the write is skipped.

Transcoding to AV1 is what makes self-hosting cheap: that clip was 78 s of a
dark room at 17 Mbps, and denoise + AV1 gives the same picture in 15 MB. Note
<video> carries a single src with no fallback, so AV1 excludes Safari < 17;
--raw skips the transcode when that matters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.forgejo/workflows/deploy.yaml

@@ -97,10 +97,16 @@ jobs:
         # loads is third-party — which is what keeps the privacy notice's
         # "everything comes from catcrafts.net" true.
         #
-        # Content-addressed and incremental: a file already on the media mount is
-        # never downloaded again. Writes straight into the mount so the copies
-        # persist across deploys — they are NOT always reproducible, because a
-        # source instance deleting a file leaves ours as the only one.
+        # Content-addressed: the name is the hash of the bytes. Note what that
+        # does NOT mean — a third-party file is re-fetched on every build, because
+        # the name cannot be known until the bytes are in hand; only the write is
+        # skipped when the hash is already on the mount. Media we host ourselves
+        # (tools/publish-media.sh) is the exception that is genuinely incremental:
+        # fetch-media.sh adopts an own-origin URL without any request at all.
+        #
+        # Writes straight into the mount so the copies persist across deploys —
+        # they are NOT always reproducible, because a source instance deleting a
+        # file leaves ours as the only one.
         run: |
           set -eu
           if [ -d /deploy-app ]; then

deploy/README.md

@@ -118,6 +118,7 @@ Only listed communities are fetched, so joining a new one does not silently
 publish it to the site — add a line first.
 
 ```sh
+tools/publish-media.sh FILE  # BEFORE posting: uploads to /media, prints the URL
 tools/fetch-posts.sh         # writes content/posts.json
 tools/fetch-media.sh         # mirrors the media, rewrites posts.json to /media/... paths
 ```
@@ -127,6 +128,47 @@ build. Neither fails the build on a network error — a fediverse outage leaves
 previous `posts.json` in place, and a single failed download leaves that one entry
 pointing at its original URL rather than losing the post.
 
+### Publish the media first, then post it
+
+**The recommended flow is to put a recording on catcrafts.net before writing the
+post, and use that URL as the post's link.** Run `tools/publish-media.sh
+recording.mp4`; it transcodes, uploads to the media mount under its content hash,
+and prints a `https://catcrafts.net/media/<hash>.mp4` URL to paste into the post.
+
+`fetch-media.sh` recognises its own origin and **adopts** such a URL: it rewrites
+it to `/media/<hash>` and probes the local file for dimensions, downloading
+nothing. That is not just an optimisation — it removes the whole class of build
+failure where the media step depends on a third party. A 167 MB recording on a
+file host once hit `MAX_BYTES` (64 MB), so the entry kept its original URL, and
+the deploy then failed the e2e "media origin" check on a file that was sitting on
+our own disk the entire time.
+
+The transcode matters as much as the hosting. Phone recordings are wildly
+oversized for what they show — that same 167 MB clip was 78 s of a dark room at
+17 Mbps, and denoising into AV1 gives the same picture in 15 MB. It also **bakes
+in the rotation**: phones record landscape and attach a display matrix, so an
+untouched file reports 1920x1080 while playing portrait, and the `width`/`height`
+attributes then reserve exactly the wrong box. (`fetch-media.sh` swaps the
+dimensions when it sees a quarter-turn matrix, so a straight-from-phone mirror is
+correct too — but transcoding means nothing downstream has to know.)
+
+Two things to know about publishing AV1:
+
+* Nothing emits a fallback encoding and `<video>` carries a single `src`, so
+  browsers without AV1 (Safari before 17, Apple hardware older than A17/M3) get
+  an element that will not play. Use `publish-media.sh --raw`, or a H.264
+  re-encode, when that matters for a particular post.
+* Lemmy's `pict-rs` will not generate a thumbnail from an AV1 file, so a
+  self-hosted video usually arrives with no `poster`. `publish-media.sh` uploads
+  a poster frame beside the video, named `<video-hash>.poster.webp`, and
+  `fetch-media.sh` falls back to that sibling when the instance supplied nothing.
+  A thumbnail the instance *did* provide always wins.
+
+An own-origin URL naming a file that is **not** on the mount is deliberately left
+pointing at its original URL rather than rewritten. That is a post published
+without its media, and failing the e2e origin check loudly beats shipping a 404
+inside a `<video>` tag.
+
 **`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions,
 which become the `width`/`height` attributes that stop the page reflowing as
 several 5 MB recordings arrive. It degrades to no dimensions without it —

tools/fetch-media.sh

@@ -32,6 +32,16 @@ MEDIA_DIR="${1:-media}"
 POSTS="content/posts.json"
 MAX_BYTES=$((64 * 1024 * 1024))
 
+# Media we host ourselves, published by tools/publish-media.sh before the post
+# that carries it exists. Such a URL is ALREADY the one the page should use, so
+# there is nothing to fetch: the bytes are on the media mount, and downloading
+# them back from our own web server would only mint a second copy under a second
+# name. Skipping the download also skips MAX_BYTES, which is what a 167 MB
+# recording on a third-party file host ran into — and it removes the last part of
+# a build that could fail because someone else's server was slow, rate-limiting
+# or gone.
+OWN_ORIGIN="https://catcrafts.net/media/"
+
 command -v jq >/dev/null 2>&1 || { echo "fetch-media: jq not found" >&2; exit 1; }
 [ -f "$POSTS" ] || { echo "fetch-media: $POSTS not found — run fetch-posts.sh first" >&2; exit 1; }
 
@@ -44,12 +54,56 @@ mkdir -p "$MEDIA_DIR"
 HAVE_FFPROBE=0
 command -v ffprobe >/dev/null 2>&1 && HAVE_FFPROBE=1
 
+# Sets $w and $h for the file named in $1, or leaves both 0.
+#
+# One query per dimension. Asking for both at once and splitting the CSV looked
+# simpler but was wrong: for some files ffprobe appends an empty field, so
+# `width,height` came back as "854x480x" and splitting on `x` gave a height of
+# "480x" — which the digit guard below then threw away, silently costing the
+# dimensions of exactly the videos that had the extra field. `nk=1` prints the
+# bare value, so there is nothing to split.
+#
+# ROTATION: a phone records 1920x1080 and attaches a display matrix rather than
+# rotating the pixels, so the stream reads landscape while the video plays
+# portrait. Believing the stream there reserves a landscape box for a portrait
+# video — precisely the layout shift these attributes exist to prevent — so a
+# quarter-turn swaps them. Files that went through publish-media.sh have the
+# rotation baked into the pixels and report no matrix at all; this is for
+# anything mirrored straight from a phone.
+probe_dims() {
+    w=0; h=0
+    [ "$HAVE_FFPROBE" = 1 ] || return 0
+    pw=$(ffprobe -v error -select_streams v:0 -show_entries stream=width \
+                 -of default=nw=1:nk=1 "$1" 2>/dev/null | head -n1 || true)
+    ph=$(ffprobe -v error -select_streams v:0 -show_entries stream=height \
+                 -of default=nw=1:nk=1 "$1" 2>/dev/null | head -n1 || true)
+    rot=$(ffprobe -v error -select_streams v:0 \
+                  -show_entries stream_side_data=rotation \
+                  -of default=nw=1:nk=1 "$1" 2>/dev/null | head -n1 || true)
+    case "$pw" in ''|*[!0-9]*) pw=0 ;; esac
+    case "$ph" in ''|*[!0-9]*) ph=0 ;; esac
+    # ffprobe reports this as a signed number that some builds print with a
+    # fractional part ("-90.000000"), so compare on the integer portion.
+    case "${rot%%.*}" in
+        90|-90|270|-270) t=$pw; pw=$ph; ph=$t ;;
+    esac
+    # Both or neither: a lone dimension is worse than none, because the browser
+    # derives the missing one from it and gets the aspect wrong.
+    if [ "$pw" -gt 0 ] && [ "$ph" -gt 0 ]; then w=$pw; h=$ph; fi
+    if [ "$w" = 0 ]; then
+        echo "fetch-media: no dimensions for $1; layout will shift on load" >&2
+    fi
+}
+
 MAP="$(mktemp)"
-trap 'rm -f "$MAP"' EXIT
+POSTERMAP="$(mktemp)"
+trap 'rm -f "$MAP" "$POSTERMAP"' EXIT
 printf '[]' > "$MAP"
+printf '[]' > "$POSTERMAP"
 
 downloaded=0
 reused=0
+adopted=0
 failed=0
 
 # Every distinct media URL across all posts, so a file shared by two posts is
@@ -62,6 +116,30 @@ failed=0
 while IFS= read -r src; do
     [ -n "$src" ] || continue
 
+    # Ours already — adopt the file on the mount and do no network at all.
+    # A URL that is on our origin but names a file that is NOT on the mount is
+    # left alone rather than invented: that is a post published without its
+    # media, and keeping the original URL makes the e2e origin check fail
+    # loudly instead of shipping a 404 in a <video> tag.
+    case "$src" in
+    "$OWN_ORIGIN"*)
+        name=${src#"$OWN_ORIGIN"}
+        # Refuse anything that is not a bare filename. A path separator or a
+        # traversal segment arriving from a post URL must never reach a path we
+        # then read or publish.
+        case "$name" in
+            ''|*/*|*..*)
+                echo "fetch-media: refusing suspicious own-origin URL: $src" >&2
+                failed=$((failed + 1)); continue ;;
+        esac
+        dest="$MEDIA_DIR/$name"
+        if [ ! -f "$dest" ]; then
+            echo "fetch-media: $name not on the media mount, keeping original URL: $src" >&2
+            failed=$((failed + 1)); continue
+        fi
+        adopted=$((adopted + 1))
+        ;;
+    *)
         ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z')
         case "$ext" in
             mp4|webm|mov|webp|png|jpg|jpeg|gif|avif) ;;
@@ -98,28 +176,27 @@ while IFS= read -r src; do
             chmod 0644 "$dest"
             downloaded=$((downloaded + 1))
         fi
+        ;;
+    esac
 
-    # One query per dimension. Asking for both at once and splitting the CSV
-    # looked simpler but was wrong: for some files ffprobe appends an empty
-    # field, so `width,height` came back as "854x480x" and splitting on `x` gave
-    # a height of "480x" — which the digit guard below then threw away, silently
-    # costing the dimensions of exactly the videos that had the extra field.
-    # `nk=1` prints the bare value, so there is nothing to split.
-    w=0; h=0
-    if [ "$HAVE_FFPROBE" = 1 ]; then
-        pw=$(ffprobe -v error -select_streams v:0 -show_entries stream=width \
-                     -of default=nw=1:nk=1 "$dest" 2>/dev/null | head -n1 || true)
-        ph=$(ffprobe -v error -select_streams v:0 -show_entries stream=height \
-                     -of default=nw=1:nk=1 "$dest" 2>/dev/null | head -n1 || true)
-        case "$pw" in ''|*[!0-9]*) pw=0 ;; esac
-        case "$ph" in ''|*[!0-9]*) ph=0 ;; esac
-        # Both or neither: a lone dimension is worse than none, because the
-        # browser derives the missing one from it and gets the aspect wrong.
-        if [ "$pw" -gt 0 ] && [ "$ph" -gt 0 ]; then w=$pw; h=$ph; fi
-        if [ "$w" = 0 ]; then
-            echo "fetch-media: no dimensions for $name; layout will shift on load" >&2
-        fi
+    probe_dims "$dest"
+
+    # A self-hosted video has no Lemmy thumbnail to mirror when the instance
+    # cannot decode it — AV1 is the common case, since pict-rs will not generate
+    # a still from one. publish-media.sh uploads a poster frame alongside the
+    # video under the video's own hash, so look for that sibling and offer it to
+    # the rewrite below. Without a poster a preload="metadata" video is a black
+    # box until someone presses play.
+    case "$name" in
+        *.mp4|*.webm|*.mov)
+            sibling="${name%.*}.poster.webp"
+            if [ -f "$MEDIA_DIR/$sibling" ]; then
+                jq --arg k "/media/$name" --arg v "/media/$sibling" \
+                   '. + [{key: $k, value: $v}]' "$POSTERMAP" > "$POSTERMAP.new" \
+                   && mv "$POSTERMAP.new" "$POSTERMAP"
             fi
+            ;;
+    esac
 
     jq --arg src "$src" --arg path "/media/$name" \
        --argjson w "${w:-0}" --argjson h "${h:-0}" \
@@ -129,14 +206,15 @@ done <<EOF
 $(jq -r '[.[].media[]? | .src, (.poster // empty)] | map(select(. != "")) | unique[]' "$POSTS")
 EOF
 
-echo "fetch-media: $downloaded new, $reused already present, $failed failed"
+echo "fetch-media: $downloaded new, $reused already present, $adopted self-hosted, $failed failed"
 
 # Rewrite each media entry to the local path. An entry with no mapping (download
 # failed) keeps its original src, so the page still shows something rather than
 # silently dropping the post's whole point.
 TMP_POSTS="$(mktemp)"
-if jq --slurpfile map "$MAP" '
+if jq --slurpfile map "$MAP" --slurpfile posters "$POSTERMAP" '
     ($map[0] | map({key: .src, value: .}) | from_entries) as $m
+    | ($posters[0] | from_entries) as $pm
     | map(.media = ((.media // []) | map(
         . as $item
         | ($m[$item.src] // null) as $hit
@@ -148,7 +226,14 @@ if jq --slurpfile map "$MAP" '
         # poster'\''s size to the <video> element would set the wrong aspect ratio.
         | if (.poster // "") == "" then .
           else . + { poster: (($m[.poster].path) // .poster) }
-          end)))
+          end
+        # Last resort, and only for media we host: the sibling poster frame
+        # publish-media.sh uploaded next to the video. Runs after the rewrite
+        # above so it sees the LOCAL src, and only fills a poster that is still
+        # empty — a thumbnail the instance did provide always wins.
+        | if ((.poster // "") == "") and (($pm[.src] // "") != "")
+          then . + { poster: $pm[.src] }
+          else . end)))
     ' "$POSTS" > "$TMP_POSTS" 2>/dev/null; then
     # Same reason as the chmod on each mirrored file: mktemp is 0600 and the
     # mode survives to production, where other users must read this.

tools/publish-media.sh

@@ -0,0 +1,152 @@
+#!/bin/sh
+# Publish a recording or screenshot to catcrafts.net and print the URL to put in
+# the fediverse post.
+#
+#   usage: tools/publish-media.sh [-r] [-c CRF] <file>...
+#
+#     -r, --raw       upload the file unchanged (no transcode, no poster)
+#     -c, --crf N     AV1 quality, lower is better (default 32)
+#
+# WHY THIS EXISTS: the media used to be uploaded to a third-party file host and
+# then mirrored back at build time by tools/fetch-media.sh. That worked until a
+# 167 MB recording hit the mirror's size cap, and the deploy failed on a file we
+# had sitting on disk the whole time. Publishing to our own origin FIRST inverts
+# it — the URL in the post is already the URL the page wants, so there is nothing
+# to download, no size cap to clear, and no third party who can be slow, rate-
+# limit us, or delete the file that IS the post.
+#
+# The name is the content hash, matching what fetch-media.sh produces, which is
+# what makes Caddy's one-year immutable cache on /media honest: different bytes
+# can never appear under a name someone already has cached.
+#
+# TRANSCODE: phone recordings are enormous for what they show — the one that
+# started this was 78 s of a dark room at 17 Mbps, 167 MB, where denoising and
+# AV1 give the same picture in 15 MB. Dark handheld footage is mostly sensor
+# noise, which is expensive to encode and worth nothing, so hqdn3d earns its
+# place before the encoder rather than after it.
+#
+# ROTATION: phones record landscape and attach a display matrix instead of
+# rotating pixels. Transcoding bakes the rotation into the frames, so what every
+# downstream consumer measures is what the viewer actually sees — see the
+# rotation note in fetch-media.sh for what goes wrong when it does not.
+#
+# NOTE ON AV1: nothing here emits a fallback encoding, and the <video> tag the
+# site renders carries a single src. Browsers without AV1 (Safari before 17, and
+# Apple hardware older than A17/M3) get an element that will not play rather than
+# a degraded one. Pass --raw, or re-encode to H.264, if that audience matters for
+# a particular post.
+
+set -eu
+
+MEDIA_HOST="${MEDIA_HOST:-root@catcrafts.net}"
+MEDIA_PATH="${MEDIA_PATH:-/srv/catcrafts-app/media}"
+SITE_ORIGIN="${SITE_ORIGIN:-https://catcrafts.net}"
+CRF="${CRF:-32}"
+RAW=0
+
+for c in ffmpeg ffprobe sha256sum ssh scp; do
+    command -v "$c" >/dev/null 2>&1 || { echo "publish-media: $c not found" >&2; exit 1; }
+done
+
+FILES=""
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -r|--raw)  RAW=1; shift ;;
+        -c|--crf)  CRF="$2"; shift 2 ;;
+        -h|--help) sed -n '2,10p' "$0"; exit 0 ;;
+        -*)        echo "publish-media: unknown option: $1" >&2; exit 1 ;;
+        *)         FILES="$FILES $1"; shift ;;
+    esac
+done
+[ -n "$FILES" ] || { echo "publish-media: no input file. See --help." >&2; exit 1; }
+
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+
+# Upload only what is not already there. The name is the content hash, so a name
+# that exists on the mount holds these exact bytes and re-sending them would
+# change nothing — which makes re-running this script on the same file free, and
+# makes a half-finished batch safe to just run again.
+publish_file() {
+    local_file=$1
+    remote_name=$2
+    if ssh -o BatchMode=yes "$MEDIA_HOST" "test -f '$MEDIA_PATH/$remote_name'"; then
+        echo "  already published: $remote_name"
+        return 0
+    fi
+    # Land it under a temporary name and move it into place, so a reader can
+    # never see a partial file under the name its hash promises.
+    scp -q -o BatchMode=yes "$local_file" "$MEDIA_HOST:$MEDIA_PATH/.$remote_name.part"
+    ssh -o BatchMode=yes "$MEDIA_HOST" \
+        "chmod 0644 '$MEDIA_PATH/.$remote_name.part' && \
+         mv -f '$MEDIA_PATH/.$remote_name.part' '$MEDIA_PATH/$remote_name'"
+    echo "  uploaded: $remote_name"
+}
+
+for src in $FILES; do
+    [ -f "$src" ] || { echo "publish-media: no such file: $src" >&2; exit 1; }
+    echo "$src"
+
+    ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z')
+    kind=other
+    case "$ext" in
+        mp4|webm|mov|mkv|avi) kind=video ;;
+        png|jpg|jpeg|webp|gif|avif) kind=image ;;
+    esac
+
+    poster=""
+    if [ "$RAW" = 1 ] || [ "$kind" = other ]; then
+        payload="$src"
+        out_ext="$ext"
+    elif [ "$kind" = video ]; then
+        payload="$WORK/out.mp4"
+        out_ext=mp4
+        echo "  transcoding to AV1 (crf $CRF)…"
+        ffmpeg -nostdin -v error -i "$src" \
+               -vf "hqdn3d=4:3:6:4.5" \
+               -c:v libsvtav1 -preset 4 -crf "$CRF" -pix_fmt yuv420p -g 240 \
+               -movflags +faststart \
+               -c:a aac -b:a 96k -ac 1 \
+               "$payload" -y
+        # AAC rather than Opus on purpose: Opus-in-MP4 is still uneven in exactly
+        # the players most likely to be shaky about AV1 anyway, and the audio is
+        # a rounding error next to the video either way.
+        poster="$WORK/poster.webp"
+        # A frame a third of the way in, which beats frame 0 — recordings tend to
+        # open on a lock screen, a hand moving into place, or a fade.
+        dur=$(ffprobe -v error -show_entries format=duration \
+                      -of default=nw=1:nk=1 "$payload" 2>/dev/null || echo 0)
+        at=$(awk -v d="$dur" 'BEGIN { printf "%.2f", (d > 3 ? d / 3 : 0) }')
+        ffmpeg -nostdin -v error -ss "$at" -i "$payload" -frames:v 1 \
+               -vf "scale=720:-2" -c:v libwebp -quality 80 "$poster" -y
+    else
+        payload="$WORK/out.webp"
+        out_ext=webp
+        echo "  converting to webp…"
+        ffmpeg -nostdin -v error -i "$src" -c:v libwebp -quality 82 "$payload" -y
+    fi
+
+    hash=$(sha256sum "$payload" | cut -c1-16)
+    name="$hash.$out_ext"
+    publish_file "$payload" "$name"
+
+    # The poster is named after the VIDEO's hash, not its own. That is the whole
+    # point: fetch-media.sh finds it by name, with nothing to look up, when the
+    # instance did not manage to make a thumbnail — pict-rs will not read AV1, so
+    # for these uploads that is the normal case rather than the exception.
+    if [ -n "$poster" ] && [ -f "$poster" ]; then
+        publish_file "$poster" "$hash.poster.webp"
+    fi
+
+    url="$SITE_ORIGIN/media/$name"
+    # Confirm it is actually being served before handing over a URL that is about
+    # to be pasted into a post, where a 404 is public and permanent.
+    code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$url" || echo 000)
+    if [ "$code" = 200 ]; then
+        printf '  %s\n' "$url"
+    else
+        echo "  WARNING: $url answered HTTP $code — do not post this link yet" >&2
+    fi
+    printf '  %s -> %s (%s)\n' \
+        "$(du -h "$src" | cut -f1)" "$(du -h "$payload" | cut -f1)" "$name"
+done