media: publish to our own origin instead of mirroring a file host
All checks were successful
Deploy / build-deploy (push) Successful in 7m34s
All checks were successful
Deploy / build-deploy (push) Successful in 7m34s
A 167 MB screen recording uploaded to catbox.moe hit fetch-media.sh's 64 MB
MAX_BYTES, so curl refused it, the entry kept its third-party URL, and e2e
failed "/posts media origin" on a file we had on disk the whole time. Raising
the cap would have papered over it: the mirror step still depends on someone
else's server being up, fast, and still holding the file.
Invert it. tools/publish-media.sh uploads a recording to the media mount under
its content hash BEFORE the post exists and prints the URL to post, and
fetch-media.sh adopts an own-origin URL by rewriting it to /media/<hash> with no
request at all — no size cap, no third party in the build.
Also here, because publishing exposed them:
* Rotation. Phones record 1920x1080 and attach a display matrix rather than
rotating pixels, so an untouched file reports landscape while playing
portrait and width/height reserve exactly the wrong box. publish-media.sh
bakes rotation into the frames; fetch-media.sh swaps the dimensions on a
quarter-turn matrix for anything mirrored straight from a phone.
* Posters. pict-rs will not thumbnail AV1, so a self-hosted video usually
arrives with no poster. publish-media.sh uploads <hash>.poster.webp beside
the video and fetch-media.sh falls back to it — a real thumbnail still wins.
* The workflow comment claiming a file on the mount is never downloaded again
was wrong: the name is the hash of the bytes, so third-party media is
re-fetched every build and only the write is skipped.
Transcoding to AV1 is what makes self-hosting cheap: that clip was 78 s of a
dark room at 17 Mbps, and denoise + AV1 gives the same picture in 15 MB. Note
<video> carries a single src with no fallback, so AV1 excludes Safari < 17;
--raw skips the transcode when that matters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
fca089c20d
commit
c1b0c29af6
4 changed files with 351 additions and 66 deletions
152
tools/publish-media.sh
Executable file
152
tools/publish-media.sh
Executable file
|
|
@ -0,0 +1,152 @@
|
|||
#!/bin/sh
|
||||
# Publish a recording or screenshot to catcrafts.net and print the URL to put in
|
||||
# the fediverse post.
|
||||
#
|
||||
# usage: tools/publish-media.sh [-r] [-c CRF] <file>...
|
||||
#
|
||||
# -r, --raw upload the file unchanged (no transcode, no poster)
|
||||
# -c, --crf N AV1 quality, lower is better (default 32)
|
||||
#
|
||||
# WHY THIS EXISTS: the media used to be uploaded to a third-party file host and
|
||||
# then mirrored back at build time by tools/fetch-media.sh. That worked until a
|
||||
# 167 MB recording hit the mirror's size cap, and the deploy failed on a file we
|
||||
# had sitting on disk the whole time. Publishing to our own origin FIRST inverts
|
||||
# it — the URL in the post is already the URL the page wants, so there is nothing
|
||||
# to download, no size cap to clear, and no third party who can be slow, rate-
|
||||
# limit us, or delete the file that IS the post.
|
||||
#
|
||||
# The name is the content hash, matching what fetch-media.sh produces, which is
|
||||
# what makes Caddy's one-year immutable cache on /media honest: different bytes
|
||||
# can never appear under a name someone already has cached.
|
||||
#
|
||||
# TRANSCODE: phone recordings are enormous for what they show — the one that
|
||||
# started this was 78 s of a dark room at 17 Mbps, 167 MB, where denoising and
|
||||
# AV1 give the same picture in 15 MB. Dark handheld footage is mostly sensor
|
||||
# noise, which is expensive to encode and worth nothing, so hqdn3d earns its
|
||||
# place before the encoder rather than after it.
|
||||
#
|
||||
# ROTATION: phones record landscape and attach a display matrix instead of
|
||||
# rotating pixels. Transcoding bakes the rotation into the frames, so what every
|
||||
# downstream consumer measures is what the viewer actually sees — see the
|
||||
# rotation note in fetch-media.sh for what goes wrong when it does not.
|
||||
#
|
||||
# NOTE ON AV1: nothing here emits a fallback encoding, and the <video> tag the
|
||||
# site renders carries a single src. Browsers without AV1 (Safari before 17, and
|
||||
# Apple hardware older than A17/M3) get an element that will not play rather than
|
||||
# a degraded one. Pass --raw, or re-encode to H.264, if that audience matters for
|
||||
# a particular post.
|
||||
|
||||
set -eu
|
||||
|
||||
MEDIA_HOST="${MEDIA_HOST:-root@catcrafts.net}"
|
||||
MEDIA_PATH="${MEDIA_PATH:-/srv/catcrafts-app/media}"
|
||||
SITE_ORIGIN="${SITE_ORIGIN:-https://catcrafts.net}"
|
||||
CRF="${CRF:-32}"
|
||||
RAW=0
|
||||
|
||||
for c in ffmpeg ffprobe sha256sum ssh scp; do
|
||||
command -v "$c" >/dev/null 2>&1 || { echo "publish-media: $c not found" >&2; exit 1; }
|
||||
done
|
||||
|
||||
FILES=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-r|--raw) RAW=1; shift ;;
|
||||
-c|--crf) CRF="$2"; shift 2 ;;
|
||||
-h|--help) sed -n '2,10p' "$0"; exit 0 ;;
|
||||
-*) echo "publish-media: unknown option: $1" >&2; exit 1 ;;
|
||||
*) FILES="$FILES $1"; shift ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$FILES" ] || { echo "publish-media: no input file. See --help." >&2; exit 1; }
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
|
||||
# Upload only what is not already there. The name is the content hash, so a name
|
||||
# that exists on the mount holds these exact bytes and re-sending them would
|
||||
# change nothing — which makes re-running this script on the same file free, and
|
||||
# makes a half-finished batch safe to just run again.
|
||||
publish_file() {
|
||||
local_file=$1
|
||||
remote_name=$2
|
||||
if ssh -o BatchMode=yes "$MEDIA_HOST" "test -f '$MEDIA_PATH/$remote_name'"; then
|
||||
echo " already published: $remote_name"
|
||||
return 0
|
||||
fi
|
||||
# Land it under a temporary name and move it into place, so a reader can
|
||||
# never see a partial file under the name its hash promises.
|
||||
scp -q -o BatchMode=yes "$local_file" "$MEDIA_HOST:$MEDIA_PATH/.$remote_name.part"
|
||||
ssh -o BatchMode=yes "$MEDIA_HOST" \
|
||||
"chmod 0644 '$MEDIA_PATH/.$remote_name.part' && \
|
||||
mv -f '$MEDIA_PATH/.$remote_name.part' '$MEDIA_PATH/$remote_name'"
|
||||
echo " uploaded: $remote_name"
|
||||
}
|
||||
|
||||
for src in $FILES; do
|
||||
[ -f "$src" ] || { echo "publish-media: no such file: $src" >&2; exit 1; }
|
||||
echo "$src"
|
||||
|
||||
ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z')
|
||||
kind=other
|
||||
case "$ext" in
|
||||
mp4|webm|mov|mkv|avi) kind=video ;;
|
||||
png|jpg|jpeg|webp|gif|avif) kind=image ;;
|
||||
esac
|
||||
|
||||
poster=""
|
||||
if [ "$RAW" = 1 ] || [ "$kind" = other ]; then
|
||||
payload="$src"
|
||||
out_ext="$ext"
|
||||
elif [ "$kind" = video ]; then
|
||||
payload="$WORK/out.mp4"
|
||||
out_ext=mp4
|
||||
echo " transcoding to AV1 (crf $CRF)…"
|
||||
ffmpeg -nostdin -v error -i "$src" \
|
||||
-vf "hqdn3d=4:3:6:4.5" \
|
||||
-c:v libsvtav1 -preset 4 -crf "$CRF" -pix_fmt yuv420p -g 240 \
|
||||
-movflags +faststart \
|
||||
-c:a aac -b:a 96k -ac 1 \
|
||||
"$payload" -y
|
||||
# AAC rather than Opus on purpose: Opus-in-MP4 is still uneven in exactly
|
||||
# the players most likely to be shaky about AV1 anyway, and the audio is
|
||||
# a rounding error next to the video either way.
|
||||
poster="$WORK/poster.webp"
|
||||
# A frame a third of the way in, which beats frame 0 — recordings tend to
|
||||
# open on a lock screen, a hand moving into place, or a fade.
|
||||
dur=$(ffprobe -v error -show_entries format=duration \
|
||||
-of default=nw=1:nk=1 "$payload" 2>/dev/null || echo 0)
|
||||
at=$(awk -v d="$dur" 'BEGIN { printf "%.2f", (d > 3 ? d / 3 : 0) }')
|
||||
ffmpeg -nostdin -v error -ss "$at" -i "$payload" -frames:v 1 \
|
||||
-vf "scale=720:-2" -c:v libwebp -quality 80 "$poster" -y
|
||||
else
|
||||
payload="$WORK/out.webp"
|
||||
out_ext=webp
|
||||
echo " converting to webp…"
|
||||
ffmpeg -nostdin -v error -i "$src" -c:v libwebp -quality 82 "$payload" -y
|
||||
fi
|
||||
|
||||
hash=$(sha256sum "$payload" | cut -c1-16)
|
||||
name="$hash.$out_ext"
|
||||
publish_file "$payload" "$name"
|
||||
|
||||
# The poster is named after the VIDEO's hash, not its own. That is the whole
|
||||
# point: fetch-media.sh finds it by name, with nothing to look up, when the
|
||||
# instance did not manage to make a thumbnail — pict-rs will not read AV1, so
|
||||
# for these uploads that is the normal case rather than the exception.
|
||||
if [ -n "$poster" ] && [ -f "$poster" ]; then
|
||||
publish_file "$poster" "$hash.poster.webp"
|
||||
fi
|
||||
|
||||
url="$SITE_ORIGIN/media/$name"
|
||||
# Confirm it is actually being served before handing over a URL that is about
|
||||
# to be pasted into a post, where a 404 is public and permanent.
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$url" || echo 000)
|
||||
if [ "$code" = 200 ]; then
|
||||
printf ' %s\n' "$url"
|
||||
else
|
||||
echo " WARNING: $url answered HTTP $code — do not post this link yet" >&2
|
||||
fi
|
||||
printf ' %s -> %s (%s)\n' \
|
||||
"$(du -h "$src" | cut -f1)" "$(du -h "$payload" | cut -f1)" "$name"
|
||||
done
|
||||
Loading…
Reference in a new issue