media: publish to our own origin instead of mirroring a file host
All checks were successful
Deploy / build-deploy (push) Successful in 7m34s
All checks were successful
Deploy / build-deploy (push) Successful in 7m34s
A 167 MB screen recording uploaded to catbox.moe hit fetch-media.sh's 64 MB
MAX_BYTES, so curl refused it, the entry kept its third-party URL, and e2e
failed "/posts media origin" on a file we had on disk the whole time. Raising
the cap would have papered over it: the mirror step still depends on someone
else's server being up, fast, and still holding the file.
Invert it. tools/publish-media.sh uploads a recording to the media mount under
its content hash BEFORE the post exists and prints the URL to post, and
fetch-media.sh adopts an own-origin URL by rewriting it to /media/<hash> with no
request at all — no size cap, no third party in the build.
Also here, because publishing exposed them:
* Rotation. Phones record 1920x1080 and attach a display matrix rather than
rotating pixels, so an untouched file reports landscape while playing
portrait and width/height reserve exactly the wrong box. publish-media.sh
bakes rotation into the frames; fetch-media.sh swaps the dimensions on a
quarter-turn matrix for anything mirrored straight from a phone.
* Posters. pict-rs will not thumbnail AV1, so a self-hosted video usually
arrives with no poster. publish-media.sh uploads <hash>.poster.webp beside
the video and fetch-media.sh falls back to it — a real thumbnail still wins.
* The workflow comment claiming a file on the mount is never downloaded again
was wrong: the name is the hash of the bytes, so third-party media is
re-fetched every build and only the write is skipped.
Transcoding to AV1 is what makes self-hosting cheap: that clip was 78 s of a
dark room at 17 Mbps, and denoise + AV1 gives the same picture in 15 MB. Note
<video> carries a single src with no fallback, so AV1 excludes Safari < 17;
--raw skips the transcode when that matters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
fca089c20d
commit
c1b0c29af6
4 changed files with 351 additions and 66 deletions
|
|
@ -97,10 +97,16 @@ jobs:
|
||||||
# loads is third-party — which is what keeps the privacy notice's
|
# loads is third-party — which is what keeps the privacy notice's
|
||||||
# "everything comes from catcrafts.net" true.
|
# "everything comes from catcrafts.net" true.
|
||||||
#
|
#
|
||||||
# Content-addressed and incremental: a file already on the media mount is
|
# Content-addressed: the name is the hash of the bytes. Note what that
|
||||||
# never downloaded again. Writes straight into the mount so the copies
|
# does NOT mean — a third-party file is re-fetched on every build, because
|
||||||
# persist across deploys — they are NOT always reproducible, because a
|
# the name cannot be known until the bytes are in hand; only the write is
|
||||||
# source instance deleting a file leaves ours as the only one.
|
# skipped when the hash is already on the mount. Media we host ourselves
|
||||||
|
# (tools/publish-media.sh) is the exception that is genuinely incremental:
|
||||||
|
# fetch-media.sh adopts an own-origin URL without any request at all.
|
||||||
|
#
|
||||||
|
# Writes straight into the mount so the copies persist across deploys —
|
||||||
|
# they are NOT always reproducible, because a source instance deleting a
|
||||||
|
# file leaves ours as the only one.
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
if [ -d /deploy-app ]; then
|
if [ -d /deploy-app ]; then
|
||||||
|
|
|
||||||
|
|
@ -118,8 +118,9 @@ Only listed communities are fetched, so joining a new one does not silently
|
||||||
publish it to the site — add a line first.
|
publish it to the site — add a line first.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
tools/fetch-posts.sh # writes content/posts.json
|
tools/publish-media.sh FILE # BEFORE posting: uploads to /media, prints the URL
|
||||||
tools/fetch-media.sh # mirrors the media, rewrites posts.json to /media/... paths
|
tools/fetch-posts.sh # writes content/posts.json
|
||||||
|
tools/fetch-media.sh # mirrors the media, rewrites posts.json to /media/... paths
|
||||||
```
|
```
|
||||||
|
|
||||||
Order matters: the second reads what the first wrote. CI runs both before the
|
Order matters: the second reads what the first wrote. CI runs both before the
|
||||||
|
|
@ -127,6 +128,47 @@ build. Neither fails the build on a network error — a fediverse outage leaves
|
||||||
previous `posts.json` in place, and a single failed download leaves that one entry
|
previous `posts.json` in place, and a single failed download leaves that one entry
|
||||||
pointing at its original URL rather than losing the post.
|
pointing at its original URL rather than losing the post.
|
||||||
|
|
||||||
|
### Publish the media first, then post it
|
||||||
|
|
||||||
|
**The recommended flow is to put a recording on catcrafts.net before writing the
|
||||||
|
post, and use that URL as the post's link.** Run `tools/publish-media.sh
|
||||||
|
recording.mp4`; it transcodes, uploads to the media mount under its content hash,
|
||||||
|
and prints a `https://catcrafts.net/media/<hash>.mp4` URL to paste into the post.
|
||||||
|
|
||||||
|
`fetch-media.sh` recognises its own origin and **adopts** such a URL: it rewrites
|
||||||
|
it to `/media/<hash>` and probes the local file for dimensions, downloading
|
||||||
|
nothing. That is not just an optimisation — it removes the whole class of build
|
||||||
|
failure where the media step depends on a third party. A 167 MB recording on a
|
||||||
|
file host once hit `MAX_BYTES` (64 MB), so the entry kept its original URL, and
|
||||||
|
the deploy then failed the e2e "media origin" check on a file that was sitting on
|
||||||
|
our own disk the entire time.
|
||||||
|
|
||||||
|
The transcode matters as much as the hosting. Phone recordings are wildly
|
||||||
|
oversized for what they show — that same 167 MB clip was 78 s of a dark room at
|
||||||
|
17 Mbps, and denoising into AV1 gives the same picture in 15 MB. It also **bakes
|
||||||
|
in the rotation**: phones record landscape and attach a display matrix, so an
|
||||||
|
untouched file reports 1920x1080 while playing portrait, and the `width`/`height`
|
||||||
|
attributes then reserve exactly the wrong box. (`fetch-media.sh` swaps the
|
||||||
|
dimensions when it sees a quarter-turn matrix, so a straight-from-phone mirror is
|
||||||
|
correct too — but transcoding means nothing downstream has to know.)
|
||||||
|
|
||||||
|
Two things to know about publishing AV1:
|
||||||
|
|
||||||
|
* Nothing emits a fallback encoding and `<video>` carries a single `src`, so
|
||||||
|
browsers without AV1 (Safari before 17, Apple hardware older than A17/M3) get
|
||||||
|
an element that will not play. Use `publish-media.sh --raw`, or a H.264
|
||||||
|
re-encode, when that matters for a particular post.
|
||||||
|
* Lemmy's `pict-rs` will not generate a thumbnail from an AV1 file, so a
|
||||||
|
self-hosted video usually arrives with no `poster`. `publish-media.sh` uploads
|
||||||
|
a poster frame beside the video, named `<video-hash>.poster.webp`, and
|
||||||
|
`fetch-media.sh` falls back to that sibling when the instance supplied nothing.
|
||||||
|
A thumbnail the instance *did* provide always wins.
|
||||||
|
|
||||||
|
An own-origin URL naming a file that is **not** on the mount is deliberately left
|
||||||
|
pointing at its original URL rather than rewritten. That is a post published
|
||||||
|
without its media, and failing the e2e origin check loudly beats shipping a 404
|
||||||
|
inside a `<video>` tag.
|
||||||
|
|
||||||
**`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions,
|
**`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions,
|
||||||
which become the `width`/`height` attributes that stop the page reflowing as
|
which become the `width`/`height` attributes that stop the page reflowing as
|
||||||
several 5 MB recordings arrive. It degrades to no dimensions without it —
|
several 5 MB recordings arrive. It degrades to no dimensions without it —
|
||||||
|
|
|
||||||
|
|
@ -32,6 +32,16 @@ MEDIA_DIR="${1:-media}"
|
||||||
POSTS="content/posts.json"
|
POSTS="content/posts.json"
|
||||||
MAX_BYTES=$((64 * 1024 * 1024))
|
MAX_BYTES=$((64 * 1024 * 1024))
|
||||||
|
|
||||||
|
# Media we host ourselves, published by tools/publish-media.sh before the post
|
||||||
|
# that carries it exists. Such a URL is ALREADY the one the page should use, so
|
||||||
|
# there is nothing to fetch: the bytes are on the media mount, and downloading
|
||||||
|
# them back from our own web server would only mint a second copy under a second
|
||||||
|
# name. Skipping the download also skips MAX_BYTES, which is what a 167 MB
|
||||||
|
# recording on a third-party file host ran into — and it removes the last part of
|
||||||
|
# a build that could fail because someone else's server was slow, rate-limiting
|
||||||
|
# or gone.
|
||||||
|
OWN_ORIGIN="https://catcrafts.net/media/"
|
||||||
|
|
||||||
command -v jq >/dev/null 2>&1 || { echo "fetch-media: jq not found" >&2; exit 1; }
|
command -v jq >/dev/null 2>&1 || { echo "fetch-media: jq not found" >&2; exit 1; }
|
||||||
[ -f "$POSTS" ] || { echo "fetch-media: $POSTS not found — run fetch-posts.sh first" >&2; exit 1; }
|
[ -f "$POSTS" ] || { echo "fetch-media: $POSTS not found — run fetch-posts.sh first" >&2; exit 1; }
|
||||||
|
|
||||||
|
|
@ -44,12 +54,56 @@ mkdir -p "$MEDIA_DIR"
|
||||||
HAVE_FFPROBE=0
|
HAVE_FFPROBE=0
|
||||||
command -v ffprobe >/dev/null 2>&1 && HAVE_FFPROBE=1
|
command -v ffprobe >/dev/null 2>&1 && HAVE_FFPROBE=1
|
||||||
|
|
||||||
|
# Sets $w and $h for the file named in $1, or leaves both 0.
|
||||||
|
#
|
||||||
|
# One query per dimension. Asking for both at once and splitting the CSV looked
|
||||||
|
# simpler but was wrong: for some files ffprobe appends an empty field, so
|
||||||
|
# `width,height` came back as "854x480x" and splitting on `x` gave a height of
|
||||||
|
# "480x" — which the digit guard below then threw away, silently costing the
|
||||||
|
# dimensions of exactly the videos that had the extra field. `nk=1` prints the
|
||||||
|
# bare value, so there is nothing to split.
|
||||||
|
#
|
||||||
|
# ROTATION: a phone records 1920x1080 and attaches a display matrix rather than
|
||||||
|
# rotating the pixels, so the stream reads landscape while the video plays
|
||||||
|
# portrait. Believing the stream there reserves a landscape box for a portrait
|
||||||
|
# video — precisely the layout shift these attributes exist to prevent — so a
|
||||||
|
# quarter-turn swaps them. Files that went through publish-media.sh have the
|
||||||
|
# rotation baked into the pixels and report no matrix at all; this is for
|
||||||
|
# anything mirrored straight from a phone.
|
||||||
|
probe_dims() {
|
||||||
|
w=0; h=0
|
||||||
|
[ "$HAVE_FFPROBE" = 1 ] || return 0
|
||||||
|
pw=$(ffprobe -v error -select_streams v:0 -show_entries stream=width \
|
||||||
|
-of default=nw=1:nk=1 "$1" 2>/dev/null | head -n1 || true)
|
||||||
|
ph=$(ffprobe -v error -select_streams v:0 -show_entries stream=height \
|
||||||
|
-of default=nw=1:nk=1 "$1" 2>/dev/null | head -n1 || true)
|
||||||
|
rot=$(ffprobe -v error -select_streams v:0 \
|
||||||
|
-show_entries stream_side_data=rotation \
|
||||||
|
-of default=nw=1:nk=1 "$1" 2>/dev/null | head -n1 || true)
|
||||||
|
case "$pw" in ''|*[!0-9]*) pw=0 ;; esac
|
||||||
|
case "$ph" in ''|*[!0-9]*) ph=0 ;; esac
|
||||||
|
# ffprobe reports this as a signed number that some builds print with a
|
||||||
|
# fractional part ("-90.000000"), so compare on the integer portion.
|
||||||
|
case "${rot%%.*}" in
|
||||||
|
90|-90|270|-270) t=$pw; pw=$ph; ph=$t ;;
|
||||||
|
esac
|
||||||
|
# Both or neither: a lone dimension is worse than none, because the browser
|
||||||
|
# derives the missing one from it and gets the aspect wrong.
|
||||||
|
if [ "$pw" -gt 0 ] && [ "$ph" -gt 0 ]; then w=$pw; h=$ph; fi
|
||||||
|
if [ "$w" = 0 ]; then
|
||||||
|
echo "fetch-media: no dimensions for $1; layout will shift on load" >&2
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
MAP="$(mktemp)"
|
MAP="$(mktemp)"
|
||||||
trap 'rm -f "$MAP"' EXIT
|
POSTERMAP="$(mktemp)"
|
||||||
|
trap 'rm -f "$MAP" "$POSTERMAP"' EXIT
|
||||||
printf '[]' > "$MAP"
|
printf '[]' > "$MAP"
|
||||||
|
printf '[]' > "$POSTERMAP"
|
||||||
|
|
||||||
downloaded=0
|
downloaded=0
|
||||||
reused=0
|
reused=0
|
||||||
|
adopted=0
|
||||||
failed=0
|
failed=0
|
||||||
|
|
||||||
# Every distinct media URL across all posts, so a file shared by two posts is
|
# Every distinct media URL across all posts, so a file shared by two posts is
|
||||||
|
|
@ -62,64 +116,87 @@ failed=0
|
||||||
while IFS= read -r src; do
|
while IFS= read -r src; do
|
||||||
[ -n "$src" ] || continue
|
[ -n "$src" ] || continue
|
||||||
|
|
||||||
ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z')
|
# Ours already — adopt the file on the mount and do no network at all.
|
||||||
case "$ext" in
|
# A URL that is on our origin but names a file that is NOT on the mount is
|
||||||
mp4|webm|mov|webp|png|jpg|jpeg|gif|avif) ;;
|
# left alone rather than invented: that is a post published without its
|
||||||
*) echo "fetch-media: skipping unexpected extension: $src" >&2; continue ;;
|
# media, and keeping the original URL makes the e2e origin check fail
|
||||||
|
# loudly instead of shipping a 404 in a <video> tag.
|
||||||
|
case "$src" in
|
||||||
|
"$OWN_ORIGIN"*)
|
||||||
|
name=${src#"$OWN_ORIGIN"}
|
||||||
|
# Refuse anything that is not a bare filename. A path separator or a
|
||||||
|
# traversal segment arriving from a post URL must never reach a path we
|
||||||
|
# then read or publish.
|
||||||
|
case "$name" in
|
||||||
|
''|*/*|*..*)
|
||||||
|
echo "fetch-media: refusing suspicious own-origin URL: $src" >&2
|
||||||
|
failed=$((failed + 1)); continue ;;
|
||||||
|
esac
|
||||||
|
dest="$MEDIA_DIR/$name"
|
||||||
|
if [ ! -f "$dest" ]; then
|
||||||
|
echo "fetch-media: $name not on the media mount, keeping original URL: $src" >&2
|
||||||
|
failed=$((failed + 1)); continue
|
||||||
|
fi
|
||||||
|
adopted=$((adopted + 1))
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z')
|
||||||
|
case "$ext" in
|
||||||
|
mp4|webm|mov|webp|png|jpg|jpeg|gif|avif) ;;
|
||||||
|
*) echo "fetch-media: skipping unexpected extension: $src" >&2; continue ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
tmp="$(mktemp)"
|
||||||
|
# --max-filesize refuses an oversized body before writing it; the explicit
|
||||||
|
# size check afterwards covers servers that do not send Content-Length.
|
||||||
|
if ! curl -fsSL --max-time 120 --max-filesize "$MAX_BYTES" \
|
||||||
|
-A 'catcrafts.net-buildfetch/1.0 (+https://catcrafts.net)' \
|
||||||
|
"$src" -o "$tmp" 2>/dev/null; then
|
||||||
|
echo "fetch-media: download failed, keeping original URL: $src" >&2
|
||||||
|
rm -f "$tmp"
|
||||||
|
failed=$((failed + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if [ "$(wc -c < "$tmp")" -gt "$MAX_BYTES" ]; then
|
||||||
|
echo "fetch-media: oversized, keeping original URL: $src" >&2
|
||||||
|
rm -f "$tmp"
|
||||||
|
failed=$((failed + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
hash=$(sha256sum "$tmp" | cut -c1-16)
|
||||||
|
name="$hash.$ext"
|
||||||
|
dest="$MEDIA_DIR/$name"
|
||||||
|
|
||||||
|
if [ -f "$dest" ]; then
|
||||||
|
rm -f "$tmp"
|
||||||
|
reused=$((reused + 1))
|
||||||
|
else
|
||||||
|
mv "$tmp" "$dest"
|
||||||
|
chmod 0644 "$dest"
|
||||||
|
downloaded=$((downloaded + 1))
|
||||||
|
fi
|
||||||
|
;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
tmp="$(mktemp)"
|
probe_dims "$dest"
|
||||||
# --max-filesize refuses an oversized body before writing it; the explicit
|
|
||||||
# size check afterwards covers servers that do not send Content-Length.
|
|
||||||
if ! curl -fsSL --max-time 120 --max-filesize "$MAX_BYTES" \
|
|
||||||
-A 'catcrafts.net-buildfetch/1.0 (+https://catcrafts.net)' \
|
|
||||||
"$src" -o "$tmp" 2>/dev/null; then
|
|
||||||
echo "fetch-media: download failed, keeping original URL: $src" >&2
|
|
||||||
rm -f "$tmp"
|
|
||||||
failed=$((failed + 1))
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
if [ "$(wc -c < "$tmp")" -gt "$MAX_BYTES" ]; then
|
|
||||||
echo "fetch-media: oversized, keeping original URL: $src" >&2
|
|
||||||
rm -f "$tmp"
|
|
||||||
failed=$((failed + 1))
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
|
|
||||||
hash=$(sha256sum "$tmp" | cut -c1-16)
|
# A self-hosted video has no Lemmy thumbnail to mirror when the instance
|
||||||
name="$hash.$ext"
|
# cannot decode it — AV1 is the common case, since pict-rs will not generate
|
||||||
dest="$MEDIA_DIR/$name"
|
# a still from one. publish-media.sh uploads a poster frame alongside the
|
||||||
|
# video under the video's own hash, so look for that sibling and offer it to
|
||||||
if [ -f "$dest" ]; then
|
# the rewrite below. Without a poster a preload="metadata" video is a black
|
||||||
rm -f "$tmp"
|
# box until someone presses play.
|
||||||
reused=$((reused + 1))
|
case "$name" in
|
||||||
else
|
*.mp4|*.webm|*.mov)
|
||||||
mv "$tmp" "$dest"
|
sibling="${name%.*}.poster.webp"
|
||||||
chmod 0644 "$dest"
|
if [ -f "$MEDIA_DIR/$sibling" ]; then
|
||||||
downloaded=$((downloaded + 1))
|
jq --arg k "/media/$name" --arg v "/media/$sibling" \
|
||||||
fi
|
'. + [{key: $k, value: $v}]' "$POSTERMAP" > "$POSTERMAP.new" \
|
||||||
|
&& mv "$POSTERMAP.new" "$POSTERMAP"
|
||||||
# One query per dimension. Asking for both at once and splitting the CSV
|
fi
|
||||||
# looked simpler but was wrong: for some files ffprobe appends an empty
|
;;
|
||||||
# field, so `width,height` came back as "854x480x" and splitting on `x` gave
|
esac
|
||||||
# a height of "480x" — which the digit guard below then threw away, silently
|
|
||||||
# costing the dimensions of exactly the videos that had the extra field.
|
|
||||||
# `nk=1` prints the bare value, so there is nothing to split.
|
|
||||||
w=0; h=0
|
|
||||||
if [ "$HAVE_FFPROBE" = 1 ]; then
|
|
||||||
pw=$(ffprobe -v error -select_streams v:0 -show_entries stream=width \
|
|
||||||
-of default=nw=1:nk=1 "$dest" 2>/dev/null | head -n1 || true)
|
|
||||||
ph=$(ffprobe -v error -select_streams v:0 -show_entries stream=height \
|
|
||||||
-of default=nw=1:nk=1 "$dest" 2>/dev/null | head -n1 || true)
|
|
||||||
case "$pw" in ''|*[!0-9]*) pw=0 ;; esac
|
|
||||||
case "$ph" in ''|*[!0-9]*) ph=0 ;; esac
|
|
||||||
# Both or neither: a lone dimension is worse than none, because the
|
|
||||||
# browser derives the missing one from it and gets the aspect wrong.
|
|
||||||
if [ "$pw" -gt 0 ] && [ "$ph" -gt 0 ]; then w=$pw; h=$ph; fi
|
|
||||||
if [ "$w" = 0 ]; then
|
|
||||||
echo "fetch-media: no dimensions for $name; layout will shift on load" >&2
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
jq --arg src "$src" --arg path "/media/$name" \
|
jq --arg src "$src" --arg path "/media/$name" \
|
||||||
--argjson w "${w:-0}" --argjson h "${h:-0}" \
|
--argjson w "${w:-0}" --argjson h "${h:-0}" \
|
||||||
|
|
@ -129,14 +206,15 @@ done <<EOF
|
||||||
$(jq -r '[.[].media[]? | .src, (.poster // empty)] | map(select(. != "")) | unique[]' "$POSTS")
|
$(jq -r '[.[].media[]? | .src, (.poster // empty)] | map(select(. != "")) | unique[]' "$POSTS")
|
||||||
EOF
|
EOF
|
||||||
|
|
||||||
echo "fetch-media: $downloaded new, $reused already present, $failed failed"
|
echo "fetch-media: $downloaded new, $reused already present, $adopted self-hosted, $failed failed"
|
||||||
|
|
||||||
# Rewrite each media entry to the local path. An entry with no mapping (download
|
# Rewrite each media entry to the local path. An entry with no mapping (download
|
||||||
# failed) keeps its original src, so the page still shows something rather than
|
# failed) keeps its original src, so the page still shows something rather than
|
||||||
# silently dropping the post's whole point.
|
# silently dropping the post's whole point.
|
||||||
TMP_POSTS="$(mktemp)"
|
TMP_POSTS="$(mktemp)"
|
||||||
if jq --slurpfile map "$MAP" '
|
if jq --slurpfile map "$MAP" --slurpfile posters "$POSTERMAP" '
|
||||||
($map[0] | map({key: .src, value: .}) | from_entries) as $m
|
($map[0] | map({key: .src, value: .}) | from_entries) as $m
|
||||||
|
| ($posters[0] | from_entries) as $pm
|
||||||
| map(.media = ((.media // []) | map(
|
| map(.media = ((.media // []) | map(
|
||||||
. as $item
|
. as $item
|
||||||
| ($m[$item.src] // null) as $hit
|
| ($m[$item.src] // null) as $hit
|
||||||
|
|
@ -148,7 +226,14 @@ if jq --slurpfile map "$MAP" '
|
||||||
# poster'\''s size to the <video> element would set the wrong aspect ratio.
|
# poster'\''s size to the <video> element would set the wrong aspect ratio.
|
||||||
| if (.poster // "") == "" then .
|
| if (.poster // "") == "" then .
|
||||||
else . + { poster: (($m[.poster].path) // .poster) }
|
else . + { poster: (($m[.poster].path) // .poster) }
|
||||||
end)))
|
end
|
||||||
|
# Last resort, and only for media we host: the sibling poster frame
|
||||||
|
# publish-media.sh uploaded next to the video. Runs after the rewrite
|
||||||
|
# above so it sees the LOCAL src, and only fills a poster that is still
|
||||||
|
# empty — a thumbnail the instance did provide always wins.
|
||||||
|
| if ((.poster // "") == "") and (($pm[.src] // "") != "")
|
||||||
|
then . + { poster: $pm[.src] }
|
||||||
|
else . end)))
|
||||||
' "$POSTS" > "$TMP_POSTS" 2>/dev/null; then
|
' "$POSTS" > "$TMP_POSTS" 2>/dev/null; then
|
||||||
# Same reason as the chmod on each mirrored file: mktemp is 0600 and the
|
# Same reason as the chmod on each mirrored file: mktemp is 0600 and the
|
||||||
# mode survives to production, where other users must read this.
|
# mode survives to production, where other users must read this.
|
||||||
|
|
|
||||||
152
tools/publish-media.sh
Executable file
152
tools/publish-media.sh
Executable file
|
|
@ -0,0 +1,152 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Publish a recording or screenshot to catcrafts.net and print the URL to put in
|
||||||
|
# the fediverse post.
|
||||||
|
#
|
||||||
|
# usage: tools/publish-media.sh [-r] [-c CRF] <file>...
|
||||||
|
#
|
||||||
|
# -r, --raw upload the file unchanged (no transcode, no poster)
|
||||||
|
# -c, --crf N AV1 quality, lower is better (default 32)
|
||||||
|
#
|
||||||
|
# WHY THIS EXISTS: the media used to be uploaded to a third-party file host and
|
||||||
|
# then mirrored back at build time by tools/fetch-media.sh. That worked until a
|
||||||
|
# 167 MB recording hit the mirror's size cap, and the deploy failed on a file we
|
||||||
|
# had sitting on disk the whole time. Publishing to our own origin FIRST inverts
|
||||||
|
# it — the URL in the post is already the URL the page wants, so there is nothing
|
||||||
|
# to download, no size cap to clear, and no third party who can be slow, rate-
|
||||||
|
# limit us, or delete the file that IS the post.
|
||||||
|
#
|
||||||
|
# The name is the content hash, matching what fetch-media.sh produces, which is
|
||||||
|
# what makes Caddy's one-year immutable cache on /media honest: different bytes
|
||||||
|
# can never appear under a name someone already has cached.
|
||||||
|
#
|
||||||
|
# TRANSCODE: phone recordings are enormous for what they show — the one that
|
||||||
|
# started this was 78 s of a dark room at 17 Mbps, 167 MB, where denoising and
|
||||||
|
# AV1 give the same picture in 15 MB. Dark handheld footage is mostly sensor
|
||||||
|
# noise, which is expensive to encode and worth nothing, so hqdn3d earns its
|
||||||
|
# place before the encoder rather than after it.
|
||||||
|
#
|
||||||
|
# ROTATION: phones record landscape and attach a display matrix instead of
|
||||||
|
# rotating pixels. Transcoding bakes the rotation into the frames, so what every
|
||||||
|
# downstream consumer measures is what the viewer actually sees — see the
|
||||||
|
# rotation note in fetch-media.sh for what goes wrong when it does not.
|
||||||
|
#
|
||||||
|
# NOTE ON AV1: nothing here emits a fallback encoding, and the <video> tag the
|
||||||
|
# site renders carries a single src. Browsers without AV1 (Safari before 17, and
|
||||||
|
# Apple hardware older than A17/M3) get an element that will not play rather than
|
||||||
|
# a degraded one. Pass --raw, or re-encode to H.264, if that audience matters for
|
||||||
|
# a particular post.
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
MEDIA_HOST="${MEDIA_HOST:-root@catcrafts.net}"
|
||||||
|
MEDIA_PATH="${MEDIA_PATH:-/srv/catcrafts-app/media}"
|
||||||
|
SITE_ORIGIN="${SITE_ORIGIN:-https://catcrafts.net}"
|
||||||
|
CRF="${CRF:-32}"
|
||||||
|
RAW=0
|
||||||
|
|
||||||
|
for c in ffmpeg ffprobe sha256sum ssh scp; do
|
||||||
|
command -v "$c" >/dev/null 2>&1 || { echo "publish-media: $c not found" >&2; exit 1; }
|
||||||
|
done
|
||||||
|
|
||||||
|
FILES=""
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
-r|--raw) RAW=1; shift ;;
|
||||||
|
-c|--crf) CRF="$2"; shift 2 ;;
|
||||||
|
-h|--help) sed -n '2,10p' "$0"; exit 0 ;;
|
||||||
|
-*) echo "publish-media: unknown option: $1" >&2; exit 1 ;;
|
||||||
|
*) FILES="$FILES $1"; shift ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[ -n "$FILES" ] || { echo "publish-media: no input file. See --help." >&2; exit 1; }
|
||||||
|
|
||||||
|
WORK="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$WORK"' EXIT
|
||||||
|
|
||||||
|
# Upload only what is not already there. The name is the content hash, so a name
|
||||||
|
# that exists on the mount holds these exact bytes and re-sending them would
|
||||||
|
# change nothing — which makes re-running this script on the same file free, and
|
||||||
|
# makes a half-finished batch safe to just run again.
|
||||||
|
publish_file() {
|
||||||
|
local_file=$1
|
||||||
|
remote_name=$2
|
||||||
|
if ssh -o BatchMode=yes "$MEDIA_HOST" "test -f '$MEDIA_PATH/$remote_name'"; then
|
||||||
|
echo " already published: $remote_name"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
# Land it under a temporary name and move it into place, so a reader can
|
||||||
|
# never see a partial file under the name its hash promises.
|
||||||
|
scp -q -o BatchMode=yes "$local_file" "$MEDIA_HOST:$MEDIA_PATH/.$remote_name.part"
|
||||||
|
ssh -o BatchMode=yes "$MEDIA_HOST" \
|
||||||
|
"chmod 0644 '$MEDIA_PATH/.$remote_name.part' && \
|
||||||
|
mv -f '$MEDIA_PATH/.$remote_name.part' '$MEDIA_PATH/$remote_name'"
|
||||||
|
echo " uploaded: $remote_name"
|
||||||
|
}
|
||||||
|
|
||||||
|
for src in $FILES; do
|
||||||
|
[ -f "$src" ] || { echo "publish-media: no such file: $src" >&2; exit 1; }
|
||||||
|
echo "$src"
|
||||||
|
|
||||||
|
ext=$(printf '%s' "$src" | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | tr 'A-Z' 'a-z')
|
||||||
|
kind=other
|
||||||
|
case "$ext" in
|
||||||
|
mp4|webm|mov|mkv|avi) kind=video ;;
|
||||||
|
png|jpg|jpeg|webp|gif|avif) kind=image ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
poster=""
|
||||||
|
if [ "$RAW" = 1 ] || [ "$kind" = other ]; then
|
||||||
|
payload="$src"
|
||||||
|
out_ext="$ext"
|
||||||
|
elif [ "$kind" = video ]; then
|
||||||
|
payload="$WORK/out.mp4"
|
||||||
|
out_ext=mp4
|
||||||
|
echo " transcoding to AV1 (crf $CRF)…"
|
||||||
|
ffmpeg -nostdin -v error -i "$src" \
|
||||||
|
-vf "hqdn3d=4:3:6:4.5" \
|
||||||
|
-c:v libsvtav1 -preset 4 -crf "$CRF" -pix_fmt yuv420p -g 240 \
|
||||||
|
-movflags +faststart \
|
||||||
|
-c:a aac -b:a 96k -ac 1 \
|
||||||
|
"$payload" -y
|
||||||
|
# AAC rather than Opus on purpose: Opus-in-MP4 is still uneven in exactly
|
||||||
|
# the players most likely to be shaky about AV1 anyway, and the audio is
|
||||||
|
# a rounding error next to the video either way.
|
||||||
|
poster="$WORK/poster.webp"
|
||||||
|
# A frame a third of the way in, which beats frame 0 — recordings tend to
|
||||||
|
# open on a lock screen, a hand moving into place, or a fade.
|
||||||
|
dur=$(ffprobe -v error -show_entries format=duration \
|
||||||
|
-of default=nw=1:nk=1 "$payload" 2>/dev/null || echo 0)
|
||||||
|
at=$(awk -v d="$dur" 'BEGIN { printf "%.2f", (d > 3 ? d / 3 : 0) }')
|
||||||
|
ffmpeg -nostdin -v error -ss "$at" -i "$payload" -frames:v 1 \
|
||||||
|
-vf "scale=720:-2" -c:v libwebp -quality 80 "$poster" -y
|
||||||
|
else
|
||||||
|
payload="$WORK/out.webp"
|
||||||
|
out_ext=webp
|
||||||
|
echo " converting to webp…"
|
||||||
|
ffmpeg -nostdin -v error -i "$src" -c:v libwebp -quality 82 "$payload" -y
|
||||||
|
fi
|
||||||
|
|
||||||
|
hash=$(sha256sum "$payload" | cut -c1-16)
|
||||||
|
name="$hash.$out_ext"
|
||||||
|
publish_file "$payload" "$name"
|
||||||
|
|
||||||
|
# The poster is named after the VIDEO's hash, not its own. That is the whole
|
||||||
|
# point: fetch-media.sh finds it by name, with nothing to look up, when the
|
||||||
|
# instance did not manage to make a thumbnail — pict-rs will not read AV1, so
|
||||||
|
# for these uploads that is the normal case rather than the exception.
|
||||||
|
if [ -n "$poster" ] && [ -f "$poster" ]; then
|
||||||
|
publish_file "$poster" "$hash.poster.webp"
|
||||||
|
fi
|
||||||
|
|
||||||
|
url="$SITE_ORIGIN/media/$name"
|
||||||
|
# Confirm it is actually being served before handing over a URL that is about
|
||||||
|
# to be pasted into a post, where a 404 is public and permanent.
|
||||||
|
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$url" || echo 000)
|
||||||
|
if [ "$code" = 200 ]; then
|
||||||
|
printf ' %s\n' "$url"
|
||||||
|
else
|
||||||
|
echo " WARNING: $url answered HTTP $code — do not post this link yet" >&2
|
||||||
|
fi
|
||||||
|
printf ' %s -> %s (%s)\n' \
|
||||||
|
"$(du -h "$src" | cut -f1)" "$(du -h "$payload" | cut -f1)" "$name"
|
||||||
|
done
|
||||||
Loading…
Reference in a new issue