catcrafts.net/.forgejo/workflows/deploy.yaml
Jorijn van der Graaf 0bfa715e6a
Some checks failed
Deploy / build-deploy (push) Failing after 6m19s
CI: build with the Crafter.Build Flatpak on a repo-scoped privileged runner
Crafter.Build's CI publishes only the Flatpak since 2026-10-02, so the
rolling `latest` tarball the Install crafter-build step downloaded is gone
and the 2026-10-12 deploy failed with a 404 before building anything.

Take the route imsd's package job takes. The job runs on `flatpak`, a
privileged runner registered to this repository alone
(forgejo-runner-catcrafts on the box, same arrangement as imsd's), with the
installed Flatpak and its Sdk kept on a docker volume between runs. The
container installs nothing that compiles any more: clang, lld, libc++, the
WASI sysroot, cmake, perl, gpg and OpenSSL all come with the Flatpak and its
Sdk. `crafter-build` is a wrapper around `flatpak run`, `crafter-run` runs a
built binary inside the sandbox, cast is installed under /opt where the
sandbox can see it, and the cache path is the Flatpak's own.

The server binary links the Flatpak's libc++, libc++abi and libunwind, which
are newer than the host's, so the deploy ships them next to the binary the
way it already ships libmsquic; the rpath finds them there. They need at
most GLIBC_2.38, the host has 2.41.

ShouldMatchBankTransfers passed braced lists straight into a std::span
parameter. That compiled only because libc++ 22 had the C++26
span(initializer_list) constructor; the libc++ 23 in the Flatpak does not,
so the lists now go through a vector like the rest of the file.

deploy/README.md documents the runner, the sandbox's private /tmp and how
to run a built binary where the host lacks a matching libc++.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-12 16:32:57 +02:00

481 lines
25 KiB
YAML

name: Deploy
on:
push:
branches: [master]
workflow_dispatch:
# One deploy at a time; if you push twice quickly, cancel the older run so the
# newest commit is what lands on the server.
concurrency:
group: deploy
cancel-in-progress: true
jobs:
build-deploy:
# `flatpak` is a privileged runner registered to this repository alone
# (/etc/forgejo-runner/config-catcrafts.yaml on the box, label
# flatpak:docker://archlinux:latest): crafter-build is the Crafter.Build
# Flatpak, and bubblewrap inside the job container needs the privilege.
# The same runner carries the two deploy bind mounts (/deploy, /deploy-app)
# and a docker volume at /var/lib/flatpak that keeps the installed Flatpak
# and its Sdk between runs. Same route as imsd's package job; see
# deploy/README.md "Runner configuration".
runs-on: flatpak
steps:
- name: Check live payment test credentials
# ShouldSettleEurcOnTestnet is a mandatory deploy gate, so a missing
# secret fails HERE, before any build time is spent — never as a silent
# skip. The secret lives in Forgejo: repo Settings -> Actions ->
# Secrets. See deploy/README.md "Live payment suites in CI" for what it
# is and how to keep the testnet wallet funded.
#
# There is ONE live suite now, not two. The bank half used to be a
# hosted provider with a test-mode API, and that gate died with the
# account on 2026-08-20. Its replacement is a transfer to our own
# account, which has no counterpart to call and therefore no credential
# to check: ShouldSettleBankTransfers drives the real rail with no
# network at all and runs unconditionally with the other black-box
# suites. Losing a live gate here is a fact about the rail, not a
# weakening of the standard.
env:
EURC_E2E_PRIVATE_KEY: ${{ secrets.EURC_E2E_PRIVATE_KEY }}
run: |
set -eu
if [ -z "${EURC_E2E_PRIVATE_KEY:-}" ]; then
echo "ERROR: EURC_E2E_PRIVATE_KEY secret is not set (an Ethereum Sepolia key holding testnet EURC + gas)." >&2
exit 1
fi
echo "Live payment test credentials are present."
- name: Install build dependencies
run: |
# Same keyring bootstrap the Crafter.Build CI does: the slim
# archlinux:latest image ships without a populated pacman keyring
# or local master key.
pacman-key --init
pacman-key --populate archlinux
pacman -Sy --noconfirm --needed archlinux-keyring
# nodejs is required for the JS-based actions (checkout, cache) to
# run inside this archlinux container — the runner execs them with
# node. This shell step needs no node, so installing it here (before
# Checkout) is enough.
# ffmpeg/ffprobe are both used by tools/fetch-media.sh. ffprobe reads
# the pixel dimensions of each mirrored file, which become the
# width/height attributes that stop the posts page reflowing as 5 MB
# recordings arrive. ffmpeg transcodes each still image into the AVIF
# and PNG renditions the page serves it between. The
# ShouldServePostPages suite asserts both are present — so without
# this package the deploy fails at the test gate rather than
# shipping a janky page.
#
# Nothing that compiles is installed here. clang, lld, libc++ and the
# WASI sysroot come with the crafter-build Flatpak (next step), and
# its Sdk has the cmake, make and perl that build msquic and quictls
# (Crafter.Network's ExternalBuilder::CMake dependency), the OpenSSL
# the server links, and the gpg the suites call: the suites run
# inside the sandbox, so what this container has is beside the point.
pacman -Syu --noconfirm --needed \
nodejs flatpak \
git curl tar rsync zstd gzip jq ffmpeg
# Container runs as root; workspace may be owned by another uid.
git config --global --add safe.directory '*'
- name: Install crafter-build
# crafter-build is the Crafter.Build Flatpak, the route imsd's package
# job takes: it carries its own clang, lld, libc++ and WASI sysroot, and
# its Sdk (org.freedesktop.Sdk, from Flathub) has the cmake, make, perl,
# git, gpg and OpenSSL the build and the suites need. `stable` is the
# branch Crafter.Build publishes from its release tags (`beta` follows
# every push to its master), so the toolchain moves only with a release
# of the toolchain. Crafter.Build's CI publishes the Flatpak and nothing
# else since 2026-10-02: the rolling `latest` release's
# crafter-build-linux-x86_64-v2.tar.gz this step used to download is
# gone (404), which is what failed the deploy on 2026-10-12.
#
# /var/lib/flatpak is a docker volume the runner mounts, so the Sdk and
# the Flatpak (about 4 GB on disk) are downloaded once; `install` is a
# no-op after that and `update` follows new stable releases. The install is retried: a
# flaky download must not fail a deploy before any build time is spent.
run: |
set -eu
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
flatpak remote-add --if-not-exists catcrafts https://flatpak.catcrafts.net/catcrafts.flatpakrepo
for i in 1 2 3; do
flatpak install -y --noninteractive catcrafts net.catcrafts.CrafterBuild//stable && break
if [ "$i" -eq 3 ]; then
echo "ERROR: installing the crafter-build Flatpak failed 3 times." >&2
exit 1
fi
echo "install failed (attempt $i/3), retrying in 15s..." >&2
sleep 15
done
flatpak update -y --noninteractive
# The Flatpak under the name the rest of this file uses. Inside the
# sandbox PATH is /app/bin:/usr/bin (the Flatpak's tools, the Sdk's);
# /opt/foundry/bin is appended because the suites run INSIDE the
# sandbox and ShouldSettleEurcOnTestnet shells out to cast, which
# the next step installs there: the sandbox sees the host's /opt but
# not its /usr/local.
cat > /usr/local/bin/crafter-build <<'EOF'
#!/bin/sh
exec flatpak run --env=PATH=/app/bin:/usr/bin:/opt/foundry/bin net.catcrafts.CrafterBuild//stable "$@"
EOF
# A binary crafter-build builds for the host links the Flatpak's
# libc++ (and the Sdk's glibc, newer than this container's), so it
# runs inside the sandbox, not in this shell: `crafter-run <binary>
# <args>` is how the steps below call catcrafts-server. The host
# itself gets those libraries shipped next to the binary, see
# "Deploy backend".
cat > /usr/local/bin/crafter-run <<'EOF'
#!/bin/sh
cmd=$1; shift
case $cmd in /*) ;; *) cmd=$PWD/$cmd ;; esac
exec flatpak run --env=PATH=/app/bin:/usr/bin:/opt/foundry/bin --command="$cmd" net.catcrafts.CrafterBuild//stable "$@"
EOF
chmod +x /usr/local/bin/crafter-build /usr/local/bin/crafter-run
flatpak info net.catcrafts.CrafterBuild//stable | sed -n 's/^ *\(Version\|Commit\|Date\): */crafter-build \1: /p'
crafter-build --version
- name: Install cast (Foundry)
# The ShouldSettleEurcOnTestnet gate pays its own test order with a
# real ERC-20 transfer on Base Sepolia, and cast is the only tool a
# stock shell has that can sign one — the server deliberately cannot,
# that is the EURC rail's design. Extracted alone: forge/anvil/chisel
# are dead weight here.
#
# Into /opt, not /usr/local/bin: the suite runs inside the crafter-build
# sandbox, which sees the host's /opt but has the Sdk's own /usr. The
# crafter-build wrapper above puts /opt/foundry/bin on the sandbox PATH;
# the last line checks that is where the suite will find it.
run: |
set -eux
url="https://github.com/foundry-rs/foundry/releases/download/stable/foundry_stable_linux_amd64.tar.gz"
curl -fsSL "$url" -o /tmp/foundry.tar.gz
mkdir -p /opt/foundry/bin
tar -xzf /tmp/foundry.tar.gz -C /opt/foundry/bin cast
crafter-run /usr/bin/sh -c 'command -v cast && cast --version'
- name: Checkout
uses: actions/checkout@v4
- name: Cache crafter-build dependency clones
# crafter-build's cache holds the Crafter.Graphics and Crafter.Network
# clones, the prebuilt module cache and the msquic/quictls external
# build. crafter-build still git-pulls each dep every run, so a stale
# cache only means a smaller delta fetch, never a stale build.
#
# The path is Flatpak's: it sets XDG_CACHE_HOME to the app's own
# directory under ~/.var/app, crafter-build honors that, and the
# sandbox shares the host's home, so the directory is the same inside
# and out. The "Deploy backend" step reads libmsquic out of the same
# tree.
uses: actions/cache@v4
with:
path: ~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build
key: crafter-cache-${{ runner.os }}-flatpak-${{ hashFiles('project.cpp') }}
restore-keys: |
crafter-cache-${{ runner.os }}-flatpak-
- name: Fetch ECB reference rates
# Feeds the indicative national-currency line on order pages. Every
# charge is in euros; this is display only, labelled with its date —
# which is why build-time freshness is enough and no rate service is
# ever called at page-view time. Exits 0 on failure: a stale rate
# (or none — the page then shows only euros) must not fail a deploy.
run: tools/fetch-rates.sh
- name: Fetch fediverse posts
# Build-time, not run-time: the site hosts the owner's own posts in full
# and links out for the discussion, so there is no sync service and no
# runtime dependency on the instance being up. The script leaves the
# committed content/posts.json untouched and exits 0 on any failure, so a
# fediverse outage cannot fail a deploy.
run: tools/fetch-posts.sh
- name: Mirror post media
# Downloads the images and screen recordings the posts carry — both the
# headline file and everything embedded inside the body — and rewrites
# content/posts.json (including the body text itself) to point at our own
# copies, so nothing the browser loads is third-party. That is what keeps
# the privacy notice's "everything comes from catcrafts.net" true.
#
# Content-addressed: the name is the hash of the bytes. Note what that
# does NOT mean — a third-party file is re-fetched on every build, because
# the name cannot be known until the bytes are in hand; only the write is
# skipped when the hash is already on the mount. Media we host ourselves
# (tools/publish-media.sh) is the exception that is genuinely incremental:
# fetch-media.sh adopts an own-origin URL without any request at all.
#
# Writes straight into the mount so the copies persist across deploys —
# they are NOT always reproducible, because a source instance deleting a
# file leaves ours as the only one.
id: media
run: |
set -eu
if [ -d /deploy-app ]; then
mkdir -p /deploy-app/media
dir=/deploy-app/media
else
echo "WARNING: /deploy-app not mounted; mirroring to a throwaway dir." >&2
echo "Media will be re-downloaded on every build until the mount exists." >&2
dir=media
fi
tools/fetch-media.sh "$dir"
# Handed to the test step (E2E_MEDIA_DIR) so ShouldServePostPages
# checks the files where they actually are. It used to assume
# ./media and reported every referenced file as missing here, which
# reads as a broken site and means a wrong path.
echo "dir=$dir" >> "$GITHUB_OUTPUT"
- name: Build and test the backend
id: srv
# The server product builds Catcrafts.Shared for the host, which is the
# only way to actually RUN the code that generates every byte of markup
# the site emits. `crafter-build test` is the gate: if escaping or the
# JSON reader regress, the deploy stops here rather than shipping
# broken pages. One suite per tests/<Name>/main.cpp — the unit suites
# replaced the binary's --selftest flag, the Should{ServeRoutes,…}
# black-box suites replaced tools/e2e.sh: each spawns the freshly
# built server on its own scratch port with the fake payment rails and
# exercises it over real HTTP. ShouldBootWasmAtDepth self-skips here —
# no wasm bundle exists yet — and re-runs after the wasm build below.
#
# E2E_MEDIA_DIR points ShouldServePostPages at the mirror mount, where
# the media files actually are.
#
# EURC_E2E_PRIVATE_KEY feeds the live crypto suite, which runs as part
# of this same gate: a deploy that cannot settle a testnet EURC
# transfer does not ship. If it fails at `cast send`, the testnet
# wallet is empty — see deploy/README.md "Live payment suites in CI"
# for the faucets. The bank rail's own suite needs no secret; it drives
# the real rail with a credits file instead of a provider.
#
# Same refuse-to-guess rule as the wasm bundle below: a variant
# directory embeds a config hash, so more than one match means the tree
# is ambiguous and picking the first would deploy an arbitrary build.
#
# The binary is called through crafter-run: it links the Flatpak's
# libc++, which this container does not have.
env:
E2E_MEDIA_DIR: ${{ steps.media.outputs.dir }}
EURC_E2E_PRIVATE_KEY: ${{ secrets.EURC_E2E_PRIVATE_KEY }}
run: |
set -eux
crafter-build --product=server
crafter-build test --product=server
matches=$(find bin -maxdepth 1 -type d -name 'Catcrafts.Server-*' | sort)
count=$(printf '%s\n' "$matches" | grep -c . || true)
if [ "$count" -ne 1 ]; then
echo "Expected exactly one Catcrafts.Server-* directory, found $count:" >&2
printf '%s\n' "$matches" >&2
exit 1
fi
echo "srv=$matches" >> "$GITHUB_OUTPUT"
crafter-run "$matches/catcrafts-server" --routes
- name: Generate sitemap and Atom feed
# Both come from the same route table and Post model the pages use, so
# they cannot drift from what the site serves. Generated BEFORE the wasm
# build so cfg.files picks them up into the bundle.
env:
SRV: ${{ steps.srv.outputs.srv }}
run: |
set -eux
crafter-run "$SRV/catcrafts-server" --sitemap > sitemap.xml
crafter-run "$SRV/catcrafts-server" --feed > feed.xml
head -n 4 sitemap.xml
- name: Build (wasm bundle)
run: crafter-build
- name: Locate build output
id: out
run: |
set -eu
# The directory name embeds a config hash, so glob for it. Any change
# to compile/link flags produces a NEW hash, which is why we refuse to
# guess when more than one variant is present rather than taking
# whichever the filesystem happened to list first.
matches=$(find bin -maxdepth 1 -type d -name 'Catcrafts.Net-wasm32-wasip1-*' | sort)
count=$(printf '%s\n' "$matches" | grep -c . || true)
if [ "$count" -eq 0 ]; then
echo "No build output directory found under bin/" >&2
ls -la bin || true
exit 1
fi
if [ "$count" -gt 1 ]; then
echo "Ambiguous build output — $count variant directories under bin/:" >&2
printf '%s\n' "$matches" >&2
echo "Refusing to guess which one to deploy. Clean bin/ and rebuild." >&2
exit 1
fi
dist=$matches
echo "dist=$dist" >> "$GITHUB_OUTPUT"
echo "Built bundle: $dist"
ls -la "$dist"
- name: Make the static shell depth-safe
# Caddy serves this index.html directly when the backend is down, at
# whatever URL was requested — including two-segment ones like
# /demos/raytracer. Crafter.Build emits relative boot scripts and its
# runtime.js fetches variants.json/files.json/the wasm relative to the
# DOCUMENT, so at any depth the fallback loads nothing at all. The script
# roots the tags and adds <base href="/">, and fails loudly rather than
# silently no-opping. The SSR path handles itself; this is only the
# backend-down fallback.
env:
DIST: ${{ steps.out.outputs.dist }}
run: tools/fix-bundle-depth.sh "$DIST"
- name: Wasm boot and script-shape checks
# The bundle-sensitive slice of the black-box suites, re-run now that
# the wasm bundle exists under bin/. The server discovers the bundle
# and lifts its <script> tags from it, so before the wasm build there
# is nothing to assert about wasm booting: ShouldBootWasmAtDepth
# self-skipped in the backend step, and ShouldStayScriptFree's checks
# around the boot tags passed vacuously because no page had scripts to
# begin with. Re-running both against the real bundle is what makes
# them meaningful.
#
# A gate, not a report: a failure here stops the deploy.
run: crafter-build test ShouldBootWasmAtDepth ShouldStayScriptFree --product=server
- name: Pre-compress static assets
# Caddy's `precompressed zstd gzip` (see deploy/Caddyfile.example)
# serves these siblings straight from disk instead of re-compressing
# the ~700 KB wasm module on every request. Building them here also
# buys a better ratio than on-the-fly encoding would spend CPU on.
env:
DIST: ${{ steps.out.outputs.dist }}
run: |
set -eu
for f in "$DIST"/*.wasm "$DIST"/*.js "$DIST"/*.css "$DIST"/*.xml "$DIST"/*.svg; do
[ -f "$f" ] || continue
zstd -19 -q -f -k -- "$f"
gzip -9 -f -k -- "$f"
done
echo "Compressed artifacts:"
ls -la "$DIST"
- name: Deploy to web root
# No SSH: the job already runs on the deploy box. The server's web root
# is bind-mounted into this container at /deploy via the runner config
# (container.options: "-v /path/to/webroot:/deploy"). We just copy into
# it. --delete makes the root an exact mirror of the build; Caddyfile.coi
# is a local dev helper (its 'root' points at the build machine), so the
# server's own Caddy config supplies the headers instead.
env:
DIST: ${{ steps.out.outputs.dist }}
run: |
set -eu
if [ ! -d /deploy ]; then
echo "ERROR: /deploy is not mounted into the runner container." >&2
echo "Add '-v /your/webroot:/deploy' to the runner's container.options" >&2
echo "in config.yaml and restart the runner." >&2
exit 1
fi
rsync -a --delete --exclude 'Caddyfile.coi' "$DIST"/ /deploy/
echo "Deployed $DIST -> /deploy (host web root)"
- name: Guard against leaking runtime files into the web root
# The web root is served by Caddy's file_server AND mirrored with
# --delete, so anything runtime-owned that lands there is both published
# and destroyed on the next push. Nothing does today; this is here so a
# future cfg.files line cannot quietly change that once the shop has a
# database and bank credentials.
run: |
set -eu
# The inverse guard: everything that IS meant to be served must be
# world-readable. Caddy and catcrafts-server run as their own users,
# so a 0600 file (mktemp's default, once shipped by fetch-posts.sh)
# deploys "successfully" and then 403s / renders an empty page.
# (/deploy-app/content gets the same check in its own deploy step,
# after its rsync — here it would still hold the previous run.)
unreadable=$(find /deploy -type f ! -perm -o+r -print 2>/dev/null || true)
if [ -n "$unreadable" ]; then
echo "ERROR: deployed files not world-readable (host services cannot serve them):" >&2
echo "$unreadable" >&2
exit 1
fi
bad=$(find /deploy -maxdepth 1 \( -name '*.db' -o -name '*.db-*' \
-o -name '*.pem' -o -name '*.key' -o -name '*.env' \
-o -name '.*' ! -name '.' \) -print 2>/dev/null || true)
if [ -n "$bad" ]; then
echo "ERROR: runtime-owned files found in the public web root:" >&2
printf '%s\n' "$bad" >&2
exit 1
fi
echo "Web root contains no database, key or dotfile."
- name: Deploy backend
# Second bind mount, separate from the web root on purpose: the backend
# binary and its content must NOT be inside a directory Caddy serves and
# rsync --delete mirrors. Add to the runner's container.options:
# -v /srv/catcrafts-app:/deploy-app
#
# The restart is not done here. This job runs inside a container with no
# access to the host's systemd, and the alternatives (host root for the
# runner, or a polkit rule) grant CI far more than "restart one service"
# needs. Instead the last step touches a marker file that
# catcrafts-deploy.path watches — see deploy/catcrafts-deploy.path.
# That unit smoke-checks the new binary (--render /) before cutting
# over, so a broken deploy leaves the working server running.
env:
SRV: ${{ steps.srv.outputs.srv }}
run: |
set -eu
if [ ! -d /deploy-app ]; then
echo "ERROR: /deploy-app is not mounted into the runner container." >&2
echo "Add '-v /srv/catcrafts-app:/deploy-app' to the runner's" >&2
echo "container.options in config.yaml and restart the runner." >&2
exit 1
fi
install -m 0755 "$SRV/catcrafts-server" /deploy-app/catcrafts-server.new
mkdir -p /deploy-app/content
# --delete on content/ is safe: it is regenerated every build. Note it
# does NOT touch /deploy-app/media, which must survive.
rsync -a --delete content/ /deploy-app/content/
# The server links libmsquic as a SHARED lib out of crafter-build's
# external cache; the RUNPATH into that cache only exists on this
# build machine. The binary carries an rpath to /srv/catcrafts-app
# (project.cpp), so the .so must travel with it. install dereferences
# the libmsquic.so.2 -> .so.2.x.y symlink into a regular file.
MSQ=$(find ~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build/external -name 'libmsquic.so.2*' -type f | head -1)
if [ -z "$MSQ" ]; then
echo "ERROR: libmsquic.so.2 not found in the external build cache." >&2
exit 1
fi
install -m 0755 "$MSQ" /deploy-app/libmsquic.so.2.new
mv -f /deploy-app/libmsquic.so.2.new /deploy-app/libmsquic.so.2
# The binary also links the Flatpak's libc++, libc++abi and LLVM
# libunwind: its rpath into /app/lib exists only inside the sandbox,
# the host's own libc++ is an older release than the headers the
# binary was compiled against, and the host has no libunwind.so.1 on
# the loader path at all. They travel with the binary the way
# libmsquic does and the same rpath finds them. Nothing in them needs
# more than GLIBC_2.38 from the host's glibc (checked with objdump -T
# on 2026-10-12; the Sdk's own glibc is newer than the host's, which
# is why it matters).
APP=$(flatpak info --show-location net.catcrafts.CrafterBuild//stable)/files/lib
for lib in libc++.so.1 libc++abi.so.1 libunwind.so.1; do
if [ ! -f "$APP/$lib" ]; then
echo "ERROR: $lib not found in the crafter-build Flatpak under $APP." >&2
exit 1
fi
install -m 0755 "$APP/$lib" "/deploy-app/$lib.new"
mv -f "/deploy-app/$lib.new" "/deploy-app/$lib"
done
# Same world-readable guard as the web root: the server runs as its
# own user, and a 0600 content file renders as an empty page.
unreadable=$(find /deploy-app/content -type f ! -perm -o+r -print 2>/dev/null || true)
if [ -n "$unreadable" ]; then
echo "ERROR: deployed content not world-readable:" >&2
echo "$unreadable" >&2
exit 1
fi
# Swap the binary into place atomically, so a request arriving mid-copy
# never hits a truncated executable.
mv -f /deploy-app/catcrafts-server.new /deploy-app/catcrafts-server
date -u +%FT%TZ > /deploy-app/.deploy-stamp
echo "Deployed backend -> /deploy-app; marker touched"