CI: build with the Crafter.Build Flatpak on a repo-scoped privileged runner

Crafter.Build's CI publishes only the Flatpak since 2026-10-02, so the
rolling `latest` tarball the Install crafter-build step downloaded is gone
and the 2026-10-12 deploy failed with a 404 before building anything.

Take the route imsd's package job takes. The job runs on `flatpak`, a
privileged runner registered to this repository alone
(forgejo-runner-catcrafts on the box, same arrangement as imsd's), with the
installed Flatpak and its Sdk kept on a docker volume between runs. The
container installs nothing that compiles any more: clang, lld, libc++, the
WASI sysroot, cmake, perl, gpg and OpenSSL all come with the Flatpak and its
Sdk. `crafter-build` is a wrapper around `flatpak run`, `crafter-run` runs a
built binary inside the sandbox, cast is installed under /opt where the
sandbox can see it, and the cache path is the Flatpak's own.

The server binary links the Flatpak's libc++, libc++abi and libunwind, which
are newer than the host's, so the deploy ships them next to the binary the
way it already ships libmsquic; the rpath finds them there. They need at
most GLIBC_2.38, the host has 2.41.

ShouldMatchBankTransfers passed braced lists straight into a std::span
parameter. That compiled only because libc++ 22 had the C++26
span(initializer_list) constructor; the libc++ 23 in the Flatpak does not,
so the lists now go through a vector like the rest of the file.

deploy/README.md documents the runner, the sandbox's private /tmp and how
to run a built binary where the host lacks a matching libc++.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.forgejo/workflows/deploy.yaml

@@ -13,7 +13,15 @@ concurrency:
 
 jobs:
   build-deploy:
-    runs-on: arch-latest
+    # `flatpak` is a privileged runner registered to this repository alone
+    # (/etc/forgejo-runner/config-catcrafts.yaml on the box, label
+    # flatpak:docker://archlinux:latest): crafter-build is the Crafter.Build
+    # Flatpak, and bubblewrap inside the job container needs the privilege.
+    # The same runner carries the two deploy bind mounts (/deploy, /deploy-app)
+    # and a docker volume at /var/lib/flatpak that keeps the installed Flatpak
+    # and its Sdk between runs. Same route as imsd's package job; see
+    # deploy/README.md "Runner configuration".
+    runs-on: flatpak
     steps:
       - name: Check live payment test credentials
         # ShouldSettleEurcOnTestnet is a mandatory deploy gate, so a missing
@@ -60,33 +68,74 @@ jobs:
           # ShouldServePostPages suite asserts both are present — so without
           # this package the deploy fails at the test gate rather than
           # shipping a janky page.
+          #
+          # Nothing that compiles is installed here. clang, lld, libc++ and the
+          # WASI sysroot come with the crafter-build Flatpak (next step), and
+          # its Sdk has the cmake, make and perl that build msquic and quictls
+          # (Crafter.Network's ExternalBuilder::CMake dependency), the OpenSSL
+          # the server links, and the gpg the suites call: the suites run
+          # inside the sandbox, so what this container has is beside the point.
           pacman -Syu --noconfirm --needed \
-            nodejs \
-            clang lld libc++ \
-            wasi-libc wasi-libc++ wasi-libc++abi wasi-compiler-rt \
-            git curl tar rsync zstd gzip jq openssl ffmpeg gnupg llvm \
-            cmake make perl
-          # cmake/make/perl: the native server product pulls Crafter.Network,
-          # which builds msquic (ExternalBuilder::CMake) with quictls — CMake's
-          # default generator drives make, and quictls is an OpenSSL fork whose
-          # build system is perl. Dev machines have base-devel; this container
-          # does not.
+            nodejs flatpak \
+            git curl tar rsync zstd gzip jq ffmpeg
           # Container runs as root; workspace may be owned by another uid.
           git config --global --add safe.directory '*'
 
       - name: Install crafter-build
-        # Pull the rolling 'latest' Linux build from the Crafter.Build repo and
-        # install it distro-style so it auto-discovers its modules under
-        # /usr/share/crafter-build. v2 = SSE4.2 baseline, safe on the CI SBC.
+        # crafter-build is the Crafter.Build Flatpak, the route imsd's package
+        # job takes: it carries its own clang, lld, libc++ and WASI sysroot, and
+        # its Sdk (org.freedesktop.Sdk, from Flathub) has the cmake, make, perl,
+        # git, gpg and OpenSSL the build and the suites need. `stable` is the
+        # branch Crafter.Build publishes from its release tags (`beta` follows
+        # every push to its master), so the toolchain moves only with a release
+        # of the toolchain. Crafter.Build's CI publishes the Flatpak and nothing
+        # else since 2026-10-02: the rolling `latest` release's
+        # crafter-build-linux-x86_64-v2.tar.gz this step used to download is
+        # gone (404), which is what failed the deploy on 2026-10-12.
+        #
+        # /var/lib/flatpak is a docker volume the runner mounts, so the Sdk and
+        # the Flatpak (about 4 GB on disk) are downloaded once; `install` is a
+        # no-op after that and `update` follows new stable releases. The install is retried: a
+        # flaky download must not fail a deploy before any build time is spent.
         run: |
-          set -eux
-          url="https://forgejo.catcrafts.net/Catcrafts/Crafter.Build/releases/download/latest/crafter-build-linux-x86_64-v2.tar.gz"
-          mkdir -p /tmp/cb
-          curl -fsSL "$url" -o /tmp/cb.tar.gz
-          tar -xzf /tmp/cb.tar.gz -C /tmp/cb
-          install -Dm755 /tmp/cb/bin/crafter-build /usr/bin/crafter-build
-          cp -r /tmp/cb/share/crafter-build /usr/share/
-          crafter-build --version || true
+          set -eu
+          flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
+          flatpak remote-add --if-not-exists catcrafts https://flatpak.catcrafts.net/catcrafts.flatpakrepo
+          for i in 1 2 3; do
+            flatpak install -y --noninteractive catcrafts net.catcrafts.CrafterBuild//stable && break
+            if [ "$i" -eq 3 ]; then
+              echo "ERROR: installing the crafter-build Flatpak failed 3 times." >&2
+              exit 1
+            fi
+            echo "install failed (attempt $i/3), retrying in 15s..." >&2
+            sleep 15
+          done
+          flatpak update -y --noninteractive
+          # The Flatpak under the name the rest of this file uses. Inside the
+          # sandbox PATH is /app/bin:/usr/bin (the Flatpak's tools, the Sdk's);
+          # /opt/foundry/bin is appended because the suites run INSIDE the
+          # sandbox and ShouldSettleEurcOnTestnet shells out to cast, which
+          # the next step installs there: the sandbox sees the host's /opt but
+          # not its /usr/local.
+          cat > /usr/local/bin/crafter-build <<'EOF'
+          #!/bin/sh
+          exec flatpak run --env=PATH=/app/bin:/usr/bin:/opt/foundry/bin net.catcrafts.CrafterBuild//stable "$@"
+          EOF
+          # A binary crafter-build builds for the host links the Flatpak's
+          # libc++ (and the Sdk's glibc, newer than this container's), so it
+          # runs inside the sandbox, not in this shell: `crafter-run <binary>
+          # <args>` is how the steps below call catcrafts-server. The host
+          # itself gets those libraries shipped next to the binary, see
+          # "Deploy backend".
+          cat > /usr/local/bin/crafter-run <<'EOF'
+          #!/bin/sh
+          cmd=$1; shift
+          case $cmd in /*) ;; *) cmd=$PWD/$cmd ;; esac
+          exec flatpak run --env=PATH=/app/bin:/usr/bin:/opt/foundry/bin --command="$cmd" net.catcrafts.CrafterBuild//stable "$@"
+          EOF
+          chmod +x /usr/local/bin/crafter-build /usr/local/bin/crafter-run
+          flatpak info net.catcrafts.CrafterBuild//stable | sed -n 's/^ *\(Version\|Commit\|Date\): */crafter-build \1: /p'
+          crafter-build --version
 
       - name: Install cast (Foundry)
         # The ShouldSettleEurcOnTestnet gate pays its own test order with a
@@ -94,26 +143,39 @@ jobs:
         # stock shell has that can sign one — the server deliberately cannot,
         # that is the EURC rail's design. Extracted alone: forge/anvil/chisel
         # are dead weight here.
+        #
+        # Into /opt, not /usr/local/bin: the suite runs inside the crafter-build
+        # sandbox, which sees the host's /opt but has the Sdk's own /usr. The
+        # crafter-build wrapper above puts /opt/foundry/bin on the sandbox PATH;
+        # the last line checks that is where the suite will find it.
         run: |
           set -eux
           url="https://github.com/foundry-rs/foundry/releases/download/stable/foundry_stable_linux_amd64.tar.gz"
           curl -fsSL "$url" -o /tmp/foundry.tar.gz
-          tar -xzf /tmp/foundry.tar.gz -C /usr/local/bin cast
-          cast --version
+          mkdir -p /opt/foundry/bin
+          tar -xzf /tmp/foundry.tar.gz -C /opt/foundry/bin cast
+          crafter-run /usr/bin/sh -c 'command -v cast && cast --version'
 
       - name: Checkout
         uses: actions/checkout@v4
 
       - name: Cache crafter-build dependency clones
-        # ~/.cache/crafter.build holds the Crafter.Graphics clone and prebuilt
-        # module cache. crafter-build still git-pulls the dep each run, so a
-        # stale cache only means a smaller delta fetch, never a stale build.
+        # crafter-build's cache holds the Crafter.Graphics and Crafter.Network
+        # clones, the prebuilt module cache and the msquic/quictls external
+        # build. crafter-build still git-pulls each dep every run, so a stale
+        # cache only means a smaller delta fetch, never a stale build.
+        #
+        # The path is Flatpak's: it sets XDG_CACHE_HOME to the app's own
+        # directory under ~/.var/app, crafter-build honors that, and the
+        # sandbox shares the host's home, so the directory is the same inside
+        # and out. The "Deploy backend" step reads libmsquic out of the same
+        # tree.
         uses: actions/cache@v4
         with:
-          path: ~/.cache/crafter.build
-          key: crafter-cache-${{ runner.os }}-${{ hashFiles('project.cpp') }}
+          path: ~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build
+          key: crafter-cache-${{ runner.os }}-flatpak-${{ hashFiles('project.cpp') }}
           restore-keys: |
-            crafter-cache-${{ runner.os }}-
+            crafter-cache-${{ runner.os }}-flatpak-
 
       - name: Fetch ECB reference rates
         # Feeds the indicative national-currency line on order pages. Every
@@ -192,6 +254,9 @@ jobs:
         # Same refuse-to-guess rule as the wasm bundle below: a variant
         # directory embeds a config hash, so more than one match means the tree
         # is ambiguous and picking the first would deploy an arbitrary build.
+        #
+        # The binary is called through crafter-run: it links the Flatpak's
+        # libc++, which this container does not have.
         env:
           E2E_MEDIA_DIR: ${{ steps.media.outputs.dir }}
           EURC_E2E_PRIVATE_KEY: ${{ secrets.EURC_E2E_PRIVATE_KEY }}
@@ -207,7 +272,7 @@ jobs:
             exit 1
           fi
           echo "srv=$matches" >> "$GITHUB_OUTPUT"
-          "$matches/catcrafts-server" --routes
+          crafter-run "$matches/catcrafts-server" --routes
 
       - name: Generate sitemap and Atom feed
         # Both come from the same route table and Post model the pages use, so
@@ -217,8 +282,8 @@ jobs:
           SRV: ${{ steps.srv.outputs.srv }}
         run: |
           set -eux
-          "$SRV/catcrafts-server" --sitemap > sitemap.xml
-          "$SRV/catcrafts-server" --feed    > feed.xml
+          crafter-run "$SRV/catcrafts-server" --sitemap > sitemap.xml
+          crafter-run "$SRV/catcrafts-server" --feed    > feed.xml
           head -n 4 sitemap.xml
 
       - name: Build (wasm bundle)
@@ -376,13 +441,31 @@ jobs:
           # build machine. The binary carries an rpath to /srv/catcrafts-app
           # (project.cpp), so the .so must travel with it. install dereferences
           # the libmsquic.so.2 -> .so.2.x.y symlink into a regular file.
-          MSQ=$(find ~/.cache/crafter.build/external -name 'libmsquic.so.2*' -type f | head -1)
+          MSQ=$(find ~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build/external -name 'libmsquic.so.2*' -type f | head -1)
           if [ -z "$MSQ" ]; then
             echo "ERROR: libmsquic.so.2 not found in the external build cache." >&2
             exit 1
           fi
           install -m 0755 "$MSQ" /deploy-app/libmsquic.so.2.new
           mv -f /deploy-app/libmsquic.so.2.new /deploy-app/libmsquic.so.2
+          # The binary also links the Flatpak's libc++, libc++abi and LLVM
+          # libunwind: its rpath into /app/lib exists only inside the sandbox,
+          # the host's own libc++ is an older release than the headers the
+          # binary was compiled against, and the host has no libunwind.so.1 on
+          # the loader path at all. They travel with the binary the way
+          # libmsquic does and the same rpath finds them. Nothing in them needs
+          # more than GLIBC_2.38 from the host's glibc (checked with objdump -T
+          # on 2026-10-12; the Sdk's own glibc is newer than the host's, which
+          # is why it matters).
+          APP=$(flatpak info --show-location net.catcrafts.CrafterBuild//stable)/files/lib
+          for lib in libc++.so.1 libc++abi.so.1 libunwind.so.1; do
+            if [ ! -f "$APP/$lib" ]; then
+              echo "ERROR: $lib not found in the crafter-build Flatpak under $APP." >&2
+              exit 1
+            fi
+            install -m 0755 "$APP/$lib" "/deploy-app/$lib.new"
+            mv -f "/deploy-app/$lib.new" "/deploy-app/$lib"
+          done
           # Same world-readable guard as the web root: the server runs as its
           # own user, and a 0600 content file renders as an empty page.
           unreadable=$(find /deploy-app/content -type f ! -perm -o+r -print 2>/dev/null || true)

deploy/README.md

@@ -82,11 +82,48 @@ systemctl reload caddy
 
 ## Runner configuration
 
-Both mounts are required. In the Forgejo runner's `config.yaml`:
+The deploy job runs on a privileged runner registered to this repository alone
+(`runs-on: flatpak`), not on the shared `arch-latest` runner. crafter-build is
+the Crafter.Build Flatpak, and bubblewrap inside the job container needs the
+privilege; imsd's package job is set up the same way. On the box since
+2026-10-12:
+
+* `/etc/forgejo-runner/config-catcrafts.yaml`: label
+  `flatpak:docker://archlinux:latest`, `privileged: true`, cache proxy on port
+  34570.
+* `forgejo-runner-catcrafts.service`, a copy of the imsd unit, working
+  directory `/var/lib/forgejo-runner-catcrafts`. Its `.runner` is the
+  registration: scope `Catcrafts/catcrafts.net`, name
+  `cx43-catcrafts-net-privileged`.
+* docker volume `catcrafts-flatpak-system`, mounted at `/var/lib/flatpak` in
+  the job container: the installed Flatpak and its Sdk, about 4 GB, so a
+  deploy downloads neither. Deleting the volume only costs the next run that
+  download.
+
+To register it afresh (a lost `.runner`): as the forgejo user in
+`/var/lib/forgejo`, `forgejo -c /etc/forgejo/app.ini actions
+generate-runner-token --scope Catcrafts/catcrafts.net`, then as forgejo-runner
+in the working directory, `forgejo-runner register --no-interactive -c
+/etc/forgejo-runner/config-catcrafts.yaml --instance
+https://forgejo.catcrafts.net --token <token> --name
+cx43-catcrafts-net-privileged`.
+
+Inside the crafter-build sandbox the job container's top-level directories are
+visible at the same paths (`/deploy-app`, `/workspace`, `/opt`, `/root`), but
+`/tmp` is the sandbox's own: a file a step writes under `/tmp` does not exist
+for a suite, and the other way round. The media mirror lives on `/deploy-app`
+and cast is installed under `/opt` for that reason.
+
+Both deploy mounts are required, in that runner's config:
 
 ```yaml
 container:
-  options: "-v /srv/catcrafts.net:/deploy -v /srv/catcrafts-app:/deploy-app"
+  privileged: true
+  options: "--volume /var/www/catcrafts.net:/deploy --volume /srv/catcrafts-app:/deploy-app --volume catcrafts-flatpak-system:/var/lib/flatpak"
+  valid_volumes:
+    - /var/www/catcrafts.net
+    - /srv/catcrafts-app
+    - catcrafts-flatpak-*
 ```
 
 The deploy steps fail with an explanatory message if either is missing, rather
@@ -103,9 +140,12 @@ reacts.
 `catcrafts-deploy.service` runs `catcrafts-server --render /` as `ExecStartPre`
 before restarting. The unit tests already gated the deploy in CI
 (`crafter-build test --product=server`); this on-host check covers what CI
-cannot — the dynamic loader finding libmsquic on this machine and a page
-rendering from the deployed content — so **a broken binary leaves the working
-server running** instead of replacing it.
+cannot — the dynamic loader finding libmsquic, libc++, libc++abi and libunwind
+next to the binary on this machine and a page rendering from the deployed
+content — so **a broken binary leaves the working server running** instead of
+replacing it. The binary links the Flatpak's libc++, not the host's, which is
+why the deploy ships those three libraries into `/srv/catcrafts-app` alongside
+libmsquic; the rpath from `project.cpp` finds them there.
 
 The binary is installed as `catcrafts-server.new` and `mv`d into place, so a
 request arriving mid-copy never hits a truncated executable.
@@ -992,6 +1032,12 @@ crafter-build test --product=server   # every suite: the unit assertions
 <server>/catcrafts-server --orders FILE         # the orders ledger + manual transitions
 ```
 
+`crafter-build` is the Crafter.Build Flatpak (`net.catcrafts.CrafterBuild`),
+and a server binary it builds links the Flatpak's libc++ rather than the
+host's. Where the host has no matching libc++, run the binary inside the
+sandbox, which is what CI's `crafter-run` wrapper does:
+`flatpak run --command=$PWD/<server>/catcrafts-server net.catcrafts.CrafterBuild --routes`.
+
 ### If a `bin/` glob matches two directories
 
 The variant directory name embeds a config hash, and **`` and

tests/ShouldMatchBankTransfers/main.cpp

@@ -47,8 +47,16 @@ BankCredit Credit(std::string reference, std::int64_t amountMinor,
     return c;
 }
 
+// The braced lists below land in a vector on purpose. MatchCredits takes a
+// span, and the libc++ 23 inside the Crafter.Build 2.1.0 Flatpak (what CI
+// builds with) has no span(initializer_list) constructor; libc++ 22 had one
+// under C++26, which is the only reason the lists ever compiled.
+Server::TransferMatch Match(std::vector<BankCredit> credits, std::string_view reference) {
+    return Server::MatchCredits(credits, reference);
+}
+
 std::int64_t PaidFor(std::vector<BankCredit> credits, std::string_view reference) {
-    return Server::MatchCredits(credits, reference).paidMinor;
+    return Match(std::move(credits), reference).paidMinor;
 }
 
 } // namespace
@@ -102,7 +110,7 @@ int main() {
                     Credit("CC-2B6457", 2000, "sepa", "b") }, ref) == 5000,
           "two credits for one order are summed");
     {
-        const Server::TransferMatch m = Server::MatchCredits(
+        const Server::TransferMatch m = Match(
             { Credit("CC-2B6457", 3000, "sepa", "a"),
               Credit("CC-2B6457", 2000, "sepa", "b") }, ref);
         Check(m.count == 2, "the count reports how many credits carried it",
@@ -114,7 +122,7 @@ int main() {
     // transfer is final, and anything with a dispute window is not.
     {
         const Server::TransferMatch m =
-            Server::MatchCredits({ Credit("CC-2B6457", 5000, "ideal") }, ref);
+            Match({ Credit("CC-2B6457", 5000, "ideal") }, ref);
         Check(m.method == "ideal", "the settling method is carried out", m.method);
     }
 
@@ -129,11 +137,11 @@ int main() {
     // the same money. The matcher cannot fix it, so it flags it for a human.
     {
         const Server::TransferMatch m =
-            Server::MatchCredits({ Credit("CC-2B6457 and CC-AAAAAA", 10000) }, ref);
+            Match({ Credit("CC-2B6457 and CC-AAAAAA", 10000) }, ref);
         Check(m.ambiguous,
               "a credit quoting a second order reference is flagged ambiguous");
         const Server::TransferMatch clean =
-            Server::MatchCredits({ Credit("CC-2B6457 thanks", 10000) }, ref);
+            Match({ Credit("CC-2B6457 thanks", 10000) }, ref);
         Check(!clean.ambiguous, "an ordinary credit is not flagged");
     }