# Deploying catcrafts.net Two artifacts come out of CI and land in two different places, on purpose. | | goes to | served by | |---|---|---| | wasm bundle + static assets | `/srv/catcrafts.net` | Caddy `file_server` | | `catcrafts-server` + `content/` | `/srv/catcrafts-app` | itself, on `127.0.0.1:8081` | They are kept apart because the web root is **publicly served and mirrored with `rsync --delete`** on every push. Anything runtime-owned that lives there is both published to the internet and destroyed on the next deploy — which matters now for the content files and matters a great deal once the shop has a database and bank credentials. Runtime state goes in a third place, `/var/lib/catcrafts`, created by the service's `StateDirectory=`. Today that is `orders.jsonl` (the order event log) and `bunq-state.json` (the bunq session context, including the client RSA key the server generates on first contact). **Two things on this box cannot be regenerated.** Everything else — the wasm bundle, the content, the binary — comes back from a rebuild. The first is `orders.jsonl`. It is the ledger: every order and every status transition, append-only, and the audit trail the tax records lean on. Back it up: ```sh install -d -m 0700 /var/backups/catcrafts cp /var/lib/catcrafts/orders.jsonl \ /var/backups/catcrafts/orders-$(date -u +%F).jsonl ``` It contains names, addresses and email addresses, so it is personal data: keep it 0600, keep it off the web root, and encrypt it before it leaves the machine. (`bunq-state.json` is deliberately NOT worth backing up: delete it and the server re-onboards from the API key on the next start.) The second is `/srv/catcrafts-app/media` — the mirrored post images and screen recordings. Usually reproducible from `content/posts.json`, but **not if a source instance has deleted the file since**, and for these posts the media *is* the content. It lives on the app mount rather than the web root specifically so `rsync --delete` cannot reach it, and CI only ever adds to it. Worth including in the same backup. ## Why the backend speaks plaintext HTTP/1.1 Caddy terminates TLS and reverse-proxies to loopback. The backend uses Crafter.Network's `ListenerHTTP1` rather than its HTTP/3 `ListenerHTTP` for one concrete reason: **Caddy cannot `reverse_proxy` to an h3 upstream.** Port 8081 must never be exposed directly. ## One-time host setup ```sh # 1. service account, no login, no home useradd --system --no-create-home --shell /usr/sbin/nologin catcrafts # 2. directories mkdir -p /srv/catcrafts.net /srv/catcrafts-app chown catcrafts:catcrafts /srv/catcrafts-app # The web root stays writable by whatever uid the runner container uses. # 3. units cp deploy/catcrafts-server.service /etc/systemd/system/ cp deploy/catcrafts-deploy.service /etc/systemd/system/ cp deploy/catcrafts-deploy.path /etc/systemd/system/ systemctl daemon-reload systemctl enable --now catcrafts-deploy.path # catcrafts-server is started by the first deploy; enable it so it survives a # reboot: systemctl enable catcrafts-server # 4. Caddy — merge deploy/Caddyfile.example into your site config caddy validate --config /etc/caddy/Caddyfile systemctl reload caddy ``` ## Runner configuration Both mounts are required. In the Forgejo runner's `config.yaml`: ```yaml container: options: "-v /srv/catcrafts.net:/deploy -v /srv/catcrafts-app:/deploy-app" ``` The deploy steps fail with an explanatory message if either is missing, rather than silently succeeding into the container's own filesystem. ## How the restart happens CI runs **inside a container** and has no access to the host's systemd. Rather than give the runner host root or a polkit rule — both of which grant far more authority than "restart one service" needs — the last deploy step writes `/srv/catcrafts-app/.deploy-stamp`, and `catcrafts-deploy.path` on the host reacts. `catcrafts-deploy.service` runs `catcrafts-server --selftest` as `ExecStartPre` before restarting. That self-test exercises the HTML-escaping and JSON layers that generate every byte of markup the site emits, so **a broken binary leaves the working server running** instead of replacing it. The binary is installed as `catcrafts-server.new` and `mv`d into place, so a request arriving mid-copy never hits a truncated executable. ## If the backend is down Caddy's `handle_response @down` falls back to serving the static `index.html`, so the site degrades to the client-rendered wasm app rather than showing a 502. Content still renders; what is lost is server-side rendering and real status codes — an unknown path becomes a soft 404 again until the backend returns. ## Posts and their media `content/posts-sources.json` holds the account and a **community allowlist**. Only listed communities are fetched, so joining a new one does not silently publish it to the site — add a line first. ```sh tools/fetch-posts.sh # writes content/posts.json tools/fetch-media.sh # mirrors the media, rewrites posts.json to /media/... paths ``` Order matters: the second reads what the first wrote. CI runs both before the build. Neither fails the build on a network error — a fediverse outage leaves the previous `posts.json` in place, and a single failed download leaves that one entry pointing at its original URL rather than losing the post. **`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions, which become the `width`/`height` attributes that stop the page reflowing as several 5 MB recordings arrive. It degrades to no dimensions without it — so `tools/e2e.sh` asserts they are present, and a build host missing the package fails at the e2e gate instead of quietly shipping a janky page. **Only the post's main media is used.** Lemmy distinguishes the media a post *is* (`post.url`, with `post.thumbnail_url` as its generated still) from images merely embedded in the body. Only the former is mirrored — a post whose pictures are body-only renders as text, which is correct. Videos get the thumbnail as their `poster`; without it a `preload="metadata"` video is a black box until someone presses play. **Thread links point at the community's instance, not the author's.** A post's `ap_id` is on the account's own instance, but the discussion is in the community, which is federated elsewhere and has its own local id for the same thread. So `fetch-posts.sh` resolves each `ap_id` through the community instance's `resolve_object` and stores that URL. One request per post, at build time; a failure falls back to the `ap_id`, which still reaches a readable copy. Media is **mirrored, not hotlinked**. Three reasons: the privacy notice says everything the browser loads comes from catcrafts.net and should stay true; hotlinking would send every visitor's IP to the source instance; and these posts are their media, so a deleted upstream file would gut the page. Filenames are the content hash, which is why the cache lifetime can be a year. ## Payments: Mollie setup The rail is Mollie (bunq.me was measured and disqualified: €500/transaction on cards and no method at all for a non-EU buyer at phone prices — it is a P2P tool; the bunq client remains in the tree, unused, in case an account sweep is ever wanted). The server needs exactly one secret: the Mollie API key. ```sh # Keys live in the Mollie dashboard: Developers -> API keys. A test_… key # works against the real API from the moment the account exists — verify the # whole flow with it BEFORE swapping in the live_… key. install -d -m 0755 /etc/catcrafts cat > /etc/catcrafts/payments.env <<'ENV' MOLLIE_API_KEY=test_your-key-here ENV chmod 0600 /etc/catcrafts/payments.env systemctl restart catcrafts-server journalctl -u catcrafts-server | tail # should say "payments: mollie" ``` Without the env file the server starts with payments off: the whole site works, the product page renders, and checkout answers 503 with an honest message — degraded, not down. Mechanics worth knowing: * The reconciler polls each open order (`GET /v2/payments/{id}`) every 10 s while fresh, backing off with age. `?redirect` back from Mollie is ignored by design — only the authenticated poll moves an order to paid, and the paid event records the method (`ideal`, `creditcard`, …) in the ledger. * Mollie payments EXPIRE. A payment that reaches canceled/expired/failed lapses the order automatically — the buyer just orders again. * Card money stays disputable for months even after "paid": before shipping a large or exported order, glance at the `via` column in `--orders`. iDEAL and bank transfers are final; `creditcard` is the one with a tail. * Mollie onboarding reviews the shop: the imprint (KVK, contact address), terms and privacy pages must be real before they approve live payments. They are — and e2e now fails the build if a PLACEHOLDER marker ever reaches a rendered page again. ## Shipping rates: Sendcloud (optional) Without configuration, shipping is priced by the three-zone table in the compiled-in product data (NL / EU / world, Catcrafts.Shared-Content.cppm) — honest flat rates you set. With a Sendcloud account, the server fetches the real per-country prices of one shipping method daily and uses those instead, falling back to the zones for any country the method does not cover: ```sh # credentials from Sendcloud: Settings -> Integrations -> API cat >> /etc/catcrafts/bunq.env <<'ENV' SENDCLOUD_PUBLIC_KEY=... SENDCLOUD_SECRET_KEY=... SENDCLOUD_METHOD='PostNL Parcels non-EU,DPD Home' ENV systemctl restart catcrafts-server journalctl -u catcrafts-server | grep shipping: # "table refreshed (N countries...)" ``` `SENDCLOUD_METHOD` is a comma-separated list of name substrings, merged in order with the FIRST match per country winning — put the postal method first so non-EU destinations get post rates (a courier method that also covers Norway or Switzerland would otherwise price them at courier rates, €54 instead of €19), and the courier second to fill the EU. The fetched table is cached next to the orders file so a restart during a Sendcloud outage keeps the last known prices. Like the bunq client, this integration is UNTESTED against the live API until credentials exist — the response parser is covered by --selftest, the fetch around it is thin. The buyer sees whatever the server will charge: the checkout page embeds the active table into its live total, and the amount is computed server-side at order time from the same data. ## Invoice signing (GPG) Paid orders offer a clearsigned markdown invoice at `/order//invoice.md`. Numbering continues the pre-shop administration: one series per customer — a random UUID as the customer number, invoices counting sequentially within it (`f57c6512-…-3`), keyed by the buyer's email. Art. 226(2) permits "one or more series"; completeness is provable by reconciling the append-only ledger against the payment provider's records. The signature makes the invoice verifiable forever, independent of this server — which is why the order page tells buyers to download it rather than promising to host receipts indefinitely. One-time key setup on the server, as the service user: ```sh sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \ gpg --batch --passphrase '' --quick-gen-key 'Catcrafts invoices ' default default never # export the PUBLIC key and commit it to the repo so buyers can verify: sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \ gpg --armor --export invoices@catcrafts.net > invoice-key.asc ``` Then in `/etc/catcrafts/payments.env`: ``` INVOICE_GPG_KEY=invoices@catcrafts.net ``` and `Environment=GNUPGHOME=/var/lib/catcrafts/gnupg` in the service unit (see catcrafts-server.service). The key has no passphrase because the service signs unattended; the keyring lives in the 0700 StateDirectory. With a key configured, a signing failure is a 500 — an unsigned invoice is never served by accident. Without one (dev), invoices carry a visible UNSIGNED marker. ## Reading the orders ledger ```sh catcrafts-server --orders /var/lib/catcrafts/orders.jsonl ``` ``` orders: 2 reference status total cc created token CC-3F9A2C paid 595.00 NL 2026-08-04T14:02:11Z 3f9a2c… CC-91B04D awaiting_payment 534.34 CA 2026-08-04T15:40:03Z 91b04d… ``` Manual transitions exist for the cases automation cannot see — a payment bunq confirmed out-of-band, the parcel handed to the carrier, a refund: ```sh catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-paid catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-shipped catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --cancel ``` Each appends a status event to the log — nothing is ever rewritten, so the file remains its own audit trail. Deleting personal data on request is an edit of the fields the seven-year fiscal retention does not cover. ## Verifying a deploy ```sh systemctl status catcrafts-server curl -s localhost:8081/api/healthz # ok + content counts # real status codes, which a client-side router cannot produce curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/nope # 404 curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/blog # 301 # the SEO check: content present with no JavaScript involved curl -s https://catcrafts.net/projects | grep -c '[^<]*' ``` ## Running it locally ```sh tools/dev.sh # build both products and serve on :8080 tools/dev.sh --no-build # reuse what is already in bin/ ``` That is the whole site: Caddy in front, the backend behind, static assets from disk, cross-origin headers scoped exactly as in production. Ctrl-C stops both. Orders from the session go to a temp file and are discarded on exit; the fake payment rail is active — `touch /orders.jsonl.fake-paid` plays the part of the customer paying. **Do not run `catcrafts-server --serve` alone and expect a working site.** It serves pages only — static assets are Caddy's job — so `/styles.css` 404s and every page renders unstyled. That looks broken but isn't; it is the deployment split working as designed. ### Other useful commands ```sh tools/fetch-posts.sh # pull posts from the allowed communities tools/fetch-media.sh [dir] # mirror their media locally (run after the above) tools/fetch-rates.sh # ECB reference rates for the indicative prices tools/e2e.sh # 143 HTTP checks against a real server; the CI gate crafter-build --local -r # the wasm app alone, no backend, on :8080 /catcrafts-server --selftest # ~140 in-process assertions /catcrafts-server --routes # status + title for every route /catcrafts-server --render /projects # dump one page's HTML /catcrafts-server --orders FILE # the orders ledger + manual transitions ``` ### If a `bin/` glob matches two directories The variant directory name embeds a config hash, and **`--local` and non-`--local` builds hash differently**: `--local` resolves the Crafter libraries from sibling working trees, a plain build fetches them from Forgejo. They are genuinely different configurations and both land in `bin/`. Mixing them leaves two directories, and any script globbing for one picks arbitrarily — which in practice means testing a stale binary and believing the result. Every script here refuses to guess and tells you to `rm -rf bin`. Pick one mode and stay in it.