catcrafts.net/deploy/README.md

# Deploying catcrafts.net

Two artifacts come out of CI and land in two different places, on purpose.

| | goes to | served by |
|---|---|---|
| wasm bundle + static assets | `/srv/catcrafts.net` | Caddy `file_server` |
| `catcrafts-server` + `content/` | `/srv/catcrafts-app` | itself, on `127.0.0.1:8081` |

They are kept apart because the web root is **publicly served and mirrored with
`rsync --delete`** on every push. Anything runtime-owned that lives there is
both published to the internet and destroyed on the next deploy — which matters
now for the content files and matters a great deal once the shop has a database
and bank credentials.

Runtime state goes in a third place, `/var/lib/catcrafts`, created by the
service's `StateDirectory=`. Today that is `orders.jsonl` (the order event log)
and `bunq-state.json` (the bunq session context, including the client RSA key
the server generates on first contact).

**Two things on this box cannot be regenerated.** Everything else — the wasm
bundle, the content, the binary — comes back from a rebuild.

The first is `orders.jsonl`. It is the ledger: every order and every status
transition, append-only, and the audit trail the tax records lean on. Back it
up:

```sh
install -d -m 0700 /var/backups/catcrafts
cp /var/lib/catcrafts/orders.jsonl \
   /var/backups/catcrafts/orders-$(date -u +%F).jsonl
```

It contains names, addresses and email addresses, so it is personal data: keep
it 0600, keep it off the web root, and encrypt it before it leaves the machine.
(`bunq-state.json` is deliberately NOT worth backing up: delete it and the
server re-onboards from the API key on the next start.)

The second is `/srv/catcrafts-app/media` — the mirrored post images and screen
recordings. Usually reproducible from `content/posts.json`, but **not if a source
instance has deleted the file since**, and for these posts the media *is* the
content. It lives on the app mount rather than the web root specifically so
`rsync --delete` cannot reach it, and CI only ever adds to it. Worth including in
the same backup.

## Why the backend speaks plaintext HTTP/1.1

Caddy terminates TLS and reverse-proxies to loopback. The backend uses
Crafter.Network's `ListenerHTTP1` rather than its HTTP/3 `ListenerHTTP` for one
concrete reason: **Caddy cannot `reverse_proxy` to an h3 upstream.** Port 8081
must never be exposed directly.

## One-time host setup

```sh
# 1. service account, no login, no home
useradd --system --no-create-home --shell /usr/sbin/nologin catcrafts

# 2. directories
mkdir -p /srv/catcrafts.net /srv/catcrafts-app
chown catcrafts:catcrafts /srv/catcrafts-app
# The web root stays writable by whatever uid the runner container uses.

# 3. units
cp deploy/catcrafts-server.service /etc/systemd/system/
cp deploy/catcrafts-deploy.service /etc/systemd/system/
cp deploy/catcrafts-deploy.path    /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now catcrafts-deploy.path
# catcrafts-server is started by the first deploy; enable it so it survives a
# reboot:
systemctl enable catcrafts-server

# 4. Caddy — merge deploy/Caddyfile.example into your site config
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy
```

## Runner configuration

Both mounts are required. In the Forgejo runner's `config.yaml`:

```yaml
container:
  options: "-v /srv/catcrafts.net:/deploy -v /srv/catcrafts-app:/deploy-app"
```

The deploy steps fail with an explanatory message if either is missing, rather
than silently succeeding into the container's own filesystem.

## How the restart happens

CI runs **inside a container** and has no access to the host's systemd. Rather
than give the runner host root or a polkit rule — both of which grant far more
authority than "restart one service" needs — the last deploy step writes
`/srv/catcrafts-app/.deploy-stamp`, and `catcrafts-deploy.path` on the host
reacts.

`catcrafts-deploy.service` runs `catcrafts-server --selftest` as `ExecStartPre`
before restarting. That self-test exercises the HTML-escaping and JSON layers
that generate every byte of markup the site emits, so **a broken binary leaves
the working server running** instead of replacing it.

The binary is installed as `catcrafts-server.new` and `mv`d into place, so a
request arriving mid-copy never hits a truncated executable.

## If the backend is down

Caddy's `handle_errors` fallback serves the static `index.html`,
so the site degrades to the client-rendered wasm app rather than showing a 502.
Content still renders; what is lost is server-side rendering and real status
codes — an unknown path becomes a soft 404 again until the backend returns.

## Posts and their media

`content/posts-sources.json` holds the account and a **community allowlist**.
Only listed communities are fetched, so joining a new one does not silently
publish it to the site — add a line first.

```sh
tools/publish-media.sh FILE  # BEFORE posting: uploads to /media, prints the URL
tools/fetch-posts.sh         # writes content/posts.json
tools/fetch-media.sh         # mirrors the media, rewrites posts.json to /media/... paths
```

Order matters: the second reads what the first wrote. CI runs both before the
build. Neither fails the build on a network error — a fediverse outage leaves the
previous `posts.json` in place, and a single failed download leaves that one entry
pointing at its original URL rather than losing the post.

### Publish the media first, then post it

**The recommended flow is to put a recording on catcrafts.net before writing the
post, and use that URL as the post's link.** Run `tools/publish-media.sh
recording.mp4`; it transcodes to AV1 *and* an H.264 sibling, uploads both to the
media mount under the AV1's content hash, and prints a
`https://catcrafts.net/media/<hash>.h264.mp4` URL to paste into the post. The
H.264 one on purpose: a post's link is fetched raw — Lemmy apps and browsers play
that exact file, with no negotiation in front of it — so it must be the encoding
everything can play. (An AV1 link posted before this existed drew "bad media
error" reports from iPhones within hours.)

`fetch-media.sh` recognises its own origin and **adopts** such a URL: it rewrites
it to `/media/<hash>` and probes the local file for dimensions, downloading
nothing. That is not just an optimisation — it removes the whole class of build
failure where the media step depends on a third party. A 167 MB recording on a
file host once hit `MAX_BYTES` (64 MB), so the entry kept its original URL, and
the deploy then failed the e2e "media origin" check on a file that was sitting on
our own disk the entire time.

Adopting a `<hash>.h264.mp4` URL swaps the AV1 sibling back in as the page's
primary `<source>` when it is on the mount, keeping the H.264 as the fallback
`<source>`. So the post links the compatible file, while browsers that can take
AV1 download the small one — the `codecs` parameter on the first source is what
lets the rest skip it.

The transcode matters as much as the hosting. Phone recordings are wildly
oversized for what they show — that same 167 MB clip was 78 s of a dark room at
17 Mbps, and denoising into AV1 gives the same picture in 15 MB. It also **bakes
in the rotation**: phones record landscape and attach a display matrix, so an
untouched file reports 1920x1080 while playing portrait, and the `width`/`height`
attributes then reserve exactly the wrong box. (`fetch-media.sh` swaps the
dimensions when it sees a quarter-turn matrix, so a straight-from-phone mirror is
correct too — but transcoding means nothing downstream has to know.)

Two things to know about publishing AV1:

* Browsers without AV1 (Safari before 17, Apple hardware older than A17/M3) get
  the `<hash>.h264.mp4` sibling: on the site via the second `<source>`, on the
  fediverse because that sibling *is* the posted URL. `--raw` skips the
  transcode and the fallback both, so a raw AV1 upload recreates the
  will-not-play problem — use it for files that are already universally
  playable.
* Lemmy's `pict-rs` will not generate a thumbnail from an AV1 file, so a
  self-hosted video usually arrives with no `poster`. `publish-media.sh` uploads
  a poster frame beside the video, named `<video-hash>.poster.webp`, and
  `fetch-media.sh` falls back to that sibling when the instance supplied nothing.
  A thumbnail the instance *did* provide always wins. (Posting the H.264 URL
  also means pict-rs can thumbnail it again, so instance thumbnails come back.)

An own-origin URL naming a file that is **not** on the mount is deliberately left
pointing at its original URL rather than rewritten. That is a post published
without its media, and failing the e2e origin check loudly beats shipping a 404
inside a `<video>` tag.

**`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions,
which become the `width`/`height` attributes that stop the page reflowing as
several 5 MB recordings arrive. It degrades to no dimensions without it —
so `tools/e2e.sh` asserts they are present, and a build host missing the package
fails at the e2e gate instead of quietly shipping a janky page.

**Only the post's main media is used.** Lemmy distinguishes the media a post *is*
(`post.url`, with `post.thumbnail_url` as its generated still) from images merely
embedded in the body. Only the former is mirrored — a post whose pictures are
body-only renders as text, which is correct. Videos get the thumbnail as their
`poster`; without it a `preload="metadata"` video is a black box until someone
presses play.

**Thread links point at the community's instance, not the author's.** A post's
`ap_id` is on the account's own instance, but the discussion is in the community,
which is federated elsewhere and has its own local id for the same thread. So
`fetch-posts.sh` resolves each `ap_id` through the community instance's
`resolve_object` and stores that URL. One request per post, at build time; a
failure falls back to the `ap_id`, which still reaches a readable copy.

Media is **mirrored, not hotlinked**. Three reasons: the privacy notice says
everything the browser loads comes from catcrafts.net and should stay true;
hotlinking would send every visitor's IP to the source instance; and these posts
are their media, so a deleted upstream file would gut the page. Filenames are the
content hash, which is why the cache lifetime can be a year.

## Payments: Mollie setup

The rail is Mollie (bunq.me was measured and disqualified: €500/transaction on
cards and no method at all for a non-EU buyer at phone prices — it is a P2P
tool; the bunq client remains in the tree, unused, in case an account sweep is
ever wanted). The server needs exactly one secret: the Mollie API key.

```sh
# Keys live in the Mollie dashboard: Developers -> API keys. A test_… key
# works against the real API from the moment the account exists — verify the
# whole flow with it BEFORE swapping in the live_… key.
install -d -m 0755 /etc/catcrafts
cat > /etc/catcrafts/payments.env <<'ENV'
MOLLIE_API_KEY=test_your-key-here
ENV
chmod 0600 /etc/catcrafts/payments.env
systemctl restart catcrafts-server
journalctl -u catcrafts-server | tail   # should say "payments: mollie"
```

Without the env file the server starts with payments off: the whole site works,
the product page renders, and checkout answers 503 with an honest message —
degraded, not down.

Mechanics worth knowing:

* The reconciler polls each open order (`GET /v2/payments/{id}`) every 10 s
  while fresh, backing off with age. `?redirect` back from Mollie is ignored
  by design — only the authenticated poll moves an order to paid, and the
  paid event records the method (`ideal`, `creditcard`, …) in the ledger.
* Mollie payments EXPIRE. A payment that reaches canceled/expired/failed
  lapses the order automatically — the buyer just orders again.
* Card money stays disputable for months even after "paid": before shipping a
  large or exported order, glance at the `via` column in `--orders`. iDEAL
  and bank transfers are final; `creditcard` is the one with a tail.
* Mollie onboarding reviews the shop: the imprint (KVK, contact address),
  terms and privacy pages must be real before they approve live payments.
  They are — and e2e now fails the build if a PLACEHOLDER marker ever
  reaches a rendered page again.

## Shipping rates: Sendcloud (optional)

Without configuration, shipping is priced by the three-zone table in
the compiled-in product data (NL / EU / world, Catcrafts.Shared-Content.cppm) — honest flat rates you set. With a
Sendcloud account, the server fetches the real per-country prices of one
shipping method daily and uses those instead, falling back to the zones for
any country the method does not cover:

```sh
# credentials from Sendcloud: Settings -> Integrations -> API
cat >> /etc/catcrafts/bunq.env <<'ENV'
SENDCLOUD_PUBLIC_KEY=...
SENDCLOUD_SECRET_KEY=...
SENDCLOUD_METHOD='PostNL Parcels non-EU,DPD Home'
ENV
systemctl restart catcrafts-server
journalctl -u catcrafts-server | grep shipping:   # "table refreshed (N countries...)"
```

`SENDCLOUD_METHOD` is a comma-separated list of name substrings, merged in
order with the FIRST match per country winning — put the postal method
first so non-EU destinations get post rates (a courier method that also
covers Norway or Switzerland would otherwise price them at courier rates,
€54 instead of €19), and the courier second to fill the EU. The fetched table is
cached next to the orders file so a restart during a Sendcloud outage keeps
the last known prices. Like the bunq client, this integration is UNTESTED
against the live API until credentials exist — the response parser is covered
by --selftest, the fetch around it is thin.

The buyer sees whatever the server will charge: the checkout page embeds the
active table into its live total, and the amount is computed server-side at
order time from the same data.

## Invoice signing (GPG)

Paid orders offer a clearsigned markdown invoice at `/order/<token>/invoice.md`.
Numbering continues the pre-shop administration: one series per customer — a
random UUID as the customer number, invoices counting sequentially within it
(`f57c6512-…-3`), keyed by the buyer's email. Art. 226(2) permits "one or more
series"; completeness is provable by reconciling the append-only ledger against
the payment provider's records. The signature makes the invoice verifiable
forever, independent of this server — which is why the order page tells buyers
to download it rather than promising to host receipts indefinitely.

One-time key setup on the server, as the service user:

```sh
sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \
  gpg --batch --passphrase '' --quick-gen-key 'Catcrafts invoices <invoices@catcrafts.net>' default default never
# export the PUBLIC key and commit it to the repo so buyers can verify:
sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \
  gpg --armor --export invoices@catcrafts.net > invoice-key.asc
```

Then in `/etc/catcrafts/payments.env`:

```
INVOICE_GPG_KEY=invoices@catcrafts.net
```

and `Environment=GNUPGHOME=/var/lib/catcrafts/gnupg` in the service unit (see
catcrafts-server.service). The key has no passphrase because the service signs
unattended; the keyring lives in the 0700 StateDirectory. With a key configured,
a signing failure is a 500 — an unsigned invoice is never served by accident.
Without one (dev), invoices carry a visible UNSIGNED marker.

## Order email (confirmation + invoice)

A paid order gets one confirmation email with the clearsigned invoice
attached — plain text plus a markdown attachment, no HTML part, no remote
resources, nothing the privacy notice would have to explain. The mailer
watches the ledger, so every path to paid (reconciler, arrival poll, a manual
`--mark-paid` even on a later restart) results in exactly one email: the
`notified` event, appended only after the mail command accepts the message,
is what stops a resend.

Delivery shells out to a **sendmail-compatible command** rather than speaking
SMTP itself, for the same reason invoices shell out to gpg: TLS, AUTH and
deliverability are exactly what msmtp already does well, and the volume is a
handful of messages per week. Deliverability stays the mailbox provider's
problem (SPF/DKIM are theirs), and no third party beyond the provider that
already handles `info@catcrafts.net` ever sees order data — which is what the
privacy page implies.

```sh
apt install msmtp

cat > /etc/msmtprc <<'CONF'
defaults
auth on
tls on
tls_starttls on
account catcrafts
host smtp.your-mail-provider.example
port 587
from info@catcrafts.net
user info@catcrafts.net
passwordeval cat /etc/catcrafts/smtp-password
account default : catcrafts
CONF
chmod 0644 /etc/msmtprc

# msmtp runs as the service user, so the password file must be readable by
# it — unlike payments.env, which only root (systemd) reads.
install -o catcrafts -g catcrafts -m 0600 /dev/null /etc/catcrafts/smtp-password
# ...then put the SMTP password in that file.
```

Then in `/etc/catcrafts/payments.env`:

```
MAIL_COMMAND=msmtp -t
MAIL_FROM=Catcrafts <info@catcrafts.net>
```

and `systemctl restart catcrafts-server` — the journal should say
`mail: order confirmations via 'msmtp -t'`. Unset, no email is sent and the
order page plus the invoice download remain the buyer's receipt: degraded,
not down, like every optional integration here.

Worth knowing:

* A failed handoff retries with exponential backoff (1 min doubling to a cap
  of ~an hour), forever — a broken relay delays the email, it never eats it.
  Watch `journalctl -u catcrafts-server | grep 'mail:'` after changing config.
* With a signing key configured, a gpg failure means the email WAITS — an
  unsigned invoice never leaves by accident, same rule as the download.
* Send a real test: `--rail=fake` locally with `MAIL_COMMAND` pointing at
  msmtp and your own address in the order form, or just run `tools/e2e.sh`,
  which captures the messages with a fake sendmail and verifies the attached
  signature.

## Reading the orders ledger

```sh
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl
```

```
orders: 2

reference  status             total  cc  created               token
CC-3F9A2C  paid              595.00  NL  2026-08-04T14:02:11Z  3f9a2c…
CC-91B04D  awaiting_payment  534.34  CA  2026-08-04T15:40:03Z  91b04d…
```

Manual transitions exist for the cases automation cannot see — a payment bunq
confirmed out-of-band, the parcel handed to the carrier, a refund:

```sh
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-paid    <token>
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-shipped <token>
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --cancel       <token>
```

Each appends a status event to the log — nothing is ever rewritten, so the
file remains its own audit trail. Deleting personal data on request is an edit
of the fields the seven-year fiscal retention does not cover.

## Verifying a deploy

```sh
systemctl status catcrafts-server
curl -s localhost:8081/api/healthz            # ok + content counts

# real status codes, which a client-side router cannot produce
curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/nope        # 404
curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/blog        # 301

# the SEO check: content present with no JavaScript involved
curl -s https://catcrafts.net/projects | grep -c '<script'              # 0
curl -s https://catcrafts.net/projects | grep -o '<title>[^<]*'
```

## Analytics

Server-side only — the privacy policy promises request logging and nothing
else, so there is no client-side analytics anywhere on the site. GoAccess
(Debian package) turns Caddy's JSON access logs into two static HTML
reports, each with its own persistent DB and ingest ledger:

- `https://catcrafts.net/analytics/` — **public, censored.** Visitor IPs are
  anonymized at ingest (last octet zeroed before anything reaches its DB),
  no HOSTS or full-URL REFERRERS panels, and log lines matching `CENSOR_RE`
  in the script never enter its DB at all — the public tier cannot leak
  what it never ingested. Extend `CENSOR_RE` when the shop launches so
  order/payment URLs can never surface; keep secrets out of URL *paths*
  regardless (query strings are already stripped).
- `https://catcrafts.net/analytics/private/` — **uncensored** (basic auth,
  hash in the Caddyfile): full IPs, all panels.

Raw logs keep full IPs either way — that is the request logging the privacy
policy declares; per-IP forensics work from the logs and the private tier,
never from the public page.

Three pieces, all in `deploy/`:

- `catcrafts-analytics` → `/usr/local/bin/` — ingests each rotated
  `catcrafts.net-*.log.gz` exactly once into a persistent GoAccess DB
  (`/var/lib/goaccess/db`, tracked in `/var/lib/goaccess/ingested`), then
  renders the report from DB + live log. The live file is never persisted,
  so its lines don't double-count when Caddy rotates it. History therefore
  survives log deletion: the DB keeps aggregates forever.
- `catcrafts-analytics.service` — oneshot, runs as `caddy` (owner of the
  0600 logs).
- `catcrafts-analytics.timer` — hourly at :07.

Bot filtering is the load-bearing part: measured on real traffic, 57% of
requests were headerless vulnerability scanners and another 19% self-declared
bots (mostly ClaudeBot) — only ~24% human. `--ignore-crawlers
--unknowns-as-crawlers` drops both groups. The flags in the script apply at
ingest time and the DB stores aggregated data, so changing filters later only
affects new lines — re-ingesting history means deleting
`/var/lib/goaccess/{db,ingested}` and letting the next run rebuild from
whatever raw logs retention still holds (a year, per the Caddyfile).

```sh
apt install goaccess
install -m 755 deploy/catcrafts-analytics /usr/local/bin/
install -m 644 deploy/catcrafts-analytics.{service,timer} /etc/systemd/system/
mkdir -p /etc/goaccess /var/lib/goaccess /var/www/analytics /var/www/analytics-private
install -m 644 deploy/goaccess-browsers.list /etc/goaccess/browsers.list
# own IPs to keep out of the numbers - host-only file, NOT in this repo
echo "203.0.113.7" > /etc/goaccess/exclude-ips
chown -R caddy:caddy /var/lib/goaccess /var/www/analytics /var/www/analytics-private
systemctl daemon-reload && systemctl enable --now catcrafts-analytics.timer
```

## Running it locally

```sh
tools/dev.sh              # build both products and serve on :8080
tools/dev.sh --no-build   # reuse what is already in bin/
```

That is the whole site: Caddy in front, the backend behind, static assets from
disk, cross-origin headers scoped exactly as in production. Ctrl-C stops both.
Orders from the session go to a temp file and are discarded on exit; the
fake payment rail is active — `touch <workdir>/orders.jsonl.fake-paid` plays
the part of the customer paying.

**Do not run `catcrafts-server --serve` alone and expect a working site.** It
serves pages only — static assets are Caddy's job — so `/styles.css` 404s and
every page renders unstyled. That looks broken but isn't; it is the deployment
split working as designed.

### Other useful commands

```sh
tools/fetch-posts.sh            # pull posts from the allowed communities
tools/fetch-media.sh [dir]      # mirror their media locally (run after the above)
tools/fetch-rates.sh            # ECB reference rates for the indicative prices
tools/e2e.sh                    # ~200 HTTP checks against a real server (135 while
                                # coming-soon; the rest re-arm at launch); the CI gate
crafter-build --local -r        # the wasm app alone, no backend, on :8080

<server>/catcrafts-server --selftest            # ~260 in-process assertions
<server>/catcrafts-server --routes              # status + title for every route
<server>/catcrafts-server --render /projects    # dump one page's HTML
<server>/catcrafts-server --orders FILE         # the orders ledger + manual transitions
```

### If a `bin/` glob matches two directories

The variant directory name embeds a config hash, and **`--local` and
non-`--local` builds hash differently**: `--local` resolves the Crafter libraries
from sibling working trees, a plain build fetches them from Forgejo. They are
genuinely different configurations and both land in `bin/`.

Mixing them leaves two directories, and any script globbing for one picks
arbitrarily — which in practice means testing a stale binary and believing the
result. Every script here refuses to guess and tells you to `rm -rf bin`. Pick
one mode and stay in it.