347 lines
15 KiB
Markdown
347 lines
15 KiB
Markdown
# Deploying catcrafts.net
|
|
|
|
Two artifacts come out of CI and land in two different places, on purpose.
|
|
|
|
| | goes to | served by |
|
|
|---|---|---|
|
|
| wasm bundle + static assets | `/srv/catcrafts.net` | Caddy `file_server` |
|
|
| `catcrafts-server` + `content/` | `/srv/catcrafts-app` | itself, on `127.0.0.1:8081` |
|
|
|
|
They are kept apart because the web root is **publicly served and mirrored with
|
|
`rsync --delete`** on every push. Anything runtime-owned that lives there is
|
|
both published to the internet and destroyed on the next deploy — which matters
|
|
now for the content files and matters a great deal once the shop has a database
|
|
and bank credentials.
|
|
|
|
Runtime state goes in a third place, `/var/lib/catcrafts`, created by the
|
|
service's `StateDirectory=`. Today that is `orders.jsonl` (the order event log)
|
|
and `bunq-state.json` (the bunq session context, including the client RSA key
|
|
the server generates on first contact).
|
|
|
|
**Two things on this box cannot be regenerated.** Everything else — the wasm
|
|
bundle, the content, the binary — comes back from a rebuild.
|
|
|
|
The first is `orders.jsonl`. It is the ledger: every order and every status
|
|
transition, append-only, and the audit trail the tax records lean on. Back it
|
|
up:
|
|
|
|
```sh
|
|
install -d -m 0700 /var/backups/catcrafts
|
|
cp /var/lib/catcrafts/orders.jsonl \
|
|
/var/backups/catcrafts/orders-$(date -u +%F).jsonl
|
|
```
|
|
|
|
It contains names, addresses and email addresses, so it is personal data: keep
|
|
it 0600, keep it off the web root, and encrypt it before it leaves the machine.
|
|
(`bunq-state.json` is deliberately NOT worth backing up: delete it and the
|
|
server re-onboards from the API key on the next start.)
|
|
|
|
The second is `/srv/catcrafts-app/media` — the mirrored post images and screen
|
|
recordings. Usually reproducible from `content/posts.json`, but **not if a source
|
|
instance has deleted the file since**, and for these posts the media *is* the
|
|
content. It lives on the app mount rather than the web root specifically so
|
|
`rsync --delete` cannot reach it, and CI only ever adds to it. Worth including in
|
|
the same backup.
|
|
|
|
## Why the backend speaks plaintext HTTP/1.1
|
|
|
|
Caddy terminates TLS and reverse-proxies to loopback. The backend uses
|
|
Crafter.Network's `ListenerHTTP1` rather than its HTTP/3 `ListenerHTTP` for one
|
|
concrete reason: **Caddy cannot `reverse_proxy` to an h3 upstream.** Port 8081
|
|
must never be exposed directly.
|
|
|
|
## One-time host setup
|
|
|
|
```sh
|
|
# 1. service account, no login, no home
|
|
useradd --system --no-create-home --shell /usr/sbin/nologin catcrafts
|
|
|
|
# 2. directories
|
|
mkdir -p /srv/catcrafts.net /srv/catcrafts-app
|
|
chown catcrafts:catcrafts /srv/catcrafts-app
|
|
# The web root stays writable by whatever uid the runner container uses.
|
|
|
|
# 3. units
|
|
cp deploy/catcrafts-server.service /etc/systemd/system/
|
|
cp deploy/catcrafts-deploy.service /etc/systemd/system/
|
|
cp deploy/catcrafts-deploy.path /etc/systemd/system/
|
|
systemctl daemon-reload
|
|
systemctl enable --now catcrafts-deploy.path
|
|
# catcrafts-server is started by the first deploy; enable it so it survives a
|
|
# reboot:
|
|
systemctl enable catcrafts-server
|
|
|
|
# 4. Caddy — merge deploy/Caddyfile.example into your site config
|
|
caddy validate --config /etc/caddy/Caddyfile
|
|
systemctl reload caddy
|
|
```
|
|
|
|
## Runner configuration
|
|
|
|
Both mounts are required. In the Forgejo runner's `config.yaml`:
|
|
|
|
```yaml
|
|
container:
|
|
options: "-v /srv/catcrafts.net:/deploy -v /srv/catcrafts-app:/deploy-app"
|
|
```
|
|
|
|
The deploy steps fail with an explanatory message if either is missing, rather
|
|
than silently succeeding into the container's own filesystem.
|
|
|
|
## How the restart happens
|
|
|
|
CI runs **inside a container** and has no access to the host's systemd. Rather
|
|
than give the runner host root or a polkit rule — both of which grant far more
|
|
authority than "restart one service" needs — the last deploy step writes
|
|
`/srv/catcrafts-app/.deploy-stamp`, and `catcrafts-deploy.path` on the host
|
|
reacts.
|
|
|
|
`catcrafts-deploy.service` runs `catcrafts-server --selftest` as `ExecStartPre`
|
|
before restarting. That self-test exercises the HTML-escaping and JSON layers
|
|
that generate every byte of markup the site emits, so **a broken binary leaves
|
|
the working server running** instead of replacing it.
|
|
|
|
The binary is installed as `catcrafts-server.new` and `mv`d into place, so a
|
|
request arriving mid-copy never hits a truncated executable.
|
|
|
|
## If the backend is down
|
|
|
|
Caddy's `handle_response @down` falls back to serving the static `index.html`,
|
|
so the site degrades to the client-rendered wasm app rather than showing a 502.
|
|
Content still renders; what is lost is server-side rendering and real status
|
|
codes — an unknown path becomes a soft 404 again until the backend returns.
|
|
|
|
## Posts and their media
|
|
|
|
`content/posts-sources.json` holds the account and a **community allowlist**.
|
|
Only listed communities are fetched, so joining a new one does not silently
|
|
publish it to the site — add a line first.
|
|
|
|
```sh
|
|
tools/fetch-posts.sh # writes content/posts.json
|
|
tools/fetch-media.sh # mirrors the media, rewrites posts.json to /media/... paths
|
|
```
|
|
|
|
Order matters: the second reads what the first wrote. CI runs both before the
|
|
build. Neither fails the build on a network error — a fediverse outage leaves the
|
|
previous `posts.json` in place, and a single failed download leaves that one entry
|
|
pointing at its original URL rather than losing the post.
|
|
|
|
**`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions,
|
|
which become the `width`/`height` attributes that stop the page reflowing as
|
|
several 5 MB recordings arrive. It degrades to no dimensions without it —
|
|
so `tools/e2e.sh` asserts they are present, and a build host missing the package
|
|
fails at the e2e gate instead of quietly shipping a janky page.
|
|
|
|
**Only the post's main media is used.** Lemmy distinguishes the media a post *is*
|
|
(`post.url`, with `post.thumbnail_url` as its generated still) from images merely
|
|
embedded in the body. Only the former is mirrored — a post whose pictures are
|
|
body-only renders as text, which is correct. Videos get the thumbnail as their
|
|
`poster`; without it a `preload="metadata"` video is a black box until someone
|
|
presses play.
|
|
|
|
**Thread links point at the community's instance, not the author's.** A post's
|
|
`ap_id` is on the account's own instance, but the discussion is in the community,
|
|
which is federated elsewhere and has its own local id for the same thread. So
|
|
`fetch-posts.sh` resolves each `ap_id` through the community instance's
|
|
`resolve_object` and stores that URL. One request per post, at build time; a
|
|
failure falls back to the `ap_id`, which still reaches a readable copy.
|
|
|
|
Media is **mirrored, not hotlinked**. Three reasons: the privacy notice says
|
|
everything the browser loads comes from catcrafts.net and should stay true;
|
|
hotlinking would send every visitor's IP to the source instance; and these posts
|
|
are their media, so a deleted upstream file would gut the page. Filenames are the
|
|
content hash, which is why the cache lifetime can be a year.
|
|
|
|
## Payments: Mollie setup
|
|
|
|
The rail is Mollie (bunq.me was measured and disqualified: €500/transaction on
|
|
cards and no method at all for a non-EU buyer at phone prices — it is a P2P
|
|
tool; the bunq client remains in the tree, unused, in case an account sweep is
|
|
ever wanted). The server needs exactly one secret: the Mollie API key.
|
|
|
|
```sh
|
|
# Keys live in the Mollie dashboard: Developers -> API keys. A test_… key
|
|
# works against the real API from the moment the account exists — verify the
|
|
# whole flow with it BEFORE swapping in the live_… key.
|
|
install -d -m 0755 /etc/catcrafts
|
|
cat > /etc/catcrafts/payments.env <<'ENV'
|
|
MOLLIE_API_KEY=test_your-key-here
|
|
ENV
|
|
chmod 0600 /etc/catcrafts/payments.env
|
|
systemctl restart catcrafts-server
|
|
journalctl -u catcrafts-server | tail # should say "payments: mollie"
|
|
```
|
|
|
|
Without the env file the server starts with payments off: the whole site works,
|
|
the product page renders, and checkout answers 503 with an honest message —
|
|
degraded, not down.
|
|
|
|
Mechanics worth knowing:
|
|
|
|
* The reconciler polls each open order (`GET /v2/payments/{id}`) every 10 s
|
|
while fresh, backing off with age. `?redirect` back from Mollie is ignored
|
|
by design — only the authenticated poll moves an order to paid, and the
|
|
paid event records the method (`ideal`, `creditcard`, …) in the ledger.
|
|
* Mollie payments EXPIRE. A payment that reaches canceled/expired/failed
|
|
lapses the order automatically — the buyer just orders again.
|
|
* Card money stays disputable for months even after "paid": before shipping a
|
|
large or exported order, glance at the `via` column in `--orders`. iDEAL
|
|
and bank transfers are final; `creditcard` is the one with a tail.
|
|
* Mollie onboarding reviews the shop: the imprint (KVK, contact address),
|
|
terms and privacy pages must be real before they approve live payments.
|
|
They are — and e2e now fails the build if a PLACEHOLDER marker ever
|
|
reaches a rendered page again.
|
|
|
|
## Shipping rates: Sendcloud (optional)
|
|
|
|
Without configuration, shipping is priced by the three-zone table in
|
|
the compiled-in product data (NL / EU / world, Catcrafts.Shared-Content.cppm) — honest flat rates you set. With a
|
|
Sendcloud account, the server fetches the real per-country prices of one
|
|
shipping method daily and uses those instead, falling back to the zones for
|
|
any country the method does not cover:
|
|
|
|
```sh
|
|
# credentials from Sendcloud: Settings -> Integrations -> API
|
|
cat >> /etc/catcrafts/bunq.env <<'ENV'
|
|
SENDCLOUD_PUBLIC_KEY=...
|
|
SENDCLOUD_SECRET_KEY=...
|
|
SENDCLOUD_METHOD='PostNL Parcels non-EU,DPD Home'
|
|
ENV
|
|
systemctl restart catcrafts-server
|
|
journalctl -u catcrafts-server | grep shipping: # "table refreshed (N countries...)"
|
|
```
|
|
|
|
`SENDCLOUD_METHOD` is a comma-separated list of name substrings, merged in
|
|
order with the FIRST match per country winning — put the postal method
|
|
first so non-EU destinations get post rates (a courier method that also
|
|
covers Norway or Switzerland would otherwise price them at courier rates,
|
|
€54 instead of €19), and the courier second to fill the EU. The fetched table is
|
|
cached next to the orders file so a restart during a Sendcloud outage keeps
|
|
the last known prices. Like the bunq client, this integration is UNTESTED
|
|
against the live API until credentials exist — the response parser is covered
|
|
by --selftest, the fetch around it is thin.
|
|
|
|
The buyer sees whatever the server will charge: the checkout page embeds the
|
|
active table into its live total, and the amount is computed server-side at
|
|
order time from the same data.
|
|
|
|
## Invoice signing (GPG)
|
|
|
|
Paid orders offer a clearsigned markdown invoice at `/order/<token>/invoice.md`.
|
|
Numbering continues the pre-shop administration: one series per customer — a
|
|
random UUID as the customer number, invoices counting sequentially within it
|
|
(`f57c6512-…-3`), keyed by the buyer's email. Art. 226(2) permits "one or more
|
|
series"; completeness is provable by reconciling the append-only ledger against
|
|
the payment provider's records. The signature makes the invoice verifiable
|
|
forever, independent of this server — which is why the order page tells buyers
|
|
to download it rather than promising to host receipts indefinitely.
|
|
|
|
One-time key setup on the server, as the service user:
|
|
|
|
```sh
|
|
sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \
|
|
gpg --batch --passphrase '' --quick-gen-key 'Catcrafts invoices <invoices@catcrafts.net>' default default never
|
|
# export the PUBLIC key and commit it to the repo so buyers can verify:
|
|
sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \
|
|
gpg --armor --export invoices@catcrafts.net > invoice-key.asc
|
|
```
|
|
|
|
Then in `/etc/catcrafts/payments.env`:
|
|
|
|
```
|
|
INVOICE_GPG_KEY=invoices@catcrafts.net
|
|
```
|
|
|
|
and `Environment=GNUPGHOME=/var/lib/catcrafts/gnupg` in the service unit (see
|
|
catcrafts-server.service). The key has no passphrase because the service signs
|
|
unattended; the keyring lives in the 0700 StateDirectory. With a key configured,
|
|
a signing failure is a 500 — an unsigned invoice is never served by accident.
|
|
Without one (dev), invoices carry a visible UNSIGNED marker.
|
|
|
|
## Reading the orders ledger
|
|
|
|
```sh
|
|
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl
|
|
```
|
|
|
|
```
|
|
orders: 2
|
|
|
|
reference status total cc created token
|
|
CC-3F9A2C paid 595.00 NL 2026-08-04T14:02:11Z 3f9a2c…
|
|
CC-91B04D awaiting_payment 534.34 CA 2026-08-04T15:40:03Z 91b04d…
|
|
```
|
|
|
|
Manual transitions exist for the cases automation cannot see — a payment bunq
|
|
confirmed out-of-band, the parcel handed to the carrier, a refund:
|
|
|
|
```sh
|
|
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-paid <token>
|
|
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-shipped <token>
|
|
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --cancel <token>
|
|
```
|
|
|
|
Each appends a status event to the log — nothing is ever rewritten, so the
|
|
file remains its own audit trail. Deleting personal data on request is an edit
|
|
of the fields the seven-year fiscal retention does not cover.
|
|
|
|
## Verifying a deploy
|
|
|
|
```sh
|
|
systemctl status catcrafts-server
|
|
curl -s localhost:8081/api/healthz # ok + content counts
|
|
|
|
# real status codes, which a client-side router cannot produce
|
|
curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/nope # 404
|
|
curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/blog # 301
|
|
|
|
# the SEO check: content present with no JavaScript involved
|
|
curl -s https://catcrafts.net/projects | grep -c '<script' # 0
|
|
curl -s https://catcrafts.net/projects | grep -o '<title>[^<]*'
|
|
```
|
|
|
|
## Running it locally
|
|
|
|
```sh
|
|
tools/dev.sh # build both products and serve on :8080
|
|
tools/dev.sh --no-build # reuse what is already in bin/
|
|
```
|
|
|
|
That is the whole site: Caddy in front, the backend behind, static assets from
|
|
disk, cross-origin headers scoped exactly as in production. Ctrl-C stops both.
|
|
Orders from the session go to a temp file and are discarded on exit; the
|
|
fake payment rail is active — `touch <workdir>/orders.jsonl.fake-paid` plays
|
|
the part of the customer paying.
|
|
|
|
**Do not run `catcrafts-server --serve` alone and expect a working site.** It
|
|
serves pages only — static assets are Caddy's job — so `/styles.css` 404s and
|
|
every page renders unstyled. That looks broken but isn't; it is the deployment
|
|
split working as designed.
|
|
|
|
### Other useful commands
|
|
|
|
```sh
|
|
tools/fetch-posts.sh # pull posts from the allowed communities
|
|
tools/fetch-media.sh [dir] # mirror their media locally (run after the above)
|
|
tools/fetch-rates.sh # ECB reference rates for the indicative prices
|
|
tools/e2e.sh # 143 HTTP checks against a real server; the CI gate
|
|
crafter-build --local -r # the wasm app alone, no backend, on :8080
|
|
|
|
<server>/catcrafts-server --selftest # ~140 in-process assertions
|
|
<server>/catcrafts-server --routes # status + title for every route
|
|
<server>/catcrafts-server --render /projects # dump one page's HTML
|
|
<server>/catcrafts-server --orders FILE # the orders ledger + manual transitions
|
|
```
|
|
|
|
### If a `bin/` glob matches two directories
|
|
|
|
The variant directory name embeds a config hash, and **`--local` and
|
|
non-`--local` builds hash differently**: `--local` resolves the Crafter libraries
|
|
from sibling working trees, a plain build fetches them from Forgejo. They are
|
|
genuinely different configurations and both land in `bin/`.
|
|
|
|
Mixing them leaves two directories, and any script globbing for one picks
|
|
arbitrarily — which in practice means testing a stale binary and believing the
|
|
result. Every script here refuses to guess and tells you to `rm -rf bin`. Pick
|
|
one mode and stay in it.
|