rewrite

deploy/README.md

@@ -0,0 +1,347 @@
+# Deploying catcrafts.net
+
+Two artifacts come out of CI and land in two different places, on purpose.
+
+| | goes to | served by |
+|---|---|---|
+| wasm bundle + static assets | `/srv/catcrafts.net` | Caddy `file_server` |
+| `catcrafts-server` + `content/` | `/srv/catcrafts-app` | itself, on `127.0.0.1:8081` |
+
+They are kept apart because the web root is **publicly served and mirrored with
+`rsync --delete`** on every push. Anything runtime-owned that lives there is
+both published to the internet and destroyed on the next deploy — which matters
+now for the content files and matters a great deal once the shop has a database
+and bank credentials.
+
+Runtime state goes in a third place, `/var/lib/catcrafts`, created by the
+service's `StateDirectory=`. Today that is `orders.jsonl` (the order event log)
+and `bunq-state.json` (the bunq session context, including the client RSA key
+the server generates on first contact).
+
+**Two things on this box cannot be regenerated.** Everything else — the wasm
+bundle, the content, the binary — comes back from a rebuild.
+
+The first is `orders.jsonl`. It is the ledger: every order and every status
+transition, append-only, and the audit trail the tax records lean on. Back it
+up:
+
+```sh
+install -d -m 0700 /var/backups/catcrafts
+cp /var/lib/catcrafts/orders.jsonl \
+   /var/backups/catcrafts/orders-$(date -u +%F).jsonl
+```
+
+It contains names, addresses and email addresses, so it is personal data: keep
+it 0600, keep it off the web root, and encrypt it before it leaves the machine.
+(`bunq-state.json` is deliberately NOT worth backing up: delete it and the
+server re-onboards from the API key on the next start.)
+
+The second is `/srv/catcrafts-app/media` — the mirrored post images and screen
+recordings. Usually reproducible from `content/posts.json`, but **not if a source
+instance has deleted the file since**, and for these posts the media *is* the
+content. It lives on the app mount rather than the web root specifically so
+`rsync --delete` cannot reach it, and CI only ever adds to it. Worth including in
+the same backup.
+
+## Why the backend speaks plaintext HTTP/1.1
+
+Caddy terminates TLS and reverse-proxies to loopback. The backend uses
+Crafter.Network's `ListenerHTTP1` rather than its HTTP/3 `ListenerHTTP` for one
+concrete reason: **Caddy cannot `reverse_proxy` to an h3 upstream.** Port 8081
+must never be exposed directly.
+
+## One-time host setup
+
+```sh
+# 1. service account, no login, no home
+useradd --system --no-create-home --shell /usr/sbin/nologin catcrafts
+
+# 2. directories
+mkdir -p /srv/catcrafts.net /srv/catcrafts-app
+chown catcrafts:catcrafts /srv/catcrafts-app
+# The web root stays writable by whatever uid the runner container uses.
+
+# 3. units
+cp deploy/catcrafts-server.service /etc/systemd/system/
+cp deploy/catcrafts-deploy.service /etc/systemd/system/
+cp deploy/catcrafts-deploy.path    /etc/systemd/system/
+systemctl daemon-reload
+systemctl enable --now catcrafts-deploy.path
+# catcrafts-server is started by the first deploy; enable it so it survives a
+# reboot:
+systemctl enable catcrafts-server
+
+# 4. Caddy — merge deploy/Caddyfile.example into your site config
+caddy validate --config /etc/caddy/Caddyfile
+systemctl reload caddy
+```
+
+## Runner configuration
+
+Both mounts are required. In the Forgejo runner's `config.yaml`:
+
+```yaml
+container:
+  options: "-v /srv/catcrafts.net:/deploy -v /srv/catcrafts-app:/deploy-app"
+```
+
+The deploy steps fail with an explanatory message if either is missing, rather
+than silently succeeding into the container's own filesystem.
+
+## How the restart happens
+
+CI runs **inside a container** and has no access to the host's systemd. Rather
+than give the runner host root or a polkit rule — both of which grant far more
+authority than "restart one service" needs — the last deploy step writes
+`/srv/catcrafts-app/.deploy-stamp`, and `catcrafts-deploy.path` on the host
+reacts.
+
+`catcrafts-deploy.service` runs `catcrafts-server --selftest` as `ExecStartPre`
+before restarting. That self-test exercises the HTML-escaping and JSON layers
+that generate every byte of markup the site emits, so **a broken binary leaves
+the working server running** instead of replacing it.
+
+The binary is installed as `catcrafts-server.new` and `mv`d into place, so a
+request arriving mid-copy never hits a truncated executable.
+
+## If the backend is down
+
+Caddy's `handle_response @down` falls back to serving the static `index.html`,
+so the site degrades to the client-rendered wasm app rather than showing a 502.
+Content still renders; what is lost is server-side rendering and real status
+codes — an unknown path becomes a soft 404 again until the backend returns.
+
+## Posts and their media
+
+`content/posts-sources.json` holds the account and a **community allowlist**.
+Only listed communities are fetched, so joining a new one does not silently
+publish it to the site — add a line first.
+
+```sh
+tools/fetch-posts.sh     # writes content/posts.json
+tools/fetch-media.sh     # mirrors the media, rewrites posts.json to /media/... paths
+```
+
+Order matters: the second reads what the first wrote. CI runs both before the
+build. Neither fails the build on a network error — a fediverse outage leaves the
+previous `posts.json` in place, and a single failed download leaves that one entry
+pointing at its original URL rather than losing the post.
+
+**`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions,
+which become the `width`/`height` attributes that stop the page reflowing as
+several 5 MB recordings arrive. It degrades to no dimensions without it —
+so `tools/e2e.sh` asserts they are present, and a build host missing the package
+fails at the e2e gate instead of quietly shipping a janky page.
+
+**Only the post's main media is used.** Lemmy distinguishes the media a post *is*
+(`post.url`, with `post.thumbnail_url` as its generated still) from images merely
+embedded in the body. Only the former is mirrored — a post whose pictures are
+body-only renders as text, which is correct. Videos get the thumbnail as their
+`poster`; without it a `preload="metadata"` video is a black box until someone
+presses play.
+
+**Thread links point at the community's instance, not the author's.** A post's
+`ap_id` is on the account's own instance, but the discussion is in the community,
+which is federated elsewhere and has its own local id for the same thread. So
+`fetch-posts.sh` resolves each `ap_id` through the community instance's
+`resolve_object` and stores that URL. One request per post, at build time; a
+failure falls back to the `ap_id`, which still reaches a readable copy.
+
+Media is **mirrored, not hotlinked**. Three reasons: the privacy notice says
+everything the browser loads comes from catcrafts.net and should stay true;
+hotlinking would send every visitor's IP to the source instance; and these posts
+are their media, so a deleted upstream file would gut the page. Filenames are the
+content hash, which is why the cache lifetime can be a year.
+
+## Payments: Mollie setup
+
+The rail is Mollie (bunq.me was measured and disqualified: €500/transaction on
+cards and no method at all for a non-EU buyer at phone prices — it is a P2P
+tool; the bunq client remains in the tree, unused, in case an account sweep is
+ever wanted). The server needs exactly one secret: the Mollie API key.
+
+```sh
+# Keys live in the Mollie dashboard: Developers -> API keys. A test_… key
+# works against the real API from the moment the account exists — verify the
+# whole flow with it BEFORE swapping in the live_… key.
+install -d -m 0755 /etc/catcrafts
+cat > /etc/catcrafts/payments.env <<'ENV'
+MOLLIE_API_KEY=test_your-key-here
+ENV
+chmod 0600 /etc/catcrafts/payments.env
+systemctl restart catcrafts-server
+journalctl -u catcrafts-server | tail   # should say "payments: mollie"
+```
+
+Without the env file the server starts with payments off: the whole site works,
+the product page renders, and checkout answers 503 with an honest message —
+degraded, not down.
+
+Mechanics worth knowing:
+
+* The reconciler polls each open order (`GET /v2/payments/{id}`) every 10 s
+  while fresh, backing off with age. `?redirect` back from Mollie is ignored
+  by design — only the authenticated poll moves an order to paid, and the
+  paid event records the method (`ideal`, `creditcard`, …) in the ledger.
+* Mollie payments EXPIRE. A payment that reaches canceled/expired/failed
+  lapses the order automatically — the buyer just orders again.
+* Card money stays disputable for months even after "paid": before shipping a
+  large or exported order, glance at the `via` column in `--orders`. iDEAL
+  and bank transfers are final; `creditcard` is the one with a tail.
+* Mollie onboarding reviews the shop: the imprint (KVK, contact address),
+  terms and privacy pages must be real before they approve live payments.
+  They are — and e2e now fails the build if a PLACEHOLDER marker ever
+  reaches a rendered page again.
+
+## Shipping rates: Sendcloud (optional)
+
+Without configuration, shipping is priced by the three-zone table in
+the compiled-in product data (NL / EU / world, Catcrafts.Shared-Content.cppm) — honest flat rates you set. With a
+Sendcloud account, the server fetches the real per-country prices of one
+shipping method daily and uses those instead, falling back to the zones for
+any country the method does not cover:
+
+```sh
+# credentials from Sendcloud: Settings -> Integrations -> API
+cat >> /etc/catcrafts/bunq.env <<'ENV'
+SENDCLOUD_PUBLIC_KEY=...
+SENDCLOUD_SECRET_KEY=...
+SENDCLOUD_METHOD='PostNL Parcels non-EU,DPD Home'
+ENV
+systemctl restart catcrafts-server
+journalctl -u catcrafts-server | grep shipping:   # "table refreshed (N countries...)"
+```
+
+`SENDCLOUD_METHOD` is a comma-separated list of name substrings, merged in
+order with the FIRST match per country winning — put the postal method
+first so non-EU destinations get post rates (a courier method that also
+covers Norway or Switzerland would otherwise price them at courier rates,
+€54 instead of €19), and the courier second to fill the EU. The fetched table is
+cached next to the orders file so a restart during a Sendcloud outage keeps
+the last known prices. Like the bunq client, this integration is UNTESTED
+against the live API until credentials exist — the response parser is covered
+by --selftest, the fetch around it is thin.
+
+The buyer sees whatever the server will charge: the checkout page embeds the
+active table into its live total, and the amount is computed server-side at
+order time from the same data.
+
+## Invoice signing (GPG)
+
+Paid orders offer a clearsigned markdown invoice at `/order/<token>/invoice.md`.
+Numbering continues the pre-shop administration: one series per customer — a
+random UUID as the customer number, invoices counting sequentially within it
+(`f57c6512-…-3`), keyed by the buyer's email. Art. 226(2) permits "one or more
+series"; completeness is provable by reconciling the append-only ledger against
+the payment provider's records. The signature makes the invoice verifiable
+forever, independent of this server — which is why the order page tells buyers
+to download it rather than promising to host receipts indefinitely.
+
+One-time key setup on the server, as the service user:
+
+```sh
+sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \
+  gpg --batch --passphrase '' --quick-gen-key 'Catcrafts invoices <invoices@catcrafts.net>' default default never
+# export the PUBLIC key and commit it to the repo so buyers can verify:
+sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \
+  gpg --armor --export invoices@catcrafts.net > invoice-key.asc
+```
+
+Then in `/etc/catcrafts/payments.env`:
+
+```
+INVOICE_GPG_KEY=invoices@catcrafts.net
+```
+
+and `Environment=GNUPGHOME=/var/lib/catcrafts/gnupg` in the service unit (see
+catcrafts-server.service). The key has no passphrase because the service signs
+unattended; the keyring lives in the 0700 StateDirectory. With a key configured,
+a signing failure is a 500 — an unsigned invoice is never served by accident.
+Without one (dev), invoices carry a visible UNSIGNED marker.
+
+## Reading the orders ledger
+
+```sh
+catcrafts-server --orders /var/lib/catcrafts/orders.jsonl
+```
+
+```
+orders: 2
+
+reference  status             total  cc  created               token
+CC-3F9A2C  paid              595.00  NL  2026-08-04T14:02:11Z  3f9a2c…
+CC-91B04D  awaiting_payment  534.34  CA  2026-08-04T15:40:03Z  91b04d…
+```
+
+Manual transitions exist for the cases automation cannot see — a payment bunq
+confirmed out-of-band, the parcel handed to the carrier, a refund:
+
+```sh
+catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-paid    <token>
+catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-shipped <token>
+catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --cancel       <token>
+```
+
+Each appends a status event to the log — nothing is ever rewritten, so the
+file remains its own audit trail. Deleting personal data on request is an edit
+of the fields the seven-year fiscal retention does not cover.
+
+## Verifying a deploy
+
+```sh
+systemctl status catcrafts-server
+curl -s localhost:8081/api/healthz            # ok + content counts
+
+# real status codes, which a client-side router cannot produce
+curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/nope        # 404
+curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/blog        # 301
+
+# the SEO check: content present with no JavaScript involved
+curl -s https://catcrafts.net/projects | grep -c '<script'              # 0
+curl -s https://catcrafts.net/projects | grep -o '<title>[^<]*'
+```
+
+## Running it locally
+
+```sh
+tools/dev.sh              # build both products and serve on :8080
+tools/dev.sh --no-build   # reuse what is already in bin/
+```
+
+That is the whole site: Caddy in front, the backend behind, static assets from
+disk, cross-origin headers scoped exactly as in production. Ctrl-C stops both.
+Orders from the session go to a temp file and are discarded on exit; the
+fake payment rail is active — `touch <workdir>/orders.jsonl.fake-paid` plays
+the part of the customer paying.
+
+**Do not run `catcrafts-server --serve` alone and expect a working site.** It
+serves pages only — static assets are Caddy's job — so `/styles.css` 404s and
+every page renders unstyled. That looks broken but isn't; it is the deployment
+split working as designed.
+
+### Other useful commands
+
+```sh
+tools/fetch-posts.sh            # pull posts from the allowed communities
+tools/fetch-media.sh [dir]      # mirror their media locally (run after the above)
+tools/fetch-rates.sh            # ECB reference rates for the indicative prices
+tools/e2e.sh                    # 143 HTTP checks against a real server; the CI gate
+crafter-build --local -r        # the wasm app alone, no backend, on :8080
+
+<server>/catcrafts-server --selftest            # ~140 in-process assertions
+<server>/catcrafts-server --routes              # status + title for every route
+<server>/catcrafts-server --render /projects    # dump one page's HTML
+<server>/catcrafts-server --orders FILE         # the orders ledger + manual transitions
+```
+
+### If a `bin/` glob matches two directories
+
+The variant directory name embeds a config hash, and **`--local` and
+non-`--local` builds hash differently**: `--local` resolves the Crafter libraries
+from sibling working trees, a plain build fetches them from Forgejo. They are
+genuinely different configurations and both land in `bin/`.
+
+Mixing them leaves two directories, and any script globbing for one picks
+arbitrarily — which in practice means testing a stale binary and believing the
+result. Every script here refuses to guess and tells you to `rm -rf bin`. Pick
+one mode and stay in it.