catcrafts.net/deploy/README.md

# Deploying catcrafts.net

Two artifacts come out of CI and land in two different places, on purpose.

| | goes to | served by |
|---|---|---|
| wasm bundle + static assets | `/srv/catcrafts.net` | Caddy `file_server` |
| `catcrafts-server` + `content/` | `/srv/catcrafts-app` | itself, on `127.0.0.1:8081` |

They are kept apart because the web root is **publicly served and mirrored with
`rsync --delete`** on every push. Anything runtime-owned that lives there is
both published to the internet and destroyed on the next deploy — which matters
now for the content files and matters a great deal once the shop has a database
and bank credentials.

Runtime state goes in a third place, `/var/lib/catcrafts`, created by the
service's `StateDirectory=`. Today that is `orders.jsonl` (the order event log),
`orders.jsonl.shipping.json` (the cached carrier rate table) and
`orders.jsonl.financials.json` (the published running totals behind the public
/financials page — see "The open financials page"). Neither payment provider
needs stored state — both authenticate with a bearer token per request.

**Two things on this box cannot be regenerated.** Everything else — the wasm
bundle, the content, the binary — comes back from a rebuild.

The first is `orders.jsonl`. It is the ledger: every order and every status
transition, append-only, and the audit trail the tax records lean on. Back it
up:

```sh
install -d -m 0700 /var/backups/catcrafts
cp /var/lib/catcrafts/orders.jsonl \
   /var/backups/catcrafts/orders-$(date -u +%F).jsonl
```

It contains names, addresses and email addresses, so it is personal data: keep
it 0600, keep it off the web root, and encrypt it before it leaves the machine.
(The cached shipping table is deliberately NOT worth backing up: delete it and
the next Sendcloud refresh rebuilds it. The financials aggregates file is a few
hundred bytes the owner's tooling rewrites — keep a copy with the ledger backup
all the same, since nothing on this box can regenerate it.)

The second is `/srv/catcrafts-app/media` — the mirrored post images and screen
recordings. Usually reproducible from `content/posts.json`, but **not if a source
instance has deleted the file since**, and for these posts the media *is* the
content. It lives on the app mount rather than the web root specifically so
`rsync --delete` cannot reach it, and CI only ever adds to it. Worth including in
the same backup.

## Why the backend speaks plaintext HTTP/1.1

Caddy terminates TLS and reverse-proxies to loopback. The backend uses
Crafter.Network's `ListenerHTTP1` rather than its HTTP/3 `ListenerHTTP` for one
concrete reason: **Caddy cannot `reverse_proxy` to an h3 upstream.** Port 8081
must never be exposed directly.

## One-time host setup

```sh
# 1. service account, no login, no home
useradd --system --no-create-home --shell /usr/sbin/nologin catcrafts

# 2. directories
mkdir -p /srv/catcrafts.net /srv/catcrafts-app
chown catcrafts:catcrafts /srv/catcrafts-app
# The web root stays writable by whatever uid the runner container uses.

# 3. units
cp deploy/catcrafts-server.service /etc/systemd/system/
cp deploy/catcrafts-deploy.service /etc/systemd/system/
cp deploy/catcrafts-deploy.path    /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now catcrafts-deploy.path
# catcrafts-server is started by the first deploy; enable it so it survives a
# reboot:
systemctl enable catcrafts-server

# 4. Caddy — merge deploy/Caddyfile.example into your site config
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy
```

## Runner configuration

Both mounts are required. In the Forgejo runner's `config.yaml`:

```yaml
container:
  options: "-v /srv/catcrafts.net:/deploy -v /srv/catcrafts-app:/deploy-app"
```

The deploy steps fail with an explanatory message if either is missing, rather
than silently succeeding into the container's own filesystem.

## How the restart happens

CI runs **inside a container** and has no access to the host's systemd. Rather
than give the runner host root or a polkit rule — both of which grant far more
authority than "restart one service" needs — the last deploy step writes
`/srv/catcrafts-app/.deploy-stamp`, and `catcrafts-deploy.path` on the host
reacts.

`catcrafts-deploy.service` runs `catcrafts-server --render /` as `ExecStartPre`
before restarting. The unit tests already gated the deploy in CI
(`crafter-build test --product=server`); this on-host check covers what CI
cannot — the dynamic loader finding libmsquic on this machine and a page
rendering from the deployed content — so **a broken binary leaves the working
server running** instead of replacing it.

The binary is installed as `catcrafts-server.new` and `mv`d into place, so a
request arriving mid-copy never hits a truncated executable.

## If the backend is down

Caddy's `handle_errors` fallback serves the static `index.html`,
so the site degrades to the client-rendered wasm app rather than showing a 502.
Content still renders; what is lost is server-side rendering and real status
codes — an unknown path becomes a soft 404 again until the backend returns.

## Posts and their media

`content/posts-sources.json` holds the account and a **community allowlist**.
Only listed communities are fetched, so joining a new one does not silently
publish it to the site — add a line first.

```sh
tools/publish-media.sh FILE  # BEFORE posting: uploads to /media, prints the URL
tools/fetch-posts.sh         # writes content/posts.json
tools/fetch-media.sh         # mirrors the media, rewrites posts.json to /media/... paths
```

Order matters: the second reads what the first wrote. CI runs both before the
build. Neither fails the build on a network error — a fediverse outage leaves the
previous `posts.json` in place, and a single failed download leaves that one entry
pointing at its original URL rather than losing the post.

### The full body, and its inline media

Each post is hosted whole at `/posts/<slug>`, with the card on `/posts` linking
to it — the writing is what the site is about, and a page this site can name as
canonical is the only version a search engine can be pointed at. The comments
are *not* mirrored: every post page links out to the thread, which is where the
discussion belongs.

The body stays Markdown in `posts.json` and is rendered by
`Catcrafts.Shared:Markdown` at page-render time, never converted to HTML by the
shell. That is deliberate: the renderer is inside the escaping guarantee, and
text fetched from someone else's server must not be able to become markup
anywhere else. Raw HTML in a body is always shown as text.

`fetch-media.sh` mirrors what the body embeds as well as the headline file, and
rewrites the URLs **inside the Markdown**, so a post page loads nothing
third-party either. It also writes a `body_media` list per post — dimensions,
poster frame and format renditions for each inline file, which Markdown syntax
has nowhere to carry. Slugs come from the title; a duplicate title takes the
post's numeric id as a suffix, so an old post's URL is never renumbered by a new
one. Re-running the script is a no-op: local paths are adopted from the mount
rather than re-fetched.

### The image format ladder

Every mirrored still image is transcoded to two siblings named after its content
hash, and `Catcrafts.Shared:Media` serves all three as one `<picture>` — so the
browser fetches **exactly one**:

| tier | file | size vs. WebP | who gets it |
|---|---|---|---|
| `<source type="image/avif">` | `<hash>.avif` | **76%** | almost everyone |
| `<source type="image/webp">` | `<hash>.webp` (the mirrored original) | 100% | Safari 14–16 |
| `<img src>` | `<hash>.png` | **875%** | neither of the above |

The middle tier is why the PNG being ~9× the WebP does not matter: it is free
(the mirror already downloaded that file) and it is what the small number of
non-AVIF browsers actually land on. The PNG is the floor nothing can refuse.

AVIF is encoded at `crf 26, cpu-used 6, yuv444p` — measured at SSIM 0.997
against the source and still smaller than it. Full chroma is deliberate: these
are screenshots of text, and re-subsampling chroma that pict-rs already
subsampled once fringes coloured text visibly, for about 3% more bytes.

Encoding is skipped when the sibling is already on the mount, so only genuinely
new images cost encoder time (~0.5 s each). Animated sources are left alone
entirely — one moving GIF beats three copies of its first frame. Video posters
are skipped too: `poster` takes exactly one URL, so a `<video>` cannot negotiate
a format the way `<picture>` can and the renditions would be unreachable.

Both encodes pin `-c:v` and then **verify the codec that actually came out**.
That check earned its place immediately: `-f image2 out.png` without an explicit
codec makes ffmpeg fall back to the muxer default, which is MJPEG — it silently
produced a full set of lossy JPEGs under `.png` names, served to browsers as
`image/png`. A rendition that fails the check is discarded and its tier dropped.

### Publish the media first, then post it

**The recommended flow is to put a recording on catcrafts.net before writing the
post, and use that URL as the post's link.** Run `tools/publish-media.sh
recording.mp4`; it transcodes to AV1 *and* an H.264 sibling, uploads both to the
media mount under the AV1's content hash, and prints a
`https://catcrafts.net/media/<hash>.h264.mp4` URL to paste into the post. The
H.264 one on purpose: a post's link is fetched raw — Lemmy apps and browsers play
that exact file, with no negotiation in front of it — so it must be the encoding
everything can play. (An AV1 link posted before this existed drew "bad media
error" reports from iPhones within hours.)

`fetch-media.sh` recognises its own origin and **adopts** such a URL: it rewrites
it to `/media/<hash>` and probes the local file for dimensions, downloading
nothing. That is not just an optimisation — it removes the whole class of build
failure where the media step depends on a third party. A 167 MB recording on a
file host once hit `MAX_BYTES` (64 MB), so the entry kept its original URL, and
the deploy then failed the "media origin" check (ShouldServePostPages) on a file that was sitting on
our own disk the entire time.

Adopting a `<hash>.h264.mp4` URL swaps the AV1 sibling back in as the page's
primary `<source>` when it is on the mount, keeping the H.264 as the fallback
`<source>`. So the post links the compatible file, while browsers that can take
AV1 download the small one — the `codecs` parameter on the first source is what
lets the rest skip it.

The transcode matters as much as the hosting. Phone recordings are wildly
oversized for what they show — that same 167 MB clip was 78 s of a dark room at
17 Mbps, and denoising into AV1 gives the same picture in 15 MB. It also **bakes
in the rotation**: phones record landscape and attach a display matrix, so an
untouched file reports 1920x1080 while playing portrait, and the `width`/`height`
attributes then reserve exactly the wrong box. (`fetch-media.sh` swaps the
dimensions when it sees a quarter-turn matrix, so a straight-from-phone mirror is
correct too — but transcoding means nothing downstream has to know.)

Two things to know about publishing AV1:

* Browsers without AV1 (Safari before 17, Apple hardware older than A17/M3) get
  the `<hash>.h264.mp4` sibling: on the site via the second `<source>`, on the
  fediverse because that sibling *is* the posted URL. `--raw` skips the
  transcode and the fallback both, so a raw AV1 upload recreates the
  will-not-play problem — use it for files that are already universally
  playable.
* Lemmy's `pict-rs` will not generate a thumbnail from an AV1 file, so a
  self-hosted video usually arrives with no `poster`. `publish-media.sh` uploads
  a poster frame beside the video, named `<video-hash>.poster.webp`, and
  `fetch-media.sh` falls back to that sibling when the instance supplied nothing.
  A thumbnail the instance *did* provide always wins. (Posting the H.264 URL
  also means pict-rs can thumbnail it again, so instance thumbnails come back.)

An own-origin URL naming a file that is **not** on the mount is deliberately left
pointing at its original URL rather than rewritten. That is a post published
without its media, and failing the origin check (ShouldServePostPages) loudly beats shipping a 404
inside a `<video>` tag.

**`fetch-media.sh` wants `ffprobe`** (Arch: `ffmpeg`) to read pixel dimensions,
which become the `width`/`height` attributes that stop the page reflowing as
several 5 MB recordings arrive. It degrades to no dimensions without it —
so the ShouldServePostPages suite asserts they are present, and a build host
missing the package fails at the test gate instead of quietly shipping a janky
page.

**Only the post's main media is used.** Lemmy distinguishes the media a post *is*
(`post.url`, with `post.thumbnail_url` as its generated still) from images merely
embedded in the body. Only the former is mirrored — a post whose pictures are
body-only renders as text, which is correct. Videos get the thumbnail as their
`poster`; without it a `preload="metadata"` video is a black box until someone
presses play.

**Thread links point at the community's instance, not the author's.** A post's
`ap_id` is on the account's own instance, but the discussion is in the community,
which is federated elsewhere and has its own local id for the same thread. So
`fetch-posts.sh` resolves each `ap_id` through the community instance's
`resolve_object` and stores that URL. One request per post, at build time; a
failure falls back to the `ap_id`, which still reaches a readable copy.

Media is **mirrored, not hotlinked**. Three reasons: the privacy notice says
everything the browser loads comes from catcrafts.net and should stay true;
hotlinking would send every visitor's IP to the source instance; and these posts
are their media, so a deleted upstream file would gut the page. Filenames are the
content hash, which is why the cache lifetime can be a year.

## Payments: bank transfer and EURC

Checkout offers the buyer **two choices**, each served by its own rail:

| Choice   | Rail     | Env var             | What the buyer gets                     |
|----------|----------|---------------------|-----------------------------------------|
| `bank`   | transfer | `TRANSFER_IBAN`     | a plain SEPA transfer to our own account |
| `crypto` | EURC     | `EURC_CHAINS`       | self-hosted EURC on the configured EVM chains |

**Both rails are self-hosted, and that is deliberate rather than
incidental.** On 2026-08-20 the shop's hosted payment provider closed its
account after a risk review, with no appeal and no reason given beyond
"outside our acceptance criteria". Every payment method the shop offered
through that provider died in one email. The rails above cannot be switched
off by a third party: one is a transfer to our own bank account, the other is
an address we generated ourselves. The hosted rail's implementation has been
removed from the tree — the decision was final, and a dead integration costs a
CI gate, a secret, and confusion about which rail is actually serving.

What this costs, stated honestly so nobody re-litigates it from memory: there
is no iDEAL and there are no cards. iDEAL requires an acquiring contract with
a bank (or a PSP), which is the same kind of relationship that just ended, and
cards additionally require PCI DSS SAQ D plus an EMVCo-certified 3DS server.
In the last measured week before the change, cards and Bancontact were 40% of
donation value, so this is a real loss of reach and not a free win.

The slots are independent. Set one and the form offers only that method;
set both and the buyer picks; set neither and checkout answers 503 with an
honest message — the whole site still works, degraded rather than down. A
mode whose configuration is missing is a **startup refusal**, not a silent
downgrade: a checkout that 502s at the last step is worse than one that
never offered.

### The bank-transfer rail

No processor and no credential: the buyer sends a normal SEPA transfer to our
IBAN quoting the order's reference, and the server settles the order when a
matching credit appears on the account. The order page is the payment page.

- `TRANSFER_IBAN=NL..` — the account the money lands in. Setting this IS
  selecting the rail, the same convention as `EURC_CHAINS`.
- `TRANSFER_BENEFICIARY='J. van der Graaf'` — **the account-holder name
  exactly as the bank holds it, character for character.** Not the trading
  name. Since 2025-10-09 every euro-area transfer is name-checked against the
  IBAN by Verification of Payee, and the payer sees a mismatch warning at the
  moment of paying; a friendly-looking "Catcrafts" here would scare buyers off
  at the last step. Startup refuses without it, because an order page missing
  this loses the money rather than the sale.
- `TRANSFER_BIC` — optional. Shown on the order page only when set, and
  labelled "only if your bank asks for it, usually outside Europe". Inside SEPA
  an IBAN alone has been sufficient since 2016, so an unconditional BIC row is
  just one more field for a Dutch buyer to fill in and mistype; a payer sending
  by SWIFT from outside SEPA genuinely cannot proceed without it.
- `TRANSFER_WINDOW_HOURS` (default 336, i.e. 14 days) — how long an order
  waits before it lapses. Generous on purpose: a non-instant transfer from
  outside the euro area can legitimately take a business day, and there is no
  provider-side expiry forcing our hand. A lapsed order is **not** bounced
  money: the IBAN stays ours, a late payment still arrives, and it is settled
  with `--mark-paid`. The server logs exactly this when it lapses one.
- `TRANSFER_CREDITS` (default `<orders>.transfer-credits.jsonl`) — where the
  rail reads incoming credits from. One JSON object per line:
  `{"id":"…","reference":"…","amount_minor":2500,"method":"sepa"}`. A missing
  file is an empty list, not an error, since a shop that has taken no
  transfers yet has no file.

#### Filling the credits file: the bunq reader

`--pull-credits` reads the bunq account once and appends anything new to the
credits file, deduplicated by bunq's own payment id, then prints how many
arrived. Every pull re-reads an overlapping window, so that dedupe is what
stops a credit being counted twice and settling an order nobody paid twice
for. It appends rather than rewrites, so lines added by hand survive.

```sh
BUNQ_API_KEY=… TRANSFER_IBAN=NL.. \
  catcrafts-server --pull-credits --orders /var/lib/catcrafts/orders.jsonl
# -> pulled 2 new credit(s) into /var/lib/catcrafts/orders.jsonl.transfer-credits.jsonl
```

**Read this before deciding where to put the key.** A bunq API key can
**initiate payments**, and bunq offers no read-only scope, so there is no such
thing as a key that can only read. That gives two deployment shapes, and they
are not equivalent:

1. **Key off the public host (intended).** Run `--pull-credits` on a trusted
   machine on a timer, and ship the credits file to the server (rsync, scp,
   whatever). The server settles orders while holding **no credential that can
   move a cent**, so even a full compromise of the web host cannot spend the
   account. This is the same rule the EURC rail follows by never holding a
   wallet key.
2. **Key on the server (simpler, strictly worse).** Set `BUNQ_API_KEY` in
   `payments.env` and the rail reads the bank itself, no file shipping. The
   startup log prints a warning saying exactly what has been traded away.

Either way, set **`BUNQ_PERMITTED_IPS`** to the egress address of whatever
machine holds the key. bunq registers it once, at device-server time, and it is
the only thing standing between a leaked key and someone spending the balance.
The default is `*`, which works and is announced loudly. Changing it later
means deleting the context file and re-onboarding.

Other bunq settings: `BUNQ_STATE` (default `<credits>.bunq-context.json`) holds
the generated RSA keypair and the session tokens at 0600 — **back it up or at
least know that deleting it forces a re-onboard**, and bunq allows as few as
ten setup calls PER DAY, so do not delete it casually. If the key can see more
than one active account, `TRANSFER_IBAN` picks which; without it, a key seeing
several accounts is a refusal rather than a guess, because guessing means
reconciling the shop against its savings.

Rate limits worth respecting: bunq caps reads at roughly **3 GET per 3
seconds** per method. One pull is one request, and the rail additionally shares
a single read across all orders in a sweep, so neither path is anywhere near
the cap. What WOULD hit it is retrying the onboarding calls in a loop; the
context file is what prevents that.

The file source stays regardless — it is how the suites drive real settlement
with no network, and it is the manual escape hatch for settling a transfer by
hand. `--mark-paid` remains the other one.

Two matching properties worth knowing, both pinned in
`ShouldMatchBankTransfers`:

* **The reference matches in whatever form the payer typed it.** Both sides
  are reduced to upper-case alphanumerics and the match is on the reference
  *body*, so `CC-2B6457`, `cc2b6457`, `CC 2B 64 57` and the structured
  `RF70CC2B6457` all settle the same order. The order page prints the
  structured ISO 11649 form as well, because banks with a dedicated
  payment-reference field validate its check digits and refuse a mistyped one
  *before the money leaves* — which is what makes unattended matching safe.
* **Partial payments accumulate.** Two credits quoting one reference are
  summed, which is what makes the page's "send the rest the same way" true.
  Outgoing amounts are ignored, so a refund quoting the reference cannot pay
  for the order it refunded.

One known limitation, logged rather than solved: a single transfer quoting
**two** order references cannot be attributed by a per-order matcher, and both
orders would settle on the same money. The rail detects the shape and logs
"settle this one by hand"; at this volume that is proportionate, but do not
assume the matcher is total.

### The EURC rail

No processor, no API key, no account anywhere: the buyer sends EURC (Circle's
euro stablecoin, pegged 1:1 — the token amount IS the euro total) to an
address from a pool this shop generated offline, and the server notices by
polling `balanceOf` over JSON-RPC at a finalized block. The order page is the
payment page. Three pieces of configuration:

- `EURC_CHAINS=/etc/catcrafts/eurc-chains.json` — which chains to watch.
  Selecting this rail IS setting this variable. One EVM address is valid on
  every chain at once, so the buyer pays on whichever is cheapest; **file
  order is display order** — put the cheap chain (Base) first, and its
  `note` ("lowest network fees") is the nudge the buyer sees.

  ```json
  {"chains": [
    {"name": "base", "rpc": "https://mainnet.base.org",
     "contract": "0x60a3E35Cc302bFA44Cb288Bc5a4F316Fdb1adb42",
     "chain_id": 8453, "note": "lowest network fees"},
    {"name": "ethereum", "rpc": "https://ethereum-rpc.publicnode.com",
     "contract": "0x1aBaEA1f7C830bD89Acc67eC4af516284b1bC33c",
     "chain_id": 1}
  ]}
  ```

  Contract addresses come from Circle's own list
  (developers.circle.com/stablecoins/eurc-contract-addresses) and nowhere
  else — matching the CONTRACT, not the ticker, is what makes a fake "EURC"
  worthless here. EURC exists on Ethereum, Base, Avalanche, Cronos and World
  Chain (not Arbitrum/Optimism/Polygon/BNB).

- `EURC_POOL` (default `<orders>.eurc-addresses`) — one receiving address
  per line, `#` comments allowed. Generated by the wallet at home; the box
  holds NO key, NO xpub, and can only hand out addresses it was given.
  **Copy addresses, never retype**: the server cannot verify the EIP-55
  checksum, so a retyped-but-still-hex address is accepted and published.
  **Append only**: `<pool>.cursor` is an index into this file, so reordering
  re-issues addresses already bound to old orders. Duplicates and malformed
  lines are startup refusals; exhaustion too — the low-water warning fires
  at 25 left.

- `EURC_WINDOW_HOURS` (default 24) — how long an order may sit unpaid. A
  lapsed window is NOT bounced money: the address stays ours, a late payment
  still lands there (the server logs exactly this), and it is settled by
  hand with `--mark-paid`.

```sh
# Neither rail has a credential, so neither has a "test mode" to rehearse in:
# what selects each one is naming where the money lands. To rehearse the crypto
# rail against worthless tokens, point EURC_CHAINS at a testnet chains file
# (Ethereum Sepolia / Base Sepolia contracts are in Circle's list). To rehearse
# the bank rail, run it and write a credits line by hand — see the transfer
# section above, or tools/dev-credit.sh locally.
install -d -m 0755 /etc/catcrafts
cat > /etc/catcrafts/payments.env <<'ENV'
TRANSFER_IBAN=NL..
TRANSFER_BENEFICIARY=exactly as the bank holds it
TRANSFER_BIC=BUNQNL2A
EURC_CHAINS=/etc/catcrafts/eurc-chains.json
ENV
chmod 0600 /etc/catcrafts/payments.env
systemctl restart catcrafts-server
journalctl -u catcrafts-server | tail  # "payments: bank=transfer crypto=eurc"
```

Mechanics worth knowing:

* The reconciler polls each open order against **the rail that issued its
  link** — the ledger records `pay_choice` per order for exactly this reason.
  the bank rail every 60 s (and it shares ONE account read across the whole
  sweep, so a dozen open orders is still one request), the EURC rail every
  30 s (a finalized block will not arrive faster), both backing off with age. Arriving back on the order page
  is ignored as evidence by design: only the rail's own authenticated read of
  the bank account, or its RPC check on-chain, moves an order to paid, and the paid event
  records the method (`ideal`, `creditcard`, `eurc-base`, …) in the ledger.
* **Unpaid orders lapse automatically** and the buyer just orders again.
  the bank rail closes its window after 14 days; the EURC rail after
  `EURC_WINDOW_HOURS` (24 by default). A lapsed EURC order is NOT bounced
  money — see above.
* The shop **holds EURC**: crypto revenue sits at the pool addresses until
  swept, and nothing lands on the bank account until it is sold for euros at
  an exchange. Bookkeeping note: a EURC sale is still a euro sale on the
  invoice — EURC is euro-denominated, so there is no rate and no revaluation —
  but the financials bank-callback never sees it, so crypto sales exist in
  the ledger only until the sweep's SEPA leg arrives.
* Card money stays disputable for months even after "paid": before shipping a
  large or exported order, glance at the `via` column in `--orders`. Bank
  transfers and EURC are final; `creditcard` is the one with a tail. This is
  the one real advantage of the crypto rail — no chargebacks — and it matters
  most on exactly the non-EU orders where cards get declined.
* **iDEAL's finality has an expiry date, and it is inside the launch window.**
  Written here because it invalidates the obvious shipping rule: "iDEAL says
  paid, so it is safe to post" stops being true as iDEAL becomes Wero through
  2026-2027. Wero carries a **120-calendar-day dispute window** for online
  purchases (goods not delivered, not as described), and the bank acts as
  facilitator, NOT as insurer — verified in bunq's business terms ch. 8.1, and
  it is a scheme property rather than one provider's policy, so changing PSP
  does not avoid it. Consequence for whoever runs this shop: treat a settled
  iDEAL/Wero order like a card order, not like a bank transfer, and keep the
  proof-of-dispatch that a dispute is answered with.
* Neither rail has onboarding, a KYB review, or an account that a provider can
  close. That is the reason both exist. The imprint, terms and privacy pages
  still have to be real, of course, and ShouldStayScriptFree fails the build if
  a PLACEHOLDER marker ever reaches a rendered page again.
* Accepting crypto for goods does **not** make this shop a CASP under MiCA —
  no custody for others, no transfer for third parties, so no licence is
  required, with or without a processor. Cashing EURC out to the bank goes
  through an exchange, and THAT party must be a MiCA-authorised CASP (check
  the ESMA register) — but it stands after the checkout, not in it.
* VAT is unchanged by payment method: the sale is priced and invoiced in euro
  and taxed on the euro value, whichever rail settled it.

### Payment rail suites in CI

Every deploy exercises both rails end to end, as a mandatory gate. What differs
between them is whether a real counterparty is involved, and that difference is
a fact about the rails rather than a difference in standard.

**`ShouldSettleBankTransfers` needs nothing and always runs.** A self-hosted
rail has no provider to authenticate to, so there is nothing to call and no
secret to hold. The suite drives the REAL transfer rail — not a stand-in — and
plays the part of the bank by writing a credits line, the same file
`--pull-credits` fills in production. It covers the whole lifecycle: the order
page's account details, the reference in all the mangled forms a payer might
type, partial payments accumulating, an outgoing amount being ignored, and the
reconciler flipping the order to paid with the right `via`.

**`ShouldSettleEurcOnTestnet` is the one live-counterparty suite**: a €1
donation paid with 1 real testnet EURC on Ethereum Sepolia, settled by the same
two-endpoint quorum as mainnet. It is a deliberate caution-over-convenience
trade — a public RPC hiccup can fail a deploy (re-run the workflow), but the
rail code can never drift from the chain unnoticed. Locally it skips unless
`EURC_E2E_PRIVATE_KEY` is exported; in CI a missing secret is a failure,
checked as the workflow's first step.

There used to be a second live suite, against the hosted bank provider's
test-mode API. It went when the account did, on 2026-08-20.

One Forgejo secret feeds this (repo → Settings → Actions → Secrets):

* `EURC_E2E_PRIVATE_KEY` — a throwaway Ethereum Sepolia key
  (`cast wallet new`), used ONLY for this: it holds worthless testnet tokens,
  never mainnet funds. Fund it with testnet EURC at
  <https://faucet.circle.com> — **network set to Ethereum Sepolia**, one claim
  is ~20 EURC — and Sepolia ETH for gas from the pk910 PoW faucet
  (<https://sepolia-faucet.pk910.de>, no account, it mines in a browser tab;
  one ~0.05 ETH claim covers hundreds of transfers). **Each deploy spends
  1 EURC plus gas**, and the suite's receiving addresses are random — the
  EURC is gone after the run. When the wallet runs dry the EURC suite fails
  at `cast send` with the faucet pointers in its output; top up and re-run.

The workflow installs Foundry's `cast` for the transfer — the server itself
cannot sign transactions, by design. The EURC suite watches `block_tag:
"latest"` (not production's `finalized`) because Sepolia finality is ~13
minutes; the rail's reorg warning in that suite's server log is expected. To
prove the funded wallet end-to-end before it gates a deploy, run the suite
live once from a dev shell:

```sh
EURC_E2E_PRIVATE_KEY=0x... crafter-build test ShouldSettleEurcOnTestnet --product=server
```

## Opening the shop: what is owed before the status flips

Launch is one line in `Catcrafts.Shared-Content.cppm` (`p.status` from
`coming-soon` to `available`), and the comment above that line repeats this
list. Both items are invisible today because the only purchasable thing is the
donation, which ships nothing and is a gift rather than a purchase.

1. **The withdrawal button (herroepingsknop) — legally required, currently
   absent.** Since **19 June 2026**, art. 11a of the Consumer Rights Directive,
   implemented as **art. 6:230oa BW**, requires a clearly labelled button in the
   online interface by which a consumer can withdraw from a distance contract.
   Pointing them at an email address, which is what the terms page does today,
   is no longer sufficient on its own. The button must stay available for the
   whole withdrawal period, ask only for what identifies the order, **not**
   require creating an account, and confirm immediately on a durable medium.
   **Sanction for not having one: the withdrawal period extends from 14 days to
   twelve months**, so every EU phone sale stays unwindable for a year.
   The order page is the natural home, since it already identifies exactly one
   order through an unguessable link and therefore needs no login. Sketch:
   `POST /order/<token>/withdraw`, a button rendered while the window is open
   on orders where the right applies, a `withdrawn` event in the ledger, and the
   confirmation through the existing mailer.
   Note this is easy to miss because the directive that introduced it is
   otherwise about financial services; the ACM states it applies to webshops.
   Worth confirming with a lawyer in the same pass as anything else.

2. **The shipping rate table.** Covered in the next section: with no
   `SENDCLOUD_*` credentials and no cache file there is no table, and checkout
   refuses every order that ships.

## Shipping rates: Sendcloud (REQUIRED to sell)

Sendcloud is the only source of shipping prices. There is no compiled-in
fallback table: a country the carrier has no rate for is a country the shop
cannot post a parcel to, so checkout **refuses** it rather than quoting a price
that would then have to be refunded or absorbed. The consequence is blunt and
intended — **with no rate table, every checkout refuses**, and the server says
so at startup:

```
shipping: NO RATE TABLE — checkout will refuse every order until Sendcloud answers
```

```sh
# credentials from Sendcloud: Settings -> Integrations -> API
cat >> /etc/catcrafts/payments.env <<'ENV'
SENDCLOUD_PUBLIC_KEY=...
SENDCLOUD_SECRET_KEY=...
SENDCLOUD_METHOD='PostNL Parcels non-EU,DPD Home'
ENV
systemctl restart catcrafts-server
journalctl -u catcrafts-server | grep shipping:   # "table refreshed (N countries...)"
```

`SENDCLOUD_METHOD` is a comma-separated list of name substrings, merged in
order with the FIRST FILTER to cover a country winning it — put the postal
method first so non-EU destinations get post rates (a courier method that also
covers Norway or Switzerland would otherwise price them at courier rates, €54
instead of €19), and the courier second to fill the EU.

Every method matching a filter is kept, not just the first: Sendcloud lists the
same service once per **weight band**, so the matches for `DPD Home` are that
service's ladder. A parcel is priced at the cheapest band that can carry it,
where the weight is the product's boxed unit weight (`shipWeightGrams` in
Catcrafts.Shared-Content.cppm) times the quantity ordered. That also sets the
quantity ceiling: one order is one parcel, so an order heavier than every band
is refused with the number that *would* fit, and the buy form's `max` shows the
best case across destinations.

A method that publishes no `max_weight` is skipped rather than treated as
unlimited — same principle, no invented numbers.

### The cache is the resilience layer

The fetched table is cached next to the orders file (`<orders>.shipping.json`)
and read at startup **whether or not credentials are configured**, so a
Sendcloud outage keeps selling at the last known prices. Format is
`country -> [[maxWeightGrams, consumerCents], …]`, prices already VAT-inclusive
(EU rates are grossed up once, at fetch, so the shop nets the carrier's cost).

That also means a hand-written cache file is a complete rate table, which is how
dev and the black-box suites run with no Sendcloud account at all. A cache written by
an older build (flat `country -> cents`, no weight bands) is ignored on load and
replaced by the next refresh — those numbers were an unknown weight band and
re-serving them would price parcels by guess.

This integration is UNTESTED against the live API until credentials exist —
the response parser is covered by the ShouldParseSendcloudRates test, the fetch around it is thin. **Verify one real fetch before opening the shop**: check that
the logged country count and the weight bands match what the Sendcloud panel
shows, because that table is now the difference between a shop that sells and
one that refuses everything.

The buyer sees whatever the server will charge: the checkout page embeds the
active table into its live total (picking the same band, refusing in the same
places), and the amount is computed server-side at order time from the same
data.

## Invoice signing (GPG)

Paid orders offer a clearsigned markdown invoice at `/order/<token>/invoice.md`.
Numbering continues the pre-shop administration: one series per customer — a
random UUID as the customer number, invoices counting sequentially within it
(`f57c6512-…-3`), keyed by the buyer's email. Art. 226(2) permits "one or more
series"; completeness is provable by reconciling the append-only ledger against
the payment provider's records. The signature makes the invoice verifiable
forever, independent of this server — which is why the order page tells buyers
to download it rather than promising to host receipts indefinitely.

One-time key setup on the server, as the service user:

```sh
sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \
  gpg --batch --passphrase '' --quick-gen-key 'Catcrafts invoices <invoices@catcrafts.net>' default default never
# export the PUBLIC key and commit it to the repo so buyers can verify:
sudo -u catcrafts env GNUPGHOME=/var/lib/catcrafts/gnupg \
  gpg --armor --export invoices@catcrafts.net > invoice-key.asc
```

Then in `/etc/catcrafts/payments.env`:

```
INVOICE_GPG_KEY=invoices@catcrafts.net
```

and `Environment=GNUPGHOME=/var/lib/catcrafts/gnupg` in the service unit (see
catcrafts-server.service). The key has no passphrase because the service signs
unattended; the keyring lives in the 0700 StateDirectory. With a key configured,
a signing failure is a 500 — an unsigned invoice is never served by accident.
Without one (dev), invoices carry a visible UNSIGNED marker.

## Order email (confirmation + invoice)

A paid order gets one confirmation email with the clearsigned invoice
attached — plain text plus a markdown attachment, no HTML part, no remote
resources, nothing the privacy notice would have to explain. The mailer
watches the ledger, so every path to paid (reconciler, arrival poll, a manual
`--mark-paid` even on a later restart) results in exactly one email: the
`notified` event, appended only after the mail command accepts the message,
is what stops a resend.

Delivery shells out to a **sendmail-compatible command** rather than speaking
SMTP itself, for the same reason invoices shell out to gpg: TLS, AUTH and
deliverability are exactly what msmtp already does well, and the volume is a
handful of messages per week. Deliverability stays the mailbox provider's
problem (SPF/DKIM are theirs), and no third party beyond the provider that
already handles `info@catcrafts.net` ever sees order data — which is what the
privacy page implies.

```sh
apt install msmtp

cat > /etc/msmtprc <<'CONF'
defaults
auth on
tls on
tls_starttls on
account catcrafts
host smtp.your-mail-provider.example
port 587
from info@catcrafts.net
user info@catcrafts.net
passwordeval cat /etc/catcrafts/smtp-password
account default : catcrafts
CONF
chmod 0644 /etc/msmtprc

# msmtp runs as the service user, so the password file must be readable by
# it — unlike payments.env, which only root (systemd) reads.
install -o catcrafts -g catcrafts -m 0600 /dev/null /etc/catcrafts/smtp-password
# ...then put the SMTP password in that file.
```

Then in `/etc/catcrafts/payments.env`:

```
MAIL_COMMAND=msmtp -t
MAIL_FROM=Catcrafts <info@catcrafts.net>
```

and `systemctl restart catcrafts-server` — the journal should say
`mail: order confirmations via 'msmtp -t'`. Unset, no email is sent and the
order page plus the invoice download remain the buyer's receipt: degraded,
not down, like every optional integration here.

Worth knowing:

* A failed handoff retries with exponential backoff (1 min doubling to a cap
  of ~an hour), forever — a broken relay delays the email, it never eats it.
  Watch `journalctl -u catcrafts-server | grep 'mail:'` after changing config.
* With a signing key configured, a gpg failure means the email WAITS — an
  unsigned invoice never leaves by accident, same rule as the download.
* Send a real test: `--rail=fake` locally with `MAIL_COMMAND` pointing at
  msmtp and your own address in the order form, or just run the
  ShouldProcessCheckout suite (`crafter-build test ShouldProcessCheckout
  --product=server`), which captures the messages with a fake sendmail and
  verifies the attached signature.

## Reading the orders ledger

```sh
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl
```

```
orders: 2

reference  status             total  cc  created               token
CC-3F9A2C  paid              595.00  NL  2026-08-04T14:02:11Z  3f9a2c…
CC-91B04D  awaiting_payment  534.34  CA  2026-08-04T15:40:03Z  91b04d…
```

Manual transitions exist for the cases automation cannot see — a payment
confirmed out-of-band, the parcel handed to the carrier, a refund:

```sh
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-paid    <token>
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --mark-shipped <token>
catcrafts-server --orders /var/lib/catcrafts/orders.jsonl --cancel       <token>
```

Each appends a status event to the log — nothing is ever rewritten, so the
file remains its own audit trail. Deleting personal data on request is an edit
of the fields the seven-year fiscal retention does not cover.

## The open financials page

`/financials` publishes running totals only: sales, donations, and expenses
as one flat list of categories. Sales — and donations made through the shop —
fold out of `orders.jsonl` on every request and need no setup at all.

The bank side (donations to the bank account, and the expense categories) is
`orders.jsonl.financials.json`, written by the owner's own tooling off this
box and re-read on every request: updating the file is all it takes to update
the page. Aggregates by construction — category totals and an as-of date are
all the file can carry, which is the page's privacy design. No bank credential
of any kind lives on this box.

(A bunq mutation callback used to keep these numbers live; it was retired
2026-08-17 — webhook deregistered, endpoint removed. The code and its
documentation are in git history if a bank feed ever comes back.)

## Verifying a deploy

```sh
systemctl status catcrafts-server
curl -s localhost:8081/api/healthz            # ok + content counts

# real status codes, which a client-side router cannot produce
curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/nope        # 404
curl -o /dev/null -w '%{http_code}\n' https://catcrafts.net/blog        # 301

# the SEO check: content present with no JavaScript involved
curl -s https://catcrafts.net/projects | grep -c '<script'              # 0
curl -s https://catcrafts.net/projects | grep -o '<title>[^<]*'
```

## Analytics

Server-side only — the privacy policy promises request logging and nothing
else, so there is no client-side analytics anywhere on the site. GoAccess
(Debian package) turns Caddy's JSON access logs into two static HTML
reports, each with its own persistent DB and ingest ledger:

- `https://catcrafts.net/analytics/` — **public, censored.** Visitor IPs are
  anonymized at ingest (last octet zeroed before anything reaches its DB),
  no HOSTS or full-URL REFERRERS panels, and log lines matching `CENSOR_RE`
  in the script never enter its DB at all — the public tier cannot leak
  what it never ingested. `CENSOR_RE` covers `/api` **and `/order`**: an
  order token is the entire capability to read that buyer's status page and
  their invoice (name, street, postal code, city), so publishing the path
  publishes the buyer. Keep secrets out of URL *paths* regardless — query
  strings are already stripped, paths are not.
- `https://catcrafts.net/analytics/private/` — **uncensored** (basic auth,
  hash in the Caddyfile): full IPs, all panels.

Raw logs keep full IPs either way — that is the request logging the privacy
policy declares; per-IP forensics work from the logs and the private tier,
never from the public page.

Three pieces, all in `deploy/`:

- `catcrafts-analytics` → `/usr/local/bin/` — ingests each rotated
  `catcrafts.net-*.log.gz` exactly once into a persistent GoAccess DB
  (`/var/lib/goaccess/db`, tracked in `/var/lib/goaccess/ingested`), then
  renders the report from DB + live log. The live file is never persisted,
  so its lines don't double-count when Caddy rotates it. History therefore
  survives log deletion: the DB keeps aggregates forever.
- `catcrafts-analytics.service` — oneshot, runs as `caddy` (owner of the
  0600 logs).
- `catcrafts-analytics.timer` — hourly at :07.

Bot filtering is the load-bearing part: measured on real traffic, 57% of
requests were headerless vulnerability scanners and another 19% self-declared
bots (mostly ClaudeBot) — only ~24% human. `--ignore-crawlers
--unknowns-as-crawlers` drops both groups. The flags in the script apply at
ingest time and the DB stores aggregated data, so changing filters later only
affects new lines — re-ingesting history means deleting
`/var/lib/goaccess/{db,ingested}` and letting the next run rebuild from
whatever raw logs retention still holds (a year, per the Caddyfile).

```sh
apt install goaccess
install -m 755 deploy/catcrafts-analytics /usr/local/bin/
install -m 644 deploy/catcrafts-analytics.{service,timer} /etc/systemd/system/
mkdir -p /etc/goaccess /var/lib/goaccess /var/www/analytics /var/www/analytics-private
install -m 644 deploy/goaccess-browsers.list /etc/goaccess/browsers.list
# own IPs to keep out of the numbers - host-only file, NOT in this repo
echo "203.0.113.7" > /etc/goaccess/exclude-ips
chown -R caddy:caddy /var/lib/goaccess /var/www/analytics /var/www/analytics-private
systemctl daemon-reload && systemctl enable --now catcrafts-analytics.timer
```

## Running it locally

```sh
tools/dev.sh              # build both products and serve on :8080
tools/dev.sh --no-build   # reuse what is already in bin/
```

That is the whole site: Caddy in front, the backend behind, static assets from
disk, cross-origin headers scoped exactly as in production. Ctrl-C stops both.
Orders from the session go to a temp file and are discarded on exit; the
fake payment rail is active — `touch <workdir>/orders.jsonl.fake-paid` plays
the part of the customer paying.

**Do not run `catcrafts-server --serve` alone and expect a working site.** It
serves pages only — static assets are Caddy's job — so `/styles.css` 404s and
every page renders unstyled. That looks broken but isn't; it is the deployment
split working as designed.

### Other useful commands

```sh
tools/fetch-posts.sh            # pull posts from the allowed communities
tools/fetch-media.sh [dir]      # mirror their media locally (run after the above)
tools/fetch-rates.sh            # ECB reference rates for the indicative prices
crafter-build  -r        # the wasm app alone, no backend, on :8080

crafter-build test --product=server   # every suite: the unit assertions
                                # AND the black-box HTTP suites, which spawn the
                                # real server on scratch ports (the CI gate; the
                                # checkout lifecycle re-arms at launch)
<server>/catcrafts-server --routes              # status + title for every route
<server>/catcrafts-server --render /projects    # dump one page's HTML
<server>/catcrafts-server --orders FILE         # the orders ledger + manual transitions
```

### If a `bin/` glob matches two directories

The variant directory name embeds a config hash, and **`` and
non-`` builds hash differently**: `` resolves the Crafter libraries
from sibling working trees, a plain build fetches them from Forgejo. They are
genuinely different configurations and both land in `bin/`.

Mixing them leaves two directories, and any script globbing for one picks
arbitrarily — which in practice means testing a stale binary and believing the
result. Every script here refuses to guess and tells you to `rm -rf bin`. Pick
one mode and stay in it.