analytics: GoAccess reports from the Caddy logs, public and private tiers

Server-side only, per the privacy policy: no client-side analytics. Rotated
logs ingest exactly once into per-tier DBs; the public tier is anonymized
at ingest and never receives censored paths. Log retention raised to a year.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-08-08 02:50:52 +02:00
commit 8cd4d55d00
6 changed files with 251 additions and 0 deletions

View file

@ -343,6 +343,60 @@ curl -s https://catcrafts.net/projects | grep -c '<script' # 0
curl -s https://catcrafts.net/projects | grep -o '<title>[^<]*'
```
## Analytics
Server-side only — the privacy policy promises request logging and nothing
else, so there is no client-side analytics anywhere on the site. GoAccess
(Debian package) turns Caddy's JSON access logs into two static HTML
reports, each with its own persistent DB and ingest ledger:
- `https://catcrafts.net/analytics/`**public, censored.** Visitor IPs are
anonymized at ingest (last octet zeroed before anything reaches its DB),
no HOSTS or full-URL REFERRERS panels, and log lines matching `CENSOR_RE`
in the script never enter its DB at all — the public tier cannot leak
what it never ingested. Extend `CENSOR_RE` when the shop launches so
order/payment URLs can never surface; keep secrets out of URL *paths*
regardless (query strings are already stripped).
- `https://catcrafts.net/analytics/private/`**uncensored** (basic auth,
hash in the Caddyfile): full IPs, all panels.
Raw logs keep full IPs either way — that is the request logging the privacy
policy declares; per-IP forensics work from the logs and the private tier,
never from the public page.
Three pieces, all in `deploy/`:
- `catcrafts-analytics``/usr/local/bin/` — ingests each rotated
`catcrafts.net-*.log.gz` exactly once into a persistent GoAccess DB
(`/var/lib/goaccess/db`, tracked in `/var/lib/goaccess/ingested`), then
renders the report from DB + live log. The live file is never persisted,
so its lines don't double-count when Caddy rotates it. History therefore
survives log deletion: the DB keeps aggregates forever.
- `catcrafts-analytics.service` — oneshot, runs as `caddy` (owner of the
0600 logs).
- `catcrafts-analytics.timer` — hourly at :07.
Bot filtering is the load-bearing part: measured on real traffic, 57% of
requests were headerless vulnerability scanners and another 19% self-declared
bots (mostly ClaudeBot) — only ~24% human. `--ignore-crawlers
--unknowns-as-crawlers` drops both groups. The flags in the script apply at
ingest time and the DB stores aggregated data, so changing filters later only
affects new lines — re-ingesting history means deleting
`/var/lib/goaccess/{db,ingested}` and letting the next run rebuild from
whatever raw logs retention still holds (a year, per the Caddyfile).
```sh
apt install goaccess
install -m 755 deploy/catcrafts-analytics /usr/local/bin/
install -m 644 deploy/catcrafts-analytics.{service,timer} /etc/systemd/system/
mkdir -p /etc/goaccess /var/lib/goaccess /var/www/analytics /var/www/analytics-private
install -m 644 deploy/goaccess-browsers.list /etc/goaccess/browsers.list
# own IPs to keep out of the numbers - host-only file, NOT in this repo
echo "203.0.113.7" > /etc/goaccess/exclude-ips
chown -R caddy:caddy /var/lib/goaccess /var/www/analytics /var/www/analytics-private
systemctl daemon-reload && systemctl enable --now catcrafts-analytics.timer
```
## Running it locally
```sh