Compare commits
7 changed files with 3 additions and 255 deletions
|
|
@ -43,37 +43,6 @@ catcrafts.net {
|
|||
header /styles.css Cross-Origin-Resource-Policy "same-origin"
|
||||
header /favicon.svg Cross-Origin-Resource-Policy "same-origin"
|
||||
|
||||
# ── analytics, two tiers ──────────────────────────────────────────────
|
||||
#
|
||||
# GoAccess reports rebuilt hourly by catcrafts-analytics.timer from the
|
||||
# access logs below. Server-side only: no client-side analytics anywhere
|
||||
# on the site, per the privacy policy (request logging only).
|
||||
#
|
||||
# /analytics/ is public and censored - visitor IPs are anonymized at
|
||||
# ingest, there is no host panel, and sensitive paths never enter its DB
|
||||
# (see CENSOR_RE in deploy/catcrafts-analytics). /analytics/private/ is
|
||||
# the uncensored report. Generate the hash with `caddy hash-password`.
|
||||
redir /analytics /analytics/ 308
|
||||
redir /analytics/private /analytics/private/ 308
|
||||
|
||||
# The privacy policy's "view previous versions" promise. A redirect so
|
||||
# the policy text stays short and survives repo or path moves - update
|
||||
# the target here, never the policy wording.
|
||||
redir /legal/privacy/history https://forgejo.catcrafts.net/Catcrafts/catcrafts.net/commits/branch/master/shared/interfaces/Catcrafts.Shared-Content.cppm 302
|
||||
handle_path /analytics/private/* {
|
||||
basic_auth {
|
||||
jorijn <bcrypt-hash-here>
|
||||
}
|
||||
root * /var/www/analytics-private
|
||||
header Cache-Control "private, no-store"
|
||||
file_server
|
||||
}
|
||||
handle_path /analytics/* {
|
||||
root * /var/www/analytics
|
||||
header Cache-Control "public, max-age=600"
|
||||
file_server
|
||||
}
|
||||
|
||||
# ── build artifacts: served from disk ─────────────────────────────────
|
||||
#
|
||||
# file_server does sendfile, precompressed variants and range requests far
|
||||
|
|
@ -129,18 +98,4 @@ catcrafts.net {
|
|||
header Cache-Control "no-store"
|
||||
file_server
|
||||
}
|
||||
|
||||
# ── access log ────────────────────────────────────────────────────────
|
||||
#
|
||||
# Feeds the GoAccess analytics (see deploy/catcrafts-analytics). Retention
|
||||
# sized for that: ~15 MB/day means a roll roughly weekly; 52 compressed
|
||||
# rolls ~ a year of raw logs (~6 MB each gzipped). roll_keep_for must be
|
||||
# explicit — lumberjack's default silently deletes after 90 days.
|
||||
log {
|
||||
output file /var/log/caddy/catcrafts.net.log {
|
||||
roll_size 100MiB
|
||||
roll_keep 52
|
||||
roll_keep_for 8760h
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -343,60 +343,6 @@ curl -s https://catcrafts.net/projects | grep -c '<script' # 0
|
|||
curl -s https://catcrafts.net/projects | grep -o '<title>[^<]*'
|
||||
```
|
||||
|
||||
## Analytics
|
||||
|
||||
Server-side only — the privacy policy promises request logging and nothing
|
||||
else, so there is no client-side analytics anywhere on the site. GoAccess
|
||||
(Debian package) turns Caddy's JSON access logs into two static HTML
|
||||
reports, each with its own persistent DB and ingest ledger:
|
||||
|
||||
- `https://catcrafts.net/analytics/` — **public, censored.** Visitor IPs are
|
||||
anonymized at ingest (last octet zeroed before anything reaches its DB),
|
||||
no HOSTS or full-URL REFERRERS panels, and log lines matching `CENSOR_RE`
|
||||
in the script never enter its DB at all — the public tier cannot leak
|
||||
what it never ingested. Extend `CENSOR_RE` when the shop launches so
|
||||
order/payment URLs can never surface; keep secrets out of URL *paths*
|
||||
regardless (query strings are already stripped).
|
||||
- `https://catcrafts.net/analytics/private/` — **uncensored** (basic auth,
|
||||
hash in the Caddyfile): full IPs, all panels.
|
||||
|
||||
Raw logs keep full IPs either way — that is the request logging the privacy
|
||||
policy declares; per-IP forensics work from the logs and the private tier,
|
||||
never from the public page.
|
||||
|
||||
Three pieces, all in `deploy/`:
|
||||
|
||||
- `catcrafts-analytics` → `/usr/local/bin/` — ingests each rotated
|
||||
`catcrafts.net-*.log.gz` exactly once into a persistent GoAccess DB
|
||||
(`/var/lib/goaccess/db`, tracked in `/var/lib/goaccess/ingested`), then
|
||||
renders the report from DB + live log. The live file is never persisted,
|
||||
so its lines don't double-count when Caddy rotates it. History therefore
|
||||
survives log deletion: the DB keeps aggregates forever.
|
||||
- `catcrafts-analytics.service` — oneshot, runs as `caddy` (owner of the
|
||||
0600 logs).
|
||||
- `catcrafts-analytics.timer` — hourly at :07.
|
||||
|
||||
Bot filtering is the load-bearing part: measured on real traffic, 57% of
|
||||
requests were headerless vulnerability scanners and another 19% self-declared
|
||||
bots (mostly ClaudeBot) — only ~24% human. `--ignore-crawlers
|
||||
--unknowns-as-crawlers` drops both groups. The flags in the script apply at
|
||||
ingest time and the DB stores aggregated data, so changing filters later only
|
||||
affects new lines — re-ingesting history means deleting
|
||||
`/var/lib/goaccess/{db,ingested}` and letting the next run rebuild from
|
||||
whatever raw logs retention still holds (a year, per the Caddyfile).
|
||||
|
||||
```sh
|
||||
apt install goaccess
|
||||
install -m 755 deploy/catcrafts-analytics /usr/local/bin/
|
||||
install -m 644 deploy/catcrafts-analytics.{service,timer} /etc/systemd/system/
|
||||
mkdir -p /etc/goaccess /var/lib/goaccess /var/www/analytics /var/www/analytics-private
|
||||
install -m 644 deploy/goaccess-browsers.list /etc/goaccess/browsers.list
|
||||
# own IPs to keep out of the numbers - host-only file, NOT in this repo
|
||||
echo "203.0.113.7" > /etc/goaccess/exclude-ips
|
||||
chown -R caddy:caddy /var/lib/goaccess /var/www/analytics /var/www/analytics-private
|
||||
systemctl daemon-reload && systemctl enable --now catcrafts-analytics.timer
|
||||
```
|
||||
|
||||
## Running it locally
|
||||
|
||||
```sh
|
||||
|
|
|
|||
|
|
@ -1,119 +0,0 @@
|
|||
#!/bin/sh
|
||||
# Rebuilds the analytics reports from Caddy's access logs, in two tiers:
|
||||
#
|
||||
# /var/www/analytics/index.html PUBLIC - censored. IPs are
|
||||
# anonymized at ingest, there is no HOSTS or full-URL REFERRERS panel,
|
||||
# and log lines matching $CENSOR_RE never reach its DB at all: the
|
||||
# public tier cannot leak what it never ingested.
|
||||
# /var/www/analytics-private/index.html PRIVATE (basic auth in Caddy) -
|
||||
# uncensored: full IPs, all panels.
|
||||
#
|
||||
# Each tier has its own persistent DB and ingest ledger: rotated logs are
|
||||
# ingested exactly once per tier; the live file is layered on at report time
|
||||
# WITHOUT --persist, so its lines never double-count when Caddy rolls it.
|
||||
#
|
||||
# Filters live at INGEST: the DBs store aggregated, filtered data, so a
|
||||
# filter change only applies to new lines. To re-filter history:
|
||||
# rm -rf /var/lib/goaccess/db-* /var/lib/goaccess/ingested-*
|
||||
# systemctl start catcrafts-analytics # rebuilds from retained raw logs
|
||||
#
|
||||
# Runs hourly as the caddy user (owner of the 0600 logs) via
|
||||
# catcrafts-analytics.timer.
|
||||
set -eu
|
||||
|
||||
LOG_DIR=/var/log/caddy
|
||||
LIVE=$LOG_DIR/catcrafts.net.log
|
||||
STATE_DIR=/var/lib/goaccess
|
||||
OUT_PUBLIC=/var/www/analytics/index.html
|
||||
OUT_PRIVATE=/var/www/analytics-private/index.html
|
||||
|
||||
# Log lines whose URI matches this never enter the public tier. Extend it
|
||||
# when the shop launches so order/payment URLs can never surface publicly.
|
||||
CENSOR_RE='"uri":"/api'
|
||||
|
||||
# Serialize runs: a manual run racing the hourly timer once ingested the same
|
||||
# rotated log twice (both processes passed the ledger check before either
|
||||
# appended). Skip quietly if another run holds the lock - the timer comes
|
||||
# around hourly anyway.
|
||||
exec 9>$STATE_DIR/.lock
|
||||
flock -n 9 || exit 0
|
||||
|
||||
# goaccess refuses a missing --db-path; recreate after a re-filter wipe.
|
||||
mkdir -p "$STATE_DIR/db-private" "$STATE_DIR/db-public"
|
||||
|
||||
# Own IPs to keep out of the numbers (data quality, both tiers). The IPs
|
||||
# live in /etc/goaccess/exclude-ips (one per line, # comments allowed) - ON
|
||||
# THE HOST ONLY, never in this public repo.
|
||||
exclude_args() {
|
||||
[ -r /etc/goaccess/exclude-ips ] || return 0
|
||||
while IFS= read -r ip; do
|
||||
case "$ip" in ''|'#'*) continue ;; esac
|
||||
printf -- '--exclude-ip=%s\n' "$ip"
|
||||
done < /etc/goaccess/exclude-ips
|
||||
}
|
||||
|
||||
# Shared quality filters, both tiers:
|
||||
# --ignore-crawlers drops self-declared bots (ClaudeBot alone was 18% of
|
||||
# traffic); --unknowns-as-crawlers additionally drops the headerless
|
||||
# vulnerability scanners (57% of traffic, no User-Agent at all); the
|
||||
# browsers-file catches known fake-browser scrapers.
|
||||
# --ignore-referrer drops referrer values that can never carry real source
|
||||
# information: '*catcrafts.*' covers self-referrals AND the parked redirect
|
||||
# TLDs (a 301 never sets the redirecting host as referrer, so those are
|
||||
# always crawlers stamping their pre-redirect URL); '*localhost*' is Android
|
||||
# fediverse apps whose WebView UI lives on an embedded localhost page.
|
||||
# $(exclude_args) is unquoted on purpose: one word per --exclude-ip flag.
|
||||
common() {
|
||||
goaccess "$@" \
|
||||
--log-format=CADDY \
|
||||
--ignore-crawlers \
|
||||
--unknowns-as-crawlers \
|
||||
--browsers-file=/etc/goaccess/browsers.list \
|
||||
$(exclude_args) \
|
||||
--ignore-referrer='*catcrafts.*' \
|
||||
--ignore-referrer='*localhost*' \
|
||||
--no-query-string \
|
||||
--tz=Europe/Amsterdam
|
||||
}
|
||||
|
||||
run_private() {
|
||||
common "$@" --db-path="$STATE_DIR/db-private"
|
||||
}
|
||||
|
||||
# Public extras: --anonymize-ip zeroes the last octet before anything is
|
||||
# stored; visitor IPs are personal data and stay out of the public page.
|
||||
run_public() {
|
||||
common "$@" --db-path="$STATE_DIR/db-public" \
|
||||
--anonymize-ip \
|
||||
--ignore-panel=HOSTS \
|
||||
--ignore-panel=REFERRERS
|
||||
}
|
||||
|
||||
# grep -v exits 1 when it outputs nothing - not an error here.
|
||||
censor() {
|
||||
grep -vE "$CENSOR_RE" || true
|
||||
}
|
||||
|
||||
ingest() { # $1 = tier
|
||||
ledger=$STATE_DIR/ingested-$1
|
||||
touch "$ledger"
|
||||
for f in "$LOG_DIR"/catcrafts.net-*.log.gz; do
|
||||
[ -e "$f" ] || continue
|
||||
grep -qxF "$f" "$ledger" && continue
|
||||
case "$1" in
|
||||
private) zcat "$f" | run_private - --persist --restore --process-and-exit ;;
|
||||
public) zcat "$f" | censor | run_public - --persist --restore --process-and-exit ;;
|
||||
esac
|
||||
printf '%s\n' "$f" >>"$ledger"
|
||||
done
|
||||
}
|
||||
|
||||
ingest private
|
||||
ingest public
|
||||
|
||||
run_private "$LIVE" --restore \
|
||||
--html-report-title="catcrafts.net (private)" \
|
||||
-o "$OUT_PRIVATE"
|
||||
censor <"$LIVE" | run_public - --restore \
|
||||
--html-report-title="catcrafts.net" \
|
||||
-o "$OUT_PUBLIC"
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
[Unit]
|
||||
Description=Rebuild GoAccess analytics report for catcrafts.net
|
||||
Documentation=https://goaccess.io/man
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# caddy owns the 0600 access logs; DB and output dirs are chowned to match.
|
||||
User=caddy
|
||||
Group=caddy
|
||||
ExecStart=/usr/local/bin/catcrafts-analytics
|
||||
Nice=10
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
[Unit]
|
||||
Description=Hourly GoAccess analytics report rebuild
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* *:07:00
|
||||
RandomizedDelaySec=3m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
# Extra crawler classifications for GoAccess (--browsers-file), TAB-separated.
|
||||
# Installed at /etc/goaccess/browsers.list; extends the builtin list.
|
||||
#
|
||||
# Frozen-UA scraper rotating through ~60 IPs: fakes Referer as reddit.com /
|
||||
# localhost / none, never sends Sec-Fetch-* or Range, re-downloads media
|
||||
# whole, probes /test.php. The UA itself is the proof of fakery: real Chrome
|
||||
# mobile has sent a reduced UA ("Linux; Android 10; K", no device model)
|
||||
# since ~2023, so bare Chrome claiming "Android 14; Pixel 8" cannot be real.
|
||||
# Keep patterns EXACT: Voyager/Summit Lemmy apps on genuine Pixel 8s appear
|
||||
# as "Pixel 8 Build/...; wv)" WebView UAs and must keep counting as humans.
|
||||
Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Crawlers
|
||||
Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Crawlers
|
||||
|
|
@ -199,7 +199,7 @@ export const std::vector<LegalPage>& LegalPages() {
|
|||
{
|
||||
.slug = "privacy",
|
||||
.title = "Privacy",
|
||||
.updated = "2026-08-08",
|
||||
.updated = "2026-08-04",
|
||||
.lede = "What this site collects, why, and how to get rid of it. Written to describe what the code actually does. If you find a discrepancy, the code is the bug and a report is very welcome.",
|
||||
.sections = {
|
||||
{ "Who is responsible",
|
||||
|
|
@ -220,13 +220,12 @@ export const std::vector<LegalPage>& LegalPages() {
|
|||
} },
|
||||
{ "What this site does not do",
|
||||
{
|
||||
"No analytics in your browser. No cookies, none at all, which is why there is no cookie banner. No third-party scripts, no fonts loaded from anyone else's server, no embedded video, no social buttons, no advertising, no profiling, no automated decision-making.",
|
||||
"No analytics. No cookies, none at all, which is why there is no cookie banner. No third-party scripts, no fonts loaded from anyone else's server, no embedded video, no social buttons, no advertising, no profiling, no automated decision-making.",
|
||||
"Everything the browser loads comes from catcrafts.net. Following a link out (to a fediverse thread, to Forgejo, to the Mollie payment page) puts you on that site under its terms, and Catcrafts has no visibility into what happens there.",
|
||||
} },
|
||||
{ "Server logs",
|
||||
{
|
||||
"The web server keeps ordinary request logs. Those exist to debug faults and spot abuse, and are not connected to order records or used to build any kind of profile.",
|
||||
"Those logs are also this site's only analytics: a page of aggregate statistics, computed on the server, with visitor addresses anonymized before anything is stored. A tally like that is nothing to keep secret, so you are free to take a look yourself at catcrafts.net/analytics.",
|
||||
} },
|
||||
{ "Your rights",
|
||||
{
|
||||
|
|
@ -234,7 +233,7 @@ export const std::vector<LegalPage>& LegalPages() {
|
|||
} },
|
||||
{ "Changes",
|
||||
{
|
||||
"This page has a date at the top. Anything that changes what is collected or why gets a new date, and you do not have to take the date's word for it: every previous version of this page, with the exact diff, is public at catcrafts.net/legal/privacy/history.",
|
||||
"This page has a date at the top. Anything that changes what is collected or why will be a new date and a note in the posts feed, not a silent edit.",
|
||||
} },
|
||||
},
|
||||
},
|
||||
|
|
|
|||
Loading…
Reference in a new issue