Compare commits
7 changed files with 3 additions and 255 deletions
|
|
@ -43,37 +43,6 @@ catcrafts.net {
|
||||||
header /styles.css Cross-Origin-Resource-Policy "same-origin"
|
header /styles.css Cross-Origin-Resource-Policy "same-origin"
|
||||||
header /favicon.svg Cross-Origin-Resource-Policy "same-origin"
|
header /favicon.svg Cross-Origin-Resource-Policy "same-origin"
|
||||||
|
|
||||||
# ── analytics, two tiers ──────────────────────────────────────────────
|
|
||||||
#
|
|
||||||
# GoAccess reports rebuilt hourly by catcrafts-analytics.timer from the
|
|
||||||
# access logs below. Server-side only: no client-side analytics anywhere
|
|
||||||
# on the site, per the privacy policy (request logging only).
|
|
||||||
#
|
|
||||||
# /analytics/ is public and censored - visitor IPs are anonymized at
|
|
||||||
# ingest, there is no host panel, and sensitive paths never enter its DB
|
|
||||||
# (see CENSOR_RE in deploy/catcrafts-analytics). /analytics/private/ is
|
|
||||||
# the uncensored report. Generate the hash with `caddy hash-password`.
|
|
||||||
redir /analytics /analytics/ 308
|
|
||||||
redir /analytics/private /analytics/private/ 308
|
|
||||||
|
|
||||||
# The privacy policy's "view previous versions" promise. A redirect so
|
|
||||||
# the policy text stays short and survives repo or path moves - update
|
|
||||||
# the target here, never the policy wording.
|
|
||||||
redir /legal/privacy/history https://forgejo.catcrafts.net/Catcrafts/catcrafts.net/commits/branch/master/shared/interfaces/Catcrafts.Shared-Content.cppm 302
|
|
||||||
handle_path /analytics/private/* {
|
|
||||||
basic_auth {
|
|
||||||
jorijn <bcrypt-hash-here>
|
|
||||||
}
|
|
||||||
root * /var/www/analytics-private
|
|
||||||
header Cache-Control "private, no-store"
|
|
||||||
file_server
|
|
||||||
}
|
|
||||||
handle_path /analytics/* {
|
|
||||||
root * /var/www/analytics
|
|
||||||
header Cache-Control "public, max-age=600"
|
|
||||||
file_server
|
|
||||||
}
|
|
||||||
|
|
||||||
# ── build artifacts: served from disk ─────────────────────────────────
|
# ── build artifacts: served from disk ─────────────────────────────────
|
||||||
#
|
#
|
||||||
# file_server does sendfile, precompressed variants and range requests far
|
# file_server does sendfile, precompressed variants and range requests far
|
||||||
|
|
@ -129,18 +98,4 @@ catcrafts.net {
|
||||||
header Cache-Control "no-store"
|
header Cache-Control "no-store"
|
||||||
file_server
|
file_server
|
||||||
}
|
}
|
||||||
|
|
||||||
# ── access log ────────────────────────────────────────────────────────
|
|
||||||
#
|
|
||||||
# Feeds the GoAccess analytics (see deploy/catcrafts-analytics). Retention
|
|
||||||
# sized for that: ~15 MB/day means a roll roughly weekly; 52 compressed
|
|
||||||
# rolls ~ a year of raw logs (~6 MB each gzipped). roll_keep_for must be
|
|
||||||
# explicit — lumberjack's default silently deletes after 90 days.
|
|
||||||
log {
|
|
||||||
output file /var/log/caddy/catcrafts.net.log {
|
|
||||||
roll_size 100MiB
|
|
||||||
roll_keep 52
|
|
||||||
roll_keep_for 8760h
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -343,60 +343,6 @@ curl -s https://catcrafts.net/projects | grep -c '<script' # 0
|
||||||
curl -s https://catcrafts.net/projects | grep -o '<title>[^<]*'
|
curl -s https://catcrafts.net/projects | grep -o '<title>[^<]*'
|
||||||
```
|
```
|
||||||
|
|
||||||
## Analytics
|
|
||||||
|
|
||||||
Server-side only — the privacy policy promises request logging and nothing
|
|
||||||
else, so there is no client-side analytics anywhere on the site. GoAccess
|
|
||||||
(Debian package) turns Caddy's JSON access logs into two static HTML
|
|
||||||
reports, each with its own persistent DB and ingest ledger:
|
|
||||||
|
|
||||||
- `https://catcrafts.net/analytics/` — **public, censored.** Visitor IPs are
|
|
||||||
anonymized at ingest (last octet zeroed before anything reaches its DB),
|
|
||||||
no HOSTS or full-URL REFERRERS panels, and log lines matching `CENSOR_RE`
|
|
||||||
in the script never enter its DB at all — the public tier cannot leak
|
|
||||||
what it never ingested. Extend `CENSOR_RE` when the shop launches so
|
|
||||||
order/payment URLs can never surface; keep secrets out of URL *paths*
|
|
||||||
regardless (query strings are already stripped).
|
|
||||||
- `https://catcrafts.net/analytics/private/` — **uncensored** (basic auth,
|
|
||||||
hash in the Caddyfile): full IPs, all panels.
|
|
||||||
|
|
||||||
Raw logs keep full IPs either way — that is the request logging the privacy
|
|
||||||
policy declares; per-IP forensics work from the logs and the private tier,
|
|
||||||
never from the public page.
|
|
||||||
|
|
||||||
Three pieces, all in `deploy/`:
|
|
||||||
|
|
||||||
- `catcrafts-analytics` → `/usr/local/bin/` — ingests each rotated
|
|
||||||
`catcrafts.net-*.log.gz` exactly once into a persistent GoAccess DB
|
|
||||||
(`/var/lib/goaccess/db`, tracked in `/var/lib/goaccess/ingested`), then
|
|
||||||
renders the report from DB + live log. The live file is never persisted,
|
|
||||||
so its lines don't double-count when Caddy rotates it. History therefore
|
|
||||||
survives log deletion: the DB keeps aggregates forever.
|
|
||||||
- `catcrafts-analytics.service` — oneshot, runs as `caddy` (owner of the
|
|
||||||
0600 logs).
|
|
||||||
- `catcrafts-analytics.timer` — hourly at :07.
|
|
||||||
|
|
||||||
Bot filtering is the load-bearing part: measured on real traffic, 57% of
|
|
||||||
requests were headerless vulnerability scanners and another 19% self-declared
|
|
||||||
bots (mostly ClaudeBot) — only ~24% human. `--ignore-crawlers
|
|
||||||
--unknowns-as-crawlers` drops both groups. The flags in the script apply at
|
|
||||||
ingest time and the DB stores aggregated data, so changing filters later only
|
|
||||||
affects new lines — re-ingesting history means deleting
|
|
||||||
`/var/lib/goaccess/{db,ingested}` and letting the next run rebuild from
|
|
||||||
whatever raw logs retention still holds (a year, per the Caddyfile).
|
|
||||||
|
|
||||||
```sh
|
|
||||||
apt install goaccess
|
|
||||||
install -m 755 deploy/catcrafts-analytics /usr/local/bin/
|
|
||||||
install -m 644 deploy/catcrafts-analytics.{service,timer} /etc/systemd/system/
|
|
||||||
mkdir -p /etc/goaccess /var/lib/goaccess /var/www/analytics /var/www/analytics-private
|
|
||||||
install -m 644 deploy/goaccess-browsers.list /etc/goaccess/browsers.list
|
|
||||||
# own IPs to keep out of the numbers - host-only file, NOT in this repo
|
|
||||||
echo "203.0.113.7" > /etc/goaccess/exclude-ips
|
|
||||||
chown -R caddy:caddy /var/lib/goaccess /var/www/analytics /var/www/analytics-private
|
|
||||||
systemctl daemon-reload && systemctl enable --now catcrafts-analytics.timer
|
|
||||||
```
|
|
||||||
|
|
||||||
## Running it locally
|
## Running it locally
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|
|
||||||
|
|
@ -1,119 +0,0 @@
|
||||||
#!/bin/sh
|
|
||||||
# Rebuilds the analytics reports from Caddy's access logs, in two tiers:
|
|
||||||
#
|
|
||||||
# /var/www/analytics/index.html PUBLIC - censored. IPs are
|
|
||||||
# anonymized at ingest, there is no HOSTS or full-URL REFERRERS panel,
|
|
||||||
# and log lines matching $CENSOR_RE never reach its DB at all: the
|
|
||||||
# public tier cannot leak what it never ingested.
|
|
||||||
# /var/www/analytics-private/index.html PRIVATE (basic auth in Caddy) -
|
|
||||||
# uncensored: full IPs, all panels.
|
|
||||||
#
|
|
||||||
# Each tier has its own persistent DB and ingest ledger: rotated logs are
|
|
||||||
# ingested exactly once per tier; the live file is layered on at report time
|
|
||||||
# WITHOUT --persist, so its lines never double-count when Caddy rolls it.
|
|
||||||
#
|
|
||||||
# Filters live at INGEST: the DBs store aggregated, filtered data, so a
|
|
||||||
# filter change only applies to new lines. To re-filter history:
|
|
||||||
# rm -rf /var/lib/goaccess/db-* /var/lib/goaccess/ingested-*
|
|
||||||
# systemctl start catcrafts-analytics # rebuilds from retained raw logs
|
|
||||||
#
|
|
||||||
# Runs hourly as the caddy user (owner of the 0600 logs) via
|
|
||||||
# catcrafts-analytics.timer.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
LOG_DIR=/var/log/caddy
|
|
||||||
LIVE=$LOG_DIR/catcrafts.net.log
|
|
||||||
STATE_DIR=/var/lib/goaccess
|
|
||||||
OUT_PUBLIC=/var/www/analytics/index.html
|
|
||||||
OUT_PRIVATE=/var/www/analytics-private/index.html
|
|
||||||
|
|
||||||
# Log lines whose URI matches this never enter the public tier. Extend it
|
|
||||||
# when the shop launches so order/payment URLs can never surface publicly.
|
|
||||||
CENSOR_RE='"uri":"/api'
|
|
||||||
|
|
||||||
# Serialize runs: a manual run racing the hourly timer once ingested the same
|
|
||||||
# rotated log twice (both processes passed the ledger check before either
|
|
||||||
# appended). Skip quietly if another run holds the lock - the timer comes
|
|
||||||
# around hourly anyway.
|
|
||||||
exec 9>$STATE_DIR/.lock
|
|
||||||
flock -n 9 || exit 0
|
|
||||||
|
|
||||||
# goaccess refuses a missing --db-path; recreate after a re-filter wipe.
|
|
||||||
mkdir -p "$STATE_DIR/db-private" "$STATE_DIR/db-public"
|
|
||||||
|
|
||||||
# Own IPs to keep out of the numbers (data quality, both tiers). The IPs
|
|
||||||
# live in /etc/goaccess/exclude-ips (one per line, # comments allowed) - ON
|
|
||||||
# THE HOST ONLY, never in this public repo.
|
|
||||||
exclude_args() {
|
|
||||||
[ -r /etc/goaccess/exclude-ips ] || return 0
|
|
||||||
while IFS= read -r ip; do
|
|
||||||
case "$ip" in ''|'#'*) continue ;; esac
|
|
||||||
printf -- '--exclude-ip=%s\n' "$ip"
|
|
||||||
done < /etc/goaccess/exclude-ips
|
|
||||||
}
|
|
||||||
|
|
||||||
# Shared quality filters, both tiers:
|
|
||||||
# --ignore-crawlers drops self-declared bots (ClaudeBot alone was 18% of
|
|
||||||
# traffic); --unknowns-as-crawlers additionally drops the headerless
|
|
||||||
# vulnerability scanners (57% of traffic, no User-Agent at all); the
|
|
||||||
# browsers-file catches known fake-browser scrapers.
|
|
||||||
# --ignore-referrer drops referrer values that can never carry real source
|
|
||||||
# information: '*catcrafts.*' covers self-referrals AND the parked redirect
|
|
||||||
# TLDs (a 301 never sets the redirecting host as referrer, so those are
|
|
||||||
# always crawlers stamping their pre-redirect URL); '*localhost*' is Android
|
|
||||||
# fediverse apps whose WebView UI lives on an embedded localhost page.
|
|
||||||
# $(exclude_args) is unquoted on purpose: one word per --exclude-ip flag.
|
|
||||||
common() {
|
|
||||||
goaccess "$@" \
|
|
||||||
--log-format=CADDY \
|
|
||||||
--ignore-crawlers \
|
|
||||||
--unknowns-as-crawlers \
|
|
||||||
--browsers-file=/etc/goaccess/browsers.list \
|
|
||||||
$(exclude_args) \
|
|
||||||
--ignore-referrer='*catcrafts.*' \
|
|
||||||
--ignore-referrer='*localhost*' \
|
|
||||||
--no-query-string \
|
|
||||||
--tz=Europe/Amsterdam
|
|
||||||
}
|
|
||||||
|
|
||||||
run_private() {
|
|
||||||
common "$@" --db-path="$STATE_DIR/db-private"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Public extras: --anonymize-ip zeroes the last octet before anything is
|
|
||||||
# stored; visitor IPs are personal data and stay out of the public page.
|
|
||||||
run_public() {
|
|
||||||
common "$@" --db-path="$STATE_DIR/db-public" \
|
|
||||||
--anonymize-ip \
|
|
||||||
--ignore-panel=HOSTS \
|
|
||||||
--ignore-panel=REFERRERS
|
|
||||||
}
|
|
||||||
|
|
||||||
# grep -v exits 1 when it outputs nothing - not an error here.
|
|
||||||
censor() {
|
|
||||||
grep -vE "$CENSOR_RE" || true
|
|
||||||
}
|
|
||||||
|
|
||||||
ingest() { # $1 = tier
|
|
||||||
ledger=$STATE_DIR/ingested-$1
|
|
||||||
touch "$ledger"
|
|
||||||
for f in "$LOG_DIR"/catcrafts.net-*.log.gz; do
|
|
||||||
[ -e "$f" ] || continue
|
|
||||||
grep -qxF "$f" "$ledger" && continue
|
|
||||||
case "$1" in
|
|
||||||
private) zcat "$f" | run_private - --persist --restore --process-and-exit ;;
|
|
||||||
public) zcat "$f" | censor | run_public - --persist --restore --process-and-exit ;;
|
|
||||||
esac
|
|
||||||
printf '%s\n' "$f" >>"$ledger"
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
ingest private
|
|
||||||
ingest public
|
|
||||||
|
|
||||||
run_private "$LIVE" --restore \
|
|
||||||
--html-report-title="catcrafts.net (private)" \
|
|
||||||
-o "$OUT_PRIVATE"
|
|
||||||
censor <"$LIVE" | run_public - --restore \
|
|
||||||
--html-report-title="catcrafts.net" \
|
|
||||||
-o "$OUT_PUBLIC"
|
|
||||||
|
|
@ -1,11 +0,0 @@
|
||||||
[Unit]
|
|
||||||
Description=Rebuild GoAccess analytics report for catcrafts.net
|
|
||||||
Documentation=https://goaccess.io/man
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=oneshot
|
|
||||||
# caddy owns the 0600 access logs; DB and output dirs are chowned to match.
|
|
||||||
User=caddy
|
|
||||||
Group=caddy
|
|
||||||
ExecStart=/usr/local/bin/catcrafts-analytics
|
|
||||||
Nice=10
|
|
||||||
|
|
@ -1,10 +0,0 @@
|
||||||
[Unit]
|
|
||||||
Description=Hourly GoAccess analytics report rebuild
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
OnCalendar=*-*-* *:07:00
|
|
||||||
RandomizedDelaySec=3m
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
|
|
@ -1,12 +0,0 @@
|
||||||
# Extra crawler classifications for GoAccess (--browsers-file), TAB-separated.
|
|
||||||
# Installed at /etc/goaccess/browsers.list; extends the builtin list.
|
|
||||||
#
|
|
||||||
# Frozen-UA scraper rotating through ~60 IPs: fakes Referer as reddit.com /
|
|
||||||
# localhost / none, never sends Sec-Fetch-* or Range, re-downloads media
|
|
||||||
# whole, probes /test.php. The UA itself is the proof of fakery: real Chrome
|
|
||||||
# mobile has sent a reduced UA ("Linux; Android 10; K", no device model)
|
|
||||||
# since ~2023, so bare Chrome claiming "Android 14; Pixel 8" cannot be real.
|
|
||||||
# Keep patterns EXACT: Voyager/Summit Lemmy apps on genuine Pixel 8s appear
|
|
||||||
# as "Pixel 8 Build/...; wv)" WebView UAs and must keep counting as humans.
|
|
||||||
Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Crawlers
|
|
||||||
Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Crawlers
|
|
||||||
|
|
@ -199,7 +199,7 @@ export const std::vector<LegalPage>& LegalPages() {
|
||||||
{
|
{
|
||||||
.slug = "privacy",
|
.slug = "privacy",
|
||||||
.title = "Privacy",
|
.title = "Privacy",
|
||||||
.updated = "2026-08-08",
|
.updated = "2026-08-04",
|
||||||
.lede = "What this site collects, why, and how to get rid of it. Written to describe what the code actually does. If you find a discrepancy, the code is the bug and a report is very welcome.",
|
.lede = "What this site collects, why, and how to get rid of it. Written to describe what the code actually does. If you find a discrepancy, the code is the bug and a report is very welcome.",
|
||||||
.sections = {
|
.sections = {
|
||||||
{ "Who is responsible",
|
{ "Who is responsible",
|
||||||
|
|
@ -220,13 +220,12 @@ export const std::vector<LegalPage>& LegalPages() {
|
||||||
} },
|
} },
|
||||||
{ "What this site does not do",
|
{ "What this site does not do",
|
||||||
{
|
{
|
||||||
"No analytics in your browser. No cookies, none at all, which is why there is no cookie banner. No third-party scripts, no fonts loaded from anyone else's server, no embedded video, no social buttons, no advertising, no profiling, no automated decision-making.",
|
"No analytics. No cookies, none at all, which is why there is no cookie banner. No third-party scripts, no fonts loaded from anyone else's server, no embedded video, no social buttons, no advertising, no profiling, no automated decision-making.",
|
||||||
"Everything the browser loads comes from catcrafts.net. Following a link out (to a fediverse thread, to Forgejo, to the Mollie payment page) puts you on that site under its terms, and Catcrafts has no visibility into what happens there.",
|
"Everything the browser loads comes from catcrafts.net. Following a link out (to a fediverse thread, to Forgejo, to the Mollie payment page) puts you on that site under its terms, and Catcrafts has no visibility into what happens there.",
|
||||||
} },
|
} },
|
||||||
{ "Server logs",
|
{ "Server logs",
|
||||||
{
|
{
|
||||||
"The web server keeps ordinary request logs. Those exist to debug faults and spot abuse, and are not connected to order records or used to build any kind of profile.",
|
"The web server keeps ordinary request logs. Those exist to debug faults and spot abuse, and are not connected to order records or used to build any kind of profile.",
|
||||||
"Those logs are also this site's only analytics: a page of aggregate statistics, computed on the server, with visitor addresses anonymized before anything is stored. A tally like that is nothing to keep secret, so you are free to take a look yourself at catcrafts.net/analytics.",
|
|
||||||
} },
|
} },
|
||||||
{ "Your rights",
|
{ "Your rights",
|
||||||
{
|
{
|
||||||
|
|
@ -234,7 +233,7 @@ export const std::vector<LegalPage>& LegalPages() {
|
||||||
} },
|
} },
|
||||||
{ "Changes",
|
{ "Changes",
|
||||||
{
|
{
|
||||||
"This page has a date at the top. Anything that changes what is collected or why gets a new date, and you do not have to take the date's word for it: every previous version of this page, with the exact diff, is public at catcrafts.net/legal/privacy/history.",
|
"This page has a date at the top. Anything that changes what is collected or why will be a new date and a note in the posts feed, not a silent edit.",
|
||||||
} },
|
} },
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue