Crafter.Build/AGENTS.md

AGENTS.md

Guidance for coding agents and contributors working on crafter-build itself. The README is the user-facing manual; this file holds what you need to change the build system. When the two disagree, the code wins: check the interface in interfaces/ and the --help output of the built binary.

Project overview

crafter-build is a C++26 modules build system. A project is described by a project.cpp that returns a Crafter::Configuration; the tool compiles that file to a shared object, loads it in-process and builds what it describes. This repository is self-hosting: its own project.cpp builds crafter-build with crafter-build, and its tests link the library and drive Build() in-process.

Path	What lives there
interfaces/Crafter.Build-<Partition>.cppm	Module interface partitions. Crafter.Build.cppm is the umbrella that re-exports them.
implementations/Crafter.Build-<Partition>.cpp	Implementation units, one per partition, named Crafter.Build:<Partition>_impl. main.cpp is a thin wrapper around Crafter::Run.
project.cpp	The self-host build description: a crafter.build-lib library, a crafter.build-exe executable, the test declarations and the lint rules.
tests/<Name>/main.cpp	One self-test per directory; tests/<Name>/fixture/ holds the sample project it builds.
examples/	Standalone user-facing projects, MIT licensed.
build.sh	Bootstrap script. It compiles everything by hand, without crafter-build.
wasi-runtime/	runtime.js and index.html.in, shipped for browser targets.
lib/	Vendored third-party headers (nlohmann/json, glslang's DirStackFileIncluder). Do not edit.
share/crafter-build/, bin/, build/, lib/*.a	Generated and gitignored.
packaging/flatpak/, .forgejo/workflows/ci.yaml	The Flatpak manifest with its metainfo, repo and ref files, and the Forgejo Actions CI that builds and publishes it.

Partitions and what they own:

• :Toolchain resolves the compiler once per process (GetToolchain): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from CRAFTER_BUILD_LLVM_DIR, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. SysrootStdModule picks the std module source for a sysroot build by the sysroot's libc++ release.
• :Clang holds Configuration, Build(), VariantId(), ApplyStandardArgs, HostTarget, the CLI entry point Crafter::Run, and the lint and test data types.
• :Platform runs commands, implements LoadProject (compile and load project.cpp), the host PCM cache, #pragma crafter dependency, and all OS-specific code.
• :Interface and :Implementation model module units, their staleness checks and their compiles.
• :External handles git + cmake dependencies, GitProject and LocalProject.
• :Test holds AddTest, RunTests and TestRunner.
• :Lint is the lint/format driver plus the libclang token and AST layer.
• :Progress is console output. :Shader compiles shaders through glslang. :Asset is the hook into Crafter.Asset.

Dev environment

The shipped artifact is the Flatpak (see "CI and publishing"); the loop below is the native development loop.

Prerequisites on Linux: clang (with libclang, which the AST lint rules load at runtime), lld, libc++ with its std module source (share/libc++/v1/std.cppm), cmake, git. crafter-build --version prints which compiler was resolved; CRAFTER_BUILD_LLVM_DIR points it at another LLVM prefix, and build.sh honours the same variable. Network access to forgejo.catcrafts.net and github.com is needed: loading project.cpp fetches Crafter.Math, Crafter.Asset and Crafter.LintRules into ~/.cache/crafter.build/external/, and the bootstrap clones glslang. Those sources are unpinned, so every invocation runs git pull on them.

./build.sh              # bootstrap: bin/crafter-build, lib/libcrafter-build.a, share/crafter-build/
./bin/crafter-build     # self-host build through project.cpp

Three things to know about this loop:

• bin/crafter-build is the bootstrap binary and the harness for every command below. The self-host build writes to bin/crafter.build-exe-<variant>/crafter-build and does not replace it.
• Changes under implementations/ are picked up by the self-host build and by the tests, which link the local library. Changes under interfaces/ are not: the harness compiles against the copies in share/crafter-build/ and the PCM cache built from them, so a new or changed declaration is invisible until you run ./build.sh again. The symptom is use of undeclared identifier for the symbol you just added.
• A new exported symbol that a project.cpp reaches on its default path also needs ./build.sh: the old binary does not export it, and LoadProject fails with an undefined symbol.

./build.sh clones and builds glslang on first run and takes several minutes; later runs are faster. ./bin/crafter-build clean deletes bin/ and build/ without loading project.cpp. Set CRAFTER_BUILD_HOME only when running a binary from outside this tree; CRAFTER_BUILD_MARCH and CRAFTER_BUILD_MTUNE default to native.

Build, test and style commands

./bin/crafter-build                          # build (self-host)
./bin/crafter-build test                     # build and run every self-test
./bin/crafter-build test 'Incremental*'      # name glob
./bin/crafter-build test --list              # enumerate without running
./bin/crafter-build test --jobs=1 --timeout=120
./bin/crafter-build lint                     # dry style gate, exit 1 on any finding
./bin/crafter-build format                   # apply the transform rules in place
./bin/crafter-build format --check           # CI-style dry run
./bin/crafter-build clean

Before you call a change done, all three must exit 0: format (then re-read your diff), lint, test. CI runs ./build.sh and crafter-build test on every PR; its lint step is currently commented out, so lint is enforced only by you. Failed, crashed and timed-out tests leave their output in build/test-logs/<name>.log.

Testing instructions

Tests are declared, not discovered. Each one is a line in project.cpp:

cfg.AddTest("IncrementalHeaderChange").Dependencies({ CrafterBuildLib.get() });

The source is tests/<Name>/main.cpp. The test inherits target, march and the rest from the root configuration and links the local crafter.build-lib, so import Crafter.Build; resolves to the code in this checkout. A test that calls LoadProject also needs .LinkFlag("-Wl,--export-dynamic").LinkFlag("-ldl"), as the existing ones show. The declarations sit in an if (linuxTarget) block that matches any *-linux-gnu or *-linux-musl target: the suite is Linux-host only today.

A test is an ordinary int main(). Exit 0 passes, nonzero fails, 77 skips (for a missing tool, never for a failed assertion). Each runs in its own subprocess, in parallel, with a 60 s default timeout.

Conventions the existing tests follow:

• Open main.cpp with a comment that states the behaviour under test and the failure it guards against. tests/CompilerWarning/main.cpp shows the shape.
• Collect failures with a Check(cond, msg) helper that prints FAIL: ... to stderr and counts, then return nonzero at the end. Do not stop at the first failure unless later steps are meaningless without it.
• Drive the engine in-process: Build(cfg, depResults, depMutex), RunTests, the lint driver. Do not shell out to bin/crafter-build and grep its stdout.
• Fixtures under tests/<Name>/fixture/ are inputs. A test that edits or touches files first stages a copy under fs::temp_directory_path() / "crafter-build-<name>" and removes it on the next run. Variant sources are checked in as *.in files and copied over the original inside the staged copy.
• Use a fresh depResults map per build pass: the map memoises each configuration's result for one pass, so reusing it skips rebuilds you meant to observe. Build() itself is safe to call repeatedly on the same Configuration.
• Set cfg.path before calling GetInterfacesAndImplementations, and spell cFiles as absolute paths: both resolve against the working directory, not the configuration.

A bug fix ships with a test that fails before the fix and passes after it, in its own test: commit when it documents the behaviour on its own.

Cross-target runs use test --target=<triple> with --runner=local or --runner=cmd:<wrapper> (wine, qemu-user, valgrind), or CRAFTER_BUILD_RUNNER_<TRIPLE> as the persistent form. A failed runner probe skips its tests rather than failing the suite.

Code style

Style is enforced by the 17 rules in Crafter.LintRules, which project.cpp pulls in through #pragma crafter dependency and applies with crafter-build format and lint. Most rules are transforms, so run format and let it restyle rather than hand-formatting. In short:

• The first three lines of every source: // SPDX-License-Identifier: LGPL-3.0-only, // SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®, blank. Examples use MIT.
• Four-space indentation, no tabs, no trailing whitespace, final newline.
• PascalCase for types, functions, statics and constants; camelCase for variables; a trailing _ is allowed.
• enum class only. std::println instead of iostream. std::format instead of string concatenation. std::int32_t and friends instead of int or long (exempt: main, argc/argv, extern "C" and foreign API signatures). No char* outside C interop.
• One declaration per statement. K&R braces. if (x) return; on one line when it fits. The formatter joins wrapped statements that fit on one line, so do not hand-wrap short ones. if ( and for ( with a space; no space before a call's (.
• const on scalar locals that are never written; constexpr on constants with constant initialisers.

Suppress one finding with // lint-disable-next-line <rule> or a whole file with // lint-disable-file <rule>. Both are rare in this tree and need a reason.

Beyond the rules:

• import std; everywhere; standard headers are never #included. C headers (clang-c/Index.h, dlfcn.h) go in the global module fragment after module;.
• Comments explain why: the failure mode, the constraint, the rejected alternative. The codebase carries long comments on purpose. Do not strip them, and write them for new non-obvious code. Commit bodies follow the same rule.
• There is no host-platform preprocessor split: the host is Linux. Target differences (mingw, wasm) branch on cfg.target strings at runtime, in the library as well as in project.cpp and tests.
• A build step fails when its command exits nonzero, never because it printed. Use RunBuildCommand or RunCommandChecked; warnings go to Progress::Diagnostic.
• Every output path goes through Configuration::BinDir(), BuildDir() and PcmDir(). Hand-rolled path / "bin" / ... clobbered variants before; do not add new ones.
• Anything that changes what gets compiled or bundled belongs in VariantId(). It already hashes type, debug, sysroot, defines, compile flags, wasm variant flags and uninterpreted project args.

Design rules

These were decided deliberately. Do not undo them without the maintainer's say-so.

1. Configuration is C++. No JSON, YAML, TOML or DSL for project, runner or lint configuration. Lint rules are C++ callbacks; there is no clang-format or clang-tidy integration and no separate "format rule" type. A rule that calls ctx.SetContent is a transform, and that is the whole mechanism.
2. Per-project artifacts go in <project>/build and <project>/bin. Only cross-project things go in ~/.cache/crafter.build/: the std PCM and Crafter.Build PCMs per target and march, and external clones keyed by their spec hash.
3. Sources ship, PCMs build locally. The Flatpak installs *.cppm under /app/share/crafter-build/. Nothing target-baked ships except the executable and the toolchain beside it, and the executable is built for a portable -march.
4. crafter-build is a monolithic executable with --export-dynamic. project.so resolves Crafter:: symbols from the running binary at dlopen time. There is no Windows host; x86_64-w64-mingw32 is a cross-compile target for user projects only.
5. Convention over configuration. Lead with the zero-config path. Env var, then CLI flag, then full project.cpp are the escalation, not the entry point.
6. One skip mechanism (runner probe plus exit 77) and one TestRunner shape (template fields with {bin}, {bundle} and similar placeholders; Cmd(...) covers any wrapper). Do not add a special-case runner class or a hardcoded "skip on this OS" branch.
7. The house-style rules stay in their own repo. They are not library API of crafter-build and are not forced on downstream projects.
8. Fix warts inline; do not stack workarounds. The previous version of this tool died of accumulated hacks. If a change needs a shortcut, say so in the PR rather than hiding it.
9. project.cpp keeps the library/executable split. crafter.build-lib owns the sources, dependencies and glslang; crafter.build-exe is main.cpp plus link flags.
10. Host-tooling builds (the target of a #pragma crafter dependency) get -fPIC and the host triple scoped to that build. Keep it scoped.

Commit and PR guidelines

• Branch from master and open the PR against master on the project's Forgejo at forgejo.catcrafts.net/Catcrafts/Crafter.Build. Issue work uses claude/issue-<n> branch names. Pushes go over SSH; HTTPS push is disabled on the server.
• Subject line: type(scope): imperative summary, with feat, fix, refactor, docs or test; the scope (lint, incremental, ...) is optional. Body: what was wrong, why, and why the fix has this shape, wrapped at about 75 columns. Read git log for the voice.
• Separate commits for the fix, its test and the README change when each stands on its own.
• Update the README in the same PR when user-visible behaviour changes.
• format, lint and test green before pushing. CI builds the Flatpak and runs the suite inside it on PRs, publishes the beta branch of the Catcrafts repo on every push to master, and the stable branch plus a release on an X.Y.Z tag.
• Never commit bin/, build/, share/crafter-build/, lib/*.a or fixture build outputs (all gitignored), and never edit the vendored files in lib/.
• Keep CRAFTER_BUILD_MARCH=x86-64-v2 in the manifest for the shipped binary: it has to run on any x86_64 machine. User projects still default to native.

Security considerations

• project.cpp is compiled and loaded into the crafter-build process, and every GitProject, #pragma crafter dependency and ExternalDependency clones a repository and runs its build on the host. Treat a dependency URL as code you are choosing to run. This repo's own dependencies are sibling Catcrafts repos tracking their default branch on purpose; pin with commit when adding anything else, and whenever reproducibility matters.
• Unpinned dependencies git pull on every invocation, so builds and tests need network. A missing or private repo surfaces as a clone failure before anything is built.
• --runner=cmd:<command> executes the named wrapper around every test binary. Wrappers come from --runner= or CRAFTER_BUILD_RUNNER_*; do not hardcode hosts, wrappers or credentials in project.cpp or tests.
• The linter loads libclang at runtime by name. CRAFTER_BUILD_LIBCLANG pins an exact path and must fail loudly when wrong, never fall back to another library.
• CI force-pushes the rolling latest tag and publishes a prerelease on every push to master with the workflow's own token, and signs the Flatpak repo with the key mounted from the runner host. Do not add secrets to the workflow or echo the environment in it.
• Tests write only under fs::temp_directory_path() and the fixture's own bin/ and build/. Keep it that way.

CI and publishing

.forgejo/workflows/ci.yaml runs on the flatpak runner label: a dedicated privileged forgejo-runner on the Hetzner box (forgejo-runner-flatpak.service, config /etc/forgejo-runner/config-flatpak.yaml), registered to this repository only. It is privileged because bubblewrap inside the job container needs it, and separate from the other runners so that privilege is not shared with any other repository. Its config allows four volumes, which the workflow mounts: crafter-flatpak-system (the Sdk install at /var/lib/flatpak), crafter-flatpak-builder (flatpak-builder state and ccache, so the LLVM and WASI modules are rebuilt only when their sources change; a cold run takes about 80 minutes on 8 cores, a warm one a fraction of that), /var/www/flatpak, the web root Caddy serves as https://flatpak.catcrafts.net/, and /etc/crafter-flatpak, which holds the repo signing key (signing-key.asc, no passphrase, and key-id), generated once on the host and mounted read-only into the job; nothing is stored in the forge. The live repo at /var/www/flatpak/repo is canonical; builds are promoted into it with flatpak build-commit-from, never built into it.

The key is made once, as root on the runner host, and the step fails with a message pointing here when it is missing:

install -d -m 700 /etc/crafter-flatpak/gnupg && export GNUPGHOME=/etc/crafter-flatpak/gnupg
gpg --batch --pinentry-mode loopback --passphrase '' --quick-generate-key 'Catcrafts Flatpak Repository <flatpak@catcrafts.net>' ed25519 sign never
KEYID=$(gpg --list-secret-keys --with-colons | awk -F: '/^sec/{print $5; exit}')
gpg --batch --pinentry-mode loopback --passphrase '' --export-secret-keys --armor "$KEYID" > /etc/crafter-flatpak/signing-key.asc
printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id

To release: add a <release> to packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml (its description becomes the release notes), merge, and push the tag X.Y.Z. CI refuses a tag that does not match the newest entry. Local builds: flatpak-builder --user --force-clean --ccache --repo=repo build-dir packaging/flatpak/net.catcrafts.CrafterBuild.yaml, then flatpak-builder --run build-dir <manifest> crafter-build --version.

Gotchas

• A #pragma crafter dependency target is rebuilt on every load on purpose: clang crashes rather than erroring when it imports a BMI whose own imported PCM was rebuilt underneath it, and the depfile check is what catches a regenerated host cache. Do not optimise the rebuild away.
• A host-tooling library such as Crafter.LintRules may import Crafter.Build; without declaring it as a dependency, because LoadProject fills the cache it resolves from. Listing Crafter.Build in its cfg.dependencies would clone and build all of crafter-build for every consumer. Its test does link crafter-build, so that test is wired behind IsTopLevelProject(): without the guard, crafter-build loading the library as a dependency would load its own project.cpp again through the test's GitProject and hit the cycle guard.
• crafter-build clean leaves a pragma dependency's build alone; it lives with the clone in the cache.
• The bootstrap binary is compiled without CRAFTER_BUILD_HAS_ASSET, so cfg.assets only works in the self-hosted binary. Changing asset extensions means updating two maps in Crafter.Build-Clang.cpp: assetOutput and the transitive copy loop.
• A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone.
• A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under share/crafter-build/libc++-modules/<major>/, one libcxx-<major>-modules manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under share/crafter-build/ to test against a foreign sysroot.
• VariantId() names the output directory, so CI globs bin/crafter.build-exe-<target>-<march>-*/ rather than predicting the hash.