Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues 5 Pull requests Projects Releases 7 Packages Wiki Activity Actions
Crafter.Build/AGENTS.md

197 lines
26 KiB
Markdown
Raw Permalink Normal View History

docs change
2026-10-02 20:35:38 +02:00
# AGENTS.md
Guidance for coding agents and contributors working on crafter-build itself. The [README](README.md) is the user-facing manual; this file holds what you need to change the build system. When the two disagree, the code wins: check the interface in `interfaces/` and the `--help` output of the built binary.
## Project overview
crafter-build is a C++26 modules build system. A project is described by a `project.cpp` that returns a `Crafter::Configuration`; the tool compiles that file to a shared object, loads it in-process and builds what it describes. This repository is self-hosting: its own `project.cpp` builds crafter-build with crafter-build, and its tests link the library and drive `Build()` in-process.
| Path | What lives there |
|---|---|
docs: describe the Linux-only host README and AGENTS.md stop documenting build.cmd, LIBCXX_DIR, the DLL-plus-launcher design rule and the cmd-over-SSH gotcha. The runner section also drops Ssh, SshWin and Wsl, which were removed from the code earlier but still had README entries, and the stale pointer to a deleted cross-arch test. ci.yaml loses the commented-out Windows build, staging and upload steps and the comments that counted them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 21:56:38 +02:00
| `interfaces/Crafter.Build-<Partition>.cppm` | Module interface partitions. `Crafter.Build.cppm` is the umbrella that re-exports them. |
docs change
2026-10-02 20:35:38 +02:00
| `implementations/Crafter.Build-<Partition>.cpp` | Implementation units, one per partition, named `Crafter.Build:<Partition>_impl`. `main.cpp` is a thin wrapper around `Crafter::Run`. |
feat(packaging): `run flatpak` and `run install` for a local Flatpak The local Flatpak build was one long flatpak-builder line in AGENTS.md, with its state wherever the caller happened to stand, and installing the result was not written down anywhere. It is not a rare chore: the development box compiles with the distro's clang and the Flatpak with its bundled LLVM, a major version apart at the moment, so anything toolchain-sensitive has to be tried in the Flatpak too. Both are now scripts in project.cpp, the first users of cfg.scripts. `run flatpak` builds the manifest as it is, from the working tree, and ends with `crafter-build --version` from inside the result. Its state is build/flatpak/ (state/, repo/, build-dir/). That is a fixed name and not a BuildDir() on purpose: the Flatpak is compiled by the toolchain in the sandbox, so it does not depend on the host variant, and a variant-keyed directory would strand the LLVM cache on every host compiler update. The manifest already skips build/ when it copies the sources. flatpak-builder keys its cache on the manifest's path, so the manifest is never rewritten into a copy the way CI stamps the version in, and a local build reports 0.0.0-dev. The only argument accepted is --jobs=<N>: anything that ends the run without a build would leave the smoke test checking the previous one. `run install` installs the last export as branch `local` from a user remote of its own and makes it current, leaving the published branch installed. It does not build first: flatpak-builder never caches a `type: dir` module, so every build reruns build.sh and the suite. remote-add --if-not-exists is followed by remote-modify --url, because the remote may exist and point at another checkout. Both refuse to run from the Flatpak build of crafter-build, which has no flatpak to drive. The smoke test goes through `flatpak build` and not `<builder> --run`, which is what CI uses. flatpak-builder 1.4.9 run as the org.flatpak.Builder app exits 1 when the command it ran succeeded and 0 when it failed (`true` gives 1, `exit 3` gives 0), so the script reported a good build as a failure. Measured on the 56-core box with LLVM and the WASI sysroot cached: 260 seconds, the crafter-build module only, 29 tests passing in the sandbox. A second `run install` after a rebuild moved the installed commit. `clean` now also drops that state, about 2 GB and a cold LLVM build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:29:32 +02:00
| `project.cpp` | The self-host build description: a `crafter.build-lib` library, a `crafter.build-exe` executable, the test declarations, the lint rules and the `flatpak` and `install` scripts. |
docs change
2026-10-02 20:35:38 +02:00
| `tests/<Name>/main.cpp` | One self-test per directory; `tests/<Name>/fixture/` holds the sample project it builds. |
| `examples/` | Standalone user-facing projects, MIT licensed. |
docs: describe the Linux-only host README and AGENTS.md stop documenting build.cmd, LIBCXX_DIR, the DLL-plus-launcher design rule and the cmd-over-SSH gotcha. The runner section also drops Ssh, SshWin and Wsl, which were removed from the code earlier but still had README entries, and the stale pointer to a deleted cross-arch test. ci.yaml loses the commented-out Windows build, staging and upload steps and the comments that counted them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 21:56:38 +02:00
| `build.sh` | Bootstrap script. It compiles everything by hand, without crafter-build. |
docs change
2026-10-02 20:35:38 +02:00
| `wasi-runtime/` | `runtime.js` and `index.html.in`, shipped for browser targets. |
| `lib/` | Vendored third-party headers (nlohmann/json, glslang's DirStackFileIncluder). Do not edit. |
| `share/crafter-build/`, `bin/`, `build/`, `lib/*.a` | Generated and gitignored. |
feat(ci): build and publish the Flatpak One job replaces the tarball pipeline. On a pull request it builds the Flatpak with flatpak-builder, which runs the bootstrap, the self-host build and the test suite inside the sandbox, and smoke-tests the result with `crafter-build --version` and the hello-world example. On a push to master it also bundles the build, promotes it into the live repo as the `beta` branch and refreshes the rolling `latest` prerelease. On a tag X.Y.Z it promotes as `stable`, refuses to proceed unless the metainfo's newest release is that version, and publishes a release whose notes are that entry's description. The job runs on the `flatpak` runner label: the privileged forgejo-runner on the Hetzner box, with the Sdk install, the flatpak-builder state and the repo's web root as volumes. The state volume is what keeps a normal run short: flatpak-builder rebuilds the LLVM and WASI modules only when their sources change. The live repo is canonical and never built into; builds are promoted with build-commit-from, then build-update-repo signs and prunes it and generates static deltas. The musl job, the per-march tarballs, the toolchain packages the runner used to install and the PKGBUILD go: the Flatpak is the one artifact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:36:55 +02:00
| `packaging/flatpak/`, `.forgejo/workflows/ci.yaml` | The Flatpak manifest with its metainfo, repo and ref files, and the Forgejo Actions CI that builds and publishes it. |
docs change
2026-10-02 20:35:38 +02:00
Partitions and what they own:
docs: describe how a sysroot build finds its std module source The README said a cross build precompiles the sysroot's own std.cppm, which was never the whole story for sysroots that do not ship one. It now states the order (the sysroot's, the toolchain's for the same major release, the sources crafter-build carries for other releases), what the failure looks like when nothing matches and how to resolve it, and lists the new directory in the install layout. AGENTS.md records the rule for the next LLVM bump: the release the bundled toolchain leaves behind stays available as a module-source module, because stable distros remain on it long after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 17:01:21 +02:00
- `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release.
feat(script): scripts in project.cpp, run with `crafter-build run` A project.cpp could describe how a project is built, tested and linted, but not the chores around it: packaging, installing, publishing. Those stayed shell lines in a README or in someone's history. Configuration gains `scripts`, a plain list like `defines`. A Script is a name and a C++ callback, as a lint rule is; there is no script language and no config file. The callback gets a ScriptContext (the project, the project file's directory, the arguments) and returns the exit status. ctx.Run runs a shell command with the terminal attached and returns its status. It is only stored when project.cpp loads, which happens for every verb and for every project that depends on this one. `crafter-build run <name> [args...]` runs one, a bare `run` lists them. Only the scripts of the project crafter-build was pointed at are runnable, never a dependency's. Everything after the script's name is handed to the script untouched. The two branches that do this sit ahead of every other branch of the parse loop, --version and --help included. Below the verbs, `run package clean` would delete build/, and any other word would land in the project's args and so in the variant hash, moving the build to another directory. Options before the name keep their meaning, so project args go there. The driver is its own partition, :Script. It refuses a project with no scripts, a name that is empty or starts with '-' (it could never be typed), a name declared twice (running either would be a guess) and an unknown name, each with exit 1 and without running anything. A script that throws exits 1. A return value outside 0..255 becomes 1: an exit status is eight bits, and a raw wait status such as 256 would otherwise wrap to success. Two helpers in :Platform: - RunCommandAttached runs a command with inherited stdio. Every helper so far captured through popen, which is right for a compile step and useless for a command that runs for half an hour or asks a question. It reads the wait status through DecodeWaitStatus, which RunCommandChecked and RunCommandWithTimeout now share. - ShellQuote is exported. It existed twice, as ShellQuote in :External and ShellQuoteSh in :Test, with the same behaviour; both are gone. A script that puts a path into a command needs it too. Not included: a string shorthand for one-command scripts (a one-line lambda covers it) and a ctx.Build() helper (Build() is already exported). A new partition and new exported symbols: run ./build.sh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:16:00 +02:00
- `:Clang` holds `Configuration`, `Build()`, `VariantId()`, `ApplyStandardArgs`, `HostTarget`, the CLI entry point `Crafter::Run`, and the lint, test and script data types.
docs change
2026-10-02 20:35:38 +02:00
- `:Platform` runs commands, implements `LoadProject` (compile and load `project.cpp`), the host PCM cache, `#pragma crafter dependency`, and all OS-specific code.
- `:Interface` and `:Implementation` model module units, their staleness checks and their compiles.
- `:External` handles git + cmake dependencies, `GitProject` and `LocalProject`.
- `:Test` holds `AddTest`, `RunTests` and `TestRunner`.
- `:Lint` is the lint/format driver plus the libclang token and AST layer.
feat(script): scripts in project.cpp, run with `crafter-build run` A project.cpp could describe how a project is built, tested and linted, but not the chores around it: packaging, installing, publishing. Those stayed shell lines in a README or in someone's history. Configuration gains `scripts`, a plain list like `defines`. A Script is a name and a C++ callback, as a lint rule is; there is no script language and no config file. The callback gets a ScriptContext (the project, the project file's directory, the arguments) and returns the exit status. ctx.Run runs a shell command with the terminal attached and returns its status. It is only stored when project.cpp loads, which happens for every verb and for every project that depends on this one. `crafter-build run <name> [args...]` runs one, a bare `run` lists them. Only the scripts of the project crafter-build was pointed at are runnable, never a dependency's. Everything after the script's name is handed to the script untouched. The two branches that do this sit ahead of every other branch of the parse loop, --version and --help included. Below the verbs, `run package clean` would delete build/, and any other word would land in the project's args and so in the variant hash, moving the build to another directory. Options before the name keep their meaning, so project args go there. The driver is its own partition, :Script. It refuses a project with no scripts, a name that is empty or starts with '-' (it could never be typed), a name declared twice (running either would be a guess) and an unknown name, each with exit 1 and without running anything. A script that throws exits 1. A return value outside 0..255 becomes 1: an exit status is eight bits, and a raw wait status such as 256 would otherwise wrap to success. Two helpers in :Platform: - RunCommandAttached runs a command with inherited stdio. Every helper so far captured through popen, which is right for a compile step and useless for a command that runs for half an hour or asks a question. It reads the wait status through DecodeWaitStatus, which RunCommandChecked and RunCommandWithTimeout now share. - ShellQuote is exported. It existed twice, as ShellQuote in :External and ShellQuoteSh in :Test, with the same behaviour; both are gone. A script that puts a path into a command needs it too. Not included: a string shorthand for one-command scripts (a one-line lambda covers it) and a ctx.Build() helper (Build() is already exported). A new partition and new exported symbols: run ./build.sh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:16:00 +02:00
- `:Script` is the driver behind `crafter-build run`: it looks a name up in `cfg.scripts`, calls the callback and turns what it returns into the exit code.
feat(agents): keep dependencies' AGENTS.consumer.md in AGENTS.md An agent working on a crafter-build project reads its AGENTS.md, and learns nothing there about the libraries the project builds against unless someone copied that in by hand. This repository shows the cost: the code style section of its AGENTS.md restates what Crafter.LintRules enforces, and goes stale whenever the rules change. A library now ships those notes itself, as AGENTS.consumer.md beside its project.cpp; nothing in its project.cpp changes. When a consumer has an AGENTS.md beside its own project.cpp, `crafter-build` and `crafter-build test` keep a section in it, between two marker comments, holding its dependencies' notes. The section is appended the first time and rewritten in place after that. The file is written only when the section changes, and an AGENTS.md is never created. lint, format, run and clean leave it alone. The update runs after loading, which has just fetched every dependency, and before compiling, so a build that fails still has the notes. cfg.dependencyNotes picks the dependencies. Transitive, the default, follows `dependencies` all the way down from the project's own configurations, its tests and its `#pragma crafter dependency` tooling; Direct stops at the first loaded project on each path; None removes the section. A dependency's own pragma tooling never counts: it is about how that dependency is built, not how it is used, and being no `dependencies` edge it is never reached. Nor are a dependency's tests: nothing builds them for the consumer, and their configurations can point at a local their project file has since moved out of. The walk has to know which Configurations belong to another project, and where that project lives, and cfg.path cannot say: only the Configuration GitProject and LocalProject return is rebased, so a dependency's internal configurations keep "./", which resolves to the consumer. The loader records it instead, LoadedProjectOf beside the project cache, and keeps what the root's pragmas resolved to past the frame LoadProject pops, PragmaDependencies. Neither is exported. The markers count only as whole lines outside fenced code blocks, so documentation can quote them. Unbalanced markers and a document ending inside a code block leave the file alone with a warning: guessing which part of someone's file is generated would overwrite their text, and missing the section would add another one on every build. A note that would break the section when it is read back, through a marker line of its own or a code block it never closes, is left out with a warning. The file is replaced through a temporary renamed onto the symlink's target, so a linked AGENTS.md stays a link and a write cut short leaves the old file rather than part of the new one. The code is a new partition, Crafter.Build:Agents, listed in build.sh, CrafterBuildModules, project.cpp and the umbrella. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 03:18:06 +01:00
- `:Agents` keeps the dependency notes section of a project's AGENTS.md: it walks the dependencies `cfg.dependencyNotes` selects, reads each one's `AGENTS.consumer.md` and merges them between the section's markers. Which Configurations belong to another project, and where it lives, comes from the loader (`LoadedProjectOf` in `:External`, `PragmaDependencies` in `:Platform`), because only the Configuration `GitProject` and `LocalProject` return has its `path` rebased.
docs change
2026-10-02 20:35:38 +02:00
- `:Progress` is console output. `:Shader` compiles shaders through glslang. `:Asset` is the hook into Crafter.Asset.
## Dev environment
docs: install from the Catcrafts Flatpak repo The README leads with the Flatpak install, keeps the native loop as the way to build from source, and drops the tarball, musl and makepkg text that no longer describes anything CI produces. The install layout shows the /app tree and the per-app cache directory. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:36:56 +02:00
The shipped artifact is the Flatpak (see "CI and publishing"); the loop below is the native development loop.
docs: describe the toolchain resolver and --version Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:26:31 +02:00
Prerequisites on Linux: `clang` (with `libclang`, which the AST lint rules load at runtime), `lld`, `libc++` with its std module source (`share/libc++/v1/std.cppm`), `cmake`, `git`. `crafter-build --version` prints which compiler was resolved; `CRAFTER_BUILD_LLVM_DIR` points it at another LLVM prefix, and `build.sh` honours the same variable. Network access to `forgejo.catcrafts.net` and `github.com` is needed: loading `project.cpp` fetches Crafter.Math, Crafter.Asset and Crafter.LintRules into `~/.cache/crafter.build/external/`, and the bootstrap clones glslang. Those sources are unpinned, so every invocation runs `git pull` on them.
docs change
2026-10-02 20:35:38 +02:00
```bash
./build.sh # bootstrap: bin/crafter-build, lib/libcrafter-build.a, share/crafter-build/
./bin/crafter-build # self-host build through project.cpp
```
Three things to know about this loop:
- `bin/crafter-build` is the bootstrap binary and the harness for every command below. The self-host build writes to `bin/crafter.build-exe-<variant>/crafter-build` and does not replace it.
docs: correct what the sibling-args docs promise The README suggested giving one sibling a list without a flag only the project reads, so that the sibling would not get a build directory of its own. When another path loads the same sibling with the full list, that filter is what gives it the second directory: two builds of one project, both in the program (#41). It now says to build such a list once and give it to every sibling. It also says the default is read when the spec is made, so a spec made at namespace scope, on another thread or in a script forwards nothing, and that cfg.projectArgs, the part ApplyStandardArgs did not interpret, is no substitute: it drops the target and the build type. The comments on ProjectArgs() and on Configuration::projectArgs say the same. The spec comment named --local as a flag the default carries down a chain of siblings. Through a GitProject hop it breaks: the clone has no checkouts beside it. The GitProject comment said the clone is keyed by the args; only the Configuration is, and every arg list builds in the one clone. AGENTS.md now notes that a stale harness gives no symptom for a changed default member initialiser: project.cpp silently gets the old default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 05:10:48 +01:00
- Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added. A changed default member initialiser has no symptom at all: `project.cpp` compiles against the old one and quietly gets the old default, as `project.cpp`'s sibling loads do with a harness that predates `GitProjectSpec::args = ProjectArgs()`.
docs: describe the Linux-only host README and AGENTS.md stop documenting build.cmd, LIBCXX_DIR, the DLL-plus-launcher design rule and the cmd-over-SSH gotcha. The runner section also drops Ssh, SshWin and Wsl, which were removed from the code earlier but still had README entries, and the stale pointer to a deleted cross-arch test. ci.yaml loses the commented-out Windows build, staging and upload steps and the comments that counted them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 21:56:38 +02:00
- A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol.
docs change
2026-10-02 20:35:38 +02:00
feat(packaging): `run flatpak` and `run install` for a local Flatpak The local Flatpak build was one long flatpak-builder line in AGENTS.md, with its state wherever the caller happened to stand, and installing the result was not written down anywhere. It is not a rare chore: the development box compiles with the distro's clang and the Flatpak with its bundled LLVM, a major version apart at the moment, so anything toolchain-sensitive has to be tried in the Flatpak too. Both are now scripts in project.cpp, the first users of cfg.scripts. `run flatpak` builds the manifest as it is, from the working tree, and ends with `crafter-build --version` from inside the result. Its state is build/flatpak/ (state/, repo/, build-dir/). That is a fixed name and not a BuildDir() on purpose: the Flatpak is compiled by the toolchain in the sandbox, so it does not depend on the host variant, and a variant-keyed directory would strand the LLVM cache on every host compiler update. The manifest already skips build/ when it copies the sources. flatpak-builder keys its cache on the manifest's path, so the manifest is never rewritten into a copy the way CI stamps the version in, and a local build reports 0.0.0-dev. The only argument accepted is --jobs=<N>: anything that ends the run without a build would leave the smoke test checking the previous one. `run install` installs the last export as branch `local` from a user remote of its own and makes it current, leaving the published branch installed. It does not build first: flatpak-builder never caches a `type: dir` module, so every build reruns build.sh and the suite. remote-add --if-not-exists is followed by remote-modify --url, because the remote may exist and point at another checkout. Both refuse to run from the Flatpak build of crafter-build, which has no flatpak to drive. The smoke test goes through `flatpak build` and not `<builder> --run`, which is what CI uses. flatpak-builder 1.4.9 run as the org.flatpak.Builder app exits 1 when the command it ran succeeded and 0 when it failed (`true` gives 1, `exit 3` gives 0), so the script reported a good build as a failure. Measured on the 56-core box with LLVM and the WASI sysroot cached: 260 seconds, the crafter-build module only, 29 tests passing in the sandbox. A second `run install` after a rebuild moved the installed commit. `clean` now also drops that state, about 2 GB and a cold LLVM build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:29:32 +02:00
`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. That takes `bin/crafter-build` itself with it, and `build/flatpak`, the state of the local Flatpak build (see "CI and publishing"): about 2 GB, and the next `run flatpak` rebuilds LLVM. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`.
docs change
2026-10-02 20:35:38 +02:00
## Build, test and style commands
```bash
./bin/crafter-build # build (self-host)
./bin/crafter-build test # build and run every self-test
./bin/crafter-build test 'Incremental*' # name glob
./bin/crafter-build test --list # enumerate without running
./bin/crafter-build test --jobs=1 --timeout=120
./bin/crafter-build lint # dry style gate, exit 1 on any finding
./bin/crafter-build format # apply the transform rules in place
./bin/crafter-build format --check # CI-style dry run
feat(script): scripts in project.cpp, run with `crafter-build run` A project.cpp could describe how a project is built, tested and linted, but not the chores around it: packaging, installing, publishing. Those stayed shell lines in a README or in someone's history. Configuration gains `scripts`, a plain list like `defines`. A Script is a name and a C++ callback, as a lint rule is; there is no script language and no config file. The callback gets a ScriptContext (the project, the project file's directory, the arguments) and returns the exit status. ctx.Run runs a shell command with the terminal attached and returns its status. It is only stored when project.cpp loads, which happens for every verb and for every project that depends on this one. `crafter-build run <name> [args...]` runs one, a bare `run` lists them. Only the scripts of the project crafter-build was pointed at are runnable, never a dependency's. Everything after the script's name is handed to the script untouched. The two branches that do this sit ahead of every other branch of the parse loop, --version and --help included. Below the verbs, `run package clean` would delete build/, and any other word would land in the project's args and so in the variant hash, moving the build to another directory. Options before the name keep their meaning, so project args go there. The driver is its own partition, :Script. It refuses a project with no scripts, a name that is empty or starts with '-' (it could never be typed), a name declared twice (running either would be a guess) and an unknown name, each with exit 1 and without running anything. A script that throws exits 1. A return value outside 0..255 becomes 1: an exit status is eight bits, and a raw wait status such as 256 would otherwise wrap to success. Two helpers in :Platform: - RunCommandAttached runs a command with inherited stdio. Every helper so far captured through popen, which is right for a compile step and useless for a command that runs for half an hour or asks a question. It reads the wait status through DecodeWaitStatus, which RunCommandChecked and RunCommandWithTimeout now share. - ShellQuote is exported. It existed twice, as ShellQuote in :External and ShellQuoteSh in :Test, with the same behaviour; both are gone. A script that puts a path into a command needs it too. Not included: a string shorthand for one-command scripts (a one-line lambda covers it) and a ctx.Build() helper (Build() is already exported). A new partition and new exported symbols: run ./build.sh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:16:00 +02:00
./bin/crafter-build run # list the scripts project.cpp defines
./bin/crafter-build run <name> [args...] # run one; everything after the name is the script's
docs change
2026-10-02 20:35:38 +02:00
./bin/crafter-build clean
```
Before you call a change done, all three must exit 0: `format` (then re-read your diff), `lint`, `test`. CI runs `./build.sh` and `crafter-build test` on every PR; its lint step is currently commented out, so lint is enforced only by you. Failed, crashed and timed-out tests leave their output in `build/test-logs/<name>.log`.
## Testing instructions
Tests are declared, not discovered. Each one is a line in `project.cpp`:
```cpp
cfg.AddTest("IncrementalHeaderChange").Dependencies({ CrafterBuildLib.get() });
```
Merge branch 'claude/project-dependencies': project.cpp dependencies, AGENTS.md Brings in `#pragma crafter dependency`, the switch to Crafter.LintRules for the house style, and the contributor guide in AGENTS.md. The branch predates the musl host work on master, and both rewrote the command that compiles project.so, so the merge reconciles them by hand: - LoadProject compiles project.so for hostConfig.target (HostTarget(), from master) with the pre-dependency module path and archives from the branch. -fuse-ld=lld appears once, with both reasons in the comment. - ForceHostTooling added -fPIC only for the literal x86_64-pc-linux-gnu triple. hostConfig now comes from HostTarget(), so on Alpine the host is x86_64-alpine-linux-musl and the dependency archive would not have been position-independent. It now matches any *-linux-gnu or *-linux-musl host, as project.cpp and the cmake flags do since the musl change, and its comment no longer argues against HostTarget(). - The ProjectDependency test compared the dependency's target against the glibc triple, which the musl CI job would fail; it compares against HostTarget(). - README keeps master's paragraph on the bootstrap triple. AGENTS.md describes the test block as `if (linuxTarget)`, which is what project.cpp has now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 20:47:23 +02:00
The source is `tests/<Name>/main.cpp`. The test inherits target, march and the rest from the root configuration and links the local `crafter.build-lib`, so `import Crafter.Build;` resolves to the code in this checkout. A test that calls `LoadProject` also needs `.LinkFlag("-Wl,--export-dynamic").LinkFlag("-ldl")`, as the existing ones show. The declarations sit in an `if (linuxTarget)` block that matches any `*-linux-gnu` or `*-linux-musl` target: the suite is Linux-host only today.
docs change
2026-10-02 20:35:38 +02:00
A test is an ordinary `int main()`. Exit 0 passes, nonzero fails, 77 skips (for a missing tool, never for a failed assertion). Each runs in its own subprocess, in parallel, with a 60 s default timeout.
Conventions the existing tests follow:
- Open `main.cpp` with a comment that states the behaviour under test and the failure it guards against. `tests/CompilerWarning/main.cpp` shows the shape.
- Collect failures with a `Check(cond, msg)` helper that prints `FAIL: ...` to stderr and counts, then return nonzero at the end. Do not stop at the first failure unless later steps are meaningless without it.
- Drive the engine in-process: `Build(cfg, depResults, depMutex)`, `RunTests`, the lint driver. Do not shell out to `bin/crafter-build` and grep its stdout.
- Fixtures under `tests/<Name>/fixture/` are inputs. A test that edits or touches files first stages a copy under `fs::temp_directory_path() / "crafter-build-<name>"` and removes it on the next run. Variant sources are checked in as `*.in` files and copied over the original inside the staged copy.
fix(external): fetch and build each clone once per build pass `crafter-build test` builds a project's tests side by side, and a project whose tests each copy its externalDependencies hands every one of them the same unpinned dependency. Each test's Build() pulled the shared clone. Before the clone had a lock, those pulls ran at the same time and broke its .git/FETCH_HEAD, which failed every test of Crafter.Graphics (issue #34). Since 653dd0b they wait for each other, but every one is still a network round trip: 26 tests cost 26 pulls of each dependency, one after another, and for a CMake dependency as many configure checks and builds. depResults already records one build pass, keyed by the PcmDir of every dependency configuration built in it. An external dependency now goes in the same map, keyed by the directory its build leaves (ExternalBuildDir: the clone, or for CMake its build directory). The first configuration that needs it fetches and builds it; the others wait for that and take its outcome. Keying on the build directory rather than the clone keeps a CMake declaration from taking the outcome of a header-only declaration of the same clone, which only fetched it. A fresh depResults is a fresh pass, so the next build still pulls, and every caller of Build() gets this: RunTests, a plain build, and the pragma dependencies LoadProject builds. To make that possible BuildExternal is split. BuildExternal is now the shared work, lock, fetch, configure and build, and returns the error. ExternalOutputs computes what one declaration compiles and links with, because two declarations of one clone can differ in includeDirs, libDirs and libs; ExternalBuildResult loses its error field. The work no longer stops between steps when the configuration that started it is cancelled: others may be waiting on it, and a run cut short would leave them a clone nobody fetched. A dependency configuration's Build() already runs to completion for the same reason. The promise and future the dependency threads used to build or wait for a configuration are now one helper, OncePerPass, which both kinds use. An exception thrown in BuildExternal used to escape its thread into std::terminate; it is now a build error, as for a dependency. With eight tests that share a local dependency, a test run fetched it eight times before this and fetches it once now (SharedExternalFetch, next commit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 02:09:24 +01:00
- Use a fresh `depResults` map per build pass: the map memoises each configuration's result, and each external dependency's fetch and build, for one pass, so reusing it skips rebuilds and pulls you meant to observe. `Build()` itself is safe to call repeatedly on the same `Configuration`.
docs change
2026-10-02 20:35:38 +02:00
- Set `cfg.path` before calling `GetInterfacesAndImplementations`, and spell `cFiles` as absolute paths: both resolve against the working directory, not the configuration.
A bug fix ships with a test that fails before the fix and passes after it, in its own `test:` commit when it documents the behaviour on its own.
Cross-target runs use `test --target=<triple>` with `--runner=local` or `--runner=cmd:<wrapper>` (wine, qemu-user, valgrind), or `CRAFTER_BUILD_RUNNER_<TRIPLE>` as the persistent form. A failed runner probe skips its tests rather than failing the suite.
## Code style
refactor(lint): check this tree with the lite house rules Crafter.LintRules now has AddLiteRules: the house rules without the ones that only decide layout, whitespace and naming. This tree switches to it. The full set's naming rule now wants camelCase statics and constants, which this tree spells PascalCase throughout, so `lint` with it reported 49 findings on code that follows the documented convention; with the lite set `lint` and `format --check` are clean. AGENTS.md described the 17 rules as the enforced style and told contributors to let `format` restyle their code. It now says which 8 rules `lint` enforces, that `format` only rewrites string concatenation and builtin integer types, and lists layout and naming as conventions kept by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 02:58:41 +01:00
`project.cpp` pulls in [Crafter.LintRules](https://forgejo.catcrafts.net/Catcrafts/Crafter.LintRules) through `#pragma crafter dependency` and applies its lite set, `LintRules::AddLiteRules`: the 8 rules that ask for different code, without the ones that only decide layout, whitespace and naming. `lint` reports all 8; `format` applies the two transforms among them, `format-concat` and `fixed-width-types`, and leaves everything else as written. Enforced:
docs change
2026-10-02 20:35:38 +02:00
- The first three lines of every source: `// SPDX-License-Identifier: LGPL-3.0-only`, `// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®`, blank. Examples use MIT.
- `enum class` only. `std::println` instead of iostream. `std::format` instead of string concatenation. `std::int32_t` and friends instead of `int` or `long` (exempt: `main`, `argc`/`argv`, `extern "C"` and foreign API signatures). No `char*` outside C interop.
- `const` on scalar locals that are never written; `constexpr` on constants with constant initialisers.
refactor(lint): check this tree with the lite house rules Crafter.LintRules now has AddLiteRules: the house rules without the ones that only decide layout, whitespace and naming. This tree switches to it. The full set's naming rule now wants camelCase statics and constants, which this tree spells PascalCase throughout, so `lint` with it reported 49 findings on code that follows the documented convention; with the lite set `lint` and `format --check` are clean. AGENTS.md described the 17 rules as the enforced style and told contributors to let `format` restyle their code. It now says which 8 rules `lint` enforces, that `format` only rewrites string concatenation and builtin integer types, and lists layout and naming as conventions kept by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 02:58:41 +01:00
Not checked, so match the surrounding code:
- Four-space indentation, no tabs, no trailing whitespace, final newline.
- `PascalCase` for types, functions, statics and constants; `camelCase` for variables; a trailing `_` is allowed.
- One declaration per statement. K&R braces. `if (x) return;` on one line when it fits, and no hand-wrapping of a statement that fits on one line. `if (` and `for (` with a space; no space before a call's `(`.
docs change
2026-10-02 20:35:38 +02:00
Suppress one finding with `// lint-disable-next-line <rule>` or a whole file with `// lint-disable-file <rule>`. Both are rare in this tree and need a reason.
Beyond the rules:
docs: describe the Linux-only host README and AGENTS.md stop documenting build.cmd, LIBCXX_DIR, the DLL-plus-launcher design rule and the cmd-over-SSH gotcha. The runner section also drops Ssh, SshWin and Wsl, which were removed from the code earlier but still had README entries, and the stale pointer to a deleted cross-arch test. ci.yaml loses the commented-out Windows build, staging and upload steps and the comments that counted them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 21:56:38 +02:00
- `import std;` everywhere; standard headers are never `#include`d. C headers (`clang-c/Index.h`, `dlfcn.h`) go in the global module fragment after `module;`.
docs change
2026-10-02 20:35:38 +02:00
- Comments explain why: the failure mode, the constraint, the rejected alternative. The codebase carries long comments on purpose. Do not strip them, and write them for new non-obvious code. Commit bodies follow the same rule.
docs: describe the Linux-only host README and AGENTS.md stop documenting build.cmd, LIBCXX_DIR, the DLL-plus-launcher design rule and the cmd-over-SSH gotcha. The runner section also drops Ssh, SshWin and Wsl, which were removed from the code earlier but still had README entries, and the stale pointer to a deleted cross-arch test. ci.yaml loses the commented-out Windows build, staging and upload steps and the comments that counted them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 21:56:38 +02:00
- There is no host-platform preprocessor split: the host is Linux. Target differences (mingw, wasm) branch on `cfg.target` strings at runtime, in the library as well as in `project.cpp` and tests.
docs change
2026-10-02 20:35:38 +02:00
- A build step fails when its command exits nonzero, never because it printed. Use `RunBuildCommand` or `RunCommandChecked`; warnings go to `Progress::Diagnostic`.
- Every output path goes through `Configuration::BinDir()`, `BuildDir()` and `PcmDir()`. Hand-rolled `path / "bin" / ...` clobbered variants before; do not add new ones.
docs: wasm targets and their C++ exceptions The README described the mingw target and said nothing about wasm. It now says where the WASI sysroot comes from, that C++ exceptions work as on the other targets and what that takes (-fwasm-exceptions in the standardized form, the sysroot's exception-enabled libc++, -lunwind added to the link), which engines run the result, and that an exception escaping main ends the program without calling std::terminate. AGENTS.md says where the wasm flags live, that VariantId and the std PCM directory hash them, and that the wasm tests skip where the distro's sysroot lacks the eh/ libraries, so they run in the Flatpak. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 05:55:00 +01:00
- Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, the wasm baseline and variant flags, uninterpreted project args and the toolchain (`Toolchain::Key()`).
docs change
2026-10-02 20:35:38 +02:00
## Design rules
These were decided deliberately. Do not undo them without the maintainer's say-so.
1. **Configuration is C++.** No JSON, YAML, TOML or DSL for project, runner or lint configuration. Lint rules are C++ callbacks; there is no clang-format or clang-tidy integration and no separate "format rule" type. A rule that calls `ctx.SetContent` is a transform, and that is the whole mechanism.
2. **Per-project artifacts go in `<project>/build` and `<project>/bin`. Only cross-project things go in `~/.cache/crafter.build/`:** the std PCM and Crafter.Build PCMs per target and march, and external clones keyed by their spec hash.
feat(ci): build and publish the Flatpak One job replaces the tarball pipeline. On a pull request it builds the Flatpak with flatpak-builder, which runs the bootstrap, the self-host build and the test suite inside the sandbox, and smoke-tests the result with `crafter-build --version` and the hello-world example. On a push to master it also bundles the build, promotes it into the live repo as the `beta` branch and refreshes the rolling `latest` prerelease. On a tag X.Y.Z it promotes as `stable`, refuses to proceed unless the metainfo's newest release is that version, and publishes a release whose notes are that entry's description. The job runs on the `flatpak` runner label: the privileged forgejo-runner on the Hetzner box, with the Sdk install, the flatpak-builder state and the repo's web root as volumes. The state volume is what keeps a normal run short: flatpak-builder rebuilds the LLVM and WASI modules only when their sources change. The live repo is canonical and never built into; builds are promoted with build-commit-from, then build-update-repo signs and prunes it and generates static deltas. The musl job, the per-march tarballs, the toolchain packages the runner used to install and the PKGBUILD go: the Flatpak is the one artifact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:36:55 +02:00
3. **Sources ship, PCMs build locally.** The Flatpak installs `*.cppm` under `/app/share/crafter-build/`. Nothing target-baked ships except the executable and the toolchain beside it, and the executable is built for a portable `-march`.
docs: describe the Linux-only host README and AGENTS.md stop documenting build.cmd, LIBCXX_DIR, the DLL-plus-launcher design rule and the cmd-over-SSH gotcha. The runner section also drops Ssh, SshWin and Wsl, which were removed from the code earlier but still had README entries, and the stale pointer to a deleted cross-arch test. ci.yaml loses the commented-out Windows build, staging and upload steps and the comments that counted them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 21:56:38 +02:00
4. **crafter-build is a monolithic executable with `--export-dynamic`.** `project.so` resolves `Crafter::` symbols from the running binary at dlopen time. There is no Windows host; `x86_64-w64-mingw32` is a cross-compile target for user projects only.
docs change
2026-10-02 20:35:38 +02:00
5. **Convention over configuration.** Lead with the zero-config path. Env var, then CLI flag, then full `project.cpp` are the escalation, not the entry point.
6. **One skip mechanism** (runner probe plus exit 77) and **one `TestRunner` shape** (template fields with `{bin}`, `{bundle}` and similar placeholders; `Cmd(...)` covers any wrapper). Do not add a special-case runner class or a hardcoded "skip on this OS" branch.
7. **The house-style rules stay in their own repo.** They are not library API of crafter-build and are not forced on downstream projects.
8. **Fix warts inline; do not stack workarounds.** The previous version of this tool died of accumulated hacks. If a change needs a shortcut, say so in the PR rather than hiding it.
9. **`project.cpp` keeps the library/executable split.** `crafter.build-lib` owns the sources, dependencies and glslang; `crafter.build-exe` is `main.cpp` plus link flags.
10. **Host-tooling builds** (the target of a `#pragma crafter dependency`) get `-fPIC` and the host triple scoped to that build. Keep it scoped.
## Commit and PR guidelines
- Branch from `master` and open the PR against `master` on the project's Forgejo at `forgejo.catcrafts.net/Catcrafts/Crafter.Build`. Issue work uses `claude/issue-<n>` branch names. Pushes go over SSH; HTTPS push is disabled on the server.
- Subject line: `type(scope): imperative summary`, with `feat`, `fix`, `refactor`, `docs` or `test`; the scope (`lint`, `incremental`, ...) is optional. Body: what was wrong, why, and why the fix has this shape, wrapped at about 75 columns. Read `git log` for the voice.
- Separate commits for the fix, its test and the README change when each stands on its own.
- Update the README in the same PR when user-visible behaviour changes.
feat(ci): build and publish the Flatpak One job replaces the tarball pipeline. On a pull request it builds the Flatpak with flatpak-builder, which runs the bootstrap, the self-host build and the test suite inside the sandbox, and smoke-tests the result with `crafter-build --version` and the hello-world example. On a push to master it also bundles the build, promotes it into the live repo as the `beta` branch and refreshes the rolling `latest` prerelease. On a tag X.Y.Z it promotes as `stable`, refuses to proceed unless the metainfo's newest release is that version, and publishes a release whose notes are that entry's description. The job runs on the `flatpak` runner label: the privileged forgejo-runner on the Hetzner box, with the Sdk install, the flatpak-builder state and the repo's web root as volumes. The state volume is what keeps a normal run short: flatpak-builder rebuilds the LLVM and WASI modules only when their sources change. The live repo is canonical and never built into; builds are promoted with build-commit-from, then build-update-repo signs and prunes it and generates static deltas. The musl job, the per-march tarballs, the toolchain packages the runner used to install and the PKGBUILD go: the Flatpak is the one artifact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:36:55 +02:00
- `format`, `lint` and `test` green before pushing. CI builds the Flatpak and runs the suite inside it on PRs, publishes the `beta` branch of the Catcrafts repo on every push to `master`, and the `stable` branch plus a release on an `X.Y.Z` tag.
docs change
2026-10-02 20:35:38 +02:00
- Never commit `bin/`, `build/`, `share/crafter-build/`, `lib/*.a` or fixture build outputs (all gitignored), and never edit the vendored files in `lib/`.
feat(ci): build and publish the Flatpak One job replaces the tarball pipeline. On a pull request it builds the Flatpak with flatpak-builder, which runs the bootstrap, the self-host build and the test suite inside the sandbox, and smoke-tests the result with `crafter-build --version` and the hello-world example. On a push to master it also bundles the build, promotes it into the live repo as the `beta` branch and refreshes the rolling `latest` prerelease. On a tag X.Y.Z it promotes as `stable`, refuses to proceed unless the metainfo's newest release is that version, and publishes a release whose notes are that entry's description. The job runs on the `flatpak` runner label: the privileged forgejo-runner on the Hetzner box, with the Sdk install, the flatpak-builder state and the repo's web root as volumes. The state volume is what keeps a normal run short: flatpak-builder rebuilds the LLVM and WASI modules only when their sources change. The live repo is canonical and never built into; builds are promoted with build-commit-from, then build-update-repo signs and prunes it and generates static deltas. The musl job, the per-march tarballs, the toolchain packages the runner used to install and the PKGBUILD go: the Flatpak is the one artifact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:36:55 +02:00
- Keep `CRAFTER_BUILD_MARCH=x86-64-v2` in the manifest for the shipped binary: it has to run on any x86_64 machine. User projects still default to `native`.
docs change
2026-10-02 20:35:38 +02:00
## Security considerations
- `project.cpp` is compiled and loaded into the crafter-build process, and every `GitProject`, `#pragma crafter dependency` and `ExternalDependency` clones a repository and runs its build on the host. Treat a dependency URL as code you are choosing to run. This repo's own dependencies are sibling Catcrafts repos tracking their default branch on purpose; pin with `commit` when adding anything else, and whenever reproducibility matters.
- Unpinned dependencies `git pull` on every invocation, so builds and tests need network. A missing or private repo surfaces as a clone failure before anything is built.
- `--runner=cmd:<command>` executes the named wrapper around every test binary. Wrappers come from `--runner=` or `CRAFTER_BUILD_RUNNER_*`; do not hardcode hosts, wrappers or credentials in `project.cpp` or tests.
feat(script): scripts in project.cpp, run with `crafter-build run` A project.cpp could describe how a project is built, tested and linted, but not the chores around it: packaging, installing, publishing. Those stayed shell lines in a README or in someone's history. Configuration gains `scripts`, a plain list like `defines`. A Script is a name and a C++ callback, as a lint rule is; there is no script language and no config file. The callback gets a ScriptContext (the project, the project file's directory, the arguments) and returns the exit status. ctx.Run runs a shell command with the terminal attached and returns its status. It is only stored when project.cpp loads, which happens for every verb and for every project that depends on this one. `crafter-build run <name> [args...]` runs one, a bare `run` lists them. Only the scripts of the project crafter-build was pointed at are runnable, never a dependency's. Everything after the script's name is handed to the script untouched. The two branches that do this sit ahead of every other branch of the parse loop, --version and --help included. Below the verbs, `run package clean` would delete build/, and any other word would land in the project's args and so in the variant hash, moving the build to another directory. Options before the name keep their meaning, so project args go there. The driver is its own partition, :Script. It refuses a project with no scripts, a name that is empty or starts with '-' (it could never be typed), a name declared twice (running either would be a guess) and an unknown name, each with exit 1 and without running anything. A script that throws exits 1. A return value outside 0..255 becomes 1: an exit status is eight bits, and a raw wait status such as 256 would otherwise wrap to success. Two helpers in :Platform: - RunCommandAttached runs a command with inherited stdio. Every helper so far captured through popen, which is right for a compile step and useless for a command that runs for half an hour or asks a question. It reads the wait status through DecodeWaitStatus, which RunCommandChecked and RunCommandWithTimeout now share. - ShellQuote is exported. It existed twice, as ShellQuote in :External and ShellQuoteSh in :Test, with the same behaviour; both are gone. A script that puts a path into a command needs it too. Not included: a string shorthand for one-command scripts (a one-line lambda covers it) and a ctx.Build() helper (Build() is already exported). A new partition and new exported symbols: run ./build.sh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:16:00 +02:00
- `crafter-build run <name>` executes whatever the script does, with your permissions. Only the scripts of the project crafter-build was pointed at can be run, never a dependency's; keep it that way.
docs change
2026-10-02 20:35:38 +02:00
- The linter loads `libclang` at runtime by name. `CRAFTER_BUILD_LIBCLANG` pins an exact path and must fail loudly when wrong, never fall back to another library.
fix(ci): sign with a key kept on the runner host The runner is our own machine, so the repo signing key does not need to pass through the forge as a secret. It is generated once under /etc/crafter-flatpak on the host and bind-mounted read-only into the job container, where each run imports it into a throwaway keyring. One less secret to manage, and the private key never leaves the box that uses it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 23:51:23 +02:00
- CI force-pushes the rolling `latest` tag and publishes a prerelease on every push to `master` with the workflow's own token, and signs the Flatpak repo with the key mounted from the runner host. Do not add secrets to the workflow or echo the environment in it.
docs change
2026-10-02 20:35:38 +02:00
- Tests write only under `fs::temp_directory_path()` and the fixture's own `bin/` and `build/`. Keep it that way.
feat(ci): build and publish the Flatpak One job replaces the tarball pipeline. On a pull request it builds the Flatpak with flatpak-builder, which runs the bootstrap, the self-host build and the test suite inside the sandbox, and smoke-tests the result with `crafter-build --version` and the hello-world example. On a push to master it also bundles the build, promotes it into the live repo as the `beta` branch and refreshes the rolling `latest` prerelease. On a tag X.Y.Z it promotes as `stable`, refuses to proceed unless the metainfo's newest release is that version, and publishes a release whose notes are that entry's description. The job runs on the `flatpak` runner label: the privileged forgejo-runner on the Hetzner box, with the Sdk install, the flatpak-builder state and the repo's web root as volumes. The state volume is what keeps a normal run short: flatpak-builder rebuilds the LLVM and WASI modules only when their sources change. The live repo is canonical and never built into; builds are promoted with build-commit-from, then build-update-repo signs and prunes it and generates static deltas. The musl job, the per-march tarballs, the toolchain packages the runner used to install and the PKGBUILD go: the Flatpak is the one artifact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:36:55 +02:00
## CI and publishing
fix(ci): run on a label only the Crafter.Build runner carries Master's last two runs (#368 and #370) failed in the Build step with The state dir (/root/.cache/flatpak-builder) is not on the same filesystem as the target dir (/workspace/Catcrafts/Crafter.Build) while #369, the 2.2.0 tag run on the same commit with the same flatpak-builder, passed. The difference was the runner. Since 14 October the Hetzner box has a second privileged runner, cx33-org-flatpak, which serves every Catcrafts repository under the label `flatpak` so their package jobs can run crafter-build as a Flatpak. It mounts no host volumes, deliberately, since any repository in the organisation can send it jobs. Whenever it was idle first it took this job too, logged our four volumes as "not a valid volume, will be ignored", and left the flatpak-builder state on the container's overlay while the workspace sat on a Docker volume. Even past that check it would have had no build cache, no web root and no signing key. The org runner's label is right for the jobs it was made for, and giving it our volumes would hand the signing key to every repository in the organisation. So the job asks for a label only its own runner carries: `crafter-flatpak`, after the volumes and key directory that runner alone allows. The runner on the host is relabelled to match (/etc/forgejo-runner/config-flatpak.yaml, previous copy kept as config-flatpak.yaml.bak-2026-10-31-label), so until this reaches master a push there with the old `runs-on: flatpak` goes only to the org runner and fails the same way. AGENTS.md now says the label must stay unique and quotes the error a job on the wrong runner stops with, so the next collision is found from the log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 01:25:44 +01:00
`.forgejo/workflows/ci.yaml` runs on the `crafter-flatpak` runner label: a dedicated privileged forgejo-runner on the Hetzner box (`forgejo-runner-flatpak.service`, config `/etc/forgejo-runner/config-flatpak.yaml`), registered to this repository only and the only runner with that label. It is privileged because bubblewrap inside the job container needs it, and separate from the other runners so that privilege is not shared with any other repository. Its config allows four volumes, which the workflow mounts: `crafter-flatpak-system` (the Sdk install at `/var/lib/flatpak`), `crafter-flatpak-builder` (flatpak-builder state and ccache, so the LLVM and WASI modules are rebuilt only when their sources change; a cold run takes about 80 minutes on 8 cores, a warm one a fraction of that), `/var/www/flatpak`, the web root Caddy serves as `https://flatpak.catcrafts.net/`, and `/etc/crafter-flatpak`, which holds the repo signing key (`signing-key.asc`, no passphrase, and `key-id`), generated once on the host and mounted read-only into the job; nothing is stored in the forge. The live repo at `/var/www/flatpak/repo` is canonical; builds are promoted into it with `flatpak build-commit-from`, never built into it.
Keep the label unique to that runner. Plain `flatpak` also belongs to the organisation-wide runner (`forgejo-runner-org-flatpak.service`), which serves every Catcrafts repository and on purpose allows no volumes. Whenever this job lands there, flatpak-builder stops with `The state dir (/root/.cache/flatpak-builder) is not on the same filesystem as the target dir`; the top of the job log shows which runner took it and lists every volume it ignored.
fix(ci): sign with a key kept on the runner host The runner is our own machine, so the repo signing key does not need to pass through the forge as a secret. It is generated once under /etc/crafter-flatpak on the host and bind-mounted read-only into the job container, where each run imports it into a throwaway keyring. One less secret to manage, and the private key never leaves the box that uses it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 23:51:23 +02:00
The key is made once, as root on the runner host, and the step fails with a message pointing here when it is missing:
```bash
install -d -m 700 /etc/crafter-flatpak/gnupg && export GNUPGHOME=/etc/crafter-flatpak/gnupg
gpg --batch --pinentry-mode loopback --passphrase '' --quick-generate-key 'Catcrafts Flatpak Repository <flatpak@catcrafts.net>' ed25519 sign never
KEYID=$(gpg --list-secret-keys --with-colons | awk -F: '/^sec/{print $5; exit}')
gpg --batch --pinentry-mode loopback --passphrase '' --export-secret-keys --armor "$KEYID" > /etc/crafter-flatpak/signing-key.asc
printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id
```
feat(ci): build and publish the Flatpak One job replaces the tarball pipeline. On a pull request it builds the Flatpak with flatpak-builder, which runs the bootstrap, the self-host build and the test suite inside the sandbox, and smoke-tests the result with `crafter-build --version` and the hello-world example. On a push to master it also bundles the build, promotes it into the live repo as the `beta` branch and refreshes the rolling `latest` prerelease. On a tag X.Y.Z it promotes as `stable`, refuses to proceed unless the metainfo's newest release is that version, and publishes a release whose notes are that entry's description. The job runs on the `flatpak` runner label: the privileged forgejo-runner on the Hetzner box, with the Sdk install, the flatpak-builder state and the repo's web root as volumes. The state volume is what keeps a normal run short: flatpak-builder rebuilds the LLVM and WASI modules only when their sources change. The live repo is canonical and never built into; builds are promoted with build-commit-from, then build-update-repo signs and prunes it and generates static deltas. The musl job, the per-march tarballs, the toolchain packages the runner used to install and the PKGBUILD go: the Flatpak is the one artifact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:36:55 +02:00
feat(packaging): `run flatpak` and `run install` for a local Flatpak The local Flatpak build was one long flatpak-builder line in AGENTS.md, with its state wherever the caller happened to stand, and installing the result was not written down anywhere. It is not a rare chore: the development box compiles with the distro's clang and the Flatpak with its bundled LLVM, a major version apart at the moment, so anything toolchain-sensitive has to be tried in the Flatpak too. Both are now scripts in project.cpp, the first users of cfg.scripts. `run flatpak` builds the manifest as it is, from the working tree, and ends with `crafter-build --version` from inside the result. Its state is build/flatpak/ (state/, repo/, build-dir/). That is a fixed name and not a BuildDir() on purpose: the Flatpak is compiled by the toolchain in the sandbox, so it does not depend on the host variant, and a variant-keyed directory would strand the LLVM cache on every host compiler update. The manifest already skips build/ when it copies the sources. flatpak-builder keys its cache on the manifest's path, so the manifest is never rewritten into a copy the way CI stamps the version in, and a local build reports 0.0.0-dev. The only argument accepted is --jobs=<N>: anything that ends the run without a build would leave the smoke test checking the previous one. `run install` installs the last export as branch `local` from a user remote of its own and makes it current, leaving the published branch installed. It does not build first: flatpak-builder never caches a `type: dir` module, so every build reruns build.sh and the suite. remote-add --if-not-exists is followed by remote-modify --url, because the remote may exist and point at another checkout. Both refuse to run from the Flatpak build of crafter-build, which has no flatpak to drive. The smoke test goes through `flatpak build` and not `<builder> --run`, which is what CI uses. flatpak-builder 1.4.9 run as the org.flatpak.Builder app exits 1 when the command it ran succeeded and 0 when it failed (`true` gives 1, `exit 3` gives 0), so the script reported a good build as a failure. Measured on the 56-core box with LLVM and the WASI sysroot cached: 260 seconds, the crafter-build module only, 29 tests passing in the sandbox. A second `run install` after a rebuild moved the installed commit. `clean` now also drops that state, about 2 GB and a cold LLVM build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:29:32 +02:00
To release: add a `<release>` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry.
Local builds are two scripts in `project.cpp`. Run them with the native binary: the Flatpak build of crafter-build has no `flatpak` to drive.
```bash
./bin/crafter-build run flatpak [--jobs=<N>] # build the Flatpak, then print its version from inside the sandbox
./bin/crafter-build run install # install that build as branch `local` and make it current
```
`run flatpak` needs `flatpak-builder` or the `org.flatpak.Builder` app, and the Sdk (`flatpak install flathub org.freedesktop.Sdk//26.08`). It builds the working tree, uncommitted changes included, runs the whole suite inside the sandbox and reports version `0.0.0-dev`. Its state is `build/flatpak/` (`state/`, `repo/`, `build-dir/`). The first run builds LLVM and the WASI sysroot, about 25 minutes on a 56-core machine; later runs rebuild only the crafter-build module, a few minutes, because flatpak-builder never caches a `type: dir` source. The checkout must not be under `/tmp`, which `org.flatpak.Builder` cannot see. Use it for anything toolchain-sensitive: the native loop compiles with the distro's clang and the Flatpak with its bundled LLVM, and the two can be a major version apart.
`run install` does not build; it installs what `run flatpak` last exported. It points the user remote `crafter-build-local` at `build/flatpak/repo`, installs `net.catcrafts.CrafterBuild//local` from it next to whichever published branch is installed, and makes it the branch `flatpak run net.catcrafts.CrafterBuild` starts. `flatpak make-current --user net.catcrafts.CrafterBuild <branch>` switches back. Before `crafter-build clean`, or before deleting the checkout, run `flatpak uninstall --user net.catcrafts.CrafterBuild//local` and `flatpak remote-delete --user crafter-build-local`; otherwise the remote is left pointing at a repo that is gone.
feat(ci): build and publish the Flatpak One job replaces the tarball pipeline. On a pull request it builds the Flatpak with flatpak-builder, which runs the bootstrap, the self-host build and the test suite inside the sandbox, and smoke-tests the result with `crafter-build --version` and the hello-world example. On a push to master it also bundles the build, promotes it into the live repo as the `beta` branch and refreshes the rolling `latest` prerelease. On a tag X.Y.Z it promotes as `stable`, refuses to proceed unless the metainfo's newest release is that version, and publishes a release whose notes are that entry's description. The job runs on the `flatpak` runner label: the privileged forgejo-runner on the Hetzner box, with the Sdk install, the flatpak-builder state and the repo's web root as volumes. The state volume is what keeps a normal run short: flatpak-builder rebuilds the LLVM and WASI modules only when their sources change. The live repo is canonical and never built into; builds are promoted with build-commit-from, then build-update-repo signs and prunes it and generates static deltas. The musl job, the per-march tarballs, the toolchain packages the runner used to install and the PKGBUILD go: the Flatpak is the one artifact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:36:55 +02:00
docs change
2026-10-02 20:35:38 +02:00
## Gotchas
feat(script): scripts in project.cpp, run with `crafter-build run` A project.cpp could describe how a project is built, tested and linted, but not the chores around it: packaging, installing, publishing. Those stayed shell lines in a README or in someone's history. Configuration gains `scripts`, a plain list like `defines`. A Script is a name and a C++ callback, as a lint rule is; there is no script language and no config file. The callback gets a ScriptContext (the project, the project file's directory, the arguments) and returns the exit status. ctx.Run runs a shell command with the terminal attached and returns its status. It is only stored when project.cpp loads, which happens for every verb and for every project that depends on this one. `crafter-build run <name> [args...]` runs one, a bare `run` lists them. Only the scripts of the project crafter-build was pointed at are runnable, never a dependency's. Everything after the script's name is handed to the script untouched. The two branches that do this sit ahead of every other branch of the parse loop, --version and --help included. Below the verbs, `run package clean` would delete build/, and any other word would land in the project's args and so in the variant hash, moving the build to another directory. Options before the name keep their meaning, so project args go there. The driver is its own partition, :Script. It refuses a project with no scripts, a name that is empty or starts with '-' (it could never be typed), a name declared twice (running either would be a guess) and an unknown name, each with exit 1 and without running anything. A script that throws exits 1. A return value outside 0..255 becomes 1: an exit status is eight bits, and a raw wait status such as 256 would otherwise wrap to success. Two helpers in :Platform: - RunCommandAttached runs a command with inherited stdio. Every helper so far captured through popen, which is right for a compile step and useless for a command that runs for half an hour or asks a question. It reads the wait status through DecodeWaitStatus, which RunCommandChecked and RunCommandWithTimeout now share. - ShellQuote is exported. It existed twice, as ShellQuote in :External and ShellQuoteSh in :Test, with the same behaviour; both are gone. A script that puts a path into a command needs it too. Not included: a string shorthand for one-command scripts (a one-line lambda covers it) and a ctx.Build() helper (Build() is already exported). A new partition and new exported symbols: run ./build.sh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 18:16:00 +02:00
- A new partition is listed in four places, in import order: `PARTITIONS` in `build.sh`, `CrafterBuildModules` in `Crafter.Build-Platform.cpp`, both arrays in `project.cpp`, and the umbrella `Crafter.Build.cppm`. Then `./build.sh`.
docs change
2026-10-02 20:35:38 +02:00
- A `#pragma crafter dependency` target is rebuilt on every load on purpose: clang crashes rather than erroring when it imports a BMI whose own imported PCM was rebuilt underneath it, and the depfile check is what catches a regenerated host cache. Do not optimise the rebuild away.
docs: describe IsTopLevelProject for test-only dependencies Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-02 22:02:40 +02:00
- A host-tooling library such as Crafter.LintRules may `import Crafter.Build;` without declaring it as a dependency, because `LoadProject` fills the cache it resolves from. Listing Crafter.Build in its `cfg.dependencies` would clone and build all of crafter-build for every consumer. Its test does link crafter-build, so that test is wired behind `IsTopLevelProject()`: without the guard, crafter-build loading the library as a dependency would load its own `project.cpp` again through the test's `GitProject` and hit the cycle guard.
docs change
2026-10-02 20:35:38 +02:00
- `crafter-build clean` leaves a pragma dependency's build alone; it lives with the clone in the cache.
- The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop.
- A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone.
docs: describe how a sysroot build finds its std module source The README said a cross build precompiles the sysroot's own std.cppm, which was never the whole story for sysroots that do not ship one. It now states the order (the sysroot's, the toolchain's for the same major release, the sources crafter-build carries for other releases), what the failure looks like when nothing matches and how to resolve it, and lists the new directory in the install layout. AGENTS.md records the rule for the next LLVM bump: the release the bundled toolchain leaves behind stays available as a module-source module, because stable distros remain on it long after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 17:01:21 +02:00
- A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under `share/crafter-build/libc++-modules/<major>/`, one `libcxx-<major>-modules` manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under `share/crafter-build/` to test against a foreign sysroot.
docs: wasm targets and their C++ exceptions The README described the mingw target and said nothing about wasm. It now says where the WASI sysroot comes from, that C++ exceptions work as on the other targets and what that takes (-fwasm-exceptions in the standardized form, the sysroot's exception-enabled libc++, -lunwind added to the link), which engines run the result, and that an exception escaping main ends the program without calling std::terminate. AGENTS.md says where the wasm flags live, that VariantId and the std PCM directory hash them, and that the wasm tests skip where the distro's sysroot lacks the eh/ libraries, so they run in the Flatpak. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 05:55:00 +01:00
- The wasm compile flags live in one place, `WasmBaselineFlags` in `Crafter.Build-Clang.cppm`: the std module and every unit have to agree on them, and `VariantId()` and the std PCM directory hash them, so a change rebuilds instead of mixing old and new BMIs. They enable C++ exceptions, so the link takes the exception-enabled libc++ and libunwind from the sysroot's `lib/<triple>/eh/` (wasi-sdk 33 and later). The Flatpak's sysroot has them and a distro's may not; where they are missing the wasm tests skip, so run those through the Flatpak. A browser page checks the engine with the `WasmBaselineProbes` named in `variants.json`.
docs: wasm exceptions as the review left them The README called an exception escaping main a trap. It is an uncaught WebAssembly exception, and nothing prints its type or what(); the README says so now. It also lists what a wasm build refuses and why, says that setjmp needs -lsetjmp and that a longjmp across a noexcept function ends the program, that a wasmtime older than 47 needs -W exceptions=y, and that an old wasm-opt on PATH breaks release links. It documents exit and park in a page from EnableWasiBrowserRuntime, with the declaration of the park import. The wasi example still sent Arch users to wasi-libc++, which has no exception-enabled libraries. It asks for a wasi-sdk 33 sysroot or the Flatpak now, and names the browsers a page needs. AGENTS.md notes that the runtime test needs node, so it runs in the native loop and not in CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 06:10:41 +01:00
- `WasiBrowserRuntimeExit` runs `wasi-runtime/runtime.js` under node and skips where there is none. The Flatpak has no node, so CI never runs it: after touching `runtime.js`, run the suite in the native loop with node on `PATH`.
docs: describe the output directory name and what a compiler update does The README still gave the output path as bin/<name>-<target>-<march>/, which has not been its shape since the variant hash was introduced. It now shows the real one and says what the hash stands for, including that a compiler update starts a new directory and `clean` removes the old. AGENTS.md lists the toolchain among VariantId()'s inputs and notes that a glob over variant directories matches two after an update. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-03 20:09:22 +02:00
- `VariantId()` names the output directory, so CI globs `bin/crafter.build-exe-<target>-<march>-*/` rather than predicting the hash. The hash includes the compiler, so after a toolchain update the previous variant's directories stay in `bin/` and `build/` until `clean`, and such a glob matches both.
docs: dependency notes as the review left them The README's section: notes are labelled by name, the file is written in place, a project.cpp that fails to load gets no update, `test --list` writes nothing, the section follows what the build loaded, and tests' dependencies don't count. The AGENTS.md gotcha: an unpinned dependency can change this file's section on any build here, and the dirty tree then rebuilds crafter.build-lib under a new version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 05:02:36 +01:00
- The self-host build and `test` keep the dependency notes section of this very file. None of Crafter.Math, Crafter.Asset and Crafter.LintRules ships an `AGENTS.consumer.md` yet. Once one does, a build here can change AGENTS.md whenever that repository changes its notes, since all three are unpinned. The tree is then dirty, so the next build's `git describe --dirty` version differs, which is a new `crafter.build-lib` variant and a full rebuild. Commit the section to settle it.
Reference in a new issue Copy permalink