Compare commits

...
Author SHA1 Message Date
00f07dae88
docs: push reviewed ranges straight to master
All checks were successful
Flatpak / flatpak (push) Successful in 30m58s
AGENTS.template.md had every change go through a pull request against
master. The owner has made pushing straight to master the default.

The commit and push guidelines say so: an approved range is pushed to
master as a fast-forward, with no merge commit, and a pull request is
opened only when the owner asks for one; when master has moved since
the freeze, the range is rebased, frozen and reviewed again. What
follows from that: the README changes in the same push; without a pull
request CI does not run on a range before it reaches master unless
someone starts it by hand, so the review's gate0 is the check before the
push; and the freeze exception for a merge commit applies only to a pull
request. Design rule 8, the testing note and the release steps lose
their pull-request wording, and the testing note no longer speaks of a
commented-out lint step: ci.yaml has no lint step at all.
2026-11-01 23:21:35 +01:00
03951ff458
docs: update the first local Flatpak build's time and size
AGENTS.template.md and project.cpp gave the first run flatpak as about
25 minutes on a 56-core machine and its state as about 2 GB, figures
written on 2026-10-06, before lldb and the Vulkan and SPIR-V modules
were in the manifest. A first run of this commit's manifest, in a fresh
worktree, took 30 min 36 s with 56 CPU threads, 14 min of it for LLVM
with lldb, and left build/flatpak at 4.1 GB, 3.3 GB of it state/.

The figures become those, and the line on later runs says that they
rebuild only the crafter-build module while the manifest is unchanged.
2026-11-01 22:28:08 +01:00
40dcb18a1e
feat(flatpak): ship the SPIRV-Tools programs
Crafter.Graphics' PushConstantRewrite and UIFusedShader tests require
tool:spirv-val, and the Sdk has no spirv-val, so inside the Flatpak they
are skipped. The spirv-tools module the validation layer needs built
only the libraries, and removed them afterwards.

It now builds the programs as well, spirv-val, spirv-opt, spirv-dis and
the rest, which link SPIRV-Tools statically, and ships them. The
headers, the static libraries, the CMake and pkg-config files and
libSPIRV-Tools-shared.so, which nothing here links, are still removed,
and a post-install step runs spirv-val --version.
2026-11-01 22:28:08 +01:00
c8517a41cf
docs(readme): --die-with-parent in the install alias
flatpak run becomes bubblewrap, and without --die-with-parent the
sandbox outlives it. With crafter-build started through the alias in an
interactive bash or zsh on a terminal, SIGKILL to its flatpak run
process alone left a -r program running and a build writing into bin/
and build/. With the flag, both ended with it.

bubblewrap kills the sandbox when flatpak run or its parent dies, and
for crafter-build typed at the prompt, in the foreground or with &, that
parent is the shell: with the flag, the sandbox also ended when the
shell was SIGKILLed, or exited after a disown. In a (crafter-build -r &)
subshell, flatpak run outlives the subshell and is adopted by another
process, so its sandbox outlived the shell with the flag as well. Ctrl-C
ended the program either way.

The README's alias gets the flag, and the paragraph after it says what
that changes: a program started through the alias, as crafter-build ...
or crafter-build ... & at a bash or zsh prompt, ends with its shell,
even after a disown.
2026-11-01 22:28:08 +01:00
a32b8c898a
docs: debugging inside the Flatpak
The README gets a Debugging section: the debugging tools the sandbox
carries, that a program to debug needs cfg.debug (which --debug sets
where the project calls ApplyStandardArgs), how to start the tools with
--command, why a debugger in one flatpak run cannot see a program that
another one started (each has process IDs of its own), and flatpak
enter as the way into a running program's sandbox, with the ptrace
opt-in that takes where ptrace_scope is 1. It warns off lldb's
gdb-remote from a second sandbox: when a process in lldb's own sandbox
has the remote program's process ID, lldb takes that process's
executable for the program's. And it shows which variables turn the
layer and GPU-AV on from the environment, and that VK_LAYER_PATH and
VK_ADD_LAYER_PATH do not reach the sandbox.

AGENTS.template.md gets the module layout and the rule that the layer
follows the Sdk's Vulkan loader. Its CI section says that a change to a
module's sources or build-commands builds it again and every module
after it, so that a new version of swig or libedit, which come before
LLVM, now rebuilds it too, and loses the cold-run time it gave, written
on 2026-10-03, before any of these modules were in the manifest.
2026-11-01 22:28:08 +01:00
16db37bbe6
feat(flatpak): ship lldb and the Khronos validation layers
Programs that -r and the test runner start run inside the Flatpak's
sandbox, and the Sdk it runs on has gdb but neither lldb nor the Khronos
validation layer: in the sandbox of beta 5.0.0, a vkCreateInstance that
asks for VK_LAYER_KHRONOS_validation returns VK_ERROR_LAYER_NOT_PRESENT.
flatpak run clears VK_LAYER_PATH and VK_ADD_LAYER_PATH, so the layer the
host has cannot stand in.

lldb becomes a project of the llvm module and brings lldb-server,
lldb-dap and lldb-mcp. Its Python, libedit, curses, libxml2 and lzma
support are set ON instead of being left to detection, so a dependency
that goes missing fails the configure step rather than shipping an lldb
without it. The Sdk has neither SWIG nor libedit, so both become modules
before llvm, SWIG as build-only.

The layer is Vulkan-ValidationLayers vulkan-sdk-1.4.357.0, the Vulkan
SDK release of the Sdk's loader (libvulkan.so.1.4.357), built from
Vulkan-Headers, Vulkan-Utility-Libraries, SPIRV-Headers and SPIRV-Tools
at the revisions that release's scripts/known_good.json names. Those
four are built into /app at those revisions and removed afterwards. All
five come after the LLVM and WASI modules, so that bumping them reuses
the cached LLVM and WASI builds. UPDATE_DEPS is turned off: it defaults
to ON, which has update_deps.py clone the dependencies, and of all the
modules only crafter-build's build-args share the network.
2026-11-01 19:20:22 +01:00
4 changed files with 198 additions and 19 deletions

View file

@ -51,7 +51,7 @@ Three things to know about this loop:
- Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added. A changed default member initialiser has no symptom at all: `project.cpp` compiles against the old one and quietly gets the old default, as `project.cpp`'s sibling loads do with a harness that predates `GitProjectSpec::args = ProjectArgs()`. - Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added. A changed default member initialiser has no symptom at all: `project.cpp` compiles against the old one and quietly gets the old default, as `project.cpp`'s sibling loads do with a harness that predates `GitProjectSpec::args = ProjectArgs()`.
- A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol. - A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol.
`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. That takes `bin/crafter-build` itself with it, and `build/flatpak`, the state of the local Flatpak build (see "CI and publishing"): about 2 GB, and the next `run flatpak` rebuilds LLVM. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`. `./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. That takes `bin/crafter-build` itself with it, and `build/flatpak`, the state of the local Flatpak build (see "CI and publishing"): about 4 GB after a first run, and the next `run flatpak` rebuilds LLVM. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`.
## Build, test and style commands ## Build, test and style commands
@ -69,7 +69,7 @@ Three things to know about this loop:
./bin/crafter-build clean ./bin/crafter-build clean
``` ```
Before you call a change done, all three must exit 0: `format` (then re-read your diff), `lint`, `test`. CI runs `./build.sh` and `crafter-build test` on every PR; its lint step is currently commented out, so lint is enforced only by you. Failed, crashed and timed-out tests leave their output in `build/test-logs/<name>.log`. Before you call a change done, all three must exit 0: `format` (then re-read your diff), `lint`, `test`. CI runs `./build.sh` and `crafter-build test` on every push to `master` and on pull requests, and has no lint step, so lint is enforced only by you. Failed, crashed and timed-out tests leave their output in `build/test-logs/<name>.log`.
## Testing instructions ## Testing instructions
@ -132,18 +132,18 @@ These were decided deliberately. Do not undo them without the maintainer's say-s
5. **Convention over configuration.** Lead with the zero-config path. Env var, then CLI flag, then full `project.cpp` are the escalation, not the entry point. 5. **Convention over configuration.** Lead with the zero-config path. Env var, then CLI flag, then full `project.cpp` are the escalation, not the entry point.
6. **One skip mechanism** (runner probe plus exit 77) and **one `TestRunner` shape** (template fields with `{bin}`, `{bundle}` and similar placeholders; `Cmd(...)` covers any wrapper). Do not add a special-case runner class or a hardcoded "skip on this OS" branch. 6. **One skip mechanism** (runner probe plus exit 77) and **one `TestRunner` shape** (template fields with `{bin}`, `{bundle}` and similar placeholders; `Cmd(...)` covers any wrapper). Do not add a special-case runner class or a hardcoded "skip on this OS" branch.
7. **The house-style rules stay in their own repo.** They are not library API of crafter-build and are not forced on downstream projects. 7. **The house-style rules stay in their own repo.** They are not library API of crafter-build and are not forced on downstream projects.
8. **Fix warts inline; do not stack workarounds.** The previous version of this tool died of accumulated hacks. If a change needs a shortcut, say so in the PR rather than hiding it. 8. **Fix warts inline; do not stack workarounds.** The previous version of this tool died of accumulated hacks. If a change needs a shortcut, say so in its commit message rather than hiding it.
9. **`project.cpp` keeps the library/executable split.** `crafter.build-lib` owns the sources, dependencies and glslang; `crafter.build-exe` is `main.cpp` plus link flags. 9. **`project.cpp` keeps the library/executable split.** `crafter.build-lib` owns the sources, dependencies and glslang; `crafter.build-exe` is `main.cpp` plus link flags.
10. **Host-tooling builds** (the target of a `#pragma crafter dependency`) get `-fPIC` and the host triple scoped to that build. Keep it scoped. 10. **Host-tooling builds** (the target of a `#pragma crafter dependency`) get `-fPIC` and the host triple scoped to that build. Keep it scoped.
## Commit and PR guidelines ## Commit and push guidelines
- Branch from `master` and open the PR against `master` on the project's Forgejo at `forgejo.catcrafts.net/Catcrafts/Crafter.Build`. Issue work uses `claude/issue-<n>` branch names. Pushes go over SSH; HTTPS push is disabled on the server. - Branch from `master`; issue work uses `claude/issue-<n>` branch names. Once the owner approves its frozen hash, a range goes straight to `master` on the project's Forgejo at `forgejo.catcrafts.net/Catcrafts/Crafter.Build`, as a fast-forward push with no merge commit and no pull request; open a pull request against `master` only when the owner asks for one. If `master` has moved since the freeze, rebase onto it, then freeze and review the rebased range again. Pushes go over SSH; HTTPS push is disabled on the server.
- Subject line: `type(scope): imperative summary`, with `feat`, `fix`, `refactor`, `docs` or `test`; the scope (`lint`, `incremental`, ...) is optional. Body: what was wrong, why, and why the fix has this shape, wrapped at about 75 columns. Read `git log` for the voice. - Subject line: `type(scope): imperative summary`, with `feat`, `fix`, `refactor`, `docs` or `test`; the scope (`lint`, `incremental`, ...) is optional. Body: what was wrong, why, and why the fix has this shape, wrapped at about 75 columns. Read `git log` for the voice.
- Separate commits for the fix, its test and the README change when each stands on its own. - Separate commits for the fix, its test and the README change when each stands on its own.
- Update the README in the same PR when user-visible behaviour changes. - Update the README in the same push when user-visible behaviour changes.
- `format`, `lint` and `test` green before pushing. CI builds the Flatpak and runs the suite inside it on PRs, publishes the `beta` branch of the Catcrafts repo on every push to `master`, and the `stable` branch plus a release on an `X.Y.Z` tag. - `format`, `lint` and `test` green before pushing. CI builds the Flatpak and runs the suite inside it on every push to `master` and on pull requests; a push to `master` then publishes the `beta` branch of the Catcrafts repo, and an `X.Y.Z` tag the `stable` branch plus a release. Without a pull request, CI does not run on a range before it reaches `master` unless someone starts it by hand (`workflow_dispatch`), so the review's gate0, the commands in `.review-gates/config.json`, is the check before the push.
- Every push of new commits goes through Crafter.Review's pre-push review first, and goes out only once the owner approves the frozen hash. Run `flatpak run net.catcrafts.CrafterReview init` once in a fresh checkout or worktree: it writes the `/prepush` skill that drives the review, and the skill stays untracked. `freeze` refuses merge commits, so the merge of a reviewed branch into `master` is pushed without a freeze of its own. - Every push of new commits goes through Crafter.Review's pre-push review first, and goes out only once the owner approves the frozen hash. Run `flatpak run net.catcrafts.CrafterReview init` once in a fresh checkout or worktree: it writes the `/prepush` skill that drives the review, and the skill stays untracked. `freeze` refuses merge commits: a range reaches `master` as a fast-forward, and when the owner asks for a pull request instead, the merge of the reviewed branch into `master` is pushed without a freeze of its own.
- Every commit of a pushed range is GPG-signed, has `jorijnvdgraaf@catcrafts.net` as its author and committer, and carries no AI attribution trailer in its message, neither `Co-Authored-By:` nor `Assisted-by:`. `freeze` refuses a range with a commit that breaks any of these (`requireSigned`, `authors` and `aiTrailers` in `.review-gates/config.json`). - Every commit of a pushed range is GPG-signed, has `jorijnvdgraaf@catcrafts.net` as its author and committer, and carries no AI attribution trailer in its message, neither `Co-Authored-By:` nor `Assisted-by:`. `freeze` refuses a range with a commit that breaks any of these (`requireSigned`, `authors` and `aiTrailers` in `.review-gates/config.json`).
- Never commit `bin/`, `build/`, `share/crafter-build/`, `lib/*.a`, `AGENTS.md` or fixture build outputs (all gitignored), and never edit the vendored files in `lib/`. - Never commit `bin/`, `build/`, `share/crafter-build/`, `lib/*.a`, `AGENTS.md` or fixture build outputs (all gitignored), and never edit the vendored files in `lib/`.
- Keep `CRAFTER_BUILD_MARCH=x86-64-v2` in the manifest for the shipped binary: it has to run on any x86_64 machine. User projects still default to `native`. - Keep `CRAFTER_BUILD_MARCH=x86-64-v2` in the manifest for the shipped binary: it has to run on any x86_64 machine. User projects still default to `native`.
@ -160,7 +160,7 @@ These were decided deliberately. Do not undo them without the maintainer's say-s
## CI and publishing ## CI and publishing
`.forgejo/workflows/ci.yaml` runs on the `crafter-flatpak` runner label: a dedicated privileged forgejo-runner on the Hetzner box (`forgejo-runner-flatpak.service`, config `/etc/forgejo-runner/config-flatpak.yaml`), registered to this repository only and the only runner with that label. It is privileged because bubblewrap inside the job container needs it, and separate from the other runners so that privilege is not shared with any other repository. Its config allows four volumes, which the workflow mounts: `crafter-flatpak-system` (the Sdk install at `/var/lib/flatpak`), `crafter-flatpak-builder` (flatpak-builder state and ccache, so the LLVM and WASI modules are rebuilt only when their sources change; a cold run takes about 80 minutes on 8 cores, a warm one a fraction of that), `/var/www/flatpak`, the web root Caddy serves as `https://flatpak.catcrafts.net/`, and `/etc/crafter-flatpak`, which holds the repo signing key (`signing-key.asc`, no passphrase, and `key-id`), generated once on the host and mounted read-only into the job; nothing is stored in the forge. The live repo at `/var/www/flatpak/repo` is canonical; builds are promoted into it with `flatpak build-commit-from`, never built into it. `.forgejo/workflows/ci.yaml` runs on the `crafter-flatpak` runner label: a dedicated privileged forgejo-runner on the Hetzner box (`forgejo-runner-flatpak.service`, config `/etc/forgejo-runner/config-flatpak.yaml`), registered to this repository only and the only runner with that label. It is privileged because bubblewrap inside the job container needs it, and separate from the other runners so that privilege is not shared with any other repository. Its config allows four volumes, which the workflow mounts: `crafter-flatpak-system` (the Sdk install at `/var/lib/flatpak`), `crafter-flatpak-builder` (flatpak-builder state and ccache, which later runs reuse: flatpak-builder builds a module again when its sources or its `build-commands` change, and every module after it with it, so a new version of `swig` or `libedit`, which come before LLVM, also rebuilds the LLVM and WASI modules), `/var/www/flatpak`, the web root Caddy serves as `https://flatpak.catcrafts.net/`, and `/etc/crafter-flatpak`, which holds the repo signing key (`signing-key.asc`, no passphrase, and `key-id`), generated once on the host and mounted read-only into the job; nothing is stored in the forge. The live repo at `/var/www/flatpak/repo` is canonical; builds are promoted into it with `flatpak build-commit-from`, never built into it.
Keep the label unique to that runner. Plain `flatpak` also belongs to the organisation-wide runner (`forgejo-runner-org-flatpak.service`), which serves every Catcrafts repository and on purpose allows no volumes. Whenever this job lands there, flatpak-builder stops with `The state dir (/root/.cache/flatpak-builder) is not on the same filesystem as the target dir`; the top of the job log shows which runner took it and lists every volume it ignored. Keep the label unique to that runner. Plain `flatpak` also belongs to the organisation-wide runner (`forgejo-runner-org-flatpak.service`), which serves every Catcrafts repository and on purpose allows no volumes. Whenever this job lands there, flatpak-builder stops with `The state dir (/root/.cache/flatpak-builder) is not on the same filesystem as the target dir`; the top of the job log shows which runner took it and lists every volume it ignored.
@ -174,7 +174,7 @@ gpg --batch --pinentry-mode loopback --passphrase '' --export-secret-keys --armo
printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id
``` ```
To release: add a `<release>` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry. To release: add a `<release>` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), push it to `master`, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry.
Local builds are two scripts in `project.cpp`. Run them with the native binary: the Flatpak build of crafter-build has no `flatpak` to drive. Local builds are two scripts in `project.cpp`. Run them with the native binary: the Flatpak build of crafter-build has no `flatpak` to drive.
```bash ```bash
@ -182,7 +182,7 @@ Local builds are two scripts in `project.cpp`. Run them with the native binary:
./bin/crafter-build run install # install that build as branch `local` and make it current ./bin/crafter-build run install # install that build as branch `local` and make it current
``` ```
`run flatpak` needs `flatpak-builder` or the `org.flatpak.Builder` app, and the Sdk (`flatpak install flathub org.freedesktop.Sdk//26.08`). It builds the working tree, uncommitted changes included, runs the whole suite inside the sandbox and reports version `0.0.0-dev`. Its state is `build/flatpak/` (`state/`, `repo/`, `build-dir/`). The first run builds LLVM and the WASI sysroot, about 25 minutes on a 56-core machine; later runs rebuild only the crafter-build module, a few minutes, because flatpak-builder never caches a `type: dir` source. The checkout must not be under `/tmp`, which `org.flatpak.Builder` cannot see. Use it for anything toolchain-sensitive: the native loop compiles with the distro's clang and the Flatpak with its bundled LLVM, and the two can be a major version apart. `run flatpak` needs `flatpak-builder` or the `org.flatpak.Builder` app, and the Sdk (`flatpak install flathub org.freedesktop.Sdk//26.08`). It builds the working tree, uncommitted changes included, runs the whole suite inside the sandbox and reports version `0.0.0-dev`. Its state is `build/flatpak/` (`state/`, `repo/`, `build-dir/`). The first run builds every module, LLVM with lldb, the WASI sysroot and the Vulkan validation layer among them, in about 30 minutes with 56 CPU threads; later runs rebuild only the crafter-build module while the manifest is unchanged, a few minutes, because flatpak-builder never caches a `type: dir` source. The checkout must not be under `/tmp`, which `org.flatpak.Builder` cannot see. Use it for anything toolchain-sensitive: the native loop compiles with the distro's clang and the Flatpak with its bundled LLVM, and the two can be a major version apart.
`run install` does not build; it installs what `run flatpak` last exported. It points the user remote `crafter-build-local` at `build/flatpak/repo`, installs `net.catcrafts.CrafterBuild//local` from it next to whichever published branch is installed, and makes it the branch `flatpak run net.catcrafts.CrafterBuild` starts. `flatpak make-current --user net.catcrafts.CrafterBuild <branch>` switches back. Before `crafter-build clean`, or before deleting the checkout, run `flatpak uninstall --user net.catcrafts.CrafterBuild//local` and `flatpak remote-delete --user crafter-build-local`; otherwise the remote is left pointing at a repo that is gone. `run install` does not build; it installs what `run flatpak` last exported. It points the user remote `crafter-build-local` at `build/flatpak/repo`, installs `net.catcrafts.CrafterBuild//local` from it next to whichever published branch is installed, and makes it the branch `flatpak run net.catcrafts.CrafterBuild` starts. `flatpak make-current --user net.catcrafts.CrafterBuild <branch>` switches back. Before `crafter-build clean`, or before deleting the checkout, run `flatpak uninstall --user net.catcrafts.CrafterBuild//local` and `flatpak remote-delete --user crafter-build-local`; otherwise the remote is left pointing at a repo that is gone.
@ -197,6 +197,7 @@ Local builds are two scripts in `project.cpp`. Run them with the native binary:
- The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop. - The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop.
- A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone. - A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone.
- A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under `share/crafter-build/libc++-modules/<major>/`, one `libcxx-<major>-modules` manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under `share/crafter-build/` to test against a foreign sysroot. - A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under `share/crafter-build/libc++-modules/<major>/`, one `libcxx-<major>-modules` manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under `share/crafter-build/` to test against a foreign sysroot.
- The Flatpak carries the debugging tools for what it builds, because those programs run in its sandbox: lldb is a project of the `llvm` module, with `LLDB_ENABLE_PYTHON`, `LIBEDIT`, `CURSES`, `LIBXML2` and `LZMA` set ON so that a missing dependency fails the configure step (SWIG through Python, which needs it), and the `swig` and `libedit` modules come before it. The Khronos validation layer is the `vulkan-validation-layers` module plus four modules before it, build-only except for SPIRV-Tools' programs (`spirv-val` and the rest), all after the LLVM and WASI modules, so that bumping them reuses the cached LLVM and WASI builds. The layer follows the Sdk's Vulkan loader: when `runtime-version` moves to a runtime with a newer loader, move the layer to the Vulkan SDK release of that loader and take the other four revisions from that release's `scripts/known_good.json`.
- The wasm compile flags live in one place, `WasmBaselineFlags` in `Crafter.Build-Clang.cppm`: the std module and every unit have to agree on them, and `VariantId()` and the std PCM directory hash them, so a change rebuilds instead of mixing old and new BMIs. They enable C++ exceptions, so the link takes the exception-enabled libc++ and libunwind from the sysroot's `lib/<triple>/eh/` (wasi-sdk 33 and later). The Flatpak's sysroot has them and a distro's may not; where they are missing the wasm tests skip, so run those through the Flatpak. A browser page checks the engine with the `WasmBaselineProbes` named in `variants.json`. - The wasm compile flags live in one place, `WasmBaselineFlags` in `Crafter.Build-Clang.cppm`: the std module and every unit have to agree on them, and `VariantId()` and the std PCM directory hash them, so a change rebuilds instead of mixing old and new BMIs. They enable C++ exceptions, so the link takes the exception-enabled libc++ and libunwind from the sysroot's `lib/<triple>/eh/` (wasi-sdk 33 and later). The Flatpak's sysroot has them and a distro's may not; where they are missing the wasm tests skip, so run those through the Flatpak. A browser page checks the engine with the `WasmBaselineProbes` named in `variants.json`.
- `WasiBrowserRuntimeExit` runs `wasi-runtime/runtime.js` under node and skips where there is none. The Flatpak has no node, so CI never runs it: after touching `runtime.js`, run the suite in the native loop with node on `PATH`. - `WasiBrowserRuntimeExit` runs `wasi-runtime/runtime.js` under node and skips where there is none. The Flatpak has no node, so CI never runs it: after touching `runtime.js`, run the suite in the native loop with node on `PATH`.
- `VariantId()` names the output directory, so CI globs `bin/crafter.build-exe-<target>-<march>-*/` rather than predicting the hash. The hash includes the compiler, so after a toolchain update the previous variant's directories stay in `bin/` and `build/` until `clean`, and such a glob matches both. - `VariantId()` names the output directory, so CI globs `bin/crafter.build-exe-<target>-<march>-*/` rather than predicting the hash. The hash includes the compiler, so after a toolchain update the previous variant's directories stay in `bin/` and `build/` until `clean`, and such a glob matches both.

View file

@ -4,14 +4,16 @@ A C++26 modules build system. Project descriptions are written in C++ with no JS
## Install ## Install
crafter-build ships as a Flatpak that carries its own clang, lld, libc++ and WASI sysroot, so every machine builds with the same compiler: crafter-build ships as a Flatpak that carries its own clang, lld, libc++ and WASI sysroot, so every machine builds with the same compiler, and lldb and the Vulkan validation layers to debug the result (see [Debugging](#debugging)):
```bash ```bash
flatpak remote-add --if-not-exists catcrafts https://flatpak.catcrafts.net/catcrafts.flatpakrepo flatpak remote-add --if-not-exists catcrafts https://flatpak.catcrafts.net/catcrafts.flatpakrepo
flatpak install catcrafts net.catcrafts.CrafterBuild flatpak install catcrafts net.catcrafts.CrafterBuild
alias crafter-build='flatpak run net.catcrafts.CrafterBuild' alias crafter-build='flatpak run --die-with-parent net.catcrafts.CrafterBuild'
``` ```
`--die-with-parent` ends the sandbox with the `flatpak run` that started it. Without the flag, killing only the `flatpak run` process leaves the sandbox running: a build carries on writing into `bin/` and `build/`, and a program started with `-r` keeps running. With it, bubblewrap kills everything in the sandbox with SIGKILL when the `flatpak run` process dies or when that process's parent does. For `crafter-build …` or `crafter-build … &` typed at a bash or zsh prompt, that parent is the shell, so the sandbox ends with the shell, even after a `disown`. Ctrl-C reaches the program either way.
Pre-release builds of `master` are the `beta` branch (`flatpak install catcrafts net.catcrafts.CrafterBuild//beta`), and every release also attaches a single-file bundle on the Forgejo releases page (`flatpak install --user crafter-build-<version>.flatpak`). The sandbox sees the whole host filesystem, so projects can live anywhere. Binaries built for the host link the bundled `libc++.so.1` through an rpath into the Flatpak: they run inside it (`-r`, the test runner), and are not meant to be copied out onto the host. Pre-release builds of `master` are the `beta` branch (`flatpak install catcrafts net.catcrafts.CrafterBuild//beta`), and every release also attaches a single-file bundle on the Forgejo releases page (`flatpak install --user crafter-build-<version>.flatpak`). The sandbox sees the whole host filesystem, so projects can live anywhere. Binaries built for the host link the bundled `libc++.so.1` through an rpath into the Flatpak: they run inside it (`-r`, the test runner), and are not meant to be copied out onto the host.
## Writing a project.cpp ## Writing a project.cpp
@ -364,6 +366,32 @@ Things to know:
- Inside the Flatpak, commands run in its sandbox: the host's files are visible, but the tools on `PATH` are the ones the Flatpak ships (`git`, `cmake`, `python3`, `tar`, ...), not everything installed on the host. - Inside the Flatpak, commands run in its sandbox: the host's files are visible, but the tools on `PATH` are the ones the Flatpak ships (`git`, `cmake`, `python3`, `tar`, ...), not everything installed on the host.
- A return value outside 0 to 255 is not an exit status and becomes 1. So does a script that throws, with the exception's message printed. - A return value outside 0 to 255 is not an exit status and becomes 1. So does a script that throws, with the exception's message printed.
## Debugging
Programs that `-r` and the test runner start run inside the Flatpak's sandbox, and so do the tools to debug them: lldb, with `lldb-server`, `lldb-dap` and `lldb-mcp`, from the same LLVM build as the compiler; `gdb`, `gdbserver`, `valgrind` and `strace` from the Freedesktop Sdk; the Khronos Vulkan validation layer; and the SPIRV-Tools programs (`spirv-val`, `spirv-opt`, `spirv-dis`, ...). A default build is optimised (`-O3`, and ThinLTO except for wasm) and carries no debug information: set `cfg.debug = true` for a program you want to debug, which compiles its C++ units with `-g` and links it without LTO; its C and CUDA sources (`cfg.cFiles`, `cfg.cuda`) stay at `-O3` without debug information. A `project.cpp` that calls `ApplyStandardArgs` sets it from the `--debug` argument, as in `crafter-build --debug -r`. Start the tools with `--command`:
```bash
flatpak run --die-with-parent --command=lldb net.catcrafts.CrafterBuild ./bin/myapp-<variant>/myapp
flatpak run --die-with-parent --command=valgrind net.catcrafts.CrafterBuild ./bin/myapp-<variant>/myapp
```
Every `flatpak run` is a sandbox of its own, with process IDs of its own, so a debugger in one cannot see a program that another one, such as `crafter-build -r`, started. Either start the program under the debugger, or run the debugger inside that program's sandbox with `flatpak enter`, giving the instance `flatpak ps` lists for net.catcrafts.CrafterBuild and the debugger's full path:
```bash
flatpak ps --columns=instance,application
flatpak enter <instance> /app/bin/lldb -n myapp
```
Where `/proc/sys/kernel/yama/ptrace_scope` is 1, a debugger in the sandbox, where processes hold no capabilities, can attach to a program that is not its descendant only when the program allowed it with `prctl(PR_SET_PTRACER, …)`: with the debugger's process ID, or with `PR_SET_PTRACER_ANY` for any process. Otherwise the attach fails. A program started under the debugger is its descendant. Prefer `flatpak enter` to `lldb-server gdbserver` in one sandbox and lldb's `gdb-remote` from another: when a process in lldb's own sandbox has the remote program's process ID, lldb takes that process's executable for the program's: symbols come out wrong, and a breakpoint in the program's source finds no location.
The validation layer is enabled the usual ways, by the program itself or from the environment: `flatpak run` passes `VK_INSTANCE_LAYERS`, `VK_LOADER_LAYERS_ENABLE` and the layer's `VK_KHRONOS_VALIDATION_*` settings into the sandbox. `VK_KHRONOS_VALIDATION_GPUAV_ENABLE=1` adds GPU-assisted validation (GPU-AV):
```bash
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation VK_KHRONOS_VALIDATION_GPUAV_ENABLE=1 crafter-build -r
```
The layer is the Vulkan SDK release of the runtime's own Vulkan loader. `VK_LAYER_PATH` and `VK_ADD_LAYER_PATH` are cleared on the way into the sandbox, so a layer installed on the host is out of reach.
## Examples ## Examples
[examples/](examples/) contains progressively larger self-contained projects: [examples/](examples/) contains progressively larger self-contained projects:

View file

@ -5,9 +5,15 @@
# Platform: a build tool needs glibc's headers and crt objects, gcc's # Platform: a build tool needs glibc's headers and crt objects, gcc's
# libgcc_s, cmake, ninja, make, git and python at run time. Everything # libgcc_s, cmake, ninja, make, git and python at run time. Everything
# compiler-shaped is ours and lives under /app: clang, lld, libclang, libc++, # compiler-shaped is ours and lives under /app: clang, lld, libclang, libc++,
# compiler-rt (native and wasm32) and the WASI sysroot, all built from one # compiler-rt (native and wasm32), lldb and the WASI sysroot, all built from
# LLVM source tree so the modules crafter-build builds and the compiler that # one LLVM source tree so the modules crafter-build builds and the compiler
# consumes them can never drift apart. # that consumes them can never drift apart.
#
# Programs that `-r` and the test runner start run in this sandbox, so the
# tools to debug them live here too: lldb (above), the Sdk's gdb, valgrind
# and strace, the Khronos Vulkan validation layers, GPU-assisted validation
# included, which neither the Sdk nor its GL extensions carry, and the
# SPIRV-Tools programs such as spirv-val.
app-id: net.catcrafts.CrafterBuild app-id: net.catcrafts.CrafterBuild
runtime: org.freedesktop.Sdk runtime: org.freedesktop.Sdk
runtime-version: '26.08' runtime-version: '26.08'
@ -25,13 +31,38 @@ finish-args:
- --socket=fallback-x11 - --socket=fallback-x11
- --socket=pulseaudio - --socket=pulseaudio
- --device=all - --device=all
# gdb and friends from the Sdk # lldb, and gdb and friends from the Sdk
- --allow=devel - --allow=devel
- --env=PATH=/app/bin:/usr/bin - --env=PATH=/app/bin:/usr/bin
cleanup: cleanup:
- /share/man - /share/man
- /share/doc - /share/doc
modules: modules:
- name: swig
# Generates lldb's Python bindings while the llvm module builds; nothing
# of it ships.
cleanup:
- '*'
sources:
- type: archive
url: https://downloads.sourceforge.net/project/swig/swig/swig-4.4.1/swig-4.4.1.tar.gz
sha256: 40162a706c56f7592d08fd52ef5511cb7ac191f3593cf07306a0a554c6281fcf
- name: libedit
# Line editing, history and completion at lldb's prompt. The Sdk has the
# ncurses it needs but not libedit itself; liblldb links the shared
# library, so that ships and the rest goes.
config-opts:
- --disable-static
cleanup:
- /include
- /lib/pkgconfig
- /lib/*.la
sources:
- type: archive
url: https://thrysoee.dk/editline/libedit-20260512-3.1.tar.gz
sha256: 432d5e7ea8b0116dd39f2eca7bc11d0eed77faa6b77ea526ace89907c23ea4a0
- name: llvm - name: llvm
buildsystem: cmake-ninja buildsystem: cmake-ninja
builddir: true builddir: true
@ -50,8 +81,20 @@ modules:
config-opts: config-opts:
- -DCMAKE_BUILD_TYPE=Release - -DCMAKE_BUILD_TYPE=Release
- -DLLVM_ENABLE_ASSERTIONS=OFF - -DLLVM_ENABLE_ASSERTIONS=OFF
- -DLLVM_ENABLE_PROJECTS=clang;lld - -DLLVM_ENABLE_PROJECTS=clang;lld;lldb
- -DLLVM_ENABLE_RUNTIMES=compiler-rt;libcxx;libcxxabi;libunwind - -DLLVM_ENABLE_RUNTIMES=compiler-rt;libcxx;libcxxabi;libunwind
# lldb with Python scripting (its bindings generated by the swig module
# above, against the Sdk's Python), libedit (the module above), curses,
# libxml2 and lzma, each set ON rather than left to the default of "if
# found", so a dependency gone missing fails the configure step instead
# of shipping an lldb without it. lldb-server, lldb-dap and lldb-mcp
# come with it.
- -DLLDB_ENABLE_PYTHON=ON
- -DLLDB_ENABLE_LIBEDIT=ON
- -DLLDB_ENABLE_CURSES=ON
- -DLLDB_ENABLE_LIBXML2=ON
- -DLLDB_ENABLE_LZMA=ON
- -DLLDB_ENABLE_LUA=OFF
- -DLLVM_TARGETS_TO_BUILD=X86;AArch64;WebAssembly - -DLLVM_TARGETS_TO_BUILD=X86;AArch64;WebAssembly
# Classic layout: libc++ in lib/ and include/c++/v1, not under a # Classic layout: libc++ in lib/ and include/c++/v1, not under a
# per-target subdirectory, the same shape as a distro install. # per-target subdirectory, the same shape as a distro install.
@ -82,6 +125,7 @@ modules:
- /include/clang - /include/clang
- /include/clang-tidy - /include/clang-tidy
- /include/lld - /include/lld
- /include/lldb
- /lib/cmake - /lib/cmake
- /lib/*.a - /lib/*.a
- /share/opt-viewer - /share/opt-viewer
@ -219,6 +263,111 @@ modules:
url: https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/llvm-project-22.1.8.src.tar.xz url: https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/llvm-project-22.1.8.src.tar.xz
sha256: 922f1817a0df7b1489272d18134ee0087a8b068828f87ac63b9861b1a9965888 sha256: 922f1817a0df7b1489272d18134ee0087a8b068828f87ac63b9861b1a9965888
# The Khronos validation layers and the four modules they are built from,
# after llvm, so that bumping them reuses the cached LLVM build. All five
# are pinned to the Vulkan SDK release of the Sdk's own Vulkan loader and
# headers (1.4.357), at the dependency revisions that release's
# scripts/known_good.json names. When the runtime moves to a newer loader,
# move the layers to the SDK release of that loader and take the other
# four revisions from its known_good.json again. The four are built into
# /app at those revisions rather than taken from the Sdk, whose versions
# follow the runtime, and of them only SPIRV-Tools' programs ship.
- name: vulkan-headers
buildsystem: cmake-ninja
builddir: true
cleanup:
- '*'
sources:
- type: archive
url: https://github.com/KhronosGroup/Vulkan-Headers/archive/refs/tags/v1.4.357.tar.gz
sha256: 7dc0dbcf1d49dd3d7da3761c251c6097dfbaac475321a4a8a99269d3d5abecdc
- name: vulkan-utility-libraries
buildsystem: cmake-ninja
builddir: true
config-opts:
- -DCMAKE_BUILD_TYPE=Release
# CMake searches its install prefix after /usr, where the Sdk has
# Vulkan headers of its own.
- -DCMAKE_PREFIX_PATH=/app
cleanup:
- '*'
sources:
- type: archive
url: https://github.com/KhronosGroup/Vulkan-Utility-Libraries/archive/refs/tags/v1.4.357.tar.gz
sha256: e8688013cdc67dad26f15dcece91fa6b80ad595d551fceb56397fc4795efd00a
- name: spirv-headers
buildsystem: cmake-ninja
builddir: true
cleanup:
- '*'
sources:
- type: archive
url: https://github.com/KhronosGroup/SPIRV-Headers/archive/29981f65241605e08b0ede4cfeb999fe3b723c6a.tar.gz
sha256: 232899f1ad4104fb5bc377b94596c7621575eee62ad9a9e8f929b63a7dd8a7ad
- name: spirv-tools
# The optimizer GPU-assisted validation instruments shaders with, linked
# into the layer statically, and SPIRV-Tools' programs (spirv-val,
# spirv-opt, spirv-dis and the rest), which ship: a program or test that
# checks its SPIR-V with spirv-val, as Crafter.Graphics' do, finds it in
# the sandbox. The libraries and headers stay build-only.
buildsystem: cmake-ninja
builddir: true
build-options:
no-debuginfo: true
strip: true
config-opts:
- -DCMAKE_BUILD_TYPE=Release
# The installed SPIRV-Headers: its include/ holds the grammar files
# SPIRV-Tools generates its tables from, as the source tree does.
- -DSPIRV-Headers_SOURCE_DIR=/app
- -DSPIRV_SKIP_TESTS=ON
- -DSPIRV_WERROR=OFF
cleanup:
- /include/spirv-tools
- /lib/libSPIRV-Tools*.a
# Always built besides the static libraries, and linked by nothing
# here: the programs and the layer take SPIRV-Tools statically.
- /lib/libSPIRV-Tools-shared.so
- /lib/cmake/SPIRV-Tools*
- /lib/pkgconfig/SPIRV-Tools*
post-install:
- spirv-val --version
sources:
- type: archive
url: https://github.com/KhronosGroup/SPIRV-Tools/archive/b707790a898e44038547df54580022fc1cf89c3d.tar.gz
sha256: 05d8af89737bde57571c48dbd36714c9f520a69623e14de72c3be6b600e277d6
- name: vulkan-validation-layers
# VK_LAYER_KHRONOS_validation. The loader finds its manifest under
# /app/share/vulkan/explicit_layer.d through XDG_DATA_DIRS, and the
# manifest names the library without a path, which the sandbox's loader
# finds in /app/lib. GPU-AV is a setting of this layer, not a layer of
# its own (VK_KHRONOS_VALIDATION_GPUAV_ENABLE=1, or the layer settings).
buildsystem: cmake-ninja
builddir: true
build-options:
no-debuginfo: true
strip: true
config-opts:
- -DCMAKE_BUILD_TYPE=Release
- -DCMAKE_PREFIX_PATH=/app
# ON by default: it clones every dependency with update_deps.py, which
# the build sandbox has no network for, instead of using the four
# modules above.
- -DUPDATE_DEPS=OFF
- -DBUILD_TESTS=OFF
- -DBUILD_WERROR=OFF
post-install:
- test -f /app/share/vulkan/explicit_layer.d/VkLayer_khronos_validation.json
- test -f /app/lib/libVkLayer_khronos_validation.so
sources:
- type: archive
url: https://github.com/KhronosGroup/Vulkan-ValidationLayers/archive/refs/tags/vulkan-sdk-1.4.357.0.tar.gz
sha256: 73180b11992a3554e97ebc18e6bf2b45ff9790ded9d87fc526c7bab865d1303f
- name: crafter-build - name: crafter-build
buildsystem: simple buildsystem: simple
build-options: build-options:

View file

@ -16,7 +16,8 @@ namespace {
struct FlatpakPaths { struct FlatpakPaths {
fs::path manifest; fs::path manifest;
// flatpak-builder's downloads, ccache and per-module cache. The // flatpak-builder's downloads, ccache and per-module cache. The
// LLVM build lives here: about 2 GB, and 25 minutes to rebuild. // LLVM build lives here: about 3 GB in all after a first run,
// which takes about 30 minutes with 56 CPU threads.
fs::path state; fs::path state;
// The OSTree repo the build is exported to; `install` reads it. // The OSTree repo the build is exported to; `install` reads it.
fs::path repo; fs::path repo;