Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
00f07dae88 |
|||
|
03951ff458 |
|||
|
40dcb18a1e |
|||
|
c8517a41cf |
|||
|
a32b8c898a |
|||
|
16db37bbe6 |
4 changed files with 198 additions and 19 deletions
|
|
@ -51,7 +51,7 @@ Three things to know about this loop:
|
||||||
- Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added. A changed default member initialiser has no symptom at all: `project.cpp` compiles against the old one and quietly gets the old default, as `project.cpp`'s sibling loads do with a harness that predates `GitProjectSpec::args = ProjectArgs()`.
|
- Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added. A changed default member initialiser has no symptom at all: `project.cpp` compiles against the old one and quietly gets the old default, as `project.cpp`'s sibling loads do with a harness that predates `GitProjectSpec::args = ProjectArgs()`.
|
||||||
- A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol.
|
- A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol.
|
||||||
|
|
||||||
`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. That takes `bin/crafter-build` itself with it, and `build/flatpak`, the state of the local Flatpak build (see "CI and publishing"): about 2 GB, and the next `run flatpak` rebuilds LLVM. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`.
|
`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. That takes `bin/crafter-build` itself with it, and `build/flatpak`, the state of the local Flatpak build (see "CI and publishing"): about 4 GB after a first run, and the next `run flatpak` rebuilds LLVM. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`.
|
||||||
|
|
||||||
## Build, test and style commands
|
## Build, test and style commands
|
||||||
|
|
||||||
|
|
@ -69,7 +69,7 @@ Three things to know about this loop:
|
||||||
./bin/crafter-build clean
|
./bin/crafter-build clean
|
||||||
```
|
```
|
||||||
|
|
||||||
Before you call a change done, all three must exit 0: `format` (then re-read your diff), `lint`, `test`. CI runs `./build.sh` and `crafter-build test` on every PR; its lint step is currently commented out, so lint is enforced only by you. Failed, crashed and timed-out tests leave their output in `build/test-logs/<name>.log`.
|
Before you call a change done, all three must exit 0: `format` (then re-read your diff), `lint`, `test`. CI runs `./build.sh` and `crafter-build test` on every push to `master` and on pull requests, and has no lint step, so lint is enforced only by you. Failed, crashed and timed-out tests leave their output in `build/test-logs/<name>.log`.
|
||||||
|
|
||||||
## Testing instructions
|
## Testing instructions
|
||||||
|
|
||||||
|
|
@ -132,18 +132,18 @@ These were decided deliberately. Do not undo them without the maintainer's say-s
|
||||||
5. **Convention over configuration.** Lead with the zero-config path. Env var, then CLI flag, then full `project.cpp` are the escalation, not the entry point.
|
5. **Convention over configuration.** Lead with the zero-config path. Env var, then CLI flag, then full `project.cpp` are the escalation, not the entry point.
|
||||||
6. **One skip mechanism** (runner probe plus exit 77) and **one `TestRunner` shape** (template fields with `{bin}`, `{bundle}` and similar placeholders; `Cmd(...)` covers any wrapper). Do not add a special-case runner class or a hardcoded "skip on this OS" branch.
|
6. **One skip mechanism** (runner probe plus exit 77) and **one `TestRunner` shape** (template fields with `{bin}`, `{bundle}` and similar placeholders; `Cmd(...)` covers any wrapper). Do not add a special-case runner class or a hardcoded "skip on this OS" branch.
|
||||||
7. **The house-style rules stay in their own repo.** They are not library API of crafter-build and are not forced on downstream projects.
|
7. **The house-style rules stay in their own repo.** They are not library API of crafter-build and are not forced on downstream projects.
|
||||||
8. **Fix warts inline; do not stack workarounds.** The previous version of this tool died of accumulated hacks. If a change needs a shortcut, say so in the PR rather than hiding it.
|
8. **Fix warts inline; do not stack workarounds.** The previous version of this tool died of accumulated hacks. If a change needs a shortcut, say so in its commit message rather than hiding it.
|
||||||
9. **`project.cpp` keeps the library/executable split.** `crafter.build-lib` owns the sources, dependencies and glslang; `crafter.build-exe` is `main.cpp` plus link flags.
|
9. **`project.cpp` keeps the library/executable split.** `crafter.build-lib` owns the sources, dependencies and glslang; `crafter.build-exe` is `main.cpp` plus link flags.
|
||||||
10. **Host-tooling builds** (the target of a `#pragma crafter dependency`) get `-fPIC` and the host triple scoped to that build. Keep it scoped.
|
10. **Host-tooling builds** (the target of a `#pragma crafter dependency`) get `-fPIC` and the host triple scoped to that build. Keep it scoped.
|
||||||
|
|
||||||
## Commit and PR guidelines
|
## Commit and push guidelines
|
||||||
|
|
||||||
- Branch from `master` and open the PR against `master` on the project's Forgejo at `forgejo.catcrafts.net/Catcrafts/Crafter.Build`. Issue work uses `claude/issue-<n>` branch names. Pushes go over SSH; HTTPS push is disabled on the server.
|
- Branch from `master`; issue work uses `claude/issue-<n>` branch names. Once the owner approves its frozen hash, a range goes straight to `master` on the project's Forgejo at `forgejo.catcrafts.net/Catcrafts/Crafter.Build`, as a fast-forward push with no merge commit and no pull request; open a pull request against `master` only when the owner asks for one. If `master` has moved since the freeze, rebase onto it, then freeze and review the rebased range again. Pushes go over SSH; HTTPS push is disabled on the server.
|
||||||
- Subject line: `type(scope): imperative summary`, with `feat`, `fix`, `refactor`, `docs` or `test`; the scope (`lint`, `incremental`, ...) is optional. Body: what was wrong, why, and why the fix has this shape, wrapped at about 75 columns. Read `git log` for the voice.
|
- Subject line: `type(scope): imperative summary`, with `feat`, `fix`, `refactor`, `docs` or `test`; the scope (`lint`, `incremental`, ...) is optional. Body: what was wrong, why, and why the fix has this shape, wrapped at about 75 columns. Read `git log` for the voice.
|
||||||
- Separate commits for the fix, its test and the README change when each stands on its own.
|
- Separate commits for the fix, its test and the README change when each stands on its own.
|
||||||
- Update the README in the same PR when user-visible behaviour changes.
|
- Update the README in the same push when user-visible behaviour changes.
|
||||||
- `format`, `lint` and `test` green before pushing. CI builds the Flatpak and runs the suite inside it on PRs, publishes the `beta` branch of the Catcrafts repo on every push to `master`, and the `stable` branch plus a release on an `X.Y.Z` tag.
|
- `format`, `lint` and `test` green before pushing. CI builds the Flatpak and runs the suite inside it on every push to `master` and on pull requests; a push to `master` then publishes the `beta` branch of the Catcrafts repo, and an `X.Y.Z` tag the `stable` branch plus a release. Without a pull request, CI does not run on a range before it reaches `master` unless someone starts it by hand (`workflow_dispatch`), so the review's gate0, the commands in `.review-gates/config.json`, is the check before the push.
|
||||||
- Every push of new commits goes through Crafter.Review's pre-push review first, and goes out only once the owner approves the frozen hash. Run `flatpak run net.catcrafts.CrafterReview init` once in a fresh checkout or worktree: it writes the `/prepush` skill that drives the review, and the skill stays untracked. `freeze` refuses merge commits, so the merge of a reviewed branch into `master` is pushed without a freeze of its own.
|
- Every push of new commits goes through Crafter.Review's pre-push review first, and goes out only once the owner approves the frozen hash. Run `flatpak run net.catcrafts.CrafterReview init` once in a fresh checkout or worktree: it writes the `/prepush` skill that drives the review, and the skill stays untracked. `freeze` refuses merge commits: a range reaches `master` as a fast-forward, and when the owner asks for a pull request instead, the merge of the reviewed branch into `master` is pushed without a freeze of its own.
|
||||||
- Every commit of a pushed range is GPG-signed, has `jorijnvdgraaf@catcrafts.net` as its author and committer, and carries no AI attribution trailer in its message, neither `Co-Authored-By:` nor `Assisted-by:`. `freeze` refuses a range with a commit that breaks any of these (`requireSigned`, `authors` and `aiTrailers` in `.review-gates/config.json`).
|
- Every commit of a pushed range is GPG-signed, has `jorijnvdgraaf@catcrafts.net` as its author and committer, and carries no AI attribution trailer in its message, neither `Co-Authored-By:` nor `Assisted-by:`. `freeze` refuses a range with a commit that breaks any of these (`requireSigned`, `authors` and `aiTrailers` in `.review-gates/config.json`).
|
||||||
- Never commit `bin/`, `build/`, `share/crafter-build/`, `lib/*.a`, `AGENTS.md` or fixture build outputs (all gitignored), and never edit the vendored files in `lib/`.
|
- Never commit `bin/`, `build/`, `share/crafter-build/`, `lib/*.a`, `AGENTS.md` or fixture build outputs (all gitignored), and never edit the vendored files in `lib/`.
|
||||||
- Keep `CRAFTER_BUILD_MARCH=x86-64-v2` in the manifest for the shipped binary: it has to run on any x86_64 machine. User projects still default to `native`.
|
- Keep `CRAFTER_BUILD_MARCH=x86-64-v2` in the manifest for the shipped binary: it has to run on any x86_64 machine. User projects still default to `native`.
|
||||||
|
|
@ -160,7 +160,7 @@ These were decided deliberately. Do not undo them without the maintainer's say-s
|
||||||
|
|
||||||
## CI and publishing
|
## CI and publishing
|
||||||
|
|
||||||
`.forgejo/workflows/ci.yaml` runs on the `crafter-flatpak` runner label: a dedicated privileged forgejo-runner on the Hetzner box (`forgejo-runner-flatpak.service`, config `/etc/forgejo-runner/config-flatpak.yaml`), registered to this repository only and the only runner with that label. It is privileged because bubblewrap inside the job container needs it, and separate from the other runners so that privilege is not shared with any other repository. Its config allows four volumes, which the workflow mounts: `crafter-flatpak-system` (the Sdk install at `/var/lib/flatpak`), `crafter-flatpak-builder` (flatpak-builder state and ccache, so the LLVM and WASI modules are rebuilt only when their sources change; a cold run takes about 80 minutes on 8 cores, a warm one a fraction of that), `/var/www/flatpak`, the web root Caddy serves as `https://flatpak.catcrafts.net/`, and `/etc/crafter-flatpak`, which holds the repo signing key (`signing-key.asc`, no passphrase, and `key-id`), generated once on the host and mounted read-only into the job; nothing is stored in the forge. The live repo at `/var/www/flatpak/repo` is canonical; builds are promoted into it with `flatpak build-commit-from`, never built into it.
|
`.forgejo/workflows/ci.yaml` runs on the `crafter-flatpak` runner label: a dedicated privileged forgejo-runner on the Hetzner box (`forgejo-runner-flatpak.service`, config `/etc/forgejo-runner/config-flatpak.yaml`), registered to this repository only and the only runner with that label. It is privileged because bubblewrap inside the job container needs it, and separate from the other runners so that privilege is not shared with any other repository. Its config allows four volumes, which the workflow mounts: `crafter-flatpak-system` (the Sdk install at `/var/lib/flatpak`), `crafter-flatpak-builder` (flatpak-builder state and ccache, which later runs reuse: flatpak-builder builds a module again when its sources or its `build-commands` change, and every module after it with it, so a new version of `swig` or `libedit`, which come before LLVM, also rebuilds the LLVM and WASI modules), `/var/www/flatpak`, the web root Caddy serves as `https://flatpak.catcrafts.net/`, and `/etc/crafter-flatpak`, which holds the repo signing key (`signing-key.asc`, no passphrase, and `key-id`), generated once on the host and mounted read-only into the job; nothing is stored in the forge. The live repo at `/var/www/flatpak/repo` is canonical; builds are promoted into it with `flatpak build-commit-from`, never built into it.
|
||||||
|
|
||||||
Keep the label unique to that runner. Plain `flatpak` also belongs to the organisation-wide runner (`forgejo-runner-org-flatpak.service`), which serves every Catcrafts repository and on purpose allows no volumes. Whenever this job lands there, flatpak-builder stops with `The state dir (/root/.cache/flatpak-builder) is not on the same filesystem as the target dir`; the top of the job log shows which runner took it and lists every volume it ignored.
|
Keep the label unique to that runner. Plain `flatpak` also belongs to the organisation-wide runner (`forgejo-runner-org-flatpak.service`), which serves every Catcrafts repository and on purpose allows no volumes. Whenever this job lands there, flatpak-builder stops with `The state dir (/root/.cache/flatpak-builder) is not on the same filesystem as the target dir`; the top of the job log shows which runner took it and lists every volume it ignored.
|
||||||
|
|
||||||
|
|
@ -174,7 +174,7 @@ gpg --batch --pinentry-mode loopback --passphrase '' --export-secret-keys --armo
|
||||||
printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id
|
printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id
|
||||||
```
|
```
|
||||||
|
|
||||||
To release: add a `<release>` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry.
|
To release: add a `<release>` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), push it to `master`, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry.
|
||||||
Local builds are two scripts in `project.cpp`. Run them with the native binary: the Flatpak build of crafter-build has no `flatpak` to drive.
|
Local builds are two scripts in `project.cpp`. Run them with the native binary: the Flatpak build of crafter-build has no `flatpak` to drive.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -182,7 +182,7 @@ Local builds are two scripts in `project.cpp`. Run them with the native binary:
|
||||||
./bin/crafter-build run install # install that build as branch `local` and make it current
|
./bin/crafter-build run install # install that build as branch `local` and make it current
|
||||||
```
|
```
|
||||||
|
|
||||||
`run flatpak` needs `flatpak-builder` or the `org.flatpak.Builder` app, and the Sdk (`flatpak install flathub org.freedesktop.Sdk//26.08`). It builds the working tree, uncommitted changes included, runs the whole suite inside the sandbox and reports version `0.0.0-dev`. Its state is `build/flatpak/` (`state/`, `repo/`, `build-dir/`). The first run builds LLVM and the WASI sysroot, about 25 minutes on a 56-core machine; later runs rebuild only the crafter-build module, a few minutes, because flatpak-builder never caches a `type: dir` source. The checkout must not be under `/tmp`, which `org.flatpak.Builder` cannot see. Use it for anything toolchain-sensitive: the native loop compiles with the distro's clang and the Flatpak with its bundled LLVM, and the two can be a major version apart.
|
`run flatpak` needs `flatpak-builder` or the `org.flatpak.Builder` app, and the Sdk (`flatpak install flathub org.freedesktop.Sdk//26.08`). It builds the working tree, uncommitted changes included, runs the whole suite inside the sandbox and reports version `0.0.0-dev`. Its state is `build/flatpak/` (`state/`, `repo/`, `build-dir/`). The first run builds every module, LLVM with lldb, the WASI sysroot and the Vulkan validation layer among them, in about 30 minutes with 56 CPU threads; later runs rebuild only the crafter-build module while the manifest is unchanged, a few minutes, because flatpak-builder never caches a `type: dir` source. The checkout must not be under `/tmp`, which `org.flatpak.Builder` cannot see. Use it for anything toolchain-sensitive: the native loop compiles with the distro's clang and the Flatpak with its bundled LLVM, and the two can be a major version apart.
|
||||||
|
|
||||||
`run install` does not build; it installs what `run flatpak` last exported. It points the user remote `crafter-build-local` at `build/flatpak/repo`, installs `net.catcrafts.CrafterBuild//local` from it next to whichever published branch is installed, and makes it the branch `flatpak run net.catcrafts.CrafterBuild` starts. `flatpak make-current --user net.catcrafts.CrafterBuild <branch>` switches back. Before `crafter-build clean`, or before deleting the checkout, run `flatpak uninstall --user net.catcrafts.CrafterBuild//local` and `flatpak remote-delete --user crafter-build-local`; otherwise the remote is left pointing at a repo that is gone.
|
`run install` does not build; it installs what `run flatpak` last exported. It points the user remote `crafter-build-local` at `build/flatpak/repo`, installs `net.catcrafts.CrafterBuild//local` from it next to whichever published branch is installed, and makes it the branch `flatpak run net.catcrafts.CrafterBuild` starts. `flatpak make-current --user net.catcrafts.CrafterBuild <branch>` switches back. Before `crafter-build clean`, or before deleting the checkout, run `flatpak uninstall --user net.catcrafts.CrafterBuild//local` and `flatpak remote-delete --user crafter-build-local`; otherwise the remote is left pointing at a repo that is gone.
|
||||||
|
|
||||||
|
|
@ -197,6 +197,7 @@ Local builds are two scripts in `project.cpp`. Run them with the native binary:
|
||||||
- The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop.
|
- The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop.
|
||||||
- A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone.
|
- A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone.
|
||||||
- A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under `share/crafter-build/libc++-modules/<major>/`, one `libcxx-<major>-modules` manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under `share/crafter-build/` to test against a foreign sysroot.
|
- A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under `share/crafter-build/libc++-modules/<major>/`, one `libcxx-<major>-modules` manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under `share/crafter-build/` to test against a foreign sysroot.
|
||||||
|
- The Flatpak carries the debugging tools for what it builds, because those programs run in its sandbox: lldb is a project of the `llvm` module, with `LLDB_ENABLE_PYTHON`, `LIBEDIT`, `CURSES`, `LIBXML2` and `LZMA` set ON so that a missing dependency fails the configure step (SWIG through Python, which needs it), and the `swig` and `libedit` modules come before it. The Khronos validation layer is the `vulkan-validation-layers` module plus four modules before it, build-only except for SPIRV-Tools' programs (`spirv-val` and the rest), all after the LLVM and WASI modules, so that bumping them reuses the cached LLVM and WASI builds. The layer follows the Sdk's Vulkan loader: when `runtime-version` moves to a runtime with a newer loader, move the layer to the Vulkan SDK release of that loader and take the other four revisions from that release's `scripts/known_good.json`.
|
||||||
- The wasm compile flags live in one place, `WasmBaselineFlags` in `Crafter.Build-Clang.cppm`: the std module and every unit have to agree on them, and `VariantId()` and the std PCM directory hash them, so a change rebuilds instead of mixing old and new BMIs. They enable C++ exceptions, so the link takes the exception-enabled libc++ and libunwind from the sysroot's `lib/<triple>/eh/` (wasi-sdk 33 and later). The Flatpak's sysroot has them and a distro's may not; where they are missing the wasm tests skip, so run those through the Flatpak. A browser page checks the engine with the `WasmBaselineProbes` named in `variants.json`.
|
- The wasm compile flags live in one place, `WasmBaselineFlags` in `Crafter.Build-Clang.cppm`: the std module and every unit have to agree on them, and `VariantId()` and the std PCM directory hash them, so a change rebuilds instead of mixing old and new BMIs. They enable C++ exceptions, so the link takes the exception-enabled libc++ and libunwind from the sysroot's `lib/<triple>/eh/` (wasi-sdk 33 and later). The Flatpak's sysroot has them and a distro's may not; where they are missing the wasm tests skip, so run those through the Flatpak. A browser page checks the engine with the `WasmBaselineProbes` named in `variants.json`.
|
||||||
- `WasiBrowserRuntimeExit` runs `wasi-runtime/runtime.js` under node and skips where there is none. The Flatpak has no node, so CI never runs it: after touching `runtime.js`, run the suite in the native loop with node on `PATH`.
|
- `WasiBrowserRuntimeExit` runs `wasi-runtime/runtime.js` under node and skips where there is none. The Flatpak has no node, so CI never runs it: after touching `runtime.js`, run the suite in the native loop with node on `PATH`.
|
||||||
- `VariantId()` names the output directory, so CI globs `bin/crafter.build-exe-<target>-<march>-*/` rather than predicting the hash. The hash includes the compiler, so after a toolchain update the previous variant's directories stay in `bin/` and `build/` until `clean`, and such a glob matches both.
|
- `VariantId()` names the output directory, so CI globs `bin/crafter.build-exe-<target>-<march>-*/` rather than predicting the hash. The hash includes the compiler, so after a toolchain update the previous variant's directories stay in `bin/` and `build/` until `clean`, and such a glob matches both.
|
||||||
|
|
|
||||||
32
README.md
32
README.md
|
|
@ -4,14 +4,16 @@ A C++26 modules build system. Project descriptions are written in C++ with no JS
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
crafter-build ships as a Flatpak that carries its own clang, lld, libc++ and WASI sysroot, so every machine builds with the same compiler:
|
crafter-build ships as a Flatpak that carries its own clang, lld, libc++ and WASI sysroot, so every machine builds with the same compiler, and lldb and the Vulkan validation layers to debug the result (see [Debugging](#debugging)):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
flatpak remote-add --if-not-exists catcrafts https://flatpak.catcrafts.net/catcrafts.flatpakrepo
|
flatpak remote-add --if-not-exists catcrafts https://flatpak.catcrafts.net/catcrafts.flatpakrepo
|
||||||
flatpak install catcrafts net.catcrafts.CrafterBuild
|
flatpak install catcrafts net.catcrafts.CrafterBuild
|
||||||
alias crafter-build='flatpak run net.catcrafts.CrafterBuild'
|
alias crafter-build='flatpak run --die-with-parent net.catcrafts.CrafterBuild'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`--die-with-parent` ends the sandbox with the `flatpak run` that started it. Without the flag, killing only the `flatpak run` process leaves the sandbox running: a build carries on writing into `bin/` and `build/`, and a program started with `-r` keeps running. With it, bubblewrap kills everything in the sandbox with SIGKILL when the `flatpak run` process dies or when that process's parent does. For `crafter-build …` or `crafter-build … &` typed at a bash or zsh prompt, that parent is the shell, so the sandbox ends with the shell, even after a `disown`. Ctrl-C reaches the program either way.
|
||||||
|
|
||||||
Pre-release builds of `master` are the `beta` branch (`flatpak install catcrafts net.catcrafts.CrafterBuild//beta`), and every release also attaches a single-file bundle on the Forgejo releases page (`flatpak install --user crafter-build-<version>.flatpak`). The sandbox sees the whole host filesystem, so projects can live anywhere. Binaries built for the host link the bundled `libc++.so.1` through an rpath into the Flatpak: they run inside it (`-r`, the test runner), and are not meant to be copied out onto the host.
|
Pre-release builds of `master` are the `beta` branch (`flatpak install catcrafts net.catcrafts.CrafterBuild//beta`), and every release also attaches a single-file bundle on the Forgejo releases page (`flatpak install --user crafter-build-<version>.flatpak`). The sandbox sees the whole host filesystem, so projects can live anywhere. Binaries built for the host link the bundled `libc++.so.1` through an rpath into the Flatpak: they run inside it (`-r`, the test runner), and are not meant to be copied out onto the host.
|
||||||
|
|
||||||
## Writing a project.cpp
|
## Writing a project.cpp
|
||||||
|
|
@ -364,6 +366,32 @@ Things to know:
|
||||||
- Inside the Flatpak, commands run in its sandbox: the host's files are visible, but the tools on `PATH` are the ones the Flatpak ships (`git`, `cmake`, `python3`, `tar`, ...), not everything installed on the host.
|
- Inside the Flatpak, commands run in its sandbox: the host's files are visible, but the tools on `PATH` are the ones the Flatpak ships (`git`, `cmake`, `python3`, `tar`, ...), not everything installed on the host.
|
||||||
- A return value outside 0 to 255 is not an exit status and becomes 1. So does a script that throws, with the exception's message printed.
|
- A return value outside 0 to 255 is not an exit status and becomes 1. So does a script that throws, with the exception's message printed.
|
||||||
|
|
||||||
|
## Debugging
|
||||||
|
|
||||||
|
Programs that `-r` and the test runner start run inside the Flatpak's sandbox, and so do the tools to debug them: lldb, with `lldb-server`, `lldb-dap` and `lldb-mcp`, from the same LLVM build as the compiler; `gdb`, `gdbserver`, `valgrind` and `strace` from the Freedesktop Sdk; the Khronos Vulkan validation layer; and the SPIRV-Tools programs (`spirv-val`, `spirv-opt`, `spirv-dis`, ...). A default build is optimised (`-O3`, and ThinLTO except for wasm) and carries no debug information: set `cfg.debug = true` for a program you want to debug, which compiles its C++ units with `-g` and links it without LTO; its C and CUDA sources (`cfg.cFiles`, `cfg.cuda`) stay at `-O3` without debug information. A `project.cpp` that calls `ApplyStandardArgs` sets it from the `--debug` argument, as in `crafter-build --debug -r`. Start the tools with `--command`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
flatpak run --die-with-parent --command=lldb net.catcrafts.CrafterBuild ./bin/myapp-<variant>/myapp
|
||||||
|
flatpak run --die-with-parent --command=valgrind net.catcrafts.CrafterBuild ./bin/myapp-<variant>/myapp
|
||||||
|
```
|
||||||
|
|
||||||
|
Every `flatpak run` is a sandbox of its own, with process IDs of its own, so a debugger in one cannot see a program that another one, such as `crafter-build -r`, started. Either start the program under the debugger, or run the debugger inside that program's sandbox with `flatpak enter`, giving the instance `flatpak ps` lists for net.catcrafts.CrafterBuild and the debugger's full path:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
flatpak ps --columns=instance,application
|
||||||
|
flatpak enter <instance> /app/bin/lldb -n myapp
|
||||||
|
```
|
||||||
|
|
||||||
|
Where `/proc/sys/kernel/yama/ptrace_scope` is 1, a debugger in the sandbox, where processes hold no capabilities, can attach to a program that is not its descendant only when the program allowed it with `prctl(PR_SET_PTRACER, …)`: with the debugger's process ID, or with `PR_SET_PTRACER_ANY` for any process. Otherwise the attach fails. A program started under the debugger is its descendant. Prefer `flatpak enter` to `lldb-server gdbserver` in one sandbox and lldb's `gdb-remote` from another: when a process in lldb's own sandbox has the remote program's process ID, lldb takes that process's executable for the program's: symbols come out wrong, and a breakpoint in the program's source finds no location.
|
||||||
|
|
||||||
|
The validation layer is enabled the usual ways, by the program itself or from the environment: `flatpak run` passes `VK_INSTANCE_LAYERS`, `VK_LOADER_LAYERS_ENABLE` and the layer's `VK_KHRONOS_VALIDATION_*` settings into the sandbox. `VK_KHRONOS_VALIDATION_GPUAV_ENABLE=1` adds GPU-assisted validation (GPU-AV):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation VK_KHRONOS_VALIDATION_GPUAV_ENABLE=1 crafter-build -r
|
||||||
|
```
|
||||||
|
|
||||||
|
The layer is the Vulkan SDK release of the runtime's own Vulkan loader. `VK_LAYER_PATH` and `VK_ADD_LAYER_PATH` are cleared on the way into the sandbox, so a layer installed on the host is out of reach.
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
[examples/](examples/) contains progressively larger self-contained projects:
|
[examples/](examples/) contains progressively larger self-contained projects:
|
||||||
|
|
|
||||||
|
|
@ -5,9 +5,15 @@
|
||||||
# Platform: a build tool needs glibc's headers and crt objects, gcc's
|
# Platform: a build tool needs glibc's headers and crt objects, gcc's
|
||||||
# libgcc_s, cmake, ninja, make, git and python at run time. Everything
|
# libgcc_s, cmake, ninja, make, git and python at run time. Everything
|
||||||
# compiler-shaped is ours and lives under /app: clang, lld, libclang, libc++,
|
# compiler-shaped is ours and lives under /app: clang, lld, libclang, libc++,
|
||||||
# compiler-rt (native and wasm32) and the WASI sysroot, all built from one
|
# compiler-rt (native and wasm32), lldb and the WASI sysroot, all built from
|
||||||
# LLVM source tree so the modules crafter-build builds and the compiler that
|
# one LLVM source tree so the modules crafter-build builds and the compiler
|
||||||
# consumes them can never drift apart.
|
# that consumes them can never drift apart.
|
||||||
|
#
|
||||||
|
# Programs that `-r` and the test runner start run in this sandbox, so the
|
||||||
|
# tools to debug them live here too: lldb (above), the Sdk's gdb, valgrind
|
||||||
|
# and strace, the Khronos Vulkan validation layers, GPU-assisted validation
|
||||||
|
# included, which neither the Sdk nor its GL extensions carry, and the
|
||||||
|
# SPIRV-Tools programs such as spirv-val.
|
||||||
app-id: net.catcrafts.CrafterBuild
|
app-id: net.catcrafts.CrafterBuild
|
||||||
runtime: org.freedesktop.Sdk
|
runtime: org.freedesktop.Sdk
|
||||||
runtime-version: '26.08'
|
runtime-version: '26.08'
|
||||||
|
|
@ -25,13 +31,38 @@ finish-args:
|
||||||
- --socket=fallback-x11
|
- --socket=fallback-x11
|
||||||
- --socket=pulseaudio
|
- --socket=pulseaudio
|
||||||
- --device=all
|
- --device=all
|
||||||
# gdb and friends from the Sdk
|
# lldb, and gdb and friends from the Sdk
|
||||||
- --allow=devel
|
- --allow=devel
|
||||||
- --env=PATH=/app/bin:/usr/bin
|
- --env=PATH=/app/bin:/usr/bin
|
||||||
cleanup:
|
cleanup:
|
||||||
- /share/man
|
- /share/man
|
||||||
- /share/doc
|
- /share/doc
|
||||||
modules:
|
modules:
|
||||||
|
- name: swig
|
||||||
|
# Generates lldb's Python bindings while the llvm module builds; nothing
|
||||||
|
# of it ships.
|
||||||
|
cleanup:
|
||||||
|
- '*'
|
||||||
|
sources:
|
||||||
|
- type: archive
|
||||||
|
url: https://downloads.sourceforge.net/project/swig/swig/swig-4.4.1/swig-4.4.1.tar.gz
|
||||||
|
sha256: 40162a706c56f7592d08fd52ef5511cb7ac191f3593cf07306a0a554c6281fcf
|
||||||
|
|
||||||
|
- name: libedit
|
||||||
|
# Line editing, history and completion at lldb's prompt. The Sdk has the
|
||||||
|
# ncurses it needs but not libedit itself; liblldb links the shared
|
||||||
|
# library, so that ships and the rest goes.
|
||||||
|
config-opts:
|
||||||
|
- --disable-static
|
||||||
|
cleanup:
|
||||||
|
- /include
|
||||||
|
- /lib/pkgconfig
|
||||||
|
- /lib/*.la
|
||||||
|
sources:
|
||||||
|
- type: archive
|
||||||
|
url: https://thrysoee.dk/editline/libedit-20260512-3.1.tar.gz
|
||||||
|
sha256: 432d5e7ea8b0116dd39f2eca7bc11d0eed77faa6b77ea526ace89907c23ea4a0
|
||||||
|
|
||||||
- name: llvm
|
- name: llvm
|
||||||
buildsystem: cmake-ninja
|
buildsystem: cmake-ninja
|
||||||
builddir: true
|
builddir: true
|
||||||
|
|
@ -50,8 +81,20 @@ modules:
|
||||||
config-opts:
|
config-opts:
|
||||||
- -DCMAKE_BUILD_TYPE=Release
|
- -DCMAKE_BUILD_TYPE=Release
|
||||||
- -DLLVM_ENABLE_ASSERTIONS=OFF
|
- -DLLVM_ENABLE_ASSERTIONS=OFF
|
||||||
- -DLLVM_ENABLE_PROJECTS=clang;lld
|
- -DLLVM_ENABLE_PROJECTS=clang;lld;lldb
|
||||||
- -DLLVM_ENABLE_RUNTIMES=compiler-rt;libcxx;libcxxabi;libunwind
|
- -DLLVM_ENABLE_RUNTIMES=compiler-rt;libcxx;libcxxabi;libunwind
|
||||||
|
# lldb with Python scripting (its bindings generated by the swig module
|
||||||
|
# above, against the Sdk's Python), libedit (the module above), curses,
|
||||||
|
# libxml2 and lzma, each set ON rather than left to the default of "if
|
||||||
|
# found", so a dependency gone missing fails the configure step instead
|
||||||
|
# of shipping an lldb without it. lldb-server, lldb-dap and lldb-mcp
|
||||||
|
# come with it.
|
||||||
|
- -DLLDB_ENABLE_PYTHON=ON
|
||||||
|
- -DLLDB_ENABLE_LIBEDIT=ON
|
||||||
|
- -DLLDB_ENABLE_CURSES=ON
|
||||||
|
- -DLLDB_ENABLE_LIBXML2=ON
|
||||||
|
- -DLLDB_ENABLE_LZMA=ON
|
||||||
|
- -DLLDB_ENABLE_LUA=OFF
|
||||||
- -DLLVM_TARGETS_TO_BUILD=X86;AArch64;WebAssembly
|
- -DLLVM_TARGETS_TO_BUILD=X86;AArch64;WebAssembly
|
||||||
# Classic layout: libc++ in lib/ and include/c++/v1, not under a
|
# Classic layout: libc++ in lib/ and include/c++/v1, not under a
|
||||||
# per-target subdirectory, the same shape as a distro install.
|
# per-target subdirectory, the same shape as a distro install.
|
||||||
|
|
@ -82,6 +125,7 @@ modules:
|
||||||
- /include/clang
|
- /include/clang
|
||||||
- /include/clang-tidy
|
- /include/clang-tidy
|
||||||
- /include/lld
|
- /include/lld
|
||||||
|
- /include/lldb
|
||||||
- /lib/cmake
|
- /lib/cmake
|
||||||
- /lib/*.a
|
- /lib/*.a
|
||||||
- /share/opt-viewer
|
- /share/opt-viewer
|
||||||
|
|
@ -219,6 +263,111 @@ modules:
|
||||||
url: https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/llvm-project-22.1.8.src.tar.xz
|
url: https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/llvm-project-22.1.8.src.tar.xz
|
||||||
sha256: 922f1817a0df7b1489272d18134ee0087a8b068828f87ac63b9861b1a9965888
|
sha256: 922f1817a0df7b1489272d18134ee0087a8b068828f87ac63b9861b1a9965888
|
||||||
|
|
||||||
|
# The Khronos validation layers and the four modules they are built from,
|
||||||
|
# after llvm, so that bumping them reuses the cached LLVM build. All five
|
||||||
|
# are pinned to the Vulkan SDK release of the Sdk's own Vulkan loader and
|
||||||
|
# headers (1.4.357), at the dependency revisions that release's
|
||||||
|
# scripts/known_good.json names. When the runtime moves to a newer loader,
|
||||||
|
# move the layers to the SDK release of that loader and take the other
|
||||||
|
# four revisions from its known_good.json again. The four are built into
|
||||||
|
# /app at those revisions rather than taken from the Sdk, whose versions
|
||||||
|
# follow the runtime, and of them only SPIRV-Tools' programs ship.
|
||||||
|
- name: vulkan-headers
|
||||||
|
buildsystem: cmake-ninja
|
||||||
|
builddir: true
|
||||||
|
cleanup:
|
||||||
|
- '*'
|
||||||
|
sources:
|
||||||
|
- type: archive
|
||||||
|
url: https://github.com/KhronosGroup/Vulkan-Headers/archive/refs/tags/v1.4.357.tar.gz
|
||||||
|
sha256: 7dc0dbcf1d49dd3d7da3761c251c6097dfbaac475321a4a8a99269d3d5abecdc
|
||||||
|
|
||||||
|
- name: vulkan-utility-libraries
|
||||||
|
buildsystem: cmake-ninja
|
||||||
|
builddir: true
|
||||||
|
config-opts:
|
||||||
|
- -DCMAKE_BUILD_TYPE=Release
|
||||||
|
# CMake searches its install prefix after /usr, where the Sdk has
|
||||||
|
# Vulkan headers of its own.
|
||||||
|
- -DCMAKE_PREFIX_PATH=/app
|
||||||
|
cleanup:
|
||||||
|
- '*'
|
||||||
|
sources:
|
||||||
|
- type: archive
|
||||||
|
url: https://github.com/KhronosGroup/Vulkan-Utility-Libraries/archive/refs/tags/v1.4.357.tar.gz
|
||||||
|
sha256: e8688013cdc67dad26f15dcece91fa6b80ad595d551fceb56397fc4795efd00a
|
||||||
|
|
||||||
|
- name: spirv-headers
|
||||||
|
buildsystem: cmake-ninja
|
||||||
|
builddir: true
|
||||||
|
cleanup:
|
||||||
|
- '*'
|
||||||
|
sources:
|
||||||
|
- type: archive
|
||||||
|
url: https://github.com/KhronosGroup/SPIRV-Headers/archive/29981f65241605e08b0ede4cfeb999fe3b723c6a.tar.gz
|
||||||
|
sha256: 232899f1ad4104fb5bc377b94596c7621575eee62ad9a9e8f929b63a7dd8a7ad
|
||||||
|
|
||||||
|
- name: spirv-tools
|
||||||
|
# The optimizer GPU-assisted validation instruments shaders with, linked
|
||||||
|
# into the layer statically, and SPIRV-Tools' programs (spirv-val,
|
||||||
|
# spirv-opt, spirv-dis and the rest), which ship: a program or test that
|
||||||
|
# checks its SPIR-V with spirv-val, as Crafter.Graphics' do, finds it in
|
||||||
|
# the sandbox. The libraries and headers stay build-only.
|
||||||
|
buildsystem: cmake-ninja
|
||||||
|
builddir: true
|
||||||
|
build-options:
|
||||||
|
no-debuginfo: true
|
||||||
|
strip: true
|
||||||
|
config-opts:
|
||||||
|
- -DCMAKE_BUILD_TYPE=Release
|
||||||
|
# The installed SPIRV-Headers: its include/ holds the grammar files
|
||||||
|
# SPIRV-Tools generates its tables from, as the source tree does.
|
||||||
|
- -DSPIRV-Headers_SOURCE_DIR=/app
|
||||||
|
- -DSPIRV_SKIP_TESTS=ON
|
||||||
|
- -DSPIRV_WERROR=OFF
|
||||||
|
cleanup:
|
||||||
|
- /include/spirv-tools
|
||||||
|
- /lib/libSPIRV-Tools*.a
|
||||||
|
# Always built besides the static libraries, and linked by nothing
|
||||||
|
# here: the programs and the layer take SPIRV-Tools statically.
|
||||||
|
- /lib/libSPIRV-Tools-shared.so
|
||||||
|
- /lib/cmake/SPIRV-Tools*
|
||||||
|
- /lib/pkgconfig/SPIRV-Tools*
|
||||||
|
post-install:
|
||||||
|
- spirv-val --version
|
||||||
|
sources:
|
||||||
|
- type: archive
|
||||||
|
url: https://github.com/KhronosGroup/SPIRV-Tools/archive/b707790a898e44038547df54580022fc1cf89c3d.tar.gz
|
||||||
|
sha256: 05d8af89737bde57571c48dbd36714c9f520a69623e14de72c3be6b600e277d6
|
||||||
|
|
||||||
|
- name: vulkan-validation-layers
|
||||||
|
# VK_LAYER_KHRONOS_validation. The loader finds its manifest under
|
||||||
|
# /app/share/vulkan/explicit_layer.d through XDG_DATA_DIRS, and the
|
||||||
|
# manifest names the library without a path, which the sandbox's loader
|
||||||
|
# finds in /app/lib. GPU-AV is a setting of this layer, not a layer of
|
||||||
|
# its own (VK_KHRONOS_VALIDATION_GPUAV_ENABLE=1, or the layer settings).
|
||||||
|
buildsystem: cmake-ninja
|
||||||
|
builddir: true
|
||||||
|
build-options:
|
||||||
|
no-debuginfo: true
|
||||||
|
strip: true
|
||||||
|
config-opts:
|
||||||
|
- -DCMAKE_BUILD_TYPE=Release
|
||||||
|
- -DCMAKE_PREFIX_PATH=/app
|
||||||
|
# ON by default: it clones every dependency with update_deps.py, which
|
||||||
|
# the build sandbox has no network for, instead of using the four
|
||||||
|
# modules above.
|
||||||
|
- -DUPDATE_DEPS=OFF
|
||||||
|
- -DBUILD_TESTS=OFF
|
||||||
|
- -DBUILD_WERROR=OFF
|
||||||
|
post-install:
|
||||||
|
- test -f /app/share/vulkan/explicit_layer.d/VkLayer_khronos_validation.json
|
||||||
|
- test -f /app/lib/libVkLayer_khronos_validation.so
|
||||||
|
sources:
|
||||||
|
- type: archive
|
||||||
|
url: https://github.com/KhronosGroup/Vulkan-ValidationLayers/archive/refs/tags/vulkan-sdk-1.4.357.0.tar.gz
|
||||||
|
sha256: 73180b11992a3554e97ebc18e6bf2b45ff9790ded9d87fc526c7bab865d1303f
|
||||||
|
|
||||||
- name: crafter-build
|
- name: crafter-build
|
||||||
buildsystem: simple
|
buildsystem: simple
|
||||||
build-options:
|
build-options:
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,8 @@ namespace {
|
||||||
struct FlatpakPaths {
|
struct FlatpakPaths {
|
||||||
fs::path manifest;
|
fs::path manifest;
|
||||||
// flatpak-builder's downloads, ccache and per-module cache. The
|
// flatpak-builder's downloads, ccache and per-module cache. The
|
||||||
// LLVM build lives here: about 2 GB, and 25 minutes to rebuild.
|
// LLVM build lives here: about 3 GB in all after a first run,
|
||||||
|
// which takes about 30 minutes with 56 CPU threads.
|
||||||
fs::path state;
|
fs::path state;
|
||||||
// The OSTree repo the build is exported to; `install` reads it.
|
// The OSTree repo the build is exported to; `install` reads it.
|
||||||
fs::path repo;
|
fs::path repo;
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue