diff --git a/AGENTS.md b/AGENTS.md index 574abeb..ff93b96 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,7 @@ crafter-build is a C++26 modules build system. A project is described by a `proj |---|---| | `interfaces/Crafter.Build-.cppm` | Module interface partitions. `Crafter.Build.cppm` is the umbrella that re-exports them. | | `implementations/Crafter.Build-.cpp` | Implementation units, one per partition, named `Crafter.Build:_impl`. `main.cpp` is a thin wrapper around `Crafter::Run`. | -| `project.cpp` | The self-host build description: a `crafter.build-lib` library, a `crafter.build-exe` executable, the test declarations and the lint rules. | +| `project.cpp` | The self-host build description: a `crafter.build-lib` library, a `crafter.build-exe` executable, the test declarations, the lint rules and the `flatpak` and `install` scripts. | | `tests//main.cpp` | One self-test per directory; `tests//fixture/` holds the sample project it builds. | | `examples/` | Standalone user-facing projects, MIT licensed. | | `build.sh` | Bootstrap script. It compiles everything by hand, without crafter-build. | @@ -22,12 +22,13 @@ crafter-build is a C++26 modules build system. A project is described by a `proj Partitions and what they own: - `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release. -- `:Clang` holds `Configuration`, `Build()`, `VariantId()`, `ApplyStandardArgs`, `HostTarget`, the CLI entry point `Crafter::Run`, and the lint and test data types. +- `:Clang` holds `Configuration`, `Build()`, `VariantId()`, `ApplyStandardArgs`, `HostTarget`, the CLI entry point `Crafter::Run`, and the lint, test and script data types. - `:Platform` runs commands, implements `LoadProject` (compile and load `project.cpp`), the host PCM cache, `#pragma crafter dependency`, and all OS-specific code. - `:Interface` and `:Implementation` model module units, their staleness checks and their compiles. - `:External` handles git + cmake dependencies, `GitProject` and `LocalProject`. - `:Test` holds `AddTest`, `RunTests` and `TestRunner`. - `:Lint` is the lint/format driver plus the libclang token and AST layer. +- `:Script` is the driver behind `crafter-build run`: it looks a name up in `cfg.scripts`, calls the callback and turns what it returns into the exit code. - `:Progress` is console output. `:Shader` compiles shaders through glslang. `:Asset` is the hook into Crafter.Asset. ## Dev environment @@ -47,7 +48,7 @@ Three things to know about this loop: - Changes under `implementations/` are picked up by the self-host build and by the tests, which link the local library. Changes under `interfaces/` are not: the harness compiles against the copies in `share/crafter-build/` and the PCM cache built from them, so a new or changed declaration is invisible until you run `./build.sh` again. The symptom is `use of undeclared identifier` for the symbol you just added. - A new exported symbol that a `project.cpp` reaches on its default path also needs `./build.sh`: the old binary does not export it, and `LoadProject` fails with an undefined symbol. -`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`. +`./build.sh` clones and builds glslang on first run and takes several minutes; later runs are faster. `./bin/crafter-build clean` deletes `bin/` and `build/` without loading `project.cpp`. That takes `bin/crafter-build` itself with it, and `build/flatpak`, the state of the local Flatpak build (see "CI and publishing"): about 2 GB, and the next `run flatpak` rebuilds LLVM. Set `CRAFTER_BUILD_HOME` only when running a binary from outside this tree; `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` default to `native`. ## Build, test and style commands @@ -60,6 +61,8 @@ Three things to know about this loop: ./bin/crafter-build lint # dry style gate, exit 1 on any finding ./bin/crafter-build format # apply the transform rules in place ./bin/crafter-build format --check # CI-style dry run +./bin/crafter-build run # list the scripts project.cpp defines +./bin/crafter-build run [args...] # run one; everything after the name is the script's ./bin/crafter-build clean ``` @@ -142,6 +145,7 @@ These were decided deliberately. Do not undo them without the maintainer's say-s - `project.cpp` is compiled and loaded into the crafter-build process, and every `GitProject`, `#pragma crafter dependency` and `ExternalDependency` clones a repository and runs its build on the host. Treat a dependency URL as code you are choosing to run. This repo's own dependencies are sibling Catcrafts repos tracking their default branch on purpose; pin with `commit` when adding anything else, and whenever reproducibility matters. - Unpinned dependencies `git pull` on every invocation, so builds and tests need network. A missing or private repo surfaces as a clone failure before anything is built. - `--runner=cmd:` executes the named wrapper around every test binary. Wrappers come from `--runner=` or `CRAFTER_BUILD_RUNNER_*`; do not hardcode hosts, wrappers or credentials in `project.cpp` or tests. +- `crafter-build run ` executes whatever the script does, with your permissions. Only the scripts of the project crafter-build was pointed at can be run, never a dependency's; keep it that way. - The linter loads `libclang` at runtime by name. `CRAFTER_BUILD_LIBCLANG` pins an exact path and must fail loudly when wrong, never fall back to another library. - CI force-pushes the rolling `latest` tag and publishes a prerelease on every push to `master` with the workflow's own token, and signs the Flatpak repo with the key mounted from the runner host. Do not add secrets to the workflow or echo the environment in it. - Tests write only under `fs::temp_directory_path()` and the fixture's own `bin/` and `build/`. Keep it that way. @@ -160,10 +164,21 @@ gpg --batch --pinentry-mode loopback --passphrase '' --export-secret-keys --armo printf '%s\n' "$KEYID" > /etc/crafter-flatpak/key-id && chmod 600 /etc/crafter-flatpak/signing-key.asc /etc/crafter-flatpak/key-id ``` -To release: add a `` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry. Local builds: `flatpak-builder --user --force-clean --ccache --repo=repo build-dir packaging/flatpak/net.catcrafts.CrafterBuild.yaml`, then `flatpak-builder --run build-dir crafter-build --version`. +To release: add a `` to `packaging/flatpak/net.catcrafts.CrafterBuild.metainfo.xml` (its description becomes the release notes), merge, and push the tag `X.Y.Z`. CI refuses a tag that does not match the newest entry. +Local builds are two scripts in `project.cpp`. Run them with the native binary: the Flatpak build of crafter-build has no `flatpak` to drive. + +```bash +./bin/crafter-build run flatpak [--jobs=] # build the Flatpak, then print its version from inside the sandbox +./bin/crafter-build run install # install that build as branch `local` and make it current +``` + +`run flatpak` needs `flatpak-builder` or the `org.flatpak.Builder` app, and the Sdk (`flatpak install flathub org.freedesktop.Sdk//26.08`). It builds the working tree, uncommitted changes included, runs the whole suite inside the sandbox and reports version `0.0.0-dev`. Its state is `build/flatpak/` (`state/`, `repo/`, `build-dir/`). The first run builds LLVM and the WASI sysroot, about 25 minutes on a 56-core machine; later runs rebuild only the crafter-build module, a few minutes, because flatpak-builder never caches a `type: dir` source. The checkout must not be under `/tmp`, which `org.flatpak.Builder` cannot see. Use it for anything toolchain-sensitive: the native loop compiles with the distro's clang and the Flatpak with its bundled LLVM, and the two can be a major version apart. + +`run install` does not build; it installs what `run flatpak` last exported. It points the user remote `crafter-build-local` at `build/flatpak/repo`, installs `net.catcrafts.CrafterBuild//local` from it next to whichever published branch is installed, and makes it the branch `flatpak run net.catcrafts.CrafterBuild` starts. `flatpak make-current --user net.catcrafts.CrafterBuild ` switches back. Before `crafter-build clean`, or before deleting the checkout, run `flatpak uninstall --user net.catcrafts.CrafterBuild//local` and `flatpak remote-delete --user crafter-build-local`; otherwise the remote is left pointing at a repo that is gone. ## Gotchas +- A new partition is listed in four places, in import order: `PARTITIONS` in `build.sh`, `CrafterBuildModules` in `Crafter.Build-Platform.cpp`, both arrays in `project.cpp`, and the umbrella `Crafter.Build.cppm`. Then `./build.sh`. - A `#pragma crafter dependency` target is rebuilt on every load on purpose: clang crashes rather than erroring when it imports a BMI whose own imported PCM was rebuilt underneath it, and the depfile check is what catches a regenerated host cache. Do not optimise the rebuild away. - A host-tooling library such as Crafter.LintRules may `import Crafter.Build;` without declaring it as a dependency, because `LoadProject` fills the cache it resolves from. Listing Crafter.Build in its `cfg.dependencies` would clone and build all of crafter-build for every consumer. Its test does link crafter-build, so that test is wired behind `IsTopLevelProject()`: without the guard, crafter-build loading the library as a dependency would load its own `project.cpp` again through the test's `GitProject` and hit the cycle guard. - `crafter-build clean` leaves a pragma dependency's build alone; it lives with the clone in the cache. diff --git a/README.md b/README.md index 07ae901..b7e0d46 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Crafter Build -A C++26 modules build system. Project descriptions are written in C++ with no JSON, no Lua, no embedded DSL. You write a `project.cpp` that constructs a `Configuration` and returns it; Crafter Build compiles, loads, and executes it. +A C++26 modules build system. Project descriptions are written in C++ with no JSON, no Lua, no embedded DSL. You write a `project.cpp` that constructs a `Configuration` and returns it. Crafter Build compiles, loads, and executes it. ## Install @@ -14,19 +14,6 @@ alias crafter-build='flatpak run net.catcrafts.CrafterBuild' Pre-release builds of `master` are the `beta` branch (`flatpak install catcrafts net.catcrafts.CrafterBuild//beta`), and every release also attaches a single-file bundle on the Forgejo releases page (`flatpak install --user crafter-build-.flatpak`). The sandbox sees the whole host filesystem, so projects can live anywhere. Binaries built for the host link the bundled `libc++.so.1` through an rpath into the Flatpak: they run inside it (`-r`, the test runner), and are not meant to be copied out onto the host. -## Building from source (Linux) - -The development loop is native. It needs `clang`, `cmake`, `git`, `lld` and `libc++` with its std module source: - -```bash -./build.sh # produces bin/crafter-build -./bin/crafter-build # rebuild via project.cpp -``` - -The compiler is resolved once per run: `CRAFTER_BUILD_LLVM_DIR` if set, else a `clang++` installed beside crafter-build (`/bin`), else PATH. `crafter-build --version` prints the version and the compiler in use; `crafter-build --help` lists the other overrides (`CRAFTER_BUILD_LIBCXX_DIR`, `CRAFTER_BUILD_LIBCLANG`, `CRAFTER_BUILD_WASI_SYSROOT`, `CRAFTER_MINGW_DIR`). A `Configuration` targets the host triple unless `cfg.target` says otherwise. - -The bootstrap targets the toolchain's own triple (`clang++ -print-target-triple`); override with `CRAFTER_BUILD_TARGET=`. The Flatpak is built from `packaging/flatpak/net.catcrafts.CrafterBuild.yaml`, which runs this same bootstrap, the self-host build and the test suite inside the sandbox against the bundled toolchain (`CRAFTER_BUILD_LLVM_DIR=/app`). - ## Writing a project.cpp Crafter Build loads a `project.cpp` from the current directory. The file exports one function that returns a populated `Configuration`: @@ -52,7 +39,7 @@ extern "C" Configuration CrafterBuildProject(std::span a } ``` -Run `crafter-build` in that directory; outputs land at `bin/myapp----/myapp` and intermediates at `build/myapp----/`. The hash covers everything that changes what is compiled (build type, defines, flags, sysroot, the compiler itself), so two such builds never share a directory. After a compiler update, a new crafter-build release included, the first build starts a fresh directory and recompiles; `crafter-build clean` removes the ones left behind. Add `-r` to run the produced executable straight after a successful build. +Run `crafter-build` in that directory, outputs land at `bin/myapp----/myapp` and intermediates at `build/myapp----/`. The hash covers everything that changes what is compiled (build type, defines, flags, sysroot, the compiler itself), so two such builds never share a directory. After a compiler update, a new crafter-build release included, the first build starts a fresh directory and recompiles. `crafter-build clean` removes the ones left behind. Add `-r` to run the produced executable straight after a successful build. ## Dependencies @@ -73,7 +60,7 @@ glslang.includeDirs = { "" }; glslang.libs = { "SPIRV", "glslang", "OSDependent" }; ``` -Clones live at `~/.cache/crafter.build/external/-/`, keyed by `(url, branch, commit, options)` so different projects sharing the same dep with the same spec reuse the clone and cmake build, while projects with diverging specs each get their own cache entry. The clone is resilient to partial-clone recovery and CMake reconfigure is triggered automatically on `options` changes. +Clones live at `~/.cache/crafter.build/external/-/`, keyed by `(url, branch, commit, options)` so different projects sharing the same dep with the same spec reuse the clone and cmake build, while projects with diverging specs each get their own cache entry. The clone is resilient to partial-clone recovery and CMake reconfigure is triggered automatically on `options` changes. Submodules are checked out at the revisions the fetched commit records, after every clone, switch, checkout and pull, so a pinned commit builds against its own submodules and not the default branch's. **Header-only git deps** — same as cmake deps but `builder = ExternalBuilder::None`. Just clones and propagates `-I` flags. @@ -88,7 +75,7 @@ Configuration* sibling = LocalProject({ .projectFile = "../Crafter.Math/project. cfg.dependencies = { math }; ``` -Clones share the external cache and identical specs share one `Configuration`. An unpinned source is `git pull`ed on every invocation; set `.commit` for reproducible, offline-after-first-fetch builds. +Clones share the external cache and identical specs share one `Configuration`. An unpinned source is `git pull`ed on every invocation, set `.commit` for reproducible, offline-after-first-fetch builds. A project loaded this way runs its `CrafterBuildProject` as a dependency, and `IsTopLevelProject()` is `false` while it does (`true` in the project crafter-build was pointed at, and outside any project file). Wire test-only dependencies behind it: a library whose test links crafter-build itself would otherwise load crafter-build's `project.cpp` from inside crafter-build and recurse. @@ -107,24 +94,6 @@ import Crafter.LintRules; #pragma crafter dependency "" [branch "" | commit ""] [project ""] ``` -## Install layout - -Inside the Flatpak: - -``` -/app/bin/crafter-build # the executable -/app/bin/clang++ lld llvm-ar wasmtime # the bundled toolchain the executable resolves first -/app/lib/libcrafter-build.a # static archive for downstream consumers -/app/share/crafter-build/*.cppm # module sources (rebuilt on the user's machine on first run) -/app/share/crafter-build/libc++-modules// - # std module sources of other libc++ releases, for sysroots without their own -/app/share/wasi-sysroot/ # WASI sysroot for wasm32 targets -~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build/---/ - # std.pcm + Crafter.Build*.pcm, built locally per toolchain -``` - -A source build uses the same shape under the checkout (`bin/`, `lib/`, `share/crafter-build/`) with the system compiler and `~/.cache/crafter.build/`. - ## Tests Tests live under `tests//`. The simplest case is a single C++ file: @@ -143,7 +112,7 @@ Test conventions: - **Exit codes**: `0` = ✅ pass, nonzero = ❌ fail, **`77`** = ⏭ skipped (autoconf convention; use it for runtime opt-outs like "tool not on PATH"). - Each test runs in its own subprocess — a segfault doesn't take the runner down. -- Default timeout is 60 s per test; runs in parallel up to `hardware_concurrency()`. +- Default timeout is 60 s per test, runs in parallel up to `hardware_concurrency()`. - Stdout+stderr is captured and printed atomically on completion. Failed/crashed/timed-out tests also write to `build/test-logs/.log`. - A broken fixture `project.cpp` is reported as a single failed test, not a fatal error — other tests still run. @@ -161,7 +130,7 @@ crafter-build test --timeout=5 # override per-t Outcomes: ✅ pass, ❌ fail, 💥 crash (with signal name), ⏱ timeout, ⏭ skipped. The runner exits 0 only if every non-skipped test passed. -`--target=` filters the run to tests whose `cfg.target` matches; default is the host triple. Per-target tests parse `--target=...` from their own args span and build for it; tests with hardcoded targets (e.g. host-only meta-tests) are filtered out under non-matching `--target=`. +`--target=` filters the run to tests whose `cfg.target` matches, default is the host triple. Per-target tests parse `--target=...` from their own args span and build for it, tests with hardcoded targets (e.g. host-only meta-tests) are filtered out under non-matching `--target=`. ### Cross-target test runners @@ -176,9 +145,9 @@ TestRunner::Cmd("wine") // local exec via prefix co For elaborate setups (rsync, ssh ControlMaster, env vars on the spawn, `QEMU_LD_PREFIX` for sysroots), construct `TestRunner` directly — `copy`/`exec`/`cleanup`/`probe` are public string fields with `{bundle}`, `{remote_bundle}`, `{bin}`, `{bin_name}`, `{args}` placeholders. -For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=` is passed to clang, and the std module is precompiled against the sysroot's libc++ headers. The module source has to come from the same libc++ release as those headers, so it is taken from `/usr/share/libc++/v1/std.cppm` when the sysroot ships it, else from the toolchain when both are the same major release, else from the sources crafter-build carries for other releases (`share/crafter-build/libc++-modules//`; `crafter-build --version` lists them). Alpine is the common case for the last two: its `libc++-dev` has the headers and no module source. When nothing matches, the build stops with one line naming both releases; installing that release's `share/libc++/v1` directory into the sysroot resolves it. The cheapest aarch64 sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root --config --dbpath /var/lib/pacman -Sy …` (point pacman at ALARM's mirrors via explicit config to avoid pulling host-arch packages). +For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=` is passed to clang, and the std module is precompiled against the sysroot's libc++ headers. Ordinary compiles are pointed at the same headers (`-nostdinc++ -isystem /usr/include/c++/v1`), so a unit that `#include`s a standard header gets the sysroot's libc++ and not the host toolchain's. The module source has to come from the same libc++ release as those headers, so it is taken from `/usr/share/libc++/v1/std.cppm` when the sysroot ships it, else from the toolchain when both are the same major release, else from the sources crafter-build carries for other releases (`share/crafter-build/libc++-modules//`, `crafter-build --version` lists them). Alpine is the common case for the last two: its `libc++-dev` has the headers and no module source. When nothing matches, the build stops with one line naming both releases, installing that release's `share/libc++/v1` directory into the sysroot resolves it. The cheapest aarch64 sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root --config --dbpath /var/lib/pacman -Sy …` (point pacman at ALARM's mirrors via explicit config to avoid pulling host-arch packages). -For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported; there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. +For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed, with libstdc++ 15 or newer for its std module. clang finds it by the `x86_64-w64-mingw32-gcc` on `PATH`; `CRAFTER_MINGW_DIR` names another prefix (the directory holding `x86_64-w64-mingw32/` and `lib/gcc/`), and inside the Flatpak the host's `/usr` is used when it has mingw-w64. Without one the build stops with a message saying where it looked. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported, there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. ### Configuring runners @@ -207,7 +176,7 @@ local cmd: # cmd:wine, cmd:qemu-aarch64, cmd:valgrind, ... ``` -Each runner carries a `probe` command (e.g. `which wine`); if it fails, every test using that runner is reported as ⏭ skipped and the rest of the suite continues. Skipped tests count toward the summary but don't fail the suite (`AllPassed()` returns true on skip-only runs). +Each runner carries a `probe` command (e.g. `which wine`), if it fails, every test using that runner is reported as ⏭ skipped and the rest of the suite continues. Skipped tests count toward the summary but don't fail the suite (`AllPassed()` returns true on skip-only runs). ### Linking the parent project's library @@ -234,7 +203,7 @@ extern "C" Configuration CrafterBuildProject(std::span) ## Linting & formatting -Lint rules are C++ in `project.cpp` with no `.lintrc`, no YAML, and no built-in rules. A rule is a named callback that runs once per source file; register it on the `Configuration` before returning: +Lint rules are C++ in `project.cpp` with no `.lintrc`, no YAML, and no built-in rules. A rule is a named callback that runs once per source file, register it on the `Configuration` before returning: ```cpp cfg.AddLintRule("no-tabs", [](Crafter::LintContext& ctx) { @@ -275,7 +244,7 @@ Lint findings print compiler-style — `file:line: warning: message [rule]` — | `Extension()` | file extension, e.g. `".cppm"` | | `CommentStripped()` | `content` with comments and string/char literal bodies blanked to spaces — newlines kept, so line numbers stay valid (raw strings not recognized yet) | | `Report(line, msg)` | record a finding (`line == 0` for whole-file) | -| `SetContent(str)` | replace the file's content — makes the rule a transform; `lines` re-splits, `CommentStripped()` re-derives. Build the new string, call once at the end; don't assign `content` directly | +| `SetContent(str)` | replace the file's content — makes the rule a transform, `lines` re-splits, `CommentStripped()` re-derives. Build the new string, call once at the end, don't assign `content` directly | Suppression is comment-based : @@ -287,6 +256,50 @@ Suppression is comment-based : // lint-disable-file — all rules, whole file ``` +## Scripts + +A project has chores that are not part of building it: packaging, installing, publishing, generating documentation. A script gives one a name. Scripts are C++ in `project.cpp` like everything else, a name and a callback pushed onto `cfg.scripts` before returning the `Configuration`: + +```cpp +cfg.scripts.push_back({"docs", [](Crafter::ScriptContext& ctx) { + return ctx.Run("python3 tools/gen-docs.py"); +}}); +``` + +```bash +crafter-build run # list the project's scripts +crafter-build run docs # run one +crafter-build run docs --open api # everything after the name is handed to the script +``` + +The callback returns the exit status, and `crafter-build run` exits with it. `ctx.Run(command)` runs a shell command with the terminal attached, so its output appears as it is produced and it can ask questions, and returns the command's exit code (128 plus the signal number when it was killed). A script with several steps stops at the first failure by returning that code: + +```cpp +cfg.scripts.push_back({"package", [](Crafter::ScriptContext& ctx) { + const std::string dist = Crafter::ShellQuote((ctx.projectDir / "dist").string()); + if (const std::int32_t code = ctx.Run(std::format("mkdir -p {}", dist)); code != 0) return code; + return ctx.Run(std::format("tar -czf {}/myapp.tar.gz -C {} .", dist, Crafter::ShellQuote(ctx.project.BinDir().string()))); +}}); +``` + +`ScriptContext` gives each script: + +| Member | Meaning | +|---|---| +| `project` | the `Configuration` that `project.cpp` returned. Read the project through this; a `cfg` captured by reference is gone by the time a script runs | +| `projectDir` | absolute directory of the project file | +| `args` | everything typed after the script's name, untouched | +| `Run(command)` | run a shell command with the terminal attached and return its exit code | + +Things to know: + +- Everything after the script's name belongs to the script, including what looks like an option of crafter-build (`--help`, `-v`). Options and project args for crafter-build go before the name: `crafter-build --debug run package`. +- `run` loads `project.cpp` but does not build the project. A script that needs the build output runs after `crafter-build`. +- Commands run in the directory crafter-build was started from. Build paths from `ctx.projectDir`, and quote anything that is not a literal with `Crafter::ShellQuote`. +- Only the scripts of the project you run crafter-build in are available. A dependency's scripts are not. +- Inside the Flatpak, commands run in its sandbox: the host's files are visible, but the tools on `PATH` are the ones the Flatpak ships (`git`, `cmake`, `python3`, `tar`, ...), not everything installed on the host. +- A return value outside 0 to 255 is not an exit status and becomes 1. So does a script that throws, with the exception's message printed. + ## Examples [examples/](examples/) contains progressively larger self-contained projects: @@ -306,7 +319,7 @@ This project is licensed under the GNU Lesser General Public License, version 3 The example code under [examples/](examples/) is licensed under the MIT license (see [examples/LICENSE](examples/LICENSE)), so it can be freely copied into your own projects. -The [lib/](lib/) directory contains vendored third-party code (nlohmann/json, glslang's DirStackFileIncluder) that remains under its own permissive licenses; see the headers of those files. +The [lib/](lib/) directory contains vendored third-party code (nlohmann/json, glslang's DirStackFileIncluder) that remains under its own permissive licenses, see the headers of those files. ## Copyright diff --git a/build.sh b/build.sh index 2d7df47..312abde 100755 --- a/build.sh +++ b/build.sh @@ -48,7 +48,7 @@ cp wasi-runtime/index.html.in share/crafter-build/wasi-runtime/ # Partition order is the import order: every unit is listed before the # units that import it, and the umbrella last. -PARTITIONS="Toolchain Shader Platform Interface Implementation External Clang Test Lint Progress Asset" +PARTITIONS="Toolchain Shader Platform Interface Implementation External Clang Test Lint Script Progress Asset" cp interfaces/Crafter.Build.cppm share/crafter-build/ for p in $PARTITIONS; do diff --git a/implementations/Crafter.Build-Clang.cpp b/implementations/Crafter.Build-Clang.cpp index d2aee22..416856e 100644 --- a/implementations/Crafter.Build-Clang.cpp +++ b/implementations/Crafter.Build-Clang.cpp @@ -12,6 +12,7 @@ import :Platform; import :Toolchain; import :Test; import :Lint; +import :Script; import :Progress; import :Asset; namespace fs = std::filesystem; @@ -296,6 +297,21 @@ CompileCommand Crafter::GetCompileCommand(const Configuration& config) { if (!config.sysroot.empty()) { out.command += std::format(" --sysroot={}", config.sysroot); + // Same reason as in BuildStdPcm: the driver ranks the host + // toolchain's own libc++ headers (/../include/c++/v1) above + // --sysroot. A unit that only does `import std;` never notices, but + // one that includes a standard header textually (vendored C++ such as + // Crafter.Asset's gdeflate) would get host libc++ configured for the + // host libc, which fails on a musl sysroot with "unknown rune table + // for this platform". Force the sysroot's copy. -nostdinc++ leaves + // -stdlib=libc++ unused on every compile ("argument unused during + // compilation"), hence the silencer; the price is that it also hides + // that diagnostic for a user's own misplaced flag (a -l or -Wl, + // option in compileFlags) on sysroot builds. Not for mingw, which + // uses libstdc++ and has no libc++ in its sysroot. + if (!isWasm && config.target != "x86_64-w64-mingw32") { + out.command += std::format(" -nostdinc++ -isystem {}/usr/include/c++/v1 -Wno-unused-command-line-argument", config.sysroot); + } } if (isWasm) { // -mllvm is consumed by codegen but not the link driver, which is the @@ -502,13 +518,31 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map buildCancelled{false}; + // Two entries with the same output name would compile in parallel into + // one file: whichever finished last won, the build reported success, and + // Check() then called both up to date. The usual way in is one source + // registered twice with different defines and no variant; a.vert next to + // a.frag is another. + { + std::unordered_map outputs; + for (const Shader& shader : config.shaders) { + const auto [existing, inserted] = outputs.emplace(shader.OutputName().string(), &shader); + if (inserted) continue; + buildError = std::format("shaders {} and {} both compile to {}: give one of them a distinct Shader::variant", existing->second->path.string(), shader.path.string(), existing->first); + buildCancelled.store(true); + break; + } + } for (const Shader& shader : config.shaders) { - if (shader.Check(outputDir)) continue; - threads.emplace_back([&shader, &outputDir, &shaderIncludeDirs, &buildError, &buildCancelled]() { - Progress::Task task(std::format("Compiling shader {}", shader.path.filename().string())); + if (buildCancelled.load(std::memory_order_relaxed)) break; + if (shader.Check(outputDir, buildDir, shaderIncludeDirs)) continue; + threads.emplace_back([&shader, &outputDir, &buildDir, &shaderIncludeDirs, &buildError, &buildCancelled]() { + // A source registered more than once shows up once per variant; + // name the variant so the lines can be told apart. + Progress::Task task(shader.variant.empty() ? std::format("Compiling shader {}", shader.path.filename().string()) : std::format("Compiling shader {} ({})", shader.path.filename().string(), shader.variant)); if (buildCancelled.load(std::memory_order_relaxed)) return; - std::string result = shader.Compile(outputDir, shaderIncludeDirs); + std::string result = shader.Compile(outputDir, buildDir, shaderIncludeDirs); if (result.empty()) return; bool expected = false; @@ -665,7 +699,13 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map libSet; - std::unordered_set publicFlagSet; + std::set libSet; + std::set publicFlagSet; std::mutex fileMutex; std::vector depThreads; depThreads.reserve(config.dependencies.size()); @@ -770,6 +810,9 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_mapBinDir(); for (const Shader& shader : dep->shaders) { - fs::path src = depBinDir / shader.path.filename().replace_extension("spv"); + fs::path src = depBinDir / shader.OutputName(); if (!fs::exists(src)) continue; copyTree(src, outputDir / src.filename()); } @@ -1115,8 +1158,48 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map.dll + lib.dll.a (lld --out-implib) + // unix: lib.so (rpath $ORIGIN) + fs::path dll = outputDir / std::format("{}.dll", config.outputName); + fs::path implib = outputDir / std::format("lib{}.dll.a", config.outputName); + linkCommand = std::format("{}{} -shared -o {} -Wl,--out-implib,{} -fuse-ld=lld{}", command, files, dll.string(), implib.string(), linkExtras); + } else { + linkCommand = std::format("{}{} -shared -o {}.so -Wl,-rpath,'$ORIGIN' -fuse-ld=lld{}", command, files, (outputDir/(std::string("lib")+config.outputName)).string(), linkExtras); + } + const fs::path linkRecord = buildDir / std::format("{}.link", config.outputName); + if (!buildResult.repack) { + std::ifstream recorded(linkRecord, std::ios::binary); + const std::string previous((std::istreambuf_iterator(recorded)), std::istreambuf_iterator()); + if (previous != linkCommand) buildResult.repack = true; + } + if(buildResult.repack) { Progress::Task task(std::format("Linking {}", config.outputName)); + // The record goes first, so a link that fails leaves it stale. + std::error_code removed; + fs::remove(linkRecord, removed); if(config.type == ConfigurationType::Executable) { if(config.target == "x86_64-w64-mingw32") { try { @@ -1147,30 +1230,10 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map.dll + lib.dll.a (lld --out-implib) - // unix: lib.so (rpath $ORIGIN) - if (config.target == "x86_64-w64-mingw32") { - fs::path dll = outputDir / std::format("{}.dll", config.outputName); - fs::path implib = outputDir / std::format("lib{}.dll.a", config.outputName); - buildResult.result = RunBuildCommand(std::format("{}{} -shared -o {} -Wl,--out-implib,{} -fuse-ld=lld{}", command, files, dll.string(), implib.string(), linkExtras)); - } else { - buildResult.result = RunBuildCommand(std::format("{}{} -shared -o {}.so -Wl,-rpath,'$ORIGIN' -fuse-ld=lld{}", command, files, (outputDir/(std::string("lib")+config.outputName)).string(), linkExtras)); - } + } + buildResult.result = RunBuildCommand(linkCommand); + if (buildResult.result.empty()) { + std::ofstream(linkRecord, std::ios::binary | std::ios::trunc) << linkCommand; } } @@ -1561,6 +1624,7 @@ R"(Usage: {0} test [test-options] [globs...] Build and run the project's tests {0} lint [lint-options] [globs...] Run the project's lint rules over its sources {0} format [format-options] [globs...] Apply the project's transform rules (rewrites files) + {0} run [script [args...]] Run a script the project defines, or list them {0} clean Delete the project's bin/ and build/ trees {0} help | -h | --help Show this help {0} --version Print the version and the compiler in use @@ -1613,6 +1677,23 @@ Format options (after the `format` subcommand): transform. `format` rewrites changed files in place; `lint` reports the same transforms as would-reformat findings without writing. +Run (after the `run` subcommand): +