Crafter.Build/README.md

# Crafter Build

A C++26 modules build system. Project descriptions are written in C++ — no JSON, no Lua, no embedded DSL. You write a `project.cpp` that constructs a `Configuration` and returns it; Crafter Build compiles, loads, and executes it.

## Status

- **Linux**: working end-to-end, packageable as a distro package. Verified on Arch Linux via [PKGBUILD](PKGBUILD) in a fresh container.
- **Windows**: working end-to-end. Bootstrap (`build.cmd`) produces `crafter-build.exe` + `crafter-build.dll` + `crafter-build.lib`; user `project.cpp` links against the import lib via the host-controlled DLL boundary.

## Quick start (Linux)

Bootstrap requires `clang`, `cmake`, `git`, `lld`, and `libc++`.

```bash
./build.sh                                                              # produces bin/crafter-build
CRAFTER_BUILD_HOME=$PWD/build ./bin/crafter-build                       # rebuild via project.cpp
```

For distro-packaged installs, `crafter-build` finds its modules at `<prefix>/share/crafter-build/` automatically — no env var required.

To build the system as a distro package on Arch:

```bash
makepkg -si        # uses CRAFTER_BUILD_MARCH=x86-64-v3 by default for portability
```

## Writing a project.cpp

Crafter Build loads a `project.cpp` from the current directory. The file exports one function that returns a populated `Configuration`:

```cpp
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;

extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> args) {
    Configuration cfg;
    cfg.path = "./";
    cfg.name = "myapp";
    cfg.outputName = "myapp";
    cfg.target = "x86_64-pc-linux-gnu";
    cfg.type = ConfigurationType::Executable;

    std::array<fs::path, 1> ifaces = { "interfaces/Hello" };
    std::array<fs::path, 2> impls  = { "implementations/Hello", "implementations/main" };
    cfg.GetInterfacesAndImplementations(ifaces, impls);

    return cfg;
}
```

Run `crafter-build` in that directory; outputs land at `bin/myapp-<target>-<march>/myapp` and intermediates at `build/myapp-<target>-<march>/`.

## Dependencies

Three kinds, all fetched/built incrementally:

**Cross-project Crafter sub-projects** — point `cfg.dependencies` at another `Configuration*`. Cross-project module imports (`import OtherProject;`) are tracked per-import: only the modules that actually import a changed dep rebuild.

**External git+cmake deps** — for things like glslang. Declare in `cfg.externalDependencies`:

```cpp
ExternalDependency& glslang = cfg.externalDependencies.emplace_back();
glslang.name = "glslang";
glslang.source.url = "https://github.com/KhronosGroup/glslang.git";
glslang.source.branch = "main";
glslang.builder = ExternalBuilder::CMake;
glslang.options = { "-DENABLE_OPT=OFF" };
glslang.includeDirs = { "" };
glslang.libs = { "SPIRV", "glslang", "OSDependent" };
```

Clones live at `~/.cache/crafter.build/external/<name>-<hash>/`, keyed by `(url, branch, commit, options)` so different projects sharing the same dep with the same spec reuse the clone and cmake build, while projects with diverging specs each get their own cache entry. The clone is resilient to partial-clone recovery and CMake reconfigure is triggered automatically on `options` changes.

**Header-only git deps** — same as cmake deps but `builder = ExternalBuilder::None`. Just clones and propagates `-I` flags.

## Install layout

```
<prefix>/bin/crafter-build               # the executable
<prefix>/lib/libcrafter-build.a          # static archive for downstream consumers
<prefix>/share/crafter-build/*.cppm      # module sources (rebuilt on user machine on first run)
~/.cache/crafter.build/<target>-<march>/ # std.pcm + Crafter.Build*.pcm, built locally
```

Sources ship instead of PCMs because libc++ ABI varies between machines — the user's machine builds its own PCMs against its own libc++ on first run (~6-8s one-time cost), making the install resilient to libc++ version differences across distros.

The static archive is for downstream projects that want to use Crafter.Build *as a library* (construct `Configuration` objects, call `Build()`, etc., directly in C++) without invoking the CLI. Link with `-lcrafter-build` plus the project's external dep libs (`-lSPIRV -lglslang -lOSDependent -lMachineIndependent -lglslang-default-resource-limits -lGenericCodeGen`) and `-ldl` (for `LoadProject`'s dlopen path on Linux). On Windows the equivalent is `bin/crafter-build-static.lib`.

## Build incrementality

Per-import precise tracking for both within-project and cross-project module dependencies:

- Touch `lib/Hello.cppm` → only consumers of `Hello` rebuild.
- Touch `lib/Other.cppm` → only consumers of `Other` rebuild.
- External CMake dep produces fresh `.a` files → whole project rebuilds (deliberately coarse — cmake-dep changes are rare).

Diamond deps (`A → {B, C}; B → X; C → X`) build `X` exactly once via a `std::shared_future<BuildResult>` cache.

## Tests

Tests are normal executables — any `int main()`, exit code `0` = pass, nonzero = fail, captured stdout+stderr is the failure message. There's no Crafter-specific entry-point convention, so a GoogleTest/Catch2/doctest binary works as-is.

Declare tests by pushing onto `cfg.tests` in your `project.cpp`:

```cpp
Test t;
t.config.path = "./";
t.config.name = "VectorMath";
t.config.outputName = "VectorMath";
t.config.target = "x86_64-pc-linux-gnu";
t.config.type = ConfigurationType::Executable;
std::array<fs::path, 0> ifaces = {};
std::array<fs::path, 1> impls  = { "tests/VectorMath" };
t.config.GetInterfacesAndImplementations(ifaces, impls);
cfg.tests.push_back(std::move(t));
```

Each test runs in its own subprocess so a segfault doesn't take the runner down. Default timeout is 60 s per test (`--timeout=N` overrides). Tests run in parallel up to `hardware_concurrency()`; pass `--jobs=N` to override. Per-test stdout+stderr is buffered and printed atomically on completion so parallel output stays readable, and is also written to `build/test-logs/<name>.log` when a test fails, crashes, or times out.

```bash
crafter-build test                  # build + run all tests
crafter-build test 'Vector*'        # glob-filter test names
crafter-build test --list           # enumerate without running
crafter-build test --jobs=1         # serial mode
crafter-build test --timeout=5      # override per-test timeout
```

Outcomes are reported as ✅ pass, ❌ fail, 💥 crash (with the signal name), or ⏱ timeout. The runner exits 0 only if every test passed.

### Cross-target test runners

`Test::runner` runs the test binary somewhere other than this machine. Three factories ship today (Linux host only):

```cpp
t.runner = TestRunner::Local();                                  // default
t.runner = TestRunner::QemuUser("qemu-aarch64");                 // cross-arch via qemu user-mode
t.runner = TestRunner::Ssh("winvm", "/tmp/crafter-test");        // cross-OS via scp + ssh
```

`TestRunner::Ssh(host, remoteDir)` ships the test's whole `bin/<name>-<target>-<march>/` output dir via `scp -r`, runs the binary over `ssh`, then cleans up the remote temp dir. Each test invocation gets a unique remote subdir to keep parallel runs isolated. Assumes Unix-like shell on the remote (`mkdir -p`, `rm -rf`); for Windows remotes (e.g. `cmd.exe`) construct `TestRunner` directly with your own command templates — `copy`, `exec`, `cleanup` are public string fields with `{bundle}`, `{remote_bundle}`, `{bin}`, `{bin_name}`, `{args}` placeholders.

`TestRunner::QemuUser(qemuBin)` simply wraps `<qemuBin> {bin} {args}` — same machine, just a command prefix. Pair it with `t.config.target = "aarch64-linux-gnu"` (plus an aarch64 sysroot installed on the host) for cross-arch Linux testing.

For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=<path>` is passed to clang, and `<sysroot>/usr/share/libc++/v1/std.cppm` is precompiled instead of the host's.

For aarch64 specifically, the cheapest sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root <rootfs> --config <alarm-pacman.conf> --dbpath <rootfs>/var/lib/pacman -Sy …` (point pacman at ALARM's mirrors with an explicit config to avoid pulling host-arch packages). Then `cfg.sysroot = "<rootfs>";` plus a manual `TestRunner` whose exec template prepends `QEMU_LD_PREFIX=<rootfs> qemu-aarch64 …` so qemu-user finds `ld-linux-aarch64.so.1`. See `tests/fixtures/cross-arch-aarch64/` for a working example.

For `t.config.target = "x86_64-w64-mingw32"` (Windows from Linux) — V2's existing build pipeline cross-compiles on Linux as long as `mingw-w64` is installed. Combine with `TestRunner::Ssh("winvm", ...)` for end-to-end Windows-target testing from a Linux host. Full MSVC ABI (`x86_64-pc-windows-msvc`) cross-compile from Linux is not yet wired; use MinGW for now.

#### Global config: `TestRunner::FromEnv`

To avoid hardcoding hosts in every project's `project.cpp`, use `TestRunner::FromEnv(target)`:

```cpp
t.runner = TestRunner::FromEnv(t.config.target);
```

This reads the env var `CRAFTER_BUILD_RUNNER_<NORMALIZED_TARGET>` (target triple with `-` and `.` replaced by `_`). Set it once per shell or in CI:

```bash
export CRAFTER_BUILD_RUNNER_x86_64_w64_mingw32=ssh:winvm:/tmp/crafter-test
export CRAFTER_BUILD_RUNNER_aarch64_linux_gnu=qemu:qemu-aarch64
```

Spec grammar: `local`, `qemu:<qemu-bin>`, `ssh:<host>`, `ssh:<host>:<remoteDir>`, `sshwin:<host>`, `sshwin:<host>:<remoteDir>` (cmd.exe-shell variant for Windows remotes). Unset → falls back to the second arg of `FromEnv` (defaults to `Local()`).

For anything more elaborate (rsync, ssh ControlMaster, custom flags, env vars on the spawn, `QEMU_LD_PREFIX` for sysroots) — construct `TestRunner` manually with your own templates. The factories are sugar over the same fields.

Not yet supported: WSL2 / Windows-host execution (the runner currently requires a Linux host because `RunCommandWithTimeout` is Linux-only), MSVC ABI cross-compile from Linux.

#### Skipping unreachable runners

`TestRunner` carries a `probe` command that runs once per `RunTests` invocation (cached by `runner.name`); if it exits non-zero, every Test using that runner reports as `⏭ Skipped: runner '<name>' not available` and the suite continues with everything else. The factories set sensible defaults: `Ssh`/`SshWin` probe via `ssh -q -o BatchMode=yes -o ConnectTimeout=5 <host> …`, `QemuUser` via `which <qemu-bin>`, `Local` is unconditionally available.

Effect: if you don't have a Windows VM at all, you don't have to do anything — the Windows variants in the in-tree suite skip cleanly and the Linux variants pass. If you do have one, set up an SSH host alias matching what the project.cpp expects (in this repo: `winvm` in `~/.ssh/config`) and the Windows variants run automatically. Skipped tests count toward the summary but do not fail the suite (`AllPassed()` returns true on skip-only runs).

#### Per-(target, runner) tests

The 5 simple tests `HelloWorld`, `WithModule`, `Defines`, `CrossProjectModule`, `Diamond` are declared **directly** in the project's `project.cpp`, one entry per `(target, runner)` pair:

```cpp
std::vector<TargetRunner> targets = {
    { "x86_64-pc-linux-gnu", TestRunner::Local(), {} },
    { "x86_64-w64-mingw32",  TestRunner::SshWin("winvm", "C:/temp/crafter-tests"), {"-lstdc++exp"} },
};
```

So `./bin/crafter-build test` (no env vars) spawns one Test per `(target, runner)` pair, all in one report. With the `winvm` SSH alias reachable:

```
✅ HelloWorld (5ms)
✅ HelloWorld (sshwin:winvm) (2348ms)
✅ WithModule (3ms)
✅ WithModule (sshwin:winvm) (2204ms)
…
18 passed
```

Without `winvm` reachable, the SshWin variants skip:

```
✅ HelloWorld (5ms)
⏭  HelloWorld (sshwin:winvm) skipped: runner 'sshwin:winvm' not available
…
```

Local runner emits no parenthetical (existing default for the common case); cross-target runners show their name (e.g. `sshwin:winvm`, `qemu:qemu-aarch64`) so you see at a glance which environment ran each test.

The 8 outer-driver tests (Incremental, BuildError, RunnerClassification, QemuUser, SshRunner, WindowsViaSsh, CrossArchAarch64, Libraries) stay one-line in the report because they're meta-tests — the test driver is the test, validating host-side build behavior or runner integration. They're not multi-target.

## Architecture

- **Modules**: `Crafter.Build:Shader` / `:Platform` / `:Interface` / `:Implementation` / `:External` / `:Clang` / `:Test` partitions, re-exported by the `Crafter.Build` umbrella.
- **Build process**: parallel — interface PCMs, implementation `.o` files, external dep clones, dep-config recursive builds, and shader compilations all spawn threads, sync at well-defined join points.
- **`LoadProject`** compiles `project.cpp` to a shared object (`.so` on Linux, `.dll` on Windows) and loads it. On Linux the host exe is linked with `-Wl,--export-dynamic` so the project's undefined symbols resolve back into the running binary. On Windows the host is split into `crafter-build.dll` (everything) + a thin `crafter-build.exe` launcher; the public `Crafter::*` API is annotated with a `CRAFTER_API` macro (`__declspec(dllexport)` when the host is built, `__declspec(dllimport)` when project.dll consumes it via the rebuilt user-cache PCMs), and the host passes `-Wl,/EXPORT:CrafterBuildProject` when linking project.dll so user `project.cpp` files don't need platform-gated annotations.

## Compatibility / portability

- Linux: x86_64. Tested on Arch in a clean container.
- Windows: x86_64 MSVC ABI. `build.cmd` produces `bin/crafter-build.{exe,dll,lib}`; userprojs verified end-to-end (`project.cpp` → `project.dll` loaded by the host → `userapp.exe` built and run).

The `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` env vars override the default `-march=native` / `-mtune=native` for distro CI builds:

```bash
CRAFTER_BUILD_MARCH=x86-64-v3 CRAFTER_BUILD_MTUNE=generic ./build.sh
```

## Roadmap

- [ ] Target-triple auto-detection (`clang -print-target-triple`)
- [ ] Prefer system glslang when available, fall back to git+cmake
- [ ] Order-preserving deduplication of `BuildResult.libs` (current `unordered_set` doesn't preserve link order — only matters for cyclic static libs)

## License

LGPL-3.0-only. See [LICENSE](LICENSE).