Crafter.Build/README.md

# Crafter Build

A C++26 modules build system. Project descriptions are written in C++ — no JSON, no Lua, no embedded DSL. You write a `project.cpp` that constructs a `Configuration` and returns it; Crafter Build compiles, loads, and executes it.

## Status

- **Linux**: working end-to-end, packageable as a distro package. Verified on Arch Linux via [PKGBUILD](PKGBUILD) in a fresh container.
- **Windows**: working end-to-end, including the test runner. Bootstrap (`build.cmd`) produces `crafter-build.exe` + `crafter-build.dll` + `crafter-build.lib`; user `project.cpp` links against the import lib via the host-controlled DLL boundary. `crafter-build test` runs natively on Windows hosts; cross-OS runs back to Linux via WSL2 (`--runner=wsl`).

## Quick start (Linux)

Bootstrap requires `clang`, `cmake`, `git`, `lld`, and `libc++`.

```bash
./build.sh                                                              # produces bin/crafter-build
CRAFTER_BUILD_HOME=$PWD/build ./bin/crafter-build                       # rebuild via project.cpp
```

For distro-packaged installs, `crafter-build` finds its modules at `<prefix>/share/crafter-build/` automatically — no env var required.

To build the system as a distro package on Arch:

```bash
makepkg -si        # uses CRAFTER_BUILD_MARCH=x86-64-v3 by default for portability
```

## Quick start (Windows)

Two ways to get a working `crafter-build` on Windows: native build via `build.cmd` (MSVC ABI), or download the cross-compiled mingw artifact from CI (mingw ABI). They produce different binaries with different ABI requirements; pick one and stick with it.

**Native MSVC build** — what `build.cmd` does, what CI doesn't ship.

Need clang+lld targeting `x86_64-pc-windows-msvc` and a Windows libc++ install pointed to by `LIBCXX_DIR`. Then `build.cmd` produces `bin/{crafter-build.exe, crafter-build.dll, crafter-build.lib}`.

**Cross-compiled mingw artifact** — what CI ships from the Linux runner.

The CI's Windows zip contains `crafter-build.exe` + `crafter-build.dll` + `libcrafter-build.dll.a`. The DLL is **statically linked against libstdc++/libgcc/libwinpthread**, so it doesn't depend on a particular runtime DLL being on the consumer's PATH. To use it for builds the consumer needs the mingw-w64 sysroot installed, since `crafter-build.exe` invokes the user's clang to compile their `project.cpp`:

```powershell
# one-time setup
winget install MSYS2.MSYS2
C:\msys64\usr\bin\pacman.exe -S --noconfirm mingw-w64-ucrt-x86_64-toolchain
# add C:\msys64\ucrt64\bin to PATH (so user-built executables can find msys2 clang)

# then any project with a project.cpp:
cd <project-dir>
crafter-build.exe -r
```

Match the toolchain flavor to the artifact: **UCRT mingw** (`mingw-w64-ucrt-x86_64-toolchain`, sysroot at `C:\msys64\ucrt64`) — the artifact is cross-compiled with UCRT, the older MSVCRT mingw won't ABI-match. Override the auto-detected sysroot path with `CRAFTER_MINGW_DIR=...` if you have mingw-w64 installed somewhere else.

A project.cpp built on Windows by this artifact gets compiled with `--target=x86_64-w64-mingw32` by default, producing self-contained mingw exes (libstdc++/libgcc/libwinpthread statically linked, no runtime DLLs alongside). For MSVC-ABI output set `cfg.target = "x86_64-pc-windows-msvc"` in your `project.cpp` *and* point `LIBCXX_DIR` at a Windows libc++ install — same prerequisite as the native build.

## Writing a project.cpp

Crafter Build loads a `project.cpp` from the current directory. The file exports one function that returns a populated `Configuration`:

```cpp
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;

extern "C" Configuration CrafterBuildProject(std::span<const std::string_view> args) {
    Configuration cfg;
    cfg.path = "./";
    cfg.name = "myapp";
    cfg.outputName = "myapp";
    cfg.target = "x86_64-pc-linux-gnu";
    cfg.type = ConfigurationType::Executable;

    std::array<fs::path, 1> ifaces = { "interfaces/Hello" };
    std::array<fs::path, 2> impls  = { "implementations/Hello", "implementations/main" };
    cfg.GetInterfacesAndImplementations(ifaces, impls);

    return cfg;
}
```

Run `crafter-build` in that directory; outputs land at `bin/myapp-<target>-<march>/myapp` and intermediates at `build/myapp-<target>-<march>/`. Add `-r` to run the produced executable straight after a successful build (host targets only — for cross targets use `crafter-build test --runner=...`).

## Dependencies

Three kinds, all fetched/built incrementally:

**Cross-project Crafter sub-projects** — point `cfg.dependencies` at another `Configuration*`. Cross-project module imports (`import OtherProject;`) are tracked per-import: only the modules that actually import a changed dep rebuild.

**External git+cmake deps** — for things like glslang. Declare in `cfg.externalDependencies`:

```cpp
ExternalDependency& glslang = cfg.externalDependencies.emplace_back();
glslang.name = "glslang";
glslang.source.url = "https://github.com/KhronosGroup/glslang.git";
glslang.source.branch = "main";
glslang.builder = ExternalBuilder::CMake;
glslang.options = { "-DENABLE_OPT=OFF" };
glslang.includeDirs = { "" };
glslang.libs = { "SPIRV", "glslang", "OSDependent" };
```

Clones live at `~/.cache/crafter.build/external/<name>-<hash>/`, keyed by `(url, branch, commit, options)` so different projects sharing the same dep with the same spec reuse the clone and cmake build, while projects with diverging specs each get their own cache entry. The clone is resilient to partial-clone recovery and CMake reconfigure is triggered automatically on `options` changes.

**Header-only git deps** — same as cmake deps but `builder = ExternalBuilder::None`. Just clones and propagates `-I` flags.

## Install layout

```
<prefix>/bin/crafter-build               # the executable
<prefix>/lib/libcrafter-build.a          # static archive for downstream consumers
<prefix>/share/crafter-build/*.cppm      # module sources (rebuilt on user machine on first run)
~/.cache/crafter.build/<target>-<march>/ # std.pcm + Crafter.Build*.pcm, built locally
```

Sources ship instead of PCMs because libc++ ABI varies between machines — the user's machine builds its own PCMs against its own libc++ on first run (~6-8s one-time cost), making the install resilient to libc++ version differences across distros.

The static archive is for downstream projects that want to use Crafter.Build *as a library* (construct `Configuration` objects, call `Build()`, etc., directly in C++) without invoking the CLI. Link with `-lcrafter-build` plus the project's external dep libs (`-lSPIRV -lglslang -lOSDependent -lMachineIndependent -lglslang-default-resource-limits -lGenericCodeGen`) and `-ldl` (for `LoadProject`'s dlopen path on Linux). On Windows the equivalent is `bin/crafter-build-static.lib`.

## Build incrementality

Per-import precise tracking for both within-project and cross-project module dependencies:

- Touch `lib/Hello.cppm` → only consumers of `Hello` rebuild.
- Touch `lib/Other.cppm` → only consumers of `Other` rebuild.
- External CMake dep produces fresh `.a` files → whole project rebuilds (deliberately coarse — cmake-dep changes are rare).

Diamond deps (`A → {B, C}; B → X; C → X`) build `X` exactly once via a `std::shared_future<BuildResult>` cache.

## Tests

Tests live under `tests/<Name>/`. The simplest case is a single C++ file:

```
tests/Smoke/main.cpp        # int main() — exit 0 = pass, nonzero = fail, 77 = skipped
```

Run them all:

```bash
crafter-build test
```

Auto-discovery walks `tests/` one level deep. Three escalation layers:

1. **`tests/<Name>/main.cpp`** with no `project.cpp` — a Configuration is synthesized: top-level `*.cpp` files become implementations, `interfaces/*.cppm` become module interfaces, target = the run's `--target=` (host triple by default). Most tests need nothing more.
2. **`tests/<Name>/project.cpp`** — full control. Use this for tests with defines, dependencies, or a non-default target.
3. Folders starting with `_` or `.` are skipped, so `tests/_shared/` can hold cross-test code without becoming a test itself.

Test conventions:

- **Exit codes**: `0` = ✅ pass, nonzero = ❌ fail, **`77`** = ⏭ skipped (autoconf convention; use it for runtime opt-outs like "tool not on PATH").
- Each test runs in its own subprocess — a segfault doesn't take the runner down.
- Default timeout is 60 s per test; runs in parallel up to `hardware_concurrency()`.
- Stdout+stderr is captured and printed atomically on completion. Failed/crashed/timed-out tests also write to `build/test-logs/<name>.log`.
- A broken fixture `project.cpp` is reported as a single failed test, not a fatal error — other tests still run.

CLI:

```bash
crafter-build test                                              # run all
crafter-build test 'Unit*'                                      # glob-filter test names
crafter-build test --list                                       # enumerate without running
crafter-build test --target=x86_64-w64-mingw32                  # only tests for this target
crafter-build test --target=x86_64-w64-mingw32 --runner=cmd:wine
crafter-build test --jobs=1                                     # serial mode
crafter-build test --timeout=5                                  # override per-test timeout
```

Outcomes: ✅ pass, ❌ fail, 💥 crash (with signal name), ⏱ timeout, ⏭ skipped. The runner exits 0 only if every non-skipped test passed.

`--target=` filters the run to tests whose `cfg.target` matches; default is the host triple. Per-target tests parse `--target=...` from their own args span and build for it; tests with hardcoded targets (e.g. host-only meta-tests) are filtered out under non-matching `--target=`.

### Cross-target test runners

`Test::runner` runs the test binary somewhere other than this machine. Four factories:

```cpp
TestRunner::Local()                                  // run on this machine (default)
TestRunner::Cmd("wine")                              // local exec via prefix command
TestRunner::Ssh("hostname")                          // remote linux via scp + ssh
TestRunner::SshWin("winhost")                        // remote windows via scp + ssh-cmd.exe
TestRunner::Wsl()                                    // Linux binaries via WSL2 (Windows host)
```

`Cmd(cmd)` is a generic `<cmd> <bin> <args>` prefix runner — covers wine, qemu-user, valgrind, gdbserver, anything with that shape. Probes via `which <cmd>` (or `where` on Windows hosts). The `Ssh` variants `scp -r` the test's whole bin dir, run the binary remotely, then clean up; each invocation uses a unique remote subdir so parallel runs don't collide. `Wsl()` mirrors the SSH model but copies the bundle into WSL's native filesystem (faster than executing in-place from `/mnt/c`) — useful when a Windows host wants to exercise Linux test binaries.

For elaborate setups (rsync, ssh ControlMaster, env vars on the spawn, `QEMU_LD_PREFIX` for sysroots), construct `TestRunner` directly — `copy`/`exec`/`cleanup`/`probe` are public string fields with `{bundle}`, `{remote_bundle}`, `{bin}`, `{bin_name}`, `{args}` placeholders.

For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=<path>` is passed to clang, and `<sysroot>/usr/share/libc++/v1/std.cppm` is precompiled instead of the host's. The cheapest aarch64 sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root <rootfs> --config <alarm-pacman.conf> --dbpath <rootfs>/var/lib/pacman -Sy …` (point pacman at ALARM's mirrors via explicit config to avoid pulling host-arch packages). See `tests/CrossArchAarch64/inner/` for a working sysroot+qemu example.

For `--target=x86_64-w64-mingw32` (Windows from Linux), V2's pipeline cross-compiles as long as `mingw-w64` is installed. Pair it with `--runner=cmd:wine` (no VM needed) or `--runner=sshwin:winhost` (real Windows). Full MSVC ABI (`x86_64-pc-windows-msvc`) cross-compile from Linux is not yet wired; use MinGW for now. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags.

### Configuring runners

Two layers, CLI wins over env, both override the per-test default of `Local()`:

**Env var** (persistent, per shell or in CI):

```bash
export CRAFTER_BUILD_RUNNER_x86_64_w64_mingw32=cmd:wine
export CRAFTER_BUILD_RUNNER_aarch64_linux_gnu=cmd:qemu-aarch64
crafter-build test --target=x86_64-w64-mingw32           # uses cmd:wine
```

The variable name is `CRAFTER_BUILD_RUNNER_<NORMALIZED_TARGET>` (target triple with `-` and `.` replaced by `_`).

**CLI** (per invocation, scoped to `--target=`):

```bash
crafter-build test --target=x86_64-w64-mingw32 --runner=cmd:wine
```

Spec grammar:

```
local
cmd:<command>                  # cmd:wine, cmd:qemu-aarch64, cmd:valgrind, ...
ssh:<host>[:<remoteDir>]
sshwin:<host>[:<remoteDir>]    # cmd.exe-shell variant for Windows remotes
wsl[:<remoteDir>]              # Windows host -> Linux binaries via WSL2
```

Each runner carries a `probe` command (e.g. `which wine`, `ssh -BatchMode=yes <host> true`); if it fails, every test using that runner is reported as ⏭ skipped and the rest of the suite continues. Skipped tests count toward the summary but don't fail the suite (`AllPassed()` returns true on skip-only runs).

### Linking the parent project's library

A test fixture can depend on the project's own library so it can `import` and exercise its API:

```cpp
// tests/UnitFoo/project.cpp
extern "C" Configuration CrafterBuildProject(std::span<const std::string_view>) {
    Configuration cfg;
    cfg.path = "tests/UnitFoo/";
    cfg.name = "UnitFoo";
    cfg.outputName = "UnitFoo";
    cfg.target = "x86_64-pc-linux-gnu";
    cfg.type = ConfigurationType::Executable;
    cfg.dependencies = { ParentLib("MyLib") };       // <-- the magic line

    std::array<fs::path, 0> ifaces = {};
    std::array<fs::path, 1> impls = { "main" };
    cfg.GetInterfacesAndImplementations(ifaces, impls);
    return cfg;
}
```

`ParentLib(name)` walks the parent project's dependency graph (and the parent itself) to find a `Configuration*` by `Configuration::name`. The build links it into the test exe, so `import MyLib;` resolves naturally. See [examples/tests/](examples/tests/) for the complete worked example.

## Examples

[examples/](examples/) contains progressively larger self-contained projects:

| Example | Shows |
|---|---|
| [hello-world](examples/hello-world/) | Smallest possible project: `project.cpp` + `main.cpp` |
| [with-module](examples/with-module/) | Adding a C++ module interface |
| [library](examples/library/) | Library + executable, linked via `cfg.dependencies` |
| [tests](examples/tests/) | Auto-discovered tests + tests linking the parent library |

Each builds standalone: `cd examples/<name> && crafter-build`.

## Architecture

- **Modules**: `Crafter.Build:Shader` / `:Platform` / `:Interface` / `:Implementation` / `:External` / `:Clang` / `:Test` partitions, re-exported by the `Crafter.Build` umbrella.
- **Build process**: parallel — interface PCMs, implementation `.o` files, external dep clones, dep-config recursive builds, and shader compilations all spawn threads, sync at well-defined join points.
- **`LoadProject`** compiles `project.cpp` to a shared object (`.so` on Linux, `.dll` on Windows) and loads it. On Linux the host exe is linked with `-Wl,--export-dynamic` so the project's undefined symbols resolve back into the running binary. On Windows the host is split into `crafter-build.dll` (everything) + a thin `crafter-build.exe` launcher; the public `Crafter::*` API is annotated with a `CRAFTER_API` macro (`__declspec(dllexport)` when the host is built, `__declspec(dllimport)` when project.dll consumes it via the rebuilt user-cache PCMs), and the host passes `-Wl,/EXPORT:CrafterBuildProject` when linking project.dll so user `project.cpp` files don't need platform-gated annotations.

## Compatibility / portability

- Linux: x86_64. Tested on Arch in a clean container.
- Windows: x86_64 MSVC ABI. `build.cmd` produces `bin/crafter-build.{exe,dll,lib}`; userprojs verified end-to-end (`project.cpp` → `project.dll` loaded by the host → `userapp.exe` built and run).

The `CRAFTER_BUILD_MARCH` and `CRAFTER_BUILD_MTUNE` env vars override the default `-march=native` / `-mtune=native` for distro CI builds:

```bash
CRAFTER_BUILD_MARCH=x86-64-v3 CRAFTER_BUILD_MTUNE=generic ./build.sh
```

## Roadmap

- [ ] Target-triple auto-detection (`clang -print-target-triple`)
- [ ] Prefer system glslang when available, fall back to git+cmake
- [ ] Order-preserving deduplication of `BuildResult.libs` (current `unordered_set` doesn't preserve link order — only matters for cyclic static libs)

## License

LGPL-3.0-only. See [LICENSE](LICENSE).