test: declarative test.toml + target-derived runners (issue #8) #11
No reviewers
Labels
No labels
bug
claude:done
claude:failed
claude:in-progress
claude:ready
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Catcrafts/Crafter.Build!11
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "claude/issue-8"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes issue #8.
Implements the redesign agreed on in issue #8 comment-131 / comment-136:
test discovery splits into two distinct shapes, runner is derived from the
target triple, and the env-var-driven transport runners are retired.
Two-bucket model
test.toml+main.cpp) — for downstream test authors.Pick a target, declare preconditions, done. No
project.cppboilerplateper test.
project.cpp) — for tests that exercise the buildengine API in-process (
Incremental,Libraries,Cuda,Shader,ShaderDep,BuildError,RunnerClassification,CrossProjectModule,Diamond,UnitLib).project.cppandtest.tomlare mutually exclusive per fixture — adiscovery error flags the conflict.
test.toml schema
```toml
target = "aarch64-linux-gnu" # optional; default = host
march = "armv8-a"
mtune = "generic"
sysroot = "/opt/aarch64-rootfs" # propagated to QEMU_LD_PREFIX
timeout = 30 # seconds; default 60
args = ["--seed=42"]
requires = [
"tool:qemu-aarch64", # which/where probes PATH
"file:/opt/aarch64-rootfs/usr/share/libc++/v1/std.cppm",
"env:CRAFTER_TEST_TOKEN",
]
[defines]
CRAFTER_TEST_FOO = "42"
```
Runner derivation (TestRunner::ForTarget)
`CRAFTER_BUILD_RUNNER_` survives as a power-user override (parsed
via the narrowed FromSpec: `local` | `cmd:`).
Hard-fail-unless-declared
When the derived runner depends on a tool (qemu/wine/wasmtime) and that
tool isn't on PATH, the test fails unless the test declared
`tool:` in `requires`. The intent: surface broken cross-arch CI
config instead of letting it masquerade as a silent Skip.
Multi-target sweep
Bare `crafter-build test` (no --target=) now sweeps every distinct target
declared across the suite plus host. Tests run at their declared target
only, so cross-arch fixtures don't drag the rest of the suite into a
cross-build. `--target=X` bypasses the sweep.
Migrations
WindowsViaWine is new: declarative target=x86_64-w64-mingw32, requires
`tool:wine` + `tool:x86_64-w64-mingw32-g++`. Replaces the SSH-to-Windows
test, which needed a reachable Windows VM and a hand-rolled
ssh+scp+cmd.exe shell chain.
The defense-in-depth artifact checks (CrossArchAarch64's `file ... | grep
'ARM aarch64'` and Wasi's WASM magic bytes) are dropped on purpose: qemu
refuses to run wrong-arch ELFs and wasmtime refuses non-WASM input, so a
silent host-arch fallback would still fail the run.
Deletions
template machinery.
switch in JoinAndQuoteArgs.
({remote_bundle}, {bin_win}, {bundle_wsl} placeholder substitution).
{local, cmd}.
Vendors toml++ v3.4.0 as `lib/toml.hpp` (single-header, MIT, json.hpp
style). Added to the GMF of Crafter.Build-Test.cpp only so it doesn't
flow into PCM consumers.
Verification
`crafter-build test` on this host (no `wasmtime` installed):
```
=== target: aarch64-linux-gnu === ✅ CrossArchAarch64 (cmd:qemu-aarch64)
=== target: wasm32-wasip1 === ⏭ Wasi (cmd:wasmtime) skipped: tool 'wasmtime' not on PATH
=== target: x86_64-pc-linux-gnu === 13 passed (all outer-driver + arch-agnostic)
=== target: x86_64-w64-mingw32 === ✅ WindowsViaWine (wine)
```
Also verified single-target mode (`--target=x86_64-pc-linux-gnu`,
`--target=aarch64-linux-gnu`) works as before.
Test plan
🤖 Generated with Claude Code
Collapses three outer-driver tests into declarative top-level fixtures: - CrossArchAarch64: outer + inner pair becomes main.cpp + test.toml. The 'is this really an ARM aarch64 ELF?' artifact-introspection check is dropped — qemu-aarch64 refuses to run wrong-arch ELFs anyway, so a silent host-arch fallback would still fail the run. - Wasi: outer + inner pair becomes main.cpp + test.toml. The WASM magic-byte check is dropped on the same logic (wasmtime refuses non-WASM input). - Defines: simple defines-propagation smoke test becomes test.toml with [defines] CRAFTER_TEST_FOO = "42". Adds WindowsViaWine to replace the (forthcoming-deletion) WindowsViaSsh: declarative target=x86_64-w64-mingw32 + requires=[tool:wine, tool:x86_64-w64-mingw32-g++]. Exercises the new Wine runner and the ForTarget derivation end-to-end. Diamond and CrossProjectModule had speculative --target= reading that made them attempt cross-compilation under the multi-target sweep. They have no sysroot/toolchain plumbing, so those cross-builds always fail. Hardcoded them to HostTarget(); cross-arch tests live in their own test.toml from now on. Refs issue #8. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>With test.toml + ForTarget covering the cross-arch + Windows-on-Linux cases, the env-var-driven transport runners are dead weight. This commit removes them and the retired tests that exercised the env-var plumbing: - TestRunner::Ssh / SshWin / Wsl factories and their copy/exec/cleanup template machinery. - TestRunner::Shell enum (Host/Sh/Cmd) and the ShellQuoteSh helper — only Host shell quoting is needed once the remote shells are gone. - TestRunner::copy / cleanup / remoteDir / argsShell fields. - WindowsPathToWsl and the {remote_bundle}/{bin_win}/{bundle_wsl} placeholder substitution in RunSingleTest's transport branch. - ParseRunnerSpec narrowed from {local, cmd, ssh, sshwin, wsl} to {local, cmd} — the override hatch is preserved, just simpler. - tests/SshRunner, tests/WindowsViaSsh, tests/QemuUser: these tested the CRAFTER_BUILD_RUNNER_<target> → runner plumbing that has been replaced by ForTarget. The runner derivation is exercised every time CrossArchAarch64 / Wasi / WindowsViaWine runs. - tests/UnitLib: ssh/sshwin spec assertions become "throws on bogus spec" assertions. Refs issue #8. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>