test: declarative test.toml + target-derived runners (issue #8) #11

Merged
jorijnvdgraaf merged 3 commits from claude/issue-8 into master 2026-05-27 18:10:08 +02:00
4 changed files with 18286 additions and 22 deletions
Showing only changes of commit dc27c5c204 - Show all commits

test: introduce test.toml + target-derived runners alongside existing machinery

Vendors toml++ v3.4.0 as lib/toml.hpp and wires it into Crafter.Build-Test
to parse a declarative test.toml manifest (target/march/mtune/sysroot/
requires/timeout/args/defines). Test discovery now treats project.cpp and
test.toml as mutually exclusive: project.cpp stays the escape hatch for
outer-driver tests, test.toml gives downstream test authors a no-boilerplate
path.

Adds:
- TestRunner::Wine() and TestRunner::ForTarget(cfg) — runner is now derived
  from cfg.target (Local for host, Wine for Windows-on-Linux, wasmtime for
  WASI, qemu-<arch> with QEMU_LD_PREFIX for non-host Linux). The env-var
  override CRAFTER_BUILD_RUNNER_<target> still wins as a power-user escape
  hatch via FromEnv.
- Declarative preconditions: tool:<name>, file:<path>, env:<VAR> are
  evaluated before the build; missing preconditions Skip without paying
  the compile cost.
- Hard-fail-unless-declared: when a derived runner's tool is missing AND
  the test didn't declare 'tool:<that>' in requires, the missing runner
  is a Fail instead of a silent Skip. Surfaces broken cross-arch CI
  config that previously hid as "skipped".
- Multi-target sweep: bare `crafter-build test` (no --target=) now
  iterates every distinct test.toml-declared target plus the host, so
  cross-arch tests run by default without the user needing to know which
  targets exist. `--target=X` bypasses the sweep.

Test struct gains a `requires_` vector so project.cpp users can declare
preconditions too (matching what test.toml writes there).

Existing tests, factories (Ssh/SshWin/Wsl/Cmd), and CRAFTER_BUILD_RUNNER_*
machinery remain intact — this commit only adds; migration and deletion
follow in subsequent commits.

Refs issue #8.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Jorijn van der Graaf 2026-05-27 17:47:39 +02:00

View file

@ -17,6 +17,12 @@ License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
module;
// toml++ is consumed as a translation-unit-private dependency in the GMF: the
// parser is only needed for test.toml discovery here, so keeping it out of
// `import std`-using module purviews avoids dragging it through the module
// graph (and through every PCM consumer of Crafter.Build).
#include "../lib/toml.hpp"
export module Crafter.Build:Test_impl;
import std;
import :Test;
@ -307,6 +313,69 @@ TestRunner TestRunner::Cmd(std::string command) {
return r;
}
TestRunner TestRunner::Wine() {
TestRunner r;
r.name = "wine";
r.argsShell = Shell::Host;
r.exec = "wine {bin} {args}";
#ifdef _WIN32
r.probe = "where wine";
#else
r.probe = "which wine";
#endif
return r;
}
TestRunner TestRunner::ForTarget(const Configuration& cfg) {
const std::string& target = cfg.target;
// Same triple as the host → run the binary directly. Covers the common
// case (cfg.target defaulted to HostTarget()) without any wrapper.
if (target == HostTarget()) return Local();
// Windows targets: native on a Windows host, Wine on Linux. We don't
// distinguish mingw vs msvc here — the produced .exe runs the same way.
if (TargetIsWindows(target)) {
return TargetIsWindows(HostTarget()) ? Local() : Wine();
}
// WASI: a .wasm file isn't directly executable; wasmtime is the canonical
// runtime. wasi-cli also works but the upstream Bytecode Alliance name is
// wasmtime, so we standardize on that.
if (target.starts_with("wasm32-wasi") || target.starts_with("wasm64-wasi")) {
return Cmd("wasmtime");
}
// Non-host Linux triple: extract the architecture and route through
// qemu-user. Triple is <arch>-<vendor>-<os>-<env> (or sometimes 3 parts);
// qemu-user's binary names mostly follow the arch field, with two known
// mismatches handled below. cfg.sysroot, when set, becomes QEMU_LD_PREFIX
// so the target's dynamic linker / shared libs are reachable — without
// it qemu-user crashes on dynamic ELFs with "could not open /lib/ld...".
if (target.find("-linux-") != std::string::npos) {
auto dash = target.find('-');
std::string arch = target.substr(0, dash);
// i686-linux-gnu → qemu-i386; arm-* already matches qemu-arm; aarch64,
// riscv64, ppc64le, mips, mips64, s390x all match their qemu names.
if (arch == "i686") arch = "i386";
TestRunner r = Cmd(std::format("qemu-{}", arch));
if (!cfg.sysroot.empty()) {
// Use `env VAR=value cmd` rather than the shell's `VAR=value cmd`
// prefix syntax: RunCommandWithTimeout pipes through GNU `timeout`,
// which execvp's its argument list directly without going through a
// shell. A bare VAR=value would be exec'd as a command path and
// fail with "No such file or directory".
r.exec = std::format("env QEMU_LD_PREFIX={} {}", cfg.sysroot, r.exec);
}
return r;
}
// Unknown / bare-metal / freestanding targets: fall back to Local. The
// caller's runner-availability probe (or absence of the binary) surfaces
// the problem rather than us inventing a wrong wrapper here.
return Local();
}
namespace {
std::string NormalizeTriple(std::string_view target) {
std::string out(target);
@ -453,20 +522,165 @@ TestResult Crafter::RunSingleTest(const Test& test, const fs::path& binary, std:
return result;
}
namespace {
// Declarative test metadata loaded from tests/<Name>/test.toml. Lets a test
// ship just main.cpp + a few lines of config instead of a whole project.cpp
// when its needs are "pick a target, gate on prerequisites, run with these
// args". project.cpp stays the escape hatch for outer-driver tests that
// call Build() / inspect intermediate state.
struct TestManifest {
std::optional<std::string> target;
std::optional<std::string> march;
std::optional<std::string> mtune;
std::optional<std::string> sysroot;
std::vector<std::string> requires_;
std::optional<int> timeoutSeconds;
std::vector<std::string> args;
std::vector<std::pair<std::string, std::string>> defines;
};
TestManifest ParseTestManifest(const fs::path& path) {
// toml++ builds with exceptions enabled by default; parse_file throws
// toml::parse_error on malformed input. Rethrow with the path attached
// so the discovery loop's catch can surface "where the error came from"
// alongside toml++'s "what was wrong".
toml::table t;
try {
t = toml::parse_file(path.string());
} catch (const toml::parse_error& e) {
throw std::runtime_error(std::format(
"test.toml parse error in {}: {}",
path.string(),
std::string_view(e.description())));
}
TestManifest m;
if (auto v = t["target"].value<std::string>()) m.target = *v;
if (auto v = t["march"].value<std::string>()) m.march = *v;
if (auto v = t["mtune"].value<std::string>()) m.mtune = *v;
if (auto v = t["sysroot"].value<std::string>()) m.sysroot = *v;
if (auto v = t["timeout"].value<int64_t>()) m.timeoutSeconds = static_cast<int>(*v);
if (auto arr = t["requires"].as_array()) {
for (auto& el : *arr) {
if (auto s = el.value<std::string>()) m.requires_.push_back(*s);
}
}
if (auto arr = t["args"].as_array()) {
for (auto& el : *arr) {
if (auto s = el.value<std::string>()) m.args.push_back(*s);
}
}
if (auto tbl = t["defines"].as_table()) {
for (auto&& [k, v] : *tbl) {
if (auto s = v.value<std::string>()) {
m.defines.emplace_back(std::string(k.str()), *s);
}
}
}
return m;
}
// Apply manifest overlay onto a Configuration synthesized from the test
// folder. Target overrides come last so a manifest's `target = "..."`
// wins over the synth default (= run's targetFilter). Defines accumulate;
// they don't replace pre-existing ones.
void ApplyManifest(Configuration& cfg, const TestManifest& m) {
if (m.target) cfg.target = *m.target;
if (m.march) cfg.march = *m.march;
if (m.mtune) cfg.mtune = *m.mtune;
if (m.sysroot) cfg.sysroot = *m.sysroot;
for (auto& [k, v] : m.defines) cfg.defines.push_back({k, v});
}
bool ToolOnPath(std::string_view name) {
#ifdef _WIN32
std::string cmd = std::format("where {} > nul 2>&1", name);
#else
std::string cmd = std::format("which {} > /dev/null 2>&1", name);
#endif
return std::system(cmd.c_str()) == 0;
}
struct RequireResult {
bool ok;
std::string reason; // human-readable when !ok
};
// Evaluate each `<kind>:<arg>` precondition. Returns the first failure
// (short-circuit; reporting one missing dep at a time is enough to act on
// and keeps the test log uncluttered).
RequireResult EvaluateRequires(std::span<const std::string> reqs) {
for (const auto& r : reqs) {
auto sep = r.find(':');
if (sep == std::string::npos || sep == 0 || sep == r.size() - 1) {
return {false, std::format("malformed require '{}' (expected kind:arg)", r)};
}
std::string_view kind(r.data(), sep);
std::string_view arg(r.data() + sep + 1, r.size() - sep - 1);
if (kind == "tool") {
if (!ToolOnPath(arg)) {
return {false, std::format("tool '{}' not on PATH", arg)};
}
} else if (kind == "file") {
if (!fs::exists(std::string(arg))) {
return {false, std::format("file '{}' missing", arg)};
}
} else if (kind == "env") {
const char* v = std::getenv(std::string(arg).c_str());
if (!v || !*v) {
return {false, std::format("env '{}' unset", arg)};
}
} else {
return {false, std::format(
"unknown require kind '{}' (expected tool/file/env)", kind)};
}
}
return {true, ""};
}
// Match a runner's tool dependency against the test's declared
// requirements. Used to decide between Skip (declared, may legitimately
// be missing) and Fail (runner unavailable but test didn't declare it —
// a silent skip would mask broken cross-arch CI configuration).
bool RequiresMentionsTool(std::span<const std::string> reqs, std::string_view tool) {
std::string needle = std::format("tool:{}", tool);
return std::ranges::any_of(reqs, [&](const std::string& s) { return s == needle; });
}
// Best-effort extraction of the runner-tool name from a TestRunner so the
// hard-fail-unless-declared check can match it against `requires`. For
// Cmd("foo"), the name is "cmd:foo"; for Wine, it's "wine". Anything else
// (Local, transport runners) returns empty — those don't trigger the
// declared/undeclared gate.
std::string RunnerToolName(const TestRunner& runner) {
if (runner.name == "wine") return "wine";
if (runner.name.starts_with("cmd:")) {
std::string tool = runner.name.substr(4);
// QEMU_LD_PREFIX prefix may be glued onto exec but the runner's
// `name` field already isolates the command, so no extra parsing.
return tool;
}
return "";
}
}
namespace {
// Synthesize a Configuration for tests/<Name>/ folders that don't contain
// a project.cpp. Convention: cfg.path = the folder, cfg.name/outputName =
// folder basename, cfg.target = the run's targetFilter, cfg.type = exe.
// Sources: top-level *.cpp (excluding project.cpp) become implementations,
// interfaces/*.cppm become module interfaces (matching the layout used
// elsewhere in this codebase). Tests with deeper layouts, defines, or
// dependencies still need an explicit project.cpp.
Configuration SynthesizeTest(const fs::path& dir, std::string_view target) {
// folder basename, cfg.target = host (overridable via test.toml `target`),
// cfg.type = exe. Sources: top-level *.cpp (excluding project.cpp) become
// implementations, interfaces/*.cppm become module interfaces. Tests with
// deeper layouts or dependencies still need an explicit project.cpp.
//
// Why host-default instead of targetFilter-default: under the multi-target
// sweep, an arch-agnostic test (no test.toml target) should run at the
// host iteration only — not get rebuilt against every cross-target the
// suite happens to declare. Cross-targeting is an opt-in via test.toml.
Configuration SynthesizeTest(const fs::path& dir) {
Configuration cfg;
cfg.path = dir;
cfg.name = dir.filename().string();
cfg.outputName = cfg.name;
cfg.target = std::string(target);
cfg.target = HostTarget();
cfg.type = ConfigurationType::Executable;
std::vector<fs::path> impls;
@ -502,6 +716,61 @@ namespace {
}
TestSummary Crafter::RunTests(Configuration& projectCfg, const RunTestsOptions& opts, std::span<const std::string_view> projectArgs) {
// Multi-target sweep: when no --target= was given, the run covers every
// distinct target a test.toml declares plus the host target. Lets a bare
// `crafter-build test` exercise cross-arch tests without the user having
// to know which targets exist in this project. An explicit --target=X
// bypasses the sweep and runs that target only.
if (opts.targetFilter.empty()) {
std::set<std::string> sweep;
sweep.insert(HostTarget());
fs::path testsDir = fs::current_path() / "tests";
if (fs::exists(testsDir) && fs::is_directory(testsDir)) {
for (auto& e : fs::directory_iterator(testsDir)) {
if (!e.is_directory()) continue;
auto stem = e.path().filename().string();
if (stem.empty() || stem[0] == '_' || stem[0] == '.') continue;
fs::path tomlPath = e.path() / "test.toml";
if (!fs::exists(tomlPath)) continue;
try {
TestManifest m = ParseTestManifest(tomlPath);
if (m.target) sweep.insert(*m.target);
} catch (...) {
// Parse failures surface as discovery failures during the
// actual run; the sweep phase just collects targets.
}
}
}
TestSummary aggregate;
// Inline tests pushed by the caller (fixture-driven inner RunTests
// calls, e.g. RunnerClassification) must survive each sweep
// iteration. Configuration isn't copyable so we can't snapshot+restore;
// instead, remember the inline count and erase only the entries
// appended by the previous iteration's auto-discovery.
size_t inlineCount = projectCfg.tests.size();
for (const auto& target : sweep) {
RunTestsOptions perTarget = opts;
perTarget.targetFilter = target;
if (projectCfg.tests.size() > inlineCount) {
projectCfg.tests.erase(
projectCfg.tests.begin() + inlineCount,
projectCfg.tests.end());
}
if (sweep.size() > 1) {
Progress::Clear();
std::println("\n=== target: {} ===", target);
}
TestSummary s = RunTests(projectCfg, perTarget, projectArgs);
aggregate.passed += s.passed;
aggregate.failed += s.failed;
aggregate.crashed += s.crashed;
aggregate.timedOut += s.timedOut;
aggregate.skipped += s.skipped;
for (auto& r : s.results) aggregate.results.push_back(std::move(r));
}
return aggregate;
}
TestSummary summary;
std::vector<TestResult> discoveryFailures;
@ -517,7 +786,16 @@ TestSummary Crafter::RunTests(Configuration& projectCfg, const RunTestsOptions&
// when projectCfg.path points at a subdirectory like "./src/" or "./lib/".
fs::path testsDir = fs::current_path() / "tests";
if (fs::exists(testsDir) && fs::is_directory(testsDir)) {
struct TestEntry { fs::path dir; fs::path pcpp; }; // pcpp empty = synth
// A discovered fixture is one of:
// - project.cpp present → outer-driver test (LoadProject)
// - test.toml present → declarative synth + manifest
// - neither, just *.cpp → bare synth (host-target)
// - both project.cpp and test.toml → XOR violation, discovery Fail
struct TestEntry {
fs::path dir;
fs::path pcpp; // outer-driver path
std::optional<TestManifest> manifest;
};
std::vector<TestEntry> entries;
for (auto& entry : fs::directory_iterator(testsDir)) {
if (!entry.is_directory()) continue;
@ -526,7 +804,34 @@ TestSummary Crafter::RunTests(Configuration& projectCfg, const RunTestsOptions&
TestEntry te;
te.dir = entry.path();
auto pcpp = te.dir / "project.cpp";
if (fs::exists(pcpp)) te.pcpp = pcpp;
auto tomlPath = te.dir / "test.toml";
bool hasPcpp = fs::exists(pcpp);
bool hasToml = fs::exists(tomlPath);
if (hasPcpp && hasToml) {
TestResult r;
r.name = stem;
r.outcome = TestOutcome::Fail;
r.exitCode = -1;
r.output = "both project.cpp and test.toml present — they're "
"mutually exclusive (delete one to disambiguate "
"outer-driver vs declarative test)";
discoveryFailures.push_back(std::move(r));
continue;
}
if (hasPcpp) te.pcpp = pcpp;
if (hasToml) {
try {
te.manifest = ParseTestManifest(tomlPath);
} catch (const std::exception& e) {
TestResult r;
r.name = stem;
r.outcome = TestOutcome::Fail;
r.exitCode = -1;
r.output = e.what();
discoveryFailures.push_back(std::move(r));
continue;
}
}
entries.push_back(std::move(te));
}
std::ranges::sort(entries, [](auto& a, auto& b) { return a.dir < b.dir; });
@ -548,7 +853,15 @@ TestSummary Crafter::RunTests(Configuration& projectCfg, const RunTestsOptions&
if (!te.pcpp.empty()) {
t.config = LoadProject(te.pcpp, fixtureArgs);
} else {
t.config = SynthesizeTest(te.dir, opts.targetFilter);
t.config = SynthesizeTest(te.dir);
if (te.manifest) {
ApplyManifest(t.config, *te.manifest);
if (te.manifest->timeoutSeconds) {
t.timeout = std::chrono::seconds(*te.manifest->timeoutSeconds);
}
t.args = te.manifest->args;
t.requires_ = te.manifest->requires_;
}
}
} catch (const std::exception& e) {
// A broken fixture shouldn't kill the whole run. Surface as a
@ -564,7 +877,7 @@ TestSummary Crafter::RunTests(Configuration& projectCfg, const RunTestsOptions&
continue;
}
if (t.config.target != opts.targetFilter) continue;
t.runner = TestRunner::FromEnv(t.config.target, TestRunner::Local());
t.runner = TestRunner::FromEnv(t.config.target, TestRunner::ForTarget(t.config));
if (opts.runnerOverride) {
if (auto r = TestRunner::FromSpec(*opts.runnerOverride)) {
t.runner = std::move(*r);
@ -639,9 +952,42 @@ TestSummary Crafter::RunTests(Configuration& projectCfg, const RunTestsOptions&
TestResult r;
r.name = t.config.name;
if (!runnerAvailable(t.runner)) {
// Declarative preconditions (test.toml requires = [...] or Test.requires_
// set in project.cpp). Evaluated before the build so a missing tool/file/env
// turns into a Skip without paying the compile cost. Reports the first
// failure only — once one precondition is unmet the test couldn't run
// anyway, and a wall of "also missing X, also missing Y" buries the
// actionable root cause.
if (auto req = EvaluateRequires(t.requires_); !req.ok) {
r.outcome = TestOutcome::Skipped;
r.output = std::format("runner '{}' not available", t.runner.name);
r.output = req.reason;
{
std::lock_guard lk(printMutex);
PrintResult(r, t.runner.name);
}
results[i] = std::move(r);
continue;
}
if (!runnerAvailable(t.runner)) {
// Hard-fail-unless-declared: if the runner depends on a tool
// (qemu-aarch64, wasmtime, wine, ...) and the test didn't say
// "tool:<that>" in requires, the missing runner is a Fail. The
// intent is to surface broken cross-arch CI configuration
// instead of letting it masquerade as a Skip; tests that
// legitimately may run without their runner have to opt in.
std::string tool = RunnerToolName(t.runner);
if (!tool.empty() && !RequiresMentionsTool(t.requires_, tool)) {
r.outcome = TestOutcome::Fail;
r.exitCode = -1;
r.output = std::format(
"runner '{}' unavailable and not declared in requires "
"(add 'tool:{}' to test.toml requires to permit skipping)",
t.runner.name, tool);
} else {
r.outcome = TestOutcome::Skipped;
r.output = std::format("runner '{}' not available", t.runner.name);
}
{
std::lock_guard lk(printMutex);
PrintResult(r, t.runner.name);

View file

@ -77,8 +77,20 @@ export namespace Crafter {
static CRAFTER_API TestRunner SshWin(std::string host, std::string remoteDir = "C:/temp/crafter-tests");
static CRAFTER_API TestRunner Wsl(std::string remoteDir = "/tmp/crafter-tests-wsl");
static CRAFTER_API TestRunner Cmd(std::string command);
// Run a Windows .exe through Wine. Probes `wine` on PATH; on a Windows
// host the wine wrapper is pointless, so callers should route to Local
// before reaching here.
static CRAFTER_API TestRunner Wine();
static CRAFTER_API std::optional<TestRunner> FromSpec(std::string_view spec);
static CRAFTER_API TestRunner FromEnv(std::string_view target, TestRunner fallback = Local());
// Derive a runner from a Configuration's target triple + sysroot.
// Returns Local() when target equals the host, Wine() for Windows
// targets on a non-Windows host, `qemu-<arch>` (with QEMU_LD_PREFIX
// set when cfg.sysroot is non-empty) for non-host -linux- triples,
// `wasmtime` for wasm32-wasi/wasm64-wasi, and Local() as a last
// resort. CRAFTER_BUILD_RUNNER_<target> still wins as an override
// upstream of this — see FromEnv.
static CRAFTER_API TestRunner ForTarget(const struct Configuration& cfg);
};
enum class TestOutcome { Pass, Fail, Crash, Timeout, Skipped };
@ -185,6 +197,15 @@ export namespace Crafter {
TestRunner runner;
std::chrono::seconds timeout{60};
std::vector<std::string> args;
// Declarative preconditions. Each entry is "tool:<name>",
// "file:<path>", or "env:<VAR>". Evaluated before the test runs; any
// unmet require turns the test into a Skip with a derived reason.
// Also doubles as the "I know this runner might not be here" opt-in:
// when the test's derived runner needs a tool (e.g. qemu-aarch64,
// wasmtime, wine) and the matching tool: entry isn't present, an
// unavailable runner becomes a Fail instead of a silent Skip — the
// dependency has to be declared to be allowed to be missing.
std::vector<std::string> requires_;
};
CRAFTER_API BuildResult Build(Configuration& config, std::unordered_map<fs::path, std::shared_future<BuildResult>>& depResults, std::mutex& depMutex);

View file

@ -29,15 +29,13 @@ export namespace Crafter {
int jobs = 0;
std::optional<std::chrono::seconds> timeoutOverride;
bool listOnly = false;
// Only tests whose Configuration::target equals targetFilter are run.
// Set from --target=... (host triple if unspecified). Tests for other
// targets are silently excluded so e.g. `--target=mingw` doesn't drag
// in host-only outer-driver tests.
#ifdef _WIN32
std::string targetFilter = "x86_64-pc-windows-msvc";
#else
std::string targetFilter = "x86_64-pc-linux-gnu";
#endif
// Single-target run: only tests whose Configuration::target matches
// are included. Empty (default) = run every distinct target declared
// across discovered tests, plus the host target. Set from --target=...
// (when omitted, the harness sweeps all declared targets so cross-arch
// tests run by default without the user having to know which targets
// exist).
std::string targetFilter;
// CLI override for --runner=<spec>: applies to every test in the run.
// Target scoping is unnecessary because targetFilter ensures the run
// contains only one target's tests.

17899
lib/toml.hpp Normal file

File diff suppressed because it is too large Load diff