test: introduce test.toml + target-derived runners alongside existing machinery

Vendors toml++ v3.4.0 as lib/toml.hpp and wires it into Crafter.Build-Test
to parse a declarative test.toml manifest (target/march/mtune/sysroot/
requires/timeout/args/defines). Test discovery now treats project.cpp and
test.toml as mutually exclusive: project.cpp stays the escape hatch for
outer-driver tests, test.toml gives downstream test authors a no-boilerplate
path.

Adds:
- TestRunner::Wine() and TestRunner::ForTarget(cfg) — runner is now derived
  from cfg.target (Local for host, Wine for Windows-on-Linux, wasmtime for
  WASI, qemu-<arch> with QEMU_LD_PREFIX for non-host Linux). The env-var
  override CRAFTER_BUILD_RUNNER_<target> still wins as a power-user escape
  hatch via FromEnv.
- Declarative preconditions: tool:<name>, file:<path>, env:<VAR> are
  evaluated before the build; missing preconditions Skip without paying
  the compile cost.
- Hard-fail-unless-declared: when a derived runner's tool is missing AND
  the test didn't declare 'tool:<that>' in requires, the missing runner
  is a Fail instead of a silent Skip. Surfaces broken cross-arch CI
  config that previously hid as "skipped".
- Multi-target sweep: bare `crafter-build test` (no --target=) now
  iterates every distinct test.toml-declared target plus the host, so
  cross-arch tests run by default without the user needing to know which
  targets exist. `--target=X` bypasses the sweep.

Test struct gains a `requires_` vector so project.cpp users can declare
preconditions too (matching what test.toml writes there).

Existing tests, factories (Ssh/SshWin/Wsl/Cmd), and CRAFTER_BUILD_RUNNER_*
machinery remain intact — this commit only adds; migration and deletion
follow in subsequent commits.

Refs issue #8.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-05-27 17:47:39 +02:00
commit dc27c5c204
4 changed files with 18286 additions and 22 deletions

View file

@ -77,8 +77,20 @@ export namespace Crafter {
static CRAFTER_API TestRunner SshWin(std::string host, std::string remoteDir = "C:/temp/crafter-tests");
static CRAFTER_API TestRunner Wsl(std::string remoteDir = "/tmp/crafter-tests-wsl");
static CRAFTER_API TestRunner Cmd(std::string command);
// Run a Windows .exe through Wine. Probes `wine` on PATH; on a Windows
// host the wine wrapper is pointless, so callers should route to Local
// before reaching here.
static CRAFTER_API TestRunner Wine();
static CRAFTER_API std::optional<TestRunner> FromSpec(std::string_view spec);
static CRAFTER_API TestRunner FromEnv(std::string_view target, TestRunner fallback = Local());
// Derive a runner from a Configuration's target triple + sysroot.
// Returns Local() when target equals the host, Wine() for Windows
// targets on a non-Windows host, `qemu-<arch>` (with QEMU_LD_PREFIX
// set when cfg.sysroot is non-empty) for non-host -linux- triples,
// `wasmtime` for wasm32-wasi/wasm64-wasi, and Local() as a last
// resort. CRAFTER_BUILD_RUNNER_<target> still wins as an override
// upstream of this — see FromEnv.
static CRAFTER_API TestRunner ForTarget(const struct Configuration& cfg);
};
enum class TestOutcome { Pass, Fail, Crash, Timeout, Skipped };
@ -185,6 +197,15 @@ export namespace Crafter {
TestRunner runner;
std::chrono::seconds timeout{60};
std::vector<std::string> args;
// Declarative preconditions. Each entry is "tool:<name>",
// "file:<path>", or "env:<VAR>". Evaluated before the test runs; any
// unmet require turns the test into a Skip with a derived reason.
// Also doubles as the "I know this runner might not be here" opt-in:
// when the test's derived runner needs a tool (e.g. qemu-aarch64,
// wasmtime, wine) and the matching tool: entry isn't present, an
// unavailable runner becomes a Fail instead of a silent Skip — the
// dependency has to be declared to be allowed to be missing.
std::vector<std::string> requires_;
};
CRAFTER_API BuildResult Build(Configuration& config, std::unordered_map<fs::path, std::shared_future<BuildResult>>& depResults, std::mutex& depMutex);

View file

@ -29,15 +29,13 @@ export namespace Crafter {
int jobs = 0;
std::optional<std::chrono::seconds> timeoutOverride;
bool listOnly = false;
// Only tests whose Configuration::target equals targetFilter are run.
// Set from --target=... (host triple if unspecified). Tests for other
// targets are silently excluded so e.g. `--target=mingw` doesn't drag
// in host-only outer-driver tests.
#ifdef _WIN32
std::string targetFilter = "x86_64-pc-windows-msvc";
#else
std::string targetFilter = "x86_64-pc-linux-gnu";
#endif
// Single-target run: only tests whose Configuration::target matches
// are included. Empty (default) = run every distinct target declared
// across discovered tests, plus the host target. Set from --target=...
// (when omitted, the harness sweeps all declared targets so cross-arch
// tests run by default without the user having to know which targets
// exist).
std::string targetFilter;
// CLI override for --runner=<spec>: applies to every test in the run.
// Target scoping is unnecessary because targetFilter ensures the run
// contains only one target's tests.