diff --git a/AGENTS.template.md b/AGENTS.template.md index b60a81a..882b551 100644 --- a/AGENTS.template.md +++ b/AGENTS.template.md @@ -22,7 +22,7 @@ crafter-build is a C++26 modules build system. A project is described by a `proj Partitions and what they own: -- `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release. `MingwSysroot` makes the mingw target's `--sysroot`, a view in the cache of the mingw-w64 prefix's own directories: the prefix itself is never a sysroot, since clang would search its `include/` and `lib/`, and for `/usr` those are the Linux system's (issue #43). +- `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release. `MingwSysroot` settles the mingw target's `--sysroot`: the mingw-w64 prefix itself, or, for a prefix that also holds a Linux system (the host's `/usr`, anything whose `include/` has `features.h`), a view in the cache of its mingw-w64 directories, since clang would search its `include/` and `lib/` too (issue #43). `MingwViewToPrefix` turns paths into the view back into paths into the prefix for messages. - `:Clang` holds `Configuration`, `Build()`, `VariantId()`, `ApplyStandardArgs`, `HostTarget`, the CLI entry point `Crafter::Run`, and the lint, test and script data types. - `:Platform` runs commands, implements `LoadProject` (compile and load `project.cpp`), the host PCM cache, `#pragma crafter dependency`, and all OS-specific code. - `:Interface` and `:Implementation` model module units, their staleness checks and their compiles. @@ -118,14 +118,14 @@ Beyond the rules: - There is no host-platform preprocessor split: the host is Linux. Target differences (mingw, wasm) branch on `cfg.target` strings at runtime, in the library as well as in `project.cpp` and tests. - A build step fails when its command exits nonzero, never because it printed. Use `RunBuildCommand` or `RunCommandChecked`; warnings go to `Progress::Diagnostic`. - Every output path goes through `Configuration::BinDir()`, `BuildDir()` and `PcmDir()`. Hand-rolled `path / "bin" / ...` clobbered variants before; do not add new ones. -- Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, the wasm baseline and variant flags, the mingw-w64 prefix for the mingw target, uninterpreted project args and the toolchain (`Toolchain::Key()`). +- Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, the wasm baseline and variant flags, the std module the units import (`StdPcmKey`), uninterpreted project args and the toolchain (`Toolchain::Key()`). ## Design rules These were decided deliberately. Do not undo them without the maintainer's say-so. 1. **Configuration is C++.** No JSON, YAML, TOML or DSL for project, runner or lint configuration. Lint rules are C++ callbacks; there is no clang-format or clang-tidy integration and no separate "format rule" type. A rule that calls `ctx.SetContent` is a transform, and that is the whole mechanism. -2. **Per-project artifacts go in `/build` and `/bin`. Only cross-project things go in `~/.cache/crafter.build/`:** the std PCM and Crafter.Build PCMs per target and march, external clones keyed by their spec hash, and the mingw sysroot views keyed by prefix. +2. **Per-project artifacts go in `/build` and `/bin`. Only cross-project things go in `~/.cache/crafter.build/`:** the std PCM and Crafter.Build PCMs per target, march, toolchain and std module flags, external clones keyed by their spec hash, and the mingw sysroot views keyed by prefix. 3. **Sources ship, PCMs build locally.** The Flatpak installs `*.cppm` under `/app/share/crafter-build/`. Nothing target-baked ships except the executable and the toolchain beside it, and the executable is built for a portable `-march`. 4. **crafter-build is a monolithic executable with `--export-dynamic`.** `project.so` resolves `Crafter::` symbols from the running binary at dlopen time. There is no Windows host; `x86_64-w64-mingw32` is a cross-compile target for user projects only. 5. **Convention over configuration.** Lead with the zero-config path. Env var, then CLI flag, then full `project.cpp` are the escalation, not the entry point. @@ -189,6 +189,8 @@ Local builds are two scripts in `project.cpp`. Run them with the native binary: - A `#pragma crafter dependency` target is rebuilt on every load on purpose: clang crashes rather than erroring when it imports a BMI whose own imported PCM was rebuilt underneath it, and the depfile check is what catches a regenerated host cache. Do not optimise the rebuild away. - A host-tooling library such as Crafter.LintRules may `import Crafter.Build;` without declaring it as a dependency, because `LoadProject` fills the cache it resolves from. Listing Crafter.Build in its `cfg.dependencies` would clone and build all of crafter-build for every consumer. Its test does link crafter-build, so that test is wired behind `IsTopLevelProject()`: without the guard, crafter-build loading the library as a dependency would load its own `project.cpp` again through the test's `GitProject` and hit the cycle guard. - `crafter-build clean` leaves a pragma dependency's build alone; it lives with the clone in the cache. +- Units see a std module rebuilt in place, because their depfiles list the BMIs they import (`-fmodule-file-deps`), but not one that moved to another directory: their old depfiles still name the old one. So anything that changes how the std module is compiled goes into `StdPcmFlags`, which `BuildStdPcm` compiles with and `StdPcmKey` hashes. `VariantId()` hashes `StdPcmKey`, so the build moves with it. A flag added to the command anywhere else leaves both directories where they are. +- A CMake external whose recorded configure inputs (`/.crafter-options`) differ from this build's is built again from an empty build directory, not reconfigured in place: CMake keeps what an old compiler or `--sysroot` compiled, and the check results it cached. - The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop. - A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone. - A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under `share/crafter-build/libc++-modules//`, one `libcxx--modules` manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under `share/crafter-build/` to test against a foreign sysroot. diff --git a/README.md b/README.md index 11248aa..95d936f 100644 --- a/README.md +++ b/README.md @@ -201,7 +201,7 @@ For elaborate setups (rsync, ssh ControlMaster, env vars on the spawn, `QEMU_LD_ For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=` is passed to clang, and the std module is precompiled against the sysroot's libc++ headers. Ordinary compiles are pointed at the same headers (`-nostdinc++ -isystem /usr/include/c++/v1`), so a unit that `#include`s a standard header gets the sysroot's libc++ and not the host toolchain's. The module source has to come from the same libc++ release as those headers, so it is taken from `/usr/share/libc++/v1/std.cppm` when the sysroot ships it, else from the toolchain when both are the same major release, else from the sources crafter-build carries for other releases (`share/crafter-build/libc++-modules//`, `crafter-build --version` lists them). Alpine is the common case for the last two: its `libc++-dev` has the headers and no module source. When nothing matches, the build stops with one line naming both releases, installing that release's `share/libc++/v1` directory into the sysroot resolves it. The cheapest aarch64 sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root --config --dbpath /var/lib/pacman -Sy …` (point pacman at ALARM's mirrors via explicit config to avoid pulling host-arch packages). -For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed, with libstdc++ 15 or newer for its std module. clang finds it by the `x86_64-w64-mingw32-gcc` on `PATH`; `CRAFTER_MINGW_DIR` names another prefix (the directory holding `x86_64-w64-mingw32/` and `lib/gcc/`), and inside the Flatpak the host's `/usr` is used when it has mingw-w64. Without one the build stops with a message saying where it looked. clang is shown only the prefix's `x86_64-w64-mingw32/` and `lib/gcc/`, as links in `~/.cache/crafter.build/mingw-sysroot/`: it searches the `include/` and `lib/` of whatever it gets as the sysroot, and those of a prefix like `/usr` are the Linux system's own, glibc's headers among them. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported, there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. +For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed, with libstdc++ 15 or newer for its std module. clang finds it by the `x86_64-w64-mingw32-gcc` on `PATH`; `CRAFTER_MINGW_DIR` names another prefix (the directory holding `x86_64-w64-mingw32/` and `lib/gcc/`), and inside the Flatpak the host's `/usr` is used when it has mingw-w64. Without one the build stops with a message saying where it looked. The prefix is clang's sysroot as it is, so Windows libraries installed into its `include/` and `lib/` are found, unless it also holds a Linux system: the host's `/usr` in the Flatpak, or any prefix whose `include/` has a C library's `features.h`, as `/usr` does. clang searches a sysroot's `include/` and `lib/`, and in such a prefix those hold glibc's headers and Linux libraries, so clang is shown only its `x86_64-w64-mingw32/`, `lib/gcc/` and `lib64/gcc/`, as links in `mingw-sysroot/` in the cache (`~/.cache/crafter.build/`, in the Flatpak `~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build/`, or `$XDG_RUNTIME_DIR/crafter.build/` when the cache's filesystem takes no symbolic links). Compiler messages still name those headers by their paths in the prefix. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported, there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. For `--target=wasm32-wasip1`, the WASI sysroot is the one beside the toolchain (the Flatpak bundles one) unless `CRAFTER_BUILD_WASI_SYSROOT` names another, and tests run under `wasmtime`. C++ exceptions work as on the other targets: every unit and the std module are compiled with `-fwasm-exceptions` in the standardized form of WebAssembly exception handling (exnref), against the exception-enabled libc++ in the sysroot's `lib//eh/`, and `-lunwind` is linked automatically. A sysroot without that libc++ (wasi-sdk before 33) stops the build with one line saying so. The result runs on engines with that proposal: Chrome 137, Firefox 131 and Safari 18.4 or newer, and the wasmtime the Flatpak bundles. A page from `EnableWasiBrowserRuntime` checks first and names what an older browser lacks in the console, instead of failing to compile the module. An exception that escapes `main` leaves the program as an uncaught WebAssembly exception (wasmtime reports `thrown Wasm exception` and exits 1, a page gets a `WebAssembly.Exception`) without calling `std::terminate`, so a `std::set_terminate` handler does not run and nothing prints the exception's type or `what()`: wasm exception handling unwinds in a single phase, and nothing finds out that no handler is left until the stack is gone. Catch it in `main` to report it. diff --git a/implementations/Crafter.Build-Clang.cpp b/implementations/Crafter.Build-Clang.cpp index f5943ca..a302ca9 100644 --- a/implementations/Crafter.Build-Clang.cpp +++ b/implementations/Crafter.Build-Clang.cpp @@ -247,36 +247,51 @@ void Configuration::GetInterfacesAndImplementations(std::span interfac } } +std::string Crafter::StdPcmFlags(const Configuration& config) { + if (config.target == "x86_64-w64-mingw32") { + // -femulated-tls keeps PCM TLS access matching libstdc++'s emutls + // definitions; a mismatch surfaces as undefined std::__once_callable. + return std::format(" --target={} -march={} -mtune={} -femulated-tls -O3 -std=c++26 -Wno-reserved-identifier -Wno-reserved-module-identifier", config.target, config.march, config.mtune); + } + const bool isWasm = config.target.starts_with("wasm32"); + const std::string sysrootFlag = config.sysroot.empty() ? std::string() : std::format(" --sysroot={}", config.sysroot); + // wasm32 rejects -march, and the std module has to be compiled the way + // the units importing it are, a variant pass's flags (e.g. + // -mrelaxed-simd) included, or the BMI is not feature-compatible. + std::string archFlags = isWasm ? WasmCompileFlags(config) : std::format(" -march={} -mtune={}", config.march, config.mtune); + // non-wasm cross builds: the driver ranks the host toolchain's own + // libc++ headers (/../include/c++/v1) above --sysroot, so the + // PCM would mix host libc++ with the foreign libc. Force the sysroot's + // copy. + if (!isWasm && !config.sysroot.empty()) { + archFlags += std::format(" -nostdinc++ -isystem {}/usr/include/c++/v1", config.sysroot); + } + return std::format(" --target={} -std=c++26{}{} -O3 -Wno-reserved-identifier -Wno-reserved-module-identifier", config.target, sysrootFlag, archFlags); +} + +std::string Crafter::StdPcmKey(const Configuration& config, const Toolchain& toolchain) { + // A PCM is only valid for the clang that wrote it and the flags it was + // written with, so a compiler bump starts a cold directory instead of + // handing the new clang a PCM it will crash on, and so does any flag. + // With only some of them in the name, configurations that differed in + // the rest took turns rebuilding one std PCM in place: -mtune=generic + // against project.cpp's -mtune=native on every run, two sysroots, or two + // releases sharing a cache, as the Flatpak's branches do, each time under + // the PCMs built against the previous one, which clang crashes on rather + // than rejecting. + std::string compiled = StdPcmFlags(config); + // The base command's part: the toolchain's key covers the compiler and + // libc++, and for the mingw target what clang finds mingw-w64 through is + // the prefix, given whole or through a view (MingwSysroot), named here by + // the prefix rather than by the view's own path. + if (config.target == "x86_64-w64-mingw32" && !toolchain.mingwPrefix.empty()) { + compiled += std::format("|mingw:{}|{}", toolchain.mingwPrefix.string(), toolchain.mingwPrefixIsSystem ? "view" : "whole"); + } + return std::format("{}-{}-{}-{:08x}", config.target, config.march, toolchain.Key(), static_cast(std::hash{}(compiled))); +} + fs::path Crafter::StdPcmDir(const Configuration& config) { - // The std PCM is cached per target+march+toolchain: a PCM is only valid - // for the exact clang that wrote it, so a compiler bump starts a cold - // directory instead of handing the new clang a PCM it will crash on. A - // wasm variant pass compiles it with extra codegen flags (e.g. - // -mrelaxed-simd) — its BMI must not be shared with the baseline's, or - // the consuming TUs see a target-feature mismatch. Suffix the cache dir - // with the variant flags when present. - std::string stdPcmKey = std::format("{}-{}-{}", config.target, config.march, GetToolchain().Key()); - // The flags every wasm unit gets are compiled into the std PCM as well, - // and the importing units have to match them. Two releases that share a - // cache and a compiler, as the Flatpak's branches do, would otherwise - // take turns rebuilding this one std PCM, each time underneath the PCMs - // built against the previous one, and clang crashes on those rather than - // rejecting them. - if (config.target.starts_with("wasm32")) { - stdPcmKey += std::format("-{:08x}", static_cast(std::hash{}(WasmBaselineFlags))); - } - // The same goes for the mingw-w64 prefix the mingw std module is - // compiled against: releases up to the fix for issue #43 gave clang the - // prefix itself as the sysroot and the ones after give it a view of it, - // and two prefixes are two sets of headers. - if (config.target == "x86_64-w64-mingw32" && !GetToolchain().mingwPrefix.empty()) { - stdPcmKey += std::format("-{:08x}", static_cast(std::hash{}(GetToolchain().mingwPrefix.string()))); - } - for (const std::string& f : config.wasmVariantFlags) { - stdPcmKey += "+"; - for (char c : f) stdPcmKey += (c == '/' || c == '\\') ? '_' : c; - } - return GetCacheDir()/stdPcmKey; + return GetCacheDir() / StdPcmKey(config, GetToolchain()); } std::string Crafter::WasmCompileFlags(const Configuration& config) { @@ -935,9 +950,9 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map_impl.o.d beside the object for Check to read. - std::string result = RunBuildCommand(std::format("{0} {1}.cpp -c -MD -MF {2}_impl.o.d -o {2}_impl.o", clang, path.string(), (buildDir/path.filename()).string())); + // -MD leaves _impl.o.d beside the object for Check to read, and + // -fmodule-file-deps puts the BMIs the unit imported in it with the + // headers. The driver adds that for --precompile only, so a std + // module rebuilt in place, for a mingw-w64 or libc++ update under + // the same compiler, left the objects importing it as they were, and + // the next build only relinked them. + std::string result = RunBuildCommand(std::format("{0} {1}.cpp -c -MD -MF {2}_impl.o.d -fmodule-file-deps -o {2}_impl.o", clang, path.string(), (buildDir/path.filename()).string())); bool expected = false; if(!result.empty() && buildCancelled.compare_exchange_strong(expected, true)) { diff --git a/implementations/Crafter.Build-Lint.cpp b/implementations/Crafter.Build-Lint.cpp index 357fb33..39e4be0 100644 --- a/implementations/Crafter.Build-Lint.cpp +++ b/implementations/Crafter.Build-Lint.cpp @@ -1149,6 +1149,10 @@ format` applies it to disk, and `crafter-build lint` reports where it would. // command per configuration — GetCompileCommand walks the dependency tree. const bool anyRuleNeedsAst = std::ranges::any_of(rules, [](const LintRule* r) { return r->needsAst; }); std::unordered_map commands; + // Why a configuration of the build graph has no command: its files + // cannot be parsed, which fails the run. Cleared to nothing, the command + // read as "not a translation unit" and every semantic rule passed them. + std::unordered_map commandErrors; if (anyRuleNeedsAst && !opts.noAst) { for (Configuration* c : localConfigs) { auto ensure = [&](const Configuration& cfg) { @@ -1166,10 +1170,14 @@ format` applies it to disk, and `crafter-build lint` reports where it would. Progress::Task task(std::format("Building std PCM ({}-{})", cfg.target, cfg.march)); fs::create_directories(assembled.stdPcmDir); std::string error = BuildStdPcm(cfg, assembled.stdPcmDir/"std.pcm"); - if (!error.empty()) command.clear(); + if (!error.empty()) { + command.clear(); + commandErrors.emplace(&cfg, std::format("building the std module for {}: {}", cfg.target, error)); + } } - } catch (const std::exception&) { - command.clear(); // surfaced per file as an AST reason + } catch (const std::exception& e) { + command.clear(); + commandErrors.emplace(&cfg, std::format("assembling the compile command for {}: {}", cfg.target, e.what())); } commands.emplace(&cfg, std::move(command)); }; @@ -1200,6 +1208,11 @@ format` applies it to disk, and `crafter-build lint` reports where it would. // decides what is safe to touch. Skip it and make the run fail. if (rule->needsAst) { if (opts.noAst) continue; + if (auto failed = commandErrors.find(owner); failed != commandErrors.end()) { + astFailures.emplace(file, failed->second); + skippedAstRules.insert(rule->name); + continue; + } // No compile command means this file is not a translation unit // of the build graph — project.cpp, which LoadProject compiles // with its own flags, or a header. A semantic rule does not diff --git a/implementations/Crafter.Build-Platform.cpp b/implementations/Crafter.Build-Platform.cpp index e0f2065..5d0c5f4 100644 --- a/implementations/Crafter.Build-Platform.cpp +++ b/implementations/Crafter.Build-Platform.cpp @@ -281,6 +281,9 @@ bool Crafter::MatchAny(std::span globs, std::string_view name std::string Crafter::RunBuildCommand(std::string_view cmd) { Progress::EchoCommand(cmd); CommandResult r = RunCommandChecked(cmd); + // A mingw build against a system prefix sees mingw-w64 through a view in + // the cache, and clang names every header by the path it found it at. + r.output = MingwViewToPrefix(std::move(r.output)); if (r.exitCode == 0) { // Warnings from a step that succeeded. Show them — until now they were @@ -737,10 +740,13 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) { // library, naming the std module's source relative to itself. It is // looked for in the library directories the compiler searches for // this target, as it would look for libstdc++ itself, but only in - // those that belong to the mingw toolchain: the list also has the - // generic /lib, where the host's own (Linux) libstdc++ keeps - // its manifest. A toolchain that isn't there is a build error, not - // an exception: Build() runs this with its worker threads started. + // those that belong to the mingw toolchain. The list also holds + // clang's resource directory, the sysroot's generic lib/, where a + // prefix given whole can keep a libstdc++ built for another target, + // and, when clang finds no gcc directory, an empty entry that would + // name a manifest in the working directory. A toolchain that isn't + // there is a build error, not an exception: Build() runs this with + // its worker threads started. const Toolchain& tc = GetToolchain(); const std::string where = tc.mingwPrefix.empty() ? std::string("where clang finds it by the x86_64-w64-mingw32 gcc on PATH") @@ -775,24 +781,19 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) { } } if (manifestPath.empty()) return missing; - // Found through the view MingwSysroot() makes of the prefix, and - // named from here on by its place in the prefix, which is the - // directory the user knows and can go and look at. - if (!tc.mingwPrefix.empty()) { - const fs::path inPrefix = manifestPath.lexically_relative(MingwSysroot()); - if (!inPrefix.empty() && *inPrefix.begin() != "..") manifestPath = tc.mingwPrefix / inPrefix; - } + // Found through the view MingwSysroot() may have made of the prefix, + // and named from here on by its place in the prefix, the directory + // the user knows and can go and look at. + manifestPath = fs::path(MingwViewToPrefix(manifestPath.string())); const std::optional source = StdModuleSourcePath(FileBytes(manifestPath)); if (!source) return std::format("{} names no std module", manifestPath.string()); const fs::path stdCc = (manifestPath.parent_path() / *source).lexically_normal(); if (!fs::exists(stdCc)) return std::format("{}, which {} names as the std module, is missing", stdCc.string(), manifestPath.string()); - // -femulated-tls keeps PCM TLS access matching libstdc++'s emutls - // definitions; mismatch surfaces as undefined std::__once_callable. // The std.cc → std.cppm copy uses a plain filename: held under the // cache lock, no other builder writes this path concurrently. fs::path stdCppm = stdPcm.parent_path() / "std.cppm"; - std::string command = std::format("{} --target={} -march={} -mtune={} -femulated-tls -O3 -std=c++26 -Wno-reserved-identifier -Wno-reserved-module-identifier --precompile {} -o {}", GetBaseCommand(config), config.target, config.march, config.mtune, stdCppm.string(), stdPcm.string()); + std::string command = std::format("{}{} --precompile {} -o {}", GetBaseCommand(config), StdPcmFlags(config), stdCppm.string(), stdPcm.string()); std::string stamp = HashStamp(command + '\0' + FileBytes(stdCc)); if (fs::exists(stdPcm) && ReadStamp(StampPath(stdPcm)) == stamp) return ""; std::error_code copyEc; @@ -822,27 +823,11 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) { if (!source.error.empty()) return source.error; stdCppm = source.stdCppm.string(); } - std::string sysrootFlag = config.sysroot.empty() - ? std::string() - : std::format(" --sysroot={}", config.sysroot); - // wasm32 rejects -march, and the std module has to be compiled the way - // the units importing it are, a variant pass's flags (e.g. - // -mrelaxed-simd) included, or the BMI is not feature-compatible. - std::string archFlags = isWasm - ? WasmCompileFlags(config) - : std::format(" -march={} -mtune={}", config.march, config.mtune); - // non-wasm cross builds: the driver ranks the host toolchain's own - // libc++ headers (/../include/c++/v1) above --sysroot, so - // the PCM would mix host libc++ with the foreign libc — force the - // sysroot's copy. - if (!isWasm && !config.sysroot.empty()) { - archFlags += std::format(" -nostdinc++ -isystem {}/usr/include/c++/v1", config.sysroot); - } // Keyed on the command and the module source's bytes, never on // mtimes: an installed std.cppm can be dated at the epoch (OSTree), // and a libc++ update under the same clang changes the bytes but // not the path. - std::string command = std::format("{} --target={} -std=c++26{}{} -O3 -Wno-reserved-identifier -Wno-reserved-module-identifier --precompile {} -o {}", GetBaseCommand(config), config.target, sysrootFlag, archFlags, stdCppm, stdPcm.string()); + std::string command = std::format("{}{} --precompile {} -o {}", GetBaseCommand(config), StdPcmFlags(config), stdCppm, stdPcm.string()); std::string stamp = HashStamp(command + '\0' + FileBytes(stdCppm)); if (fs::exists(stdPcm) && ReadStamp(StampPath(stdPcm)) == stamp) return ""; std::string result = RunBuildCommand(command); diff --git a/implementations/Crafter.Build-Toolchain.cpp b/implementations/Crafter.Build-Toolchain.cpp index c75f627..84a1de3 100644 --- a/implementations/Crafter.Build-Toolchain.cpp +++ b/implementations/Crafter.Build-Toolchain.cpp @@ -54,9 +54,26 @@ namespace { fs::create_directory_symlink(target, link, ec); std::error_code ignored; if (ec && fs::read_symlink(link, ignored) != target) { - throw fs::filesystem_error("cannot make the mingw-w64 sysroot view", target, link, ec); + throw fs::filesystem_error("cannot make a link", target, link, ec); } } + + // `dir` made absolute and spelled one way, once, when the toolchain is + // resolved. The mingw prefix ends up as the target of links and in the + // names of build and cache directories: a relative path read later from + // another working directory would name something else, and /opt/mingw/ + // next to /opt/mingw would rebuild everything for the same directory. + fs::path PrefixPath(const fs::path& dir) { + fs::path normal = fs::absolute(dir).lexically_normal(); + if (!normal.has_filename() && normal.has_relative_path()) normal = normal.parent_path(); + return normal; + } + + // The view MingwSysroot() made and the prefix it shows, for + // MingwViewToPrefix. Written once, before ViewMade is set. + fs::path MadeView; + fs::path ViewedPrefix; + std::atomic ViewMade = false; } std::string Toolchain::LibcxxCompileFlags() const { @@ -150,14 +167,19 @@ Toolchain Crafter::ResolveToolchain(const ToolchainInputs& in) { tc.wasiSysroot = "/usr/share/wasi-sysroot"; } if (std::string_view dir = Env(in, "CRAFTER_MINGW_DIR"); !dir.empty()) { - // Made absolute here, once: the prefix becomes the target of links - // in the cache and part of directory names, and a relative path read - // later from another working directory would name something else. - tc.mingwPrefix = fs::absolute(fs::path(dir)); + tc.mingwPrefix = PrefixPath(fs::path(dir)); + // A prefix that holds a Linux system as well as mingw-w64, /usr + // itself, has that system's C library in include/: features.h, which + // every glibc and musl header includes and mingw-w64 has no use for. + // One that holds mingw-w64 alone stays the sysroot, so Windows + // libraries installed into its include/ and lib/ stay visible. + tc.mingwPrefixIsSystem = fs::exists(tc.mingwPrefix / "include" / "features.h"); } else if (!in.hostRoot.empty() && fs::exists(in.hostRoot / "usr" / "x86_64-w64-mingw32")) { // The sandbox has no mingw-w64 of its own, and no mingw gcc on PATH - // for clang to find one by. - tc.mingwPrefix = in.hostRoot / "usr"; + // for clang to find one by. The host's /usr is the host's system, + // whether or not its C library's headers are installed. + tc.mingwPrefix = PrefixPath(in.hostRoot / "usr"); + tc.mingwPrefixIsSystem = true; } return tc; } @@ -203,31 +225,69 @@ const Toolchain& Crafter::RequireToolchain() { return tc; } -const fs::path& Crafter::MingwSysroot() { - // Made on first use rather than by GetToolchain(), so only a mingw build - // writes to the cache for it, and a failure surfaces as that build's - // error instead of on every command. - static const fs::path Sysroot = []() -> fs::path { - const fs::path& prefix = GetToolchain().mingwPrefix; - if (prefix.empty()) return {}; - // A view per prefix keeps two shells with different - // CRAFTER_MINGW_DIRs, which share one cache, from rewriting each - // other's. - const fs::path sysroot = GetCacheDir() / "mingw-sysroot" / std::format("{:016x}", std::hash{}(prefix.string())); - // All clang's mingw driver takes from a sysroot other than its - // include/ and the rest of its lib/: the mingw-w64 tree (headers, - // libraries, and libstdc++ where a distro's cross gcc installs it) - // and gcc's per-target directory (crtbegin.o, libgcc, and the version - // that names libstdc++'s header directory), which openSUSE keeps - // under lib64. A link to nothing is skipped like a missing directory. - for (std::string_view entry : {"x86_64-w64-mingw32", "lib/gcc", "lib64/gcc"}) { - EnsureSymlink(prefix / entry, sysroot / entry); +fs::path Crafter::MakeMingwSysroot(const fs::path& prefix, std::span parents) { + std::string reasons; + for (const fs::path& parent : parents) { + // One view per prefix: two shells with different CRAFTER_MINGW_DIRs + // share one cache and would otherwise rewrite each other's. + const fs::path sysroot = parent / "mingw-sysroot" / std::format("{:016x}", std::hash{}(prefix.string())); + try { + // All clang's mingw driver takes from a sysroot other than its + // include/ and the rest of its lib/: the mingw-w64 tree (headers, + // libraries, and libstdc++ where a distro's cross gcc installs + // it) and gcc's per-target directory (crtbegin.o, libgcc, and the + // version that names libstdc++'s header directory), which + // openSUSE keeps under lib64. A link to nothing is skipped like a + // missing directory. + for (std::string_view entry : {"x86_64-w64-mingw32", "lib/gcc", "lib64/gcc"}) { + EnsureSymlink(prefix / entry, sysroot / entry); + } + return sysroot; + } catch (const std::exception& e) { + reasons += std::format("; {}", e.what()); } - return sysroot; + } + throw std::runtime_error(std::format("cannot make a sysroot that shows clang only the mingw-w64 parts of {}{}", prefix.string(), reasons)); +} + +const fs::path& Crafter::MingwSysroot() { + // Made on first use rather than by GetToolchain(), so that only what + // compiles for the mingw target, a build or an AST lint of one, writes + // anything for it, and a failure surfaces as that command's error + // instead of on every command. + static const fs::path Sysroot = []() -> fs::path { + const Toolchain& tc = GetToolchain(); + if (!tc.mingwPrefixIsSystem) return tc.mingwPrefix; + // In the cache, or where the XDG spec promises symbolic links when + // the cache's filesystem takes none (vfat, exFAT, CIFS without unix + // extensions), which costs nothing else crafter-build caches. + // Absolute, because CMake runs the compiler from directories of its + // own. + std::vector parents = { fs::absolute(GetCacheDir()) }; + if (const char* runtime = std::getenv("XDG_RUNTIME_DIR"); runtime && *runtime) { + parents.push_back(fs::absolute(fs::path(runtime) / "crafter.build")); + } + fs::path view = MakeMingwSysroot(tc.mingwPrefix, parents); + MadeView = view; + ViewedPrefix = tc.mingwPrefix; + ViewMade.store(true, std::memory_order_release); + return view; }(); return Sysroot; } +std::string Crafter::MingwViewToPrefix(std::string text) { + if (!ViewMade.load(std::memory_order_acquire)) return text; + // With the separator, so a --sysroot= naming the view itself, in a + // command line the output quotes, stays what was run. + const std::string view = (MadeView / "").string(); + const std::string prefix = (ViewedPrefix / "").string(); + for (std::size_t at = text.find(view); at != std::string::npos; at = text.find(view, at + prefix.size())) { + text.replace(at, view.size(), prefix); + } + return text; +} + std::int32_t Crafter::LibcxxVersion(const fs::path& includeDir) { // "# define _LIBCPP_VERSION 230102", the one place libc++ states its // release in a form the preprocessor does not have to be run for. diff --git a/interfaces/Crafter.Build-Clang.cppm b/interfaces/Crafter.Build-Clang.cppm index 89583cc..50903d6 100644 --- a/interfaces/Crafter.Build-Clang.cppm +++ b/interfaces/Crafter.Build-Clang.cppm @@ -392,6 +392,15 @@ export namespace Crafter { }; struct Configuration; + // The name StdPcmDir gives the directory of this configuration's std + // module under `toolchain`: target, march and the toolchain's key, which + // say what it is, then a hash of everything else it is compiled with, + // each flag BuildStdPcm passes (StdPcmFlags) and for the mingw target the + // prefix behind its --sysroot. Two configurations that would compile the + // std module differently never share a directory, so neither rebuilds it + // under the PCMs the other compiled against it. Declared ahead of + // Configuration, whose VariantId keys on it. + std::string StdPcmKey(const Configuration& config, const Toolchain& toolchain); // What a Script's callback is handed when `crafter-build run ` // starts it. @@ -580,7 +589,8 @@ export namespace Crafter { void AddAstLintRule(std::string name, std::function check); // Suffix that uniquely identifies this Configuration's compile state. // target+march+mtune are spelled out for readability; the rest - // (type, debug, sysroot, defines, compileFlags, the toolchain) + // (type, debug, sysroot, defines, compileFlags, the std module, the + // toolchain) // collapse into a short hash so two Configurations sharing a path but // with different compile state can't clobber each other's outputs. std::string VariantId() const { return VariantId(GetToolchain()); } @@ -628,17 +638,15 @@ export namespace Crafter { compileKey += "|B:"; compileKey += WasmBaselineFlags; } - // The mingw-w64 prefix decides what the mingw target compiles - // against, and until issue #43 clang got it as the sysroot - // itself, which put the host's own headers on the include path. - // Unkeyed, the release that fixed it kept the objects compiled - // against glibc's headers: the build after the update only - // relinked them, and failed to link as before. No prefix keeps - // the id as it was, since clang's own discovery has not changed. - if (target == "x86_64-w64-mingw32" && !toolchain.mingwPrefix.empty()) { - compileKey += "|M:"; - compileKey += toolchain.mingwPrefix.string(); - } + // The std module the units import. Its PCMs and objects record + // it by path, and units do not notice it moving: when the mingw + // prefix stopped being clang's sysroot as is (issue #43), the + // std module was rebuilt and the objects compiled against the + // host's glibc headers were only relinked, failing as before. A + // std module that moves directories, because anything it is + // compiled with changed, moves the build with it. + compileKey += "|S:"; + compileKey += StdPcmKey(*this, toolchain); // Sorted by ApplyStandardArgs, so flag order on the command line // doesn't split the cache. for (const std::string& a : projectArgs) { @@ -673,10 +681,9 @@ export namespace Crafter { std::vector requires_; }; - // Directory holding the std module PCM for this Configuration. Keyed on - // target+march because a BMI is only loadable by a TU with matching target - // features, and suffixed with the wasm flags (WasmBaselineFlags, then the - // variant's) for the same reason. + // Directory holding the std module PCM for this Configuration, + // GetCacheDir() / StdPcmKey(config, GetToolchain()): a BMI is only + // loadable by a unit compiled the way it was. fs::path StdPcmDir(const Configuration& config); // Where Build leaves this Configuration's linked artifact: the executable @@ -805,4 +812,9 @@ namespace Crafter { // compile command and of the std module's, which have to agree flag for // flag. std::string WasmCompileFlags(const Configuration& config); + // The flags BuildStdPcm compiles the std module with, after the base + // command and before the source, each with a leading space. StdPcmKey + // hashes the same string, so the command and the directory it writes to + // cannot disagree about what decides the BMI. + std::string StdPcmFlags(const Configuration& config); } diff --git a/interfaces/Crafter.Build-Toolchain.cppm b/interfaces/Crafter.Build-Toolchain.cppm index d6b360a..6f1b68c 100644 --- a/interfaces/Crafter.Build-Toolchain.cppm +++ b/interfaces/Crafter.Build-Toolchain.cppm @@ -39,14 +39,20 @@ export namespace Crafter { fs::path wasiSysroot; // The mingw-w64 install prefix, the directory that holds // x86_64-w64-mingw32/ and lib/gcc/x86_64-w64-mingw32/ (/usr for a - // distro package), as an absolute path. Empty means clang's own - // discovery, which follows the mingw gcc on PATH. Set by + // distro package), absolute and lexically normal. Empty means clang's + // own discovery, which follows the mingw gcc on PATH. Set by // CRAFTER_MINGW_DIR, else the host's /usr when running sandboxed and - // it has mingw-w64 (the sandbox's own /usr never does). clang never - // gets the prefix itself: the mingw target's --sysroot is - // MingwSysroot(), a view of it. Keyed into the mingw target's - // VariantId and std PCM directory. + // it has mingw-w64 (the sandbox's own /usr never does). What the + // mingw target passes as --sysroot is MingwSysroot(), not this: + // see mingwPrefixIsSystem. fs::path mingwPrefix; + // The prefix holds a Linux system as well as mingw-w64: the host's + // /usr in the sandbox, or a prefix whose include/ has a C library's + // features.h, as /usr has. clang searches a mingw sysroot's include/ + // and lib/, so such a prefix is shown to it only through a view of + // its mingw-w64 parts; any other prefix is the sysroot itself, and + // Windows libraries installed into its include/ and lib/ are found. + bool mingwPrefixIsSystem = false; // `clang++ -print-target-triple` and the first line of // `clang++ --version`; both empty when clang++ could not be run. std::string hostTriple; @@ -89,18 +95,32 @@ export namespace Crafter { const Toolchain& GetToolchain(); // GetToolchain(), or a std::runtime_error carrying its error. const Toolchain& RequireToolchain(); + // Makes, in the first of `parents` that can hold one, a directory that + // shows clang only the mingw-w64 parts of `prefix`: links named + // x86_64-w64-mingw32, lib/gcc and lib64/gcc to the same paths under + // `prefix`, at /mingw-sysroot/, and returns it. + // A view already there is reused; one another process makes at the same + // moment is as good as one made here. Throws when no parent can hold + // one, with the reason for each. + fs::path MakeMingwSysroot(const fs::path& prefix, std::span parents); // The --sysroot of every compile, link and CMake configure for the mingw - // target: a directory in crafter-build's cache holding only links to the - // x86_64-w64-mingw32/, lib/gcc/ and lib64/gcc/ of GetToolchain()'s - // mingwPrefix, made on the first call of a process. Empty without a - // prefix. The prefix cannot be the sysroot: clang takes a mingw sysroot - // for a tree that holds mingw-w64 alone and searches its include/ and - // lib/ as well, which for /usr (the host's, in the Flatpak) are the - // Linux system's own. A Windows build then saw glibc's headers, and + // target, settled on the first call of a process: empty without a + // mingwPrefix, the prefix itself for one that holds mingw-w64 alone, and + // for a system prefix the view MakeMingwSysroot makes in crafter-build's + // cache, or in $XDG_RUNTIME_DIR/crafter.build when the cache's + // filesystem takes no symbolic links. A system prefix cannot be the + // sysroot: clang takes a mingw sysroot for a tree that holds mingw-w64 + // alone and searches its include/ and lib/ as well, which for /usr are + // the Linux system's own. A Windows build then saw glibc's headers, and // libstdc++, finding , had shared_ptr read // __libc_single_threaded, which nothing on Windows defines (issue #43). // Throws when the view cannot be made. const fs::path& MingwSysroot(); + // `text` with every path into the view MingwSysroot() made written as + // the same path into the prefix it shows, so that compiler messages name + // a header where the user installed it rather than in a hashed cache + // directory; `text` unchanged when no view was made. + std::string MingwViewToPrefix(std::string text); // _LIBCPP_VERSION as the <__config> in `includeDir` (an include/c++/v1) // defines it, XXYYZZ; 0 when the file is missing or has no such line. std::int32_t LibcxxVersion(const fs::path& includeDir); diff --git a/project.cpp b/project.cpp index adedaef..38db3a4 100644 --- a/project.cpp +++ b/project.cpp @@ -247,6 +247,7 @@ extern "C" Configuration CrafterBuildProject(std::span a cfg.AddTest("IncrementalInterfaceChange").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("TransitiveInterfaceChange").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("IncrementalHeaderChange").Dependencies({ CrafterBuildLib.get() }); + cfg.AddTest("StdPcmRebuild").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("CompilerWarning").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("CleanProject").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ShaderCompile").Dependencies({ CrafterBuildLib.get() }); @@ -257,6 +258,7 @@ extern "C" Configuration CrafterBuildProject(std::span a cfg.AddTest("GitSubmodules").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("LinkFlagsChange").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ExternalArtifactTime").Dependencies({ CrafterBuildLib.get() }); + cfg.AddTest("ExternalReconfigure").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ConcurrentExternalFetch").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("SharedExternalFetch").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("StandardArgs").Dependencies({ CrafterBuildLib.get() }); diff --git a/tests/ExternalReconfigure/main.cpp b/tests/ExternalReconfigure/main.cpp new file mode 100644 index 0000000..57d50ba --- /dev/null +++ b/tests/ExternalReconfigure/main.cpp @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +import std; +import Crafter.Build; +namespace fs = std::filesystem; +using namespace Crafter; + +// A CMake dependency configured with other inputs than this build's (another +// compiler or --sysroot, the flags crafter-build injects, which it records in +// /.crafter-options) has to be built again from nothing. It was +// reconfigured in place, and CMake does not recompile what a changed +// --sysroot compiled (the Makefiles generator writes it into build.make, +// which make does not compare) and keeps the check results of the old +// configure. A mingw dependency built by a release that gave clang the +// host's /usr as the sysroot (issue #43) kept its objects reading +// __libc_single_threaded after the update. +// +// The dependency is a local repository with a one-file C library. A file +// planted in its build directory stands for what the old configure left, +// and the recorded inputs are rewritten the way another crafter-build would +// have left them. Everything is local; XDG_CACHE_HOME keeps the clone out of +// the real cache. + +extern "C" int setenv(const char* name, const char* value, int overwrite); + +namespace { + std::int32_t failures = 0; + + void Check(bool cond, std::string_view msg) { + if (!cond) { + std::println(std::cerr, "FAIL: {}", msg); + ++failures; + } + } + + void Git(const fs::path& dir, std::string_view args) { + const std::string command = std::format("git -C {} {}", ShellQuote(dir.string()), args); + auto result = RunCommandWithTimeout(command, std::chrono::seconds(60)); + if (result.exitCode != 0) throw std::runtime_error(std::format("{} failed: {}", command, result.output)); + } + + void Write(const fs::path& file, std::string_view text) { + std::ofstream(file, std::ios::binary | std::ios::trunc) << text; + } +} + +int main() { + const fs::path scratch = fs::temp_directory_path() / "crafter-build-external-reconfigure"; + fs::remove_all(scratch); + fs::create_directories(scratch); + setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1); + setenv("GIT_CONFIG_NOSYSTEM", "1", 1); + setenv("GIT_CONFIG_GLOBAL", "/dev/null", 1); + for (const char* name : { "GIT_AUTHOR_NAME", "GIT_COMMITTER_NAME" }) setenv(name, "crafter-build test", 1); + for (const char* name : { "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_EMAIL" }) setenv(name, "test@example.invalid", 1); + + const fs::path repo = scratch / "reconf"; + fs::create_directories(repo); + Write(repo / "lib.c", "int Reconfigured(void) { return 1; }\n"); + Write(repo / "CMakeLists.txt", + "cmake_minimum_required(VERSION 3.16)\n" + "project(reconf C)\n" + "add_library(reconf STATIC lib.c)\n"); + Git(repo, "init -q -b main"); + Git(repo, "add ."); + Git(repo, "commit -q -m reconf"); + + ExternalDependency dep; + dep.name = "reconf"; + dep.source.url = std::format("file://{}", repo.string()); + dep.builder = ExternalBuilder::CMake; + dep.libs = { "reconf" }; + std::string error = BuildExternal(dep, HostTarget()); + Check(error.empty(), std::format("the dependency builds: {}", error)); + + const fs::path build = ExternalBuildDir(dep, HostTarget()); + const fs::path leftover = build / "left-by-the-old-configure"; + Write(leftover, ""); + error = BuildExternal(dep, HostTarget()); + Check(error.empty(), std::format("and builds again: {}", error)); + Check(fs::exists(leftover), "with the same inputs its build directory is kept"); + + Write(build / ".crafter-options", " -DCMAKE_SYSROOT=/usr\n"); + error = BuildExternal(dep, HostTarget()); + Check(error.empty(), std::format("configured with other inputs, it builds: {}", error)); + Check(!fs::exists(leftover), "from an empty build directory"); + Check(fs::exists(build / "libreconf.a"), "and the library is built again"); + + std::error_code ec; + fs::remove_all(scratch, ec); + if (failures > 0) { + std::println(std::cerr, "{} assertions failed", failures); + return 1; + } + return 0; +} diff --git a/tests/Lint/main.cpp b/tests/Lint/main.cpp index 69bd436..4e07d94 100644 --- a/tests/Lint/main.cpp +++ b/tests/Lint/main.cpp @@ -6,6 +6,9 @@ import Crafter.Build; namespace fs = std::filesystem; using namespace Crafter; +extern "C" int setenv(const char* name, const char* value, int overwrite); +extern "C" int unsetenv(const char* name); + namespace { std::int32_t Failures = 0; @@ -705,6 +708,31 @@ int main() { // two assertions above cover it. } + // A configuration whose std module cannot be built is in the build graph + // all the same: its files cannot be parsed, which has to fail the run. + // Its command was cleared, which read as "not a translation unit", and + // every semantic rule passed the files. A sysroot without libc++ is one + // way there. The std module's directory is made before its build fails, + // so the cache is redirected around this run. + { + Scratch s("ast-no-std-module"); + s.Write("f", "int Value = 1;\n"); + Configuration cfg = s.Config({"f"}); + cfg.sysroot = (s.dir / "empty-sysroot").string(); + fs::create_directories(cfg.sysroot); + bool ran = false; + cfg.AddAstLintRule("needs-ast", [&ran](LintContext&) { ran = true; }); + const char* realCache = std::getenv("XDG_CACHE_HOME"); + const std::string saved = realCache ? realCache : ""; + setenv("XDG_CACHE_HOME", (s.dir / "cache").c_str(), 1); + LintSummary summary = RunLint(cfg, Mode(LintMode::Report)); + if (saved.empty()) unsetenv("XDG_CACHE_HOME"); + else setenv("XDG_CACHE_HOME", saved.c_str(), 1); + Check(!ran, "ast: rule is skipped when the std module cannot be built"); + Check(summary.errors > 0, "ast: a std module that cannot be built fails the run"); + Check(!summary.Clean(), "ast: a std module that cannot be built is not Clean"); + } + // --no-ast skips those rules deliberately and exits normally. { Scratch s("ast-optout"); diff --git a/tests/MingwSysroot/fixture/failing.cpp b/tests/MingwSysroot/fixture/failing.cpp new file mode 100644 index 0000000..254ec5b --- /dev/null +++ b/tests/MingwSysroot/fixture/failing.cpp @@ -0,0 +1,6 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +// Stops in a header of the stand-in mingw-w64, so that the error clang +// reports is located in one. +#include diff --git a/tests/MingwSysroot/main.cpp b/tests/MingwSysroot/main.cpp index 2388acf..3964b36 100644 --- a/tests/MingwSysroot/main.cpp +++ b/tests/MingwSysroot/main.cpp @@ -17,17 +17,17 @@ using namespace Crafter; // did the same outside the Flatpak. // // The prefix staged here has the shape of such a /usr and needs no real -// mingw-w64: glibc's in include/, gcc's per-target -// directory, which clang derives libstdc++'s header directory from, and a -// stand-in mingw-w64 tree whose libstdc++ has a std module with nothing in -// it. The fixture is a static library, so nothing is linked, and its C++ and -// C sources stop the compile when they see the prefix's include/ or miss -// mingw-w64's or libstdc++'s headers. +// mingw-w64: glibc's features.h and in include/, +// gcc's per-target directory, which clang derives libstdc++'s header +// directory from, and a stand-in mingw-w64 tree whose libstdc++ has a std +// module with nothing in it. The fixture is a static library, so nothing is +// linked, and its C++ and C sources stop the compile when they see the +// prefix's include/ or miss mingw-w64's or libstdc++'s headers. // -// The std module such a build compiles is not the one releases before the -// fix compiled against the prefix itself, so it has to live in a directory -// of its own: the Flatpak's branches share one cache, and two releases -// taking turns rebuilding one PCM do it under the PCMs built against it. +// clang reaches such a prefix through a view of it (MakeMingwSysroot), which +// has to be made where symbolic links can be, and a compile error in one of +// its headers has to name the header where it is installed, not in the +// view. extern "C" int setenv(const char* name, const char* value, int overwrite); @@ -45,6 +45,26 @@ namespace { fs::create_directories(file.parent_path()); std::ofstream(file, std::ios::binary | std::ios::trunc) << text; } + + BuildResult BuildLibrary(const fs::path& project, std::string_view name, const fs::path& cFile) { + Configuration lib; + lib.path = project; + lib.name = std::string(name); + lib.outputName = std::string(name); + lib.target = "x86_64-w64-mingw32"; + // Spelled out: the default, native, names the host's CPU, which is + // no x86 CPU on an aarch64 host. + lib.march = "x86-64"; + lib.mtune = "x86-64"; + lib.type = ConfigurationType::LibraryStatic; + std::array ifaces = {}; + std::array impls = { fs::path(name) }; + lib.GetInterfacesAndImplementations(ifaces, impls); + if (!cFile.empty()) lib.cFiles = { cFile }; + std::unordered_map> depResults; + std::mutex depMutex; + return Build(lib, depResults, depMutex); + } } int main() { @@ -56,37 +76,71 @@ int main() { fs::copy(fs::current_path() / "tests" / "MingwSysroot" / "fixture", project, fs::copy_options::recursive); const fs::path prefix = scratch / "usr"; + Write(prefix / "include" / "features.h", "#define __GLIBC__ 2\n"); Write(prefix / "include" / "sys" / "single_threaded.h", "extern char __libc_single_threaded;\n"); fs::create_directories(prefix / "lib" / "gcc" / "x86_64-w64-mingw32" / "15.1.0"); const fs::path mingw = prefix / "x86_64-w64-mingw32"; Write(mingw / "include" / "_mingw.h", ""); + Write(mingw / "include" / "crafter_probe_error.h", "#error stopped in a mingw-w64 header\n"); Write(mingw / "include" / "c++" / "15.1.0" / "memory", ""); // Through std::format because the build's import scanner reads string // literals as code: one holding "module std;" made this file a unit of a // module std, and the test did not build. Write(mingw / "include" / "c++" / "15.1.0" / "bits" / "std.cc", std::format("export module {};\n", "std")); Write(mingw / "lib" / "libstdc++.modules.json", R"({"version": 1, "modules": [{"logical-name": "std", "source-path": "../include/c++/15.1.0/bits/std.cc", "is-std-library": true}]})"); + + { + // The view is made in the first place that can hold one. A cache on + // a filesystem that takes no symbolic links (vfat, exFAT, some CIFS + // and FUSE mounts) holds everything else crafter-build caches, so + // its failure must not be every mingw build's: this one can make + // nothing under mingw-sysroot, which is a file. + const fs::path blocked = scratch / "blocked"; + Write(blocked / "mingw-sysroot", ""); + const fs::path runtime = scratch / "runtime"; + const std::array parents = { blocked, runtime }; + fs::path view; + try { + view = MakeMingwSysroot(prefix, parents); + } catch (const std::exception& e) { + Check(false, std::format("a view is made in the second place when the first can hold none: {}", e.what())); + } + if (!view.empty()) { + Check(view.string().starts_with(runtime.string()), std::format("the view is in the place that could hold it: {}", view.string())); + Check(fs::equivalent(view / "x86_64-w64-mingw32", mingw, ec), "the view shows the prefix's mingw-w64 tree"); + Check(fs::equivalent(view / "lib" / "gcc", prefix / "lib" / "gcc", ec), "and gcc's directories"); + Check(!fs::exists(view / "include") && !fs::exists(view / "lib" / "libstdc++.modules.json"), "and nothing else of the prefix's include/ and lib/"); + fs::path again; + try { + again = MakeMingwSysroot(prefix, parents); + } catch (const std::exception& e) { + Check(false, std::format("a view already there is reused: {}", e.what())); + } + Check(again == view, "a view already there is reused"); + } + const std::array nowhere = { blocked }; + std::string error; + try { + MakeMingwSysroot(prefix, nowhere); + } catch (const std::exception& e) { + error = e.what(); + } + Check(error.contains(prefix.string()) && error.contains((blocked / "mingw-sysroot").string()), std::format("with no place for a view, the error names the prefix and why each place failed: {}", error)); + } + // Before anything resolves the toolchain, which reads both once. The - // std module cache goes to the scratch directory too. + // view and the std module cache go to the scratch directory too. setenv("CRAFTER_MINGW_DIR", prefix.c_str(), 1); setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1); - Configuration lib; - lib.path = project; - lib.name = "probe"; - lib.outputName = "probe"; - lib.target = "x86_64-w64-mingw32"; - lib.type = ConfigurationType::LibraryStatic; - std::array ifaces = {}; - std::array impls = { "probe" }; - lib.GetInterfacesAndImplementations(ifaces, impls); - lib.cFiles = { project / "cprobe" }; - std::unordered_map> depResults; - std::mutex depMutex; - const BuildResult built = Build(lib, depResults, depMutex); + const BuildResult built = BuildLibrary(project, "probe", project / "cprobe"); Check(built.result.empty(), std::format("a Windows build sees mingw-w64's and libstdc++'s headers and not the prefix's own: {}", built.result)); - const std::string unkeyed = std::format("{}-{}-{}", lib.target, lib.march, GetToolchain().Key()); - Check(StdPcmDir(lib).filename() != unkeyed, std::format("the std module of a mingw build with a prefix is not cached where older releases built it against the prefix itself: {}", StdPcmDir(lib).string())); + Check(MingwSysroot() != prefix && MingwSysroot().string().starts_with((scratch / "cache").string()), std::format("a prefix holding a Linux system reaches clang through a view in the cache: {}", MingwSysroot().string())); + + const BuildResult failed = BuildLibrary(project, "failing", {}); + const std::string header = (mingw / "include" / "crafter_probe_error.h").string(); + Check(failed.result.contains("stopped in a mingw-w64 header"), std::format("an error in a mingw-w64 header fails the build: {}", failed.result)); + Check(failed.result.contains(header) && !failed.result.contains(MingwSysroot().string()), std::format("and names the header in the prefix, not in the view: {}", failed.result)); fs::remove_all(scratch, ec); if (failures > 0) { diff --git a/tests/StdPcmRebuild/fixture/main.cpp b/tests/StdPcmRebuild/fixture/main.cpp new file mode 100644 index 0000000..45992f4 --- /dev/null +++ b/tests/StdPcmRebuild/fixture/main.cpp @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +import std; + +int main() { + std::println("rebuilt"); +} diff --git a/tests/StdPcmRebuild/main.cpp b/tests/StdPcmRebuild/main.cpp new file mode 100644 index 0000000..291bee6 --- /dev/null +++ b/tests/StdPcmRebuild/main.cpp @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +import std; +import Crafter.Build; +namespace fs = std::filesystem; +using namespace Crafter; + +// A std module rebuilt in place has to recompile the units that import it. +// The std PCM is rebuilt in its directory whenever its source changes under +// the same compiler and flags: a libc++ update, or mingw-w64's under the +// same prefix. A .cpp unit's depfile listed only its source and headers, +// because the driver records imported BMIs for --precompile alone, so such +// a unit kept its object and the next build only relinked it. +// +// The project is one main.cpp that imports std. Its object is dated before +// std.pcm and after its source, as a std module rebuilt after the object +// leaves them, and the next build has to compile it again. XDG_CACHE_HOME keeps the std PCM out of the real +// cache, whose other users the change of date would rebuild. + +extern "C" int setenv(const char* name, const char* value, int overwrite); + +namespace { + std::int32_t failures = 0; + + void Check(bool cond, std::string_view msg) { + if (!cond) { + std::println(std::cerr, "FAIL: {}", msg); + ++failures; + } + } + + BuildResult BuildOnce(Configuration& cfg) { + std::unordered_map> depResults; + std::mutex depMutex; + return Build(cfg, depResults, depMutex); + } +} + +int main() { + const fs::path scratch = fs::temp_directory_path() / "crafter-build-std-pcm-rebuild"; + std::error_code ec; + fs::remove_all(scratch, ec); + fs::create_directories(scratch); + const fs::path project = scratch / "project"; + fs::copy(fs::current_path() / "tests" / "StdPcmRebuild" / "fixture", project, fs::copy_options::recursive); + setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1); + + Configuration cfg; + cfg.path = project; + cfg.name = "rebuilt"; + cfg.outputName = "rebuilt"; + cfg.type = ConfigurationType::Executable; + std::array ifaces = {}; + std::array impls = { "main" }; + cfg.GetInterfacesAndImplementations(ifaces, impls); + + BuildResult built = BuildOnce(cfg); + Check(built.result.empty(), std::format("the project builds: {}", built.result)); + const fs::path object = cfg.BuildDir() / "main_impl.o"; + const fs::path stdPcm = StdPcmDir(cfg) / "std.pcm"; + Check(fs::exists(object) && fs::exists(stdPcm), std::format("{} and {} exist", object.string(), stdPcm.string())); + if (!fs::exists(object) || !fs::exists(stdPcm)) return 1; + + const fs::file_time_type first = fs::last_write_time(object); + built = BuildOnce(cfg); + Check(built.result.empty(), std::format("an idle build succeeds: {}", built.result)); + Check(fs::last_write_time(object) == first, "and compiles nothing again"); + + // The source before the object and the object before std.pcm, so that + // nothing but the std module makes the object stale. + const fs::file_time_type older = fs::last_write_time(stdPcm) - std::chrono::seconds(10); + fs::last_write_time(project / "main.cpp", older - std::chrono::seconds(10)); + fs::last_write_time(object, older); + built = BuildOnce(cfg); + Check(built.result.empty(), std::format("the build after the std module succeeds: {}", built.result)); + Check(fs::last_write_time(object) > older, "a unit importing a std module newer than its object is compiled again"); + + fs::remove_all(scratch, ec); + if (failures > 0) { + std::println(std::cerr, "{} assertions failed", failures); + return 1; + } + return 0; +} diff --git a/tests/SysrootCompileFlags/main.cpp b/tests/SysrootCompileFlags/main.cpp index bfaf95b..6734089 100644 --- a/tests/SysrootCompileFlags/main.cpp +++ b/tests/SysrootCompileFlags/main.cpp @@ -19,6 +19,9 @@ using namespace Crafter; // sysroot's copy. It must not do so where there is no sysroot, for wasm (which // has its own sysroot handling) or for mingw (libstdc++, no libc++ at all). // The command is checked as text: no compiler runs. + +extern "C" int setenv(const char* name, const char* value, int overwrite); + namespace { std::int32_t Failures = 0; @@ -44,6 +47,15 @@ namespace { } int main() { + // A compile command names the std module's directory in the cache, and + // the mingw one can make the view of a system mingw prefix there + // (MingwSysroot), as with CRAFTER_MINGW_DIR=/usr or inside the Flatpak + // on a host with mingw-w64. Set before anything resolves the toolchain. + const fs::path scratch = fs::temp_directory_path() / "crafter-build-sysroot-compile-flags-cache"; + std::error_code ec; + fs::remove_all(scratch, ec); + setenv("XDG_CACHE_HOME", scratch.c_str(), 1); + const std::string sysroot = "/opt/crafter-build-test-sysroot"; const std::string forced = std::format("-nostdinc++ -isystem {}/usr/include/c++/v1", sysroot); @@ -57,6 +69,7 @@ int main() { Check(!CommandFor("wasm32-wasip1", sysroot).contains(forced), "wasm: left to its own sysroot handling"); Check(!CommandFor("x86_64-w64-mingw32", sysroot).contains(forced), "mingw: libstdc++ target, libc++ headers are not forced"); + fs::remove_all(scratch, ec); if (Failures > 0) { std::println(std::cerr, "{} assertions failed", Failures); return 1; diff --git a/tests/ToolchainResolve/main.cpp b/tests/ToolchainResolve/main.cpp index 1912418..d92a1c8 100644 --- a/tests/ToolchainResolve/main.cpp +++ b/tests/ToolchainResolve/main.cpp @@ -147,6 +147,31 @@ int main() { Check(tc.mingwPrefix == "/opt/mingw", "CRAFTER_MINGW_DIR names the mingw prefix"); in.env["CRAFTER_MINGW_DIR"] = "mingw"; Check(ResolveToolchain(in).mingwPrefix == fs::absolute("mingw"), "a relative CRAFTER_MINGW_DIR is resolved against the working directory once, when the toolchain is"); + // The prefix names directories, so one directory has one spelling: + // otherwise a trailing slash rebuilds every mingw build for nothing. + for (const char* spelling : {"/opt/mingw/", "/opt//mingw", "/opt/./mingw", "/opt/x/../mingw"}) { + in.env["CRAFTER_MINGW_DIR"] = spelling; + Check(ResolveToolchain(in).mingwPrefix == "/opt/mingw", std::format("{} is spelled /opt/mingw (got '{}')", spelling, ResolveToolchain(in).mingwPrefix.string())); + } + } + { + // A prefix that holds a Linux system's C library, as /usr does, is + // shown to clang through a view; one that holds mingw-w64 alone is + // the sysroot itself, Windows libraries in its include/ and lib/ + // included. + const fs::path staged = fs::temp_directory_path() / "crafter-build-toolchain-mingw-prefixes"; + fs::remove_all(staged); + Touch(staged / "system" / "include" / "features.h"); + Touch(staged / "system" / "x86_64-w64-mingw32" / "include" / "_mingw.h"); + Touch(staged / "dedicated" / "include" / "SDL2" / "SDL.h"); + Touch(staged / "dedicated" / "x86_64-w64-mingw32" / "include" / "_mingw.h"); + ToolchainInputs in; + in.env["CRAFTER_MINGW_DIR"] = (staged / "system").string(); + Check(ResolveToolchain(in).mingwPrefixIsSystem, "a prefix with a C library's features.h holds a Linux system"); + in.env["CRAFTER_MINGW_DIR"] = (staged / "dedicated").string(); + Check(!ResolveToolchain(in).mingwPrefixIsSystem, "a prefix of mingw-w64 and Windows libraries does not"); + Check(!ResolveToolchain({}).mingwPrefixIsSystem, "no prefix, nothing to show"); + fs::remove_all(staged); } { // Sandboxed, the host's /usr stands in for the missing mingw-w64, @@ -158,6 +183,7 @@ int main() { Check(ResolveToolchain(in).mingwPrefix.empty(), "a host without mingw-w64 gives no mingw prefix"); fs::create_directories(host / "usr" / "x86_64-w64-mingw32"); Check(ResolveToolchain(in).mingwPrefix == host / "usr", "a host with mingw-w64 gives its /usr as the mingw prefix"); + Check(ResolveToolchain(in).mingwPrefixIsSystem, "the host's /usr is the host's system, with or without its C library's headers"); in.env["CRAFTER_MINGW_DIR"] = "/opt/mingw"; Check(ResolveToolchain(in).mingwPrefix == "/opt/mingw", "CRAFTER_MINGW_DIR wins over the host's"); fs::remove_all(host); diff --git a/tests/VariantId/main.cpp b/tests/VariantId/main.cpp index e658b3e..0b4472d 100644 --- a/tests/VariantId/main.cpp +++ b/tests/VariantId/main.cpp @@ -178,6 +178,46 @@ int main() { Check(mingw.VariantId(hostUsr) != mingw.VariantId(optMingw), "two mingw prefixes are two variants"); Configuration host = MakeBase(); Check(host.VariantId(discovered) == host.VariantId(hostUsr), "the mingw prefix leaves other targets' variants alone"); + + // The std module's directory is keyed the same way, and so is a + // prefix given whole against one reached through a view. + Check(StdPcmKey(mingw, discovered) != StdPcmKey(mingw, hostUsr), "a mingw prefix moves the mingw std module"); + Check(StdPcmKey(mingw, hostUsr) != StdPcmKey(mingw, optMingw), "two mingw prefixes are two std modules"); + Toolchain hostUsrWhole = hostUsr; + hostUsrWhole.mingwPrefixIsSystem = !hostUsr.mingwPrefixIsSystem; + Check(StdPcmKey(mingw, hostUsr) != StdPcmKey(mingw, hostUsrWhole), "a prefix given whole and one shown through a view are two std modules"); + Check(mingw.VariantId(hostUsr) != mingw.VariantId(hostUsrWhole), "and two variants, through the std module"); + Check(StdPcmKey(host, discovered) == StdPcmKey(host, hostUsr), "the mingw prefix leaves other targets' std modules alone"); + } + { + // Everything the std module is compiled with keys its directory, not + // only target and march. -mtune was left out, so a project built + // with -mtune=generic shared one std PCM with project.cpp's + // -mtune=native, and the two rebuilt it in turn on every run, under + // every module built against it; two sysroots did the same. + Toolchain tc; + tc.clangxx = "/app/bin/clang++"; + tc.clangVersion = "clang version 23.1.2"; + Configuration native = MakeBase(); + Configuration generic = MakeBase(); + generic.mtune = "generic"; + Check(StdPcmKey(native, tc) != StdPcmKey(generic, tc), "-mtune keys the std module"); + Configuration sysrootA = MakeBase(); + sysrootA.sysroot = "/opt/sysroot-a"; + Configuration sysrootB = MakeBase(); + sysrootB.sysroot = "/opt/sysroot-b"; + Check(StdPcmKey(sysrootA, tc) != StdPcmKey(sysrootB, tc) && StdPcmKey(sysrootA, tc) != StdPcmKey(native, tc), "the sysroot keys the std module"); + Configuration wasm = MakeBase(); + wasm.target = "wasm32-wasip1"; + wasm.sysroot = "/opt/wasi-sysroot"; + Configuration relaxed = MakeBase(); + relaxed.target = wasm.target; + relaxed.sysroot = wasm.sysroot; + relaxed.wasmVariantFlags = {"-mrelaxed-simd"}; + Check(StdPcmKey(wasm, tc) != StdPcmKey(relaxed, tc), "a wasm variant pass's flags key the std module"); + Check(StdPcmKey(native, tc) == StdPcmKey(MakeBase(), tc), "one configuration, one std module"); + Check(StdPcmKey(native, tc).starts_with(std::format("{}-{}-{}-", native.target, native.march, tc.Key())), std::format("the directory still says target, march and toolchain (got '{}')", StdPcmKey(native, tc))); + Check(native.VariantId(tc) != generic.VariantId(tc) && sysrootA.VariantId(tc) != sysrootB.VariantId(tc), "a build moves with its std module"); } if (Failures > 0) {