From ab998081e1b0dbd4709bf6a74fae81fd7dad8563 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 20:48:06 +0100 Subject: [PATCH 01/12] fix: give the mingw target a sysroot that holds mingw-w64 alone Inside the Flatpak the mingw-w64 prefix is the host's /usr, and it was clang's --sysroot as is. clang's mingw driver takes a sysroot for a tree that holds mingw-w64 alone, and adds its include/ to the include path and its lib/ to the library path. For /usr those are the Linux system's own: every header mingw-w64 lacks came from glibc, and libstdc++'s , finding , had shared_ptr read __libc_single_threaded, which nothing on Windows defines. Every program that used one failed to link, among them every Crafter.Graphics Win32 executable, through Crafter.Asset's GLB loader. CRAFTER_MINGW_DIR=/usr did the same outside the Flatpak (#43). The sysroot is now MingwSysroot(): a directory in the cache, one per prefix, holding only links to the prefix's x86_64-w64-mingw32/, lib/gcc/ and lib64/gcc/ (where openSUSE keeps gcc). Through it clang finds mingw-w64, gcc's version and libstdc++ exactly as it does through the mingw gcc on PATH, and nothing else. It is made by the first mingw compile, link or CMake configure of a process, so no other command writes to the cache for it and a failure to make it is that build's error. The Toolchain field is now mingwPrefix, since clang never gets the prefix itself, and it is made absolute when the toolchain is resolved, as it ends up in link targets and directory names. Messages keep naming paths under the prefix rather than under the view. The prefix is also part of the mingw target's VariantId and of its std PCM directory. Units do not track the std PCM they import: when the fixed release built in a directory a broken build had left, the std module was rebuilt and the objects compiled against glibc's headers were only relinked, failing as before. As with a compiler update, a new directory is what gets them rebuilt. It also keeps two releases that share a cache, as the Flatpak's branches do, from rebuilding one std PCM under each other's PCMs, and two prefixes apart. Without a prefix both stay as they were: clang's own discovery has not changed. Two shorter ways do not work: --sysroot=/x86_64-w64-mingw32 loses the directory gcc's version names, so is not found, and clang ignores --gcc-toolchain and --gcc-install-dir for this target. Fixes #43 Co-Authored-By: Claude Opus 5.5 --- implementations/Crafter.Build-Clang.cpp | 17 +++++--- implementations/Crafter.Build-External.cpp | 2 +- implementations/Crafter.Build-Platform.cpp | 15 +++++-- implementations/Crafter.Build-Toolchain.cpp | 46 ++++++++++++++++++++- interfaces/Crafter.Build-Clang.cppm | 11 +++++ interfaces/Crafter.Build-Toolchain.cppm | 25 ++++++++--- tests/ToolchainResolve/main.cpp | 12 +++--- 7 files changed, 106 insertions(+), 22 deletions(-) diff --git a/implementations/Crafter.Build-Clang.cpp b/implementations/Crafter.Build-Clang.cpp index c37c362..f5943ca 100644 --- a/implementations/Crafter.Build-Clang.cpp +++ b/implementations/Crafter.Build-Clang.cpp @@ -265,6 +265,13 @@ fs::path Crafter::StdPcmDir(const Configuration& config) { if (config.target.starts_with("wasm32")) { stdPcmKey += std::format("-{:08x}", static_cast(std::hash{}(WasmBaselineFlags))); } + // The same goes for the mingw-w64 prefix the mingw std module is + // compiled against: releases up to the fix for issue #43 gave clang the + // prefix itself as the sysroot and the ones after give it a view of it, + // and two prefixes are two sets of headers. + if (config.target == "x86_64-w64-mingw32" && !GetToolchain().mingwPrefix.empty()) { + stdPcmKey += std::format("-{:08x}", static_cast(std::hash{}(GetToolchain().mingwPrefix.string()))); + } for (const std::string& f : config.wasmVariantFlags) { stdPcmKey += "+"; for (char c : f) stdPcmKey += (c == '/' || c == '\\') ? '_' : c; @@ -928,9 +935,9 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map source = StdModuleSourcePath(FileBytes(manifestPath)); if (!source) return std::format("{} names no std module", manifestPath.string()); const fs::path stdCc = (manifestPath.parent_path() / *source).lexically_normal(); @@ -862,9 +869,9 @@ std::string Crafter::GetBaseCommand(const Configuration& config) { if (config.target == "x86_64-w64-mingw32") { // libstdc++ from the mingw toolchain, said outright: a clang built to // default to libc++ (the Flatpak's) would otherwise reach for that. - // Found by clang's own mingw discovery unless there is a sysroot. + // Found by clang's own mingw discovery unless there is a prefix. command += " -stdlib=libstdc++"; - if (!tc.mingwSysroot.empty()) command += std::format(" --sysroot={}", tc.mingwSysroot.string()); + if (const fs::path& sysroot = MingwSysroot(); !sysroot.empty()) command += std::format(" --sysroot={}", sysroot.string()); return command; } command += " -stdlib=libc++"; diff --git a/implementations/Crafter.Build-Toolchain.cpp b/implementations/Crafter.Build-Toolchain.cpp index 68d9f8f..c75f627 100644 --- a/implementations/Crafter.Build-Toolchain.cpp +++ b/implementations/Crafter.Build-Toolchain.cpp @@ -43,6 +43,20 @@ namespace { while (!s.empty() && (s.back() == '\n' || s.back() == '\r' || s.back() == ' ')) s.pop_back(); return s; } + + // `link` made a symbolic link to `target`, or found one already. Another + // crafter-build may make the same link at the same moment, which is as + // good as making it; a link to anything else is an error. + void EnsureSymlink(const fs::path& target, const fs::path& link) { + std::error_code ec; + if (fs::read_symlink(link, ec) == target) return; + fs::create_directories(link.parent_path()); + fs::create_directory_symlink(target, link, ec); + std::error_code ignored; + if (ec && fs::read_symlink(link, ignored) != target) { + throw fs::filesystem_error("cannot make the mingw-w64 sysroot view", target, link, ec); + } + } } std::string Toolchain::LibcxxCompileFlags() const { @@ -136,11 +150,14 @@ Toolchain Crafter::ResolveToolchain(const ToolchainInputs& in) { tc.wasiSysroot = "/usr/share/wasi-sysroot"; } if (std::string_view dir = Env(in, "CRAFTER_MINGW_DIR"); !dir.empty()) { - tc.mingwSysroot = fs::path(dir); + // Made absolute here, once: the prefix becomes the target of links + // in the cache and part of directory names, and a relative path read + // later from another working directory would name something else. + tc.mingwPrefix = fs::absolute(fs::path(dir)); } else if (!in.hostRoot.empty() && fs::exists(in.hostRoot / "usr" / "x86_64-w64-mingw32")) { // The sandbox has no mingw-w64 of its own, and no mingw gcc on PATH // for clang to find one by. - tc.mingwSysroot = in.hostRoot / "usr"; + tc.mingwPrefix = in.hostRoot / "usr"; } return tc; } @@ -186,6 +203,31 @@ const Toolchain& Crafter::RequireToolchain() { return tc; } +const fs::path& Crafter::MingwSysroot() { + // Made on first use rather than by GetToolchain(), so only a mingw build + // writes to the cache for it, and a failure surfaces as that build's + // error instead of on every command. + static const fs::path Sysroot = []() -> fs::path { + const fs::path& prefix = GetToolchain().mingwPrefix; + if (prefix.empty()) return {}; + // A view per prefix keeps two shells with different + // CRAFTER_MINGW_DIRs, which share one cache, from rewriting each + // other's. + const fs::path sysroot = GetCacheDir() / "mingw-sysroot" / std::format("{:016x}", std::hash{}(prefix.string())); + // All clang's mingw driver takes from a sysroot other than its + // include/ and the rest of its lib/: the mingw-w64 tree (headers, + // libraries, and libstdc++ where a distro's cross gcc installs it) + // and gcc's per-target directory (crtbegin.o, libgcc, and the version + // that names libstdc++'s header directory), which openSUSE keeps + // under lib64. A link to nothing is skipped like a missing directory. + for (std::string_view entry : {"x86_64-w64-mingw32", "lib/gcc", "lib64/gcc"}) { + EnsureSymlink(prefix / entry, sysroot / entry); + } + return sysroot; + }(); + return Sysroot; +} + std::int32_t Crafter::LibcxxVersion(const fs::path& includeDir) { // "# define _LIBCPP_VERSION 230102", the one place libc++ states its // release in a form the preprocessor does not have to be run for. diff --git a/interfaces/Crafter.Build-Clang.cppm b/interfaces/Crafter.Build-Clang.cppm index b4bbc68..89583cc 100644 --- a/interfaces/Crafter.Build-Clang.cppm +++ b/interfaces/Crafter.Build-Clang.cppm @@ -628,6 +628,17 @@ export namespace Crafter { compileKey += "|B:"; compileKey += WasmBaselineFlags; } + // The mingw-w64 prefix decides what the mingw target compiles + // against, and until issue #43 clang got it as the sysroot + // itself, which put the host's own headers on the include path. + // Unkeyed, the release that fixed it kept the objects compiled + // against glibc's headers: the build after the update only + // relinked them, and failed to link as before. No prefix keeps + // the id as it was, since clang's own discovery has not changed. + if (target == "x86_64-w64-mingw32" && !toolchain.mingwPrefix.empty()) { + compileKey += "|M:"; + compileKey += toolchain.mingwPrefix.string(); + } // Sorted by ApplyStandardArgs, so flag order on the command line // doesn't split the cache. for (const std::string& a : projectArgs) { diff --git a/interfaces/Crafter.Build-Toolchain.cppm b/interfaces/Crafter.Build-Toolchain.cppm index 797fd94..d6b360a 100644 --- a/interfaces/Crafter.Build-Toolchain.cppm +++ b/interfaces/Crafter.Build-Toolchain.cppm @@ -39,11 +39,14 @@ export namespace Crafter { fs::path wasiSysroot; // The mingw-w64 install prefix, the directory that holds // x86_64-w64-mingw32/ and lib/gcc/x86_64-w64-mingw32/ (/usr for a - // distro package), passed to clang as --sysroot for the mingw target. - // Empty means clang's own discovery, which follows the mingw gcc on - // PATH. Set by CRAFTER_MINGW_DIR, else the host's /usr when running - // sandboxed and it has mingw-w64 (the sandbox's own /usr never does). - fs::path mingwSysroot; + // distro package), as an absolute path. Empty means clang's own + // discovery, which follows the mingw gcc on PATH. Set by + // CRAFTER_MINGW_DIR, else the host's /usr when running sandboxed and + // it has mingw-w64 (the sandbox's own /usr never does). clang never + // gets the prefix itself: the mingw target's --sysroot is + // MingwSysroot(), a view of it. Keyed into the mingw target's + // VariantId and std PCM directory. + fs::path mingwPrefix; // `clang++ -print-target-triple` and the first line of // `clang++ --version`; both empty when clang++ could not be run. std::string hostTriple; @@ -86,6 +89,18 @@ export namespace Crafter { const Toolchain& GetToolchain(); // GetToolchain(), or a std::runtime_error carrying its error. const Toolchain& RequireToolchain(); + // The --sysroot of every compile, link and CMake configure for the mingw + // target: a directory in crafter-build's cache holding only links to the + // x86_64-w64-mingw32/, lib/gcc/ and lib64/gcc/ of GetToolchain()'s + // mingwPrefix, made on the first call of a process. Empty without a + // prefix. The prefix cannot be the sysroot: clang takes a mingw sysroot + // for a tree that holds mingw-w64 alone and searches its include/ and + // lib/ as well, which for /usr (the host's, in the Flatpak) are the + // Linux system's own. A Windows build then saw glibc's headers, and + // libstdc++, finding , had shared_ptr read + // __libc_single_threaded, which nothing on Windows defines (issue #43). + // Throws when the view cannot be made. + const fs::path& MingwSysroot(); // _LIBCPP_VERSION as the <__config> in `includeDir` (an include/c++/v1) // defines it, XXYYZZ; 0 when the file is missing or has no such line. std::int32_t LibcxxVersion(const fs::path& includeDir); diff --git a/tests/ToolchainResolve/main.cpp b/tests/ToolchainResolve/main.cpp index 81969eb..1912418 100644 --- a/tests/ToolchainResolve/main.cpp +++ b/tests/ToolchainResolve/main.cpp @@ -61,7 +61,7 @@ int main() { Check(tc.libcxxVersion == LibcxxVersion("/usr/include/c++/v1"), "the system libc++ headers state the release of the system module source"); Check(tc.stdModulesDir.empty(), "no crafter-build prefix, no shipped module sources"); Check(tc.wasiSysroot == "/usr/share/wasi-sysroot", "the system wasi sysroot is the default"); - Check(tc.mingwSysroot.empty(), "outside a sandbox, clang finds mingw-w64 itself: no --sysroot"); + Check(tc.mingwPrefix.empty(), "outside a sandbox, clang finds mingw-w64 itself: no prefix"); Check(tc.LibcxxCompileFlags().empty() && tc.LibcxxLinkFlags().empty(), "the PATH fallback adds no libc++ flags"); } { @@ -144,7 +144,9 @@ int main() { in.env["CRAFTER_MINGW_DIR"] = "/opt/mingw"; Toolchain tc = ResolveToolchain(in); Check(tc.wasiSysroot == "/opt/wasi", "CRAFTER_BUILD_WASI_SYSROOT overrides the wasi sysroot"); - Check(tc.mingwSysroot == "/opt/mingw", "CRAFTER_MINGW_DIR names the mingw prefix passed as --sysroot"); + Check(tc.mingwPrefix == "/opt/mingw", "CRAFTER_MINGW_DIR names the mingw prefix"); + in.env["CRAFTER_MINGW_DIR"] = "mingw"; + Check(ResolveToolchain(in).mingwPrefix == fs::absolute("mingw"), "a relative CRAFTER_MINGW_DIR is resolved against the working directory once, when the toolchain is"); } { // Sandboxed, the host's /usr stands in for the missing mingw-w64, @@ -153,11 +155,11 @@ int main() { fs::remove_all(host); ToolchainInputs in; in.hostRoot = host; - Check(ResolveToolchain(in).mingwSysroot.empty(), "a host without mingw-w64 gives no --sysroot"); + Check(ResolveToolchain(in).mingwPrefix.empty(), "a host without mingw-w64 gives no mingw prefix"); fs::create_directories(host / "usr" / "x86_64-w64-mingw32"); - Check(ResolveToolchain(in).mingwSysroot == host / "usr", "a host with mingw-w64 gives its /usr as the mingw prefix"); + Check(ResolveToolchain(in).mingwPrefix == host / "usr", "a host with mingw-w64 gives its /usr as the mingw prefix"); in.env["CRAFTER_MINGW_DIR"] = "/opt/mingw"; - Check(ResolveToolchain(in).mingwSysroot == "/opt/mingw", "CRAFTER_MINGW_DIR wins over the host's"); + Check(ResolveToolchain(in).mingwPrefix == "/opt/mingw", "CRAFTER_MINGW_DIR wins over the host's"); fs::remove_all(host); } { From b162cc05529d8a8df031287951f7b8b6ed86ca1a Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 20:48:06 +0100 Subject: [PATCH 02/12] test: a mingw build sees no header from its prefix's include/ MingwSysroot stages a prefix shaped like a Linux system's /usr with a mingw-w64 cross toolchain in it, without needing a real one: glibc's in include/, gcc's per-target directory, and a stand-in mingw-w64 tree whose libstdc++ has an empty std module. A static library for x86_64-w64-mingw32, so that nothing is linked, compiles a C++ and a C source that stop with #error when they see the prefix's include/ or miss mingw-w64's or libstdc++'s headers. Against the library before the fix, the C++ source stopped on . The test also checks that the std module is not cached in the directory older releases built it in against the prefix itself. It needs only clang, so it runs in CI too, where no mingw-w64 is installed. The stand-in std module's source is written through std::format: the build's import scanner reads string literals as code, and the literal "export module std;" made the test itself a unit of a module std. VariantId checks that a prefix gives a mingw configuration a variant of its own, that two prefixes give two, and that other targets ignore it. Co-Authored-By: Claude Opus 5.5 --- project.cpp | 1 + tests/MingwSysroot/fixture/cprobe.c | 13 ++++ tests/MingwSysroot/fixture/probe.cpp | 16 +++++ tests/MingwSysroot/main.cpp | 97 ++++++++++++++++++++++++++++ tests/VariantId/main.cpp | 19 ++++++ 5 files changed, 146 insertions(+) create mode 100644 tests/MingwSysroot/fixture/cprobe.c create mode 100644 tests/MingwSysroot/fixture/probe.cpp create mode 100644 tests/MingwSysroot/main.cpp diff --git a/project.cpp b/project.cpp index 83f70d6..adedaef 100644 --- a/project.cpp +++ b/project.cpp @@ -253,6 +253,7 @@ extern "C" Configuration CrafterBuildProject(std::span a cfg.AddTest("ShaderVariantBuild").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ShaderIncludeChange").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("MingwMissing").Dependencies({ CrafterBuildLib.get() }); + cfg.AddTest("MingwSysroot").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("GitSubmodules").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("LinkFlagsChange").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ExternalArtifactTime").Dependencies({ CrafterBuildLib.get() }); diff --git a/tests/MingwSysroot/fixture/cprobe.c b/tests/MingwSysroot/fixture/cprobe.c new file mode 100644 index 0000000..3cd5215 --- /dev/null +++ b/tests/MingwSysroot/fixture/cprobe.c @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +// probe.cpp's checks for the C sources, which are compiled by a command of +// their own. +#if __has_include() +#error the include/ of the prefix is on the C include path: a Windows build sees the headers of the Linux system +#endif +#if !__has_include(<_mingw.h>) +#error the mingw-w64 headers are not on the C include path +#endif + +void CProbe(void) {} diff --git a/tests/MingwSysroot/fixture/probe.cpp b/tests/MingwSysroot/fixture/probe.cpp new file mode 100644 index 0000000..704d42b --- /dev/null +++ b/tests/MingwSysroot/fixture/probe.cpp @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +// Built for x86_64-w64-mingw32 against the prefix tests/MingwSysroot stages: +// a Linux system's /usr with a mingw-w64 cross toolchain in it. +#if __has_include() +#error the include/ of the prefix is on the include path: a Windows build sees the headers of the Linux system +#endif +#if !__has_include(<_mingw.h>) +#error the mingw-w64 headers are not on the include path +#endif +#if !__has_include() +#error the libstdc++ headers are not on the include path: clang found no gcc version under lib/gcc of the prefix +#endif + +void Probe() {} diff --git a/tests/MingwSysroot/main.cpp b/tests/MingwSysroot/main.cpp new file mode 100644 index 0000000..2388acf --- /dev/null +++ b/tests/MingwSysroot/main.cpp @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +import std; +import Crafter.Build; +namespace fs = std::filesystem; +using namespace Crafter; + +// A mingw-w64 prefix that is a whole Linux system's /usr must not hand that +// system's headers to a Windows build. Inside the Flatpak the prefix is the +// host's /usr, and clang got it as the --sysroot. clang's mingw driver takes +// a sysroot for a tree that holds mingw-w64 alone and searches its include/ +// after mingw-w64's own headers, so every header mingw-w64 lacks came from +// glibc: libstdc++ found and had shared_ptr read +// __libc_single_threaded, which nothing on Windows defines, and every +// program that used one failed to link (issue #43). CRAFTER_MINGW_DIR=/usr +// did the same outside the Flatpak. +// +// The prefix staged here has the shape of such a /usr and needs no real +// mingw-w64: glibc's in include/, gcc's per-target +// directory, which clang derives libstdc++'s header directory from, and a +// stand-in mingw-w64 tree whose libstdc++ has a std module with nothing in +// it. The fixture is a static library, so nothing is linked, and its C++ and +// C sources stop the compile when they see the prefix's include/ or miss +// mingw-w64's or libstdc++'s headers. +// +// The std module such a build compiles is not the one releases before the +// fix compiled against the prefix itself, so it has to live in a directory +// of its own: the Flatpak's branches share one cache, and two releases +// taking turns rebuilding one PCM do it under the PCMs built against it. + +extern "C" int setenv(const char* name, const char* value, int overwrite); + +namespace { + std::int32_t failures = 0; + + void Check(bool cond, std::string_view msg) { + if (!cond) { + std::println(std::cerr, "FAIL: {}", msg); + ++failures; + } + } + + void Write(const fs::path& file, std::string_view text) { + fs::create_directories(file.parent_path()); + std::ofstream(file, std::ios::binary | std::ios::trunc) << text; + } +} + +int main() { + const fs::path scratch = fs::temp_directory_path() / "crafter-build-mingw-sysroot"; + std::error_code ec; + fs::remove_all(scratch, ec); + fs::create_directories(scratch); + const fs::path project = scratch / "project"; + fs::copy(fs::current_path() / "tests" / "MingwSysroot" / "fixture", project, fs::copy_options::recursive); + + const fs::path prefix = scratch / "usr"; + Write(prefix / "include" / "sys" / "single_threaded.h", "extern char __libc_single_threaded;\n"); + fs::create_directories(prefix / "lib" / "gcc" / "x86_64-w64-mingw32" / "15.1.0"); + const fs::path mingw = prefix / "x86_64-w64-mingw32"; + Write(mingw / "include" / "_mingw.h", ""); + Write(mingw / "include" / "c++" / "15.1.0" / "memory", ""); + // Through std::format because the build's import scanner reads string + // literals as code: one holding "module std;" made this file a unit of a + // module std, and the test did not build. + Write(mingw / "include" / "c++" / "15.1.0" / "bits" / "std.cc", std::format("export module {};\n", "std")); + Write(mingw / "lib" / "libstdc++.modules.json", R"({"version": 1, "modules": [{"logical-name": "std", "source-path": "../include/c++/15.1.0/bits/std.cc", "is-std-library": true}]})"); + // Before anything resolves the toolchain, which reads both once. The + // std module cache goes to the scratch directory too. + setenv("CRAFTER_MINGW_DIR", prefix.c_str(), 1); + setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1); + + Configuration lib; + lib.path = project; + lib.name = "probe"; + lib.outputName = "probe"; + lib.target = "x86_64-w64-mingw32"; + lib.type = ConfigurationType::LibraryStatic; + std::array ifaces = {}; + std::array impls = { "probe" }; + lib.GetInterfacesAndImplementations(ifaces, impls); + lib.cFiles = { project / "cprobe" }; + std::unordered_map> depResults; + std::mutex depMutex; + const BuildResult built = Build(lib, depResults, depMutex); + Check(built.result.empty(), std::format("a Windows build sees mingw-w64's and libstdc++'s headers and not the prefix's own: {}", built.result)); + const std::string unkeyed = std::format("{}-{}-{}", lib.target, lib.march, GetToolchain().Key()); + Check(StdPcmDir(lib).filename() != unkeyed, std::format("the std module of a mingw build with a prefix is not cached where older releases built it against the prefix itself: {}", StdPcmDir(lib).string())); + + fs::remove_all(scratch, ec); + if (failures > 0) { + std::println(std::cerr, "{} assertions failed", failures); + return 1; + } + return 0; +} diff --git a/tests/VariantId/main.cpp b/tests/VariantId/main.cpp index e4d2a36..e658b3e 100644 --- a/tests/VariantId/main.cpp +++ b/tests/VariantId/main.cpp @@ -160,6 +160,25 @@ int main() { Check(cfg.VariantId() == cfg.VariantId(GetToolchain()), "VariantId() is the variant under the running toolchain"); Check(cfg.BuildDir().filename() == cfg.VariantId(GetToolchain()), "BuildDir is named for the running toolchain's variant"); } + { + // The mingw-w64 prefix decides what a mingw build compiles against. + // Before issue #43 it was clang's sysroot as is, which put the host's + // headers on the include path, and with the prefix left out of the + // id, the release that fixed that reused the objects built against + // them: the next build only relinked, and failed to link as before. + Configuration mingw = MakeBase(); + mingw.target = "x86_64-w64-mingw32"; + Toolchain discovered; + discovered.clangxx = "/app/bin/clang++"; + Toolchain hostUsr = discovered; + hostUsr.mingwPrefix = "/run/host/usr"; + Toolchain optMingw = discovered; + optMingw.mingwPrefix = "/opt/mingw"; + Check(mingw.VariantId(discovered) != mingw.VariantId(hostUsr), "a mingw prefix perturbs a mingw configuration's VariantId"); + Check(mingw.VariantId(hostUsr) != mingw.VariantId(optMingw), "two mingw prefixes are two variants"); + Configuration host = MakeBase(); + Check(host.VariantId(discovered) == host.VariantId(hostUsr), "the mingw prefix leaves other targets' variants alone"); + } if (Failures > 0) { std::println(std::cerr, "{} assertions failed", Failures); From 5fdbf2baee3d99028df5a6a13e1af7d9411ec5e6 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 20:48:06 +0100 Subject: [PATCH 03/12] docs: the mingw sysroot view in the README and AGENTS.template.md The README says which directories of a mingw-w64 prefix clang sees, and why the rest of a prefix like /usr is kept out of a Windows build. AGENTS.template.md names MingwSysroot in the :Toolchain partition, with the reason the prefix itself is never a sysroot, counts the views among what goes in the cache, and adds the mingw prefix to what VariantId hashes. Co-Authored-By: Claude Opus 5.5 --- AGENTS.template.md | 6 +++--- README.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/AGENTS.template.md b/AGENTS.template.md index cb23933..b60a81a 100644 --- a/AGENTS.template.md +++ b/AGENTS.template.md @@ -22,7 +22,7 @@ crafter-build is a C++26 modules build system. A project is described by a `proj Partitions and what they own: -- `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release. +- `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release. `MingwSysroot` makes the mingw target's `--sysroot`, a view in the cache of the mingw-w64 prefix's own directories: the prefix itself is never a sysroot, since clang would search its `include/` and `lib/`, and for `/usr` those are the Linux system's (issue #43). - `:Clang` holds `Configuration`, `Build()`, `VariantId()`, `ApplyStandardArgs`, `HostTarget`, the CLI entry point `Crafter::Run`, and the lint, test and script data types. - `:Platform` runs commands, implements `LoadProject` (compile and load `project.cpp`), the host PCM cache, `#pragma crafter dependency`, and all OS-specific code. - `:Interface` and `:Implementation` model module units, their staleness checks and their compiles. @@ -118,14 +118,14 @@ Beyond the rules: - There is no host-platform preprocessor split: the host is Linux. Target differences (mingw, wasm) branch on `cfg.target` strings at runtime, in the library as well as in `project.cpp` and tests. - A build step fails when its command exits nonzero, never because it printed. Use `RunBuildCommand` or `RunCommandChecked`; warnings go to `Progress::Diagnostic`. - Every output path goes through `Configuration::BinDir()`, `BuildDir()` and `PcmDir()`. Hand-rolled `path / "bin" / ...` clobbered variants before; do not add new ones. -- Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, the wasm baseline and variant flags, uninterpreted project args and the toolchain (`Toolchain::Key()`). +- Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, the wasm baseline and variant flags, the mingw-w64 prefix for the mingw target, uninterpreted project args and the toolchain (`Toolchain::Key()`). ## Design rules These were decided deliberately. Do not undo them without the maintainer's say-so. 1. **Configuration is C++.** No JSON, YAML, TOML or DSL for project, runner or lint configuration. Lint rules are C++ callbacks; there is no clang-format or clang-tidy integration and no separate "format rule" type. A rule that calls `ctx.SetContent` is a transform, and that is the whole mechanism. -2. **Per-project artifacts go in `/build` and `/bin`. Only cross-project things go in `~/.cache/crafter.build/`:** the std PCM and Crafter.Build PCMs per target and march, and external clones keyed by their spec hash. +2. **Per-project artifacts go in `/build` and `/bin`. Only cross-project things go in `~/.cache/crafter.build/`:** the std PCM and Crafter.Build PCMs per target and march, external clones keyed by their spec hash, and the mingw sysroot views keyed by prefix. 3. **Sources ship, PCMs build locally.** The Flatpak installs `*.cppm` under `/app/share/crafter-build/`. Nothing target-baked ships except the executable and the toolchain beside it, and the executable is built for a portable `-march`. 4. **crafter-build is a monolithic executable with `--export-dynamic`.** `project.so` resolves `Crafter::` symbols from the running binary at dlopen time. There is no Windows host; `x86_64-w64-mingw32` is a cross-compile target for user projects only. 5. **Convention over configuration.** Lead with the zero-config path. Env var, then CLI flag, then full `project.cpp` are the escalation, not the entry point. diff --git a/README.md b/README.md index c57beb6..11248aa 100644 --- a/README.md +++ b/README.md @@ -201,7 +201,7 @@ For elaborate setups (rsync, ssh ControlMaster, env vars on the spawn, `QEMU_LD_ For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=` is passed to clang, and the std module is precompiled against the sysroot's libc++ headers. Ordinary compiles are pointed at the same headers (`-nostdinc++ -isystem /usr/include/c++/v1`), so a unit that `#include`s a standard header gets the sysroot's libc++ and not the host toolchain's. The module source has to come from the same libc++ release as those headers, so it is taken from `/usr/share/libc++/v1/std.cppm` when the sysroot ships it, else from the toolchain when both are the same major release, else from the sources crafter-build carries for other releases (`share/crafter-build/libc++-modules//`, `crafter-build --version` lists them). Alpine is the common case for the last two: its `libc++-dev` has the headers and no module source. When nothing matches, the build stops with one line naming both releases, installing that release's `share/libc++/v1` directory into the sysroot resolves it. The cheapest aarch64 sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root --config --dbpath /var/lib/pacman -Sy …` (point pacman at ALARM's mirrors via explicit config to avoid pulling host-arch packages). -For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed, with libstdc++ 15 or newer for its std module. clang finds it by the `x86_64-w64-mingw32-gcc` on `PATH`; `CRAFTER_MINGW_DIR` names another prefix (the directory holding `x86_64-w64-mingw32/` and `lib/gcc/`), and inside the Flatpak the host's `/usr` is used when it has mingw-w64. Without one the build stops with a message saying where it looked. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported, there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. +For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed, with libstdc++ 15 or newer for its std module. clang finds it by the `x86_64-w64-mingw32-gcc` on `PATH`; `CRAFTER_MINGW_DIR` names another prefix (the directory holding `x86_64-w64-mingw32/` and `lib/gcc/`), and inside the Flatpak the host's `/usr` is used when it has mingw-w64. Without one the build stops with a message saying where it looked. clang is shown only the prefix's `x86_64-w64-mingw32/` and `lib/gcc/`, as links in `~/.cache/crafter.build/mingw-sysroot/`: it searches the `include/` and `lib/` of whatever it gets as the sysroot, and those of a prefix like `/usr` are the Linux system's own, glibc's headers among them. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported, there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. For `--target=wasm32-wasip1`, the WASI sysroot is the one beside the toolchain (the Flatpak bundles one) unless `CRAFTER_BUILD_WASI_SYSROOT` names another, and tests run under `wasmtime`. C++ exceptions work as on the other targets: every unit and the std module are compiled with `-fwasm-exceptions` in the standardized form of WebAssembly exception handling (exnref), against the exception-enabled libc++ in the sysroot's `lib//eh/`, and `-lunwind` is linked automatically. A sysroot without that libc++ (wasi-sdk before 33) stops the build with one line saying so. The result runs on engines with that proposal: Chrome 137, Firefox 131 and Safari 18.4 or newer, and the wasmtime the Flatpak bundles. A page from `EnableWasiBrowserRuntime` checks first and names what an older browser lacks in the console, instead of failing to compile the module. An exception that escapes `main` leaves the program as an uncaught WebAssembly exception (wasmtime reports `thrown Wasm exception` and exits 1, a page gets a `WebAssembly.Exception`) without calling `std::terminate`, so a `std::set_terminate` handler does not run and nothing prints the exception's type or `what()`: wasm exception handling unwinds in a single phase, and nothing finds out that no handler is left until the stack is gone. Catch it in `main` to report it. From 06b61e84f9c8caa450ffe51e1bd135e47133eba9 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:58:30 +0100 Subject: [PATCH 04/12] fix(mingw): view only a prefix that holds a Linux system The first fix showed clang every mingw prefix through a view of its mingw-w64 parts. That broke a prefix that holds mingw-w64 alone, where Windows libraries installed with --prefix land in include/ and lib/: with CRAFTER_MINGW_DIR=/opt/mingw, and -lSDL2 were no longer found. Such a prefix is clang's sysroot whole again. Only a prefix that also holds a Linux system gets the view (Toolchain::mingwPrefixIsSystem): the host's /usr inside the Flatpak, whatever its include/ holds, and a prefix whose include/ has a C library's features.h, which every glibc and musl header includes and mingw-w64 has no use for, as CRAFTER_MINGW_DIR=/usr has. MakeMingwSysroot makes the view in the cache, or in $XDG_RUNTIME_DIR/crafter.build when the cache's filesystem takes no symbolic links (vfat, exFAT, CIFS without unix extensions): the XDG spec promises them there, and nothing else in the cache needs them. Its path is absolute, because CMake runs the compiler from directories of its own and a relative cache did not resolve from there. The prefix is resolved once, absolute and lexically normal, so /opt/mingw/ and /opt/mingw are one prefix and not two builds. clang names a header by the path it found it at, so every diagnostic from a viewed mingw-w64 header named a hashed cache directory. RunBuildCommand now writes paths into the view as paths into the prefix, as BuildStdPcm did for its manifest messages already. Also from the review of PR #44: SysrootCompileFlags builds a mingw command, which can make a view, so it redirects the cache like the other tests; the manifest filter's comment names what it still keeps out; the C path asks MingwSysroot() once. Co-Authored-By: Claude Opus 5.5 --- implementations/Crafter.Build-Clang.cpp | 8 +- implementations/Crafter.Build-Platform.cpp | 25 +++-- implementations/Crafter.Build-Toolchain.cpp | 114 +++++++++++++++----- interfaces/Crafter.Build-Toolchain.cppm | 46 +++++--- tests/MingwSysroot/fixture/failing.cpp | 6 ++ tests/MingwSysroot/main.cpp | 106 +++++++++++++----- tests/SysrootCompileFlags/main.cpp | 13 +++ tests/ToolchainResolve/main.cpp | 26 +++++ 8 files changed, 264 insertions(+), 80 deletions(-) create mode 100644 tests/MingwSysroot/fixture/failing.cpp diff --git a/implementations/Crafter.Build-Clang.cpp b/implementations/Crafter.Build-Clang.cpp index f5943ca..138ffd7 100644 --- a/implementations/Crafter.Build-Clang.cpp +++ b/implementations/Crafter.Build-Clang.cpp @@ -935,9 +935,9 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map globs, std::string_view name std::string Crafter::RunBuildCommand(std::string_view cmd) { Progress::EchoCommand(cmd); CommandResult r = RunCommandChecked(cmd); + // A mingw build against a system prefix sees mingw-w64 through a view in + // the cache, and clang names every header by the path it found it at. + r.output = MingwViewToPrefix(std::move(r.output)); if (r.exitCode == 0) { // Warnings from a step that succeeded. Show them — until now they were @@ -737,10 +740,13 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) { // library, naming the std module's source relative to itself. It is // looked for in the library directories the compiler searches for // this target, as it would look for libstdc++ itself, but only in - // those that belong to the mingw toolchain: the list also has the - // generic /lib, where the host's own (Linux) libstdc++ keeps - // its manifest. A toolchain that isn't there is a build error, not - // an exception: Build() runs this with its worker threads started. + // those that belong to the mingw toolchain. The list also holds + // clang's resource directory, the sysroot's generic lib/, where a + // prefix given whole can keep a libstdc++ built for another target, + // and, when clang finds no gcc directory, an empty entry that would + // name a manifest in the working directory. A toolchain that isn't + // there is a build error, not an exception: Build() runs this with + // its worker threads started. const Toolchain& tc = GetToolchain(); const std::string where = tc.mingwPrefix.empty() ? std::string("where clang finds it by the x86_64-w64-mingw32 gcc on PATH") @@ -775,13 +781,10 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) { } } if (manifestPath.empty()) return missing; - // Found through the view MingwSysroot() makes of the prefix, and - // named from here on by its place in the prefix, which is the - // directory the user knows and can go and look at. - if (!tc.mingwPrefix.empty()) { - const fs::path inPrefix = manifestPath.lexically_relative(MingwSysroot()); - if (!inPrefix.empty() && *inPrefix.begin() != "..") manifestPath = tc.mingwPrefix / inPrefix; - } + // Found through the view MingwSysroot() may have made of the prefix, + // and named from here on by its place in the prefix, the directory + // the user knows and can go and look at. + manifestPath = fs::path(MingwViewToPrefix(manifestPath.string())); const std::optional source = StdModuleSourcePath(FileBytes(manifestPath)); if (!source) return std::format("{} names no std module", manifestPath.string()); const fs::path stdCc = (manifestPath.parent_path() / *source).lexically_normal(); diff --git a/implementations/Crafter.Build-Toolchain.cpp b/implementations/Crafter.Build-Toolchain.cpp index c75f627..84a1de3 100644 --- a/implementations/Crafter.Build-Toolchain.cpp +++ b/implementations/Crafter.Build-Toolchain.cpp @@ -54,9 +54,26 @@ namespace { fs::create_directory_symlink(target, link, ec); std::error_code ignored; if (ec && fs::read_symlink(link, ignored) != target) { - throw fs::filesystem_error("cannot make the mingw-w64 sysroot view", target, link, ec); + throw fs::filesystem_error("cannot make a link", target, link, ec); } } + + // `dir` made absolute and spelled one way, once, when the toolchain is + // resolved. The mingw prefix ends up as the target of links and in the + // names of build and cache directories: a relative path read later from + // another working directory would name something else, and /opt/mingw/ + // next to /opt/mingw would rebuild everything for the same directory. + fs::path PrefixPath(const fs::path& dir) { + fs::path normal = fs::absolute(dir).lexically_normal(); + if (!normal.has_filename() && normal.has_relative_path()) normal = normal.parent_path(); + return normal; + } + + // The view MingwSysroot() made and the prefix it shows, for + // MingwViewToPrefix. Written once, before ViewMade is set. + fs::path MadeView; + fs::path ViewedPrefix; + std::atomic ViewMade = false; } std::string Toolchain::LibcxxCompileFlags() const { @@ -150,14 +167,19 @@ Toolchain Crafter::ResolveToolchain(const ToolchainInputs& in) { tc.wasiSysroot = "/usr/share/wasi-sysroot"; } if (std::string_view dir = Env(in, "CRAFTER_MINGW_DIR"); !dir.empty()) { - // Made absolute here, once: the prefix becomes the target of links - // in the cache and part of directory names, and a relative path read - // later from another working directory would name something else. - tc.mingwPrefix = fs::absolute(fs::path(dir)); + tc.mingwPrefix = PrefixPath(fs::path(dir)); + // A prefix that holds a Linux system as well as mingw-w64, /usr + // itself, has that system's C library in include/: features.h, which + // every glibc and musl header includes and mingw-w64 has no use for. + // One that holds mingw-w64 alone stays the sysroot, so Windows + // libraries installed into its include/ and lib/ stay visible. + tc.mingwPrefixIsSystem = fs::exists(tc.mingwPrefix / "include" / "features.h"); } else if (!in.hostRoot.empty() && fs::exists(in.hostRoot / "usr" / "x86_64-w64-mingw32")) { // The sandbox has no mingw-w64 of its own, and no mingw gcc on PATH - // for clang to find one by. - tc.mingwPrefix = in.hostRoot / "usr"; + // for clang to find one by. The host's /usr is the host's system, + // whether or not its C library's headers are installed. + tc.mingwPrefix = PrefixPath(in.hostRoot / "usr"); + tc.mingwPrefixIsSystem = true; } return tc; } @@ -203,31 +225,69 @@ const Toolchain& Crafter::RequireToolchain() { return tc; } -const fs::path& Crafter::MingwSysroot() { - // Made on first use rather than by GetToolchain(), so only a mingw build - // writes to the cache for it, and a failure surfaces as that build's - // error instead of on every command. - static const fs::path Sysroot = []() -> fs::path { - const fs::path& prefix = GetToolchain().mingwPrefix; - if (prefix.empty()) return {}; - // A view per prefix keeps two shells with different - // CRAFTER_MINGW_DIRs, which share one cache, from rewriting each - // other's. - const fs::path sysroot = GetCacheDir() / "mingw-sysroot" / std::format("{:016x}", std::hash{}(prefix.string())); - // All clang's mingw driver takes from a sysroot other than its - // include/ and the rest of its lib/: the mingw-w64 tree (headers, - // libraries, and libstdc++ where a distro's cross gcc installs it) - // and gcc's per-target directory (crtbegin.o, libgcc, and the version - // that names libstdc++'s header directory), which openSUSE keeps - // under lib64. A link to nothing is skipped like a missing directory. - for (std::string_view entry : {"x86_64-w64-mingw32", "lib/gcc", "lib64/gcc"}) { - EnsureSymlink(prefix / entry, sysroot / entry); +fs::path Crafter::MakeMingwSysroot(const fs::path& prefix, std::span parents) { + std::string reasons; + for (const fs::path& parent : parents) { + // One view per prefix: two shells with different CRAFTER_MINGW_DIRs + // share one cache and would otherwise rewrite each other's. + const fs::path sysroot = parent / "mingw-sysroot" / std::format("{:016x}", std::hash{}(prefix.string())); + try { + // All clang's mingw driver takes from a sysroot other than its + // include/ and the rest of its lib/: the mingw-w64 tree (headers, + // libraries, and libstdc++ where a distro's cross gcc installs + // it) and gcc's per-target directory (crtbegin.o, libgcc, and the + // version that names libstdc++'s header directory), which + // openSUSE keeps under lib64. A link to nothing is skipped like a + // missing directory. + for (std::string_view entry : {"x86_64-w64-mingw32", "lib/gcc", "lib64/gcc"}) { + EnsureSymlink(prefix / entry, sysroot / entry); + } + return sysroot; + } catch (const std::exception& e) { + reasons += std::format("; {}", e.what()); } - return sysroot; + } + throw std::runtime_error(std::format("cannot make a sysroot that shows clang only the mingw-w64 parts of {}{}", prefix.string(), reasons)); +} + +const fs::path& Crafter::MingwSysroot() { + // Made on first use rather than by GetToolchain(), so that only what + // compiles for the mingw target, a build or an AST lint of one, writes + // anything for it, and a failure surfaces as that command's error + // instead of on every command. + static const fs::path Sysroot = []() -> fs::path { + const Toolchain& tc = GetToolchain(); + if (!tc.mingwPrefixIsSystem) return tc.mingwPrefix; + // In the cache, or where the XDG spec promises symbolic links when + // the cache's filesystem takes none (vfat, exFAT, CIFS without unix + // extensions), which costs nothing else crafter-build caches. + // Absolute, because CMake runs the compiler from directories of its + // own. + std::vector parents = { fs::absolute(GetCacheDir()) }; + if (const char* runtime = std::getenv("XDG_RUNTIME_DIR"); runtime && *runtime) { + parents.push_back(fs::absolute(fs::path(runtime) / "crafter.build")); + } + fs::path view = MakeMingwSysroot(tc.mingwPrefix, parents); + MadeView = view; + ViewedPrefix = tc.mingwPrefix; + ViewMade.store(true, std::memory_order_release); + return view; }(); return Sysroot; } +std::string Crafter::MingwViewToPrefix(std::string text) { + if (!ViewMade.load(std::memory_order_acquire)) return text; + // With the separator, so a --sysroot= naming the view itself, in a + // command line the output quotes, stays what was run. + const std::string view = (MadeView / "").string(); + const std::string prefix = (ViewedPrefix / "").string(); + for (std::size_t at = text.find(view); at != std::string::npos; at = text.find(view, at + prefix.size())) { + text.replace(at, view.size(), prefix); + } + return text; +} + std::int32_t Crafter::LibcxxVersion(const fs::path& includeDir) { // "# define _LIBCPP_VERSION 230102", the one place libc++ states its // release in a form the preprocessor does not have to be run for. diff --git a/interfaces/Crafter.Build-Toolchain.cppm b/interfaces/Crafter.Build-Toolchain.cppm index d6b360a..6f1b68c 100644 --- a/interfaces/Crafter.Build-Toolchain.cppm +++ b/interfaces/Crafter.Build-Toolchain.cppm @@ -39,14 +39,20 @@ export namespace Crafter { fs::path wasiSysroot; // The mingw-w64 install prefix, the directory that holds // x86_64-w64-mingw32/ and lib/gcc/x86_64-w64-mingw32/ (/usr for a - // distro package), as an absolute path. Empty means clang's own - // discovery, which follows the mingw gcc on PATH. Set by + // distro package), absolute and lexically normal. Empty means clang's + // own discovery, which follows the mingw gcc on PATH. Set by // CRAFTER_MINGW_DIR, else the host's /usr when running sandboxed and - // it has mingw-w64 (the sandbox's own /usr never does). clang never - // gets the prefix itself: the mingw target's --sysroot is - // MingwSysroot(), a view of it. Keyed into the mingw target's - // VariantId and std PCM directory. + // it has mingw-w64 (the sandbox's own /usr never does). What the + // mingw target passes as --sysroot is MingwSysroot(), not this: + // see mingwPrefixIsSystem. fs::path mingwPrefix; + // The prefix holds a Linux system as well as mingw-w64: the host's + // /usr in the sandbox, or a prefix whose include/ has a C library's + // features.h, as /usr has. clang searches a mingw sysroot's include/ + // and lib/, so such a prefix is shown to it only through a view of + // its mingw-w64 parts; any other prefix is the sysroot itself, and + // Windows libraries installed into its include/ and lib/ are found. + bool mingwPrefixIsSystem = false; // `clang++ -print-target-triple` and the first line of // `clang++ --version`; both empty when clang++ could not be run. std::string hostTriple; @@ -89,18 +95,32 @@ export namespace Crafter { const Toolchain& GetToolchain(); // GetToolchain(), or a std::runtime_error carrying its error. const Toolchain& RequireToolchain(); + // Makes, in the first of `parents` that can hold one, a directory that + // shows clang only the mingw-w64 parts of `prefix`: links named + // x86_64-w64-mingw32, lib/gcc and lib64/gcc to the same paths under + // `prefix`, at /mingw-sysroot/, and returns it. + // A view already there is reused; one another process makes at the same + // moment is as good as one made here. Throws when no parent can hold + // one, with the reason for each. + fs::path MakeMingwSysroot(const fs::path& prefix, std::span parents); // The --sysroot of every compile, link and CMake configure for the mingw - // target: a directory in crafter-build's cache holding only links to the - // x86_64-w64-mingw32/, lib/gcc/ and lib64/gcc/ of GetToolchain()'s - // mingwPrefix, made on the first call of a process. Empty without a - // prefix. The prefix cannot be the sysroot: clang takes a mingw sysroot - // for a tree that holds mingw-w64 alone and searches its include/ and - // lib/ as well, which for /usr (the host's, in the Flatpak) are the - // Linux system's own. A Windows build then saw glibc's headers, and + // target, settled on the first call of a process: empty without a + // mingwPrefix, the prefix itself for one that holds mingw-w64 alone, and + // for a system prefix the view MakeMingwSysroot makes in crafter-build's + // cache, or in $XDG_RUNTIME_DIR/crafter.build when the cache's + // filesystem takes no symbolic links. A system prefix cannot be the + // sysroot: clang takes a mingw sysroot for a tree that holds mingw-w64 + // alone and searches its include/ and lib/ as well, which for /usr are + // the Linux system's own. A Windows build then saw glibc's headers, and // libstdc++, finding , had shared_ptr read // __libc_single_threaded, which nothing on Windows defines (issue #43). // Throws when the view cannot be made. const fs::path& MingwSysroot(); + // `text` with every path into the view MingwSysroot() made written as + // the same path into the prefix it shows, so that compiler messages name + // a header where the user installed it rather than in a hashed cache + // directory; `text` unchanged when no view was made. + std::string MingwViewToPrefix(std::string text); // _LIBCPP_VERSION as the <__config> in `includeDir` (an include/c++/v1) // defines it, XXYYZZ; 0 when the file is missing or has no such line. std::int32_t LibcxxVersion(const fs::path& includeDir); diff --git a/tests/MingwSysroot/fixture/failing.cpp b/tests/MingwSysroot/fixture/failing.cpp new file mode 100644 index 0000000..254ec5b --- /dev/null +++ b/tests/MingwSysroot/fixture/failing.cpp @@ -0,0 +1,6 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +// Stops in a header of the stand-in mingw-w64, so that the error clang +// reports is located in one. +#include diff --git a/tests/MingwSysroot/main.cpp b/tests/MingwSysroot/main.cpp index 2388acf..3964b36 100644 --- a/tests/MingwSysroot/main.cpp +++ b/tests/MingwSysroot/main.cpp @@ -17,17 +17,17 @@ using namespace Crafter; // did the same outside the Flatpak. // // The prefix staged here has the shape of such a /usr and needs no real -// mingw-w64: glibc's in include/, gcc's per-target -// directory, which clang derives libstdc++'s header directory from, and a -// stand-in mingw-w64 tree whose libstdc++ has a std module with nothing in -// it. The fixture is a static library, so nothing is linked, and its C++ and -// C sources stop the compile when they see the prefix's include/ or miss -// mingw-w64's or libstdc++'s headers. +// mingw-w64: glibc's features.h and in include/, +// gcc's per-target directory, which clang derives libstdc++'s header +// directory from, and a stand-in mingw-w64 tree whose libstdc++ has a std +// module with nothing in it. The fixture is a static library, so nothing is +// linked, and its C++ and C sources stop the compile when they see the +// prefix's include/ or miss mingw-w64's or libstdc++'s headers. // -// The std module such a build compiles is not the one releases before the -// fix compiled against the prefix itself, so it has to live in a directory -// of its own: the Flatpak's branches share one cache, and two releases -// taking turns rebuilding one PCM do it under the PCMs built against it. +// clang reaches such a prefix through a view of it (MakeMingwSysroot), which +// has to be made where symbolic links can be, and a compile error in one of +// its headers has to name the header where it is installed, not in the +// view. extern "C" int setenv(const char* name, const char* value, int overwrite); @@ -45,6 +45,26 @@ namespace { fs::create_directories(file.parent_path()); std::ofstream(file, std::ios::binary | std::ios::trunc) << text; } + + BuildResult BuildLibrary(const fs::path& project, std::string_view name, const fs::path& cFile) { + Configuration lib; + lib.path = project; + lib.name = std::string(name); + lib.outputName = std::string(name); + lib.target = "x86_64-w64-mingw32"; + // Spelled out: the default, native, names the host's CPU, which is + // no x86 CPU on an aarch64 host. + lib.march = "x86-64"; + lib.mtune = "x86-64"; + lib.type = ConfigurationType::LibraryStatic; + std::array ifaces = {}; + std::array impls = { fs::path(name) }; + lib.GetInterfacesAndImplementations(ifaces, impls); + if (!cFile.empty()) lib.cFiles = { cFile }; + std::unordered_map> depResults; + std::mutex depMutex; + return Build(lib, depResults, depMutex); + } } int main() { @@ -56,37 +76,71 @@ int main() { fs::copy(fs::current_path() / "tests" / "MingwSysroot" / "fixture", project, fs::copy_options::recursive); const fs::path prefix = scratch / "usr"; + Write(prefix / "include" / "features.h", "#define __GLIBC__ 2\n"); Write(prefix / "include" / "sys" / "single_threaded.h", "extern char __libc_single_threaded;\n"); fs::create_directories(prefix / "lib" / "gcc" / "x86_64-w64-mingw32" / "15.1.0"); const fs::path mingw = prefix / "x86_64-w64-mingw32"; Write(mingw / "include" / "_mingw.h", ""); + Write(mingw / "include" / "crafter_probe_error.h", "#error stopped in a mingw-w64 header\n"); Write(mingw / "include" / "c++" / "15.1.0" / "memory", ""); // Through std::format because the build's import scanner reads string // literals as code: one holding "module std;" made this file a unit of a // module std, and the test did not build. Write(mingw / "include" / "c++" / "15.1.0" / "bits" / "std.cc", std::format("export module {};\n", "std")); Write(mingw / "lib" / "libstdc++.modules.json", R"({"version": 1, "modules": [{"logical-name": "std", "source-path": "../include/c++/15.1.0/bits/std.cc", "is-std-library": true}]})"); + + { + // The view is made in the first place that can hold one. A cache on + // a filesystem that takes no symbolic links (vfat, exFAT, some CIFS + // and FUSE mounts) holds everything else crafter-build caches, so + // its failure must not be every mingw build's: this one can make + // nothing under mingw-sysroot, which is a file. + const fs::path blocked = scratch / "blocked"; + Write(blocked / "mingw-sysroot", ""); + const fs::path runtime = scratch / "runtime"; + const std::array parents = { blocked, runtime }; + fs::path view; + try { + view = MakeMingwSysroot(prefix, parents); + } catch (const std::exception& e) { + Check(false, std::format("a view is made in the second place when the first can hold none: {}", e.what())); + } + if (!view.empty()) { + Check(view.string().starts_with(runtime.string()), std::format("the view is in the place that could hold it: {}", view.string())); + Check(fs::equivalent(view / "x86_64-w64-mingw32", mingw, ec), "the view shows the prefix's mingw-w64 tree"); + Check(fs::equivalent(view / "lib" / "gcc", prefix / "lib" / "gcc", ec), "and gcc's directories"); + Check(!fs::exists(view / "include") && !fs::exists(view / "lib" / "libstdc++.modules.json"), "and nothing else of the prefix's include/ and lib/"); + fs::path again; + try { + again = MakeMingwSysroot(prefix, parents); + } catch (const std::exception& e) { + Check(false, std::format("a view already there is reused: {}", e.what())); + } + Check(again == view, "a view already there is reused"); + } + const std::array nowhere = { blocked }; + std::string error; + try { + MakeMingwSysroot(prefix, nowhere); + } catch (const std::exception& e) { + error = e.what(); + } + Check(error.contains(prefix.string()) && error.contains((blocked / "mingw-sysroot").string()), std::format("with no place for a view, the error names the prefix and why each place failed: {}", error)); + } + // Before anything resolves the toolchain, which reads both once. The - // std module cache goes to the scratch directory too. + // view and the std module cache go to the scratch directory too. setenv("CRAFTER_MINGW_DIR", prefix.c_str(), 1); setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1); - Configuration lib; - lib.path = project; - lib.name = "probe"; - lib.outputName = "probe"; - lib.target = "x86_64-w64-mingw32"; - lib.type = ConfigurationType::LibraryStatic; - std::array ifaces = {}; - std::array impls = { "probe" }; - lib.GetInterfacesAndImplementations(ifaces, impls); - lib.cFiles = { project / "cprobe" }; - std::unordered_map> depResults; - std::mutex depMutex; - const BuildResult built = Build(lib, depResults, depMutex); + const BuildResult built = BuildLibrary(project, "probe", project / "cprobe"); Check(built.result.empty(), std::format("a Windows build sees mingw-w64's and libstdc++'s headers and not the prefix's own: {}", built.result)); - const std::string unkeyed = std::format("{}-{}-{}", lib.target, lib.march, GetToolchain().Key()); - Check(StdPcmDir(lib).filename() != unkeyed, std::format("the std module of a mingw build with a prefix is not cached where older releases built it against the prefix itself: {}", StdPcmDir(lib).string())); + Check(MingwSysroot() != prefix && MingwSysroot().string().starts_with((scratch / "cache").string()), std::format("a prefix holding a Linux system reaches clang through a view in the cache: {}", MingwSysroot().string())); + + const BuildResult failed = BuildLibrary(project, "failing", {}); + const std::string header = (mingw / "include" / "crafter_probe_error.h").string(); + Check(failed.result.contains("stopped in a mingw-w64 header"), std::format("an error in a mingw-w64 header fails the build: {}", failed.result)); + Check(failed.result.contains(header) && !failed.result.contains(MingwSysroot().string()), std::format("and names the header in the prefix, not in the view: {}", failed.result)); fs::remove_all(scratch, ec); if (failures > 0) { diff --git a/tests/SysrootCompileFlags/main.cpp b/tests/SysrootCompileFlags/main.cpp index bfaf95b..6734089 100644 --- a/tests/SysrootCompileFlags/main.cpp +++ b/tests/SysrootCompileFlags/main.cpp @@ -19,6 +19,9 @@ using namespace Crafter; // sysroot's copy. It must not do so where there is no sysroot, for wasm (which // has its own sysroot handling) or for mingw (libstdc++, no libc++ at all). // The command is checked as text: no compiler runs. + +extern "C" int setenv(const char* name, const char* value, int overwrite); + namespace { std::int32_t Failures = 0; @@ -44,6 +47,15 @@ namespace { } int main() { + // A compile command names the std module's directory in the cache, and + // the mingw one can make the view of a system mingw prefix there + // (MingwSysroot), as with CRAFTER_MINGW_DIR=/usr or inside the Flatpak + // on a host with mingw-w64. Set before anything resolves the toolchain. + const fs::path scratch = fs::temp_directory_path() / "crafter-build-sysroot-compile-flags-cache"; + std::error_code ec; + fs::remove_all(scratch, ec); + setenv("XDG_CACHE_HOME", scratch.c_str(), 1); + const std::string sysroot = "/opt/crafter-build-test-sysroot"; const std::string forced = std::format("-nostdinc++ -isystem {}/usr/include/c++/v1", sysroot); @@ -57,6 +69,7 @@ int main() { Check(!CommandFor("wasm32-wasip1", sysroot).contains(forced), "wasm: left to its own sysroot handling"); Check(!CommandFor("x86_64-w64-mingw32", sysroot).contains(forced), "mingw: libstdc++ target, libc++ headers are not forced"); + fs::remove_all(scratch, ec); if (Failures > 0) { std::println(std::cerr, "{} assertions failed", Failures); return 1; diff --git a/tests/ToolchainResolve/main.cpp b/tests/ToolchainResolve/main.cpp index 1912418..d92a1c8 100644 --- a/tests/ToolchainResolve/main.cpp +++ b/tests/ToolchainResolve/main.cpp @@ -147,6 +147,31 @@ int main() { Check(tc.mingwPrefix == "/opt/mingw", "CRAFTER_MINGW_DIR names the mingw prefix"); in.env["CRAFTER_MINGW_DIR"] = "mingw"; Check(ResolveToolchain(in).mingwPrefix == fs::absolute("mingw"), "a relative CRAFTER_MINGW_DIR is resolved against the working directory once, when the toolchain is"); + // The prefix names directories, so one directory has one spelling: + // otherwise a trailing slash rebuilds every mingw build for nothing. + for (const char* spelling : {"/opt/mingw/", "/opt//mingw", "/opt/./mingw", "/opt/x/../mingw"}) { + in.env["CRAFTER_MINGW_DIR"] = spelling; + Check(ResolveToolchain(in).mingwPrefix == "/opt/mingw", std::format("{} is spelled /opt/mingw (got '{}')", spelling, ResolveToolchain(in).mingwPrefix.string())); + } + } + { + // A prefix that holds a Linux system's C library, as /usr does, is + // shown to clang through a view; one that holds mingw-w64 alone is + // the sysroot itself, Windows libraries in its include/ and lib/ + // included. + const fs::path staged = fs::temp_directory_path() / "crafter-build-toolchain-mingw-prefixes"; + fs::remove_all(staged); + Touch(staged / "system" / "include" / "features.h"); + Touch(staged / "system" / "x86_64-w64-mingw32" / "include" / "_mingw.h"); + Touch(staged / "dedicated" / "include" / "SDL2" / "SDL.h"); + Touch(staged / "dedicated" / "x86_64-w64-mingw32" / "include" / "_mingw.h"); + ToolchainInputs in; + in.env["CRAFTER_MINGW_DIR"] = (staged / "system").string(); + Check(ResolveToolchain(in).mingwPrefixIsSystem, "a prefix with a C library's features.h holds a Linux system"); + in.env["CRAFTER_MINGW_DIR"] = (staged / "dedicated").string(); + Check(!ResolveToolchain(in).mingwPrefixIsSystem, "a prefix of mingw-w64 and Windows libraries does not"); + Check(!ResolveToolchain({}).mingwPrefixIsSystem, "no prefix, nothing to show"); + fs::remove_all(staged); } { // Sandboxed, the host's /usr stands in for the missing mingw-w64, @@ -158,6 +183,7 @@ int main() { Check(ResolveToolchain(in).mingwPrefix.empty(), "a host without mingw-w64 gives no mingw prefix"); fs::create_directories(host / "usr" / "x86_64-w64-mingw32"); Check(ResolveToolchain(in).mingwPrefix == host / "usr", "a host with mingw-w64 gives its /usr as the mingw prefix"); + Check(ResolveToolchain(in).mingwPrefixIsSystem, "the host's /usr is the host's system, with or without its C library's headers"); in.env["CRAFTER_MINGW_DIR"] = "/opt/mingw"; Check(ResolveToolchain(in).mingwPrefix == "/opt/mingw", "CRAFTER_MINGW_DIR wins over the host's"); fs::remove_all(host); From 5d254956217fdd214a94035c51caa1e0447962a6 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:58:44 +0100 Subject: [PATCH 05/12] fix: key the std module's directory on everything it is compiled with StdPcmDir named the directory by target, march and toolchain, then grew a case per target for the rest: the wasm flags, and with the first fix for #43 the mingw prefix. Every other flag BuildStdPcm passes went unkeyed, so configurations that differed only there took turns rebuilding one std PCM in place, under the PCMs built against it: a project with -mtune=generic against project.cpp's -mtune=native on every run, or two sysroots for one target. And the mingw prefix was keyed only when there was one, so a build without a prefix landed in the directory an older release had filled with a CRAFTER_MINGW_DIR=/usr build, and relinked its objects. StdPcmFlags is now the one source of the flags BuildStdPcm compiles with, and StdPcmKey names the directory by target, march and toolchain followed by a hash of those flags and, for the mingw target, of the prefix behind its --sysroot. VariantId hashes StdPcmKey in place of the mingw prefix: units do not notice a std module that moved directories, so a build has to move with it. Every std PCM directory and build directory moves once, as after a compiler update; the old ones stay until `clean` and the cache is cleared by hand. Co-Authored-By: Claude Opus 5.5 --- implementations/Crafter.Build-Clang.cpp | 73 +++++++++++++--------- implementations/Crafter.Build-Platform.cpp | 22 +------ interfaces/Crafter.Build-Clang.cppm | 44 ++++++++----- tests/VariantId/main.cpp | 40 ++++++++++++ 4 files changed, 114 insertions(+), 65 deletions(-) diff --git a/implementations/Crafter.Build-Clang.cpp b/implementations/Crafter.Build-Clang.cpp index 138ffd7..a302ca9 100644 --- a/implementations/Crafter.Build-Clang.cpp +++ b/implementations/Crafter.Build-Clang.cpp @@ -247,36 +247,51 @@ void Configuration::GetInterfacesAndImplementations(std::span interfac } } +std::string Crafter::StdPcmFlags(const Configuration& config) { + if (config.target == "x86_64-w64-mingw32") { + // -femulated-tls keeps PCM TLS access matching libstdc++'s emutls + // definitions; a mismatch surfaces as undefined std::__once_callable. + return std::format(" --target={} -march={} -mtune={} -femulated-tls -O3 -std=c++26 -Wno-reserved-identifier -Wno-reserved-module-identifier", config.target, config.march, config.mtune); + } + const bool isWasm = config.target.starts_with("wasm32"); + const std::string sysrootFlag = config.sysroot.empty() ? std::string() : std::format(" --sysroot={}", config.sysroot); + // wasm32 rejects -march, and the std module has to be compiled the way + // the units importing it are, a variant pass's flags (e.g. + // -mrelaxed-simd) included, or the BMI is not feature-compatible. + std::string archFlags = isWasm ? WasmCompileFlags(config) : std::format(" -march={} -mtune={}", config.march, config.mtune); + // non-wasm cross builds: the driver ranks the host toolchain's own + // libc++ headers (/../include/c++/v1) above --sysroot, so the + // PCM would mix host libc++ with the foreign libc. Force the sysroot's + // copy. + if (!isWasm && !config.sysroot.empty()) { + archFlags += std::format(" -nostdinc++ -isystem {}/usr/include/c++/v1", config.sysroot); + } + return std::format(" --target={} -std=c++26{}{} -O3 -Wno-reserved-identifier -Wno-reserved-module-identifier", config.target, sysrootFlag, archFlags); +} + +std::string Crafter::StdPcmKey(const Configuration& config, const Toolchain& toolchain) { + // A PCM is only valid for the clang that wrote it and the flags it was + // written with, so a compiler bump starts a cold directory instead of + // handing the new clang a PCM it will crash on, and so does any flag. + // With only some of them in the name, configurations that differed in + // the rest took turns rebuilding one std PCM in place: -mtune=generic + // against project.cpp's -mtune=native on every run, two sysroots, or two + // releases sharing a cache, as the Flatpak's branches do, each time under + // the PCMs built against the previous one, which clang crashes on rather + // than rejecting. + std::string compiled = StdPcmFlags(config); + // The base command's part: the toolchain's key covers the compiler and + // libc++, and for the mingw target what clang finds mingw-w64 through is + // the prefix, given whole or through a view (MingwSysroot), named here by + // the prefix rather than by the view's own path. + if (config.target == "x86_64-w64-mingw32" && !toolchain.mingwPrefix.empty()) { + compiled += std::format("|mingw:{}|{}", toolchain.mingwPrefix.string(), toolchain.mingwPrefixIsSystem ? "view" : "whole"); + } + return std::format("{}-{}-{}-{:08x}", config.target, config.march, toolchain.Key(), static_cast(std::hash{}(compiled))); +} + fs::path Crafter::StdPcmDir(const Configuration& config) { - // The std PCM is cached per target+march+toolchain: a PCM is only valid - // for the exact clang that wrote it, so a compiler bump starts a cold - // directory instead of handing the new clang a PCM it will crash on. A - // wasm variant pass compiles it with extra codegen flags (e.g. - // -mrelaxed-simd) — its BMI must not be shared with the baseline's, or - // the consuming TUs see a target-feature mismatch. Suffix the cache dir - // with the variant flags when present. - std::string stdPcmKey = std::format("{}-{}-{}", config.target, config.march, GetToolchain().Key()); - // The flags every wasm unit gets are compiled into the std PCM as well, - // and the importing units have to match them. Two releases that share a - // cache and a compiler, as the Flatpak's branches do, would otherwise - // take turns rebuilding this one std PCM, each time underneath the PCMs - // built against the previous one, and clang crashes on those rather than - // rejecting them. - if (config.target.starts_with("wasm32")) { - stdPcmKey += std::format("-{:08x}", static_cast(std::hash{}(WasmBaselineFlags))); - } - // The same goes for the mingw-w64 prefix the mingw std module is - // compiled against: releases up to the fix for issue #43 gave clang the - // prefix itself as the sysroot and the ones after give it a view of it, - // and two prefixes are two sets of headers. - if (config.target == "x86_64-w64-mingw32" && !GetToolchain().mingwPrefix.empty()) { - stdPcmKey += std::format("-{:08x}", static_cast(std::hash{}(GetToolchain().mingwPrefix.string()))); - } - for (const std::string& f : config.wasmVariantFlags) { - stdPcmKey += "+"; - for (char c : f) stdPcmKey += (c == '/' || c == '\\') ? '_' : c; - } - return GetCacheDir()/stdPcmKey; + return GetCacheDir() / StdPcmKey(config, GetToolchain()); } std::string Crafter::WasmCompileFlags(const Configuration& config) { diff --git a/implementations/Crafter.Build-Platform.cpp b/implementations/Crafter.Build-Platform.cpp index 2c33eca..5d0c5f4 100644 --- a/implementations/Crafter.Build-Platform.cpp +++ b/implementations/Crafter.Build-Platform.cpp @@ -790,12 +790,10 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) { const fs::path stdCc = (manifestPath.parent_path() / *source).lexically_normal(); if (!fs::exists(stdCc)) return std::format("{}, which {} names as the std module, is missing", stdCc.string(), manifestPath.string()); - // -femulated-tls keeps PCM TLS access matching libstdc++'s emutls - // definitions; mismatch surfaces as undefined std::__once_callable. // The std.cc → std.cppm copy uses a plain filename: held under the // cache lock, no other builder writes this path concurrently. fs::path stdCppm = stdPcm.parent_path() / "std.cppm"; - std::string command = std::format("{} --target={} -march={} -mtune={} -femulated-tls -O3 -std=c++26 -Wno-reserved-identifier -Wno-reserved-module-identifier --precompile {} -o {}", GetBaseCommand(config), config.target, config.march, config.mtune, stdCppm.string(), stdPcm.string()); + std::string command = std::format("{}{} --precompile {} -o {}", GetBaseCommand(config), StdPcmFlags(config), stdCppm.string(), stdPcm.string()); std::string stamp = HashStamp(command + '\0' + FileBytes(stdCc)); if (fs::exists(stdPcm) && ReadStamp(StampPath(stdPcm)) == stamp) return ""; std::error_code copyEc; @@ -825,27 +823,11 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) { if (!source.error.empty()) return source.error; stdCppm = source.stdCppm.string(); } - std::string sysrootFlag = config.sysroot.empty() - ? std::string() - : std::format(" --sysroot={}", config.sysroot); - // wasm32 rejects -march, and the std module has to be compiled the way - // the units importing it are, a variant pass's flags (e.g. - // -mrelaxed-simd) included, or the BMI is not feature-compatible. - std::string archFlags = isWasm - ? WasmCompileFlags(config) - : std::format(" -march={} -mtune={}", config.march, config.mtune); - // non-wasm cross builds: the driver ranks the host toolchain's own - // libc++ headers (/../include/c++/v1) above --sysroot, so - // the PCM would mix host libc++ with the foreign libc — force the - // sysroot's copy. - if (!isWasm && !config.sysroot.empty()) { - archFlags += std::format(" -nostdinc++ -isystem {}/usr/include/c++/v1", config.sysroot); - } // Keyed on the command and the module source's bytes, never on // mtimes: an installed std.cppm can be dated at the epoch (OSTree), // and a libc++ update under the same clang changes the bytes but // not the path. - std::string command = std::format("{} --target={} -std=c++26{}{} -O3 -Wno-reserved-identifier -Wno-reserved-module-identifier --precompile {} -o {}", GetBaseCommand(config), config.target, sysrootFlag, archFlags, stdCppm, stdPcm.string()); + std::string command = std::format("{}{} --precompile {} -o {}", GetBaseCommand(config), StdPcmFlags(config), stdCppm, stdPcm.string()); std::string stamp = HashStamp(command + '\0' + FileBytes(stdCppm)); if (fs::exists(stdPcm) && ReadStamp(StampPath(stdPcm)) == stamp) return ""; std::string result = RunBuildCommand(command); diff --git a/interfaces/Crafter.Build-Clang.cppm b/interfaces/Crafter.Build-Clang.cppm index 89583cc..50903d6 100644 --- a/interfaces/Crafter.Build-Clang.cppm +++ b/interfaces/Crafter.Build-Clang.cppm @@ -392,6 +392,15 @@ export namespace Crafter { }; struct Configuration; + // The name StdPcmDir gives the directory of this configuration's std + // module under `toolchain`: target, march and the toolchain's key, which + // say what it is, then a hash of everything else it is compiled with, + // each flag BuildStdPcm passes (StdPcmFlags) and for the mingw target the + // prefix behind its --sysroot. Two configurations that would compile the + // std module differently never share a directory, so neither rebuilds it + // under the PCMs the other compiled against it. Declared ahead of + // Configuration, whose VariantId keys on it. + std::string StdPcmKey(const Configuration& config, const Toolchain& toolchain); // What a Script's callback is handed when `crafter-build run ` // starts it. @@ -580,7 +589,8 @@ export namespace Crafter { void AddAstLintRule(std::string name, std::function check); // Suffix that uniquely identifies this Configuration's compile state. // target+march+mtune are spelled out for readability; the rest - // (type, debug, sysroot, defines, compileFlags, the toolchain) + // (type, debug, sysroot, defines, compileFlags, the std module, the + // toolchain) // collapse into a short hash so two Configurations sharing a path but // with different compile state can't clobber each other's outputs. std::string VariantId() const { return VariantId(GetToolchain()); } @@ -628,17 +638,15 @@ export namespace Crafter { compileKey += "|B:"; compileKey += WasmBaselineFlags; } - // The mingw-w64 prefix decides what the mingw target compiles - // against, and until issue #43 clang got it as the sysroot - // itself, which put the host's own headers on the include path. - // Unkeyed, the release that fixed it kept the objects compiled - // against glibc's headers: the build after the update only - // relinked them, and failed to link as before. No prefix keeps - // the id as it was, since clang's own discovery has not changed. - if (target == "x86_64-w64-mingw32" && !toolchain.mingwPrefix.empty()) { - compileKey += "|M:"; - compileKey += toolchain.mingwPrefix.string(); - } + // The std module the units import. Its PCMs and objects record + // it by path, and units do not notice it moving: when the mingw + // prefix stopped being clang's sysroot as is (issue #43), the + // std module was rebuilt and the objects compiled against the + // host's glibc headers were only relinked, failing as before. A + // std module that moves directories, because anything it is + // compiled with changed, moves the build with it. + compileKey += "|S:"; + compileKey += StdPcmKey(*this, toolchain); // Sorted by ApplyStandardArgs, so flag order on the command line // doesn't split the cache. for (const std::string& a : projectArgs) { @@ -673,10 +681,9 @@ export namespace Crafter { std::vector requires_; }; - // Directory holding the std module PCM for this Configuration. Keyed on - // target+march because a BMI is only loadable by a TU with matching target - // features, and suffixed with the wasm flags (WasmBaselineFlags, then the - // variant's) for the same reason. + // Directory holding the std module PCM for this Configuration, + // GetCacheDir() / StdPcmKey(config, GetToolchain()): a BMI is only + // loadable by a unit compiled the way it was. fs::path StdPcmDir(const Configuration& config); // Where Build leaves this Configuration's linked artifact: the executable @@ -805,4 +812,9 @@ namespace Crafter { // compile command and of the std module's, which have to agree flag for // flag. std::string WasmCompileFlags(const Configuration& config); + // The flags BuildStdPcm compiles the std module with, after the base + // command and before the source, each with a leading space. StdPcmKey + // hashes the same string, so the command and the directory it writes to + // cannot disagree about what decides the BMI. + std::string StdPcmFlags(const Configuration& config); } diff --git a/tests/VariantId/main.cpp b/tests/VariantId/main.cpp index e658b3e..0b4472d 100644 --- a/tests/VariantId/main.cpp +++ b/tests/VariantId/main.cpp @@ -178,6 +178,46 @@ int main() { Check(mingw.VariantId(hostUsr) != mingw.VariantId(optMingw), "two mingw prefixes are two variants"); Configuration host = MakeBase(); Check(host.VariantId(discovered) == host.VariantId(hostUsr), "the mingw prefix leaves other targets' variants alone"); + + // The std module's directory is keyed the same way, and so is a + // prefix given whole against one reached through a view. + Check(StdPcmKey(mingw, discovered) != StdPcmKey(mingw, hostUsr), "a mingw prefix moves the mingw std module"); + Check(StdPcmKey(mingw, hostUsr) != StdPcmKey(mingw, optMingw), "two mingw prefixes are two std modules"); + Toolchain hostUsrWhole = hostUsr; + hostUsrWhole.mingwPrefixIsSystem = !hostUsr.mingwPrefixIsSystem; + Check(StdPcmKey(mingw, hostUsr) != StdPcmKey(mingw, hostUsrWhole), "a prefix given whole and one shown through a view are two std modules"); + Check(mingw.VariantId(hostUsr) != mingw.VariantId(hostUsrWhole), "and two variants, through the std module"); + Check(StdPcmKey(host, discovered) == StdPcmKey(host, hostUsr), "the mingw prefix leaves other targets' std modules alone"); + } + { + // Everything the std module is compiled with keys its directory, not + // only target and march. -mtune was left out, so a project built + // with -mtune=generic shared one std PCM with project.cpp's + // -mtune=native, and the two rebuilt it in turn on every run, under + // every module built against it; two sysroots did the same. + Toolchain tc; + tc.clangxx = "/app/bin/clang++"; + tc.clangVersion = "clang version 23.1.2"; + Configuration native = MakeBase(); + Configuration generic = MakeBase(); + generic.mtune = "generic"; + Check(StdPcmKey(native, tc) != StdPcmKey(generic, tc), "-mtune keys the std module"); + Configuration sysrootA = MakeBase(); + sysrootA.sysroot = "/opt/sysroot-a"; + Configuration sysrootB = MakeBase(); + sysrootB.sysroot = "/opt/sysroot-b"; + Check(StdPcmKey(sysrootA, tc) != StdPcmKey(sysrootB, tc) && StdPcmKey(sysrootA, tc) != StdPcmKey(native, tc), "the sysroot keys the std module"); + Configuration wasm = MakeBase(); + wasm.target = "wasm32-wasip1"; + wasm.sysroot = "/opt/wasi-sysroot"; + Configuration relaxed = MakeBase(); + relaxed.target = wasm.target; + relaxed.sysroot = wasm.sysroot; + relaxed.wasmVariantFlags = {"-mrelaxed-simd"}; + Check(StdPcmKey(wasm, tc) != StdPcmKey(relaxed, tc), "a wasm variant pass's flags key the std module"); + Check(StdPcmKey(native, tc) == StdPcmKey(MakeBase(), tc), "one configuration, one std module"); + Check(StdPcmKey(native, tc).starts_with(std::format("{}-{}-{}-", native.target, native.march, tc.Key())), std::format("the directory still says target, march and toolchain (got '{}')", StdPcmKey(native, tc))); + Check(native.VariantId(tc) != generic.VariantId(tc) && sysrootA.VariantId(tc) != sysrootB.VariantId(tc), "a build moves with its std module"); } if (Failures > 0) { From 27697d06aa68ac9322f8d85d5cc305825bf99a09 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:59:05 +0100 Subject: [PATCH 06/12] fix: recompile the units whose std module was rebuilt in place A .cpp unit's depfile listed its source and headers, never the BMIs it imports: clang's driver adds -module-file-deps to --precompile only. A std module rebuilt in its directory, which happens whenever its source changes under one compiler and one set of flags (a libc++ update, or mingw-w64's under the same prefix), left every object that imports it as it was, and the next build only relinked them. Implementation units are now compiled with -fmodule-file-deps, so the std PCM and the project's own PCMs are prerequisites of their objects like headers. Depfiles written before this lack the edge, so a std module that moves directories still needs VariantId to move with it (StdPcmKey). Co-Authored-By: Claude Opus 5.5 --- implementations/Crafter.Build-Implementation.cpp | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/implementations/Crafter.Build-Implementation.cpp b/implementations/Crafter.Build-Implementation.cpp index ca39bac..6f0b355 100644 --- a/implementations/Crafter.Build-Implementation.cpp +++ b/implementations/Crafter.Build-Implementation.cpp @@ -51,8 +51,13 @@ namespace Crafter { return; } - // -MD leaves _impl.o.d beside the object for Check to read. - std::string result = RunBuildCommand(std::format("{0} {1}.cpp -c -MD -MF {2}_impl.o.d -o {2}_impl.o", clang, path.string(), (buildDir/path.filename()).string())); + // -MD leaves _impl.o.d beside the object for Check to read, and + // -fmodule-file-deps puts the BMIs the unit imported in it with the + // headers. The driver adds that for --precompile only, so a std + // module rebuilt in place, for a mingw-w64 or libc++ update under + // the same compiler, left the objects importing it as they were, and + // the next build only relinked them. + std::string result = RunBuildCommand(std::format("{0} {1}.cpp -c -MD -MF {2}_impl.o.d -fmodule-file-deps -o {2}_impl.o", clang, path.string(), (buildDir/path.filename()).string())); bool expected = false; if(!result.empty() && buildCancelled.compare_exchange_strong(expected, true)) { From 8e759a8410f4178290e25007f36bc145e827b143 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:59:06 +0100 Subject: [PATCH 07/12] test: a std module rebuilt in place recompiles the units importing it StdPcmRebuild builds a project whose main.cpp imports std, then dates the source before the object and the object before std.pcm, as a std module rebuilt after the object leaves them, so that nothing else makes the object stale. The next build has to compile main.cpp again, and the idle build before it must compile nothing. Without -fmodule-file-deps the object was only relinked. The test redirects the cache, since redating a std PCM the real cache shares would rebuild its other users. Co-Authored-By: Claude Opus 5.5 --- project.cpp | 1 + tests/StdPcmRebuild/fixture/main.cpp | 8 +++ tests/StdPcmRebuild/main.cpp | 85 ++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+) create mode 100644 tests/StdPcmRebuild/fixture/main.cpp create mode 100644 tests/StdPcmRebuild/main.cpp diff --git a/project.cpp b/project.cpp index adedaef..3c0915c 100644 --- a/project.cpp +++ b/project.cpp @@ -247,6 +247,7 @@ extern "C" Configuration CrafterBuildProject(std::span a cfg.AddTest("IncrementalInterfaceChange").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("TransitiveInterfaceChange").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("IncrementalHeaderChange").Dependencies({ CrafterBuildLib.get() }); + cfg.AddTest("StdPcmRebuild").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("CompilerWarning").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("CleanProject").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ShaderCompile").Dependencies({ CrafterBuildLib.get() }); diff --git a/tests/StdPcmRebuild/fixture/main.cpp b/tests/StdPcmRebuild/fixture/main.cpp new file mode 100644 index 0000000..45992f4 --- /dev/null +++ b/tests/StdPcmRebuild/fixture/main.cpp @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +import std; + +int main() { + std::println("rebuilt"); +} diff --git a/tests/StdPcmRebuild/main.cpp b/tests/StdPcmRebuild/main.cpp new file mode 100644 index 0000000..291bee6 --- /dev/null +++ b/tests/StdPcmRebuild/main.cpp @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +import std; +import Crafter.Build; +namespace fs = std::filesystem; +using namespace Crafter; + +// A std module rebuilt in place has to recompile the units that import it. +// The std PCM is rebuilt in its directory whenever its source changes under +// the same compiler and flags: a libc++ update, or mingw-w64's under the +// same prefix. A .cpp unit's depfile listed only its source and headers, +// because the driver records imported BMIs for --precompile alone, so such +// a unit kept its object and the next build only relinked it. +// +// The project is one main.cpp that imports std. Its object is dated before +// std.pcm and after its source, as a std module rebuilt after the object +// leaves them, and the next build has to compile it again. XDG_CACHE_HOME keeps the std PCM out of the real +// cache, whose other users the change of date would rebuild. + +extern "C" int setenv(const char* name, const char* value, int overwrite); + +namespace { + std::int32_t failures = 0; + + void Check(bool cond, std::string_view msg) { + if (!cond) { + std::println(std::cerr, "FAIL: {}", msg); + ++failures; + } + } + + BuildResult BuildOnce(Configuration& cfg) { + std::unordered_map> depResults; + std::mutex depMutex; + return Build(cfg, depResults, depMutex); + } +} + +int main() { + const fs::path scratch = fs::temp_directory_path() / "crafter-build-std-pcm-rebuild"; + std::error_code ec; + fs::remove_all(scratch, ec); + fs::create_directories(scratch); + const fs::path project = scratch / "project"; + fs::copy(fs::current_path() / "tests" / "StdPcmRebuild" / "fixture", project, fs::copy_options::recursive); + setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1); + + Configuration cfg; + cfg.path = project; + cfg.name = "rebuilt"; + cfg.outputName = "rebuilt"; + cfg.type = ConfigurationType::Executable; + std::array ifaces = {}; + std::array impls = { "main" }; + cfg.GetInterfacesAndImplementations(ifaces, impls); + + BuildResult built = BuildOnce(cfg); + Check(built.result.empty(), std::format("the project builds: {}", built.result)); + const fs::path object = cfg.BuildDir() / "main_impl.o"; + const fs::path stdPcm = StdPcmDir(cfg) / "std.pcm"; + Check(fs::exists(object) && fs::exists(stdPcm), std::format("{} and {} exist", object.string(), stdPcm.string())); + if (!fs::exists(object) || !fs::exists(stdPcm)) return 1; + + const fs::file_time_type first = fs::last_write_time(object); + built = BuildOnce(cfg); + Check(built.result.empty(), std::format("an idle build succeeds: {}", built.result)); + Check(fs::last_write_time(object) == first, "and compiles nothing again"); + + // The source before the object and the object before std.pcm, so that + // nothing but the std module makes the object stale. + const fs::file_time_type older = fs::last_write_time(stdPcm) - std::chrono::seconds(10); + fs::last_write_time(project / "main.cpp", older - std::chrono::seconds(10)); + fs::last_write_time(object, older); + built = BuildOnce(cfg); + Check(built.result.empty(), std::format("the build after the std module succeeds: {}", built.result)); + Check(fs::last_write_time(object) > older, "a unit importing a std module newer than its object is compiled again"); + + fs::remove_all(scratch, ec); + if (failures > 0) { + std::println(std::cerr, "{} assertions failed", failures); + return 1; + } + return 0; +} From 56b44f616aaf96b384ca08419c36b87dd1ba3a23 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:59:22 +0100 Subject: [PATCH 08/12] fix(external): start a CMake build over when its configure inputs change ConfigureCMake records the flags it injects (compiler, libc++, and for the mingw target CMAKE_SYSROOT) and reconfigures when they differ, but it did so in place. CMake then keeps what the old flags compiled: the Makefiles generator writes --sysroot into build.make, which make does not compare, so nothing was recompiled, and the check results the old configure cached stay too. A mingw external built by a release that gave clang the host's /usr as the sysroot kept its objects reading __libc_single_threaded after the fix for #43, and only deleting /external/-/build by hand got rid of them. A build directory with a CMakeCache.txt whose recorded inputs differ is now removed before the configure. It keeps its name, since projects point include directories at headers generated in it. The two crafter-builds the Flatpak's branches are, sharing one cache, rebuild such an external from nothing each time they take turns, where they used to reuse each other's objects. Co-Authored-By: Claude Opus 5.5 --- implementations/Crafter.Build-External.cpp | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/implementations/Crafter.Build-External.cpp b/implementations/Crafter.Build-External.cpp index 4a9fbe2..d209a00 100644 --- a/implementations/Crafter.Build-External.cpp +++ b/implementations/Crafter.Build-External.cpp @@ -179,6 +179,19 @@ std::string ConfigureCMake(const fs::path& cloneDir, const fs::path& cmakeBuildD if (!needsConfigure) return ""; + // A build directory configured with other inputs starts over instead of + // being reconfigured. CMake reconfigures in place without recompiling + // what an old compiler or --sysroot compiled (the Makefiles generator + // writes --sysroot into build.make, which make does not compare) and + // keeps the check results the old configure cached. A mingw external + // built against the host's /usr as the sysroot (issue #43) kept its + // objects reading __libc_single_threaded after the fix that way. The + // directory keeps its name, which projects point generated headers at. + if (fs::exists(cmakeBuildDir / "CMakeCache.txt")) { + std::error_code ec; + fs::remove_all(cmakeBuildDir, ec); + if (ec) return std::format("cannot clear {} for a fresh configure: {}", cmakeBuildDir.string(), ec.message()); + } if (!fs::exists(cmakeBuildDir)) fs::create_directories(cmakeBuildDir); std::string cmd = std::format("cmake -S {} -B {}{}", ShellQuote(fs::absolute(cloneDir).string()), ShellQuote(fs::absolute(cmakeBuildDir).string()), injected); From 0e69e71b11814a2ed4bc9332e021d055c128a149 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:59:23 +0100 Subject: [PATCH 09/12] test: a CMake external configured with other inputs is built from nothing ExternalReconfigure builds a one-file C library from a local repository as a CMake external, then plants a file in its build directory to stand for what an old configure left. Built again with the same inputs, the directory is kept. With its recorded inputs rewritten the way another crafter-build's configure would have left them, the next build has to start from an empty directory and build the library again. Before the fix the planted file survived the reconfigure. Co-Authored-By: Claude Opus 5.5 --- project.cpp | 1 + tests/ExternalReconfigure/main.cpp | 97 ++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+) create mode 100644 tests/ExternalReconfigure/main.cpp diff --git a/project.cpp b/project.cpp index 3c0915c..38db3a4 100644 --- a/project.cpp +++ b/project.cpp @@ -258,6 +258,7 @@ extern "C" Configuration CrafterBuildProject(std::span a cfg.AddTest("GitSubmodules").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("LinkFlagsChange").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ExternalArtifactTime").Dependencies({ CrafterBuildLib.get() }); + cfg.AddTest("ExternalReconfigure").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("ConcurrentExternalFetch").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("SharedExternalFetch").Dependencies({ CrafterBuildLib.get() }); cfg.AddTest("StandardArgs").Dependencies({ CrafterBuildLib.get() }); diff --git a/tests/ExternalReconfigure/main.cpp b/tests/ExternalReconfigure/main.cpp new file mode 100644 index 0000000..57d50ba --- /dev/null +++ b/tests/ExternalReconfigure/main.cpp @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: LGPL-3.0-only +// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts® + +import std; +import Crafter.Build; +namespace fs = std::filesystem; +using namespace Crafter; + +// A CMake dependency configured with other inputs than this build's (another +// compiler or --sysroot, the flags crafter-build injects, which it records in +// /.crafter-options) has to be built again from nothing. It was +// reconfigured in place, and CMake does not recompile what a changed +// --sysroot compiled (the Makefiles generator writes it into build.make, +// which make does not compare) and keeps the check results of the old +// configure. A mingw dependency built by a release that gave clang the +// host's /usr as the sysroot (issue #43) kept its objects reading +// __libc_single_threaded after the update. +// +// The dependency is a local repository with a one-file C library. A file +// planted in its build directory stands for what the old configure left, +// and the recorded inputs are rewritten the way another crafter-build would +// have left them. Everything is local; XDG_CACHE_HOME keeps the clone out of +// the real cache. + +extern "C" int setenv(const char* name, const char* value, int overwrite); + +namespace { + std::int32_t failures = 0; + + void Check(bool cond, std::string_view msg) { + if (!cond) { + std::println(std::cerr, "FAIL: {}", msg); + ++failures; + } + } + + void Git(const fs::path& dir, std::string_view args) { + const std::string command = std::format("git -C {} {}", ShellQuote(dir.string()), args); + auto result = RunCommandWithTimeout(command, std::chrono::seconds(60)); + if (result.exitCode != 0) throw std::runtime_error(std::format("{} failed: {}", command, result.output)); + } + + void Write(const fs::path& file, std::string_view text) { + std::ofstream(file, std::ios::binary | std::ios::trunc) << text; + } +} + +int main() { + const fs::path scratch = fs::temp_directory_path() / "crafter-build-external-reconfigure"; + fs::remove_all(scratch); + fs::create_directories(scratch); + setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1); + setenv("GIT_CONFIG_NOSYSTEM", "1", 1); + setenv("GIT_CONFIG_GLOBAL", "/dev/null", 1); + for (const char* name : { "GIT_AUTHOR_NAME", "GIT_COMMITTER_NAME" }) setenv(name, "crafter-build test", 1); + for (const char* name : { "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_EMAIL" }) setenv(name, "test@example.invalid", 1); + + const fs::path repo = scratch / "reconf"; + fs::create_directories(repo); + Write(repo / "lib.c", "int Reconfigured(void) { return 1; }\n"); + Write(repo / "CMakeLists.txt", + "cmake_minimum_required(VERSION 3.16)\n" + "project(reconf C)\n" + "add_library(reconf STATIC lib.c)\n"); + Git(repo, "init -q -b main"); + Git(repo, "add ."); + Git(repo, "commit -q -m reconf"); + + ExternalDependency dep; + dep.name = "reconf"; + dep.source.url = std::format("file://{}", repo.string()); + dep.builder = ExternalBuilder::CMake; + dep.libs = { "reconf" }; + std::string error = BuildExternal(dep, HostTarget()); + Check(error.empty(), std::format("the dependency builds: {}", error)); + + const fs::path build = ExternalBuildDir(dep, HostTarget()); + const fs::path leftover = build / "left-by-the-old-configure"; + Write(leftover, ""); + error = BuildExternal(dep, HostTarget()); + Check(error.empty(), std::format("and builds again: {}", error)); + Check(fs::exists(leftover), "with the same inputs its build directory is kept"); + + Write(build / ".crafter-options", " -DCMAKE_SYSROOT=/usr\n"); + error = BuildExternal(dep, HostTarget()); + Check(error.empty(), std::format("configured with other inputs, it builds: {}", error)); + Check(!fs::exists(leftover), "from an empty build directory"); + Check(fs::exists(build / "libreconf.a"), "and the library is built again"); + + std::error_code ec; + fs::remove_all(scratch, ec); + if (failures > 0) { + std::println(std::cerr, "{} assertions failed", failures); + return 1; + } + return 0; +} From 40dcf2271d41885f538d974b1c2879bd8fc5728b Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:59:37 +0100 Subject: [PATCH 10/12] fix(lint): fail the files whose configuration has no compile command When GetCompileCommand threw or the std module failed to build, the lint driver cleared the configuration's command, and an empty command means "not a translation unit" (project.cpp, a header): every rule that needs an AST skipped the files quietly and the run passed them as clean. MingwSysroot made that reachable for every mingw configuration whose view cannot be made. The reason is now kept per configuration and reported as an AST failure for each of its files, which fails the run and prints it. Co-Authored-By: Claude Opus 5.5 --- implementations/Crafter.Build-Lint.cpp | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/implementations/Crafter.Build-Lint.cpp b/implementations/Crafter.Build-Lint.cpp index 357fb33..39e4be0 100644 --- a/implementations/Crafter.Build-Lint.cpp +++ b/implementations/Crafter.Build-Lint.cpp @@ -1149,6 +1149,10 @@ format` applies it to disk, and `crafter-build lint` reports where it would. // command per configuration — GetCompileCommand walks the dependency tree. const bool anyRuleNeedsAst = std::ranges::any_of(rules, [](const LintRule* r) { return r->needsAst; }); std::unordered_map commands; + // Why a configuration of the build graph has no command: its files + // cannot be parsed, which fails the run. Cleared to nothing, the command + // read as "not a translation unit" and every semantic rule passed them. + std::unordered_map commandErrors; if (anyRuleNeedsAst && !opts.noAst) { for (Configuration* c : localConfigs) { auto ensure = [&](const Configuration& cfg) { @@ -1166,10 +1170,14 @@ format` applies it to disk, and `crafter-build lint` reports where it would. Progress::Task task(std::format("Building std PCM ({}-{})", cfg.target, cfg.march)); fs::create_directories(assembled.stdPcmDir); std::string error = BuildStdPcm(cfg, assembled.stdPcmDir/"std.pcm"); - if (!error.empty()) command.clear(); + if (!error.empty()) { + command.clear(); + commandErrors.emplace(&cfg, std::format("building the std module for {}: {}", cfg.target, error)); + } } - } catch (const std::exception&) { - command.clear(); // surfaced per file as an AST reason + } catch (const std::exception& e) { + command.clear(); + commandErrors.emplace(&cfg, std::format("assembling the compile command for {}: {}", cfg.target, e.what())); } commands.emplace(&cfg, std::move(command)); }; @@ -1200,6 +1208,11 @@ format` applies it to disk, and `crafter-build lint` reports where it would. // decides what is safe to touch. Skip it and make the run fail. if (rule->needsAst) { if (opts.noAst) continue; + if (auto failed = commandErrors.find(owner); failed != commandErrors.end()) { + astFailures.emplace(file, failed->second); + skippedAstRules.insert(rule->name); + continue; + } // No compile command means this file is not a translation unit // of the build graph — project.cpp, which LoadProject compiles // with its own flags, or a header. A semantic rule does not From 55ad128b79495f0bc4de8a4d51c3f532130e6f58 Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:59:38 +0100 Subject: [PATCH 11/12] test: lint fails when a configuration's std module cannot be built A host configuration pointed at a sysroot without libc++ cannot get its std module built, so its files cannot be parsed. With an AST rule registered, the run has to skip the rule and fail instead of reporting the files clean, as it did before the fix. The cache is redirected around this run only: the std module's directory is made before its build fails. Co-Authored-By: Claude Opus 5.5 --- tests/Lint/main.cpp | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tests/Lint/main.cpp b/tests/Lint/main.cpp index 69bd436..4e07d94 100644 --- a/tests/Lint/main.cpp +++ b/tests/Lint/main.cpp @@ -6,6 +6,9 @@ import Crafter.Build; namespace fs = std::filesystem; using namespace Crafter; +extern "C" int setenv(const char* name, const char* value, int overwrite); +extern "C" int unsetenv(const char* name); + namespace { std::int32_t Failures = 0; @@ -705,6 +708,31 @@ int main() { // two assertions above cover it. } + // A configuration whose std module cannot be built is in the build graph + // all the same: its files cannot be parsed, which has to fail the run. + // Its command was cleared, which read as "not a translation unit", and + // every semantic rule passed the files. A sysroot without libc++ is one + // way there. The std module's directory is made before its build fails, + // so the cache is redirected around this run. + { + Scratch s("ast-no-std-module"); + s.Write("f", "int Value = 1;\n"); + Configuration cfg = s.Config({"f"}); + cfg.sysroot = (s.dir / "empty-sysroot").string(); + fs::create_directories(cfg.sysroot); + bool ran = false; + cfg.AddAstLintRule("needs-ast", [&ran](LintContext&) { ran = true; }); + const char* realCache = std::getenv("XDG_CACHE_HOME"); + const std::string saved = realCache ? realCache : ""; + setenv("XDG_CACHE_HOME", (s.dir / "cache").c_str(), 1); + LintSummary summary = RunLint(cfg, Mode(LintMode::Report)); + if (saved.empty()) unsetenv("XDG_CACHE_HOME"); + else setenv("XDG_CACHE_HOME", saved.c_str(), 1); + Check(!ran, "ast: rule is skipped when the std module cannot be built"); + Check(summary.errors > 0, "ast: a std module that cannot be built fails the run"); + Check(!summary.Clean(), "ast: a std module that cannot be built is not Clean"); + } + // --no-ast skips those rules deliberately and exits normally. { Scratch s("ast-optout"); From c56d98abba784702ec03e799be35cfc4f2191c2e Mon Sep 17 00:00:00 2001 From: Jorijn van der Graaf Date: Sat, 31 Oct 2026 22:59:47 +0100 Subject: [PATCH 12/12] docs: which mingw prefixes are viewed, and what keys the std module The README said every prefix reached clang as a view of its x86_64-w64-mingw32/ and lib/gcc/, in ~/.cache. It now says a prefix is the sysroot as it is unless it also holds a Linux system, names lib64/gcc/ as well, gives the view's place inside the Flatpak and the XDG_RUNTIME_DIR fallback, and says messages name headers by their path in the prefix. AGENTS.template.md describes MingwSysroot's two outcomes and MingwViewToPrefix, has the std PCM keyed on its flags in the cache rule, lists StdPcmKey among what VariantId hashes, and gains two gotchas: anything that changes how the std module is compiled goes into StdPcmFlags, and a CMake external whose recorded configure inputs differ is built again from an empty directory. Co-Authored-By: Claude Opus 5.5 --- AGENTS.template.md | 8 +++++--- README.md | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/AGENTS.template.md b/AGENTS.template.md index b60a81a..882b551 100644 --- a/AGENTS.template.md +++ b/AGENTS.template.md @@ -22,7 +22,7 @@ crafter-build is a C++26 modules build system. A project is described by a `proj Partitions and what they own: -- `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release. `MingwSysroot` makes the mingw target's `--sysroot`, a view in the cache of the mingw-w64 prefix's own directories: the prefix itself is never a sysroot, since clang would search its `include/` and `lib/`, and for `/usr` those are the Linux system's (issue #43). +- `:Toolchain` resolves the compiler once per process (`GetToolchain`): clang++, clang, llvm-ar, libclang, libc++'s std module source and the default sysroots, from `CRAFTER_BUILD_LLVM_DIR`, else a toolchain installed beside crafter-build, else PATH. It also holds the version baked in at build time. `SysrootStdModule` picks the std module source for a sysroot build by the sysroot's libc++ release. `MingwSysroot` settles the mingw target's `--sysroot`: the mingw-w64 prefix itself, or, for a prefix that also holds a Linux system (the host's `/usr`, anything whose `include/` has `features.h`), a view in the cache of its mingw-w64 directories, since clang would search its `include/` and `lib/` too (issue #43). `MingwViewToPrefix` turns paths into the view back into paths into the prefix for messages. - `:Clang` holds `Configuration`, `Build()`, `VariantId()`, `ApplyStandardArgs`, `HostTarget`, the CLI entry point `Crafter::Run`, and the lint, test and script data types. - `:Platform` runs commands, implements `LoadProject` (compile and load `project.cpp`), the host PCM cache, `#pragma crafter dependency`, and all OS-specific code. - `:Interface` and `:Implementation` model module units, their staleness checks and their compiles. @@ -118,14 +118,14 @@ Beyond the rules: - There is no host-platform preprocessor split: the host is Linux. Target differences (mingw, wasm) branch on `cfg.target` strings at runtime, in the library as well as in `project.cpp` and tests. - A build step fails when its command exits nonzero, never because it printed. Use `RunBuildCommand` or `RunCommandChecked`; warnings go to `Progress::Diagnostic`. - Every output path goes through `Configuration::BinDir()`, `BuildDir()` and `PcmDir()`. Hand-rolled `path / "bin" / ...` clobbered variants before; do not add new ones. -- Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, the wasm baseline and variant flags, the mingw-w64 prefix for the mingw target, uninterpreted project args and the toolchain (`Toolchain::Key()`). +- Anything that changes what gets compiled or bundled belongs in `VariantId()`. It already hashes type, debug, sysroot, defines, compile flags, the wasm baseline and variant flags, the std module the units import (`StdPcmKey`), uninterpreted project args and the toolchain (`Toolchain::Key()`). ## Design rules These were decided deliberately. Do not undo them without the maintainer's say-so. 1. **Configuration is C++.** No JSON, YAML, TOML or DSL for project, runner or lint configuration. Lint rules are C++ callbacks; there is no clang-format or clang-tidy integration and no separate "format rule" type. A rule that calls `ctx.SetContent` is a transform, and that is the whole mechanism. -2. **Per-project artifacts go in `/build` and `/bin`. Only cross-project things go in `~/.cache/crafter.build/`:** the std PCM and Crafter.Build PCMs per target and march, external clones keyed by their spec hash, and the mingw sysroot views keyed by prefix. +2. **Per-project artifacts go in `/build` and `/bin`. Only cross-project things go in `~/.cache/crafter.build/`:** the std PCM and Crafter.Build PCMs per target, march, toolchain and std module flags, external clones keyed by their spec hash, and the mingw sysroot views keyed by prefix. 3. **Sources ship, PCMs build locally.** The Flatpak installs `*.cppm` under `/app/share/crafter-build/`. Nothing target-baked ships except the executable and the toolchain beside it, and the executable is built for a portable `-march`. 4. **crafter-build is a monolithic executable with `--export-dynamic`.** `project.so` resolves `Crafter::` symbols from the running binary at dlopen time. There is no Windows host; `x86_64-w64-mingw32` is a cross-compile target for user projects only. 5. **Convention over configuration.** Lead with the zero-config path. Env var, then CLI flag, then full `project.cpp` are the escalation, not the entry point. @@ -189,6 +189,8 @@ Local builds are two scripts in `project.cpp`. Run them with the native binary: - A `#pragma crafter dependency` target is rebuilt on every load on purpose: clang crashes rather than erroring when it imports a BMI whose own imported PCM was rebuilt underneath it, and the depfile check is what catches a regenerated host cache. Do not optimise the rebuild away. - A host-tooling library such as Crafter.LintRules may `import Crafter.Build;` without declaring it as a dependency, because `LoadProject` fills the cache it resolves from. Listing Crafter.Build in its `cfg.dependencies` would clone and build all of crafter-build for every consumer. Its test does link crafter-build, so that test is wired behind `IsTopLevelProject()`: without the guard, crafter-build loading the library as a dependency would load its own `project.cpp` again through the test's `GitProject` and hit the cycle guard. - `crafter-build clean` leaves a pragma dependency's build alone; it lives with the clone in the cache. +- Units see a std module rebuilt in place, because their depfiles list the BMIs they import (`-fmodule-file-deps`), but not one that moved to another directory: their old depfiles still name the old one. So anything that changes how the std module is compiled goes into `StdPcmFlags`, which `BuildStdPcm` compiles with and `StdPcmKey` hashes. `VariantId()` hashes `StdPcmKey`, so the build moves with it. A flag added to the command anywhere else leaves both directories where they are. +- A CMake external whose recorded configure inputs (`/.crafter-options`) differ from this build's is built again from an empty build directory, not reconfigured in place: CMake keeps what an old compiler or `--sysroot` compiled, and the check results it cached. - The bootstrap binary is compiled without `CRAFTER_BUILD_HAS_ASSET`, so `cfg.assets` only works in the self-hosted binary. Changing asset extensions means updating two maps in `Crafter.Build-Clang.cpp`: `assetOutput` and the transitive copy loop. - A warning is printed only when its unit actually recompiles. A build with nothing to do is silent, which is not evidence that a warning is gone. - A std module source only compiles against the headers of its own libc++ release, and a sysroot does not always ship one (Alpine). The Flatpak therefore carries the sources of the release next to the bundled one under `share/crafter-build/libc++-modules//`, one `libcxx--modules` manifest module each. When the bundled LLVM is bumped, keep the release it leaves behind as such a module: sysroots on the stable distro stay on it for months. The native loop has no such directory; stage one by hand under `share/crafter-build/` to test against a foreign sysroot. diff --git a/README.md b/README.md index 11248aa..95d936f 100644 --- a/README.md +++ b/README.md @@ -201,7 +201,7 @@ For elaborate setups (rsync, ssh ControlMaster, env vars on the spawn, `QEMU_LD_ For cross-arch builds, set `cfg.sysroot` on the `Configuration` to point at a sysroot for the target. `BuildStdPcm` and the compile/link commands both consume it: `--sysroot=` is passed to clang, and the std module is precompiled against the sysroot's libc++ headers. Ordinary compiles are pointed at the same headers (`-nostdinc++ -isystem /usr/include/c++/v1`), so a unit that `#include`s a standard header gets the sysroot's libc++ and not the host toolchain's. The module source has to come from the same libc++ release as those headers, so it is taken from `/usr/share/libc++/v1/std.cppm` when the sysroot ships it, else from the toolchain when both are the same major release, else from the sources crafter-build carries for other releases (`share/crafter-build/libc++-modules//`, `crafter-build --version` lists them). Alpine is the common case for the last two: its `libc++-dev` has the headers and no module source. When nothing matches, the build stops with one line naming both releases, installing that release's `share/libc++/v1` directory into the sysroot resolves it. The cheapest aarch64 sysroot is an extracted [Arch Linux ARM rootfs](http://os.archlinuxarm.org/) plus `libc++` + `libc++abi` + `gcc` installed into it via `pacman --root --config --dbpath /var/lib/pacman -Sy …` (point pacman at ALARM's mirrors via explicit config to avoid pulling host-arch packages). -For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed, with libstdc++ 15 or newer for its std module. clang finds it by the `x86_64-w64-mingw32-gcc` on `PATH`; `CRAFTER_MINGW_DIR` names another prefix (the directory holding `x86_64-w64-mingw32/` and `lib/gcc/`), and inside the Flatpak the host's `/usr` is used when it has mingw-w64. Without one the build stops with a message saying where it looked. clang is shown only the prefix's `x86_64-w64-mingw32/` and `lib/gcc/`, as links in `~/.cache/crafter.build/mingw-sysroot/`: it searches the `include/` and `lib/` of whatever it gets as the sysroot, and those of a prefix like `/usr` are the Linux system's own, glibc's headers among them. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported, there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. +For `--target=x86_64-w64-mingw32` (Windows from Linux), the pipeline cross-compiles as long as `mingw-w64` is installed, with libstdc++ 15 or newer for its std module. clang finds it by the `x86_64-w64-mingw32-gcc` on `PATH`; `CRAFTER_MINGW_DIR` names another prefix (the directory holding `x86_64-w64-mingw32/` and `lib/gcc/`), and inside the Flatpak the host's `/usr` is used when it has mingw-w64. Without one the build stops with a message saying where it looked. The prefix is clang's sysroot as it is, so Windows libraries installed into its `include/` and `lib/` are found, unless it also holds a Linux system: the host's `/usr` in the Flatpak, or any prefix whose `include/` has a C library's `features.h`, as `/usr` does. clang searches a sysroot's `include/` and `lib/`, and in such a prefix those hold glibc's headers and Linux libraries, so clang is shown only its `x86_64-w64-mingw32/`, `lib/gcc/` and `lib64/gcc/`, as links in `mingw-sysroot/` in the cache (`~/.cache/crafter.build/`, in the Flatpak `~/.var/app/net.catcrafts.CrafterBuild/cache/crafter.build/`, or `$XDG_RUNTIME_DIR/crafter.build/` when the cache's filesystem takes no symbolic links). Compiler messages still name those headers by their paths in the prefix. Pair it with `--runner=cmd:wine` to run the result. Only the MinGW ABI is supported, there is no MSVC target and no Windows host. `-fuse-ld=lld` is auto-injected on every link, and `-lstdc++exp` is auto-linked for the mingw target so user projects don't need to declare those flags. For `--target=wasm32-wasip1`, the WASI sysroot is the one beside the toolchain (the Flatpak bundles one) unless `CRAFTER_BUILD_WASI_SYSROOT` names another, and tests run under `wasmtime`. C++ exceptions work as on the other targets: every unit and the std module are compiled with `-fwasm-exceptions` in the standardized form of WebAssembly exception handling (exnref), against the exception-enabled libc++ in the sysroot's `lib//eh/`, and `-lunwind` is linked automatically. A sysroot without that libc++ (wasi-sdk before 33) stops the build with one line saying so. The result runs on engines with that proposal: Chrome 137, Firefox 131 and Safari 18.4 or newer, and the wasmtime the Flatpak bundles. A page from `EnableWasiBrowserRuntime` checks first and names what an older browser lacks in the console, instead of failing to compile the module. An exception that escapes `main` leaves the program as an uncaught WebAssembly exception (wasmtime reports `thrown Wasm exception` and exits 1, a page gets a `WebAssembly.Exception`) without calling `std::terminate`, so a `std::set_terminate` handler does not run and nothing prints the exception's type or `what()`: wasm exception handling unwinds in a single phase, and nothing finds out that no handler is left until the stack is gone. Catch it in `main` to report it.