Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues 3 Pull requests Projects Releases 4 Packages Wiki Activity Actions

fix: rebuild and relink on what changed, submodules at their revisions, and mingw found where it is
Some checks failed
Flatpak / flatpak (push) Failing after 1m12s
Details

Browse source 
- Shaders record the files glslang opened (a depfile) and their settings
  (source path, stage, entry point, defines, include directories) in the
  build directory, and rebuild when an included file or a setting changes,
  not only the shader's own source.
- An executable relinks when its link command changed, not only when an
  object is newer: a link flag in project.cpp, or a dependency's library
  that moved (an ExternalDependency built into a new cache directory). The
  command is recorded beside the objects; its flags come from ordered sets,
  so the order the dependency threads finish in can't change it.
- An external's newest artifact counts its libDirs and versioned .so names.
- Submodules are checked out at the revisions the fetched commit records,
  from the URLs its .gitmodules gives, after every clone, switch, checkout
  and pull, so a pinned commit builds against its own submodules, and one
  moved by hand is put back; with no .gitmodules, git isn't asked.
- Each external clone's fetch and build hold a file lock: parallel builds
  pulled into one clone at once ("cannot lock ref").
- mingw:
  - -stdlib=libstdc++ always;
  - the std module's source is found through libstdc++.modules.json in the
    mingw toolchain's own library directories (the generic lib/ holds the
    host's Linux one), asking the compiler once a run;
  - a toolchain that isn't there is a build error saying where it looked,
    not std::terminate;
  - CRAFTER_MINGW_DIR names the prefix, the directory holding
    x86_64-w64-mingw32/ and lib/gcc/;
  - inside the Flatpak the host's /usr mingw is used.
- RunScript's exit-status check reads as the one range it is.
- Tests: ConcurrentExternalFetch, ExternalArtifactTime, GitSubmodules,
  LinkFlagsChange, MingwMissing, ShaderIncludeChange; ShaderCompile and
  ToolchainResolve extended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-20 19:08:13 +02:00
commit 653dd0bb9b
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
26 changed files with 1075 additions and 109 deletions
Download patch file Download diff file Expand all files Collapse all files

139
implementations/Crafter.Build-Platform.cpp
View file

@ -33,27 +33,10 @@ namespace {
// pass through the lock before it reaches the compile steps that *read* those
// PCMs, so a reader never observes a half-written file.
//
// The lock lives in the OS, keyed on a `.lock` file in the cache dir, and is
// released automatically when the descriptor/handle closes — including on
// process death — so an interrupted build can never leave a stale lock. The
// descriptor is close-on-exec / non-inheritable so spawned clang processes
// don't keep the lock alive past our own release.
class CacheLock {
std::int32_t fd_ = -1;
public:
explicit CacheLock(const fs::path& cacheDir) {
fd_ = ::open((cacheDir / ".lock").c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0644);
if (fd_ >= 0) {
while (::flock(fd_, LOCK_EX) != 0 && errno == EINTR) {}
}
}
~CacheLock() {
if (fd_ >= 0) {
::close(fd_); // closing the descriptor drops the flock
}
}
CacheLock(const CacheLock&) = delete;
CacheLock& operator=(const CacheLock&) = delete;
// A FileLock on a `.lock` file in the cache dir.
struct CacheLock {
FileLock lock;
explicit CacheLock(const fs::path& cacheDir) : lock(cacheDir / ".lock") {}
};
// The cached Crafter.Build PCMs are keyed by `<target>-<march>` alone, so
@ -120,6 +103,51 @@ namespace {
buffer << in.rdbuf();
return buffer.str();
}
// The string value of `"key": "..."` in `object`, if it has one.
std::optional<std::string> JsonString(std::string_view object, std::string_view key) {
const std::string quoted = std::format("\"{}\"", key);
std::size_t at = object.find(quoted);
if (at == std::string_view::npos) return std::nullopt;
at = object.find_first_not_of(" \t\r\n", at + quoted.size());
if (at == std::string_view::npos || object[at] != ':') return std::nullopt;
at = object.find_first_not_of(" \t\r\n", at + 1);
if (at == std::string_view::npos || object[at] != '"') return std::nullopt;
std::string value;
for (std::size_t i = at + 1; i < object.size(); ++i) {
if (object[i] == '"') return value;
if (object[i] == '\\' && i + 1 < object.size()) ++i;
value += object[i];
}
return std::nullopt;
}
// The std module's "source-path" in a libstdc++.modules.json: the object
// in its "modules" array whose "logical-name" is "std". The manifest is
// a flat list of such objects, so each one is the text between its
// braces.
std::optional<std::string> StdModuleSourcePath(std::string_view manifest) {
for (std::size_t open = manifest.find('{', manifest.find("\"modules\"")); open != std::string_view::npos; open = manifest.find('{', open + 1)) {
const std::size_t close = manifest.find('}', open);
if (close == std::string_view::npos) break;
const std::string_view object = manifest.substr(open, close - open);
if (JsonString(object, "logical-name") == "std") return JsonString(object, "source-path");
}
return std::nullopt;
}
}
Crafter::FileLock::FileLock(const fs::path& path) {
fd = ::open(path.c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0644);
if (fd >= 0) {
while (::flock(fd, LOCK_EX) != 0 && errno == EINTR) {}
}
}
Crafter::FileLock::~FileLock() {
if (fd >= 0) {
::close(fd); // closing the descriptor drops the flock
}
}
fs::path Crafter::GetCrafterBuildHome() {
@ -464,7 +492,7 @@ namespace {
std::mutex depMutex;
std::unordered_set<Configuration*> forced;
std::unordered_set<Configuration*> walked;
std::unordered_set<std::string> libs;
std::set<std::string> libs;
for (const PragmaDependency& pragma : pragmas) {
Configuration* cfg = nullptr;
@ -646,23 +674,52 @@ CommandResult Crafter::RunCommandWithTimeout(std::string_view cmd, std::chrono::
std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) {
CacheLock lock(stdPcm.parent_path());
if(config.target == "x86_64-w64-mingw32") {
std::vector<std::string> folders;
// Iterate through the directory and collect all subdirectories
for (const auto& entry : fs::directory_iterator(GetToolchain().mingwSysroot / "include" / "c++")) {
if (entry.is_directory()) {
folders.push_back(entry.path().filename().string());
// libstdc++ 15 and newer install libstdc++.modules.json beside the
// library, naming the std module's source relative to itself. It is
// looked for in the library directories the compiler searches for
// this target, as it would look for libstdc++ itself, but only in
// those that belong to the mingw toolchain: the list also has the
// generic <prefix>/lib, where the host's own (Linux) libstdc++ keeps
// its manifest. A toolchain that isn't there is a build error, not
// an exception: Build() runs this with its worker threads started.
const Toolchain& tc = GetToolchain();
const std::string where = tc.mingwSysroot.empty()
? std::string("where clang finds it by the x86_64-w64-mingw32 gcc on PATH")
: std::format("under {}", tc.mingwSysroot.string());
const std::string missing = std::format("no mingw-w64 libstdc++ with the std module (libstdc++ 15 or newer) {}. Install mingw-w64, or point CRAFTER_MINGW_DIR at its prefix, the directory holding x86_64-w64-mingw32/ and lib/gcc/", where);
// Asked once a run for each compiler and target: every mingw
// configuration comes through here, up to date or not, and the answer
// costs a clang.
static std::mutex probeMutex;
static std::unordered_map<std::string, CommandResult> probes;
const std::string probe = std::format("{} --target={} -print-search-dirs", GetBaseCommand(config), config.target);
CommandResult dirs;
{
std::lock_guard guard(probeMutex);
auto found = probes.find(probe);
if (found == probes.end()) found = probes.emplace(probe, RunCommandChecked(probe)).first;
dirs = found->second;
}
if (dirs.exitCode != 0) return std::format("{}: {}", missing, dirs.output);
fs::path manifestPath;
constexpr std::string_view librariesLine = "libraries: =";
if (const std::size_t at = dirs.output.find(librariesLine); at != std::string::npos) {
const std::size_t end = dirs.output.find('\n', at);
const std::string list = dirs.output.substr(at + librariesLine.size(), end == std::string::npos ? std::string::npos : end - at - librariesLine.size());
for (const auto part : std::views::split(list, ':')) {
const fs::path dir(std::string_view(part.begin(), part.end()));
const bool mingwOwn = std::ranges::contains(dir, fs::path("x86_64-w64-mingw32"));
if (mingwOwn && fs::exists(dir / "libstdc++.modules.json")) {
manifestPath = dir / "libstdc++.modules.json";
break;
}
}
}
// Sort the folders by version in descending order
std::sort(folders.begin(), folders.end(), [](const std::string& a, const std::string& b) {
return std::lexicographical_compare(b.begin(), b.end(), a.begin(), a.end());
});
std::string mingWversion = folders.front();
fs::path stdCc = GetToolchain().mingwSysroot / "include" / "c++" / mingWversion / "bits" / "std.cc";
if (manifestPath.empty()) return missing;
const std::optional<std::string> source = StdModuleSourcePath(FileBytes(manifestPath));
if (!source) return std::format("{} names no std module", manifestPath.string());
const fs::path stdCc = (manifestPath.parent_path() / *source).lexically_normal();
if (!fs::exists(stdCc)) return std::format("{}, which {} names as the std module, is missing", stdCc.string(), manifestPath.string());
// -femulated-tls keeps PCM TLS access matching libstdc++'s emutls
// definitions; mismatch surfaces as undefined std::__once_callable.
@ -751,9 +808,11 @@ std::string Crafter::GetBaseCommand(const Configuration& config) {
const Toolchain& tc = GetToolchain();
std::string command = tc.clangxx.string();
if (config.target == "x86_64-w64-mingw32") {
// libstdc++ from the mingw toolchain, found by clang's own mingw
// discovery unless the sysroot was given explicitly.
if (tc.mingwSysrootExplicit) command += std::format(" --sysroot={}", tc.mingwSysroot.string());
// libstdc++ from the mingw toolchain, said outright: a clang built to
// default to libc++ (the Flatpak's) would otherwise reach for that.
// Found by clang's own mingw discovery unless there is a sysroot.
command += " -stdlib=libstdc++";
if (!tc.mingwSysroot.empty()) command += std::format(" --sysroot={}", tc.mingwSysroot.string());
return command;
}
command += " -stdlib=libc++";