Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues 3 Pull requests Projects Releases 4 Packages Wiki Activity Actions

fix: rebuild and relink on what changed, submodules at their revisions, and mingw found where it is
Some checks failed
Flatpak / flatpak (push) Failing after 1m12s
Details

Browse source 
- Shaders record the files glslang opened (a depfile) and their settings
  (source path, stage, entry point, defines, include directories) in the
  build directory, and rebuild when an included file or a setting changes,
  not only the shader's own source.
- An executable relinks when its link command changed, not only when an
  object is newer: a link flag in project.cpp, or a dependency's library
  that moved (an ExternalDependency built into a new cache directory). The
  command is recorded beside the objects; its flags come from ordered sets,
  so the order the dependency threads finish in can't change it.
- An external's newest artifact counts its libDirs and versioned .so names.
- Submodules are checked out at the revisions the fetched commit records,
  from the URLs its .gitmodules gives, after every clone, switch, checkout
  and pull, so a pinned commit builds against its own submodules, and one
  moved by hand is put back; with no .gitmodules, git isn't asked.
- Each external clone's fetch and build hold a file lock: parallel builds
  pulled into one clone at once ("cannot lock ref").
- mingw:
  - -stdlib=libstdc++ always;
  - the std module's source is found through libstdc++.modules.json in the
    mingw toolchain's own library directories (the generic lib/ holds the
    host's Linux one), asking the compiler once a run;
  - a toolchain that isn't there is a build error saying where it looked,
    not std::terminate;
  - CRAFTER_MINGW_DIR names the prefix, the directory holding
    x86_64-w64-mingw32/ and lib/gcc/;
  - inside the Flatpak the host's /usr mingw is used.
- RunScript's exit-status check reads as the one range it is.
- Tests: ConcurrentExternalFetch, ExternalArtifactTime, GitSubmodules,
  LinkFlagsChange, MingwMissing, ShaderIncludeChange; ShaderCompile and
  ToolchainResolve extended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-20 19:08:13 +02:00
commit 653dd0bb9b
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
26 changed files with 1075 additions and 109 deletions
Download patch file Download diff file Expand all files Collapse all files

94
implementations/Crafter.Build-Clang.cpp
View file

@ -535,14 +535,14 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
}
for (const Shader& shader : config.shaders) {
if (buildCancelled.load(std::memory_order_relaxed)) break;
if (shader.Check(outputDir)) continue;
threads.emplace_back([&shader, &outputDir, &shaderIncludeDirs, &buildError, &buildCancelled]() {
if (shader.Check(outputDir, buildDir, shaderIncludeDirs)) continue;
threads.emplace_back([&shader, &outputDir, &buildDir, &shaderIncludeDirs, &buildError, &buildCancelled]() {
// A source registered more than once shows up once per variant;
// name the variant so the lines can be told apart.
Progress::Task task(shader.variant.empty() ? std::format("Compiling shader {}", shader.path.filename().string()) : std::format("Compiling shader {} ({})", shader.path.filename().string(), shader.variant));
if (buildCancelled.load(std::memory_order_relaxed)) return;
std::string result = shader.Compile(outputDir, shaderIncludeDirs);
std::string result = shader.Compile(outputDir, buildDir, shaderIncludeDirs);
if (result.empty()) return;
bool expected = false;
@ -699,7 +699,13 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
std::string stdPcmResult;
{
Progress::Task task(std::format("Building std PCM ({}-{})", config.target, config.march));
stdPcmResult = BuildStdPcm(config, stdPcmDir/"std.pcm");
// The shader and external threads are running: an exception leaving
// here would destroy them joinable, which terminates the process.
try {
stdPcmResult = BuildStdPcm(config, stdPcmDir/"std.pcm");
} catch (const std::exception& e) {
stdPcmResult = std::format("building the std module for {}: {}", config.target, e.what());
}
}
if(!stdPcmResult.empty()) {
buildCancelled.store(true);
@ -728,8 +734,8 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
const bool isWasm = config.target.starts_with("wasm32");
std::string files;
std::unordered_set<std::string> libSet;
std::unordered_set<std::string> publicFlagSet;
std::set<std::string> libSet;
std::set<std::string> publicFlagSet;
std::mutex fileMutex;
std::vector<std::thread> depThreads;
depThreads.reserve(config.dependencies.size());
@ -804,6 +810,9 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
: std::format(" -march={} -mtune={}", config.march, config.mtune);
if (!config.sysroot.empty()) {
cArchFlags += std::format(" --sysroot={}", config.sysroot);
} else if (config.target == "x86_64-w64-mingw32" && !GetToolchain().mingwSysroot.empty()) {
// The C++ side gets it through GetBaseCommand.
cArchFlags += std::format(" --sysroot={}", GetToolchain().mingwSysroot.string());
}
if (cIsWasm) {
// Matches the C++ path's wasi flag set so any libc shim defines
@ -1149,8 +1158,48 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
}
}
// The exact command the link (or archive) step runs, kept beside the
// objects once it has succeeded. A different one relinks even with no
// object newer than the output: a link flag changed in project.cpp, or a
// dependency's library moved, as an ExternalDependency's does when its
// options change and it is built into a new cache directory.
std::string linkCommand;
if (config.type == ConfigurationType::Executable) {
if (config.target.starts_with("wasm32")) {
linkCommand = std::format("{}{} -o {}.wasm -fuse-ld=lld{}", command, files, (outputDir/config.outputName).string(), linkExtras);
} else {
linkCommand = std::format("{}{} -o {} -fuse-ld=lld{}", command, files, (outputDir/config.outputName).string(), linkExtras);
}
} else if (config.type == ConfigurationType::LibraryStatic) {
// ThinLTO emits LLVM bitcode objects; plain `ar` writes an archive
// index that omits their symbols, so a consumer's lld link can't
// pull the needed members. llvm-ar writes a bitcode-aware index
// and handles plain ELF and COFF objects just as well, so every
// build uses it.
linkCommand = std::format("{} rcs {}.a {}", GetToolchain().llvmAr.string(), (outputDir/fs::path(std::string("lib")+config.outputName)).string(), files);
} else if (config.target == "x86_64-w64-mingw32") {
// LibraryDynamic. Output names follow each target's convention so
// consumers can link via the standard linker search:
// mingw: <name>.dll + lib<name>.dll.a (lld --out-implib)
// unix: lib<name>.so (rpath $ORIGIN)
fs::path dll = outputDir / std::format("{}.dll", config.outputName);
fs::path implib = outputDir / std::format("lib{}.dll.a", config.outputName);
linkCommand = std::format("{}{} -shared -o {} -Wl,--out-implib,{} -fuse-ld=lld{}", command, files, dll.string(), implib.string(), linkExtras);
} else {
linkCommand = std::format("{}{} -shared -o {}.so -Wl,-rpath,'$ORIGIN' -fuse-ld=lld{}", command, files, (outputDir/(std::string("lib")+config.outputName)).string(), linkExtras);
}
const fs::path linkRecord = buildDir / std::format("{}.link", config.outputName);
if (!buildResult.repack) {
std::ifstream recorded(linkRecord, std::ios::binary);
const std::string previous((std::istreambuf_iterator<char>(recorded)), std::istreambuf_iterator<char>());
if (previous != linkCommand) buildResult.repack = true;
}
if(buildResult.repack) {
Progress::Task task(std::format("Linking {}", config.outputName));
// The record goes first, so a link that fails leaves it stale.
std::error_code removed;
fs::remove(linkRecord, removed);
if(config.type == ConfigurationType::Executable) {
if(config.target == "x86_64-w64-mingw32") {
try {
@ -1181,30 +1230,10 @@ BuildResult Crafter::Build(Configuration& config, std::unordered_map<fs::path, s
return {e.what(), false, {}};
}
}
if (config.target.starts_with("wasm32")) {
buildResult.result = RunBuildCommand(std::format("{}{} -o {}.wasm -fuse-ld=lld{}", command, files, (outputDir/config.outputName).string(), linkExtras));
} else {
buildResult.result = RunBuildCommand(std::format("{}{} -o {} -fuse-ld=lld{}", command, files, (outputDir/config.outputName).string(), linkExtras));
}
} else if(config.type == ConfigurationType::LibraryStatic) {
// ThinLTO emits LLVM bitcode objects; plain `ar` writes an archive
// index that omits their symbols, so a consumer's lld link can't
// pull the needed members. llvm-ar writes a bitcode-aware index
// and handles plain ELF and COFF objects just as well, so every
// build uses it.
buildResult.result = RunBuildCommand(std::format("{} rcs {}.a {}", GetToolchain().llvmAr.string(), (outputDir/fs::path(std::string("lib")+config.outputName)).string(), files));
} else {
// LibraryDynamic. Output names follow each target's convention so
// consumers can link via the standard linker search:
// mingw: <name>.dll + lib<name>.dll.a (lld --out-implib)
// unix: lib<name>.so (rpath $ORIGIN)
if (config.target == "x86_64-w64-mingw32") {
fs::path dll = outputDir / std::format("{}.dll", config.outputName);
fs::path implib = outputDir / std::format("lib{}.dll.a", config.outputName);
buildResult.result = RunBuildCommand(std::format("{}{} -shared -o {} -Wl,--out-implib,{} -fuse-ld=lld{}", command, files, dll.string(), implib.string(), linkExtras));
} else {
buildResult.result = RunBuildCommand(std::format("{}{} -shared -o {}.so -Wl,-rpath,'$ORIGIN' -fuse-ld=lld{}", command, files, (outputDir/(std::string("lib")+config.outputName)).string(), linkExtras));
}
}
buildResult.result = RunBuildCommand(linkCommand);
if (buildResult.result.empty()) {
std::ofstream(linkRecord, std::ios::binary | std::ios::trunc) << linkCommand;
}
}
@ -1689,7 +1718,10 @@ Environment:
instead of the compiler's own.
CRAFTER_BUILD_LIBCLANG Exact libclang to load for lint.
CRAFTER_BUILD_WASI_SYSROOT WASI sysroot for wasm32 targets.
CRAFTER_MINGW_DIR mingw-w64 sysroot, passed as --sysroot.
CRAFTER_MINGW_DIR mingw-w64 prefix (holding x86_64-w64-mingw32/
and lib/gcc/), passed as --sysroot. Default:
the mingw gcc on PATH; in the Flatpak, the
host's /usr when it has mingw-w64.
CRAFTER_BUILD_HOME Directory holding the module sources.
Exit status: