Catcrafts/Crafter.Build
Watch
2
0
Fork
You've already forked Crafter.Build
 0
Code Issues 3 Pull requests Projects Releases 4 Packages Wiki Activity Actions

fix: key PCM and project.so freshness on content, not installed mtimes

Browse source 
std.pcm was rebuilt when it was older than std.cppm, and project.so when
it was older than the crafter-build executable. Both inputs are installed
files, and an OSTree deployment (the Flatpak) gives every installed file
mtime 0: std.cppm and the executable always looked older than the cache,
so a clang bump would never rebuild std.pcm and a crafter-build update
would never recompile project.so. clang crashes on a PCM from another
compiler instead of reporting it, so the failure would have been a
backtrace on the user's first build after an update.

Each artifact now has a one-line stamp beside it. std.pcm's covers the
precompile command and the module source's bytes, which also catches a
libc++ update under the same clang. project.so's covers the crafter-build
version, the toolchain key, the module source stamp and the compile
command; project.cpp's and the pre-dependency artifacts' mtimes remain
inputs, since those live next to it with real timestamps. The module
source stamp stands in for the executable: project.so resolves Crafter::
symbols from the running binary, so only declarations matter, and an
implementation-only rebuild of crafter-build no longer recompiles it.
The std.pcm stamp folds into the module cache stamp, so a rebuilt
std.pcm rebuilds the PCMs that import it.

The cache directory carries the toolchain key
(<target>-<march>-<clang version>-<hash>), so a compiler bump starts
cold instead of handing the new clang PCMs it will crash on. LoadProject
and Build() now derive the directory from the same StdPcmDir. Old
<target>-<march> directories are left behind; `clean` never touched the
cache and still does not.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-02 22:32:35 +02:00
commit 59fc9501f1
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
4 changed files with 256 additions and 56 deletions
Download patch file Download diff file Expand all files Collapse all files

141
implementations/Crafter.Build-Platform.cpp
View file

@ -90,15 +90,35 @@ namespace {
return cacheDir / "crafter-build-sources.stamp";
}
std::string ReadCacheStamp(const fs::path& cacheDir) {
std::ifstream in(CacheStampPath(cacheDir), std::ios::binary);
// A stamp is a one-line content hash kept next to the artifact it
// describes. Written last, after the artifact, so an interrupted build
// leaves a stamp that disagrees and the next run starts over.
std::string ReadStamp(const fs::path& file) {
std::ifstream in(file, std::ios::binary);
std::string stamp;
std::getline(in, stamp);
return stamp;
}
void WriteCacheStamp(const fs::path& cacheDir, const std::string& stamp) {
std::ofstream(CacheStampPath(cacheDir), std::ios::binary | std::ios::trunc) << stamp << '\n';
void WriteStamp(const fs::path& file, std::string_view stamp) {
std::ofstream(file, std::ios::binary | std::ios::trunc) << stamp << '\n';
}
fs::path StampPath(const fs::path& artifact) {
fs::path stamp = artifact;
stamp += ".stamp";
return stamp;
}
std::string HashStamp(std::string_view content) {
return std::format("{:016x}", std::hash<std::string_view>{}(content));
}
std::string FileBytes(const fs::path& file) {
std::ifstream in(file, std::ios::binary);
std::ostringstream buffer;
buffer << in.rdbuf();
return buffer.str();
}
}
@ -495,12 +515,17 @@ namespace {
return out;
}
// project.so predates its source, the crafter-build that will load it, or
// a pre-dependency artifact — any of them means recompiling.
bool ProjectFileStale(const fs::path& artifact, const fs::path& project, const fs::path& hostExe, const ProjectFileDependencies& deps) {
// project.so is stale when its stamp (crafter-build version, toolchain,
// module sources and compile command) disagrees, or when it predates its
// source or a pre-dependency artifact. The executable's mtime is not an
// input any more: an OSTree deployment dates every installed file at the
// epoch, so a comparison against it never fired after an update and the
// old project.so was loaded into a binary it did not match.
bool ProjectFileStale(const fs::path& artifact, const fs::path& project, std::string_view stamp, const ProjectFileDependencies& deps) {
if (!fs::exists(artifact)) return true;
if (ReadStamp(StampPath(artifact)) != stamp) return true;
const fs::file_time_type built = fs::last_write_time(artifact);
return built < fs::last_write_time(project) || built < fs::last_write_time(hostExe) || built < deps.newestArtifact;
return built < fs::last_write_time(project) || built < deps.newestArtifact;
}
}
@ -611,21 +636,22 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) {
fs::path stdCc = GetToolchain().mingwSysroot / "include" / "c++" / mingWversion / "bits" / "std.cc";
if(!fs::exists(stdPcm) || fs::last_write_time(stdPcm) < fs::last_write_time(stdCc)) {
// -femulated-tls keeps PCM TLS access matching libstdc++'s emutls
// definitions; mismatch surfaces as undefined std::__once_callable.
// The std.cc → std.cppm copy uses a plain filename: held under the
// cache lock, no other builder writes this path concurrently.
fs::path stdCppm = stdPcm.parent_path() / "std.cppm";
std::error_code copyEc;
fs::copy_file(stdCc, stdCppm, fs::copy_options::overwrite_existing, copyEc);
if (copyEc) {
return std::format("copy {} -> {}: {}", stdCc.string(), stdCppm.string(), copyEc.message());
}
return RunBuildCommand(std::format("{} --target={} -march={} -mtune={} -femulated-tls -O3 -std=c++26 -Wno-reserved-identifier -Wno-reserved-module-identifier --precompile {} -o {}", GetBaseCommand(config), config.target, config.march, config.mtune, stdCppm.string(), stdPcm.string()));
} else {
return "";
// -femulated-tls keeps PCM TLS access matching libstdc++'s emutls
// definitions; mismatch surfaces as undefined std::__once_callable.
// The std.cc → std.cppm copy uses a plain filename: held under the
// cache lock, no other builder writes this path concurrently.
fs::path stdCppm = stdPcm.parent_path() / "std.cppm";
std::string command = std::format("{} --target={} -march={} -mtune={} -femulated-tls -O3 -std=c++26 -Wno-reserved-identifier -Wno-reserved-module-identifier --precompile {} -o {}", GetBaseCommand(config), config.target, config.march, config.mtune, stdCppm.string(), stdPcm.string());
std::string stamp = HashStamp(command + '\0' + FileBytes(stdCc));
if (fs::exists(stdPcm) && ReadStamp(StampPath(stdPcm)) == stamp) return "";
std::error_code copyEc;
fs::copy_file(stdCc, stdCppm, fs::copy_options::overwrite_existing, copyEc);
if (copyEc) {
return std::format("copy {} -> {}: {}", stdCc.string(), stdCppm.string(), copyEc.message());
}
std::string result = RunBuildCommand(command);
if (result.empty()) WriteStamp(StampPath(stdPcm), stamp);
return result;
} else {
const bool isWasm = config.target.starts_with("wasm32");
// wasi-sdk drops std.cppm at <sysroot>/share/libc++/v1/, the rest of
@ -669,11 +695,16 @@ std::string Crafter::BuildStdPcm(const Configuration& config, fs::path stdPcm) {
if (!isWasm && !config.sysroot.empty()) {
archFlags += std::format(" -nostdinc++ -isystem {}/usr/include/c++/v1", config.sysroot);
}
if(!fs::exists(stdPcm) || fs::last_write_time(stdPcm) < fs::last_write_time(stdCppm)) {
return RunBuildCommand(std::format("{} --target={} -std=c++26{}{} -O3 -Wno-reserved-identifier -Wno-reserved-module-identifier --precompile {} -o {}", GetBaseCommand(config), config.target, sysrootFlag, archFlags, stdCppm, stdPcm.string()));
} else {
return "";
}
// Keyed on the command and the module source's bytes, never on
// mtimes: an installed std.cppm can be dated at the epoch (OSTree),
// and a libc++ update under the same clang changes the bytes but
// not the path.
std::string command = std::format("{} --target={} -std=c++26{}{} -O3 -Wno-reserved-identifier -Wno-reserved-module-identifier --precompile {} -o {}", GetBaseCommand(config), config.target, sysrootFlag, archFlags, stdCppm, stdPcm.string());
std::string stamp = HashStamp(command + '\0' + FileBytes(stdCppm));
if (fs::exists(stdPcm) && ReadStamp(StampPath(stdPcm)) == stamp) return "";
std::string result = RunBuildCommand(command);
if (result.empty()) WriteStamp(StampPath(stdPcm), stamp);
return result;
}
}
@ -721,12 +752,17 @@ namespace {
"Crafter.Build",
};
void EnsureCrafterBuildPcms(const fs::path& sourceDir, const fs::path& cacheDir, const Configuration& host) {
// Returns the stamp of the module sources the cache was built from; it is
// one of project.so's inputs.
std::string EnsureCrafterBuildPcms(const fs::path& sourceDir, const fs::path& cacheDir, const Configuration& host) {
CacheLock lock(cacheDir);
// Re-derived under the lock: another builder may have refreshed the
// cache from its own sources while we waited.
std::string stamp = CrafterBuildSourceStamp(sourceDir, CrafterBuildModules);
const bool upToDate = ReadCacheStamp(cacheDir) == stamp;
// The std PCM is an input to every module PCM: its stamp is folded in
// so a rebuilt std.pcm (a libc++ update under the same clang) rebuilds
// them too, instead of leaving PCMs that import a std.pcm that is gone.
std::string stamp = HashStamp(CrafterBuildSourceStamp(sourceDir, CrafterBuildModules) + '\0' + ReadStamp(StampPath(cacheDir / "std.pcm")));
const bool upToDate = ReadStamp(CacheStampPath(cacheDir)) == stamp;
for (std::string_view name : CrafterBuildModules) {
fs::path cppmPath = sourceDir / std::format("{}.cppm", name);
fs::path pcmPath = cacheDir / std::format("{}.pcm", name);
@ -751,7 +787,8 @@ namespace {
// Last, so an interrupted rebuild leaves the stamp disagreeing with the
// sources and the next run starts over rather than trusting a half-built
// set of PCMs.
if (!upToDate) WriteCacheStamp(cacheDir, stamp);
if (!upToDate) WriteStamp(CacheStampPath(cacheDir), stamp);
return stamp;
}
}
@ -765,8 +802,6 @@ Configuration Crafter::LoadProject(const fs::path& projectFile, std::span<const
}
fs::path soPath = buildDir / std::format("{}.so", absProject.stem().string());
fs::path hostExe = fs::read_symlink("/proc/self/exe");
fs::path sourceDir = Crafter::GetCrafterBuildHome();
// The host toolchain's own triple (x86_64-pc-linux-gnu on glibc distros,
@ -776,7 +811,9 @@ Configuration Crafter::LoadProject(const fs::path& projectFile, std::span<const
hostConfig.target = HostTarget();
hostConfig.march = "native";
hostConfig.mtune = "native";
fs::path cacheDir = GetCacheDir() / std::format("{}-{}", hostConfig.target, hostConfig.march);
// The same directory Build() resolves the std PCM from, toolchain key
// included (see StdPcmDir).
fs::path cacheDir = StdPcmDir(hostConfig);
if (!fs::exists(cacheDir)) fs::create_directories(cacheDir);
std::string stdResult = BuildStdPcm(hostConfig, cacheDir / "std.pcm");
@ -784,7 +821,7 @@ Configuration Crafter::LoadProject(const fs::path& projectFile, std::span<const
throw std::runtime_error(std::format("Failed to build std.pcm: {}", stdResult));
}
EnsureCrafterBuildPcms(sourceDir, cacheDir, hostConfig);
const std::string sourcesStamp = EnsureCrafterBuildPcms(sourceDir, cacheDir, hostConfig);
// After the host PCMs exist (a dependency's build imports Crafter.Build
// from that cache) and before the staleness check (its artifacts are
@ -793,26 +830,30 @@ Configuration Crafter::LoadProject(const fs::path& projectFile, std::span<const
// clang crashes on it.
ProjectFileDependencies preDeps = ResolveProjectFileDependencies(absProject, hostConfig);
const bool stale = ProjectFileStale(soPath, absProject, hostExe, preDeps);
if (stale) {
// -fuse-ld=lld like every other link the tool performs: lld is a hard
// dependency, binutils' ld is not (Alpine's clang would default to it),
// and a dependency archive built in release holds ThinLTO bitcode,
// which GNU ld can't read.
std::string compileCmd = std::format(
"{} --target={} -march=native -mtune=native "
"-std=c++26 -shared -fPIC -O3 -fuse-ld=lld "
"-Wno-return-type-c-linkage "
"-fprebuilt-module-path={}{} "
"{} -o {}{}{}",
GetBaseCommand(hostConfig), hostConfig.target, cacheDir.string(), preDeps.compileFlags,
absProject.string(), soPath.string(), preDeps.linkFlags, GetToolchain().LibcxxLinkFlags());
// -fuse-ld=lld like every other link the tool performs: lld is a hard
// dependency, binutils' ld is not (Alpine's clang would default to it),
// and a dependency archive built in release holds ThinLTO bitcode,
// which GNU ld can't read.
std::string compileCmd = std::format(
"{} --target={} -march=native -mtune=native "
"-std=c++26 -shared -fPIC -O3 -fuse-ld=lld "
"-Wno-return-type-c-linkage "
"-fprebuilt-module-path={}{} "
"{} -o {}{}{}",
GetBaseCommand(hostConfig), hostConfig.target, cacheDir.string(), preDeps.compileFlags,
absProject.string(), soPath.string(), preDeps.linkFlags, GetToolchain().LibcxxLinkFlags());
// What project.so was compiled against. The module source stamp stands
// in for the executable: project.so resolves Crafter:: symbols from the
// running binary, so only the declarations (the .cppm files) matter,
// and an implementation-only rebuild of crafter-build leaves it valid.
const std::string projectStamp = HashStamp(std::format("{}\0{}\0{}\0{}", CrafterBuildVersion(), GetToolchain().Key(), sourcesStamp, compileCmd));
if (ProjectFileStale(soPath, absProject, projectStamp, preDeps)) {
std::string result = RunBuildCommand(compileCmd);
if (!result.empty()) {
throw std::runtime_error(std::format("Failed to compile project {}: {}", absProject.string(), result));
}
WriteStamp(StampPath(soPath), projectStamp);
}
void* handle = dlopen(soPath.c_str(), RTLD_NOW | RTLD_GLOBAL);