Crafter.Build/tests/ExternalArtifactTime/main.cpp

89 lines
3.7 KiB
C++
Raw Normal View History

fix: rebuild and relink on what changed, submodules at their revisions, and mingw found where it is - Shaders record the files glslang opened (a depfile) and their settings (source path, stage, entry point, defines, include directories) in the build directory, and rebuild when an included file or a setting changes, not only the shader's own source. - An executable relinks when its link command changed, not only when an object is newer: a link flag in project.cpp, or a dependency's library that moved (an ExternalDependency built into a new cache directory). The command is recorded beside the objects; its flags come from ordered sets, so the order the dependency threads finish in can't change it. - An external's newest artifact counts its libDirs and versioned .so names. - Submodules are checked out at the revisions the fetched commit records, from the URLs its .gitmodules gives, after every clone, switch, checkout and pull, so a pinned commit builds against its own submodules, and one moved by hand is put back; with no .gitmodules, git isn't asked. - Each external clone's fetch and build hold a file lock: parallel builds pulled into one clone at once ("cannot lock ref"). - mingw: - -stdlib=libstdc++ always; - the std module's source is found through libstdc++.modules.json in the mingw toolchain's own library directories (the generic lib/ holds the host's Linux one), asking the compiler once a run; - a toolchain that isn't there is a build error saying where it looked, not std::terminate; - CRAFTER_MINGW_DIR names the prefix, the directory holding x86_64-w64-mingw32/ and lib/gcc/; - inside the Flatpak the host's /usr mingw is used. - RunScript's exit-status check reads as the one range it is. - Tests: ConcurrentExternalFetch, ExternalArtifactTime, GitSubmodules, LinkFlagsChange, MingwMissing, ShaderIncludeChange; ShaderCompile and ToolchainResolve extended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:08:13 +02:00
// SPDX-License-Identifier: LGPL-3.0-only
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
import std;
import Crafter.Build;
namespace fs = std::filesystem;
using namespace Crafter;
// A CMake dependency's newest library decides when its consumers are stale
fix(external): fetch and build each clone once per build pass `crafter-build test` builds a project's tests side by side, and a project whose tests each copy its externalDependencies hands every one of them the same unpinned dependency. Each test's Build() pulled the shared clone. Before the clone had a lock, those pulls ran at the same time and broke its .git/FETCH_HEAD, which failed every test of Crafter.Graphics (issue #34). Since 653dd0b they wait for each other, but every one is still a network round trip: 26 tests cost 26 pulls of each dependency, one after another, and for a CMake dependency as many configure checks and builds. depResults already records one build pass, keyed by the PcmDir of every dependency configuration built in it. An external dependency now goes in the same map, keyed by the directory its build leaves (ExternalBuildDir: the clone, or for CMake its build directory). The first configuration that needs it fetches and builds it; the others wait for that and take its outcome. Keying on the build directory rather than the clone keeps a CMake declaration from taking the outcome of a header-only declaration of the same clone, which only fetched it. A fresh depResults is a fresh pass, so the next build still pulls, and every caller of Build() gets this: RunTests, a plain build, and the pragma dependencies LoadProject builds. To make that possible BuildExternal is split. BuildExternal is now the shared work, lock, fetch, configure and build, and returns the error. ExternalOutputs computes what one declaration compiles and links with, because two declarations of one clone can differ in includeDirs, libDirs and libs; ExternalBuildResult loses its error field. The work no longer stops between steps when the configuration that started it is cancelled: others may be waiting on it, and a run cut short would leave them a clone nobody fetched. A dependency configuration's Build() already runs to completion for the same reason. The promise and future the dependency threads used to build or wait for a configuration are now one helper, OncePerPass, which both kinds use. An exception thrown in BuildExternal used to escape its thread into std::terminate; it is now a build error, as for a dependency. With eight tests that share a local dependency, a test run fetched it eight times before this and fetches it once now (SharedExternalFetch, next commit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 02:09:24 +01:00
// (the latestArtifact of ExternalOutputs). Only files directly in the CMake
// build directory named *.a or *.so were looked at, so msquic's
fix: rebuild and relink on what changed, submodules at their revisions, and mingw found where it is - Shaders record the files glslang opened (a depfile) and their settings (source path, stage, entry point, defines, include directories) in the build directory, and rebuild when an included file or a setting changes, not only the shader's own source. - An executable relinks when its link command changed, not only when an object is newer: a link flag in project.cpp, or a dependency's library that moved (an ExternalDependency built into a new cache directory). The command is recorded beside the objects; its flags come from ordered sets, so the order the dependency threads finish in can't change it. - An external's newest artifact counts its libDirs and versioned .so names. - Submodules are checked out at the revisions the fetched commit records, from the URLs its .gitmodules gives, after every clone, switch, checkout and pull, so a pinned commit builds against its own submodules, and one moved by hand is put back; with no .gitmodules, git isn't asked. - Each external clone's fetch and build hold a file lock: parallel builds pulled into one clone at once ("cannot lock ref"). - mingw: - -stdlib=libstdc++ always; - the std module's source is found through libstdc++.modules.json in the mingw toolchain's own library directories (the generic lib/ holds the host's Linux one), asking the compiler once a run; - a toolchain that isn't there is a build error saying where it looked, not std::terminate; - CRAFTER_MINGW_DIR names the prefix, the directory holding x86_64-w64-mingw32/ and lib/gcc/; - inside the Flatpak the host's /usr mingw is used. - RunScript's exit-status check reads as the one range it is. - Tests: ConcurrentExternalFetch, ExternalArtifactTime, GitSubmodules, LinkFlagsChange, MingwMissing, ShaderIncludeChange; ShaderCompile and ToolchainResolve extended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:08:13 +02:00
// bin/Release/libmsquic.so.2, in one of the dependency's libDirs and with a
// version after the .so, never counted: a rebuilt msquic left everything
// built against the old one in place.
//
// The dependency here is a local repository whose CMake project puts a
// versioned shared library in bin/Release, declared as a libDir. Everything
// is local; XDG_CACHE_HOME keeps the clone out of the real cache.
extern "C" int setenv(const char* name, const char* value, int overwrite);
namespace {
std::int32_t failures = 0;
void Check(bool cond, std::string_view msg) {
if (!cond) {
std::println(std::cerr, "FAIL: {}", msg);
++failures;
}
}
void Git(const fs::path& dir, std::string_view args) {
const std::string command = std::format("git -C {} {}", ShellQuote(dir.string()), args);
auto result = RunCommandWithTimeout(command, std::chrono::seconds(60));
if (result.exitCode != 0) throw std::runtime_error(std::format("{} failed: {}", command, result.output));
}
void Write(const fs::path& file, std::string_view text) {
std::ofstream(file, std::ios::binary | std::ios::trunc) << text;
}
}
int main() {
const fs::path scratch = fs::temp_directory_path() / "crafter-build-external-artifact-time";
fs::remove_all(scratch);
fs::create_directories(scratch);
setenv("XDG_CACHE_HOME", (scratch / "cache").c_str(), 1);
setenv("GIT_CONFIG_NOSYSTEM", "1", 1);
setenv("GIT_CONFIG_GLOBAL", "/dev/null", 1);
for (const char* name : { "GIT_AUTHOR_NAME", "GIT_COMMITTER_NAME" }) setenv(name, "crafter-build test", 1);
for (const char* name : { "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_EMAIL" }) setenv(name, "test@example.invalid", 1);
const fs::path repo = scratch / "versioned";
fs::create_directories(repo);
Write(repo / "lib.c", "int Versioned(void) { return 2; }\n");
Write(repo / "CMakeLists.txt",
"cmake_minimum_required(VERSION 3.16)\n"
"project(versioned C)\n"
"add_library(versioned SHARED lib.c)\n"
"set_target_properties(versioned PROPERTIES VERSION 2.0.0 SOVERSION 2\n"
" LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin/Release)\n");
Git(repo, "init -q -b main");
Git(repo, "add .");
Git(repo, "commit -q -m versioned");
ExternalDependency dep;
dep.name = "versioned";
dep.source.url = std::format("file://{}", repo.string());
dep.builder = ExternalBuilder::CMake;
dep.libDirs = { "bin/Release" };
dep.libs = { "versioned" };
fix(external): fetch and build each clone once per build pass `crafter-build test` builds a project's tests side by side, and a project whose tests each copy its externalDependencies hands every one of them the same unpinned dependency. Each test's Build() pulled the shared clone. Before the clone had a lock, those pulls ran at the same time and broke its .git/FETCH_HEAD, which failed every test of Crafter.Graphics (issue #34). Since 653dd0b they wait for each other, but every one is still a network round trip: 26 tests cost 26 pulls of each dependency, one after another, and for a CMake dependency as many configure checks and builds. depResults already records one build pass, keyed by the PcmDir of every dependency configuration built in it. An external dependency now goes in the same map, keyed by the directory its build leaves (ExternalBuildDir: the clone, or for CMake its build directory). The first configuration that needs it fetches and builds it; the others wait for that and take its outcome. Keying on the build directory rather than the clone keeps a CMake declaration from taking the outcome of a header-only declaration of the same clone, which only fetched it. A fresh depResults is a fresh pass, so the next build still pulls, and every caller of Build() gets this: RunTests, a plain build, and the pragma dependencies LoadProject builds. To make that possible BuildExternal is split. BuildExternal is now the shared work, lock, fetch, configure and build, and returns the error. ExternalOutputs computes what one declaration compiles and links with, because two declarations of one clone can differ in includeDirs, libDirs and libs; ExternalBuildResult loses its error field. The work no longer stops between steps when the configuration that started it is cancelled: others may be waiting on it, and a run cut short would leave them a clone nobody fetched. A dependency configuration's Build() already runs to completion for the same reason. The promise and future the dependency threads used to build or wait for a configuration are now one helper, OncePerPass, which both kinds use. An exception thrown in BuildExternal used to escape its thread into std::terminate; it is now a build error, as for a dependency. With eight tests that share a local dependency, a test run fetched it eight times before this and fetches it once now (SharedExternalFetch, next commit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 02:09:24 +01:00
const std::string error = BuildExternal(dep, HostTarget());
Check(error.empty(), std::format("the dependency builds: {}", error));
const ExternalBuildResult result = ExternalOutputs(dep, HostTarget());
fix: rebuild and relink on what changed, submodules at their revisions, and mingw found where it is - Shaders record the files glslang opened (a depfile) and their settings (source path, stage, entry point, defines, include directories) in the build directory, and rebuild when an included file or a setting changes, not only the shader's own source. - An executable relinks when its link command changed, not only when an object is newer: a link flag in project.cpp, or a dependency's library that moved (an ExternalDependency built into a new cache directory). The command is recorded beside the objects; its flags come from ordered sets, so the order the dependency threads finish in can't change it. - An external's newest artifact counts its libDirs and versioned .so names. - Submodules are checked out at the revisions the fetched commit records, from the URLs its .gitmodules gives, after every clone, switch, checkout and pull, so a pinned commit builds against its own submodules, and one moved by hand is put back; with no .gitmodules, git isn't asked. - Each external clone's fetch and build hold a file lock: parallel builds pulled into one clone at once ("cannot lock ref"). - mingw: - -stdlib=libstdc++ always; - the std module's source is found through libstdc++.modules.json in the mingw toolchain's own library directories (the generic lib/ holds the host's Linux one), asking the compiler once a run; - a toolchain that isn't there is a build error saying where it looked, not std::terminate; - CRAFTER_MINGW_DIR names the prefix, the directory holding x86_64-w64-mingw32/ and lib/gcc/; - inside the Flatpak the host's /usr mingw is used. - RunScript's exit-status check reads as the one range it is. - Tests: ConcurrentExternalFetch, ExternalArtifactTime, GitSubmodules, LinkFlagsChange, MingwMissing, ShaderIncludeChange; ShaderCompile and ToolchainResolve extended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:08:13 +02:00
const fs::path library = ExternalCloneDir(dep, HostTarget()) / "build" / "bin" / "Release" / "libversioned.so.2.0.0";
Check(fs::exists(library), std::format("the library is where its CMake project puts it: {}", library.string()));
if (fs::exists(library)) {
Check(result.latestArtifact == fs::last_write_time(library), "its newest artifact is the versioned library in the libDir");
}
std::error_code ec;
fs::remove_all(scratch, ec);
if (failures > 0) {
std::println(std::cerr, "{} assertions failed", failures);
return 1;
}
return 0;
}