`crafter-build test` builds a project's tests side by side, and a
project whose tests each copy its externalDependencies hands every one
of them the same unpinned dependency. Each test's Build() pulled the
shared clone. Before the clone had a lock, those pulls ran at the same
time and broke its .git/FETCH_HEAD, which failed every test of
Crafter.Graphics (issue #34). Since 653dd0b they wait for each other,
but every one is still a network round trip: 26 tests cost 26 pulls of
each dependency, one after another, and for a CMake dependency as many
configure checks and builds.
depResults already records one build pass, keyed by the PcmDir of every
dependency configuration built in it. An external dependency now goes
in the same map, keyed by the directory its build leaves
(ExternalBuildDir: the clone, or for CMake its build directory). The
first configuration that needs it fetches and builds it; the others
wait for that and take its outcome. Keying on the build directory
rather than the clone keeps a CMake declaration from taking the outcome
of a header-only declaration of the same clone, which only fetched it.
A fresh depResults is a fresh pass, so the next build still pulls, and
every caller of Build() gets this: RunTests, a plain build, and the
pragma dependencies LoadProject builds.
To make that possible BuildExternal is split. BuildExternal is now the
shared work, lock, fetch, configure and build, and returns the error.
ExternalOutputs computes what one declaration compiles and links with,
because two declarations of one clone can differ in includeDirs,
libDirs and libs; ExternalBuildResult loses its error field. The work
no longer stops between steps when the configuration that started it
is cancelled: others may be waiting on it, and a run cut short would
leave them a clone nobody fetched. A dependency configuration's Build()
already runs to completion for the same reason.
The promise and future the dependency threads used to build or wait for
a configuration are now one helper, OncePerPass, which both kinds use.
An exception thrown in BuildExternal used to escape its thread into
std::terminate; it is now a build error, as for a dependency.
With eight tests that share a local dependency, a test run fetched it
eight times before this and fetches it once now (SharedExternalFetch,
next commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
116 lines
5.9 KiB
C++
116 lines
5.9 KiB
C++
// SPDX-License-Identifier: LGPL-3.0-only
|
|
// SPDX-FileCopyrightText: Copyright (C) 2026 Catcrafts®
|
|
|
|
export module Crafter.Build:External;
|
|
import std;
|
|
namespace fs = std::filesystem;
|
|
|
|
export namespace Crafter {
|
|
struct Configuration;
|
|
|
|
struct GitSource {
|
|
std::string url;
|
|
std::string branch;
|
|
std::string commit;
|
|
};
|
|
|
|
enum class ExternalBuilder {
|
|
None,
|
|
CMake,
|
|
};
|
|
|
|
struct ExternalDependency {
|
|
std::string name;
|
|
GitSource source;
|
|
ExternalBuilder builder = ExternalBuilder::None;
|
|
std::vector<std::string> options;
|
|
std::vector<fs::path> includeDirs;
|
|
// Extra library search paths (each becomes a -L flag), interpreted
|
|
// relative to the CMake build dir. Currently only honoured for
|
|
// ExternalBuilder::CMake. The CMake builder always adds its
|
|
// top-level build dir as a -L; libDirs is for projects (msquic,
|
|
// others) whose CMakeLists set LIBRARY_OUTPUT_DIRECTORY to a subdir
|
|
// of the build tree.
|
|
std::vector<fs::path> libDirs;
|
|
std::vector<std::string> libs;
|
|
};
|
|
|
|
struct ExternalBuildResult {
|
|
std::vector<std::string> compileFlags;
|
|
std::vector<std::string> linkFlags;
|
|
fs::file_time_type latestArtifact = fs::file_time_type::min();
|
|
};
|
|
|
|
// Where this dependency's clone lives in the cache. A pure function of the
|
|
// declaration and target — BuildExternal derives its own working directory
|
|
// through this, so a caller that only needs to know where the headers ended
|
|
// up cannot drift from where they actually are.
|
|
fs::path ExternalCloneDir(const ExternalDependency& dep, std::string_view target);
|
|
// Where BuildExternal leaves this dependency: the CMake build directory,
|
|
// or the clone itself when it is only fetched. Pure, like ExternalCloneDir.
|
|
// Build() keys a pass's record of the work on it rather than on the clone,
|
|
// so a declaration that shares the clone but builds it with CMake is not
|
|
// handed the outcome of a fetch alone.
|
|
fs::path ExternalBuildDir(const ExternalDependency& dep, std::string_view target);
|
|
// The -I flags this dependency contributes, from its declared includeDirs.
|
|
// Also pure, which is what lets the linter's AST layer parse a source that
|
|
// includes an external's headers without running a build to find out where
|
|
// they are. The paths only resolve once something has fetched the clone.
|
|
std::vector<std::string> ExternalIncludeFlags(const ExternalDependency& dep, std::string_view target);
|
|
// Fetches the clone and, for ExternalBuilder::CMake, configures and builds
|
|
// it. Empty on success, otherwise what failed. A lock on the clone is held
|
|
// throughout, so another crafter-build, or another build pass in this one,
|
|
// works on it before or after this call and never during it. Not
|
|
// cancellable once started: Build() hands the outcome to every
|
|
// configuration of the pass that declares the dependency, and a run cut
|
|
// short would leave them a clone nobody fetched.
|
|
std::string BuildExternal(const ExternalDependency& dep, std::string_view target);
|
|
// What a configuration that declares `dep` compiles and links with once
|
|
// BuildExternal has run: the -I flags and, for CMake, the -L, -rpath and -l
|
|
// flags and the newest library. Per declaration, because two that share a
|
|
// clone can still differ in includeDirs, libDirs and libs.
|
|
ExternalBuildResult ExternalOutputs(const ExternalDependency& dep, std::string_view target);
|
|
|
|
// Specification for a sibling crafter-build project to fetch and depend on.
|
|
// GitSource picks the revision: leave branch + commit empty for the
|
|
// remote's default branch HEAD (sloppy but convenient), set branch to
|
|
// track a branch tip, or set commit to pin to an immutable SHA. args are
|
|
// forwarded verbatim to the dep's CrafterBuildProject — typically you
|
|
// pass through the parent's args so target/march/debug propagate.
|
|
struct GitProjectSpec {
|
|
GitSource source;
|
|
fs::path projectFile = "project.cpp";
|
|
std::vector<std::string> args;
|
|
};
|
|
|
|
// Clones the spec's git URL into the per-project external cache (keyed by
|
|
// url+branch+commit+args+projectFile), recursively LoadProject's the
|
|
// remote's project.cpp, and returns a stable Configuration* you can drop
|
|
// into cfg.dependencies. The Configuration is owned by an internal cache
|
|
// for the lifetime of the build; identical specs share one Configuration.
|
|
Configuration* GitProject(const GitProjectSpec& spec);
|
|
|
|
// Specification for a local sibling crafter-build project (e.g. an
|
|
// example folder depending on its parent project, or a workspace where
|
|
// multiple projects live side-by-side without going through git).
|
|
struct LocalProjectSpec {
|
|
fs::path projectFile; // relative to cwd or absolute
|
|
std::vector<std::string> args; // forwarded to dep's CrafterBuildProject
|
|
};
|
|
|
|
// Same as GitProject but for a local on-disk project — no fetch, no
|
|
// cache copy. Resolves projectFile to its canonical absolute path and
|
|
// recursively LoadProject's it. Returns a stable Configuration* owned
|
|
// by the same internal cache as GitProject.
|
|
Configuration* LocalProject(const LocalProjectSpec& spec);
|
|
|
|
// Clone (or update) the given git source into the shared external
|
|
// cache and return the path to the cloned working tree. Intended for
|
|
// example projects that need large asset bundles which shouldn't
|
|
// live in the parent repo — push the returned path (or specific
|
|
// files inside it) into `cfg.assets`. Reuses the same cache root as
|
|
// ExternalDependency / GitProject. Caching key = url + branch + commit;
|
|
// identical calls reuse the existing clone. Pin `source.commit` to a
|
|
// SHA for reproducible, offline-after-first-fetch builds.
|
|
fs::path GitFetch(const GitSource& source);
|
|
}
|