fix(external): fetch and build each clone once per build pass #39
2026-10-31
docs: an unpinned external is pulled once per build
Say how often an unpinned ExternalDependency is pulled now that the configurations of a build share it, and list the target in the clone's cache key, which it is part of. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test(external): race BuildExternal as well as GitFetch
ConcurrentExternalFetch raced eight threads through GitFetch only. BuildExternal takes the same lock on a line of its own, so that line could go without any test noticing. Now that the configurations of one pass share a single BuildExternal, its lock is what keeps two crafter-builds, or two passes, out of each other's pulls. The same threads now fetch through both, each into a clone of its own, and both clones must end at the last round. With the lock in BuildExternal removed, the test failed in two of three runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test(external): tests that share a dependency fetch it once a run
Eight tests that include a header from the same tracked branch of a
local repository go through RunTests at once, and again after the
branch has moved. GIT_TRACE logs every git process, and each fetch of
the repository, clone or pull, runs one `git upload-pack` in it, so the
trace counts the fetches: one per run rather than one per test, and the
second run still pulls and brings the clone to the new commit.
On the commit before the fix both runs fetch eight times. The tests
themselves pass there, because the lock from 653dd0b already keeps the
pulls apart. Going through RunTests, rather than Build() on threads of
the test's own, also holds RunTests to building its tests with one
depResults.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(external): fetch and build each clone once per build pass
`crafter-build test` builds a project's tests side by side, and a
project whose tests each copy its externalDependencies hands every one
of them the same unpinned dependency. Each test's Build() pulled the
shared clone. Before the clone had a lock, those pulls ran at the same
time and broke its .git/FETCH_HEAD, which failed every test of
Crafter.Graphics (issue #34). Since 653dd0b they wait for each other,
but every one is still a network round trip: 26 tests cost 26 pulls of
each dependency, one after another, and for a CMake dependency as many
configure checks and builds.
depResults already records one build pass, keyed by the PcmDir of every
dependency configuration built in it. An external dependency now goes
in the same map, keyed by the directory its build leaves
(ExternalBuildDir: the clone, or for CMake its build directory). The
first configuration that needs it fetches and builds it; the others
wait for that and take its outcome. Keying on the build directory
rather than the clone keeps a CMake declaration from taking the outcome
of a header-only declaration of the same clone, which only fetched it.
A fresh depResults is a fresh pass, so the next build still pulls, and
every caller of Build() gets this: RunTests, a plain build, and the
pragma dependencies LoadProject builds.
To make that possible BuildExternal is split. BuildExternal is now the
shared work, lock, fetch, configure and build, and returns the error.
ExternalOutputs computes what one declaration compiles and links with,
because two declarations of one clone can differ in includeDirs,
libDirs and libs; ExternalBuildResult loses its error field. The work
no longer stops between steps when the configuration that started it
is cancelled: others may be waiting on it, and a run cut short would
leave them a clone nobody fetched. A dependency configuration's Build()
already runs to completion for the same reason.
The promise and future the dependency threads used to build or wait for
a configuration are now one helper, OncePerPass, which both kinds use.
An exception thrown in BuildExternal used to escape its thread into
std::terminate; it is now a build error, as for a dependency.
With eight tests that share a local dependency, a test run fetched it
eight times before this and fetches it once now (SharedExternalFetch,
next commit).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>