crafter-build test: parallel jobs race on git pull in a shared external clone, every test fails #34

New issue

Open

opened 2026-10-06 13:25:47 +00:00 by jorijnvdgraaf · 0 comments

jorijnvdgraaf commented

2026-10-06 13:25:47 +00:00

Owner

Summary

crafter-build test with the default job count fails every test of a project whose test configurations share an unpinned external dependency. Seen in Crafter.Graphics (26 tests, 56 hardware threads), twice in a row:

❌ UIFallbackParity (0ms) exit -1
    build failed: git fetch for 'Vulkan-Headers': `git -C '~/.cache/crafter.build/external/Vulkan-Headers-3035f4a8827e4e9d' pull`
    failed (exit 128): fatal: not something we can merge in .git/FETCH_HEAD: ithub.com/KhronosGroup/Vulkan-Headers
… (the same for all 26)
26 failed

crafter-build test --jobs=1 passes 26 of 26 (three runs).
Running that git pull by hand succeeds ("Already up to date").
Building or testing a single configuration works.

crafter-build 2.1.0-2-gc6b0bba, with local changes that do not touch dependency fetching. Line numbers refer to c6b0bba.

Likely cause

Read from the code; I did not instrument it.

RunTests builds the test configurations on jobs worker threads (implementations/Crafter.Build-Test.cpp:530, :604, :654).
In Crafter.Graphics every test Configuration carries its own copy of externalDependencies (tc.externalDependencies = cfg.externalDependencies, once per test in its project.cpp), including the unpinned Vulkan-Headers and Vulkan-Utility-Libraries.
For an unpinned source, FetchGit runs git pull in the shared clone directory (implementations/Crafter.Build-External.cpp:43-74). Nothing serialises that per clone: ProjectCacheMutex (:302) guards the project cache only.

Concurrent pulls in one clone race on .git/FETCH_HEAD; the truncated ithub.com/… in the message looks like a torn read of that file. I did not work out why all 26 fail rather than a few.

Suggested fix

Fetch each external clone at most once per process: memoise by clone directory, the way depResults memoises configurations.
Take a file lock in the clone directory around FetchGit, so two crafter-build processes are safe too. The cache already keeps a .lock next to each std PCM.
A regression test in the style of ConcurrentCacheRace: several configurations with the same unpinned external dependency, built in parallel.

## Summary `crafter-build test` with the default job count fails every test of a project whose test configurations share an unpinned external dependency. Seen in Crafter.Graphics (26 tests, 56 hardware threads), twice in a row: ``` ❌ UIFallbackParity (0ms) exit -1 build failed: git fetch for 'Vulkan-Headers': `git -C '~/.cache/crafter.build/external/Vulkan-Headers-3035f4a8827e4e9d' pull` failed (exit 128): fatal: not something we can merge in .git/FETCH_HEAD: ithub.com/KhronosGroup/Vulkan-Headers … (the same for all 26) 26 failed ``` - `crafter-build test --jobs=1` passes 26 of 26 (three runs). - Running that `git pull` by hand succeeds ("Already up to date"). - Building or testing a single configuration works. crafter-build `2.1.0-2-gc6b0bba`, with local changes that do not touch dependency fetching. Line numbers refer to `c6b0bba`. ## Likely cause Read from the code; I did not instrument it. - `RunTests` builds the test configurations on `jobs` worker threads (`implementations/Crafter.Build-Test.cpp:530`, `:604`, `:654`). - In Crafter.Graphics every test `Configuration` carries its own copy of `externalDependencies` (`tc.externalDependencies = cfg.externalDependencies`, once per test in its `project.cpp`), including the unpinned `Vulkan-Headers` and `Vulkan-Utility-Libraries`. - For an unpinned source, `FetchGit` runs `git pull` in the shared clone directory (`implementations/Crafter.Build-External.cpp:43-74`). Nothing serialises that per clone: `ProjectCacheMutex` (`:302`) guards the project cache only. Concurrent pulls in one clone race on `.git/FETCH_HEAD`; the truncated `ithub.com/…` in the message looks like a torn read of that file. I did not work out why all 26 fail rather than a few. ## Suggested fix - Fetch each external clone at most once per process: memoise by clone directory, the way `depResults` memoises configurations. - Take a file lock in the clone directory around `FetchGit`, so two crafter-build processes are safe too. The cache already keeps a `.lock` next to each std PCM. - A regression test in the style of `ConcurrentCacheRace`: several configurations with the same unpinned external dependency, built in parallel.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Catcrafts/Crafter.Build#34

No description provided.

Rows
Columns