Loading a project scans the same sources once per Configuration: a Crafter.Graphics consumer spends ~10 s before anything builds #47

New issue

Open

opened 2026-11-01 01:01:36 +00:00 by jorijnvdgraaf · 0 comments

jorijnvdgraaf commented

2026-11-01 01:01:36 +00:00

Owner

Summary

Configuration::GetInterfacesAndImplementations reads and regex-scans every source it is given, every time it is called, with nothing cached between calls. When a project declares many configurations over the same files, each file is scanned once per configuration. Crafter.Graphics gives each of its tests the library's whole interface and implementation list, so its sources are scanned dozens of times.

Every consumer pays this on every load, including crafter-build run --list and crafter-build -r, because a dependency's project.cpp, test declarations included, runs whenever its consumer is loaded. In a Crafter.Graphics consumer it is about 10 s before anything is built or run.

Line numbers refer to a3fe1bd (5.0.0).

Measured

On 2026-11-01, with flatpak run net.catcrafts.CrafterBuild run --list in 3DForts, which depends on Crafter.Graphics:

The load takes 13.4–14.3 s with the network cut off (--unshare=network, plus a git wrapper that makes pull a no-op), and 20–28 s online.
strace -f -T on flock, fcntl, openat and execve shows no call and no gap over 0.5 s, so this is not a lock wait.
From second 2 to second 11 the process opens about 300 files a second, all inside the Crafter.Graphics clone. That is 3,423 opens of 117 distinct files, about 29 per file. Window.cppm, WebGPU.cppm and WebGPUComputeShader.cppm are each opened 59 times.
crafter-build -r on a Crafter.Graphics example spends about 14 s before the program starts.
For comparison, Crafter.Network, with one unpinned dependency, loads in 0.33 s with pulls skipped.
The eight unpinned dependencies' git pulls run one after another, about 0.2 s each. That is a small share of the total.

Where

implementations/Crafter.Build-Clang.cpp:81, GetInterfacesAndImplementations. For each interface (:89-100) and each implementation (:188-196), it reads the whole file, strips comments with two std::regex_replace passes, and then runs the module and import patterns (:110, :123, :153, :164, :179, :201, :208, :223, :238). Nothing remembers a file from one call to the next.
Crafter.Graphics' project.cpp:389, in the lambda that declares each test: c.GetInterfacesAndImplementations(ifaces, sources), where sources is the library's implementations plus the test's main. It runs once per test, for 61 test directories.

Proposal

Cache the per-file scan for the life of the process, keyed on the absolute path plus size and mtime, or a content hash. Store what the scan finds, the module or partition declared and the imports, and let every Configuration that names the file reuse it. Resolving those imports against the configuration's own dependencies (ResolveImportName) stays per configuration; only reading and matching are shared.

#45 would replace the regexes with clang-scan-deps, a preprocessor run per file. That makes a per-file cache more important, not less, and the two fit together: the cache stores whatever the scanner returns.

Independent of this, a library can avoid the cost for its consumers by declaring its tests only when IsTopLevelProject(), as Crafter.LintRules does for its test. That would be a change in Crafter.Graphics, not here.

## Summary `Configuration::GetInterfacesAndImplementations` reads and regex-scans every source it is given, every time it is called, with nothing cached between calls. When a project declares many configurations over the same files, each file is scanned once per configuration. Crafter.Graphics gives each of its tests the library's whole interface and implementation list, so its sources are scanned dozens of times. Every consumer pays this on every load, including `crafter-build run --list` and `crafter-build -r`, because a dependency's `project.cpp`, test declarations included, runs whenever its consumer is loaded. In a Crafter.Graphics consumer it is about 10 s before anything is built or run. Line numbers refer to `a3fe1bd` (5.0.0). ## Measured On 2026-11-01, with `flatpak run net.catcrafts.CrafterBuild run --list` in 3DForts, which depends on Crafter.Graphics: - The load takes 13.4–14.3 s with the network cut off (`--unshare=network`, plus a `git` wrapper that makes `pull` a no-op), and 20–28 s online. - `strace -f -T` on `flock`, `fcntl`, `openat` and `execve` shows no call and no gap over 0.5 s, so this is not a lock wait. - From second 2 to second 11 the process opens about 300 files a second, all inside the Crafter.Graphics clone. That is 3,423 opens of 117 distinct files, about 29 per file. `Window.cppm`, `WebGPU.cppm` and `WebGPUComputeShader.cppm` are each opened 59 times. - `crafter-build -r` on a Crafter.Graphics example spends about 14 s before the program starts. - For comparison, Crafter.Network, with one unpinned dependency, loads in 0.33 s with pulls skipped. - The eight unpinned dependencies' `git pull`s run one after another, about 0.2 s each. That is a small share of the total. ## Where - `implementations/Crafter.Build-Clang.cpp:81`, `GetInterfacesAndImplementations`. For each interface (`:89-100`) and each implementation (`:188-196`), it reads the whole file, strips comments with two `std::regex_replace` passes, and then runs the module and import patterns (`:110`, `:123`, `:153`, `:164`, `:179`, `:201`, `:208`, `:223`, `:238`). Nothing remembers a file from one call to the next. - Crafter.Graphics' `project.cpp:389`, in the lambda that declares each test: `c.GetInterfacesAndImplementations(ifaces, sources)`, where `sources` is the library's implementations plus the test's `main`. It runs once per test, for 61 test directories. ## Proposal Cache the per-file scan for the life of the process, keyed on the absolute path plus size and mtime, or a content hash. Store what the scan finds, the module or partition declared and the imports, and let every Configuration that names the file reuse it. Resolving those imports against the configuration's own dependencies (`ResolveImportName`) stays per configuration; only reading and matching are shared. #45 would replace the regexes with `clang-scan-deps`, a preprocessor run per file. That makes a per-file cache more important, not less, and the two fit together: the cache stores whatever the scanner returns. Independent of this, a library can avoid the cost for its consumers by declaring its tests only when `IsTopLevelProject()`, as Crafter.LintRules does for its test. That would be a change in Crafter.Graphics, not here.

jorijnvdgraaf referenced this issue

2026-11-01 01:09:24 +00:00

Send progress output to stderr, so stdout carries only a command's own output #48

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Catcrafts/Crafter.Build#47

No description provided.

Rows
Columns