test(bench): measure net perf gain from #40 to master in Sponza (#155) #156

Merged

catbot merged 2 commits from claude/issue-155 into master

2026-06-18 21:36:37 +02:00

catbot commented

2026-06-18 21:36:31 +02:00

Member

Resolves #155.

What

Issue #155 asked for an actual measurement of the net performance gain
from #40 to current master, in a representative scene (Sponza). This adds a
headless benchmark harness (examples/SponzaBench) and reports the measured
result.

SponzaBench is the native Vulkan Sponza RT scene, but it times setup and a
measured Render() loop and prints BENCH … lines instead of opening an
interactive window. Its closest-hit shades from barycentrics (samples no
texture), so unlike the interactive Sponza it builds the full 25-mesh
atrium (~262 k triangles) as one multi-instance TLAS — the dynamic-heap
hit-shader driver fault that forces single-material on native doesn't apply.

A second small change adds CRAFTER_PRESENT_IMMEDIATE to Window so frame
pacing can be uncapped from the compositor's vblank for the measurement
(FIFO stays the default).

Method

Built the same SponzaBench sources against each library revision (#40 =
1451e3a, HEAD = master, via a git worktree). RTX 4090, driver 610.43.02,
1280×720, validation disabled, IMMEDIATE present, 200 warmup + 2000
measured frames, median of 9 runs.

Result — the net gain in Sponza is small (~1–2 % frames, ~2–3 % setup)

Full atrium (GPU-traversal bound):

metric	#40	HEAD	Δ
setup (cold) ms	322.7	316.6	−1.9 %
setup (warm cache #69) ms	322.7*	313.9	−2.7 %
fps	7 560	7 637	+1.0 %
frame p50 ms	0.1311	0.1298	−1.0 %
peak RSS warm MB	336.0	332.0	−1.2 %

Light scene (CPU-bound): fps 15 022 → 15 237 (+1.4 %), frame p50
0.0662 → 0.0650 ms (−1.8 %).

*#40 predates the disk pipeline cache (#69), so its setup is always cold.

Why so small (the honest part)

This is a real result, not a flat run — it's about what Sponza exercises:

The biggest block of post-#40 PRs is UI / text (#46–57, #61,
#122–129, #132). Sponza has no UI → zero contribution.
The next block optimises per-frame dynamic uploads (#118 TLAS
dirty-track, #119 mesh refit, #120 staging ring). A static scene
builds its TLAS once and never re-uploads → not exercised in steady state.
WebGPU-only (#130/#131/#133) and Win32-only (#134) PRs don't apply to a
native Vulkan build.
The PRs that do apply to native static RT (device-local placement
#65/#72/#73/#75, staging release #66/#67/#114, pipeline cache #69,
deferred deletion #101/#116, barrier scoping #48/#115, heap-bind caching
#42/#43) act on setup cost and memory, not frame time — which is
exactly where the measured ~2–3 % setup change and the warm-cache RSS drop
appear. Steady-state frame time is GPU-traversal bound, so CPU savings are
hidden behind traversal.

Takeaway: the perf work is real but concentrated in UI/text and
dynamic-upload paths; a static UI-less RT scene is the wrong workload to see
most of it. Quantifying those would need a text-heavy UIRenderer benchmark
or an animated/deforming scene. Full analysis in
examples/SponzaBench/README.md.

Note (pre-existing, unrelated)

Stock examples/Sponza native no longer compiles against the unpinned
Vulkan-Headers main — VkResourceDescriptorDataEXT dropped the
pCombinedImageSampler member it uses. SponzaBench avoids it; porting
Sponza to the split sampled-image + sampler-heap model is a good follow-up.

Tests

crafter-build test → 25 passed (run with --jobs=1; the default
parallel run hits a pre-existing race in the external-dep git pull, not
related to this change).

Resolves #155. ## What Issue #155 asked for an **actual measurement** of the net performance gain from #40 to current master, in a representative scene (Sponza). This adds a headless benchmark harness (`examples/SponzaBench`) and reports the measured result. SponzaBench is the native Vulkan Sponza RT scene, but it times setup and a measured `Render()` loop and prints `BENCH …` lines instead of opening an interactive window. Its closest-hit shades from barycentrics (samples no texture), so unlike the interactive Sponza it builds the **full 25-mesh atrium (~262 k triangles)** as one multi-instance TLAS — the dynamic-heap hit-shader driver fault that forces single-material on native doesn't apply. A second small change adds `CRAFTER_PRESENT_IMMEDIATE` to `Window` so frame pacing can be uncapped from the compositor's vblank for the measurement (FIFO stays the default). ## Method Built the *same* SponzaBench sources against each library revision (#40 = `1451e3a`, HEAD = master, via a git worktree). RTX 4090, driver `610.43.02`, 1280×720, validation disabled, `IMMEDIATE` present, 200 warmup + 2000 measured frames, **median of 9 runs**. ## Result — the net gain in Sponza is small (~1–2 % frames, ~2–3 % setup) Full atrium (GPU-traversal bound): | metric | #40 | HEAD | Δ | |---|---:|---:|---:| | setup (cold) ms | 322.7 | 316.6 | −1.9 % | | setup (warm cache #69) ms | 322.7* | 313.9 | −2.7 % | | fps | 7 560 | 7 637 | +1.0 % | | frame p50 ms | 0.1311 | 0.1298 | −1.0 % | | peak RSS warm MB | 336.0 | 332.0 | −1.2 % | Light scene (CPU-bound): fps 15 022 → 15 237 (+1.4 %), frame p50 0.0662 → 0.0650 ms (−1.8 %). *#40 predates the disk pipeline cache (#69), so its setup is always cold. ## Why so small (the honest part) This is a real result, not a flat run — it's about *what Sponza exercises*: - The biggest block of post-#40 PRs is **UI / text** (#46–57, #61, #122–129, #132). Sponza has no UI → zero contribution. - The next block optimises **per-frame dynamic uploads** (#118 TLAS dirty-track, #119 mesh refit, #120 staging ring). A **static** scene builds its TLAS once and never re-uploads → not exercised in steady state. - WebGPU-only (#130/#131/#133) and Win32-only (#134) PRs don't apply to a native Vulkan build. - The PRs that *do* apply to native static RT (device-local placement #65/#72/#73/#75, staging release #66/#67/#114, pipeline cache #69, deferred deletion #101/#116, barrier scoping #48/#115, heap-bind caching #42/#43) act on **setup cost and memory**, not frame time — which is exactly where the measured ~2–3 % setup change and the warm-cache RSS drop appear. Steady-state frame time is GPU-traversal bound, so CPU savings are hidden behind traversal. **Takeaway:** the perf work is real but concentrated in UI/text and dynamic-upload paths; a static UI-less RT scene is the wrong workload to see most of it. Quantifying those would need a text-heavy `UIRenderer` benchmark or an animated/deforming scene. Full analysis in `examples/SponzaBench/README.md`. ## Note (pre-existing, unrelated) Stock `examples/Sponza` native no longer compiles against the unpinned Vulkan-Headers `main` — `VkResourceDescriptorDataEXT` dropped the `pCombinedImageSampler` member it uses. SponzaBench avoids it; porting Sponza to the split sampled-image + sampler-heap model is a good follow-up. ## Tests `crafter-build test` → **25 passed** (run with `--jobs=1`; the default parallel run hits a pre-existing race in the external-dep `git pull`, not related to this change).

catbot added 2 commits

2026-06-18 21:36:31 +02:00

feat(window): env-gated uncapped present mode for benchmarking 3e116e6e43

CRAFTER_PRESENT_IMMEDIATE selects IMMEDIATE (then MAILBOX) when the
surface offers it, instead of the default FIFO. Needed to measure
steady-state frame throughput without the compositor's vblank cap; FIFO
remains the default when the variable is unset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

test(bench): SponzaBench harness + #40→HEAD perf measurement (#155 ) 619e39369d

Headless benchmark around the native Sponza RT scene: times setup and a
measured Render() loop over the full multi-mesh atrium, prints BENCH
metrics, and exits. Includes run-bench.sh and a README documenting the
methodology and the measured net gain from #40 to current master.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

catbot merged commit 47bd4da0e3 into master

2026-06-18 21:36:37 +02:00

catbot deleted branch claude/issue-155

2026-06-18 21:36:37 +02:00

catbot referenced this pull request from a commit

2026-06-18 21:36:38 +02:00

Merge pull request 'test(bench): measure net perf gain from #40 to master in Sponza (#155)' (#156) from claude/issue-155 into master