Crafter.Graphics/examples/SponzaBench/README.md

SponzaBench — measuring the post-#40 performance work

This example exists to answer issue #155: starting with #40 a lot of performance-related issues were merged — what is the net performance gain, measured in a representative scene (Sponza)?

It is a headless benchmark harness built around the native Vulkan Sponza ray-tracing scene. Same asset bundle and camera as examples/Sponza, but instead of opening an interactive window it times the work and prints machine-readable BENCH … lines, then exits.

What it measures

• setup — process start through the first command submission: asset decompression, BLAS build per mesh group, the multi-instance TLAS, the RT pipeline, GPU memory placement and the descriptor writes. This is the window most of the post-#40 native perf work acts on.
• frames — a warmup followed by a measured loop calling Window::Render(), reporting per-frame wall-clock stats and throughput.

Unlike the interactive Sponza example — which is single-material on native because of the hit-shader dynamic-descriptor_heap-index driver fault, fixed in NVIDIA 615.78.08 (see examples/Sponza/README.md) — SponzaBench's closest-hit shades from barycentric coordinates and samples no texture, so it can build the full multi-mesh atrium (25 mesh groups, ~262 k triangles) as one multi-instance TLAS. The albedo is still decompressed and uploaded during setup (to keep that path in the measurement) but is not bound.

Running

cd examples/SponzaBench
crafter-build                       # native Vulkan
# from the produced bin dir:
VK_LOADER_LAYERS_DISABLE='~all~' \
CRAFTER_PRESENT_IMMEDIATE=1 \
BENCH_WARMUP=200 BENCH_FRAMES=2000 BENCH_MESHES=25 ./SponzaBench

run-bench.sh <bindir> <label> <meshes> <reps> [--cold] runs it N times and reports the median / min / max of each metric.

Environment knobs:

var	effect
CRAFTER_PRESENT_IMMEDIATE=1	uncapped present mode — without it FIFO pins frame time to the compositor's vblank (~60 Hz) and steady-state throughput can't be seen.
VK_LOADER_LAYERS_DISABLE='~all~'	disable the validation layer. The engine enables Khronos validation and GPU-assisted validation unconditionally; that adds large, version-dependent overhead and must be off for a representative measurement.
BENCH_WARMUP / BENCH_FRAMES	warmup and measured frame counts (default 200 / 2000).
BENCH_MESHES	cap on mesh groups loaded (default: all 25). A small cap shrinks GPU work so per-frame CPU cost dominates; the full scene is GPU-traversal bound.
BENCH_CLEAN_EXIT=1	exit via std::exit (runs atexit, which writes the #69 pipeline cache) instead of the default hard _Exit. Used once to seed pipeline_cache.bin for a warm-cache measurement.

Results — #40 vs current master

Measured on this repo's CI box: NVIDIA RTX 4090, driver 610.43.02, 1280×720, validation disabled, IMMEDIATE present, 200 warmup + 2000 measured frames, median of 9 runs. "#40" is commit 1451e3a (the #40 merge); "HEAD" is current master. The same SponzaBench sources were built against each library revision (a git worktree at #40).

Full atrium — 25 meshes / ~262 k triangles (GPU-traversal bound)

metric	#40	HEAD	Δ
setup (cold) ms	322.7	316.6	−1.9 %
setup (warm pipeline cache, #69) ms	322.7¹	313.9	−2.7 %
throughput fps	7 560	7 637	+1.0 %
frame time p50 ms	0.1311	0.1298	−1.0 %
peak host RSS (cold) MB	336.0	344.2	+2.4 %
peak host RSS (warm cache) MB	336.0¹	332.0	−1.2 %

¹ #40 predates the disk pipeline cache (#69), so its setup is always the cold-compile path.

Light scene — 1 mesh (deliberately CPU-bound)

metric	#40	HEAD	Δ
throughput fps	15 022	15 237	+1.4 %
frame time p50 ms	0.0662	0.0650	−1.8 %

Interpretation

The net measured gain in a static Sponza RT scene is small: ~1–2 % faster frames, ~2–3 % faster setup, and roughly flat host memory. That is an honest result, and the reason is what Sponza exercises, not that the perf work was ineffective:

• Most post-#40 PRs don't touch this workload. The largest block is UI / text rendering (shaped-run cache, font-atlas dirty uploads, the UI compute-shader rewrites — #46–#57, #61, #122–#129, #132). Sponza has no UI, so those contribute zero. A second block optimises per-frame dynamic uploads (TLAS dirty-tracking #118, deforming-mesh refit #119, the staging ring #120). A static scene builds its TLAS once and never re-uploads, so these don't fire in steady state either. WebGPU-only (#130/#131/#133) and Win32-only (#134) PRs don't apply to a native Vulkan build at all.
• The PRs that do apply act on setup and memory, not frame time. Device-local placement (#65/#72/#73/#75), staging release (#66/#67/#114), the pipeline cache (#69) and the deferred-deletion queue (#101/#116) move setup cost and peak memory — which is exactly where the measured ~2–3 % setup change and the warm-cache RSS drop show up. The pipeline cache itself saves ~3 ms here (one RT pipeline) and, more visibly, ~12 MB of peak RSS by skipping the cold shader-compile allocations.
• Steady-state frame time is GPU-traversal bound. With 262 k triangles the per-frame CPU work (barrier scoping #48/#115, cached heap-bind structs #42/#43) is hidden behind GPU traversal, so it can't move the frame time. Shrinking the scene until it is CPU-bound (the light scene above) surfaces the per-frame CPU savings — and even then they are only ~1.8 %, because that CPU path was already cheap.

Takeaway: the post-#40 work is real but concentrated in UI/text and per-frame dynamic-upload paths; a static, UI-less RT scene is the wrong workload to see most of it. To quantify the UI/text gains a separate benchmark over a text-heavy UIRenderer scene (or an animated/deforming scene for the dynamic-upload PRs) would be needed.

Caveats / notes

• The stock examples/Sponza native path does not currently compile against the (unpinned) Vulkan-Headers main: the VkResourceDescriptorDataEXT union no longer has the pCombinedImageSampler member the example uses. SponzaBench sidesteps this by not binding a combined image+sampler. (Pre-existing, unrelated to #155 — worth a follow-up to port Sponza to the split sampled-image + sampler-heap model the engine's own UI renderer already uses.)
• Numbers are CPU/GPU specific. Re-run run-bench.sh locally for your hardware; the deltas between revisions are the point, not the absolute figures.