Catcrafts/Crafter.Graphics
Watch
2
0
Fork
You've already forked Crafter.Graphics
 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

feat: windows on their own clock, deferred deletion for every handle owner, staged re-uploads, and RTPass push data

Browse source 
Window
- updateInterval runs Update on the window's own clock while the
  compositor sends no frame callbacks (hidden, minimized): the Wayland loop
  is a poll() loop, and the Win32 loop sleeps until the next update. Both
  read the interval each pass.
- SetPresentMode (FIFO, Mailbox, Immediate). A mode the surface lacks, or
  whose swapchain has other than numFrames images (Mesa's Mailbox has four),
  falls back to FIFO until the mode is set again.
- AddPollFd / RemovePollFd wake the Wayland loop on the app's descriptors.
- The swapchain has TRANSFER_DST where the surface allows it.
- CaptureNextFrame replaces SaveFrame: it records the copy in the frame,
  before the present, where SaveFrame read an image already presented. A
  frame unwound past before the write still frees the readback buffer.
- Inter-pass barriers on the swapchain image carry the access masks of the
  stages the passes use.

Deferred deletion (issue #101)
- VulkanBuffer, Mesh, VulkanShader, ComputeShader and PipelineRTVulkan hand
  their handles to the deletion queue from their destructors and moves, as
  a frame in flight may still use them; a static owner goes after the
  driver has torn down. VulkanShader and PipelineRTVulkan move and can't be
  copied; re-Load and re-Init queue the old pipeline.
- The queue and its mutex are never destroyed, and the queue takes handles
  from any thread; Device::frameCounter is atomic. Both the queue and the
  staging rings count frames as one window renders them (documented).
- DrainDeletions also lets every buffer's staging ring start over: without
  a Window nothing else reclaims either, so an app without one calls it
  after waiting for its work.

Meshes and buffers
- A Mesh rebuild queues the old acceleration structure and builds into the
  same storage after a barrier, so the BLAS address stays put when it fits.
  The refit and rebuild barriers cover compute ray queries too.
- RTBuildOptions::noDuplicateAnyHit. Mesh::geometryFlags replaces opaque,
  and the refit and rebuild fallbacks keep the flags.
- A re-upload into a buffer a frame in flight may read (UploadDeviceLocal
  of the same size, UploadDeviceLocalRange) copies on the GPU, after those
  reads, through a per-frame staging ring; on ReBAR it used to write the
  buffer in place. Empty uploads record nothing.
- A compressed Mesh::Build that keeps its buffers decompresses into them
  after the reads of frames in flight, and the hit shaders and ray queries
  wait for the decompress as well as the BLAS build. Its vertex buffer
  takes a ranged refit's copies.
- Mesh move-assigns (queueing the BLAS it replaces), so it can be replaced
  in place or erased from a vector.
- VulkanBuffer::Create asks for a device address only for buffers with
  SHADER_DEVICE_ADDRESS usage, as vkGetBufferDeviceAddress requires; the
  others have address 0. UIRenderer::RegisterBuffer refuses those.
- Build inputs are synchronized as shader reads.

Other
- RTPass pushes the app's own push data and traces at an explicit size.
- DescriptorHeapVulkan hands freed slot ranges out again (they leaked), and
  keeps its free lists sorted.
- ImageVulkan's mip chain clamps each level to 1 texel: a non-square
  image's shorter side went to 0.
- VulkanShader throws on a missing file instead of failing in bad_alloc.
- CRAFTER_GRAPHICS_DEVICE picks the GPU by its number among the distinct
  GPUs Vulkan lists; anything but such a number is an error.
- The debug messenger prints info messages only with
  CRAFTER_GRAPHICS_VALIDATION set.
- Windows: the WinRT gamepad code moved to a C file (mingw-w64's WinRT
  headers don't compile as C++), -lcombase dropped, a reference leak fixed.
- project.cpp honours --local for Crafter.Event, Math and Asset, as Asset
  does for Math, and builds every test through one helper, projectArgs
  included.
- Tests: DebugMessengerSeverity, DeletionQueueThreads, DeviceSelection,
  HeapSlotReuse, MeshLifetime, MipChainNonSquare, PipelineLifetime,
  RTPassPushData, SwapchainTransferDst, UploadInFlight, VulkanBufferAddress,
  VulkanShaderLifetime, WindowUpdateClock; others extended.

Breaking: SaveFrame is CaptureNextFrame; Mesh::opaque is geometryFlags; a
buffer without SHADER_DEVICE_ADDRESS usage has no address; frameCounter is
std::atomic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jorijn van der Graaf 2026-10-20 19:07:54 +02:00
commit 079b6b0e1f
Signed by: jorijnvdgraaf
GPG key ID: 2937E59CDCC1BCFB
47 changed files with 3572 additions and 1430 deletions
Download patch file Download diff file Expand all files Collapse all files

338
interfaces/Crafter.Graphics-VulkanBuffer.cppm
View file

@ -109,11 +109,17 @@ namespace Crafter {
.memoryTypeIndex = memoryTypeIndex
};
// Only a buffer created for SHADER_DEVICE_ADDRESS may have its
// address queried (VUID-VkBufferDeviceAddressInfo-buffer-02601);
// the others keep address 0.
const bool addressable = (usageFlags & VK_BUFFER_USAGE_2_SHADER_DEVICE_ADDRESS_BIT) != 0;
VkMemoryAllocateFlagsInfoKHR allocFlagsInfo {
.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO_KHR,
.flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT_KHR,
};
memAlloc.pNext = &allocFlagsInfo;
if (addressable) {
memAlloc.pNext = &allocFlagsInfo;
}
Device::CheckVkResult(vkAllocateMemory(Device::device, &memAlloc, nullptr, &memory));
Device::CheckVkResult(vkBindBufferMemory(Device::device, buffer, memory, 0));
@ -122,14 +128,12 @@ namespace Crafter {
.sType = VK_STRUCTURE_TYPE_BUFFER_DEVICE_ADDRESS_INFO,
.buffer = buffer
};
address = vkGetBufferDeviceAddress(Device::device, &addressInfo);
address = addressable ? vkGetBufferDeviceAddress(Device::device, &addressInfo) : 0;
// Record the allocation's byte size (≥ `size`) for every buffer,
// not just mapped ones: UploadDeviceLocalRange's direct path flushes
// only the dirty sub-range and clamps its rounded-up upper bound to
// this (the nonCoherentAtomSize-aligned-end exception, see
// AlignMappedFlushRange). A non-mapped buffer never maps persistently,
// but its memory is still the allocation a transient map writes into.
// Record the allocation's byte size (≥ `size`): a ranged
// FlushDevice flushes only the dirty sub-range and clamps its
// rounded-up upper bound to this (the nonCoherentAtomSize-aligned-end
// exception, see AlignMappedFlushRange).
mappedSize = memReqs.size;
if constexpr(Mapped) {
Device::CheckVkResult(vkMapMemory(Device::device, memory, 0, memReqs.size, 0, reinterpret_cast<void**>(&(VulkanBufferMappedConditional<T, true>::value))));
@ -227,97 +231,62 @@ namespace Crafter {
// Upload `count` host elements from `src` into this buffer as
// device-local, GPU-read geometry, choosing placement at runtime via
// the #89 upload strategy (Device::PreferDirectDeviceWrite) and then
// recording a barrier from the upload to (dstStageMask, dstAccessMask)
// on `cmd`:
// the #89 upload strategy (Device::PreferDirectDeviceWrite), and order
// the upload before (dstStageMask, dstAccessMask) on `cmd`:
// ReBAR / UMA (direct) — allocate HOST_VISIBLE with DEVICE_LOCAL as a
// best-effort preference (GetMemoryType lands the host-visible
// device-local type that the strategy already proved exists), map
// transiently, memcpy, flush if the chosen type isn't coherent,
// unmap. No staging buffer: it would be pure overhead on a bar where
// the device-local heap is itself host-writable.
// No / small BAR (staged) — allocate pure DEVICE_LOCAL (+ TRANSFER_DST),
// fill a transient HOST_VISIBLE staging buffer, vkCmdCopyBuffer into
// this buffer, then hand the staging buffer to the fence-keyed
// deferred-deletion queue (#101/#102) so it outlives the copy submit.
// Requires !Mapped: the destination must be free to be device-local-only
// (a persistent map would force HOST_VISIBLE), so the direct path maps
// just long enough to write. A same-size re-upload reuses the allocation
// (Resize), so the device address stays stable across an in-place AS
// UPDATE refit that re-calls this.
// No / small BAR (staged) — allocate pure DEVICE_LOCAL and copy from
// the staging ring on the GPU (CopyThroughStaging).
// The direct write is only for a fresh allocation, which nothing can be
// reading yet. A same-size re-upload reuses the allocation (Resize), so
// the device address stays stable across an in-place AS UPDATE refit
// that re-calls this, but a frame in flight may still read it: that
// upload always copies, on the GPU timeline, after those reads.
// dstStageMask must name every stage that reads the buffer, since the
// next upload waits for them. Requires !Mapped: the destination must be
// free to be device-local-only (a persistent map would force
// HOST_VISIBLE), so the direct path maps just long enough to write.
void UploadDeviceLocal(VkBufferUsageFlags2 usageFlags, const T* src, std::uint32_t count, VkCommandBuffer cmd, VkAccessFlags dstAccessMask, VkPipelineStageFlags dstStageMask) requires(!Mapped) {
VkDeviceSize bytes = static_cast<VkDeviceSize>(count) * sizeof(T);
VkAccessFlags srcAccessMask;
VkPipelineStageFlags srcStageMask;
if (Device::PreferDirectDeviceWrite(bytes)) {
// Nothing to put anywhere: no buffer of no bytes, no copy of none.
if (count == 0) return;
const VkDeviceSize bytes = static_cast<VkDeviceSize>(count) * sizeof(T);
// Any upload may be followed by one that copies in.
usageFlags |= VK_BUFFER_USAGE_TRANSFER_DST_BIT;
const bool direct = Device::PreferDirectDeviceWrite(bytes);
const VkBuffer previous = buffer;
if (direct) {
Resize(usageFlags, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT, count, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT);
void* mapped = nullptr;
Device::CheckVkResult(vkMapMemory(Device::device, memory, 0, VK_WHOLE_SIZE, 0, &mapped));
std::memcpy(mapped, src, bytes);
// Match FlushDevice()'s gate: a non-coherent type needs an
// explicit flush before the device reads the written range.
if (!(memoryPropertyFlagsChosen & VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)) {
VkMappedMemoryRange range {
.sType = VK_STRUCTURE_TYPE_MAPPED_MEMORY_RANGE,
.memory = memory,
.offset = 0,
.size = VK_WHOLE_SIZE
};
vkFlushMappedMemoryRanges(Device::device, 1, &range);
}
vkUnmapMemory(Device::device, memory);
srcAccessMask = VK_ACCESS_HOST_WRITE_BIT;
srcStageMask = VK_PIPELINE_STAGE_HOST_BIT;
} else {
Resize(usageFlags | VK_BUFFER_USAGE_TRANSFER_DST_BIT, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT, count);
// Persistent staging instead of a Create+DeferredClear per call
// (issue #120): on no-/small-BAR hardware Mesh::Refit /
// RecordProceduralBuild hit this branch every frame for deforming
// meshes, so a fresh alloc+map+free cycle per upload was pure
// churn on exactly the hardware this path targets. Keep a small
// ring of staging buffers, each grown to a high-water mark and
// reused — reallocated (by Resize, which defers the outgrown
// allocation) only when a larger upload arrives.
//
// The ring is grown lazily, on first entry into this staged
// branch, so ReBAR/UMA hardware (which always takes the direct
// `if` branch above) never constructs a staging allocation it
// does not use.
//
// Why a ring and not one shared buffer: the vkCmdCopyBuffer below
// still reads the staging buffer after this call returns — the
// reason the old code used DeferredClear. With frames pipelined
// framesInFlight deep, overwriting one shared staging buffer next
// frame would clobber data the previous frame's copy is still
// reading. Indexing by frameCounter % framesInFlight gives each
// in-flight frame its own slot; a slot is rewritten only after
// framesInFlight frames have elapsed — the exact window after
// which single-queue submission order (the same guarantee the
// #101 deletion queue relies on) ensures that copy has completed.
const std::uint32_t ringSize =
Device::framesInFlight ? Device::framesInFlight : 1;
if (stagingRing.size() < ringSize) {
stagingRing.resize(ringSize);
}
VulkanBuffer<T, true>& staging =
stagingRing[Device::frameCounter % ringSize];
// SHADER_DEVICE_ADDRESS: Create always queries the buffer device
// address (and allocates with the device-address bit); the
// staging buffer's own address is otherwise unused. Resize reuses
// the existing allocation (and its mapped pointer) when the
// upload still fits the slot's high-water capacity.
staging.Resize(VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT, count);
std::memcpy(staging.value, src, bytes);
staging.FlushDevice();
VkBufferCopy region { .srcOffset = 0, .dstOffset = 0, .size = bytes };
vkCmdCopyBuffer(cmd, staging.buffer, buffer, 1, &region);
srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT;
srcStageMask = VK_PIPELINE_STAGE_TRANSFER_BIT;
Resize(usageFlags, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT, count);
}
if (!direct || buffer == previous) {
CopyThroughStaging(src, 0, count, cmd, dstAccessMask, dstStageMask);
return;
}
void* mapped = nullptr;
Device::CheckVkResult(vkMapMemory(Device::device, memory, 0, VK_WHOLE_SIZE, 0, &mapped));
std::memcpy(mapped, src, bytes);
// Match FlushDevice()'s gate: a non-coherent type needs an
// explicit flush before the device reads the written range.
if (!(memoryPropertyFlagsChosen & VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)) {
VkMappedMemoryRange range {
.sType = VK_STRUCTURE_TYPE_MAPPED_MEMORY_RANGE,
.memory = memory,
.offset = 0,
.size = VK_WHOLE_SIZE
};
vkFlushMappedMemoryRanges(Device::device, 1, &range);
}
vkUnmapMemory(Device::device, memory);
VkBufferMemoryBarrier barrier = {
.sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER,
.srcAccessMask = srcAccessMask,
.srcAccessMask = VK_ACCESS_HOST_WRITE_BIT,
.dstAccessMask = dstAccessMask,
.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
@ -325,79 +294,55 @@ namespace Crafter {
.offset = 0,
.size = VK_WHOLE_SIZE
};
vkCmdPipelineBarrier(cmd, srcStageMask, dstStageMask, 0, 0, NULL, 1, &barrier, 0, NULL);
vkCmdPipelineBarrier(cmd, VK_PIPELINE_STAGE_HOST_BIT, dstStageMask, 0, 0, NULL, 1, &barrier, 0, NULL);
}
// Re-upload only the half-open sub-range [offset, offset+count) of an
// already-allocated device-local buffer — the dirty-range counterpart of
// Re-upload only the half-open sub-range [offset, offset+count) of a
// buffer UploadDeviceLocal made — the dirty-range counterpart of
// UploadDeviceLocal. `src` points at the first changed element (not the
// start of the whole array); `offset` is that element's index in this
// buffer. The buffer must already exist at its full size (a prior
// UploadDeviceLocal / Build sized it): this never Resizes, so the device
// address stays stable and the untouched elements keep their last
// contents. Only the dirty bytes are written + flushed (direct path) or
// staged + copied (staged path), and the post-upload barrier covers only
// that sub-range — so a deforming-mesh refit that nudges a few vertices
// pays for those vertices, not a full-array re-upload + flush + copy (#119).
//
// The direct-vs-staged choice is read from the memory type the buffer was
// actually allocated with (HOST_VISIBLE → map + write in place;
// device-local-only → stage + GPU copy), NOT from the sub-range size: the
// allocation is fixed, so a small dirty range must not be mis-routed to a
// map of a non-host-visible buffer the way UploadDeviceLocal's size-based
// PreferDirectDeviceWrite check would. The staged path's GPU copy needs
// VK_BUFFER_USAGE_TRANSFER_DST_BIT on the destination — already present,
// since UploadDeviceLocal only allocates device-local-only memory on the
// staged path, where it adds that bit.
// buffer. This never Resizes, so the device address stays stable and the
// untouched elements keep their last contents. Only the dirty elements
// are staged and copied, and the barriers cover only that sub-range — so
// a deforming-mesh refit that nudges a few vertices pays for those
// vertices, not a full-array re-upload (#119). The buffer is one a frame
// in flight may still read, so the copy waits for its readers
// (dstStageMask, as in UploadDeviceLocal) on the GPU, whatever memory
// type it lives in.
void UploadDeviceLocalRange(const T* src, std::uint32_t offset, std::uint32_t count, VkCommandBuffer cmd, VkAccessFlags dstAccessMask, VkPipelineStageFlags dstStageMask) requires(!Mapped) {
VkDeviceSize byteOffset = static_cast<VkDeviceSize>(offset) * sizeof(T);
VkDeviceSize bytes = static_cast<VkDeviceSize>(count) * sizeof(T);
VkAccessFlags srcAccessMask;
VkPipelineStageFlags srcStageMask;
if (memoryPropertyFlagsChosen & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) {
void* mapped = nullptr;
Device::CheckVkResult(vkMapMemory(Device::device, memory, 0, VK_WHOLE_SIZE, 0, &mapped));
std::memcpy(static_cast<std::byte*>(mapped) + byteOffset, src, bytes);
// Non-coherent memory needs an explicit flush — but only of the
// sub-range actually written, rounded outward to
// nonCoherentAtomSize (and clamped to the allocation size).
if (!(memoryPropertyFlagsChosen & VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)) {
MappedFlushRange r = AlignMappedFlushRange(
byteOffset, bytes, Device::nonCoherentAtomSize, mappedSize);
VkMappedMemoryRange range {
.sType = VK_STRUCTURE_TYPE_MAPPED_MEMORY_RANGE,
.memory = memory,
.offset = r.offset,
.size = r.size
};
vkFlushMappedMemoryRanges(Device::device, 1, &range);
}
vkUnmapMemory(Device::device, memory);
srcAccessMask = VK_ACCESS_HOST_WRITE_BIT;
srcStageMask = VK_PIPELINE_STAGE_HOST_BIT;
} else {
// Device-local-only: stage just the dirty elements and copy them
// into place at byteOffset. The staging buffer outlives the queued
// copy via the fence-keyed deletion queue (#101/#102), exactly as
// the full-buffer staged path does.
VulkanBuffer<T, true> staging;
staging.Create(VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT, count);
std::memcpy(staging.value, src, bytes);
staging.FlushDevice();
VkBufferCopy region { .srcOffset = 0, .dstOffset = byteOffset, .size = bytes };
vkCmdCopyBuffer(cmd, staging.buffer, buffer, 1, &region);
staging.DeferredClear();
srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT;
srcStageMask = VK_PIPELINE_STAGE_TRANSFER_BIT;
}
CopyThroughStaging(src, offset, count, cmd, dstAccessMask, dstStageMask);
}
// Order only the written sub-range before the consumer reads it. The
// untouched bytes were made visible by their own prior upload's
// barrier and are not written in this submit, so they need no fresh
// dependency even though the build reads the whole buffer.
VkBufferMemoryBarrier barrier = {
// Copy `count` elements from `src` to element `offset` of this buffer on
// the GPU timeline: after the reads by `dstStageMask` recorded earlier
// or still in flight, and before the ones that follow. The bytes go
// through this frame's slot of the staging ring first.
void CopyThroughStaging(const T* src, std::uint32_t offset, std::uint32_t count, VkCommandBuffer cmd, VkAccessFlags dstAccessMask, VkPipelineStageFlags dstStageMask) requires(!Mapped) {
// A copy or barrier of no bytes is invalid usage.
if (count == 0) return;
const VkDeviceSize byteOffset = static_cast<VkDeviceSize>(offset) * sizeof(T);
const VkDeviceSize bytes = static_cast<VkDeviceSize>(count) * sizeof(T);
// Write after read: an execution dependency on the earlier readers.
VkBufferMemoryBarrier readsDone = {
.sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER,
.srcAccessMask = srcAccessMask,
.srcAccessMask = 0,
.dstAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT,
.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
.buffer = buffer,
.offset = byteOffset,
.size = bytes
};
vkCmdPipelineBarrier(cmd, dstStageMask, VK_PIPELINE_STAGE_TRANSFER_BIT, 0, 0, NULL, 1, &readsDone, 0, NULL);
const auto [staging, stagingOffset] = Stage(src, count);
VkBufferCopy region { .srcOffset = stagingOffset, .dstOffset = byteOffset, .size = bytes };
vkCmdCopyBuffer(cmd, staging, buffer, 1, &region);
VkBufferMemoryBarrier copied = {
.sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER,
.srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT,
.dstAccessMask = dstAccessMask,
.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
@ -405,7 +350,47 @@ namespace Crafter {
.offset = byteOffset,
.size = bytes
};
vkCmdPipelineBarrier(cmd, srcStageMask, dstStageMask, 0, 0, NULL, 1, &barrier, 0, NULL);
vkCmdPipelineBarrier(cmd, VK_PIPELINE_STAGE_TRANSFER_BIT, dstStageMask, 0, 0, NULL, 1, &copied, 0, NULL);
}
// Put `count` elements in this frame's slot of the staging ring, after
// whatever this frame staged before, and say where: the slot's buffer
// and the byte offset. A copy still in flight never reads the slot, as
// a slot is used again only framesInFlight frames later — the window
// after which single-queue submission order (the same guarantee the
// #101 deletion queue relies on) ensures that copy has completed. Each
// slot keeps its high-water size (issue #120); outgrowing it mid-frame
// hands the old allocation, with this frame's earlier bytes, to the
// deletion queue. Without a window no frame passes; after a
// Device::DrainDeletions (which follows a wait for the queue) nothing
// reads any slot, so they all start over.
std::pair<VkBuffer, VkDeviceSize> Stage(const T* src, std::uint32_t count) requires(!Mapped) {
const std::uint32_t ringSize = Device::framesInFlight ? Device::framesInFlight : 1;
if (stagingRing.size() < ringSize) {
stagingRing.resize(ringSize);
stagingFrame.resize(ringSize, std::numeric_limits<std::uint64_t>::max());
stagingUsed.resize(ringSize, 0);
}
if (stagingDrains != Device::drains) {
stagingDrains = Device::drains;
std::ranges::fill(stagingUsed, 0);
}
const std::size_t slot = Device::frameCounter % ringSize;
if (stagingFrame[slot] != Device::frameCounter) {
stagingFrame[slot] = Device::frameCounter;
stagingUsed[slot] = 0;
}
VulkanBuffer<T, true>& staging = stagingRing[slot];
const std::uint32_t needed = stagingUsed[slot] + count;
if (staging.buffer == VK_NULL_HANDLE || static_cast<VkDeviceSize>(needed) * sizeof(T) > staging.capacity) {
staging.Resize(VK_BUFFER_USAGE_TRANSFER_SRC_BIT, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT, needed);
stagingUsed[slot] = 0;
}
const std::uint32_t first = stagingUsed[slot];
std::memcpy(staging.value + first, src, static_cast<std::size_t>(count) * sizeof(T));
staging.FlushDevice(static_cast<VkDeviceSize>(first) * sizeof(T), static_cast<VkDeviceSize>(count) * sizeof(T));
stagingUsed[slot] = first + count;
return { staging.buffer, static_cast<VkDeviceSize>(first) * sizeof(T) };
}
void FlushDevice() requires(Mapped) {
@ -516,18 +501,31 @@ namespace Crafter {
vkInvalidateMappedMemoryRanges(Device::device, 1, &range);
}
// Persistent per-frame-in-flight staging ring for the staged
// UploadDeviceLocal path (issue #120). Empty — and so holding zero
// host-visible allocations — until the first upload that actually
// stages, which only happens on no-/small-BAR hardware (ReBAR/UMA always
// takes the direct-write branch and never touches this, so the lazy ring
// keeps the change a no-op there). Sized to Device::framesInFlight so
// each in-flight frame copies from its own slot; each slot grows to a
// high-water mark via Resize and is reused across frames rather than
// reallocated every call — mirroring the TLAS instance/metadata reuse.
// Persistent per-frame-in-flight staging ring for the copies Stage
// records (issue #120). Empty — and so holding zero host-visible
// allocations — until the first upload that stages: on ReBAR/UMA only
// re-uploads do, as a first upload writes directly. Sized to
// Device::framesInFlight so each in-flight frame copies from its own
// slot; each slot grows to a high-water mark via Resize and is reused
// across frames rather than reallocated every call — mirroring the TLAS
// instance/metadata reuse. stagingFrame and stagingUsed say, per slot,
// which frame last staged into it and how many elements it has staged;
// stagingDrains, the Device::drains it last saw.
std::vector<VulkanBuffer<T, true>> stagingRing;
std::vector<std::uint64_t> stagingFrame;
std::vector<std::uint32_t> stagingUsed;
std::uint64_t stagingDrains = 0;
VulkanBuffer(VulkanBuffer&& other) {
VulkanBuffer(VulkanBuffer&& other) noexcept {
*this = std::move(other);
}
// Takes `other`'s buffer over. The one this held goes to the deletion
// queue, as the destructor would send it: a frame in flight may still
// read it.
VulkanBuffer& operator=(VulkanBuffer&& other) noexcept {
if (this == &other) return *this;
DeferredClear();
buffer = other.buffer;
memory = other.memory;
size = other.size;
@ -538,15 +536,21 @@ namespace Crafter {
other.buffer = VK_NULL_HANDLE;
address = other.address;
stagingRing = std::move(other.stagingRing);
stagingFrame = std::move(other.stagingFrame);
stagingUsed = std::move(other.stagingUsed);
stagingDrains = other.stagingDrains;
if constexpr(Mapped) {
VulkanBufferMappedConditional<T, true>::value = other.VulkanBufferMappedConditional<T, true>::value;
}
};
return *this;
}
// Deferred, like DeferredClear: the owner may go while a frame in
// flight still reads the buffer, and a static owner goes after the
// driver has torn down. Call Clear() first to free at once when the
// GPU is known to be idle. The staging ring's buffers defer the same way.
~VulkanBuffer() {
if(buffer != VK_NULL_HANDLE) {
Clear();
}
DeferredClear();
}
VulkanBuffer(VulkanBuffer&) = delete;