fixed bug in VolumeToMesh.cuh (#2306)

Signed-off-by: Ken Museth <ken.museth@gmail.com>

.github/workflows/teachme.yml

@@ -0,0 +1,55 @@
+
+name: TEACHME
+
+# Compile-checks the NanoVDB TEACHME tutorial against the real headers. Runs
+# when EITHER the NanoVDB headers or the tutorial change, so an API change that
+# breaks a lesson code block is caught here (the whole point of the harness).
+# Compile-only: no GPU is required.
+on:
+  push:
+    branches:
+      - 'master'
+      - 'feature/**'
+      - 'pr/**'
+    paths:
+      - 'nanovdb/nanovdb/**'
+      - 'doc/nanovdb/TEACHME/**'
+      - '.github/workflows/teachme.yml'
+  pull_request:
+    branches:
+      - '**'
+    paths:
+      - 'nanovdb/nanovdb/**'
+      - 'doc/nanovdb/TEACHME/**'
+      - '.github/workflows/teachme.yml'
+  workflow_dispatch:
+
+# Allow subsequent pushes to the same PR or REF to cancel any previous jobs.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  teachme-docs-compile:
+    runs-on: ${{ (github.repository_owner == 'AcademySoftwareFoundation' && 'ubuntu-22.04-8c-32g-300h') || 'ubuntu-latest' }}
+    name: teachme-docs-compile
+    container:
+      image: aswf/ci-openvdb:2024-clang17.2
+    env:
+      # Compile-only target; matches the arch used by the other NanoVDB jobs.
+      CUDAARCHS: "80"
+      # Each TU compiles the kitchen-sink fixture (most of NanoVDB's heavy
+      # headers), so nvcc is memory-hungry. Cap parallel compiles to stay
+      # within the runner's RAM — unbounded -j OOMs the 8-core/32 GB runner.
+      TEACHME_JOBS: "4"
+    steps:
+      - uses: actions/checkout@v3
+      - name: setup_cuda_12
+        run: |
+          echo "/usr/local/cuda-12/bin" >> $GITHUB_PATH
+          echo "LD_LIBRARY_PATH=/usr/local/cuda-12/lib64:$LD_LIBRARY_PATH" >> $GITHUB_ENV
+      - name: run_harness
+        # Reference-checks every #include / prose header path, then compiles
+        # every tagged code block. The OpenVDB-dependent blocks are skipped
+        # (still reference-checked) when OpenVDB is not installed in the image.
+        run: bash doc/nanovdb/TEACHME/test/run.sh

CODEOWNERS

@@ -6,10 +6,10 @@
 
 # Admin/General
 /.github/                                                   @kmuseth @apradhana @jmlait @idclip @danrbailey @richhones
+/.github/workflows/nanovdb.yml                              @kmuseth @apradhana
 /ci/                                                        @kmuseth @apradhana @jmlait @idclip @danrbailey @richhones
 /cmake/                                                     @kmuseth @apradhana @jmlait @idclip @danrbailey @richhones
 /doc/                                                       @kmuseth @apradhana @jmlait @idclip @danrbailey @richhones
-/pendingchanges/                                            @kmuseth @apradhana @jmlait @idclip @danrbailey @richhones
 /tsc/                                                       @kmuseth @apradhana @jmlait @idclip @danrbailey @richhones
 CODEOWNERS                                                  @kmuseth @apradhana @jmlait @idclip @danrbailey @richhones
 LICENSE                                                     @kmuseth @apradhana @jmlait @idclip @danrbailey @richhones

doc/nanovdb/TEACHME/README.md

@@ -0,0 +1,67 @@
+# TEACHME
+
+This directory contains interactive lesson documents designed to be loaded by
+an LLM coding agent (Claude Code, Cursor, or similar) to teach a user NanoVDB
+interactively.
+
+The lesson teaches how to *use* NanoVDB to read, build, transform, and render
+sparse volumetric data on CPU and GPU. It is a single self-contained markdown
+file the agent reads at the start of a session.
+
+## Getting started
+
+The easiest way to use this lesson is with [Claude Code](https://docs.anthropic.com/en/docs/claude-code)
+(CLI or IDE extension) from the root of an OpenVDB checkout. The agent can
+read the lesson files *and* the NanoVDB source code, so it can verify API
+details and help you debug exercises in real time.
+
+**Example prompts:**
+
+To learn how to use NanoVDB:
+
+```
+Read doc/nanovdb/TEACHME/nanovdb_user_lesson.md and teach me how to use NanoVDB.
+```
+
+You can also give the agent context about your background so it can tailor
+the lesson:
+
+```
+Read doc/nanovdb/TEACHME/nanovdb_user_lesson.md and teach me NanoVDB. I have
+C++ and CUDA experience but I've never used OpenVDB or any sparse 3D data
+structure before.
+```
+
+Keep the matching cheatsheet (`*_cheatsheet.md`) open in your editor while
+working through exercises — it's a quick reference for the APIs and
+invariants covered in the lesson.
+
+## How it works
+
+The lesson is a self-contained markdown file that serves as both a
+curriculum and an instructor prompt. The LLM acts as an interactive
+instructor: teaching concepts module by module, quizzing the student, and
+adapting to their responses.
+
+The lesson includes:
+
+- Teacher instructions (persona, pacing, scope)
+- Module-by-module curriculum with embedded concepts and code examples
+- Quiz questions and an answer key
+- Exercises with progressive difficulty
+- A capstone project
+- A reference table at the end
+
+## Available lessons
+
+| Lesson | Cheat sheet | Covers |
+|---|---|---|
+| [nanovdb_user_lesson.md](nanovdb_user_lesson.md) | [nanovdb_user_cheatsheet.md](nanovdb_user_cheatsheet.md) | Reading `.nvdb` files, `ReadAccessor`, `NodeManager`, math + sampling, HDDA ray-march, GPU kernels with `Grid`, GPU topology builders (`PointsToGrid`, `MeshToGrid`, `DilateGrid`, etc.), `IndexGrid` + `VoxelBlockManager`, OpenVDB ↔ NanoVDB conversion |
+
+## Not covered
+
+- The OpenVDB side (the dynamic, mutable tree used at film/sim time) — this
+  lesson treats OpenVDB only at the conversion boundary.
+- Houdini / Maya integration — outside the scope.
+- Gaussian splatting and fVDB-specific topics — see the fVDB repo's own
+  TEACHME at <https://github.com/openvdb/fvdb-core/tree/main/docs/TEACHME>.

doc/nanovdb/TEACHME/capstone/.gitignore

@@ -0,0 +1,6 @@
+# Build + editor + render artifacts (not source)
+build/
+compile_commands.json
+.clangd
+*.ppm
+sphere.png

doc/nanovdb/TEACHME/capstone/CMakeLists.txt

@@ -0,0 +1,27 @@
+cmake_minimum_required(VERSION 3.20)
+project(nanovdb_capstone LANGUAGES CXX CUDA)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CUDA_STANDARD 17)
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)  # emit build/compile_commands.json
+if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
+  set(CMAKE_CUDA_ARCHITECTURES 89)  # RTX 6000 Ada; override with -DCMAKE_CUDA_ARCHITECTURES=...
+endif()
+
+# NanoVDB headers pull in CUDA lambdas / relaxed-constexpr paths.
+string(APPEND CMAKE_CUDA_FLAGS " --extended-lambda --expt-relaxed-constexpr")
+
+# This file is at doc/nanovdb/TEACHME/capstone/CMakeLists.txt; the NanoVDB
+# include root (the dir containing the nanovdb/ header subdir) is the repo's
+# `nanovdb/` subdir — repo root is four dirs up.
+get_filename_component(_teachme "${CMAKE_CURRENT_SOURCE_DIR}" DIRECTORY) # TEACHME
+get_filename_component(_docnv   "${_teachme}" DIRECTORY)                 # doc/nanovdb
+get_filename_component(_doc     "${_docnv}" DIRECTORY)                   # doc
+get_filename_component(REPO_ROOT "${_doc}" DIRECTORY)                    # repo root
+set(NANOVDB_INCLUDE_ROOT "${REPO_ROOT}/nanovdb")
+message(STATUS "NanoVDB include root: ${NANOVDB_INCLUDE_ROOT}")
+
+add_executable(raymarch raymarch.cu)
+target_include_directories(raymarch PRIVATE "${NANOVDB_INCLUDE_ROOT}")
+# Binary lands in the build dir (conventional). raymarch writes sphere.ppm to
+# the current working directory, so run it from wherever you want the output.

doc/nanovdb/TEACHME/capstone/ppm2png.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""Convert a binary P6 PPM to PNG using only the Python standard library
+(the container has no ImageMagick / PIL). Usage: python3 ppm2png.py in.ppm out.png"""
+import sys, struct, zlib, binascii
+
+def main(src, dst):
+    data = open(src, "rb").read()
+    i = 0
+    def tok(i):
+        n = len(data)
+        while i < n and data[i] in b" \t\n\r": i += 1
+        s = i
+        while i < n and data[i] not in b" \t\n\r": i += 1
+        if s == i: raise ValueError("truncated or non-binary PPM")
+        return data[s:i], i
+    magic, i = tok(i)
+    assert magic == b"P6", "not a binary PPM"
+    w, i = tok(i); h, i = tok(i); mx, i = tok(i)
+    i += 1                       # single whitespace after maxval
+    W, H = int(w), int(h)
+    px = data[i:i + W * H * 3]
+
+    def chunk(typ, payload):
+        return (struct.pack(">I", len(payload)) + typ + payload +
+                struct.pack(">I", binascii.crc32(typ + payload) & 0xffffffff))
+
+    raw = bytearray()
+    for y in range(H):
+        raw.append(0)            # filter: none
+        raw += px[y * W * 3:(y + 1) * W * 3]
+    png = (b"\x89PNG\r\n\x1a\n"
+           + chunk(b"IHDR", struct.pack(">IIBBBBB", W, H, 8, 2, 0, 0, 0))
+           + chunk(b"IDAT", zlib.compress(bytes(raw), 9))
+           + chunk(b"IEND", b""))
+    open(dst, "wb").write(png)
+    print(f"wrote {dst} ({W}x{H})")
+
+if __name__ == "__main__":
+    main(sys.argv[1], sys.argv[2])

doc/nanovdb/TEACHME/capstone/raymarch.cu

@@ -0,0 +1,111 @@
+// Capstone scaffold: GPU level-set ray-march renderer.
+//
+// The host plumbing is DONE (build SDF, upload to device, launch, read back,
+// write PPM) and the light direction + ambient level are handed to the kernel
+// as parameters. Your job is the kernel body — the per-pixel ray-march + shade.
+// As shipped it compiles and runs but only writes a placeholder gradient.
+//
+// Build:  cmake -S . -B build && cmake --build build
+// Run:    ./build/raymarch [ambient] [lx ly lz]   (writes sphere.ppm)
+// View:   python3 ppm2png.py sphere.ppm sphere.png   (then open sphere.png)
+#include <nanovdb/NanoVDB.h>
+#include <nanovdb/tools/CreatePrimitives.h>
+#include <nanovdb/cuda/DeviceBuffer.h>
+#include <nanovdb/cuda/GridHandle.cuh>
+#include <nanovdb/math/Ray.h>
+#include <nanovdb/math/HDDA.h>        // nanovdb::math::ZeroCrossing
+#include <nanovdb/math/Stencils.h>   // nanovdb::math::GradStencil
+
+#include <cstdio>
+#include <cstdlib>
+#include <cmath>
+#include <vector>
+
+using GridT = nanovdb::NanoGrid<float>;
+
+__global__ void render(const GridT* grid, unsigned char* img, int W, int H,
+                       nanovdb::Vec3f lightDir, float ambient)
+{
+    const int x = blockIdx.x * blockDim.x + threadIdx.x;
+    const int y = blockIdx.y * blockDim.y + threadIdx.y;
+    if (x >= W || y >= H) return;
+    const int pid = (y * W + x) * 3;
+
+    // ===================== YOUR WORK STARTS HERE =====================
+    // Placeholder so it compiles & runs: a horizontal grey gradient.
+    unsigned char shade = (unsigned char)(255.f * x / W);
+
+    // The aesthetic inputs are given to you as parameters: `lightDir` and
+    // `ambient` (set in main, overridable on the command line). You write the
+    // ray-march and the shading math. Signatures are in the cheat sheet.
+    //
+    //   1. Camera ray (WORLD space). Pinhole: an eye point and, per pixel, a
+    //      direction through the image plane. You decide the math — remember
+    //      aspect ratio and that image y grows downward. Wrap it in a Ray.
+    //
+    //   2. The SDF lives in INDEX space, so convert your world ray to index
+    //      space before you trace it. (Ray has a method for this.)
+    //
+    //   3. Find where the ray first crosses the surface (the SDF sign change).
+    //      Module 5 gave you a helper in math/HDDA.h that drives an HDDA and
+    //      reports the hit voxel, the value, and the ray parameter.
+    //
+    //   4. On a hit, the surface normal is the (normalized) SDF gradient at the
+    //      hit voxel. Module 5's gradient stencil computes it. (Watch its
+    //      template parameter.)
+    //
+    //   5. Shade: a Lambertian intensity from the normal and the provided
+    //      `lightDir`, lifted off the floor by `ambient`; write it as greyscale.
+    //      On a MISS, leave the background.
+    // ====================== YOUR WORK ENDS HERE ======================
+
+    img[pid] = shade; img[pid + 1] = shade; img[pid + 2] = shade;
+}
+
+int main(int argc, char** argv)
+{
+    // Shading parameters passed into the kernel. Defaults, overridable on the
+    // command line:  ./raymarch [ambient] [lx ly lz]
+    float ambient = 0.15f;
+    nanovdb::Vec3f lightDir(-0.577f, 0.577f, -0.577f);   // (-1,1,-1) normalized
+    if (argc >= 2) ambient = (float)std::atof(argv[1]);
+    if (argc >= 5) lightDir = nanovdb::Vec3f((float)std::atof(argv[2]),
+                                             (float)std::atof(argv[3]),
+                                             (float)std::atof(argv[4]));
+    lightDir.normalize();
+
+    // Build a level-set sphere SDF on the host (radius 100, voxel size 1).
+    auto handle = nanovdb::tools::createLevelSetSphere<float>(
+        /*radius=*/100.0, /*center=*/nanovdb::Vec3d(0.0),
+        /*voxelSize=*/1.0, /*halfWidth=*/3.0);
+
+    // Move to the device. copy<DeviceBuffer>() fills the HOST side of the dual
+    // buffer; deviceUpload() pushes it to the GPU — skip it and deviceGrid()
+    // returns nullptr.
+    auto devHandle = handle.copy<nanovdb::cuda::DeviceBuffer>();
+    devHandle.deviceUpload();
+    const GridT* dGrid = devHandle.deviceGrid<float>();
+    if (!dGrid) { std::printf("no device grid\n"); return 1; }
+
+    const int W = 512, H = 512;
+    unsigned char* dImg = nullptr;
+    cudaMalloc(&dImg, size_t(W) * H * 3);
+    const dim3 block(16, 16), gridDim((W + 15) / 16, (H + 15) / 16);
+    render<<<gridDim, block>>>(dGrid, dImg, W, H, lightDir, ambient);
+    cudaDeviceSynchronize();
+    if (auto e = cudaGetLastError(); e != cudaSuccess) {
+        std::printf("CUDA error: %s\n", cudaGetErrorString(e));
+        return 1;
+    }
+
+    std::vector<unsigned char> img(size_t(W) * H * 3);
+    cudaMemcpy(img.data(), dImg, img.size(), cudaMemcpyDeviceToHost);
+    cudaFree(dImg);
+
+    FILE* f = std::fopen("sphere.ppm", "wb");
+    std::fprintf(f, "P6\n%d %d\n255\n", W, H);
+    std::fwrite(img.data(), 1, img.size(), f);
+    std::fclose(f);
+    std::printf("wrote sphere.ppm (%dx%d)\n", W, H);
+    return 0;
+}

doc/nanovdb/TEACHME/capstone/raymarch_reference.cu

@@ -0,0 +1,105 @@
+// Capstone: GPU level-set ray-march renderer.
+// Build an SDF sphere -> move to device -> one thread per pixel ->
+// ZeroCrossing to find the surface -> Lambertian shade via SDF gradient
+// normal -> write PPM.
+#include <nanovdb/NanoVDB.h>
+#include <nanovdb/tools/CreatePrimitives.h>
+#include <nanovdb/cuda/DeviceBuffer.h>
+#include <nanovdb/cuda/GridHandle.cuh>
+#include <nanovdb/math/Ray.h>
+#include <nanovdb/math/HDDA.h>        // ZeroCrossing lives here
+#include <nanovdb/math/Stencils.h>   // GradStencil
+
+#include <cstdio>
+#include <cstdlib>
+#include <cmath>
+#include <vector>
+
+using GridT = nanovdb::NanoGrid<float>;
+
+__global__ void render(const GridT* grid, unsigned char* img, int W, int H,
+                       nanovdb::Vec3f lightDir, float ambient)
+{
+    const int x = blockIdx.x * blockDim.x + threadIdx.x;
+    const int y = blockIdx.y * blockDim.y + threadIdx.y;
+    if (x >= W || y >= H) return;
+    const int pid = (y * W + x) * 3;
+
+    // --- pinhole camera in WORLD space ---
+    const nanovdb::Vec3f eye(0.f, 0.f, -300.f);
+    const float aspect     = float(W) / float(H);
+    const float tanHalfFov = tanf(0.5f * 45.f * 3.14159265f / 180.f);
+    const float u = (2.f * ((x + 0.5f) / W) - 1.f) * tanHalfFov * aspect;
+    const float v = (1.f - 2.f * ((y + 0.5f) / H)) * tanHalfFov;  // flip y
+    nanovdb::Vec3f dir(u, v, 1.f);
+    dir.normalize();
+
+    nanovdb::math::Ray<float> wRay(eye, dir);
+    auto iRay = wRay.worldToIndexF(*grid);          // ray in index space
+
+    auto acc = grid->getAccessor();
+    nanovdb::Coord ijk;
+    float t = 0.f, val = 0.f;
+
+    unsigned char shade = 0;                        // background (black)
+    if (nanovdb::math::ZeroCrossing(iRay, acc, ijk, val, t)) {
+        nanovdb::math::GradStencil<GridT> stencil(*grid);
+        stencil.moveTo(ijk);
+        nanovdb::Vec3f n = stencil.gradient();      // SDF gradient = outward normal
+        n.normalize();
+        const float diff = fmaxf(0.f, n.dot(lightDir));
+        const float I = fminf(1.f, ambient + (1.f - ambient) * diff);
+        shade = (unsigned char)(255.f * I);         // greyscale
+    }
+    img[pid] = shade; img[pid + 1] = shade; img[pid + 2] = shade;
+}
+
+int main(int argc, char** argv)
+{
+    // Shading parameters (passed into the kernel). Defaults, overridable on the
+    // command line:  ./raymarch [ambient] [lx ly lz]
+    float ambient = 0.15f;
+    nanovdb::Vec3f lightDir(-0.577f, 0.577f, -0.577f);   // (-1,1,-1) normalized
+    if (argc >= 2) ambient = (float)std::atof(argv[1]);
+    if (argc >= 5) lightDir = nanovdb::Vec3f((float)std::atof(argv[2]),
+                                             (float)std::atof(argv[3]),
+                                             (float)std::atof(argv[4]));
+    lightDir.normalize();
+
+    // 1. Build a level-set sphere SDF on the host (radius 100, voxel size 1).
+    auto handle = nanovdb::tools::createLevelSetSphere<float>(
+        /*radius=*/100.0, /*center=*/nanovdb::Vec3d(0.0),
+        /*voxelSize=*/1.0, /*halfWidth=*/3.0);
+
+    // 2. Move it to the device. copy<DeviceBuffer>() fills the HOST side of the
+    //    dual buffer; deviceUpload() pushes the bytes to the device. Only then
+    //    is deviceGrid() non-null.
+    auto devHandle = handle.copy<nanovdb::cuda::DeviceBuffer>();
+    devHandle.deviceUpload();
+    const GridT* dGrid = devHandle.deviceGrid<float>();
+    if (!dGrid) { std::printf("no device grid\n"); return 1; }
+
+    // 3. Render.
+    const int W = 512, H = 512;
+    unsigned char* dImg = nullptr;
+    cudaMalloc(&dImg, size_t(W) * H * 3);
+    const dim3 block(16, 16), grid((W + 15) / 16, (H + 15) / 16);
+    render<<<grid, block>>>(dGrid, dImg, W, H, lightDir, ambient);
+    cudaDeviceSynchronize();
+    if (auto e = cudaGetLastError(); e != cudaSuccess) {
+        std::printf("CUDA error: %s\n", cudaGetErrorString(e));
+        return 1;
+    }
+
+    // 4. Copy back and write PPM.
+    std::vector<unsigned char> img(size_t(W) * H * 3);
+    cudaMemcpy(img.data(), dImg, img.size(), cudaMemcpyDeviceToHost);
+    cudaFree(dImg);
+
+    FILE* f = std::fopen("sphere.ppm", "wb");
+    std::fprintf(f, "P6\n%d %d\n255\n", W, H);
+    std::fwrite(img.data(), 1, img.size(), f);
+    std::fclose(f);
+    std::printf("wrote sphere.ppm (%dx%d)\n", W, H);
+    return 0;
+}

doc/nanovdb/TEACHME/nanovdb_user_cheatsheet.md

@@ -0,0 +1,592 @@
+# NanoVDB User Cheat Sheet
+
+A one-page reference for the APIs covered in `nanovdb_user_lesson.md`. Keep
+this open while working through exercises.
+
+---
+
+## In one paragraph
+
+NanoVDB is a single-header, read-only, GPU-friendly sparse 3D data
+structure. The whole grid (one or more) lives in one contiguous,
+32-byte-aligned byte buffer. `cudaMemcpy` it to the device and it works
+as-is. Topology is *frozen* after construction; values can change, the set
+of active voxels cannot. Every primitive is `__hostdev__` (callable from
+both host and device with identical code).
+
+---
+
+## Includes & CMake
+
+```cpp openvdb
+#include <nanovdb/NanoVDB.h>               // grid, accessor, math
+#include <nanovdb/GridHandle.h>            // owns the buffer
+#include <nanovdb/HostBuffer.h>            // CPU allocator
+#include <nanovdb/NodeManager.h>           // sequential iteration
+#include <nanovdb/io/IO.h>                 // file IO
+#include <nanovdb/math/HDDA.h>             // ray-march
+#include <nanovdb/math/SampleFromVoxels.h> // sampling
+
+// GPU
+#include <nanovdb/cuda/DeviceBuffer.h>
+#include <nanovdb/cuda/GridHandle.cuh>
+#include <nanovdb/cuda/NodeManager.cuh>
+
+// Builders & operators (GPU)
+#include <nanovdb/tools/cuda/PointsToGrid.cuh>
+#include <nanovdb/tools/cuda/MeshToGrid.cuh>
+#include <nanovdb/tools/cuda/DilateGrid.cuh>
+#include <nanovdb/tools/cuda/CoarsenGrid.cuh>
+#include <nanovdb/tools/cuda/RefineGrid.cuh>
+#include <nanovdb/tools/cuda/PruneGrid.cuh>
+#include <nanovdb/tools/cuda/MergeGrids.cuh>
+#include <nanovdb/tools/cuda/SignedFloodFill.cuh>
+
+// CPU builders & primitives
+#include <nanovdb/tools/GridBuilder.h>
+#include <nanovdb/tools/CreatePrimitives.h>
+
+// IndexGrid / VBM
+#include <nanovdb/tools/VoxelBlockManager.h>
+#include <nanovdb/tools/cuda/VoxelBlockManager.cuh>
+
+// Stats, validation, blind data
+#include <nanovdb/tools/GridStats.h>
+#include <nanovdb/tools/GridChecksum.h>
+#include <nanovdb/tools/GridValidator.h>
+#include <nanovdb/tools/cuda/AddBlindData.cuh>
+
+// OpenVDB bridge (gated behind NANOVDB_USE_OPENVDB)
+#include <nanovdb/tools/CreateNanoGrid.h>
+#include <nanovdb/tools/NanoToOpenVDB.h>
+```
+
+```cmake
+find_package(nanovdb CONFIG REQUIRED)
+target_link_libraries(my_app PRIVATE nanovdb::nanovdb)
+```
+
+---
+
+## Build types (`NanoGrid<T>`)
+
+| `T`               | Stored per voxel                          |
+|-------------------|-------------------------------------------|
+| `float`, `double` | Scalar (SDF, density)                     |
+| `int32_t`, `int64_t` | Integer                                |
+| `Vec3f`, `Vec3d`  | 3-vector (velocity, color)                |
+| `bool`            | Active state only                         |
+| `ValueIndex`      | Sequential index over all voxels          |
+| `ValueOnIndex`    | Sequential index over **active** voxels   |
+| `ValueMask`       | Topology only (no per-voxel values)       |
+| `Half`            | 16-bit float                              |
+| `Fp4`/`Fp8`/`Fp16`/`FpN` | Quantized float (4/8/16 bits)      |
+| `Point`           | Point counts (for point grids)            |
+
+---
+
+## File IO
+
+```cpp
+namespace io = nanovdb::io;
+
+auto h  = io::readGrid("file.nvdb");                 // first grid
+auto h2 = io::readGrid("file.nvdb", 2);              // grid #2 (0-based)
+auto hN = io::readGrid("file.nvdb", "density");      // by name
+auto hAll = io::readGrid("file.nvdb", -1);           // all → one handle
+auto v  = io::readGrids("file.nvdb");                // vector<handle>, one per segment
+
+io::writeGrid("out.nvdb", h);                        // single
+io::writeGrid("out.nvdb", h, io::Codec::BLOSC);      // compressed
+io::writeGrids("out.nvdb", handles);                 // many
+```
+
+---
+
+## Inspecting a grid
+
+```cpp
+const auto* grid = handle.grid<float>();             // nullptr if T mismatch
+if (!grid) throw std::runtime_error("type mismatch");
+
+grid->gridName();           // const char*
+grid->gridClass();          // GridClass enum (LevelSet, FogVolume, etc.)
+grid->gridType();           // GridType enum (Float, Vec3f, OnIndex, ...)
+grid->voxelSize();          // Vec3d
+grid->activeVoxelCount();   // uint64_t
+grid->indexBBox();          // CoordBBox
+grid->worldBBox();          // BBox<Vec3d>
+grid->tree().background();  // T
+grid->worldToIndex(p);      // Vec3<T>, p is world-space
+grid->indexToWorld(p);      // Vec3<T>, p is index-space (fractional)
+```
+
+---
+
+## Random access (ReadAccessor)
+
+```cpp
+// One accessor per thread; not thread-safe; never copy between CPU & GPU
+auto acc = grid->getAccessor();
+float v   = acc.getValue(nanovdb::Coord(1, 2, 3));
+bool  on  = acc.isActive(nanovdb::Coord(1, 2, 3));
+const auto* leaf = acc.getNode<nanovdb::NanoLeaf<float>>();  // cached leaf ptr, if needed
+
+// probe* = value/state/leaf-existence in ONE descent (disambiguates the
+// two "got background" cases: inactive-vs-no-node)
+float pv;
+bool active = acc.probeValue(nanovdb::Coord(1,2,3), pv);     // pv set; ret = state
+const nanovdb::NanoLeaf<float>* lf = acc.probeLeaf(nanovdb::Coord(1,2,3)); // nullptr if no leaf
+```
+
+---
+
+## Sequential access (NodeManager)
+
+```cpp
+#include <nanovdb/NodeManager.h>
+
+auto nmh = nanovdb::createNodeManager(*grid);
+const auto* mgr = nmh.mgr<float>();
+
+auto leafN  = mgr->leafCount();
+auto lowerN = mgr->lowerCount();
+auto upperN = mgr->upperCount();
+
+for (uint64_t i = 0; i < leafN; ++i) {
+    const auto& leaf = mgr->leaf(i);
+    for (auto it = leaf.beginValueOn(); it; ++it) {
+        float v = *it;
+        auto c = it.getCoord();
+    }
+}
+```
+
+GPU. Pass the device-side `NodeManager*` to the kernel and index leaves by
+block, voxels within a leaf by thread:
+
+```cpp no-compile
+#include <nanovdb/cuda/NodeManager.cuh>
+
+auto dnmh = nanovdb::cuda::createNodeManager<float>(dGrid);
+const auto* dmgr = dnmh.template deviceMgr<float>();   // device pointer
+
+// One block per leaf; 64 threads cover the 512 voxels in each leaf.
+perLeafKernel<<<dmgr->leafCount(), 64>>>(dmgr);   // kernel defined just below
+```
+
+The kernel:
+
+```cpp
+__global__ void perLeafKernel(const nanovdb::NodeManager<float>* mgr)
+{
+    using LeafT = nanovdb::NanoLeaf<float>;
+    const uint32_t leafIdx = blockIdx.x;
+    if (leafIdx >= mgr->leafCount()) return;
+
+    const LeafT& leaf = mgr->leaf(leafIdx);
+
+    // 512 voxels per leaf (LeafT::SIZE == 1<<3*LOG2DIM == 8^3).
+    for (uint32_t n = threadIdx.x; n < LeafT::SIZE; n += blockDim.x) {
+        if (!leaf.isActive(n)) continue;
+        float          v = leaf.getValue(n);
+        nanovdb::Coord c = leaf.offsetToGlobalCoord(n);
+        // ... do something with (v, c)
+    }
+}
+```
+
+Same methods exist on `lower(i)` and `upper(i)` (internal nodes) — you'll
+mostly want `leaf(i)` since that's where actual voxel values live.
+
+Indexed leaf-voxel API (callable from `__hostdev__` code):
+
+| Call                          | Returns                                   |
+|-------------------------------|-------------------------------------------|
+| `leaf.getValue(n)`            | voxel value at offset `n` (0..511)        |
+| `leaf.isActive(n)`            | active state at offset `n`                |
+| `leaf.offsetToGlobalCoord(n)` | the global `Coord` for offset `n`         |
+| `leaf.origin()`               | the leaf's `(8,8,8)`-aligned origin Coord |
+| `LeafT::SIZE`                 | `512`, voxels per leaf                    |
+
+---
+
+## Math primitives (all `__hostdev__`)
+
+```cpp
+nanovdb::Coord c(1, 2, 3);
+auto c2 = nanovdb::Coord::Floor(nanovdb::Vec3f(1.7f, 2.1f, 3.9f)); // (1, 2, 3)
+
+nanovdb::Vec3f v(1, 2, 3);
+auto d = v.dot(v);            // 14
+auto n = v.length();          // sqrt(14)
+auto u = v.normalize();
+
+nanovdb::math::BBox<nanovdb::Vec3f> b(nanovdb::Vec3f(0,0,0),
+                                      nanovdb::Vec3f(10,10,10));
+b.expand(nanovdb::Vec3f(5,5,5));
+b.isInside(nanovdb::Vec3f(1,1,1));
+```
+
+---
+
+## Sampling
+
+```cpp
+#include <nanovdb/math/SampleFromVoxels.h>
+
+auto acc = grid->getAccessor();
+auto sampler = nanovdb::math::createSampler<1>(acc);  // 0=nearest, 1=trilinear, 2=triquadratic, 3=tricubic
+auto idxPt = grid->worldToIndex(worldPt);
+float v = sampler(idxPt);
+```
+
+---
+
+## HDDA ray-march
+
+```cpp
+#include <nanovdb/math/Ray.h>
+#include <nanovdb/math/HDDA.h>
+
+auto acc = grid->getAccessor();
+nanovdb::math::Ray<float> wRay({0,0,-5}, {0,0,1});
+auto iRay = wRay.worldToIndexF(*grid);
+
+// High-level: ZeroCrossing drives an HDDA and stops at the first sign change.
+nanovdb::Coord ijk;
+float v, t;
+if (nanovdb::math::ZeroCrossing(iRay, acc, ijk, v, t)) { /* hit at ijk */ }
+
+// Low-level: HDDA ctor takes the ray AND the start dimension from getDim.
+nanovdb::Coord start(0, 0, 0);
+nanovdb::math::HDDA<decltype(iRay), nanovdb::Coord> hdda(iRay, acc.getDim(start, iRay));
+while (hdda.step()) {
+    if (acc.isActive(hdda.voxel())) break;   // first active voxel
+}
+```
+
+---
+
+## GPU: transfer + kernel
+
+```cpp
+#include <nanovdb/cuda/DeviceBuffer.h>
+#include <nanovdb/cuda/GridHandle.cuh>
+
+auto devH = hostH.copy<nanovdb::cuda::DeviceBuffer>();
+devH.deviceUpload();                       // REQUIRED: push to device, else deviceGrid()==nullptr
+const auto* dGrid = devH.deviceGrid<float>();
+
+__global__ void k(const nanovdb::NanoGrid<float>* g,
+                  const nanovdb::Coord* coords, float* out, int N) {
+    int tid = blockIdx.x * blockDim.x + threadIdx.x;
+    if (tid >= N) return;
+    auto acc = g->getAccessor();
+    out[tid] = acc.getValue(coords[tid]);
+}
+
+k<<<(N+127)/128, 128>>>(dGrid, dCoords, dOut, N);
+```
+
+---
+
+## GPU builders
+
+```cpp
+// Points → grid (builder-pattern class, not a free function)
+nanovdb::tools::cuda::PointsToGrid<nanovdb::Point> p2g(/*voxelSize=*/0.05);
+auto ptsH = p2g.getHandle(dPoints, nPts);
+const auto* ptsGrid = ptsH.deviceGrid<nanovdb::Point>();
+
+// Mesh → OnIndexGrid (narrow-band). Verts are Vec3f, triangles Vec3i.
+const nanovdb::Vec3f* dVerts; size_t vertN;
+const nanovdb::Vec3i* dTris;  size_t triN;
+nanovdb::tools::cuda::MeshToGrid<nanovdb::ValueOnIndex> m2g(
+    dVerts, vertN, dTris, triN, nanovdb::Map(voxelSize));
+m2g.setNarrowBandWidth(3.f);
+auto meshH = m2g.getHandle();
+const auto* meshGrid = meshH.deviceGrid<nanovdb::ValueOnIndex>();
+
+// MeshToGrid + per-active-voxel UDF sidecar
+auto [gridH, udfBuf] = m2g.getHandleAndUDF();
+```
+
+## CPU builder
+
+```cpp
+// Build a tree on the host, then bake into a packed NanoVDB grid.
+nanovdb::tools::build::Grid<float> g(/*background=*/0.0f,
+                                     "my_grid", nanovdb::GridClass::FogVolume);
+g.setTransform(/*scale=*/0.1);                   // voxel size
+auto acc = g.getAccessor();                      // not thread-safe
+acc.setValue(nanovdb::Coord(0, 0, 0), 1.0f);
+auto h = nanovdb::tools::createNanoGrid(g);      // GridHandle<HostBuffer>
+```
+
+## Primitive helpers
+
+```cpp
+auto h = nanovdb::tools::createLevelSetSphere<float>(
+    /*radius=*/10.0, /*center=*/{0,0,0}, /*voxelSize=*/0.1);
+// also: createLevelSetTorus, createLevelSetBox, createFogVolumeSphere, ...
+```
+
+---
+
+## Topological operators (GPU)
+
+```cpp
+// Builder classes (NOT free functions); ValueOnIndex grids only. Construct
+// from a device grid pointer, getHandle() returns a new GridHandle.
+using OnIdx = nanovdb::ValueOnIndex;
+auto* g = devH.deviceGrid<OnIdx>();
+
+auto dilatedH = nanovdb::tools::cuda::DilateGrid<OnIdx>(g).getHandle();
+auto coarseH  = nanovdb::tools::cuda::CoarsenGrid<OnIdx>(g).getHandle();
+auto fineH    = nanovdb::tools::cuda::RefineGrid<OnIdx>(g).getHandle();
+const nanovdb::Mask<3>* leafMask = nullptr;   // PruneGrid needs a per-leaf mask
+auto prunedH  = nanovdb::tools::cuda::PruneGrid<OnIdx>(g, leafMask).getHandle();
+auto mergedH  = nanovdb::tools::cuda::MergeGrids<OnIdx>(g, g).getHandle();
+
+// signedFloodFill is the exception: a free function, in place, on a scalar
+// SDF grid (not OnIndex).
+nanovdb::tools::cuda::signedFloodFill(devH.deviceGrid<float>());
+```
+
+CPU morphology equivalents live under `nanovdb/util/MorphologyHelpers.h`.
+
+## Re-home sidecar data after a topology change (Injection.cuh)
+
+A topology op renumbers the active-voxel indices, so the old sidecar no
+longer matches. Move values across, and find the new voxels:
+
+```cpp
+#include <nanovdb/util/cuda/Injection.cuh>
+#include <nanovdb/util/cuda/DeviceGridTraits.cuh>
+#include <nanovdb/util/cuda/Util.h>
+
+using OnIdx = nanovdb::ValueOnIndex;
+const auto* srcGrid = devH.deviceGrid<OnIdx>();      // old grid
+const auto* dstGrid = devHandle.deviceGrid<OnIdx>(); // new (e.g. dilated) grid
+const float* oldSidecar = nullptr;
+float*       newSidecar = nullptr;                   // size = new activeVoxelCount(+1)
+
+// Copy overlapping values (non-overlap dst voxels left untouched):
+auto srcLeaves = nanovdb::util::cuda::DeviceGridTraits<OnIdx>::getTreeData(srcGrid).mNodeCount[0];
+nanovdb::util::cuda::operatorKernel<
+    nanovdb::util::cuda::InjectGridDataFunctor<OnIdx, float>>
+    <<<srcLeaves, 256>>>(srcGrid, dstGrid, oldSidecar, newSidecar);
+
+// Per-leaf overlap mask (new voxels = dstMask & ~overlap):
+auto dstLeaves = nanovdb::util::cuda::DeviceGridTraits<OnIdx>::getTreeData(dstGrid).mNodeCount[0];
+auto maskBuf = nanovdb::cuda::DeviceBuffer::create(dstLeaves*sizeof(nanovdb::Mask<3>), nullptr, false);
+auto* overlap = static_cast<nanovdb::Mask<3>*>(maskBuf.deviceData());
+unsigned nt = 128, nb = (dstLeaves + nt - 1) / nt;
+nanovdb::util::cuda::lambdaKernel<<<nb, nt>>>(dstLeaves,
+    nanovdb::util::cuda::InjectGridMaskFunctor<OnIdx>(), srcGrid, dstGrid, overlap);
+```
+
+Simpler: pre-fill `newSidecar` with a sentinel; slots still holding it after
+the data inject are the new voxels.
+
+---
+
+## Stats, validation & integrity
+
+Nodes cache bbox/min/max/avg per node. Mutating values in place leaves them
+**stale** — which silently breaks HDDA space-skipping and ray-tracing.
+Recompute after any manual edit.
+
+```cpp
+#include <nanovdb/tools/GridStats.h>          // CPU
+// #include <nanovdb/tools/cuda/GridStats.cuh> // GPU (device grid + stream)
+
+nanovdb::NanoGrid<float>* mutableGrid = nullptr;   // non-const: it gets written
+nanovdb::tools::updateGridStats(mutableGrid, nanovdb::tools::StatsMode::All);
+// StatsMode: Disable | BBox | MinMax | All(=Default)
+
+// Extrema over an index-space box, computed directly (no cache):
+auto ex = nanovdb::tools::getExtrema(*grid, nanovdb::CoordBBox(nanovdb::Coord(-10), nanovdb::Coord(10)));
+float lo = ex.min(), hi = ex.max();
+```
+
+Certify a grid after load / build / device transfer:
+
+```cpp
+#include <nanovdb/tools/GridChecksum.h>
+#include <nanovdb/tools/GridValidator.h>
+
+nanovdb::NanoGrid<float>* mutableGrid = nullptr;   // non-const: edited grid
+
+bool good = nanovdb::tools::validateChecksum(grid, nanovdb::CheckMode::Full);
+nanovdb::tools::updateChecksum(mutableGrid, nanovdb::CheckMode::Full); // after a legit edit
+
+bool valid = nanovdb::tools::isValid(grid, nanovdb::CheckMode::Full, /*verbose=*/false);
+char err[(uint32_t)nanovdb::CheckMode::StrLen] = {};   // CheckMode is scoped → cast
+nanovdb::tools::checkGrid(grid, err, nanovdb::CheckMode::Full);   // err="" if valid
+// CheckMode: Disable/Empty | Half/Partial/Default | Full
+// GPU variants: tools/cuda/GridChecksum.cuh, tools/cuda/GridValidator.cuh
+```
+
+---
+
+## IndexGrid & VBM
+
+```cpp no-compile
+// Build a ValueOnIndex grid from an OpenVDB grid
+auto h = nanovdb::tools::createNanoGrid<openvdb::FloatGrid,
+                                        nanovdb::ValueOnIndex>(*ovGrid);
+
+// Or from a mesh on the GPU (already returns OnIndexGrid)
+nanovdb::tools::cuda::MeshToGrid<nanovdb::ValueOnIndex> mesher(dV, vN, dT, tN, nanovdb::Map(vs));
+auto devH = mesher.getHandle();
+const auto* g = devH.deviceGrid<nanovdb::ValueOnIndex>();
+
+// External attribute array — same length as active voxel count
+size_t N = g->activeVoxelCount();
+thrust::device_vector<float> attr(N);
+
+// Read attribute at a coordinate
+auto acc = g->getAccessor();
+uint64_t idx = acc.getValue(c);       // index into attr
+float a = attr[idx];
+
+// VBM: SIMT-friendly block iteration (Log2BlockWidth >= 6, i.e. >=64/block)
+auto vbmH = nanovdb::tools::buildVoxelBlockManager<6>(g);
+// see tools/cuda/VoxelBlockManager.cuh for the GPU decode pattern
+```
+
+ChannelAccessor — read the sidecar by attribute, not by index (one typed
+call instead of getValue(ijk) → array[idx]). Note: IndexT defaults to
+`ValueIndex`, so spell out `ValueOnIndex`:
+
+```cpp
+__global__ void readChan(const nanovdb::NanoGrid<nanovdb::ValueOnIndex>* g,
+                         float* dValues, nanovdb::Coord c, float* out) {
+    nanovdb::ChannelAccessor<float, nanovdb::ValueOnIndex> ca(*g, dValues);
+    *out = ca.getValue(c);          // attribute value (not the index)
+    // ca(c) same; ca.getIndex(c) for the raw index; ca.probeValue(c, v) for state+value
+    // ctor (*g, channelID) instead reads an internal blind-data channel
+}
+```
+
+Bake a sidecar back into a value grid (round-trip partner of the split;
+needed by value-typed tools like `signedFloodFill`):
+
+```cpp
+#include <nanovdb/tools/cuda/IndexToGrid.cuh>
+
+const auto* idxGrid = devH.deviceGrid<nanovdb::ValueOnIndex>();
+const float* dValues = nullptr;                       // per-active-voxel sidecar
+auto floatH = nanovdb::tools::cuda::indexToGrid<float>(idxGrid, dValues);
+const auto* floatGrid = floatH.deviceGrid<float>();   // values now in-grid
+```
+
+---
+
+## OpenVDB ↔ NanoVDB
+
+```cpp openvdb
+#include <nanovdb/tools/CreateNanoGrid.h>
+#include <nanovdb/tools/NanoToOpenVDB.h>
+
+// OpenVDB → NanoVDB (optionally with type change)
+auto h = nanovdb::tools::createNanoGrid<openvdb::FloatGrid, float>(*ovGrid);
+auto h_half = nanovdb::tools::createNanoGrid<openvdb::FloatGrid,
+                                             nanovdb::Half>(*ovGrid);
+
+// NanoVDB → OpenVDB
+auto ovGrid = nanovdb::tools::nanoToOpenVDB<float>(handle);
+```
+
+---
+
+## Blind data (attached arrays)
+
+Read (host or device — `__hostdev__`):
+
+```cpp
+uint32_t n = grid->blindDataCount();
+const nanovdb::GridBlindMetaData& meta = grid->blindMetaData(0);
+// meta.mValueCount, mValueSize, mDataType, mSemantic, mDataClass, mName
+
+int id = grid->findBlindDataForSemantic(nanovdb::GridBlindDataSemantic::PointPosition);
+// or: int id = grid->findBlindData("name");
+const nanovdb::Vec3f* pos = grid->getBlindData<nanovdb::Vec3f>(id);  // nullptr if T mismatch
+```
+
+Attach on the GPU (returns a NEW handle with the array appended; fixes the
+checksum):
+
+```cpp
+#include <nanovdb/tools/cuda/AddBlindData.cuh>
+
+const float* dChannel = nullptr; uint64_t valueCount = 0;
+auto h = nanovdb::tools::cuda::addBlindData(
+    dGrid, dChannel, valueCount,
+    nanovdb::GridBlindDataClass::ChannelArray,
+    nanovdb::GridBlindDataSemantic::Unknown, "my_channel");
+const auto* g = h.deviceGrid<float>();
+```
+
+---
+
+## Common idioms
+
+```cpp
+// Iterate active voxels in a grid (CPU, leaf-by-leaf)
+auto nmh = nanovdb::createNodeManager(*grid);
+auto* mgr = nmh.mgr<float>();
+for (uint64_t L = 0; L < mgr->leafCount(); ++L) {
+    const auto& leaf = mgr->leaf(L);
+    for (auto it = leaf.beginValueOn(); it; ++it) {
+        // *it is the value, it.getCoord() is the ijk
+    }
+}
+
+// Sample a grid at a list of world-space points (pts is a std::vector<Vec3f>)
+auto sampler = nanovdb::math::createSampler<1>(acc);
+std::vector<float> sampled;
+for (const auto& w : pts)
+    sampled.push_back(sampler(grid->worldToIndex(w)));
+
+// Save a level-set sphere to disk
+auto h = nanovdb::tools::createLevelSetSphere<float>(10.0, {0,0,0}, 0.1);
+nanovdb::io::writeGrid("sphere.nvdb", h);
+```
+
+---
+
+## Pitfalls
+
+- **Type mismatch returns nullptr.** Always check `handle.grid<T>()`.
+- **Accessors are not thread-safe.** One per thread/warp.
+- **Accessors don't cross CPU/GPU.** Build inside the kernel.
+- **Topology is frozen.** To "add a voxel" you must rebuild the grid.
+- **32-byte alignment.** Don't allocate the buffer yourself with
+  `new char[N]` — use `HostBuffer` or `DeviceBuffer`, which guarantee
+  alignment.
+- **No runtime `std::` in `__hostdev__` code.** Containers, iostreams,
+  allocators, `std::sort` etc. won't work on device. For type traits and
+  compile-time helpers, NanoVDB uses its own `nanovdb::util::is_same` /
+  `util::enable_if` / `util::conditional` / `util::remove_const` (in
+  `util/Util.h`) — these compile under C++11 across compilers and don't
+  need `--expt-relaxed-constexpr`. Where NanoVDB needs more, it pulls from
+  CCCL's `cuda::std::` (e.g. `cuda::std::numeric_limits` in `math/Math.h`).
+  Plain `std::` compile-time facilities work with relaxed constexpr but
+  NanoVDB's own headers avoid them for portability.
+- **HDDA is essential for sparse ray-march.** Without it, you're stepping
+  one voxel at a time through empty space.
+
+---
+
+## Tools / binaries
+
+| Tool                | Purpose                                   |
+|---------------------|-------------------------------------------|
+| `nanovdb_print`     | Print metadata for grids in a `.nvdb`     |
+| `nanovdb_convert`   | OpenVDB ↔ NanoVDB on the CLI              |
+| `nanovdb_validate`  | ABI / format validation                   |
+
+Run with `--help` for options.

doc/nanovdb/TEACHME/test/.gitignore

@@ -0,0 +1,3 @@
+# Transient harness outputs
+gen/
+build/

doc/nanovdb/TEACHME/test/CMakeLists.txt

@@ -0,0 +1,87 @@
+cmake_minimum_required(VERSION 3.20)
+project(teachme_tests LANGUAGES CXX CUDA)
+
+# Compile-only gate: object libraries compile each generated translation unit
+# to an object file without linking. That is exactly what we want — catch API
+# errors at compile time; the fixture's extern stand-ins never need to resolve.
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CUDA_STANDARD 17)
+if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
+  set(CMAKE_CUDA_ARCHITECTURES 89)  # RTX 6000 Ada; any arch suffices to compile
+endif()
+
+# NanoVDB headers pull in CUDA lambdas / relaxed-constexpr code paths.
+# diag 177 = "function declared but never referenced": every wrapped block
+# becomes an unreferenced static function on purpose, so silence it.
+string(APPEND CMAKE_CUDA_FLAGS
+  " --extended-lambda --expt-relaxed-constexpr --diag-suppress 177")
+
+# NanoVDB include root: this file is doc/nanovdb/TEACHME/test/CMakeLists.txt, so
+# the repo root is four dirs up. Headers are at nanovdb/nanovdb/NanoVDB.h, so the
+# include root is the repo's `nanovdb/` subdir (a sibling of `doc/`).
+get_filename_component(_teachme_dir "${CMAKE_CURRENT_SOURCE_DIR}" DIRECTORY)  # TEACHME
+get_filename_component(_docnv_dir   "${_teachme_dir}" DIRECTORY)              # doc/nanovdb
+get_filename_component(_doc_dir     "${_docnv_dir}" DIRECTORY)                # doc
+get_filename_component(REPO_ROOT    "${_doc_dir}" DIRECTORY)                  # repo root
+set(NANOVDB_INCLUDE_ROOT "${REPO_ROOT}/nanovdb")
+set(OPENVDB_INCLUDE_ROOT "${REPO_ROOT}/openvdb")
+
+message(STATUS "NanoVDB include root: ${NANOVDB_INCLUDE_ROOT}")
+
+# Generated TUs + manifest live in ${GEN_DIR} (passed in, defaults to ./gen).
+if(NOT DEFINED GEN_DIR)
+  set(GEN_DIR "${CMAKE_CURRENT_SOURCE_DIR}/gen")
+endif()
+
+set(_manifest "${GEN_DIR}/manifest.txt")
+if(NOT EXISTS "${_manifest}")
+  message(FATAL_ERROR "No manifest at ${_manifest}. Run the extractor first "
+                      "(see run.sh).")
+endif()
+
+file(STRINGS "${_manifest}" _lines)
+set(_core_srcs "")
+set(_ovdb_srcs "")
+foreach(_line IN LISTS _lines)
+  if(_line STREQUAL "")
+    continue()
+  endif()
+  separate_arguments(_parts UNIX_COMMAND "${_line}")
+  list(GET _parts 0 _stem)
+  list(GET _parts 1 _needs_ovdb)
+  set(_src "${GEN_DIR}/${_stem}.cu")
+  if(_needs_ovdb STREQUAL "1")
+    list(APPEND _ovdb_srcs "${_src}")
+  else()
+    list(APPEND _core_srcs "${_src}")
+  endif()
+endforeach()
+
+# Core (NanoVDB-only) translation units.
+if(_core_srcs)
+  add_library(teachme_blocks OBJECT ${_core_srcs})
+  target_include_directories(teachme_blocks PRIVATE
+    "${NANOVDB_INCLUDE_ROOT}" "${GEN_DIR}" "${CMAKE_CURRENT_SOURCE_DIR}")
+  list(LENGTH _core_srcs _n_core)
+  message(STATUS "teachme_blocks: ${_n_core} core TU(s)")
+endif()
+
+# OpenVDB-dependent translation units — only if OpenVDB is available.
+if(_ovdb_srcs)
+  find_package(OpenVDB QUIET)
+  if(OpenVDB_FOUND)
+    add_library(teachme_blocks_openvdb OBJECT ${_ovdb_srcs})
+    target_compile_definitions(teachme_blocks_openvdb PRIVATE NANOVDB_USE_OPENVDB)
+    target_include_directories(teachme_blocks_openvdb PRIVATE
+      "${NANOVDB_INCLUDE_ROOT}" "${OPENVDB_INCLUDE_ROOT}"
+      "${GEN_DIR}" "${CMAKE_CURRENT_SOURCE_DIR}")
+    target_link_libraries(teachme_blocks_openvdb PRIVATE OpenVDB::openvdb)
+    list(LENGTH _ovdb_srcs _n_ovdb)
+    message(STATUS "teachme_blocks_openvdb: ${_n_ovdb} OpenVDB TU(s)")
+  else()
+    list(LENGTH _ovdb_srcs _n_ovdb)
+    message(WARNING "OpenVDB not found — skipping ${_n_ovdb} OpenVDB TU(s). "
+                    "These blocks are reference-checked but not compiled.")
+  endif()
+endif()

doc/nanovdb/TEACHME/test/README.md

@@ -0,0 +1,89 @@
+# TEACHME test harness
+
+Keeps the code in the lesson honest. Two gates:
+
+1. **Reference validation** (`teachme_tests.py check-refs`) — every
+   `#include <...>` in a code block must resolve to a real header in the
+   NanoVDB tree (or be explicitly exempt). Catches stale-path rot — e.g. a
+   header that was renamed out from under the docs.
+
+2. **Compilation** (`teachme_tests.py extract` + CMake) — code blocks tagged
+   as compilable are extracted into `.cu` translation units and compiled
+   (compile-only, `-c`) against the real NanoVDB headers. Catches API-shape
+   drift — wrong namespace, a free function that's actually a class, a
+   renamed method, a changed signature.
+
+Run both with `./run.sh` from this directory.
+
+## Why compile-only
+
+The harness compiles (`nvcc -c`) but does not link or run. NanoVDB is
+header-only and the goal is to catch *API* errors (does this call name a
+real symbol with a compatible signature?), which all surface at compile
+time. Skipping the link step means a fixture header can declare the
+"given" symbols a fragment assumes (`grid`, `acc`, device pointers) as
+`extern` globals — unresolved at link, but fully type-checked at compile.
+
+Everything compiles as CUDA (`.cu`) because even host-only examples pull
+in CUDA code paths (under `__CUDACC__`) when they include headers like
+`NodeManager.h` or `io/IO.h`.
+
+## Block tagging
+
+Code blocks are fenced with an info string: a language followed by
+whitespace-separated tags, e.g.
+
+    ```cpp compile
+    ```cpp compile-global continuation
+    ```cpp skip-refs
+
+**Opt-out model:** every `cpp` / `cu` block is compiled by default. A block
+is wrapped in a function body unless it opens a top-level definition
+(`__global__`, `template`, `struct`, `class`, `namespace`, `enum`, `using`,
+`typedef`), in which case it is auto-emitted at file scope. Tags only adjust
+the defaults:
+
+| Tag            | Meaning                                                                                  |
+|----------------|------------------------------------------------------------------------------------------|
+| *(none)*       | Compiled. Auto: file scope if it opens a top-level definition, else function-wrapped.    |
+| `no-compile`   | Reference-check includes only; do **not** compile. For illustrative/partial fragments.   |
+| `global`       | Force file-scope emission (override the auto-detection).                                  |
+| `main`         | Block is already a complete program (its own `main`); compile verbatim.                  |
+| `continuation` | Append this block to the previous block's translation unit (shares its symbols).         |
+| `openvdb`      | Block needs OpenVDB; compiled only when OpenVDB is found.                                 |
+| `skip-refs`    | Do **not** validate includes in this block — for intentionally-stale counter-examples.   |
+
+A translation unit is a sequence of segments. A non-`continuation` block
+starts a new TU; each following `continuation` block adds a segment. At emit
+time, file-scope segments go to file scope (in order) and function segments
+each get their own wrapper function — so a kernel definition followed by a
+`continuation` launch line compose into one valid TU.
+
+Every TU includes `fixture.cuh`, which pulls in the common NanoVDB headers
+(a deliberately broad "kitchen-sink" set, so terse fragments compile without
+restating includes) and declares the stand-in "given" symbols (`grid`,
+`acc`, device pointers, etc.) as `extern`. Including the heavy `tools/cuda/*`
+headers in every TU makes per-TU compilation slower; that's an accepted
+trade for letting fragments stay terse. If the suite grows large, switch to
+per-TU includes driven by each block's own `#include` lines.
+
+## Coverage
+
+`check-refs` reports how many code blocks are compile-tagged vs
+reference-only, so the fraction of the lesson that is actually
+compile-gated is visible (and ideally climbing).
+
+## Exemptions
+
+Includes that are intentionally not real are skipped by the validator:
+- ellipsis placeholders such as `#include <openvdb/...>`
+- any include inside a `skip-refs` block (e.g. the deprecated
+  `nanovdb/util/cuda/CudaDeviceBuffer.h` shown as a "this path is stale"
+  counter-example)
+
+## Files
+
+- `teachme_tests.py` — the extractor + reference validator
+- `fixture.cuh` — common headers + `extern` stand-in symbols
+- `CMakeLists.txt` — compiles the generated TUs
+- `run.sh` — driver: check-refs → extract → cmake configure+build → report

doc/nanovdb/TEACHME/test/fixture.cuh

@@ -0,0 +1,92 @@
+// fixture.cuh — common headers + stand-in "given" symbols for TEACHME
+// code-block compilation.
+//
+// Lesson fragments assume context: "you have a grid", "you have device
+// points", etc. The harness compiles each block (compile-only, never linked)
+// with this header in scope, so those assumed symbols need declarations with
+// the right *types* — enough for the compiler to type-check member calls and
+// catch API drift. They are declared `extern` (no definitions): a block that
+// declares its own local `grid`/`acc`/etc. simply shadows the extern, and
+// since we never link, the unresolved externs cost nothing.
+//
+// Grow this file as the tagging pass surfaces more assumed names. Keep the
+// types matching how the lesson uses each symbol.
+
+#pragma once
+
+// ---- common headers (header guards make re-includes in blocks harmless) ----
+// Core read path + IO + math + CUDA.
+#include <nanovdb/NanoVDB.h>
+#include <nanovdb/GridHandle.h>
+#include <nanovdb/HostBuffer.h>
+#include <nanovdb/NodeManager.h>
+#include <nanovdb/io/IO.h>
+#include <nanovdb/math/Ray.h>
+#include <nanovdb/math/HDDA.h>
+#include <nanovdb/math/SampleFromVoxels.h>
+#include <nanovdb/math/Stencils.h>
+#include <nanovdb/cuda/DeviceBuffer.h>
+#include <nanovdb/cuda/GridHandle.cuh>
+#include <nanovdb/cuda/NodeManager.cuh>
+// Commonly-assumed "tools" headers — ambient so terse fragments compile
+// without restating the include. (OpenVDB-conversion headers are NOT here;
+// blocks that need them are tagged `openvdb` and include them explicitly.)
+#include <nanovdb/tools/CreatePrimitives.h>
+#include <nanovdb/tools/CreateNanoGrid.h>
+#include <nanovdb/tools/GridBuilder.h>
+#include <nanovdb/tools/VoxelBlockManager.h>
+#include <nanovdb/tools/cuda/PointsToGrid.cuh>
+#include <nanovdb/tools/cuda/MeshToGrid.cuh>
+#include <nanovdb/tools/cuda/DilateGrid.cuh>
+#include <nanovdb/tools/cuda/CoarsenGrid.cuh>
+#include <nanovdb/tools/cuda/RefineGrid.cuh>
+#include <nanovdb/tools/cuda/PruneGrid.cuh>
+#include <nanovdb/tools/cuda/MergeGrids.cuh>
+#include <nanovdb/tools/cuda/SignedFloodFill.cuh>
+
+#include <cstdint>
+#include <cstddef>
+#include <cmath>
+#include <string>
+#include <vector>
+#include <iostream>
+#include <stdexcept>
+
+// ---- stand-in "given" symbols (extern: declared, never defined) ----
+// Handles
+extern nanovdb::GridHandle<nanovdb::HostBuffer>        handle;
+extern nanovdb::GridHandle<nanovdb::HostBuffer>        hostH;
+extern nanovdb::GridHandle<nanovdb::cuda::DeviceBuffer> devH;
+extern nanovdb::GridHandle<nanovdb::cuda::DeviceBuffer> devHandle;
+extern std::vector<nanovdb::GridHandle<nanovdb::HostBuffer>> handles;
+
+// Grids (host + device pointers into a float grid)
+extern const nanovdb::NanoGrid<float>* grid;
+extern const nanovdb::NanoGrid<float>* dGrid;
+
+// A read accessor + sampler over the float grid (fragments often assume these
+// already exist from a prior step).
+extern nanovdb::DefaultReadAccessor<float> acc;
+
+// Device point / coord / output buffers used by kernel launches.
+// dPoints is non-const: PointsToGrid<Point> writes input points into the
+// grid's blind data via getPoint<element_type>(...) = ..., which requires a
+// non-const element type.
+extern nanovdb::Vec3f*       dPoints;
+extern std::size_t           nPts;
+extern const nanovdb::Vec3f* dPts;
+extern const nanovdb::Vec3f* worldPts;
+extern const nanovdb::Coord* dCoords;
+extern const nanovdb::Coord* coords;
+extern float*                dOut;
+extern float*                out;
+extern int                   N;
+
+// Host-side query containers + single query points
+extern std::vector<nanovdb::Vec3f> pts;
+extern std::vector<nanovdb::Coord> queries;
+extern nanovdb::Vec3f worldPt;   // a single world-space query point
+extern nanovdb::Vec3f p;         // generic world-space point
+
+// Misc scalars
+extern double voxelSize;

doc/nanovdb/TEACHME/test/run.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# Drive the TEACHME test harness: reference-check -> extract -> compile.
+#
+# Usage: ./run.sh
+# Exits nonzero if any reference is dangling or any compile-tagged block
+# fails to compile.
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+TEACHME="$(dirname "$HERE")"
+GEN="$HERE/gen"
+BUILD="$HERE/build"
+
+MD=(
+  "$TEACHME/nanovdb_user_lesson.md"
+  "$TEACHME/nanovdb_user_cheatsheet.md"
+  "$TEACHME/README.md"
+)
+
+echo "==> [1/3] reference check"
+python3 "$HERE/teachme_tests.py" check-refs "${MD[@]}"
+
+echo "==> [2/3] extract compile-tagged blocks"
+rm -rf "$GEN"
+mkdir -p "$GEN"
+python3 "$HERE/teachme_tests.py" extract "${MD[@]}" --out "$GEN"
+
+n_tu=$(find "$GEN" -name '*.cu' | wc -l | tr -d ' ')
+if [ "$n_tu" -eq 0 ]; then
+  echo "==> [3/3] compile: no compile-tagged blocks yet — nothing to build"
+  echo "PASS (reference check only)"
+  exit 0
+fi
+
+echo "==> [3/3] compile $n_tu translation unit(s)"
+mkdir -p "$BUILD"
+cmake -S "$HERE" -B "$BUILD" -DGEN_DIR="$GEN" -DCMAKE_BUILD_TYPE=Release \
+  > "$BUILD/configure.log" 2>&1 || { cat "$BUILD/configure.log"; exit 1; }
+
+# Cap parallelism: each TU pulls in the kitchen-sink fixture (most of NanoVDB's
+# heavy headers), so nvcc is memory-hungry. Default to all cores locally; CI
+# sets TEACHME_JOBS lower so the parallel nvcc processes stay within the
+# runner's RAM (8 unbounded jobs OOM a 32 GB runner).
+jobs="${TEACHME_JOBS:-$(nproc)}"
+
+# Keep going (--keep-going / -k) so every failing TU is reported, not just the
+# first. Capture cmake's status directly (don't pipe through tee, which would
+# mask it).
+set +e
+cmake --build "$BUILD" --parallel "$jobs" -- -k > "$BUILD/build.log" 2>&1
+status=$?
+set -e
+cat "$BUILD/build.log"
+if [ "$status" -eq 0 ]; then
+  echo "PASS: all $n_tu translation unit(s) compiled"
+else
+  echo "FAIL: one or more translation units did not compile (see above)"
+  exit 1
+fi

doc/nanovdb/TEACHME/test/teachme_tests.py

@@ -0,0 +1,319 @@
+#!/usr/bin/env python3
+"""TEACHME lesson test harness: reference validation + block extraction.
+
+Two subcommands:
+
+  check-refs <md...>            Validate every #include in code blocks resolves
+                                against the NanoVDB include root. Reports
+                                compile-coverage. Exits nonzero on dangling refs.
+
+  extract <md...> --out <dir>   Extract compile-tagged code blocks into .cu
+                                translation units under <dir>, ready for CMake.
+
+See README.md for the block tagging convention.
+"""
+
+import argparse
+import os
+import re
+import sys
+
+# Repo layout: this file is at doc/nanovdb/TEACHME/test/teachme_tests.py, so the
+# repo root is four levels up. The NanoVDB include root (the dir that contains
+# the `nanovdb/` header subdir) is the repo's `nanovdb/`; OpenVDB's is `openvdb/`.
+HERE = os.path.dirname(os.path.abspath(__file__))
+REPO_ROOT = os.path.abspath(os.path.join(HERE, "..", "..", "..", ".."))
+NANOVDB_INCLUDE_ROOT = os.path.join(REPO_ROOT, "nanovdb")
+OPENVDB_INCLUDE_ROOT = os.path.join(REPO_ROOT, "openvdb")
+INCLUDE_ROOTS = [NANOVDB_INCLUDE_ROOT, OPENVDB_INCLUDE_ROOT]
+
+FENCE_RE = re.compile(r"^```(\w+)?(.*)$")
+INCLUDE_RE = re.compile(r'^\s*#\s*include\s*<([^>]+)>')
+
+# Inline-code spans in prose that look like header paths, e.g.
+# `nanovdb/tools/GridBuilder.h`. Used to catch stale "Read: ..." references.
+PROSE_PATH_RE = re.compile(r'`((?:nanovdb|openvdb)/[\w./]+\.(?:h|cuh|cc|cpp))`')
+# A line carrying this marker is exempt from prose-path checking (for
+# intentionally-stale counter-examples).
+STALE_OK = "teachme:stale-ok"
+
+
+def resolves(inc):
+    """True if an include path exists under any include root."""
+    return any(os.path.isfile(os.path.join(r, inc)) for r in INCLUDE_ROOTS)
+
+# Languages whose blocks we care about.
+CODE_LANGS = {"cpp", "cu", "c++"}
+
+# Opt-OUT model: every cpp/cu block is compiled unless tagged `no-compile`.
+# A line that begins a file-scope definition forces file-scope emission even
+# without an explicit `global` tag.
+FILE_SCOPE_RE = re.compile(r'^\s*(template\b|struct\b|class\b|namespace\b|'
+                           r'__global__\b|enum\b|using\b|typedef\b)')
+
+
+class Block:
+    __slots__ = ("path", "start_line", "lang", "tags", "lines")
+
+    def __init__(self, path, start_line, lang, tags):
+        self.path = path
+        self.start_line = start_line
+        self.lang = lang
+        self.tags = tags
+        self.lines = []
+
+    @property
+    def text(self):
+        return "".join(self.lines)
+
+
+def parse_blocks(path):
+    """Yield Block objects for every fenced code block in a markdown file."""
+    blocks = []
+    with open(path, encoding="utf-8") as f:
+        lines = f.readlines()
+    i = 0
+    in_block = None
+    for lineno, line in enumerate(lines, 1):
+        m = FENCE_RE.match(line)
+        if m:
+            if in_block is None:
+                lang = (m.group(1) or "").lower()
+                tags = set(m.group(2).split())
+                in_block = Block(path, lineno, lang, tags)
+            else:
+                blocks.append(in_block)
+                in_block = None
+            continue
+        if in_block is not None:
+            in_block.lines.append(line)
+    if in_block is not None:
+        sys.stderr.write(f"{path}: unterminated code fence opened at "
+                         f"line {in_block.start_line}\n")
+    return blocks
+
+
+def is_exempt_include(inc):
+    """Includes that are intentionally not real headers."""
+    if "..." in inc:           # ellipsis placeholder, e.g. openvdb/...
+        return True
+    return False
+
+
+def scan_prose_paths(path):
+    """Yield (lineno, token) for header-path references in prose inline code,
+    skipping lines inside fenced code blocks and lines marked stale-ok."""
+    with open(path, encoding="utf-8") as f:
+        lines = f.readlines()
+    in_fence = False
+    for lineno, line in enumerate(lines, 1):
+        if FENCE_RE.match(line) and line.lstrip().startswith("```"):
+            in_fence = not in_fence
+            continue
+        if in_fence or STALE_OK in line:
+            continue
+        for tok in PROSE_PATH_RE.findall(line):
+            yield lineno, tok
+
+
+def cmd_check_refs(args):
+    code_blocks = 0
+    compiled_blocks = 0
+    dangling = []          # (path, lineno, inc, kind)
+    for path in args.md:
+        # 1) #include lines inside code blocks
+        for b in parse_blocks(path):
+            if b.lang not in CODE_LANGS:
+                continue
+            code_blocks += 1
+            if "no-compile" not in b.tags:
+                compiled_blocks += 1
+            if "skip-refs" in b.tags:
+                continue
+            for off, line in enumerate(b.lines):
+                im = INCLUDE_RE.match(line)
+                if not im:
+                    continue
+                inc = im.group(1).strip()
+                if is_exempt_include(inc):
+                    continue
+                if not resolves(inc):
+                    dangling.append((path, b.start_line + 1 + off, inc, "include"))
+        # 2) header-path references in prose ("Read: `nanovdb/.../X.h`")
+        for lineno, tok in scan_prose_paths(path):
+            if not resolves(tok):
+                dangling.append((path, lineno, tok, "prose"))
+
+    if dangling:
+        sys.stderr.write("DANGLING REFERENCES:\n")
+        for path, lineno, ref, kind in dangling:
+            rel = os.path.relpath(path)
+            sys.stderr.write(f"  {rel}:{lineno}: [{kind}] {ref!r} does not "
+                             f"resolve under any include root\n")
+
+    pct = (100.0 * compiled_blocks / code_blocks) if code_blocks else 0.0
+    print(f"code blocks: {code_blocks}  compiled: {compiled_blocks} "
+          f"({pct:.0f}%)  no-compile: {code_blocks - compiled_blocks}")
+    if dangling:
+        print(f"FAIL: {len(dangling)} dangling reference(s)")
+        return 1
+    print("OK: all references resolve")
+    return 0
+
+
+def split_block(block):
+    """Split one block's lines into ordered (kind, lines) sub-segments.
+
+    `kind` is 'global' (file scope) or 'fn' (function body). Explicit `global`
+    / `main` tags force the whole block. Otherwise we brace-walk: a top-level
+    definition (matched by FILE_SCOPE_RE at brace depth 0, e.g. a __global__
+    kernel) becomes a file-scope chunk spanning until its braces balance;
+    everything else at depth 0 (launches, statements) becomes a function chunk.
+    This lets a single block mix a kernel definition and its launch.
+    """
+    if "global" in block.tags:
+        return [("global", block.lines)]
+    if "main" in block.tags:
+        return [("main", block.lines)]
+
+    segs = []
+    fn_buf = []
+    lines = block.lines
+    i, n = 0, len(lines)
+
+    def flush_fn():
+        if fn_buf:
+            segs.append(("fn", fn_buf[:]))
+            fn_buf.clear()
+
+    while i < n:
+        line = lines[i]
+        stripped = line.lstrip()
+        if stripped.startswith("#"):       # preprocessor: neutral (hoisted later)
+            fn_buf.append(line)
+            i += 1
+            continue
+        if FILE_SCOPE_RE.match(line):      # start of a top-level definition
+            flush_fn()
+            chunk = []
+            depth = 0
+            saw_brace = False
+            while i < n:
+                l = lines[i]
+                chunk.append(l)
+                i += 1
+                depth += l.count("{") - l.count("}")
+                if "{" in l:
+                    saw_brace = True
+                if saw_brace and depth <= 0:
+                    break
+                if not saw_brace and l.rstrip().endswith(";"):
+                    break              # brace-less decl, e.g. `using X = ...;`
+            segs.append(("global", chunk))
+            continue
+        fn_buf.append(line)                # ordinary statement → function body
+        i += 1
+    flush_fn()
+    return segs if segs else [("fn", lines)]
+
+
+def cmd_extract(args):
+    os.makedirs(args.out, exist_ok=True)
+    # A TU has a list of (kind, lines, line_no) sub-segments + an openvdb flag.
+    tus = []
+    current = None
+
+    for path in args.md:
+        for b in parse_blocks(path):
+            if b.lang not in CODE_LANGS:
+                continue
+            # Opt-out: compile everything except blocks tagged no-compile.
+            if "no-compile" in b.tags:
+                continue
+            subsegs = [(k, lines, b.start_line) for (k, lines) in split_block(b)]
+            cont = "continuation" in b.tags
+            if cont and current is not None:
+                current["segs"].extend(subsegs)
+                if "openvdb" in b.tags:
+                    current["openvdb"] = True
+            else:
+                current = {
+                    "segs": list(subsegs),
+                    "openvdb": "openvdb" in b.tags,
+                    "src": os.path.basename(path),
+                    "line": b.start_line,
+                }
+                tus.append(current)
+
+    manifest = []
+    for n, tu in enumerate(tus):
+        stem = f"tu_{n:03d}_{os.path.splitext(tu['src'])[0]}_{tu['line']}"
+        fname = os.path.join(args.out, stem + ".cu")
+        # Hoist all preprocessor directives (#include/#define/#pragma) to file
+        # scope, deduped in first-seen order, so function-wrapped segments
+        # never carry an #include into block scope (illegal for headers that
+        # define static __device__ functions).
+        hoisted = []
+        seen_hoist = set()
+        seg_render = []   # (kind, body_lines, line)
+        for kind, lines, line in tu["segs"]:
+            body = []
+            for ln in lines:
+                if ln.lstrip().startswith("#"):
+                    if ln not in seen_hoist:
+                        seen_hoist.add(ln)
+                        hoisted.append(ln)
+                else:
+                    body.append(ln)
+            if body:
+                seg_render.append((kind, body, line))
+
+        out = ['#include "fixture.cuh"\n']
+        out += hoisted
+        out.append("\n")
+        fn_count = 0
+        for kind, body, line in seg_render:
+            out.append(f"// --- from {tu['src']}:{line} ---\n")
+            if kind in ("global", "main"):
+                out += body
+            else:  # fn
+                out.append(f"static void _seg_{fn_count}() {{\n")
+                out += body
+                out.append("\n}\n")
+                fn_count += 1
+            out.append("\n")
+        with open(fname, "w", encoding="utf-8") as f:
+            f.writelines(out)
+        manifest.append((stem, tu["openvdb"]))
+
+    # Emit a manifest CMake can read: <stem> <needs_openvdb 0|1>
+    with open(os.path.join(args.out, "manifest.txt"), "w") as f:
+        for stem, ovdb in manifest:
+            f.write(f"{stem} {1 if ovdb else 0}\n")
+
+    n_ovdb = sum(1 for _, o in manifest if o)
+    print(f"extracted {len(manifest)} translation unit(s) "
+          f"({n_ovdb} need OpenVDB) into {args.out}")
+    return 0
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    p1 = sub.add_parser("check-refs", help="validate #include references")
+    p1.add_argument("md", nargs="+", help="markdown files")
+    p1.set_defaults(func=cmd_check_refs)
+
+    p2 = sub.add_parser("extract", help="extract compile-tagged blocks to .cu")
+    p2.add_argument("md", nargs="+", help="markdown files")
+    p2.add_argument("--out", required=True, help="output directory")
+    p2.set_defaults(func=cmd_extract)
+
+    args = ap.parse_args()
+    sys.exit(args.func(args))
+
+
+if __name__ == "__main__":
+    main()

doc/nanovdb/doc.md

@@ -6,3 +6,7 @@ Welcome to the NanoVDB documentation page.
 * @subpage NanoVDB_FAQ
 * @subpage NanoVDB_SourceTree
 * @subpage NanoVDB_HelloWorld
+
+For a hands-on introduction, point an LLM coding agent (Claude Code, Cursor, or
+similar) at the [TEACHME lessons](https://github.com/AcademySoftwareFoundation/openvdb/tree/master/doc/nanovdb/TEACHME)
+and ask it to teach you the NanoVDB API interactively.

nanovdb/nanovdb/CMakeLists.txt

@@ -31,6 +31,7 @@ option(NANOVDB_BUILD_PYTHON_MODULE "Build the nanovdb Python module" OFF)
 option(NANOVDB_USE_INTRINSICS "Build with hardware intrinsics support" OFF)
 option(NANOVDB_USE_CUDA "Build with CUDA support" OFF)
 option(NANOVDB_CUDA_KEEP_PTX "Keep CUDA PTX" OFF)
+option(NANOVDB_CUDA_WERROR "Treat all NVCC diagnostics as errors" ON)
 option(NANOVDB_USE_OLD_ACCESSOR "Use the old (buggy) ReadAccessor instead of the new fixed one" OFF)
 
 option(NANOVDB_USE_OPENVDB "Build with OpenVDB support" OFF)
@@ -96,6 +97,21 @@ if(NANOVDB_USE_CUDA)
 
   set(CMAKE_CUDA_FLAGS "${NANOVDB_CUDA_EXTENDED_LAMBDA} -use_fast_math -lineinfo ${CMAKE_CUDA_FLAGS}")
 
+  if(NANOVDB_CUDA_WERROR OR OPENVDB_CXX_STRICT)
+    # Device-side analog of the host -Werror configured in OpenVDBCXX.cmake:
+    # every nvcc front-end diagnostic becomes an error. NANOVDB_CUDA_WERROR
+    # enables it independently of the host-side strict flags.
+    set(CMAKE_CUDA_FLAGS "--Werror=all-warnings ${CMAKE_CUDA_FLAGS}")
+  endif()
+
+  if(WIN32)
+    # NVCC analog of the /wd4251 and /wd4275 suppressions in OpenVDBCXX.cmake:
+    # it's not possible to use STL types in DLL interfaces in a portable and
+    # reliable way, so the dll-interface diagnostics (#1394 and #1388) fire on
+    # every include of the OpenVDB core headers from a CUDA source.
+    set(CMAKE_CUDA_FLAGS "--diag-suppress=1388,1394 ${CMAKE_CUDA_FLAGS}")
+  endif()
+
   # workaround for win32 bug when nvcc "--keep" is used.
   if(WIN32)
     if(NANOVDB_CUDA_KEEP_PTX)

nanovdb/nanovdb/GridHandle.h

@@ -8,8 +8,9 @@
 
     \date January 8, 2020
 
-    \brief Defines GridHandle, which manages a host, and possibly a device,
-           memory buffer containing one or more NanoVDB grids.
+    \brief Defines GridHandle, which manages a memory buffer containing one
+           or more NanoVDB grids: host-resident, dual host/device, or a
+           single-space device-only buffer.
 */
 
 #ifndef NANOVDB_GRID_HANDLE_H_HAS_BEEN_INCLUDED
@@ -30,18 +31,63 @@ namespace nanovdb {
 
 struct GridHandleMetaData {uint64_t offset, size; GridType gridType;};
 
+namespace detail {
+
+/// @brief Allocates @c bytes of host-readable storage for a GridHandle:
+///        through @c BufferT::create for buffers that provide it, and through
+///        @c pool's resource for a host-accessible single-space buffer (e.g. a
+///        pinned-resource cuda::Buffer). The braced third argument selects the
+///        uninitialized-storage constructor without naming its tag type, so
+///        this header stays CUDA-free.
+template<typename BufferT>
+inline BufferT createHostStorage(uint64_t bytes, const BufferT& pool)
+{
+    if constexpr (BufferHasHostSingle<BufferT>::value && BufferHasStream<BufferT>::value) {
+        return BufferT(pool.stream(), pool.resource(), bytes, {});// stream-ordered resource: allocate on the pool's retained stream
+    } else if constexpr (BufferHasHostSingle<BufferT>::value) {
+        return BufferT(pool.resource(), bytes, {});
+    } else {
+        return BufferT::create(bytes, &pool);
+    }
+}
+
+}// namespace detail
+
 /// @brief This class serves to manage a buffer containing one or more NanoVDB Grids.
 ///
 /// @note  It is important to note that this class does NOT depend on OpenVDB.
 template<typename BufferT = HostBuffer>
 class GridHandle
 {
+    static_assert(!(BufferTraits<BufferT>::hasDeviceDual && BufferHasDeviceSingle<BufferT>::value),
+                  "a buffer cannot be both dual-space and single-space");
+    static_assert(!(BufferHasDeviceSingle<BufferT>::value || BufferHasHostSingle<BufferT>::value)
+                      || BufferHasByteElements<BufferT>::value,
+                  "GridHandle requires byte-addressed single-space storage, e.g. cuda::Buffer<std::byte, R>");
+
     std::vector<GridHandleMetaData> mMetaData;
     BufferT mBuffer;
 
     template <typename T>
     static T* no_const(const T* ptr) { return const_cast<T*>(ptr); }
 
+    /// @brief Shared lookup behind grid() and deviceGrid(): the @a n'th grid
+    ///        within @a base, or nullptr when @a base is null, @a n is out of
+    ///        range, or the value type does not match the grid.
+    template<typename ValueT>
+    const NanoGrid<ValueT>* gridAt(const void* base, uint32_t n) const
+    {
+        if (base == nullptr || n >= mMetaData.size() || mMetaData[n].gridType != toGridType<ValueT>()) return nullptr;
+        return util::PtrAdd<NanoGrid<ValueT>>(base, mMetaData[n].offset);
+    }
+
+    /// @brief Adopts a buffer whose metadata is already known, so a deep copy
+    ///        does not re-parse and a non-default-constructible buffer (e.g.
+    ///        over a ResourceRef) never needs default construction.
+    GridHandle(BufferT&& buffer, std::vector<GridHandleMetaData> meta)
+        : mMetaData(std::move(meta))
+        , mBuffer(std::move(buffer)) {}
+
 public:
     using BufferType = BufferT;
 
@@ -55,7 +101,15 @@ public:
     /// @brief  Move constructor from a host buffer
     /// @param buffer buffer containing one or more NanoGrids that will be moved into this GridHandle
     /// @throw Will throw and error with the buffer does not contain a valid NanoGrid!
-    template<typename T = BufferT, typename util::disable_if<BufferTraits<T>::hasDeviceDual, int>::type = 0>
+    template<typename T = BufferT, typename util::disable_if<BufferTraits<T>::hasDeviceDual || BufferHasDeviceSingle<T>::value, int>::type = 0>
+    GridHandle(T&& buffer);
+
+    /// @brief  Move constructor from a single-space device buffer: the grid
+    ///         metadata is read through the device on the buffer's stream.
+    /// @param buffer buffer containing one or more NanoGrids that will be moved into this GridHandle
+    /// @throw Will throw an error if the buffer does not contain a valid NanoGrid
+    /// @note The implementation of this template specialization is in nanovdb/cuda/GridHandle.cuh since it requires CUDA
+    template<typename T = BufferT, typename util::enable_if<BufferHasDeviceSingle<T>::value, int>::type = 0, typename = void>
     GridHandle(T&& buffer);
 
     /// @brief Constructs an empty GridHandle
@@ -65,14 +119,14 @@ public:
     GridHandle(const GridHandle&) = delete;
 
     /// @brief Move copy-constructor
-    GridHandle(GridHandle&& other) noexcept {
-        mBuffer   = std::move(other.mBuffer);
-        mMetaData = std::move(other.mMetaData);
-    }
+    GridHandle(GridHandle&& other) noexcept
+        : mMetaData(std::move(other.mMetaData))
+        , mBuffer(std::move(other.mBuffer)) {}
 
     /// @brief clear this GridHandle to an empty handle
     void reset() {
-        mBuffer.clear();
+        if constexpr (BufferHasDestroy<BufferT>::value) mBuffer.destroy();
+        else mBuffer.clear();
         mMetaData.clear();
     }
 
@@ -87,11 +141,25 @@ public:
     }
 
     /// @brief Performs a deep copy of the GridHandle, possibly templated on a different buffer type
-    /// @tparam OtherBufferT Buffer type of the deep copy
-    /// @param buffer optional buffer used for allocation
+    /// @tparam OtherBufferT Buffer type of the deep copy. A single-space device
+    ///         handle only copies to its own buffer type (device-to-device,
+    ///         ordered on the source's retained stream), so call it as
+    ///         copy<BufferT>() -- the default OtherBufferT does not compile.
+    /// @param buffer optional buffer used for allocation; a compile error for
+    ///        single-space handles, which allocate through the source buffer's
+    ///        resource (use the no-argument overload)
     /// @return A new handle of the specified buffer type that contains a deep copy of the current handle
     template <typename OtherBufferT = HostBuffer>
-    GridHandle<OtherBufferT> copy(const OtherBufferT& buffer = OtherBufferT()) const;
+    GridHandle<OtherBufferT> copy(const OtherBufferT& buffer) const;
+
+    /// @brief Deep copy without a pool argument. The single-space path
+    ///        allocates through the source buffer's resource and never
+    ///        constructs a pool buffer, so it works for buffers that are not
+    ///        default-constructible, e.g. over a ResourceRef. The host path
+    ///        default-constructs the pool argument, so it requires a
+    ///        default-constructible OtherBufferT.
+    template <typename OtherBufferT = HostBuffer>
+    GridHandle<OtherBufferT> copy() const;
 
     /// @brief Return a reference to the buffer
     BufferT&       buffer() { return mBuffer; }
@@ -99,13 +167,15 @@ public:
     /// @brief Return a const reference to the buffer
     const BufferT& buffer() const { return mBuffer; }
 
-    /// @brief Returns a non-const pointer to the data.
+    //@{
+    /// @brief Returns a pointer to the host data; not available for a
+    ///        single-space device buffer, which has no host-readable bytes.
     /// @warning Note that the return pointer can be NULL if the GridHandle was not initialized
+    template<typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     void* data() { return mBuffer.data(); }
-
-    /// @brief Returns a const pointer to the data.
-    /// @warning Note that the return pointer can be NULL if the GridHandle was not initialized
+    template<typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     const void* data() const { return mBuffer.data(); }
+    //@}
 
     template<typename U = BufferT>
     typename util::enable_if<BufferTraits<U>::hasDeviceDual, const void*>::type
@@ -120,6 +190,16 @@ public:
     typename util::enable_if<BufferTraits<U>::hasDeviceDual, void*>::type
     deviceData(int device) { return mBuffer.deviceData(device); }
 
+    //@{
+    /// @brief For a single-space buffer the device data is the buffer itself.
+    template<typename U = BufferT>
+    typename util::enable_if<BufferHasDeviceSingle<U>::value, const void*>::type
+    deviceData() const { return mBuffer.data(); }
+    template<typename U = BufferT>
+    typename util::enable_if<BufferHasDeviceSingle<U>::value, void*>::type
+    deviceData() { return mBuffer.data(); }
+    //@}
+
     //@{
     /// @brief Returns the size in bytes of the raw memory buffer managed by this GridHandle.
     [[deprecated("Use GridHandle::bufferSize instead.")]] uint64_t size() const { return mBuffer.size(); }
@@ -140,7 +220,7 @@ public:
     /// @param n Index of the (host) grid pointer to be returned
     /// @warning Note that the return pointer can be NULL if the GridHandle no host grid, @a n is invalid
     ///          or if the template parameter does not match the specified grid!
-    template<typename ValueT>
+    template<typename ValueT, typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     const NanoGrid<ValueT>* grid(uint32_t n = 0) const;
 
     /// @brief Returns a host pointer to the @a n'th  NanoVDB grid encoded in this GridHandle.
@@ -148,7 +228,7 @@ public:
     /// @param n Index of the (host) grid pointer to be returned
     /// @warning Note that the return pointer can be NULL if the GridHandle no host grid, @a n is invalid
     ///          or if the template parameter does not match the specified grid!
-    template<typename ValueT>
+    template<typename ValueT, typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     NanoGrid<ValueT>* grid(uint32_t n = 0) {return const_cast<NanoGrid<ValueT>*>(static_cast<const GridHandle*>(this)->template grid<ValueT>(n));}
 
     /// @brief Return a const pointer to the @a n'th grid encoded in this GridHandle on the device, e.g. GPU
@@ -169,6 +249,18 @@ public:
     typename util::enable_if<BufferTraits<U>::hasDeviceDual, NanoGrid<ValueT>*>::type
     deviceGrid(uint32_t n=0){return const_cast<NanoGrid<ValueT>*>(static_cast<const GridHandle*>(this)->template deviceGrid<ValueT>(n));}
 
+    //@{
+    /// @brief Return a pointer to the @a n'th grid of a single-space device buffer
+    /// @warning Note that the return pointer can be NULL if the GridHandle is empty, @a n is invalid,
+    ///          or if the template parameter does not match the specified grid.
+    template<typename ValueT, typename U = BufferT>
+    typename util::enable_if<BufferHasDeviceSingle<U>::value, const NanoGrid<ValueT>*>::type
+    deviceGrid(uint32_t n=0) const { return this->template gridAt<ValueT>(mBuffer.data(), n); }
+    template<typename ValueT, typename U = BufferT>
+    typename util::enable_if<BufferHasDeviceSingle<U>::value, NanoGrid<ValueT>*>::type
+    deviceGrid(uint32_t n=0){return const_cast<NanoGrid<ValueT>*>(static_cast<const GridHandle*>(this)->template deviceGrid<ValueT>(n));}
+    //@}
+
     /// @brief Upload the grid to the device, e.g. from CPU to GPU
     /// @note This method is only available if the buffer supports devices
     template<typename U = BufferT>
@@ -230,17 +322,21 @@ public:
     /// @brief Access to the GridData of the n'th grid in the current handle
     /// @param n zero-based ID of the grid
     /// @return Const pointer to the n'th GridData in the current handle
+    template<typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     const GridData* gridData(uint32_t n = 0) const;
 
     /// @brief Returns a const point to the @a n'th grid meta data
     /// @param n zero-based ID of the grid
     /// @warning Note that the return pointer can be NULL if the GridHandle was not initialized
+    template<typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     const GridMetaData* gridMetaData(uint32_t n = 0) const;
 
     /// @brief Write a specific grid in this buffer to an output stream
     /// @param os  output stream that the buffer will be written to
     /// @param n zero-based index of the grid to be written to stream
     void write(std::ostream& os, uint32_t n) const {
+        static_assert(!BufferHasDeviceSingle<BufferT>::value,
+                      "GridHandle::write requires host-accessible grids: cuda::copyTo a host-readable handle first");
         if (const GridData* data = this->gridData(n)) {
             os.write((const char*)data, data->mGridSize);
         } else {
@@ -251,6 +347,9 @@ public:
     /// @brief Write the entire grid buffer to an output stream
     /// @param os output stream that the buffer will be written to
     void write(std::ostream& os) const {
+        static_assert(!BufferHasDeviceSingle<BufferT>::value,
+                      "GridHandle::write requires host-accessible grids: cuda::copyTo a host-readable handle first");
+
         for (uint32_t n=0; n<this->gridCount(); ++n) this->write(os, n);
     }
 
@@ -328,6 +427,7 @@ public:
 // --------------------------> Implementation of private methods in GridHandle <------------------------------------
 
 template<typename BufferT>
+template<typename U, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type>
 inline const GridData* GridHandle<BufferT>::gridData(uint32_t n) const
 {
     const void *data = this->data();
@@ -336,6 +436,7 @@ inline const GridData* GridHandle<BufferT>::gridData(uint32_t n) const
 }// const GridData* GridHandle<BufferT>::gridData(uint32_t n) const
 
 template<typename BufferT>
+template<typename U, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type>
 inline const GridMetaData* GridHandle<BufferT>::gridMetaData(uint32_t n) const
 {
     const auto *data = this->data();
@@ -355,11 +456,11 @@ inline __hostdev__ void cpyGridHandleMeta(const GridData *data, GridHandleMetaDa
 
 // template specialization of move constructor from a host buffer
 template<typename BufferT>
-template<typename T, typename util::disable_if<BufferTraits<T>::hasDeviceDual, int>::type>
+template<typename T, typename util::disable_if<BufferTraits<T>::hasDeviceDual || BufferHasDeviceSingle<T>::value, int>::type>
 GridHandle<BufferT>::GridHandle(T&& buffer)
+    : mBuffer(std::move(buffer))
 {
     static_assert(util::is_same<T,BufferT>::value, "Expected U==BufferT");
-    mBuffer = std::move(buffer);
     if (auto *data = reinterpret_cast<const GridData*>(mBuffer.data())) {
         if (!data->isValid()) throw std::runtime_error("GridHandle was constructed with an invalid host buffer");
         mMetaData.resize(data->mGridCount);
@@ -371,19 +472,38 @@ template<typename BufferT>
 template <typename OtherBufferT>
 inline GridHandle<OtherBufferT> GridHandle<BufferT>::copy(const OtherBufferT& other) const
 {
-    if (mBuffer.isEmpty()) return GridHandle<OtherBufferT>();// return an empty handle
-    auto buffer = OtherBufferT::create(mBuffer.size(), &other);
+    static_assert(!(BufferHasDeviceSingle<BufferT>::value || BufferHasDeviceSingle<OtherBufferT>::value),
+                  "GridHandle::copy(pool) cannot honor a pool argument for a single-space device buffer, "
+                  "whose copy allocates through the source buffer's resource: use the no-argument copy()");
+    if (mBuffer.size() == 0) return GridHandle<OtherBufferT>();// return an empty handle
+    auto buffer = detail::createHostStorage<OtherBufferT>(mBuffer.size(), other);
     std::memcpy(buffer.data(), mBuffer.data(), mBuffer.size());// deep copy of buffer
     return GridHandle<OtherBufferT>(std::move(buffer));
 }// GridHandle<OtherBufferT> GridHandle<BufferT>::copy(const OtherBufferT& other) const
 
 template<typename BufferT>
-template<typename ValueT>
+template <typename OtherBufferT>
+inline GridHandle<OtherBufferT> GridHandle<BufferT>::copy() const
+{
+    if constexpr (BufferHasDeviceSingle<BufferT>::value || BufferHasDeviceSingle<OtherBufferT>::value) {
+        static_assert(util::is_same<OtherBufferT, BufferT>::value && BufferHasDeviceSingle<BufferT>::value,
+                      "GridHandle::copy is same-space only: a single-space device handle copies to its own "
+                      "buffer type; use cuda::copyTo (cuda/GridHandle.cuh) to move grids across address spaces");
+        // Device-to-device deep copy; for a stream-ordered resource it is
+        // ordered on the source's retained stream, so synchronize that stream
+        // before reading the result. Metadata is host-resident, so the copy
+        // adopts it directly with no device re-parse.
+        return GridHandle(mBuffer.copy(), mMetaData);
+    } else {
+        return this->copy(OtherBufferT());
+    }
+}// GridHandle<OtherBufferT> GridHandle<BufferT>::copy() const
+
+template<typename BufferT>
+template<typename ValueT, typename U, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type>
 inline const NanoGrid<ValueT>* GridHandle<BufferT>::grid(uint32_t n) const
 {
-    const void *data = mBuffer.data();
-    if (data == nullptr || n >= mMetaData.size() || mMetaData[n].gridType != toGridType<ValueT>()) return nullptr;
-    return util::PtrAdd<NanoGrid<ValueT>>(data, mMetaData[n].offset);
+    return this->template gridAt<ValueT>(mBuffer.data(), n);
 }// const NanoGrid<ValueT>* GridHandle<BufferT>::grid(uint32_t n) const
 
 template<typename BufferT>
@@ -391,14 +511,14 @@ template<typename ValueT, typename U>
 inline typename util::enable_if<BufferTraits<U>::hasDeviceDual, const NanoGrid<ValueT>*>::type
 GridHandle<BufferT>::deviceGrid(uint32_t n) const
 {
-    const void *data = mBuffer.deviceData();
-    if (data == nullptr || n >= mMetaData.size() || mMetaData[n].gridType != toGridType<ValueT>()) return nullptr;
-    return util::PtrAdd<NanoGrid<ValueT>>(data, mMetaData[n].offset);
+    return this->template gridAt<ValueT>(mBuffer.deviceData(), n);
 }// GridHandle<BufferT>::deviceGrid(uint32_t n) cons
 
 template<typename BufferT>
 void GridHandle<BufferT>::read(std::istream& is, const BufferT& pool)
 {
+    static_assert(!BufferHasDeviceSingle<BufferT>::value,
+                  "GridHandle::read requires a host-accessible buffer: read into a host-readable handle, then cuda::copyTo");
     const std::streampos start = is.tellg();// remember where the raw buffer begins
     GridData data;
     is.read((char*)&data, sizeof(GridData));
@@ -409,7 +529,7 @@ void GridHandle<BufferT>::read(std::istream& is, const BufferT& pool)
             is.read((char*)&data, sizeof(GridData));
             sum += data.mGridSize;
         }
-        auto buffer = BufferT::create(size + sum, &pool);
+        auto buffer = detail::createHostStorage(size + sum, pool);
         is.seekg(start);// rewind to the start of the raw buffer
         is.read((char*)(buffer.data()), buffer.size());
         *this = GridHandle(std::move(buffer));
@@ -422,6 +542,8 @@ void GridHandle<BufferT>::read(std::istream& is, const BufferT& pool)
 template<typename BufferT>
 void GridHandle<BufferT>::read(std::istream& is, uint32_t n, const BufferT& pool)
 {
+    static_assert(!BufferHasDeviceSingle<BufferT>::value,
+                  "GridHandle::read requires a host-accessible buffer: read into a host-readable handle, then cuda::copyTo");
     GridData data;
     is.read((char*)&data, sizeof(GridData));
     if (data.isValid()) {
@@ -430,7 +552,7 @@ void GridHandle<BufferT>::read(std::istream& is, uint32_t n, const BufferT& pool
             is.seekg(data.mGridSize - sizeof(GridData), std::ios::cur);// skip grid
             is.read((char*)&data, sizeof(GridData));
         }
-        auto buffer = BufferT::create(data.mGridSize, &pool);
+        auto buffer = detail::createHostStorage(data.mGridSize, pool);
         is.seekg(-sizeof(GridData), std::ios::cur);// rewind
         is.read((char*)(buffer.data()), data.mGridSize);
         tools::updateGridCount((GridData*)buffer.data(), 0u, 1u);
@@ -444,6 +566,8 @@ void GridHandle<BufferT>::read(std::istream& is, uint32_t n, const BufferT& pool
 template<typename BufferT>
 void GridHandle<BufferT>::read(std::istream& is, const std::string &gridName, const BufferT& pool)
 {
+    static_assert(!BufferHasDeviceSingle<BufferT>::value,
+                  "GridHandle::read requires a host-accessible buffer: read into a host-readable handle, then cuda::copyTo");
     static const std::streamsize byteSize = sizeof(GridData);
     GridData data;
     is.read((char*)&data, byteSize);
@@ -456,7 +580,7 @@ void GridHandle<BufferT>::read(std::istream& is, const std::string &gridName, co
             is.seekg(-byteSize, std::ios::cur);// rewind
         }
         if (n>data.mGridCount) throw std::runtime_error("No raw grid named \""+gridName+"\"");
-        auto buffer = BufferT::create(data.mGridSize, &pool);
+        auto buffer = detail::createHostStorage(data.mGridSize, pool);
         is.read((char*)(buffer.data()), data.mGridSize);
         tools::updateGridCount((GridData*)buffer.data(), 0u, 1u);
         *this = GridHandle(std::move(buffer));
@@ -476,6 +600,10 @@ template<typename BufferT, template <class, class...> class VectorT = std::vecto
 inline VectorT<GridHandle<BufferT>>
 splitGrids(const GridHandle<BufferT> &handle, const BufferT* other = nullptr)
 {
+    static_assert(!BufferHasDeviceSingle<BufferT>::value,
+                  "splitGrids requires a buffer type providing create(): cuda::copyTo a HostBuffer handle first");
+    static_assert(!BufferHasHostSingle<BufferT>::value,
+                  "splitGrids requires a buffer type providing create(): copy the handle to a HostBuffer first");
     using HandleT = GridHandle<BufferT>;
     const void *ptr = handle.data();
     if (ptr == nullptr) return VectorT<HandleT>();
@@ -493,6 +621,51 @@ splitGrids(const GridHandle<BufferT> &handle, const BufferT* other = nullptr)
     return handles;
 }// splitGrids
 
+/// @brief Combines (or merges) multiple GridHandles into a single GridHandle containing all grids
+/// @tparam BufferT Type of the input and output grid buffers
+/// @param handles array of non-owning pointers to the GridHandles to be combined
+/// @param count number of pointers in @a handles
+/// @param pool optional pool used for allocation of output GridHandle
+/// @return single GridHandle containing all input grids
+/// @note This overload borrows the input handles, so it can be used when the
+///       handles are owned elsewhere (e.g. by a language binding) and can be
+///       neither moved from nor cheaply copied.
+template<typename BufferT>
+inline GridHandle<BufferT>
+mergeGrids(const GridHandle<BufferT>* const* handles, size_t count, const BufferT* pool = nullptr)
+{
+    static_assert(!BufferHasDeviceSingle<BufferT>::value,
+                  "mergeGrids requires a buffer type providing create(): cuda::copyTo HostBuffer handles first");
+    static_assert(!BufferHasHostSingle<BufferT>::value,
+                  "mergeGrids requires a buffer type providing create(): copy the handles to HostBuffer first");
+    uint64_t size = 0u;
+    uint32_t counter = 0u, gridCount = 0u;
+    for (size_t i = 0; i < count; ++i) {
+        gridCount += handles[i]->gridCount();
+        for (uint32_t n=0; n<handles[i]->gridCount(); ++n) size += handles[i]->gridSize(n);
+    }
+    // Return an empty handle when there is nothing to merge: BufferT::create(0)
+    // can produce a non-null zero-byte region, from which the GridHandle
+    // constructor would attempt to read a GridData header.
+    if (gridCount == 0u) return GridHandle<BufferT>();
+    auto buffer = BufferT::create(size, pool);
+    void *dst = buffer.data();
+    for (size_t i = 0; i < count; ++i) {
+        const GridHandle<BufferT> &h = *handles[i];
+        for (uint32_t n=0; n<h.gridCount(); ++n) {
+            // gridData(n) applies the per-grid offset, so padded (non-contiguous)
+            // source buffers are copied correctly.
+            const GridData *src = h.gridData(n);
+            NANOVDB_ASSERT(src && src->isValid());
+            const uint64_t gridSize = h.gridSize(n);
+            std::memcpy(dst, src, gridSize);
+            tools::updateGridCount(reinterpret_cast<GridData*>(dst), counter++, gridCount);
+            dst = util::PtrAdd(dst, gridSize);
+        }
+    }
+    return GridHandle<BufferT>(std::move(buffer));
+}// mergeGrids
+
 /// @brief Combines (or merges) multiple GridHandles into a single GridHandle containing all grids
 /// @tparam BufferT Type of the input and output grid buffers
 /// @param handles Vector of GridHandles to be combined
@@ -502,26 +675,10 @@ template<typename BufferT, template <class, class...> class VectorT>
 inline GridHandle<BufferT>
 mergeGrids(const VectorT<GridHandle<BufferT>> &handles, const BufferT* pool = nullptr)
 {
-    uint64_t size = 0u;
-    uint32_t counter = 0u, gridCount = 0u;
-    for (auto &h : handles) {
-        gridCount += h.gridCount();
-        for (uint32_t n=0; n<h.gridCount(); ++n) size += h.gridSize(n);
-    }
-    auto buffer = BufferT::create(size, pool);
-    void *dst = buffer.data();
-    for (auto &h : handles) {
-        const void *src = h.data();
-        for (uint32_t n=0; n<h.gridCount(); ++n) {
-            std::memcpy(dst, src, h.gridSize(n));
-            GridData *data = reinterpret_cast<GridData*>(dst);
-            NANOVDB_ASSERT(data->isValid());
-            tools::updateGridCount(data, counter++, gridCount);
-            dst = util::PtrAdd(dst, data->mGridSize);
-            src = util::PtrAdd(src, data->mGridSize);
-        }
-    }
-    return GridHandle<BufferT>(std::move(buffer));
+    std::vector<const GridHandle<BufferT>*> ptrs;
+    ptrs.reserve(handles.size());
+    for (auto &h : handles) ptrs.push_back(&h);
+    return mergeGrids<BufferT>(ptrs.data(), ptrs.size(), pool);
 }// mergeGrids
 
 } // namespace nanovdb

nanovdb/nanovdb/HostBuffer.h

@@ -87,6 +87,7 @@
 #include <cassert>//           for assert
 #include <sstream>//           for std::stringstream
 #include <cstring>//           for memcpy
+#include <type_traits>//       for std::void_t
 
 #define checkPtr(ptr, msg) \
     { \
@@ -98,9 +99,57 @@ namespace nanovdb {
 template<typename BufferT>
 struct BufferTraits
 {
-    static constexpr bool hasDeviceDual = false;
+    static constexpr bool hasDeviceDual   = false;
+    static constexpr bool hasDeviceSingle = false;
 };
 
+/// @brief Detects whether @c BufferTraits<BufferT> defines @c hasDeviceSingle,
+///        i.e. whether the buffer manages a single device-resident allocation.
+/// @details Defaults to false when the trait member is absent, so pre-existing
+///          BufferTraits specializations (in or out of tree) that only define
+///          hasDeviceDual keep compiling unchanged.
+template<typename BufferT, typename = void>
+struct BufferHasDeviceSingle { static constexpr bool value = false; };
+template<typename BufferT>
+struct BufferHasDeviceSingle<BufferT, std::void_t<decltype(BufferTraits<BufferT>::hasDeviceSingle)>>
+{ static constexpr bool value = BufferTraits<BufferT>::hasDeviceSingle; };
+
+/// @brief Companion detection for BufferTraits<...>::hasHostSingle: a
+///        single-space buffer whose storage is host-accessible (e.g. a
+///        pinned-resource cuda::Buffer).
+template<typename BufferT, typename = void>
+struct BufferHasHostSingle { static constexpr bool value = false; };
+template<typename BufferT>
+struct BufferHasHostSingle<BufferT, std::void_t<decltype(BufferTraits<BufferT>::hasHostSingle)>>
+{ static constexpr bool value = BufferTraits<BufferT>::hasHostSingle; };
+
+/// @brief Detects whether a buffer exposes a retained stream (a stream()
+///        member), i.e. whether its resource is stream-ordered. Used to pick
+///        the buffer's stream-taking constructor without naming CUDA types.
+template<typename BufferT, typename = void>
+struct BufferHasStream { static constexpr bool value = false; };
+template<typename BufferT>
+struct BufferHasStream<BufferT, std::void_t<decltype(std::declval<const BufferT&>().stream())>>
+{ static constexpr bool value = true; };
+
+/// @brief Detects whether a buffer's elements are byte-sized. Buffers that
+///        expose no ElementType (e.g. HostBuffer) address raw bytes by
+///        definition, so the primary defaults to true.
+template<typename BufferT, typename = void>
+struct BufferHasByteElements { static constexpr bool value = true; };
+template<typename BufferT>
+struct BufferHasByteElements<BufferT, std::void_t<typename BufferT::ElementType>>
+{ static constexpr bool value = sizeof(typename BufferT::ElementType) == 1; };
+
+/// @brief Detects whether a buffer provides destroy(), the cuda::Buffer
+///        spelling for releasing its storage. Handle reset() dispatches to it
+///        when present and falls back to the legacy clear() otherwise.
+template<typename BufferT, typename = void>
+struct BufferHasDestroy { static constexpr bool value = false; };
+template<typename BufferT>
+struct BufferHasDestroy<BufferT, std::void_t<decltype(std::declval<BufferT&>().destroy())>>
+{ static constexpr bool value = true; };
+
 // ----------------------------> HostBuffer <--------------------------------------
 
 /// @brief This is a buffer that contains a shared or private pool

nanovdb/nanovdb/NanoVDB.h

@@ -1808,6 +1808,42 @@ struct ProbeValue;
 template<typename BuildT>
 struct GetNodeInfo;
 
+/// @brief Result of a fused ReadAccessor::getDimAndActive traversal: the
+/// getDim(ijk, ray) step size and the policy-dependent active state of @a ijk.
+///
+/// The low 31 bits store @c dim and bit 31 stores @c active, saving one live
+/// return register versus {uint32_t, bool}. The default FloatTree's upper-node
+/// maximum is 4096. The constructor supports `return {dim, active}`; dim() and
+/// active() decode the packed value.
+struct DimAndActive
+{
+    uint32_t packed{0u};
+
+    __hostdev__ DimAndActive() : packed(0u) {}
+    __hostdev__ DimAndActive(uint32_t d, bool a)
+        : packed((d & 0x7FFFFFFFu) | (uint32_t(a) << 31))
+    {
+        NANOVDB_ASSERT(d < (uint32_t(1) << 31)); // dim must fit in 31 bits (bit 31 = active)
+    }
+
+    __hostdev__ uint32_t dim()    const { return packed & 0x7FFFFFFFu; }
+    __hostdev__ bool     active() const { return (packed >> 31) != 0u; }
+};
+
+/// @brief Policy that exactly matches separate getDim and isActive calls. A
+/// skip flag still controls @c dim but cannot stop the active-state descent:
+/// GridStats flags value range versus render delta, not activeness. This costs
+/// a full descent through skipped InternalNodes; use it when @c active is read
+/// unconditionally, such as by HDDA iterators observing upper-level activeness.
+struct ActiveExact {};
+
+/// @brief Policy that preserves getDim's skip-flag fast path. If an
+/// InternalNode short-circuits, @c active is unspecified and must be gated on
+/// @c dim. Otherwise the leaf-level active read is fused into the descent. Use
+/// for logic such as `if (hdda.dim() > 1 || !active) continue;`, where the
+/// short-circuit ensures @c active is read only after a leaf-level descent.
+struct ActiveOnLeafOnly {};
+
 // ----------------------------> CheckMode <----------------------------------
 
 /// @brief List of different modes for computing for a checksum
@@ -3136,6 +3172,23 @@ private:
         return ChildNodeType::dim(); // background
     }
 
+    /// @brief Returns dim and active state in one traversal, sharing cache and
+    /// terminal tile/mask reads. Root has no skip flag, so @c Policy affects
+    /// only the child descent; see ActiveExact and ActiveOnLeafOnly.
+    template<typename Policy = ActiveExact, typename RayT, typename AccT>
+    __hostdev__ DimAndActive getDimAndActiveAndCache(const CoordType& ijk, const RayT& ray, const AccT& acc) const
+    {
+        if (const Tile* tile = this->probeTile(ijk)) {
+            if (tile->isChild()) {
+                const auto* child = this->getChild(tile);
+                acc.insert(ijk, child);
+                return child->template getDimAndActiveAndCache<Policy>(ijk, ray, acc);
+            }
+            return {uint32_t(1) << ChildT::TOTAL, tile->state > 0u}; // tile value
+        }
+        return {ChildNodeType::dim(), false}; // background is never active
+    }
+
     template<typename OpT, typename AccT, typename... ArgsT>
     __hostdev__ typename OpT::Type getAndCache(const CoordType& ijk, const AccT& acc, ArgsT&&... args) const
     {
@@ -3632,6 +3685,39 @@ private:
         return ChildNodeType::dim(); // tile value
     }
 
+    /// @brief Returns dim and active state in one traversal. On a skip flag,
+    /// ActiveExact resolves the true active state but returns this node's dim;
+    /// ActiveOnLeafOnly returns {dim, false} immediately. Without a skip flag
+    /// both policies are identical. See GridStats::setFlag.
+    template<typename Policy = ActiveExact, typename RayT, typename AccT>
+    __hostdev__ DimAndActive getDimAndActiveAndCache(const CoordType& ijk, const RayT& ray, const AccT& acc) const
+    {
+        //if (!ray.intersects( this->bbox() )) return {1<<TOTAL, ...};
+        // Separate paths keep `skip` out of ActiveOnLeafOnly's recursion,
+        // saving ~2 registers/thread and avoiding an occupancy cliff.
+        if constexpr (util::is_same<Policy, ActiveOnLeafOnly>::value) {
+            if (DataType::mFlags & uint32_t(1u))
+                return {this->dim(), false}; // active unspecified per policy
+            const uint32_t n = CoordToOffset(ijk);
+            if (DataType::mChildMask.isOn(n)) {
+                const ChildT* child = this->getChild(n);
+                acc.insert(ijk, child);
+                return child->template getDimAndActiveAndCache<Policy>(ijk, ray, acc);
+            }
+            return {ChildNodeType::dim(), DataType::mValueMask.isOn(n)};
+        } else {
+            const bool     skip = (DataType::mFlags & uint32_t(1u));
+            const uint32_t n    = CoordToOffset(ijk);
+            if (DataType::mChildMask.isOn(n)) {
+                const ChildT* child = this->getChild(n);
+                acc.insert(ijk, child);
+                const auto r = child->template getDimAndActiveAndCache<Policy>(ijk, ray, acc);
+                return {skip ? this->dim() : r.dim(), r.active()};
+            }
+            return {skip ? this->dim() : ChildNodeType::dim(), DataType::mValueMask.isOn(n)};
+        }
+    }
+
     template<typename OpT, typename AccT, typename... ArgsT>
     __hostdev__ typename OpT::Type getAndCache(const CoordType& ijk, const AccT& acc, ArgsT&&... args) const
     {
@@ -4572,6 +4658,21 @@ private:
         return ChildNodeType::dim();
     }
 
+    /// @brief Returns dim and active state in one traversal. @c Policy is
+    /// accepted for recursion but does not change leaf behavior: the skip flag
+    /// controls dim while active always comes from mValueMask. Branching here
+    /// raised parallelForKernel from 64 to 66 registers/thread and reduced
+    /// occupancy from four to three blocks/SM, negating the descent savings;
+    /// leaf skip flags are rare because upper-level flags usually short-circuit.
+    template<typename Policy = ActiveExact, typename RayT, typename AccT>
+    __hostdev__ DimAndActive getDimAndActiveAndCache(const CoordT& ijk, const RayT& /*ray*/, const AccT&) const
+    {
+        //if (!ray.intersects( this->bbox() )) return {1 << LOG2DIM, ...};
+        const uint32_t n      = CoordToOffset(ijk);
+        const uint32_t dimOut = (DataType::mFlags & uint8_t(1u)) ? this->dim() : ChildNodeType::dim();
+        return {dimOut, DataType::mValueMask.isOn(n)};
+    }
+
     template<typename OpT, typename AccT, typename... ArgsT>
     __hostdev__ auto
     //__hostdev__  decltype(OpT::get(util::declval<const LeafNode&>(), util::declval<uint32_t>(), util::declval<ArgsT>()...))
@@ -4878,6 +4979,15 @@ public:
     {
         return mRoot->getDimAndCache(ijk, ray, *this);
     }
+    /// @brief Fused getDim + isActive: one tree walk returns both the HDDA
+    /// step size and the active state of @a ijk, shaving the second descent
+    /// / cache-check / leaf ValueMask lookup. See @c ActiveExact /
+    /// @c ActiveOnLeafOnly for the @c Policy contract.
+    template<typename Policy = ActiveExact, typename RayT>
+    __hostdev__ DimAndActive getDimAndActive(const CoordType& ijk, const RayT& ray) const
+    {
+        return mRoot->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+    }
     template<typename OpT, typename... ArgsT>
     __hostdev__ auto get(const CoordType& ijk, ArgsT&&... args) const
     {
@@ -4993,6 +5103,17 @@ public:
         return mRoot->getDimAndCache(ijk, ray, *this);
     }
 
+    /// @brief Fused getDim + isActive: one tree walk returns both the HDDA
+    /// step size and the active state of @a ijk, shaving the second descent
+    /// / cache-check / leaf ValueMask lookup. See @c ActiveExact /
+    /// @c ActiveOnLeafOnly for the @c Policy contract.
+    template<typename Policy = ActiveExact, typename RayT>
+    __hostdev__ DimAndActive getDimAndActive(const CoordType& ijk, const RayT& ray) const
+    {
+        if (this->isCached(ijk)) return mNode->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+        return mRoot->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+    }
+
     template<typename OpT, typename... ArgsT>
     __hostdev__ typename OpT::Type get(const CoordType& ijk, ArgsT&&... args) const
     {
@@ -5175,6 +5296,26 @@ public:
         return mRoot->getDimAndCache(ijk, ray, *this);
     }
 
+    /// @brief Fused getDim + isActive: one tree walk returns both the HDDA
+    /// step size and the active state of @a ijk, shaving the second descent
+    /// / cache-check / leaf ValueMask lookup. See @c ActiveExact /
+    /// @c ActiveOnLeafOnly for the @c Policy contract.
+    template<typename Policy = ActiveExact, typename RayT>
+    __hostdev__ DimAndActive getDimAndActive(const CoordType& ijk, const RayT& ray) const
+    {
+#ifdef NANOVDB_USE_SINGLE_ACCESSOR_KEY
+        const CoordValueType dirty = this->computeDirty(ijk);
+#else
+        auto&& dirty = ijk;
+#endif
+        if (this->isCached1(dirty)) {
+            return mNode1->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+        } else if (this->isCached2(dirty)) {
+            return mNode2->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+        }
+        return mRoot->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+    }
+
     template<typename OpT, typename... ArgsT>
     __hostdev__ typename OpT::Type get(const CoordType& ijk, ArgsT&&... args) const
     {
@@ -5446,6 +5587,28 @@ public:
         return mRoot->getDimAndCache(ijk, ray, *this);
     }
 
+    /// @brief Fused getDim + isActive: one tree walk returns both the HDDA
+    /// step size and the active state of @a ijk, shaving the second descent
+    /// / cache-check / leaf ValueMask lookup. See @c ActiveExact /
+    /// @c ActiveOnLeafOnly for the @c Policy contract.
+    template<typename Policy = ActiveExact, typename RayT>
+    __hostdev__ DimAndActive getDimAndActive(const CoordType& ijk, const RayT& ray) const
+    {
+#ifdef NANOVDB_USE_SINGLE_ACCESSOR_KEY
+        const CoordValueType dirty = this->computeDirty(ijk);
+#else
+        auto&& dirty = ijk;
+#endif
+        if (this->isCached<LeafT>(dirty)) {
+            return ((LeafT*)mNode[0])->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+        } else if (this->isCached<NodeT1>(dirty)) {
+            return ((NodeT1*)mNode[1])->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+        } else if (this->isCached<NodeT2>(dirty)) {
+            return ((NodeT2*)mNode[2])->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+        }
+        return mRoot->template getDimAndActiveAndCache<Policy>(ijk, ray, *this);
+    }
+
 private:
     /// @brief Allow nodes to insert themselves into the cache.
     template<typename>

nanovdb/nanovdb/NodeManager.h

@@ -53,10 +53,14 @@ struct NodeManagerData
 template<typename BufferT>
 class NodeManagerHandle
 {
+    static_assert(!(BufferHasDeviceSingle<BufferT>::value || BufferHasHostSingle<BufferT>::value)
+                      || BufferHasByteElements<BufferT>::value,
+                  "NodeManagerHandle requires byte-addressed single-space storage, e.g. cuda::Buffer<std::byte, R>");
+
     GridType mGridType{GridType::Unknown};
     BufferT  mBuffer;
 
-    template<typename BuildT>
+    template<typename BuildT, typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     const NodeManager<BuildT>* getMgr() const {
         return mGridType == toGridType<BuildT>() ? (const NodeManager<BuildT>*)mBuffer.data() : nullptr;
     }
@@ -72,7 +76,9 @@ class NodeManagerHandle
 
 public:
     /// @brief Move constructor from a buffer
-    NodeManagerHandle(GridType gridType, BufferT&& buffer) : mGridType(gridType) { mBuffer = std::move(buffer); }
+    NodeManagerHandle(GridType gridType, BufferT&& buffer)
+        : mGridType(gridType)
+        , mBuffer(std::move(buffer)) {}// member-initialized, so a non-default-constructible buffer works
     /// @brief Empty ctor
     NodeManagerHandle() = default;
     /// @brief Disallow copy-construction
@@ -87,15 +93,19 @@ public:
         return *this;
     }
     /// @brief Move copy-constructor
-    NodeManagerHandle(NodeManagerHandle&& other) noexcept {
-        mGridType = other.mGridType;
-        mBuffer = std::move(other.mBuffer);
+    NodeManagerHandle(NodeManagerHandle&& other) noexcept
+        : mGridType(other.mGridType)
+        , mBuffer(std::move(other.mBuffer))
+    {
         other.mGridType = GridType::Unknown;
     }
     /// @brief Default destructor
     ~NodeManagerHandle() { this->reset(); }
-    /// @brief clear the buffer
-    void reset() { mBuffer.clear(); }
+    /// @brief Release the buffer's storage
+    void reset() {
+        if constexpr (BufferHasDestroy<BufferT>::value) mBuffer.destroy();
+        else mBuffer.clear();
+    }
 
     /// @brief Return a reference to the buffer
     BufferT& buffer() { return mBuffer; }
@@ -103,15 +113,15 @@ public:
     /// @brief Return a const reference to the buffer
     const BufferT& buffer() const { return mBuffer; }
 
-    /// @brief Returns a non-const pointer to the data.
-    ///
+    //@{
+    /// @brief Returns a pointer to the host data; not available for a
+    ///        single-space device buffer, which has no host-readable bytes.
     /// @warning Note that the return pointer can be NULL if the NodeManagerHandle was not initialized
+    template<typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     void* data() { return mBuffer.data(); }
-
-    /// @brief Returns a const pointer to the data.
-    ///
-    /// @warning Note that the return pointer can be NULL if the NodeManagerHandle was not initialized
+    template<typename U = BufferT, typename util::disable_if<BufferHasDeviceSingle<U>::value, int>::type = 0>
     const void* data() const { return mBuffer.data(); }
+    //@}
 
     /// @brief Returns the size in bytes of the raw memory buffer managed by this NodeManagerHandle's allocator.
     uint64_t size() const { return mBuffer.size(); }
@@ -142,6 +152,19 @@ public:
     typename util::enable_if<BufferTraits<U>::hasDeviceDual, NodeManager<BuildT>*>::type
     deviceMgr() { return no_const(this->template getDeviceMgr<BuildT>()); }
 
+    //@{
+    /// @brief Return a pointer to the NodeManager of a single-space device buffer
+    /// @warning Note that the return pointer can be NULL if the template parameter does not match the specified grid!
+    template<typename BuildT, typename U = BufferT>
+    typename util::enable_if<BufferHasDeviceSingle<U>::value, const NodeManager<BuildT>*>::type
+    deviceMgr() const {
+        return mGridType == toGridType<BuildT>() ? (const NodeManager<BuildT>*)mBuffer.data() : nullptr;
+    }
+    template<typename BuildT, typename U = BufferT>
+    typename util::enable_if<BufferHasDeviceSingle<U>::value, NodeManager<BuildT>*>::type
+    deviceMgr() { return no_const(static_cast<const NodeManagerHandle*>(this)->template deviceMgr<BuildT>()); }
+    //@}
+
     /// @brief Upload the NodeManager to the device, e.g. from CPU to GPU
     ///
     /// @note This method is only available if the buffer supports devices

nanovdb/nanovdb/Readme.md

@@ -7,6 +7,7 @@ A lightweight GPU friendly version of VDB initially targeting rendering applicat
 * [Frequently asked questions](../../doc/nanovdb/FAQ.md)
 * [Source tree](../../doc/nanovdb/SourceTree.md)
 * [Examples](../../doc/nanovdb/HelloWorld.md)
+* [TEACHME — learn NanoVDB by prompting an LLM coding agent](../../doc/nanovdb/TEACHME/README.md)
 * [Python examples](python/examples/)
 
 ### Copyright Contributors to the OpenVDB Project

nanovdb/nanovdb/cuda/Buffer.h

@@ -0,0 +1,465 @@
+// Copyright Contributors to the OpenVDB Project
+// SPDX-License-Identifier: Apache-2.0
+//
+/// @file nanovdb/cuda/Buffer.h
+///
+/// @brief Typed containers for CUDA memory: the owning, resource-aware,
+///        stream-ordered cuda::Buffer and the non-owning cuda::BufferView.
+
+#ifndef NANOVDB_CUDA_BUFFER_H_HAS_BEEN_INCLUDED
+#define NANOVDB_CUDA_BUFFER_H_HAS_BEEN_INCLUDED
+
+#include <cuda_runtime_api.h>
+#include <nanovdb/cuda/DeviceResource.h>
+#include <nanovdb/util/cuda/Util.h>
+
+#include <cstddef>
+#include <limits>
+#include <stdexcept>
+#include <type_traits>
+#include <utility>
+
+namespace nanovdb {
+
+namespace cuda {
+
+/// @brief Tag type selecting the Buffer constructors that skip element
+///        initialization, leaving the contents indeterminate.
+struct NoInit {};
+inline constexpr NoInit noInit{};
+
+namespace detail {
+
+/// @brief Conditional stream storage for Buffer. The async specialization
+///        retains the stream of the most recent allocation; the synchronous
+///        specialization is an empty base, so a Buffer over a synchronous
+///        resource carries no stream state and exposes no stream API.
+template<bool IsAsync>
+struct StreamHolder {};
+
+template<>
+struct StreamHolder<true> { cudaStream_t mStream = 0; };
+
+} // namespace detail
+
+/// @brief Owning, typed container of @c T elements allocated from a memory
+///        resource @c R held by value.
+/// @tparam T element type; sizes are expressed in elements, not bytes
+/// @tparam R memory resource, either stream-ordered (AsyncResource concept,
+///         see is_async_resource) or synchronous (Resource concept, see
+///         is_resource). When @c R provides both interfaces the stream-ordered
+///         one is used.
+/// @details With a stream-ordered resource the Buffer retains the stream of
+///          the most recent allocation (or the one supplied via set_stream)
+///          and orders its deallocation on that stream. Buffer is move-only.
+template<typename T, typename R = DeviceResource>
+class Buffer : private detail::StreamHolder<is_async_resource<R>::value>
+{
+    static_assert(is_async_resource<R>::value || is_resource<R>::value,
+                  "Buffer requires R to model the AsyncResource or the Resource concept");
+    static_assert(std::is_trivially_copyable<T>::value,
+                  "Buffer requires a trivially copyable T: elements are copied bytewise and never constructed or destroyed");
+
+    static constexpr bool IsAsync = is_async_resource<R>::value;
+
+public:
+    /// @brief Element and resource types, for generic code that rebinds one
+    ///        or constructs sibling buffers over the same resource.
+    using ElementType  = T;
+    using ResourceType = R;
+
+    /// @brief Alias for a sibling buffer over the same resource with a
+    ///        different element type.
+    template<typename U>
+    using rebind = Buffer<U, R>;
+
+private:
+    R      mResource;
+    T*     mData = nullptr;
+    size_t mSize = 0; // element count
+
+public:
+    /// @brief Default c-tor of an empty buffer; performs no allocation.
+    Buffer() = default;
+
+    /// @brief C-tor allocating @c count uninitialized elements, stream-ordered
+    ///        on @c stream. Parameter order follows cuda::buffer:
+    ///        (stream, resource, count, no_init).
+    /// @param stream cuda stream the allocation is ordered on
+    /// @param resource resource instance the buffer takes ownership of
+    /// @param count number of elements
+    template<typename S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    explicit Buffer(cudaStream_t stream, R resource, size_t count, NoInit)
+        : detail::StreamHolder<true>{stream}
+        , mResource(std::move(resource))
+    {
+        this->allocate(count, stream);
+    }
+
+    /// @brief Convenience c-tor using a default-constructed resource.
+    /// @param stream cuda stream the allocation is ordered on
+    /// @param count number of elements
+    /// @note There is deliberately no count c-tor without NoInit: implicit
+    ///       initialization of freshly allocated memory costs a hidden fill
+    ///       pass that the dominant allocate-then-overwrite pattern wastes,
+    ///       so initialization is always explicit (matching cuda::buffer,
+    ///       whose count c-tor likewise requires cuda::no_init).
+    template<typename S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    Buffer(cudaStream_t stream, size_t count, NoInit) : Buffer(stream, R(), count, noInit) {}
+
+    /// @brief C-tor allocating @c count uninitialized elements from a
+    ///        synchronous resource: the stream-less analog of
+    ///        (stream, resource, count, no_init).
+    /// @param resource resource instance the buffer takes ownership of
+    /// @param count number of elements
+    template<typename S = R, std::enable_if_t<!is_async_resource<S>::value && is_resource<S>::value, int> = 0>
+    explicit Buffer(R resource, size_t count, NoInit)
+        : mResource(std::move(resource))
+    {
+        this->allocate(count, cudaStream_t{0});
+    }
+
+    /// @brief Convenience c-tor using a default-constructed resource.
+    /// @param count number of elements
+    template<typename S = R, std::enable_if_t<!is_async_resource<S>::value && is_resource<S>::value, int> = 0>
+    Buffer(size_t count, NoInit) : Buffer(R(), count, noInit) {}
+
+    /// @brief Explicitly disallow copy construction and assignment operation
+    Buffer(const Buffer&) = delete;
+    Buffer& operator=(const Buffer&) = delete;
+
+    /// @brief Move c-tor; steals the allocation (and retained stream, if any)
+    ///        and leaves @c other empty.
+    Buffer(Buffer&& other) noexcept
+        : detail::StreamHolder<IsAsync>(other)
+        , mResource(std::move(other.mResource))
+        , mData(other.mData)
+        , mSize(other.mSize)
+    {
+        other.mData = nullptr;
+        other.mSize = 0;
+    }
+
+    /// @brief Move assignment; frees the current allocation first, then steals
+    ///        from @c other and leaves it empty. Self-move is a no-op.
+    Buffer& operator=(Buffer&& other) noexcept
+    {
+        if (this != &other) {
+            this->destroy();
+            static_cast<detail::StreamHolder<IsAsync>&>(*this) = other;
+            mResource = std::move(other.mResource);
+            mData = other.mData;
+            mSize = other.mSize;
+            other.mData = nullptr;
+            other.mSize = 0;
+        }
+        return *this;
+    }
+
+    /// @brief Returns a deep copy of this buffer, allocated from a copy of the
+    ///        resource; the allocation and element copy are ordered on @c stream,
+    ///        which becomes the copy's retained stream.
+    /// @param stream cuda stream the allocation and element copy are ordered on
+    template<typename S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    Buffer copy(cudaStream_t stream) const
+    {
+        Buffer out(stream, mResource, mSize, noInit);
+        if (mData) cudaCheck(cudaMemcpyAsync(out.mData, mData, this->size_bytes(), cudaMemcpyDefault, stream));
+        return out;
+    }
+
+    /// @brief Returns a deep copy of this buffer ordered on the retained
+    ///        stream, i.e. copy(this->stream()).
+    template<typename S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    Buffer copy() const { return this->copy(this->stream()); }
+
+    /// @brief Returns a deep copy of this buffer, allocated from a copy of the
+    ///        synchronous resource.
+    template<typename S = R, std::enable_if_t<!is_async_resource<S>::value && is_resource<S>::value, int> = 0>
+    Buffer copy() const
+    {
+        Buffer out(mResource, mSize, noInit);
+        if (mData) cudaCheck(cudaMemcpy(out.mData, mData, this->size_bytes(), cudaMemcpyDefault));
+        return out;
+    }
+
+    /// @brief D-tor. A stream-ordered resource frees on the retained stream;
+    ///        a synchronous resource frees immediately.
+    ~Buffer() { this->destroy(); }
+
+    /// @brief Returns the retained stream, i.e. the stream the buffer's memory
+    ///        will be freed on.
+    template<typename S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    cudaStream_t stream() const { return this->mStream; }
+
+    /// @brief Replaces the retained stream without synchronizing; subsequent
+    ///        deallocation (and destruction) is ordered on @c stream instead.
+    /// @param stream cuda stream subsequent deallocation is ordered on
+    /// @warning The caller is responsible for ordering @c stream after any
+    ///          in-flight work that uses the buffer's memory. This deliberately
+    ///          does not synchronize, matching cuda::buffer's set_stream, which
+    ///          avoids implicit synchronization in fundamental primitives.
+    template<typename S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    void set_stream(cudaStream_t stream) { this->mStream = stream; }
+
+    /// @brief Resizes the buffer to @c count elements, preserving the leading
+    ///        min(old, new) elements. Every operation — the new allocation, the
+    ///        prefix copy, and the free of the old block — is ordered on
+    ///        @c stream, which becomes the retained stream: the prefix copy is
+    ///        the old block's last use, so that is the stream its free must be
+    ///        ordered on.
+    /// @param count number of elements
+    /// @param stream cuda stream the reallocation is ordered on
+    /// @warning The caller is responsible for ordering @c stream after any
+    ///          in-flight work on the previously retained stream that uses the
+    ///          buffer's memory.
+    template<typename S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    void resize(size_t count, cudaStream_t stream)
+    {
+        if (count != mSize) {
+            // No member is mutated until every throwing operation has
+            // succeeded, so a failed resize leaves the buffer untouched --
+            // including its retained stream.
+            T* newData = count ? static_cast<T*>(mResource.allocate_async(checkedBytes(count), R::DEFAULT_ALIGNMENT, stream))
+                               : nullptr;
+            if (newData && mData) {
+                const size_t prefix = count < mSize ? count : mSize;
+                try {
+                    cudaCheck(cudaMemcpyAsync(newData, mData, prefix * sizeof(T), cudaMemcpyDefault, stream));
+                }
+                catch (...) {
+                    mResource.deallocate_async(newData, checkedBytes(count), R::DEFAULT_ALIGNMENT, stream);
+                    throw;
+                }
+            }
+            this->mStream = stream;
+            this->deallocate(mData, mSize); // ordered on stream: after the prefix copy
+            mData = newData;
+            mSize = count;
+        }
+        else {
+            this->mStream = stream; // no reallocation: set_stream semantics
+        }
+    }
+
+    /// @brief Resizes the buffer to @c count elements through the synchronous
+    ///        resource, preserving the leading min(old, new) elements.
+    /// @param count number of elements
+    template<typename S = R, std::enable_if_t<!is_async_resource<S>::value && is_resource<S>::value, int> = 0>
+    void resize(size_t count)
+    {
+        if (count == mSize) return;
+        // No member is mutated until every throwing operation has succeeded,
+        // so a failed resize leaves the buffer untouched.
+        T* newData = count ? static_cast<T*>(mResource.allocate(checkedBytes(count), R::DEFAULT_ALIGNMENT))
+                           : nullptr;
+        if (newData && mData) {
+            const size_t prefix = count < mSize ? count : mSize;
+            try {
+                cudaCheck(cudaMemcpy(newData, mData, prefix * sizeof(T), cudaMemcpyDefault));
+            }
+            catch (...) {
+                mResource.deallocate(newData, checkedBytes(count), R::DEFAULT_ALIGNMENT);
+                throw;
+            }
+        }
+        this->deallocate(mData, mSize);
+        mData = newData;
+        mSize = count;
+    }
+
+    /// @brief Returns a pointer to the elements, or nullptr if empty.
+    T*       data()       { return mData; }
+    const T* data() const { return mData; }
+
+    /// @brief Returns a copy of the resource; for a ResourceRef this refers
+    ///        to the same underlying instance.
+    /// @note Requires R to be copy-constructible (the cuda::mr convention:
+    ///       resources are cheap handles). A resource that owns its pool by
+    ///       value hands the caller an independent copy of that pool.
+    R resource() const { return mResource; }
+
+    /// @brief Returns the number of elements.
+    size_t size() const { return mSize; }
+
+    /// @brief Returns the size of the buffer's allocation in bytes.
+    size_t size_bytes() const { return mSize * sizeof(T); }
+
+    /// @brief Returns true if this buffer manages no memory.
+    bool empty() const { return mSize == 0; }
+
+    /// @brief Frees the buffer memory (if any) and resets to the empty state.
+    ///        A stream-ordered resource frees on the retained stream.
+    /// @note Spelled destroy to match cuda::buffer. This is the name to use.
+    void destroy()
+    {
+        this->deallocate(mData, mSize);
+        mData = nullptr;
+        mSize = 0;
+    }
+
+    /// @brief Frees the buffer memory (if any) and resets to the empty state.
+    /// @deprecated Use destroy(): the handles now dispatch to it directly.
+    [[deprecated("Use cuda::Buffer::destroy instead")]]
+    void clear() { this->destroy(); }
+
+    /// @brief Frees the buffer memory (if any) on @c stream and resets to the
+    ///        empty state. @c stream becomes the retained stream.
+    /// @param stream cuda stream the deallocation is ordered on
+    /// @warning The caller is responsible for ordering @c stream after any
+    ///          in-flight work that uses the buffer's memory.
+    template<typename S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    void destroy(cudaStream_t stream)
+    {
+        this->mStream = stream;
+        this->destroy();
+    }
+
+    /// @brief Exchanges the contents of this buffer with @c other. Neither
+    ///        buffer allocates, frees, or copies element data.
+    /// @param other buffer to exchange contents with
+    void swap(Buffer& other) noexcept
+    {
+        auto& lhs = static_cast<detail::StreamHolder<IsAsync>&>(*this);
+        auto& rhs = static_cast<detail::StreamHolder<IsAsync>&>(other);
+        std::swap(lhs, rhs);
+        std::swap(mResource, other.mResource);
+        std::swap(mData, other.mData);
+        std::swap(mSize, other.mSize);
+    }
+
+private:
+    /// @brief Returns @c count * sizeof(T), throwing std::runtime_error if the
+    ///        byte size would overflow size_t instead of silently wrapping into
+    ///        a tiny allocation.
+    /// @param count number of elements
+    static size_t checkedBytes(size_t count)
+    {
+        if (count > std::numeric_limits<size_t>::max() / sizeof(T))
+            throw std::runtime_error("nanovdb::cuda::Buffer: element count overflows the byte size");
+        return count * sizeof(T);
+    }
+
+    /// @brief Allocates @c count elements through the resource and records the
+    ///        new extent. A zero count allocates nothing.
+    /// @param count number of elements
+    /// @param stream cuda stream the allocation is ordered on; used by the
+    ///        stream-ordered form and ignored by the synchronous one
+    void allocate(size_t count, cudaStream_t stream)
+    {
+        if (count) {
+            if constexpr (IsAsync)
+                mData = static_cast<T*>(mResource.allocate_async(checkedBytes(count), R::DEFAULT_ALIGNMENT, stream));
+            else
+                mData = static_cast<T*>(mResource.allocate(checkedBytes(count), R::DEFAULT_ALIGNMENT));
+        }
+        mSize = count;
+    }
+
+    /// @brief Frees @c count elements at @c p through the resource; the
+    ///        stream-ordered form frees on the retained stream. Null is a no-op.
+    /// @param p pointer to the elements to free
+    /// @param count number of elements
+    void deallocate(T* p, size_t count)
+    {
+        if (!p) return;
+        if constexpr (IsAsync)
+            mResource.deallocate_async(p, count * sizeof(T), R::DEFAULT_ALIGNMENT, this->mStream);
+        else
+            mResource.deallocate(p, count * sizeof(T), R::DEFAULT_ALIGNMENT);
+    }
+}; // Buffer<T, R> class
+
+/// @brief Non-owning, trivially copyable view of a contiguous range of @c T
+///        elements, with span semantics.
+/// @tparam T element type; spell constness in the element type
+///         (e.g. BufferView<const std::byte> is the read-only form), since
+///         const on the view itself is shallow.
+template<typename T>
+class BufferView
+{
+    T*     mData = nullptr;
+    size_t mSize = 0; // element count
+
+public:
+    /// @brief Default c-tor of an empty view.
+    BufferView() = default;
+
+    /// @brief C-tor viewing a contiguous range; the caller guarantees the
+    ///        underlying storage outlives every use of the view.
+    /// @param data pointer to the first element
+    /// @param count number of elements
+    /// @throw std::runtime_error if @c data is null while @c count is non-zero.
+    BufferView(T* data, size_t count) : mData(data), mSize(count)
+    {
+        if (data == nullptr && count != 0)
+            throw std::runtime_error("BufferView: null data with a non-zero element count");
+    }
+
+    /// @brief Returns a pointer to the viewed elements, or nullptr if empty.
+    T* data() const { return mData; }
+
+    /// @brief Returns the number of viewed elements.
+    size_t size() const { return mSize; }
+
+    /// @brief Returns the size of the viewed range in bytes.
+    size_t size_bytes() const { return mSize * sizeof(T); }
+
+    /// @brief Returns true if this view references no elements.
+    bool empty() const { return mSize == 0; }
+
+    /// @brief Detaches the view (nulls the pointer and zeroes the size)
+    ///        without touching the underlying storage -- the view is
+    ///        non-owning, so "destroying" it releases nothing. This is the
+    ///        one deliberate deviation from std::span, required by the buffer
+    ///        static interface the handles consume through reset().
+    void destroy()
+    {
+        mData = nullptr;
+        mSize = 0;
+    }
+
+    /// @brief Detaches the view.
+    /// @deprecated Use destroy(): the handles now dispatch to it directly.
+    [[deprecated("Use cuda::BufferView::destroy instead")]]
+    void clear() { this->destroy(); }
+}; // BufferView<T> class
+
+} // namespace cuda
+
+// Primary template defined in HostBuffer.h; declared here so this header
+// stays self-contained without pulling in the host-buffer machinery.
+template<typename BufferT>
+struct BufferTraits;
+
+/// @brief GridHandle support for the single-space cuda::Buffer: the buffer
+///        owns exactly one allocation, resident on the device, so the handle
+///        parses metadata through a device read and exposes only the device
+///        accessors. Requires byte-addressed storage.
+/// @note This trait doubles as the definition of the single-space
+///       device-buffer concept: a buffer whose BufferTraits specialization
+///       sets hasDeviceSingle guarantees ElementType and ResourceType
+///       typedefs, data(), size() and size_bytes() (byte-addressed elements,
+///       enforced by the consumer), resource(), copy(), destroy(), and stream()
+///       when the resource is stream-ordered. Any consumer of hasDeviceSingle
+///       may rely on exactly this interface and nothing more; in particular,
+///       scratch allocates through resource() as a cuda::Buffer, so a
+///       conforming buffer never needs to be constructible by a consumer.
+template<typename T, typename R>
+struct BufferTraits<cuda::Buffer<T, R>>
+{
+    static constexpr bool hasDeviceDual   = false;
+    // Device-resident storage; the byte-addressed requirement is enforced by
+    // the single-space GridHandle constructor, so trait queries stay
+    // answerable for any element type.
+    static constexpr bool hasDeviceSingle = !cuda::is_host_accessible_resource<R>::value;
+    // A buffer over a host-accessible resource (e.g. PinnedResource) is
+    // host-readable single-space storage: GridHandle parses its metadata on
+    // the host, exposes the host accessors, and allocates reads and copies
+    // through the buffer's resource.
+    static constexpr bool hasHostSingle   = cuda::is_host_accessible_resource<R>::value;
+};
+
+} // namespace nanovdb
+
+#endif // end of NANOVDB_CUDA_BUFFER_H_HAS_BEEN_INCLUDED

nanovdb/nanovdb/cuda/DeviceBuffer.h

@@ -38,6 +38,7 @@ class DeviceBuffer
     uint64_t mSize; // total number of bytes managed by this buffer (assumed to be identical for host and device)
     void *mCpuData, **mGpuData; // raw pointers to the host and device buffers
     int   mDeviceCount, mManaged;// if mManaged is non-zero this class is responsible for allocating and freeing memory buffers. Otherwise this is assumed to be handled externally
+    cudaEvent_t *mEvents = nullptr;// per-device event marking the last use of each managed device buffer (parallel to mGpuData, length mDeviceCount). Every use waits on this event before issuing work and re-records it afterwards, so the single event transitively covers EVERY stream the buffer has been used on. Frees then wait on it, which orders them after all outstanding work: freeing on the default stream alone is only safe for blocking streams, and freeing on the last-used stream alone is only safe when just one stream was used.
 
     /// @brief Initialize buffer
     /// @param size byte size of buffer to be initialized
@@ -46,6 +47,33 @@ class DeviceBuffer
     /// @warning size is expected to be non-zero. Use clear() clear buffer!
     void init(uint64_t size, int device, cudaStream_t stream);
 
+    /// @brief Free every managed device allocation, each ordered after all tracked uses of the
+    ///        buffer, and destroy the tracking events.
+    /// @param stream Stream the frees are issued on for allocations owned by the CURRENT device.
+    ///        A stream belongs to a single device, so it cannot carry frees for other devices'
+    ///        memory pools; allocations on other devices are freed on their own device's default
+    ///        stream, after switching to that device.
+    /// @note Destroying an event with a pending wait is safe: CUDA releases it once the device
+    ///       has completed it.
+    void freeDeviceBuffers(cudaStream_t stream)
+    {
+        int current = 0;
+        cudaCheck(cudaGetDevice(&current));
+        for (int i = 0; i < mDeviceCount; ++i) {
+            if (mGpuData[i]) {
+                const cudaStream_t freeStream = (i == current) ? stream : cudaStream_t{0};
+                if (i != current) cudaCheck(cudaSetDevice(i));
+                this->orderAfterPriorUses(i, freeStream);
+                cudaCheck(util::cuda::freeAsync(mGpuData[i], freeStream));
+                if (i != current) cudaCheck(cudaSetDevice(current));
+            }
+            if (mEvents && mEvents[i]) {
+                cudaCheck(cudaEventDestroy(mEvents[i]));
+                mEvents[i] = nullptr;
+            }
+        }
+    }
+
 public:
 
     using PtrT = std::shared_ptr<DeviceBuffer>;
@@ -122,8 +150,10 @@ public:
         , mGpuData(other.mGpuData)
         , mDeviceCount(other.mDeviceCount)
         , mManaged(other.mManaged)
+        , mEvents(other.mEvents)
     {
         other.mCpuData = other.mGpuData = nullptr;
+        other.mEvents = nullptr;
         other.mSize = other.mDeviceCount = other.mManaged = 0;
     }
 
@@ -142,6 +172,8 @@ public:
     }
 
      /// @brief Destructor frees memory on both the host and device
+    /// @note Each managed device free waits on that device's tracking event first, so it is
+    ///       ordered after every stream the buffer was used on, not just the most recent one.
     ~DeviceBuffer() { this->clear(); };
 
     /// @brief Static factory method that return an instance of this buffer
@@ -226,8 +258,59 @@ public:
 
     ///////////////////////////////////////////////////////////////////////
 
+    /// @brief Order work subsequently issued on @a stream after every prior use of this
+    ///        device buffer, whichever stream those uses were issued on. The consume-side
+    ///        companion of recordUse: an external consumer (e.g. a zero-copy array-interface
+    ///        export) calls this with its own stream before reading, so it cannot observe a
+    ///        partially-written buffer after asynchronous uploads or recorded kernels.
+    /// @param device Device whose buffer is about to be read
+    /// @param stream Stream the consumer's work will be issued on
+    void orderAfterPriorUses(int device, cudaStream_t stream) const
+    {
+        if (mEvents && mEvents[device]) cudaCheck(cudaStreamWaitEvent(stream, mEvents[device], 0));
+    }
+
+    /// @brief Record that this buffer's device data was just used on @a stream, so that the
+    ///        buffer's device frees (destructor, move-assignment, clear) are ordered after that
+    ///        work. Uses issued through deviceUpload/deviceDownload are recorded automatically;
+    ///        callers that enqueue their own kernels or copies against the raw pointer returned
+    ///        by deviceData() should call this afterwards. Without it, such work is only safe if
+    ///        it is on a blocking stream (which the free, issued on the default stream, waits on
+    ///        implicitly) or if the caller synchronizes before the buffer is cleared/destroyed.
+    /// @param device Device whose buffer was used
+    /// @param stream Stream the work was issued on
+    /// @note Recording chains across streams: @a stream is first ordered after the previously
+    ///       recorded use (if any) so the single per-device event transitively covers every
+    ///       recorded use, not just the last one. Without this, concurrent uses on streams A
+    ///       then B would leave only B's event, and the device free could run while A's work
+    ///       is still in flight. The side effect is that work subsequently issued on @a stream
+    ///       also waits on the previously recorded use -- acceptable for a shared buffer, where
+    ///       later-recorded consumers observing earlier writes is the expected ordering. Note
+    ///       this also serializes CONCURRENT READERS that record uses (the single event cannot
+    ///       distinguish read-read from write-read); if that ever matters in a profile, the
+    ///       upgrade path is a read/write-separated or per-record event scheme, not a revert.
+    void recordUse(int device, cudaStream_t stream)
+    {
+        if (!mEvents) return;
+        if (mEvents[device] == nullptr) {// events are per-device, so create it on the right one
+            int current = 0;
+            cudaCheck(cudaGetDevice(&current));
+            if (current != device) cudaCheck(cudaSetDevice(device));
+            cudaCheck(cudaEventCreateWithFlags(&mEvents[device], cudaEventDisableTiming));
+            if (current != device) cudaCheck(cudaSetDevice(current));
+        } else {
+            // Re-recording MOVES the event; chain first so the new capture also covers the
+            // prior recorded use (waiting on a never-recorded or completed event is a no-op).
+            cudaCheck(cudaStreamWaitEvent(stream, mEvents[device], 0));
+        }
+        cudaCheck(cudaEventRecord(mEvents[device], stream));
+    }
+
     /// @brief Retuns a raw pointer to the specified device/GPU buffer managed by this allocator.
     /// @warning Note that the pointer can be NULL!
+    /// @note Work enqueued against this raw pointer is invisible to the buffer's lifetime
+    ///       tracking: on a non-blocking stream, call recordUse afterwards (or synchronize
+    ///       before the buffer is cleared/destroyed) so the device free is ordered after it.
     void* deviceData(int device) const {
         NANOVDB_ASSERT(device >= 0 && device < mDeviceCount);
         return mGpuData[device];
@@ -301,6 +384,10 @@ public:
     /// @}
 
     /// @brief De-allocate all memory managed by this allocator and set all pointers to NULL
+    /// @param stream Stream the device frees are issued on. The frees are additionally ordered
+    ///        after every stream the buffer was used on (via the per-device tracking event), so
+    ///        @a stream selects where the free is enqueued, not what it is ordered against - any
+    ///        stream is safe to pass here regardless of where the buffer was used.
     void clear(cudaStream_t stream = 0);
     void clear(void* stream){this->clear(cudaStream_t(stream));}
 
@@ -310,18 +397,22 @@ public:
 
 inline DeviceBuffer& DeviceBuffer::operator=(DeviceBuffer&& other) noexcept
 {
-    if (mManaged) {// first free all the managed data buffers
+    if (this == &other) return *this;// self-move would free our buffers and then read them back
+    if (mManaged) {// first free all the managed data buffers, ordered after every use of each
         cudaCheck(cudaFreeHost(mCpuData));
-        for (int i=0; i<mDeviceCount; ++i) cudaCheck(util::cuda::freeAsync(mGpuData[i], 0));
+        this->freeDeviceBuffers(cudaStream_t{0});
     }
     delete [] mGpuData;
+    delete [] mEvents;
     mSize    = other.mSize;
     mCpuData = other.mCpuData;
     mGpuData = other.mGpuData;
     mDeviceCount = other.mDeviceCount;
     mManaged = other.mManaged;
+    mEvents = other.mEvents;
     other.mCpuData = nullptr;
     other.mGpuData = nullptr;
+    other.mEvents = nullptr;
     other.mSize = 0;
     other.mDeviceCount = 0;
     other.mManaged = 0;
@@ -333,6 +424,7 @@ inline void DeviceBuffer::init(uint64_t size, int device, cudaStream_t stream)
     if (size==0) return;
     cudaCheck(cudaGetDeviceCount(&mDeviceCount));
     mGpuData = new void*[mDeviceCount]();// NULL initialization
+    mEvents = new cudaEvent_t[mDeviceCount]();// NULL initialization; created lazily on first use
     NANOVDB_ASSERT(device >= cudaCpuDeviceId && device < mDeviceCount);
     if (device == cudaCpuDeviceId) {
         cudaCheck(cudaMallocHost((void**)&mCpuData, size)); // un-managed pinned memory on the host (can be slow to access!). Always 32B aligned
@@ -340,6 +432,7 @@ inline void DeviceBuffer::init(uint64_t size, int device, cudaStream_t stream)
     } else {
         cudaCheck(util::cuda::mallocAsync(mGpuData+device, size, stream)); // un-managed memory on the device, always 32B aligned!
         checkPtr(mGpuData[device], "cuda::DeviceBuffer::init: failed to allocate device buffer");
+        this->recordUse(device, stream);// the free must be ordered after this allocation
     }
     mSize = size;
     mManaged = 1;// i.e. this instance is responsible for allocating and delete memory
@@ -354,7 +447,11 @@ inline void DeviceBuffer::deviceUpload(int device, cudaStream_t stream, bool syn
         cudaCheck(util::cuda::mallocAsync(mGpuData+device, mSize, stream)); // un-managed memory on the device, always 32B aligned!
     }
     checkPtr(mGpuData[device], "uninitialized gpu destination data");
+    // Order this transfer after any use of the buffer on another stream, then mark it as the
+    // latest use, so the tracking event keeps covering every stream the buffer has seen.
+    this->orderAfterPriorUses(device, stream);
     cudaCheck(cudaMemcpyAsync(mGpuData[device], mCpuData, mSize, cudaMemcpyHostToDevice, stream));
+    this->recordUse(device, stream);
     if (sync) cudaCheck(cudaStreamSynchronize(stream));
 } // DeviceBuffer::deviceUpload
 
@@ -374,7 +471,9 @@ inline void DeviceBuffer::deviceDownload(int device, cudaStream_t stream, bool s
         cudaCheck(cudaMallocHost((void**)&mCpuData, mSize)); // un-managed pinned memory on the host (can be slow to access!). Always 32B aligned
     }
     checkPtr(mCpuData, "uninitialized cpu destination data");
+    this->orderAfterPriorUses(device, stream);
     cudaCheck(cudaMemcpyAsync(mCpuData, mGpuData[device], mSize, cudaMemcpyDeviceToHost, stream));
+    this->recordUse(device, stream);
     if (sync) cudaCheck(cudaStreamSynchronize(stream));
 } // DeviceBuffer::deviceDownload
 
@@ -387,13 +486,15 @@ inline void DeviceBuffer::deviceDownload(void* stream, bool sync)
 
 inline void DeviceBuffer::clear(cudaStream_t stream)
 {
-    if (mManaged) {// free all the managed data buffers
+    if (mManaged) {// free all the managed data buffers, ordered after every use of each
         cudaCheck(cudaFreeHost(mCpuData));
-        for (int i=0; i<mDeviceCount; ++i) cudaCheck(util::cuda::freeAsync(mGpuData[i], stream));
+        this->freeDeviceBuffers(stream);
     }
     delete [] mGpuData;
+    delete [] mEvents;
     mCpuData = nullptr;
     mGpuData = nullptr;
+    mEvents = nullptr;
     mSize = 0;
     mDeviceCount = 0;
     mManaged = 0;

nanovdb/nanovdb/cuda/DeviceResource.h

@@ -18,43 +18,61 @@ namespace cuda {
 /// @brief Default stream-ordered device memory resource. Allocations are made
 ///        with cudaMallocAsync and freed with cudaFreeAsync via the
 ///        util::cuda wrappers.
-/// @note Exposes both the legacy static methods (allocateAsync /
-///       deallocateAsync, retained for source compatibility) and the
-///       instance methods (allocate_async / deallocate_async) that model
-///       the AsyncResource concept. The instance methods forward to the
-///       static ones, so a default-constructed instance is stateless and
-///       adds no overhead.
+/// @note Models the AsyncResource concept, which refines the synchronous
+///       Resource concept (as in CCCL's cuda::mr): the instance methods
+///       provide both the stream-ordered allocate_async / deallocate_async
+///       and the synchronous allocate / deallocate. The type is stateless,
+///       so a default-constructed instance adds no overhead. The static
+///       allocateAsync / deallocateAsync methods are deprecated.
 class DeviceResource
 {
 public:
     // cudaMalloc aligns memory to 256 bytes by default
     static constexpr size_t DEFAULT_ALIGNMENT = 256;
 
-    static void* allocateAsync(size_t bytes, size_t, cudaStream_t stream) {
+    /// @brief Stream-ordered allocation.
+    /// @param bytes number of bytes to allocate
+    /// @param stream cuda stream the allocation is ordered on
+    /// @note the alignment parameter is unnamed: cudaMallocAsync always
+    ///       256B-aligns
+    void* allocate_async(size_t bytes, size_t, cudaStream_t stream) {
         void* p = nullptr;
         cudaCheck(util::cuda::mallocAsync(&p, bytes, stream));
         return p;
     }
 
-    static void deallocateAsync(void *p, size_t, size_t, cudaStream_t stream) {
+    /// @brief Stream-ordered deallocation.
+    /// @param p pointer previously returned by allocate_async
+    /// @param stream cuda stream the deallocation is ordered on
+    void deallocate_async(void* p, size_t, size_t, cudaStream_t stream) {
         cudaCheck(util::cuda::freeAsync(p, stream));
     }
 
-    /// @brief Instance allocation entry point modelling the AsyncResource concept.
+    /// @brief Synchronous allocation; the returned memory is immediately
+    ///        valid on every stream.
     /// @param bytes number of bytes to allocate
-    /// @param alignment requested alignment (cudaMallocAsync always 256B-aligns)
-    /// @param stream cuda stream the allocation is ordered on
-    void* allocate_async(size_t bytes, size_t alignment, cudaStream_t stream) {
-        return allocateAsync(bytes, alignment, stream);
+    /// @param alignment requested alignment
+    void* allocate(size_t bytes, size_t alignment) {
+        void* p = this->allocate_async(bytes, alignment, cudaStream_t(0));
+        cudaCheck(cudaStreamSynchronize(cudaStream_t(0)));
+        return p;
     }
 
-    /// @brief Instance deallocation entry point modelling the AsyncResource concept.
-    /// @param p pointer previously returned by allocate_async
-    /// @param bytes size passed to the matching allocate_async
-    /// @param alignment alignment passed to the matching allocate_async
-    /// @param stream cuda stream the deallocation is ordered on
-    void deallocate_async(void* p, size_t bytes, size_t alignment, cudaStream_t stream) {
-        deallocateAsync(p, bytes, alignment, stream);
+    /// @brief Synchronous deallocation; the caller guarantees that device
+    ///        work touching the memory has completed.
+    /// @param p pointer previously returned by allocate or allocate_async
+    void deallocate(void* p, size_t bytes, size_t alignment) {
+        this->deallocate_async(p, bytes, alignment, cudaStream_t(0));
+    }
+
+    [[deprecated("use the instance method allocate_async")]]
+    static void* allocateAsync(size_t bytes, size_t alignment, cudaStream_t stream) {
+        return DeviceResource().allocate_async(bytes, alignment, stream);
+    }
+
+    [[deprecated("use the instance method deallocate_async")]]
+    static void deallocateAsync(void *p, size_t bytes, size_t alignment, cudaStream_t stream) {
+        DeviceResource().deallocate_async(p, bytes, alignment, stream);
     }
 };
 
@@ -87,15 +105,236 @@ inline R& default_resource()
 ///         return resource.allocate(bytes, alignment);               // synchronous
 /// }
 /// @endcode
+/// @note AsyncResource refines the synchronous Resource concept, matching
+///       CCCL's cuda::mr: an async resource must also provide the
+///       synchronous allocate / deallocate, so is_async_resource<R> implies
+///       is_resource<R>. A synchronous-only resource (e.g. PinnedResource)
+///       models just is_resource. The synchronous methods of a stream-ordered
+///       resource are typically thin delegates (allocate_async on the null
+///       stream followed by a stream synchronize).
 template <class R, class = void>
 struct is_async_resource : std::false_type {};
 
 template <class R>
 struct is_async_resource<R, std::void_t<
     decltype(std::declval<R&>().allocate_async(size_t{0}, size_t{0}, cudaStream_t{0})),
-    decltype(std::declval<R&>().deallocate_async(std::declval<void*>(), size_t{0}, size_t{0}, cudaStream_t{0}))>>
+    decltype(std::declval<R&>().deallocate_async(std::declval<void*>(), size_t{0}, size_t{0}, cudaStream_t{0})),
+    decltype(std::declval<R&>().allocate(size_t{0}, size_t{0})),
+    decltype(std::declval<R&>().deallocate(std::declval<void*>(), size_t{0}, size_t{0}))>>
     : std::true_type {};
 
+/// @brief Detection trait: @c is_host_accessible_resource<R>::value is true
+///        iff @c R declares `static constexpr bool HOST_ACCESSIBLE = true`,
+///        i.e. its allocations are mapped into the host address space (e.g.
+///        PinnedResource). Defaults to false: allocations are device-resident.
+/// @note Unlike the void_t detections above, this checks the member's VALUE:
+///       a resource declaring HOST_ACCESSIBLE = false stays device-resident.
+template<typename R, typename = void>
+struct is_host_accessible_resource : std::false_type {};
+template<typename R>
+struct is_host_accessible_resource<R, typename std::enable_if<bool(R::HOST_ACCESSIBLE)>::type> : std::true_type {};
+
+/// @brief Detection trait: @c is_resource<R>::value is true iff @c R models
+///        the synchronous Resource concept, i.e. exposes
+///        allocate(size_t, size_t) and deallocate(void*, size_t, size_t).
+/// @details Use it to constrain code paths that need a resource without a
+///          stream argument, e.g. host-side allocations:
+/// @code
+/// template<typename R>
+/// void* allocate(R& resource, size_t bytes, size_t alignment)
+/// {
+///     static_assert(nanovdb::cuda::is_resource<R>::value, "R must be a synchronous resource");
+///     return resource.allocate(bytes, alignment);
+/// }
+/// @endcode
+template <class R, class = void>
+struct is_resource : std::false_type {};
+
+template <class R>
+struct is_resource<R, std::void_t<
+    decltype(std::declval<R&>().allocate(size_t{0}, size_t{0})),
+    decltype(std::declval<R&>().deallocate(std::declval<void*>(), size_t{0}, size_t{0}))>>
+    : std::true_type {};
+
+/// @brief CRTP base supplying the synchronous half of the resource concept in
+///        terms of the stream-ordered half, so a custom stream-ordered resource
+///        only has to write allocate_async and deallocate_async.
+/// @tparam Derived the resource deriving from this base
+/// @details A stream-ordered resource must also model the synchronous concept
+///          (is_async_resource implies is_resource), which means writing four
+///          methods where two would do. The synchronous pair is not a bare
+///          delegate: memory from allocate must be usable immediately on any
+///          stream, so the null-stream allocation has to be synchronized before
+///          it is returned. Omitting that synchronization yields memory that
+///          satisfies the concept but is not actually synchronous -- a race
+///          rather than a compile error -- so it lives here rather than being
+///          rewritten per resource.
+/// @code
+/// struct MyResource : nanovdb::cuda::SyncFromAsync<MyResource> {
+///     static constexpr size_t DEFAULT_ALIGNMENT = 256;
+///     void* allocate_async(size_t bytes, size_t alignment, cudaStream_t stream);
+///     void  deallocate_async(void* p, size_t bytes, size_t alignment, cudaStream_t stream);
+/// };
+/// @endcode
+template <class Derived>
+struct SyncFromAsync
+{
+    /// @brief Allocates @c bytes usable on any stream when this returns.
+    /// @param bytes number of bytes to allocate
+    /// @param alignment requested alignment
+    /// @note Every call synchronizes the null stream; on hot paths prefer the
+    ///       stream-ordered pair.
+    void* allocate(size_t bytes, size_t alignment)
+    {
+        void* p = static_cast<Derived&>(*this).allocate_async(bytes, alignment, cudaStream_t{0});
+        cudaCheck(cudaStreamSynchronize(cudaStream_t{0}));
+        return p;
+    }
+
+    /// @brief Frees @c p on the null stream.
+    /// @param p pointer previously returned by allocate
+    /// @param bytes size passed to the matching allocate
+    /// @param alignment alignment passed to the matching allocate
+    /// @note No synchronization here: the synchronous concept's contract is
+    ///       that the memory is already quiescent when deallocate is called.
+    void deallocate(void* p, size_t bytes, size_t alignment)
+    {
+        static_cast<Derived&>(*this).deallocate_async(p, bytes, alignment, cudaStream_t{0});
+    }
+};
+
+/// @brief Synchronous device memory resource backed by cudaMalloc/cudaFree.
+///        Models only the Resource concept: it never touches stream-ordered
+///        allocation, so it works on devices without memory-pool support
+///        (cudaDevAttrMemoryPoolsSupported == 0), where DeviceResource's
+///        cudaMallocAsync path fails by design. Pair with AsyncFromSync to
+///        drive the stream-ordered builders on such a device.
+class MallocResource
+{
+public:
+    // cudaMalloc aligns memory to 256 bytes by default
+    static constexpr size_t DEFAULT_ALIGNMENT = 256;
+
+    /// @brief Allocates @c bytes with cudaMalloc; valid on every stream when
+    ///        this returns. A zero request returns nullptr.
+    void* allocate(size_t bytes, size_t)
+    {
+        if (bytes == 0) return nullptr;
+        void* p = nullptr;
+        cudaCheck(cudaMalloc(&p, bytes));
+        return p;
+    }
+
+    /// @brief Frees @c p with cudaFree; the caller guarantees that device work
+    ///        touching the memory has completed.
+    void deallocate(void* p, size_t, size_t) { cudaCheck(cudaFree(p)); }
+};// MallocResource
+
+/// @brief Wrapper presenting a synchronous resource as a stream-ordered one,
+///        so it can drive components that require the AsyncResource concept
+///        (TempPool and the GPU builders).
+/// @tparam R the wrapped synchronous resource, held by value; wrap a
+///         ResourceRef<R> to borrow a stateful instance instead.
+/// @details The mirror of SyncFromAsync, and the analog of cuda::mr's
+///          synchronous_resource_adapter. allocate_async forwards to
+///          R::allocate, whose memory is immediately valid on every stream --
+///          a stronger guarantee than stream-ordering requires.
+///          deallocate_async synchronizes @c stream before R::deallocate,
+///          establishing the quiescence the synchronous contract demands.
+/// @warning Every deallocation synchronizes its stream, so expect
+///          serialization relative to a genuinely stream-ordered resource.
+///          That is the unavoidable cost of a synchronous backend under a
+///          stream-ordered algorithm; this wrapper exists so the cost is
+///          explicit and chosen by the caller -- e.g. on a device without
+///          memory-pool support -- rather than silently substituted.
+template<class R>
+struct AsyncFromSync
+{
+    static_assert(is_resource<R>::value,
+                  "AsyncFromSync requires R to model the synchronous Resource concept");
+
+    static constexpr size_t DEFAULT_ALIGNMENT = R::DEFAULT_ALIGNMENT;
+
+    /// @brief The adapter is host-accessible iff the adapted resource is.
+    static constexpr bool HOST_ACCESSIBLE = is_host_accessible_resource<R>::value;
+
+    R resource;
+
+    /// @brief Allocates through the synchronous resource; the result is valid
+    ///        on every stream, hence trivially valid on @c stream.
+    void* allocate_async(size_t bytes, size_t alignment, cudaStream_t) { return resource.allocate(bytes, alignment); }
+
+    /// @brief Synchronizes @c stream, then frees through the synchronous
+    ///        resource -- the synchronize makes the quiescence contract hold.
+    ///        Null is a no-op and skips the synchronize.
+    void deallocate_async(void* p, size_t bytes, size_t alignment, cudaStream_t stream)
+    {
+        if (p == nullptr) return;
+        cudaCheck(cudaStreamSynchronize(stream));
+        resource.deallocate(p, bytes, alignment);
+    }
+
+    /// @brief Synchronous pair, forwarding to the wrapped resource.
+    void* allocate(size_t bytes, size_t alignment) { return resource.allocate(bytes, alignment); }
+    void  deallocate(void* p, size_t bytes, size_t alignment) { resource.deallocate(p, bytes, alignment); }
+};// AsyncFromSync<R>
+
+/// @brief Non-owning reference to a memory resource that is itself a resource:
+///        copying the ref shares the underlying resource rather than copying it.
+/// @tparam R the referenced resource type
+/// @details Types that hold their resource by value -- cuda::Buffer, matching
+///          cuda::buffer -- select their ownership semantics by what is placed
+///          in that slot: a concrete resource is owned as a copy, while a
+///          ResourceRef borrows. This is the same division cuda::mr draws
+///          between any_resource (owning) and resource_ref (borrowing), and the
+///          same shape as std::pmr::polymorphic_allocator over memory_resource*.
+///          Use it when a resource is stateful or long-lived and a container
+///          must allocate through *that* instance rather than a copy of it.
+/// @warning The referenced resource must outlive every use of this ref and of
+///          all copies of it, including any container holding one.
+template <class R>
+struct ResourceRef
+{
+    static_assert(is_async_resource<R>::value || is_resource<R>::value,
+                  "ResourceRef requires R to model the AsyncResource or the Resource concept");
+
+    static constexpr size_t DEFAULT_ALIGNMENT = R::DEFAULT_ALIGNMENT;
+
+    /// @brief A reference is host-accessible iff the referenced resource is.
+    static constexpr bool HOST_ACCESSIBLE = is_host_accessible_resource<R>::value;
+
+    /// @brief Constructs a ref borrowing @c resource.
+    /// @param resource resource to allocate from; must outlive this ref
+    ResourceRef(R& resource) : mResource(&resource) {}
+
+    /// @{
+    /// @brief Stream-ordered pair, present only when @c R models AsyncResource,
+    ///        so a ref over a synchronous resource does not misreport its tier.
+    template<class S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    void* allocate_async(size_t bytes, size_t alignment, cudaStream_t stream)
+    {
+        return mResource->allocate_async(bytes, alignment, stream);
+    }
+    template<class S = R, std::enable_if_t<is_async_resource<S>::value, int> = 0>
+    void deallocate_async(void* p, size_t bytes, size_t alignment, cudaStream_t stream)
+    {
+        mResource->deallocate_async(p, bytes, alignment, stream);
+    }
+    /// @}
+
+    /// @brief Synchronous pair, forwarding to the referenced resource.
+    void* allocate(size_t bytes, size_t alignment) { return mResource->allocate(bytes, alignment); }
+    void  deallocate(void* p, size_t bytes, size_t alignment) { mResource->deallocate(p, bytes, alignment); }
+
+    /// @brief Two refs compare equal iff they reference the same resource, i.e.
+    ///        memory allocated through one may be deallocated through the other.
+    friend bool operator==(ResourceRef lhs, ResourceRef rhs) { return lhs.mResource == rhs.mResource; }
+    friend bool operator!=(ResourceRef lhs, ResourceRef rhs) { return lhs.mResource != rhs.mResource; }
+
+private:
+    R* mResource;
+};// ResourceRef<R>
+
 }
 
 } // namespace nanovdb::cuda

nanovdb/nanovdb/cuda/GridHandle.cuh

@@ -17,21 +17,20 @@
 #ifndef NANOVDB_CUDA_GRIDHANDLE_CUH_HAS_BEEN_INCLUDED
 #define NANOVDB_CUDA_GRIDHANDLE_CUH_HAS_BEEN_INCLUDED
 
+#include <nanovdb/cuda/Buffer.h>// for the resource-aware scratch buffers below
 #include <nanovdb/cuda/DeviceBuffer.h>// required for instantiation of move c-tor of GridHandle
 #include <nanovdb/tools/cuda/GridChecksum.cuh>// for cuda::updateChecksum
 #include <nanovdb/GridHandle.h>
 
+#include <string>// for the grid index in chain-validation error messages
+#include <type_traits>// for std::is_default_constructible in cuda::copyTo
+
 namespace nanovdb {
 
 namespace cuda {
 
 namespace detail {
 
-static __global__ void cpyGridHandleMeta(const GridData *d_data, GridHandleMetaData *d_meta)
-{
-    nanovdb::cpyGridHandleMeta(d_data, d_meta);
-}
-
 static __global__ void updateGridCount(GridData *d_data, uint32_t gridIndex, uint32_t gridCount, bool *d_dirty)
 {
     NANOVDB_ASSERT(gridIndex < gridCount);
@@ -43,6 +42,145 @@ static __global__ void updateGridCount(GridData *d_data, uint32_t gridIndex, uin
     }
 }
 
+/// @brief Defect classes the device-side chain walk can report, mapped to
+///        exceptions on the host by parseGridChain.
+enum class ChainError : uint32_t { Ok = 0, Truncated, Invalid, Inconsistent, BadSize };
+
+/// @brief Result of the device-side chain walk: the defect class and the
+///        index of the grid the walk failed at.
+struct ChainStatus { ChainError error; uint32_t gridIndex; };
+
+/// @brief Validates every header of the grid chain and fills the metadata
+///        scratch in a single pass. Bounds are checked before every header
+///        read, so a truncated buffer or a forged header can never cause an
+///        out-of-bounds access.
+static __global__ void parseGridChainKernel(const GridData *d_head, uint64_t bytes, uint32_t count,
+                                            GridHandleMetaData *d_meta, ChainStatus *d_status)
+{
+    *d_status = ChainStatus{ChainError::Ok, 0u};
+    uint64_t offset = 0;
+    for (uint32_t i = 0; i < count; ++i) {
+        if (offset + sizeof(GridData) > bytes) { *d_status = {ChainError::Truncated, i}; return; }
+        const GridData *data = util::PtrAdd<GridData>(d_head, offset);
+        if (!data->isValid()) { *d_status = {ChainError::Invalid, i}; return; }
+        if (data->mGridIndex != i || data->mGridCount != count) { *d_status = {ChainError::Inconsistent, i}; return; }
+        if (data->mGridSize < sizeof(GridData) || data->mGridSize > bytes - offset) { *d_status = {ChainError::BadSize, i}; return; }
+        d_meta[i] = GridHandleMetaData{offset, data->mGridSize, data->mGridType};
+        offset += data->mGridSize;
+    }
+}
+
+/// @brief Reads and validates the head of the grid chain, returning the grid
+///        count the chain claims. This is the one synchronizing header read
+///        that sizing the metadata scratch requires; the rest of the chain is
+///        validated on the device by parseGridChain.
+inline uint32_t validGridChainHead(const GridData *d_head, uint64_t bytes, cudaStream_t stream)
+{
+    if (bytes < sizeof(GridData))
+        throw std::runtime_error("GridHandle: grid chain exceeds the device buffer (truncated or corrupt grid data)");
+    GridData tmp;
+    cudaCheck(cudaMemcpyAsync(&tmp, d_head, sizeof(GridData), cudaMemcpyDeviceToHost, stream));
+    cudaCheck(cudaStreamSynchronize(stream));
+    if (!tmp.isValid()) throw std::runtime_error("GridHandle was constructed with an invalid device buffer");
+    if (tmp.mGridCount == 0) throw std::runtime_error("GridHandle: device buffer contains no grids");
+    if (tmp.mGridIndex != 0)
+        throw std::runtime_error("GridHandle: inconsistent grid index/count in the device buffer's grid chain");
+    if (uint64_t(tmp.mGridCount) > bytes / sizeof(GridData))// every grid is at least one full header
+        throw std::runtime_error("GridHandle: grid chain exceeds the device buffer (truncated or corrupt grid data)");
+    return tmp.mGridCount;
+}
+
+/// @brief Bytes of device scratch parseGridChain needs for @c count grids:
+///        the metadata array followed by the status word.
+inline constexpr uint64_t chainScratchSize(uint32_t count)
+{
+    return count * sizeof(GridHandleMetaData) + sizeof(ChainStatus);
+}
+
+/// @brief Launches the combined validate-and-parse walk over the grid chain
+///        and fills @c meta from its scratch: one kernel, one readback and
+///        one synchronization regardless of the grid count.
+/// @param scratch device scratch of at least chainScratchSize(count) bytes
+/// @throw std::runtime_error naming the defect when any header fails validation
+template<typename ScratchBufferT>
+inline void parseGridChain(const GridData *d_head, uint64_t bytes, uint32_t count,
+                           ScratchBufferT &scratch, std::vector<GridHandleMetaData> &meta, cudaStream_t stream)
+{
+    auto *d_meta   = reinterpret_cast<GridHandleMetaData*>(scratch.data());
+    auto *d_status = reinterpret_cast<ChainStatus*>(scratch.data() + count * sizeof(GridHandleMetaData));
+    parseGridChainKernel<<<1, 1, 0, stream>>>(d_head, bytes, count, d_meta, d_status);
+    cudaCheckError();
+    meta.resize(count);
+    ChainStatus status;
+    cudaCheck(cudaMemcpyAsync(meta.data(), d_meta, count * sizeof(GridHandleMetaData), cudaMemcpyDeviceToHost, stream));
+    cudaCheck(cudaMemcpyAsync(&status, d_status, sizeof(ChainStatus), cudaMemcpyDeviceToHost, stream));
+    cudaCheck(cudaStreamSynchronize(stream));
+    if (status.error == ChainError::Ok) return;
+    const std::string where = " (grid " + std::to_string(status.gridIndex) + " of " + std::to_string(count) + ")";
+    switch (status.error) {
+    case ChainError::Truncated:
+        throw std::runtime_error("GridHandle: grid chain exceeds the device buffer (truncated or corrupt grid data)" + where);
+    case ChainError::Inconsistent:
+        throw std::runtime_error("GridHandle: inconsistent grid index/count in the device buffer's grid chain" + where);
+    case ChainError::BadSize:
+        throw std::runtime_error("GridHandle: grid size field exceeds the device buffer (truncated or corrupt grid data)" + where);
+    default:
+        throw std::runtime_error("GridHandle was constructed with an invalid device buffer" + where);
+    }
+}
+
+/// @brief The buffer's retained stream when its resource is stream-ordered,
+///        the default stream otherwise.
+template<typename BufferT>
+inline cudaStream_t retainedStreamOrDefault(const BufferT& buf)
+{
+    if constexpr (is_async_resource<typename BufferT::ResourceType>::value) return buf.stream();
+    else { (void)buf; return cudaStream_t(0); }
+}
+
+/// @brief Constructs the metadata scratch through @c buf's resource, on
+///        @c stream for a stream-ordered resource.
+template<typename ScratchT, typename BufferT>
+inline ScratchT makeMetaScratch(const BufferT& buf, uint64_t count, cudaStream_t stream)
+{
+    if constexpr (is_async_resource<typename BufferT::ResourceType>::value)
+        return ScratchT(stream, buf.resource(), count, noInit);
+    else { (void)stream; return ScratchT(buf.resource(), count, noInit); }
+}
+
+/// @brief Allocates @c bytes of destination storage for a cross-space
+///        transfer: single-space buffers allocate through @c proto's resource
+///        (or a default-constructed resource without one), on @c stream when
+///        the resource is stream-ordered; buffers providing create() go
+///        through it.
+template<typename DstBufferT>
+inline DstBufferT makeTransferStorage(uint64_t bytes, cudaStream_t stream, const DstBufferT* proto)
+{
+    if constexpr (BufferHasDeviceSingle<DstBufferT>::value || BufferHasHostSingle<DstBufferT>::value) {
+        using ResourceT = typename DstBufferT::ResourceType;
+        if (!proto) {
+            // both branches of a plain conditional would instantiate the
+            // default-resource constructor, breaking non-default-constructible
+            // resources (e.g. ResourceRef) even for callers that pass a proto
+            if constexpr (std::is_default_constructible<ResourceT>::value) {
+                if constexpr (is_async_resource<ResourceT>::value) return DstBufferT(stream, bytes, noInit);
+                else                                               return DstBufferT(bytes, noInit);
+            } else {
+                throw std::runtime_error("cuda::copyTo: a destination buffer over a non-default-constructible "
+                                         "resource requires a prototype buffer");
+            }
+        }
+        if constexpr (is_async_resource<ResourceT>::value) {
+            return DstBufferT(stream, proto->resource(), bytes, noInit);
+        } else {
+            (void)stream;
+            return DstBufferT(proto->resource(), bytes, noInit);
+        }
+    } else {
+        return DstBufferT::create(bytes, proto);
+    }
+}
+
 }// namespace detail
 
 template<typename BufferT, template <class, class...> class VectorT = std::vector>
@@ -52,10 +190,11 @@ splitGridHandles(const GridHandle<BufferT> &handle, const BufferT* other = nullp
     const void *ptr = handle.deviceData();
     if (ptr == nullptr) return VectorT<GridHandle<BufferT>>();
     VectorT<GridHandle<BufferT>> handles(handle.gridCount());
-    bool dirty, *d_dirty;// use this to check if the checksum needs to be recomputed
-    cudaCheck(util::cuda::mallocAsync((void**)&d_dirty, sizeof(bool), stream));
+    Buffer<bool, DeviceResource> dirtyBuf(stream, 1, noInit);
+    bool *d_dirty = dirtyBuf.data();
     int device = util::cuda::currentDevice();
     for (uint32_t n=0; n<handle.gridCount(); ++n) {
+        bool dirty = false;// set when the checksum needs to be recomputed
         auto buffer = BufferT::create(handle.gridSize(n), other, device, stream);
         GridData *dst = reinterpret_cast<GridData*>(buffer.deviceData());
         const GridData *src = reinterpret_cast<const GridData*>(ptr);
@@ -68,7 +207,6 @@ splitGridHandles(const GridHandle<BufferT> &handle, const BufferT* other = nullp
         handles[n] = nanovdb::GridHandle<BufferT>(std::move(buffer));
         ptr = util::PtrAdd(ptr, handle.gridSize(n));
     }
-    cudaCheck(util::cuda::freeAsync(d_dirty, stream));
     return handles;
 }// cuda::splitGridHandles
 
@@ -85,11 +223,12 @@ mergeGridHandles(const VectorT<GridHandle<BufferT>> &handles, const BufferT* oth
     int device = util::cuda::currentDevice();
     auto buffer = BufferT::create(size, other, device, stream);
     void *dst = buffer.deviceData();
-    bool dirty, *d_dirty;// use this to check if the checksum needs to be recomputed
-    cudaCheck(util::cuda::mallocAsync((void**)&d_dirty, sizeof(bool), stream));
+    Buffer<bool, DeviceResource> dirtyBuf(stream, 1, noInit);
+    bool *d_dirty = dirtyBuf.data();
     for (auto &h : handles) {
         const void *src = h.deviceData();
         for (uint32_t n=0; n<h.gridCount(); ++n) {
+            bool dirty = false;// set when the checksum needs to be recomputed
             cudaCheck(cudaMemcpyAsync(dst, src, h.gridSize(n), cudaMemcpyDeviceToDevice, stream));
             GridData *data = reinterpret_cast<GridData*>(dst);
             detail::updateGridCount<<<1, 1, 0, stream>>>(data, counter++, gridCount, d_dirty);
@@ -101,10 +240,82 @@ mergeGridHandles(const VectorT<GridHandle<BufferT>> &handles, const BufferT* oth
             src = util::PtrAdd(src, h.gridSize(n));
         }
     }
-    cudaCheck(util::cuda::freeAsync(d_dirty, stream));
     return GridHandle<BufferT>(std::move(buffer));
 }// cuda::mergeGridHandles
 
+/// @brief Deep-copies a grid handle into a different address space: the
+///        explicit, stream-carrying transfer between single-space device
+///        handles and host-readable handles (HostBuffer or a host-accessible
+///        single-space buffer such as a pinned-resource cuda::Buffer).
+/// @tparam DstBufferT destination buffer type (specify explicitly)
+/// @param src the handle to copy; must not be dual-space (use
+///        deviceUpload/deviceDownload on those)
+/// @param stream stream the copy is issued on; a device destination buffer
+///        with a stream-ordered resource retains it
+/// @warning Passing a stream other than the source buffer's retained stream
+///          makes the caller responsible for ordering: prior work on the
+///          source (and the source's later destruction, which frees on its
+///          own stream) must be ordered against @a stream by the caller,
+///          e.g. with cudaStreamWaitEvent or a synchronization. The
+///          stream-less overload below has no such requirement.
+/// @param proto optional buffer whose resource (or pool, for buffers
+///        providing create()) allocates the destination storage; without it
+///        the destination resource is default-constructed
+/// @return a handle of the destination buffer type with equal contents
+/// @details The returned handle is immediately usable: a host-readable
+///          destination synchronizes @c stream before returning, and a device
+///          destination parses (and validates) its metadata on the
+///          transferred bytes, which synchronizes internally. A pageable host
+///          source or destination (HostBuffer) degrades the copy to
+///          synchronous behavior; pinned single-space handles keep it
+///          asynchronous.
+template<typename DstBufferT, typename SrcBufferT>
+inline GridHandle<DstBufferT> copyTo(const GridHandle<SrcBufferT>& src, cudaStream_t stream, const DstBufferT* proto = nullptr)
+{
+    constexpr bool srcDev = BufferHasDeviceSingle<SrcBufferT>::value;
+    constexpr bool dstDev = BufferHasDeviceSingle<DstBufferT>::value;
+    static_assert(!BufferTraits<SrcBufferT>::hasDeviceDual && !BufferTraits<DstBufferT>::hasDeviceDual,
+                  "cuda::copyTo does not support dual-space buffers: use deviceUpload/deviceDownload on the handle");
+    static_assert(srcDev || dstDev,
+                  "cuda::copyTo is for cross-space transfers involving a device buffer: use GridHandle::copy for host-to-host");
+    const uint64_t bytes = src.bufferSize();
+    if (bytes == 0u) {
+        if constexpr (std::is_default_constructible<DstBufferT>::value) {
+            return GridHandle<DstBufferT>();
+        } else {
+            throw std::runtime_error("cuda::copyTo: an empty handle cannot be copied to a buffer type "
+                                     "that is not default-constructible");
+        }
+    }
+    DstBufferT dst = detail::makeTransferStorage<DstBufferT>(bytes, stream, proto);
+    const void* srcPtr;
+    if constexpr (srcDev) srcPtr = src.deviceData();
+    else                  srcPtr = src.data();
+    constexpr cudaMemcpyKind kind = srcDev ? (dstDev ? cudaMemcpyDeviceToDevice : cudaMemcpyDeviceToHost)
+                                           : cudaMemcpyHostToDevice;
+    cudaCheck(cudaMemcpyAsync(dst.data(), srcPtr, bytes, kind, stream));
+    if constexpr (dstDev) {
+        // A synchronous destination resource retains no stream: order the
+        // metadata parse (which runs on the default stream) after the copy.
+        if constexpr (!is_async_resource<typename DstBufferT::ResourceType>::value)
+            cudaCheck(cudaStreamSynchronize(stream));
+    } else {
+        cudaCheck(cudaStreamSynchronize(stream));// the host-readable result is the postcondition
+    }
+    return GridHandle<DstBufferT>(std::move(dst));// the constructor parses (and validates) the metadata
+}// cuda::copyTo
+
+/// @brief Convenience overload issuing the copy on the source buffer's
+///        retained stream when it has one (any single-space source over a
+///        stream-ordered resource), the default stream otherwise.
+template<typename DstBufferT, typename SrcBufferT>
+inline GridHandle<DstBufferT> copyTo(const GridHandle<SrcBufferT>& src, const DstBufferT* proto = nullptr)
+{
+    cudaStream_t stream = 0;
+    if constexpr (BufferHasStream<SrcBufferT>::value) stream = src.buffer().stream();
+    return copyTo<DstBufferT>(src, stream, proto);
+}// cuda::copyTo (retained stream)
+
 }// namespace cuda
 
 template<typename BufferT, template <class, class...> class VectorT = std::vector>
@@ -122,30 +333,49 @@ mergeDeviceGrids(const VectorT<GridHandle<BufferT>> &handles, const BufferT* oth
 template<typename BufferT>
 template<typename T, typename util::enable_if<BufferTraits<T>::hasDeviceDual, int>::type>
 GridHandle<BufferT>::GridHandle(T&& buffer)
+    : mBuffer(std::move(buffer))
 {
     static_assert(util::is_same<T,BufferT>::value, "Expected U==BufferT");
-    mBuffer = std::move(buffer);
     if (auto *data = reinterpret_cast<const GridData*>(mBuffer.data())) {
         if (!data->isValid()) throw std::runtime_error("GridHandle was constructed with an invalid host buffer");
         mMetaData.resize(data->mGridCount);
         cpyGridHandleMeta(data, mMetaData.data());
     } else {
         if (auto *d_data = reinterpret_cast<const GridData*>(mBuffer.deviceData())) {
-            GridData tmp;
-            cudaCheck(cudaMemcpy(&tmp, d_data, sizeof(GridData), cudaMemcpyDeviceToHost));
-            if (!tmp.isValid()) throw std::runtime_error("GridHandle was constructed with an invalid device buffer");
-            GridHandleMetaData *d_metaData;
-            cudaMalloc((void**)&d_metaData, tmp.mGridCount*sizeof(GridHandleMetaData));
-            cuda::detail::cpyGridHandleMeta<<<1,1>>>(d_data, d_metaData);
-            mMetaData.resize(tmp.mGridCount);
-            cudaCheck(cudaMemcpy(mMetaData.data(), d_metaData,tmp.mGridCount*sizeof(GridHandleMetaData), cudaMemcpyDeviceToHost));
-            cudaCheck(cudaFree(d_metaData));
+            const uint32_t count = cuda::detail::validGridChainHead(d_data, mBuffer.size(), cudaStream_t(0));
+            // MallocResource: plain cudaMalloc, so this long-standing parse path
+            // keeps working on devices without memory-pool support.
+            cuda::Buffer<std::byte, cuda::MallocResource> scratch(cuda::detail::chainScratchSize(count), cuda::noInit);
+            cuda::detail::parseGridChain(d_data, mBuffer.size(), count, scratch, mMetaData, cudaStream_t(0));
         }
     }
 }// GridHandle(T&& buffer)
 
-// Dummy function that ensures instantiation of the move-constructor above when BufferT=cuda::DeviceBuffer
-namespace {auto __dummy(){return GridHandle<cuda::DeviceBuffer>(std::move(cuda::DeviceBuffer()));}}
+// move constructor from a single-space device buffer: all device work runs on
+// the buffer's retained stream (or the default stream for a synchronous
+// resource), and the metadata scratch allocates through the buffer's resource.
+template<typename BufferT>
+template<typename T, typename util::enable_if<BufferHasDeviceSingle<T>::value, int>::type, typename>
+GridHandle<BufferT>::GridHandle(T&& buffer)
+    : mBuffer(std::move(buffer))
+{
+    static_assert(util::is_same<T,BufferT>::value, "Expected U==BufferT");
+    using ResourceT = typename BufferT::ResourceType;
+    if (const GridData *d_data = reinterpret_cast<const GridData*>(mBuffer.data())) {
+        const cudaStream_t stream = cuda::detail::retainedStreamOrDefault(mBuffer);
+        const uint32_t count = cuda::detail::validGridChainHead(d_data, mBuffer.size_bytes(), stream);
+        using ScratchT = cuda::Buffer<std::byte, ResourceT>;
+        ScratchT scratch = cuda::detail::makeMetaScratch<ScratchT>(mBuffer, cuda::detail::chainScratchSize(count), stream);
+        cuda::detail::parseGridChain(d_data, mBuffer.size_bytes(), count, scratch, mMetaData, stream);
+    }
+}// GridHandle(T&& buffer) for single-space device buffers
+
+// Emit the dual-buffer move constructor from every CUDA translation unit, so
+// host-only translation units that construct GridHandle<cuda::DeviceBuffer>
+// (they see only the declaration) always find the symbol at link time. An
+// unused private function is not enough: the optimizer may drop the
+// complete-object constructor it instantiates.
+template GridHandle<cuda::DeviceBuffer>::GridHandle(cuda::DeviceBuffer&&);
 
 } // namespace nanovdb
 

nanovdb/nanovdb/cuda/NodeManager.cuh

@@ -18,6 +18,7 @@
 #define NANOVDB_CUDA_NODE_MANAGER_CUH_HAS_BEEN_INCLUDED
 
 #include <nanovdb/util/cuda/Util.h>// for cuda::lambdaKernel
+#include <nanovdb/cuda/Buffer.h>// for the resource-aware buffers below
 #include <nanovdb/cuda/DeviceBuffer.h>
 #include <nanovdb/NodeManager.h>
 
@@ -25,6 +26,23 @@ namespace nanovdb {
 
 namespace cuda {
 
+namespace detail {
+
+/// @brief Constructs a single-space buffer of @c count elements through
+///        @c ref, on @c stream when the resource is stream-ordered.
+template<typename BufferT, typename RefT>
+inline BufferT makeNodeManagerStorage(cudaStream_t stream, const RefT& ref, uint64_t count)
+{
+    if constexpr (is_async_resource<typename BufferT::ResourceType>::value) {
+        return BufferT(stream, ref, count, noInit);
+    } else {
+        (void)stream;
+        return BufferT(ref, count, noInit);
+    }
+}
+
+}// namespace detail
+
 /// @brief Construct a NodeManager from a device grid pointer
 ///
 /// @param d_grid device grid pointer whose nodes will be accessed sequentially
@@ -41,8 +59,8 @@ createNodeManager(const NanoGrid<BuildT> *d_grid,
     cudaCheck(cudaGetDevice(&device));
     auto buffer = BufferT::create(sizeof(NodeManagerData), &pool, device, stream);
     auto *d_data = (NodeManagerData*)buffer.deviceData();
-    size_t size = 0u, *d_size;
-    cudaCheck(util::cuda::mallocAsync((void**)&d_size, sizeof(size_t), stream));
+    Buffer<size_t, DeviceResource> sizeBuf(stream, 1, noInit);
+    size_t size = 0u, *d_size = sizeBuf.data();
     util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, [=] __device__(size_t) {
         *d_data = NodeManagerData((void*)d_grid);
         *d_size = sizeof(NodeManagerData);
@@ -58,7 +76,7 @@ createNodeManager(const NanoGrid<BuildT> *d_grid,
     });
     cudaCheckError();
     cudaCheck(cudaMemcpyAsync(&size, d_size, sizeof(size_t), cudaMemcpyDeviceToHost, stream));
-    cudaCheck(util::cuda::freeAsync(d_size, stream));
+    cudaCheck(cudaStreamSynchronize(stream));// the branch below reads the copied size on the host
     if (size > sizeof(NodeManagerData)) {
         auto tmp = BufferT::create(size, &pool, device, stream);// only allocate buffer on the device
         cudaCheck(cudaMemcpyAsync(tmp.deviceData(), buffer.deviceData(), sizeof(NodeManagerData), cudaMemcpyDeviceToDevice, stream));
@@ -85,6 +103,76 @@ createNodeManager(const NanoGrid<BuildT> *d_grid,
     return NodeManagerHandle<BufferT>(toGridType<BuildT>(), std::move(buffer));
 }// cuda::createNodeManager
 
+/// @brief Construct a NodeManager over a single-space device buffer whose
+///        storage allocates through @a resource, which the returned handle
+///        borrows (it must outlive the handle) -- the same ownership contract
+///        as the builders.
+///
+/// @param d_grid device grid pointer whose nodes will be accessed sequentially
+/// @param resource device-resident memory resource the handle's storage
+///        allocates through; a host-accessible resource (e.g. PinnedResource)
+///        is excluded, since the result would carry no device NodeManager
+/// @param stream cuda stream
+/// @return Handle over cuda::Buffer<std::byte, ResourceRef<ResourceT>> that
+///         contains a device NodeManager
+template <typename BuildT, typename ResourceT>
+inline typename util::enable_if<(is_resource<ResourceT>::value || is_async_resource<ResourceT>::value)
+                                    && !is_host_accessible_resource<ResourceT>::value,
+                                NodeManagerHandle<Buffer<std::byte, ResourceRef<ResourceT>>>>::type
+createNodeManager(const NanoGrid<BuildT> *d_grid,
+                  ResourceT& resource,
+                  cudaStream_t stream = 0)
+{
+    using RefT    = ResourceRef<ResourceT>;
+    using BufferT = Buffer<std::byte, RefT>;
+    RefT ref(resource);
+    auto buffer = detail::makeNodeManagerStorage<BufferT>(stream, ref, sizeof(NodeManagerData));
+    auto *d_data = reinterpret_cast<NodeManagerData*>(buffer.data());
+    auto sizeBuf = detail::makeNodeManagerStorage<Buffer<size_t, RefT>>(stream, ref, 1);
+    size_t size = 0u, *d_size = sizeBuf.data();
+    util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, [=] __device__(size_t) {
+        *d_data = NodeManagerData((void*)d_grid);
+        *d_size = sizeof(NodeManagerData);
+        auto &tree = d_grid->tree();
+        if (NodeManager<BuildT>::FIXED_SIZE && d_grid->isBreadthFirst()) {
+            d_data->mLinear = uint8_t(1u);
+            d_data->mOff[0] = util::PtrDiff(tree.template getFirstNode<0>(), d_grid);
+            d_data->mOff[1] = util::PtrDiff(tree.template getFirstNode<1>(), d_grid);
+            d_data->mOff[2] = util::PtrDiff(tree.template getFirstNode<2>(), d_grid);
+        } else {
+            *d_size += sizeof(uint64_t)*tree.totalNodeCount();
+        }
+    });
+    cudaCheckError();
+    cudaCheck(cudaMemcpyAsync(&size, d_size, sizeof(size_t), cudaMemcpyDeviceToHost, stream));
+    cudaCheck(cudaStreamSynchronize(stream));// the branch below reads the copied size on the host
+    if (size > sizeof(NodeManagerData)) {
+        auto tmp = detail::makeNodeManagerStorage<BufferT>(stream, ref, size);
+        cudaCheck(cudaMemcpyAsync(tmp.data(), buffer.data(), sizeof(NodeManagerData), cudaMemcpyDeviceToDevice, stream));
+        buffer = std::move(tmp);
+        d_data = reinterpret_cast<NodeManagerData*>(buffer.data());
+        util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, [=] __device__ (size_t) {
+            auto &tree = d_grid->tree();
+            int64_t *ptr0 = d_data->mPtr[0] = reinterpret_cast<int64_t*>(d_data + 1);
+            int64_t *ptr1 = d_data->mPtr[1] = d_data->mPtr[0] + tree.nodeCount(0);
+            int64_t *ptr2 = d_data->mPtr[2] = d_data->mPtr[1] + tree.nodeCount(1);
+            // Performs depth first traversal but breadth first insertion
+            for (auto it2 = tree.root().cbeginChild(); it2; ++it2) {
+                *ptr2++ = util::PtrDiff(&*it2, d_grid);
+                for (auto it1 = it2->cbeginChild(); it1; ++it1) {
+                    *ptr1++ = util::PtrDiff(&*it1, d_grid);
+                    for (auto it0 = it1->cbeginChild(); it0; ++it0) {
+                        *ptr0++ = util::PtrDiff(&*it0, d_grid);
+                    }// loop over child nodes of the lower internal node
+                }// loop over child nodes of the upper internal node
+            }// loop over child nodes of the root node
+        });
+        cudaCheckError();
+    }
+
+    return NodeManagerHandle<BufferT>(toGridType<BuildT>(), std::move(buffer));
+}// cuda::createNodeManager (single-space)
+
 }// namespace cuda
 
 template <typename BuildT, typename BufferT = cuda::DeviceBuffer>

nanovdb/nanovdb/cuda/PinnedResource.h

@@ -30,6 +30,11 @@ public:
     // alignment is always satisfied; 256 is the nominal default advertised here.
     static constexpr size_t DEFAULT_ALIGNMENT = 256;
 
+    /// @brief Pinned allocations are mapped into the host address space, so
+    ///        containers over this resource remain host-readable (detected by
+    ///        nanovdb::cuda::is_host_accessible_resource).
+    static constexpr bool HOST_ACCESSIBLE = true;
+
     /// @brief Synchronous allocation of page-locked host memory.
     /// @param bytes number of bytes to allocate
     /// @param alignment requested alignment (ignored; cudaMallocHost is page-aligned)

nanovdb/nanovdb/cuda/TempPool.h

@@ -10,6 +10,7 @@
 #ifndef NANOVDB_CUDA_TEMPPOOL_H_HAS_BEEN_INCLUDED
 #define NANOVDB_CUDA_TEMPPOOL_H_HAS_BEEN_INCLUDED
 
+#include <nanovdb/cuda/Buffer.h>
 #include <nanovdb/cuda/DeviceResource.h>
 
 #include <cstddef>
@@ -21,31 +22,33 @@ namespace cuda {
 
 template <class Resource>
 class TempPool {
+    static_assert(is_async_resource<Resource>::value,
+                  "TempPool allocates stream-ordered scratch and requires an AsyncResource");
+    // The buffer borrows the pool's resource through a ResourceRef rather than
+    // copying it, preserving the pool's contract that all traffic reaches the
+    // caller's resource instance (which may be stateful).
+    using BufferT = Buffer<std::byte, ResourceRef<Resource>>;
 public:
 
     /// @brief Default c-tor of an empty memory pool that uses the default
     ///        instance of @c Resource for all allocations.
-    TempPool() : mResource(&default_resource<Resource>()), mData(nullptr), mSize(0), mRequestedSize(0), mStream(nullptr) {}
+    TempPool() : TempPool(default_resource<Resource>()) {}
 
     /// @brief C-tor of an empty memory pool that routes all allocations through
     ///        the supplied @c Resource instance.
     /// @param resource resource instance to allocate from; must outlive this pool.
-    explicit TempPool(Resource& resource) : mResource(&resource), mData(nullptr), mSize(0), mRequestedSize(0), mStream(nullptr) {}
-
-    /// @brief Destructor. Frees the managed memory on the stream of the most
-    ///        recent reallocate(), so the stream-ordered free is ordered after
-    ///        the work that used the memory (rather than on the null stream).
-    ~TempPool() {
-        mRequestedSize = 0;
-        mResource->deallocate_async(mData, mSize, Resource::DEFAULT_ALIGNMENT, mStream);
-        mData = nullptr;
-        mSize = 0;
+    explicit TempPool(Resource& resource)
+        : mResource(&resource)
+        , mBuffer(cudaStream_t{0}, ResourceRef<Resource>(resource), 0, noInit)
+    {
     }
 
     /// @brief Returns a non-const void pointer to the data managed by this instance.
-    void* data() {return mData;}
+    void* data() {return mBuffer.data();}
 
     /// @brief Returns a non-const reference to the actual size of the data managed by this instance.
+    /// @note Returned by reference because cub's two-pass API takes the storage
+    ///       size as a size_t&, so this cannot forward Buffer::size() by value.
     size_t& size() {return mSize;}
 
     /// @brief Returns a non-const reference to the requested size of the data managed by this instance.
@@ -54,25 +57,27 @@ public:
 
     /// @brief Returns the stream that the managed memory was last (re)allocated on,
     ///        i.e. the stream this pool will free on at destruction.
-    cudaStream_t stream() const {return mStream;}
+    cudaStream_t stream() const {return mBuffer.stream();}
 
     /// @brief Re-allocation of the data managed by this instance. Only has affect if the pool in empty or
     ///        the requested memory is larger than the existing size.
     /// @param stream cuda stream used for asynchronous de-allocation and allocation.
+    /// @note Scratch is discarded, never resized: preserving a prefix of
+    ///       temporary storage would be a wasted copy.
     void reallocate(cudaStream_t stream) {
-        if (!mData || mRequestedSize > mSize) {
-            mResource->deallocate_async(mData, mSize, Resource::DEFAULT_ALIGNMENT, stream);
-            mData = mResource->allocate_async(mRequestedSize, Resource::DEFAULT_ALIGNMENT, stream);
-            mSize = mRequestedSize;
+        if (mBuffer.empty() || mRequestedSize > mSize) {
+            mBuffer.destroy(stream);// free the outgrown block on this stream
+            mBuffer = BufferT(stream, ResourceRef<Resource>(*mResource), mRequestedSize, noInit);
+            mSize = mBuffer.size();
+        } else {
+            mBuffer.set_stream(stream);// retained so the d-tor frees on the most-recently-used stream
         }
-        mStream = stream;// retained so the destructor frees on the most-recently-used stream
     }
 private:
-    Resource    *mResource;
-    void        *mData;
-    size_t       mSize;
-    size_t       mRequestedSize;
-    cudaStream_t mStream;
+    Resource *mResource;// non-owning; must outlive this pool and its buffer
+    BufferT   mBuffer;
+    size_t    mSize{0};
+    size_t    mRequestedSize{0};
 };// TempPool<Resource> class
 
 using TempDevicePool = TempPool<DeviceResource>;

nanovdb/nanovdb/examples/CMakeLists.txt

@@ -117,8 +117,28 @@ nanovdb_example(NAME "ex_coarsen_nanovdb_cuda" OPENVDB)
 nanovdb_example(NAME "ex_mesh_to_grid_cuda" OPENVDB)
 
 if(CUDAToolkit_FOUND)
-  nanovdb_example(NAME "ex_make_mgpu_nanovdb") # requires cuRAND
-  target_link_libraries(ex_make_mgpu_nanovdb PRIVATE CUDA::curand)
+  # ex_make_mgpu_nanovdb uses TF32 WMMA which requires SM_80+ (Ampere or newer).
+  # Gate on CMAKE_CUDA_ARCHITECTURES (what the build actually targets) rather than
+  # probing the configure-time machine's GPU: the two can differ (cross-compiling,
+  # building on one machine to run on another, CI runners without a GPU), and
+  # probing via "bash -c nvidia-smi ..." isn't portable to Windows either way.
+  set(_nanovdb_has_sm80 OFF)
+  foreach(_cuda_arch IN LISTS CMAKE_CUDA_ARCHITECTURES)
+    string(REGEX MATCH "^[0-9]+" _cuda_arch_number "${_cuda_arch}")
+    if(_cuda_arch_number GREATER_EQUAL 80)
+      set(_nanovdb_has_sm80 ON)
+    endif()
+  endforeach()
+
+  if(_nanovdb_has_sm80)
+    nanovdb_example(NAME "ex_make_mgpu_nanovdb") # requires cuRAND and SM_80+ (TF32 WMMA)
+    target_link_libraries(ex_make_mgpu_nanovdb PRIVATE CUDA::curand)
+  else()
+    message(STATUS "Skipping ex_make_mgpu_nanovdb: requires CUDA architecture 80 or newer (CMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES})")
+  endif()
+  unset(_nanovdb_has_sm80)
+  unset(_cuda_arch_number)
+  unset(_cuda_arch)
 endif()
 
 if(NANOVDB_USE_MAGICAVOXEL)

nanovdb/nanovdb/examples/ex_collide_level_set/common.h

@@ -3,7 +3,9 @@
 
 #pragma once
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 #include <fstream>

nanovdb/nanovdb/examples/ex_collide_level_set/nanovdb.cu

@@ -1,7 +1,9 @@
 // Copyright Contributors to the OpenVDB Project
 // SPDX-License-Identifier: Apache-2.0
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 

nanovdb/nanovdb/examples/ex_collide_level_set/openvdb.cc

@@ -3,7 +3,9 @@
 
 #if defined(NANOVDB_USE_OPENVDB)
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 

nanovdb/nanovdb/examples/ex_nodemanager_cuda/nodemanager_cuda_kernel.cu

@@ -31,7 +31,7 @@ extern "C" void launch_kernels(const nanovdb::NodeManager<float>* deviceMgr,
 
 // Simple wrapper that makes sure nanovdb::cuda::createNodeManager is initiated
 extern "C" void cudaCreateNodeManager(const nanovdb::NanoGrid<float> *d_grid,
-                                      nanovdb::NodeManagerHandle<nanovdb::CudaDeviceBuffer> *handle)
+                                      nanovdb::NodeManagerHandle<nanovdb::cuda::DeviceBuffer> *handle)
 {
     *handle = std::move(nanovdb::cuda::createNodeManager<float>(d_grid));
-}
+}

nanovdb/nanovdb/examples/ex_raytrace_fog_volume/common.h

@@ -3,7 +3,9 @@
 
 #pragma once
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 #include <fstream>

nanovdb/nanovdb/examples/ex_raytrace_fog_volume/nanovdb.cu

@@ -1,7 +1,9 @@
 // Copyright Contributors to the OpenVDB Project
 // SPDX-License-Identifier: Apache-2.0
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 

nanovdb/nanovdb/examples/ex_raytrace_fog_volume/openvdb.cc

@@ -3,7 +3,9 @@
 
 #if defined(NANOVDB_USE_OPENVDB)
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 

nanovdb/nanovdb/examples/ex_raytrace_iso_surface/common.h

@@ -3,7 +3,9 @@
 
 #pragma once
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 #include <fstream>

nanovdb/nanovdb/examples/ex_raytrace_iso_surface/nanovdb.cu

@@ -1,7 +1,9 @@
 // Copyright Contributors to the OpenVDB Project
 // SPDX-License-Identifier: Apache-2.0
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 

nanovdb/nanovdb/examples/ex_raytrace_level_set/common.h

@@ -3,7 +3,9 @@
 
 #pragma once
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 #include <fstream>
@@ -44,11 +46,41 @@ inline __hostdev__ void mortonEncode(uint32_t& code, uint32_t x, uint32_t y)
 template<typename RenderFn, typename GridT>
 inline float renderImage(bool useCuda, const RenderFn renderOp, int width, int height, float* image, const GridT* grid)
 {
+#if defined(__CUDACC__)
+    // Kernel-only timing on the GPU path. cudaEventElapsedTime measures only
+    // the work between the two record points on the default stream, so it
+    // excludes host-side launch latency and the cudaDeviceSynchronize wakeup
+    // that std::chrono would otherwise pick up. Events are created/destroyed
+    // per call; the ~10us creation cost is dwarfed by the kernel itself.
+    if (useCuda) {
+        cudaEvent_t startEv, stopEv;
+        NANOVDB_CUDA_CHECK_ERROR(cudaEventCreate(&startEv), __FILE__, __LINE__);
+        NANOVDB_CUDA_CHECK_ERROR(cudaEventCreate(&stopEv), __FILE__, __LINE__);
+
+        NANOVDB_CUDA_CHECK_ERROR(cudaEventRecord(startEv, 0), __FILE__, __LINE__);
+
+        computeForEach(
+            true, width * height, 256, __FILE__, __LINE__, [renderOp, image, grid] __hostdev__(int start, int end) {
+                renderOp(start, end, image, grid);
+            });
+
+        NANOVDB_CUDA_CHECK_ERROR(cudaEventRecord(stopEv, 0), __FILE__, __LINE__);
+        NANOVDB_CUDA_CHECK_ERROR(cudaEventSynchronize(stopEv), __FILE__, __LINE__);
+
+        float ms = 0.f;
+        NANOVDB_CUDA_CHECK_ERROR(cudaEventElapsedTime(&ms, startEv, stopEv), __FILE__, __LINE__);
+
+        NANOVDB_CUDA_CHECK_ERROR(cudaEventDestroy(startEv), __FILE__, __LINE__);
+        NANOVDB_CUDA_CHECK_ERROR(cudaEventDestroy(stopEv), __FILE__, __LINE__);
+        return ms;
+    }
+#endif
+
     using ClockT = std::chrono::high_resolution_clock;
     auto t0 = ClockT::now();
 
     computeForEach(
-        useCuda, width * height, 512, __FILE__, __LINE__, [renderOp, image, grid] __hostdev__(int start, int end) {
+        useCuda, width * height, 256, __FILE__, __LINE__, [renderOp, image, grid] __hostdev__(int start, int end) {
             renderOp(start, end, image, grid);
         });
     computeSync(useCuda, __FILE__, __LINE__);

nanovdb/nanovdb/examples/ex_raytrace_level_set/main.cc

@@ -33,7 +33,7 @@ int main(int ac, char** av)
             throw std::runtime_error("Grid must be a level set");
         }
 
-        const int numIterations = 50;
+        const int numIterations = 200;
 
         const int width = 1024;
         const int height = 1024;

nanovdb/nanovdb/examples/ex_raytrace_level_set/nanovdb.cu

@@ -1,9 +1,13 @@
 // Copyright Contributors to the OpenVDB Project
 // SPDX-License-Identifier: Apache-2.0
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
+#include <algorithm>
 #include <cmath>
 #include <chrono>
+#include <vector>
 
 #if defined(NANOVDB_USE_CUDA)
 #include <nanovdb/cuda/DeviceBuffer.h>
@@ -18,6 +22,34 @@ using BufferT = nanovdb::HostBuffer;
 
 #include "common.h"
 
+// Number of untimed warmup iterations to run before the timed loop. These
+// pay the kernel cold-start, lazy module load and GPU clock ramp-up costs
+// that would otherwise contaminate the first measured iteration.
+static constexpr int NUM_WARMUP_ITERATIONS = 2;
+
+// Print min / median / mean / max instead of just mean. A single outlier
+// (context switch, OS jitter, thermal blip) pulls the mean noticeably but
+// leaves the median alone, so the median is the more trustworthy stat.
+static void
+reportStats(const char *label, const std::vector<float> &samples)
+{
+    if (samples.empty()) return;
+    std::vector<float> sorted(samples); // copy so we don't reorder the caller's data
+    std::sort(sorted.begin(), sorted.end());
+    const float minMs = sorted.front();
+    const float maxMs = sorted.back();
+    const float medianMs = sorted[sorted.size() / 2];
+    float sum = 0;
+    for (float s : sorted) sum += s;
+    const float meanMs = sum / float(sorted.size());
+    std::cout << label
+              << " min=" << minMs << " ms"
+              << "  median=" << medianMs << " ms"
+              << "  mean=" << meanMs << " ms"
+              << "  max=" << maxMs << " ms"
+              << "  (n=" << sorted.size() << ")" << std::endl;
+}
+
 void runNanoVDB(nanovdb::GridHandle<BufferT>& handle, int numIterations, int width, int height, BufferT& imageBuffer)
 {
     using GridT  = nanovdb::FloatGrid;
@@ -69,14 +101,16 @@ void runNanoVDB(nanovdb::GridHandle<BufferT>& handle, int numIterations, int wid
     };
 
     {
-        float durationAvg = 0;
-        for (int i = 0; i < numIterations; ++i) {
-            float duration = renderImage(false, renderOp, width, height, h_outImage, h_grid);
-            //std::cout << "Duration(NanoVDB-Host) = " << duration << " ms" << std::endl;
-            durationAvg += duration;
+        for (int i = 0; i < NUM_WARMUP_ITERATIONS; ++i) {
+            (void)renderImage(false, renderOp, width, height, h_outImage, h_grid);
         }
-        durationAvg /= numIterations;
-        std::cout << "Average Duration(NanoVDB-Host) = " << durationAvg << " ms" << std::endl;
+
+        std::vector<float> samples;
+        samples.reserve(numIterations);
+        for (int i = 0; i < numIterations; ++i) {
+            samples.push_back(renderImage(false, renderOp, width, height, h_outImage, h_grid));
+        }
+        reportStats("Duration(NanoVDB-Host):", samples);
 
         saveImage("raytrace_level_set-nanovdb-host.pfm", width, height, (float*)imageBuffer.data());
     }
@@ -92,14 +126,16 @@ void runNanoVDB(nanovdb::GridHandle<BufferT>& handle, int numIterations, int wid
     float* d_outImage = reinterpret_cast<float*>(imageBuffer.deviceData());
 
     {
-        float durationAvg = 0;
-        for (int i = 0; i < numIterations; ++i) {
-            float duration = renderImage(true, renderOp, width, height, d_outImage, d_grid);
-            //std::cout << "Duration(NanoVDB-Cuda) = " << duration << " ms" << std::endl;
-            durationAvg += duration;
+        for (int i = 0; i < NUM_WARMUP_ITERATIONS; ++i) {
+            (void)renderImage(true, renderOp, width, height, d_outImage, d_grid);
         }
-        durationAvg /= numIterations;
-        std::cout << "Average Duration(NanoVDB-Cuda) = " << durationAvg << " ms" << std::endl;
+
+        std::vector<float> samples;
+        samples.reserve(numIterations);
+        for (int i = 0; i < numIterations; ++i) {
+            samples.push_back(renderImage(true, renderOp, width, height, d_outImage, d_grid));
+        }
+        reportStats("Duration(NanoVDB-Cuda):", samples);
 
         imageBuffer.deviceDownload();
         saveImage("raytrace_level_set-nanovdb-cuda.pfm", width, height, (float*)imageBuffer.data());

nanovdb/nanovdb/examples/ex_raytrace_level_set/openvdb.cc

@@ -3,7 +3,9 @@
 
 #if defined(NANOVDB_USE_OPENVDB)
 
+#ifndef _USE_MATH_DEFINES
 #define _USE_MATH_DEFINES
+#endif
 #include <cmath>
 #include <chrono>
 

nanovdb/nanovdb/examples/ex_read_nanovdb_sphere_accessor_cuda/read_nanovdb_sphere_accessor_cuda.cu

@@ -17,7 +17,7 @@ int main(int, char**)
 {
     try {
         // returns a GridHandle using CUDA for memory management.
-        auto handle = nanovdb::io::readGrid<nanovdb::CudaDeviceBuffer>("data/sphere.nvdb");
+        auto handle = nanovdb::io::readGrid<nanovdb::cuda::DeviceBuffer>("data/sphere.nvdb");
 
         cudaStream_t stream; // Create a CUDA stream to allow for asynchronous copy of pinned CUDA memory.
         cudaStreamCreate(&stream);

nanovdb/nanovdb/python/PyBuildGrid.cc

@@ -21,7 +21,7 @@ namespace pynanovdb {
 
 // Bind nanovdb::tools::build::Grid<BuildT> together with its
 // ValueAccessor<BuildT> and Tree<BuildT>::WriteAccessor proxies, plus a
-// .to_nanovdb() shortcut that bakes the build grid into a host NanoGrid
+// .toNanoVDB() shortcut that bakes the build grid into a host NanoGrid
 // via tools::createNanoGrid.
 //
 // One instantiation per writable BuildT in BuildTypes.def (scalars +
@@ -44,7 +44,7 @@ static void defineBuildGrid(nb::module_& m,
     // ----- build::Grid<BuildT> -----
     nb::class_<GridT>(m, gridName,
         "Mutable host-side grid builder for this BuildT. Write voxels via "
-        "setValue / getAccessor / getWriteAccessor, then call to_nanovdb() "
+        "setValue / getAccessor / getWriteAccessor, then call toNanoVDB() "
         "to bake a frozen NanoGrid GridHandle.")
         .def(nb::init<const ValueT&, const std::string&, GridClass>(),
             "background"_a,
@@ -133,7 +133,7 @@ static void defineBuildGrid(nb::module_& m,
         // Baking a large grid is the most expensive operation on this
         // class; release the GIL so other Python threads can run during
         // the conversion (the lambda only touches C++ state).
-        .def("to_nanovdb",
+        .def("toNanoVDB",
             [](const GridT& self,
                tools::StatsMode sMode,
                CheckMode        cMode,
@@ -220,7 +220,7 @@ void defineBuildGridModule(nb::module_& toolsModule)
         "Mutable, voxel-by-voxel CPU grid builder mirroring "
         "nanovdb::tools::build::*. Construct a typed Grid (e.g. "
         "FloatGrid(0.0, 'mygrid')), populate it with setValue / "
-        "ValueAccessor / WriteAccessor, then call .to_nanovdb() to bake "
+        "ValueAccessor / WriteAccessor, then call .toNanoVDB() to bake "
         "a host NanoGrid handle.";
 
     // X-macro instantiation over every writable BuildT: scalars (full

nanovdb/nanovdb/python/PyCreateNanoGrid.cc

@@ -13,7 +13,6 @@
 
 #include <cstring>
 #include <string>
-#include <vector>
 
 namespace nb = nanobind;
 using namespace nb::literals;
@@ -107,6 +106,8 @@ nb::object tryQuantizeFpX(nb::handle       py_src,
         }
         return nb::cast(std::move(handle));
     }
+    // Invalid (not None) object signals "SrcBuildT didn't match"; the caller
+    // tests is_valid() and falls through to the next SrcBuildT.
     return nb::object();
 }
 
@@ -165,6 +166,8 @@ nb::object tryQuantizeFpN(nb::handle       py_src,
         }
         return nb::cast(std::move(handle));
     }
+    // Invalid (not None) object signals "SrcBuildT didn't match"; the caller
+    // tests is_valid() and falls through to the next SrcBuildT.
     return nb::object();
 }
 
@@ -220,6 +223,8 @@ nb::object tryIndexify(nb::handle py_src,
         }
         return nb::cast(std::move(handle));
     }
+    // Invalid (not None) object signals "SrcBuildT didn't match"; the caller
+    // tests is_valid() and falls through to the next SrcBuildT.
     return nb::object();
 }
 
@@ -362,6 +367,8 @@ private:
         using BuildSrcT = tools::build::Grid<SrcBuildT>;
         if (nb::isinstance<NanoSrcT>(mSrc)) return this->bake(nb::cast<const NanoSrcT&>(mSrc));
         if (nb::isinstance<BuildSrcT>(mSrc)) return this->bake(nb::cast<const BuildSrcT&>(mSrc));
+        // Invalid (not None) object signals "SrcBuildT didn't match"; the caller
+        // tests is_valid() and falls through to the next SrcBuildT.
         return nb::object();
     }
 

nanovdb/nanovdb/python/PyGridHandle.h

@@ -8,9 +8,7 @@
 
 #include <nanovdb/GridHandle.h>
 #include <nanovdb/NanoVDB.h>
-#include <nanovdb/tools/GridChecksum.h>  // for tools::updateGridCount
 
-#include <cstring>
 #include <vector>
 
 namespace nb = nanobind;
@@ -74,53 +72,18 @@ template<typename BufferT> void defineGridHandleUtilities(nb::module_& m)
     }, nb::arg("handle"),
        "Split a multi-grid handle into a list of single-grid handles, "
        "each owning a freshly-allocated buffer.");
-    // mergeGrids: walk the Python sequence by CONST ref to each handle and
-    // concatenate buffer bytes into a freshly-allocated output. The original
-    // nanovdb::mergeGrids takes a `const std::vector<GridHandle>&`, but
-    // building such a vector from a Python list requires moving from each
-    // wrapper (GridHandle is move-only) — which would silently empty the
-    // caller's `h1`/`h2` Python objects. Instead we inline the merge logic
-    // here so we never need to move from the inputs.
+    // mergeGrids: collect non-owning pointers to each handle and forward to
+    // the borrowing nanovdb::mergeGrids overload. The vector-of-handles
+    // overload is unusable from a Python list because GridHandle is move-only
+    // — building the vector would move from (and silently empty) the caller's
+    // Python objects.
     m.def("mergeGrids", [](nb::sequence handles) {
-        // Collect const refs so we touch each Python wrapper exactly once.
         std::vector<const HandleT*> sources;
         sources.reserve(nb::len(handles));
         for (nb::handle item : handles) {
             sources.push_back(&nb::cast<const HandleT&>(item));
         }
-
-        uint64_t totalSize = 0;
-        uint32_t totalGrids = 0;
-        for (const HandleT* h : sources) {
-            totalGrids += h->gridCount();
-            for (uint32_t n = 0; n < h->gridCount(); ++n) {
-                totalSize += h->gridSize(n);
-            }
-        }
-
-        // Nothing to merge (empty sequence, or only empty handles): return an
-        // empty handle. BufferT::create(0) is ill-defined — for HostBuffer it
-        // yields a non-null data() over a zero-byte region, and the
-        // GridHandle(buffer) ctor would then read a full GridData header out of
-        // it (heap overflow / "invalid host buffer" throw).
-        if (totalGrids == 0) return HandleT();
-
-        auto buffer = BufferT::create(totalSize);
-        uint8_t* dst = static_cast<uint8_t*>(buffer.data());
-        uint32_t writeIndex = 0;
-        for (const HandleT* h : sources) {
-            for (uint32_t n = 0; n < h->gridCount(); ++n) {
-                // gridData(n) is the authoritative per-grid start pointer (it
-                // applies mMetaData[n].offset), so we don't assume the source
-                // grids are laid out contiguously in the buffer.
-                const uint64_t gs = h->gridSize(n);
-                std::memcpy(dst, h->gridData(n), gs);
-                auto* gd = reinterpret_cast<nanovdb::GridData*>(dst);
-                nanovdb::tools::updateGridCount(gd, writeIndex++, totalGrids);
-                dst += gs;
-            }
-        }
-        return HandleT(std::move(buffer));
+        return nanovdb::mergeGrids<BufferT>(sources.data(), sources.size());
     }, nb::arg("handles"),
        "Combine a list of GridHandles into a single multi-grid GridHandle. "
        "Input handles are read by const reference; the new handle owns a "

nanovdb/nanovdb/python/examples/README.md

@@ -23,7 +23,7 @@ PYTHONPATH=. python /path/to/<example>.py
 | Script | What it shows |
 | --- | --- |
 | [`load_inspect.py`](load_inspect.py) | Polymorphic `handle.grid(n)` access, `GridMetaData` type-erased introspection, mixed-type handles via `mergeGrids`. |
-| [`build_grid.py`](build_grid.py) | Voxel-by-voxel construction with `nanovdb.tools.build.FloatGrid`, the cached `ValueAccessor`, the thread-safe `WriteAccessor`, and `.to_nanovdb()` to bake into a host `GridHandle`. |
+| [`build_grid.py`](build_grid.py) | Voxel-by-voxel construction with `nanovdb.tools.build.FloatGrid`, the cached `ValueAccessor`, the thread-safe `WriteAccessor`, and `.toNanoVDB()` to bake into a host `GridHandle`. |
 | [`bulk_leaf_numpy.py`](bulk_leaf_numpy.py) | Zero-copy `(N_leaves, 512)` NumPy view of every leaf's mValues via `grid.leaf_values()`. Includes a global-stats reduction and an in-place mutation that propagates back into the grid. Requires NumPy. |
 | [`quantize.py`](quantize.py) | Quantize a `NanoGrid<float>` through `nanovdb.tools.createNanoGridFp{4,8,16,N}`. Shows fixed-width quantization with dithering and variable-width `FpN` with both `AbsDiff` and `RelDiff` oracles. |
 | [`validate.py`](validate.py) | `nanovdb.tools.validateGrid` / `validateGrids`, `checkGrid`, `isValid`, and the `evalChecksum` / `validateChecksum` / `updateChecksum` round-trip. |

nanovdb/nanovdb/python/examples/build_grid.py

@@ -65,14 +65,14 @@ def main():
         print(f"  nodeCount (leaf, lower, upper) = {g.nodeCount()}")
 
         # Bake into a host NanoGrid<float>.
-        handle = g.to_nanovdb(sMode=nanovdb.tools.StatsMode.All)
+        handle = g.toNanoVDB(sMode=nanovdb.tools.StatsMode.All)
         ng = handle.grid()
         print(f"  baked NanoGrid: type={ng.gridType()}, "
               f"active={ng.activeVoxelCount()}, "
               f"worldBBox={ng.worldBBox()}")
 
         # The build grid is left untouched — we can bake again.
-        handle2 = g.to_nanovdb()
+        handle2 = g.toNanoVDB()
         assert handle2.grid().activeVoxelCount() == ng.activeVoxelCount()
 
 

nanovdb/nanovdb/python/examples/make_typed_grids.py

@@ -44,7 +44,7 @@ def build_typed_grids():
         grid = cls(background, name, nanovdb.GridClass.Unknown)
         for ijk in solid_sphere_coords():
             grid.setValue(ijk, value)
-        handles.append(grid.to_nanovdb())
+        handles.append(grid.toNanoVDB())
         print(f"built {name} ({cls.__name__})")
     return handles
 

nanovdb/nanovdb/python/test/TestNanoVDB.py

@@ -1774,7 +1774,7 @@ class TestGridStats(unittest.TestCase):
         g = nanovdb.tools.build.FloatGrid(0.0, "stats", nanovdb.GridClass.FogVolume)
         for i in range(5):
             g.setValue(nanovdb.math.Coord(i, 0, 0), float(i + 1))
-        return g.to_nanovdb(sMode=nanovdb.tools.StatsMode.All)
+        return g.toNanoVDB(sMode=nanovdb.tools.StatsMode.All)
 
     def test_extrema_default_and_add(self):
         ex = nanovdb.tools.FloatExtrema()
@@ -1829,7 +1829,7 @@ class TestGridStats(unittest.TestCase):
         g = nanovdb.tools.build.FloatGrid(0.0)
         for i in range(3):
             g.setValue(nanovdb.math.Coord(i, 0, 0), float(i + 10))
-        h = g.to_nanovdb(sMode=nanovdb.tools.StatsMode.Disable)
+        h = g.toNanoVDB(sMode=nanovdb.tools.StatsMode.Disable)
         ng = h.grid()
         nanovdb.tools.updateGridStats(ng, nanovdb.tools.StatsMode.All)
         # checkGrid still passes after writing stats.
@@ -2005,12 +2005,12 @@ class TestBuildGrid(unittest.TestCase):
         self.assertEqual(g.getValue(ijk), 3.5)
         self.assertTrue(g.isActive(ijk))
 
-    def test_to_nanovdb_roundtrip(self):
+    def test_tonanovdb_roundtrip(self):
         g = nanovdb.tools.build.FloatGrid(0.0, "trip", nanovdb.GridClass.FogVolume)
         g.setValue(nanovdb.math.Coord(0, 0, 0), 1.0)
         g.setValue(nanovdb.math.Coord(1, 0, 0), 2.0)
         g.setValue(nanovdb.math.Coord(2, 0, 0), 3.0)
-        h = g.to_nanovdb()
+        h = g.toNanoVDB()
         self.assertEqual(h.gridCount(), 1)
         ng = h.grid()
         self.assertEqual(ng.gridType(), nanovdb.GridType.Float)
@@ -2018,13 +2018,13 @@ class TestBuildGrid(unittest.TestCase):
         self.assertEqual(ng.gridName(), "trip")
         self.assertEqual(ng.activeVoxelCount(), 3)
 
-    def test_to_nanovdb_does_not_consume_source(self):
-        # Source build::Grid must remain usable after .to_nanovdb().
+    def test_tonanovdb_does_not_consume_source(self):
+        # Source build::Grid must remain usable after .toNanoVDB().
         g = nanovdb.tools.build.FloatGrid(0.0)
         g.setValue(nanovdb.math.Coord(0, 0, 0), 1.0)
-        _ = g.to_nanovdb()
+        _ = g.toNanoVDB()
         g.setValue(nanovdb.math.Coord(1, 0, 0), 2.0)
-        h2 = g.to_nanovdb()
+        h2 = g.toNanoVDB()
         self.assertEqual(h2.grid().activeVoxelCount(), 2)
 
     def test_int32_build_grid(self):
@@ -2033,7 +2033,7 @@ class TestBuildGrid(unittest.TestCase):
         g.setValue(nanovdb.math.Coord(1, 1, 1), -7)
         self.assertEqual(g.getValue(nanovdb.math.Coord(0, 0, 0)), 42)
         self.assertEqual(g.getValue(nanovdb.math.Coord(1, 1, 1)), -7)
-        h = g.to_nanovdb()
+        h = g.toNanoVDB()
         self.assertEqual(h.grid().gridType(), nanovdb.GridType.Int32)
         self.assertEqual(h.grid().activeVoxelCount(), 2)
 
@@ -2043,14 +2043,14 @@ class TestBuildGrid(unittest.TestCase):
         v = nanovdb.math.Vec3f(1.0, 2.0, 3.0)
         g.setValue(nanovdb.math.Coord(0, 0, 0), v)
         self.assertEqual(g.getValue(nanovdb.math.Coord(0, 0, 0)), v)
-        h = g.to_nanovdb()
+        h = g.toNanoVDB()
         self.assertEqual(h.grid().gridType(), nanovdb.GridType.Vec3f)
 
     def test_set_transform(self):
         g = nanovdb.tools.build.FloatGrid(0.0)
         g.setTransform(scale=0.5, translation=nanovdb.math.Vec3d(1.0, 2.0, 3.0))
         g.setValue(nanovdb.math.Coord(0, 0, 0), 1.0)
-        h = g.to_nanovdb()
+        h = g.toNanoVDB()
         ng = h.grid()
         vs = ng.voxelSize()
         self.assertAlmostEqual(vs[0], 0.5)

nanovdb/nanovdb/tools/VoxelBlockManager.h

@@ -142,7 +142,16 @@ public:
 
     ~VoxelBlockManagerHandle() { this->reset(); }
     /// @brief clear the buffer
-    void reset() { mFirstLeafID.clear(); mJumpMap.clear(); mBlockCount = 0; }
+    void reset() {
+        if constexpr (BufferHasDestroy<BufferT>::value) {
+            mFirstLeafID.destroy();
+            mJumpMap.destroy();
+        } else {
+            mFirstLeafID.clear();
+            mJumpMap.clear();
+        }
+        mBlockCount = 0;
+    }
 
     /// @brief Returns a non-const pointer to the firstLeafID device-hosted data
     ///

nanovdb/nanovdb/tools/cuda/AddBlindData.cuh

@@ -18,6 +18,7 @@
 #define NANOVDB_TOOLS_CUDA_ADDBLINDDATA_CUH_HAS_BEEN_INCLUDED
 
 #include <nanovdb/NanoVDB.h>
+#include <nanovdb/cuda/Buffer.h>
 #include <nanovdb/cuda/DeviceBuffer.h>
 #include <nanovdb/GridHandle.h>
 #include <nanovdb/util/cuda/Util.h>
@@ -34,6 +35,7 @@ namespace tools::cuda {// ============================================
 /// @tparam BuildT Build type of the grid
 /// @tparam BlindDataT Type of the blind data
 /// @tparam BufferT Type of the buffer used for allocation
+/// @tparam ResourceT Template type of optional resource used for internal temporary memory
 /// @param d_grid Pointer to device grid
 /// @param d_blindData Pointer to device blind data
 /// @param valueCount number of values in the blind data
@@ -43,7 +45,7 @@ namespace tools::cuda {// ============================================
 /// @param pool optional pool used for allocation
 /// @param stream optional CUDA stream (defaults to CUDA stream 0)
 /// @return GridHandle with blind data appended
-template<typename BuildT, typename BlindDataT, typename BufferT = nanovdb::cuda::DeviceBuffer>
+template<typename BuildT, typename BlindDataT, typename BufferT = nanovdb::cuda::DeviceBuffer, typename ResourceT = nanovdb::cuda::DeviceResource>
 GridHandle<BufferT>
 addBlindData(const NanoGrid<BuildT> *d_grid,
              const BlindDataT *d_blindData,
@@ -60,11 +62,16 @@ addBlindData(const NanoGrid<BuildT> *d_grid,
     //        old grid    old meta   new meta    old data    new data
 
     static_assert(BufferTraits<BufferT>::hasDeviceDual, "Expected BufferT to support device allocation");
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "addBlindData allocates stream-ordered scratch and requires an AsyncResource");
 
     // extract byte sizes of the grid, blind meta data and blind data
     enum {GRID=0, META=1, DATA=2, CHECKSUM=3};
-    uint64_t tmp[4], *d_tmp;
-    cudaCheck(util::cuda::mallocAsync((void**)&d_tmp, 4*sizeof(uint64_t), stream));
+    using ScratchT = nanovdb::cuda::Buffer<uint64_t, nanovdb::cuda::ResourceRef<ResourceT>>;
+    auto &resource = nanovdb::cuda::default_resource<ResourceT>();
+    uint64_t tmp[4];
+    ScratchT tmpBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 4, nanovdb::cuda::noInit);
+    uint64_t *d_tmp = tmpBuf.data();
     util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, [=] __device__(size_t) {
         if (auto count  = d_grid->blindDataCount()) {
             d_tmp[GRID] = util::PtrDiff(&d_grid->blindMetaData(0), d_grid);
@@ -110,13 +117,15 @@ addBlindData(const NanoGrid<BuildT> *d_grid,
     // increment grid size and blind data counter in output grid
     util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, [=] __device__(size_t) {
         auto &grid = *reinterpret_cast<NanoGrid<BuildT>*>(d_data);
+        grid.mGridIndex = 0u; // Possibly overwriting input; returned grid has batch size 1
+        grid.mGridCount = 1u;
         grid.mBlindMetadataCount += 1;
         grid.mBlindMetadataOffset = d_tmp[GRID];
         auto *meta = util::PtrAdd<GridBlindMetaData>(d_data, grid.mBlindMetadataOffset);// points to first blind meta data
         for (uint32_t i=0, n=grid.mBlindMetadataCount-1; i<n; ++i, ++meta) meta->mDataOffset += sizeof(GridBlindMetaData);
         grid.mGridSize += sizeof(GridBlindMetaData) + meta->blindDataSize();// expansion with 32 byte alignment
     }); cudaCheckError();
-    cudaCheck(util::cuda::freeAsync(d_tmp, stream));
+    tmpBuf.destroy(stream);
 
     Checksum cs(tmp[CHECKSUM]);
     cuda::updateChecksum(reinterpret_cast<GridData*>(d_data), cs.mode(), stream);

nanovdb/nanovdb/tools/cuda/CoarsenGrid.cuh

@@ -30,9 +30,12 @@ namespace nanovdb {
 
 namespace tools::cuda {
 
-template <typename BuildT>
+template <typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class CoarsenGrid
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "CoarsenGrid allocates stream-ordered scratch and requires an AsyncResource");
+
     using GridT  = NanoGrid<BuildT>;
     using TreeT  = NanoTree<BuildT>;
     using RootT  = NanoRoot<BuildT>;
@@ -43,8 +46,11 @@ public:
     /// @brief Constructor
     /// @param deviceGrid source device grid to be coarsened
     /// @param stream optional CUDA stream (defaults to CUDA stream 0)
-    CoarsenGrid(const GridT* d_srcGrid, cudaStream_t stream = 0)
-        : mBuilder(stream), mStream(stream), mTimer(stream), mDeviceSrcGrid(d_srcGrid) {}
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this operator (defaults to the per-type default resource)
+    CoarsenGrid(const GridT* d_srcGrid, cudaStream_t stream = 0,
+                ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : mBuilder(stream, resource), mStream(stream), mTimer(stream), mDeviceSrcGrid(d_srcGrid) {}
 
     /// @brief Toggle on and off verbose mode
     /// @param level Verbose level: 0=quiet, 1=timing, 2=benchmarking
@@ -74,20 +80,20 @@ private:
     static constexpr unsigned int mNumThreads = 128;// for kernels spawned via lambdaKernel (others may specialize)
     static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
 
-    TopologyBuilder<BuildT> mBuilder;
+    TopologyBuilder<BuildT, ResourceT> mBuilder;
     cudaStream_t            mStream{0};
     util::cuda::Timer       mTimer;
     int                     mVerbose{0};
     const GridT             *mDeviceSrcGrid;
     TreeData                mSrcTreeData;
-};// tools::cuda::CoarsenGrid<BuildT>
+};// tools::cuda::CoarsenGrid<BuildT, ResourceT>
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 template<typename BufferT>
 GridHandle<BufferT>
-CoarsenGrid<BuildT>::getHandle(const BufferT &pool)
+CoarsenGrid<BuildT, ResourceT>::getHandle(const BufferT &pool)
 {
     // Copy TreeData from GPU -> CPU
     cudaStreamSynchronize(mStream);
@@ -147,12 +153,12 @@ CoarsenGrid<BuildT>::getHandle(const BufferT &pool)
     cudaStreamSynchronize(mStream);
 
     return GridHandle<BufferT>(std::move(buffer));
-}// CoarsenGrid<BuildT>::getHandle
+}// CoarsenGrid<BuildT, ResourceT>::getHandle
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void CoarsenGrid<BuildT>::coarsenRoot()
+template<typename BuildT, typename ResourceT>
+void CoarsenGrid<BuildT, ResourceT>::coarsenRoot()
 {
     // This method coarsens the root tiles, to accommodate for the overall downsamping operation.
 
@@ -197,12 +203,12 @@ void CoarsenGrid<BuildT>::coarsenRoot()
     for (const auto& [key, tile] : coarsenedTiles)
         *coarsenedRootPtr->tile(t++) = tile;
     mBuilder.mProcessedRoot.deviceUpload(device, mStream, false);
-}// CoarsenGrid<BuildT>::coarsenRoot
+}// CoarsenGrid<BuildT, ResourceT>::coarsenRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void CoarsenGrid<BuildT>::coarsenInternalNodes()
+template<typename BuildT, typename ResourceT>
+void CoarsenGrid<BuildT, ResourceT>::coarsenInternalNodes()
 {
     // Computes the masks of upper and (densified) lower internal nodes, as a result of the coarsening operation
     // Masks of lower internal nodes are densified in the sense that a serialized array of them is allocated,
@@ -210,26 +216,26 @@ void CoarsenGrid<BuildT>::coarsenInternalNodes()
     if (auto srcLeafCount = mSrcTreeData.mNodeCount[0]) { // Unless it's an empty grid
         util::cuda::lambdaKernel<<<numBlocks(srcLeafCount), mNumThreads, 0, mStream>>>(
             srcLeafCount, util::morphology::cuda::CoarsenInternalNodesFunctor<BuildT>(),
-            mDeviceSrcGrid, mBuilder.deviceProcessedRoot(), mBuilder.mUpperMasks.deviceData(), mBuilder.mLowerMasks.deviceData() );
+            mDeviceSrcGrid, mBuilder.deviceProcessedRoot(), mBuilder.mUpperMasks.data(), mBuilder.mLowerMasks.data() );
     }
-}// CoarsenGrid<BuildT>::coarsenInternalNodes
+}// CoarsenGrid<BuildT, ResourceT>::coarsenInternalNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template <typename BuildT>
-void CoarsenGrid<BuildT>::processGridTreeRoot()
+template <typename BuildT, typename ResourceT>
+void CoarsenGrid<BuildT, ResourceT>::processGridTreeRoot()
 {
     // Copy GridData from source grid
     // By convention: this will duplicate grid name and map. Others will be reset later
     cudaCheck(cudaMemcpyAsync(&mBuilder.data()->getGrid(), mDeviceSrcGrid->data(), GridT::memUsage(), cudaMemcpyDeviceToDevice, mStream));
     util::cuda::lambdaKernel<<<1, 1, 0, mStream>>>(1, topology::detail::BuildGridTreeRootFunctor<BuildT>(), mBuilder.deviceData());
     cudaCheckError();
-}// CoarsenGrid<BuildT>::processGridTreeRoot
+}// CoarsenGrid<BuildT, ResourceT>::processGridTreeRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void CoarsenGrid<BuildT>::coarsenLeafNodes()
+template<typename BuildT, typename ResourceT>
+void CoarsenGrid<BuildT, ResourceT>::coarsenLeafNodes()
 {
     // Coarsens the active masks of the source grid (as indicated at the leaf level), into a new grid that
     // has been already topologically coarsened to include all necessary leaf nodes.
@@ -241,7 +247,7 @@ void CoarsenGrid<BuildT>::coarsenLeafNodes()
 
     // Update leaf offsets and prefix sums
     mBuilder.processLeafOffsets(mStream);
-}// CoarsenGrid<BuildT>::coarsenLeafNodes
+}// CoarsenGrid<BuildT, ResourceT>::coarsenLeafNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 

nanovdb/nanovdb/tools/cuda/DilateGrid.cuh

@@ -30,9 +30,12 @@ namespace nanovdb {
 
 namespace tools::cuda {
 
-template <typename BuildT>
+template <typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class DilateGrid
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "DilateGrid allocates stream-ordered scratch and requires an AsyncResource");
+
     using GridT  = NanoGrid<BuildT>;
     using TreeT  = NanoTree<BuildT>;
     using RootT  = NanoRoot<BuildT>;
@@ -43,8 +46,11 @@ public:
     /// @brief Constructor
     /// @param deviceGrid source device grid to be dilated
     /// @param stream optional CUDA stream (defaults to CUDA stream 0)
-    DilateGrid(const GridT* d_srcGrid, cudaStream_t stream = 0)
-        : mBuilder(stream), mStream(stream), mTimer(stream), mDeviceSrcGrid(d_srcGrid) {}
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this operator (defaults to the per-type default resource)
+    DilateGrid(const GridT* d_srcGrid, cudaStream_t stream = 0,
+               ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : mBuilder(stream, resource), mStream(stream), mTimer(stream), mDeviceSrcGrid(d_srcGrid) {}
 
     /// @brief Toggle on and off verbose mode
     /// @param level Verbose level: 0=quiet, 1=timing, 2=benchmarking
@@ -78,21 +84,21 @@ private:
     static constexpr unsigned int mNumThreads = 128;// for kernels spawned via lambdaKernel (others may specialize)
     static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
 
-    TopologyBuilder<BuildT>      mBuilder;
+    TopologyBuilder<BuildT, ResourceT> mBuilder;
     cudaStream_t                 mStream{0};
     util::cuda::Timer            mTimer;
     int                          mVerbose{0};
     const GridT                  *mDeviceSrcGrid;
     morphology::NearestNeighbors mOp{morphology::NN_FACE_EDGE_VERTEX};
     TreeData                     mSrcTreeData;
-};// tools::cuda::DilateGrid<BuildT>
+};// tools::cuda::DilateGrid<BuildT, ResourceT>
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 template<typename BufferT>
 GridHandle<BufferT>
-DilateGrid<BuildT>::getHandle(const BufferT &pool)
+DilateGrid<BuildT, ResourceT>::getHandle(const BufferT &pool)
 {
     // Copy TreeData from GPU -> CPU
     cudaStreamSynchronize(mStream);
@@ -152,12 +158,12 @@ DilateGrid<BuildT>::getHandle(const BufferT &pool)
     cudaStreamSynchronize(mStream);
 
     return GridHandle<BufferT>(std::move(buffer));
-}// DilateGrid<BuildT>::getHandle
+}// DilateGrid<BuildT, ResourceT>::getHandle
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void DilateGrid<BuildT>::dilateRoot()
+template<typename BuildT, typename ResourceT>
+void DilateGrid<BuildT, ResourceT>::dilateRoot()
 {
     // This method conservatively and speculatively dilates the root tiles, to accommodate
     // any new root nodes that might be introduced by the dilation operation.
@@ -220,12 +226,12 @@ void DilateGrid<BuildT>::dilateRoot()
     for (const auto& [key, tile] : dilatedTiles)
         *dilatedRootPtr->tile(t++) = tile;
     mBuilder.mProcessedRoot.deviceUpload(device, mStream, false);
-}// DilateGrid<BuildT>::dilateRoot
+}// DilateGrid<BuildT, ResourceT>::dilateRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void DilateGrid<BuildT>::dilateInternalNodes()
+template<typename BuildT, typename ResourceT>
+void DilateGrid<BuildT, ResourceT>::dilateInternalNodes()
 {
     // Computes the masks of upper and (densified) lower internal nodes, as a result of the dilation operation
     // Masks of lower internal nodes are densified in the sense that a serialized array of them is allocated,
@@ -247,24 +253,24 @@ void DilateGrid<BuildT>::dilateInternalNodes()
                 <<<dim3(mSrcTreeData.mNodeCount[1],Op::SlicesPerLowerNode,1), Op::MaxThreadsPerBlock, 0, mStream>>>
                 (mDeviceSrcGrid, mBuilder.deviceProcessedRoot(), mBuilder.deviceUpperMasks(), mBuilder.deviceLowerMasks()); }
     }
-}// DilateGrid<BuildT>::dilateInternalNodes
+}// DilateGrid<BuildT, ResourceT>::dilateInternalNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template <typename BuildT>
-void DilateGrid<BuildT>::processGridTreeRoot()
+template <typename BuildT, typename ResourceT>
+void DilateGrid<BuildT, ResourceT>::processGridTreeRoot()
 {
     // Copy GridData from source grid
     // By convention: this will duplicate grid name and map. Others will be reset later
     cudaCheck(cudaMemcpyAsync(&mBuilder.data()->getGrid(), mDeviceSrcGrid->data(), GridT::memUsage(), cudaMemcpyDeviceToDevice, mStream));
     util::cuda::lambdaKernel<<<1, 1, 0, mStream>>>(1, topology::detail::BuildGridTreeRootFunctor<BuildT>(), mBuilder.deviceData());
     cudaCheckError();
-}// DilateGrid<BuildT>::processGridTreeRoot
+}// DilateGrid<BuildT, ResourceT>::processGridTreeRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void DilateGrid<BuildT>::dilateLeafNodes()
+template<typename BuildT, typename ResourceT>
+void DilateGrid<BuildT, ResourceT>::dilateLeafNodes()
 {
     // Dilates the active masks of the source grid (as indicated at the leaf level), into a new grid that
     // has been already topologically dilated to include all necessary leaf nodes.
@@ -279,13 +285,13 @@ void DilateGrid<BuildT>::dilateLeafNodes()
         else if (mOp == morphology::NN_FACE_EDGE_VERTEX) {
             using Op = util::morphology::cuda::DilateLeafNodesFunctor<BuildT, morphology::NN_FACE_EDGE_VERTEX>;
             util::cuda::operatorKernel<Op>
-                <<<dim3(mBuilder.data()->nodeCount[1],Op::SlicesPerLowerNode,1), Op::MaxThreadsPerBlock>>>
+                <<<dim3(mBuilder.data()->nodeCount[1],Op::SlicesPerLowerNode,1), Op::MaxThreadsPerBlock, 0, mStream>>>
                 (mDeviceSrcGrid, static_cast<GridT*>(mBuilder.data()->d_bufferPtr)); }
     }
 
     // Update leaf offsets and prefix sums
     mBuilder.processLeafOffsets(mStream);
-}// DilateGrid<BuildT>::dilateLeafNodes
+}// DilateGrid<BuildT, ResourceT>::dilateLeafNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 

nanovdb/nanovdb/tools/cuda/DistributedPointsToGrid.cuh

@@ -20,6 +20,9 @@
 #include <nanovdb/cuda/UnifiedBuffer.h>
 #include <nanovdb/tools/cuda/PointsToGrid.cuh>
 #include <nanovdb/util/cuda/Util.h>
+#include <algorithm>
+
+#include <cuda/cmath>
 
 namespace nanovdb {
 
@@ -37,112 +40,84 @@ private:
     uint32_t* mNodeCounts;
 };
 
-/// @brief Indicator functor that returns 1 if the input value matches the member value, 0 otherwise
-template <typename T, typename std::enable_if<std::is_integral<T>::value>::type* = nullptr>
-struct EqualityIndicator
-{
-    EqualityIndicator(const T* value) : mValue(value) {}
-
-    __hostdev__
-    T operator()(const T& x) const
-    {
-        return x == (*mValue);
-    }
-private:
-    const T* mValue;
-};
-
-/// @brief Implements the merge path binary search algorithm in order to find the median across two sorted input key arrays
+/// @brief Find a partition at an arbitrary diagonal in the conceptual merge of two sorted input arrays.
 template<typename KeyIteratorIn>
 __device__
-void mergePath(KeyIteratorIn keys1, size_t keys1Count, KeyIteratorIn keys2, size_t keys2Count, ptrdiff_t* key1Intervals, ptrdiff_t* key2Intervals, int intervalIndex)
+void mergePath(KeyIteratorIn keys1, size_t keys1Count, KeyIteratorIn keys2, size_t keys2Count, ptrdiff_t* key1Intervals, ptrdiff_t* key2Intervals, size_t combinedIndex)
 {
-    using key_type = typename ::cuda::std::iterator_traits<KeyIteratorIn>::value_type;
-
-    const size_t combinedIndex = intervalIndex * (keys1Count + keys2Count) / 2;
-    size_t leftTop = combinedIndex > keys1Count ? keys1Count : combinedIndex;
-    size_t rightTop = combinedIndex > keys1Count ? combinedIndex - keys1Count : 0;
-    size_t leftBottom = rightTop;
-
-    key_type leftKey;
-    key_type rightKey;
-    while(true)
-    {
-        ptrdiff_t offset = (leftTop - leftBottom) / 2;
-        ptrdiff_t leftMid = leftTop - offset;
-        ptrdiff_t rightMid = rightTop + offset;
-
-        if (leftMid > keys1Count - 1 || rightMid < 1) {
-            leftKey = 1;
-            rightKey = 0;
-        }
-        else {
-            leftKey = *(keys1 + leftMid);
-            rightKey = *(keys2 + rightMid - 1);
-        }
-
-        if (leftKey > rightKey) {
-            if (rightMid > keys2Count - 1 || leftMid < 1) {
-                leftKey = 0;
-                rightKey = 1;
-            }
-            else {
-                leftKey = *(keys1 + leftMid - 1);
-                rightKey = *(keys2 + rightMid);
-            }
-
-            if (leftKey <= rightKey) {
-                *key1Intervals = leftMid;
-                *key2Intervals = rightMid;
-                break;
-            }
-            else {
-                leftTop = leftMid - 1;
-                rightTop = rightMid + 1;
-            }
-        }
-        else {
-            leftBottom = leftMid + 1;
+    size_t begin = combinedIndex > keys2Count ? combinedIndex - keys2Count : 0;
+    size_t end = combinedIndex < keys1Count ? combinedIndex : keys1Count;
+    while (begin < end) {
+        const size_t key1Index = (begin + end) / 2;
+        const size_t key2Index = combinedIndex - 1 - key1Index;
+        if (!(keys2[key2Index] < keys1[key1Index])) {
+            begin = key1Index + 1;
+        } else {
+            end = key1Index;
         }
     }
+
+    *key1Intervals = begin;
+    *key2Intervals = combinedIndex - begin;
 }
 
 namespace kernels {
 
-/// @brief Kernel wrapper for the merge path algorithm
+/// @brief Kernel wrapper for the merge path algorithm.
 template<typename KeyIteratorIn>
 __global__
-void mergePathKernel(KeyIteratorIn keys1, size_t keys1Count, KeyIteratorIn keys2, size_t keys2Count, ptrdiff_t* key1Intervals, ptrdiff_t* key2Intervals, size_t intervalOffset)
+void mergePathKernel(KeyIteratorIn keys1, size_t keys1Count, KeyIteratorIn keys2, size_t keys2Count, ptrdiff_t* key1Intervals, ptrdiff_t* key2Intervals, size_t combinedIndex)
 {
-    const unsigned int intervalIndex = threadIdx.x + blockIdx.x * blockDim.x + intervalOffset;
-    mergePath(keys1, keys1Count, keys2, keys2Count, key1Intervals, key2Intervals, intervalIndex);
+    mergePath(keys1, keys1Count, keys2, keys2Count, key1Intervals, key2Intervals, combinedIndex);
 }
 
-/// @brief Extends or shortens the left end of an array interval
-template<typename DistanceIteratorIn, typename CountIteratorOut, typename OffsetIteratorOut>
+/// @brief Snap each device boundary in stripeOffsets to the nearest edge of the run of equal
+/// keys containing it, so that no run (i.e. upper-node tile) straddles two devices. The keys
+/// must be globally sorted. The boundaries are adjusted monotonically from left to right, so a
+/// run spanning several stripes is consolidated onto one device and the fully-interior devices
+/// are left with empty stripes. Runs sequentially on a single thread: the boundary chain is a
+/// sequential dependence over deviceCount entries and each run extent is found by binary search.
+template<typename KeyT>
 __global__
-void leftRebalanceKernel(DistanceIteratorIn leftDistance, DistanceIteratorIn rightDistance, CountIteratorOut leftCount, OffsetIteratorOut leftOffset)
+void snapBoundariesToRunsKernel(const KeyT* keys, ptrdiff_t keyCount, int deviceCount, ptrdiff_t* stripeOffsets, size_t* stripeCounts)
 {
-    if (*leftDistance < *rightDistance) {
-        *leftCount -= *leftDistance;
+    ptrdiff_t previousBoundary = stripeOffsets[0]; // device 0 always starts at 0
+    for (int deviceId = 1; deviceId < deviceCount; ++deviceId) {
+        ptrdiff_t boundary = stripeOffsets[deviceId];
+        if (boundary >= keyCount) {
+            boundary = keyCount;
+        } else if (boundary > previousBoundary && keys[boundary] == keys[boundary - 1]) {
+            // The even-split boundary falls inside a run; binary search for the run's extent
+            // and snap to whichever end keeps the boundary closest to the even split without
+            // crossing the previous boundary.
+            const KeyT key = keys[boundary];
+            ptrdiff_t lo = previousBoundary, hi = boundary;
+            while (lo < hi) {
+                const ptrdiff_t mid = lo + (hi - lo) / 2;
+                if (keys[mid] < key) lo = mid + 1; else hi = mid;
+            }
+            const ptrdiff_t runStart = lo;
+            lo = boundary; hi = keyCount;
+            while (lo < hi) {
+                const ptrdiff_t mid = lo + (hi - lo) / 2;
+                if (keys[mid] <= key) lo = mid + 1; else hi = mid;
+            }
+            const ptrdiff_t runEnd = lo;
+            if (runStart <= previousBoundary) {
+                boundary = runEnd; // the run reaches the previous device, give the whole tile away
+            } else {
+                boundary = (boundary - runStart <= runEnd - boundary) ? runStart : runEnd;
+            }
+        }
+        if (boundary < previousBoundary) boundary = previousBoundary;
+        stripeOffsets[deviceId] = boundary;
+        previousBoundary = boundary;
     }
-    else {
-        *leftCount += *rightDistance;
-    }
-}
 
-/// @brief Extends or shortens the right end of an array interval
-template<typename DistanceIteratorIn, typename CountIteratorOut, typename OffsetIteratorOut>
-__global__
-void rightRebalanceKernel(DistanceIteratorIn leftDistance, DistanceIteratorIn rightDistance, CountIteratorOut rightCount, OffsetIteratorOut rightOffset)
-{
-    if (*leftDistance < *rightDistance) {
-        *rightCount += *leftDistance;
-        *rightOffset -= *leftDistance;
-    }
-    else {
-        *rightCount -= *rightDistance;
-        *rightOffset += *rightDistance;
+    // Recompute the per-device counts from the adjusted, monotonic offsets.
+    for (int deviceId = 0; deviceId < deviceCount; ++deviceId) {
+        const ptrdiff_t nextOffset = (deviceId + 1 < deviceCount) ? stripeOffsets[deviceId + 1] : keyCount;
+        stripeCounts[deviceId] = static_cast<size_t>(nextOffset - stripeOffsets[deviceId]);
     }
 }
 
@@ -163,135 +138,183 @@ void rightRebalanceKernel(DistanceIteratorIn leftDistance, DistanceIteratorIn ri
 #endif// ifdef _WIN32
 #endif// ifndef CUB_LAUNCH
 
+/// @brief Make every device stream wait for work submitted to all device streams.
+inline void mergeStreams(const nanovdb::cuda::DeviceMesh& deviceMesh, cudaEvent_t* events)
+{
+    static constexpr int mergeDeviceId = 0;
+
+    // Record an event for each device in its respective stream.
+    for (const auto& [deviceId, stream] : deviceMesh) {
+        cudaCheck(cudaSetDevice(deviceId));
+        cudaCheck(cudaEventRecord(events[deviceId], stream));
+    }
+
+    // Fan in the per-device events on the merge stream.
+    cudaCheck(cudaSetDevice(mergeDeviceId));
+    const auto mergeStream = deviceMesh[mergeDeviceId].stream;
+    for (const auto& deviceNode : deviceMesh) {
+        cudaCheck(cudaStreamWaitEvent(mergeStream, events[deviceNode.id]));
+    }
+    cudaCheck(cudaEventRecord(events[mergeDeviceId], mergeStream));
+
+    // Fan the merged dependency back out to every device stream.
+    for (const auto& [deviceId, stream] : deviceMesh) {
+        cudaCheck(cudaSetDevice(deviceId));
+        cudaCheck(cudaStreamWaitEvent(stream, events[mergeDeviceId]));
+    }
+}
+
 template<typename KeyT, typename ValueT, typename NumItemsT, typename OffsetT, typename CountT>
 void radixSortAsync(const nanovdb::cuda::DeviceMesh& deviceMesh, nanovdb::cuda::TempDevicePool* pools, KeyT* keysIn, KeyT* keysOut, ValueT* valuesIn, ValueT* valuesOut, NumItemsT numItems, OffsetT* mergeIntervals, const OffsetT* offsets, const CountT* counts, cudaEvent_t* preEvents, cudaEvent_t* postEvents)
 {
-    // Radix sort the subset of keys assigned to each device in parallel
+    if (!numItems) return;
+
+    const int deviceCount = static_cast<int>(deviceMesh.deviceCount());
+    OffsetT* leftIntervals = mergeIntervals;
+    OffsetT* rightIntervals = mergeIntervals + deviceCount + 1;
+    const auto offset = [&](int deviceIndex) {
+        return deviceIndex == deviceCount ? static_cast<OffsetT>(numItems) : offsets[deviceIndex];
+    };
+
+    // Radix sort the subset of key/value pairs assigned to each GPU.
     for (const auto& [deviceId, stream] : deviceMesh) {
         cudaCheck(cudaSetDevice(deviceId));
         cudaCheck(cudaEventSynchronize(preEvents[deviceId]));
 
+        if (!counts[deviceId]) {
+            cudaCheck(cudaEventRecord(postEvents[deviceId], stream));
+            continue;
+        }
+
         const KeyT* deviceKeysIn = keysIn + offsets[deviceId];
         const ValueT* deviceValuesIn = valuesIn + offsets[deviceId];
         KeyT* deviceKeysOut = keysOut + offsets[deviceId];
         ValueT* deviceValuesOut = valuesOut + offsets[deviceId];
 
         cudaCheck(util::cuda::memPrefetchAsync(deviceKeysIn, counts[deviceId] * sizeof(KeyT), deviceId, stream));
+        cudaCheck(util::cuda::memPrefetchAsync(deviceValuesIn, counts[deviceId] * sizeof(ValueT), deviceId, stream));
+        cudaCheck(util::cuda::memPrefetchAsync(deviceKeysOut, counts[deviceId] * sizeof(KeyT), deviceId, stream));
+        cudaCheck(util::cuda::memPrefetchAsync(deviceValuesOut, counts[deviceId] * sizeof(ValueT), deviceId, stream));
 
         // TODO: Add begin and end bit support
         CUB_LAUNCH(DeviceRadixSort::SortPairs, pools[deviceId], stream, deviceKeysIn, deviceKeysOut, deviceValuesIn, deviceValuesOut, counts[deviceId], 0, sizeof(KeyT) * 8);
         cudaCheck(cudaEventRecord(postEvents[deviceId], stream));
     }
+    mergeStreams(deviceMesh, postEvents);
 
-    // TODO: Generalize to numbers of GPUs that aren't powers of two
-    // For each pair of devices, merge the local sorts by first computing the median across the two devices followed by merging
-    // the elements less than and greater than/equal to the median onto the first and second device of the pair respectively.
-    // This avoids the allocating memory for and gathering the values from both devices onto a single device.
-    const int log2DeviceCount = log2(deviceMesh.deviceCount());
-    OffsetT* leftIntervals = mergeIntervals;
-    OffsetT* rightIntervals = mergeIntervals + deviceMesh.deviceCount();
-    for (int deviceExponent = 0; deviceExponent < log2DeviceCount; ++deviceExponent) {
-        std::swap(keysIn, keysOut);
-        std::swap(valuesIn, valuesOut);
-        const int deviceIncrement = 1 << deviceExponent;
+    KeyT* currentKeys = keysOut;
+    KeyT* nextKeys = keysIn;
+    ValueT* currentValues = valuesOut;
+    ValueT* nextValues = valuesIn;
 
-        std::vector<std::thread> threads;
-        for (int leftDeviceId = 0; leftDeviceId < static_cast<int>(deviceMesh.deviceCount()); leftDeviceId += 2 * deviceIncrement) {
-            threads.emplace_back([&, leftDeviceId]() {
-                const int rightDeviceId = leftDeviceId + deviceIncrement;
+    // At each level, split every pair of sorted runs at all of the group's output-shard boundaries.
+    // Every GPU in the group merges one balanced output shard, including at the upper merge-tree levels.
+    for (int runDeviceCount = 1; runDeviceCount < deviceCount; runDeviceCount *= 2) {
+        for (int groupStart = 0; groupStart < deviceCount; groupStart += 2 * runDeviceCount) {
+            const int groupMiddle = std::min(groupStart + runDeviceCount, deviceCount);
+            const int groupEnd = std::min(groupStart + 2 * runDeviceCount, deviceCount);
+            if (groupMiddle == groupEnd) continue;
 
-                CountT leftDeviceItemCount = 0;
-                for (int deviceId = leftDeviceId; deviceId < rightDeviceId; ++deviceId)
-                    leftDeviceItemCount += counts[deviceId];
+            const CountT leftCount = offset(groupMiddle) - offset(groupStart);
+            const CountT rightCount = offset(groupEnd) - offset(groupMiddle);
+            const KeyT* leftKeys = currentKeys + offset(groupStart);
+            const KeyT* rightKeys = currentKeys + offset(groupMiddle);
 
-                CountT rightDeviceItemCount = 0;
-                for (int deviceId = rightDeviceId; deviceId < rightDeviceId + deviceIncrement; ++deviceId)
-                    rightDeviceItemCount += counts[deviceId];
-
-                const KeyT* leftDeviceKeysIn = keysIn + offsets[leftDeviceId];
-                const ValueT* leftDeviceValuesIn = valuesIn + offsets[leftDeviceId];
-                const KeyT* rightDeviceKeysIn = keysIn + offsets[leftDeviceId] + leftDeviceItemCount;
-                const ValueT* rightDeviceValuesIn = valuesIn + offsets[leftDeviceId] + leftDeviceItemCount;
-
-                // Wait on the prior sort to finish on both devices before computing the median across both devices
-                auto mergePathSubfunc = [&](int deviceId, int otherDeviceId, int intervalIndex) {
-                    cudaCheck(cudaSetDevice(deviceId));
-
-                    cudaCheck(cudaStreamWaitEvent(deviceMesh[deviceId].stream, postEvents[otherDeviceId]));
-                    kernels::mergePathKernel<<<1, 1, 0, deviceMesh[deviceId].stream>>>(leftDeviceKeysIn, leftDeviceItemCount, rightDeviceKeysIn, rightDeviceItemCount, leftIntervals + deviceId, rightIntervals + deviceId, intervalIndex);
-                    cudaCheck(cudaEventRecord(postEvents[deviceId], deviceMesh[deviceId].stream));
-                };
-                mergePathSubfunc(leftDeviceId, rightDeviceId, 0);
-                mergePathSubfunc(rightDeviceId, leftDeviceId, 1);
-
-                cudaCheck(cudaEventSynchronize(postEvents[leftDeviceId]));
-                cudaCheck(cudaEventSynchronize(postEvents[rightDeviceId]));
-
-                // Merge the pairs less than the median to the left device
-                {
-                    cudaCheck(cudaSetDevice(leftDeviceId));
-
-                    const KeyT* leftKeysIn = leftDeviceKeysIn + leftIntervals[leftDeviceId];
-                    const ValueT* leftValuesIn = leftDeviceValuesIn + leftIntervals[leftDeviceId];
-                    CountT leftCount = leftIntervals[rightDeviceId] - leftIntervals[leftDeviceId];
-
-                    const KeyT* rightKeysIn = rightDeviceKeysIn + rightIntervals[leftDeviceId];
-                    const ValueT* rightValuesIn = rightDeviceValuesIn + rightIntervals[leftDeviceId];
-                    CountT rightCount = rightIntervals[rightDeviceId] - rightIntervals[leftDeviceId];
-
-                    OffsetT outputOffset = offsets[leftDeviceId] + leftIntervals[leftDeviceId] + rightIntervals[leftDeviceId];
-
-                    CUB_LAUNCH(DeviceMerge::MergePairs, pools[leftDeviceId], deviceMesh[leftDeviceId].stream, leftKeysIn, leftValuesIn, leftCount, rightKeysIn, rightValuesIn, rightCount, keysOut + outputOffset, valuesOut + outputOffset, {});
-                    cudaCheck(cudaEventRecord(postEvents[leftDeviceId], deviceMesh[leftDeviceId].stream));
-                };
-
-                // Merge the pairs greater than/equal to the median to the right device
-                {
-                    cudaCheck(cudaSetDevice(rightDeviceId));
-
-                    const KeyT* leftKeysIn = leftDeviceKeysIn + leftIntervals[rightDeviceId];
-                    const ValueT* leftValuesIn = leftDeviceValuesIn + leftIntervals[rightDeviceId];
-                    CountT leftCount = leftDeviceItemCount - leftIntervals[rightDeviceId];
-
-                    const KeyT* rightKeysIn = rightDeviceKeysIn + rightIntervals[rightDeviceId];
-                    const ValueT* rightValuesIn = rightDeviceValuesIn + rightIntervals[rightDeviceId];
-                    CountT rightCount = rightDeviceItemCount - rightIntervals[rightDeviceId];
-
-                    OffsetT outputOffset = offsets[leftDeviceId] + leftIntervals[rightDeviceId] + rightIntervals[rightDeviceId];
-
-                    CUB_LAUNCH(DeviceMerge::MergePairs, pools[rightDeviceId], deviceMesh[rightDeviceId].stream, leftKeysIn, leftValuesIn, leftCount, rightKeysIn, rightValuesIn, rightCount, keysOut + outputOffset, valuesOut + outputOffset, {});
-                    cudaCheck(cudaEventRecord(postEvents[rightDeviceId], deviceMesh[rightDeviceId].stream));
-                };
-
-                cudaCheck(cudaEventSynchronize(postEvents[leftDeviceId]));
-                cudaCheck(cudaEventSynchronize(postEvents[rightDeviceId]));
-            });
+            for (int boundaryDevice = groupStart + 1; boundaryDevice < groupEnd; ++boundaryDevice) {
+                cudaCheck(cudaSetDevice(boundaryDevice));
+                const auto stream = deviceMesh[boundaryDevice].stream;
+                const CountT diagonal = offset(boundaryDevice) - offset(groupStart);
+                kernels::mergePathKernel<<<1, 1, 0, stream>>>(leftKeys, leftCount, rightKeys, rightCount, leftIntervals + boundaryDevice, rightIntervals + boundaryDevice, diagonal);
+                cudaCheckError();
+                cudaCheck(cudaEventRecord(postEvents[boundaryDevice], stream));
+            }
         }
-        std::for_each(threads.begin(), threads.end(), [](std::thread& t) { t.join(); });
+
+        // CUB needs the partition sizes on the host. Synchronize only the tiny merge-path kernels.
+        for (int groupStart = 0; groupStart < deviceCount; groupStart += 2 * runDeviceCount) {
+            const int groupMiddle = std::min(groupStart + runDeviceCount, deviceCount);
+            const int groupEnd = std::min(groupStart + 2 * runDeviceCount, deviceCount);
+            if (groupMiddle == groupEnd) continue;
+
+            for (int boundaryDevice = groupStart + 1; boundaryDevice < groupEnd; ++boundaryDevice) {
+                cudaCheck(cudaEventSynchronize(postEvents[boundaryDevice]));
+            }
+        }
+
+        for (int groupStart = 0; groupStart < deviceCount; groupStart += 2 * runDeviceCount) {
+            const int groupMiddle = std::min(groupStart + runDeviceCount, deviceCount);
+            const int groupEnd = std::min(groupStart + 2 * runDeviceCount, deviceCount);
+
+            // A trailing run without a merge partner must still be copied because the
+            // current and next buffers are swapped after every merge-tree level.
+            if (groupMiddle == groupEnd) {
+                for (int outputDevice = groupStart; outputDevice < groupEnd; ++outputDevice) {
+                    cudaCheck(cudaSetDevice(outputDevice));
+                    const auto stream = deviceMesh[outputDevice].stream;
+                    if (counts[outputDevice]) {
+                        cudaCheck(cudaMemcpyAsync(nextKeys + offset(outputDevice), currentKeys + offset(outputDevice), counts[outputDevice] * sizeof(KeyT), cudaMemcpyDefault, stream));
+                        cudaCheck(cudaMemcpyAsync(nextValues + offset(outputDevice), currentValues + offset(outputDevice), counts[outputDevice] * sizeof(ValueT), cudaMemcpyDefault, stream));
+                    }
+                    cudaCheck(cudaEventRecord(postEvents[outputDevice], stream));
+                }
+                continue;
+            }
+
+            const CountT leftCount = offset(groupMiddle) - offset(groupStart);
+            const CountT rightCount = offset(groupEnd) - offset(groupMiddle);
+            const KeyT* leftKeys = currentKeys + offset(groupStart);
+            const KeyT* rightKeys = currentKeys + offset(groupMiddle);
+            const ValueT* leftValues = currentValues + offset(groupStart);
+            const ValueT* rightValues = currentValues + offset(groupMiddle);
+
+            for (int outputDevice = groupStart; outputDevice < groupEnd; ++outputDevice) {
+                cudaCheck(cudaSetDevice(outputDevice));
+                const auto stream = deviceMesh[outputDevice].stream;
+                const CountT leftBegin = outputDevice == groupStart ? 0 : leftIntervals[outputDevice];
+                const CountT leftEnd = outputDevice + 1 == groupEnd ? leftCount : leftIntervals[outputDevice + 1];
+                const CountT rightBegin = outputDevice == groupStart ? 0 : rightIntervals[outputDevice];
+                const CountT rightEnd = outputDevice + 1 == groupEnd ? rightCount : rightIntervals[outputDevice + 1];
+                const CountT leftSegmentCount = leftEnd - leftBegin;
+                const CountT rightSegmentCount = rightEnd - rightBegin;
+                const CountT outputCount = leftSegmentCount + rightSegmentCount;
+                NANOVDB_ASSERT(outputCount == counts[outputDevice]);
+
+                if (outputCount) {
+                    CUB_LAUNCH(DeviceMerge::MergePairs, pools[outputDevice], stream,
+                        leftKeys + leftBegin, leftValues + leftBegin, leftSegmentCount,
+                        rightKeys + rightBegin, rightValues + rightBegin, rightSegmentCount,
+                        nextKeys + offset(outputDevice), nextValues + offset(outputDevice), {});
+                }
+                cudaCheck(cudaEventRecord(postEvents[outputDevice], stream));
+            }
+        }
+
+        mergeStreams(deviceMesh, postEvents);
+        std::swap(currentKeys, nextKeys);
+        std::swap(currentValues, nextValues);
     }
 
-    // There is no merging required for a single device so we simply copy the sorted result to the destination array (where the sort would have been merged to).
-    if (log2DeviceCount % 2) {
-        std::swap(keysIn, keysOut);
-        std::swap(valuesIn, valuesOut);
+    // Keep the existing contract: callers always receive the final data in the output buffers.
+    if (currentKeys != keysOut) {
         for (const auto& [deviceId, stream] : deviceMesh) {
             cudaCheck(cudaSetDevice(deviceId));
-
-            cudaMemcpyAsync(keysOut + offsets[deviceId], keysIn + offsets[deviceId], counts[deviceId] * sizeof(KeyT), cudaMemcpyDefault, stream);
-            cudaMemcpyAsync(valuesOut + offsets[deviceId], valuesIn + offsets[deviceId], counts[deviceId] * sizeof(ValueT), cudaMemcpyDefault, stream);
-
-            cudaEventRecord(postEvents[deviceId], stream);
+            if (counts[deviceId]) {
+                cudaCheck(cudaMemcpyAsync(keysOut + offsets[deviceId], currentKeys + offsets[deviceId], counts[deviceId] * sizeof(KeyT), cudaMemcpyDefault, stream));
+                cudaCheck(cudaMemcpyAsync(valuesOut + offsets[deviceId], currentValues + offsets[deviceId], counts[deviceId] * sizeof(ValueT), cudaMemcpyDefault, stream));
+            }
+            cudaCheck(cudaEventRecord(postEvents[deviceId], stream));
         }
     }
+    mergeStreams(deviceMesh, postEvents);
 }
 
 template<typename KeyT, typename ValueT, typename NumItemsT, typename OffsetT, typename CountT>
 void radixSortAsync(const nanovdb::cuda::DeviceMesh& deviceMesh, nanovdb::cuda::TempDevicePool* pools, KeyT* keysIn, KeyT* keysOut, ValueT* valuesIn, ValueT* valuesOut, NumItemsT numItems, const OffsetT* offsets, const CountT* counts, cudaEvent_t* preEvents, cudaEvent_t* postEvents)
 {
     ptrdiff_t* mergeIntervals = nullptr;
-    cudaCheck(cudaMallocManaged(&mergeIntervals, 2 * deviceMesh.deviceCount() * sizeof(ptrdiff_t)));
+    cudaCheck(cudaMallocHost(&mergeIntervals, 2 * (deviceMesh.deviceCount() + 1) * sizeof(ptrdiff_t)));
     radixSortAsync(deviceMesh, pools, keysIn, keysOut, valuesIn, valuesOut, numItems, mergeIntervals, offsets, counts, preEvents, postEvents);
-    cudaCheck(cudaFree(mergeIntervals));
+    cudaCheck(cudaFreeHost(mergeIntervals));
 }
 
 /// @brief Launches an async exclusive sum operation across multiple devices. The operator waits on the per-device preEvents[deviceId] before summing over that device's contributions and records postEvents[deviceId] when the device's contribution is summed.
@@ -306,10 +329,12 @@ void exclusiveSumAsync(const nanovdb::cuda::DeviceMesh& deviceMesh, nanovdb::cud
         // Required for the host to pass the correct value of counts[deviceId]
         cudaCheck(cudaEventSynchronize(preEvents[deviceId]));
         uint32_t deviceNumItems = counts[deviceId];
-        CUB_LAUNCH(DeviceScan::ExclusiveSum, pools[deviceId], stream, deviceIn, deviceOut, deviceNumItems);
+        if (deviceNumItems) {
+            CUB_LAUNCH(DeviceScan::ExclusiveSum, pools[deviceId], stream, deviceIn, deviceOut, deviceNumItems);
+        }
         cudaCheck(cudaEventRecord(preEvents[deviceId], stream));
-        deviceIn += counts[deviceId];
-        deviceOut += counts[deviceId];
+        deviceIn += deviceNumItems;
+        deviceOut += deviceNumItems;
     }
 
     deviceIn = in;
@@ -323,7 +348,7 @@ void exclusiveSumAsync(const nanovdb::cuda::DeviceMesh& deviceMesh, nanovdb::cud
         if (counts[deviceId]) {
             auto segmentExclusiveSum = deviceOut[counts[deviceId] - 1] + deviceIn[counts[deviceId] - 1];
 
-            unsigned int numBlocks = (counts[deviceId] + NumThreads - 1) / NumThreads;
+            unsigned int numBlocks = ::cuda::ceil_div<int>(counts[deviceId], NumThreads);
             util::cuda::lambdaKernel<<<numBlocks, NumThreads, 0, stream>>>(counts[deviceId], [=] __device__ (size_t tid) { deviceOut[tid] += partialExclusiveSum; });
             cudaCheckError();
 
@@ -347,10 +372,12 @@ void inclusiveSumAsync(const nanovdb::cuda::DeviceMesh& deviceMesh, nanovdb::cud
         // Required for the host to pass the correct value of counts[deviceId]
         cudaCheck(cudaEventSynchronize(preEvents[deviceId]));
         uint32_t deviceNumItems = counts[deviceId];
-        CUB_LAUNCH(DeviceScan::InclusiveSum, pools[deviceId], stream, deviceIn, deviceOut, deviceNumItems);
+        if (deviceNumItems) {
+            CUB_LAUNCH(DeviceScan::InclusiveSum, pools[deviceId], stream, deviceIn, deviceOut, deviceNumItems);
+        }
         cudaCheck(cudaEventRecord(preEvents[deviceId], stream));
-        deviceIn += counts[deviceId];
-        deviceOut += counts[deviceId];
+        deviceIn += deviceNumItems;
+        deviceOut += deviceNumItems;
     }
 
     deviceIn = in;
@@ -364,7 +391,7 @@ void inclusiveSumAsync(const nanovdb::cuda::DeviceMesh& deviceMesh, nanovdb::cud
         if (counts[deviceId]) {
             auto segmentInclusiveSum = deviceOut[counts[deviceId] - 1];
 
-            unsigned int numBlocks = (counts[deviceId] + NumThreads - 1) / NumThreads;
+            unsigned int numBlocks = ::cuda::ceil_div<int>(counts[deviceId], NumThreads);
             util::cuda::lambdaKernel<<<numBlocks, NumThreads, 0, stream>>>(counts[deviceId], [=] __device__ (size_t tid) { deviceOut[tid] += partialInclusiveSum; });
             cudaCheckError();
 
@@ -425,7 +452,7 @@ public:
 
 private:
     static constexpr unsigned int mNumThreads = 128;
-    static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
+    static unsigned int numBlocks(unsigned int n) {return ::cuda::ceil_div(n, mNumThreads);}
 
     uint32_t* deviceNodeCount(int deviceId) const { return mNodeCounts + 3 * deviceId; }
 
@@ -476,7 +503,7 @@ DistributedPointsToGrid<BuildT>::DistributedPointsToGrid(const nanovdb::cuda::De
     mVoxelOffsets = nullptr;
     cudaCheck(cudaMallocManaged(&mVoxelOffsets, mDeviceMesh.deviceCount() * sizeof(uint32_t)));
     mIntervals = nullptr;
-    cudaCheck(cudaMallocManaged(&mIntervals, 2 * mDeviceMesh.deviceCount() * sizeof(ptrdiff_t)));
+    cudaCheck(cudaMallocHost(&mIntervals, 2 * (mDeviceMesh.deviceCount() + 1) * sizeof(ptrdiff_t)));
 }
 
 template <typename BuildT>
@@ -488,7 +515,7 @@ DistributedPointsToGrid<BuildT>::DistributedPointsToGrid(const nanovdb::cuda::De
 template <typename BuildT>
 DistributedPointsToGrid<BuildT>::~DistributedPointsToGrid()
 {
-    cudaCheck(cudaFree(mIntervals));
+    cudaCheck(cudaFreeHost(mIntervals));
     cudaCheck(cudaFree(mVoxelOffsets));
     cudaCheck(cudaFree(mVoxelCounts));
     cudaCheck(cudaFree(mNodeOffsets));
@@ -560,8 +587,6 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
     // to reduce overhead.
     std::vector<cudaEvent_t> sortEvents(mDeviceMesh.deviceCount());
     std::vector<cudaEvent_t> runLengthEncodeEvents(mDeviceMesh.deviceCount());
-    std::vector<cudaEvent_t> transformReduceEvents(mDeviceMesh.deviceCount());
-    std::vector<cudaEvent_t> rebalanceEvents(mDeviceMesh.deviceCount());
     std::vector<cudaEvent_t> tilePrefixSumEvents(mDeviceMesh.deviceCount());
     std::vector<cudaEvent_t> voxelCountEvents(mDeviceMesh.deviceCount());
     std::vector<cudaEvent_t> leafCountEvents(mDeviceMesh.deviceCount());
@@ -572,8 +597,6 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
         cudaCheck(cudaSetDevice(deviceId));
         cudaEventCreateWithFlags(&sortEvents[deviceId], cudaEventDisableTiming);
         cudaEventCreateWithFlags(&runLengthEncodeEvents[deviceId], cudaEventDisableTiming);
-        cudaEventCreateWithFlags(&transformReduceEvents[deviceId], cudaEventDisableTiming);
-        cudaEventCreateWithFlags(&rebalanceEvents[deviceId], cudaEventDisableTiming);
         cudaEventCreateWithFlags(&tilePrefixSumEvents[deviceId], cudaEventDisableTiming);
         cudaEventCreateWithFlags(&voxelCountEvents[deviceId], cudaEventDisableTiming);
         cudaEventCreateWithFlags(&leafCountEvents[deviceId], cudaEventDisableTiming);
@@ -582,33 +605,32 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
         cudaEventCreateWithFlags(&leafPrefixSumEvents[deviceId], cudaEventDisableTiming);
     }
 
-    // Advise per-coord quantities to be split even across devices
+    // Advise per-coord quantities to be split evenly across devices. Clamp each stripe to
+    // the input range so that inputs smaller than the device count produce valid trailing empty stripes.
+    const size_t deviceStripeSize = ::cuda::ceil_div(coordCount, mDeviceMesh.deviceCount());
     for (const auto& [deviceId, stream] : mDeviceMesh) {
         cudaCheck(cudaSetDevice(deviceId));
 
-        size_t deviceStripeCount = (coordCount + mDeviceMesh.deviceCount() - 1) / mDeviceMesh.deviceCount();
-        const ptrdiff_t deviceStripeOffset = deviceStripeCount * deviceId;
-        deviceStripeCount = std::min(deviceStripeCount, coordCount - deviceStripeOffset);
+        const ptrdiff_t deviceStripeOffset = std::min(deviceStripeSize * deviceId, coordCount);
+        const size_t deviceStripeCount = std::min(deviceStripeSize, coordCount - deviceStripeOffset);
 
         mStripeCounts[deviceId] = deviceStripeCount;
         mStripeOffsets[deviceId] = deviceStripeOffset;
 
-        nanovdb::Coord* deviceCoords = coords + deviceStripeOffset;
-        uint64_t* deviceInputKeys = mKeys + deviceStripeOffset;
-        uint32_t* deviceInputIndices = mIndices + deviceStripeOffset;
-        uint64_t* deviceOutputKeys = mData->d_keys + deviceStripeOffset;
-        uint32_t* deviceOutputIndices = mData->d_indx + deviceStripeOffset;
+        if (deviceStripeCount) {
+            uint64_t* deviceInputKeys = mKeys + deviceStripeOffset;
+            uint32_t* deviceInputIndices = mIndices + deviceStripeOffset;
+            uint64_t* deviceOutputKeys = mData->d_keys + deviceStripeOffset;
+            uint32_t* deviceOutputIndices = mData->d_indx + deviceStripeOffset;
 
-        util::cuda::memAdvise(deviceCoords, deviceStripeCount * sizeof(nanovdb::Coord), cudaMemAdviseSetPreferredLocation, deviceId);
-        util::cuda::memAdvise(deviceCoords, deviceStripeCount * sizeof(nanovdb::Coord), cudaMemAdviseSetReadMostly, deviceId);
+            util::cuda::memAdvise(deviceInputKeys, deviceStripeCount * sizeof(uint64_t), cudaMemAdviseSetPreferredLocation, deviceId);
+            util::cuda::memAdvise(deviceInputIndices, deviceStripeCount * sizeof(uint32_t), cudaMemAdviseSetPreferredLocation, deviceId);
+            util::cuda::memAdvise(deviceOutputKeys, deviceStripeCount * sizeof(uint64_t), cudaMemAdviseSetPreferredLocation, deviceId);
+            util::cuda::memAdvise(deviceOutputIndices, deviceStripeCount * sizeof(uint32_t), cudaMemAdviseSetPreferredLocation, deviceId);
 
-        util::cuda::memAdvise(deviceInputKeys, deviceStripeCount * sizeof(uint64_t), cudaMemAdviseSetPreferredLocation, deviceId);
-        util::cuda::memAdvise(deviceInputIndices, deviceStripeCount * sizeof(uint32_t), cudaMemAdviseSetPreferredLocation, deviceId);
-        util::cuda::memAdvise(deviceOutputKeys, deviceStripeCount * sizeof(uint64_t), cudaMemAdviseSetPreferredLocation, deviceId);
-        util::cuda::memAdvise(deviceOutputIndices, deviceStripeCount * sizeof(uint32_t), cudaMemAdviseSetPreferredLocation, deviceId);
-
-        uint32_t* devicePointsPerTile = mPointsPerTile + deviceStripeOffset;
-        util::cuda::memAdvise(devicePointsPerTile, deviceStripeCount * sizeof(uint32_t), cudaMemAdviseSetPreferredLocation, deviceId);
+            uint32_t* devicePointsPerTile = mPointsPerTile + deviceStripeOffset;
+            util::cuda::memAdvise(devicePointsPerTile, deviceStripeCount * sizeof(uint32_t), cudaMemAdviseSetPreferredLocation, deviceId);
+        }
         util::cuda::memAdvise(deviceNodeCount(deviceId), 3 * sizeof(uint32_t), cudaMemAdviseSetPreferredLocation, deviceId);
     }
 
@@ -619,67 +641,51 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
         auto deviceStripeCount = mStripeCounts[deviceId];
         auto deviceStripeOffset = mStripeOffsets[deviceId];
 
-        util::cuda::memPrefetchAsync(coords, coordCount * sizeof(nanovdb::Coord), deviceId, stream);
-
-        util::cuda::offsetLambdaKernel<<<numBlocks(deviceStripeCount), mNumThreads, 0, stream>>>(deviceStripeCount, deviceStripeOffset, TileKeyFunctor<BuildT, PtrT>(), mData, coords, mData->d_keys, mData->d_indx);
+        if (deviceStripeCount) {
+            util::cuda::memPrefetchAsync(coords, coordCount * sizeof(nanovdb::Coord), deviceId, stream);
+            nanovdb::util::cuda::offsetLambdaKernel<<<numBlocks(deviceStripeCount), mNumThreads, 0, stream>>>(deviceStripeCount, deviceStripeOffset, TileKeyFunctor<BuildT, PtrT>(), mData, coords, mData->d_keys, mData->d_indx);
+            cudaCheckError();
+        }
     }
 
     radixSortAsync(mDeviceMesh, mTempDevicePools, mData->d_keys, mKeys, mData->d_indx, mIndices, coordCount, mIntervals, mStripeOffsets, mStripeCounts, sortEvents.data(), sortEvents.data());
 
-    // For each segment of sorted keys on each device, we count how many of the leftmost key occur past the left boundary of the segment. The same is done for the rightmost key with the right boundary of the segment.
-    auto leftIntervals = mIntervals;
-    auto rightIntervals = mIntervals + mDeviceMesh.deviceCount();
-    for (const auto& [deviceId, stream] : mDeviceMesh) {
-        cudaCheck(cudaSetDevice(deviceId));
-
-        auto deviceStripeCount = mStripeCounts[deviceId];
-        auto deviceStripeOffset = mStripeOffsets[deviceId];
-        uint64_t* deviceInputKeys = mKeys + deviceStripeOffset;
-
-        if (deviceId > 0) {
-            cudaStreamWaitEvent(stream, sortEvents[deviceId - 1]);
-            EqualityIndicator<uint64_t> indicator(deviceInputKeys - 1);
-            CUB_LAUNCH(DeviceReduce::TransformReduce, mTempDevicePools[deviceId], stream, deviceInputKeys, rightIntervals + deviceId, deviceStripeCount, ::cuda::std::plus(), indicator, 0);
-        }
-        else {
-            rightIntervals[deviceId] = 0;
-        }
-
-        if (deviceId < static_cast<int>(mDeviceMesh.deviceCount() - 1)) {
-            cudaStreamWaitEvent(stream, sortEvents[deviceId + 1]);
-            EqualityIndicator<uint64_t> indicator(deviceInputKeys + deviceStripeCount);
-            CUB_LAUNCH(DeviceReduce::TransformReduce, mTempDevicePools[deviceId], stream, deviceInputKeys, leftIntervals + deviceId, deviceStripeCount, ::cuda::std::plus(), indicator, 0);
-        }
-        else {
-            leftIntervals[deviceId] = 0;
-        }
-        cudaEventRecord(transformReduceEvents[deviceId], stream);
+    // Rebalance the device segments so that a device boundary always coincides
+    // with a change in key value. Because TileKeyFunctor assigns identical keys
+    // to every point that falls in the same upper-node "tile", this aligns the
+    // device ownership boundaries with tile boundaries. Downstream construction
+    // assumes each tile (and therefore each lower node, leaf node, and voxel) is
+    // owned by exactly one device; if a tile straddled a boundary, multiple
+    // devices would concurrently build the same leaf and race on its value mask.
+    //
+    // A single tile can span three or more devices (e.g. one dense leaf whose
+    // points are split evenly across the mesh). Adjusting only adjacent pairs of
+    // boundaries cannot consolidate such a tile because a fully-interior device
+    // lies entirely within it, so we compute the boundaries globally and
+    // monotonically. mKeys is globally sorted at this point, so a tile boundary
+    // is simply a position where mKeys changes. Snapping runs in a single-thread
+    // kernel on one device (every device stream is already ordered after the
+    // whole sort by radixSortAsync's final stream merge) with binary searches for
+    // the run extents; the host only waits on its completion event before reading
+    // back the small offset/count arrays, mirroring how the sort's merge-path
+    // partitions are synchronized. Fully-interior devices are left with empty
+    // stripes, which the rest of the pipeline already handles.
+    {
+        static constexpr int snapDeviceId = 0;
+        cudaCheck(cudaSetDevice(snapDeviceId));
+        const auto snapStream = mDeviceMesh[snapDeviceId].stream;
+        kernels::snapBoundariesToRunsKernel<<<1, 1, 0, snapStream>>>(mKeys, static_cast<ptrdiff_t>(coordCount), static_cast<int>(mDeviceMesh.deviceCount()), mStripeOffsets, mStripeCounts);
+        cudaCheckError();
+        cudaCheck(cudaEventRecord(sortEvents[snapDeviceId], snapStream));
+        // CUB needs the rebalanced partition sizes on the host. Synchronize only the tiny snap kernel.
+        cudaCheck(cudaEventSynchronize(sortEvents[snapDeviceId]));
     }
 
-    // Rebalance the segments so that a device segment boundary also corresponds to a change in key value. Effectively, this aligns upper node boundaries with device ownership boundaries.
+    // Parallel RLE in order to obtain tiles. The device boundaries were finalized
+    // above before the host read them back, so no per-device rebalance event is needed.
     for (const auto& [deviceId, stream] : mDeviceMesh) {
         cudaCheck(cudaSetDevice(deviceId));
 
-        if (deviceId > 0)
-        {
-            cudaStreamWaitEvent(stream, transformReduceEvents[deviceId - 1]);
-            kernels::rightRebalanceKernel<<<1, 1, 0, stream>>>(leftIntervals + deviceId - 1, rightIntervals + deviceId, mStripeCounts + deviceId, mStripeOffsets + deviceId);
-        }
-
-        if (deviceId < static_cast<int>(mDeviceMesh.deviceCount() - 1))
-        {
-            cudaStreamWaitEvent(stream, transformReduceEvents[deviceId + 1]);
-            kernels::leftRebalanceKernel<<<1, 1, 0, stream>>>(leftIntervals + deviceId, rightIntervals + deviceId + 1, mStripeCounts + deviceId, mStripeOffsets + deviceId);
-        }
-        cudaEventRecord(rebalanceEvents[deviceId], stream);
-    }
-
-    // Parallel RLE in order to obtain tiles
-    for (const auto& [deviceId, stream] : mDeviceMesh) {
-        cudaCheck(cudaSetDevice(deviceId));
-
-        cudaCheck(cudaEventSynchronize(rebalanceEvents[deviceId]));
-
         auto deviceStripeCount = mStripeCounts[deviceId];
         auto deviceStripeOffset = mStripeOffsets[deviceId];
 
@@ -689,7 +695,11 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
 
         // util::cuda::memPrefetchAsync(deviceInputKeys, deviceStripeCount * sizeof(uint64_t), deviceId, stream);
 
-        CUB_LAUNCH(DeviceRunLengthEncode::Encode, mTempDevicePools[deviceId], stream, deviceInputKeys, deviceOutputKeys, devicePointsPerTile, deviceNodeCount(deviceId) + 2, deviceStripeCount);
+        if (deviceStripeCount) {
+            CUB_LAUNCH(DeviceRunLengthEncode::Encode, mTempDevicePools[deviceId], stream, deviceInputKeys, deviceOutputKeys, devicePointsPerTile, deviceNodeCount(deviceId) + 2, deviceStripeCount);
+        } else {
+            cudaCheck(cudaMemsetAsync(deviceNodeCount(deviceId) + 2, 0, sizeof(uint32_t), stream));
+        }
         cudaCheck(cudaEventRecord(runLengthEncodeEvents[deviceId], stream));
     }
 
@@ -699,7 +709,9 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
         cudaCheck(cudaEventSynchronize(runLengthEncodeEvents[deviceId]));
         auto deviceStripeOffset = mStripeOffsets[deviceId];
         uint64_t* deviceKeys = mData->d_keys + deviceStripeOffset;
-        cudaCheck(cudaMemcpyAsync(mData->d_tile_keys + upperOffset, deviceKeys, sizeof(uint64_t) * deviceNodeCount(deviceId)[2], cudaMemcpyDefault, stream));
+        if (deviceNodeCount(deviceId)[2]) {
+            cudaCheck(cudaMemcpyAsync(mData->d_tile_keys + upperOffset, deviceKeys, sizeof(uint64_t) * deviceNodeCount(deviceId)[2], cudaMemcpyDefault, stream));
+        }
         deviceNodeOffset(deviceId)[2] = upperOffset;
         upperOffset += deviceNodeCount(deviceId)[2];
     }
@@ -766,23 +778,25 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
             uint64_t* deviceInputKeys = mKeys + mStripeOffsets[deviceId];
             const uint64_t* deviceOutputKeys = mData->d_keys + mStripeOffsets[deviceId];
 
-            if (deviceId == 0) {
+            if (deviceId > 0) {
+                cudaCheck(cudaEventSynchronize(voxelCountEvents[deviceId - 1]));
+                devicePointsPerVoxel += mVoxelCounts[deviceId - 1];
+
+                cudaCheck(cudaEventSynchronize(leafCountEvents[deviceId - 1]));
+                devicePointsPerLeaf += deviceNodeCount(deviceId - 1)[0];
+            }
+
+            if (mStripeCounts[deviceId]) {
                 CUB_LAUNCH(DeviceRunLengthEncode::Encode, mTempDevicePools[deviceId], stream, deviceOutputKeys, deviceInputKeys, devicePointsPerVoxel, mVoxelCounts + deviceId, mStripeCounts[deviceId]);
                 cudaCheck(cudaEventRecord(voxelCountEvents[deviceId], stream));
 
                 CUB_LAUNCH(DeviceRunLengthEncode::Encode, mTempDevicePools[deviceId], stream, thrust::make_transform_iterator(deviceOutputKeys, ShiftRight<9>()), deviceInputKeys, devicePointsPerLeaf, deviceNodeCount(deviceId), mStripeCounts[deviceId]);
                 cudaCheck(cudaEventRecord(leafCountEvents[deviceId], stream));
-            }
-            else
-            {
-                cudaCheck(cudaEventSynchronize(voxelCountEvents[deviceId - 1]));
-                devicePointsPerVoxel += mVoxelCounts[deviceId - 1];
-                CUB_LAUNCH(DeviceRunLengthEncode::Encode, mTempDevicePools[deviceId], stream, deviceOutputKeys, deviceInputKeys, devicePointsPerVoxel, mVoxelCounts + deviceId, mStripeCounts[deviceId]);
+            } else {
+                cudaCheck(cudaMemsetAsync(mVoxelCounts + deviceId, 0, sizeof(uint32_t), stream));
                 cudaCheck(cudaEventRecord(voxelCountEvents[deviceId], stream));
 
-                cudaCheck(cudaEventSynchronize(leafCountEvents[deviceId - 1]));
-                devicePointsPerLeaf += deviceNodeCount(deviceId - 1)[0];
-                CUB_LAUNCH(DeviceRunLengthEncode::Encode, mTempDevicePools[deviceId], stream, thrust::make_transform_iterator(deviceOutputKeys, ShiftRight<9>()), deviceInputKeys, devicePointsPerLeaf, deviceNodeCount(deviceId), mStripeCounts[deviceId]);
+                cudaCheck(cudaMemsetAsync(deviceNodeCount(deviceId), 0, sizeof(uint32_t), stream));
                 cudaCheck(cudaEventRecord(leafCountEvents[deviceId], stream));
             }
         }
@@ -796,7 +810,9 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
     for (const auto& [deviceId, stream] : mDeviceMesh) {
         cudaCheck(cudaSetDevice(deviceId));
         uint64_t* deviceKeys = mKeys + mStripeOffsets[deviceId];
-        cudaCheck(cudaMemcpyAsync(mData->d_leaf_keys + leafOffset, deviceKeys, sizeof(uint64_t) * deviceNodeCount(deviceId)[0], cudaMemcpyDefault, stream));
+        if (deviceNodeCount(deviceId)[0]) {
+            cudaCheck(cudaMemcpyAsync(mData->d_leaf_keys + leafOffset, deviceKeys, sizeof(uint64_t) * deviceNodeCount(deviceId)[0], cudaMemcpyDefault, stream));
+        }
         deviceNodeOffset(deviceId)[0] = leafOffset;
         leafOffset += deviceNodeCount(deviceId)[0];
     }
@@ -808,8 +824,12 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
         uint64_t* deviceInputKeys = mKeys + mStripeOffsets[deviceId];
         uint64_t* deviceOutputKeys = mData->d_keys + mStripeOffsets[deviceId];
 
-        CUB_LAUNCH(DeviceSelect::Unique, mTempDevicePools[deviceId], stream, thrust::make_transform_iterator(deviceOutputKeys, ShiftRight<21>()), deviceInputKeys, deviceNodeCount(deviceId) + 1, mStripeCounts[deviceId]);
-        cudaEventRecord(lowerCountEvents[deviceId], stream);
+        if (mStripeCounts[deviceId]) {
+            CUB_LAUNCH(DeviceSelect::Unique, mTempDevicePools[deviceId], stream, thrust::make_transform_iterator(deviceOutputKeys, ShiftRight<21>()), deviceInputKeys, deviceNodeCount(deviceId) + 1, mStripeCounts[deviceId]);
+        } else {
+            cudaCheck(cudaMemsetAsync(deviceNodeCount(deviceId) + 1, 0, sizeof(uint32_t), stream));
+        }
+        cudaCheck(cudaEventRecord(lowerCountEvents[deviceId], stream));
     }
 
     uint32_t lowerOffset = 0;
@@ -817,7 +837,9 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
         cudaCheck(cudaSetDevice(deviceId));
         cudaCheck(cudaEventSynchronize(lowerCountEvents[deviceId]));
         uint64_t* deviceKeys = mKeys + mStripeOffsets[deviceId];
-        cudaCheck(cudaMemcpyAsync(mData->d_lower_keys + lowerOffset, deviceKeys, sizeof(uint64_t) * deviceNodeCount(deviceId)[1], cudaMemcpyDefault, stream));
+        if (deviceNodeCount(deviceId)[1]) {
+            cudaCheck(cudaMemcpyAsync(mData->d_lower_keys + lowerOffset, deviceKeys, sizeof(uint64_t) * deviceNodeCount(deviceId)[1], cudaMemcpyDefault, stream));
+        }
         deviceNodeOffset(deviceId)[1] = lowerOffset;
         lowerOffset += deviceNodeCount(deviceId)[1];
     }
@@ -837,8 +859,6 @@ void DistributedPointsToGrid<BuildT>::countNodes(const PtrT coords, size_t coord
         cudaCheck(cudaSetDevice(deviceId));
         cudaEventDestroy(sortEvents[deviceId]);
         cudaEventDestroy(runLengthEncodeEvents[deviceId]);
-        cudaEventDestroy(transformReduceEvents[deviceId]);
-        cudaEventDestroy(rebalanceEvents[deviceId]);
         cudaEventDestroy(tilePrefixSumEvents[deviceId]);
         cudaEventDestroy(voxelCountEvents[deviceId]);
         cudaEventDestroy(leafCountEvents[deviceId]);
@@ -909,13 +929,16 @@ inline void DistributedPointsToGrid<BuildT>::processGridTreeRoot(const PtrT poin
     util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, BuildGridTreeRootFunctor<BuildT, PtrT>(), mData, mPointType, pointCount);// lambdaKernel
     cudaCheckError();
 
+    // Zero the name field, then copy only the actual string (if any).
     char *dst = mData->getGrid().mGridName;
-    if (const char *src = mGridName.data()) {
-        cudaCheck(cudaMemcpyAsync(dst, src, GridData::MaxNameSize, cudaMemcpyHostToDevice, stream));
-    } else {
-        cudaCheck(cudaMemsetAsync(dst, 0, GridData::MaxNameSize, stream));
+    cudaCheck(cudaMemsetAsync(dst, 0, GridData::MaxNameSize, stream));
+    if (!mGridName.empty()) {
+        // Copy at most MaxNameSize-1 bytes so the memset's trailing '\0' always
+        // survives; a name >= MaxNameSize is truncated, never left unterminated.
+        const size_t nameSize = std::min<size_t>(mGridName.size(), GridData::MaxNameSize - 1);
+        cudaCheck(cudaMemcpyAsync(dst, mGridName.c_str(), nameSize, cudaMemcpyHostToDevice, stream));
     }
-    cudaEventRecord(processGridTreeRootEvent);
+    cudaCheck(cudaEventRecord(processGridTreeRootEvent, stream));
 
     for (const auto& [otherDeviceId, otherStream] : mDeviceMesh) {
         cudaSetDevice(otherDeviceId);
@@ -976,12 +999,14 @@ inline void DistributedPointsToGrid<BuildT>::processNodes()
         std::vector<cudaEvent_t> leafCountEvents(mDeviceMesh.deviceCount());
         std::vector<cudaEvent_t> valueIndexPrefixSumEvents(mDeviceMesh.deviceCount());
 
+        auto lastDeviceId = cudaInvalidDeviceId;
         for (const auto& [deviceId, stream] : mDeviceMesh) {
             cudaSetDevice(deviceId);
             cudaEventCreateWithFlags(&leafCountEvents[deviceId], cudaEventDisableTiming);
             cudaEventCreateWithFlags(&valueIndexPrefixSumEvents[deviceId], cudaEventDisableTiming);
 
             if (deviceNodeCount(deviceId)[0]) {
+                lastDeviceId = deviceId;
                 kernels::fillValueIndexKernel<BuildT><<<numBlocks(deviceNodeCount(deviceId)[0]), mNumThreads, 0, stream>>>(deviceNodeCount(deviceId)[0], deviceNodeOffset(deviceId)[0], mValueIndex, mData);
                 cudaCheckError();
             }
@@ -990,10 +1015,18 @@ inline void DistributedPointsToGrid<BuildT>::processNodes()
         LeafCountIterator leafCountIterator(mNodeCounts);
         inclusiveSumAsync(mDeviceMesh, mTempDevicePools, mValueIndex, mValueIndexPrefix, leafCountIterator, leafCountEvents.data(), valueIndexPrefixSumEvents.data());
 
+        // The first leaf on each device reads the last prefix value produced by the
+        // previous device, while the first leaf globally reads the final prefix value.
+        // Wait for the corresponding producer before launching each leaf-processing kernel.
+        auto previousDeviceId = cudaInvalidDeviceId;
         for (const auto& [deviceId, stream] : mDeviceMesh) {
             cudaSetDevice(deviceId);
-            cudaStreamWaitEvent(stream, valueIndexPrefixSumEvents.back());
             if (deviceNodeCount(deviceId)[0]) {
+                cudaStreamWaitEvent(stream, valueIndexPrefixSumEvents[lastDeviceId]);
+                if (previousDeviceId != cudaInvalidDeviceId) {
+                    cudaStreamWaitEvent(stream, valueIndexPrefixSumEvents[previousDeviceId]);
+                }
+                previousDeviceId = deviceId;
                 kernels::leafPrefixSumKernel<BuildT><<<numBlocks(deviceNodeCount(deviceId)[0]), mNumThreads, 0, stream>>>(deviceNodeCount(deviceId)[0], deviceNodeOffset(deviceId)[0], mValueIndexPrefix, mData);
                 cudaCheckError();
             }

nanovdb/nanovdb/tools/cuda/GridChecksum.cuh

@@ -109,6 +109,89 @@ inline unique_ptr<uint32_t> createCrc32Lut(size_t extra = 0, cudaStream_t stream
     return lut;
 }
 
+/// @brief Cuda kernel computing per-block CRC32 checksums via slicing-by-4.
+/// The 256-entry base LUT is staged into shared memory and three derived
+/// slice tables are built in place, so the four divergent lookups per 4-byte
+/// step hit shared memory and the dependent update chain advances four bytes
+/// per step instead of one. Bit-identical to the byte-serial crc32().
+/// The final block absorbs any remainder of @c totalSize.
+static __global__ void crc32SlicedKernel(const void *d_data, uint32_t* d_blockCRC, uint64_t blockCount, uint32_t log2BlockSize, uint64_t totalSize, const uint32_t *d_lut)
+{
+    __shared__ uint32_t sLut[4][256];
+    for (uint32_t i = threadIdx.x; i < 256; i += blockDim.x) sLut[0][i] = d_lut[i];
+    __syncthreads();
+    for (int k = 1; k < 4; ++k) {
+        for (uint32_t i = threadIdx.x; i < 256; i += blockDim.x) {
+            const uint32_t c = sLut[k-1][i];
+            sLut[k][i] = (c >> 8) ^ sLut[0][c & 0xffu];
+        }
+        __syncthreads();
+    }
+    const uint64_t tid = blockIdx.x * (uint64_t)blockDim.x + threadIdx.x;
+    if (tid >= blockCount) return;
+    const uint8_t *p = (const uint8_t*)d_data + (tid << log2BlockSize);
+    uint64_t n = uint64_t(1) << log2BlockSize;
+    if (tid + 1 == blockCount) n += totalSize - (blockCount << log2BlockSize);
+    uint32_t crc = ~0u;
+    // blocks start at power-of-two offsets of a 32B-aligned buffer -> 4B aligned
+    const uint32_t *w = (const uint32_t*)p;
+    for (uint64_t i = 0, nw = n >> 2; i < nw; ++i) {
+        const uint32_t x = crc ^ w[i];
+        crc = sLut[3][x & 0xffu] ^ sLut[2][(x >> 8) & 0xffu] ^ sLut[1][(x >> 16) & 0xffu] ^ sLut[0][x >> 24];
+    }
+    for (uint64_t i = n & ~3ull; i < n; ++i) {
+        crc ^= p[i];
+        for (int j = 0; j < 8; ++j) crc = (crc >> 1) ^ (0xEDB88320u & (-(crc & 1u)));
+    }
+    d_blockCRC[tid] = ~crc;
+}
+
+/// @brief y = M x over GF(2), M given as 32 column words
+__host__ __device__ inline uint32_t crc32Gf2MatTimes(const uint32_t *mat, uint32_t vec)
+{
+    uint32_t sum = 0;
+    while (vec) {
+        if (vec & 1u) sum ^= *mat;
+        vec >>= 1;
+        ++mat;
+    }
+    return sum;
+}
+
+/// @brief Build into @c dst[32] the GF(2) operator that advances a CRC past
+/// @c bits zero bits, by binary exponentiation of the single-bit operator
+/// (zlib crc32_combine construction). O(log bits x 32) - a few microseconds on
+/// the host, which is where it is called so the device combine stays a cheap fold.
+__host__ __device__ inline void crc32BuildShiftOp(uint32_t *dst, uint64_t bits)
+{
+    uint32_t oddBuf[32], evenBuf[32];
+    uint32_t *cur = oddBuf, *nxt = evenBuf;
+    cur[0] = 0xEDB88320u;// operator for a single zero bit
+    for (int n = 1; n < 32; ++n) cur[n] = 1u << (n - 1);
+    for (int n = 0; n < 32; ++n) dst[n] = 1u << n;// identity
+    while (bits) {
+        if (bits & 1ull) for (int n = 0; n < 32; ++n) dst[n] = crc32Gf2MatTimes(cur, dst[n]);
+        bits >>= 1;
+        if (bits) { for (int n = 0; n < 32; ++n) nxt[n] = crc32Gf2MatTimes(cur, cur[n]); uint32_t *t = cur; cur = nxt; nxt = t; }
+    }
+}
+
+/// @brief Single-thread kernel folding per-chunk CRCs into the CRC of the whole
+/// stream, using crc(A||B) = shift(crc(A), len(B)) ^ crc(B). The two GF(2) shift
+/// operators - @c d_acc for a full chunk and @c d_accLast for the final,
+/// possibly shorter, chunk - are precomputed on the host, so this is just an
+/// O(chunkCount) fold. Bit-identical to a serial crc32 over the concatenated
+/// stream. (@c d_accLast equals @c d_acc when the last chunk is full.)
+static __global__ void crc32CombineKernel(const uint32_t *d_chunkCRC, uint64_t chunkCount, const uint32_t *d_acc, const uint32_t *d_accLast, uint32_t *d_crc)
+{
+    uint32_t crc = d_chunkCRC[0];
+    for (uint64_t i = 1; i < chunkCount; ++i) {
+        const uint32_t *op = (i + 1 == chunkCount) ? d_accLast : d_acc;
+        crc = crc32Gf2MatTimes(op, crc) ^ d_chunkCRC[i];
+    }
+    *d_crc = crc;
+}
+
 /// @brief Compute CRC32 checksum of 4K block
 /// @param d_data device pointer to start of data
 /// @param size number of bytes
@@ -121,14 +204,39 @@ inline void blockedCRC32(const void *d_data, size_t size, const uint32_t *d_lut,
     const uint64_t checksumCount = size >> NANOVDB_CRC32_LOG2_BLOCK_SIZE;// 4 KB (4096 byte)
     unique_ptr<uint32_t> buffer(checksumCount, stream);// for checksums of 4 KB blocks
     uint32_t *d_checksums = buffer.get();
-    lambdaKernel<<<blocksPerGrid(checksumCount, threadsPerBlock), threadsPerBlock, 0, stream>>>(checksumCount, [=] __device__(size_t tid) {
-        uint32_t blockSize = 1 << NANOVDB_CRC32_LOG2_BLOCK_SIZE;
-        if (tid+1 == checksumCount) blockSize += size - (checksumCount<<NANOVDB_CRC32_LOG2_BLOCK_SIZE);
-        d_checksums[tid] = crc32((const uint8_t*)d_data + (tid<<NANOVDB_CRC32_LOG2_BLOCK_SIZE), blockSize, d_lut);
-    }); cudaCheckError();
-    lambdaKernel<<<1, 1, 0, stream>>>(1, [=] __device__(size_t) {// Compute CRC32 of all the 4K blocks
-        *d_crc = crc32((const uint8_t*)d_checksums, checksumCount*sizeof(uint32_t), d_lut);
-    }); cudaCheckError();
+    crc32SlicedKernel<<<blocksPerGrid(checksumCount, threadsPerBlock), threadsPerBlock, 0, stream>>>(
+        d_data, d_checksums, checksumCount, NANOVDB_CRC32_LOG2_BLOCK_SIZE, size, d_lut);
+    cudaCheckError();
+    // CRC of the block-checksum array itself. The former single-thread pass
+    // over checksumCount*4 bytes (megabytes for multi-GB grids) is replaced
+    // by parallel per-chunk CRCs plus a GF(2) combine - bit-identical result.
+    const uint64_t checksumBytes = checksumCount*sizeof(uint32_t);
+    constexpr uint64_t log2ChunkSize = 12, chunkSize = uint64_t(1) << log2ChunkSize;
+    if (checksumBytes <= 2*chunkSize) {// small: single-thread CRC is fine
+        lambdaKernel<<<1, 1, 0, stream>>>(1, [=] __device__(size_t) {
+            *d_crc = crc32((const uint8_t*)d_checksums, checksumBytes, d_lut);
+        }); cudaCheckError();
+    } else {
+        const uint64_t chunkCount = checksumBytes >> log2ChunkSize;// final chunk absorbs the remainder
+        const uint64_t lastChunkBytes = chunkSize + (checksumBytes - (chunkCount << log2ChunkSize));
+        unique_ptr<uint32_t> chunkBuffer(chunkCount, stream);
+        uint32_t *d_chunkCRC = chunkBuffer.get();
+        crc32SlicedKernel<<<blocksPerGrid(chunkCount, threadsPerBlock), threadsPerBlock, 0, stream>>>(
+            d_checksums, d_chunkCRC, chunkCount, log2ChunkSize, checksumBytes, d_lut);
+        cudaCheckError();
+        // Precompute the two GF(2) shift operators on the host (data-independent,
+        // ~microseconds) and upload them, instead of rebuilding them on a single
+        // device thread inside the combine - that build dominated the checksum
+        // for small and medium grids.
+        uint32_t hOps[64];
+        crc32BuildShiftOp(hOps,      chunkSize      * 8ull);// operator for a full chunk
+        crc32BuildShiftOp(hOps + 32, lastChunkBytes * 8ull);// operator for the last chunk
+        unique_ptr<uint32_t> opBuffer(64, stream);
+        uint32_t *d_ops = opBuffer.get();
+        cudaCheck(cudaMemcpyAsync(d_ops, hOps, 64*sizeof(uint32_t), cudaMemcpyHostToDevice, stream));
+        crc32CombineKernel<<<1, 1, 0, stream>>>(d_chunkCRC, chunkCount, d_ops, d_ops + 32, d_crc);
+        cudaCheckError();
+    }
 }// void cudaBlockedCRC32(const void *d_data, size_t size, const uint32_t *d_lut, uint32_t *d_crc, cudaStream_t stream)
 
 /// @brief Compute CRC32 checksum of 4K block
@@ -234,7 +342,7 @@ inline Checksum evalChecksum(const GridData *d_gridData, CheckMode mode, cudaStr
     if (mode != CheckMode::Empty) {
         auto d_lut = util::cuda::createCrc32Lut(1, stream);
         crc32Head(d_gridData, d_lut.get(), d_lut.get() + 256, stream);
-        cudaCheck(cudaMemcpyAsync(&(cs.head()), d_lut.get() + 256, headSize, cudaMemcpyDeviceToHost, stream));
+        cudaCheck(cudaMemcpyAsync(&(cs.head()), d_lut.get() + 256, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
         if (mode == CheckMode::Full) {
             std::unique_ptr<char[]> buffer(new char[headSize]);
             auto *gridData = (GridData*)(buffer.get());
@@ -244,7 +352,7 @@ inline Checksum evalChecksum(const GridData *d_gridData, CheckMode mode, cudaStr
             } else {
                 callNanoGrid<Crc32TailOld>(d_gridData, gridData, d_lut.get(), d_lut.get() + 256, stream);
             }
-            cudaCheck(cudaMemcpyAsync(&(cs.tail()), d_lut.get() + 256, headSize, cudaMemcpyDeviceToHost, stream));
+            cudaCheck(cudaMemcpyAsync(&(cs.tail()), d_lut.get() + 256, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
         }
     }
     return cs;
@@ -265,7 +373,7 @@ Checksum evalChecksum(const NanoGrid<BuildT> *d_grid, CheckMode mode, cudaStream
     if (mode != CheckMode::Empty) {
         auto d_lut = util::cuda::createCrc32Lut(1, stream);
         crc32Head(d_grid, d_lut.get(), d_lut.get() + 256, stream);
-        cudaCheck(cudaMemcpyAsync(&(cs.head()), d_lut.get() + 256, headSize, cudaMemcpyDeviceToHost, stream));
+        cudaCheck(cudaMemcpyAsync(&(cs.head()), d_lut.get() + 256, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
         if (mode == CheckMode::Full) {
             std::unique_ptr<char[]> buffer(new char[headSize]);
             auto *gridData = (GridData*)(buffer.get());
@@ -275,7 +383,7 @@ Checksum evalChecksum(const NanoGrid<BuildT> *d_grid, CheckMode mode, cudaStream
             } else {
                 crc32TailOld(d_grid, gridData, d_lut.get(), d_lut.get() + 256, stream);
             }
-            cudaCheck(cudaMemcpyAsync(&(cs.tail()), d_lut.get() + 256, headSize, cudaMemcpyDeviceToHost, stream));
+            cudaCheck(cudaMemcpyAsync(&(cs.tail()), d_lut.get() + 256, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
         }
     }
     return cs;

nanovdb/nanovdb/tools/cuda/GridStats.cuh

@@ -16,26 +16,33 @@
 #define NANOVDB_TOOLS_CUDA_GRIDSTATS_CUH_HAS_BEEN_INCLUDED
 
 #include <nanovdb/NanoVDB.h>
+#include <nanovdb/cuda/Buffer.h>
 #include <nanovdb/tools/GridStats.h>
 
+#include <cub/util_type.cuh>// for cub::Uninitialized
+
 namespace nanovdb {
 
 namespace tools::cuda {
 
 /// @brief Update, i.e. re-compute, grid statistics like min/max, stats and bbox
 ///        information for an existing NanoVDB Grid.
-/// @param grid   Grid whose stats to update
+/// @tparam ResourceT Template type of optional resource used for internal temporary memory
+/// @param d_grid Grid whose stats to update
 /// @param mode   Mode of computation for the statistics.
 /// @param stream Optional cuda stream (defaults to zero)
-template<typename BuildT>
+template<typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 void updateGridStats(NanoGrid<BuildT> *d_grid, StatsMode mode = StatsMode::Default, cudaStream_t stream = 0);
 
 //================================================================================================
 
-/// @brief Allows for the construction of NanoVDB grids without any dependency
-template<typename BuildT, typename StatsT = Stats<typename NanoGrid<BuildT>::ValueType>>
+/// @brief Re-computes statistics (of type @c StatsT) for each node of an existing NanoVDB grid on the device
+template<typename BuildT, typename StatsT = Stats<typename NanoGrid<BuildT>::ValueType>, typename ResourceT = nanovdb::cuda::DeviceResource>
 class GridStats
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "GridStats allocates stream-ordered scratch and requires an AsyncResource");
+
     using GridT  = NanoGrid<BuildT>;
     using TreeT  = typename GridT::TreeType;
     using ValueT = typename TreeT::ValueType;
@@ -46,10 +53,23 @@ class GridStats
     static_assert(util::is_same<ValueT, typename StatsT::ValueType>::value, "Mismatching type");
 
     ValueT mDelta; // skip rendering of node if: node.max < -mDelta || node.min > mDelta
+    ResourceT* mResource;// non-owning; all device scratch routes through this resource instance
+
+    template<typename T>
+    using BufT = nanovdb::cuda::Buffer<T, nanovdb::cuda::ResourceRef<ResourceT>>;
+    nanovdb::cuda::ResourceRef<ResourceT> ref() { return nanovdb::cuda::ResourceRef<ResourceT>(*mResource); }
 
 public:
-    GridStats(ValueT delta = ValueT(0)) : mDelta(delta) {}
+    /// @brief Constructor
+    /// @param delta skip rendering of nodes when node.max < -delta or node.min > delta
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this instance (defaults to the per-type default resource)
+    GridStats(ValueT delta = ValueT(0), ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : mDelta(delta), mResource(&resource) {}
 
+    /// @note All device allocations this method makes -- the per-node
+    ///       statistics scratch and the temporary NodeManager -- go through
+    ///       the injected resource.
     void update(GridT *d_grid, cudaStream_t stream = 0);
 
 }; // cuda::GridStats
@@ -58,65 +78,164 @@ public:
 
 namespace {// define cuda kernels in an unnamed namespace
 
+// One warp per leaf: lanes stride the 512 voxel slots (mask-gated) and merge
+// their partial statistics through shared memory; lane 0 handles the bbox
+// update and the final store. Launch with 128 threads (4 warps) per block.
 template<typename BuildT, typename StatsT>
 __global__ void processLeaf(NodeManager<BuildT> *d_nodeMgr, StatsT *d_stats)
 {
-    const uint32_t tid = blockIdx.x * blockDim.x + threadIdx.x;
+    constexpr uint32_t WarpsPerBlock = 4;
+    // CUDA does not run constructors for __shared__ variables, so wrap the scratch
+    // array in cub::Uninitialized (raw, constructor-free storage). Safe here because
+    // every slot is fully assigned before it is read.
+    __shared__ cub::Uninitialized<StatsT[WarpsPerBlock * 32]> sStatsStorage;
+    StatsT (&sStats)[WarpsPerBlock * 32] = sStatsStorage.Alias();
+
+    const uint32_t warpID = threadIdx.x >> 5, lane = threadIdx.x & 31u;
+    const uint32_t tid = blockIdx.x * WarpsPerBlock + warpID;// leaf index
     if (tid >= d_nodeMgr->leafCount()) return;
     auto &d_leaf = d_nodeMgr->leaf(tid);
 
-    if (d_leaf.updateBBox()) {// updates active bounding box (also updates data->mFlags) and return true if non-empty
+    bool nonEmpty = false;
+    // updates leaf's active bounding box (also updates data->mFlags), only run on lane 0
+    if (lane == 0) nonEmpty = d_leaf.updateBBox();
+    // broadcast nonEmpty value to all lanes; register-to-register warp shuffle with all lanes participating
+    nonEmpty = __shfl_sync(0xffffffffu, nonEmpty, 0); // 0xffffffffu is "all 32 lanes" mask
+
+    if (nonEmpty) {
         if constexpr(StatsT::hasStats()) {
+            // 1) Per-lane partial: each of the 32 lanes folds its strided share of the
+            //    512 voxel slots (lane L visits L, L+32, L+64, ...) into a local StatsT,
+            //    skipping inactive voxels via the value mask. This is the coalesced pass -
+            //    consecutive lanes touch consecutive slots on each stride.
             StatsT stats;
-            for (auto it = d_leaf.cbeginValueOn(); it; ++it) stats.add(*it);
-            if constexpr(StatsT::hasAverage()) {
-                d_stats[tid] = stats;
-                *reinterpret_cast<uint32_t*>(&d_leaf.mMinimum) = tid;
-            } else {
-                stats.setStats(d_leaf);
+            const auto &mask = d_leaf.valueMask();
+            for (uint32_t i = lane; i < NanoLeaf<BuildT>::SIZE; i += 32)
+                if (mask.isOn(i)) stats.add(d_leaf.getValue(i));
+            // 2) Warp reduction: stage the 32 partials in this warp's slice of shared
+            //    memory, then merge them pairwise in log2(32)=5 steps (16->8->4->2->1).
+            //    StatsT::add is an associative merge (exact for min/max; a Welford
+            //    combine for mean/variance), so the tree order is safe. __syncwarp
+            //    after each step orders the shared-memory writes within the warp.
+            StatsT *sWarp = sStats + (warpID << 5);// this warp's 32-slot scratch region
+            sWarp[lane] = stats;
+            __syncwarp();
+            // pairwise reduction
+            for (uint32_t d = 16; d; d >>= 1) {
+                if (lane < d) sWarp[lane].add(sWarp[lane + d]);
+                __syncwarp();
+            }
+            // 3) Publish: lane 0 now holds the leaf's fully merged stats in sWarp[0].
+            if (lane == 0) {
+                if constexpr(StatsT::hasAverage()) {
+                    // mean/variance: the parent lower node must Welford-merge this leaf's
+                    // full accumulator (count+mean+M2), not just its extrema, so publish
+                    // the whole StatsT into the per-node scratch array at this leaf's slot
+                    // and stash that slot index in mMinimum (reused as a uint32 handle) so
+                    // the parent kernel can locate it.
+                    d_stats[tid] = sWarp[0];
+                    *reinterpret_cast<uint32_t*>(&d_leaf.mMinimum) = tid;
+                } else {
+                    // min/max only: extrema compose directly from child to parent, so write
+                    // them straight into the leaf - no scratch slot needed.
+                    sWarp[0].setStats(d_leaf);
+                }
             }
         }
     }
-    d_leaf.mFlags &= ~uint8_t(1u);// enable rendering
+    if (lane == 0) d_leaf.mFlags &= ~uint8_t(1u);// enable rendering
 }// processLeaf
 
+// One block per internal node: threads stride the node's child table (4096 or
+// 32768 entries) and merge partial bboxes/statistics through shared memory -
+// the former one-thread-per-node kernel serialized up to 32768 child visits
+// per thread and left the device nearly empty at typical node counts.
 template<typename BuildT, typename StatsT, int LEVEL>
 __global__ void processInternal(NodeManager<BuildT> *d_nodeMgr, StatsT *d_stats)
 {
     using ChildT = typename NanoNode<BuildT,LEVEL-1>::type;
-    uint32_t nodeID = blockIdx.x * blockDim.x + threadIdx.x;// thread id (reused below to avoid compiler warning)
+    using NodeT = typename NanoNode<BuildT,LEVEL>::type;
+    constexpr uint32_t Threads = 128;
+    // Constructor-free shared storage, as in processLeaf: every slot is assigned
+    // before it is read.
+    __shared__ cub::Uninitialized<StatsT[Threads]> sStatsStorage;
+    __shared__ cub::Uninitialized<CoordBBox[Threads]> sBBoxStorage;
+    StatsT (&sStats)[Threads] = sStatsStorage.Alias();
+    CoordBBox (&sBBox)[Threads] = sBBoxStorage.Alias();
+
+    const uint32_t nodeID = blockIdx.x;
+    const uint32_t tID = threadIdx.x;
     if (nodeID >= d_nodeMgr->nodeCount(LEVEL)) return;
     auto &d_node = d_nodeMgr->template node<LEVEL>(nodeID);
-    auto &bbox   = d_node.mBBox;
-    bbox         = CoordBBox();// empty bbox
-    StatsT stats;
 
-    for (auto it = d_node.beginChild(); it; ++it) {
-        auto &child = *it;
-        bbox.expand( child.bbox() );
-        if constexpr(StatsT::hasAverage()) {
-            nodeID = *reinterpret_cast<uint32_t*>(&child.mMinimum);
-            StatsT &s = d_stats[nodeID];
-            s.setStats(child);
-            stats.add(s);
-        } else if constexpr(StatsT::hasMinMax()) {
-            stats.add(child.minimum());
-            stats.add(child.maximum());
+    // 1) Per-thread partial: each thread folds its strided share of the child table
+    //    (NodeT::SIZE = 4096 for lower, 32768 for upper) into a local bbox + stats.
+    //    Children were finalized by an earlier launch (leaves, then lower, then upper),
+    //    so a child's bbox()/stats are valid to read here. Each entry is one of three:
+    CoordBBox bbox;// empty
+    StatsT stats;
+    for (uint32_t i = tID; i < NodeT::SIZE; i += Threads) {
+        if (d_node.childMask().isOn(i)) {
+            // (a) a child node: union its finalized bbox and merge its statistics.
+            auto &child = *d_node.getChild(i);
+            bbox.expand( child.bbox() );
+            if constexpr(StatsT::hasAverage()) {
+                // The child published its full accumulator into d_stats and stashed the
+                // slot index in its mMinimum. We must read that handle FIRST (to index
+                // d_stats), then setStats commits the child's real min/max/avg/std into
+                // the child node - which also overwrites the mMinimum-as-slot handle with
+                // the true minimum. Finally merge the child's accumulator into ours.
+                StatsT &s = d_stats[*reinterpret_cast<const uint32_t*>(&child.mMinimum)];
+                s.setStats(child);
+                stats.add(s);
+            } else if constexpr(StatsT::hasMinMax()) {
+                // min/max compose directly - the child already holds its final extrema.
+                stats.add(child.minimum());
+                stats.add(child.maximum());
+            }
+        } else if (d_node.valueMask().isOn(i)) {
+            // (b) an active tile: one constant value covering the whole child region
+            //     (no child node). Grow the bbox to span the tile's extent, and add the
+            //     value with multiplicity NUM_VALUES (the voxel count it stands in for)
+            //     so the mean/variance are weighted correctly.
+            const Coord ijk = d_node.offsetToGlobalCoord(i);
+            bbox[0].minComponent(ijk);
+            bbox[1].maxComponent(ijk + Coord(ChildT::DIM - 1));
+            if constexpr(StatsT::hasStats()) stats.add(d_node.data()->getValue(i), ChildT::NUM_VALUES);
         }
+        // (c) otherwise inactive - contributes nothing.
     }
-    for (auto it = d_node.cbeginValueOn(); it; ++it) {
-        const Coord ijk = it.getCoord();
-        bbox[0].minComponent(ijk);
-        bbox[1].maxComponent(ijk + Coord(ChildT::DIM - 1));
-        if constexpr(StatsT::hasStats()) stats.add(*it, ChildT::NUM_VALUES);
+    // 2) Block reduction: merge the 128 per-thread partials (both stats and bbox) via a
+    //    shared-memory tree in log2(128)=7 steps. This spans 4 warps, so it needs
+    //    __syncthreads (block-wide barrier), not the __syncwarp used in processLeaf.
+    sStats[tID] = stats;
+    sBBox[tID] = bbox;
+    __syncthreads();
+    for (uint32_t d = Threads >> 1; d; d >>= 1) {
+        if (tID < d) {
+            sStats[tID].add(sStats[tID + d]);
+            sBBox[tID].expand(sBBox[tID + d]);
+        }
+        __syncthreads();
     }
-    if constexpr(StatsT::hasAverage()) {
-        d_stats[nodeID] = stats;
-        *reinterpret_cast<uint32_t*>(&d_node.mMinimum) = nodeID;
-    } else if constexpr(StatsT::hasMinMax()) {
-        stats.setStats(d_node);
+    // 3) Publish: thread 0 now holds the node's merged bbox + stats.
+    if (tID == 0) {
+        d_node.mBBox = sBBox[0];
+        if constexpr(StatsT::hasAverage()) {
+            // Each node writes its OWN unique d_stats slot (leaves occupy
+            // [0, leafCount), then lower nodes, then upper nodes), so a node
+            // with no children still has a valid slot. The previous scheme
+            // reused a child's slot, writing d_stats[-1] for a childless
+            // (fully-tiled) internal node - an out-of-bounds write.
+            const uint32_t slot = d_nodeMgr->leafCount()
+                                + (LEVEL == 2 ? d_nodeMgr->nodeCount(1) : 0u) + nodeID;
+            d_stats[slot] = sStats[0];
+            *reinterpret_cast<uint32_t*>(&d_node.mMinimum) = slot;
+        } else if constexpr(StatsT::hasMinMax()) {
+            sStats[0].setStats(d_node);
+        }
+        d_node.mFlags &= ~uint64_t(1u);// enable rendering
     }
-    d_node.mFlags &= ~uint64_t(1u);// enable rendering
 }// processInternal
 
 template<typename BuildT, typename StatsT>
@@ -184,50 +303,54 @@ __global__ void processRootAndGrid(NodeManager<BuildT> *d_nodeMgr, StatsT *d_sta
 
 //================================================================================================
 
-template<typename BuildT, typename StatsT>
-void GridStats<BuildT, StatsT>::update(NanoGrid<BuildT> *d_grid, cudaStream_t stream)
+template<typename BuildT, typename StatsT, typename ResourceT>
+void GridStats<BuildT, StatsT, ResourceT>::update(NanoGrid<BuildT> *d_grid, cudaStream_t stream)
 {
     static const uint32_t threadsPerBlock = 128;
     auto blocksPerGrid = [&](uint32_t count)->uint32_t{return (count + (threadsPerBlock - 1)) / threadsPerBlock;};
 
-    auto nodeMgrHandle = nanovdb::cuda::createNodeManager(d_grid, CudaDeviceBuffer(), stream);
+    auto nodeMgrHandle = nanovdb::cuda::createNodeManager(d_grid, *mResource, stream);
     auto *d_nodeMgr = nodeMgrHandle.template deviceMgr<BuildT>();
 
     uint32_t nodeCount[3];// {leaf, lower, upper}
     cudaCheck(cudaMemcpyAsync(nodeCount, (char*)d_grid + sizeof(GridData) + 4*sizeof(uint64_t), 3*sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
     //cudaStreamSynchronize(stream);// finish all device tasks in stream
 
-    StatsT *d_stats = nullptr;
+    // One d_stats slot per node (leaves, then lower, then upper) so every node
+    // has its own slot - see processInternal. Zero elements when the mode has
+    // no average, so nothing is allocated.
+    BufT<StatsT> statsBuf(stream, this->ref(),
+                          StatsT::hasAverage() ? nodeCount[0]+nodeCount[1]+nodeCount[2] : 0, nanovdb::cuda::noInit);
+    StatsT *d_stats = statsBuf.data();
 
-    if constexpr(StatsT::hasAverage()) cudaCheck(util::cuda::mallocAsync((void**)&d_stats, nodeCount[0]*sizeof(StatsT), stream));
+    // warp per leaf (4 warps per 128-thread block); block per internal node
+    if (nodeCount[0]) processLeaf<BuildT><<<blocksPerGrid(nodeCount[0]*32), threadsPerBlock, 0, stream>>>(d_nodeMgr, d_stats);
 
-    processLeaf<BuildT><<<blocksPerGrid(nodeCount[0]), threadsPerBlock, 0, stream>>>(d_nodeMgr, d_stats);
+    if (nodeCount[1]) processInternal<BuildT, StatsT, 1><<<nodeCount[1], threadsPerBlock, 0, stream>>>(d_nodeMgr, d_stats);
 
-    processInternal<BuildT, StatsT, 1><<<blocksPerGrid(nodeCount[1]), threadsPerBlock, 0, stream>>>(d_nodeMgr, d_stats);
-
-    processInternal<BuildT, StatsT, 2><<<blocksPerGrid(nodeCount[2]), threadsPerBlock, 0, stream>>>(d_nodeMgr, d_stats);
+    if (nodeCount[2]) processInternal<BuildT, StatsT, 2><<<nodeCount[2], threadsPerBlock, 0, stream>>>(d_nodeMgr, d_stats);
 
     processRootAndGrid<BuildT><<<1, 1, 0, stream>>>(d_nodeMgr, d_stats);
 
-    if constexpr(StatsT::hasAverage()) cudaCheck(util::cuda::freeAsync(d_stats, stream));
+    statsBuf.destroy(stream);
 
 } // cuda::GridStats::update( Grid )
 
 //================================================================================================
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 void updateGridStats(NanoGrid<BuildT> *d_grid, StatsMode mode, cudaStream_t stream)
 {
     if (d_grid == nullptr && mode == StatsMode::Disable) {
         return;
     } else if (mode == StatsMode::BBox || util::is_same<bool, BuildT>::value) {
-        GridStats<BuildT, NoopStats<BuildT> > stats;
+        GridStats<BuildT, NoopStats<BuildT>, ResourceT> stats;
         stats.update(d_grid, stream);
     } else if (mode == StatsMode::MinMax) {
-        GridStats<BuildT, Extrema<BuildT> > stats;
+        GridStats<BuildT, Extrema<BuildT>, ResourceT> stats;
         stats.update(d_grid, stream);
     } else if (mode == StatsMode::All) {
-        GridStats<BuildT, Stats<BuildT> > stats;
+        GridStats<BuildT, Stats<BuildT>, ResourceT> stats;
         stats.update(d_grid, stream);
     } else {
         throw std::runtime_error("GridStats: Unsupported statistics mode.");

nanovdb/nanovdb/tools/cuda/IndexToGrid.cuh

@@ -18,6 +18,7 @@
 #define NANOVDB_TOOLS_CUDA_INDEXTOGRID_CUH_HAS_BEEN_INCLUDED
 
 #include <nanovdb/NanoVDB.h>
+#include <nanovdb/cuda/Buffer.h>
 #include <nanovdb/cuda/DeviceBuffer.h>
 #include <nanovdb/GridHandle.h>
 #include <nanovdb/util/cuda/Timer.h>
@@ -31,42 +32,49 @@ namespace tools::cuda {// ======================================================
 /// @tparam DstBuildT Build time of the destination/output Grid
 /// @tparam SrcBuildT  Build type of the source/input IndexGrid
 /// @tparam BufferT Type of the buffer used for allocation of the destination Grid
+/// @tparam ResourceT Template type of optional resource used for internal temporary memory
 /// @param d_srcGrid Device pointer to source/input IndexGrid, i.e. SrcBuildT={ValueIndex,ValueOnIndex}
 /// @param d_srcValues Device pointer to an array of values
 /// @param pool Memory pool used to create a buffer for the destination/output Grid
-/// @param stream optional CUDA stream (defaults to CUDA stream 0
+/// @param stream optional CUDA stream (defaults to CUDA stream 0)
 /// @note If d_srcGrid has stats (min,max,avg,std-div), the d_srcValues is also assumed
 ///       to have the same information, all of which are then copied to the destination/output grid.
 ///       An exception to this rule is if the type of d_srcValues is different from the stats type
 ///       NanoRoot<DstBuildT>::FloatType, e.g. if DstBuildT=Vec3f then NanoRoot<DstBuildT>::FloatType=float,
 ///       in which case average and standard-deviation is undefined in the output grid.
 /// @return returns handle to grid that combined IndexGrid and values
-template<typename DstBuildT, typename SrcBuildT, typename BufferT = nanovdb::cuda::DeviceBuffer>
+template<typename DstBuildT, typename SrcBuildT, typename BufferT = nanovdb::cuda::DeviceBuffer, typename ResourceT = nanovdb::cuda::DeviceResource>
 typename util::enable_if<BuildTraits<SrcBuildT>::is_index, GridHandle<BufferT>>::type
 indexToGrid(const NanoGrid<SrcBuildT> *d_srcGrid, const typename BuildToValueMap<DstBuildT>::type *d_srcValues, const BufferT &pool = BufferT(), cudaStream_t stream = 0);
 
 
-template<typename DstBuildT, typename SrcBuildT, typename BufferT = nanovdb::cuda::DeviceBuffer>
+template<typename DstBuildT, typename SrcBuildT, typename BufferT = nanovdb::cuda::DeviceBuffer, typename ResourceT = nanovdb::cuda::DeviceResource>
 typename util::enable_if<BuildTraits<SrcBuildT>::is_index, GridHandle<BufferT>>::type
 createNanoGrid(const NanoGrid<SrcBuildT> *d_srcGrid, const typename BuildToValueMap<DstBuildT>::type *d_srcValues, const BufferT &pool = BufferT(), cudaStream_t stream = 0)
 {
-    return indexToGrid<DstBuildT, SrcBuildT, BufferT>(d_srcGrid, d_srcValues, pool, stream);
+    return indexToGrid<DstBuildT, SrcBuildT, BufferT, ResourceT>(d_srcGrid, d_srcValues, pool, stream);
 }
 
 namespace {// anonymous namespace
 
-template<typename SrcBuildT>
+template<typename SrcBuildT> struct IndexToGridNodeAccessor;
+
+template<typename SrcBuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class IndexToGrid
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "IndexToGrid allocates stream-ordered scratch and requires an AsyncResource");
     using SrcGridT = NanoGrid<SrcBuildT>;
 public:
-    struct NodeAccessor;
+    using NodeAccessor = IndexToGridNodeAccessor<SrcBuildT>;
 
     /// @brief Constructor from a source IndeGrid
-    /// @param srcGrid Device pointer to IndexGrid used as the source
-    IndexToGrid(const SrcGridT *d_srcGrid, cudaStream_t stream = 0);
-
-    ~IndexToGrid() {cudaCheck(util::cuda::freeAsync(mDevNodeAcc, mStream));}
+    /// @param d_srcGrid Device pointer to IndexGrid used as the source
+    /// @param stream optional CUDA stream (defaults to CUDA stream 0)
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this instance (defaults to the per-type default resource)
+    IndexToGrid(const SrcGridT *d_srcGrid, cudaStream_t stream = 0,
+                ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>());
 
     /// @brief Toggle on and off verbose mode
     /// @param on if true verbose is turned on
@@ -90,7 +98,14 @@ private:
     util::cuda::Timer mTimer;
     std::string       mGridName;
     bool              mVerbose{false};
-    NodeAccessor      mNodeAcc, *mDevNodeAcc;
+    ResourceT*        mResource;// non-owning; all device scratch routes through this resource instance
+    template<typename T>
+    using BufT = nanovdb::cuda::Buffer<T, nanovdb::cuda::ResourceRef<ResourceT>>;
+    nanovdb::cuda::ResourceRef<ResourceT> ref() { return nanovdb::cuda::ResourceRef<ResourceT>(*mResource); }
+    NodeAccessor      mNodeAcc;
+    BufT<NodeAccessor> mDevNodeAccBuf;
+    NodeAccessor      *mDevNodeAcc{nullptr};// view of mDevNodeAccBuf
+    BufT<char>        mGridNameBuf;// owns the device copy of mGridName
 
     template<typename DstBuildT, typename BufferT>
     BufferT getBuffer(const BufferT &pool);
@@ -99,8 +114,9 @@ private:
 //================================================================================================
 
 template<typename SrcBuildT>
-struct IndexToGrid<SrcBuildT>::NodeAccessor
+struct IndexToGridNodeAccessor
 {
+    using SrcGridT = NanoGrid<SrcBuildT>;
     uint64_t grid, tree, root, node[3], meta, blind, size;// byte offsets, node: 0=leaf,1=lower, 2=upper
     const SrcGridT *d_srcGrid;// device point to source IndexGrid
     void *d_dstPtr;// device pointer to buffer with destination Grid
@@ -125,12 +141,12 @@ struct IndexToGrid<SrcBuildT>::NodeAccessor
     __device__ typename NanoNode<DstBuildT, LEVEL>::type& dstNode(int i) const {
         return *(util::PtrAdd<typename NanoNode<DstBuildT,LEVEL>::type>(d_dstPtr, node[LEVEL])+i);
     }
-};// IndexToGrid<SrcBuildT>::NodeAccessor
+};// IndexToGridNodeAccessor<SrcBuildT>
 
 //================================================================================================
 
 template<typename SrcBuildT, typename DstBuildT>
-__global__ void processGridTreeRootKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor *nodeAcc,
+__global__ void processGridTreeRootKernel(IndexToGridNodeAccessor<SrcBuildT> *nodeAcc,
                                           const typename BuildToValueMap<DstBuildT>::type *srcValues)
 {
     using SrcValueT = typename BuildToValueMap<DstBuildT>::type;
@@ -147,6 +163,8 @@ __global__ void processGridTreeRootKernel(typename IndexToGrid<SrcBuildT>::NodeA
     *dstGrid.data() = *srcGrid.data();
     dstGrid.mGridType = toGridType<DstBuildT>();
     dstGrid.mData1 = 0u;
+    dstGrid.mGridIndex = 0u; // Possibly overwriting input; returned grid has batch size 1
+    dstGrid.mGridCount = 1u;
     // we will recompute GridData::mChecksum later
 
     // process Tree
@@ -173,7 +191,7 @@ __global__ void processGridTreeRootKernel(typename IndexToGrid<SrcBuildT>::NodeA
 //================================================================================================
 
 template<typename SrcBuildT, typename DstBuildT>
-__global__ void processRootTilesKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor *nodeAcc,
+__global__ void processRootTilesKernel(IndexToGridNodeAccessor<SrcBuildT> *nodeAcc,
                                        const typename BuildToValueMap<DstBuildT>::type *srcValues)
 {
     const auto tileID = blockIdx.x, tileCount = nodeAcc->nodeCount[3];// note: tileID != childID!
@@ -205,7 +223,7 @@ __global__ void processRootTilesKernel(typename IndexToGrid<SrcBuildT>::NodeAcce
 //================================================================================================
 
 template<typename SrcBuildT, typename DstBuildT, int LEVEL>
-__global__ void processNodesKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor *nodeAcc,
+__global__ void processNodesKernel(IndexToGridNodeAccessor<SrcBuildT> *nodeAcc,
                                    const typename BuildToValueMap<DstBuildT>::type *srcValues)
 {
     using SrcNodeT  = typename NanoNode<SrcBuildT, LEVEL>::type;
@@ -218,11 +236,11 @@ __global__ void processNodesKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor
     auto &srcNode = nodeAcc->template srcNode<LEVEL>(blockIdx.x);
     auto &dstNode = nodeAcc->template dstNode<DstBuildT, LEVEL>(blockIdx.x);
 
-    if (threadIdx.x == 0 && threadIdx.y == 0) {
+    const int tid = threadIdx.y*blockDim.x + threadIdx.x;
+    const int nThreads = blockDim.x*blockDim.y;
+    if (tid == 0) {
         dstNode.mBBox = srcNode.mBBox;
         dstNode.mFlags = srcNode.mFlags;
-        dstNode.mValueMask = srcNode.mValueMask;
-        dstNode.mChildMask = srcNode.mChildMask;
         auto &srcGrid = nodeAcc->srcGrid();
         if (srcGrid.hasMinMax()) {
             dstNode.mMinimum = srcValues[srcNode.mMinimum];
@@ -233,9 +251,13 @@ __global__ void processNodesKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor
             if (srcGrid.hasStdDeviation()) dstNode.mStdDevi = srcValues[srcNode.mStdDevi];
         }
     }
-    const int off = blockDim.x*blockDim.y*threadIdx.x + blockDim.x*threadIdx.y;
-    for (int threadIdx_z=0; threadIdx_z<blockDim.x; ++threadIdx_z) {
-        const int i = off + threadIdx_z;
+    // Cooperative, coalesced mask copies
+    for (int w = tid; w < srcNode.mValueMask.wordCount(); w += nThreads) {
+        dstNode.mValueMask.words()[w] = srcNode.mValueMask.words()[w];
+        dstNode.mChildMask.words()[w] = srcNode.mChildMask.words()[w];
+    }
+    // Consecutive threads process consecutive table entries (coalesced)
+    for (int i = tid; i < SrcNodeT::SIZE; i += nThreads) {
         if (srcNode.mChildMask.isOn(i)) {
             if constexpr(sizeof(SrcNodeT)==sizeof(DstNodeT) && sizeof(SrcChildT)==sizeof(DstChildT)) {
                 dstNode.mTable[i].child = srcNode.mTable[i].child;
@@ -253,7 +275,7 @@ __global__ void processNodesKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor
 //================================================================================================
 
 template<typename SrcBuildT, typename DstBuildT>
-__global__ void processLeafsKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor *nodeAcc,
+__global__ void processLeafsKernel(IndexToGridNodeAccessor<SrcBuildT> *nodeAcc,
                                      const typename BuildToValueMap<DstBuildT>::type *srcValues)
 {
     using SrcValueT = typename BuildToValueMap<DstBuildT>::type;
@@ -261,11 +283,20 @@ __global__ void processLeafsKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor
     static_assert(!BuildTraits<DstBuildT>::is_special, "Invalid destination type!");
     auto &srcLeaf = nodeAcc->template srcNode<0>(blockIdx.x);
     auto &dstLeaf = nodeAcc->template dstNode<DstBuildT,0>(blockIdx.x);
-    if (threadIdx.x == 0 && threadIdx.y == 0) {
+    const int tid = threadIdx.y*blockDim.x + threadIdx.x;
+    const int nThreads = blockDim.x*blockDim.y;
+    if (tid == 0) {
         dstLeaf.mBBoxMin = srcLeaf.mBBoxMin;
         for (int i=0; i<3; ++i) dstLeaf.mBBoxDif[i] = srcLeaf.mBBoxDif[i];
         dstLeaf.mFlags = srcLeaf.mFlags;
         dstLeaf.mValueMask = srcLeaf.mValueMask;
+        // The leaf array is excluded from the buffer zero-init in getBuffer (it
+        // is the bulk of the grid and every mValues[i] is written below), so
+        // make the only otherwise-unwritten leaf bytes deterministic here: the
+        // stats fields (absent when the source has no stats) and any alignment
+        // padding before the 32-aligned mValues array. Real stats, if present,
+        // overwrite the zeros just below. Byte-identical to a full zero-init.
+        for (uint8_t *p = (uint8_t*)&dstLeaf.mMinimum, *e = (uint8_t*)dstLeaf.mValues; p < e; ++p) *p = 0;
         ///
         auto &srcGrid = nodeAcc->srcGrid();
         if (srcGrid.hasMinMax()) {
@@ -277,19 +308,16 @@ __global__ void processLeafsKernel(typename IndexToGrid<SrcBuildT>::NodeAccessor
             if (srcGrid.hasStdDeviation()) dstLeaf.mStdDevi = srcValues[srcLeaf.getDev()];
         }
     }
-    const int off = blockDim.x*blockDim.y*threadIdx.x + blockDim.x*threadIdx.y;
-    auto *dst = dstLeaf.mValues + off;
-    for (int threadIdx_z=0; threadIdx_z<blockDim.x; ++threadIdx_z) {
-        const int i = off + threadIdx_z;
-        *dst++ = srcValues[srcLeaf.getValue(i)];
-    }
+    // Consecutive threads write consecutive values (coalesced)
+    for (int i = tid; i < NanoLeaf<DstBuildT>::SIZE; i += nThreads)
+        dstLeaf.mValues[i] = srcValues[srcLeaf.getValue(i)];
 }// processLeafsKernel
 
 //================================================================================================
 
 template <typename SrcBuildT>
 __global__ void cpyNodeCountKernel(const NanoGrid<SrcBuildT> *srcGrid,
-                                   typename IndexToGrid<SrcBuildT>::NodeAccessor *nodeAcc)
+                                   IndexToGridNodeAccessor<SrcBuildT> *nodeAcc)
 {
     assert(srcGrid->isSequential());
     nodeAcc->d_srcGrid = srcGrid;
@@ -301,12 +329,14 @@ __global__ void cpyNodeCountKernel(const NanoGrid<SrcBuildT> *srcGrid,
 
 //================================================================================================
 
-template <typename SrcBuildT>
-IndexToGrid<SrcBuildT>::IndexToGrid(const SrcGridT *d_srcGrid, cudaStream_t stream)
-    : mStream(stream), mTimer(stream)
+template <typename SrcBuildT, typename ResourceT>
+IndexToGrid<SrcBuildT, ResourceT>::IndexToGrid(const SrcGridT *d_srcGrid, cudaStream_t stream, ResourceT& resource)
+    : mStream(stream), mTimer(stream), mResource(&resource),
+      mDevNodeAccBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 1, nanovdb::cuda::noInit),
+      mGridNameBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
 {
     NANOVDB_ASSERT(d_srcGrid);
-    cudaCheck(util::cuda::mallocAsync((void**)&mDevNodeAcc, sizeof(NodeAccessor), mStream));
+    mDevNodeAcc = mDevNodeAccBuf.data();
     cpyNodeCountKernel<SrcBuildT><<<1, 1, 0, mStream>>>(d_srcGrid, mDevNodeAcc);
     cudaCheckError();
     cudaCheck(cudaMemcpyAsync(&mNodeAcc, mDevNodeAcc, sizeof(NodeAccessor), cudaMemcpyDeviceToHost, mStream));// mNodeAcc = *mDevNodeAcc
@@ -314,9 +344,9 @@ IndexToGrid<SrcBuildT>::IndexToGrid(const SrcGridT *d_srcGrid, cudaStream_t stre
 
 //================================================================================================
 
-template <typename SrcBuildT>
+template <typename SrcBuildT, typename ResourceT>
 template <typename DstBuildT, typename BufferT>
-GridHandle<BufferT> IndexToGrid<SrcBuildT>::getHandle(const typename BuildToValueMap<DstBuildT>::type *srcValues,
+GridHandle<BufferT> IndexToGrid<SrcBuildT, ResourceT>::getHandle(const typename BuildToValueMap<DstBuildT>::type *srcValues,
                                                           const BufferT &pool)
 {
     if (mVerbose) mTimer.start("Initiate buffer");
@@ -330,7 +360,7 @@ GridHandle<BufferT> IndexToGrid<SrcBuildT>::getHandle(const typename BuildToValu
     processRootTilesKernel<SrcBuildT,DstBuildT><<<mNodeAcc.nodeCount[3], 1, 0, mStream>>>(mDevNodeAcc, srcValues);
     cudaCheckError();
 
-    cudaCheck(util::cuda::freeAsync(mNodeAcc.d_gridName, mStream));
+    mGridNameBuf.destroy(mStream);
 
     if (mVerbose) mTimer.restart("Process upper internal nodes");
     processNodesKernel<SrcBuildT,DstBuildT,2><<<mNodeAcc.nodeCount[2], dim3(32,32), 0, mStream>>>(mDevNodeAcc, srcValues);
@@ -355,9 +385,9 @@ GridHandle<BufferT> IndexToGrid<SrcBuildT>::getHandle(const typename BuildToValu
 
 //================================================================================================
 
-template <typename SrcBuildT>
+template <typename SrcBuildT, typename ResourceT>
 template <typename DstBuildT, typename BufferT>
-inline BufferT IndexToGrid<SrcBuildT>::getBuffer(const BufferT &pool)
+inline BufferT IndexToGrid<SrcBuildT, ResourceT>::getBuffer(const BufferT &pool)
 {
     mNodeAcc.grid  = 0;// grid is always stored at the start of the buffer!
     mNodeAcc.tree  = NanoGrid<DstBuildT>::memUsage(); // grid ends and tree begins
@@ -373,9 +403,19 @@ inline BufferT IndexToGrid<SrcBuildT>::getBuffer(const BufferT &pool)
     auto buffer = BufferT::create(mNodeAcc.size, &pool, device, mStream);
     mNodeAcc.d_dstPtr = buffer.deviceData();
     if (mNodeAcc.d_dstPtr == nullptr) throw std::runtime_error("Failed memory allocation on the device");
+    // Zero the non-leaf region: grid, tree, root, root tiles and the internal
+    // nodes. Bytes the kernels below do not explicitly write - stats fields
+    // absent from the source and struct alignment padding - would otherwise
+    // carry recycled allocator bytes, making the output nondeterministic and
+    // leaking heap contents into written files. The leaf array [node[0], size)
+    // is the bulk of the buffer and is fully overwritten by processLeafsKernel
+    // (values + header, which zeroes its own stats/padding gap), so it is
+    // excluded here to avoid a redundant multi-GB memset.
+    cudaCheck(cudaMemsetAsync(mNodeAcc.d_dstPtr, 0, mNodeAcc.node[0], mStream));
 
     if (size_t size = mGridName.size()) {
-        cudaCheck(util::cuda::mallocAsync((void**)&mNodeAcc.d_gridName, size, mStream));
+        mGridNameBuf = BufT<char>(mStream, this->ref(), size, nanovdb::cuda::noInit);
+        mNodeAcc.d_gridName = mGridNameBuf.data();
         cudaCheck(cudaMemcpyAsync(mNodeAcc.d_gridName, mGridName.data(), size, cudaMemcpyHostToDevice, mStream));
     } else {
         mNodeAcc.d_gridName = nullptr;
@@ -386,11 +426,11 @@ inline BufferT IndexToGrid<SrcBuildT>::getBuffer(const BufferT &pool)
 
 //================================================================================================
 
-template<typename DstBuildT, typename SrcBuildT, typename BufferT>
+template<typename DstBuildT, typename SrcBuildT, typename BufferT, typename ResourceT>
 typename util::enable_if<BuildTraits<SrcBuildT>::is_index, GridHandle<BufferT>>::type
 indexToGrid(const NanoGrid<SrcBuildT> *d_srcGrid, const typename BuildToValueMap<DstBuildT>::type *d_srcValues, const BufferT &pool, cudaStream_t stream)
 {
-    IndexToGrid<SrcBuildT> converter(d_srcGrid, stream);
+    IndexToGrid<SrcBuildT, ResourceT> converter(d_srcGrid, stream);
     return converter.template getHandle<DstBuildT>(d_srcValues, pool);
 }
 

nanovdb/nanovdb/tools/cuda/MergeGrids.cuh

@@ -33,9 +33,12 @@ namespace nanovdb {
 
 namespace tools::cuda {
 
-template <typename BuildT>
+template <typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class MergeGrids
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "MergeGrids allocates stream-ordered scratch and requires an AsyncResource");
+
     using GridT  = NanoGrid<BuildT>;
     using TreeT  = NanoTree<BuildT>;
     using RootT  = NanoRoot<BuildT>;
@@ -46,15 +49,21 @@ public:
     /// @brief Constructor for an N-ary merge
     /// @param d_srcGrids list of source device grids to be merged (active-mask union)
     /// @param stream optional CUDA stream (defaults to CUDA stream 0)
-    MergeGrids(const std::vector<const GridT*>& d_srcGrids, cudaStream_t stream = 0)
-        : mBuilder(stream), mStream(stream), mTimer(stream), mDeviceSrcGrids(d_srcGrids) {}
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this operator (defaults to the per-type default resource)
+    MergeGrids(const std::vector<const GridT*>& d_srcGrids, cudaStream_t stream = 0,
+               ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : mBuilder(stream, resource), mStream(stream), mTimer(stream), mDeviceSrcGrids(d_srcGrids) {}
 
     /// @brief Convenience constructor for the common binary merge
     /// @param d_srcGrid1 first source device grid to be merged
     /// @param d_srcGrid2 second source device grid to be merged
     /// @param stream optional CUDA stream (defaults to CUDA stream 0)
-    MergeGrids(const GridT* d_srcGrid1, const GridT* d_srcGrid2, cudaStream_t stream = 0)
-        : MergeGrids(std::vector<const GridT*>{d_srcGrid1, d_srcGrid2}, stream) {}
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this operator (defaults to the per-type default resource)
+    MergeGrids(const GridT* d_srcGrid1, const GridT* d_srcGrid2, cudaStream_t stream = 0,
+               ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : MergeGrids(std::vector<const GridT*>{d_srcGrid1, d_srcGrid2}, stream, resource) {}
 
     /// @brief Toggle on and off verbose mode
     /// @param level Verbose level: 0=quiet, 1=timing, 2=benchmarking
@@ -84,20 +93,20 @@ private:
     static constexpr unsigned int mNumThreads = 128;// for kernels spawned via lambdaKernel (others may specialize)
     static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
 
-    TopologyBuilder<BuildT> mBuilder;
+    TopologyBuilder<BuildT, ResourceT> mBuilder;
     cudaStream_t            mStream{0};
     util::cuda::Timer       mTimer;
     int                     mVerbose{0};
     std::vector<const GridT*> mDeviceSrcGrids;
     std::vector<TreeData>     mSrcTreeData;
-};// tools::cuda::MergeGrids<BuildT>
+};// tools::cuda::MergeGrids<BuildT, ResourceT>
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 template<typename BufferT>
 GridHandle<BufferT>
-MergeGrids<BuildT>::getHandle(const BufferT &pool)
+MergeGrids<BuildT, ResourceT>::getHandle(const BufferT &pool)
 {
     if (mDeviceSrcGrids.empty())
         throw std::runtime_error("MergeGrids: no input grids");
@@ -163,12 +172,12 @@ MergeGrids<BuildT>::getHandle(const BufferT &pool)
     cudaStreamSynchronize(mStream);
 
     return GridHandle<BufferT>(std::move(buffer));
-}// MergeGrids<BuildT>::getHandle
+}// MergeGrids<BuildT, ResourceT>::getHandle
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void MergeGrids<BuildT>::mergeRoot()
+template<typename BuildT, typename ResourceT>
+void MergeGrids<BuildT, ResourceT>::mergeRoot()
 {
     // Creates a new merged tree root with the merged tiles of the two input root topologies
 
@@ -218,12 +227,12 @@ void MergeGrids<BuildT>::mergeRoot()
     for (const auto& [key, tile] : mergedTiles)
         *mergedRootPtr->tile(t++) = tile;
     mBuilder.mProcessedRoot.deviceUpload(device, mStream, false);
-}// MergeGrids<BuildT>::mergeRoot
+}// MergeGrids<BuildT, ResourceT>::mergeRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void MergeGrids<BuildT>::mergeInternalNodes()
+template<typename BuildT, typename ResourceT>
+void MergeGrids<BuildT, ResourceT>::mergeInternalNodes()
 {
     // Merges the masks of upper and lower nodes from both input topologies into the
     // densified, pre-allocated mask arrays of the merged result
@@ -236,12 +245,12 @@ void MergeGrids<BuildT>::mergeInternalNodes()
             <<<mSrcTreeData[i].mNodeCount[1], Op::MaxThreadsPerBlock, 0, mStream>>>
             (mDeviceSrcGrids[i], mBuilder.deviceProcessedRoot(), mBuilder.deviceUpperMasks(), mBuilder.deviceLowerMasks());
     }
-}// MergeGrids<BuildT>::mergeInternalNodes
+}// MergeGrids<BuildT, ResourceT>::mergeInternalNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template <typename BuildT>
-void MergeGrids<BuildT>::processGridTreeRoot()
+template <typename BuildT, typename ResourceT>
+void MergeGrids<BuildT, ResourceT>::processGridTreeRoot()
 {
     // Copy GridData from the first source grid
     // TODO: Check for instances where extra processing is needed
@@ -249,12 +258,12 @@ void MergeGrids<BuildT>::processGridTreeRoot()
     cudaCheck(cudaMemcpyAsync(&mBuilder.data()->getGrid(), mDeviceSrcGrids.front()->data(), GridT::memUsage(), cudaMemcpyDeviceToDevice, mStream));
     util::cuda::lambdaKernel<<<1, 1, 0, mStream>>>(1, topology::detail::BuildGridTreeRootFunctor<BuildT>(), mBuilder.deviceData());
     cudaCheckError();
-}// MergeGrids<BuildT>::processGridTreeRoot
+}// MergeGrids<BuildT, ResourceT>::processGridTreeRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void MergeGrids<BuildT>::mergeLeafNodes()
+template<typename BuildT, typename ResourceT>
+void MergeGrids<BuildT, ResourceT>::mergeLeafNodes()
 {
     using Op = util::morphology::cuda::MergeLeafNodesFunctor<BuildT>;
     // Each input ORs its leaf active masks into the merged leaf topology.
@@ -267,7 +276,7 @@ void MergeGrids<BuildT>::mergeLeafNodes()
 
     // Update leaf offsets and prefix sums
     mBuilder.processLeafOffsets(mStream);
-}// MergeGrids<BuildT>::mergeLeafNodes
+}// MergeGrids<BuildT, ResourceT>::mergeLeafNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 

nanovdb/nanovdb/tools/cuda/MeshToGrid.cuh

@@ -17,6 +17,7 @@
 #define NVIDIA_TOOLS_CUDA_MESHTOGRID_CUH_HAS_BEEN_INCLUDED
 
 #include <cub/cub.cuh>
+#include <algorithm>
 
 #include <nanovdb/NanoVDB.h>
 #include <nanovdb/GridHandle.h>
@@ -48,9 +49,20 @@ struct Triangle {
     __hostdev__       nanovdb::Vec3f& operator[](int i)       { return v[i]; }
 };
 
-template <typename BuildT>
+/// @brief Pairing of a leaf-node origin with a triangle id. Independent of the
+///        resource the converter allocates from, so it lives outside MeshToGrid
+///        and stays one type across every ResourceT instantiation.
+struct alignas(16) MeshToGridBoxTrianglePair { // sizeof = 16B
+    nanovdb::Coord origin; // 12B
+    uint32_t triangleID;   // 4B
+};
+
+template <typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class MeshToGrid
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "MeshToGrid allocates stream-ordered scratch and requires an AsyncResource");
+
     using PointT = nanovdb::Vec3f;
     using TriangleIndexT = nanovdb::Vec3i;
     using TriangleT = Triangle;
@@ -62,10 +74,7 @@ class MeshToGrid
     using LeafT  = NanoLeaf<BuildT>;
 
 public:
-    struct alignas(16) BoxTrianglePair { // sizeof(BoxTrianglePair) = 16B
-        nanovdb::Coord origin; // 12B
-        uint32_t triangleID;   // 4B
-    };
+    using BoxTrianglePair = MeshToGridBoxTrianglePair;
 
     /// @brief Constructor
     /// @param devicePoints Vertex list for input triangle surface
@@ -79,10 +88,11 @@ public:
         const nanovdb::Vec3i *deviceTriangles,
         const uint32_t triangleCount,
         const nanovdb::Map map = nanovdb::Map(),
-        cudaStream_t stream = 0
+        cudaStream_t stream = 0,
+        ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>()
     )
-        : mStream(stream), mTimer(stream), mBuilder(stream), mDevicePoints(devicePoints), mPointCount(pointCount),
-         mDeviceTriangles(deviceTriangles), mTriangleCount(triangleCount), mMap(map)
+        : mStream(stream), mTimer(stream), mBuilder(stream, resource), mDevicePoints(devicePoints), mPointCount(pointCount),
+         mDeviceTriangles(deviceTriangles), mTriangleCount(triangleCount), mMap(map), mTempDevicePool(resource)
     {}
 
     /// @brief Toggle on and off verbose mode
@@ -155,7 +165,7 @@ private:
     static constexpr unsigned int mNumThreads = 128;// for kernels spawned via lambdaKernel (others may specialize)
     static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
 
-    TopologyBuilder<BuildT>      mBuilder;
+    TopologyBuilder<BuildT, ResourceT> mBuilder;
     cudaStream_t                 mStream{0};
     std::string                  mGridName;
     util::cuda::Timer            mTimer;
@@ -178,8 +188,8 @@ private:
     auto deviceBoxTrianglePairs()  { return static_cast<BoxTrianglePair*>(mBoxTrianglePairsBuffer.deviceData()); }
     auto deviceUniqueRootOrigins() const { return static_cast<nanovdb::Coord*>(mUniqueRootOriginsBuffer.deviceData()); }
 
-    nanovdb::cuda::TempDevicePool mTempDevicePool;
-}; // tools::cuda::MeshToGrid<BuildT>
+    nanovdb::cuda::TempPool<ResourceT> mTempDevicePool;
+}; // tools::cuda::MeshToGrid<BuildT, ResourceT>
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -200,10 +210,9 @@ private:
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 template<typename BufferT>
-GridHandle<BufferT>
-MeshToGrid<BuildT>::getHandle(const BufferT &buffer)
+GridHandle<BufferT> MeshToGrid<BuildT, ResourceT>::getHandle(const BufferT &buffer)
 {
     cudaStreamSynchronize(mStream);
 
@@ -310,7 +319,7 @@ MeshToGrid<BuildT>::getHandle(const BufferT &buffer)
     }
     if (mVerbose==1) mTimer.stop();
     return handle;
-} // MeshToGrid<BuildT>::getHandle
+} // MeshToGrid<BuildT, ResourceT>::getHandle
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -335,8 +344,8 @@ struct TransformTrianglesFunctor
 
 } // namespace topology::detail
 
-template<typename BuildT>
-void MeshToGrid<BuildT>::transformTriangles()
+template<typename BuildT, typename ResourceT>
+void MeshToGrid<BuildT, ResourceT>::transformTriangles()
 {
     if (mTriangleCount == 0) return;
 
@@ -355,7 +364,7 @@ void MeshToGrid<BuildT>::transformTriangles()
 
     cudaCheckError();
 
-} // MeshToGrid<BuildT>::transformTriangles
+} // MeshToGrid<BuildT, ResourceT>::transformTriangles
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -416,7 +425,7 @@ struct CountRootBoxesFunctor
 template <typename BuildT>
 struct ScatterRootTrianglePairsFunctor
 {
-    using PairT = typename MeshToGrid<BuildT>::BoxTrianglePair;
+    using PairT = MeshToGridBoxTrianglePair;
 
     const Triangle* dXformedTriangles;
     const uint64_t* dOffsets;
@@ -471,8 +480,8 @@ struct ScatterRootTrianglePairsFunctor
 
 } // namespace topology::detail
 
-template<typename BuildT>
-void MeshToGrid<BuildT>::processRootTrianglePairs()
+template<typename BuildT, typename ResourceT>
+void MeshToGrid<BuildT, ResourceT>::processRootTrianglePairs()
 {
     if (mTriangleCount == 0) { mBoxTrianglePairCount = 0; return; }
 
@@ -512,7 +521,7 @@ void MeshToGrid<BuildT>::processRootTrianglePairs()
     // Pass 3: Re-enumerate intersections of (padded) root boxes and triangles, and scatter to allocated list
 
     mBoxTrianglePairsBuffer = nanovdb::cuda::DeviceBuffer::create(
-        mBoxTrianglePairCount * sizeof(typename MeshToGrid<BuildT>::BoxTrianglePair), nullptr, device, mStream);
+        mBoxTrianglePairCount * sizeof(MeshToGridBoxTrianglePair), nullptr, device, mStream);
     if (mBoxTrianglePairsBuffer.deviceData() == nullptr) throw std::runtime_error("Failed to allocate pairs buffer");
 
     util::cuda::lambdaKernel<<<numBlocks(mTriangleCount), mNumThreads, 0, mStream>>>(
@@ -525,7 +534,7 @@ void MeshToGrid<BuildT>::processRootTrianglePairs()
         }
     );
 
-} // MeshToGrid<BuildT>::processRootTrianglePairs
+} // MeshToGrid<BuildT, ResourceT>::processRootTrianglePairs
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -545,7 +554,7 @@ namespace topology::detail {
 template <typename BuildT>
 struct ScatterChildPairsFunctor
 {
-    using PairT = typename MeshToGrid<BuildT>::BoxTrianglePair;
+    using PairT = MeshToGridBoxTrianglePair;
 
     const PairT*            dParents;
     const nanovdb::Mask<3>* dMasks;
@@ -657,7 +666,7 @@ __device__ inline bool testTriangleAABB(
 
 template <typename BuildT, bool OnlyUseAABB>
 __global__ void evaluateAndCountSubBoxesKernel(
-    const typename MeshToGrid<BuildT>::BoxTrianglePair* dParents,
+    const MeshToGridBoxTrianglePair* dParents,
     const Triangle* dXformedTriangles,
     nanovdb::Mask<3>* dMasks,
     uint64_t* dCounts,
@@ -746,7 +755,7 @@ __device__ inline nanovdb::Coord keyToCoord(uint64_t key)
 template <typename BuildT>
 struct EncodeRootOriginsFunctor
 {
-    const typename MeshToGrid<BuildT>::BoxTrianglePair* dPairs;
+    const MeshToGridBoxTrianglePair* dPairs;
     uint64_t*                                           dKeys;
 
     __device__ void operator()(size_t i) const { dKeys[i] = coordToKey(dPairs[i].origin); }
@@ -763,8 +772,8 @@ struct DecodeRootOriginsFunctor
 
 } // namespace topology::detail
 
-template<typename BuildT>
-void MeshToGrid<BuildT>::enumerateRootTiles()
+template<typename BuildT, typename ResourceT>
+void MeshToGrid<BuildT, ResourceT>::enumerateRootTiles()
 {
     if (mBoxTrianglePairCount == 0) return;
 
@@ -816,12 +825,12 @@ void MeshToGrid<BuildT>::enumerateRootTiles()
     );
     cudaCheckError();
 
-} // MeshToGrid<BuildT>::enumerateRootTiles
+} // MeshToGrid<BuildT, ResourceT>::enumerateRootTiles
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void MeshToGrid<BuildT>::buildRasterizedRoot()
+template<typename BuildT, typename ResourceT>
+void MeshToGrid<BuildT, ResourceT>::buildRasterizedRoot()
 {
     int device = 0;
     cudaGetDevice(&device);
@@ -850,12 +859,12 @@ void MeshToGrid<BuildT>::buildRasterizedRoot()
         mBuilder.mProcessedRoot.deviceUpload(device, mStream, false);
         mUniqueRootOriginsBuffer.clear(mStream);
     }
-} // MeshToGrid<BuildT>::buildRasterizedRoot
+} // MeshToGrid<BuildT, ResourceT>::buildRasterizedRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void MeshToGrid<BuildT>::rasterizeInternalNodes()
+template<typename BuildT, typename ResourceT>
+void MeshToGrid<BuildT, ResourceT>::rasterizeInternalNodes()
 {
     if (mBoxTrianglePairCount == 0) return;
 
@@ -870,12 +879,12 @@ void MeshToGrid<BuildT>::rasterizeInternalNodes()
     );
     cudaCheckError();
 
-} // MeshToGrid<BuildT>::rasterizeInternalNodes
+} // MeshToGrid<BuildT, ResourceT>::rasterizeInternalNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void MeshToGrid<BuildT>::processGridTreeRoot()
+template<typename BuildT, typename ResourceT>
+void MeshToGrid<BuildT, ResourceT>::processGridTreeRoot()
 {
     // Initialize grid/tree/root metadata from scratch using the provided map.
     // InitGridTreeRootFunctor sets all GridData fields explicitly (magic, version,
@@ -884,20 +893,23 @@ void MeshToGrid<BuildT>::processGridTreeRoot()
         topology::detail::InitGridTreeRootFunctor<BuildT>{mMap}, mBuilder.deviceData());
     cudaCheckError();
 
-    // Copy grid name into the output grid's name field
+    // Copy grid name into the output grid's name field. Zero the field first
+    // and copy only the actual string.
     char *dst = mBuilder.data()->getGrid().mGridName;
+    cudaCheck(cudaMemsetAsync(dst, 0, GridData::MaxNameSize, mStream));
     if (!mGridName.empty()) {
-        cudaCheck(cudaMemcpyAsync(dst, mGridName.data(), GridData::MaxNameSize, cudaMemcpyHostToDevice, mStream));
-    } else {
-        cudaCheck(cudaMemsetAsync(dst, 0, GridData::MaxNameSize, mStream));
+        // Copy at most MaxNameSize-1 bytes so the memset's trailing '\0' always
+        // survives; a name >= MaxNameSize is truncated, never left unterminated.
+        const size_t nameSize = std::min<size_t>(mGridName.size(), GridData::MaxNameSize - 1);
+        cudaCheck(cudaMemcpyAsync(dst, mGridName.c_str(), nameSize, cudaMemcpyHostToDevice, mStream));
     }
 
-} // MeshToGrid<BuildT>::processGridTreeRoot
+} // MeshToGrid<BuildT, ResourceT>::processGridTreeRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void MeshToGrid<BuildT>::rasterizeLeafNodes()
+template<typename BuildT, typename ResourceT>
+void MeshToGrid<BuildT, ResourceT>::rasterizeLeafNodes()
 {
     if (mBoxTrianglePairCount == 0) return;
 
@@ -908,12 +920,12 @@ void MeshToGrid<BuildT>::rasterizeLeafNodes()
                       &mBuilder.data()->getGrid(), mBandWidth * mBandWidth });
     cudaCheckError();
 
-} // MeshToGrid<BuildT>::rasterizeLeafNodes
+} // MeshToGrid<BuildT, ResourceT>::rasterizeLeafNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void MeshToGrid<BuildT>::processLeafTrianglePairs()
+template<typename BuildT, typename ResourceT>
+void MeshToGrid<BuildT, ResourceT>::processLeafTrianglePairs()
 {
     if (mBoxTrianglePairCount == 0) return;
 
@@ -999,7 +1011,7 @@ void MeshToGrid<BuildT>::processLeafTrianglePairs()
         scale /= 8;
     }
 
-} // MeshToGrid<BuildT>::processLeafTrianglePairs
+} // MeshToGrid<BuildT, ResourceT>::processLeafTrianglePairs
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -1038,10 +1050,10 @@ struct FinalizeSidecarFunctor
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 template<typename GridBufferT, typename SidecarBufferT>
 std::pair<GridHandle<GridBufferT>, SidecarBufferT>
-MeshToGrid<BuildT>::getHandleAndUDF(const GridBufferT& buffer, const SidecarBufferT&)
+MeshToGrid<BuildT, ResourceT>::getHandleAndUDF(const GridBufferT& buffer, const SidecarBufferT&)
 {
     cudaStreamSynchronize(mStream);
 
@@ -1172,7 +1184,14 @@ MeshToGrid<BuildT>::getHandleAndUDF(const GridBufferT& buffer, const SidecarBuff
     cudaStreamSynchronize(mStream);
 
     return { std::move(handle), std::move(sidecarBuffer) };
-} // MeshToGrid<BuildT>::getHandleAndUDF
+} // MeshToGrid<BuildT, ResourceT>::getHandleAndUDF
+
+//-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+// Undefine utility macro for cub functions
+#ifdef CALL_CUBS
+#undef CALL_CUBS
+#endif
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 

nanovdb/nanovdb/tools/cuda/PointsToGrid.cuh

@@ -22,6 +22,7 @@
 #include <vector>
 #include <tuple>
 #include <cinttypes>
+#include <algorithm>
 
 #include <nanovdb/NanoVDB.h>
 #include <nanovdb/cuda/DeviceBuffer.h>
@@ -294,6 +295,8 @@ struct PointsToGridData {
 template<typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class PointsToGrid
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "PointsToGrid allocates stream-ordered scratch and requires an AsyncResource");
 public:
     /// @brief Map constructor, which other constructors might call
     /// @param map Map to be used for the output device grid
@@ -303,10 +306,20 @@ public:
         , mResource(&resource)
         , mTimer(stream)
         , mPointType(util::is_same<BuildT,Point>::value ? PointType::Default : PointType::Disable)
+        , mDeviceDataBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 1, nanovdb::cuda::noInit)
+        , mKeysBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
+        , mTileKeysBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
+        , mLeafKeysBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
+        , mLowerKeysBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
+        , mIndxBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
+        , mPointsPerVoxelBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
+        , mPointsPerVoxelPrefixBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
+        , mPointsPerLeafBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
+        , mPointsPerLeafPrefixBuf(stream, nanovdb::cuda::ResourceRef<ResourceT>(resource), 0, nanovdb::cuda::noInit)
         , mTempDevicePool(resource)
     {
         mData.map = map;
-        mDeviceData = static_cast<PointsToGridData<BuildT>*>(mResource->allocate_async(sizeof(PointsToGridData<BuildT>), ResourceT::DEFAULT_ALIGNMENT, mStream));
+        mDeviceData = mDeviceDataBuf.data();
     }
 
     /// @brief Default constructor that calls the Map constructor defined above
@@ -327,7 +340,6 @@ public:
         mMaxIterations = maxIterations;
     }
 
-    ~PointsToGrid(){ mResource->deallocate_async(mDeviceData, sizeof(PointsToGridData<BuildT>), ResourceT::DEFAULT_ALIGNMENT, mStream); }
 
     /// @brief Toggle on and off verbose mode
     /// @param level Verbose level: 0=quiet, 1=timing, 2=benchmarking
@@ -373,6 +385,13 @@ public:
     template<typename PtrT>
     void processPoints(const PtrT points, size_t pointCount);
 
+    // Only instantiated when BuildT == Point, from the corresponding branch of
+    // processPoints. Holds the encode kernels because nvcc rejects an extended
+    // lambda defined directly inside the block of an if constexpr statement on
+    // some host compilers.
+    template<typename PtrT>
+    void encodePoints(const PtrT points, size_t pointCount);
+
     void processBBox();
 
     // the following methods are only defined when BuildT == Point
@@ -385,13 +404,26 @@ private:
     static constexpr unsigned int mNumThreads = 128;// seems faster than the old value of 256!
     static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
 
+    template<typename T>
+    using BufT = nanovdb::cuda::Buffer<T, nanovdb::cuda::ResourceRef<ResourceT>>;
+    nanovdb::cuda::ResourceRef<ResourceT> ref() { return nanovdb::cuda::ResourceRef<ResourceT>(*mResource); }
+
     cudaStream_t             mStream{0};
     ResourceT*               mResource;// non-owning; all device allocations (mDeviceData + scratch) route through this resource instance
     util::cuda::Timer        mTimer;
     PointType                mPointType;
     std::string              mGridName;
     int                      mVerbose{0};
-    PointsToGridData<BuildT> mData, *mDeviceData;
+    PointsToGridData<BuildT> mData, *mDeviceData;// mDeviceData views mDeviceDataBuf
+    // Owners of the device arrays that mData's raw pointers view. Raw views are
+    // refreshed immediately after every owner event (assign, swap, destroy), so
+    // the views -- including the device-visible fields inside mData -- are
+    // never stale. Members rather than locals because the pipeline frees them
+    // across member functions (countNodes allocates; processUpperNodes,
+    // processLeafNodes, processPoints and processBBox release).
+    BufT<PointsToGridData<BuildT>> mDeviceDataBuf;
+    BufT<uint64_t>           mKeysBuf, mTileKeysBuf, mLeafKeysBuf, mLowerKeysBuf;
+    BufT<uint32_t>           mIndxBuf, mPointsPerVoxelBuf, mPointsPerVoxelPrefixBuf, mPointsPerLeafBuf, mPointsPerLeafPrefixBuf;
     uint32_t                 mMaxPointsPerVoxel{0u}, mMaxPointsPerLeaf{0u};
     int                      mTolerance{1}, mMaxIterations{1};
     CheckMode                mChecksum{CheckMode::Disable};
@@ -603,126 +635,165 @@ void PointsToGrid<BuildT, ResourceT>::countNodes(const PtrT points, size_t point
         bool operator<(const Foo &rhs) const {return density < rhs.density || (density == rhs.density && dx < rhs.dx);}
     } min{0.0, 1}, max{0.0, 0};// min: as dx -> 0 density -> 1 point per voxel, max: density is 0 i.e. undefined
 
-jump:// this marks the beginning of the actual algorithm
+    // Declared ahead of the search loop below: d_keys and d_node_count carry
+    // results past it, and the event is recorded and re-used across iterations
+    // (previously it was re-created per iteration, leaking the prior handle).
+    BufT<uint64_t> keysScratch(mStream, this->ref(), 0, nanovdb::cuda::noInit);
+    BufT<uint32_t> indxScratch(mStream, this->ref(), 0, nanovdb::cuda::noInit);
+    BufT<uint32_t> nodeCountScratch(mStream, this->ref(), 0, nanovdb::cuda::noInit);
+    uint64_t *d_keys = nullptr;
+    uint32_t *d_indx = nullptr, *d_points_per_tile = nullptr, *d_node_count = nullptr;
+    // Owns the copy event so the too-many-points-per-leaf throw below cannot
+    // leak the handle.
+    struct EventGuard {
+        cudaEvent_t event;
+        EventGuard() { cudaCheck(cudaEventCreate(&event)); }
+        ~EventGuard() { cudaCheck(cudaEventDestroy(event)); }
+        EventGuard(const EventGuard&) = delete;
+        EventGuard& operator=(const EventGuard&) = delete;
+    } eventGuard;
+    cudaEvent_t copyEvent = eventGuard.event;
 
-    mData.d_keys = static_cast<uint64_t*>(mResource->allocate_async(pointCount*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-    mData.d_indx = static_cast<uint32_t*>(mResource->allocate_async(pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));// uint32_t can index 4.29 billion Coords, corresponding to 48 GB
-    cudaCheck(cudaMemcpyAsync(mDeviceData, &mData, sizeof(PointsToGridData<BuildT>), cudaMemcpyHostToDevice, mStream));// copy mData from CPU -> GPU
+    // Bisection search for the voxel size dx that yields the target point
+    // density: each iteration builds tile and voxel keys at the current dx,
+    // then either converges or frees this iteration's buffers and retries.
+    while (true) {
 
-    if (mVerbose==2) mTimer.start("\nAllocating arrays for keys and indices");
-    auto *d_keys = static_cast<uint64_t*>(mResource->allocate_async(pointCount*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-    auto *d_indx = static_cast<uint32_t*>(mResource->allocate_async(pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
+        mKeysBuf = BufT<uint64_t>(mStream, this->ref(), pointCount, nanovdb::cuda::noInit);
+        mIndxBuf = BufT<uint32_t>(mStream, this->ref(), pointCount, nanovdb::cuda::noInit);// uint32_t can index 4.29 billion Coords, corresponding to 48 GB
+        mData.d_keys = mKeysBuf.data();
+        mData.d_indx = mIndxBuf.data();
+        cudaCheck(cudaMemcpyAsync(mDeviceData, &mData, sizeof(PointsToGridData<BuildT>), cudaMemcpyHostToDevice, mStream));// copy mData from CPU -> GPU
 
-    if (mVerbose==2) mTimer.restart("Generate tile keys");
-    util::cuda::lambdaKernel<<<numBlocks(pointCount), mNumThreads, 0, mStream>>>(pointCount, TileKeyFunctor<BuildT, PtrT>(), mDeviceData, points, d_keys, d_indx);
-    cudaCheckError();
-    if (mVerbose==2) mTimer.restart("DeviceRadixSort of "+std::to_string(pointCount)+" tile keys");
-    CALL_CUBS(DeviceRadixSort::SortPairs, d_keys, mData.d_keys, d_indx, mData.d_indx, pointCount, 0, 63);// 21 bits per coord
-    std::swap(d_indx, mData.d_indx);// sorted indices are now in d_indx
+        if (mVerbose==2) mTimer.start("\nAllocating arrays for keys and indices");
+        keysScratch = BufT<uint64_t>(mStream, this->ref(), pointCount, nanovdb::cuda::noInit);
+        indxScratch = BufT<uint32_t>(mStream, this->ref(), pointCount, nanovdb::cuda::noInit);
+        d_keys = keysScratch.data();
+        d_indx = indxScratch.data();
 
-    if (mVerbose==2) mTimer.restart("Allocate runs");
-    auto *d_points_per_tile = static_cast<uint32_t*>(mResource->allocate_async(pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-    uint32_t *d_node_count  = static_cast<uint32_t*>(mResource->allocate_async(3*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-
-    if (mVerbose==2) mTimer.restart("DeviceRunLengthEncode tile keys");
-    CALL_CUBS(DeviceRunLengthEncode::Encode, mData.d_keys, d_keys, d_points_per_tile, d_node_count+2, pointCount);
-    cudaCheck(cudaMemcpyAsync(mData.nodeCount+2, d_node_count+2, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
-    cudaCheck(cudaStreamSynchronize(mStream));
-    mData.d_tile_keys = static_cast<uint64_t*>(mResource->allocate_async(mData.nodeCount[2]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-    cudaCheck(cudaMemcpyAsync(mData.d_tile_keys, d_keys, mData.nodeCount[2]*sizeof(uint64_t), cudaMemcpyDeviceToDevice, mStream));
-
-    static constexpr uint32_t SEGMENTED_SORT_TILE_THRESHOLD = 32;
-    if (mData.nodeCount[2] >= SEGMENTED_SORT_TILE_THRESHOLD) {
-        // Bulk segmented sort: one kernel launch + one segmented radix sort (faster for many tiles)
-        if (mVerbose==2) mTimer.restart("Segmented radix sort of " + std::to_string(pointCount) + " voxel keys in " + std::to_string(mData.nodeCount[2]) + " tiles");
-        auto *d_tile_offsets = static_cast<uint32_t*>(mResource->allocate_async((mData.nodeCount[2]+1)*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-        cudaCheck(cudaMemsetAsync(d_tile_offsets, 0, sizeof(uint32_t), mStream));
-        CALL_CUBS(DeviceScan::InclusiveSum, d_points_per_tile, d_tile_offsets + 1, mData.nodeCount[2]);
-        mResource->deallocate_async(d_points_per_tile, pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-
-        util::cuda::lambdaKernel<<<numBlocks(pointCount), mNumThreads, 0, mStream>>>(pointCount, BulkVoxelKeyFunctor<BuildT, PtrT>(), mDeviceData, points, d_tile_offsets, mData.nodeCount[2], d_keys, d_indx, uint32_t(0));
+        if (mVerbose==2) mTimer.restart("Generate tile keys");
+        util::cuda::lambdaKernel<<<numBlocks(pointCount), mNumThreads, 0, mStream>>>(pointCount, TileKeyFunctor<BuildT, PtrT>(), mDeviceData, points, d_keys, d_indx);
         cudaCheckError();
-        CALL_CUBS(DeviceSegmentedRadixSort::SortPairs, d_keys, mData.d_keys, d_indx, mData.d_indx, (int)pointCount, (int)mData.nodeCount[2], d_tile_offsets, d_tile_offsets + 1, 0, 36);
-        mResource->deallocate_async(d_tile_offsets, (mData.nodeCount[2]+1)*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-    } else {
-        // Serial per-tile sort: individual kernel + sort per tile (lower overhead for few tiles)
-        if (mVerbose==2) mTimer.restart("DeviceRadixSort of " + std::to_string(pointCount) + " voxel keys in " + std::to_string(mData.nodeCount[2]) + " tiles");
-        uint32_t *points_per_tile = new uint32_t[mData.nodeCount[2]];
-        cudaCheck(cudaMemcpyAsync(points_per_tile, d_points_per_tile, mData.nodeCount[2]*sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
-        mResource->deallocate_async(d_points_per_tile, pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-        for (uint32_t id = 0, offset = 0; id < mData.nodeCount[2]; ++id) {
-            const uint32_t count = points_per_tile[id];
-            util::cuda::offsetLambdaKernel<<<numBlocks(count), mNumThreads, 0, mStream>>>(count, offset, VoxelKeyFunctor<BuildT, PtrT>(), mDeviceData, points, id, d_keys, d_indx);
+        if (mVerbose==2) mTimer.restart("DeviceRadixSort of "+std::to_string(pointCount)+" tile keys");
+        CALL_CUBS(DeviceRadixSort::SortPairs, d_keys, mData.d_keys, d_indx, mData.d_indx, pointCount, 0, 63);// 21 bits per coord
+        mIndxBuf.swap(indxScratch);// the sorted indices' owner is now indxScratch
+        d_indx = indxScratch.data();// sorted indices
+        mData.d_indx = mIndxBuf.data();// receives the voxel-sorted indices below
+
+        if (mVerbose==2) mTimer.restart("Allocate runs");
+        BufT<uint32_t> pointsPerTileScratch(mStream, this->ref(), pointCount, nanovdb::cuda::noInit);
+        d_points_per_tile = pointsPerTileScratch.data();
+        nodeCountScratch = BufT<uint32_t>(mStream, this->ref(), 3, nanovdb::cuda::noInit);
+        d_node_count = nodeCountScratch.data();
+
+        if (mVerbose==2) mTimer.restart("DeviceRunLengthEncode tile keys");
+        CALL_CUBS(DeviceRunLengthEncode::Encode, mData.d_keys, d_keys, d_points_per_tile, d_node_count+2, pointCount);
+        cudaCheck(cudaMemcpyAsync(mData.nodeCount+2, d_node_count+2, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
+        cudaCheck(cudaStreamSynchronize(mStream));
+        mTileKeysBuf = BufT<uint64_t>(mStream, this->ref(), mData.nodeCount[2], nanovdb::cuda::noInit);
+        mData.d_tile_keys = mTileKeysBuf.data();
+        cudaCheck(cudaMemcpyAsync(mData.d_tile_keys, d_keys, mData.nodeCount[2]*sizeof(uint64_t), cudaMemcpyDeviceToDevice, mStream));
+
+        static constexpr uint32_t SEGMENTED_SORT_TILE_THRESHOLD = 32;
+        if (mData.nodeCount[2] >= SEGMENTED_SORT_TILE_THRESHOLD) {
+            // Bulk segmented sort: one kernel launch + one segmented radix sort (faster for many tiles)
+            if (mVerbose==2) mTimer.restart("Segmented radix sort of " + std::to_string(pointCount) + " voxel keys in " + std::to_string(mData.nodeCount[2]) + " tiles");
+            BufT<uint32_t> tileOffsetsScratch(mStream, this->ref(), mData.nodeCount[2]+1, nanovdb::cuda::noInit);
+            auto *d_tile_offsets = tileOffsetsScratch.data();
+            cudaCheck(cudaMemsetAsync(d_tile_offsets, 0, sizeof(uint32_t), mStream));
+            CALL_CUBS(DeviceScan::InclusiveSum, d_points_per_tile, d_tile_offsets + 1, mData.nodeCount[2]);
+            pointsPerTileScratch.destroy(); d_points_per_tile = nullptr;
+
+            util::cuda::lambdaKernel<<<numBlocks(pointCount), mNumThreads, 0, mStream>>>(pointCount, BulkVoxelKeyFunctor<BuildT, PtrT>(), mDeviceData, points, d_tile_offsets, mData.nodeCount[2], d_keys, d_indx, uint32_t(0));
             cudaCheckError();
-            CALL_CUBS(DeviceRadixSort::SortPairs, d_keys + offset, mData.d_keys + offset, d_indx + offset, mData.d_indx + offset, count, 0, 36);
-            offset += count;
-        }
-        delete [] points_per_tile;
-    }
-    mResource->deallocate_async(d_indx, pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-
-    if (mVerbose==2) mTimer.restart("Count points per voxel");
-
-    cudaEvent_t copyEvent;
-    cudaCheck(cudaEventCreate(&copyEvent));
-    mData.pointsPerVoxel    = static_cast<uint32_t*>(mResource->allocate_async(pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-    uint32_t *d_voxel_count = static_cast<uint32_t*>(mResource->allocate_async(sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-    CALL_CUBS(DeviceRunLengthEncode::Encode, mData.d_keys, d_keys, mData.pointsPerVoxel, d_voxel_count, pointCount);
-    cudaCheck(cudaMemcpyAsync(&mData.voxelCount, d_voxel_count, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
-    cudaCheck(cudaEventRecord(copyEvent, mStream));
-    mResource->deallocate_async(d_voxel_count, sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-
-    if (util::is_same<BuildT, Point>::value) {
-        if (mVerbose==2) mTimer.restart("Count max points per voxel");
-        uint32_t *d_maxPointsPerVoxel = static_cast<uint32_t*>(mResource->allocate_async(sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream)), maxPointsPerVoxel;
-        cudaCheck(cudaEventSynchronize(copyEvent));
-        CALL_CUBS(DeviceReduce::Max, mData.pointsPerVoxel, d_maxPointsPerVoxel, mData.voxelCount);
-        cudaCheck(cudaMemcpyAsync(&maxPointsPerVoxel, d_maxPointsPerVoxel, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
-        cudaCheck(cudaEventRecord(copyEvent, mStream));
-        mResource->deallocate_async(d_maxPointsPerVoxel, sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-        double dx = mData.map.getVoxelSize()[0];
-        cudaCheck(cudaEventSynchronize(copyEvent));
-        if (++iterCounter >= mMaxIterations || pointCount == 1u || math::Abs((int)maxPointsPerVoxel - (int)mMaxPointsPerVoxel) <= mTolerance) {
-            mMaxPointsPerVoxel = maxPointsPerVoxel;
+            CALL_CUBS(DeviceSegmentedRadixSort::SortPairs, d_keys, mData.d_keys, d_indx, mData.d_indx, (int)pointCount, (int)mData.nodeCount[2], d_tile_offsets, d_tile_offsets + 1, 0, 36);
         } else {
-            const Foo tmp{dx, maxPointsPerVoxel};
-            if (maxPointsPerVoxel < mMaxPointsPerVoxel) {
-                if (min < tmp) min = tmp;
-            } else if (max.density == 0 || tmp < max) {
-                max = tmp;
+            // Serial per-tile sort: individual kernel + sort per tile (lower overhead for few tiles)
+            if (mVerbose==2) mTimer.restart("DeviceRadixSort of " + std::to_string(pointCount) + " voxel keys in " + std::to_string(mData.nodeCount[2]) + " tiles");
+            uint32_t *points_per_tile = new uint32_t[mData.nodeCount[2]];
+            cudaCheck(cudaMemcpyAsync(points_per_tile, d_points_per_tile, mData.nodeCount[2]*sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
+            pointsPerTileScratch.destroy(); d_points_per_tile = nullptr;
+            for (uint32_t id = 0, offset = 0; id < mData.nodeCount[2]; ++id) {
+                const uint32_t count = points_per_tile[id];
+                util::cuda::offsetLambdaKernel<<<numBlocks(count), mNumThreads, 0, mStream>>>(count, offset, VoxelKeyFunctor<BuildT, PtrT>(), mDeviceData, points, id, d_keys, d_indx);
+                cudaCheckError();
+                CALL_CUBS(DeviceRadixSort::SortPairs, d_keys + offset, mData.d_keys + offset, d_indx + offset, mData.d_indx + offset, count, 0, 36);
+                offset += count;
             }
-            if (max.density) {
-                dx = (min.dx*(max.density - mMaxPointsPerVoxel) + max.dx*(mMaxPointsPerVoxel-min.density))/double(max.density-min.density);
-            } else if (maxPointsPerVoxel > 1u) {
-                dx *= (mMaxPointsPerVoxel-1.0)/(maxPointsPerVoxel-1.0);
-            } else {// maxPointsPerVoxel = 1 so increase dx significantly
-                dx *= 10.0;
-            }
-            if (mVerbose==2) printf("\ntarget density = %" PRIu32 ", current density = %" PRIu32 ", current dx = %f, next dx = %f\n", mMaxPointsPerVoxel, maxPointsPerVoxel, tmp.dx, dx);
-            mData.map = Map(dx);
-            mResource->deallocate_async(mData.d_keys, pointCount*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-            mResource->deallocate_async(mData.d_indx, pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-            mResource->deallocate_async(d_keys, pointCount*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-            mResource->deallocate_async(mData.d_tile_keys, mData.nodeCount[2]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-            mResource->deallocate_async(d_node_count, 3*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-            mResource->deallocate_async(mData.pointsPerVoxel, pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-            goto jump;
+            delete [] points_per_tile;
         }
-    }
+        indxScratch.destroy(); d_indx = nullptr;// tile-order copy, superseded by the voxel sort
+
+        if (mVerbose==2) mTimer.restart("Count points per voxel");
+
+        mPointsPerVoxelBuf = BufT<uint32_t>(mStream, this->ref(), pointCount, nanovdb::cuda::noInit);
+        mData.pointsPerVoxel = mPointsPerVoxelBuf.data();
+        BufT<uint32_t> voxelCountScratch(mStream, this->ref(), 1, nanovdb::cuda::noInit);
+        uint32_t *d_voxel_count = voxelCountScratch.data();
+        CALL_CUBS(DeviceRunLengthEncode::Encode, mData.d_keys, d_keys, mData.pointsPerVoxel, d_voxel_count, pointCount);
+        cudaCheck(cudaMemcpyAsync(&mData.voxelCount, d_voxel_count, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
+        cudaCheck(cudaEventRecord(copyEvent, mStream));
+        voxelCountScratch.destroy(); d_voxel_count = nullptr;
+
+        if (util::is_same<BuildT, Point>::value) {
+            if (mVerbose==2) mTimer.restart("Count max points per voxel");
+            BufT<uint32_t> maxPointsPerVoxelScratch(mStream, this->ref(), 1, nanovdb::cuda::noInit);
+            uint32_t *d_maxPointsPerVoxel = maxPointsPerVoxelScratch.data(), maxPointsPerVoxel;
+            cudaCheck(cudaEventSynchronize(copyEvent));
+            CALL_CUBS(DeviceReduce::Max, mData.pointsPerVoxel, d_maxPointsPerVoxel, mData.voxelCount);
+            cudaCheck(cudaMemcpyAsync(&maxPointsPerVoxel, d_maxPointsPerVoxel, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
+            cudaCheck(cudaEventRecord(copyEvent, mStream));
+            maxPointsPerVoxelScratch.destroy(); d_maxPointsPerVoxel = nullptr;
+            double dx = mData.map.getVoxelSize()[0];
+            cudaCheck(cudaEventSynchronize(copyEvent));
+            if (++iterCounter >= mMaxIterations || pointCount == 1u || math::Abs((int)maxPointsPerVoxel - (int)mMaxPointsPerVoxel) <= mTolerance) {
+                mMaxPointsPerVoxel = maxPointsPerVoxel;
+            } else {
+                const Foo tmp{dx, maxPointsPerVoxel};
+                if (maxPointsPerVoxel < mMaxPointsPerVoxel) {
+                    if (min < tmp) min = tmp;
+                } else if (max.density == 0 || tmp < max) {
+                    max = tmp;
+                }
+                if (max.density) {
+                    dx = (min.dx*(max.density - mMaxPointsPerVoxel) + max.dx*(mMaxPointsPerVoxel-min.density))/double(max.density-min.density);
+                } else if (maxPointsPerVoxel > 1u) {
+                    dx *= (mMaxPointsPerVoxel-1.0)/(maxPointsPerVoxel-1.0);
+                } else {// maxPointsPerVoxel = 1 so increase dx significantly
+                    dx *= 10.0;
+                }
+                if (mVerbose==2) printf("\ntarget density = %" PRIu32 ", current density = %" PRIu32 ", current dx = %f, next dx = %f\n", mMaxPointsPerVoxel, maxPointsPerVoxel, tmp.dx, dx);
+                mData.map = Map(dx);
+                // free before the next iteration reallocates, so peak device
+                // memory matches the pre-loop behavior
+                mKeysBuf.destroy();          mData.d_keys = nullptr;
+                mIndxBuf.destroy();          mData.d_indx = nullptr;
+                keysScratch.destroy();       d_keys = nullptr;
+                mTileKeysBuf.destroy();      mData.d_tile_keys = nullptr;
+                nodeCountScratch.destroy();  d_node_count = nullptr;
+                mPointsPerVoxelBuf.destroy();mData.pointsPerVoxel = nullptr;
+                continue;
+            }
+        }
+        break;
+    }// while (true)
     if (iterCounter>1 && mVerbose) std::cerr << "Used " << iterCounter << " attempts to determine dx that produces a target dpoint denisty\n\n";
 
     if (mVerbose==2) mTimer.restart("Compute prefix sum of points per voxel");
     cudaCheck(cudaEventSynchronize(copyEvent));
-    mData.pointsPerVoxelPrefix = static_cast<uint32_t*>(mResource->allocate_async(mData.voxelCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
+    mPointsPerVoxelPrefixBuf = BufT<uint32_t>(mStream, this->ref(), mData.voxelCount, nanovdb::cuda::noInit);
+    mData.pointsPerVoxelPrefix = mPointsPerVoxelPrefixBuf.data();
     CALL_CUBS(DeviceScan::ExclusiveSum, mData.pointsPerVoxel, mData.pointsPerVoxelPrefix, mData.voxelCount);
 
-    mData.pointsPerLeaf = static_cast<uint32_t*>(mResource->allocate_async(pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
+    mPointsPerLeafBuf = BufT<uint32_t>(mStream, this->ref(), pointCount, nanovdb::cuda::noInit);
+    mData.pointsPerLeaf = mPointsPerLeafBuf.data();
     CALL_CUBS(DeviceRunLengthEncode::Encode, thrust::make_transform_iterator(mData.d_keys, ShiftRight<9>()), d_keys, mData.pointsPerLeaf, d_node_count, pointCount);
     cudaCheck(cudaMemcpyAsync(mData.nodeCount, d_node_count, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
     cudaCheck(cudaEventRecord(copyEvent, mStream));
 
     if constexpr(util::is_same<BuildT, Point>::value) {
-        uint32_t *d_maxPointsPerLeaf = static_cast<uint32_t*>(mResource->allocate_async(sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
+        BufT<uint32_t> maxPointsPerLeafScratch(mStream, this->ref(), 1, nanovdb::cuda::noInit);
+        uint32_t *d_maxPointsPerLeaf = maxPointsPerLeafScratch.data();
         cudaCheck(cudaEventSynchronize(copyEvent));
         CALL_CUBS(DeviceReduce::Max, mData.pointsPerLeaf, d_maxPointsPerLeaf, mData.nodeCount[0]);
         cudaCheck(cudaMemcpyAsync(&mMaxPointsPerLeaf, d_maxPointsPerLeaf, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
@@ -730,27 +801,26 @@ jump:// this marks the beginning of the actual algorithm
         if (mMaxPointsPerLeaf > std::numeric_limits<uint16_t>::max()) {
             throw std::runtime_error("Too many points per leaf: "+std::to_string(mMaxPointsPerLeaf));
         }
-        mResource->deallocate_async(d_maxPointsPerLeaf, sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
     }
 
     cudaCheck(cudaEventSynchronize(copyEvent));
-    mData.pointsPerLeafPrefix = static_cast<uint32_t*>(mResource->allocate_async(mData.nodeCount[0]*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
+    mPointsPerLeafPrefixBuf = BufT<uint32_t>(mStream, this->ref(), mData.nodeCount[0], nanovdb::cuda::noInit);
+    mData.pointsPerLeafPrefix = mPointsPerLeafPrefixBuf.data();
     CALL_CUBS(DeviceScan::ExclusiveSum, mData.pointsPerLeaf, mData.pointsPerLeafPrefix, mData.nodeCount[0]);
 
     cudaCheck(cudaStreamSynchronize(mStream));
-    mData.d_leaf_keys = static_cast<uint64_t*>(mResource->allocate_async(mData.nodeCount[0]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
+    mLeafKeysBuf = BufT<uint64_t>(mStream, this->ref(), mData.nodeCount[0], nanovdb::cuda::noInit);
+    mData.d_leaf_keys = mLeafKeysBuf.data();
     cudaCheck(cudaMemcpyAsync(mData.d_leaf_keys, d_keys, mData.nodeCount[0]*sizeof(uint64_t), cudaMemcpyDeviceToDevice, mStream));
 
     CALL_CUBS(DeviceSelect::Unique, thrust::make_transform_iterator(mData.d_leaf_keys, ShiftRight<12>()), d_keys, d_node_count+1, mData.nodeCount[0]);// count lower nodes
     cudaCheck(cudaMemcpyAsync(mData.nodeCount+1, d_node_count+1, sizeof(uint32_t), cudaMemcpyDeviceToHost, mStream));
     cudaCheck(cudaStreamSynchronize(mStream));
-    mData.d_lower_keys = static_cast<uint64_t*>(mResource->allocate_async(mData.nodeCount[1]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
+    mLowerKeysBuf = BufT<uint64_t>(mStream, this->ref(), mData.nodeCount[1], nanovdb::cuda::noInit);
+    mData.d_lower_keys = mLowerKeysBuf.data();
     cudaCheck(cudaMemcpyAsync(mData.d_lower_keys, d_keys, mData.nodeCount[1]*sizeof(uint64_t), cudaMemcpyDeviceToDevice, mStream));
 
-    mResource->deallocate_async(d_keys, pointCount*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-    mResource->deallocate_async(d_node_count, 3*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
     if (mVerbose==2) mTimer.stop();
-    cudaCheck(cudaEventDestroy(copyEvent));
 
     //printf("Leaf count = %u, lower count = %u, upper count = %u\n", mData.nodeCount[0], mData.nodeCount[1], mData.nodeCount[2]);
 }// PointsToGrid<BuildT, ResourceT>::countNodes
@@ -792,6 +862,12 @@ inline BufferT PointsToGrid<BuildT, ResourceT>::getBuffer(const PtrT, size_t poi
 
     mData.d_bufferPtr = buffer.deviceData();
     if (mData.d_bufferPtr == nullptr) throw std::runtime_error("Failed to allocate grid buffer on the device");
+    // Zero the whole grid buffer up front. This (a) makes the dense background
+    // fills (upper/lower value tables, inactive leaf values - all zero in this
+    // builder) redundant, so those kernels are skipped, and (b) makes the
+    // output bit-deterministic: alignment padding and stats fields no longer
+    // carry recycled pool bytes (which previously leaked into written files).
+    cudaCheck(cudaMemsetAsync(mData.d_bufferPtr, 0, mData.size, mStream));
     cudaCheck(cudaMemcpyAsync(mDeviceData, &mData, sizeof(PointsToGridData<BuildT>), cudaMemcpyHostToDevice, mStream));// copy Data CPU -> GPU
     return buffer;
 }// PointsToGrid<BuildT, ResourceT>::getBuffer
@@ -938,11 +1014,18 @@ inline void PointsToGrid<BuildT, ResourceT>::processGridTreeRoot(const PtrT poin
     util::cuda::lambdaKernel<<<1, 1, 0, mStream>>>(1, BuildGridTreeRootFunctor<BuildT, PtrT>(), mDeviceData, mPointType, pointCount);// lambdaKernel
     cudaCheckError();
 
+    // Zero the name field, then copy only the actual string (if any). The
+    // previous code copied MaxNameSize bytes from the std::string buffer
+    // (whose .data() is never null, so the memset branch was dead), reading
+    // up to 255 bytes past the allocation and leaking host heap contents
+    // into the grid - nondeterministic output and a hygiene issue for files.
     char *dst = mData.getGrid().mGridName;
-    if (const char *src = mGridName.data()) {
-        cudaCheck(cudaMemcpyAsync(dst, src, GridData::MaxNameSize, cudaMemcpyHostToDevice, mStream));
-    } else {
-        cudaCheck(cudaMemsetAsync(dst, 0, GridData::MaxNameSize, mStream));
+    cudaCheck(cudaMemsetAsync(dst, 0, GridData::MaxNameSize, mStream));
+    if (!mGridName.empty()) {
+        // Copy at most MaxNameSize-1 bytes so the memset's trailing '\0' always
+        // survives; a name >= MaxNameSize is truncated, never left unterminated.
+        const size_t nameSize = std::min<size_t>(mGridName.size(), GridData::MaxNameSize - 1);
+        cudaCheck(cudaMemcpyAsync(dst, mGridName.c_str(), nameSize, cudaMemcpyHostToDevice, mStream));
     }
 }// PointsToGrid<BuildT, ResourceT>::processGridTreeRoot
 
@@ -993,11 +1076,8 @@ inline void PointsToGrid<BuildT, ResourceT>::processUpperNodes()
     util::cuda::lambdaKernel<<<numBlocks(mData.nodeCount[2]), mNumThreads, 0, mStream>>>(mData.nodeCount[2], BuildUpperNodesFunctor<BuildT>(), mDeviceData);
     cudaCheckError();
 
-    mResource->deallocate_async(mData.d_tile_keys, mData.nodeCount[2]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
+    mTileKeysBuf.destroy(); mData.d_tile_keys = nullptr;
 
-    const uint64_t valueCount = mData.nodeCount[2] << 15;
-    util::cuda::lambdaKernel<<<numBlocks(valueCount), mNumThreads, 0, mStream>>>(valueCount, SetUpperBackgroundValuesFunctor<BuildT>(), mDeviceData);
-    cudaCheckError();
 }// PointsToGrid<BuildT, ResourceT>::processUpperNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -1039,9 +1119,6 @@ inline void PointsToGrid<BuildT, ResourceT>::processLowerNodes()
     util::cuda::lambdaKernel<<<numBlocks(mData.nodeCount[1]), mNumThreads, 0, mStream>>>(mData.nodeCount[1], BuildLowerNodesFunctor<BuildT>(), mDeviceData);
     cudaCheckError();
 
-    const uint64_t valueCount = mData.nodeCount[1] << 12;
-    util::cuda::lambdaKernel<<<numBlocks(valueCount), mNumThreads, 0, mStream>>>(valueCount, SetLowerBackgroundValuesFunctor<BuildT>(), mDeviceData);
-    cudaCheckError();
 }// PointsToGrid<BuildT, ResourceT>::processLowerNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -1128,28 +1205,35 @@ inline void PointsToGrid<BuildT, ResourceT>::processLeafNodes(size_t pointCount)
     util::cuda::lambdaKernel<<<numBlocks(mData.voxelCount), mNumThreads, 0, mStream>>>(mData.voxelCount, SetLeafActiveVoxelStateAndValuesFunctor<BuildT>(), mDeviceData);
     cudaCheckError();
 
-    mResource->deallocate_async(mData.d_keys, pointCount*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-    mResource->deallocate_async(mData.pointsPerVoxel, pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-    mResource->deallocate_async(mData.pointsPerVoxelPrefix, mData.voxelCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-    mResource->deallocate_async(mData.pointsPerLeafPrefix, pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-    mResource->deallocate_async(mData.pointsPerLeaf,mData.nodeCount[0]*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
+    mKeysBuf.destroy();                mData.d_keys = nullptr;
+    mPointsPerVoxelBuf.destroy();      mData.pointsPerVoxel = nullptr;
+    mPointsPerVoxelPrefixBuf.destroy();mData.pointsPerVoxelPrefix = nullptr;
+    mPointsPerLeafPrefixBuf.destroy(); mData.pointsPerLeafPrefix = nullptr;
+    mPointsPerLeafBuf.destroy();       mData.pointsPerLeaf = nullptr;
 
-    if (mVerbose==2) mTimer.restart("set inactive voxel values");
-    const uint64_t denseVoxelCount = mData.nodeCount[0] << 9;
-    util::cuda::lambdaKernel<<<numBlocks(denseVoxelCount), mNumThreads, 0, mStream>>>(denseVoxelCount, SetLeafInactiveVoxelValuesFunctor<BuildT>(), mDeviceData);
-    cudaCheckError();
+    // Inactive voxel values are zero for every build type except Point (which
+    // copies the previous active value for rank queries); the zero cases are
+    // covered by the buffer memset in getBuffer, and for index grids this
+    // dense pass (nodeCount[0]<<9 threads) is a no-op.
+    if constexpr(util::is_same<BuildT, Point>::value) {
+        if (mVerbose==2) mTimer.restart("set inactive voxel values");
+        const uint64_t denseVoxelCount = mData.nodeCount[0] << 9;
+        util::cuda::lambdaKernel<<<numBlocks(denseVoxelCount), mNumThreads, 0, mStream>>>(denseVoxelCount, SetLeafInactiveVoxelValuesFunctor<BuildT>(), mDeviceData);
+        cudaCheckError();
+    }
 
     if constexpr(BuildTraits<BuildT>::is_onindex) {
         if (mVerbose==2) mTimer.restart("prefix-sum for index grid");
-        auto devValueIndex = static_cast<uint64_t*>(mResource->allocate_async(mData.nodeCount[0]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
-        auto devValueIndexPrefix = static_cast<uint64_t*>(mResource->allocate_async(mData.nodeCount[0]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream));
+        BufT<uint64_t> valueIndexScratch(mStream, this->ref(), mData.nodeCount[0], nanovdb::cuda::noInit);
+        BufT<uint64_t> valueIndexPrefixScratch(mStream, this->ref(), mData.nodeCount[0], nanovdb::cuda::noInit);
+        auto devValueIndex = valueIndexScratch.data();
+        auto devValueIndexPrefix = valueIndexPrefixScratch.data();
         kernels::fillValueIndexKernel<BuildT><<<numBlocks(mData.nodeCount[0]), mNumThreads, 0, mStream>>>(mData.nodeCount[0], 0, devValueIndex, mDeviceData);
         cudaCheckError();
         CALL_CUBS(DeviceScan::InclusiveSum, devValueIndex, devValueIndexPrefix, mData.nodeCount[0]);
-        mResource->deallocate_async(devValueIndex, mData.nodeCount[0]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
+        valueIndexScratch.destroy(); devValueIndex = nullptr;
         kernels::leafPrefixSumKernel<BuildT><<<numBlocks(mData.nodeCount[0]), mNumThreads, 0, mStream>>>(mData.nodeCount[0], 0, devValueIndexPrefix, mDeviceData);
         cudaCheckError();
-        mResource->deallocate_async(devValueIndexPrefix, mData.nodeCount[0]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
     }
 
     if (mVerbose==2) mTimer.stop();
@@ -1164,23 +1248,24 @@ inline void PointsToGrid<BuildT, ResourceT>::processLeafNodes(size_t pointCount)
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
+// The point-encode kernels operate on device data and are independent of the
+// resource type; only the trailing d_indx deallocation routes through ResourceT.
+// Dispatching the encode with if constexpr keeps it reachable for any ResourceT.
 template<typename BuildT, typename ResourceT>
 template<typename PtrT>
-inline void PointsToGrid<BuildT, ResourceT>::processPoints(const PtrT, size_t pointCount)
+inline void PointsToGrid<BuildT, ResourceT>::processPoints(const PtrT points, size_t pointCount)
 {
-    mResource->deallocate_async(mData.d_indx, pointCount*sizeof(uint32_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-}
+    if constexpr(util::is_same<BuildT, Point>::value) this->encodePoints(points, pointCount);
+    mIndxBuf.destroy(); mData.d_indx = nullptr;
+}// PointsToGrid<BuildT, ResourceT>::processPoints
 
-//-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-// Template specialization with BuildT = Point
-template<>
+template<typename BuildT, typename ResourceT>
 template<typename PtrT>
-inline void PointsToGrid<Point>::processPoints(const PtrT points, size_t pointCount)
+inline void PointsToGrid<BuildT, ResourceT>::encodePoints(const PtrT points, size_t pointCount)
 {
     switch (mPointType){
     case PointType::Disable:
-        throw std::runtime_error("PointsToGrid<Point>::processPoints: mPointType == PointType::Disable\n");
+        throw std::runtime_error("PointsToGrid<Point, ResourceT>::encodePoints: mPointType == PointType::Disable\n");
     case PointType::PointID:
         util::cuda::lambdaKernel<<<numBlocks(pointCount), mNumThreads, 0, mStream>>>(pointCount, [=] __device__(size_t tid, PointsToGridData<Point> *d_data) {
             d_data->template getPoint<uint32_t>(tid) = d_data->d_indx[tid];
@@ -1227,12 +1312,9 @@ inline void PointsToGrid<Point>::processPoints(const PtrT points, size_t pointCo
         }, mDeviceData); cudaCheckError();
         break;
     default:
-        printf("Internal error in PointsToGrid<Point>::processPoints\n");
+        printf("Internal error in PointsToGrid<Point, ResourceT>::encodePoints\n");
     }
-    // This is the PointsToGrid<Point> (== <Point, DeviceResource>) member specialization,
-    // so ResourceT is not a name here; mResource is a DeviceResource* and the alignment is concrete.
-    mResource->deallocate_async(mData.d_indx, pointCount*sizeof(uint32_t), nanovdb::cuda::DeviceResource::DEFAULT_ALIGNMENT, mStream);
-}// PointsToGrid<Point>::processPoints
+}// PointsToGrid<BuildT, ResourceT>::encodePoints
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -1310,7 +1392,7 @@ inline void PointsToGrid<BuildT, ResourceT>::processBBox()
 
     // update and propagate bbox from leaf -> lower/parent nodes
     util::cuda::lambdaKernel<<<numBlocks(mData.nodeCount[0]), mNumThreads, 0, mStream>>>(mData.nodeCount[0], UpdateAndPropagateLeafBBoxFunctor<BuildT>(), mDeviceData);
-    mResource->deallocate_async(mData.d_leaf_keys, mData.nodeCount[0]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
+    mLeafKeysBuf.destroy(); mData.d_leaf_keys = nullptr;
     cudaCheckError();
 
     // reset bbox in upper nodes
@@ -1319,8 +1401,8 @@ inline void PointsToGrid<BuildT, ResourceT>::processBBox()
 
     // propagate bbox from lower -> upper/parent node
     util::cuda::lambdaKernel<<<numBlocks(mData.nodeCount[1]), mNumThreads, 0, mStream>>>(mData.nodeCount[1], PropagateLowerBBoxFunctor<BuildT>(), mDeviceData);
-    mResource->deallocate_async(mData.d_lower_keys, mData.nodeCount[1]*sizeof(uint64_t), ResourceT::DEFAULT_ALIGNMENT, mStream);
-    cudaCheckError()
+    mLowerKeysBuf.destroy(); mData.d_lower_keys = nullptr;
+    cudaCheckError();
 
     // propagate bbox from upper -> root/parent node
     util::cuda::lambdaKernel<<<numBlocks(mData.nodeCount[2]), mNumThreads, 0, mStream>>>(mData.nodeCount[2], PropagateUpperBBoxFunctor<BuildT>(), mDeviceData);

nanovdb/nanovdb/tools/cuda/PruneGrid.cuh

@@ -30,9 +30,12 @@ namespace nanovdb {
 
 namespace tools::cuda {
 
-template <typename BuildT>
+template <typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class PruneGrid
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "PruneGrid allocates stream-ordered scratch and requires an AsyncResource");
+
     using GridT  = NanoGrid<BuildT>;
     using TreeT  = NanoTree<BuildT>;
     using RootT  = NanoRoot<BuildT>;
@@ -44,8 +47,11 @@ public:
     /// @param d_srcGrid source device grid to be pruned
     /// @param d_srcLeafMask sidecar array of leaf masks for voxels to retain
     /// @param stream optional CUDA stream (defaults to CUDA stream 0)
-    PruneGrid(const GridT* d_srcGrid, const Mask<3>* d_srcLeafMask, cudaStream_t stream = 0)
-        : mBuilder(stream), mStream(stream), mTimer(stream), mDeviceSrcGrid(d_srcGrid), mDeviceSrcLeafMask(d_srcLeafMask) {}
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this operator (defaults to the per-type default resource)
+    PruneGrid(const GridT* d_srcGrid, const Mask<3>* d_srcLeafMask, cudaStream_t stream = 0,
+              ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : mBuilder(stream, resource), mStream(stream), mTimer(stream), mDeviceSrcGrid(d_srcGrid), mDeviceSrcLeafMask(d_srcLeafMask) {}
 
     /// @brief Toggle on and off verbose mode
     /// @param level Verbose level: 0=quiet, 1=timing, 2=benchmarking
@@ -75,21 +81,21 @@ private:
     static constexpr unsigned int mNumThreads = 128;// for kernels spawned via lambdaKernel (others may specialize)
     static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
 
-    TopologyBuilder<BuildT> mBuilder;
+    TopologyBuilder<BuildT, ResourceT> mBuilder;
     cudaStream_t            mStream{0};
     util::cuda::Timer       mTimer;
     int                     mVerbose{0};
     const GridT             *mDeviceSrcGrid;
     const Mask<3>           *mDeviceSrcLeafMask;
     TreeData                mSrcTreeData;
-};// tools::cuda::PruneGrid<BuildT>
+};// tools::cuda::PruneGrid<BuildT, ResourceT>
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 template<typename BufferT>
 GridHandle<BufferT>
-PruneGrid<BuildT>::getHandle(const BufferT &pool)
+PruneGrid<BuildT, ResourceT>::getHandle(const BufferT &pool)
 {
     // Copy TreeData from GPU -> CPU
     cudaStreamSynchronize(mStream);
@@ -150,12 +156,12 @@ PruneGrid<BuildT>::getHandle(const BufferT &pool)
     cudaStreamSynchronize(mStream);
 
     return GridHandle<BufferT>(std::move(buffer));
-}// PruneGrid<BuildT>::getHandle
+}// PruneGrid<BuildT, ResourceT>::getHandle
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void PruneGrid<BuildT>::pruneRoot()
+template<typename BuildT, typename ResourceT>
+void PruneGrid<BuildT, ResourceT>::pruneRoot()
 {
     // This method conservatively (and trivially) prunes the root tile table.
     // For this simple approximation, it is assumed that all root tiles currently present will presist,
@@ -200,12 +206,12 @@ void PruneGrid<BuildT>::pruneRoot()
     for (const auto& [key, tile] : prunedTiles)
         *prunedRootPtr->tile(t++) = tile;
     mBuilder.mProcessedRoot.deviceUpload(device, mStream, false);
-}// PruneGrid<BuildT>::pruneRoot
+}// PruneGrid<BuildT, ResourceT>::pruneRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void PruneGrid<BuildT>::pruneInternalNodes()
+template<typename BuildT, typename ResourceT>
+void PruneGrid<BuildT, ResourceT>::pruneInternalNodes()
 {
     // Computes the masks of upper and (densified) lower internal nodes, as a result of the pruning operation
     // Masks of lower internal nodes are densified in the sense that a serialized array of them is allocated,
@@ -213,26 +219,26 @@ void PruneGrid<BuildT>::pruneInternalNodes()
     if (auto srcLeafCount = mSrcTreeData.mNodeCount[0]) { // Unless it's an empty grid
         util::cuda::lambdaKernel<<<numBlocks(srcLeafCount), mNumThreads, 0, mStream>>>(
             srcLeafCount, util::morphology::cuda::PruneInternalNodesFunctor<BuildT>(),
-            mDeviceSrcGrid, mBuilder.deviceProcessedRoot(), mDeviceSrcLeafMask, mBuilder.mUpperMasks.deviceData(), mBuilder.mLowerMasks.deviceData() );
+            mDeviceSrcGrid, mBuilder.deviceProcessedRoot(), mDeviceSrcLeafMask, mBuilder.mUpperMasks.data(), mBuilder.mLowerMasks.data() );
     }
-}// PruneGrid<BuildT>::pruneInternalNodes
+}// PruneGrid<BuildT, ResourceT>::pruneInternalNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template <typename BuildT>
-void PruneGrid<BuildT>::processGridTreeRoot()
+template <typename BuildT, typename ResourceT>
+void PruneGrid<BuildT, ResourceT>::processGridTreeRoot()
 {
     // Copy GridData from source grid
     // By convention: this will duplicate grid name and map. Others will be reset later
     cudaCheck(cudaMemcpyAsync(&mBuilder.data()->getGrid(), mDeviceSrcGrid->data(), GridT::memUsage(), cudaMemcpyDeviceToDevice, mStream));
     util::cuda::lambdaKernel<<<1, 1, 0, mStream>>>(1, topology::detail::BuildGridTreeRootFunctor<BuildT>(), mBuilder.deviceData());
     cudaCheckError();
-}// PruneGrid<BuildT>::processGridTreeRoot
+}// PruneGrid<BuildT, ResourceT>::processGridTreeRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void PruneGrid<BuildT>::pruneLeafNodes()
+template<typename BuildT, typename ResourceT>
+void PruneGrid<BuildT, ResourceT>::pruneLeafNodes()
 {
     // Prunes the active masks of the source grid to the intersection with the leaf-mask sidecar
     // followed by rebuilding the leaf offsets
@@ -244,7 +250,7 @@ void PruneGrid<BuildT>::pruneLeafNodes()
 
     // Update leaf offsets and prefix sums
     mBuilder.processLeafOffsets(mStream);
-}// PruneGrid<BuildT>::pruneLeafNodes
+}// PruneGrid<BuildT, ResourceT>::pruneLeafNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 

nanovdb/nanovdb/tools/cuda/RefineGrid.cuh

@@ -30,9 +30,12 @@ namespace nanovdb {
 
 namespace tools::cuda {
 
-template <typename BuildT>
+template <typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class RefineGrid
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "RefineGrid allocates stream-ordered scratch and requires an AsyncResource");
+
     using GridT  = NanoGrid<BuildT>;
     using TreeT  = NanoTree<BuildT>;
     using RootT  = NanoRoot<BuildT>;
@@ -43,8 +46,11 @@ public:
     /// @brief Constructor
     /// @param deviceGrid source device grid to be refined
     /// @param stream optional CUDA stream (defaults to CUDA stream 0)
-    RefineGrid(const GridT* d_srcGrid, cudaStream_t stream = 0)
-        : mBuilder(stream), mStream(stream), mTimer(stream), mDeviceSrcGrid(d_srcGrid) {}
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this operator (defaults to the per-type default resource)
+    RefineGrid(const GridT* d_srcGrid, cudaStream_t stream = 0,
+               ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : mBuilder(stream, resource), mStream(stream), mTimer(stream), mDeviceSrcGrid(d_srcGrid) {}
 
     /// @brief Toggle on and off verbose mode
     /// @param level Verbose level: 0=quiet, 1=timing, 2=benchmarking
@@ -74,20 +80,20 @@ private:
     static constexpr unsigned int mNumThreads = 128;// for kernels spawned via lambdaKernel (others may specialize)
     static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
 
-    TopologyBuilder<BuildT> mBuilder;
+    TopologyBuilder<BuildT, ResourceT> mBuilder;
     cudaStream_t            mStream{0};
     util::cuda::Timer       mTimer;
     int                     mVerbose{0};
     const GridT             *mDeviceSrcGrid;
     TreeData                mSrcTreeData;
-};// tools::cuda::RefineGrid<BuildT>
+};// tools::cuda::RefineGrid<BuildT, ResourceT>
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 template<typename BufferT>
 GridHandle<BufferT>
-RefineGrid<BuildT>::getHandle(const BufferT &pool)
+RefineGrid<BuildT, ResourceT>::getHandle(const BufferT &pool)
 {
     // Copy TreeData from GPU -> CPU
     cudaStreamSynchronize(mStream);
@@ -147,12 +153,12 @@ RefineGrid<BuildT>::getHandle(const BufferT &pool)
     cudaStreamSynchronize(mStream);
 
     return GridHandle<BufferT>(std::move(buffer));
-}// RefineGrid<BuildT>::getHandle
+}// RefineGrid<BuildT, ResourceT>::getHandle
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void RefineGrid<BuildT>::refineRoot()
+template<typename BuildT, typename ResourceT>
+void RefineGrid<BuildT, ResourceT>::refineRoot()
 {
     // This method conservatively and speculatively refines the root tiles, to accommodate
     // any new root nodes that might be introduced by the upsampling operation.
@@ -212,12 +218,12 @@ void RefineGrid<BuildT>::refineRoot()
     for (const auto& [key, tile] : refinedTiles)
         *refinedRootPtr->tile(t++) = tile;
     mBuilder.mProcessedRoot.deviceUpload(device, mStream, false);
-}// RefineGrid<BuildT>::refineRoot
+}// RefineGrid<BuildT, ResourceT>::refineRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void RefineGrid<BuildT>::refineInternalNodes()
+template<typename BuildT, typename ResourceT>
+void RefineGrid<BuildT, ResourceT>::refineInternalNodes()
 {
     // Computes the masks of upper and (densified) lower internal nodes, as a result of the refinement operation
     // Masks of lower internal nodes are densified in the sense that a serialized array of them is allocated,
@@ -225,26 +231,26 @@ void RefineGrid<BuildT>::refineInternalNodes()
     if (auto srcLeafCount = mSrcTreeData.mNodeCount[0]) { // Unless it's an empty grid
         util::cuda::lambdaKernel<<<numBlocks(srcLeafCount), mNumThreads, 0, mStream>>>(
             srcLeafCount, util::morphology::cuda::RefineInternalNodesFunctor<BuildT>(),
-            mDeviceSrcGrid, mBuilder.deviceProcessedRoot(), mBuilder.mUpperMasks.deviceData(), mBuilder.mLowerMasks.deviceData() );
+            mDeviceSrcGrid, mBuilder.deviceProcessedRoot(), mBuilder.mUpperMasks.data(), mBuilder.mLowerMasks.data() );
     }
-}// RefineGrid<BuildT>::refineInternalNodes
+}// RefineGrid<BuildT, ResourceT>::refineInternalNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template <typename BuildT>
-void RefineGrid<BuildT>::processGridTreeRoot()
+template <typename BuildT, typename ResourceT>
+void RefineGrid<BuildT, ResourceT>::processGridTreeRoot()
 {
     // Copy GridData from source grid
     // By convention: this will duplicate grid name and map. Others will be reset later
     cudaCheck(cudaMemcpyAsync(&mBuilder.data()->getGrid(), mDeviceSrcGrid->data(), GridT::memUsage(), cudaMemcpyDeviceToDevice, mStream));
     util::cuda::lambdaKernel<<<1, 1, 0, mStream>>>(1, topology::detail::BuildGridTreeRootFunctor<BuildT>(), mBuilder.deviceData());
     cudaCheckError();
-}// RefineGrid<BuildT>::processGridTreeRoot
+}// RefineGrid<BuildT, ResourceT>::processGridTreeRoot
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void RefineGrid<BuildT>::refineLeafNodes()
+template<typename BuildT, typename ResourceT>
+void RefineGrid<BuildT, ResourceT>::refineLeafNodes()
 {
     // Refines the active masks of the source grid (as indicated at the leaf level), into a new grid that
     // has been already topologically refined to include all necessary leaf nodes.
@@ -256,7 +262,7 @@ void RefineGrid<BuildT>::refineLeafNodes()
 
     // Update leaf offsets and prefix sums
     mBuilder.processLeafOffsets(mStream);
-}// RefineGrid<BuildT>::refineLeafNodes
+}// RefineGrid<BuildT, ResourceT>::refineLeafNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 

nanovdb/nanovdb/tools/cuda/SignedFloodFill.cuh

@@ -23,6 +23,7 @@
 #define NANOVDB_TOOLS_CUDA_SIGNEDFLOODFILL_CUH_HAS_BEEN_INCLUDED
 
 #include <nanovdb/NanoVDB.h>
+#include <nanovdb/cuda/Buffer.h>
 #include <nanovdb/GridHandle.h>
 #include <nanovdb/cuda/UnifiedBuffer.h>
 #include <nanovdb/util/cuda/Timer.h>
@@ -36,19 +37,28 @@ namespace tools::cuda {
 
 /// @brief Performs signed flood-fill operation on the hierarchical tree structure on the device
 /// @tparam BuildT Build type of the grid to be flood-filled
+/// @tparam ResourceT Template type of optional resource used for internal temporary memory
 /// @param d_grid Non-const device pointer to the grid that will be flood-filled
 /// @param verbose If true timing information will be printed to the terminal
 /// @param stream optional cuda stream
-template<typename BuildT>
+template<typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 typename util::enable_if<BuildTraits<BuildT>::is_float, void>::type
 signedFloodFill(NanoGrid<BuildT> *d_grid, bool verbose = false, cudaStream_t stream = 0);
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class SignedFloodFill
 {
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "SignedFloodFill allocates stream-ordered scratch and requires an AsyncResource");
 public:
-    SignedFloodFill(bool verbose = false, cudaStream_t stream = 0)
-        : mStream(stream), mVerbose(verbose) {}
+    /// @brief Constructor
+    /// @param verbose if true timing information is printed to the terminal
+    /// @param stream optional CUDA stream (defaults to CUDA stream 0)
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this instance (defaults to the per-type default resource)
+    SignedFloodFill(bool verbose = false, cudaStream_t stream = 0,
+                    ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : mStream(stream), mVerbose(verbose), mResource(&resource) {}
 
     /// @brief Toggle on and off verbose mode
     /// @param on if true verbose is turned on
@@ -60,6 +70,11 @@ private:
     cudaStream_t      mStream{0};
     util::cuda::Timer mTimer;
     bool              mVerbose{false};
+    ResourceT*        mResource;// non-owning; all device scratch routes through this resource instance
+
+    template<typename T>
+    using BufT = nanovdb::cuda::Buffer<T, nanovdb::cuda::ResourceRef<ResourceT>>;
+    nanovdb::cuda::ResourceRef<ResourceT> ref() { return nanovdb::cuda::ResourceRef<ResourceT>(*mResource); }
 
 };// SignedFloodFill
 
@@ -78,7 +93,7 @@ struct RootChild {
 
 // CPU kernel!
 template<typename BuildT>
-void processRoot(NanoTree<BuildT> *d_tree)
+void processRoot(NanoTree<BuildT> *d_tree, cudaStream_t stream = 0)
 {// the root needs special care since unlike other nodes it's sparse and not dense!
     using TreeT  = NanoTree<BuildT>;
     using RootT  = NanoRoot<BuildT>;
@@ -87,6 +102,12 @@ void processRoot(NanoTree<BuildT> *d_tree)
     using ChildT = RootChild<ValueT>;
     static const int dim = int(RootT::ChildNodeType::DIM);
 
+    // Ensure node passes issued on a non-default 'stream' have completed before
+    // the synchronous default-stream cudaMemcpy below reads d_tree. For the
+    // default stream (0) that blocking copy already orders after prior stream-0
+    // work, so the extra sync there is pure overhead - skip it.
+    if (stream != cudaStream_t{0}) cudaCheck(cudaStreamSynchronize(stream));
+
     // First copy the tree and root and then its tiles, which is of unknown size
     nanovdb::cuda::UnifiedBuffer uBuffer(sizeof(TreeT) + sizeof(RootT), sizeof(TreeT) + sizeof(RootT) + 64*sizeof(TileT));
     cudaCheck(cudaMemcpy(uBuffer.data(), d_tree, uBuffer.size(), cudaMemcpyDeviceToHost));// copy Tree and Root (minus tiles)
@@ -108,11 +129,11 @@ void processRoot(NanoTree<BuildT> *d_tree)
     for (ChildT *a = first, *b = a+1; b!=last; ++a, ++b) {// loop over pairs of adjacent child nodes
         const Coord d = b->ijk - a->ijk;// coord delta of adjacent child nodes
         if (d[0]!=0 || d[1]!=0 || d[2]==dim) continue;// not same z-scanline or they are neighbors
-        util::cuda::lambdaKernel<<<1, 1>>>(1, [=] __device__(size_t) {
+        util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, [=] __device__(size_t) {
             a->val[1] = root->getChild(root->tile(a->idx))->getLastValue();
             b->val[0] = root->getChild(root->tile(b->idx))->getFirstValue();
         });
-        cudaCheck(cudaDeviceSynchronize());// required for host access to RootChild::val[2]
+        cudaCheck(cudaStreamSynchronize(stream));// required for host access to RootChild::val[2]
         if (a->val[1] > 0 || b->val[0] > 0) continue; // scanline is not inside a surface
         for (Coord c = a->ijk.offsetBy(0,0,dim); c[2] != b->ijk[2]; c[2] += dim) {
             TileT *tile = root->probeTile(c);
@@ -179,17 +200,18 @@ __global__ void cpyNodeCountKernel(NanoGrid<BuildT> *d_grid, uint64_t *d_count)
 
 //================================================================================================
 
-template <typename BuildT>
-void SignedFloodFill<BuildT>::operator()(NanoGrid<BuildT> *d_grid)
+template <typename BuildT, typename ResourceT>
+void SignedFloodFill<BuildT, ResourceT>::operator()(NanoGrid<BuildT> *d_grid)
 {
     static_assert(BuildTraits<BuildT>::is_float, "cuda::SignedFloodFill only works on float grids");
     NANOVDB_ASSERT(d_grid);
-    uint64_t count[4], *d_count = nullptr;
-    cudaCheck(util::cuda::mallocAsync((void**)&d_count, 4*sizeof(uint64_t), mStream));
+    uint64_t count[4];
+    BufT<uint64_t> countBuf(mStream, this->ref(), 4, nanovdb::cuda::noInit);
+    uint64_t *d_count = countBuf.data();
     kernels::cpyNodeCountKernel<BuildT><<<1, 1, 0, mStream>>>(d_grid, d_count);
     cudaCheckError();
     cudaCheck(cudaMemcpyAsync(&count, d_count, 4*sizeof(uint64_t), cudaMemcpyDeviceToHost, mStream));
-    cudaCheck(util::cuda::freeAsync(d_count, mStream));
+    countBuf.destroy(mStream);
 
     static const int threadsPerBlock = 128;
     auto blocksPerGrid = [&](size_t count)->uint32_t{return (count + (threadsPerBlock - 1)) / threadsPerBlock;};
@@ -208,18 +230,18 @@ void SignedFloodFill<BuildT>::operator()(NanoGrid<BuildT> *d_grid)
     cudaCheckError();
 
     if (mVerbose) mTimer.restart("Process root node");
-    kernels::processRoot<BuildT>(d_tree);
+    kernels::processRoot<BuildT>(d_tree, mStream);
     if (mVerbose) mTimer.stop();
     cudaCheckError();
 }// SignedFloodFill::operator()
 
 //================================================================================================
 
-template<typename BuildT>
+template<typename BuildT, typename ResourceT>
 typename util::enable_if<BuildTraits<BuildT>::is_float, void>::type
 signedFloodFill(NanoGrid<BuildT> *d_grid, bool verbose, cudaStream_t stream)
 {
-    SignedFloodFill<BuildT> sff(verbose, stream);
+    SignedFloodFill<BuildT, ResourceT> sff(verbose, stream);
     sff(d_grid);
     auto *d_gridData = d_grid->data();
     Checksum cs = getChecksum(d_gridData, stream);

nanovdb/nanovdb/tools/cuda/TopologyBuilder.cuh

@@ -17,6 +17,8 @@
 
 #include <nanovdb/NanoVDB.h>
 #include <nanovdb/cuda/TempPool.h>
+#include <nanovdb/cuda/Buffer.h>
+#include <nanovdb/cuda/DeviceResource.h>
 #include <nanovdb/cuda/DeviceBuffer.h>
 #include <nanovdb/util/cuda/Morphology.cuh>
 
@@ -24,7 +26,25 @@ namespace nanovdb {
 
 namespace tools::cuda {
 
+/// @brief Shared grid/tree offsets and node counts handed to the device
+///        functors. Independent of the resource the builder allocates from,
+///        so it lives outside TopologyBuilder and stays one type across every
+///        ResourceT instantiation.
 template <typename BuildT>
+struct TopologyBuilderData {
+    void     *d_bufferPtr;
+    uint64_t grid, tree, root, upper, lower, leaf, size;// byte offsets to nodes in buffer
+    uint32_t nodeCount[3];// 0=leaf,1=lower, 2=upper
+    uint32_t *d_upperOffsets;
+    __hostdev__ NanoGrid<BuildT>&  getGrid() const {return *util::PtrAdd<NanoGrid<BuildT>>(d_bufferPtr, grid);}
+    __hostdev__ NanoTree<BuildT>&  getTree() const {return *util::PtrAdd<NanoTree<BuildT>>(d_bufferPtr, tree);}
+    __hostdev__ NanoRoot<BuildT>&  getRoot() const {return *util::PtrAdd<NanoRoot<BuildT>>(d_bufferPtr, root);}
+    __hostdev__ NanoUpper<BuildT>& getUpper(int i) const {return *(util::PtrAdd<NanoUpper<BuildT>>(d_bufferPtr, upper)+i);}
+    __hostdev__ NanoLower<BuildT>& getLower(int i) const {return *(util::PtrAdd<NanoLower<BuildT>>(d_bufferPtr, lower)+i);}
+    __hostdev__ NanoLeaf<BuildT>&  getLeaf(int i) const {return *(util::PtrAdd<NanoLeaf<BuildT>>(d_bufferPtr, leaf)+i);}
+};// TopologyBuilderData
+
+template <typename BuildT, typename ResourceT = nanovdb::cuda::DeviceResource>
 class TopologyBuilder
 {
     static_assert(nanovdb::BuildTraits<BuildT>::is_onindex);// For now, only OnIndexGrids supported
@@ -36,25 +56,40 @@ class TopologyBuilder
     using LowerT = NanoLower<BuildT>;
     using LeafT  = NanoLeaf<BuildT>;
 
+    static_assert(nanovdb::cuda::is_async_resource<ResourceT>::value,
+                  "TopologyBuilder allocates stream-ordered scratch and requires an AsyncResource");
+    static_assert(ResourceT::DEFAULT_ALIGNMENT >= alignof(uint64_t),
+                  "TopologyBuilder reinterprets byte scratch as word-sized types and requires word-aligned allocations");
+
+    /// @brief Device-only scratch storage, borrowing the injected resource
+    ///        through a ResourceRef so all traffic reaches the caller's
+    ///        instance (which may be stateful) rather than a copy. These
+    ///        buffers are never read on the host, so they use the single-space
+    ///        Buffer rather than the dual DeviceBuffer, whose host pointer and
+    ///        per-device array they would leave unused.
+    using ScratchT = nanovdb::cuda::Buffer<std::byte, nanovdb::cuda::ResourceRef<ResourceT>>;
+
 public:
 
-    TopologyBuilder(cudaStream_t stream)
+    /// @param stream cuda stream the scratch allocations are ordered on
+    /// @param resource resource instance all device scratch is allocated from;
+    ///        must outlive this builder
+    TopologyBuilder(cudaStream_t stream, ResourceT& resource = nanovdb::cuda::default_resource<ResourceT>())
+        : mUpperMasks(stream, resource, 0, nanovdb::cuda::noInit)
+        , mLowerMasks(stream, resource, 0, nanovdb::cuda::noInit)
+        , mUpperOffsets(stream, resource, 0, nanovdb::cuda::noInit)
+        , mLowerOffsets(stream, resource, 0, nanovdb::cuda::noInit)
+        , mLeafOffsets(stream, resource, 0, nanovdb::cuda::noInit)
+        , mVoxelOffsets(stream, resource, 0, nanovdb::cuda::noInit)
+        , mLowerParents(stream, resource, 0, nanovdb::cuda::noInit)
+        , mLeafParents(stream, resource, 0, nanovdb::cuda::noInit)
+        , mResource(&resource)
+        , mTempDevicePool(resource)
     {
         mData = nanovdb::cuda::DeviceBuffer::create(sizeof(Data));
     }
 
-    struct Data {
-        void     *d_bufferPtr;
-        uint64_t grid, tree, root, upper, lower, leaf, size;// byte offsets to nodes in buffer
-        uint32_t nodeCount[3];// 0=leaf,1=lower, 2=upper
-        uint32_t *d_upperOffsets;
-        __hostdev__ GridT&  getGrid() const {return *util::PtrAdd<GridT>(d_bufferPtr, grid);}
-        __hostdev__ TreeT&  getTree() const {return *util::PtrAdd<TreeT>(d_bufferPtr, tree);}
-        __hostdev__ RootT&  getRoot() const {return *util::PtrAdd<RootT>(d_bufferPtr, root);}
-        __hostdev__ UpperT& getUpper(int i) const {return *(util::PtrAdd<UpperT>(d_bufferPtr, upper)+i);}
-        __hostdev__ LowerT& getLower(int i) const {return *(util::PtrAdd<LowerT>(d_bufferPtr, lower)+i);}
-        __hostdev__ LeafT&  getLeaf(int i) const {return *(util::PtrAdd<LeafT>(d_bufferPtr, leaf)+i);}
-    };// Data
+    using Data = TopologyBuilderData<BuildT>;
 
     void allocateInternalMaskBuffers(cudaStream_t stream);
 
@@ -74,21 +109,21 @@ public:
     void postProcessGridTree(cudaStream_t stream);
 
     nanovdb::cuda::DeviceBuffer  mProcessedRoot;
-    nanovdb::cuda::DeviceBuffer  mUpperMasks;
-    nanovdb::cuda::DeviceBuffer  mLowerMasks;
-    nanovdb::cuda::DeviceBuffer  mUpperOffsets;
-    nanovdb::cuda::DeviceBuffer  mLowerOffsets;
-    nanovdb::cuda::DeviceBuffer  mLeafOffsets;
-    nanovdb::cuda::DeviceBuffer  mVoxelOffsets;
-    nanovdb::cuda::DeviceBuffer  mLowerParents;
-    nanovdb::cuda::DeviceBuffer  mLeafParents;
+    ScratchT                     mUpperMasks;
+    ScratchT                     mLowerMasks;
+    ScratchT                     mUpperOffsets;
+    ScratchT                     mLowerOffsets;
+    ScratchT                     mLeafOffsets;
+    ScratchT                     mVoxelOffsets;
+    ScratchT                     mLowerParents;
+    ScratchT                     mLeafParents;
     nanovdb::cuda::DeviceBuffer  mData;
     CheckMode                    mChecksum{CheckMode::Disable};
 
     auto deviceProcessedRoot() { return static_cast<RootT*>(mProcessedRoot.deviceData()); }
     auto hostProcessedRoot()   { return static_cast<RootT*>(mProcessedRoot.data()); }
-    void* deviceUpperMasks() { return mUpperMasks.deviceData(); }
-    void* deviceLowerMasks() { return mLowerMasks.deviceData(); }
+    void* deviceUpperMasks() { return mUpperMasks.data(); }
+    void* deviceLowerMasks() { return mLowerMasks.data(); }
     Data* data()             { return static_cast<Data*>(mData.data()); }
     Data* deviceData()       { return static_cast<Data*>(mData.deviceData()); }
 
@@ -96,8 +131,9 @@ private:
     static constexpr unsigned int mNumThreads = 128;// for kernels spawned via lambdaKernel (others may specialize)
     static unsigned int numBlocks(unsigned int n) {return (n + mNumThreads - 1) / mNumThreads;}
 
-    nanovdb::cuda::TempDevicePool mTempDevicePool;
-};// tools::cuda::TopologyBuilder<BuildT>
+    ResourceT*                        mResource;// non-owning; all device scratch routes through this instance
+    nanovdb::cuda::TempPool<ResourceT> mTempDevicePool;
+};// tools::cuda::TopologyBuilder<BuildT, ResourceT>
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -118,30 +154,28 @@ private:
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void TopologyBuilder<BuildT>::allocateInternalMaskBuffers(cudaStream_t stream)
+template<typename BuildT, typename ResourceT>
+void TopologyBuilder<BuildT, ResourceT>::allocateInternalMaskBuffers(cudaStream_t stream)
 {
     if (hostProcessedRoot()->tileCount() == 0) return; // Processing empty grid(s); nothing to allocate
 
     // Allocate (and zero-fill) buffers large enough to hold:
     // (a) The serialized masks of all upper nodes, for all tiles in the updated root node, and
     // (b) The serialized masks of all densified lower nodes, as if every upper node had a full set of 32^3 lower children
-    int device = 0;
-    cudaGetDevice(&device);
     uint64_t upperSize = hostProcessedRoot()->tileCount() * sizeof(Mask<5>);
     uint64_t lowerSize = hostProcessedRoot()->tileCount() * Mask<5>::SIZE * sizeof(Mask<4>);
-    mUpperMasks = nanovdb::cuda::DeviceBuffer::create(upperSize, nullptr, device, stream);
-    if (mUpperMasks.deviceData() == nullptr) throw std::runtime_error("Failed to allocate upper mask buffer on device");
-    cudaCheck(cudaMemsetAsync(mUpperMasks.deviceData(), 0, upperSize, stream));
-    mLowerMasks = nanovdb::cuda::DeviceBuffer::create( lowerSize, nullptr, device, stream );
-    if (mLowerMasks.deviceData() == nullptr) throw std::runtime_error("Failed to allocate lower mask buffer on device");
-    cudaCheck(cudaMemsetAsync(mLowerMasks.deviceData(), 0, lowerSize, stream));
-}// TopologyBuilder<BuildT>::allocateInternalMaskBuffers
+    mUpperMasks = ScratchT(stream, *mResource, upperSize, nanovdb::cuda::noInit);
+    if (mUpperMasks.data() == nullptr) throw std::runtime_error("Failed to allocate upper mask buffer on device");
+    cudaCheck(cudaMemsetAsync(mUpperMasks.data(), 0, upperSize, stream));
+    mLowerMasks = ScratchT(stream, *mResource, lowerSize, nanovdb::cuda::noInit);
+    if (mLowerMasks.data() == nullptr) throw std::runtime_error("Failed to allocate lower mask buffer on device");
+    cudaCheck(cudaMemsetAsync(mLowerMasks.data(), 0, lowerSize, stream));
+}// TopologyBuilder<BuildT, ResourceT>::allocateInternalMaskBuffers
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template<typename BuildT>
-void TopologyBuilder<BuildT>::countNodes(cudaStream_t stream)
+template<typename BuildT, typename ResourceT>
+void TopologyBuilder<BuildT, ResourceT>::countNodes(cudaStream_t stream)
 {
     auto processedTileCount = hostProcessedRoot()->tileCount();
     if (processedTileCount == 0) { // Processing empty grid(s); zero nodes at all levels
@@ -155,59 +189,57 @@ void TopologyBuilder<BuildT>::countNodes(cudaStream_t stream)
     // as well as the tile table at the root.
     std::size_t size = processedTileCount*Mask<5>::SIZE;
 
-    int device = 0;
-    cudaGetDevice(&device);
-    nanovdb::cuda::DeviceBuffer upperCountsBuffer = nanovdb::cuda::DeviceBuffer::create(processedTileCount*sizeof(uint32_t), nullptr, device, stream);
-    nanovdb::cuda::DeviceBuffer lowerCountsBuffer = nanovdb::cuda::DeviceBuffer::create(size*sizeof(uint32_t), nullptr, device, stream);
-    nanovdb::cuda::DeviceBuffer leafCountsBuffer = nanovdb::cuda::DeviceBuffer::create(size*sizeof(uint32_t), nullptr, device, stream);
+    ScratchT upperCountsBuffer = ScratchT(stream, *mResource, processedTileCount*sizeof(uint32_t), nanovdb::cuda::noInit);
+    ScratchT lowerCountsBuffer = ScratchT(stream, *mResource, size*sizeof(uint32_t), nanovdb::cuda::noInit);
+    ScratchT leafCountsBuffer = ScratchT(stream, *mResource, size*sizeof(uint32_t), nanovdb::cuda::noInit);
 
     using CountType = uint32_t (*)[Mask<5>::SIZE];
-    auto lowerCounts = reinterpret_cast<CountType>( lowerCountsBuffer.deviceData() );
-    auto leafCounts = reinterpret_cast<CountType>( leafCountsBuffer.deviceData() );
+    auto lowerCounts = reinterpret_cast<CountType>(lowerCountsBuffer.data());
+    auto leafCounts = reinterpret_cast<CountType>(leafCountsBuffer.data());
 
     using Op = util::morphology::cuda::EnumerateNodesFunctor;
     util::cuda::operatorKernel<Op>
         <<<dim3(processedTileCount, Op::SlicesPerUpperNode, 1), Op::MaxThreadsPerBlock, 0, stream>>>
         (deviceUpperMasks(), deviceLowerMasks(), lowerCounts, leafCounts);
 
-    mUpperOffsets = nanovdb::cuda::DeviceBuffer::create((processedTileCount+1)*sizeof(uint32_t), nullptr, device, stream);
-    mLowerOffsets = nanovdb::cuda::DeviceBuffer::create((size+1)*sizeof(uint32_t), nullptr, device, stream);
-    mLeafOffsets = nanovdb::cuda::DeviceBuffer::create((size+1)*sizeof(uint32_t), nullptr, device, stream);
+    mUpperOffsets = ScratchT(stream, *mResource, (processedTileCount+1)*sizeof(uint32_t), nanovdb::cuda::noInit);
+    mLowerOffsets = ScratchT(stream, *mResource, (size+1)*sizeof(uint32_t), nanovdb::cuda::noInit);
+    mLeafOffsets = ScratchT(stream, *mResource, (size+1)*sizeof(uint32_t), nanovdb::cuda::noInit);
 
-    cudaCheck(cudaMemsetAsync(mLowerOffsets.deviceData(), 0, sizeof(uint32_t), stream));
+    cudaCheck(cudaMemsetAsync(mLowerOffsets.data(), 0, sizeof(uint32_t), stream));
     CALL_CUBS(DeviceScan::InclusiveSum,
-        static_cast<uint32_t*>(lowerCountsBuffer.deviceData()),
-        static_cast<uint32_t*>(mLowerOffsets.deviceData())+1,
+        reinterpret_cast<uint32_t*>(lowerCountsBuffer.data()),
+        reinterpret_cast<uint32_t*>(mLowerOffsets.data())+1,
         size);
-    cudaCheck(cudaMemcpyAsync(&data()->nodeCount[1], static_cast<uint32_t*>(mLowerOffsets.deviceData())+size, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
+    cudaCheck(cudaMemcpyAsync(&data()->nodeCount[1], reinterpret_cast<uint32_t*>(mLowerOffsets.data())+size, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
 
-    cudaCheck(cudaMemsetAsync(mLeafOffsets.deviceData(), 0, sizeof(uint32_t), stream));
+    cudaCheck(cudaMemsetAsync(mLeafOffsets.data(), 0, sizeof(uint32_t), stream));
     CALL_CUBS(DeviceScan::InclusiveSum,
-        static_cast<uint32_t*>(leafCountsBuffer.deviceData()),
-        static_cast<uint32_t*>(mLeafOffsets.deviceData())+1,
+        reinterpret_cast<uint32_t*>(leafCountsBuffer.data()),
+        reinterpret_cast<uint32_t*>(mLeafOffsets.data())+1,
         size);
-    cudaCheck(cudaMemcpyAsync(&data()->nodeCount[0], static_cast<uint32_t*>(mLeafOffsets.deviceData())+size, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
+    cudaCheck(cudaMemcpyAsync(&data()->nodeCount[0], reinterpret_cast<uint32_t*>(mLeafOffsets.data())+size, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
 
     util::cuda::lambdaKernel<<<numBlocks(processedTileCount), mNumThreads, 0, stream>>>(
         processedTileCount,
         [] __device__(size_t tileID, CountType lowerOffsets, uint32_t* upperCounts)
             { upperCounts[tileID] = (lowerOffsets[tileID+1][0] > lowerOffsets[tileID][0]) ? 1 : 0; },
-        static_cast<CountType>(mLowerOffsets.deviceData()),
-        static_cast<uint32_t*>(upperCountsBuffer.deviceData()));
+        reinterpret_cast<CountType>(mLowerOffsets.data()),
+        reinterpret_cast<uint32_t*>(upperCountsBuffer.data()));
 
-    cudaCheck(cudaMemsetAsync( mUpperOffsets.deviceData(), 0, sizeof(uint32_t), stream));
+    cudaCheck(cudaMemsetAsync( mUpperOffsets.data(), 0, sizeof(uint32_t), stream));
     CALL_CUBS(DeviceScan::InclusiveSum,
-        static_cast<uint32_t*>(upperCountsBuffer.deviceData()),
-        static_cast<uint32_t*>(mUpperOffsets.deviceData())+1,
+        reinterpret_cast<uint32_t*>(upperCountsBuffer.data()),
+        reinterpret_cast<uint32_t*>(mUpperOffsets.data())+1,
         processedTileCount);
-    cudaCheck(cudaMemcpyAsync(&data()->nodeCount[2], static_cast<uint32_t*>(mUpperOffsets.deviceData())+processedTileCount, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
-}// TopologyBuilder<BuildT>::countNodes
+    cudaCheck(cudaMemcpyAsync(&data()->nodeCount[2], reinterpret_cast<uint32_t*>(mUpperOffsets.data())+processedTileCount, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream));
+}// TopologyBuilder<BuildT, ResourceT>::countNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template <typename BuildT>
+template <typename BuildT, typename ResourceT>
 template <typename BufferT>
-BufferT TopologyBuilder<BuildT>::getBuffer(const BufferT &pool, cudaStream_t stream)
+BufferT TopologyBuilder<BuildT, ResourceT>::getBuffer(const BufferT &pool, cudaStream_t stream)
 {
     // Allocates a device buffer for the destination grid, once the topology/size of the tree is known
     data()->grid  = 0;// grid is always stored at the start of the buffer!
@@ -226,11 +258,11 @@ BufferT TopologyBuilder<BuildT>::getBuffer(const BufferT &pool, cudaStream_t str
     data()->d_bufferPtr = buffer.deviceData();
     if (data()->d_bufferPtr == nullptr) throw std::runtime_error("Failed to allocate grid buffer on the device");
     if (data()->nodeCount[2] != 0) // Unless the result is an empty grid
-        data()->d_upperOffsets = static_cast<uint32_t*>(mUpperOffsets.deviceData());
+        data()->d_upperOffsets = reinterpret_cast<uint32_t*>(mUpperOffsets.data());
     mData.deviceUpload(device, stream, false);
 
     return buffer;
-}// TopologyBuilder<BuildT>::getBuffer
+}// TopologyBuilder<BuildT, ResourceT>::getBuffer
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -240,7 +272,7 @@ template <typename BuildT>
 struct BuildGridTreeRootFunctor
 {
     __device__
-    void operator()(size_t, typename TopologyBuilder<BuildT>::Data *d_data) {
+    void operator()(size_t, TopologyBuilderData<BuildT> *d_data) {
 
         // process Root
         auto &root = d_data->getRoot();
@@ -320,7 +352,7 @@ struct InitGridTreeRootFunctor
     Map map; // transform to embed in the output grid
 
     __device__
-    void operator()(size_t, typename TopologyBuilder<BuildT>::Data *d_data) {
+    void operator()(size_t, TopologyBuilderData<BuildT> *d_data) {
 
         // process Root (identical to BuildGridTreeRootFunctor)
         auto &root = d_data->getRoot();
@@ -389,7 +421,7 @@ template <typename BuildT>
 struct BuildUpperNodesFunctor
 {
     __device__
-    void operator()(size_t processedTileID, typename TopologyBuilder<BuildT>::Data *d_data, NanoRoot<BuildT> *d_processedRoot) {
+    void operator()(size_t processedTileID, TopologyBuilderData<BuildT> *d_data, NanoRoot<BuildT> *d_processedRoot) {
         uint32_t tileID = d_data->d_upperOffsets[processedTileID];
         if (tileID != d_data->d_upperOffsets[processedTileID+1]) // if the offsets are the same, this was a speculatively introduced tile which was not necessary
         {
@@ -406,8 +438,8 @@ struct BuildUpperNodesFunctor
 
 }// namespace topology::detail
 
-template <typename BuildT>
-inline void TopologyBuilder<BuildT>::processUpperNodes(cudaStream_t stream)
+template<typename BuildT, typename ResourceT>
+inline void TopologyBuilder<BuildT, ResourceT>::processUpperNodes(cudaStream_t stream)
 {
     // Connect all newly allocated upper nodes to their respective tiles
     // Also fill in any necessary part of the preamble (in InternalData) of upper nodes
@@ -418,12 +450,12 @@ inline void TopologyBuilder<BuildT>::processUpperNodes(cudaStream_t stream)
             processedTileCount, topology::detail::BuildUpperNodesFunctor<BuildT>(), deviceData(), deviceProcessedRoot());
         cudaCheckError();
     }
-}// TopologyBuilder<BuildT>::processUpperNodes
+}// TopologyBuilder<BuildT, ResourceT>::processUpperNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-template <typename BuildT>
-inline void TopologyBuilder<BuildT>::processLowerNodes(cudaStream_t stream)
+template<typename BuildT, typename ResourceT>
+inline void TopologyBuilder<BuildT, ResourceT>::processLowerNodes(cudaStream_t stream)
 {
     // Fill out the contents of all newly allocated lower nodes (using the densified upper/lower mask arrays)
     // Also fill in the preamble (most of LeafData) for their leaf children
@@ -431,34 +463,32 @@ inline void TopologyBuilder<BuildT>::processLowerNodes(cudaStream_t stream)
     using CountType = uint32_t (*)[Mask<5>::SIZE];
  
     if (processedTileCount) { // Unless output grid is empty
-        int device = 0;
-        cudaGetDevice(&device);
         std::size_t lowerCount = data()->nodeCount[1];
-        mLowerParents = nanovdb::cuda::DeviceBuffer::create(lowerCount*sizeof(uint32_t), nullptr, device, stream);
+        mLowerParents = ScratchT(stream, *mResource, lowerCount*sizeof(uint32_t), nanovdb::cuda::noInit);
         std::size_t leafCount = data()->nodeCount[0];
-        mLeafParents = nanovdb::cuda::DeviceBuffer::create(leafCount*sizeof(uint32_t), nullptr, device, stream);
+        mLeafParents = ScratchT(stream, *mResource, leafCount*sizeof(uint32_t), nanovdb::cuda::noInit);
 
         using Op = util::morphology::cuda::ProcessLowerNodesFunctor<BuildT>;
         util::cuda::operatorKernel<Op>
             <<<dim3(processedTileCount, Op::SlicesPerUpperNode, 1), Op::MaxThreadsPerBlock, 0, stream>>>(
                 deviceUpperMasks(),
                 deviceLowerMasks(),
-                static_cast<uint32_t*>(mUpperOffsets.deviceData()),
-                static_cast<CountType>(mLowerOffsets.deviceData()),
-                static_cast<CountType>(mLeafOffsets.deviceData()),
+                reinterpret_cast<uint32_t*>(mUpperOffsets.data()),
+                reinterpret_cast<CountType>(mLowerOffsets.data()),
+                reinterpret_cast<CountType>(mLeafOffsets.data()),
                 static_cast<GridT*>(data()->d_bufferPtr),
-                static_cast<uint32_t*>(mLowerParents.deviceData()),
-                static_cast<uint32_t*>(mLeafParents.deviceData())
+                reinterpret_cast<uint32_t*>(mLowerParents.data()),
+                reinterpret_cast<uint32_t*>(mLeafParents.data())
             );
         cudaCheckError();
     }
 
     mProcessedRoot.clear(stream);
-    mUpperMasks.clear(stream);
-    mLowerMasks.clear(stream);
-    mLowerOffsets.clear(stream);
-    mLeafOffsets.clear(stream);
-}// TopologyBuilder<BuildT>::processLowerNodes
+    mUpperMasks.destroy(stream);
+    mLowerMasks.destroy(stream);
+    mLowerOffsets.destroy(stream);
+    mLeafOffsets.destroy(stream);
+}// TopologyBuilder<BuildT, ResourceT>::processLowerNodes
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -468,7 +498,7 @@ template <typename BuildT>
 struct UpdateLeafVoxelCountsAndPrefixSumFunctor
 {
     __device__
-    void operator()(size_t leafID, typename TopologyBuilder<BuildT>::Data *d_data, uint64_t *d_voxelCounts) {
+    void operator()(size_t leafID, TopologyBuilderData<BuildT> *d_data, uint64_t *d_voxelCounts) {
         auto &leaf = d_data->getGrid().tree().template getFirstNode<0>()[leafID];
         const uint64_t *w = leaf.mValueMask.words();
         uint64_t prefixSum = 0, sum = util::countOn(*w++);
@@ -485,32 +515,30 @@ template <typename BuildT>
 struct UpdateLeafVoxelOffsetsFunctor
 {
     __device__
-    void operator()(size_t leafID, typename TopologyBuilder<BuildT>::Data *d_data, uint64_t *d_voxelOffsets) {
+    void operator()(size_t leafID, TopologyBuilderData<BuildT> *d_data, uint64_t *d_voxelOffsets) {
         auto &leaf = d_data->getGrid().tree().template getFirstNode<0>()[leafID];
         leaf.mOffset = d_voxelOffsets[leafID]+1; }
 };
 
 }// namespace topology::detail
 
-template<typename BuildT>
-inline void TopologyBuilder<BuildT>::processLeafOffsets(cudaStream_t stream)
+template<typename BuildT, typename ResourceT>
+inline void TopologyBuilder<BuildT, ResourceT>::processLeafOffsets(cudaStream_t stream)
 {
-    int device = 0;
-    cudaGetDevice(&device);
     std::size_t leafCount = data()->nodeCount[0];
     if (leafCount) { // Unless output grid is empty
-        mVoxelOffsets = nanovdb::cuda::DeviceBuffer::create((leafCount+1)*sizeof(uint64_t), nullptr, device, stream);
-        cudaCheck(cudaMemsetAsync(mVoxelOffsets.deviceData(), 0, sizeof(uint64_t), stream));
+        mVoxelOffsets = ScratchT(stream, *mResource, (leafCount+1)*sizeof(uint64_t), nanovdb::cuda::noInit);
+        cudaCheck(cudaMemsetAsync(mVoxelOffsets.data(), 0, sizeof(uint64_t), stream));
         util::cuda::lambdaKernel<<<numBlocks(leafCount), mNumThreads, 0, stream>>>(
-            leafCount, topology::detail::UpdateLeafVoxelCountsAndPrefixSumFunctor<BuildT>(), deviceData(), static_cast<uint64_t*>(mVoxelOffsets.deviceData())+1);
+            leafCount, topology::detail::UpdateLeafVoxelCountsAndPrefixSumFunctor<BuildT>(), deviceData(), reinterpret_cast<uint64_t*>(mVoxelOffsets.data())+1);
         CALL_CUBS(DeviceScan::InclusiveSum,
-            static_cast<uint64_t*>(mVoxelOffsets.deviceData())+1,
-            static_cast<uint64_t*>(mVoxelOffsets.deviceData())+1,
+            reinterpret_cast<uint64_t*>(mVoxelOffsets.data())+1,
+            reinterpret_cast<uint64_t*>(mVoxelOffsets.data())+1,
             leafCount);
         util::cuda::lambdaKernel<<<numBlocks(leafCount), mNumThreads, 0, stream>>>(
-            leafCount, topology::detail::UpdateLeafVoxelOffsetsFunctor<BuildT>(), deviceData(), static_cast<uint64_t*>(mVoxelOffsets.deviceData()));
+            leafCount, topology::detail::UpdateLeafVoxelOffsetsFunctor<BuildT>(), deviceData(), reinterpret_cast<uint64_t*>(mVoxelOffsets.data()));
     }
-}// TopologyBuilder<BuildT>::processLeafOffsets
+}// TopologyBuilder<BuildT, ResourceT>::processLeafOffsets
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -527,7 +555,7 @@ template <typename BuildT>
 struct UpdateAndPropagateLeafBBoxFunctor
 {
     __device__
-    void operator()(size_t tid, typename TopologyBuilder<BuildT>::Data *d_data, const uint32_t* leafParents) {
+    void operator()(size_t tid, TopologyBuilderData<BuildT> *d_data, const uint32_t* leafParents) {
         auto &lower = d_data->getLower(leafParents[tid]);
         auto &leaf = d_data->getLeaf(tid);
         leaf.updateBBox();
@@ -539,7 +567,7 @@ template <typename BuildT>
 struct PropagateLowerBBoxFunctor
 {
     __device__
-    void operator()(size_t tid, typename TopologyBuilder<BuildT>::Data *d_data, const uint32_t* lowerParents) {
+    void operator()(size_t tid, TopologyBuilderData<BuildT> *d_data, const uint32_t* lowerParents) {
         auto &upper = d_data->getUpper(lowerParents[tid]);
         auto &lower = d_data->getLower(tid);
         upper.mBBox.expandAtomic(lower.bbox()); }
@@ -549,7 +577,7 @@ template <typename BuildT>
 struct PropagateUpperBBoxFunctor
 {
     __device__
-    void operator()(size_t tid, typename TopologyBuilder<BuildT>::Data *d_data) {
+    void operator()(size_t tid, TopologyBuilderData<BuildT> *d_data) {
         d_data->getRoot().mBBox.expandAtomic(d_data->getUpper(tid).bbox());
     }
 };
@@ -558,7 +586,7 @@ template <typename BuildT>
 struct UpdateRootWorldBBoxFunctor
 {
     __device__
-    void operator()(size_t tid, typename TopologyBuilder<BuildT>::Data *d_data) {
+    void operator()(size_t tid, TopologyBuilderData<BuildT> *d_data) {
         // TODO: check that the correct semantics are followed in this transformation
         auto BBox = d_data->getRoot().mBBox;
         BBox.max() += 1;
@@ -569,8 +597,8 @@ struct UpdateRootWorldBBoxFunctor
 
 }// namespace topology::detail
 
-template <typename BuildT>
-inline void TopologyBuilder<BuildT>::processBBox(cudaStream_t stream)
+template<typename BuildT, typename ResourceT>
+inline void TopologyBuilder<BuildT, ResourceT>::processBBox(cudaStream_t stream)
 {
     if (data()->nodeCount[0] == 0) return; // Output grid is empty; retain empty bounding box
 
@@ -578,14 +606,14 @@ inline void TopologyBuilder<BuildT>::processBBox(cudaStream_t stream)
 
     // update and propagate bbox from leaf -> lower/parent nodes
     util::cuda::lambdaKernel<<<numBlocks(data()->nodeCount[0]), mNumThreads, 0, stream>>>(
-        data()->nodeCount[0], topology::detail::UpdateAndPropagateLeafBBoxFunctor<BuildT>(), deviceData(), static_cast<uint32_t*>(mLeafParents.deviceData()));
-    mLeafParents.clear(stream);
+        data()->nodeCount[0], topology::detail::UpdateAndPropagateLeafBBoxFunctor<BuildT>(), deviceData(), reinterpret_cast<uint32_t*>(mLeafParents.data()));
+    mLeafParents.destroy(stream);
     cudaCheckError();
 
     // propagate bbox from lower -> upper/parent node
     util::cuda::lambdaKernel<<<numBlocks(data()->nodeCount[1]), mNumThreads, 0, stream>>>(
-        data()->nodeCount[1], topology::detail::PropagateLowerBBoxFunctor<BuildT>(), deviceData(), static_cast<uint32_t*>(mLowerParents.deviceData()));
-    mLowerParents.clear(stream);
+        data()->nodeCount[1], topology::detail::PropagateLowerBBoxFunctor<BuildT>(), deviceData(), reinterpret_cast<uint32_t*>(mLowerParents.data()));
+    mLowerParents.destroy(stream);
     cudaCheckError();
 
     // propagate bbox from upper -> root/parent node
@@ -595,7 +623,7 @@ inline void TopologyBuilder<BuildT>::processBBox(cudaStream_t stream)
     // update the world-bbox in the root node
     util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, topology::detail::UpdateRootWorldBBoxFunctor<BuildT>(), deviceData());
     cudaCheckError();
-}// TopologyBuilder<BuildT>::processBBox
+}// TopologyBuilder<BuildT, ResourceT>::processBBox
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -605,7 +633,7 @@ template <typename BuildT>
 struct PostProcessGridTreeFunctor
 {
     __device__
-    void operator()(size_t tid, typename TopologyBuilder<BuildT>::Data *d_data, uint64_t* d_voxelOffsets) {
+    void operator()(size_t tid, TopologyBuilderData<BuildT> *d_data, uint64_t* d_voxelOffsets) {
         auto& grid = d_data->getGrid();
         auto& tree = grid.tree();
         auto leafCount = tree.mNodeCount[0];
@@ -616,17 +644,17 @@ struct PostProcessGridTreeFunctor
 
 }// namespace topology::detail
 
-template <typename BuildT>
-inline void TopologyBuilder<BuildT>::postProcessGridTree(cudaStream_t stream)
+template<typename BuildT, typename ResourceT>
+inline void TopologyBuilder<BuildT, ResourceT>::postProcessGridTree(cudaStream_t stream)
 {
     // Finish updates to GridData/TreeData and (optionally) update checksum
     if (data()->nodeCount[0]) // if grid is empty, the default values are correct
-        util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, topology::detail::PostProcessGridTreeFunctor<BuildT>(), deviceData(), static_cast<uint64_t*>(mVoxelOffsets.deviceData()));
+        util::cuda::lambdaKernel<<<1, 1, 0, stream>>>(1, topology::detail::PostProcessGridTreeFunctor<BuildT>(), deviceData(), reinterpret_cast<uint64_t*>(mVoxelOffsets.data()));
     cudaCheckError();
-    mVoxelOffsets.clear(stream);
+    mVoxelOffsets.destroy(stream);
 
     tools::cuda::updateChecksum((GridData*)data()->d_bufferPtr, mChecksum, stream);
-}// TopologyBuilder<BuildT>::postProcessGridTree
+}// TopologyBuilder<BuildT, ResourceT>::postProcessGridTree
 
 //-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 

nanovdb/nanovdb/tools/cuda/VoxelBlockManager.cuh

@@ -17,9 +17,17 @@
       of an OnIndexGrid, independent of occupancy.  This file provides:
       - buildVoxelBlockManager (device): constructs the firstLeafID array and
         jumpMap on the GPU from a device-resident NanoGrid.
-      - decodeInverseMaps (device): per-block SIMT decode of the inverse maps
-        (sequential active-voxel index -> leaf ID + intra-leaf voxel offset),
-        executed cooperatively across a CUDA thread block.
+      - decodeInverseMap (device): per-slot, thread-local decode of one inverse
+        map entry (sequential active-voxel index -> leaf ID + intra-leaf voxel
+        offset) into registers; no shared memory or synchronization, callable
+        from divergent threads. Block-level facts a cooperative consumer might
+        otherwise read from a materialized map are derivable directly from the
+        VBM metadata: the block's first leaf is firstLeafID, slot p starts a
+        new leaf iff jumpMap bit p is set, and the number of leaves spanned by
+        the block is 1 + the jumpMap popcount.
+      - decodeInverseMaps (device): cooperative wrapper over decodeInverseMap
+        that materializes all BlockWidth slots into shared memory, preserving
+        the pre-existing signature for consumers that read across slots.
 */
 
 #ifndef NANOVDB_VOXELBLOCKMANAGER_CUH_HAS_BEEN_INCLUDED
@@ -48,15 +56,129 @@ struct VoxelBlockManager : nanovdb::tools::VoxelBlockManagerBase<Log2BlockWidth>
     using Base::UnusedLeafIndex;
     using Base::UnusedVoxelOffset;
 
-    // The efficiency of the functions in this class are contingent on
-    // threadblock-level coordination, which manifests either as using shared
-    // memory for synchronization, or warp-level shift operations.
+    // The decode is a rank + select over the VBM's bit-vectors and is
+    // thread-local per output slot: decodeInverseMap requires no threadblock
+    // coordination. Consumers that need cross-slot information can derive it
+    // from the VBM metadata (firstLeafID + jumpMap), or materialize all slots
+    // in shared memory via the cooperative decodeInverseMaps wrapper.
 
-    /// @brief Decode the inverse maps for a single voxel block on the device.
+    /// @brief Rank one output slot into its leaf: the number of leaves that
+    /// begin at in-block positions [1, blockOffset] (bit 0 is never set),
+    /// counted via popcounts over the block's jumpMap.
+    /// Thread-local; no synchronization.
+    __device__
+    static uint32_t jumpMapRank(const uint64_t *jumpMap, const int blockOffset)
+    {
+        uint32_t leafRank = 0;
+        const int jumpWord = blockOffset >> 6;  // index into jumpMap
+        #pragma unroll
+        for (int i = 0; i < JumpMapLength; ++i) {
+            if (i < jumpWord) leafRank += util::countOn(jumpMap[i]); // count leaves before the current jump word
+            else if (i == jumpWord) {
+                // count leaves in the current jump word, masking those that are at or before blockOffset
+                leafRank += util::countOn(jumpMap[i] & ((uint64_t(2) << (blockOffset & 63)) - 1u));
+            }
+        }
+        return leafRank;
+    }
+
+    /// @brief Select the voxel with sequential index @c globalOffset inside leaf
+    /// @c leafID: find its 64-bit mask word via the leaf's precomputed 9-bit
+    /// prefix sums (mPrefixSum), then its bit within that word via __fns.
+    /// Writes the sentinels if globalOffset lies beyond the leaf's last active
+    /// voxel (i.e. beyond the last active voxel of the grid).
+    /// Thread-local; no synchronization.
+    template <class BuildT>
+    __device__
+    static typename util::enable_if<BuildTraits<BuildT>::is_index, void>::type
+    selectVoxelInLeaf(
+        const NanoGrid<BuildT> *grid,
+        const uint32_t leafID,
+        const uint64_t globalOffset,
+        uint32_t &leafIndex,
+        uint16_t &voxelOffset)
+    {
+        const auto& leafData = *grid->tree().template getFirstNode<0>()[leafID].data();
+        // 0-based rank among the leaf's actives
+        const uint64_t rankInLeaf = globalOffset - leafData.firstOffset();
+        if (rankInLeaf < leafData.valueCount()) { // if the rank is within the leaf's active voxels (bounds check)
+            int wordID = 0;
+            uint32_t activesBeforeWord = 0;
+            #pragma unroll
+            for (int candidateWord = 1; candidateWord < 8; ++candidateWord) {
+                // the number of active voxels before the candidateWord's mask word (& 0x1ffu masks to 9 bits)
+                const uint32_t cumulative = uint32_t(leafData.mPrefixSum >> (9*(candidateWord-1))) & 0x1ffu;
+                if (cumulative <= rankInLeaf) {
+                    wordID = candidateWord; // word ID of the mask word that contains the rankInLeaf-th active voxel
+                    activesBeforeWord = cumulative; // the number of active voxels before the wordID's mask word
+                }
+            }
+            uint32_t rankInWord = uint32_t(rankInLeaf) - activesBeforeWord; // active voxel's rank within the mask word (0-based)
+            // select the in-word bit position of the voxel using __fns (find n-th set bit)
+            // __fns(mask, base, k) is the hardware find-nth-set-bit intrinsic - the k-th set bit (1-based) at/after base
+            // but it's a 32-bit op while `maskWord` is 64-bit, so we need to split it into two 32-bit halves
+            const uint64_t maskWord = leafData.mValueMask.words()[wordID];
+            const uint32_t lowHalf = uint32_t(maskWord);  // low 32 bits of the mask word
+            const uint32_t lowHalfCount = __popc(lowHalf);
+            int bit;
+            // if rank is less than the number of active voxels in the low half, __fns finds the bit in the lower half
+            // otherwise, shift maskWord by 32 bits and __fns finds the bit in the upper half
+            if (rankInWord < lowHalfCount) bit = __fns(lowHalf, 0, rankInWord + 1);
+            else                           bit = 32 + __fns(uint32_t(maskWord >> 32), 0, rankInWord - lowHalfCount + 1);
+            leafIndex = leafID;
+            voxelOffset = uint16_t((wordID << 6) + bit);
+        } else { // beyond the last active voxel in the grid
+            leafIndex = UnusedLeafIndex;
+            voxelOffset = UnusedVoxelOffset;
+        }
+    }
+
+    /// @brief Decode a single inverse-map entry into registers on the device.
     ///
     /// Given the VBM metadata for one block (firstLeafID and the block's slice of
-    /// the jumpMap) and the block's base sequential offset, fills smem_leafIndex[]
-    /// and smem_voxelOffset[] in shared memory so that for each position p in
+    /// the jumpMap), the block's base sequential offset, and a slot position
+    /// blockOffset in [0, BlockWidth), computes:
+    ///   - leafIndex   = index of the leaf node containing sequential voxel
+    ///                   (blockFirstOffset + blockOffset), or UnusedLeafIndex if
+    ///                   that index is beyond the last active voxel.
+    ///   - voxelOffset = local (0..511) offset of that voxel within its leaf,
+    ///                   or UnusedVoxelOffset.
+    ///
+    /// No shared memory or synchronization; may be called from divergent threads.
+    ///
+    /// @tparam BuildT  Build type of the grid (must be an index type)
+    /// @param grid              Device-accessible OnIndex grid
+    /// @param firstLeafID       Index of the first leaf overlapping this block
+    /// @param jumpMap           Pointer to the JumpMapLength words for this block
+    /// @param blockFirstOffset  Sequential index of the first voxel in this block
+    /// @param blockOffset       Slot position within the block, in [0, BlockWidth)
+    /// @param leafIndex         Output leaf index (register)
+    /// @param voxelOffset       Output intra-leaf voxel offset (register)
+    template <class BuildT>
+    __device__
+    static typename util::enable_if<BuildTraits<BuildT>::is_index, void>::type
+    decodeInverseMap(
+        const NanoGrid<BuildT> *grid,
+        const uint32_t firstLeafID,
+        const uint64_t *jumpMap,
+        const uint64_t blockFirstOffset,
+        const int blockOffset,
+        uint32_t &leafIndex,
+        uint16_t &voxelOffset)
+    {
+        // Verify that the nodes can be accessed linearly
+        NANOVDB_ASSERT(grid->isSequential());
+        NANOVDB_ASSERT(blockOffset >= 0 && blockOffset < BlockWidth);
+
+        const uint32_t leafRank = jumpMapRank(jumpMap, blockOffset);
+        selectVoxelInLeaf(grid, firstLeafID + leafRank,
+                          blockFirstOffset + blockOffset, leafIndex, voxelOffset);
+    }
+
+    /// @brief Decode the inverse maps for a single voxel block into shared memory.
+    ///
+    /// Cooperative form of decodeInverseMap, preserving the pre-existing signature:
+    /// fills smem_leafIndex[] and smem_voxelOffset[] so that for each position p in
     /// [0, BlockWidth):
     ///   - smem_leafIndex[p]   = index of the leaf node containing sequential voxel
     ///                           (blockFirstOffset + p), or UnusedLeafIndex if that
@@ -64,8 +186,12 @@ struct VoxelBlockManager : nanovdb::tools::VoxelBlockManagerBase<Log2BlockWidth>
     ///   - smem_voxelOffset[p] = local (0..511) offset of that voxel within its leaf,
     ///                           or UnusedVoxelOffset.
     ///
-    /// Must be called by all threads in the block (uses __syncthreads internally).
-    /// Do not call from divergent threads within a thread block.
+    /// Each slot is decoded independently by decodeInverseMap; this wrapper only
+    /// materializes the results, striding over the slots so any blockDim.x <= 512
+    /// fills the maps correctly. Must be called by all threads in the block (ends
+    /// in __syncthreads); do not call from divergent threads. Consumers that only
+    /// read their own slot's result should prefer decodeInverseMap, which needs no
+    /// shared memory or barrier.
     ///
     /// @tparam BuildT  Build type of the grid (must be an index type)
     /// @param grid              Device-accessible OnIndex grid
@@ -85,57 +211,22 @@ struct VoxelBlockManager : nanovdb::tools::VoxelBlockManagerBase<Log2BlockWidth>
         uint32_t *smem_leafIndex,
         uint16_t *smem_voxelOffset)
     {
-        // Verify that the nodes can be accessed linearly
-        NANOVDB_ASSERT(grid->isSequential());
-
-        int tID = threadIdx.x;
-
-        // Count how many additional leaves (following the one indicated by firstLeafID)
-        // overlap with this voxel block
-        int nExtraLeaves = 0;
-        for (int i = 0; i < JumpMapLength; i++)
-            nExtraLeaves += util::countOn(jumpMap[i]);
-
-        // Initialize leafIndex & voxelOffset to sentinel values
-        // for blocks that extend beyond the last active voxel in the grid
-        if (tID < BlockWidth)
-            #pragma unroll
-            for (int i = 0; i < BlockWidth; i += blockDim.x) {
-                smem_leafIndex[i+tID] = UnusedLeafIndex;
-                smem_voxelOffset[i+tID] = UnusedVoxelOffset;
-            }
-        __syncthreads();
-
         NANOVDB_ASSERT(blockDim.x <= 512);
-        const auto& tree = grid->tree();
-        // Loop through all leafNodes overlapping the voxel block
-        // with all threads in threadblock working collaboratively within each leafNode
-        for (int leafID = firstLeafID; leafID <= firstLeafID + nExtraLeaves; leafID++) {
-            const auto& leaf = tree.template getFirstNode<0>()[leafID];
-            if (leaf.data()->firstOffset() >= blockFirstOffset + BlockWidth) break;
-            const Coord origin = leaf.origin();
-            for (int threadOffset = 0; threadOffset < 512; threadOffset += blockDim.x) {
-                int localOffset = threadOffset + tID;
-                auto index = leaf.data()->getValue(localOffset);
-                if ((index >= blockFirstOffset) && (index < blockFirstOffset + BlockWidth)) {
-                    int blockOffset = index - blockFirstOffset;
-                    // Write inverse map to shared memory; no collisions
-                    smem_leafIndex[blockOffset] = leafID;
-                    smem_voxelOffset[blockOffset] = localOffset;
-                }
-            }
-        }
+        for (int blockOffset = threadIdx.x; blockOffset < BlockWidth; blockOffset += blockDim.x)
+            decodeInverseMap(grid, firstLeafID, jumpMap, blockFirstOffset, blockOffset,
+                             smem_leafIndex[blockOffset], smem_voxelOffset[blockOffset]);
         __syncthreads();
     }
 
-    /// @brief Given a grid and its decoded voxel map, compute the stencil.
-    /// This function accesses shared memory but does not synchronize threads
-    /// so it may be called from divergent threads within a thread block.
-    /// offsets for a 3x3x3 box stencil.
+    /// @brief Given a grid and one decoded inverse-map entry (from
+    /// decodeInverseMap), compute the stencil indices for a 3x3x3 box stencil.
+    /// Thread-local: no shared memory, no synchronization; may be called from
+    /// divergent threads within a thread block. Leaves stencilIndices untouched
+    /// when leafIndex is UnusedLeafIndex (a slot beyond the last active voxel).
     /// @tparam BuildT Build type of the grid
-    /// @param grid
-    /// @param smem_leafIndex Leaf indices stored in shared memory
-    /// @param smem_voxelOffset Voxel offsets stored in shared memory
+    /// @param grid         Device-accessible OnIndex grid
+    /// @param leafIndex    This thread's decoded leaf index (or UnusedLeafIndex)
+    /// @param voxelOffset  This thread's decoded intra-leaf voxel offset
     /// @param stencilIndices Pointer to output stencil indices. Must have
     /// length of at least 27 (corresponding to the 3x3x3 stencil)
     template <class BuildT>
@@ -143,20 +234,18 @@ struct VoxelBlockManager : nanovdb::tools::VoxelBlockManagerBase<Log2BlockWidth>
     static typename util::enable_if<BuildTraits<BuildT>::is_index, void>::type
     computeBoxStencil(
         const NanoGrid<BuildT> *grid,
-        const uint32_t *smem_leafIndex,
-        const uint16_t *smem_voxelOffset,
+        const uint32_t leafIndex,
+        const uint16_t voxelOffset,
         uint64_t *stencilIndices)
     {
         // Verify that the nodes can be accessed linearly
         NANOVDB_ASSERT(grid->isSequential());
 
-        int tID = threadIdx.x;
         const auto& tree = grid->tree();
-        if (smem_leafIndex[tID] != UnusedLeafIndex) {
+        if (leafIndex != UnusedLeafIndex) {
             // This presumes that leaf nodes are fixed-size and sequentially accessible in memory
-            const auto& leaf = tree.template getFirstNode<0>()[ smem_leafIndex[tID] ];
-            const Coord coord = leaf.offsetToGlobalCoord( smem_voxelOffset[tID] );
-            const auto index = leaf.getValue( smem_voxelOffset[tID] );
+            const auto& leaf = tree.template getFirstNode<0>()[ leafIndex ];
+            const Coord coord = leaf.offsetToGlobalCoord( voxelOffset );
             for (int di = -1; di <= 1; di++)
             for (int dj = -1; dj <= 1; dj++)
             for (int dk = -1; dk <= 1; dk++) {

nanovdb/nanovdb/unittest/CMakeLists.txt

@@ -62,6 +62,35 @@ if(NANOVDB_USE_CUDA)
   target_link_libraries(nanovdb_test_cuda_memory_resource PRIVATE nanovdb GTest::GTest GTest::Main)
   set_target_properties(nanovdb_test_cuda_memory_resource PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
   add_test(nanovdb_cuda_memory_resource_unit_test nanovdb_test_cuda_memory_resource)
+
+  add_executable(nanovdb_test_cuda_util "TestUtilCuda.cu")
+  target_link_libraries(nanovdb_test_cuda_util PRIVATE nanovdb GTest::GTest GTest::Main)
+  set_target_properties(nanovdb_test_cuda_util PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
+  add_test(nanovdb_cuda_util_unit_test nanovdb_test_cuda_util)
+
+  # Same tests with every allocation forced through cudaMalloc/cudaFree --
+  # the configuration a pool-less vGPU relies on.
+  add_executable(nanovdb_test_cuda_util_sync "TestUtilCuda.cu")
+  target_compile_definitions(nanovdb_test_cuda_util_sync PRIVATE NANOVDB_USE_SYNC_CUDA_MALLOC)
+  target_link_libraries(nanovdb_test_cuda_util_sync PRIVATE nanovdb GTest::GTest GTest::Main)
+  set_target_properties(nanovdb_test_cuda_util_sync PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
+  add_test(nanovdb_cuda_util_sync_unit_test nanovdb_test_cuda_util_sync)
+  add_executable(nanovdb_test_cuda_buffer "TestBuffer.cu")
+  target_link_libraries(nanovdb_test_cuda_buffer PRIVATE nanovdb GTest::GTest GTest::Main)
+  set_target_properties(nanovdb_test_cuda_buffer PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
+  add_test(nanovdb_cuda_buffer_unit_test nanovdb_test_cuda_buffer)
+
+  if(WIN32)
+    # gtest's TEST macro declares each test's static test_info_ member with an
+    # unused-attribute on GCC/Clang but not under MSVC, so for tests defined
+    # inside an anonymous namespace the CUDA front end can prove it
+    # unreferenced and reports #177, which NANOVDB_CUDA_WERROR promotes to an
+    # error. Suppress #177 for the CUDA test sources that use that style.
+    foreach(_cuda_gtest nanovdb_test_cuda_buffer nanovdb_test_cuda_memory_resource
+            nanovdb_test_cuda_util nanovdb_test_cuda_util_sync)
+      target_compile_options(${_cuda_gtest} PRIVATE "$<$<COMPILE_LANGUAGE:CUDA>:--diag-suppress=177>")
+    endforeach()
+  endif()
 endif()
 
 # -----------------------------------------------------------------------------

nanovdb/nanovdb/unittest/TestMemoryResource.cu

@@ -4,13 +4,23 @@
 /// @file TestMemoryResource.cu
 ///
 /// @brief Unit tests for the CUDA memory-resource concept (cuda::DeviceResource,
-///        cuda::PinnedResource) and the cuda::TempPool / tools::cuda::PointsToGrid
-///        resource plumbing.
+///        cuda::PinnedResource) and the resource plumbing of cuda::TempPool and
+///        the tools::cuda builders (PointsToGrid, the TopologyBuilder consumers,
+///        IndexToGrid, GridStats, SignedFloodFill, addBlindData).
 
 #include <nanovdb/cuda/DeviceResource.h>
 #include <nanovdb/cuda/PinnedResource.h>
 #include <nanovdb/cuda/TempPool.h>
 #include <nanovdb/tools/cuda/PointsToGrid.cuh>
+#include <nanovdb/tools/cuda/DilateGrid.cuh>
+#include <nanovdb/tools/cuda/MergeGrids.cuh>
+#include <nanovdb/tools/cuda/PruneGrid.cuh>
+#include <nanovdb/tools/cuda/RefineGrid.cuh>
+#include <nanovdb/tools/cuda/CoarsenGrid.cuh>
+#include <nanovdb/tools/cuda/IndexToGrid.cuh>
+#include <nanovdb/tools/cuda/SignedFloodFill.cuh>
+#include <nanovdb/tools/cuda/AddBlindData.cuh>
+#include <nanovdb/tools/cuda/GridStats.cuh>
 
 #include <cuda_runtime_api.h>
 #include <gtest/gtest.h>
@@ -25,7 +35,7 @@ namespace {
 
 /// @brief Resource that counts (non-null) allocations and deallocations so
 ///        leaks can be asserted. Delegates the actual work to DeviceResource.
-struct CountingResource
+struct CountingResource : nanovdb::cuda::SyncFromAsync<CountingResource>
 {
     static constexpr size_t DEFAULT_ALIGNMENT = nanovdb::cuda::DeviceResource::DEFAULT_ALIGNMENT;
     int allocs = 0;
@@ -43,7 +53,7 @@ struct CountingResource
 
 /// @brief Resource that records the stream of every allocation/deallocation,
 ///        to verify stream-ordered teardown. Delegates work to DeviceResource.
-struct StreamRecordingResource
+struct StreamRecordingResource : nanovdb::cuda::SyncFromAsync<StreamRecordingResource>
 {
     static constexpr size_t DEFAULT_ALIGNMENT = nanovdb::cuda::DeviceResource::DEFAULT_ALIGNMENT;
     std::vector<cudaStream_t> allocStreams;
@@ -136,6 +146,57 @@ TEST(TestMemoryResource, PinnedResource_DefaultResourceRoundTrip)
 // retained stream (the stream of the most recent reallocate), not the null stream.
 //======================================================================
 
+// Synchronous-only and stateful: the shape of a vGPU or arena backend.
+struct SyncCountingResource
+{
+    static constexpr size_t DEFAULT_ALIGNMENT = nanovdb::cuda::MallocResource::DEFAULT_ALIGNMENT;
+    int allocs = 0, deallocs = 0;
+    void* allocate(size_t bytes, size_t alignment) {
+        void* p = nanovdb::cuda::MallocResource{}.allocate(bytes, alignment);
+        if (p) ++allocs;
+        return p;
+    }
+    void deallocate(void* p, size_t bytes, size_t alignment) {
+        if (p) ++deallocs;
+        nanovdb::cuda::MallocResource{}.deallocate(p, bytes, alignment);
+    }
+};
+
+static_assert(nanovdb::cuda::is_resource<nanovdb::cuda::MallocResource>::value,
+              "MallocResource must model the synchronous Resource concept");
+static_assert(!nanovdb::cuda::is_async_resource<nanovdb::cuda::MallocResource>::value,
+              "MallocResource must not claim the AsyncResource concept");
+static_assert(nanovdb::cuda::is_async_resource<
+                  nanovdb::cuda::AsyncFromSync<nanovdb::cuda::MallocResource>>::value,
+              "AsyncFromSync must lift a synchronous resource to AsyncResource");
+
+TEST(TestMemoryResource, PointsToGrid_RunsOnSynchronousResource)
+{
+    // The pool-less-device path: every scratch allocation routes through
+    // cudaMalloc/cudaFree via AsyncFromSync, never touching cudaMallocAsync.
+    // The grid handle's output buffer is the exception -- getHandle allocates
+    // it through BufferT::create, not through the injected resource.
+    using RefT  = nanovdb::cuda::ResourceRef<SyncCountingResource>;
+    using VgpuT = nanovdb::cuda::AsyncFromSync<RefT>;
+    SyncCountingResource base;
+    VgpuT res{RefT(base)};
+
+    const std::vector<nanovdb::Coord> voxels = {{0,0,0},{1,2,3},{8,8,8},{100,100,100},{-50,20,7}};
+    nanovdb::Coord* d_voxels = nullptr;
+    ASSERT_EQ(cudaMalloc(&d_voxels, voxels.size()*sizeof(nanovdb::Coord)), cudaSuccess);
+    ASSERT_EQ(cudaMemcpy(d_voxels, voxels.data(), voxels.size()*sizeof(nanovdb::Coord), cudaMemcpyHostToDevice), cudaSuccess);
+    {
+        nanovdb::tools::cuda::PointsToGrid<nanovdb::ValueOnIndex, VgpuT> converter(nanovdb::Map(1.0), cudaStream_t{0}, res);
+        auto handle = converter.getHandle(d_voxels, voxels.size());
+        auto* grid = handle.deviceGrid<nanovdb::ValueOnIndex>();
+        EXPECT_NE(grid, nullptr);
+    }
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    ASSERT_EQ(cudaFree(d_voxels), cudaSuccess);
+    EXPECT_GT(base.allocs, 0);                  // scratch really routed through the sync resource
+    EXPECT_EQ(base.allocs, base.deallocs);      // and every allocation was freed through it
+}
+
 TEST(TestMemoryResource, TempPool_FreesOnRetainedStream)
 {
     cudaStream_t s = nullptr;
@@ -209,4 +270,302 @@ TEST(TestMemoryResource, PointsToGrid_InjectedInstanceSeam)
     ASSERT_EQ(cudaFree(d_coords), cudaSuccess);
 }
 
+//======================================================================
+// A Point grid built through a custom (non-default) resource must still
+// encode its point coordinates. The encode path must not depend on the
+// resource type; only allocation does.
+//======================================================================
+
+TEST(TestMemoryResource, PointsToGrid_PointEncodedWithCustomResource)
+{
+    using BuildT = nanovdb::Point;
+    using Vec3T  = nanovdb::Vec3d;
+
+    const size_t pointCount = 256;
+    std::vector<Vec3T> points;
+    points.reserve(pointCount);
+    std::srand(98765);
+    const int max = 128, min = -max;
+    auto op = [&]() { return double(std::rand() % (max - min) + min); };
+    while (points.size() < pointCount) points.emplace_back(op(), op(), op());
+
+    Vec3T* d_points = nullptr;
+    const size_t pointSize = points.size() * sizeof(Vec3T);
+    ASSERT_EQ(cudaMalloc(&d_points, pointSize), cudaSuccess);
+    ASSERT_EQ(cudaMemcpy(d_points, points.data(), pointSize, cudaMemcpyHostToDevice), cudaSuccess);
+
+    const double voxelSize = 8.0;
+    CountingResource res;
+    nanovdb::tools::cuda::PointsToGrid<BuildT, CountingResource> converter(voxelSize, nanovdb::Vec3d(0.0), 0, res);
+    auto handle = converter.getHandle(d_points, pointCount);
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    ASSERT_EQ(cudaFree(d_points), cudaSuccess);
+
+    EXPECT_TRUE(handle.deviceData());
+    EXPECT_TRUE(handle.deviceGrid<BuildT>());
+
+    handle.deviceDownload();
+    auto* grid = handle.grid<BuildT>();
+    ASSERT_TRUE(grid);
+    EXPECT_EQ(pointCount, grid->pointCount());
+
+    // Every input point must be recoverable from the encoded per-voxel point data.
+    nanovdb::PointAccessor<Vec3T, BuildT> acc(*grid);
+    ASSERT_TRUE(acc);
+    for (size_t i = 0; i < points.size(); ++i) {
+        const nanovdb::Coord ijk = grid->worldToIndex(points[i]).round();
+        ASSERT_TRUE(acc.probeLeaf(ijk) != nullptr);
+        ASSERT_TRUE(acc.isActive(ijk));
+        const Vec3T *start = nullptr, *stop = nullptr;
+        const uint64_t count = acc.voxelPoints(ijk, start, stop);
+        ASSERT_TRUE(start);
+        ASSERT_TRUE(stop);
+        bool found = false;
+        for (uint64_t j = 0; !found && j < count; ++j)
+            found = nanovdb::math::isApproxZero<double>((points[i] - start[j]).lengthSqr());
+        EXPECT_TRUE(found);
+    }
+
+    EXPECT_GT(res.allocs, 0);
+}
+
+//======================================================================
+// TopologyBuilder consumers (DilateGrid, MergeGrids, PruneGrid, RefineGrid,
+// CoarsenGrid) route their builder's stream-ordered scratch through an
+// injected resource instance (B5, openvdb #2232).
+//======================================================================
+
+/// @brief Build a small device grid of type @c BuildT from a handful of voxels.
+///        The default resource is fine for this setup step; the CountingResource
+///        of the op under test only observes that op's scratch.
+template<typename BuildT>
+static nanovdb::GridHandle<nanovdb::cuda::DeviceBuffer>
+buildGrid(const std::vector<nanovdb::Coord>& voxels)
+{
+    nanovdb::Coord* d_voxels = nullptr;
+    cudaCheck(cudaMalloc(&d_voxels, voxels.size() * sizeof(nanovdb::Coord)));
+    cudaCheck(cudaMemcpy(d_voxels, voxels.data(), voxels.size() * sizeof(nanovdb::Coord), cudaMemcpyHostToDevice));
+    nanovdb::tools::cuda::PointsToGrid<BuildT> converter(nanovdb::Map(1.0));
+    auto handle = converter.getHandle(d_voxels, voxels.size());
+    cudaCheck(cudaFree(d_voxels));
+    return handle;
+}
+
+static nanovdb::GridHandle<nanovdb::cuda::DeviceBuffer>
+buildIndexGrid(const std::vector<nanovdb::Coord>& voxels) { return buildGrid<nanovdb::ValueOnIndex>(voxels); }
+
+TEST(TestMemoryResource, DilateGrid_InjectedResourceSeam)
+{
+    // The dilated grid handle's output buffer goes through BufferT::create, and the
+    // dual-space mProcessedRoot / builder mData go through DeviceBuffer -- none of these
+    // is observed by the CountingResource; only the builder's stream-ordered scratch is.
+    auto src = buildIndexGrid({{0,0,0},{1,2,3},{4,4,4}});
+    auto* d_srcGrid = src.deviceGrid<nanovdb::ValueOnIndex>();
+    ASSERT_NE(d_srcGrid, nullptr);
+
+    CountingResource res;
+    {
+        nanovdb::tools::cuda::DilateGrid<nanovdb::ValueOnIndex, CountingResource> op(d_srcGrid, 0, res);
+        auto handle = op.getHandle();
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_TRUE(handle.deviceData());
+    }
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    EXPECT_GT(res.allocs, 0);              // builder scratch routed through the injected instance
+    EXPECT_EQ(res.allocs, res.deallocs);   // and every allocation was freed through it
+}
+
+TEST(TestMemoryResource, MergeGrids_InjectedResourceSeam)
+{
+    // As above, the merged handle's output buffer and the dual-space mProcessedRoot / mData
+    // go through DeviceBuffer and are not counted -- only the builder's scratch is.
+    auto srcA = buildIndexGrid({{0,0,0},{1,1,1}});
+    auto srcB = buildIndexGrid({{5,5,5},{6,6,6}});
+    auto* gridA = srcA.deviceGrid<nanovdb::ValueOnIndex>();
+    auto* gridB = srcB.deviceGrid<nanovdb::ValueOnIndex>();
+    ASSERT_NE(gridA, nullptr);
+    ASSERT_NE(gridB, nullptr);
+
+    CountingResource res;
+    {
+        nanovdb::tools::cuda::MergeGrids<nanovdb::ValueOnIndex, CountingResource> op(gridA, gridB, 0, res);
+        auto handle = op.getHandle();
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_TRUE(handle.deviceData());
+    }
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    EXPECT_GT(res.allocs, 0);
+    EXPECT_EQ(res.allocs, res.deallocs);
+}
+
+TEST(TestMemoryResource, PruneGrid_InjectedResourceSeam)
+{
+    // Voxels are kept within a single 8^3 leaf, so the source grid has exactly one leaf
+    // and the retain-mask sidecar is a single all-on Mask<3>. As above, only the builder's
+    // scratch is counted; the output buffer and dual-space mProcessedRoot / mData are not.
+    auto src = buildIndexGrid({{0,0,0},{1,1,1},{2,2,2}});
+    auto* d_srcGrid = src.deviceGrid<nanovdb::ValueOnIndex>();
+    ASSERT_NE(d_srcGrid, nullptr);
+
+    nanovdb::Mask<3> hostMask;
+    hostMask.setOn(); // retain every voxel
+    nanovdb::Mask<3>* d_mask = nullptr;
+    ASSERT_EQ(cudaMalloc(&d_mask, sizeof(nanovdb::Mask<3>)), cudaSuccess);
+    ASSERT_EQ(cudaMemcpy(d_mask, &hostMask, sizeof(nanovdb::Mask<3>), cudaMemcpyHostToDevice), cudaSuccess);
+
+    CountingResource res;
+    {
+        nanovdb::tools::cuda::PruneGrid<nanovdb::ValueOnIndex, CountingResource> op(d_srcGrid, d_mask, 0, res);
+        auto handle = op.getHandle();
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_TRUE(handle.deviceData());
+    }
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    ASSERT_EQ(cudaFree(d_mask), cudaSuccess);
+    EXPECT_GT(res.allocs, 0);
+    EXPECT_EQ(res.allocs, res.deallocs);
+}
+
+TEST(TestMemoryResource, RefineGrid_InjectedResourceSeam)
+{
+    // As above, only the builder's scratch is counted; the output buffer and dual-space
+    // mProcessedRoot / mData go through DeviceBuffer and are not.
+    auto src = buildIndexGrid({{0,0,0},{1,2,3},{4,4,4}});
+    auto* d_srcGrid = src.deviceGrid<nanovdb::ValueOnIndex>();
+    ASSERT_NE(d_srcGrid, nullptr);
+
+    CountingResource res;
+    {
+        nanovdb::tools::cuda::RefineGrid<nanovdb::ValueOnIndex, CountingResource> op(d_srcGrid, 0, res);
+        auto handle = op.getHandle();
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_TRUE(handle.deviceData());
+    }
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    EXPECT_GT(res.allocs, 0);
+    EXPECT_EQ(res.allocs, res.deallocs);
+}
+
+TEST(TestMemoryResource, CoarsenGrid_InjectedResourceSeam)
+{
+    // Voxels span a 2x2x2 block of leaves (leaf DIM = 8) so coarsening is non-degenerate.
+    // As above, only the builder's scratch is counted; the output buffer and dual-space
+    // mProcessedRoot / mData go through DeviceBuffer and are not.
+    auto src = buildIndexGrid({{0,0,0},{8,0,0},{0,8,0},{0,0,8},{8,8,0},{8,0,8},{0,8,8},{8,8,8}});
+    auto* d_srcGrid = src.deviceGrid<nanovdb::ValueOnIndex>();
+    ASSERT_NE(d_srcGrid, nullptr);
+
+    CountingResource res;
+    {
+        nanovdb::tools::cuda::CoarsenGrid<nanovdb::ValueOnIndex, CountingResource> op(d_srcGrid, 0, res);
+        auto handle = op.getHandle();
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_TRUE(handle.deviceData());
+    }
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    EXPECT_GT(res.allocs, 0);
+    EXPECT_EQ(res.allocs, res.deallocs);
+}
+
+//======================================================================
+// Small builders (IndexToGrid, SignedFloodFill, GridStats, AddBlindData)
+// route their device scratch through cuda::Buffer over an injectable
+// resource instead of raw mallocAsync/freeAsync pairs (openvdb #2232).
+// Their scratch is small and fixed, so the counts are exact.
+//======================================================================
+
+TEST(TestMemoryResource, IndexToGrid_InjectedResourceSeam)
+{
+    auto src = buildIndexGrid({{0,0,0},{1,2,3},{4,4,4}});
+    auto* d_srcGrid = src.deviceGrid<nanovdb::ValueOnIndex>();
+    ASSERT_NE(d_srcGrid, nullptr);
+
+    float* d_values = nullptr;// one value per index; over-provisioned
+    cudaCheck(cudaMalloc(&d_values, 64 * sizeof(float)));
+    cudaCheck(cudaMemset(d_values, 0, 64 * sizeof(float)));
+
+    CountingResource res;
+    {
+        nanovdb::tools::cuda::IndexToGrid<nanovdb::ValueOnIndex, CountingResource> op(d_srcGrid, 0, res);
+        op.setGridName("seam");// exercises the device grid-name buffer as well
+        auto handle = op.getHandle<float>(d_values);
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_TRUE(handle.deviceData());
+        EXPECT_EQ(res.allocs, 2);            // node accessor + grid name
+    }                                        // op destroyed -> node accessor freed via res
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    EXPECT_EQ(res.allocs, res.deallocs);
+    ASSERT_EQ(cudaFree(d_values), cudaSuccess);
+}
+
+TEST(TestMemoryResource, SignedFloodFill_InjectedResourceSeam)
+{
+    auto src = buildGrid<float>({{0,0,0},{1,2,3},{4,4,4}});
+    auto* d_grid = src.deviceGrid<float>();
+    ASSERT_NE(d_grid, nullptr);
+
+    CountingResource res;
+    nanovdb::tools::cuda::SignedFloodFill<float, CountingResource> op(false, 0, res);
+    op(d_grid);
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    EXPECT_EQ(res.allocs, 1);                // the node-count scratch
+    EXPECT_EQ(res.deallocs, 1);
+}
+
+TEST(TestMemoryResource, GridStats_InjectedResourceSeam)
+{
+    auto src = buildGrid<float>({{0,0,0},{1,2,3},{4,4,4}});
+    auto* d_grid = src.deviceGrid<float>();
+    ASSERT_NE(d_grid, nullptr);
+
+    CountingResource res;
+    {   // Stats has an average, so the per-node scratch is allocated; the
+        // temporary NodeManager (storage + size scratch) routes through the
+        // resource as well
+        nanovdb::tools::cuda::GridStats<float, nanovdb::tools::Stats<float>, CountingResource> stats(0.0f, res);
+        stats.update(d_grid);
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_EQ(res.allocs, 3);
+        EXPECT_EQ(res.deallocs, 3);
+    }
+    {   // Extrema has no average: the zero-element stats buffer must not
+        // allocate, leaving only the NodeManager pair
+        nanovdb::tools::cuda::GridStats<float, nanovdb::tools::Extrema<float>, CountingResource> stats(0.0f, res);
+        stats.update(d_grid);
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_EQ(res.allocs, 5);
+        EXPECT_EQ(res.deallocs, 5);
+    }
+    {   // The free function forwards ResourceT to the per-type default instance
+        auto& def = nanovdb::cuda::default_resource<CountingResource>();
+        const int a0 = def.allocs, d0 = def.deallocs;
+        nanovdb::tools::cuda::updateGridStats<float, CountingResource>(d_grid, nanovdb::tools::StatsMode::All);
+        ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+        EXPECT_EQ(def.allocs - a0, 3);
+        EXPECT_EQ(def.deallocs - d0, 3);
+    }
+}
+
+TEST(TestMemoryResource, AddBlindData_InjectedResourceSeam)
+{
+    auto src = buildGrid<float>({{0,0,0},{1,2,3},{4,4,4}});
+    auto* d_grid = src.deviceGrid<float>();
+    ASSERT_NE(d_grid, nullptr);
+
+    float* d_blind = nullptr;
+    cudaCheck(cudaMalloc(&d_blind, 8 * sizeof(float)));
+    cudaCheck(cudaMemset(d_blind, 0, 8 * sizeof(float)));
+
+    // The free function routes through the per-type default resource instance.
+    auto& res = nanovdb::cuda::default_resource<CountingResource>();
+    const int a0 = res.allocs, d0 = res.deallocs;
+    auto handle = nanovdb::tools::cuda::addBlindData<float, float, nanovdb::cuda::DeviceBuffer, CountingResource>(
+        d_grid, d_blind, 8);
+    ASSERT_EQ(cudaStreamSynchronize(0), cudaSuccess);
+    EXPECT_TRUE(handle.deviceData());
+    EXPECT_EQ(res.allocs - a0, 1);           // the byte-size scratch
+    EXPECT_EQ(res.deallocs - d0, 1);
+    ASSERT_EQ(cudaFree(d_blind), cudaSuccess);
+}
+
 } // unnamed namespace

nanovdb/nanovdb/unittest/TestMultiGPU.cu

@@ -10,6 +10,9 @@
 #include <cuda_runtime_api.h>
 #include <gtest/gtest.h>
 #include <thread> // for std::thread
+#include <algorithm> // for std::sort, std::unique
+#include <cstdint>
+#include <vector>
 
 #include <thrust/fill.h>
 #include <thrust/universal_vector.h>
@@ -71,6 +74,46 @@ TEST(TestNanoVDBMultiGPU, RadixSort)
     }
 }
 
+/// @brief Tests the merge path split for empty partitions. An empty side
+///        yields a trivial split where everything up to the diagonal comes
+///        from the non-empty side. Runs on a single GPU.
+TEST(TestNanoVDBMultiGPU, MergePathEmptyPartition)
+{
+    using KeyT = int;
+    constexpr size_t count = 4;
+    KeyT* keys = nullptr;
+    cudaCheck(cudaMallocManaged(&keys, 2 * count * sizeof(KeyT)));
+    for (size_t i = 0; i < count; ++i) keys[i] = 10 * (int(i) + 1);              // 10 20 30 40
+    for (size_t i = 0; i < count; ++i) keys[count + i] = 10 * (int(i) + 1) + 5;  // 15 25 35 45
+    ptrdiff_t* intervals = nullptr;
+    cudaCheck(cudaMallocManaged(&intervals, 2 * sizeof(ptrdiff_t)));
+
+    const auto expectMergePath = [&](const KeyT* keys1, size_t keys1Count,
+                                     const KeyT* keys2, size_t keys2Count,
+                                     size_t diagonal, ptrdiff_t expected1, ptrdiff_t expected2) {
+        intervals[0] = intervals[1] = -1;
+        nanovdb::tools::cuda::kernels::mergePathKernel<<<1, 1>>>(keys1, keys1Count, keys2, keys2Count, intervals, intervals + 1, diagonal);
+        cudaCheck(cudaDeviceSynchronize());
+        EXPECT_EQ(intervals[0], expected1);
+        EXPECT_EQ(intervals[1], expected2);
+    };
+
+    // With an empty right partition, all elements come from keys1.
+    expectMergePath(keys, count, keys + count, 0, count / 2, ptrdiff_t(count / 2), ptrdiff_t(0));
+
+    // With an empty left partition, all elements come from keys2.
+    expectMergePath(keys, 0, keys + count, count, count / 2, ptrdiff_t(0), ptrdiff_t(count / 2));
+
+    // Both partitions empty.
+    expectMergePath(keys, 0, keys + count, 0, 0, ptrdiff_t(0), ptrdiff_t(0));
+
+    // Merged head is 10 15 20 25, so the median splits both arrays at 2.
+    expectMergePath(keys, count, keys + count, count, count, ptrdiff_t(2), ptrdiff_t(2));
+
+    cudaCheck(cudaFree(intervals));
+    cudaCheck(cudaFree(keys));
+}
+
 /// @brief Tests the correctness of multi-GPU exclusive sums against an equivalent CPU implementation
 TEST(TestNanoVDBMultiGPU, ExclusiveSum)
 {
@@ -201,6 +244,51 @@ TEST(TestNanoVDBMultiGPU, InclusiveSum)
     }
 }
 
+/// @brief Tests multi-GPU creation of a grid containing a single voxel
+TEST(TestNanoVDBMultiGPU, SingleVoxel_DistributedCudaPointsToGrid_UnifiedBuffer)
+{
+    int current = 0;
+    cudaCheck(cudaGetDevice(&current));
+
+    using BufferT = nanovdb::cuda::UnifiedBuffer;
+    using BuildT = nanovdb::ValueOnIndex;
+    const size_t voxelCount = 1;
+    nanovdb::Coord* voxels = nullptr;
+    const size_t voxelSize = voxelCount * sizeof(nanovdb::Coord);
+    cudaCheck(cudaMallocManaged(&voxels, voxelSize));
+    voxels[0] = nanovdb::Coord(1, 2, 3);
+
+    nanovdb::cuda::DeviceMesh deviceMesh;
+    nanovdb::tools::cuda::DistributedPointsToGrid<BuildT> converter(deviceMesh);
+    auto handle = converter.getHandle(voxels, voxelCount, BufferT());
+
+    EXPECT_TRUE(handle.deviceData());// grid exists on the GPU
+    EXPECT_TRUE(handle.deviceGrid<BuildT>());
+    EXPECT_FALSE(handle.deviceGrid<int>(0));
+    EXPECT_TRUE(handle.deviceGrid<BuildT>(0));
+    EXPECT_FALSE(handle.deviceGrid<BuildT>(1));
+    EXPECT_TRUE(handle.data());// grid also exists on the CPU
+
+    auto* grid = handle.grid<BuildT>();// grid also exists on the CPU
+    ASSERT_TRUE(grid);
+    handle.deviceDownload();// creates a copy on the CPU
+    EXPECT_TRUE(handle.deviceData());
+    EXPECT_TRUE(handle.data());
+    auto* data = handle.gridData();
+    EXPECT_TRUE(data);
+    grid = handle.grid<BuildT>();
+    ASSERT_TRUE(grid);
+    EXPECT_EQ(1u, grid->activeVoxelCount());
+    EXPECT_EQ(nanovdb::Vec3d(1.0), grid->voxelSize());
+
+    auto acc = grid->getAccessor();
+    EXPECT_TRUE(acc.isActive(voxels[0]));
+    EXPECT_GT(acc.getValue(voxels[0]), 0u);
+
+    cudaCheck(cudaFree(voxels));
+    cudaSetDevice(current); // restore device so subsequent tests don't fail
+}// SingleVoxel_DistributedCudaPointsToGrid_UnifiedBuffer
+
 /// @brief Tests multi-GPU creation of grids for a single dense leaf
 TEST(TestNanoVDBMultiGPU, DenseLeaf_DistributedCudaPointsToGrid_UnifiedBuffer)
 {
@@ -243,7 +331,7 @@ TEST(TestNanoVDBMultiGPU, DenseLeaf_DistributedCudaPointsToGrid_UnifiedBuffer)
     EXPECT_TRUE(data);
     grid = handle.grid<BuildT>();
     EXPECT_TRUE(grid);
-    EXPECT_TRUE(grid->activeVoxelCount() == 512);
+    EXPECT_EQ(voxelCount, grid->activeVoxelCount());
     EXPECT_EQ(nanovdb::Vec3d(1.0), grid->voxelSize());
 
     cudaCheck(cudaFree(voxels));
@@ -415,3 +503,118 @@ TEST(TestNanoVDBMultiGPU, ManyTiles_DistributedCudaPointsToGrid)
     cudaCheck(cudaFree(voxels));
     cudaSetDevice(current);
 }// ManyTiles_DistributedCudaPointsToGrid
+
+/// @brief Regression test for the multi-GPU shared-tile race. All coordinates
+///        fall inside a single upper-node tile (each upper node spans 4096
+///        voxels per axis, so ijk in [0, 4095] maps to tile (0,0,0)), yet they
+///        populate many lower and leaf nodes. With the default even initial
+///        split this one tile is shared across every GPU, so the builder must
+///        consolidate it onto a single device. A cross-device race in leaf
+///        construction would drop a device's contribution and undercount the
+///        active voxels, which the exact-count assertion below catches
+///        deterministically.
+TEST(TestNanoVDBMultiGPU, SingleUpperNode_DistributedCudaPointsToGrid_UnifiedBuffer)
+{
+    int current = 0;
+    cudaCheck(cudaGetDevice(&current));
+
+    using BufferT = nanovdb::cuda::UnifiedBuffer;
+    using BuildT = nanovdb::ValueOnIndex;
+
+    const size_t inputCount = 1 << 18;// 262144
+    nanovdb::Coord* voxels = nullptr;
+    cudaCheck(cudaMallocManaged(&voxels, inputCount * sizeof(nanovdb::Coord)));
+    std::srand(24680);
+    auto op = [](){ return rand() % 4096; };// stays within a single upper node
+    for (size_t i = 0; i < inputCount; ++i)
+        voxels[i] = nanovdb::Coord(op(), op(), op());
+
+    // Deterministic expected active-voxel count (the input may contain duplicates).
+    std::vector<uint64_t> packed(inputCount);
+    for (size_t i = 0; i < inputCount; ++i)
+        packed[i] = (uint64_t(voxels[i][0]) << 24) | (uint64_t(voxels[i][1]) << 12) | uint64_t(voxels[i][2]);
+    std::sort(packed.begin(), packed.end());
+    const size_t uniqueCount = static_cast<size_t>(std::unique(packed.begin(), packed.end()) - packed.begin());
+
+    nanovdb::cuda::DeviceMesh deviceMesh;
+    nanovdb::tools::cuda::DistributedPointsToGrid<BuildT> converter(deviceMesh);
+    auto handle = converter.getHandle(voxels, inputCount);
+
+    EXPECT_TRUE(handle.deviceData());
+    EXPECT_TRUE(handle.deviceGrid<BuildT>());
+    handle.deviceDownload();
+    auto *grid = handle.grid<BuildT>();
+    EXPECT_TRUE(grid);
+    EXPECT_EQ(nanovdb::Vec3d(1.0), grid->voxelSize());
+    // The input occupies exactly one upper-node tile, so the shared-tile
+    // consolidation path is exercised.
+    EXPECT_EQ(1u, grid->tree().nodeCount(2));
+    // Every unique input voxel must be active exactly once.
+    EXPECT_EQ(static_cast<uint64_t>(uniqueCount), grid->activeVoxelCount());
+
+    nanovdb::util::forEach(0, inputCount, 1, [&](const nanovdb::util::Range1D &r){
+        auto acc = grid->getAccessor();
+        for (size_t i=r.begin(); i!=r.end(); ++i) {
+            const nanovdb::Coord &ijk = voxels[i];
+            EXPECT_TRUE(acc.probeLeaf(ijk)!=nullptr);
+            EXPECT_TRUE(acc.isActive(ijk));
+            EXPECT_TRUE(acc.getValue(ijk) > 0u);
+        }
+    });
+
+    cudaCheck(cudaFree(voxels));
+    cudaSetDevice(current);
+}// SingleUpperNode_DistributedCudaPointsToGrid_UnifiedBuffer
+
+/// @brief Cross-checks the distributed builder against the trusted single-GPU
+///        PointsToGrid on an input that forces a single tile to be split across
+///        devices. Index assignment order may differ between the two builders,
+///        so we compare topology (node counts, active-voxel count) and voxel
+///        occupancy rather than the ValueOnIndex indices themselves.
+TEST(TestNanoVDBMultiGPU, MatchesSingleGpu_DistributedCudaPointsToGrid)
+{
+    int current = 0;
+    cudaCheck(cudaGetDevice(&current));
+
+    using BufferT = nanovdb::cuda::UnifiedBuffer;
+    using BuildT = nanovdb::ValueOnIndex;
+
+    const size_t inputCount = 1 << 17;// 131072, all within a single upper node
+    nanovdb::Coord* voxels = nullptr;
+    cudaCheck(cudaMallocManaged(&voxels, inputCount * sizeof(nanovdb::Coord)));
+    std::srand(1357);
+    auto op = [](){ return rand() % 4096; };
+    for (size_t i = 0; i < inputCount; ++i)
+        voxels[i] = nanovdb::Coord(op(), op(), op());
+
+    nanovdb::cuda::DeviceMesh deviceMesh;
+    nanovdb::tools::cuda::DistributedPointsToGrid<BuildT> converter(deviceMesh);
+    auto distributedHandle = converter.getHandle(voxels, inputCount);
+    distributedHandle.deviceDownload();
+    auto *distributedGrid = distributedHandle.grid<BuildT>();
+    EXPECT_TRUE(distributedGrid);
+
+    cudaSetDevice(current);
+    auto referenceHandle = nanovdb::tools::cuda::voxelsToGrid<BuildT, nanovdb::Coord*, BufferT>(voxels, inputCount);
+    referenceHandle.deviceDownload();
+    auto *referenceGrid = referenceHandle.grid<BuildT>();
+    EXPECT_TRUE(referenceGrid);
+
+    EXPECT_EQ(referenceGrid->activeVoxelCount(), distributedGrid->activeVoxelCount());
+    EXPECT_EQ(referenceGrid->tree().nodeCount(0), distributedGrid->tree().nodeCount(0));
+    EXPECT_EQ(referenceGrid->tree().nodeCount(1), distributedGrid->tree().nodeCount(1));
+    EXPECT_EQ(referenceGrid->tree().nodeCount(2), distributedGrid->tree().nodeCount(2));
+
+    nanovdb::util::forEach(0, inputCount, 1, [&](const nanovdb::util::Range1D &r){
+        auto distributedAcc = distributedGrid->getAccessor();
+        auto referenceAcc = referenceGrid->getAccessor();
+        for (size_t i=r.begin(); i!=r.end(); ++i) {
+            const nanovdb::Coord &ijk = voxels[i];
+            EXPECT_TRUE(referenceAcc.isActive(ijk));
+            EXPECT_TRUE(distributedAcc.isActive(ijk));
+        }
+    });
+
+    cudaCheck(cudaFree(voxels));
+    cudaSetDevice(current);
+}// MatchesSingleGpu_DistributedCudaPointsToGrid

nanovdb/nanovdb/unittest/TestNanoVDB.cc

@@ -1623,6 +1623,107 @@ TEST_F(TestNanoVDB, HDDA)
 #endif
 } // HDDA
 
+TEST_F(TestNanoVDB, GetDimAndActive)
+{
+    // Verifies that the fused ReadAccessor::getDimAndActive matches the result
+    // of the two separate queries it replaces -- getDim(ijk, ray) for the dim
+    // half and isActive(ijk) for the active half -- under both policy tags.
+    using GridT = nanovdb::FloatGrid;
+    using CoordT = nanovdb::Coord;
+    using RayT = nanovdb::math::Ray<float>;
+
+    const int    R = 50;
+    const double radius = double(R), voxelSize = 1.0, halfWidth = 3.0;
+    auto handle = nanovdb::tools::createLevelSetSphere(radius, nanovdb::Vec3d(0),
+                                                       voxelSize, halfWidth,
+                                                       nanovdb::Vec3d(0), "sphere");
+    auto* grid = handle.grid<float>();
+    ASSERT_TRUE(grid);
+    EXPECT_TRUE(grid->isLevelSet());
+
+    // The createLevelSetSphere path runs GridStats with delta=0, which leaves
+    // the InternalNode skip-flag (mFlags bit 0) clear everywhere. Re-run stats
+    // with delta=voxelSize so nodes whose value range falls entirely outside
+    // the [-delta, +delta] band get the skip-flag set; this is what makes
+    // getDim short-circuit above the leaf and exercises the skip-flag handling
+    // in both getDimAndActive policies below.
+    nanovdb::tools::GridStats<GridT> stats;
+    stats.update(*grid, float(voxelSize));
+
+    // The ray only feeds getDim's (currently disabled) bbox-intersection early
+    // out, so its direction does not affect any result checked here.
+    const RayT ray(RayT::Vec3T(0.0f), RayT::Vec3T(1.0f, 0.0f, 0.0f));
+
+    { // Sweep a box straddling interior, surface and exterior so the descent
+      // terminates at background, tile values, skip-flagged leaves and active
+      // leaf voxels. A single accessor is reused: its cache only changes the
+      // node a descent starts from, never the value it returns.
+        const nanovdb::CoordBBox bbox(CoordT(-R - 6), CoordT(R + 6));
+        auto acc = grid->getAccessor();
+        uint64_t nActive = 0, nLeafResolved = 0;
+        uint64_t exactDimBad = 0, exactActiveBad = 0, leafDimBad = 0, leafActiveBad = 0;
+        for (auto it = bbox.begin(); it; ++it) {
+            const CoordT   ijk    = *it;
+            const uint32_t dim    = acc.getDim(ijk, ray);
+            const bool     active = acc.isActive(ijk);
+            if (active) ++nActive;
+
+            // ActiveExact: both halves must equal the separate calls exactly.
+            const auto e = acc.getDimAndActive<nanovdb::ActiveExact>(ijk, ray);
+            if (e.dim() != dim) ++exactDimBad;
+            if (e.active() != active) ++exactActiveBad;
+
+            // ActiveOnLeafOnly: dim still matches getDim; active is only
+            // contractually defined when dim==1 (a true leaf-voxel descent).
+            const auto l = acc.getDimAndActive<nanovdb::ActiveOnLeafOnly>(ijk, ray);
+            if (l.dim() != dim) ++leafDimBad;
+            if (l.dim() == 1) {
+                ++nLeafResolved;
+                if (l.active() != active) ++leafActiveBad;
+            }
+        }
+        EXPECT_EQ(0u, exactDimBad);
+        EXPECT_EQ(0u, exactActiveBad);
+        EXPECT_EQ(0u, leafDimBad);
+        EXPECT_EQ(0u, leafActiveBad);
+        EXPECT_GT(nActive, 0u);       // we actually visited active voxels
+        EXPECT_GT(nLeafResolved, 0u); // ... and descents that reached leaf voxels
+    }
+
+    { // For a sphere the active narrow band is too thin to fill an InternalNode,
+      // so no skip-flagged internal node ever sits above an active voxel -- the
+      // headline divergence (getDim short-circuits at an internal node while
+      // active is still true) cannot arise from the geometry alone. Force it by
+      // setting the skip-flag on the lower node above a known-active voxel.
+        CoordT ijk0;
+        bool   found = false;
+        for (int x = R - 4; x <= R + 4 && !found; ++x) {
+            const CoordT c(x, 0, 0);
+            if (grid->getAccessor().isActive(c)) { ijk0 = c; found = true; }
+        }
+        ASSERT_TRUE(found);
+
+        const auto* lower = grid->getAccessor().get<nanovdb::GetLower<float>>(ijk0);
+        ASSERT_TRUE(lower);
+        const uint32_t lowerDim = lower->dim();
+        EXPECT_GT(lowerDim, 1u);
+        const_cast<nanovdb::NanoLower<float>::DataType*>(lower->data())->mFlags |= uint64_t(1u);
+
+        // Fresh accessors per query so a cached leaf can't bypass the lower-node
+        // descent where the skip-flag now lives.
+        EXPECT_EQ(lowerDim, grid->getAccessor().getDim(ijk0, ray));
+        EXPECT_TRUE(grid->getAccessor().isActive(ijk0));
+
+        const auto e = grid->getAccessor().getDimAndActive<nanovdb::ActiveExact>(ijk0, ray);
+        EXPECT_EQ(lowerDim, e.dim()); // dim short-circuits like getDim
+        EXPECT_TRUE(e.active());      // ... but active descends past the skip-flag
+
+        const auto l = grid->getAccessor().getDimAndActive<nanovdb::ActiveOnLeafOnly>(ijk0, ray);
+        EXPECT_EQ(lowerDim, l.dim());
+        EXPECT_FALSE(l.active());     // short-circuits: active is "unspecified" (false)
+    }
+} // GetDimAndActive
+
 TEST_F(TestNanoVDB, Mask)
 {
     using MaskT = nanovdb::Mask<3>;

nanovdb/nanovdb/unittest/TestNanoVDB.cu

@@ -2594,6 +2594,83 @@ TEST(TestNanoVDBCUDA, cudaAddBlindData)
     for (size_t i=0; i<num_points; ++i) EXPECT_EQ(blind2[i], dataPtr2[i]);
 }// cudaAddBlindData
 
+TEST(TestNanoVDBCUDA, cudaAddBlindDataFromMultiGridSource)
+{
+    using BufferT = nanovdb::cuda::DeviceBuffer;
+    // two grids merged into one device buffer give source grids whose headers
+    // carry a non-trivial grid index/count, which the output must not inherit
+    std::vector<nanovdb::GridHandle<BufferT>> handles;
+    handles.emplace_back(nanovdb::tools::createLevelSetSphere<float, BufferT>(20, nanovdb::Vec3d(0), 1, 3, nanovdb::Vec3d(0), "a"));
+    handles.emplace_back(nanovdb::tools::createLevelSetSphere<float, BufferT>(30, nanovdb::Vec3d(0), 1, 3, nanovdb::Vec3d(0), "b"));
+    for (auto &h : handles) h.deviceUpload();
+    auto merged = nanovdb::cuda::mergeGridHandles<BufferT, std::vector>(handles);
+    EXPECT_EQ(2u, merged.gridCount());
+
+    const size_t num_values = 2;
+    float *d_blind = nullptr, blind[num_values] = {1.2f, 3.0f};
+    cudaCheck(cudaMalloc(&d_blind, num_values * sizeof(float)));
+    cudaCheck(cudaMemcpy(d_blind, blind, num_values * sizeof(float), cudaMemcpyHostToDevice));
+    for (uint32_t n = 0; n < merged.gridCount(); ++n) {// source header: mGridIndex=n, mGridCount=2
+        auto *d_grid = merged.deviceGrid<float>(n);
+        EXPECT_TRUE(d_grid);
+        auto out = nanovdb::tools::cuda::addBlindData(d_grid, d_blind, num_values);
+        EXPECT_EQ(1u, out.gridCount());
+        out.deviceDownload();
+        auto *gridData = out.gridData();
+        EXPECT_TRUE(gridData);
+        EXPECT_EQ(0u, gridData->mGridIndex);
+        EXPECT_EQ(1u, gridData->mGridCount);
+        auto *grid = out.grid<float>();
+        EXPECT_TRUE(grid);
+        EXPECT_EQ(1u, grid->blindDataCount());
+    }
+    cudaCheck(cudaFree(d_blind));
+}// cudaAddBlindDataFromMultiGridSource
+
+TEST(TestNanoVDBCUDA, cudaIndexToGridFromMultiGridSource)
+{
+    using BufferT = nanovdb::cuda::DeviceBuffer;
+    auto floatHdl = nanovdb::tools::createLevelSetSphere<float, nanovdb::HostBuffer>(20, nanovdb::Vec3d(0), 1, 3, nanovdb::Vec3d(0), "sphere");
+    auto *floatGrid = floatHdl.grid<float>();
+    EXPECT_TRUE(floatGrid);
+    std::vector<nanovdb::GridHandle<BufferT>> idxHandles;
+    idxHandles.emplace_back(nanovdb::tools::createNanoGrid<nanovdb::FloatGrid, nanovdb::ValueIndex, BufferT>(*floatGrid, 0u, false, false));
+    idxHandles.emplace_back(nanovdb::tools::createNanoGrid<nanovdb::FloatGrid, nanovdb::ValueIndex, BufferT>(*floatGrid, 0u, false, false));
+    auto *idxGrid = idxHandles[0].grid<nanovdb::ValueIndex>();
+    EXPECT_TRUE(idxGrid);
+
+    // host-side value list for the (identical) index grids
+    std::vector<float> values(idxGrid->valueCount(), 0.0f);
+    values[0] = floatGrid->tree().root().background();
+    auto acc = floatGrid->getAccessor();
+    for (auto it = floatGrid->indexBBox().begin(); it; ++it) {
+        const uint64_t idx = idxGrid->tree().getValue(*it);
+        if (idx > 0) values[idx] = acc.getValue(*it);
+    }
+    float *d_values = nullptr;
+    cudaCheck(cudaMalloc(&d_values, values.size() * sizeof(float)));
+    cudaCheck(cudaMemcpy(d_values, values.data(), values.size() * sizeof(float), cudaMemcpyHostToDevice));
+
+    for (auto &h : idxHandles) h.deviceUpload();
+    auto merged = nanovdb::cuda::mergeGridHandles<BufferT, std::vector>(idxHandles);
+    EXPECT_EQ(2u, merged.gridCount());
+    for (uint32_t n = 0; n < merged.gridCount(); ++n) {// source header: mGridIndex=n, mGridCount=2
+        auto *d_idxGrid = merged.deviceGrid<nanovdb::ValueIndex>(n);
+        EXPECT_TRUE(d_idxGrid);
+        auto out = nanovdb::tools::cuda::indexToGrid<float>(d_idxGrid, d_values);
+        EXPECT_EQ(1u, out.gridCount());
+        out.deviceDownload();
+        auto *gridData = out.gridData();
+        EXPECT_TRUE(gridData);
+        EXPECT_EQ(0u, gridData->mGridIndex);
+        EXPECT_EQ(1u, gridData->mGridCount);
+        auto *outGrid = out.grid<float>();
+        EXPECT_TRUE(outGrid);
+        EXPECT_EQ(floatGrid->indexBBox(), outGrid->indexBBox());
+    }
+    cudaCheck(cudaFree(d_values));
+}// cudaIndexToGridFromMultiGridSource
+
 TEST(TestNanoVDBCUDA, testGridHandleCopy)
 {
     auto cudaHandle = nanovdb::tools::createLevelSetSphere<float, nanovdb::cuda::DeviceBuffer>(100);
@@ -3465,6 +3542,8 @@ __global__ void testComputeStencilNeighborsKernel(
     uint64_t *jumpMap = jumpMapArray + JumpMapLength * bID;
     int firstOffset = 1;
     int blockFirstOffset = firstOffset + bID * BlockWidth;
+    // Decode through the cooperative wrapper (which exercises the per-slot
+    // decodeInverseMap internally), then resolve this thread's own slot.
     __shared__ uint32_t leafIndex[BlockWidth];
     __shared__ uint16_t voxelOffset[BlockWidth];
 
@@ -3473,8 +3552,7 @@ __global__ void testComputeStencilNeighborsKernel(
 
     uint64_t localNeighbors[27] = {};
     nanovdb::tools::cuda::VoxelBlockManager<Log2BlockWidth>::computeBoxStencil(
-        grid, &leafIndex[0], &voxelOffset[0], localNeighbors);
-    __syncthreads();
+        grid, leafIndex[tID], voxelOffset[tID], localNeighbors);
 
     using StencilNeighborsType = uint64_t (*)[27];
     auto stencilNeighbors = reinterpret_cast<StencilNeighborsType>(stencilNeighborsArray+27*BlockWidth*bID);
@@ -3607,6 +3685,362 @@ TEST(TestNanoVDBCUDA, DilateInjectPrune_ValueOnIndex)
     EXPECT_EQ(inputHandle.grid<BuildT>()->mChecksum.full(), prunedHandle.grid<BuildT>()->mChecksum.full());
 }// DilateInjectPrune_ValueOnIndex
 
+// Busy-wait on whatever stream it is launched on; used to occupy the default
+// stream so that a kernel mistakenly launched there (instead of the caller's
+// stream) is reordered relative to the rest of the operation.
+__global__ void streamBusyWaitKernel(unsigned long long cycles)
+{
+    const unsigned long long t0 = clock64();
+    while (clock64() - t0 < cycles) {}
+}
+
+// Regression test: the NN_FACE_EDGE_VERTEX (26-neighbour) leaf dilation kernel
+// must run on the caller's stream. It previously launched on the default stream,
+// silently corrupting the result when the caller used a non-blocking stream.
+// Dilating on the default stream and on a non-blocking stream (while the default
+// stream is occupied) must produce identical output.
+TEST(TestNanoVDBCUDA, NonBlockingStreamDilate_ValueOnIndex)
+{
+    using BuildT = nanovdb::ValueOnIndex;
+    std::vector<nanovdb::Coord> pts;
+    for (int i = 0; i < 64; ++i) pts.emplace_back(i * 3, (i * 7) % 40, (i * 13) % 50);
+    pts.emplace_back(127, 127, 127);
+    auto inBuf = nanovdb::cuda::DeviceBuffer::create(pts.size() * sizeof(nanovdb::Coord), nullptr, false);
+    cudaCheck(cudaMemcpy(inBuf.deviceData(), pts.data(), pts.size() * sizeof(nanovdb::Coord), cudaMemcpyHostToDevice));
+    nanovdb::tools::cuda::PointsToGrid<BuildT> conv;
+    conv.setChecksum(nanovdb::CheckMode::Full);
+    auto inHandle = conv.getHandle(static_cast<nanovdb::Coord*>(inBuf.deviceData()), pts.size());
+    auto* inGrid = inHandle.deviceGrid<BuildT>();
+    EXPECT_TRUE(inGrid);
+    cudaCheck(cudaDeviceSynchronize());
+
+    auto dilateOn = [&](cudaStream_t stream, bool occupyDefault) {
+        if (occupyDefault) {// keep the default stream busy for ~20 ms
+            int clockKHz = 0, dev = 0;
+            cudaCheck(cudaGetDevice(&dev));
+            cudaCheck(cudaDeviceGetAttribute(&clockKHz, cudaDevAttrClockRate, dev));
+            streamBusyWaitKernel<<<1, 1, 0, 0>>>(static_cast<unsigned long long>(clockKHz) * 20ull);
+            cudaCheck(cudaGetLastError());
+        }
+        nanovdb::tools::cuda::DilateGrid<BuildT> dilator(inGrid, stream);
+        dilator.setOperation(nanovdb::tools::morphology::NN_FACE_EDGE_VERTEX);
+        dilator.setChecksum(nanovdb::CheckMode::Full);
+        dilator.setVerbose(0);
+        auto handle = dilator.getHandle();
+        cudaCheck(cudaStreamSynchronize(stream));
+        const auto treeData = nanovdb::util::cuda::DeviceGridTraits<BuildT>::getTreeData(handle.deviceGrid<BuildT>());
+        handle.deviceDownload();
+        return std::make_pair(treeData.mVoxelCount, handle.grid<BuildT>()->mChecksum.full());
+    };
+
+    const auto reference = dilateOn(static_cast<cudaStream_t>(0), false);
+    cudaCheck(cudaDeviceSynchronize());
+    cudaStream_t nb = nullptr;
+    cudaCheck(cudaStreamCreateWithFlags(&nb, cudaStreamNonBlocking));
+    const auto candidate = dilateOn(nb, /*occupyDefault=*/true);
+    cudaCheck(cudaStreamSynchronize(nb));
+    cudaCheck(cudaDeviceSynchronize());// drain the default-stream busy-wait so it can't leak into later tests
+    cudaCheck(cudaStreamDestroy(nb));
+
+    EXPECT_EQ(reference.first, candidate.first);   // identical active-voxel count
+    EXPECT_EQ(reference.second, candidate.second); // identical full checksum
+}// NonBlockingStreamDilate_ValueOnIndex
+
+// Cross-stream determinism coverage: signedFloodFill must run entirely on the caller's stream.
+// Mirrors the dilation test above: repair the same corrupted level set once on the default
+// stream and once on a non-blocking stream while the default stream is occupied, reading each
+// result back on the stream that produced it (no default-stream rescue), and require identical
+// output. Node passes escaping to the occupied default stream show up as an unrepaired or
+// partially repaired grid.
+//
+// Scope note: this locks the observable contract but cannot discriminate processRoot's internal
+// ordering on constructible inputs - its root-tile scanline repair only does work when interior
+// root-level tiles exist, which needs a level set thousands of voxels across, and the pre-fix
+// code was accidentally host-synchronous for other topologies (a blocking legacy-stream memcpy).
+TEST(TestNanoVDBCUDA, NonBlockingStreamSignedFloodFill)
+{
+    using BufferT = nanovdb::cuda::DeviceBuffer;
+    auto runOn = [](cudaStream_t stream, bool occupyDefault) {
+        auto hdl = nanovdb::tools::createLevelSetSphere<float, BufferT>(100);
+        auto* grid = hdl.grid<float>();
+        auto acc = grid->getAccessor();
+        using OpT = nanovdb::SetVoxel<float>;
+        acc.set<OpT>(nanovdb::Coord(103,0,0), -1.0f);// flip sign and value of an inactive voxel
+        acc.set<OpT>(nanovdb::Coord( 97,0,0),  1.0f);// (the corruption CudaSignedFloodFill uses)
+        hdl.deviceUpload(0, stream, true);
+        if (occupyDefault) {// keep the default stream busy for ~20 ms
+            int clockKHz = 0, dev = 0;
+            cudaCheck(cudaGetDevice(&dev));
+            cudaCheck(cudaDeviceGetAttribute(&clockKHz, cudaDevAttrClockRate, dev));
+            streamBusyWaitKernel<<<1, 1, 0, 0>>>(static_cast<unsigned long long>(clockKHz) * 20ull);
+            cudaCheck(cudaGetLastError());
+        }
+        nanovdb::tools::cuda::signedFloodFill(hdl.deviceGrid<float>(), false, stream);
+        hdl.deviceDownload(0, stream, true);
+        auto* out = hdl.grid<float>();
+        auto outAcc = out->getAccessor();
+        EXPECT_EQ( 3.0f, outAcc(103,0,0));
+        EXPECT_EQ( 0.0f, outAcc(100,0,0));
+        EXPECT_EQ(-3.0f, outAcc( 97,0,0));
+        return out->mChecksum.full();
+    };
+    const uint64_t reference = runOn(static_cast<cudaStream_t>(0), false);
+    cudaCheck(cudaDeviceSynchronize());
+    cudaStream_t nb = nullptr;
+    cudaCheck(cudaStreamCreateWithFlags(&nb, cudaStreamNonBlocking));
+    const uint64_t candidate = runOn(nb, /*occupyDefault=*/true);
+    cudaCheck(cudaStreamSynchronize(nb));
+    cudaCheck(cudaDeviceSynchronize());// drain the default-stream busy-wait so it can't leak into later tests
+    cudaCheck(cudaStreamDestroy(nb));
+    EXPECT_EQ(reference, candidate);
+}// NonBlockingStreamSignedFloodFill
+
+// Regression test: the grid-name copy must not over-read the source string.
+// It previously copied a fixed MaxNameSize bytes from the (possibly shorter)
+// std::string, over-reading host heap into the grid; now it copies only
+// min(name.size()+1, MaxNameSize) bytes. Empty, short, and maximum-length
+// names must round-trip.
+TEST(TestNanoVDBCUDA, GridName_CudaPointsToGrid)
+{
+    using BuildT = nanovdb::ValueOnIndex;
+    const std::vector<nanovdb::Coord> pts{ {0, 0, 0}, {1, 2, 3} };
+    auto build = [&](const std::string& name) {
+        auto buf = nanovdb::cuda::DeviceBuffer::create(pts.size() * sizeof(nanovdb::Coord), nullptr, false);
+        cudaCheck(cudaMemcpy(buf.deviceData(), pts.data(), pts.size() * sizeof(nanovdb::Coord), cudaMemcpyHostToDevice));
+        nanovdb::tools::cuda::PointsToGrid<BuildT> conv;
+        conv.setGridName(name);
+        conv.setChecksum(nanovdb::CheckMode::Full);
+        auto h = conv.getHandle(static_cast<nanovdb::Coord*>(buf.deviceData()), pts.size());
+        h.deviceDownload();
+        return std::string(h.template grid<BuildT>()->gridName());
+    };
+    EXPECT_EQ(std::string(""),    build(""));       // empty name
+    EXPECT_EQ(std::string("rho"), build("rho"));    // short name
+    // Maximum name that fits with a null terminator (MaxNameSize includes it).
+    const std::string maxName(nanovdb::GridData::MaxNameSize - 1, 'x');
+    EXPECT_EQ(maxName, build(maxName));
+    // Over-long name (>= MaxNameSize) must be truncated to MaxNameSize-1 chars
+    // and stay null-terminated - never a non-terminated field that gridName()
+    // would over-read. The result is the max-length string, and reading it back
+    // as a C-string does not exceed MaxNameSize.
+    const std::string tooLong(nanovdb::GridData::MaxNameSize + 37, 'y');
+    const std::string got = build(tooLong);
+    EXPECT_EQ(size_t(nanovdb::GridData::MaxNameSize - 1), got.size());
+    EXPECT_EQ(std::string(nanovdb::GridData::MaxNameSize - 1, 'y'), got);
+}// GridName_CudaPointsToGrid
+
+// Regression test: PointsToGrid output must be byte-deterministic. Alignment
+// padding and stats fields previously carried recycled pool bytes, so the output
+// (and its full checksum) was nondeterministic; the buffer-wide zero-init fixes
+// this. Two independent builds of the same input must have identical full
+// checksums.
+TEST(TestNanoVDBCUDA, DeterministicOutput_CudaPointsToGrid)
+{
+    using BuildT = nanovdb::ValueOnIndex;
+    std::vector<nanovdb::Coord> pts;
+    for (int i = 0; i < 256; ++i) pts.emplace_back((i * 5) % 97, (i * 11) % 53, (i * 17) % 71);
+    auto buildChecksum = [&]() {
+        auto buf = nanovdb::cuda::DeviceBuffer::create(pts.size() * sizeof(nanovdb::Coord), nullptr, false);
+        cudaCheck(cudaMemcpy(buf.deviceData(), pts.data(), pts.size() * sizeof(nanovdb::Coord), cudaMemcpyHostToDevice));
+        nanovdb::tools::cuda::PointsToGrid<BuildT> conv;
+        conv.setChecksum(nanovdb::CheckMode::Full);
+        auto h = conv.getHandle(static_cast<nanovdb::Coord*>(buf.deviceData()), pts.size());
+        h.deviceDownload();
+        return h.template grid<BuildT>()->mChecksum.full();
+    };
+    EXPECT_EQ(buildChecksum(), buildChecksum());
+}// DeterministicOutput_CudaPointsToGrid
+
+// Smoke test: repeated allocate/use/free on a non-blocking stream must complete without error.
+// Note this is only a smoke test - DeviceBufferMultiStreamFreeOrdering below is what actually
+// discriminates a premature free (compute-sanitizer memcheck on this path is a further gate).
+TEST(TestNanoVDBCUDA, DeviceBufferNonBlockingStreamLifetime)
+{
+    cudaStream_t stream = nullptr;
+    cudaCheck(cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking));
+    for (int i = 0; i < 16; ++i) {
+        auto buf = nanovdb::cuda::DeviceBuffer::create(size_t(1) << 20, nullptr, 0, stream);// 1 MiB on stream
+        EXPECT_TRUE(buf.deviceData());
+        cudaCheck(cudaMemsetAsync(buf.deviceData(), i & 0xff, buf.size(), stream));
+        // buf is destroyed here and must free on 'stream' (still alive).
+    }
+    cudaCheck(cudaStreamSynchronize(stream));
+    cudaCheck(cudaStreamDestroy(stream));
+    EXPECT_EQ(cudaSuccess, cudaGetLastError());
+}// DeviceBufferNonBlockingStreamLifetime
+
+__global__ void deviceBufferFillKernel(unsigned char *p, size_t n, unsigned char v)
+{
+    for (size_t i = blockIdx.x*(size_t)blockDim.x + threadIdx.x; i < n; i += (size_t)gridDim.x*blockDim.x) p[i] = v;
+}
+__global__ void deviceBufferCountKernel(const unsigned char *p, size_t n, unsigned char v, unsigned long long *bad)
+{
+    for (size_t i = blockIdx.x*(size_t)blockDim.x + threadIdx.x; i < n; i += (size_t)gridDim.x*blockDim.x)
+        if (p[i] != v) atomicAdd(bad, 1ull);
+}
+
+// Shared body of the two free-ordering regression tests below. A device-only buffer is used on
+// purpose: it owns no pinned host memory, so clear()'s cudaFreeHost (which implicitly
+// synchronizes) cannot mask a premature free. A second stream 'user' is parked behind a
+// busy-wait with a write to the buffer's raw pointer queued behind it, then the buffer is
+// destroyed. If the free is not ordered after 'user', the allocator recycles the block into the
+// next allocation and the late write corrupts it.
+//
+// The two scenarios cover the two distinct ways the free can be ordered:
+//  - blocking 'user', write NOT registered: the free, issued on the legacy default stream, is
+//    implicitly ordered after all blocking streams. Freeing on any other single stream (e.g.
+//    the buffer's most recently used one) would drop that and fail here.
+//  - non-blocking 'user', write registered via recordUse: no implicit ordering exists for
+//    non-blocking streams, so only the recorded event orders the free. Freeing on the default
+//    stream without the event fails here.
+static void testDeviceBufferFreeOrdering(bool nonBlockingUser, bool registerUse)
+{
+    const size_t N = size_t(64) << 20;// large enough that a recycled block is the same address
+    const unsigned char LATE = 0xAA, VICTIM = 0x55;
+    const unsigned long long CYCLES = 400000000ull;// parks 'user' for O(100 ms)
+
+    cudaStream_t user = nullptr, other = nullptr;
+    if (nonBlockingUser) {
+        cudaCheck(cudaStreamCreateWithFlags(&user, cudaStreamNonBlocking));
+    } else {
+        cudaCheck(cudaStreamCreate(&user));
+    }
+    cudaCheck(cudaStreamCreate(&other));
+    unsigned long long *bad = nullptr;
+    cudaCheck(cudaMallocManaged(&bad, sizeof(*bad)));
+
+    {// warm-up: on a cold context the first launches serialize, which would hide the race
+        unsigned char *w = nullptr;
+        cudaCheck(cudaMallocAsync((void**)&w, N, other));
+        streamBusyWaitKernel<<<1,1,0,user>>>(CYCLES/10);
+        deviceBufferFillKernel<<<1024,256,0,other>>>(w, N, 0);
+        deviceBufferCountKernel<<<1024,256,0,other>>>(w, N, 0, bad);
+        cudaCheck(cudaFreeAsync(w, other));
+        cudaCheck(cudaDeviceSynchronize());
+    }
+
+    void *devPtr = nullptr;
+    {
+        auto buf = nanovdb::cuda::DeviceBuffer::create(N, nullptr, 0, other);// device-only
+        devPtr = buf.deviceData(0);
+        ASSERT_TRUE(devPtr);
+        streamBusyWaitKernel<<<1,1,0,user>>>(CYCLES);// park 'user'
+        deviceBufferFillKernel<<<1024,256,0,user>>>((unsigned char*)devPtr, N, LATE);
+        if (registerUse) buf.recordUse(0, user);// raw-pointer use: tell the buffer about it
+    }// destroyed here; the free must be ordered after 'user' as well as 'other'
+
+    unsigned char *victim = nullptr;
+    cudaCheck(cudaMallocAsync((void**)&victim, N, other));
+    deviceBufferFillKernel<<<1024,256,0,other>>>(victim, N, VICTIM);
+    cudaCheck(cudaStreamSynchronize(other));
+
+    // Diagnostics only. These are deliberately NOT preconditions: with a correctly ordered free
+    // the allocator cannot hand this block out again until 'user' has drained, so observing
+    // "not recycled" or "no longer pending" here is the fix working, not a reason to skip.
+    const bool stillPending = (cudaStreamQuery(user) == cudaErrorNotReady);
+    cudaGetLastError();// clear the cudaErrorNotReady left by the query above
+    const bool recycled = (victim == devPtr);
+
+    cudaCheck(cudaStreamSynchronize(user));// let the late write land
+    *bad = 0;
+    deviceBufferCountKernel<<<1024,256>>>(victim, N, VICTIM, bad);
+    cudaCheck(cudaDeviceSynchronize());
+    const unsigned long long clobbered = *bad;
+
+    cudaCheck(cudaFreeAsync(victim, other));
+    cudaCheck(cudaStreamSynchronize(other));
+    cudaCheck(cudaFree(bad));
+    cudaCheck(cudaStreamDestroy(user));
+    cudaCheck(cudaStreamDestroy(other));
+
+    EXPECT_EQ(0u, clobbered) << "device memory was freed while another stream still had work in "
+                                "flight (block recycled: " << recycled
+                             << ", work still pending when it was reused: " << stillPending << ")";
+}// testDeviceBufferFreeOrdering
+
+TEST(TestNanoVDBCUDA, DeviceBufferMultiStreamFreeOrdering)
+{
+    testDeviceBufferFreeOrdering(/*nonBlockingUser=*/false, /*registerUse=*/false);
+}// DeviceBufferMultiStreamFreeOrdering
+
+TEST(TestNanoVDBCUDA, DeviceBufferNonBlockingFreeOrdering)
+{
+    testDeviceBufferFreeOrdering(/*nonBlockingUser=*/true, /*registerUse=*/true);
+}// DeviceBufferNonBlockingFreeOrdering
+
+TEST(TestNanoVDBCUDA, DeviceBufferChainedRecordUse)
+{
+    // Regression: recordUse re-records the single per-device tracking event, and re-recording
+    // MOVES an event. Two non-blocking streams recording uses in sequence must therefore CHAIN
+    // (the second record first waits on the first capture); otherwise the second record
+    // discards the only coverage of the first stream's in-flight work and the free races it.
+    // Same shape as testDeviceBufferFreeOrdering, with the late writer recorded FIRST and an
+    // idle second stream recorded after it.
+    const size_t N = size_t(64) << 20;
+    const unsigned char LATE = 0xAA, VICTIM = 0x55;
+    const unsigned long long CYCLES = 400000000ull;// parks 'userA' for O(100 ms)
+
+    cudaStream_t userA = nullptr, userB = nullptr, other = nullptr;
+    cudaCheck(cudaStreamCreateWithFlags(&userA, cudaStreamNonBlocking));
+    cudaCheck(cudaStreamCreateWithFlags(&userB, cudaStreamNonBlocking));
+    cudaCheck(cudaStreamCreate(&other));
+    unsigned long long *bad = nullptr;
+    cudaCheck(cudaMallocManaged(&bad, sizeof(*bad)));
+
+    {// warm-up (see testDeviceBufferFreeOrdering)
+        unsigned char *w = nullptr;
+        cudaCheck(cudaMallocAsync((void**)&w, N, other));
+        streamBusyWaitKernel<<<1,1,0,userA>>>(CYCLES/10);
+        deviceBufferFillKernel<<<1024,256,0,other>>>(w, N, 0);
+        deviceBufferCountKernel<<<1024,256,0,other>>>(w, N, 0, bad);
+        cudaCheck(cudaFreeAsync(w, other));
+        cudaCheck(cudaDeviceSynchronize());
+    }
+
+    void *devPtr = nullptr;
+    {
+        auto buf = nanovdb::cuda::DeviceBuffer::create(N, nullptr, 0, other);// device-only
+        devPtr = buf.deviceData(0);
+        ASSERT_TRUE(devPtr);
+        streamBusyWaitKernel<<<1,1,0,userA>>>(CYCLES);// park 'userA'
+        deviceBufferFillKernel<<<1024,256,0,userA>>>((unsigned char*)devPtr, N, LATE);
+        buf.recordUse(0, userA);// covers the in-flight write...
+        buf.recordUse(0, userB);// ...and must NOT be discarded by a later record
+    }// destroyed here; the free must still be ordered after 'userA'
+
+    unsigned char *victim = nullptr;
+    cudaCheck(cudaMallocAsync((void**)&victim, N, other));
+    deviceBufferFillKernel<<<1024,256,0,other>>>(victim, N, VICTIM);
+    cudaCheck(cudaStreamSynchronize(other));
+
+    const bool stillPending = (cudaStreamQuery(userA) == cudaErrorNotReady);
+    cudaGetLastError();// clear the cudaErrorNotReady left by the query above
+    const bool recycled = (victim == devPtr);
+
+    cudaCheck(cudaStreamSynchronize(userA));// let the late write land
+    *bad = 0;
+    deviceBufferCountKernel<<<1024,256>>>(victim, N, VICTIM, bad);
+    cudaCheck(cudaDeviceSynchronize());
+    const unsigned long long clobbered = *bad;
+
+    cudaCheck(cudaFreeAsync(victim, other));
+    cudaCheck(cudaStreamSynchronize(other));
+    cudaCheck(cudaFree(bad));
+    cudaCheck(cudaStreamDestroy(userA));
+    cudaCheck(cudaStreamDestroy(userB));
+    cudaCheck(cudaStreamDestroy(other));
+
+    // Detection relies on the pool recycling the freed block into 'victim' (stream-ordered
+    // pools recycle WITH the dependency attached, so recycling is expected even with a
+    // correctly ordered free). If it did not recycle, the chain was never exercised -- make
+    // that visible instead of a vacuous pass.
+    if (!recycled) GTEST_SKIP() << "allocator did not recycle the block; ordering not exercised";
+
+    EXPECT_EQ(0u, clobbered) << "a later recordUse on another stream discarded the tracking "
+                                "event covering in-flight work (block recycled: " << recycled
+                             << ", work still pending when it was reused: " << stillPending << ")";
+}// DeviceBufferChainedRecordUse
+
 TEST(TestNanoVDBCUDA, RefineCoarsen_ValueOnIndex)
 {
     using BuildT = nanovdb::ValueOnIndex;

nanovdb/nanovdb/unittest/TestUtilCuda.cu

@@ -0,0 +1,71 @@
+// Copyright Contributors to the OpenVDB Project
+// SPDX-License-Identifier: Apache-2.0
+
+/// @file TestUtilCuda.cu
+///
+/// @brief Unit tests for the util::cuda allocation wrappers: the cached
+///        memory-pool capability query, and an allocate/use/free round-trip
+///        through whichever mode the build selects (stream-ordered by default,
+///        synchronous under NANOVDB_USE_SYNC_CUDA_MALLOC). This file is compiled
+///        into two test binaries -- one per mode -- so both are exercised. On a
+///        device without memory pools the default (async) wrappers fail by
+///        design; such a device must build with NANOVDB_USE_SYNC_CUDA_MALLOC,
+///        so the async round-trip skips when pools are absent.
+
+#include <nanovdb/util/cuda/Util.h>
+
+#include <cuda_runtime_api.h>
+#include <gtest/gtest.h>
+
+#include <vector>
+
+namespace {
+
+TEST(TestUtilCuda, MemoryPoolsSupportedMatchesAttribute)
+{
+    int count = 0;
+    ASSERT_EQ(cudaGetDeviceCount(&count), cudaSuccess);
+    ASSERT_GT(count, 0);
+    for (int i = 0; i < count; ++i) {
+        int attr = 0;
+        ASSERT_EQ(cudaDeviceGetAttribute(&attr, cudaDevAttrMemoryPoolsSupported, i), cudaSuccess);
+        EXPECT_EQ(nanovdb::util::cuda::memoryPoolsSupported(i), attr != 0) << "device " << i;
+    }
+    // Out-of-range device ids are unsupported, not undefined.
+    EXPECT_FALSE(nanovdb::util::cuda::memoryPoolsSupported(-1));
+    EXPECT_FALSE(nanovdb::util::cuda::memoryPoolsSupported(count));
+}
+
+TEST(TestUtilCuda, MallocAsyncRoundTrip)
+{
+    // Allocate/use/free through the wrapper in whichever mode the build selects:
+    // synchronous under the macro, stream-ordered otherwise. Without the macro
+    // the async path requires memory pools, so skip when the device lacks them.
+    int device = 0;
+    ASSERT_EQ(cudaGetDevice(&device), cudaSuccess);
+#ifndef NANOVDB_USE_SYNC_CUDA_MALLOC
+    if (!nanovdb::util::cuda::memoryPoolsSupported(device))
+        GTEST_SKIP() << "device " << device << " lacks memory pools; the async wrappers fail by design "
+                        "(build with NANOVDB_USE_SYNC_CUDA_MALLOC)";
+#endif
+    cudaStream_t s = nullptr;
+    ASSERT_EQ(cudaStreamCreate(&s), cudaSuccess);
+
+    const size_t n = 4096;
+    void* d = nullptr;
+    ASSERT_EQ(nanovdb::util::cuda::mallocAsync(&d, n, s), cudaSuccess);
+    ASSERT_NE(d, nullptr);
+    ASSERT_EQ(cudaMemsetAsync(d, 0x3C, n, s), cudaSuccess);
+
+    std::vector<unsigned char> host(n, 0);
+    ASSERT_EQ(cudaMemcpyAsync(host.data(), d, n, cudaMemcpyDeviceToHost, s), cudaSuccess);
+    ASSERT_EQ(cudaStreamSynchronize(s), cudaSuccess);
+    for (size_t i = 0; i < n; ++i)
+        ASSERT_EQ(host[i], 0x3Cu) << "byte " << i;
+
+    ASSERT_EQ(nanovdb::util::cuda::freeAsync(d, s), cudaSuccess);
+    ASSERT_EQ(cudaStreamSynchronize(s), cudaSuccess);
+    ASSERT_EQ(cudaStreamDestroy(s), cudaSuccess);
+}
+
+} // unnamed namespace

nanovdb/nanovdb/util/cuda/Util.h

@@ -16,6 +16,7 @@
 
 #include <cuda.h>
 #include <cuda_runtime_api.h>
+#include <vector>
 #include <nanovdb/util/Util.h> // for stderr and NANOVDB_ASSERT
 
 // change 1 -> 0 to only perform asserts during debug builds
@@ -70,23 +71,54 @@ namespace nanovdb {// =========================================================
 namespace util::cuda {// ======================================================
 
 //#define NANOVDB_USE_SYNC_CUDA_MALLOC
-// cudaMallocAsync and cudaFreeAsync were introduced in CUDA 11.2 so we introduce
-// custom implementations that map to cudaMalloc and cudaFree below. If NANOVDB_USE_SYNC_CUDA_MALLOC
-// is defined these implementations will also be defined, which is useful in virtualized environments
-// that slice up the GPU and share it between instances as vGPU's. GPU unified memory is usually disabled
-// out of security considerations. Asynchronous CUDA malloc/free depends on GPU unified memory, so it
-// is not possible to use cudaMallocAsync and cudaFreeAsync in such environments.
+// cudaMallocAsync and cudaFreeAsync were introduced in CUDA 11.2, and even on newer
+// toolkits a device may not expose stream-ordered memory pools at runtime — fractional
+// vGPU configurations commonly disable them, making cudaMallocAsync fail with
+// cudaErrorNotSupported. The wrappers below therefore behave in two modes: when
+// CUDA < 11.2 or NANOVDB_USE_SYNC_CUDA_MALLOC is defined they map to synchronous
+// cudaMalloc/cudaFree; otherwise they use cudaMallocAsync/cudaFreeAsync and, on a
+// device that lacks memory pools, fail with an actionable diagnostic rather than
+// silently substituting synchronous allocation (which would make an async resource
+// misrepresent its own semantics — the choice of a synchronous backend belongs to the
+// caller, not to a hidden runtime fallback). Callers select synchronous allocation by
+// defining NANOVDB_USE_SYNC_CUDA_MALLOC, which -- because the wrappers are inline
+// functions -- must be defined identically in every translation unit to avoid violating
+// the one-definition rule.
+
+/// @brief Returns true if @c device supports stream-ordered memory pools, i.e.
+///        cudaMallocAsync/cudaFreeAsync. Queried once per process for all
+///        devices and cached; out-of-range device ids return false.
+inline bool memoryPoolsSupported(int device)
+{
+#if (CUDART_VERSION < 11020)
+    (void)device;
+    return false;
+#else
+    static const auto supported = [] {
+        int count = 0;
+        if (cudaGetDeviceCount(&count) != cudaSuccess || count < 0) count = 0;
+        std::vector<char> s(static_cast<size_t>(count), 0);
+        for (int i = 0; i < count; ++i) {
+            int attr = 0;
+            if (cudaDeviceGetAttribute(&attr, cudaDevAttrMemoryPoolsSupported, i) == cudaSuccess)
+                s[static_cast<size_t>(i)] = char(attr != 0);
+        }
+        return s;
+    }();
+    return device >= 0 && static_cast<size_t>(device) < supported.size() && supported[static_cast<size_t>(device)] != 0;
+#endif
+}
 
 #if (CUDART_VERSION < 11020) || defined(NANOVDB_USE_SYNC_CUDA_MALLOC) // 11.2 introduced cudaMallocAsync and cudaFreeAsync
 
-/// @brief Simple wrapper that calls cudaMalloc
+/// @brief Wrapper forced to synchronous cudaMalloc; see the mode comment above.
 /// @param d_ptr Device pointer to allocated device memory
 /// @param size  Number of bytes to allocate
 /// @param dummy The stream establishing the stream ordering contract and the memory pool to allocate from (ignored)
 /// @return Cuda error code
 inline cudaError_t mallocAsync(void** d_ptr, size_t size, cudaStream_t){return cudaMalloc(d_ptr, size);}
 
-/// @brief Simple wrapper that calls cudaFree
+/// @brief Wrapper forced to synchronous cudaFree; see the mode comment above.
 /// @param d_ptr Device pointer that will be freed
 /// @param dummy The stream establishing the stream ordering promise (ignored)
 /// @return Cuda error code
@@ -94,18 +126,43 @@ inline cudaError_t freeAsync(void* d_ptr, cudaStream_t){return cudaFree(d_ptr);}
 
 #else
 
-/// @brief Simple wrapper that calls cudaMallocAsync
+/// @brief Wrapper that calls cudaMallocAsync. On a device without stream-ordered
+///        memory pools it emits an actionable diagnostic and returns
+///        cudaErrorNotSupported rather than silently allocating synchronously.
 /// @param d_ptr Device pointer to allocated device memory
 /// @param size  Number of bytes to allocate
 /// @param stream The stream establishing the stream ordering contract and the memory pool to allocate from
 /// @return Cuda error code
-inline cudaError_t mallocAsync(void** d_ptr, size_t size, cudaStream_t stream){return cudaMallocAsync(d_ptr, size, stream);}
+inline cudaError_t mallocAsync(void** d_ptr, size_t size, cudaStream_t stream)
+{
+    int device = 0;
+    const cudaError_t err = cudaGetDevice(&device);
+    if (err != cudaSuccess) return err;
+    if (!memoryPoolsSupported(device)) {
+        fprintf(stderr,
+                "NanoVDB: device %d does not support stream-ordered CUDA memory pools required by "
+                "cudaMallocAsync. Define NANOVDB_USE_SYNC_CUDA_MALLOC when building to allocate "
+                "synchronously with cudaMalloc/cudaFree instead.\n",
+                device);
+        return cudaErrorNotSupported;
+    }
+    return cudaMallocAsync(d_ptr, size, stream);
+}
 
-/// @brief Simple wrapper that calls cudaFreeAsync
+/// @brief Wrapper that calls cudaFreeAsync. Mirrors mallocAsync's pool check so a
+///        device without memory pools reports cudaErrorNotSupported rather than
+///        calling cudaFreeAsync where it cannot succeed.
 /// @param d_ptr Device pointer that will be freed
 /// @param stream The stream establishing the stream ordering promise
 /// @return Cuda error code
-inline cudaError_t freeAsync(void* d_ptr, cudaStream_t stream){return cudaFreeAsync(d_ptr, stream);}
+inline cudaError_t freeAsync(void* d_ptr, cudaStream_t stream)
+{
+    int device = 0;
+    const cudaError_t err = cudaGetDevice(&device);
+    if (err != cudaSuccess) return err;
+    if (!memoryPoolsSupported(device)) return cudaErrorNotSupported;
+    return cudaFreeAsync(d_ptr, stream);
+}
 
 #endif
 

openvdb/openvdb/math/Stats.h

@@ -308,8 +308,14 @@ public:
     /// @return @c true if the sample value falls within this histogram's value range.
     inline bool add(double val, uint64_t n = 1)
     {
-        if (val<mMin || val>mMax) return false;
-        mBins[size_t(mDelta*(val-mMin))] += n;
+        if (std::isnan(val) || val<mMin || val>mMax) return false;
+        // Clamp the bin index to the last bin. The range test above admits values up
+        // to mMax, which is the upper bound plus a small epsilon, so a sample equal
+        // to (or within the epsilon of) that bound scales to exactly numBins -- one
+        // past the end of mBins. That is trivially reached in practice, e.g. by
+        // tools::histogram() over a fog volume whose interior is exactly its maximum.
+        const size_t i = size_t(mDelta*(val-mMin));
+        mBins[i < mBins.size() ? i : mBins.size()-1] += n;
         mSize += n;
         return true;
     }
@@ -332,11 +338,11 @@ public:
     /// Return the upper bound of this histogram's value range.
     inline double max() const { return mMax; }
     /// Return the minimum value in the <i>n</i>th bin.
-    inline double min(int n) const { return mMin+n/mDelta; }
+    inline double min(size_t n) const { return mMin+double(n)/mDelta; }
     /// Return the maximum value in the <i>n</i>th bin.
-    inline double max(int n) const { return mMin+(n+1)/mDelta; }
+    inline double max(size_t n) const { return mMin+double(n+1)/mDelta; }
     /// Return the number of samples in the <i>n</i>th bin.
-    inline uint64_t count(int n) const { return mBins[n]; }
+    inline uint64_t count(size_t n) const { return mBins[n]; }
     /// Return the population size, i.e., the total number of samples.
     inline uint64_t size() const { return mSize; }
 

openvdb/openvdb/points/PointRasterizeSDF.h

@@ -282,7 +282,25 @@ OPENVDB_NO_DEPRECATION_WARNING_BEGIN
 ///   point. Typically (for our intended surfacing) these transformations are
 ///   built by analysing each points neighbourhood distributions and
 ///   constructing tight ellipsoids that orient themselves to follow these
-///   point distributions.
+///   point distributions. Example:
+/// @code
+///    auto vs = points->voxelSize()[0];
+///
+///    points::PcaSettings s;
+///    s.searchRadius = vs*2.0f;
+///    s.nonAnisotropicStretch = 0.2f;
+///
+///    // could alternatively use stretch as the radius
+///    points::PcaAttributes a;
+///    a.xformOutput = points::PcaAttributes::XformOutput::COMBINED_TRANSFORM;
+///    points::pca(*points,s, a);
+
+///    points::EllipsoidSettings<> es;
+///    es.xform = a.xform;
+///    es.pws = a.positionWS;
+///    es.radiusScale = Vec3f(vs);
+///    auto outgrids = points::rasterizeSdf(*points, es);
+/// @endcode
 /// @note  Protected inheritance prevents accidental struct slicing
 template <typename AttributeTs = TypeList<>,
     typename RadiusAttributeT = Vec3f,

openvdb/openvdb/points/PrincipalComponentAnalysis.h

@@ -24,6 +24,7 @@
 #include <openvdb/thread/Threading.h>
 #include <openvdb/util/NullInterrupter.h>
 #include <openvdb/util/Assert.h>
+#include <openvdb/simd/Simd.h>
 #include <openvdb/points/PointAttribute.h>
 #include <openvdb/points/PointGroup.h>
 #include <openvdb/points/PointTransfer.h>

openvdb/openvdb/points/impl/PrincipalComponentAnalysisImpl.h

@@ -53,8 +53,15 @@ struct PcaTimer final : public TimerT
 };
 
 using WeightSumT = double;
-using WeightedPositionSumT = Vec3d;
+using WeightedPositionSumT = math::Vec3<WeightSumT>;
 using GroupIndexT = points::AttributeSet::Descriptor::GroupIndex;
+// Batched PCA types. Either double, or some simd type
+using NativeT = simd::NativeSimdOrScalar<WeightSumT>::Type;
+// 1 (for double) or the size of our simd type
+inline constexpr size_t kPcaBatchSize = simd::NativeSimdOrScalar<WeightSumT>::Size;
+// Expect power of two
+static_assert((kPcaBatchSize > 0) &&
+    !(kPcaBatchSize & (kPcaBatchSize - 1)));
 
 struct AttrIndices
 {
@@ -93,7 +100,7 @@ inline T* initPcaArrayAttribute(LeafNodeT& leaf, const size_t idx, const bool fi
 ///   it might be worth falling back to a gather stlye approach when the MPPV
 ///   and radius values are relatively small. We should investigate this in the
 ///   future.
-template <typename PointDataTreeT>
+template <typename DerivedT, typename PointDataTreeT> // CRTP for shared interface
 struct PcaTransfer
     : public VolumeTransfer<PointDataTreeT>
     , public InterruptableTransfer
@@ -162,6 +169,9 @@ struct PcaTransfer
         BaseT::initialize(origin, idx, bounds);
         auto& leaf = mManager.leaf(idx);
         mTargetPosition = std::make_unique<PositionHandleT>(leaf.constAttributeArray(mIndices.mPWsIndex));
+#if OPENVDB_PCA_SELF_CONTRIBUTION == 0
+        mCurrent = &leaf;
+#endif
         return &leaf;
     }
 
@@ -169,13 +179,109 @@ struct PcaTransfer
     {
         mSourcePosition = std::make_unique<PositionHandleT>(leaf.constAttributeArray(mIndices.mPWsIndex));
 #if OPENVDB_PCA_SELF_CONTRIBUTION == 0
-        mIsSameLeaf = this->mTargetPosition->array() == this->mSourcePosition->array();
+        mIsSameLeaf = (&leaf == mCurrent);
 #endif
         return true;
     }
 
+    /// @brief  Multi point rasterization
+    /// @note   This is the only allowed entry point from the transfer schemes
+    inline void rasterizePoints(const Coord& ijk,
+                    const Index start,
+                    const Index end,
+                    const CoordBBox& bounds)
+    {
+        const Index step = std::max(Index(1), Index((end - start) / this->maxSourcePointsPerVoxel()));
+
+        if constexpr(kPcaBatchSize == 1) {
+            // Fallback to per point rasterization as batch size is explicitly 1
+            // i.e. no simd containers in use
+            for (Index i = start; i < end; i += step) {
+                this->rasterizePoint(ijk, i, bounds);
+            }
+        }
+        else {
+            std::array<int64_t, kPcaBatchSize> ids;
+            Index offset = 0;
+            for (Index i = start; i < end; i += step) {
+                ids[offset++] = int64_t(i);
+                if (offset == kPcaBatchSize) {
+                   this->rasterizeN2<kPcaBatchSize>(ijk, ids, bounds);
+                    offset = 0;
+                }
+            }
+            if (offset == 0) return;
+            for (; offset < kPcaBatchSize; ++offset) {
+                ids[offset] = int64_t(-1);
+            }
+           this->rasterizeN2<kPcaBatchSize>(ijk, ids, bounds);
+        }
+    }
+
     bool endPointLeaf(const typename PointDataTreeT::LeafNodeType&) { return true; }
 
+private:
+    template <size_t N2>
+    inline void rasterizeN2(const Coord&,
+        const std::array<int64_t, N2>& points,
+        const CoordBBox& bounds)
+    {
+        // expected pow2 and at least 2 elements
+        static_assert((N2 > 1) && !(N2 & (N2 - 1)));
+        using SimdT  = typename simd::SimdT<WeightSumT, N2>::Type;
+        using SimdIT = typename simd::SimdT<int64_t, N2>::Type;
+        using SimdMT = typename simd::SimdTraits<SimdT>::MaskT;
+
+        OPENVDB_ASSERT(points[0] != -1);
+
+        std::array<WeightSumT, 3*N2> cache;
+        math::Vec3<WeightSumT> tmp;
+        // convert AoS to SoA
+        for (size_t i = 0; i < N2; ++i) {
+            if (points[i] != -1) {
+                tmp = math::Vec3<WeightSumT>(this->mSourcePosition->get(Index(points[i])));
+            }
+            cache[i+(N2*0)] = tmp[0];
+            cache[i+(N2*1)] = tmp[1];
+            cache[i+(N2*2)] = tmp[2];
+        }
+
+        const SimdT dxinv(this->mDxInv);
+        const SimdT pwx = simd::load<N2>(cache.data() + (N2*0));
+        const SimdT pwy = simd::load<N2>(cache.data() + (N2*1));
+        const SimdT pwz = simd::load<N2>(cache.data() + (N2*2));
+        const SimdT pix = pwx * dxinv;
+        const SimdT piy = pwy * dxinv;
+        const SimdT piz = pwz * dxinv;
+        const SimdIT ids = simd::load<N2>(points.data());
+        const SimdMT validMask = ids != SimdIT(-1);
+
+        this->derived().stamp(pwx, pwy, pwz, pix, piy, piz, validMask, bounds);
+    }
+
+    /// @brief   Single point rasterization
+    /// @warning If batching, we never invoke stamp() with a single scalar. This
+    ///   makes the arithmetic a bit simpler as we don't have to worry about
+    ///   intermediately storing simdt's with accumulating scalars accidently
+    ///   filling all lanes
+    inline void rasterizePoint(const Coord& ijk,
+                    const Index id,
+                    const CoordBBox& bounds)
+    {
+        if (kPcaBatchSize > 1) {
+            this->rasterizePoints(ijk, id, id+1, bounds);
+        }
+        else {
+            const Vec3d Pws = math::Vec3<WeightSumT>(this->mSourcePosition->get(id));
+            const Vec3d Pis = Pws * this->mDxInv;
+            this->derived().stamp(Pws.x(), Pws.y(), Pws.z(), Pis.x(), Pis.y(), Pis.z(), true, bounds);
+        }
+    }
+
+    inline DerivedT& derived() {
+        return *(static_cast<DerivedT*>(this));
+    }
+
 protected:
     const AttrIndices& mIndices;
     const PcaSettings& mSettings;
@@ -185,14 +291,15 @@ protected:
     std::unique_ptr<PositionHandleT> mSourcePosition;
 #if OPENVDB_PCA_SELF_CONTRIBUTION == 0
     bool mIsSameLeaf {false};
+    LeafNodeType* mCurrent {nullptr};
 #endif
 };
 
 template <typename PointDataTreeT>
 struct WeightPosSumsTransfer
-    : public PcaTransfer<PointDataTreeT>
+    : public PcaTransfer<WeightPosSumsTransfer<PointDataTreeT>, PointDataTreeT>
 {
-    using BaseT = PcaTransfer<PointDataTreeT>;
+    using BaseT = PcaTransfer<WeightPosSumsTransfer<PointDataTreeT>, PointDataTreeT>;
 
     static const Index DIM = PointDataTreeT::LeafNodeType::DIM;
     static const Index LOG2DIM = PointDataTreeT::LeafNodeType::LOG2DIM;
@@ -204,106 +311,40 @@ struct WeightPosSumsTransfer
                           util::NullInterrupter* interrupt)
         : BaseT(indices, settings, vs, manager, interrupt)
         , mWeights()
-        , mWeightedPositions()
-        , mCounts() {}
+        , mCounts()
+        , mBatchedWeightedPositions()
+        , mWeightedPositions() {}
 
     WeightPosSumsTransfer(const WeightPosSumsTransfer& other)
         : BaseT(other)
         , mWeights()
-        , mWeightedPositions()
-        , mCounts() {}
+        , mCounts()
+        , mBatchedWeightedPositions()
+        , mWeightedPositions() {}
 
     inline void initialize(const Coord& origin, const size_t idx, const CoordBBox& bounds)
     {
         auto& leaf = (*BaseT::initialize(origin, idx, bounds));
         mWeights = initPcaArrayAttribute<WeightSumT>(leaf, this->mIndices.mWeightSumIndex);
-        mWeightedPositions = initPcaArrayAttribute<WeightedPositionSumT>(leaf, this->mIndices.mPosSumIndex);
+
+        // If we're batching, don't init the WeightedPositionSumT buffer yet.
+        // Instead create an intermediate buffer to avoid the horizontal
+        // accumulations until finalize.
+        // @warning  This does increase peak memory relative to the size of
+        //   the native simdt in use.
+        if constexpr (kPcaBatchSize > 1) {
+            mBatchedWeightedPositions.assign(this->mTargetPosition->size() * 3, NativeT(0));
+            mWeightedPositions = mBatchedWeightedPositions.data();
+        }
+        else {
+            auto* weightedPos = initPcaArrayAttribute<WeightedPositionSumT>(leaf, this->mIndices.mPosSumIndex);
+            mWeightedPositions = weightedPos[0].asPointer();
+        }
+
         // track neighbours
         mCounts.assign(this->mTargetPosition->size(), 0);
     }
 
-    inline void rasterizePoints(const Coord&, const Index start, const Index end, const CoordBBox& bounds)
-    {
-        const Index step = std::max(Index(1), Index((end - start) / this->maxSourcePointsPerVoxel()));
-
-        const auto* const data = this->template buffer<0>();
-        const auto& mask = *(this->template mask<0>());
-
-        const float searchRadiusInv = 1.0f / this->searchRadius();
-        const Real searchRadius2 = math::Pow2(this->searchRadius());
-        const Real searchRadiusIS2 = math::Pow2(this->searchRadius() * this->mDxInv);
-
-        OPENVDB_ASSERT(std::isfinite(searchRadiusInv));
-        OPENVDB_ASSERT(std::isfinite(searchRadius2));
-        OPENVDB_ASSERT(std::isfinite(searchRadiusIS2));
-
-        const Coord& a(bounds.min());
-        const Coord& b(bounds.max());
-
-        for (Index srcid = start; srcid < end; srcid += step) {
-            const Vec3d Psrc(this->mSourcePosition->get(srcid));
-            const Vec3d PsrcIS = Psrc * this->mDxInv;
-
-            for (Coord c = a; c.x() <= b.x(); ++c.x()) {
-                const Real minx = c.x() - 0.5;
-                const Real maxx = c.x() + 0.5;
-                const Real dminx =
-                    (PsrcIS[0] < minx ? math::Pow2(PsrcIS[0] - minx) :
-                    (PsrcIS[0] > maxx ? math::Pow2(PsrcIS[0] - maxx) : 0));
-                if (dminx > searchRadiusIS2) continue; // next target voxel
-                const Index i = ((c.x() & (DIM-1u)) << 2*LOG2DIM); // unsigned bit shift mult
-                for (c.y() = a.y(); c.y() <= b.y(); ++c.y()) {
-                    const Real miny = c.y() - 0.5;
-                    const Real maxy = c.y() + 0.5;
-                    const Real dminxy = dminx +
-                        (PsrcIS[1] < miny ? math::Pow2(PsrcIS[1] - miny) :
-                        (PsrcIS[1] > maxy ? math::Pow2(PsrcIS[1] - maxy) : 0));
-                    if (dminxy > searchRadiusIS2) continue; // next target voxel
-                    const Index ij = i + ((c.y() & (DIM-1u)) << LOG2DIM);
-                    for (c.z() = a.z(); c.z() <= b.z(); ++c.z()) {
-                        const Index offset = ij + /*k*/(c.z() & (DIM-1u));
-                        if (!mask.isOn(offset)) continue; // next target voxel
-
-                        const Real minz = c.z() - 0.5;
-                        const Real maxz = c.z() + 0.5;
-                        const Real dminxyz = dminxy +
-                            (PsrcIS[2] < minz ? math::Pow2(PsrcIS[2] - minz) :
-                            (PsrcIS[2] > maxz ? math::Pow2(PsrcIS[2] - maxz) : 0));
-                        // Does this point's radius overlap the voxel c
-                        if (dminxyz > searchRadiusIS2) continue; // next target voxel
-
-                        // src point overlaps voxel c
-                        const Index targetEnd = data[offset];
-                        const Index targetStart = (offset == 0) ? 0 : Index(data[offset - 1]);
-                        const Index targetStep =
-                            std::max(Index(1), Index((targetEnd - targetStart) / this->maxTargetPointsPerVoxel()));
-
-                        /// @warning  stepping in this way does not guarantee
-                        ///   we get a self contribution, could guarantee this
-                        ///   by enabling the OPENVDB_PCA_SELF_CONTRIBUTION == 0
-                        ///   check and adding it afterwards.
-                        for (Index tgtid = targetStart; tgtid < targetEnd; tgtid += targetStep) {
-#if OPENVDB_PCA_SELF_CONTRIBUTION == 0
-                            if (OPENVDB_UNLIKELY(this->mIsSameLeaf && tgtid == srcid)) continue;
-#endif
-                            const Vec3d Ptgt(this->mTargetPosition->get(tgtid));
-                            const Real d2 = (Psrc - Ptgt).lengthSqr();
-                            if (d2 > searchRadius2) continue;
-
-                            // src point (srcid) reaches target point (tgtid)
-                            const float weight = 1.0f - math::Pow3(float(math::Sqrt(d2)) * searchRadiusInv);
-                            OPENVDB_ASSERT(weight >= 0.0f && weight <= 1.0f);
-
-                            mWeights[tgtid] += weight;
-                            mWeightedPositions[tgtid] += Psrc * weight; // @note: world space position is weighted
-                            ++mCounts[tgtid];
-                        }
-                    }
-                }
-            } // outer sdf voxel
-        } // point idx
-    }
-
     bool finalize(const Coord&, size_t idx)
     {
         // Add points to group with counts which are more than the neighbouring threshold
@@ -319,32 +360,180 @@ struct WeightPosSumsTransfer
 
         points::GroupWriteHandle group(leaf.groupWriteHandle(this->mIndices.mEllipsesGroupIndex));
 
+        // Init the actual buffer data if we're batching
+        double* const weightedPositions = [&]() {
+            if constexpr (kPcaBatchSize > 1) {
+                OPENVDB_ASSERT(mWeightedPositions == mBatchedWeightedPositions.data());
+                // assume trivial type and contiguously allocated
+                auto* wp = initPcaArrayAttribute<WeightedPositionSumT>(leaf, this->mIndices.mPosSumIndex, /*fill=*/false);
+                return wp[0].asPointer();
+            }
+            else {
+                return mWeightedPositions;
+            }
+        }();
+
         const int32_t threshold = int32_t(this->neighbourThreshold());
-        for (Index i = 0; i < this->mTargetPosition->size(); ++i) {
+        for (Index i = 0; i < this->mTargetPosition->size(); ++i)
+        {
             // turn points OFF if they are ON and don't meet max neighbour requirements
             if ((threshold == 0 || (mCounts[i] < threshold)) && group.getUnsafe(i)) {
                 group.setUnsafe(i, false);
             }
+
+            // If we were batching, hadd and set each dest component
+            double* const wp = &weightedPositions[i*3];
+            if constexpr (kPcaBatchSize > 1)
+            {
+                NativeT* const comp = &mWeightedPositions[i*3];
+                for (size_t j = 0; j < 3; ++j) {
+                    wp[j] = simd::horizontal_add(comp[j]);
+                }
+            }
             if (mCounts[i] <= 0) continue;
             // Normalize weights
             OPENVDB_ASSERT(mWeights[i] >= 0.0f);
             mWeights[i] = 1.0 / mWeights[i];
-            mWeightedPositions[i] *= mWeights[i];
+            wp[0] *= mWeights[i];
+            wp[1] *= mWeights[i];
+            wp[2] *= mWeights[i];
         }
         return true;
     }
 
+private:
+    friend BaseT;
+
+    template<typename ScalarT, typename MaskT> /// Real or SimdT
+    inline void stamp(const ScalarT& Pwx,
+                    const ScalarT& Pwy,
+                    const ScalarT& Pwz,
+                    const ScalarT& Pix,
+                    const ScalarT& Piy,
+                    const ScalarT& Piz,
+                    const MaskT& m,
+                    const CoordBBox& intersection)
+    {
+        // Some of the arithmetic in this function assumes these are the same,
+        // otherwise we can end up writing scalars to all lanes of a simdt
+        // (e.g. wp[0] += (Pwx * weights), where wp is a simdt and Pwx is
+        // a scalar - this ends up filing all the lanes).
+        static_assert(std::is_same_v<ScalarT, NativeT>);
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Pwx)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Pwy)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Pwz)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Pix)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Piy)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Piz)));
+
+        const auto* const data = this->template buffer<0>();
+        const auto& mask = *(this->template mask<0>());
+
+        const ScalarT searchRadiusInv(1.0f / this->searchRadius());
+        const ScalarT searchRadius2(math::Pow2(this->searchRadius()));
+        const Real searchRadiusIS2 = math::Pow2(this->searchRadius() * this->mDxInv);
+
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(searchRadiusInv)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(searchRadius2)));
+        OPENVDB_ASSERT(std::isfinite(searchRadiusIS2));
+
+        ScalarT tx, ty, tz;
+
+        const Coord& a(intersection.min());
+        const Coord& b(intersection.max());
+        for (Coord c = a; c.x() <= b.x(); ++c.x()) {
+            const ScalarT minx(c.x() - 0.5);
+            const ScalarT maxx(c.x() + 0.5);
+            const ScalarT dminx =
+                (simd::select(Pix < minx, simd::pow2(Pix - minx),
+                    (simd::select(Pix > maxx, simd::pow2(Pix - maxx), ScalarT(0)))));
+            if (simd::horizontal_min(dminx) > searchRadiusIS2) continue;
+            const Index i = ((c.x() & (DIM-1u)) << 2*LOG2DIM); // unsigned bit shift mult
+            for (c.y() = a.y(); c.y() <= b.y(); ++c.y()) {
+                const ScalarT miny(c.y() - 0.5);
+                const ScalarT maxy(c.y() + 0.5);
+                const ScalarT dminxy = dminx +
+                    (simd::select(Piy < miny, simd::pow2(Piy - miny),
+                        (simd::select(Piy > maxy, simd::pow2(Piy - maxy), ScalarT(0)))));
+                if (simd::horizontal_min(dminxy) > searchRadiusIS2) continue;
+                const Index ij = i + ((c.y() & (DIM-1u)) << LOG2DIM);
+                for (c.z() = a.z(); c.z() <= b.z(); ++c.z()) {
+                    const Index offset = ij + /*k*/(c.z() & (DIM-1u));
+                    if (!mask.isOn(offset)) continue; // next target voxel
+
+                    const ScalarT minz(c.z() - 0.5);
+                    const ScalarT maxz(c.z() + 0.5);
+                    const ScalarT dminxyz = dminxy +
+                        (simd::select(Piz < minz, simd::pow2(Piz - minz),
+                            (simd::select(Piz > maxz, simd::pow2(Piz - maxz), ScalarT(0)))));
+                    // Does this point's radius overlap the voxel c
+                    if (simd::horizontal_min(dminxyz) > searchRadiusIS2) continue;
+
+                    // src point overlaps voxel c
+                    const Index targetEnd = data[offset];
+                    const Index targetStart = (offset == 0) ? 0 : Index(data[offset - 1]);
+                    const Index targetStep =
+                        std::max(Index(1), Index((targetEnd - targetStart) / this->maxTargetPointsPerVoxel()));
+
+                    /// @warning  stepping in this way does not guarantee
+                    ///   we get a self contribution, could guarantee this
+                    ///   by enabling the OPENVDB_PCA_SELF_CONTRIBUTION == 0
+                    ///   check and adding it afterwards.
+                    for (Index tgtid = targetStart; tgtid < targetEnd; tgtid += targetStep)
+                    {
+#if OPENVDB_PCA_SELF_CONTRIBUTION == 0
+                        if (OPENVDB_UNLIKELY(this->mIsSameLeaf && tgtid == srcid)) continue;
+#endif
+                        const math::Vec3<WeightSumT> Ptgt(this->mTargetPosition->get(tgtid));
+                        // compute length to Ptgt and store in tx
+                        tx = Pwx - ScalarT(Ptgt[0]);
+                        ty = Pwy - ScalarT(Ptgt[1]);
+                        tz = Pwz - ScalarT(Ptgt[2]);
+                        tx = simd::pow2(tx) + simd::pow2(ty) + simd::pow2(tz);
+
+                        const auto pmask = ((tx <= searchRadius2) && m);
+                        const int c = simd::horizontal_count(pmask);
+                        if (c == 0) continue;
+
+                        // some of src point (srcid) reaches target point (tgtid)
+                        ScalarT weights = (ScalarT(1.0) - simd::pow3(simd::sqrt(tx) * searchRadiusInv));
+                        weights = simd::select(pmask, weights, ScalarT(0.0));
+                        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(weights)));
+                        OPENVDB_ASSERT(simd::horizontal_min(weights) >= 0.0);
+                        OPENVDB_ASSERT(simd::horizontal_max(weights) <= 1.0);
+
+                        const WeightSumT weight = simd::horizontal_add(weights);
+                        mWeights[tgtid] += weight;
+
+                        NativeT* const wp = &mWeightedPositions[tgtid*3];
+                        wp[0] += (Pwx * weights); // @note: world space position is weighted
+                        wp[1] += (Pwy * weights); // @note: world space position is weighted
+                        wp[2] += (Pwz * weights); // @note: world space position is weighted
+                        mCounts[tgtid] += c;
+                    } //point idx
+                }
+            }
+        } // outer loop
+    }
+
 private:
     WeightSumT* mWeights;
-    WeightedPositionSumT* mWeightedPositions;
     std::vector<int32_t> mCounts;
+    // For batched PCA, we avoid the horizontal reductions in the hot path with
+    // the intermediate SimdT storage. For non-batched, this is unused, and we
+    // write directly to the attribute buffer
+    std::vector<NativeT> mBatchedWeightedPositions;
+    // Points to either the array of doubles (direct write) or the batched data
+    // storage.
+    NativeT* mWeightedPositions;
 };
 
 template <typename PointDataTreeT>
 struct CovarianceTransfer
-    : public PcaTransfer<PointDataTreeT>
+    : public PcaTransfer<CovarianceTransfer<PointDataTreeT>, PointDataTreeT>
 {
-    using BaseT = PcaTransfer<PointDataTreeT>;
+    using BaseT = PcaTransfer<CovarianceTransfer<PointDataTreeT>, PointDataTreeT>;
+    using CovStorageT = std::conditional_t<kPcaBatchSize == 1, float, NativeT>;
 
     static const Index DIM = PointDataTreeT::LeafNodeType::DIM;
     static const Index LOG2DIM = PointDataTreeT::LeafNodeType::LOG2DIM;
@@ -358,14 +547,16 @@ struct CovarianceTransfer
         , mInclusionGroupHandle()
         , mWeights()
         , mWeightedPositions()
-        , mCovMats() {}
+        , mBatchedCovStorage()
+        , mCovComponents() {}
 
     CovarianceTransfer(const CovarianceTransfer& other)
         : BaseT(other)
         , mInclusionGroupHandle()
         , mWeights()
         , mWeightedPositions()
-        , mCovMats() {}
+        , mBatchedCovStorage()
+        , mCovComponents() {}
 
     inline void initialize(const Coord& origin, const size_t idx, const CoordBBox& bounds)
     {
@@ -373,123 +564,203 @@ struct CovarianceTransfer
         mInclusionGroupHandle = std::make_unique<points::GroupHandle>(leaf.groupHandle(this->mIndices.mEllipsesGroupIndex));
         mWeights = initPcaArrayAttribute<WeightSumT>(leaf, this->mIndices.mWeightSumIndex);
         mWeightedPositions = initPcaArrayAttribute<WeightedPositionSumT>(leaf, this->mIndices.mPosSumIndex);
-        mCovMats = initPcaArrayAttribute<math::Mat3s>(leaf, this->mIndices.mCovMatrixIndex);
+
+        // If we're batching, don't init the math::Mat3s buffer yet.
+        // Instead create an intermediate buffer to avoid the horizontal
+        // accumulations until finalize.
+        // @warning  This does increase peak memory relative to the size of
+        //   the native simdt in use.
+        if constexpr (kPcaBatchSize > 1) {
+            mCovComponents = [&]() { // force evaluate in templated context
+                mBatchedCovStorage.assign(this->mTargetPosition->size() * 9, NativeT(0));
+                return mBatchedCovStorage.data();
+            }();
+        }
+        else {
+            // assume contiguously allocated
+            auto* mats = initPcaArrayAttribute<math::Mat3s>(leaf, this->mIndices.mCovMatrixIndex);
+            mCovComponents = mats[0].asPointer();
+        }
     }
 
-    inline void rasterizePoints(const Coord&, const Index start, const Index end, const CoordBBox& bounds)
+    bool finalize(const Coord&, size_t idx)
     {
-        const Index step = std::max(Index(1), Index((end - start) / this->maxSourcePointsPerVoxel()));
+        // if we've written directly to the attribute buffer, return
+        if constexpr (kPcaBatchSize == 1) return true;
+
+        OPENVDB_ASSERT(mCovComponents == mBatchedCovStorage.data());
+        auto& leaf = this->mManager.leaf(idx);
+        // now init the attribute buffer. no need to fill a value as we're
+        // overwriting everything
+        math::Mat3s* mats = initPcaArrayAttribute<math::Mat3s>(leaf, this->mIndices.mCovMatrixIndex, /*fill=*/false);
+
+        for (Index i = 0; i < this->mTargetPosition->size(); ++i) {
+            float* const dst = mats[i].asPointer();
+            CovStorageT* const src = &mCovComponents[i*9];
+            for (size_t j = 0; j < 9; ++j) {
+                dst[j] = static_cast<float>(simd::horizontal_add(src[j]));
+            }
+        }
+
+        return true;
+    }
+
+private:
+    friend BaseT;
+
+    template<typename ScalarT, typename MaskT> /// Real or SimdT
+    inline void stamp(const ScalarT& Pwx,
+                    const ScalarT& Pwy,
+                    const ScalarT& Pwz,
+                    const ScalarT& Pix,
+                    const ScalarT& Piy,
+                    const ScalarT& Piz,
+                    const MaskT& m,
+                    const CoordBBox& intersection)
+    {
+        // Some of the arithmetic in this function assumes these are the same,
+        // otherwise we can end up writing scalars to all lanes of a simdt
+        // (e.g. m[0] += (wx * tx), where wp is a simdt and wx is a scalar -
+        //  this ends up filing all the lanes).
+        static_assert(std::is_same_v<ScalarT, NativeT>);
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Pwx)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Pwy)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Pwz)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Pix)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Piy)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(Piz)));
 
         const auto* const data = this->template buffer<0>();
         const auto& mask = *(this->template mask<0>());
 
-        const float searchRadiusInv = 1.0f/this->searchRadius();
-        const Real searchRadius2 = math::Pow2(this->searchRadius());
+        const ScalarT searchRadiusInv(1.0f / this->searchRadius());
+        const ScalarT searchRadius2(math::Pow2(this->searchRadius()));
         const Real searchRadiusIS2 = math::Pow2(this->searchRadius() * this->mDxInv);
 
-        OPENVDB_ASSERT(std::isfinite(searchRadiusInv));
-        OPENVDB_ASSERT(std::isfinite(searchRadius2));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(searchRadiusInv)));
+        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(searchRadius2)));
         OPENVDB_ASSERT(std::isfinite(searchRadiusIS2));
 
-        const Coord& a(bounds.min());
-        const Coord& b(bounds.max());
+        ScalarT tx, ty, tz, wx, wy, wz;
 
-        for (Index srcid = start; srcid < end; srcid += step) {
-            const Vec3d Psrc(this->mSourcePosition->get(srcid));
-            const Vec3d PsrcIS = Psrc * this->mDxInv;
+        const Coord& a(intersection.min());
+        const Coord& b(intersection.max());
+        for (Coord c = a; c.x() <= b.x(); ++c.x()) {
+            const ScalarT minx(c.x() - 0.5);
+            const ScalarT maxx(c.x() + 0.5);
+            const ScalarT dminx =
+                (simd::select(Pix < minx, simd::pow2(Pix - minx),
+                    (simd::select(Pix > maxx, simd::pow2(Pix - maxx), ScalarT(0)))));
+            if (simd::horizontal_min(dminx) > searchRadiusIS2) continue;
+            const Index i = ((c.x() & (DIM-1u)) << 2*LOG2DIM); // unsigned bit shift mult
+            for (c.y() = a.y(); c.y() <= b.y(); ++c.y()) {
+                const ScalarT miny(c.y() - 0.5);
+                const ScalarT maxy(c.y() + 0.5);
+                const ScalarT dminxy = dminx +
+                    (simd::select(Piy < miny, simd::pow2(Piy - miny),
+                        (simd::select(Piy > maxy, simd::pow2(Piy - maxy), ScalarT(0)))));
+                if (simd::horizontal_min(dminxy) > searchRadiusIS2) continue;
+                const Index ij = i + ((c.y() & (DIM-1u)) << LOG2DIM);
+                for (c.z() = a.z(); c.z() <= b.z(); ++c.z()) {
+                    const Index offset = ij + /*k*/(c.z() & (DIM-1u));
+                    if (!mask.isOn(offset)) continue; // next target voxel
 
-            for (Coord c = a; c.x() <= b.x(); ++c.x()) {
-                const Real minx = c.x() - 0.5;
-                const Real maxx = c.x() + 0.5;
-                const Real dminx =
-                    (PsrcIS[0] < minx ? math::Pow2(PsrcIS[0] - minx) :
-                    (PsrcIS[0] > maxx ? math::Pow2(PsrcIS[0] - maxx) : 0));
-                if (dminx > searchRadiusIS2) continue;
-                const Index i = ((c.x() & (DIM-1u)) << 2*LOG2DIM); // unsigned bit shift mult
-                for (c.y() = a.y(); c.y() <= b.y(); ++c.y()) {
-                    const Real miny = c.y() - 0.5;
-                    const Real maxy = c.y() + 0.5;
-                    const Real dminxy = dminx +
-                        (PsrcIS[1] < miny ? math::Pow2(PsrcIS[1] - miny) :
-                        (PsrcIS[1] > maxy ? math::Pow2(PsrcIS[1] - maxy) : 0));
-                    if (dminxy > searchRadiusIS2) continue;
-                    const Index ij = i + ((c.y() & (DIM-1u)) << LOG2DIM);
-                    for (c.z() = a.z(); c.z() <= b.z(); ++c.z()) {
-                        const Index offset = ij + /*k*/(c.z() & (DIM-1u));
-                        if (!mask.isOn(offset)) continue;
+                    const ScalarT minz(c.z() - 0.5);
+                    const ScalarT maxz(c.z() + 0.5);
+                    const ScalarT dminxyz = dminxy +
+                        (simd::select(Piz < minz, simd::pow2(Piz - minz),
+                            (simd::select(Piz > maxz, simd::pow2(Piz - maxz), ScalarT(0)))));
+                    // Does this point's radius overlap the voxel c
+                    if (simd::horizontal_min(dminxyz) > searchRadiusIS2) continue;
 
-                        const Real minz = c.z() - 0.5;
-                        const Real maxz = c.z() + 0.5;
-                        const Real dminxyz = dminxy +
-                            (PsrcIS[2] < minz ? math::Pow2(PsrcIS[2] - minz) :
-                            (PsrcIS[2] > maxz ? math::Pow2(PsrcIS[2] - maxz) : 0));
-                        // Does this point's radius overlap the voxel c
-                        if (dminxyz > searchRadiusIS2) continue;
+                    const Index targetEnd = data[offset];
+                    const Index targetStart = (offset == 0) ? 0 : Index(data[offset - 1]);
+                    const Index targetStep =
+                        std::max(Index(1), Index((targetEnd - targetStart) / this->maxTargetPointsPerVoxel()));
 
-                        const Index targetEnd = data[offset];
-                        const Index targetStart = (offset == 0) ? 0 : Index(data[offset - 1]);
-                        const Index targetStep =
-                            std::max(Index(1), Index((targetEnd - targetStart) / this->maxTargetPointsPerVoxel()));
-
-                        for (Index tgtid = targetStart; tgtid < targetEnd; tgtid += targetStep) {
-                            if (!mInclusionGroupHandle->get(tgtid)) continue;
+                    for (Index tgtid = targetStart; tgtid < targetEnd; tgtid += targetStep) {
+                        if (!mInclusionGroupHandle->get(tgtid)) continue;
 #if OPENVDB_PCA_SELF_CONTRIBUTION == 0
-                            if (OPENVDB_UNLIKELY(this->mIsSameLeaf && tgtid == srcid)) continue;
+                        if (OPENVDB_UNLIKELY(this->mIsSameLeaf && tgtid == srcid)) continue;
 #endif
-                            const Vec3d Ptgt(this->mTargetPosition->get(tgtid));
-                            const Real d2 = (Psrc - Ptgt).lengthSqr();
-                            if (d2 > searchRadius2) continue;
+                        const math::Vec3<WeightSumT> Ptgt(this->mTargetPosition->get(tgtid));
+                        tx = Pwx - ScalarT(Ptgt[0]);
+                        ty = Pwy - ScalarT(Ptgt[1]);
+                        tz = Pwz - ScalarT(Ptgt[2]);
+                        tx = simd::pow2(tx) + simd::pow2(ty) + simd::pow2(tz);
 
-                            // @note  I've observed some performance degradation if
-                            //   we don't take copies of the buffers here (aliasing?)
-                            const WeightSumT totalWeightInv = mWeights[tgtid];
-                            const WeightedPositionSumT currWeightSum = mWeightedPositions[tgtid];
+                        const MaskT pmask = ((tx <= searchRadius2) && m);
+                        if (!simd::horizontal_or(pmask)) continue;
 
-                            // re-compute weight
-                            // @note  A gather style approach might be better,
-                            //   where each point appends weights/positions into
-                            //   a container. We lose some time having to re-
-                            //   iterate, but this is far more memory efficient.
-                            const WeightSumT weight = 1.0f - math::Pow3(float(math::Sqrt(d2)) * searchRadiusInv);
-                            const WeightedPositionSumT posMeanDiff = Psrc - currWeightSum;
-                            // @note  Could extract the mult by totalWeightInv
-                            //   and put it into a loop in finalize() - except
-                            //   it would be mat3*float rather than vec*float,
-                            //   which would probably better as maxppv increases
-                            const WeightedPositionSumT x = (totalWeightInv * weight) * posMeanDiff;
+                        // @note  I've observed some performance degradation if
+                        //   we don't take copies of the buffers here (aliasing?)
+                        const WeightSumT totalWeightInv = mWeights[tgtid];
+                        const WeightedPositionSumT currWeightSum = mWeightedPositions[tgtid];
 
-                            float* const m = mCovMats[tgtid].asPointer();
-                            /// @note: equal to:
-                            // mat.setCol(0, mat.col(0) + (x * posMeanDiff[0]));
-                            // mat.setCol(1, mat.col(1) + (x * posMeanDiff[1]));
-                            // mat.setCol(2, mat.col(2) + (x * posMeanDiff[2]));
-                            // @todo formalize precision of these methods
-                            m[0] += float(x[0] * posMeanDiff[0]);
-                            m[1] += float(x[0] * posMeanDiff[1]);
-                            m[2] += float(x[0] * posMeanDiff[2]);
-                            //
-                            m[3] += float(x[1] * posMeanDiff[0]);
-                            m[4] += float(x[1] * posMeanDiff[1]);
-                            m[5] += float(x[1] * posMeanDiff[2]);
-                            //
-                            m[6] += float(x[2] * posMeanDiff[0]);
-                            m[7] += float(x[2] * posMeanDiff[1]);
-                            m[8] += float(x[2] * posMeanDiff[2]);
-                        } //point idx
-                    }
+                        // re-compute weight
+                        // @note  A gather style approach might be better,
+                        //   where each point appends weights/positions into
+                        //   a container. We lose some time having to re-
+                        //   iterate, but this is far more memory efficient.
+                        ScalarT weights = ScalarT(1.0) - simd::pow3(simd::sqrt(tx) * searchRadiusInv);
+                        weights = simd::select(pmask, weights, ScalarT(0.0));
+                        OPENVDB_ASSERT(simd::horizontal_and(simd::is_finite(weights)));
+                        OPENVDB_ASSERT(simd::horizontal_min(weights) >= 0.0);
+                        OPENVDB_ASSERT(simd::horizontal_max(weights) <= 1.0);
+
+                        // posMeanDiff
+                        tx = Pwx - ScalarT(currWeightSum[0]);
+                        ty = Pwy - ScalarT(currWeightSum[1]);
+                        tz = Pwz - ScalarT(currWeightSum[2]);
+
+                        // @note  Could extract the mult by totalWeightInv
+                        //   and put it into a loop in finalize() - except
+                        //   it would be mat3*float rather than vec*float,
+                        //   which would probably better as maxppv increases
+                        weights = weights * ScalarT(totalWeightInv);
+                        wx = tx * weights;
+                        wy = ty * weights;
+                        wz = tz * weights;
+
+                        // Either float or SimdT
+                        // @todo  SimdT will be of double precision to avoid
+                        //   the casting backwards and fowards. Might be worth
+                        //   just making the resulting matrix double at some
+                        //   point
+                        auto* const m = &mCovComponents[tgtid * 9];
+                        /// @note: equal to:
+                        // mat.setCol(0, mat.col(0) + (x * posMeanDiff[0]));
+                        // mat.setCol(1, mat.col(1) + (x * posMeanDiff[1]));
+                        // mat.setCol(2, mat.col(2) + (x * posMeanDiff[2]));
+                        m[0] += (wx * tx);
+                        m[1] += (wx * ty);
+                        m[2] += (wx * tz);
+                        //
+                        m[3] += (wy * tx);
+                        m[4] += (wy * ty);
+                        m[5] += (wy * tz);
+                        //
+                        m[6] += (wz * tx);
+                        m[7] += (wz * ty);
+                        m[8] += (wz * tz);
+                    } //point idx
                 }
-            } // outer sdf voxel
-        } // point idx
+            }
+        } // outer sdf voxel
     }
 
-    bool finalize(const Coord&, size_t) { return true; }
-
 private:
     points::GroupHandle::UniquePtr mInclusionGroupHandle;
     const WeightSumT* mWeights;
     const WeightedPositionSumT* mWeightedPositions;
-    math::Mat3s* mCovMats;
     std::vector<int32_t> mCounts;
+    // For batched PCA, we avoid the horizontal reductions in the hot path with
+    // the intermediate SimdT storage. For non-batched, this is unused, and we
+    // write directly to the attribute buffer
+    std::vector<NativeT> mBatchedCovStorage;
+    // Points to either the array of floats (direct write) or the batched data
+    // storage.
+    CovStorageT* mCovComponents;
 };
 
 /// @brief Sort a vector into descending order and output a vector of the resulting order

openvdb/openvdb/simd/Simd.h

@@ -229,6 +229,33 @@ template <> struct SimdT<double, 8> { using Type = simd::Vec8d; };
 template <typename T>
 using SimdNativeT = SimdT<T, OPENVDB_MAX_REGISTER_SIZE/(sizeof(T)*CHAR_BIT)>;
 
+/// @brief  Native type selection for compatible POD types when there
+///   may not be an available Simd type. For example:
+///
+/// @code
+///    // compiled WITHOUT -msse2 (or higher)
+///    static_assert(std::is_same_v<NativeSimdOrScalar<double>::Type, double>);
+///    // compiled WITH -mavx
+///    static_assert(std::is_same_v<NativeSimdOrScalar<double>::Type, SimdT<double, 4>>);
+///@endcode
+template <typename T, typename = void>
+struct NativeSimdOrScalar
+{
+    using Type = T;
+    static constexpr size_t Size = 1;
+};
+
+template <typename T>
+struct NativeSimdOrScalar<T, std::void_t<typename SimdNativeT<T>::Type>>
+{
+    using Type = typename SimdNativeT<T>::Type;
+#ifdef OPENVDB_USE_VCL
+    static constexpr size_t Size = Type::size();
+#else
+    static constexpr size_t Size = Type::size;
+#endif
+};
+
 /// @brief  Traits for determining if a type is a SIMD type
 template <typename T> struct IsSimdMaskT : std::false_type {};
 template <typename T> struct IsSimdIntT : std::false_type {};
@@ -402,6 +429,18 @@ OPENVDB_ENABLE_IF_VCL_MASK inline auto horizontal_or(const T& a) { return openvd
 OPENVDB_ENABLE_IF_VCL_MASK inline int horizontal_count(const T& a) { return openvdb_ext_vcl::horizontal_count(a); }
 OPENVDB_ENABLE_IF_VCL_MASK inline int horizontal_find_first(const T& a) { return openvdb_ext_vcl::horizontal_find_first(a); }
 
+/// @brief  ostream operator for vcl types
+OPENVDB_ENABLE_IF_VCL inline std::ostream& operator<<(std::ostream& os, const T v)
+{
+    os << "[";
+    for (unsigned j(0); j < T::size(); j++) {
+        if (j) os << ", ";
+        os << PrintCast(v[j]);
+    }
+    os << "]";
+    return os;
+}
+
 #else
 
 /// Tuple API

openvdb/openvdb/thread/Threading.h

@@ -13,18 +13,24 @@
     #define OPENVDB_THREAD_THREADING_RESTORE_DEPRECATION_MESSAGE_TASK
 #endif
 
+/// @warning  the below changes what includes are available from this header
+///   depending on whether version.h is available
+#if __has_include(<tbb/version.h>)
+    #include <tbb/version.h>
+#else
+    /// @note tbb/task_arena.h is the ONLY include that persists from TBB 2020
+    ///   to TBB 2021 to TBB 2023 that itself includes the TBB specific version
+    ///   header files.
+    ///   In TBB 2020, the version header was called tbb/stddef.h. In 2021+, it's
+    ///   called tbb/version.h. We include tbb/task_arena.h here to indirectly
+    ///   access the version defines in a consistent way so that downstream
+    ///   software doesn't need to provide compile time defines.
+    #include <tbb/task_arena.h>
+#endif
 
+#include <tbb/task_group.h>
 #include <tbb/blocked_range.h>
 #include <tbb/task.h>
-/// @note tbb/task_arena.h is the ONLY include that persists from TBB 2020
-///   to TBB 2021 to TBB 2023 that itself includes the TBB specific version
-///   header files.
-///   In TBB 2020, the version header was called tbb/stddef.h. In 2021+, it's
-///   called tbb/version.h. We include tbb/task_arena.h here to indirectly
-///   access the version defines in a consistent way so that downstream
-///   software doesn't need to provide compile time defines.
-#include <tbb/task_arena.h>
-#include <tbb/task_group.h>
 
 namespace openvdb {
 OPENVDB_USE_VERSION_NAMESPACE

openvdb/openvdb/tools/Diagnostics.h

@@ -847,7 +847,7 @@ public:
     std::string checkBackground(Real halfWidth = LEVEL_SET_HALF_WIDTH) const
     {
         const Real w = mDiagnose.grid().background() / mDiagnose.grid().voxelSize()[0];
-        if (w < halfWidth) {
+        if (w < halfWidth * (Real(1) - Real(1e-6))) {
             std::ostringstream ss;
             ss << "The background value ("<< mDiagnose.grid().background()<<") is less than "
                << halfWidth << " voxel units\n";

openvdb/openvdb/tools/MeshToVolume.h

@@ -1645,7 +1645,7 @@ checkNeighbours(const Index pos, const typename LeafNodeType::ValueType * data,
     // i + 1, j, k
     if (mask[0] && Compare::check(data[pos + NodeT::DIM * NodeT::DIM]))                   return true;
     // i+1, j, k-1
-    if (mask[6] && Compare::check(data[pos + NodeT::DIM * NodeT::DIM]))                   return true;
+    if (mask[6] && Compare::check(data[pos + NodeT::DIM * NodeT::DIM - 1]))               return true;
     // i-1, j, k-1
     if (mask[7] && Compare::check(data[pos - NodeT::DIM * NodeT::DIM - 1]))               return true;
     // i+1, j, k+1

openvdb/openvdb/tools/PolySoupToLevelSet.h

@@ -285,6 +285,15 @@ PolySoupToLevelSet<GridType>::PolySoupToLevelSet(PolySoup &&poly, int dim, float
         OPENVDB_THROW(ArithmeticError, "polySoupToLevelSet: computed voxel size is not "
             "finite and positive (is dim too small for the given halfWidth?)");
     }
+    // The coarse-to-fine hierarchy in process() requires at least two resolution
+    // levels: the finest (dx = mMinVoxelSize) and one coarser (dx = 2*mMinVoxelSize).
+    // A single level would skip all upsample/shrinkWrap iterations and return only
+    // the raw coarse offset, defeating the purpose of the algorithm. The bound is
+    // therefore mMaxVoxelSize/2, not mMaxVoxelSize. Enforced here as an assert
+    // rather than a throw because a violation always reflects a bad constructor
+    // argument (geometry too small for the requested voxel size or dim), not a
+    // recoverable runtime condition. process() separately guards against the
+    // stricter mMinVoxelSize > mMaxVoxelSize case with an explicit throw.
     OPENVDB_ASSERT(2*mMinVoxelSize <= mMaxVoxelSize);
 }// tools::PolySoupToLevelSet::PolySoupToLevelSet()
 
@@ -314,6 +323,9 @@ PolySoupToLevelSet<GridType>::PolySoupToLevelSet(PolySoup &&poly, float voxelSiz
     if (!math::isFinite(mMaxVoxelSize) || !(mMaxVoxelSize > 0.0f)) {
         OPENVDB_THROW(ArithmeticError, "polySoupToLevelSet: computed voxel size is not finite and positive");
     }
+    // See note in the dim-based constructor above: the bound is mMaxVoxelSize/2
+    // (not mMaxVoxelSize) because one hierarchy level is not enough. Here that
+    // means voxelSize must not exceed maxLength/4 = mMaxVoxelSize/2.
     OPENVDB_ASSERT(2*mMinVoxelSize <= mMaxVoxelSize);
 }// tools::PolySoupToLevelSet::PolySoupToLevelSet()
 
@@ -331,6 +343,17 @@ void PolySoupToLevelSet<GridType>::process(const ShrinkWrapT &D, ProgressT *prog
         mGrids.push_back(this->offset(dx, offset_mode));
     }
 
+    // The loop above produces no grids when mMinVoxelSize > mMaxVoxelSize (i.e. the
+    // requested voxel size exceeds maxLength/2, half the largest bounding-box
+    // dimension). Guard against that here: mGrids.back() below is otherwise
+    // undefined behaviour on an empty vector and crashes in optimized builds.
+    if (mGrids.empty()) {
+        OPENVDB_THROW(ValueError, "PolySoupToLevelSet::process: voxel size (" +
+            std::to_string(mMinVoxelSize) + ") is too large for this mesh; it must not "
+            "exceed maxLength/2 = " + std::to_string(mMaxVoxelSize) +
+            " (half the largest bounding-box dimension)");
+    }
+
     // Coarse to fine shrink wrap algorithm
     double vol[2] = {0.0, 0.0};// levelSetVolume returns Real (double); keep full precision.
                                // Zero-init silences a GCC -Wmaybe-uninitialized false positive:

openvdb/openvdb/tree/ValueAccessor.h

@@ -997,7 +997,7 @@ private:
         return this->evalFirstPred([&](const auto Idx) -> bool
             {
                 if constexpr(Idx < Start)                  return false;
-                else if constexpr(Idx > NumCacheLevels+1)  return false;
+                else if constexpr(static_cast<ptrdiff_t>(Idx) > static_cast<ptrdiff_t>(NumCacheLevels) + 1)  return false;
                 else {
                     using NodeType = typename NodeLevelList::template Get<Idx>;
                     return this->isHashed<NodeType>(xyz);

openvdb/openvdb/unittest/TestDiagnostics.cc

@@ -278,6 +278,16 @@ TEST_F(TestDiagnostics, testCheckLevelSet)
     EXPECT_TRUE(str.empty());
     //std::cerr << "\n" << str << std::endl;
 
+    // A sphere created with voxelSize=0.008 (dim=256, radius=1) exercises a floating-point
+    // edge case where background / voxelSize computes to just below LEVEL_SET_HALF_WIDTH
+    // due to non-representable values, which previously caused a false-positive failure.
+    {
+        FloatGrid::Ptr grid2 = tools::createLevelSetSphere<FloatGrid>(
+            1.0f, Vec3f(0), 0.008f, LEVEL_SET_HALF_WIDTH);
+        str = tools::checkLevelSet(*grid2);
+        EXPECT_TRUE(str.empty());
+    }
+
     grid->tree().setValue(Coord(0,0,0), voxelSize*(width+0.5f));
     //str = c.check();
     str = tools::checkLevelSet(*grid);

openvdb/openvdb/unittest/TestMeshToVolume.cc

@@ -60,6 +60,39 @@ TEST_F(TestMeshToVolume, testUtils)
     EXPECT_TRUE( mZ == 9);
 }
 
+// Regression test for the checkNeighbours(pos, data, mask) overload in
+// mesh_to_volume_internal: each of its 26 branches hand-computes a flat-array
+// offset for one entry of util::COORD_OFFSETS from the node's i/j/k strides, so a
+// copy-paste slip in any single branch (as happened for i+1,j,k-1, which reused
+// the i+1,j,k offset and so silently ignored a negative neighbour on read) goes
+// undetected by callers that never place a negative value at that exact offset.
+// This checks all 26 branches against the offset table directly.
+TEST_F(TestMeshToVolume, testCheckNeighboursOffsets)
+{
+    using namespace openvdb;
+    using namespace openvdb::tools::mesh_to_volume_internal;
+
+    using LeafT = FloatTree::LeafNodeType;
+    struct IsNegative { static bool check(float v) { return v < 0.0f; } };
+
+    // interior voxel, far enough from every face that all 26 neighbours are in-node
+    const Coord center(LeafT::DIM / 2, LeafT::DIM / 2, LeafT::DIM / 2);
+    const Index pos = LeafT::coordToOffset(center);
+
+    for (int m = 0; m < 26; ++m) {
+        std::vector<float> data(LeafT::NUM_VALUES, /*background=*/1.0f);
+        const Coord neighbourIjk = center + util::COORD_OFFSETS[m];
+        data[LeafT::coordToOffset(neighbourIjk)] = -1.0f;
+
+        bool mask[26] = {false};
+        mask[m] = true;
+
+        EXPECT_TRUE((checkNeighbours<IsNegative, LeafT>(pos, data.data(), mask)))
+            << "checkNeighbours missed neighbour offset " << m
+            << " = " << util::COORD_OFFSETS[m];
+    }
+}
+
 TEST_F(TestMeshToVolume, testConversion)
 {
     using namespace openvdb;

openvdb/openvdb/unittest/TestStats.cc

@@ -343,6 +343,31 @@ TEST_F(TestStats, testHistogram)
         for (int i=0; i<N; ++i) EXPECT_TRUE(h.add(N/2-i));
         //h.print("print-test");
     }
+    {// A sample equal to the upper bound must land in the LAST bin, not one past it.
+     // Histogram(min, max, numBins) sets mDelta = numBins/(max-min), so val == max
+     // scales to exactly numBins; the range check admits it (mMax is max plus a small
+     // epsilon), which used to index one element past the end of the bin array. This
+     // is trivially reached in practice, e.g. tools::histogram() over a fog volume
+     // whose interior is exactly its maximum value.
+        const size_t numBins = 8;
+        openvdb::math::Histogram h(0.0, 1.0, numBins);
+        EXPECT_TRUE(h.add(1.0));// the upper bound itself
+        EXPECT_EQ(uint64_t(1), h.size());
+        EXPECT_EQ(uint64_t(1), h.count(numBins-1));// counted in the last bin
+        uint64_t total = 0;
+        for (size_t i=0; i<numBins; ++i) total += h.count(i);
+        EXPECT_EQ(uint64_t(1), total);// and counted exactly once overall
+        // The lower bound still lands in the first bin, and out-of-range is rejected.
+        EXPECT_TRUE(h.add(0.0));
+        EXPECT_EQ(uint64_t(1), h.count(0));
+        EXPECT_FALSE(h.add(1.5));
+        EXPECT_FALSE(h.add(-0.5));
+    }
+    {// NaN must be rejected without undefined behaviour (converting NaN to size_t is UB)
+        openvdb::math::Histogram h(0.0, 1.0, 8);
+        EXPECT_FALSE(h.add(std::numeric_limits<double>::quiet_NaN()));
+        EXPECT_EQ(uint64_t(0), h.size());
+    }
 }
 
 namespace {

openvdb_cmd/vdb_tool/CMakeLists.txt

@@ -256,6 +256,9 @@ if(OPENVDB_BUILD_VDB_TOOL_UNITTESTS)
 
   add_executable(vdb_tool_test src/unittest.cpp)
   target_include_directories(vdb_tool_test PRIVATE vdb_tool_common)
-  target_link_libraries(vdb_tool_test PRIVATE vdb_tool_common GTest::gmock GTest::gtest GTest::gmock_main GTest::gtest_main)
+  # The test provides its own main() and uses only gtest (no gmock), so link just
+  # GTest::gtest. Listing gmock / *_main as well pulls gtest/gmock onto the link
+  # line multiple times, which the macOS linker flags as "ignoring duplicate libraries".
+  target_link_libraries(vdb_tool_test PRIVATE vdb_tool_common GTest::gtest)
   add_test(vdb_tool_unit_test vdb_tool_test)
 endif()

openvdb_cmd/vdb_tool/README.md

@@ -7,25 +7,28 @@ The vdb_tool is a versatile yet lightweight command-line utility that chains tog
 | Action | Description |
 |---|---|
 | **ax** | run an OpenVDB AX expression over selected grids (requires the openvdb_ax library and LLVM) |
-| **calc** | calculate string expression |
+| **calc** (alias: math) | calculate string expression |
 | **case** | case branch inside a -switch scope. Body runs only if key matches the parent -switch's selector. Use key=* or key=default for a catch-all that fires when no earlier case matched. Closed by -end. |
 | **clear** | Deletes geometry, VDB grids and local variables |
 | **clip** | Clip a VDB grid against another grid, a bbox or frustum |
-| **close** | morphological closing, i.e. dilation followed by erosion, of level set surface by a fixed radius |
-| **config** | Import and process one or more configuration files |
+| **close** (alias: closeLS) | morphological closing, i.e. dilation followed by erosion, of level set surface by a fixed radius |
+| **config** (alias: c) | Import and process one or more configuration files |
+| **copy** | Deep-copy VDB grids and/or Geometry by index onto the top of their respective stacks |
 | **cpt** | generate a vector grid with the closest-point-transform to a level set surface |
 | **curl** | generate a vector grid with the curl of another vector grid |
 | **curvature** | generate scalar grid with the mean curvature of a level set surface |
 | **debug** | print debugging information to the terminal |
 | **default** | define default values to be used by subsequent actions |
+| **diagnose** | Run OpenVDB diagnostics checks on one or more VDB grids |
 | **difference** | CSG difference of two level sets surfaces |
-| **dilate** | dilate level set surface by a fixed radius |
+| **dilate** (alias: dilateLS) | dilate level set surface by a fixed radius |
 | **div** | generate a scalar grid with the divergence of a vector grid |
+| **divide** | Given grids A and B, compute a / b per voxel |
 | **each** | start of each-loop over a user-defined loop variable and list of values. |
 | **end** | marks the end scope of "-for, -each, -files, -if, -switch and -case" control actions |
 | **enright** | Performs Enright advection benchmark test on a level set |
-| **erode** | erode level set surface by a fixed radius |
-| **errorOnWarning** | stop on warnings, i.e. treat warnings as errors |
+| **erode** (alias: erodeLS) | erode level set surface by a fixed radius |
+| **errorOnWarning** (alias: stopOnWarning) | stop on warnings, i.e. treat warnings as errors |
 | **eval** | evaluate string expression |
 | **examples** | print examples to the terminal and terminate |
 | **expand** | expand narrow band of level set |
@@ -36,50 +39,56 @@ The vdb_tool is a versatile yet lightweight command-line utility that chains tog
 | **forAllValues** | Applied a simple computational kernel to ALL values in a grid. |
 | **forOffValues** | Applied a simple computational kernel to OFF values in a grid. |
 | **forOnValues** | Applied a simple computational kernel to ON values in a grid. |
-| **gauss** | gaussian convolution of a level set surface |
+| **gauss** (alias: gaussLS) | gaussian convolution of a level set surface |
 | **grad** | generate a vector grid with the gradient of a scalar grid |
-| **help** | Print documentation for one, multiple or all available actions |
+| **help** (alias: h) | Print documentation for one, multiple or all available actions |
+| **histogram** (alias: hist) | Print an ASCII bar histogram of the distribution of active values for one or more VDB grids |
 | **if** | start of if-scope. If the value of its option, named test, evaluates to false the entire scope is skipped |
 | **intersection** | CSG intersection of two level sets surfaces |
-| **iso2ls** | Convert an iso-surface of a scalar field into a level set (i.e. SDF) |
+| **iso2ls** (aliases: lsRebuild, i2l) | Convert an iso-surface of a scalar field into a level set (i.e. SDF) |
 | **length** | generate a scalar grid with the magnitude of a vector grid |
 | **log** | enable logging to file |
-| **ls2fog** | Convert a level set VDB into a VDB with a fog volume, i.e. normalized density. |
-| **ls2mesh** | Convert a level set to an adaptive polygon mesh |
+| **ls2fog** (aliases: l2f, sdf2fog) | Convert a level set VDB into a VDB with a fog volume, i.e. normalized density. |
+| **ls2mesh** (alias: sdf2mesh) | Convert a level set to an adaptive polygon mesh |
 | **max** | Given grids A and B, compute max(a, b) per voxel |
-| **mean** | mean value filtering of a level set surface |
-| **median** | median value filtering of a level set surface |
-| **mesh2ls** | Convert a watertight polygon surface into a narrow-band level set, i.e. a narrow-band signed distance to a polygon mesh |
+| **mean** (alias: meanLS) | mean value filtering of a level set surface |
+| **median** (alias: medianLS) | median value filtering of a level set surface |
+| **mesh2ls** (alias: mesh2sdf) | Convert a watertight polygon surface into a narrow-band level set, i.e. a narrow-band signed distance to a polygon mesh |
 | **min** | Given grids A and B, compute min(a, b) per voxel |
-| **movie** | Convert image and movie files to mpeg or animated gif files |
+| **movie** (aliases: img2mpeg, mov2mpeg, mov2gif, img2gif) | Convert image and movie files to mpeg or animated gif files |
+| **multiply** (alias: mul) | Given grids A and B, compute a * b per voxel |
 | **multires** | construct a LoD sequences of VDB trees with powers of two refinements |
-| **open** | morphological opening, i.e. erosion followed by dilation, of a level set surface by a fixed radius |
+| **open** (alias: openLS) | morphological opening, i.e. erosion followed by dilation, of a level set surface by a fixed radius |
 | **platonic** | Create a level set shape with the specified number of polygon faces |
-| **points2ls** | Convert geometry points into a narrow-band level set |
-| **points2vdb** | Encode geometry points into a VDB grid |
-| **print** | prints information to the terminal about the current stack of VDB grids and Geometry |
+| **points2ls** (aliases: points2sdf, p2l, pts2sdf) | Convert geometry points into a narrow-band level set |
+| **points2vdb** (alias: p2v) | Encode geometry points into a VDB grid |
+| **print** (alias: p) | prints information to the terminal about the current stack of VDB grids and Geometry |
 | **prune** | prune away inactive values in a VDB grid |
-| **quad2tri** | Convert all quads in mesh to triangles, assuming they are both planar and convex |
+| **quad2tri** (alias: q2t) | Convert all quads in mesh to triangles, assuming they are both planar and convex |
 | **quiet** | disable printing to the terminal |
-| **read** | Read one or more geometry or VDB files from disk or STDIN. |
+| **read** (aliases: import, load, i) | Read one or more geometry or VDB files from disk or STDIN. |
+| **rename** | Rename a VDB grid and/or Geometry on the stack by age index |
 | **render** | ray-tracing of level set surfaces and volume rendering of fog volumes |
 | **resample** | resample one VDB grid into another VDB grid or a transformation of the input grid |
 | **scatter** | Scatter point into the active values of an input VDB grid |
 | **sdf2udf** | Converts a signed distance field into an unsigned distance field, i.e. performs the Abs of all values and changes GridClass to UNKNOWN. |
 | **segment** | segment an input VDB into a list if topologically disconnected VDB grids |
 | **slice** | Generate images of slices of a VDB grid |
+| **soup2ls** (aliases: soup2sdf, shrinkwrap) | Convert a polygon soup into a narrow-band level set, i.e. a narrow-band signed distance to a polygon mesh |
 | **soup2offset** | Convert a polygon soup into an offset narrow-band level set, i.e. a narrow-band signed distance to a polygon mesh |
-| **soup2udf** | Convert a polygon soup into a to a unsigned distance field with an symmetrical narrow band |
+| **soup2udf** (alias: mesh2udf) | Convert a polygon soup into a to a unsigned distance field with an symmetrical narrow band |
 | **sphere** | Create a level set sphere, i.e. a narrow-band signed distance to a sphere |
+| **stats** | Print value statistics (min, max, mean, std. dev.) of active voxels for one or more VDB grids |
 | **sum** | Given grids A and B, compute sum(a, b) per voxel |
+| **swap** | Swap two VDB grids and/or two Geometry entries on their respective stacks |
 | **switch** | start of switch-scope. The selector value (on=) is compared against each enclosed -case's key; only the matching case body runs (or the '*'/'default' case if nothing else matched). Closed by -end. |
 | **transform** | apply affine transformations (uniform scale -> rotation -> translation) to a VDB grids and geometry |
 | **union** | CSG union of two level sets surfaces |
-| **vdb2points** | Extract points encoded in a VDB to points in a geometry format |
+| **vdb2points** (alias: v2p) | Extract points encoded in a VDB to points in a geometry format |
 | **verbose** | print timing information to the terminal |
 | **version** | write timing information to the terminal |
-| **vol2mesh** | Convert a scalar volume to an adaptive polygon mesh |
-| **write** | Write list of geometry, VDB or config files to disk or STDOUT |
+| **vol2mesh** (alias: vdb2mesh) | Convert a scalar volume to an adaptive polygon mesh |
+| **write** (aliases: export, save, o) | Write list of geometry, VDB or config files to disk or STDOUT |
 <!-- END AUTO-GENERATED ACTION TABLE -->
 
 For support, bug-reports or ideas for improvements please contact ken.museth@gmail.com
@@ -104,6 +113,20 @@ For support, bug-reports or ideas for improvements please contact ken.museth@gma
 | jpg | optional write | Binary JPEG image file |
 | exr | optional write | Binary OpenEXR image file |
 
+# Inspecting grids: -print vs -stats
+
+Two actions are available for inspecting grids on the stack, and they are intentionally complementary and non-overlapping:
+
+- **`-print`** reports *structural* metadata that is independent of voxel values: grid name, value type, grid class, active voxel count, voxel size, memory usage, and (at `level=1`) the active-voxel bounding box in index and world space plus tree node counts. Nothing printed by `-print` requires reading voxel data.
+
+- **`-stats`** reports *value-derived* information computed from the voxels themselves: background value, min, max, mean, standard deviation of active voxels, and — for level set grids — the world-space surface area and enclosed volume. All columns require touching voxel data.
+
+```bash
+vdb_tool -sphere -print          # structural: type, class, dim, voxels, dx, size
+vdb_tool -sphere -print level=1  # + bounding box and tree topology
+vdb_tool -sphere -stats          # value: background, min, max, mean, stddev, area, volume
+```
+
 # Terminology
 
 We introduce the following terms: **actions**, **options**, **expressions**, and **instructions**. Actions are high-level openvdb tools, which each have unique options, e.g. -mesh2ls geo=1 voxel=0.1, where "-mesh2ls" is an action with two options "geo" and "voxel". Expressions are strings of code with one or more low-level instructions in our stack-based programming language (see below). These expressions start with "{" and ends with "}", and ":" is used to separate values and instructions. E.g. {1:2:+} is an expression with two values (1 and 2) and one instruction "+", and it reduces to the string value "3". See section on the "Stack-based string expressions" below for more details.
@@ -129,6 +152,13 @@ The format is intentionally tiny and line-oriented:
 5. **Leading and trailing whitespace are ignored**, so indenting nested control-flow scopes (`-for` / `-each` / `-files` / `-if` / `-switch` / `-case`) for readability has no effect on behavior.
 6. **No shell quoting is needed** — the line is parsed verbatim. Characters that would otherwise need escaping on the command line (`*`, `{`, `}`, `$`, `(`, `)` etc.) are written plain.
 7. **Blank lines are skipped.**
+8. **Line continuation with `\`** — a line ending in `\` is joined to the next line before parsing, allowing long action invocations to be split across multiple lines for readability:
+   ```
+   sphere \
+     voxel=0.1 \
+     radius=2.0 \
+     name=mysphere
+   ```
 
 Example: the [switch-statement example above](#switch-statement-to-pick-one-of-n-branches) written as a config file demonstrates all of these rules &mdash; one action per line, no leading `-`, nested scopes indented, and `key=*` unquoted.
 
@@ -521,7 +551,7 @@ git clone https://github.com/PixarAnimationStudios/OpenUSD.git
 python3 OpenUSD/build_scripts/build_usd.py \
     --no-imaging --no-usdview --no-alembic --no-draco --no-openimageio \
     --no-tutorials --no-examples ~/dev/src/openusd
-cmake -DOPENVDB_TOOL_USE_USD=ON -Dpxr_DIR=$HOME/local/openusd/lib/cmake/pxr ..
+cmake -DOPENVDB_TOOL_USE_USD=ON -Dpxr_DIR=$HOME/dev/openusd/lib/cmake/pxr ..
 ```
 
 For a slimmer build (skips Alembic, Draco, OpenImageIO, materials, imaging, etc., cutting build time substantially), pass `--no-imaging --no-alembic --no-draco --no-openimageio` to `build_usd.py`. vdb_tool only needs the core USD libraries (`usd`, `usdGeom`, `sdf`, `gf`, `vt`, `tf`).

openvdb_cmd/vdb_tool/include/Geometry.h

@@ -82,7 +82,6 @@
 #include "pdal/PipelineManager.hpp"
 #include "pdal/PipelineReaderJSON.hpp"
 #include "pdal/util/FileUtils.hpp"
-#include <sstream>
 #endif
 
 #if defined(_WIN32)

openvdb_cmd/vdb_tool/include/Parser.h

@@ -1100,6 +1100,17 @@ struct Parser {
     ///        If it returns false, the error is logged as a skip and execution continues.
     ///        The callback receives (action_name, exception_message).
     std::function<bool(const std::string&, const std::string&)> onActionError = nullptr;
+    /// @brief Optional callback applied to an option's value every time it is read,
+    ///        after {} expression expansion but before the caller sees it. Receives the
+    ///        option's name and its expanded value, which it may rewrite in place (e.g.
+    ///        resolving a grid name to a stack age) using state the generic Parser has
+    ///        no access to.
+    /// @note  This deliberately post-processes a COPY on each read rather than rewriting
+    ///        Option::value, so the stored source expression survives. Overwriting it
+    ///        would break loops, where the same Action is revisited and must re-evaluate
+    ///        its expressions each iteration (e.g. "-for v=0,3 -render vdb={$v} -end"),
+    ///        and would also corrupt the source form written out by -write to a config file.
+    std::function<void(const std::string&, std::string&)> onGetOption = nullptr;
 };// Parser struct
 
 // ==============================================================================================================
@@ -1111,6 +1122,8 @@ std::string Parser::getStr(const std::string &name) const
       if (opt.name != name) continue;// linear search
       std::string str = opt.value;// deep copy since it might get modified by map
       processor(str);
+      // Post-process the COPY, never opt.value itself -- see onGetOption's docstring.
+      if (onGetOption) onGetOption(name, str);
       return str;
   }
   throw std::invalid_argument(iter->names[0]+": Parser::getStr: no option named \""+name+"\"");

openvdb_cmd/vdb_tool/include/Tool.h

@@ -34,6 +34,8 @@
 #include <openvdb/util/Assert.h>
 #include <openvdb/tools/Composite.h>
 #include <openvdb/tools/Count.h>// for tools::minMax (used by -print level=2)
+#include <openvdb/tools/Diagnostics.h>
+#include <openvdb/tools/Statistics.h>
 #include <openvdb/tools/FastSweeping.h>
 #include <openvdb/tools/LevelSetAdvect.h>
 #include <openvdb/tools/LevelSetDilatedMesh.h>
@@ -51,11 +53,9 @@
 #include <openvdb/tools/PolySoupToLevelSet.h>
 #include <openvdb/tools/PointScatter.h>
 #include <openvdb/tools/PointsToMask.h>
-#include <openvdb/tools/Composite.h>
 #include <openvdb/tools/VolumeToMesh.h>
 #include <openvdb/tools/GridOperators.h>
 #include <openvdb/tools/GridTransformer.h>
-#include <openvdb/tools/FastSweeping.h>
 #include <openvdb/tools/Prune.h>
 #include <openvdb/tools/Clip.h>
 #include <openvdb/tools/Mask.h> // for tools::interiorMask()
@@ -218,6 +218,24 @@ private:
     /// @brief Delete all queued Geometry, VDB grids, and local variables.
     void clear();
 
+    /// @brief Deep-copy VDB grids and/or Geometry by index onto their respective stacks.
+    void copy();
+
+    /// @brief Run OpenVDB diagnostics checks on VDB grids on the stack.
+    void diagnose();
+
+    /// @brief Print value statistics (min, max, mean, std. dev.) for VDB grids on the stack.
+    void stats();
+
+    /// @brief Print an ASCII bar histogram of the active values of VDB grids on the stack.
+    void histogram();
+
+    /// @brief Rename a VDB grid and/or Geometry on the stack by age index.
+    void rename();
+
+    /// @brief Swap two VDB grids and/or two Geometry entries on their respective stacks.
+    void swap();
+
     /// @brief Clip an input VDB grid against another grid, a bbox, or a frustum.
     /// @tparam GridType Type of the input grid being clipped.
     /// @param v     Numeric parameters defining the clipping region (interpretation depends on mode).
@@ -393,6 +411,22 @@ private:
     /// @throw std::invalid_argument if @a age exceeds the current stack depth.
     inline auto getGeom(size_t age) const;
 
+    /// @brief Translate one "vdb"/"geo" option value, replacing any non-numeric (name)
+    ///        token with the stack age of the entry carrying that exact name.
+    /// @details Installed as mParser.onGetOption so it runs on every read of a "vdb"/"geo"
+    ///          option, letting each action's existing numeric parsing and error handling
+    ///          (age lists, "*", out-of-range checks, etc.) operate unchanged downstream.
+    ///          Resolution happens per-read against the CURRENT stack rather than being
+    ///          written back into the option, so loop bodies re-resolve each iteration.
+    /// @param optName Option name, either "vdb" or "geo" (selects which stack to search).
+    /// @param raw     Fully expression-resolved option value (i.e. after {} substitution).
+    /// @return The translated value: unchanged if empty or "*", otherwise every token is numeric.
+    /// @throw std::out_of_range   if a name token matches no entry on the corresponding stack.
+    /// @throw std::invalid_argument if a name token matches more than one entry. A name must
+    ///        identify exactly one entry, since most actions read "vdb"/"geo" as a single age
+    ///        and those that read a list may mutate the stack while consuming it.
+    std::string resolveStackOption(const std::string &optName, const std::string &raw) const;
+
     /// @brief Convert the output of a VolumeToMesh pass into a Geometry instance.
     Geometry::Ptr mesherToGeometry(tools::VolumeToMesh&) const;
     /// @brief Adaptively mesh a scalar grid at @a isoValue and return the result as a Geometry.
@@ -436,6 +470,13 @@ Tool::Tool(int argc, char *argv[])
         mParser.onActionError = [this](const std::string&, const std::string&) {
             return mErrorOnWarning; // true = fatal (re-throw), false = skip
         };
+        // Translate "vdb"/"geo" name tokens into stack ages on every read of those
+        // options. Every option so named refers to an entry on one of the two stacks,
+        // so no action needs to opt out of this.
+        mParser.onGetOption = [this](const std::string &name, std::string &value) {
+            if (name != "vdb" && name != "geo") return;
+            value = this->resolveStackOption(name, value);
+        };
         mParser.parse(argc, argv);// extremely fast, but might throw
     } catch (const std::exception& e) {
         this->endLog();
@@ -521,6 +562,57 @@ auto Tool::getGeom(size_t age) const
 
 // ==============================================================================================================
 
+std::string Tool::resolveStackOption(const std::string &optName, const std::string &raw) const
+{
+    if (raw.empty() || raw == "*") return raw;// unchanged: every action already special-cases these
+
+    const bool isVdb = optName == "vdb";
+    const size_t stackSize = isVdb ? mGrid.size() : mGeom.size();
+    const std::string &actionName = mParser.getAction().names[0];
+
+    VecS out;
+    for (const std::string &tok : tokenize(raw, "(),")) {
+        int age;
+        if (isInt(tok, age)) { out.push_back(tok); continue; }// already a stack age
+        // Name lookup: scan age 0 (top/most-recent) upward, collecting every match.
+        std::vector<int> matches;
+        int a = 0;
+        if (isVdb) {
+            for (auto it = mGrid.crbegin(); it != mGrid.crend(); ++it, ++a)
+                if ((*it)->getName() == tok) matches.push_back(a);
+        } else {
+            for (auto it = mGeom.crbegin(); it != mGeom.crend(); ++it, ++a)
+                if ((*it)->getName() == tok) matches.push_back(a);
+        }
+        const std::string what = isVdb ? "VDB grid" : "Geometry";
+        if (matches.empty()) {
+            throw std::out_of_range(actionName + ": no " + what + " named \"" + tok +
+                                    "\" on the stack (stack size " + std::to_string(stackSize) + ")");
+        }
+        // A name must identify exactly one entry. Expanding it to several ages would be
+        // unsafe: most actions read "vdb"/"geo" as a single age via get<int>() and cannot
+        // parse a list at all, and those that do read a list may erase entries by index
+        // while consuming it, so a multi-age expansion can delete the wrong grid.
+        if (matches.size() > 1) {
+            std::string ages;
+            for (size_t k = 0; k < matches.size(); ++k) ages += (k ? "," : "") + std::to_string(matches[k]);
+            throw std::invalid_argument(actionName + ": the name \"" + tok + "\" is ambiguous -- it matches " +
+                                        std::to_string(matches.size()) + " " + what + " entries at ages " +
+                                        ages + ". Use an explicit age index to select one.");
+        }
+        out.push_back(std::to_string(matches[0]));
+    }
+
+    std::string result;
+    for (size_t i = 0; i < out.size(); ++i) {
+        if (i) result += ",";
+        result += out[i];
+    }
+    return result;
+}// Tool::resolveStackOption
+
+// ==============================================================================================================
+
 void Tool::run()
 {
     if (mParser.verbose>1) this->print_args();
@@ -642,6 +734,26 @@ void Tool::init()
      {"checksum", "", "none|partial|full", "specify the type of checksum to compute for NanoVDBs"}},
      [&](){mParser.setDefaults();}, [&](){this->write();}, 0);// anonymous options are treated as to the first option,i.e. "files"
 
+  mParser.addAction(
+     {"rename"}, "Rename a VDB grid and/or Geometry on the stack by age index",
+    {{"vdb",  "", "0",      "age index of the VDB grid to rename (omit to skip VDB)"},
+     {"geo",  "", "0",      "age index of the Geometry to rename (omit to skip Geometry)"},
+     {"name", "", "sphere", "new name to assign"}},
+     [](){}, [&](){this->rename();});
+
+  mParser.addAction(
+     {"swap"}, "Swap two VDB grids and/or two Geometry entries on their respective stacks",
+    {{"vdb", "", "0,1", "comma-separated pair of age indices of VDB grids to swap (omit to skip VDB)"},
+     {"geo", "", "0,1", "comma-separated pair of age indices of Geometry to swap (omit to skip Geometry)"}},
+     [](){}, [&](){this->swap();});
+
+  mParser.addAction(
+     {"copy"}, "Deep-copy VDB grids and/or Geometry by index onto the top of their respective stacks",
+    {{"vdb",    "", "0|0,1,2|*", "comma-separated age index/indices of VDB grids to copy, or \"*\" for all (omit to skip VDB)"},
+     {"geo",    "",  "0|0,1|*",  "comma-separated age index/indices of Geometry to copy, or \"*\" for all (omit to skip Geometry)"},
+     {"prefix", "", "copy_",     "prefix prepended to the name of each copy (default is empty, preserving the original name)"}},
+     [&](){mParser.setDefaults();}, [&](){this->copy();});
+
   mParser.addAction(
      {"clear"}, "Deletes geometry, VDB grids and local variables",
     {{"geo", "*", "*|0,1,...", "list of geometries to delete (defaults to all)"},
@@ -749,13 +861,14 @@ void Tool::init()
   // local build with: cmake -DCMAKE_CXX_FLAGS="-DVDB_TOOL_USE_SHRINKWRAP" ..
   mParser.addAction(
      {"soup2ls", "soup2sdf", "shrinkwrap"}, "Convert a polygon soup into a narrow-band level set, i.e. a narrow-band signed distance to a polygon mesh",
-    {{"dim", "", "256", "largest dimension in voxel units of the mesh bbox (defaults to 256). If \"vdb\" or \"voxel\" is defined then \"dim\" is ignored"},
+    {{"dim", "", "256", "largest dimension in voxel units of the mesh bbox (defaults to 256). If \"voxel\" is defined then \"dim\" is ignored"},
      {"voxel", "", "0.01", "voxel size in world units (by defaults \"dim\" is used to derive \"voxel\"). If specified this option takes precedence over \"dim\""},
      {"width", "", "3.0", "half-width in voxel units of the output narrow-band level set (defaults to 3 units on either side of the zero-crossing)"},
      {"mode", "0", "0", "mode of offset operator: 0) old method (using mesh -> UDF -> mesh -> SDF), 1) Mihai's signed-flood-fill and 2) Greg's createLevelSetDilatedMesh. Defaults to 0, i.e. paper."},
      {"geo", "0", "0", "age (i.e. stack index) of the geometry to be processed. Defaults to 0, i.e. most recently inserted geometry."},
      {"erode", "8", "2", "number of iterations of constrained erosion. Defaults to 8."},
      {"thres", "0", "0.01", "closing (or engineering) threshold. Defaults to 0, i.e. it\'s diabled."},
+     {"levels", "0", "0|0,1,2|*", "selects which of the level-set grids generated by the hierarchical shrink-wrap algorithm to output, by resolution level: 0 is the finest (highest-resolution) grid, 1 the next-finest, and so on. Accepts a single index (default 0, i.e. only the finest grid), a comma-separated list (e.g. \"0,1,2\" outputs the three finest grids), or \"*\" to output every generated grid. A runtime error is thrown if a requested level does not exist. Note that these are resolution levels of the output hierarchy, not ages on the VDB stack, which is why this option is named \"levels\" rather than \"vdb\"."},
      {"keep", "", "1|0|true|false", "toggle wether the input geometry is preserved or deleted after the conversion"},
      {"name", "", "soup2ls_input", "specify the name of the resulting vdb (by default it's derived from the input geometry)"}},
      [&](){mParser.setDefaults();}, [&](){this->soupToLevelSet();});
@@ -1018,6 +1131,18 @@ void Tool::init()
      {"keep", "", "1|0|true|false", "toggle wether the input VDBs is preserved or deleted after the processing"}},
      [&](){mParser.setDefaults();}, [&](){this->composite();});
 
+  mParser.addAction(
+     {"multiply", "mul"}, "Given grids A and B, compute a * b per voxel",
+    {{"vdb", "0,1", "0,1", "ages (i.e. stack indices) of the two VDB grids to composit. Defaults to 0,1, i.e. two most recently inserted VDBs."},
+     {"keep", "", "1|0|true|false", "toggle wether the input VDBs is preserved or deleted after the processing"}},
+     [&](){mParser.setDefaults();}, [&](){this->composite();});
+
+  mParser.addAction(
+     {"divide"}, "Given grids A and B, compute a / b per voxel",
+    {{"vdb", "0,1", "0,1", "ages (i.e. stack indices) of the two VDB grids to composit. Defaults to 0,1, i.e. two most recently inserted VDBs."},
+     {"keep", "", "1|0|true|false", "toggle wether the input VDBs is preserved or deleted after the processing"}},
+     [&](){mParser.setDefaults();}, [&](){this->composite();});
+
   mParser.addAction(
      {"multires"}, "construct a LoD sequences of VDB trees with powers of two refinements",
     {{"levels", "2", "2", "number of multi-resolution grids in the output LoD sequence"},
@@ -1124,9 +1249,31 @@ void Tool::init()
       {{"vdb", "*", "*", "print information about VDB grids"},
        {"geo", "*", "*", "print information about geometries"},
        {"mem", "0", "0|1|false|true", "print a list of all stored variables"},
-       {"level", "0", "0|1|2", "detail level: 0=base table, 1=+bbox column, 2=+value range column"}},
+       {"level", "0", "0|1", "detail level: 0=base table, 1=+bbox and node-count columns"}},
       [](){}, [&](){this->print();});
 
+  mParser.addAction(
+      {"diagnose"}, "Run OpenVDB diagnostics checks on one or more VDB grids",
+     {{"vdb",    "*", "*|0|0,1,2", "comma-separated age indices of VDB grids to check, or \"*\" for all (default)"},
+      {"checks", "9", "1|...|9",   "number of checks to run (level set: 1-9, fog volume: 1-6). Lower values skip slower or stricter checks"},
+      {"fatal",  "0", "1|0|true|false", "if true, throw an exception when any check fails (default false)"}},
+     [&](){mParser.setDefaults();}, [&](){this->diagnose();});
+
+  mParser.addAction(
+      {"stats"}, "Print value statistics (min, max, mean, std. dev.) of active voxels for one or more VDB grids",
+     {{"vdb", "*", "*|0|0,1,2", "comma-separated age indices of VDB grids to analyze, or \"*\" for all (default)"}},
+     [](){}, [&](){this->stats();});
+
+  mParser.addAction(
+      {"histogram", "hist"}, "Print an ASCII bar histogram of the distribution of active values for one or more VDB grids",
+     {{"vdb",  "*",  "*|0|0,1,2", "comma-separated age indices of VDB grids to analyze, or \"*\" for all (default)"},
+      {"bins", "10", "10|20|50",  "number of histogram bins (defaults to 10)"},
+      {"min",  "",   "0.0",       "lower bound of the histogram range (defaults to the grid's smallest active value)"},
+      {"max",  "",   "1.0",       "upper bound of the histogram range (defaults to the grid's largest active value)"},
+      {"cols", "40", "40",        "width in characters of the longest bar (defaults to 40)"},
+      {"log",  "false", "1|0|true|false", "use a logarithmic scale for the bar lengths, which makes highly peaked distributions (common for level sets and fog volumes) readable. Defaults to false, i.e. a linear scale"}},
+     [](){}, [&](){this->histogram();});
+
   mParser.addAction(
       {"version"}, "write timing information to the terminal", {},
       [&](){std::clog << mCmdName << ": version " << Tool::version() << std::endl;std::exit(EXIT_SUCCESS);}, [](){});
@@ -1323,7 +1470,13 @@ void Tool::help()
         desc.replace(bar, 1, "\\|");
         bar += 2;
       }
-      std::clog << "| **" << a.names[0] << "** | " << desc << " |\n";
+      std::clog << "| **" << a.names[0] << "**";
+      if (a.names.size() > 1) {
+        std::clog << " (alias" << (a.names.size() > 2 ? "es" : "") << ": ";
+        for (size_t i = 1; i < a.names.size(); ++i) std::clog << (i>1 ? ", " : "") << a.names[i];
+        std::clog << ")";
+      }
+      std::clog << " | " << desc << " |\n";
     }
     if (stop) std::exit(EXIT_SUCCESS);
     return;
@@ -1381,21 +1534,36 @@ std::string Tool::examples() const
 void Tool::clear()
 {
   OPENVDB_ASSERT(mParser.getAction().names[0] == "clear");
+  // Resolve every requested age to a stable list iterator BEFORE erasing anything.
+  // Erasing inside the loop instead would (a) leave earlier deletions in place when
+  // a later age turns out to be out of range, and (b) shift the remaining entries,
+  // since an age is a distance from the back of the list -- so "-clear vdb=0,2" on a
+  // three-grid stack erased age 0 and then failed on age 2, having already destroyed
+  // a grid. Duplicate ages are removed first: resolving one twice would yield the
+  // same iterator twice, and erasing it twice is undefined behaviour. This is safe
+  // because mGrid/mGeom are std::lists, whose iterators stay valid when *other*
+  // elements are erased.
+  auto resolveVictims = [](VecI ages, auto &stack, auto &&get) {
+    std::sort(ages.begin(), ages.end());
+    ages.erase(std::unique(ages.begin(), ages.end()), ages.end());
+    std::vector<typename std::decay_t<decltype(stack)>::const_iterator> victims;
+    victims.reserve(ages.size());
+    for (int a : ages) victims.push_back(std::next(get(a)).base());// throws if out of range
+    return victims;
+  };
   if (mParser.get<std::string>("geo") == "*") {
     mGeom.clear();
   } else {
-    for (int a : mParser.getVec<int>("geo")) {
-      auto it = this->getGeom(a);
-      mGeom.erase(std::next(it).base());
-    }
+    const auto victims = resolveVictims(mParser.getVec<int>("geo"), mGeom,
+                                        [this](int a){ return this->getGeom(a); });
+    for (auto it : victims) mGeom.erase(it);
   }
   if (mParser.get<std::string>("vdb")  == "*") {
     mGrid.clear();
   } else {
-    for (int a : mParser.getVec<int>("vdb")) {
-      auto it = this->getGrid(a);
-      mGrid.erase(std::next(it).base());
-    }
+    const auto victims = resolveVictims(mParser.getVec<int>("vdb"), mGrid,
+                                        [this](int a){ return this->getGrid(a); });
+    for (auto it : victims) mGrid.erase(it);
   }
   if (mParser.get<bool>("variables")) {
     mParser.processor.memory().clear();
@@ -1404,6 +1572,468 @@ void Tool::clear()
 
 // ==============================================================================================================
 
+void Tool::copy()
+{
+  OPENVDB_ASSERT(mParser.getAction().names[0] == "copy");
+  mParser.printAction();
+  const std::string vdb_str = mParser.get<std::string>("vdb");
+  const std::string geo_str = mParser.get<std::string>("geo");
+  const std::string prefix  = mParser.get<std::string>("prefix");
+  if (vdb_str.empty() && geo_str.empty()) {
+    throw std::invalid_argument("copy: at least one of \"vdb\" or \"geo\" must be specified");
+  }
+  auto applyPrefix = [&](const std::string& name) { return prefix + name; };
+  if (!vdb_str.empty()) {
+    std::vector<GridBase::Ptr> copies;
+    if (vdb_str == "*") {
+      for (auto it = mGrid.crbegin(); it != mGrid.crend(); ++it)
+        copies.push_back((*it)->deepCopyGrid());
+    } else {
+      const auto indices = mParser.getVec<int>("vdb");
+      for (int a : indices) {
+        if (size_t(a) >= mGrid.size()) {
+          throw std::out_of_range("copy: vdb index " + std::to_string(a) +
+                                  " is out of range (stack size " + std::to_string(mGrid.size()) + ")");
+        }
+      }
+      copies.reserve(indices.size());
+      for (int a : indices) copies.push_back((*this->getGrid(a))->deepCopyGrid());
+    }
+    for (auto& c : copies) {
+      if (!prefix.empty()) c->setName(applyPrefix(c->getName()));
+      mGrid.push_back(c);
+    }
+  }
+  if (!geo_str.empty()) {
+    std::vector<Geometry::Ptr> copies;
+    if (geo_str == "*") {
+      for (auto it = mGeom.crbegin(); it != mGeom.crend(); ++it)
+        copies.push_back((*it)->deepCopy());
+    } else {
+      const auto indices = mParser.getVec<int>("geo");
+      for (int a : indices) {
+        if (size_t(a) >= mGeom.size()) {
+          throw std::out_of_range("copy: geo index " + std::to_string(a) +
+                                  " is out of range (stack size " + std::to_string(mGeom.size()) + ")");
+        }
+      }
+      copies.reserve(indices.size());
+      for (int a : indices) copies.push_back((*this->getGeom(a))->deepCopy());
+    }
+    for (auto& c : copies) {
+      if (!prefix.empty()) c->setName(applyPrefix(c->getName()));
+      mGeom.push_back(c);
+    }
+  }
+}// Tool::copy
+
+// ==============================================================================================================
+
+void Tool::rename()
+{
+  OPENVDB_ASSERT(mParser.getAction().names[0] == "rename");
+  mParser.printAction();
+  const std::string vdb_str  = mParser.get<std::string>("vdb");
+  const std::string geo_str  = mParser.get<std::string>("geo");
+  const std::string new_name = mParser.get<std::string>("name");
+  if (vdb_str.empty() && geo_str.empty()) {
+    throw std::invalid_argument("rename: at least one of \"vdb\" or \"geo\" must be specified");
+  }
+  if (new_name.empty()) {
+    throw std::invalid_argument("rename: \"name\" must not be empty");
+  }
+  if (!vdb_str.empty()) {
+    const int a = mParser.get<int>("vdb");
+    if (size_t(a) >= mGrid.size()) {
+      throw std::out_of_range("rename: vdb index " + std::to_string(a) +
+                              " is out of range (stack size " + std::to_string(mGrid.size()) + ")");
+    }
+    (*this->getGrid(a))->setName(new_name);
+  }
+  if (!geo_str.empty()) {
+    const int a = mParser.get<int>("geo");
+    if (size_t(a) >= mGeom.size()) {
+      throw std::out_of_range("rename: geo index " + std::to_string(a) +
+                              " is out of range (stack size " + std::to_string(mGeom.size()) + ")");
+    }
+    (*this->getGeom(a))->setName(new_name);
+  }
+}// Tool::rename
+
+// ==============================================================================================================
+
+void Tool::diagnose()
+{
+  OPENVDB_ASSERT(mParser.getAction().names[0] == "diagnose");
+  mParser.printAction();
+  const std::string vdb_str = mParser.get<std::string>("vdb");
+  const bool fatal    = mParser.get<bool>("fatal");
+  const size_t checks = static_cast<size_t>(mParser.get<int>("checks"));
+
+  std::vector<GridBase::Ptr> grids;
+  if (vdb_str == "*") {
+    for (auto it = mGrid.crbegin(); it != mGrid.crend(); ++it) grids.push_back(*it);
+  } else {
+    for (int a : mParser.getVec<int>("vdb")) {
+      if (size_t(a) >= mGrid.size())
+        throw std::out_of_range("diagnose: vdb index " + std::to_string(a) +
+                                " is out of range (stack size " + std::to_string(mGrid.size()) + ")");
+      grids.push_back(*this->getGrid(a));
+    }
+  }
+
+  if (grids.empty()) { std::clog << "diagnose: no grids on stack\n"; return; }
+
+  bool anyFailed = false;
+  for (auto& base : grids) {
+    const std::string& name = base->getName();
+    auto grid = gridPtrCast<FloatGrid>(base);
+    if (!grid) {
+      std::clog << "diagnose: [" << name << "] skipped (not a FloatGrid)\n";
+      continue;
+    }
+    std::string msg;
+    if (base->getGridClass() == GRID_LEVEL_SET) {
+      msg = tools::checkLevelSet(*grid, checks);
+    } else if (base->getGridClass() == GRID_FOG_VOLUME) {
+      msg = tools::checkFogVolume(*grid, checks);
+    } else {
+      // For generic float grids run NaN and infinity checks.
+      tools::Diagnose<FloatGrid> d(*grid);
+      tools::CheckNan<FloatGrid> checkNan;
+      tools::CheckInf<FloatGrid> checkInf;
+      msg += d.check(checkNan, /*updateMask*/false, /*voxels*/true, /*tiles*/true, /*background*/true);
+      msg += d.check(checkInf, /*updateMask*/false, /*voxels*/true, /*tiles*/true, /*background*/true);
+    }
+    if (msg.empty()) {
+      std::clog << "diagnose: [" << name << "] OK\n";
+    } else {
+      anyFailed = true;
+      std::clog << "diagnose: [" << name << "] FAILED\n";
+      // indent each line of the diagnostic message
+      std::istringstream ss(msg);
+      std::string line;
+      while (std::getline(ss, line))
+        if (!line.empty()) std::clog << "  " << line << "\n";
+    }
+  }
+  if (fatal && anyFailed)
+    throw std::runtime_error("diagnose: one or more grids failed validation");
+}// Tool::diagnose
+
+// ==============================================================================================================
+
+void Tool::stats()
+{
+  OPENVDB_ASSERT(mParser.getAction().names[0] == "stats");
+  mParser.printAction();
+  const std::string vdb_str = mParser.get<std::string>("vdb");
+
+  std::vector<GridBase::Ptr> grids;
+  std::vector<int> ages;
+  if (vdb_str == "*") {
+    int age = 0;
+    for (auto it = mGrid.crbegin(); it != mGrid.crend(); ++it, ++age) {
+      grids.push_back(*it);
+      ages.push_back(age);
+    }
+  } else {
+    for (int a : mParser.getVec<int>("vdb")) {
+      if (size_t(a) >= mGrid.size())
+        throw std::out_of_range("stats: vdb index " + std::to_string(a) +
+                                " is out of range (stack size " + std::to_string(mGrid.size()) + ")");
+      grids.push_back(*this->getGrid(a));
+      ages.push_back(a);
+    }
+  }
+
+  if (grids.empty()) { std::clog << "stats: no grids on stack\n"; return; }
+
+  const int aw = 5, w = 14;
+  int nw = 4;// minimum width for "name" header
+  for (auto& g : grids) nw = std::max(nw, (int)g->getName().size());
+  nw += 2;// two-space gap after the longest name
+  std::clog << std::right << std::setw(aw) << "age"
+            << "  "
+            << std::left  << std::setw(nw) << "name"
+            << std::right << std::setw(w)  << "background"
+                          << std::setw(w)  << "min"
+                          << std::setw(w)  << "max"
+                          << std::setw(w)  << "mean"
+                          << std::setw(w)  << "stddev"
+                          << std::setw(w)  << "area"
+                          << std::setw(w)  << "volume"
+            << "\n" << std::string(aw + 2 + nw + 7*w, '-') << "\n";
+
+  // Background column: numeric grid types print their scalar background, vector
+  // grids print the whole vector as "[x, y, z]" (NOT its magnitude, unlike the
+  // value columns below), anything else is "(n/a)".
+  auto backgroundStr = [](const GridBase::Ptr& base) -> std::string {
+    std::stringstream ss;
+    if      (auto p = gridPtrCast<FloatGrid>(base))  ss << p->background();
+    else if (auto p = gridPtrCast<DoubleGrid>(base)) ss << p->background();
+    else if (auto p = gridPtrCast<Int32Grid>(base))  ss << p->background();
+    else if (auto p = gridPtrCast<Int64Grid>(base))  ss << p->background();
+    else if (auto p = gridPtrCast<BoolGrid>(base))   ss << p->background();
+    else if (auto p = gridPtrCast<Vec3SGrid>(base))  ss << p->background();
+    else if (auto p = gridPtrCast<Vec3DGrid>(base))  ss << p->background();
+    else if (auto p = gridPtrCast<Vec3IGrid>(base))  ss << p->background();
+    else return "(n/a)";
+    return ss.str();
+  };
+
+  /// @brief True if @a base is a vector-valued grid, whose value columns therefore
+  ///        report magnitudes rather than the values themselves.
+  auto isVectorGrid = [](const GridBase::Ptr& base) {
+    return gridPtrCast<Vec3SGrid>(base) || gridPtrCast<Vec3DGrid>(base) || gridPtrCast<Vec3IGrid>(base);
+  };
+
+  // min/max/mean/stddev of active voxels, dispatched by value type. tools::statistics()
+  // returns a (non-templated) math::Stats regardless of the grid's value type, so this
+  // covers every scalar type the file formats can round-trip. For vector grids it
+  // reduces each value to its magnitude (see stats_internal::GetValImpl), so these
+  // four columns report statistics of |v| -- flagged by a legend under the table.
+  auto valueStatsStr = [&](const GridBase::Ptr& base) -> std::string {
+    std::stringstream ss;
+    ss << std::fixed << std::setprecision(6);
+    auto printFromStats = [&](const math::Stats& s) {
+      // An empty grid (no active voxels) has no min/max/mean/stddev to report;
+      // math::Stats would otherwise print the ±inf/NaN sentinels it initializes to.
+      if (s.size() == 0) {
+        ss << std::setw(w) << "(empty)" << std::setw(w) << "(empty)"
+           << std::setw(w) << "(empty)" << std::setw(w) << "(empty)";
+      } else {
+        ss << std::setw(w) << s.min() << std::setw(w) << s.max()
+           << std::setw(w) << s.mean() << std::setw(w) << s.stdDev();
+      }
+    };
+    if      (auto p = gridPtrCast<FloatGrid>(base))  printFromStats(tools::statistics(p->cbeginValueOn()));
+    else if (auto p = gridPtrCast<DoubleGrid>(base)) printFromStats(tools::statistics(p->cbeginValueOn()));
+    else if (auto p = gridPtrCast<Int32Grid>(base))  printFromStats(tools::statistics(p->cbeginValueOn()));
+    else if (auto p = gridPtrCast<Int64Grid>(base))  printFromStats(tools::statistics(p->cbeginValueOn()));
+    else if (auto p = gridPtrCast<BoolGrid>(base))   printFromStats(tools::statistics(p->cbeginValueOn()));
+    else if (auto p = gridPtrCast<Vec3SGrid>(base))  printFromStats(tools::statistics(p->cbeginValueOn()));
+    else if (auto p = gridPtrCast<Vec3DGrid>(base))  printFromStats(tools::statistics(p->cbeginValueOn()));
+    else if (auto p = gridPtrCast<Vec3IGrid>(base))  printFromStats(tools::statistics(p->cbeginValueOn()));
+    else ss << std::setw(w) << "(n/a)" << std::setw(w) << "(n/a)" << std::setw(w) << "(n/a)" << std::setw(w) << "(n/a)";
+    return ss.str();
+  };
+
+  bool anyVector = false;
+  for (size_t i = 0; i < grids.size(); ++i) {
+    auto& base = grids[i];
+    const int age = ages[i];
+    anyVector |= isVectorGrid(base);
+    std::clog << std::right << std::setw(aw) << age << "  "
+              << std::left  << std::setw(nw) << base->getName()
+              << std::right << std::setw(w)  << backgroundStr(base)
+                            << valueStatsStr(base);
+    // area and volume are only defined for float level sets
+    auto grid = gridPtrCast<FloatGrid>(base);
+    if (grid && base->getGridClass() == GRID_LEVEL_SET) {
+      try { std::clog << std::setw(w) << tools::levelSetArea(*grid); }
+      catch (...) { std::clog << std::setw(w) << "(n/a)"; }
+      try { std::clog << std::setw(w) << tools::levelSetVolume(*grid); }
+      catch (...) { std::clog << std::setw(w) << "(n/a)"; }
+    } else {
+      std::clog << std::setw(w) << "(n/a)" << std::setw(w) << "(n/a)";
+    }
+    std::clog << "\n";
+  }
+  if (anyVector) {
+    std::clog << "(for vector grids min/max/mean/stddev are of the vector magnitude |v|,"
+                 " while background is the vector itself)\n";
+  }
+}// Tool::stats
+
+// ==============================================================================================================
+
+void Tool::histogram()
+{
+  OPENVDB_ASSERT(mParser.getAction().names[0] == "histogram");
+  mParser.printAction();
+  const std::string vdb_str = mParser.get<std::string>("vdb");
+  const std::string min_str = mParser.get<std::string>("min");
+  const std::string max_str = mParser.get<std::string>("max");
+  const int  bins   = mParser.get<int>("bins");
+  const int  cols   = mParser.get<int>("cols");
+  const bool useLog = mParser.get<bool>("log");
+
+  if (bins < 1) throw std::invalid_argument("histogram: \"bins\" must be at least 1, got "+std::to_string(bins));
+  if (cols < 1) throw std::invalid_argument("histogram: \"cols\" must be at least 1, got "+std::to_string(cols));
+  // An explicit range is optional: either bound may be given on its own, in which
+  // case the other is still derived from the grid's own extrema.
+  const bool hasMin = !min_str.empty(), hasMax = !max_str.empty();
+  const double userMin = hasMin ? mParser.get<double>("min") : 0.0;
+  const double userMax = hasMax ? mParser.get<double>("max") : 0.0;
+  if (hasMin && hasMax && userMin >= userMax) {
+    throw std::invalid_argument("histogram: \"min\" must be less than \"max\", got min="+min_str+" max="+max_str);
+  }
+
+  std::vector<GridBase::Ptr> grids;
+  std::vector<int> ages;
+  if (vdb_str == "*") {
+    int age = 0;
+    for (auto it = mGrid.crbegin(); it != mGrid.crend(); ++it, ++age) {
+      grids.push_back(*it);
+      ages.push_back(age);
+    }
+  } else {
+    for (int a : mParser.getVec<int>("vdb")) {
+      if (size_t(a) >= mGrid.size())
+        throw std::out_of_range("histogram: vdb index " + std::to_string(a) +
+                                " is out of range (stack size " + std::to_string(mGrid.size()) + ")");
+      grids.push_back(*this->getGrid(a));
+      ages.push_back(a);
+    }
+  }
+
+  if (grids.empty()) { std::clog << "histogram: no grids on stack\n"; return; }
+
+  // Bin the active values of one typed grid.
+  // Returns null when there is nothing to plot; the caller uses @a n and @a constant
+  // to distinguish cases: n==0 → no active voxels; n>0 && constant → every active
+  // voxel shares a single value; n>0 && !constant → user-supplied bounds exclude all
+  // grid values (tools::histogram requires a non-empty range).
+  auto build = [&](const auto &grid, double &lo, double &hi, uint64_t &n, bool &constant)
+      -> std::unique_ptr<math::Histogram> {
+    const math::Extrema ex = tools::extrema(grid.cbeginValueOn());
+    n = ex.size();
+    if (n == 0) return nullptr;
+    lo = hasMin ? userMin : ex.min();
+    hi = hasMax ? userMax : ex.max();
+    if (lo >= hi) {
+      // If neither bound came from the user, lo/hi equal ex.min()/ex.max(), so lo>=hi
+      // means the grid is truly constant.  If at least one bound is user-supplied the
+      // grid may span a valid range that simply doesn't intersect [lo, hi).
+      constant = !hasMin && !hasMax;
+      if (constant) lo = ex.min(); // report the actual constant, not a user value
+      return nullptr;
+    }
+    return std::make_unique<math::Histogram>(
+        tools::histogram(grid.cbeginValueOn(), lo, hi, static_cast<size_t>(bins)));
+  };
+
+  // Every value below is printed with an explicit format, since the per-bin rows
+  // switch std::clog to fixed/low precision and that state would otherwise leak
+  // into the next grid's summary line (and into whatever action runs next).
+  const std::ios_base::fmtflags oldFlags = std::clog.flags();
+  const std::streamsize oldPrecision = std::clog.precision();
+  auto summaryFmt = [](std::ostream &os) -> std::ostream& {
+    return os << std::defaultfloat << std::setprecision(6);
+  };
+
+  for (size_t i = 0; i < grids.size(); ++i) {
+    auto &base = grids[i];
+    if (i) std::clog << "\n";
+    std::clog << "histogram: [" << ages[i] << "] \"" << base->getName() << "\" ("
+              << base->valueType() << ")\n";
+
+    double lo = 0.0, hi = 0.0;
+    uint64_t n = 0;
+    bool isVec = false, constant = false;
+    std::unique_ptr<math::Histogram> hist;
+    if      (auto p = gridPtrCast<FloatGrid>(base))  hist = build(*p, lo, hi, n, constant);
+    else if (auto p = gridPtrCast<DoubleGrid>(base)) hist = build(*p, lo, hi, n, constant);
+    else if (auto p = gridPtrCast<Int32Grid>(base))  hist = build(*p, lo, hi, n, constant);
+    else if (auto p = gridPtrCast<Int64Grid>(base))  hist = build(*p, lo, hi, n, constant);
+    else if (auto p = gridPtrCast<BoolGrid>(base))   hist = build(*p, lo, hi, n, constant);
+    // tools::histogram reduces a vector value to its magnitude (see
+    // stats_internal::GetValImpl), so vector grids are binned by |v|. This is what
+    // makes "-grad -histogram" a useful check of the Eikonal condition |grad(phi)|=1.
+    else if (auto p = gridPtrCast<Vec3SGrid>(base)) { hist = build(*p, lo, hi, n, constant); isVec = true; }
+    else if (auto p = gridPtrCast<Vec3DGrid>(base)) { hist = build(*p, lo, hi, n, constant); isVec = true; }
+    else if (auto p = gridPtrCast<Vec3IGrid>(base)) { hist = build(*p, lo, hi, n, constant); isVec = true; }
+    else {
+      std::clog << "  (unsupported value type, skipped)\n";
+      continue;
+    }
+    const char *unit = isVec ? " (by vector magnitude)" : "";
+    if (n == 0) { std::clog << "  (no active voxels)\n"; continue; }
+    if (!hist && constant) {
+      summaryFmt(std::clog) << "  all " << n << " active voxels have the constant "
+                            << (isVec ? "magnitude " : "value ") << lo << "\n";
+      continue;
+    }
+    if (!hist || hist->size() == 0) {// user-supplied range excludes all grid values
+      summaryFmt(std::clog) << "  none of the " << n << " active voxels fall within the requested range ["
+                            << lo << ", " << hi << "]\n";
+      continue;
+    }
+
+    uint64_t peak = 0;
+    for (size_t b = 0; b < hist->numBins(); ++b) peak = std::max(peak, hist->count(b));
+
+    summaryFmt(std::clog) << "  " << hist->size() << " of " << n << " active voxels" << unit
+                          << " in " << hist->numBins() << " bins over [" << lo << ", " << hi << "]"
+                          << (useLog ? ", log scale" : "") << "\n";
+
+    for (size_t b = 0; b < hist->numBins(); ++b) {
+      const uint64_t c = hist->count(b);
+      // Scale bars against the tallest bin so the plot always spans "cols" characters.
+      // log1p keeps empty bins at zero length while compressing very peaked
+      // distributions (common for level sets and fog volumes) into a readable range.
+      int len = 0;
+      if (peak > 0 && c > 0) {
+        const double frac = useLog ? std::log1p(double(c)) / std::log1p(double(peak))
+                                   : double(c) / double(peak);
+        len = std::max(1, static_cast<int>(frac * cols + 0.5));
+      }
+      std::clog << "  [" << std::right << std::setw(12) << std::fixed << std::setprecision(4) << hist->min(b)
+                << ", " << std::setw(12) << hist->max(b)
+                << (b + 1 == hist->numBins() ? "] " : ") ")
+                << std::setw(10) << c << " "
+                << std::setw(5) << std::setprecision(1) << (100.0 * double(c) / double(hist->size())) << "% "
+                << std::defaultfloat << std::string(size_t(len), '#') << "\n";
+    }
+  }
+  std::clog.flags(oldFlags);
+  std::clog.precision(oldPrecision);
+}// Tool::histogram
+
+// ==============================================================================================================
+
+void Tool::swap()
+{
+  OPENVDB_ASSERT(mParser.getAction().names[0] == "swap");
+  mParser.printAction();
+  const std::string vdb_str = mParser.get<std::string>("vdb");
+  const std::string geo_str = mParser.get<std::string>("geo");
+  if (vdb_str.empty() && geo_str.empty()) {
+    throw std::invalid_argument("swap: at least one of \"vdb\" or \"geo\" must be specified");
+  }
+  if (!vdb_str.empty()) {
+    const auto idx = mParser.getVec<int>("vdb");
+    if (idx.size() != 2) {
+      throw std::invalid_argument("swap: \"vdb\" requires exactly two indices, e.g. vdb=0,1");
+    }
+    if (size_t(idx[0]) >= mGrid.size() || size_t(idx[1]) >= mGrid.size()) {
+      throw std::out_of_range("swap: vdb index out of range (stack size " + std::to_string(mGrid.size()) + ")");
+    }
+    if (idx[0] != idx[1]) {
+      auto it0 = mGrid.rbegin(); std::advance(it0, idx[0]);
+      auto it1 = mGrid.rbegin(); std::advance(it1, idx[1]);
+      std::iter_swap(it0, it1);
+    }
+  }
+  if (!geo_str.empty()) {
+    const auto idx = mParser.getVec<int>("geo");
+    if (idx.size() != 2) {
+      throw std::invalid_argument("swap: \"geo\" requires exactly two indices, e.g. geo=0,1");
+    }
+    if (size_t(idx[0]) >= mGeom.size() || size_t(idx[1]) >= mGeom.size()) {
+      throw std::out_of_range("swap: geo index out of range (stack size " + std::to_string(mGeom.size()) + ")");
+    }
+    if (idx[0] != idx[1]) {
+      auto it0 = mGeom.rbegin(); std::advance(it0, idx[0]);
+      auto it1 = mGeom.rbegin(); std::advance(it1, idx[1]);
+      std::iter_swap(it0, it1);
+    }
+  }
+}// Tool::swap
+
+// ==============================================================================================================
+
 void Tool::read()
 {
   OPENVDB_ASSERT(mParser.getAction().names[0] == "read");
@@ -1555,11 +2185,22 @@ void Tool::config()
             Header header(line);
             if (!header.isCompatible()) throw std::invalid_argument("readConf: incompatible version \""+line+"\"");
             std::vector<char*> args({&header.mMagic[0]});//parser is expecting first argument to the name of the executable
+            std::string accum;
             while (getline(file, line)) {
                 const size_t start = line.find_first_not_of(" \t"), stop = line.find_first_of("#%");
+                // A blank or comment-only line is skipped without touching accum, so a
+                // comment line in the middle of a backslash continuation is transparently
+                // absorbed rather than breaking (or being appended to) the continued line.
                 if (start >= stop) continue;// line is empty or starts with a comment
                 line = line.substr(start, stop - start);// remove leading whitespaces and tailing comments
                 line = line.substr(0, line.find_last_not_of(" \t") + 1);// remove tailing whitespaces
+                if (!line.empty() && line.back() == '\\') {
+                    accum += line.substr(0, line.size() - 1);// strip \ and accumulate
+                    accum += ' ';// separate this line's tokens from the next line's
+                    continue;
+                }
+                line = accum + line;
+                accum.clear();
                 VecS tmp = vdb_tool::tokenize(line, " ");
                 tmp[0].insert (0, 1, '-');// first token is an action
                 std::transform(tmp.begin(), tmp.end(), std::back_inserter(args), [](const std::string &s){
@@ -1568,6 +2209,9 @@ void Tool::config()
                     return c;
                 });
             }
+            if (!accum.empty()) {
+                throw std::invalid_argument("readConf: unterminated line continuation at end of file \""+fileName+"\"");
+            }
             file.close();
             mParser.parse(static_cast<int>(args.size()), args.data());
             if (mParser.verbose) mTimer.stop();
@@ -2180,6 +2824,9 @@ void Tool::soupToLevelSet()
   const float thres = mParser.get<float>("thres");
   const bool keep = mParser.get<bool>("keep");
   std::string grid_name = mParser.get<std::string>("name");
+  // Output selector (NB: selects outputs, not inputs): "*" for all generated
+  // grids, otherwise a list of resolution levels (0 = finest). Defaults to "0".
+  const std::string level_sel = mParser.get<std::string>("levels");
 
   auto it = this->getGeom(geo_age);
   Geometry::Ptr mesh = *it;
@@ -2193,13 +2840,48 @@ void Tool::soupToLevelSet()
   Spinner spin, *progress = mParser.verbose ? &spin : nullptr;
   const tools::ShrinkWrapLimit D(nErode, thres);
   tools::PolySoup poly{std::move(mesh->vtx()), std::move(mesh->tri()), std::move(mesh->quad()), mesh->bbox()};
-  auto grid = tools::polySoupToLevelSet<GridT>(std::move(poly), dim, voxel, D, width, progress, offset_mode);
+
+  // Build the LOD hierarchy directly (rather than the single-grid free function)
+  // so we can output more than just the finest level. grids() is ordered
+  // finest(0) -> coarsest(size-1), matching the "levels" indices below.
+  using SW = tools::PolySoupToLevelSet<GridT>;
+  auto sw = voxel > 0.0f ? std::make_unique<SW>(std::move(poly), voxel, width)
+                         : std::make_unique<SW>(std::move(poly), dim, width);
+  sw->process(D, progress, offset_mode);
 
   if (mParser.verbose) mTimer.stop();
 
+  const std::vector<GridT::Ptr> grids = sw->grids();// finest(0) -> coarsest
+  const int count = static_cast<int>(grids.size());
+
+  // Resolve the selector into a list of level indices. "*" -> every level.
+  std::vector<int> levels;
+  if (level_sel == "*") {
+    for (int i = 0; i < count; ++i) levels.push_back(i);
+  } else {
+    levels = mParser.getVec<int>("levels");
+    if (levels.empty()) levels.push_back(0);
+  }
+
+  // Validate every requested level BEFORE pushing any grid, so an out-of-range
+  // index produces a clean error with no partial output on the stack.
+  for (const int lvl : levels) {
+    if (lvl < 0 || lvl >= count) {
+      throw std::invalid_argument("soup2ls: requested output grid levels=" + std::to_string(lvl) +
+          " does not exist; the shrink-wrap hierarchy generated " + std::to_string(count) +
+          " grid(s), so valid levels are 0 (finest) .. " + std::to_string(count - 1) + " (coarsest)");
+    }
+  }
+
   if (grid_name.empty()) grid_name = "soup2ls_" + mesh->getName();
-  grid->setName(grid_name);
-  mGrid.push_back(grid);
+  const bool multi = levels.size() > 1;
+  for (const int lvl : levels) {
+    GridT::Ptr g = grids[lvl];
+    // Disambiguate names only when several grids are output; a single output
+    // keeps the plain name (preserving the historical single-grid behavior).
+    g->setName(multi ? grid_name + "_" + std::to_string(lvl) : grid_name);
+    mGrid.push_back(g);
+  }
   if (!keep) mGeom.erase(std::next(it).base());
 }// Tool::soupToLevelSet
 #endif// VDB_TOOL_USE_SHRINKWRAP
@@ -2563,7 +3245,7 @@ void Tool::compute()
 
 void Tool::composite()
 {
-  OPENVDB_ASSERT(findMatch(mParser.getAction().names[0], {"min","max","sum"}));
+  OPENVDB_ASSERT(findMatch(mParser.getAction().names[0], {"min","max","sum","multiply","divide"}));
   mParser.printAction();
   const std::string &action_name = mParser.getAction().names[0];
   const VecI ij = mParser.getVec<int>("vdb");
@@ -2594,6 +3276,10 @@ void Tool::composite()
     tools::compMax(*tmpA, *tmpB);// Store the result in the A grid and leave the B grid empty.
   } else if (action_name == "sum") {
     tools::compSum(*tmpA, *tmpB);// Store the result in the A grid and leave the B grid empty.
+  } else if (action_name == "multiply") {
+    tools::compMul(*tmpA, *tmpB);// Store the result in the A grid and leave the B grid empty.
+  } else if (action_name == "divide") {
+    tools::compDiv(*tmpA, *tmpB);// Store the result in the A grid and leave the B grid empty.
   } else {
     throw std::invalid_argument(action_name+": invalid operation");
   }
@@ -3884,80 +4570,6 @@ inline std::string formatBBoxd(const BBoxT& bbox)
     return ss.str();
 }
 
-template <typename GridT>
-inline std::string gridRangeStr(const GridT& grid)
-{
-    if (grid.activeVoxelCount() == 0) return "(empty)";
-    const auto mm = tools::minMax(grid.tree());
-    std::stringstream ss;
-    ss << "[" << mm.min() << ", " << mm.max() << "]";
-    return ss.str();
-}
-
-// Dispatch min/max stringification across the scalar grid types we know how
-// to compare. Vec3-typed grids etc. get "(n/a)" because ordering isn't
-// well-defined for them.
-inline std::string formatRange(const openvdb::GridBase& grid)
-{
-    if (auto p = dynamic_cast<const openvdb::FloatGrid*>(&grid))  return gridRangeStr(*p);
-    if (auto p = dynamic_cast<const openvdb::DoubleGrid*>(&grid)) return gridRangeStr(*p);
-    if (auto p = dynamic_cast<const openvdb::Int32Grid*>(&grid))  return gridRangeStr(*p);
-    if (auto p = dynamic_cast<const openvdb::Int64Grid*>(&grid))  return gridRangeStr(*p);
-    if (auto p = dynamic_cast<const openvdb::BoolGrid*>(&grid))   return gridRangeStr(*p);
-    return "(n/a)";
-}
-
-// Background value, dispatched by grid type. Falls back to the typed grid's
-// own background() accessor, which prints scalars as numbers and Vec3s as
-// "[x, y, z]".
-template <typename GridT>
-inline std::string gridBgStr(const GridT& grid)
-{
-    std::stringstream ss;
-    ss << grid.background();
-    return ss.str();
-}
-inline std::string formatBackground(const openvdb::GridBase& grid)
-{
-    if (auto p = dynamic_cast<const openvdb::FloatGrid*>(&grid))  return gridBgStr(*p);
-    if (auto p = dynamic_cast<const openvdb::DoubleGrid*>(&grid)) return gridBgStr(*p);
-    if (auto p = dynamic_cast<const openvdb::Int32Grid*>(&grid))  return gridBgStr(*p);
-    if (auto p = dynamic_cast<const openvdb::Int64Grid*>(&grid))  return gridBgStr(*p);
-    if (auto p = dynamic_cast<const openvdb::BoolGrid*>(&grid))   return gridBgStr(*p);
-    if (auto p = dynamic_cast<const openvdb::Vec3SGrid*>(&grid))  return gridBgStr(*p);
-    return "(n/a)";
-}
-
-// World-space surface area / enclosed volume of a level set, via
-// tools::levelSetArea / tools::levelSetVolume. Both throw TypeError unless the
-// grid is a non-empty scalar float level set, so only Float GRID_LEVEL_SET
-// grids are measured; everything else (and any measurement failure) renders
-// as "(n/a)".
-inline std::string formatArea(const openvdb::GridBase& grid)
-{
-    auto p = dynamic_cast<const openvdb::FloatGrid*>(&grid);
-    if (p == nullptr || grid.getGridClass() != openvdb::GRID_LEVEL_SET) return "(n/a)";
-    try {
-        std::stringstream ss;
-        ss << openvdb::tools::levelSetArea(*p);
-        return ss.str();
-    } catch (const std::exception&) {
-        return "(n/a)";
-    }
-}
-inline std::string formatVolume(const openvdb::GridBase& grid)
-{
-    auto p = dynamic_cast<const openvdb::FloatGrid*>(&grid);
-    if (p == nullptr || grid.getGridClass() != openvdb::GRID_LEVEL_SET) return "(n/a)";
-    try {
-        std::stringstream ss;
-        ss << openvdb::tools::levelSetVolume(*p);
-        return ss.str();
-    } catch (const std::exception&) {
-        return "(n/a)";
-    }
-}
-
 // Per-level node counts, top-down, e.g. "2->10->58" = 2 upper-internal nodes,
 // 10 lower-internal nodes, 58 leaf nodes. TreeBase::nodeCount() returns counts
 // bottom-up (index 0 = leaf) sized treeDepth, with the last element being the
@@ -4044,7 +4656,6 @@ void Tool::print(std::ostream& os) const
     std::vector<std::string> geomHeaders = {"age", "name", "vtx", "tri", "quad", "size"};
     std::vector<Align>       geomAligns  = {RIGHT, LEFT,   RIGHT, RIGHT, RIGHT,  RIGHT};
     if (level >= 1) { geomHeaders.push_back("bbox"); geomAligns.push_back(LEFT); }
-    if (level >= 2) { geomHeaders.push_back("rgb");  geomAligns.push_back(RIGHT); }
 
     auto buildGeomRow = [&](int age, const Geometry& geom) {
         const std::uint64_t mem = sizeof(geom)
@@ -4061,7 +4672,6 @@ void Tool::print(std::ostream& os) const
             formatBytes(mem),
         };
         if (level >= 1) row.push_back(formatBBoxd(geom.bbox()));
-        if (level >= 2) row.push_back(formatCommas(geom.rgb().size()));
         return row;
     };
 
@@ -4088,10 +4698,6 @@ void Tool::print(std::ostream& os) const
     if (level >= 1) { headers.push_back("bbox(index)"); aligns.push_back(LEFT); }
     if (level >= 1) { headers.push_back("bbox(world)"); aligns.push_back(LEFT); }
     if (level >= 1) { headers.push_back("32->16->8");   aligns.push_back(RIGHT); }
-    if (level >= 2) { headers.push_back("background"); aligns.push_back(LEFT); }
-    if (level >= 2) { headers.push_back("range");      aligns.push_back(LEFT); }
-    if (level >= 2) { headers.push_back("area");       aligns.push_back(RIGHT); }
-    if (level >= 2) { headers.push_back("volume");     aligns.push_back(RIGHT); }
 
     auto buildRow = [&](int age, const GridBase& grid) {
         const auto bbox = grid.evalActiveVoxelBoundingBox();
@@ -4114,10 +4720,6 @@ void Tool::print(std::ostream& os) const
         if (level >= 1) row.push_back(formatBBox(bbox));
         if (level >= 1) row.push_back(formatBBoxd(grid.transform().indexToWorld(bbox)));
         if (level >= 1) row.push_back(formatNodes(grid));
-        if (level >= 2) row.push_back(formatBackground(grid));
-        if (level >= 2) row.push_back(formatRange(grid));
-        if (level >= 2) row.push_back(formatArea(grid));
-        if (level >= 2) row.push_back(formatVolume(grid));
         return row;
     };
 

openvdb_cmd/vdb_tool/src/unittest.cpp

@@ -12,6 +12,7 @@
 #include <fstream>
 #include <set>
 #include <thread>
+#include <cmath>// for std::isinf, std::isnan
 
 #if defined(_WIN32)
 #include <direct.h>// for mkdir
@@ -3140,6 +3141,861 @@ TEST_F(Test_vdb_tool, ToolConfigHeader)
     std::remove(cfg.c_str());
 }
 
+// Runtime action failures are caught centrally by Parser::run() and logged
+// as "<action>: skipping due to: <what>" (see Tool's onActionError handler);
+// they do not propagate as C++ exceptions out of Tool::run() unless
+// -errorOnWarning is set, in which case Tool::run() reports the error and
+// calls std::exit() rather than throwing. So the way to observe an
+// action-level error from a test is to capture std::clog, not EXPECT_THROW.
+static std::string runCapturingClog(const std::string& cmd)
+{
+    using namespace openvdb::vdb_tool;
+    const auto tmp = tokenize(cmd, " ");
+    std::vector<std::unique_ptr<char[]>> owned;
+    std::vector<char*> args;
+    for (const auto& s : tmp) {
+        owned.emplace_back(new char[s.size()+1]);
+        std::strcpy(owned.back().get(), s.c_str());
+        args.push_back(owned.back().get());
+    }
+    std::ostringstream oss;
+    auto *old = std::clog.rdbuf(oss.rdbuf());
+    try {
+        Tool tool(int(args.size()), args.data());
+        tool.run();
+    } catch (...) {
+        std::clog.rdbuf(old);
+        throw;
+    }
+    std::clog.rdbuf(old);
+    return oss.str();
+}
+
+// Read back just the (single) grid name stored in a .vdb file, without
+// loading its tree.
+static std::string readGridName(const std::string& path)
+{
+    openvdb::io::File f(path);
+    f.open();
+    const std::string name = f.beginName().gridName();
+    f.close();
+    return name;
+}
+
+TEST_F(Test_vdb_tool, ToolConfigLineContinuation)
+{
+    using namespace openvdb::vdb_tool;
+
+    const std::string cfg = "data/continuation_test.txt";
+    const std::string out_vdb = "data/continuation_out.vdb";
+
+    // Reads back the single grid written by a config run and checks that
+    // the continuation actually joined the sphere's options: the grid must
+    // be named as given on the (continued) name= line, and its world-space
+    // active-voxel bbox must reflect the continued radius= line, not a
+    // truncated/ignored one. A no-op line-continuation implementation (one
+    // that dropped everything after the first backslash) would instead see
+    // a bare "sphere" action with default name/radius and fail these checks.
+    auto checkSphereOutput = [](const std::string& path, const std::string& expectedName, double expectedRadius) {
+        ASSERT_TRUE(openvdb::vdb_tool::fileExists(path));
+        openvdb::io::File f(path);
+        f.open();
+        std::vector<std::string> names;
+        for (auto it = f.beginName(); it != f.endName(); ++it) names.push_back(*it);
+        f.close();
+        ASSERT_EQ(size_t(1), names.size());
+        EXPECT_EQ(expectedName, names[0]);
+
+        openvdb::io::File f2(path);
+        f2.open();
+        auto grid = openvdb::gridPtrCast<openvdb::FloatGrid>(f2.readGrid(names[0]));
+        f2.close();
+        ASSERT_TRUE(grid != nullptr);
+        const auto bbox = grid->transform().indexToWorld(grid->evalActiveVoxelBoundingBox());
+        const double dx = bbox.max().x() - bbox.min().x();
+        // The active bbox spans roughly the diameter (plus narrow-band padding);
+        // a generous tolerance still clearly distinguishes this from other radii.
+        EXPECT_NEAR(2.0 * expectedRadius, dx, 0.5 * expectedRadius);
+    };
+
+    // Write a config that uses backslash line continuation to split a
+    // sphere action and its options (including name=) across multiple lines.
+    std::remove(cfg.c_str());
+    std::remove(out_vdb.c_str());
+    {
+        std::ofstream out(cfg);
+        out << "vdb_tool " << Tool::version() << "\n";
+        out << "sphere \\\n";
+        out << "  dim=64 \\\n";
+        out << "  radius=2.0 \\\n";
+        out << "  name=cont_sphere\n";
+        out << "write " << out_vdb << "\n";
+    }
+    ASSERT_TRUE(fileExists(cfg));
+
+    // The config must load and produce a grid named "cont_sphere" with
+    // radius=2.0 actually applied.
+    EXPECT_NO_THROW({
+      auto args = getArgs("vdb_tool -quiet -config " + cfg);
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+    checkSphereOutput(out_vdb, "cont_sphere", 2.0);
+    std::remove(out_vdb.c_str());
+
+    // Verify that comments on a continuation line are stripped before the
+    // backslash is checked, so the backslash must precede any comment, and
+    // that the resulting sphere still gets its continued options (a
+    // different radius than the first case, so the two can't be confused).
+    std::remove(cfg.c_str());
+    {
+        std::ofstream out(cfg);
+        out << "vdb_tool " << Tool::version() << "\n";
+        out << "sphere \\\n";
+        out << "  dim=64 \\\n";
+        out << "  radius=1.5 \\ # backslash before comment continues the line\n";
+        out << "  name=cont_sphere2\n";
+        out << "write " << out_vdb << "\n";
+    }
+    EXPECT_NO_THROW({
+      auto args = getArgs("vdb_tool -quiet -config " + cfg);
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+    checkSphereOutput(out_vdb, "cont_sphere2", 1.5);
+
+    // A continuation joined without a space before the backslash (e.g.
+    // "radius=2.5\") must still get a token separator inserted, so it
+    // doesn't concatenate directly onto the next line's first token (which
+    // would produce a garbled "radius=2.5name=..." token and a confusing
+    // parse error).
+    std::remove(cfg.c_str());
+    std::remove(out_vdb.c_str());
+    {
+        std::ofstream out(cfg);
+        out << "vdb_tool " << Tool::version() << "\n";
+        out << "sphere dim=64 radius=2.5\\\n";
+        out << "name=cont_sphere3\n";
+        out << "write " << out_vdb << "\n";
+    }
+    EXPECT_NO_THROW({
+      auto args = getArgs("vdb_tool -quiet -config " + cfg);
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+    checkSphereOutput(out_vdb, "cont_sphere3", 2.5);
+
+    // A trailing "\" on the last line of a config (nothing left to continue
+    // onto) must be reported, not silently dropped.
+    std::remove(cfg.c_str());
+    std::remove(out_vdb.c_str());
+    {
+        std::ofstream out(cfg);
+        out << "vdb_tool " << Tool::version() << "\n";
+        out << "sphere dim=64 radius=1.0\\\n";
+    }
+    EXPECT_THROW({
+      auto args = getArgs("vdb_tool -quiet -config " + cfg);
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    }, std::invalid_argument);
+
+    std::remove(cfg.c_str());
+    std::remove(out_vdb.c_str());
+}
+
+TEST_F(Test_vdb_tool, ActionCopy)
+{
+    // -copy: "vdb" and "geo" both default to empty ("omit to skip"), so at
+    // least one must be given explicitly; specifying only one must not touch
+    // the other stack, even when it is empty.
+    using namespace openvdb::vdb_tool;
+
+    // Neither vdb= nor geo= given: the "at least one" guard must fire.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere dim=8 -copy");
+      EXPECT_NE(out.find("copy: skipping due to: copy: at least one of"), std::string::npos);
+    }
+
+    // geo=0 with no VDB grids on the stack must not touch (or throw over) the
+    // empty VDB stack, and must not implicitly copy a grid. Write to a .obj
+    // (not .vdb) file so the write actually exercises the Geometry stack.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere -ls2mesh -clear vdb=* geo="
+          " -copy geo=0 -write geo=0 keep=true data/test_copy_geo.obj");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+    ASSERT_TRUE(fileExists("data/test_copy_geo.obj"));
+    std::remove("data/test_copy_geo.obj");
+
+    // vdb=0 with a prefix produces a second, independently-named grid; the
+    // original grid must be left untouched.
+    std::remove("data/test_copy_vdb.vdb");
+    EXPECT_NO_THROW({
+      auto args = getArgs("vdb_tool -quiet -sphere name=orig dim=8"
+                          " -copy vdb=0 prefix=copy_"
+                          " -write vdb=* keep=true data/test_copy_vdb.vdb");
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+    ASSERT_TRUE(fileExists("data/test_copy_vdb.vdb"));
+    {
+      openvdb::io::File f("data/test_copy_vdb.vdb");
+      f.open();
+      std::set<std::string> names;
+      for (auto it = f.beginName(); it != f.endName(); ++it) names.insert(*it);
+      f.close();
+      EXPECT_EQ(size_t(2), names.size());
+      EXPECT_TRUE(names.count("orig"));
+      EXPECT_TRUE(names.count("copy_orig"));
+    }
+    std::remove("data/test_copy_vdb.vdb");
+
+    // copy must be an independent deep copy, not an alias: overwriting the
+    // copy's background via -forAllValues' explicit "background=" option
+    // must leave the original's background untouched.
+    std::remove("data/test_copy_indep_copy.vdb");
+    std::remove("data/test_copy_indep_orig.vdb");
+    EXPECT_NO_THROW({
+      auto args = getArgs("vdb_tool -quiet -sphere name=orig dim=16"
+                          " -copy vdb=0 prefix=copy_"
+                          " -forAllValues vdb=0 v background=999.0"
+                          " -write vdb=0 keep=true data/test_copy_indep_copy.vdb"
+                          " -write vdb=1 data/test_copy_indep_orig.vdb");
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+    {
+      openvdb::io::File cf("data/test_copy_indep_copy.vdb");
+      cf.open();
+      auto copyGrid = openvdb::gridPtrCast<openvdb::FloatGrid>(cf.readGrid(cf.beginName().gridName()));
+      cf.close();
+      openvdb::io::File of("data/test_copy_indep_orig.vdb");
+      of.open();
+      auto origGrid = openvdb::gridPtrCast<openvdb::FloatGrid>(of.readGrid(of.beginName().gridName()));
+      of.close();
+      ASSERT_TRUE(copyGrid != nullptr);
+      ASSERT_TRUE(origGrid != nullptr);
+      EXPECT_NEAR(999.0f, copyGrid->background(), 1e-4f);
+      EXPECT_NE(999.0f, origGrid->background());
+    }
+    std::remove("data/test_copy_indep_copy.vdb");
+    std::remove("data/test_copy_indep_orig.vdb");
+
+    // Indices are validated up front: an out-of-range index in the same
+    // vdb= list must abort the whole copy, leaving the stack untouched
+    // rather than applying the valid indices and skipping the bad one.
+    std::remove("data/test_copy_oob.vdb");
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere name=only dim=8"
+          " -copy vdb=0,5"
+          " -write vdb=* keep=true data/test_copy_oob.vdb");
+      EXPECT_NE(out.find("copy: skipping due to"), std::string::npos);
+    }
+    ASSERT_TRUE(fileExists("data/test_copy_oob.vdb"));
+    {
+      openvdb::io::File f("data/test_copy_oob.vdb");
+      f.open();
+      int count = 0;
+      for (auto it = f.beginName(); it != f.endName(); ++it) ++count;
+      f.close();
+      EXPECT_EQ(1, count);// no copy was made: still just "only" on the stack
+    }
+    std::remove("data/test_copy_oob.vdb");
+}// ActionCopy
+
+TEST_F(Test_vdb_tool, ActionRenameSwap)
+{
+    using namespace openvdb::vdb_tool;
+
+    std::remove("data/test_rename.vdb");
+    EXPECT_NO_THROW({
+      auto args = getArgs("vdb_tool -quiet -sphere name=a dim=8 -sphere name=b dim=8"
+                          " -rename vdb=0 name=renamed"
+                          " -write vdb=* keep=true data/test_rename.vdb");
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+    {
+      openvdb::io::File f("data/test_rename.vdb");
+      f.open();
+      std::set<std::string> names;
+      for (auto it = f.beginName(); it != f.endName(); ++it) names.insert(*it);
+      f.close();
+      // grid 0 (the most recently added, "b") was renamed; "a" is untouched.
+      EXPECT_TRUE(names.count("a"));
+      EXPECT_TRUE(names.count("renamed"));
+      EXPECT_FALSE(names.count("b"));
+    }
+    std::remove("data/test_rename.vdb");
+
+    // rename rejects an empty name and leaves the grid untouched.
+    std::remove("data/test_rename_empty.vdb");
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere name=a dim=8 -rename vdb=0 name="
+          " -write vdb=0 data/test_rename_empty.vdb");
+      EXPECT_NE(out.find("rename: skipping due to"), std::string::npos);
+    }
+    ASSERT_TRUE(fileExists("data/test_rename_empty.vdb"));
+    EXPECT_EQ(std::string("a"), readGridName("data/test_rename_empty.vdb"));
+    std::remove("data/test_rename_empty.vdb");
+
+    // swap actually reorders the stack: before the swap, age 0 is "b" (most
+    // recently added) and age 1 is "a"; after "-swap vdb=0,1" the names at
+    // those two ages must be exchanged.
+    std::remove("data/test_swap_0.vdb");
+    std::remove("data/test_swap_1.vdb");
+    EXPECT_NO_THROW({
+      auto args = getArgs("vdb_tool -quiet -sphere name=a dim=8 -sphere name=b dim=8"
+                          " -swap vdb=0,1"
+                          " -write vdb=0 keep=true data/test_swap_0.vdb"
+                          " -write vdb=1 data/test_swap_1.vdb");
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+    ASSERT_TRUE(fileExists("data/test_swap_0.vdb"));
+    ASSERT_TRUE(fileExists("data/test_swap_1.vdb"));
+    EXPECT_EQ(std::string("a"), readGridName("data/test_swap_0.vdb"));
+    EXPECT_EQ(std::string("b"), readGridName("data/test_swap_1.vdb"));
+    std::remove("data/test_swap_0.vdb");
+    std::remove("data/test_swap_1.vdb");
+
+    // swap requires exactly two distinct ages; a single-entry stack must be
+    // reported as a (non-fatal, by default) action error, not crash the run.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere name=a dim=8 -swap vdb=0,1");
+      EXPECT_NE(out.find("swap: skipping due to"), std::string::npos);
+    }
+
+    // Since -difference is not commutative (A-B != B-A), swapping the two
+    // inputs beforehand must change the result. Two overlapping, differently
+    // sized/offset spheres on a shared transform (same voxel size) so the
+    // difference is purely a CSG effect, not a levelSetRebuild side-effect.
+    auto activeVoxelsAfterDifference = [](const std::string& cmd, const std::string& path) {
+        auto args = getArgs(cmd);
+        Tool tool(int(args.size()), args.data());
+        tool.run();
+        openvdb::io::File f(path);
+        f.open();
+        auto grid = openvdb::gridPtrCast<openvdb::FloatGrid>(f.readGrid(f.beginName().gridName()));
+        f.close();
+        return grid ? grid->activeVoxelCount() : openvdb::Index64(0);
+    };
+    const std::string spheres = " -sphere name=a voxel=0.05 radius=1.0 center=(0,0,0)"
+                                 " -sphere name=b voxel=0.05 radius=0.6 center=(0.5,0,0)";
+    std::remove("data/test_csg_noswap.vdb");
+    std::remove("data/test_csg_swap.vdb");
+    const auto noSwapCount = activeVoxelsAfterDifference(
+        "vdb_tool -quiet" + spheres + " -difference -write vdb=0 data/test_csg_noswap.vdb",
+        "data/test_csg_noswap.vdb");
+    const auto swapCount = activeVoxelsAfterDifference(
+        "vdb_tool -quiet" + spheres + " -swap vdb=0,1 -difference -write vdb=0 data/test_csg_swap.vdb",
+        "data/test_csg_swap.vdb");
+    EXPECT_NE(noSwapCount, swapCount);
+    std::remove("data/test_csg_noswap.vdb");
+    std::remove("data/test_csg_swap.vdb");
+}// ActionRenameSwap
+
+TEST_F(Test_vdb_tool, ActionDiagnose)
+{
+    // -diagnose runs OpenVDB's built-in level-set/fog-volume checks. Build a
+    // level set with an invalid (non-positive) background directly, since
+    // that violates the level-set invariant checkLevelSet() looks for.
+    using namespace openvdb::vdb_tool;
+    using GridT = openvdb::FloatGrid;
+
+    {
+      GridT::Ptr bad = GridT::create(/*background=*/0.0f);
+      bad->setGridClass(openvdb::GRID_LEVEL_SET);
+      bad->setName("bad_ls");
+      openvdb::io::File("data/test_diagnose_bad.vdb").write({bad});
+    }
+
+    // Default (fatal=false): a failing grid is reported as FAILED, and the
+    // action does not escalate to an error.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -read data/test_diagnose_bad.vdb -diagnose");
+      EXPECT_NE(out.find("bad_ls] FAILED"), std::string::npos);
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+
+    // fatal=true: a failing grid raises internally, which Tool::run() (via
+    // the default non-fatal onActionError policy) reports as a skipped action.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -read data/test_diagnose_bad.vdb -diagnose fatal=true");
+      EXPECT_NE(out.find("diagnose: skipping due to: diagnose: one or more grids failed validation"), std::string::npos);
+    }
+
+    // A valid level set must pass, even with fatal=true.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere dim=32 -diagnose fatal=true");
+      EXPECT_NE(out.find("] OK"), std::string::npos);
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+
+    std::remove("data/test_diagnose_bad.vdb");
+}// ActionDiagnose
+
+TEST_F(Test_vdb_tool, ActionStats)
+{
+    // -stats must not silently skip non-FloatGrid value types: background is
+    // reported for any numeric grid type, not just FloatGrid.
+    using namespace openvdb::vdb_tool;
+
+    {
+      openvdb::Int32Grid::Ptr grid = openvdb::Int32Grid::create(/*background=*/7);
+      grid->setName("int_grid");
+      auto acc = grid->getAccessor();
+      acc.setValue(openvdb::Coord(0, 0, 0), 3);
+      acc.setValue(openvdb::Coord(1, 0, 0), 9);
+      openvdb::io::File("data/test_stats_int.vdb").write({grid});
+    }
+
+    std::ostringstream oss;
+    auto *old = std::clog.rdbuf(oss.rdbuf());
+    try {
+      auto args = getArgs("vdb_tool -quiet -read data/test_stats_int.vdb -stats");
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+      std::clog.rdbuf(old);
+    } catch (...) {
+      std::clog.rdbuf(old);
+      throw;
+    }
+    const std::string out = oss.str();
+    // Background (7) must be printed; the old "(not a FloatGrid, skipped)"
+    // fallback must be gone.
+    EXPECT_NE(out.find("int_grid"), std::string::npos);
+    EXPECT_NE(out.find("7"), std::string::npos);
+    EXPECT_EQ(out.find("skipped"), std::string::npos);
+
+    std::remove("data/test_stats_int.vdb");
+
+    // A grid with no active voxels has no min/max/mean/stddev; -stats must
+    // print "(empty)" rather than the +-inf/NaN sentinels math::Stats
+    // initializes to when nothing was ever added to it.
+    {
+      openvdb::FloatGrid::Ptr grid = openvdb::FloatGrid::create(0.0f);
+      grid->setName("empty_grid");
+      openvdb::io::File("data/test_stats_empty.vdb").write({grid});
+    }
+
+    std::ostringstream oss2;
+    auto *old2 = std::clog.rdbuf(oss2.rdbuf());
+    try {
+      auto args = getArgs("vdb_tool -quiet -read data/test_stats_empty.vdb -stats");
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+      std::clog.rdbuf(old2);
+    } catch (...) {
+      std::clog.rdbuf(old2);
+      throw;
+    }
+    const std::string out2 = oss2.str();
+    EXPECT_NE(out2.find("empty_grid"), std::string::npos);
+    EXPECT_NE(out2.find("(empty)"), std::string::npos);
+    EXPECT_EQ(out2.find("inf"), std::string::npos);
+    EXPECT_EQ(out2.find("nan"), std::string::npos);
+
+    std::remove("data/test_stats_empty.vdb");
+
+    // Vector grids report statistics of the vector MAGNITUDE (tools::statistics
+    // reduces a vector value to val.length()), rather than being skipped. The
+    // background column still shows the vector itself, so a legend explains the
+    // mixed interpretation.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=32 -grad -stats vdb=0");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      EXPECT_NE(out.find("grad_sphere"), std::string::npos);
+      EXPECT_NE(out.find("[0, 0, 0]"), std::string::npos);// background, as a vector
+      EXPECT_NE(out.find("vector magnitude |v|"), std::string::npos);// legend
+      // The magnitude columns must hold real numbers, not the old "(n/a)" filler.
+      EXPECT_EQ(out.find("(n/a)      (n/a)"), std::string::npos);
+    }
+
+    // ...and the legend must NOT appear when only scalar grids were reported.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere r=1 dim=16 -stats vdb=0");
+      EXPECT_EQ(out.find("vector magnitude |v|"), std::string::npos);
+    }
+}// ActionStats
+
+TEST_F(Test_vdb_tool, ActionCompositeDivide)
+{
+    // -divide computes A/B in place via tools::compDiv, which is plain
+    // floating-point division: division by zero yields +/-inf rather than
+    // throwing. Exercise both the ordinary and the zero-divisor case.
+    using namespace openvdb::vdb_tool;
+    using GridT = openvdb::FloatGrid;
+
+    auto readResult = [](const std::string& path) {
+        openvdb::io::File f(path);
+        f.open();
+        return openvdb::gridPtrCast<GridT>(f.readGrid(f.beginName().gridName()));
+    };
+
+    // (0,0,0): ordinary division. (1,0,0): active-zero divisor. (2,0,0): A is
+    // active but B is left untouched (inactive, so B contributes its 0.0f
+    // background) -- the "inactive-B-voxel" case called out in review.
+    {
+      GridT::Ptr a = GridT::create(0.0f);
+      a->setName("A");
+      auto aAcc = a->getAccessor();
+      aAcc.setValue(openvdb::Coord(0, 0, 0), 10.0f);
+      aAcc.setValue(openvdb::Coord(1, 0, 0), 4.0f);
+      aAcc.setValue(openvdb::Coord(2, 0, 0), 6.0f);
+      openvdb::io::File("data/test_div_a.vdb").write({a});
+
+      GridT::Ptr b = GridT::create(0.0f);
+      b->setName("B");
+      auto bAcc = b->getAccessor();
+      bAcc.setValue(openvdb::Coord(0, 0, 0), 2.0f);
+      bAcc.setValue(openvdb::Coord(1, 0, 0), 0.0f);
+      // (2,0,0) intentionally left inactive in B.
+      openvdb::io::File("data/test_div_b.vdb").write({b});
+    }
+
+    EXPECT_NO_THROW({
+      // Stack after both reads (top=most recent): [A, B] => age 0 = B, age 1 = A.
+      auto args = getArgs("vdb_tool -quiet -read data/test_div_a.vdb -read data/test_div_b.vdb"
+                          " -divide vdb=1,0"
+                          " -write vdb=0 data/test_div_out.vdb");
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+
+    auto result = readResult("data/test_div_out.vdb");
+    ASSERT_TRUE(result != nullptr);
+    auto acc = result->getConstAccessor();
+    EXPECT_NEAR(5.0f, acc.getValue(openvdb::Coord(0, 0, 0)), 1e-5f);// 10/2
+    EXPECT_TRUE(std::isinf(acc.getValue(openvdb::Coord(1, 0, 0))));// 4/0 (active zero)
+    EXPECT_TRUE(std::isinf(acc.getValue(openvdb::Coord(2, 0, 0))));// 6/0 (inactive-B background)
+
+    std::remove("data/test_div_a.vdb");
+    std::remove("data/test_div_b.vdb");
+    std::remove("data/test_div_out.vdb");
+}// ActionCompositeDivide
+
+TEST_F(Test_vdb_tool, ActionMultiply)
+{
+    // -multiply computes A*B per voxel in place via tools::compMul.
+    using namespace openvdb::vdb_tool;
+    using GridT = openvdb::FloatGrid;
+
+    GridT::Ptr a = GridT::create(0.0f);
+    a->setName("A");
+    a->getAccessor().setValue(openvdb::Coord(0, 0, 0), 3.0f);
+    openvdb::io::File("data/test_mul_a.vdb").write({a});
+
+    GridT::Ptr b = GridT::create(0.0f);
+    b->setName("B");
+    b->getAccessor().setValue(openvdb::Coord(0, 0, 0), 7.0f);
+    openvdb::io::File("data/test_mul_b.vdb").write({b});
+
+    EXPECT_NO_THROW({
+      // Stack after both reads (top=most recent): [A, B] => age 0 = B, age 1 = A.
+      auto args = getArgs("vdb_tool -quiet -read data/test_mul_a.vdb -read data/test_mul_b.vdb"
+                          " -multiply vdb=1,0"
+                          " -write vdb=0 data/test_mul_out.vdb");
+      Tool tool(int(args.size()), args.data());
+      tool.run();
+    });
+
+    openvdb::io::File f("data/test_mul_out.vdb");
+    f.open();
+    auto result = openvdb::gridPtrCast<GridT>(f.readGrid(f.beginName().gridName()));
+    f.close();
+    ASSERT_TRUE(result != nullptr);
+    EXPECT_NEAR(21.0f, result->getConstAccessor().getValue(openvdb::Coord(0, 0, 0)), 1e-5f);// 3*7
+
+    std::remove("data/test_mul_a.vdb");
+    std::remove("data/test_mul_b.vdb");
+    std::remove("data/test_mul_out.vdb");
+}// ActionMultiply
+
+TEST_F(Test_vdb_tool, StackOptionNameResolution)
+{
+    // "vdb"/"geo" options accept a grid/geometry name wherever a stack age is
+    // accepted: Tool::resolveStackOption() (installed as Parser::onGetOption)
+    // translates each non-numeric token into the stack age of the entry with that
+    // exact name, on every read of the option.
+    using namespace openvdb::vdb_tool;
+
+    // A plain name resolves to its (single) age, same as the equivalent index.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere name=a dim=8 -sphere name=b dim=8 -stats vdb=b");
+      EXPECT_NE(out.find("age"), std::string::npos);
+      // age 0 is "b" (most recently added); the name lookup must resolve to it,
+      // not to age 1 ("a").
+      EXPECT_NE(out.find("0  b"), std::string::npos);
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+
+    // A name matching MORE than one entry is rejected rather than expanded to a
+    // list of ages. Expansion would be unsafe: most actions read "vdb"/"geo" as a
+    // single age via get<int>() and cannot parse a list, and list consumers such as
+    // -clear erase by index while iterating, so a multi-age value can delete the
+    // wrong grid. The error names the ambiguous ages so the user can pick one.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere name=a dim=8 -sphere name=other dim=8 -sphere name=a dim=8"
+          " -stats vdb=1,a");
+      EXPECT_NE(out.find("the name \"a\" is ambiguous"), std::string::npos);
+      EXPECT_NE(out.find("ages 0,2"), std::string::npos);
+    }
+
+    // ...and an ambiguous name must abort the action BEFORE it mutates the stack,
+    // so -clear leaves every grid in place rather than partially deleting.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -sphere name=x dim=8 -sphere name=mid dim=8 -sphere name=x dim=8"
+          " -clear vdb=x -print");
+      EXPECT_NE(out.find("is ambiguous"), std::string::npos);
+      // All three grids survive: two named "x" (ages 0 and 2) plus "mid".
+      EXPECT_NE(out.find("0  x"), std::string::npos);
+      EXPECT_NE(out.find("1  mid"), std::string::npos);
+      EXPECT_NE(out.find("2  x"), std::string::npos);
+    }
+
+    // A singleton consumer (rename reads get<int>("vdb")) rejects an ambiguous
+    // name with the same clear message rather than a raw "invalid int" parse error.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere name=x dim=8 -sphere name=mid dim=8 -sphere name=x dim=8"
+          " -rename vdb=x name=r");
+      EXPECT_NE(out.find("is ambiguous"), std::string::npos);
+      EXPECT_EQ(out.find("invalid int"), std::string::npos);
+    }
+
+    // A name that matches nothing on the stack is reported exactly like an
+    // out-of-range numeric index: a non-fatal, logged "skipping due to" error
+    // naming the culprit, not a silent no-op.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere name=a dim=8 -stats vdb=nope");
+      EXPECT_NE(out.find("stats: skipping due to: stats: no VDB grid named \"nope\" on the stack"),
+                std::string::npos);
+    }
+
+    // "*" and "" (omitted) must still pass through untouched.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere name=a dim=8 -stats vdb=*");
+      EXPECT_NE(out.find("0  a"), std::string::npos);
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+
+    // "geo" resolves names against the Geometry stack independently of "vdb".
+    {
+      std::remove("data/test_geo_name.obj");
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere dim=8 -ls2mesh name=mesh_a"
+          " -rename geo=mesh_a name=renamed"
+          " -write geo=0 data/test_geo_name.obj");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      EXPECT_TRUE(fileExists("data/test_geo_name.obj"));
+      std::remove("data/test_geo_name.obj");
+    }
+
+    // REGRESSION: resolution must NOT be written back into the option. A loop body
+    // revisits the same Action every iteration, so the stored value has to remain the
+    // source expression "{$v}" and be re-expanded (and re-resolved) each time. Writing
+    // the first iteration's resolved age back into the option would pin every later
+    // iteration to that same grid.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere name=a dim=8 -sphere name=b dim=8"
+          " -for v=0,2 -stats vdb={$v} -end");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      // Iteration v=0 reports age 0 ("b"), iteration v=1 reports age 1 ("a").
+      EXPECT_NE(out.find("0  b"), std::string::npos);
+      EXPECT_NE(out.find("1  a"), std::string::npos);
+    }
+
+    // Same guarantee for a NAME inside a loop body: it must be re-resolved against
+    // the current stack on every iteration, not frozen after the first.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere name=a dim=8 -for v=0,2 -sphere name=b dim=8"
+          " -stats vdb=a -clear vdb=0 -end");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      // "a" sits at age 1 while the loop-created "b" is on top, both iterations.
+      EXPECT_NE(out.find("1  a"), std::string::npos);
+      EXPECT_EQ(out.find("0  a"), std::string::npos);
+    }
+}// StackOptionNameResolution
+
+TEST_F(Test_vdb_tool, ActionClearMultipleAges)
+{
+    // -clear resolves every requested age to a list iterator before erasing any of
+    // them. Erasing inside the loop shifted the remaining entries (an age is a
+    // distance from the back of the stack) and left earlier deletions in place when
+    // a later age turned out to be invalid.
+    using namespace openvdb::vdb_tool;
+
+    // Stack [a,b,c] => age 0=c, 1=b, 2=a. Clearing 0 and 2 must remove c and a and
+    // leave exactly b. Previously age 0 was erased, the stack shrank to 2, and the
+    // action then failed on age 2 having already destroyed a grid.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -sphere name=a dim=8 -sphere name=b dim=8 -sphere name=c dim=8"
+          " -clear vdb=0,2 -print");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      EXPECT_NE(out.find("0  b"), std::string::npos);
+      EXPECT_EQ(out.find("  a "), std::string::npos);
+      EXPECT_EQ(out.find("  c "), std::string::npos);
+    }
+
+    // An out-of-range age anywhere in the list must abort with NOTHING deleted,
+    // matching how -copy validates every index before making any copies.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -sphere name=a dim=8 -sphere name=b dim=8 -clear vdb=1,99 -print");
+      EXPECT_NE(out.find("skipping due to"), std::string::npos);
+      EXPECT_NE(out.find("0  b"), std::string::npos);
+      EXPECT_NE(out.find("1  a"), std::string::npos);
+    }
+
+    // A repeated age refers to one entry, so it must delete one grid -- and must not
+    // erase the same list iterator twice, which is undefined behaviour.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -sphere name=a dim=8 -sphere name=b dim=8 -sphere name=c dim=8"
+          " -clear vdb=0,0 -print");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      EXPECT_NE(out.find("0  b"), std::string::npos);
+      EXPECT_NE(out.find("1  a"), std::string::npos);
+      EXPECT_EQ(out.find("  c "), std::string::npos);
+    }
+
+    // The Geometry stack takes the identical path.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -sphere dim=8 -ls2mesh name=m1 -sphere dim=8 -ls2mesh name=m2"
+          " -sphere dim=8 -ls2mesh name=m3 -clear vdb=* geo=0,2 -print");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      EXPECT_NE(out.find("0  m2"), std::string::npos);
+      EXPECT_EQ(out.find("m1"), std::string::npos);
+      EXPECT_EQ(out.find("m3"), std::string::npos);
+    }
+}// ActionClearMultipleAges
+TEST_F(Test_vdb_tool, ActionHistogram)
+{
+    // -histogram bins the active values of a grid and renders an ASCII bar chart.
+    using namespace openvdb::vdb_tool;
+
+    // A fog volume is the important case: its interior is exactly its maximum
+    // value, and every one of those voxels lands on the histogram's upper bound.
+    // (That used to overflow math::Histogram's bin array -- see TestStats.cc.)
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=32 -ls2fog -histogram bins=4");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      EXPECT_NE(out.find("4 bins over"), std::string::npos);
+      EXPECT_NE(out.find("#"), std::string::npos);// at least one bar was drawn
+      // The last bin is inclusive ("]"), every other bin half-open (")").
+      EXPECT_NE(out.find("]"), std::string::npos);
+    }
+
+    // Bin counts must sum to the reported sample total.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=32 -histogram bins=5");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      EXPECT_NE(out.find("5 bins over"), std::string::npos);
+    }
+
+    // A constant-valued grid has no range to bin, and must say so rather than
+    // throwing out of math::Histogram (which requires min < max).
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=16 -forOnValues v=1 -histogram");
+      EXPECT_NE(out.find("constant value"), std::string::npos);
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+
+    // Vector grids are binned by magnitude (tools::histogram reduces a vector to
+    // val.length()), NOT skipped. The gradient of a level set is the useful case:
+    // |grad(phi)| should be ~1 for a true signed distance field, so the histogram
+    // doubles as an Eikonal-condition check.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=32 -grad -histogram vdb=0 bins=4");
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+      EXPECT_NE(out.find("(vec3s)"), std::string::npos);
+      // The summary must flag that the bins are magnitudes, not raw values.
+      EXPECT_NE(out.find("(by vector magnitude)"), std::string::npos);
+      EXPECT_NE(out.find("4 bins over"), std::string::npos);
+      EXPECT_EQ(out.find("unsupported value type"), std::string::npos);
+    }
+
+    // A genuinely unsupported value type (e.g. a point grid) is still skipped
+    // with a note rather than raising.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=16 -scatter count=64 -points2vdb -histogram vdb=0");
+      EXPECT_NE(out.find("unsupported value type"), std::string::npos);
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+
+    // An explicit range clips the data, and the summary reports how many of the
+    // active voxels actually fell inside it.
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=32 -ls2fog -histogram bins=2 min=0.9 max=1.0");
+      EXPECT_NE(out.find("2 bins over [0.9, 1]"), std::string::npos);
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+    {// a range that excludes everything is reported, not silently empty
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=16 -ls2fog -histogram min=5 max=6");
+      EXPECT_NE(out.find("none of the"), std::string::npos);
+    }
+
+    // Degenerate bins/cols and an inverted range are rejected with clear errors.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere dim=16 -histogram bins=0");
+      EXPECT_NE(out.find("\"bins\" must be at least 1"), std::string::npos);
+    }
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere dim=16 -histogram cols=0");
+      EXPECT_NE(out.find("\"cols\" must be at least 1"), std::string::npos);
+    }
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere dim=16 -histogram min=1 max=0");
+      EXPECT_NE(out.find("\"min\" must be less than \"max\""), std::string::npos);
+    }
+
+    // An out-of-range age is reported like every other action's vdb= handling.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -sphere dim=16 -histogram vdb=7");
+      EXPECT_NE(out.find("is out of range"), std::string::npos);
+    }
+
+    // With no grids on the stack the action is a no-op, not an error.
+    {
+      const std::string out = runCapturingClog("vdb_tool -quiet -histogram");
+      EXPECT_NE(out.find("no grids on stack"), std::string::npos);
+      EXPECT_EQ(out.find("skipping due to"), std::string::npos);
+    }
+
+    // Per-bin rows switch std::clog to fixed/low precision; that must not leak
+    // into the next grid's summary line (whose range must match its own bin edges).
+    {
+      const std::string out = runCapturingClog(
+          "vdb_tool -quiet -sphere r=1 dim=32 name=a -sphere r=2 dim=32 name=b"
+          " -histogram vdb=* bins=2");
+      EXPECT_NE(out.find("\"a\""), std::string::npos);
+      EXPECT_NE(out.find("\"b\""), std::string::npos);
+      // Six significant digits in both summaries, i.e. not truncated to "[-0.2, 0.2]".
+      EXPECT_NE(out.find("over [-0.230769,"), std::string::npos);
+      EXPECT_NE(out.find("over [-0.461538,"), std::string::npos);
+    }
+}// ActionHistogram
 
 int main(int argc, char** argv)
 {

pendingchanges/distributedtileownership.txt

@@ -0,0 +1,10 @@
+NanoVDB:
+  Bug Fixes:
+  - Fixed a cross-device race in tools::cuda::DistributedPointsToGrid. The
+    device segment boundaries were rebalanced by adjusting only adjacent pairs
+    of boundaries, which cannot consolidate an upper-node tile that spans three
+    or more GPUs, because a fully-interior device lies entirely within the tile.
+    Multiple devices then built the same leaf concurrently and raced on its
+    value mask, silently dropping active voxels. The boundaries are now snapped
+    globally and monotonically to tile boundaries, so every tile - and therefore
+    every lower node, leaf node and voxel - is owned by exactly one device.

pendingchanges/meshtogridcallcubsundef.txt

@@ -0,0 +1,4 @@
+NanoVDB:
+  Bug Fixes:
+  - Fixed tools::cuda::MeshToGrid leaving the CALL_CUBS macro defined past the end of
+    the header, unlike its sibling CUDA tools.

pendingchanges/nanovdb.txt

@@ -2,10 +2,23 @@ NanoVDB:
 
   New Features:
   - Added new _hostdev_ function named nanovdb::math::isoCrossing, which intersects a ray against a user-defined iso-surface.
+  - Added doc/nanovdb/TEACHME, an interactive tutorial for the NanoVDB user API: lesson documents written to be loaded by an LLM coding agent, which then teaches the API interactively to a developer new to NanoVDB. Ships with a cheat sheet, a GPU ray-marching capstone, and a CI harness that compiles every code block in the lesson against the NanoVDB headers so the tutorial cannot drift from the API.
+  - Added nanovdb::cuda::Buffer and nanovdb::cuda::BufferView (CUDA): a typed, resource-aware, stream-ordered container that allocates from an injectable memory resource and frees on its retained stream, and a non-owning view over externally managed memory that a GridHandle can wrap without copying. Member names follow cuda::buffer (destroy, set_stream, swap). Also added the synchronous resource concept nanovdb::cuda::is_resource alongside is_async_resource.
+  - Added nanovdb::cuda::copyTo (CUDA), the explicit, stream-carrying transfer between grid handles in different address spaces: single-space device handles copy to host-readable handles (HostBuffer or a pinned-resource cuda::Buffer) and back, and device-to-device across different resources. An optional prototype buffer routes the destination allocation through a caller-supplied resource. Dual-space handles keep using deviceUpload/deviceDownload.
 
   Improvements:
+  - The GPU builders now allocate all scratch through an injectable memory resource: tools::cuda::TopologyBuilder and tools::cuda::MeshToGrid gained a ResourceT template parameter (defaulted, so existing code is unaffected), joining PointsToGrid. Added nanovdb::cuda::SyncFromAsync, a CRTP base that derives the synchronous half of the resource concept from the stream-ordered half, and nanovdb::cuda::ResourceRef, a non-owning reference to a resource that is itself a resource, for containers that hold their resource by value. PointsToGrid's device scratch and intermediate arrays are now owned by cuda::Buffer as well, replacing all of its hand-paired allocate/free calls. Added nanovdb::cuda::MallocResource, a synchronous cudaMalloc-backed resource that works on devices without memory-pool support, and nanovdb::cuda::AsyncFromSync, which presents any synchronous resource as a stream-ordered one by synchronizing before each deallocation, so the builders can run with an injected synchronous resource.
+  - Added the CMake option NANOVDB_CUDA_WERROR (default ON, also implied by OPENVDB_CXX_STRICT) which passes --Werror=all-warnings to NVCC so that every device-side diagnostic becomes an error when building the NanoVDB tests, tools and examples. Disable it if a newer CUDA toolkit introduces diagnostics that block your build.
+  - nanovdb::GridHandle can now manage a single-space device buffer: GridHandle<nanovdb::cuda::Buffer<std::byte, R>> parses its grid metadata through the device on the buffer's retained stream, exposes deviceData()/deviceGrid() (host accessors and stream I/O are compile-time errors for such handles), and deep-copies device-to-device via copy<BufferT>() ordered on the source's retained stream. Buffers over host-accessible resources (e.g. PinnedResource, detected via the new nanovdb::cuda::is_host_accessible_resource trait) are host-readable single-space handles: metadata parses on the host, the host accessors (grid, gridData, write, read) apply, and reads and copies allocate through the buffer's resource. All handle-internal scratch allocates through the buffer's resource, and pre-existing BufferTraits specializations compile unchanged (the new hasDeviceSingle trait member is detected, not required).
+  - The remaining GPU tools now allocate their device scratch through an injectable memory resource as well: tools::cuda::indexToGrid, tools::cuda::updateGridStats, tools::cuda::signedFloodFill and tools::cuda::addBlindData (and their implementation classes) gained a defaulted ResourceT template parameter, and their internal mallocAsync/freeAsync pairs are replaced by RAII nanovdb::cuda::Buffer scratch. Existing code is unaffected.
+  - nanovdb::cuda::createNodeManager gained a single-space overload: passing a memory resource instead of a pool buffer returns a NodeManagerHandle over cuda::Buffer<std::byte, ResourceRef<ResourceT>> whose storage (and size scratch) allocates through that resource, which must outlive the handle. tools::cuda::GridStats now builds its temporary NodeManager through its injected resource, so every device allocation it makes honors ResourceT. NodeManagerHandle supports single-space buffers (deviceMgr maps onto the buffer, host accessors are compile-time errors) and no longer requires a default-constructible buffer type.
+  - GridHandle::reset, NodeManagerHandle::reset and tools::VoxelBlockManager::reset now release storage through the buffer's destroy() when it provides one, and cuda::Buffer::clear/cuda::BufferView::clear are deprecated in favor of destroy() (BufferView::destroy detaches the non-owning view). The legacy HostBuffer/DeviceBuffer clear() methods are unaffected.
   - The bug-fix to the nanovdb::ReadAccessor (see below) improves random-access performance in some use-cases (especially on the CPU).
 
   Fixes:
+  - nanovdb::cuda::createNodeManager now synchronizes the stream before reading back the device-computed NodeManager size: the host previously consumed the destination of an asynchronous copy without ordering, which happened to work only because device-to-pageable copies degrade to synchronous behavior.
+  - tools::cuda::addBlindData and tools::cuda::indexToGrid now normalize the output grid's index and count fields (mGridIndex=0, mGridCount=1): both tools copy the source grid's header, so a source grid taken from a multi-grid buffer used to leave a stale index/count in the single-grid output, corrupting the metadata parse of the returned handle.
   - Fixed a caching bug in nanovdb::ReadAccessor<LEVEL0, LEVEL1, LEVEL2>, e.g ReadAccessor<0,1,2> and ReadAccessor<0,1>: the accessors now cache nodes at all requested levels (matching the behaviour of openvdb::ValueAccessor) rather than caching only leaf nodes. This is the new default behaviour. The previous (buggy) behaviour can be restored by setting the CMake option -DNANOVDB_USE_OLD_ACCESSOR=ON, or by defining the macro NANOVDB_USE_OLD_ACCESSOR before including NanoVDB.h.
-  - Deprecated nanovdb::math::ZeroCrossingNode and renamed it nanovdb::math::zeroCrossingNode to follow our naming convention.
+  - Deprecated nanovdb::math::ZeroCrossingNode and renamed it nanovdb::math::zeroCrossingNode to follow our naming convention.
+  - Fixed the nvcc #20054-D warnings emitted by the nanovdb::tools::cuda::GridStats kernels: the __shared__ statistics arrays now use constructor-free storage (cub::Uninitialized), since CUDA does not support dynamic initialization of __shared__ variables.
+  - Fixed the nvcc #20050-D warnings emitted by the nanovdb::tools::cuda::GridChecksum CRC32 kernels: the inline qualifier is ignored for __global__ functions, so the header-defined kernels are now static instead.

pendingchanges/nanovdbcorrectness.txt

@@ -0,0 +1,16 @@
+NanoVDB:
+  Bug Fixes:
+  - Fixed an out-of-bounds device read and host stack corruption in
+    tools::cuda::evalChecksum.
+  - Fixed tools::cuda::DilateGrid and signedFloodFill launching kernels on the
+    default stream instead of the caller's stream, corrupting output on
+    non-blocking streams.
+  - Fixed a host-heap over-read when copying grid names in the CUDA
+    PointsToGrid, DistributedPointsToGrid and MeshToGrid tools.
+  - Fixed nondeterministic output from the CUDA PointsToGrid and IndexToGrid tools.
+  - Fixed cuda::DeviceBuffer freeing device memory without ordering the free
+    after work still outstanding on the buffer, a latent use-after-free for
+    callers using non-blocking streams. Each managed device allocation now
+    tracks its uses with an event, so a free is ordered after every stream the
+    buffer was used on rather than only those the freeing stream happens to
+    synchronize with.