acceptodds
Under review as a conference paper at ICLR 2027

BoundedGS: Memory-Efficient High-Resolution Training for 3D Gaussian Splatting

Abstract

High-resolution 3D Gaussian Splatting training increases the number of pixels and Gaussian–tile intersections, expanding rasterization caches that overlap with projection and gradient state at the peak of a training view. A parameter update needs the gradients from the complete view, but the forward and backward caches of a local rasterization region can be released as soon as its contribution has been collected. Based on this observation, we introduce BoundedGS, which separates rasterization and projection backward passes and bounds the active local rasterization working set according to its measured load. Candidate-indexed projection and compact gradient storage address the view-level peaks exposed by this change, while CPU-assisted structure rebuilding and bounded image supply manage resources outside rendering. In a fixed-model resolution sweep with about 6M Gaussians, the measured rendering working set stays within 1.073–1.077 GiB from 1K to 16K; at the shared 11K setting, a warm pass takes 19.79 rather than 21.76 seconds. Across five native 5K outdoor scenes, full training reduces the mean rendering working set from 9.97 to 1.76 GiB (82.3%), retains similar reconstruction quality, and shortens training time from 36.7 to 33.9 minutes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, High-Resolution Training, Memory-Efficient Training, GPU Memory Optimization, Differentiable Rendering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_THw3OpenReview ↗
Rating
3(+1)
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper investigates the GPU memory bottleneck during high-resolution 3D Gaussian Splatting (3DGS) training (3K–5K+). It observes that while full-view aggregation is mathematically necessary for parameter updates, intermediate rasterization caches do not need to remain concurrently resident across the entire image. The authors propose BoundedGS, an execution pipeline that decouples rasterization and projection backward passes. By partitioning the image into bounded sub-regions (leaves), BoundedGS releases regional forward and backward rasterization states immediately after accumulating gradients on projected Gaussian attributes. To prevent secondary peaks, it introduces candidate-indexed projection (frustum-level screening), compact parameter gradient accumulation, and CPU-assisted structure rebuilding with pinned prefetch queues. Evaluated on Mip-NeRF 360 and Tanks and Temples, the paper reports an 82.3% reduction in peak rendering working set (from 9.97 GiB to 1.76 GiB at 5K) while maintaining reconstruction fidelity and comparable training time.

Critical strengths

S1: Principled mathematical decoupling and engineering execution. Decomposing the full-frame rasterization backward pass from the projection backward pass is mathematically grounded in the spatial locality of tile-based rasterization. By accumulating intermediate gradients ∂L/∂qi\partial L / \partial q_i on projected attributes across sequential leaf regions via the chain rule, the method strictly preserves complete-view rasterization semantics without lossy approximations. The underlying GPU systems engineering—including dynamic leaf splitting/merging based on measured load, candidate ID mapping, and direct gradient accumulation—is coherent, well-formulated, and addresses an authentic hardware bottleneck in high-resolution volumetric rendering.

S2: Thorough memory lifetime profiling and component diagnostics. The paper provides detailed diagnostic traces (Figure 3 and Section 4.3) analyzing the overlap and lifetime of intermediate buffers (rasterization caches, projection states, and parameter gradients). Table 2 systematically validates how each component progressively shaves off memory spikes, demonstrating conclusively that the active rasterization working set can be capped to an invariant bound regardless of observation resolution.

Critical weaknesses

W1: Complete absence of direct comparisons against specialized high-resolution baselines. The introduction explicitly highlights InfiniGS (Liu et al., 2026) as prior art tackling memory pressures in high-resolution 3DGS via image blocking and asynchronous optimization. However, in Table 1, the method is evaluated solely against vanilla 3DGS (Original). There is zero empirical comparison—in terms of peak VRAM, wall-clock time, or rendering quality—against InfiniGS or other memory-efficient 3DGS variants. Without benchmarking against direct competitors targeting the same design space, it is impossible to substantiate whether BoundedGS offers a superior Pareto frontier or distinct system advantages over existing blocking paradigms.

W2: Restricted memory accounting and lack of end-to-end VRAM closure. The paper heavily advertises an 82.3% memory reduction (from 9.97 GiB to 1.76 GiB at 5K), but Section 4.1 explicitly defines this metric as the "temporary rendering working set," excluding model parameters, initialized Adam optimizer states, and persistent density statistics. For scenes with 5M–6M Gaussians, model weights and Adam momentums alone occupy multiple gigabytes of VRAM. Furthermore, by offloading structure updates and image decoding to host RAM via pinned memory transfers, part of the memory pressure is shifted to the host system. The paper fails to report the actual physical peak GPU memory allocation (torch.cuda.max_memory_allocated), total host RAM consumption, or transfer bottlenecks across different PCIe bandwidth configurations.

W3: Confounded speedup attribution and missing Gaussian counts. Table 1 indicates that BoundedGS slightly reduces training time at 5K (from 36.7 to 33.9 minutes) despite the overhead of multi-kernel leaf dispatching and synchronization. This speedup appears to stem largely from candidate screening pruning invisible Gaussians prior to expensive spherical harmonics evaluation, rather than the bounded rasterization itself. The experimental evaluation does not decouple the runtime overhead of leaf tiling from the computational savings of candidate culling. Additionally, Table 1 omits the final Gaussian count per scene, leaving open the possibility that slight variations in densification dynamics between the runs contributed to the timing discrepancy.

W4: Substantial presentation gap with no methodological pipeline diagram. The method relies entirely on dense, text-heavy descriptions across Sections 3.1–3.4 and Appendix B to articulate complex multi-stage mechanisms (leaf trees, candidate ID mapping, compact gradients, pinned image queues, asynchronous host streams). The complete absence of an architectural overview or pipeline data-flow figure significantly hurts clarity and makes the system execution unnecessarily difficult to comprehend and reproduce.

Questions for authors

Q1: How does BoundedGS compare quantitatively against InfiniGS (or other tile/block-based memory-efficient 3DGS methods) on native 5K outdoor scenes in terms of final PSNR/SSIM, total training time, and peak memory?

Q2: What is the true end-to-end peak GPU memory allocation (torch.cuda.max_memory_allocated) during full 30K training for both Original and BoundedGS in Table 1? What is the peak Host RAM usage resulting from the CPU-assisted layout reconstruction and prefetch queues?

Q3: Could you provide a runtime ablation isolating the wall-clock time spent on: (a) candidate frustum screening, (b) leaf tree scheduling and rasterization dispatch, and (c) host-to-device transfers and structure updates? What are the exact final Gaussian counts for both methods across the scenes in Table 1?

Q4: How sensitive is training throughput to PCIe bandwidth constraints (e.g., PCIe Gen4 x16 vs. Gen4 x8 or Gen3 x16), given the continuous pinned image streaming and full-model parameter re-uploads during densification steps?

Reviewer_LxZxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

3DGS training incurs a high peak memory, especially at high resolutions, making its training on devices with limited memory prohibitive. The authors study the memory footprint during training and observe that this is due to keeping the rendering state of the whole image in memory for backpropagation. They propose splitting the image’s pixel tiles into different regions (raster leaves), and rendering and backpropagating each region separately, while accumulating their gradients into a single update per image. Processing each region separately leads to a low peak memory footprint.

Critical strengths
  1. Authors achieve a significant reduction in memory.
  2. Ablation results are very thorough.
Critical weaknesses
  1. Weak baseline, limited prior work comparison;
  2. Seems like memory reduction can be achieved with a more trivial implementation instead.
Questions for authors

I like the main idea of the paper. However, I do have some concerns with the baseline used. Please see my comments below:

  1. The peak memory of the model is high in the current implementation because of gradient checkpoints stored during rasterization and compositing (the yellow part of Figure 1). This is implementation dependent. For example, the original implementation of 3DGS (diff-gaussian-rasterization) does not need to checkpoints at all and would incur lower peak memory. Thus, the authors need to compare with this version as a baseline. Thus, a fair point of comparison would be this implementation combined with warp-level gradient reduction (LiteGS [1], gsplat [2], DISTWAR [3]) to reduce atomic contention in the backward pass. The authors already cite LiteGS’s warp-level gradient reduction and Taming3DGS’s checkpoints (lines 209–210). Please compare peak memory and training time against such a warp-reduction implementation.

  2. Additionally, peak memory could be reduced simply by using fewer checkpoints, i.e., larger buckets, in the forward and backward kernels of the bucketed rasterizer (e.g., Taming3DGS / the 3dgs_accel branch of diff-gaussian-rasterization). The current implementation uses a bucket size of 32. Can the authors report peak memory and training time with larger bucket sizes?

  3. There are simpler approaches to reducing peak memory that the authors have not evaluated against. One approach is training on a subset of tiles of pixels at each step instead of the entire high-resolution image, as in Turbo-GS [4], which the authors cite. Please compare against such an approach in quality and training time at equal peak memory.

  4. Additional prior work: ContraGS (ICCV ‘25) [5] tackles a similar problem of reducing peak memory and training time using an orthogonal approach. This should be discussed in related work.

Minor:

  1. Original - cap in line 155.

  2. Clearly define the parameters in the legend of Figure 1. For example, what is "Handoff"?

References:

[1] Liao, Kaimin, et al. "LiteGS: A high-performance framework to train 3DGS in subminutes via system and algorithm codesign." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[2] Ye, Vickie, et al. "gsplat: An open-source library for Gaussian splatting." Journal of Machine Learning Research 26.34 (2025): 1-17.

[3] Durvasula, Sankeerth, et al. "Distwar: Fast differentiable rendering on raster-based rendering pipelines." arXiv preprint arXiv:2401.05345 (2023).

[4] Dhiman, Ankit, et al. "Turbo-gs: Accelerating 3d gaussian fitting for high-quality radiance fields." arXiv preprint arXiv:2412.13547 (2024).

[5] Durvasula, Sankeerth, et al. "Contrags: Codebook-condensed and trainable gaussian splatting for fast, memory-efficient reconstruction." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Reviewer_ZsBJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This approach decouples the rasterization process from the backpropagation of projections, while introducing projection partition and a compact gradient storage scheme. It preserves the global Gaussian model, rather than achieving its objectives by reducing the number of Gaussians.

Critical strengths

1.Adjusting the rendering order reduces peak VRAM usage, making training more feasible on GPUs with limited memory.

2.The method does not simply reduce the number of Gaussians; instead, it preserves the global model.

3.It achieves significant temporary VRAM savings at a relatively low temporal cost, and through other screening mechanism, the complete method ends up being slightly faster than the original.

Critical weaknesses
  1. Novelty: The paper proposed several Incremental mechanisms closely related to existing techniques, including compact gradients (which reduce projection input to decrease state size) and image-block partitioning. The authors should further elaborate on how this system differs from CLM, InfiniGS, and gsplat.

  2. The experiment is insufficient; only a comparison with the original 3dgs is reported.

  3. Auto Leaf is a central component of the proposed method. However, important technical details, particularly the memory estimation model and precise partitioning rules, are deferred to Appendix B.3.

Questions for authors

1.Given that the original 3DGS renderer is already tile-based, to what extent can similar memory savings be achieved through alternative tile scheduling and early cache release without the proposed adaptive leaf tree?

2.Stage separation and early cache release reduce runtime from 12.19s to 11.35s in table2, despite not explicitly reducing the underlying rendering workload. Could the authors explain the source of this speedup?

3.Could the authors further clarify the technical differences from gsplat, CLM, and InfiniGS, and explain which performance improvements are specifically attributable to the newly proposed mechanisms rather than established memory-efficient techniques?

1 comment

Thank you for separating the effects of fixed tile scheduling, adaptive leaves, and the gradient implementation.

Fixed tiles and early release (Table 2 and Appendix C.2). In the five-scene component study, stage separation and early release reduce the temporary working set from 9.980 to 7.638 GiB before adding Auto Leaf. We also ran a direct leaf-layout control on Bicycle, Flowers, and Stump at 16K. The layouts share the same projection, compact gradients, immediate leaf-state release, and one update after all pixels in the view. Fixed 1,024-pixel leaves achieve peak working sets within 0.003 GiB of Auto Leaf. At this memory level, Auto Leaf shortens the synchronized warm pass by 15.4%, 14.5%, and 14.1%, respectively. Fixed 2,048-pixel leaves run slightly faster than Auto Leaf and increase the working set to 1.747–1.862 GiB, compared with 0.982–1.173 GiB for Auto Leaf.

Source of the first timing change (Appendix A.4, Table 5). The original transition from 12.19 to 11.35 s changed both the state lifetime and the projection-gradient implementation. We added a whole-image control that uses the same raw-parameter projection backward kernel as stage separation. In that matched-kernel test, releasing rasterization state before projection backward lowers the working set from 9.62 to 7.64 GiB, and warm-pass time is 11.16 versus 11.17 s. The whole-image control reproduces the timing improvement when the backward kernel changes. We corrected the speed attribution and identify lower peak memory as the independently measured effect of stage separation.

Execution rule (Section 3.2 and Figure 1). Equation (4) now gives the four terms of the leaf estimate in the main text. They cover intersections, pixel and tile state, backward checkpoints, and output buffers. When a region exceeds the estimate budget, the scheduler bisects it along the dimension with more tiles, splitting horizontally on a tie. It stops when the estimate fits or the leaf is a single 16 × 16 tile. A single tile is processed even if its estimate exceeds the budget. Figure 1 shows how the leaf gradients are accumulated before the complete-view update.

Relation to prior systems and new comparisons (Section 2, Table 2, and Appendices E and G). gsplat includes packed projection and sparse gradients. The tested full-view path still materializes the view's intersection and pixel state together. CLM-GS uses view filtering and host offloading to make room for larger models. InfiniGS combines image blocking with asynchronous optimization. BoundedGS processes image regions sequentially at fixed model parameters, releases each region's temporary state, and applies one update per view. Table 2 reports separate increments for adaptive splitting, indexed projection, and compact gradients, along with same-checkpoint comparisons to upstream 3DGS, DISTWAR, LiteGS, gsplat, and Turbo-GS. Table 9 reports the two 30K Turbo-GS training comparisons. In the separate CLM-GS control, offloading reduces GPU allocation peaks to 6.752 and 6.407 GiB on Bicycle and Stump, below the same-PLY BoundedGS references. This adds about 8.7 GiB of warm-pass host RSS and 19.4–19.6% wall time within CLM-GS (Table 14). The two systems use different update schedules.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Manuscript Revision Update

Thank you to all reviewers for your time and feedback. We have revised the BoundedGS manuscript in response to the discussion, with the following updates.

1. Method overview and key details. Figure 1 on page 2 now illustrates workload-based subdivision, release of local buffers, gradient accumulation, and one parameter update per view. We have also brought Auto Leaf's memory estimate and subdivision rules into Section 3.2 so that the main text explains how regions are partitioned and processed.

2. Timing attribution. Appendix A.4 and Table 4 incorporate the additional control reported during the rebuttal, using the same raw-parameter backward implementation in the whole-image and staged paths. Their runtimes are nearly identical, while stage separation further reduces the temporary working set. Section 4.3 now explains this distinction when discussing the first two rows of Table 2.

3. Experimental reporting. We clarified the shared training configuration and the meaning of Original, distinguished temporary rendering memory from total structure-update peaks, and made the leaf-local SSIM evaluation explicit. Appendix D now reports the actual final Gaussian counts for all scenes. Appendix B.7 adds a transfer-volume analysis to explain how effective bandwidth affects transfer costs.

4. Related work and readability. We expanded the discussion of gsplat, CLM, and InfiniGS, and clarified the role of established candidate filtering and compact storage techniques in the component analysis. We also standardized terminology, simplified repetitive passages and captions, and improved figure placement and reference formatting.

We appreciate the reviewers' suggestions, which helped us identify explanations that needed improvement. We hope these revisions make the method and experimental evidence easier to follow, and would be grateful for your consideration of the updated manuscript. We welcome further questions and discussion.

Dear Area Chair and reviewers,

Thank you for your detailed feedback on BoundedGS. The revision adds stronger memory baselines, isolates the effect of leaf scheduling, and accounts for the cost of full training. Figure 1 now shows how BoundedGS frees each region's temporary state after accumulating its gradients, then updates the model after the whole view. Major additions and clarifications are marked in blue.

  • Checkpoint-free renderers and pixel sampling (LxZx). On the same Bicycle checkpoint and 5K view, the unmodified upstream 3DGS renderer uses 6.431 GiB of temporary working memory and BoundedGS uses 1.080 GiB. Upstream 3DGS runs out of memory at 12K, and BoundedGS completes the sweep to 16K. We also report DISTWAR and LiteGS stage timings and the larger-checkpoint-interval control (Figure 5, Tables 2, 10, and 11). In 30K Bicycle training, BoundedGS and Turbo-GS have peaks of 8.607 and 8.784 GiB, within 2.05% of each other. BoundedGS gains 1.185 dB in held-out PSNR and takes 3.53 min longer in training (Table 9).
  • Packed gsplat and training resources (THw3). On five native 5K scenes from shared checkpoints at fixed count, packed gsplat with sparse gradients and SparseAdam averages 5.107 GiB and 17.60 s per warm pass, versus 1.662 GiB and 10.46 s for BoundedGS (Table 2). Across the five native 5K full training runs per method, total training allocation drops from 18.182 to 9.976 GiB. With evaluation and saving, the peaks are 18.182 and 14.286 GiB. Mean process peak RSS is about 13.4 GiB for both methods. Tables 3, 4, and 12–14 report the training peaks, host timing, image upload sensitivity, and CLM-GS offloading. The bandwidth test controls image uploads on a physical PCIe 4.0 ×16 link. The external full-training comparison is with Turbo-GS on two scenes.
  • Fixed tiles and timing attribution (ZsBJ). In three 16K scenes, fixed 1,024-pixel leaves reach nearly the same working set as adaptive leaves, and adaptive leaves shorten the warm pass by 14.1–15.4%. Fixed 2,048-pixel leaves are slightly faster and use more memory (Table 6). With the same projection backward kernel, stage separation lowers the working set from 9.62 to 7.64 GiB with nearly identical times of 11.16 and 11.17 s (Table 5). The revised text attributes the earlier timing change to the gradient implementation and the independently measured memory reduction to stage separation.

These measurements give the conditions under which BoundedGS reduces high-resolution training memory while supervising every pixel and updating once per view. The individual replies give the protocols and remaining limits for each comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.