acceptodds
Under review as a conference paper at ICLR 2027

Every Gaussian Counts: ReCap-GS for Resource-Constrained 3D Gaussian Splatting

Abstract

3D Gaussian Splatting achieves high-quality novel-view synthesis, but its aggressive densification can make training and deployment costly under tight memory and representation-size budgets. We formulate budget-constrained 3D Gaussian Splatting as a capacity-allocation problem and introduce ReCap-GS (Recovering Capacity for Gaussian Splatting), a fixed-budget framework centered on stale-capacity recovery. Through a controlled opacity perturbation and recovery phase, ReCap-GS identifies primitives that fail to regain responsibility and reclaims their slots for subsequent demand-driven refinement. The method combines early multi-view residual-consensus growth, budget-preserving recycling, and transmittance-aware opacity redistribution while enforcing a fixed upper bound on the Gaussian population throughout training. After topology optimization has ended, we additionally adapt the Multi-view Update strategy introduced in Improved-GS. We evaluate ReCap-GS on the seven public Mip-NeRF360 scenes against a broad suite of budget-aware, pruning-based, relocation-based, and adaptive-densification baselines. Under strict budgets of 50k, 100k, and 200k Gaussians, ReCap-GS improves PSNR over Improved-GS by 0.43, 0.36, and 0.24 dB, respectively, while matching or improving SSIM and LPIPS. Compared with Improved-GS, peak VRAM is reduced by approximately 76 %, 73 %, and 65 % at these budgets, respectively. ReCap-GS remains competitive as capacity increases, suggesting that under tight budgets, recovering stale capacity is an effective complement to demand-driven representation growth.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Resource-Constrained Neural Rendering, Novel View Synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XxA3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReCap-GS, a 3D Gaussian Splatting framework that maintains a strict Gaussian budget throughout training. It identifies stale primitives through opacity-rank changes after a perturbation-and-recovery phase and recycles their slots for demand-driven refinement.

Critical strengths
  • The distinction between strict training budgets and final representation size is clear and practically useful.
  • Stale selection is tested against probe-only, random, and post-recovery opacity controls.
  • Broad budget sweeps and a common evaluation pipeline support the reconstruction-quality comparisons.
Critical weaknesses
  • The large VRAM savings are confounded by CPU image staging; a matched data-staging comparison is missing.
  • Stale recovery adds only 0.07 dB in the 300k ablation. Its contribution at the main 50k–200k budgets is not isolated.
  • Evaluation covers only seven scenes from one dataset and two seeds, leaving broader generalization unclear.
  • Sensitivity to the recovery duration, opacity threshold, and reclamation fraction is not evaluated.
  • There is a lack of an overview of the proposed method, making it hard to understand the techniques.
Questions for authors

How much VRAM improvement remains if Improved-GS uses the same CPU image-staging strategy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer XxA3 (1/2): questions

We thank the reviewer for the careful assessment. We have added matched-staging VRAM measurements, three-seed component ablations across the main budgets, and evaluations of recovery sensitivity and additional datasets. Cross-cutting methodological corrections, including those affecting the original ablation tables, are described in our public manuscript-update comment. We address the VRAM question and stale recovery here; the remaining weaknesses are addressed in our second comment.

W1 and Question. Peak VRAM with matched image staging

The reviewer is correct that our original VRAM reduction claim was confounded by image staging. We repeated peak-VRAM measurements on seven Mip-NeRF360 scenes (seed 1) with the same GPU- or CPU-resident image strategy for both methods (Table 4):

BudgetReCap GPUImproved GPUReCap CPUImproved CPU
50k7654882821822010
100k8018918024582457
200k8896982934473212

Mean peak VRAM in MiB over seven scenes.

With CPU-resident images, no systematic VRAM advantage remains: ReCap-GS uses 9%/0%/7% more memory on average at 50k/100k/200k; per-scene differences range from −24% to +31%. With GPU-resident images, ReCap-GS uses 9–13% less peak VRAM on average (less on every scene), consistent with Improved-GS retaining per-image edge maps in GPU memory. This is an implementation difference, not a demonstrated saving from stale recovery.

We have withdrawn the original 65–76% VRAM-reduction claim and replaced it with matched-staging comparisons (Sec. 4.3). A strict Gaussian-population budget constrains the number of Gaussian parameters and their optimizer states, not total VRAM, which also depends on image staging and other implementation details.

W2. Contribution of stale recovery at low budgets

We expanded the component ablation to 50k, 100k, 200k, and 300k (seven Mip-NeRF360 scenes × three seeds; revised Table 5), with a consistent budget-growth schedule. The correction to the original cross-campaign scheduling discrepancy is documented in our public update; it does not affect the main benchmark.

The incremental contribution of stale recovery is measured by Full − (backbone + MU):

BudgetΔPSNR (dB)ΔSSIMΔLPIPSPSNR winsSSIM winsLPIPS wins
50k+0.056+0.0058−0.004412/2117/2115/21
100k+0.064+0.0052−0.003612/2120/2118/21
200k+0.062+0.0049−0.004713/2120/2121/21
300k+0.063+0.0046−0.005215/2121/2121/21

Mean differences over 21 scene–seed pairs, calculated before rounding. All ablation variants use the adopted opacity correction of Rota Bulò et al.

The reviewer's observation is confirmed: stale recovery adds about 0.06 dB PSNR at every budget, including 50k–200k. At 300k, backbone+stale exceeds the backbone by 0.069 dB, matching the originally reported 0.07 dB. Stale recovery's gains are more consistent in SSIM (17–21/21 pairs) and LPIPS (15–21/21 pairs). At 50k, stale recovery and late MU-mean contribute similar PSNR improvements (+0.056 vs. +0.052 dB); above 50k, MU-mean contributes more PSNR. The full method achieves the best mean PSNR, SSIM, and LPIPS among the ablated configurations at all four budgets, but its advantage should not be attributed primarily to stale recovery.

We have sharpened this interpretation in Sec. 4.4: ReCap-GS combines strict-budget capacity allocation and stale-capacity recovery, with the latter providing complementary gains most consistently reflected by SSIM/LPIPS. The full component-ablation results are also presented in our response to BSXZ (W2).

AuthorsOpenReview ↗

Response to Reviewer XxA3 (2/2): weaknesses

We thank the reviewer for highlighting the need for broader evaluation and a clearer method overview. We address W3–W5 below; cross-cutting corrections are documented in our public update.

W3. Generalization across seeds and datasets

We extended the main Mip-NeRF360 benchmark from two to three seeds. At 50k/100k/200k, ReCap-GS improves mean PSNR over Improved-GS by +0.41/+0.36/+0.24 dB, winning 20/21, 19/21, and 15/21 scene–seed pairs. The gains diminish at higher budgets (absolute mean differences below 0.06 dB at 500k–2M; Table 1; Appendix C.1).

We also evaluated three seeds across seven budgets (50k–2M) on Tanks&Temples (truck, train) and Deep Blending (drjohnson, playroom), using the same evaluation pipeline:

ΔPSNR vs. Improved-GS50k100k200k
Tanks&Temples+0.47+0.59+0.46
Deep Blending+0.36+0.14−0.17

All six Tanks&Temples scene–seed pairs favor ReCap-GS at 50k, 100k, and 200k; the gains on Deep Blending are smaller and reverse at 200k. The four-scene mean advantage is +0.41/+0.37/+0.15 dB at 50k/100k/200k, but −0.16 dB at 2M. Thus, the low-budget advantage extends beyond Mip-NeRF360 on these scenes, without establishing universal superiority. Recovery-policy controls were also expanded to all seven Mip-NeRF360 benchmark scenes and these four scenes (Table 6). The flowers/treehill study is ongoing; results will be shared when complete.

W4. Sensitivity to recovery hyperparameters

We performed a one-factor-at-a-time study at 100k on seven Mip-NeRF360 scenes (seed 1; Appendix D.2). Reference: τ=0.05\tau=0.05, U=300U=300 recovery updates, ρ=0.20\rho=0.20, cycles at 3k/6k, topology freeze at 25k.

VariantΔPSNRΔSSIMΔLPIPS
τ=.02−.003+.0001+.0003
τ=.10+.007+.0001+.0004
U=100−.049−.0010+.0012
U=1000+.024−.0012+.0014
ρ=.10−.033−.0019+.0016
ρ=.30+.002+.0003+.0004
One cycle (3k)−.009−.0020+.0016
Four cycles (3k/6k/9k/12k)−.091−.0007+.0017
Late cycles (12k/24k)−.558−.0128+.0126
No recovery−.089−.0055+.0038

Changes relative to reference; lower LPIPS is better.

Moderate variations in τ, U and ρ shift mean PSNR by −.049 to +.024 dB, without a consistent improvement across metrics. Schedule matters more: one cycle nearly matches PSNR but worsens SSIM/LPIPS; four cycles reduce PSNR; moving both cycles to 12k/24k is worse than removing recovery. The second late cycle finishes at 24.3k, leaving only 700 iterations before topology freeze, although we have not isolated the cause. This study uses one seed and budget, so it does not establish universal hyperparameter robustness.

W5. Method overview and presentation

We agree that the original explanation did not sufficiently connect the components. The expanded Sec. 3 overview (with Figure 1) and Appendix A (including Algorithm 1) now document the training sequence. In brief:

  1. Allocate under a strict budget. Starting from a deterministically capped SfM initialization, early residual-consensus insertion (until 1.8k) adds Gaussians where several views agree on high residual error; absolute-gradient clone/split refinement then continues until topology freeze. A stability-adaptive ceiling paces growth toward BB; once the population is near BB, budget-preserving slot exchanges reuse low-support slots instead of exceeding BB.
  2. Identify stale capacity. At 3k and 6k, all opacities are clamped to at most τ\tau, without changing geometry or appearance, and optimization continues for UU views. For primitives that persist through this window, the stale score is the drop in normalized opacity rank from before to after recovery.
  3. Reclaim and reallocate. The fraction ρ\rho of eligible primitives with the largest rank drops is removed; the freed slots return to the capacity pool and are consumed by subsequent gradient-driven refinement within the same budget. Structural updates use the opacity correction of Rota Bulò et al.; the absolute-gradient criterion follows AbsGS, and late MU-mean is adopted from Improved-GS.

The contribution is the integration of residual-consensus allocation, budget-preserving slot exchanges, and stale-capacity recovery under a training-time population constraint—not a new gradient criterion, opacity correction, or faster renderer. The expanded ablations also clarify that stale recovery's most consistent incremental benefits appear in SSIM/LPIPS, rather than accounting for most of the PSNR gain (Table 5). We hope this overview makes the roles and novelty of the components easier to assess.

Reviewer_BSXZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Fitting a radiance field using 3DGS optimization starts from an initial sparse COLMAP reconstruction, densifies the scene and can grow to millions of Gaussian primitives due to adaptive densification control. The authors investigate resource-constrained 3DGS optimization under a tight primitive budget and argue that when capacity is maxed out, the key question is not only where to grow but also which occupied capacity has become stale. The authors propose ReCap-GS ng timand the main contributions are as follows:

  • A stale capacity recovery mechanism that clamps all opacities to a threshold τ\tau at a recovery cycle and reclaims those primitives whose normalized opacity rank drops most from pre-perturbation to post-perturbation cycle (Sec. 3.3).
  • A transmittance-aware opacity redistribution for the replaced primitives (Sec. 3.4).

Their key result is that, under strict budgets of 50k, 100k, and 200k Gaussians, ReCAP-GS improves PSNR over ImprovedGS by 0.43, 0.36, and 0.24 dB, while reducing peak VRAM usage.

Critical strengths
  • [S1] Useful for practical applications: It is impractical to store millions of Gaussian primitives, as it increases the memory requirement. Hence, fitting a radiance field with strict budget control is very useful for practical scenarios on resource-constrained devices.
  • [S2] Novel stale capacity recovery mechanism: The authors start with the motivation that under tight budget constraints, it is also important to reclaim the stale primitives from the occupied regions. Building on this motivation, the authors propose a stale capacity recovery mechanism that tracks the importance of primitives over a recovery cycle under controlled perturbation. This is different from a "current low opacity" pruning mechanism, as it tracks the state of primitives over a cycle and makes an informed decision. Further, the authors validate this mechanism in Tab. 3 with different controls: probe-only, random reclamation and post-recovery opacity selection. However, the authors report only on 3 scenes. Check W1.
Critical weaknesses
  • [W1] Limited scenes for recovery-policy ablation: The authors report on three scenes: bicycle, stump and kitchen from the Mip-NeRF-360 scene. For a comprehensive evaluation, the authors should have validated the paper's key hypothesis across all 9 scenes of Mip-NeRF 360 and additionally shown whether this holds for other datasets such as Tanks&Temples and Deep Blending.
  • [W2] Limited ablation study on different budgets: Tab. 2 reports an ablation study at a 300K budget. Leaving "stale + opacity redistribution" costs only 0.036 dB and 0.0045 SSIM. Whereas leaving "late MU-mean" costs 0.192 dB and 0.0025 SSIM. This contradicts the central claim of the paper. To further validate their claims, the authors should report this ablation at different budgets (50K, 100K, 200k).
  • [W3] Missing comparison on standard Novel-View Synthesis dataset: The authors do not report results on standard NVS datasets: Tanks & Temples [R1] and Deep Blending [R2].
  • [W4] Missing analysis on key hyperparameters: The current manuscript lacks sensitivity analysis for ReCap-GS' hyperparameters: τ=0.05\tau = 0.05, U=300U = 300, ρ=0.2\rho = 0.2 and the choice of exactly two cycles at 3k and 6k.
  • [W5] Missing comparison with relevant baseline methods: The authors do not compare with GaussianSpa [R3] and different opacity based scores methods like (LightGaussian (Fan et. al), PUP-3DGS [R4]).
  • [W6] Equation 6 in Sec 3.4 is exactly same as the mechanism proposed in "Revising Densification in Gaussian Splatting" [R5]. The authors do not cite this paper or discuss comparison against this.

References:

  • [R1] Knapitsch, A., Park, J., Zhou, Q. Y., & Koltun, V. (2017). Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4), 1-13.
  • [R2] Hedman, P., Philip, J., Price, T., Frahm, J. M., Drettakis, G., & Brostow, G. (2018). Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics (ToG), 37(6), 1-15.
  • [R3] Zhang, Y., Jia, W., Niu, W., & Yin, M. (2025, June). Gaussianspa: An" optimizing-sparsifying" simplification framework for compact and high-quality 3d gaussian splatting. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26673-26682). IEEE.
  • [R4] Hanson, A., Tu, A., Singla, V., Jayawardhana, M., Zwicker, M., & Goldstein, T. (2025, June). Pup 3d-gs: Principled uncertainty pruning for 3d gaussian splatting. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5949-5958). IEEE.
  • [R5] Rota Bulò, S., Porzi, L., & Kontschieder, P. (2024, September). Revising densification in gaussian splatting. In European Conference on Computer Vision (pp. 347-362). Cham: Springer Nature Switzerland.
Questions for authors
  • [Q1] What is ReCap-GS peak VRAM with a normal GPU dataloader, and Improved-GS peak VRAM with a CPU dataloader? The authors mention in Appendix D.1 that, for ReCap-GS training images remain on the CPU. Hence, it is not a fair comparison with Improved-GS.
  • [Q2] Are ablations in Tab. 2 reported for all the scenes of MipNeRF-360 or selected scenes?
  • [Q3] What was the impact on the overall optimization time and rendering FPS from ReCAP-GS?
  • [Q4] What % of Gaussian primitives are reclaimed at each cycle? Why are the beginning iterations selected for stale recovery?
6 comments
AuthorsOpenReview ↗

Response to Reviewer BSXZ (1/2): questions

We thank the reviewer for the constructive questions. We have added matched-staging measurements, three-seed runtime results, expanded ablations, and recovery-schedule analyses. Methodological corrections, including those affecting the original Tables 2–3, are documented in our public manuscript-update comment. We address W1–W6 separately.

Q1. Peak VRAM with matched image staging

We repeated the comparison with GPU- and CPU-resident training images for both methods (7 Mip-NeRF360 scenes, seed 1):

BudgetReCap GPUImproved GPUReCap CPUImproved CPU
50k7654882821822010
100k8018918024582457
200k8896982934473212

Peak VRAM (MiB), averaged over seven scenes (Table 4).

With GPU-resident images, ReCap-GS uses 9–13% less peak VRAM, consistent with Improved-GS storing additional per-image edge maps; this is not attributable to stale recovery. With CPU-resident images, ReCap-GS uses comparable or slightly higher mean VRAM (up to 9%; per-scene differences −24% to +31%). We have withdrawn the original 65–76% reduction claim: it was confounded by different image-staging strategies. A strict population cap bounds Gaussian parameters and optimizer states, not total training VRAM (Sec. 4.3).

Q2. Scene coverage of the original Table 2

The original component ablation covered all seven benchmark Mip-NeRF360 scenes (bicycle, bonsai, counter, garden, kitchen, room, stump), not a selected subset; flowers and treehill are discussed in W1 of our second comment. The original recovery-policy controls (Table 3) covered three scenes. We have rerun the component ablations at 50k/100k/200k/300k over seven scenes and three seeds (revised Table 5), and extended recovery-policy controls to seven Mip-NeRF360 scenes plus four Tanks&Temples/Deep Blending scenes, also with three seeds (Table 6). Both now use the same budget-growth schedule; the correction is explained in our public update.

Q3. Optimization time and rendering FPS

We measured end-to-end training wall time and rendering FPS across seven scenes and three seeds, using NVIDIA RTX A6000 GPUs. Time includes loading, checkpointing, and native evaluation (Appendix E).

BudgetReCap (s)Improved (s)ReCap FPSImproved FPSImproved mean final NN
50k35638928726450k
100k394408258263100k
200k456455230248200k
300k507498216230300k
500k601572196201500k
1M817691156169969k
2M12047971201471453k

Means over seven scenes and three seeds; ReCap-GS fills the target budget.

ReCap-GS trains faster at 50k–100k, comparably at 200k–300k, and slower at larger budgets. At 2M, Improved-GS finishes at ~1.45M rather than 2M Gaussians, partly limiting comparability. Rendering FPS is similar at 50k–100k and generally lower for ReCap-GS at higher budgets, including at equal Gaussian counts (230 vs. 248 FPS at 200k). We have not isolated the cause of this difference and do not claim a faster renderer. Timing includes implementation-dependent image transfers, so it is not an isolated algorithmic speed comparison. Comparing full ReCap-GS with backbone+MU suggests ~3–5% additional wall time associated with recovery, but these are separate jobs rather than a controlled timing isolation.

Q4. Reclaimed fraction and early recovery

At 100k, across seven Mip-NeRF360 scenes and three seeds, the actual fractions reclaimed at the two cycles are 19.42% and 19.53% of the active population (Sec. 3.3), slightly below the target ρ=20%\rho=20\% because primitives created or replaced during recovery are ineligible.

We place recovery at 3k/6k because (i) population growth reaches its scheduled ceiling by 3k, making capacity reuse necessary as slots fill, and (ii) structural refinement continues until 25k, leaving time to reallocate and optimize freed slots. The 100k sensitivity study (7 scenes, seed 1; Appendix D.2) supports this timing: moving recovery to 12k/24k reduces mean PSNR by 0.558 dB, compared with −0.089 dB when recovery is removed. The second late cycle ends at 24.3k, leaving only 700 iterations before topology freeze; insufficient refinement time is plausible, but the two late cycles were not isolated individually. One early cycle nearly matches two in PSNR (−0.009 dB) but degrades SSIM/LPIPS (−0.0020/+0.0016), while four cycles at 3k–12k reduce PSNR by 0.091 dB.

AuthorsOpenReview ↗

Response to Reviewer BSXZ (2/2): weaknesses

We address W1–W6; see also our public update.

W1. Recovery policies and scene coverage

We expanded the 300k policy study from three scenes to seven Mip-NeRF360 scenes (21 scene–seed pairs) and four Tanks&Temples/Deep Blending scenes (12 pairs), with three seeds (Table 6) and nearly matched victim counts across reclamation policies.

Dataset / policyPSNRSSIMLPIPS
Mip / probe-only29.003.8535.2198
Mip / random29.011.8534.2210
Mip / post-opacity29.053.8563.2177
Mip / stale29.059.8573.2154
T&T+DB / probe-only26.964.8759.2231
T&T+DB / random27.007.8758.2236
T&T+DB / post-opacity27.126.8770.2231
T&T+DB / stale27.058.8770.2219

Using unrounded results, stale improves Mip PSNR over probe-only/random by +.056/+.047 dB and SSIM/LPIPS in 21/21 pairs against both. On T&T+DB, PSNR gains are +.095/+.052 dB, but only 7/12 pairwise wins for each. Versus post-opacity, stale changes PSNR by +.006 dB (Mip) and −.067 dB (T&T+DB), yet improves LPIPS on 20/21 and 12/12 pairs. Reclamation helps on average; rank-drop's most consistent additional benefit is perceptual. The flowers/treehill study is ongoing; results will follow.

W2. Multi-budget component ablation

We reran ablations at 50k–300k on seven scenes × three seeds with a consistent budget-growth schedule (Table 5; see public update).

BudgetBackbone+MU+StaleFullImproved
50k26.46026.55926.56226.61426.201
100k27.69527.78327.72827.84727.487
200k28.60328.74428.66428.80728.565
300k28.99029.13629.05929.19929.078

Mean PSNR (dB); all variants use the adopted opacity correction. Adding stale to backbone+MU yields approximately +.06 dB PSNR across budgets, but larger SSIM gains (+.0046–.0058; 17–21/21 wins) and LPIPS gains (15–21/21 wins). At 300k, this increment is +.063 dB, versus +.036 dB in the original Table 2. The backbone+MU mean changed by −.028 dB across campaigns; the change cannot be attributed solely to the corrected schedule. Late MU-mean adds +.05 dB at 50k and +.12–.14 dB at 100k–300k to backbone+stale. We have reframed the contribution accordingly: the abstract presents a strict-budget framework combining allocation with stale-capacity recovery, while Sec. 4.4 identifies stale recovery's gains as most consistent in SSIM/LPIPS rather than PSNR. We do not attribute most of the PSNR gain to recovery.

W3. Additional datasets

We evaluated three seeds over seven budgets (50k–2M) on the standard 3DGS evaluation scenes truck, train, drjohnson, and playroom (Table 2). Against Improved-GS, ReCap gains +.41/+.37/+.15 dB PSNR at 50k/100k/200k (12/12, 11/12, 9/12 wins). By dataset, gains are +.47/+.59/+.46 dB on Tanks&Temples and +.36/+.14/−.17 dB on Deep Blending. At 2M, the four-scene mean is −.16 dB. Benefits are dataset- and budget-dependent.

W4. Sensitivity

At 100k (seven scenes, seed 1; Appendix D.2), varying τ=.02/.10, U=100/1000, or ρ=.10/.30 changes PSNR by −.049 to +.024 dB. One cycle at 3k changes PSNR/SSIM/LPIPS by −.009/−.0020/+.0016; four cycles give −.091 dB PSNR. Late cycles at 12k/24k give −.558 dB PSNR, −.0128 SSIM, +.0126 LPIPS, versus −.089 dB PSNR without recovery.

W5. Requested compression baselines

We evaluated GaussianSpa, LightGaussian, and PUP-3DGS on seven Mip scenes × three seeds, at matched final counts (Table 3):

Final budgetReCapGaussianSpaPUPLightGaussian
50k26.6126.4525.3924.09
100k27.8527.5026.9125.52
200k28.8128.5128.0026.93

Mean PSNR (dB). GaussianSpa is strongest among these baselines: it has slightly better LPIPS at 100k (.2827 vs. .2837) and better SSIM/LPIPS at 200k (.8512/.2280 vs. .8435/.2376). These methods reach compact final counts from larger training populations: GaussianSpa sparsifies during optimization from a peak near 4M Gaussians, whereas LightGaussian and PUP-3DGS prune fully trained 3DGS models (83–99% pruning). Unlike ReCap, none enforces the population budget throughout training. Within the strict-budget setting, the closest opacity-based victim-selection control is post-recovery opacity in W1.

W6. Opacity correction

We thank the reviewer for flagging this omission. The correction α′=1−1−α\alpha'=1-\sqrt{1-\alpha} comes from Rota Bulò et al. (ECCV 2024), not our contribution. We now attribute it in the introduction, Sec. 3.2, Related Work, and Appendix A.3; all component ablations use it. Disabling it throughout the pipeline lowers the 300k stale configuration by .06 dB on Mip and .20 dB on T&T+DB. Our novelty is strict-budget allocation and stale-capacity recovery.

Thanks for the authors' response.

My following queries are resolved:

  • [Q1] Thanks for being candid and withdrawing the reduction claim after the experimentation.
  • [Q2] Tables 12 and 13 show that ReCap-GS performs better in budget-constrained settings for the additional scenes from the Deep Blending and Tanks & Temples datasets.
  • [W3] The authors have updated the manuscript with standard datasets.
  • [W4*] My concern related to sensitivity to τ\tau, ρ\rho and UU is resolved.
  • [W5] Thanks for comparing against the requested baselines. ReCap-GS shows better performance.
  • [W6] I appreciate the authors citing the relevant paper for the opacity correction.

However, I still have the following concerns:

  • [W1] Stale selection improves over the post-recovery policy by only a marginal $ +0.006$ dB on Mip-NeRF 360 scenes and is worse on the Tanks & Temples dataset. These results contradict the claims made in Section 4.4. Even a random policy is only −0.048-0.048 dB worse than the proposed policy for Mip-NeRF 360.
  • [W2] This shows that the major contribution is from the backbone + MU. Is stale-capacity recovery really contributing significantly to the gains?
  • [W4*] It is still not clear how the schedule is decided for the recovery. Check [Q4] as well. -[Q3] ReCap-GS takes less time at reduced capacity < 200K. However, at higher capacity, it trains more slowly. Further, ReCap-GS renders more slowly at higher capacity budgets. It seems absurd that with an equal number of primitives, the rendering FPS can differ. Do the authors have an analysis for this behavior?
  • [Q4] The authors presented the fractions of primitives reclaimed at each cycle. However, Tab. 14 suggests that "one cycle" performs similarly to two cycles, and "four cycles" further degrades the results. Is this opacity recovery too sensitive to the number of cycles? Why does performance degrade with four cycles?

Further, MipNeRF-360 has two extra challenging scenes: flowers and treehill. It would be beneficial for the manuscript to show results on them as well and also analyze how recovery behaves for such scenes.

Reviewer_QaPhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates budget-constrained 3D Gaussian Splatting as a capacity-allocation problem: the Gaussian population must satisfy N(t) ≤ B at every training iteration, including SfM initialization. The proposed framework, ReCap-GS, (i) grows toward the budget via early multi-view residual-consensus allocation followed by absolute-gradient refinement (adopted from Improved-GS), with budget-preserving slot exchanges when demand exceeds free capacity; (ii) introduces stale-capacity recovery as the central mechanism: a controlled opacity probe (all opacities clamped to τ = 0.05) followed by U = 300 recovery updates, after which primitives whose relative opacity rank drops from pre- to post-recovery (the "stale score") are reclaimed and their slots returned to a capacity pool for demand-driven reuse; (iii) applies transmittance-aware opacity redistribution (α′ = 1 − √(1−α)) when recycling slots; and (iv) after topology freeze, adapts Improved-GS's Multi-view Update with gradient averaging. Validation covers the seven Mip-NeRF360 scenes at budgets from 50k to 2M, against a broad suite of budget-aware baselines under an explicitly defined strict protocol (certified trajectory peak, deterministic initialization capping) and a common matched-PNG evaluation pipeline, with two seeds. Under strict 50k/100k/200k budgets, ReCap-GS reports +0.43/0.36/0.24 dB PSNR over Improved-GS with 76/73/65% lower peak VRAM; ablations separate stale recovery from late MU-mean and from random / post-recovery-opacity reclamation policies.

Critical strengths
  1. The stale-capacity recovery mechanism is simple, well-motivated, and cleanly validated. The rank-based stale score captures a history-dependent notion of obsolescence that instantaneous opacity cannot express, and the recovery-policy diagnostic (Table 3, with nearly matched realized victim counts) directly substantiates the central claim that the recovery trajectory carries information beyond the perturbation itself. The component ablation (Table 2) further separates stale recovery, opacity redistribution, and late MU-mean.

  2. Unusually careful and transparent evaluation methodology: strict vs. native protocols with certified trajectory peaks, per-baseline capping effects (Table 7), a common matched-PNG pipeline, precise resource definitions, and unfavorable results stated rather than hidden (3DGS-MCMC wins SSIM; gains vanish at large budgets; wall time is not uniformly better). Complete schedule, hyperparameters, and pseudocode are provided.

Critical weaknesses
  1. The headline VRAM reduction is substantially confounded by an implementation choice: ReCap-GS stages training images in CPU memory on a dedicated CUDA stream, while resource measurements "reflect the complete implementations, including their data-staging strategies." The 76/73/65% reduction over Improved-GS cannot be attributed to the method itself without identical staging on baselines; wall time is also slower from 200k onward (7.72 vs. 7.61 min at 200k).

  2. Gains are modest and confined to a narrow regime, with fairness questions at 50k: improvements over Improved-GS shrink to <0.06 dB beyond 300k; 3DGS-MCMC achieves better SSIM at 50k–200k; under the native 50k protocol SalientGS (27.215) and Prune Wisely (26.942) outperform ReCap-GS (26.625), and Table 7 shows capping costs SalientGS 1.405 dB at 50k while ReCap-GS is strict by design — so part of the strict 50k win may stem from the capping adaptation. Results are means over seven scenes from only two seeds, with no per-scene main-text tables or evaluation beyond Mip-NeRF360. Table 2 further shows the backbone alone trails Improved-GS at 300k, and the late MU-mean stage adapted from Improved-GS contributes more (+0.170 dB) than stale recovery (+0.069 dB), bounding the measured impact of the core novelty.

Questions for authors
  1. Can you quantify how much of the VRAM reduction comes from the CPU-staging strategy alone, e.g., by running Improved-GS with the same staging pipeline?
  2. How sensitive are results to the recovery schedule (cycle placement, τ, U, ρ)? Would additional or later recovery cycles near the topology freeze help or hurt?
  3. Given capping costs SalientGS 1.4 dB at 50k, can you compare against native-protocol SalientGS/Prune Wisely at matched final counts, or justify why the strict protocol is the fairer basis for the 50k claim?
2 comments
AuthorsOpenReview ↗

Response to Reviewer QaPh (1/2): questions

We thank the reviewer for the careful assessment. The concerns raised prompted additional experiments, now included in the revised manuscript. We also provide a general update documenting methodological corrections and revised interpretations, including those affecting the original Tables 2 and 3.

Q1. Peak VRAM and CPU image staging

We agree that the original VRAM reductions cannot be attributed to our method. We repeated the comparison with matched GPU- or CPU-resident images (7 Mip-NeRF360 scenes, seed 1).

BudgetReCap GPUImproved GPUReCap CPUImproved CPU
50k7,6548,8282,1822,010
100k8,0189,1802,4582,457
200k8,8969,8293,4473,212

Peak VRAM in MiB, averaged over seven scenes (Table 4).

With CPU-resident images, there is no systematic VRAM advantage (per-scene differences: −24% to +31%); mean VRAM is comparable or up to 9% higher for ReCap-GS. With GPU-resident images, ReCap-GS uses 9–13% less peak VRAM on average, consistently across scenes. This is consistent with Improved-GS storing precomputed per-image edge maps, not an intrinsic saving from stale recovery.

We have withdrawn the original 65–76% VRAM reduction claim and replaced it with matched-staging results (Sec. 4.3). A strict Gaussian budget bounds primitive and optimizer-state memory, not total VRAM. Image staging does not materially change reconstruction quality (mean PSNR differences ≤0.03 dB).

Q2. Recovery schedule and hyperparameter sensitivity

We conducted a one-factor-at-a-time study at 100k (7 scenes, seed 1; Appendix D.2), relative to τ=0.05\tau=0.05, U=300U=300, ρ=0.20\rho=0.20, two cycles at 3k/6k, and topology freeze at 25k.

VariantΔPSNR (dB)ΔSSIMΔLPIPS
τ=0.02 / 0.10−0.003 / +0.007+0.0001 / +0.0001+0.0003 / +0.0004
U=100 / 1000−0.049 / +0.024−0.0010 / −0.0012+0.0012 / +0.0014
ρ=0.10 / 0.30−0.033 / +0.002−0.0019 / +0.0003+0.0016 / +0.0004
1 cycle (3k)−0.009−0.0020+0.0016
4 cycles (3k–12k)−0.091−0.0007+0.0017
Late cycles (12k/24k)−0.558−0.0128+0.0126
No recovery−0.089−0.0055+0.0038

Differences from default; positive ΔLPIPS is worse.

Varying τ, U and ρ changes PSNR by −0.049 to +0.024 dB. These single-seed differences should be viewed against seed-to-seed PSNR standard deviations of 0.003–0.034 dB in the component ablations at 100k; we do not claim optimal hyperparameters. In our tests, both additional and later cycles reduce PSNR: four cycles give −0.091 dB, nearly matching no recovery (−0.089 dB), although SSIM/LPIPS remain better. Moving the cycles to 12k/24k gives −0.558 dB, with lower PSNR on all seven scenes and worse mean SSIM/LPIPS. The second late cycle ends at 24.3k, just 700 iterations before topology freeze. Insufficient time to refine reclaimed capacity is plausible, but the late cycles were not isolated individually.

Q3. Strict versus native protocols: SalientGS and Prune Wisely

We agree that initialization capping affects baselines, particularly SalientGS. We therefore compared native runs using actual Gaussian counts (7 scenes, 3 seeds).

SalientGS. At nominal 50k, native SalientGS achieves 27.22 dB versus 26.61 dB for ReCap-GS (3 seeds), but retains excess SfM points (mean final ~137k; median ~139k; up to 241k on kitchen). For each scene and seed, we compared it against the largest tested ReCap-GS budget not exceeding SalientGS's actual final count. ReCap-GS reaches 28.00 versus 27.22 dB (+0.78 dB; 21/21 wins); at nominal 100k, +0.84 dB (21/21). Capping SalientGS reduces its 50k PSNR from 27.22 to 25.82 dB. Capping does not systematically hurt baselines: at 50k, it improves Taming-3DGS and Taming+SPARE by 1.77 and 1.71 dB (Appendix C.2).

Prune Wisely. Native Prune Wisely ends exactly at the target count, so this is already a matched-final-count comparison. It beats ReCap-GS at 50k (26.94 vs. 26.61 dB), but reaches a mean training peak of 2.48M Gaussians (median 1.54M), nearly 50× its final count. At 100k it leads by 0.08 dB; at 200k ReCap-GS leads by 0.13 dB. At 50k ReCap-GS has better SSIM/LPIPS on 18/21 and 17/21 scene–seed pairs. With GPU-resident images, Prune Wisely uses 15.0–15.3 GiB peak VRAM (3 seeds) versus 7.5–8.7 GiB for ReCap-GS (seed 1) across 50k–200k (Appendix E): its training memory barely changes with the final budget.

The strict protocol is appropriate for our 50k claim because it matches the studied constraint: N(t)≤BN(t)\le B throughout training, including initialization. Native comparisons answer a different question, allowing much larger intermediate populations. The revised manuscript presents both protocols and peak/final counts (Table 3; Appendix B.2).

Remaining critical weaknesses are addressed in a second comment.

AuthorsOpenReview ↗

Response to Reviewer QaPh (2/2): weaknesses

We also address the wall-time concern in Critical Weakness 1 and the issues raised in Critical Weakness 2. The VRAM confound and strict/native protocol comparisons are addressed in Q1 and Q3 of our first comment.

W1. Wall time

With three seeds, ReCap-GS trains faster than Improved-GS at 50k/100k (356/394 vs. 389/408 s), similarly at 200k/300k (456/507 vs. 455/498 s), and slower at larger budgets (+18% at 1M, +51% at 2M). These end-to-end measurements include different image-staging strategies and do not establish an algorithmic speed advantage (Appendix E).

W2-A. Low-budget gains and 3DGS-MCMC

We agree that ReCap-GS's PSNR gains are concentrated at low budgets. Across 7 Mip-NeRF360 scenes and 3 seeds, it gains +0.41/+0.36/+0.24 dB over Improved-GS at 50k/100k/200k (20/21, 19/21, 15/21 scene–seed wins). The advantage drops to +0.12 dB at 300k and remains within ±0.06 dB at 500k–2M. At 2M, Improved-GS ends with ~1.45M Gaussians on average (median 1.17M). We now frame ReCap-GS as targeting tight training-time population budgets, not uniform gains across budgets.

3DGS-MCMC has higher SSIM on Mip-NeRF360 at 50k–200k (0.7693 vs. 0.7662 at 50k), while ReCap-GS gains +0.54/+0.62/+0.59 dB PSNR, with comparable LPIPS. On Tanks&Temples and Deep Blending, ReCap-GS has higher SSIM on all 12 scene–seed pairs at each of these budgets. These trade-offs depend on metric and dataset (Tables 1–2; Appendix C.4).

W2-B. Seeds, per-scene results, additional datasets

We extended the main budget sweeps, baseline comparisons, component ablations, and recovery-policy controls from 2 to 3 seeds. Sensitivity and matched-staging VRAM studies use seed 1. For the full ReCap-GS configuration, the across-seed standard deviation of seven-scene mean PSNR is 0.003–0.023 dB across budgets; gains over Improved-GS hold in every seed (+0.39 to +0.43 dB at 50k, +0.36 at 100k, +0.23 to +0.25 at 200k).

Per-scene ΔPSNR (ReCap − Improved), mean over 3 seeds (Appendix C.3):

Scene50k100k200k
bicycle+0.27−0.05−0.23
bonsai+1.06+0.60+0.32
counter+0.35+0.35+0.12
garden+0.25+0.27+0.35
kitchen+0.68+1.08+1.02
room+0.02+0.23+0.23
stump+0.27+0.03−0.12

ReCap-GS wins on 7/7, 6/7, and 5/7 scenes at 50k/100k/200k. We also evaluated Tanks&Temples (truck, train) and Deep Blending (drjohnson, playroom) across seven budgets (50k–2M), three seeds and a common evaluation pipeline:

ΔPSNR vs. Improved50k100k200k
Tanks&Temples+0.47+0.59+0.46
Deep Blending+0.36+0.14−0.17

At 50k, 100k, and 200k, Tanks&Temples wins are 6/6 at each budget, while Deep Blending wins are 6/6, 5/6 and 3/6, respectively. Generalization is therefore scene- and budget-dependent (Table 2; Appendix C.4).

W2-C. Stale recovery versus late MU-mean

We expanded the component ablation to 50k–300k (7 scenes, 3 seeds), using a consistent budget-growth schedule. The correction of the original cross-campaign inconsistency is documented in our public manuscript-update comment. Revised results are reported in Table 5:

BudgetBackbone+MU+StaleFullImproved
50k26.46026.55926.56226.61426.201
100k27.69527.78327.72827.84727.487
200k28.60328.74428.66428.80728.565
300k28.99029.13629.05929.19929.078

Mean PSNR; all variants include the adopted opacity correction.

Stale recovery added to backbone+MU improves mean PSNR by approximately 0.06 dB across budgets, winning 12–15/21 scene–seed pairs. Its benefits are more consistent in SSIM (+0.0046–0.0058; 17–21/21 wins) and LPIPS (15–21/21 wins). Late MU-mean, adapted from Improved-GS, contributes +0.05 dB PSNR at 50k and +0.12 to +0.14 dB at 100k–300k when added to backbone+stale, but only +0.0012–0.0013 SSIM.

At 50k, the backbone alone already beats Improved-GS by +0.259 dB; recovery and MU together add +0.154 dB. At 300k, the backbone trails Improved-GS by 0.09 dB, while the full method leads by 0.12 dB. We do not attribute most PSNR gains to stale recovery. The backbone combines residual-consensus allocation and slot exchanges with adopted absolute-gradient refinement; MU-mean and opacity correction are also adopted.

The contribution is therefore the strict-budget allocation framework as a whole, with stale recovery providing complementary gains, most consistently in SSIM and LPIPS. The rank-drop criterion itself improves LPIPS over post-recovery opacity more consistently than PSNR (Table 6).

Comments on the paper

5 comments
AuthorsOpenReview ↗

Summary of manuscript revisions and corrections

We thank all reviewers for their feedback. The revised manuscript extends the evaluation from two to three seeds, adds datasets and baselines, and clarifies implementation details. The 50k PSNR gain over Improved-GS changes from +0.43 to +0.41 dB. We summarize substantive changes here for all reviewers.

1. Ablation schedule correction

During code cleanup, we found that separately executed component-ablation and recovery-policy-control runs used budget_growth_end_iter=30000, while reference configurations reused from the main sweep used 3000. This parameter controls the horizon of the population-growth ceiling, not training duration. All main-benchmark ReCap-GS runs used 3000 and were unaffected. The discrepancy concerned the original component ablations (Table 2) and policy controls (Table 3), including the former probe-only comparison.

We reran both studies with a common setting of 3000 and three seeds (revised Tables 5–6). At 300k, old/new matched-seed runs differ by at most ~0.03 dB in mean PSNR; these cross-campaign differences do not isolate the effect of that setting.

2. Algorithm documentation

Appendix A.1, A.4 and Algorithm 1 now provide implementation details previously omitted or insufficiently documented, including the stability-adaptive growth ceiling MtM_t, slot-exchange conditions near budget saturation, low-opacity pruning (<0.005<0.005) after structural updates, and residual-consensus target initialization.

We also explicitly attribute absolute-gradient refinement to AbsGS and opacity redistribution to Rota Bulò et al. (2024). These are clarifications of the existing implementation, not newly introduced benchmark operations.

3. Revised interpretation of mechanisms

MU-mean vs. MU-sum. At 300k, MU-sum changes PSNR by −0.002 dB on Mip-NeRF360 and −0.039 dB on Tanks&Temples/Deep Blending relative to MU-mean, with slightly better LPIPS. The revised controls do not support a systematic quality advantage for MU-mean. We retain MU-mean to normalize gradients across different view-group sizes; the experiments show comparable reconstruction quality with both aggregation strategies (Appendix D).

Rank-drop vs. post-recovery opacity. Rank-drop changes PSNR by +0.006 dB on Mip-NeRF360 and −0.067 dB on Tanks&Temples/Deep Blending, but improves LPIPS on 20/21 and 12/12 scene–seed pairs, respectively. We narrow the claim accordingly: its advantage over post-recovery opacity is mainly perceptual, while reclamation outperforms probe-only and random controls on average, most consistently in SSIM/LPIPS on Mip-NeRF360 (Table 6).

4. Baseline and resource-reporting changes

The original resource figure included GS2 and official 3DGS+SPARE only as isolated operating points at their natural representation sizes, not in controlled budget sweeps. We removed that figure and its isolated points after withdrawing the implementation-confounded 65–76% VRAM reduction claim. GS2 remains in Related Work; Taming+SPARE remains labeled as our adaptation, as in the original submission.

The original VRAM measurements were not erroneous, but different image-staging strategies confounded their interpretation. We now report matched-staging VRAM (Sec. 4.3), and wall time/FPS (Appendix E). GaussianSpa, LightGaussian, and PUP-3DGS were added under native/final-budget protocols (Table 3).

5. Additional scenes

The revised manuscript reports three-seed results on seven Mip-NeRF360 scenes and four additional scenes from Tanks&Temples and Deep Blending. We are also extending the recovery-policy study to the two remaining Mip-NeRF360 scenes, flowers and treehill. We will share the additional results in a follow-up comment once the evaluation is complete.

6. Reproducibility

Additional per-scene and per-seed statistics, including pairwise win counts and count-matched baseline comparisons, are reported in our individual responses to provide further context beyond the revised manuscript.

An anonymized source-code ZIP with the revised experiments will be posted in the coming days.

AuthorsOpenReview ↗

Follow-up to BSXZ (W1): Recovery policies on flowers and treehill

We have completed the recovery-policy study on the two remaining Mip-NeRF360 scenes, flowers and treehill, using the same protocol as Table 6 (300k Gaussians, three seeds, identical recovery settings; outdoor scenes at 4× downsampling, as in Appendix B.1). All runs respect the 300k Gaussian budget.

Policy (flowers + treehill)PSNR ↑SSIM ↑LPIPS ↓
Backbone22.093±.059.5911±.0008.4139±.0009
Probe only22.163±.050.5926±.0008.4128±.0003
Random reclamation22.194±.023.5922±.0003.4142±.0002
Post-recovery opacity22.253±.017.5974±.0008.4111±.0005
Stale (rank-drop)22.268±.040.5994±.0004.4089±.0005
Stale (no opacity correction)22.260±.043.5990±.0007.4085±.0005

Mean ± sample standard deviation across the three seeds of the two-scene means, as in Table 6.

On these two scenes, stale recovery improves SSIM and LPIPS over probe-only training, random reclamation, and post-recovery opacity on all six scene–seed pairs. Using unrounded results, its mean PSNR is also higher than these three policies by +0.105, +0.073, and +0.015 dB, respectively.

Across all nine Mip-NeRF360 scenes (27 paired scene–seed comparisons), stale recovery improves both SSIM and LPIPS over probe-only training and random reclamation on 27/27 pairs. Compared with post-recovery opacity, it improves SSIM on 25/27 and LPIPS on 26/27 pairs, at similar mean PSNR (+0.008 dB).

These additional scenes reinforce our revised conclusion: rank-drop selection improves over perturbation alone and random reclamation, while its additional benefit over post-recovery opacity selection is most consistent in SSIM and LPIPS rather than PSNR.

On these two scenes, the opacity correction has little effect (+0.007 dB PSNR using unrounded results, with slightly worse LPIPS: .4089 vs. .4085). Its effect varies across scenes: it is near zero on bicycle and garden (−0.002 and −0.004 dB), but +0.061 dB on average over the seven original scenes and +0.20 dB on Tanks&Temples/Deep Blending (Appendix D.1).

The per-run results will be included in the anonymized code archive.

AuthorsOpenReview ↗

Anonymized code and per-run results now available

As announced in our general update, the anonymized implementation and per-run results are now available as supplementary material (recap-gs-code-results.zip).

The archive contains:

  • ReCap-GS implementation (code/), with the patches applied to the 3DGS rasterizer.
  • Training and evaluation scripts for the main benchmark, ablations, recovery-policy controls, sensitivity study, and matched-staging VRAM measurements (experiments/).
  • Baseline adaptations as patches against the official repositories, with the scripts to fetch and apply them (third_party/).
  • Per-run results for all experiments of the revised manuscript, including the flowers/treehill recovery-policy runs reported in our follow-up, and scripts that regenerate every table of the revised paper (reports/; see reports/README.md).
  • Results of the submitted version, kept for traceability and labeled as such.

File-level checksums for the reports are provided in reports/SHA256SUMS. The SHA-256 of the archive is 5494c25463a5b45a34f1a9c4e6e1fd64fd4e357935a8734c9f3de07c7a1bb772.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.