- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a learned cost for selecting Gaussian pairs during recursive simplification. It retains an analytic parent-construction rule and trains a predictor using rendered interventions at intermediate simplification states. The supervision distinguishes immediate rendering changes from changes in fidelity to the original representation. The idea is interesting.
Critical strengths
The rendered-intervention supervision is well motivated.
The evaluation covers 46 scenes from five datasets and three training seeds, with per-scene results and paired confidence intervals.
The matched-refinement experiment is particularly useful. Training separate refiners for analytic and learned selection helps isolate the contribution of the learned merge cost.
Critical weaknesses
Some evaluation scenes were leaked during MLP training. The paper should identify these scenes and report results separately on the test scenes.
I have a concern about the supervision. The MLP is trained on individual pair merges, while each simplification pass merges several disjoint pairs. Their visual effects may interact through occlusion and alpha compositing, so the individual scores may not fully capture the joint effect. I need some explanations.
Questions for authors
What is the main difference compared to other importance-based score pruning methods?
How much does it cost to get all training datasets?
Could a MLP determine the parameters of the merged Gaussians?
6 comments
Response to Reviewer gnwz (2/3): W2, interactions between the merges of a pass
W2. Single-merge labels and the joint effect of a pass
“Their visual effects may interact through occlusion and alpha compositing, so the individual scores may not fully capture the joint effect.”
The concern is correct, and the paper did not quantify it; we now measure the gap and test one of its consequences. A training label is the rendered effect of one merge in an otherwise unchanged state; a pass applies up to 0.5N disjoint merges (N: active Gaussians), and the labels miss their joint effect. Every reported quality result includes it, since quality is measured after the complete simplification.
How large is the gap? On 14 DL3DV development scenes (one seed), as a fraction of the image change of the same merges: sets of 4 or 16 scattered merges differ from the sum of their single effects by less than 1% on average, but their joint effect equals the sum exactly in 86% of the sets of 4 and 49% of the sets of 16 (the merges do not overlap in the image); a whole pass differs from ten times a random tenth of its merges by +17% at 0.1% and by −3% to +9% at 1–25% (three of five intervals exclude zero).
The order changes more: 21–33% of the pairs of probed merges change order when rendered inside their pass instead of alone, with either cost.
A tested consequence. If interactions inside a pass matter, a cost trained on single merges should do better with smaller passes.
After the reviews, we capped the passes of learned selection at 0.25N (size chosen among three on six development scenes); the submitted method keeps 0.5N. Analytic selection keeps 0.5N, its best measured setting (capped at 0.25N, it loses 0.3 to 2.1 dB on the 33 scenes below). Learned − analytic, each with its refiner, in dB; in parentheses, the submitted 0.5N on the same scenes:
| Scenes | 0.1% | 1% | 5% | 10% | 25% |
|---|---|---|---|---|---|
| Original 46-scene panel | −0.33 (−0.17) | +0.29 (+0.12) | +0.59 (+0.29) | +0.62 (+0.30) | +0.39 (+0.18) |
| 15 DL3DV scenes of Table B | −0.25 (−0.25) | +0.40 (+0.16) | +0.66 (+0.35) | +0.76 (+0.35) | +0.64 (+0.22) |
| 25 other DL3DV scenes | −0.27 (−0.15) | +0.27 (+0.13) | +0.57 (+0.38) | +0.72 (+0.37) | +0.60 (+0.19) |
| Tanks and Temples (5), one seed and one analytic refiner | −0.13 (−0.05) | +0.38 (−0.08) | +0.55 (+0.23) | +0.46 (+0.21) | +0.31 (+0.06) |
| 33 scenes of five other datasets | −0.11 (0.00) | +0.25 (+0.06) | +0.55 (+0.28) | +0.50 (+0.24) | +0.26 (+0.13) |
Overall, at 1–25% smaller passes raise the gain to +0.25 to +0.76 dB on all five sets (every interval above zero), consistent with the reviewer's concern; the loss at 0.1% is not reduced, and is larger on the 46 scenes and the 25 DL3DV scenes.
-
The effect of the pass size on our own selection is not uniform: below zero at 0.1% on the 33 scenes (−0.10 dB), positive on ETH3D, Deep Blending and LLFF, close to zero on Nerfstudio, and negative on the single Ref-NeRF real scene (−0.26 to −0.43 dB at 5–25%). Retraining our refiner on 0.25N states changes little.
-
Time: 1.37 times the end-to-end time of the submitted setting (95% interval 1.33–1.40); this is 0.78 times the wall-clock time of the official optimized NanoGS (k = 4), and 1.48 times its CPU time.
This pass-size sensitivity analysis uses the original 46-scene panel, including the six scenes inspected to check transfer during development.
Limits: this analysis does not isolate the mechanism, since the two sides do not use the same pass size and smaller passes also recompute the costs twice as often.
Revision: Section 3.3 states the single-merge approximation; Appendix D.6 reports this sensitivity analysis and its run time.
Response to Reviewer gnwz (3/3): Q2 and Q3
Q2. Cost of the training data
“How much does it cost to get all training datasets?”
A one-time cost: under an hour of one workstation (24 CPU cores; the collection is CPU-bound) for the predictor labels, and about 5 CPU job-hours per predictor seed for the refiner's training states. The data are generated automatically from 64 public DL3DV scenes and their released 3DGS models, without annotation.
| Item | Cost |
|---|---|
| Predictor labels (1.1 GB) | about 48 min of one workstation (estimated from six of the 128 collection jobs, timed and extrapolated) |
| Refiner training states (12.7 GB), per predictor seed | about 5 CPU job-hours (13.7 when we repeated one seed on a shared machine) |
| Training the predictor | 25–30 s in the logs of the submitted checkpoints |
| Training the refiner | 18–20 min of optimization on one RTX A6000, measured after the submission on three re-trainings with the same recipe (one for the submitted predictor) (43–58 min for the whole process) |
| Deployment, per asset | our full method: 7.3 s on four CPU cores; official NanoGS (k = 4): 12.6 s (Table 8) |
For scale, collecting the predictor labels takes about as long as fitting a single 3DGS scene, and the two networks (62 kB and 38 kB) are then used unchanged on every new asset.
Q3. A network for the parameters of merged Gaussians
“Could a MLP determine the parameters of the merged Gaussians?”
Yes for the final Gaussians: this is what our refiner does. For the parents built during the recursion, we keep the analytic moments; two small prototypes of learned parents were not conclusive.
-
Final Gaussians. The refiner is an MLP (7,751 parameters) shared by all scenes. It uses no photograph and no camera at deployment, and applies bounded corrections to the color, opacity and three scales of every merged final Gaussian: +0.75 dB after learned selection and +0.96 dB after analytic selection (Table 2). At the six budgets from 0.5% to 20% that it was not trained on, it adds +0.54 to +0.99 dB on the 40 primary evaluation scenes.
-
Parents of the recursion: why analytic. The moments keep every covariance valid by construction, whereas a predicted parent needs constraints or a fallback. A parent becomes a child in later passes: at 0.1%, a final Gaussian results from up to about ten levels of pairwise merges, so errors of a learned parent would compound, whereas an error of the cost changes which pairs are merged but leaves every parent a moment match of its two children. Our labels, descriptors and training states also depend on the parent rule, so learning the parent means training parent and cost together.
-
Prototypes. A learned correction of the parent color (before the submission, earlier version of the system): the best of 16 variants, a linear model, was 0.014 dB below the analytic parent. Learned parents in the last passes (analytic selection): +0.47 / +0.17 dB at 0.1 / 1% without refiner, but +0.10 / −0.03 dB with a refiner for each variant (+0.02 dB at 0.1% without the scene that had served to choose the checkpoints).
Limits: we did not test larger models or more training data for the parents, and we do not conclude that learned parents cannot help.
Revision: the revised paper distinguishes the two cases (final Gaussians; parents of the recursion).
Thank you for these questions. The revised manuscript reports the primary evaluation on 40 scenes, the six transfer checks in Appendix D.7, and results on 97 additional scenes in Appendix D.1. Our reply to W2 measures how merges interact within a pass, and Appendix D.6 reports how changing the pass size affects quality and runtime. The revision also includes the pruning comparison in Appendix D.5 and the training costs in Table 10. We would be happy to discuss any of these results further.
Response to Reviewer gnwz (1/3): W1 and Q1
We thank the reviewer for the positive assessment, for finding the supervision well motivated and the matched-refinement experiment useful, and for the questions. We answer each weakness and question below; Tables A and B are in “General response (2/3)”, and the revisions are in blue in the revised PDF.
W1. Evaluation scenes and training
None of the original 46 scenes is among the training scenes of the submitted networks: the predictor and the refiners were trained on 64 DL3DV scenes only.
After fixing the submitted method, hyperparameters, and checkpoints, we checked cross-dataset transfer during development on nightsnow and playroom (Deep Blending), chair and drums (NeRF Synthetic), and train and truck (Tanks and Temples). This check did not inform those choices. Table A and manuscript Table 16 report the other 40 scenes from the submitted panel, which form the primary evaluation. Appendix D.7 and Table 31 report the six separately.
Scenes outside the 46. With the submitted networks unchanged, Table B reports 97 scenes outside the 46, from nine datasets; six of these datasets appear neither in training nor among the 46. They range from close-up objects to city districts seen from the air, in handheld and DSLR photos, videos and rendered aerial views, whereas all training scenes come from DL3DV.
Over the 97 scenes, learned − analytic selection (PSNR, each with its own refiner) is above zero from 1% to 25% and below zero at 0.1% (−0.08 dB); at 5–15% it is above zero in 7 of the 8 groups of four or more scenes, while on LLFF (8) the interval contains zero at every budget. Our full method is +0.8 to +1.5 dB above official NanoGS in these groups (mean of the five paper budgets).
Revision: Table B is in Appendix D.1 (Table 17).
Q1. Difference from importance-based pruning
Pruning scores single Gaussians and deletes them; we score pairs and replace each selected pair by one parent that still covers both. Regions covered only by deleted Gaussians lose their coverage, unless the survivors are re-optimized with photographs; our parent keeps the mass-weighted moments (position, covariance, color) of its two children, and the costs are recomputed after each pass.
Our score is learned from the rendered effect of the merge itself, with the target budget as an input, and needs no camera at deployment, whereas the scores of LightGaussian, PUP and REFINE are computed from the training cameras. Our pipeline does contain one pruning step, the opacity prefilter.
Original 46-scene panel, PSNR in dB, without fine-tuning (pruning step only for LightGaussian and PUP). We ran this comparison in the campaign for the submitted Table 1 and report it in Appendix D.5 of the revision.
Without refiner, our selection is ahead of LightGaussian in PSNR, the best of the three pruning methods we added, by +5.3 dB at 0.1% down to +1.1 dB at 25%, in 45 or 46 of the 46 scenes from 0.1% to 10% and in 31 at 25%.
Revision: Appendix D.5.