- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a learned controller for accelerating 3D Gaussian Splatting training. A PPO-trained policy observes a compact summary of optimization progress and controls densification, pruning, etc. The controller is trained beforehand and deployed on new scenes without policy updates. Experiments report improvements in time to target quality across several training backends.
Critical strengths
The paper addresses a useful problem. RL agents improve the allocation of computation during 3DGS optimization. This approach can improvements to rasterizers and optimizers.
The analysis provides an interpretable explanation of the learned behavior. The policy generally favors compact models on expensive backends and greater capacity on Faster-GS.
Critical weaknesses
-
The train scene was used to select checkpoints, but it also appears in the final evaluation. The cross-backend transfer experiments use only this scene, so stronger independent testing is needed.
-
It is unclear how much the adaptive policy helps. A tuned fixed schedule or replaying a learned schedule could explain some of the gains. These comparisons are missing.
-
The shared training framework changes some of the original backends’ behavior, including resets and pruning. It is unclear whether the same gains would hold in their native implementations.
Questions for authors
-
Can you test cross-backend transfer on multiple scenes that were not used for checkpoint selection?
-
How does PACE compare with a tuned fixed schedule or replaying a learned schedule?
-
Do the gains hold when using the backends’ native training implementations?
6 comments
Response (part 2)
W2 / Q2 — Tuned fixed schedule and open-loop replay
Response. We ran three non-learned controls on all eight Table 2 scenes under the same protocol (Table R9): a tuned fixed schedule (3×3 grid over densification interval 100, 200, 400 × threshold multiplier 0.5, 1, 2, selected per backend on train and ignatius); a hand-written rule (densify until the validation gain per block drops below 0.03 dB, then freeze and prune); and open-loop replay of a learned schedule. For the replay we use a single recording: the per-block action sequence the 3DGS policy emitted on train on the 3DGS backend. It is re-issued without observations on the other seven scenes and, unchanged, on the Faster-GS and Dash backends (Table R9) and on LeGS, FastGS and SAD-GS (Table R10), each time against the same 3DGS policy run closed-loop on that backend. On the three Table 2 backends (24 cells): tuned 1.11× (below 1× on 4 cells), rule 1.25× (below 1× on 3), PACE 1.43× (never below 1×); the 3DGS recording 1.40× on its own backend (7 scenes), 1.25× on Dash and 0.88× on Faster-GS.
| backend | tuned fixed schedule | 3DGS-schedule replay | hand-written rule | PACE |
|---|---|---|---|---|
| 3DGS | 1.08× (-0.08 dB / 0.99) | 1.40× (+0.46 dB / 0.71) | 1.27× (+0.34 dB / 0.32) | 1.49× (+0.46 dB / 0.65) |
| Faster-GS | 1.07× (+0.11 dB / 0.42) | 0.88× (+0.02 dB / 0.59) | 1.23× (+0.14 dB / 0.31) | 1.27× (+0.59 dB / 1.06) |
| Dash | 1.19× (+0.18 dB / 0.43) | 1.25× (+0.41 dB / 0.75) | 1.25× (+0.42 dB / 0.37) | 1.54× (+0.29 dB / 0.42) |
| all scenes | 1.11× (+0.07 dB / 0.57) | 1.15× (+0.30 dB / 0.68) | 1.25× (+0.30 dB / 0.33) | 1.43× (+0.45 dB / 0.66) |
(columns 2-5: gmean speed-up over the fixed schedule (mean ΔPSNR / gmean #G ratio, vs the fixed schedule))
Table R9. Non-RL schedules vs PACE, per backend and over all eight Table 2 scenes: gmean time-to-target speed-up over the backend’s stock fixed schedule, with the mean final-PSNR difference and the gmean Gaussian ratio to the fixed schedule in brackets. Replay = the 3DGS policy’s train recording re-issued open-loop (seven scenes on 3DGS, where it was recorded on train); PACE = the backend’s own policy. Per-scene values, with SSIM, are in Table 19 of the revised manuscript.
Response (part 3)
| backend | scenes | closed-loop policy · gmean speed-up over the fixed schedule | open-loop replay · gmean speed-up over the fixed schedule | mean ΔPSNR closed / replay · (dB, vs the fixed schedule) |
|---|---|---|---|---|
| 3DGS (own) | 7 | 1.45× | 1.40× | +0.53 / +0.46 |
| Faster-GS | 7 | 0.93× | 0.88× | -0.02 / -0.04 |
| Dash | 7 | 1.27× | 1.25× | +0.18 / +0.31 |
| LeGS | 7 | 1.40× | 1.41× | -0.31 / -0.29 |
| FastGS | 8 | 1.17× | 1.17× | -0.18 / -0.34 |
| SAD-GS | 8 | 1.15× | 0.70× | -0.09 / -0.04 |
| SAD-GS (stretched) | 8 | 1.15× | 0.91× | -0.09 / -0.04 |
Table R10. The 3DGS policy closed-loop vs its train recording replayed open-loop, per backend: gmean over scenes of the speed-up over the backend’s fixed schedule, and the mean final-PSNR difference to the fixed schedule. SAD-GS cells use the 7k / 3k-iteration horizons of Table R5; “stretched” indexes the 30k-iteration recording by training fraction instead of block count; courthouse has no closed-loop cell on Faster-GS, Dash and LeGS. Per-scene values, with SSIM, are in Table 20 of the revised manuscript.
Two things follow. First, the schedule itself carries most of the gain. On 3DGS, Dash, LeGS and FastGS the recording is within 3% of the closed-loop policy (Table R10): the RL formulation discovered offline a block schedule that transfers across scenes and across four backends, which neither the grid over the schedule’s own parameters nor the hand-written rule found. Closed-loop and open-loop end within 0.2 dB of each other on average on every backend. Second, the feedback matters where the controlled backend does not behave like the one the recording was taken on. On Faster-GS the 3DGS policy itself transfers poorly on barn and meetingroom (Table R8), and the recording inherits it (0.88× vs 0.93×). On SAD-GS, whose structure-aware split multiplies the model by 10–20× in a single event and whose horizon is 7k iterations, the recording keeps issuing the growth phase of the 30k-iteration 3DGS schedule and ends with 1.3–5× more Gaussians at 0.70×; the closed-loop policy, which observes the population and the validation gain, gates the growth and reaches 1.15× at the same quality (−0.09 dB vs −0.04 dB to the fixed schedule). Re-indexing the recording by training fraction, so that the 7k-iteration cells see the whole schedule including its freeze-and-prune phase (“stretched” in Table R10), recovers part of the loss (0.91×; below the closed-loop policy on all six Tanks&Temples scenes, above it on the two Mip-360 scenes), so the re-timing is not the whole story: on SAD-GS the closed-loop policy is 1.26× faster than the best open-loop variant of its own schedule. The revision states the contribution accordingly: the RL formulation discovers offline a block schedule that transfers across scenes and backends; closed-loop deployment costs 8 ms per block and is what keeps the gain on a backend whose training dynamics differ from the training backend’s (SAD-GS: 1.15× closed-loop vs 0.70× open-loop). In the revised manuscript: Appendix D.7 “Non-learned schedule baselines and open-loop replay” (Tables 19 and 20, in blue) and the conclusion (Section 6, in blue).
Response (part 1)
W1 / Q1 —
trainin checkpoint selection; transfer on one sceneResponse. Yes. (a) The headline is now quoted over the seven scenes not used for checkpoint selection (conclusion, Section 6): 3DGS 1.45×, Faster-GS 1.26×, Dash 1.48×, all 21 cells above 1×; the camera-ready Table 2 marks
trainas the checkpoint-selection scene. (b) The full 3-policy × 4-backend matrix was rerun on six held-out scenes (Table R8). Over the 54 transfer cells: 43 of 51 non-failing cells ≥ 1×, gmean 1.22× over the 47 reliable cells (at least three rungs, final PSNR within 1 dB); per policy 1.26× (3DGS), 1.14× (Faster-GS), 1.27× (Dash). The LeGS column is above 1× in all 18 cells. One pair fails systematically: the Dash policy on Faster-GS (no rung reached on three scenes, <0.25× on three more), because a policy trained where compaction pays never densifies on a backend that rewards growth. One further cell degrades quality (3DGS policy on Faster-GS, meetingroom, −3.3 dB). In the camera-ready PDF the abstract’s “eight of nine pairs” becomes “eight of nine pairs transfer on the majority of seven held-out scenes; one pair fails”, with every cell in the appendix. In the revised manuscript: the LeGS-only table of Section 5.3 is replaced by the full matrix on seven held-out scenes (Table 4, with the paragraph that follows it, in blue); Table 4 addstrainto the six scenes of Table R8 and omitscourthouse(native cells only), so its totals read 51 of 60 non-failing transfer cells ≥ 1× and a gmean of 1.26× once the four cells below 0.25× are set aside. Thetrain-only detail stays in Appendix D.6.Table R8. Cross-backend transfer on six held-out scenes: speed-up over the backend’s fixed schedule. * native cell; “fail” = no rung reached; PSNR deficits above 1 dB in brackets.