acceptodds
Under review as a conference paper at ICLR 2027

PACE: A Reinforcement-Learning Agent for Accelerated 3D Gaussian Splatting

62 views

Abstract

Accelerating 3D Gaussian Splatting (3DGS) training is a central open problem for real-world deployment: per-scene optimization still costs minutes of GPU time, which highly depends on a fixed, hand-tuned schedule of densification, pruning, opacity resets, and learning-rate decay. Heuristic alternatives require significant domain expertise and manual tuning, while existing learned controllers are trained online per scene, increasing training time. To address these challenges, we propose Policy for Adaptive Compute-Efficient (PACE) 3DGS training, a reinforcement-learning agent that accelerates training and generalizes across scenes and 3DGS training backends. We cast 3DGS training as a block-wise Markov decision process over a pluggable training backend. At each 3DGS training block, the agent observes the optimization state (a compact summary of training progress) and selects controls (Gaussian thresholds, learning rates, etc.) for the next block. PACE uses a lightweight MLP trained offline on a small set of 3D scenes, with a time-value reward that favors achieving higher quality more quickly. At deployment, PACE only requires feedforward evaluation of the trained policy. Evaluated on four representative backends (3DGS, Faster-GS, DashGaussian, and LeGS) across different scene types, PACE's acceleration has a consistent mechanism in the Gaussians optimization: where optimizing a Gaussian is expensive (e.g., 3DGS, DashGaussian), it densifies sparingly and freezes growth early, matching the fixed schedule's quality with half the primitives; where it is cheap (Faster-GS), it densifies aggressively, converting capacity into quality during early training stages. The result is up to faster training (e.g., to seconds to reach  dB on 3DGS), faster than the fixed schedule in all scenebackend cases, up to smaller models, up to lower peak training VRAM, and up to faster rendering. The learned schedules also transfer across backends without retraining in eight of the nine policy–backend pairs we evaluate, retaining –, and on LeGS reaching . The exception is predicted by the discovered mechanism, where a policy trained such that compaction is desired (DashGaussian) underperforms on the one backend where growth is optimal (Faster-GS).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian splatting, Reinforcement Learning, Learned Optimization Control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HvbpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a learned controller for accelerating 3D Gaussian Splatting training. A PPO-trained policy observes a compact summary of optimization progress and controls densification, pruning, etc. The controller is trained beforehand and deployed on new scenes without policy updates. Experiments report improvements in time to target quality across several training backends.

Critical strengths

The paper addresses a useful problem. RL agents improve the allocation of computation during 3DGS optimization. This approach can improvements to rasterizers and optimizers.

The analysis provides an interpretable explanation of the learned behavior. The policy generally favors compact models on expensive backends and greater capacity on Faster-GS.

Critical weaknesses
  1. The train scene was used to select checkpoints, but it also appears in the final evaluation. The cross-backend transfer experiments use only this scene, so stronger independent testing is needed.

  2. It is unclear how much the adaptive policy helps. A tuned fixed schedule or replaying a learned schedule could explain some of the gains. These comparisons are missing.

  3. The shared training framework changes some of the original backends’ behavior, including resets and pruning. It is unclear whether the same gains would hold in their native implementations.

Questions for authors
  1. Can you test cross-backend transfer on multiple scenes that were not used for checkpoint selection?

  2. How does PACE compare with a tuned fixed schedule or replaying a learned schedule?

  3. Do the gains hold when using the backends’ native training implementations?

6 comments
AuthorsOpenReview ↗

Response (part 1)

W1 / Q1 — train in checkpoint selection; transfer on one scene

Response. Yes. (a) The headline is now quoted over the seven scenes not used for checkpoint selection (conclusion, Section 6): 3DGS 1.45×, Faster-GS 1.26×, Dash 1.48×, all 21 cells above 1×; the camera-ready Table 2 marks train as the checkpoint-selection scene. (b) The full 3-policy × 4-backend matrix was rerun on six held-out scenes (Table R8). Over the 54 transfer cells: 43 of 51 non-failing cells ≥ 1×, gmean 1.22× over the 47 reliable cells (at least three rungs, final PSNR within 1 dB); per policy 1.26× (3DGS), 1.14× (Faster-GS), 1.27× (Dash). The LeGS column is above 1× in all 18 cells. One pair fails systematically: the Dash policy on Faster-GS (no rung reached on three scenes, <0.25× on three more), because a policy trained where compaction pays never densifies on a backend that rewards growth. One further cell degrades quality (3DGS policy on Faster-GS, meetingroom, −3.3 dB). In the camera-ready PDF the abstract’s “eight of nine pairs” becomes “eight of nine pairs transfer on the majority of seven held-out scenes; one pair fails”, with every cell in the appendix. In the revised manuscript: the LeGS-only table of Section 5.3 is replaced by the full matrix on seven held-out scenes (Table 4, with the paragraph that follows it, in blue); Table 4 adds train to the six scenes of Table R8 and omits courthouse (native cells only), so its totals read 51 of 60 non-failing transfer cells ≥ 1× and a gmean of 1.26× once the four cells below 0.25× are set aside. The train-only detail stays in Appendix D.6.

scenepolicy3DGSFaster-GSDashLeGS
ignatius3DGS2.03×*1.51×1.52×1.51×
Faster-GS0.84×1.45×*1.23×1.22×
Dash1.76×fail1.81×*1.29×
caterpillar3DGS1.28×*1.25×1.37×1.25×
Faster-GS1.46×1.38×*1.30×1.40×
Dash1.41×0.10× (−3.3 dB)1.43×*1.30×
barn3DGS1.26×*0.74×1.16×1.20×
Faster-GS1.06×1.05×*1.19×1.26×
Dash1.19×0.66×1.32×*1.26×
meetingroom3DGS1.19×*0.21× (−3.3 dB)1.24× (−1.0 dB)1.32×
Faster-GS1.06×1.16×*1.27×1.44×
Dash1.29×0.08× (−4.2 dB)1.41×*1.27×
bicycle3DGS1.21×*1.31×1.06×1.60×
Faster-GS1.17×1.28×*1.02×1.30×
Dash1.39×fail1.41×*1.58×
stump3DGS1.44×*1.28×1.21×1.18×
Faster-GS0.73×1.22×*0.91×1.05×
Dash1.06×fail1.14×*1.40×

Table R8. Cross-backend transfer on six held-out scenes: speed-up over the backend’s fixed schedule. * native cell; “fail” = no rung reached; PSNR deficits above 1 dB in brackets.

AuthorsOpenReview ↗

Response (part 2)

W2 / Q2 — Tuned fixed schedule and open-loop replay

Response. We ran three non-learned controls on all eight Table 2 scenes under the same protocol (Table R9): a tuned fixed schedule (3×3 grid over densification interval 100, 200, 400 × threshold multiplier 0.5, 1, 2, selected per backend on train and ignatius); a hand-written rule (densify until the validation gain per block drops below 0.03 dB, then freeze and prune); and open-loop replay of a learned schedule. For the replay we use a single recording: the per-block action sequence the 3DGS policy emitted on train on the 3DGS backend. It is re-issued without observations on the other seven scenes and, unchanged, on the Faster-GS and Dash backends (Table R9) and on LeGS, FastGS and SAD-GS (Table R10), each time against the same 3DGS policy run closed-loop on that backend. On the three Table 2 backends (24 cells): tuned 1.11× (below 1× on 4 cells), rule 1.25× (below 1× on 3), PACE 1.43× (never below 1×); the 3DGS recording 1.40× on its own backend (7 scenes), 1.25× on Dash and 0.88× on Faster-GS.

backendtuned fixed schedule3DGS-schedule replayhand-written rulePACE
3DGS1.08× (-0.08 dB / 0.99)1.40× (+0.46 dB / 0.71)1.27× (+0.34 dB / 0.32)1.49× (+0.46 dB / 0.65)
Faster-GS1.07× (+0.11 dB / 0.42)0.88× (+0.02 dB / 0.59)1.23× (+0.14 dB / 0.31)1.27× (+0.59 dB / 1.06)
Dash1.19× (+0.18 dB / 0.43)1.25× (+0.41 dB / 0.75)1.25× (+0.42 dB / 0.37)1.54× (+0.29 dB / 0.42)
all scenes1.11× (+0.07 dB / 0.57)1.15× (+0.30 dB / 0.68)1.25× (+0.30 dB / 0.33)1.43× (+0.45 dB / 0.66)

(columns 2-5: gmean speed-up over the fixed schedule (mean ΔPSNR / gmean #G ratio, vs the fixed schedule))

Table R9. Non-RL schedules vs PACE, per backend and over all eight Table 2 scenes: gmean time-to-target speed-up over the backend’s stock fixed schedule, with the mean final-PSNR difference and the gmean Gaussian ratio to the fixed schedule in brackets. Replay = the 3DGS policy’s train recording re-issued open-loop (seven scenes on 3DGS, where it was recorded on train); PACE = the backend’s own policy. Per-scene values, with SSIM, are in Table 19 of the revised manuscript.

AuthorsOpenReview ↗

Response (part 3)

backendscenesclosed-loop policy · gmean speed-up over the fixed scheduleopen-loop replay · gmean speed-up over the fixed schedulemean ΔPSNR closed / replay · (dB, vs the fixed schedule)
3DGS (own)71.45×1.40×+0.53 / +0.46
Faster-GS70.93×0.88×-0.02 / -0.04
Dash71.27×1.25×+0.18 / +0.31
LeGS71.40×1.41×-0.31 / -0.29
FastGS81.17×1.17×-0.18 / -0.34
SAD-GS81.15×0.70×-0.09 / -0.04
SAD-GS (stretched)81.15×0.91×-0.09 / -0.04

Table R10. The 3DGS policy closed-loop vs its train recording replayed open-loop, per backend: gmean over scenes of the speed-up over the backend’s fixed schedule, and the mean final-PSNR difference to the fixed schedule. SAD-GS cells use the 7k / 3k-iteration horizons of Table R5; “stretched” indexes the 30k-iteration recording by training fraction instead of block count; courthouse has no closed-loop cell on Faster-GS, Dash and LeGS. Per-scene values, with SSIM, are in Table 20 of the revised manuscript.

Two things follow. First, the schedule itself carries most of the gain. On 3DGS, Dash, LeGS and FastGS the recording is within 3% of the closed-loop policy (Table R10): the RL formulation discovered offline a block schedule that transfers across scenes and across four backends, which neither the grid over the schedule’s own parameters nor the hand-written rule found. Closed-loop and open-loop end within 0.2 dB of each other on average on every backend. Second, the feedback matters where the controlled backend does not behave like the one the recording was taken on. On Faster-GS the 3DGS policy itself transfers poorly on barn and meetingroom (Table R8), and the recording inherits it (0.88× vs 0.93×). On SAD-GS, whose structure-aware split multiplies the model by 10–20× in a single event and whose horizon is 7k iterations, the recording keeps issuing the growth phase of the 30k-iteration 3DGS schedule and ends with 1.3–5× more Gaussians at 0.70×; the closed-loop policy, which observes the population and the validation gain, gates the growth and reaches 1.15× at the same quality (−0.09 dB vs −0.04 dB to the fixed schedule). Re-indexing the recording by training fraction, so that the 7k-iteration cells see the whole schedule including its freeze-and-prune phase (“stretched” in Table R10), recovers part of the loss (0.91×; below the closed-loop policy on all six Tanks&Temples scenes, above it on the two Mip-360 scenes), so the re-timing is not the whole story: on SAD-GS the closed-loop policy is 1.26× faster than the best open-loop variant of its own schedule. The revision states the contribution accordingly: the RL formulation discovers offline a block schedule that transfers across scenes and backends; closed-loop deployment costs 8 ms per block and is what keeps the gain on a backend whose training dynamics differ from the training backend’s (SAD-GS: 1.15× closed-loop vs 0.70× open-loop). In the revised manuscript: Appendix D.7 “Non-learned schedule baselines and open-loop replay” (Tables 19 and 20, in blue) and the conclusion (Section 6, in blue).

Reviewer_6d1oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents PACE, a reinforcement-learning agent that accelerates 3D Gaussian Splatting training. Training is cast as a block-wise Markov decision process over a pluggable backend: at each block boundary, a lightweight, PPO-trained MLP observes a compact normalized state (progress, loss, validation, compute, population, and previous action) and selects a mixed action setting the block length, densification/pruning modes and intervals, opacity resets, and continuous threshold and learning-rate multipliers. The policy is trained offline once on a small scene set, frozen, and generalizes zero-shot to unseen scenes across four backends. The method in this paper delivers up to 2.1× faster training and smaller models.

Critical strengths
  1. PACE advances LeGS, which learns a per-Gaussian, online density controller, to a block-level controller that output five categorical decisions, and a vector of seven continuous learning rate multipliers. The results show the acceleration ratio of PACE over LeGS.
  2. When evaluated on held-out scenes across distinct backends, it transfers zero-shot while retaining consistent speed-ups and matching or improving the quality of each backend's fixed schedule.
Critical weaknesses
  1. The reported acceleration is uneven across backends and shrinks as the backend itself becomes more heavily engineered. The cross-scene geometric-mean speed-up is 1.49× on official 3DGS and 1.54× on DashGaussian, but only 1.27× on Faster-GS—the backend that already fuses the rasterizer, sorting, and optimizer—with individual scenes as low as 1.05× (barn).
  2. The manuscript is internally inconsistent about the state-vector size: Section 4.2 states sk∈[0,1]115s_k \in [0,1]^{115}, whereas the introduction and Appendix C.1 describe a 45-dimensional state.
  3. There are formatting issues with the paragraph spacing on Page 9. It is not a nice way to compact the paper size.
Questions for authors

I am wondering how PACE works for recent 3DGS acceleration methods, such as TurboGS, ICM 2026?Is it possible to train a policy network to predict the sampling probability of each pixel in that paper?

3 comments

W1 — Uneven acceleration across backends

Response. Agreed: the factor depends on the structure of the backend, and the mechanism of Section 5.1 predicts it. PACE decides block by block, from the training signal, when to densify, prune and reset; the time it saves is the time the backend would spend on Gaussians the scene does not need, so the saving scales with the cost of a Gaussian. On 3DGS and DashGaussian a Gaussian is expensive, so the policy densifies sparingly, freezes growth early and prunes harder, ending with 0.35–0.8× the fixed schedule’s Gaussians and the largest factors (1.45×, 1.48×). On Faster-GS the fused rasterizer and optimizer make a Gaussian cheap, so the same policy family grows the model instead (0.8–1.3×), trading the headroom for quality (+0.6 to +1.5 dB on five of eight scenes) at a smaller factor (1.26×); barn is the case where its fixed schedule is already near optimum (1.05×, +0.6 dB). DL3DV shows the same ordering (3DGS 1.21× > Dash 1.15× > Faster-GS 1.05×), and on FastGS, cheap but less tuned, the gain returns (1.4–1.6×). The revision states this as a prediction and quotes the headline over the seven scenes not used for checkpoint selection: 3DGS 1.45×, Faster-GS 1.26×, Dash 1.48×.

W2 — State-vector size

Response. The state is 45-dimensional and is now unified throughout the manuscript

W3 — Page-9 spacing

Response. Restored: the revised PDF removes every negative paragraph and heading spacing in Section 5 and the conclusion, and the main text still ends on page 10. The LeGS-only table that forced the squeeze has been replaced by the full transfer matrix (Table 4)

Q — TurboGS and per-pixel sampling

Response. We could not include a TurboGS comparison: there is no official implementation of the paper (its project page lists the code as “coming soon”). We would like to use the question to make explicit how PACE relates to methods such as TurboGS. Faster 3DGS trainers appear every few months and each supersedes the previous one; PACE is not a competitor in that sequence but a controller that sits on top of whichever trainer is current. TurboGS changes the rendering loop (pixel subsampling), not the densification schedule, so like FastGS and SAD-GS it is a substrate PACE can schedule. Neither was a backend of the submission, and the policy trained once on 3DGS accelerated their official loops without retraining (1.4–1.6× and 1.16× over the authors’ schedules, tGmC W4); on SAD-GS the closed-loop policy is also what keeps that gain where a fixed recording of the schedule fails (Hvbp W2). We expect the same to hold for TurboGS once its code is available, and a new state-of-the-art trainer does not make PACE obsolete: it becomes a new substrate for the same policy. Sampling itself we consider future work: a learned per-pixel sampling probability is compatible with our formulation as an extra continuous action (a global rate, or the parameters of the probability map) under the same time-value reward, whereas a full per-pixel map at every iteration would need a spatial policy rather than PACE’s block-wise controller.

Dear reviewer,

Please let us know if you have any follow-up questions about our work.

Thank you

Thanks for the clarification. I accept the explaination on TurboGS issue and Uneven acceleration across backends. I will maintain my weak acceptance score.

Reviewer_tGmCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores RL-based control of the Gaussian fitting process and demonstrates improvements in convergence speed. While RL-based Gaussian optimization has already been explored by LeGS, the use of an offline-trained, reusable scheduling policy is a meaningful distinction. However, the evidence for generalization, quality preservation, and competitiveness with existing acceleration methods remains insufficient.

Critical strengths
  1. Offline RL is a promising approach to controlling Gaussian optimization, allowing the learned policy to be reused across scenes without per-scene policy training.
  2. The analysis of model size and backend-dependent behavior provides useful insights into how different scheduling strategies affect efficiency.
Critical weaknesses
  1. Limited comparison with LeGS. The direct comparison is restricted to a single scene, train. Moreover, the reported PSNR is around 21 dB, substantially below the approximately 22.8 dB reported in the original LeGS paper. Although differences in evaluation protocols may contribute to this gap, the authors should explain it and compare performance at comparable quality levels. Faster progress during early optimization does not establish a better overall speed–quality trade-off or comparable final quality.

  2. Insufficient evidence of generalization. Training the policy on ten scenes and evaluating it on eight provides preliminary evidence, but the scene coverage is too limited to substantiate broad generalization claims. Evaluation on additional scenes from diverse datasets, such as RealEstate10K, DL3DV, or ScanNet++, would help establish robustness to different scene geometries, camera distributions, and capture conditions.

  3. Incomplete evaluation of perceptual quality. Most quantitative tables omit SSIM and LPIPS. PSNR alone is insufficient to establish rendering quality, particularly when aggressive model compaction may affect fine structures and textures. Table 19 already shows LPIPS degradation in some settings; for example, on the 3DGS backend, LPIPS worsens from 0.238 to 0.251 despite a slight PSNR improvement. Consistent reporting of all three metrics across scenes and final controller configurations is necessary to assess whether acceleration preserves visual quality.

  4. Insufficient baselines and justification for RL. Direct comparisons with FastGS, FastGS-Big, and SAD-GS are missing. These methods report strong speed–quality trade-offs without RL-based control, making them relevant baselines. Comparisons under matched hardware and evaluation settings are needed to establish practical competitiveness. The authors should also compare against tuned fixed schedules or simple adaptive scheduling rules to clarify how much of the improvement specifically requires RL.

Questions for authors

Please refer to the weakness.

9 comments
AuthorsOpenReview ↗

Response (part 1)

W1 — LeGS comparison

Response. Seven scenes. Table R1 runs each policy on top of LeGS’s own per-Gaussian controller against native LeGS (same code, same GPU). All 21 cells are above 1× (gmean 1.38×), with final PSNR within 1 dB of native LeGS in every cell and within 0.5 dB in 18.

scene3DGS policyFaster-GS policyDash policy
train1.86× (20.9/20.8; 399k/609k)1.62× (21.4/20.9; 720k/642k)2.02× (20.9/20.3; 482k/607k)
ignatius1.51× (23.9/24.2; 937k/1093k)1.22× (24.5/24.1; 1387k/1129k)1.29× (24.3/24.4; 1065k/1093k)
caterpillar1.25× (23.1/23.1; 485k/629k)1.40× (23.6/23.0; 821k/625k)1.30× (22.6/23.1; 542k/630k)
barn1.20× (28.0/28.5; 381k/558k)1.26× (28.8/28.3; 525k/542k)1.26× (28.1/28.2; 417k/572k)
meetingroom1.32× (26.1/26.9; 421k/579k)1.44× (26.9/26.3; 557k/597k)1.27× (25.9/26.8; 523k/585k)
bicycle1.60× (25.1/25.4; 1277k/2052k)1.30× (25.6/25.4; 2563k/1987k)1.58× (25.3/25.4; 1805k/1964k)
stump1.18× (26.1/26.4; 998k/1464k)1.05× (26.4/26.4; 1536k/1388k)1.40× (26.3/26.4; 1215k/1346k)

Table R1. PACE on LeGS vs native LeGS, seven held-out scenes: speed-up (final PSNR PACE/LeGS; Gaussians PACE/LeGS).

The gap to 22.8 dB has two causes. (i) Protocol: we evaluate on 12 of the 38 test views. The official FastGS train model scores 22.35 dB on all 38 views and 21.61 dB on our 12, so about 0.7 dB is the view subset, which is identical for every method in a table. (ii) Settings: our LeGS reference used FastGS’s generic densification thresholds instead of LeGS’s released train recipe (grad_abs_thresh 1e-4, highfeature_lr 0.042, mult 0.7) and omitted LeGS’s post-15k opacity prune. We reran native LeGS and PACE-on-LeGS with the release recipe (Table R2). Native LeGS then reproduces the paper (22.89 dB on the full split vs 22.84 reported; 1.33M Gaussians), and PACE still gives 1.6–1.9× on top of it.

sceneLeGS settingsnative LeGS · PSNR (full split) / N3DGS policyFaster-GS policyDash policy
trainrelease recipe22.25 (22.89) / 1329k1.87× (21.5 / 1531k)1.68× (22.8 / 2711k)1.74× (21.6 / 2718k)
traingeneric (submission)20.83 / 608k1.86× (20.9 / 398k)1.62× (21.4 / 719k)2.02× (20.9 / 481k)
caterpillarrelease recipe24.35 (25.01) / 1768k1.76× (24.1 / 1906k)1.60× (24.4 / 2808k)1.60× (24.2 / 3309k)
caterpillargeneric (submission)23.06 / 629k1.25× (23.1 / 485k)1.40× (23.6 / 820k)1.30× (22.6 / 542k)

Table R2. LeGS with its release recipe vs. the submission’s settings. Full-split PSNR in brackets; speed-up over native LeGS at the same settings.

At comparable quality. The speed-up is a geometric mean over the ladder up to the lower of the two final PSNRs, not an early-rung number. With the release recipe, the Faster-GS policy ends above native LeGS on train (+0.5 dB, 1.68×) and the 3DGS policy within 0.3 dB on caterpillar (1.76×); on train the 3DGS and Dash policies are 1.7–1.9× faster to 21 dB but end 0.6–0.8 dB below native LeGS, which we state. The camera-ready PDF will carry the release-recipe version of Table R1 on all seven scenes, with full-split PSNR next to the 12-view numbers and the protocol difference stated next to every table; at the time of writing, the release-recipe runs exist for train and caterpillar (Table R2), and the LeGS column of the manuscript’s Table 4 still uses the submission’s settings.

W2 — Generalisation

Response. We ran five DL3DV-10K scenes on all three backends (Table R3): 1.13× overall, 12 of 15 cells ≥ 1×, final PSNR within 0.35 dB of the fixed schedule in all 15. Gains are smaller than on Tanks&Temples because these captures are small (a few hundred thousand Gaussians) and the fixed schedule converges in minutes, so there is less schedule to save. With the six new transfer scenes, PACE is now evaluated on 13 held-out scenes from three datasets, none used for training or checkpoint selection. ScanNet++ needs an access agreement we do not hold, and RealEstate10K needs per-scene COLMAP; both are future work. In the revised manuscript: Table 3 and the paragraph “DL3DV” at the end of Section 5.2, in blue.

scene3DGSFaster-GSDash
6d81c5ab0.83× (22.3/22.4)1.38× (22.9/22.6)1.13× (22.4/22.8)
374ffd0c1.32× (30.6/30.5)0.86× (30.8/30.7)1.11× (30.5/30.6)
7705a2ed1.18× (29.5/29.5)0.93× (29.8/29.5)1.09× (29.3/29.5)
cbd44beb1.44× (29.8/29.7)1.01× (29.8/29.7)1.22× (29.4/29.7)
6d2216251.41× (24.4/24.4)1.14× (24.8/24.5)1.19× (24.2/24.5)
gmean1.21×1.05×1.15×

Table R3. DL3DV: speed-up over the fixed schedule (final PSNR PACE / fixed).

AuthorsOpenReview ↗

Response (part 2)

W3 — SSIM and LPIPS

Response. SSIM is logged in every run on the same test views as PSNR; the revision reports it next to PSNR in every results table (Tables 2, 3, 19, 20 and 21, and Tables R8 and R5 below) and next to LPIPS in the new perceptual table (Table 25). LPIPS (VGG) was measured at 30k on four scenes × three backends (Table R4). The pattern follows the mechanism: where PACE compacts, LPIPS gets worse (3DGS, 50–66% of the fixed Gaussians, +0.009 on average; Dash, 35–42%, +0.034) at equal or higher PSNR; where it grows (Faster-GS, 110–130%), LPIPS improves by 0.015–0.031 with +0.4 to +0.8 dB. SSIM moves with LPIPS. The revision states this trade-off: at matched PSNR the compaction policies cost perceptual detail, which the reward’s time and count weights can trade back. In the revised manuscript: Appendix D.12 “Perceptual quality on held-out scenes” (Table 25: PSNR, SSIM and LPIPS, fixed vs PACE, four scenes × three backends), in blue.

scene3DGSFaster-GSDash
train0.238 / 0.2510.239 / 0.2080.240 / 0.271
ignatius0.183 / 0.1840.183 / 0.2000.189 / 0.230
caterpillar0.241 / 0.2560.242 / 0.2190.248 / 0.291
barn0.220 / 0.2290.222 / 0.2070.224 / 0.243

Table R4. LPIPS (VGG, lower is better) at 30k, fixed schedule / PACE. F

AuthorsOpenReview ↗

Response (part 3)

W4 — Baselines and the need for RL

Response. FastGS / FastGS-Big (official code, same scenes, split, views, GPU and clock). FastGS is a different backend and is faster than our 3DGS runs: 3DGS+PACE is 0.47× of FastGS, Faster-GS+PACE 1.00×, Dash+PACE 1.07×; FastGS-Big is 1.07× over FastGS. Since PACE schedules a backend rather than replacing it, we added the official FastGS loop as a fifth backend (its scoring, pruning and optimiser kept verbatim; the policy only gates densification, thresholds, pruning and resets) and ran the three existing policies zero-shot (Table R6): 1.38–1.56× over FastGS and 1.22–1.39× over FastGS-Big, all 24 cells ≥ 1× over FastGS, final PSNR within about 0.3 dB on average. Caveat: FastGS publishes recipes for train, bicycle and stump only; the other scenes use the train recipe for both. SAD-GS (structure-aware densification, SIGGRAPH 2026; official code) was run the same way: the authors’ script with their Tanks&Temples and Mip-360 recipes (7k / 3k iterations) on the eight scenes, and PACE on top of the SAD-GS loop as a sixth backend (its structure-aware split at its own 500-iteration cadence, warm-up densification, end-of-densification prune, L2 term and hybrid optimizer kept verbatim; the policy gates the events and scales the thresholds), same horizon for both arms (Table R5). SAD-GS is a strong recipe on our GPU: 1.55× over FastGS and 1.35× over FastGS-Big, mostly from Mip-360. Our fixed schedule on the SAD-GS loop reproduces it (0.99×). Zero-shot, the 3DGS policy is 1.16× faster than the authors’ recipe at matched PSNR with 0.6× the Gaussians (1.62× over FastGS, 1.49× over FastGS-Big), the Dash policy 1.07× at the same size and PSNR (−0.03 dB), and the Faster-GS policy 0.90× with a quarter of the Gaussians. Final PSNR of the 3DGS policy is within 0.25 dB of SAD-GS on the six Tanks&Temples scenes and 0.7–0.9 dB below on the two Mip-360 scenes, where it prunes to a quarter of SAD-GS’s 5M Gaussians (a capacity ceiling of the policy’s observation scaling, not of the scheduler). PACE on FastGS with the Dash policy is 1.05× over SAD-GS (ahead on the six Tanks&Temples scenes, behind on Mip-360). TurboGS has no official code release (its project page lists the code as “coming soon”), so it cannot be compared under matched hardware. In the revised manuscript: FastGS / FastGS-Big and PACE on FastGS as Table 5 with the paragraph “Released acceleration methods” at the end of Section 5.3; SAD-GS as Appendix D.8 “SAD-GS as a backend” (Table 21); the two backends’ code bases, rasterizers, optimizers, densification rules, release settings and the controls PACE drives on each in Table 6 and Appendix B.7–B.8, read from the code; all in blue.

sceneSAD-GS (official) · final PSNR / SSIM / N3DGS policyFaster-GS policyDash policy
train21.70 / 0.823 / 2238k1.38× (21.94 / 0.819 / 1711k)1.51× (21.51 / 0.784 / 516k)1.29× (21.91 / 0.821 / 2642k)
ignatius22.58 / 0.831 / 3449k1.20× (22.43 / 0.815 / 1495k)0.90× (22.78 / 0.785 / 602k)1.04× (22.93 / 0.822 / 2358k)
caterpillar23.41 / 0.830 / 2269k1.49× (23.63 / 0.817 / 1401k)0.96× (23.00 / 0.779 / 520k)1.00× (23.70 / 0.826 / 2153k)
barn28.87 / 0.901 / 2087k0.99× (28.76 / 0.895 / 1816k)0.68× (27.74 / 0.853 / 395k)1.03× (28.86 / 0.900 / 2436k)
meetingroom25.23 / 0.884 / 1693k1.04× (25.24 / 0.883 / 1218k)0.89× (25.03 / 0.867 / 252k)1.05× (25.17 / 0.882 / 1637k)
courthouse22.03 / 0.794 / 1235k1.24× (22.14 / 0.794 / 1224k)0.82× (20.95 / 0.762 / 173k)0.99× (22.14 / 0.795 / 1642k)
bicycle25.17 / 0.776 / 5227k1.16× (24.32 / 0.711 / 1094k)0.64× (25.21 / 0.776 / 5844k)1.16× (24.75 / 0.751 / 4015k)
stump26.36 / 0.774 / 4833k0.88× (25.70 / 0.741 / 2907k)1.01× (26.41 / 0.776 / 1441k)1.02× (25.64 / 0.738 / 3424k)
gmean / mean ΔPSNR / gmean N ratio1.16× (-0.15 dB / 0.60)0.90× (-0.34 dB / 0.24)1.07× (-0.03 dB / 0.94)

Table R5. Official SAD-GS vs the three PACE policies on the SAD-GS loop, eight Table 2 scenes. Speed-up = gmean over the scene’s PSNR ladder of the time the official SAD-GS recipe needs over the time the policy needs (12 test views, training clock); final PSNR and SSIM on the full test split and model size N at SAD-GS’s horizon (7k iterations on Tanks&Temples, 3k on Mip-360). Last row: gmean speed-up, mean PSNR difference to SAD-GS, gmean ratio of N to SAD-GS’s.

over FastGSover FastGS-Big
FastGS-Big (released)1.07×—
FastGS + 3DGS policy1.39×1.24×
FastGS + Faster-GS policy1.38×1.22×
FastGS + Dash policy1.56×1.39×

Table R6. PACE on the FastGS backend vs official FastGS and FastGS-Big.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response summary

We thank the reviewers. The shared concerns were that the transfer evidence rested on one scene also used for checkpoint selection, that non-learned schedules and recent acceleration methods were missing as baselines, and that generalisation was untested beyond eight scenes. During the rebuttal we ran, under the paper’s protocol:

  • Cross-backend transfer on six further held-out scenes (3 policies × 4 backends): gmean 1.22× over the reliable transfer cells; one policy–backend pair (Dash policy on Faster-GS) fails systematically and is reported as such (Hvbp W1).

  • Non-RL schedule baselines on all eight scenes and three backends: a tuned fixed schedule 1.11×, a hand-written rule 1.20×, PACE closed-loop 1.39×. The 3DGS policy’s train recording replayed open-loop on six backends matches the closed-loop policy within 3% on 3DGS, Dash, LeGS and FastGS but collapses on SAD-GS (0.70× vs 1.15×), where the closed-loop feedback is what keeps the gain (Hvbp W2, tGmC W4).

  • FastGS, FastGS-Big and SAD-GS, official code, same GPU, and PACE run on top of both loops: 1.38–1.56× over FastGS on all eight scenes; 1.16× over SAD-GS’s hand-tuned recipe at matched PSNR with the 3DGS policy (tGmC W4).

  • LeGS on seven scenes, and the LeGS PSNR gap explained and corrected with LeGS’s release recipe (tGmC W1).

  • Five DL3DV scenes (1.13×) and LPIPS on four scenes × three backends (tGmC W2, W3).

The state-dimension typo is fixed (45, not 115) and the squeezed paragraph spacing of the experiments section is removed; SSIM columns in every results table follow in the camera-ready PDF. Every addition to the revised manuscript is typeset in blue, and each response below states where it was added; tables of this response are numbered R1, R2, …, plain “Table n” refers to the manuscript (the list at the end collects the changes and what is still pending).

Finally, a remark on how PACE should be positioned. Faster 3DGS trainers arrive every few months: Taming 3DGS, DashGaussian, FastGS, SAD-GS and TurboGS each superseded their predecessors as the fastest recipe, and each will be superseded in turn. PACE is not part of that sequence. It is a controller that sits on top of whichever trainer is current and decides, from the training signal, when to densify, prune, and reset. Every trainer above exposes those decisions as a hand-tuned schedule, and the experiments in this response show that a policy trained once on 3DGS accelerates the released FastGS loop by 1.4–1.6× and the released SAD-GS loop by 1.16× without retraining. A new state-of-the-art trainer therefore does not make PACE obsolete; it becomes a new substrate on which the same policy is expected to recover the same kind of gain, and the stronger the trainer, the better the combined system. This is the contribution we ask the reviewers to weigh: not a faster recipe, but a reusable scheduling policy that complements any recipe.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.