- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies static multi-view Gaussian reconstruction. It uses a frozen geometry backbone to lift image observations into depth-adaptive voxel tokens, jointly processes image and voxel tokens, and decodes 32 Gaussians per occupied voxel. It further predicts per-Gaussian Adam states and learning rates for context-view test-time refinement. An Accumulated Adam Jacobian (AAJ) is proposed to reduce training memory when differentiating through this refinement. The method is evaluated on DL3DV and three out-of-distribution datasets, primarily with calibrated cameras.
Critical strengths
-
Adaptive allocation gives a useful quality–compactness trade-off. Rather than fixing the number of output tokens or producing Gaussians at every input location, the method allocates them at occupied 3D voxels. On DL3DV with 2/4/6 input views, its feed-forward results reach 22.38/25.84/27.10 dB with 0.12/0.17/0.20M Gaussians (Table 1). The controlled allocation ablations further support this design choice.
-
The refinement mechanism is effective and carefully implemented. Predicting the initial Adam state improves over cold-start Adam on the same representation (Fig. 7). AAJ also reduces measured peak memory for the outer backward pass from 29.44 to 8.75 GiB in the reported setting (supplementary Table 2). These are substantive contributions beyond the initial Gaussian prediction.
Critical weaknesses
-
The feed-forward result is weaker in the many-view setting. On DL3DV with 32/64 input views, the E=0 results are 24.94/25.44 dB, compared with MVP’s 25.73/27.44 dB. At 64 views, context processing also takes 6.4 s versus 0.4 s. The strongest many-view results therefore depend on refinement. Could finer voxels or a larger voxel-token budget improve the single-pass prediction, and what limits its current quality?
-
The independent contribution of learned Adam needs stronger controls. Predicting optimizer state from voxel features is a coherent design, and Fig. 7 shows gains over cold-start Adam. However, nonzero second-moment initialization and online Jacobian accumulation have precedents, while Fig. 6 shows that learned Adam benefits other allocation schemes as well. Comparisons with fixed or data-driven nonzero and carefully tuned attribute-wise learning rates would better isolate the value of predicting the full state and its interaction with voxel tokens.
-
The choice of gradient-based refinement needs clarification. The E>0 results require explicit rendering gradients and per-scene Adam steps. ReSplat also refines iteratively, but learns Gaussian updates from rendering residuals without test-time gradient computation. Why is predicting Adam’s initial state preferable to a voxel-conditioned network that processes rendering errors or gradients and predicts updates directly? A comparison using the same initializer and a matched wall-clock budget would clarify this choice. E>0 results should be distinguished clearly from single-pass feed-forward results.
-
Positioning against closely related methods could be stronger. Please discuss how the voxel decoder and refinement differ from AnchorSplat, and clarify the comparison with TokenGS’s test-time token optimization, since the main table reports TokenGS’s feed-forward result. A method-level discussion would already help where a controlled numerical comparison is impractical.
Questions for authors
See weakness.
3 comments
W3. Predicting Adam's initial state versus predicting updates directly
We trained an MLP Gaussian refiner using the same frozen initializer. Rendering residuals are in image space, while the refiner operates on voxel-grouped Gaussians. We therefore backpropagate the rendering loss and feed the resulting Gaussian-attribute gradients to the MLP instead of image residuals, unlike the gradient-free updates in ReSplat: Learning Recurrent Gaussian Splatting (ECCV 2026). Both our learned Adam and the MLP refiner are supervised at E=1 on the training scenes. When evaluated on test scenes, they achieve similar PSNR at E=1. However, our method outperforms the MLP refiner by 0.53 dB at E=8, suggesting better refinement-step generalization than the tested MLP refiner. Our learned Adam also requires less refinement training time and substantially less peak memory. Due to time constraints, we have not tested a more capable attention-based refiner, as used in the concurrent ReSplat. We do not expect attention alone to resolve this refinement-step generalization issue. To some extent, rendering gradients can also provide non-local information, since each Gaussian's gradient aggregates contributions from multiple pixels and reflects interactions with other Gaussians during rendering.
PSNR comparison. Entries are PSNR (dB). The MLP is selected by its E=8 PSNR.
| Method | E=1 | E=2 | E=4 | E=8 |
|---|---|---|---|---|
| Learned Adam | 24.18 | 24.87 | 25.46 | 25.89 |
| MLP | 24.16 | 24.63 | 25.06 | 25.36 |
Refinement cost breakdown (E=1). Measurements cover only the post-refinement stage. Forward includes rendering-gradient computation, and backward is the outer training backward pass.
| Method | Training forward (ms) ↓ | Training backward (ms) ↓ | Inference refinement (ms) ↓ | Training peak (GiB) ↓ |
|---|---|---|---|---|
| Learned Adam (AAJ) | 628.11 | 6.05 | 533.82 | 5.080 |
| MLP | 646.67 | 160.21 | 566.50 | 18.788 |
W4a. How the voxel decoder and refinement differ from AnchorSplat
AnchorSplat: Feed-Forward 3D Gaussian Splatting with 3D Geometric Priors (CVPR 2026) uses uniform-scale voxelization to determine the anchor count and farthest point sampling to construct its decoder input tokens. We use depth-adaptive voxelization tied to the projected image footprint, followed by joint patch–voxel attention for voxel-grouped Gaussian prediction. AnchorSplat uses a rendering-error-conditioned neural refiner. We predict Adam moments and learning rates for explicit Gaussian optimization.
W4b. TokenGS's test-time token optimization
We now report TokenGS with test-time token optimization in Table 1, following TokenGS's official setup. It improves PSNR by 1.16/1.76 dB at 4/6 views. At 2 views, PSNR decreases by 0.03 dB, although SSIM and LPIPS improve. Our feed-forward (E=0) results already outperform their token-tuned results. Token tuning also takes more than 10 seconds in our tests, while our learned Adam completes 16 refinement epochs in less than half a second and achieves substantially higher final quality.
TokenGS and SplatFoundry on DL3DV. 256×448 evaluation. Our results are from Table 1. E denotes our refinement epochs. Best values within each view count are bold.
| Views | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Total context time (s) ↓ |
|---|---|---|---|---|---|
| 2 | TokenGS (feed-forward) | 20.07 | 0.636 | 0.420 | 0.14 |
| 2 | TokenGS (token tuning) | 20.03 | 0.642 | 0.396 | 12.44 |
| 2 | Ours (E=0) | 22.38 | 0.725 | 0.258 | 0.13 |
| 2 | Ours (E=1) | 22.69 | 0.736 | 0.251 | 0.15 |
| 2 | Ours (E=16) | 23.06 | 0.750 | 0.232 | 0.34 |
| 4 | TokenGS (feed-forward) | 23.11 | 0.746 | 0.319 | 0.17 |
| 4 | TokenGS (token tuning) | 24.27 | 0.788 | 0.280 | 15.80 |
| 4 | Ours (E=0) | 25.84 | 0.835 | 0.165 | 0.15 |
| 4 | Ours (E=1) | 26.50 | 0.851 | 0.155 | 0.17 |
| 4 | Ours (E=16) | 27.56 | 0.876 | 0.127 | 0.34 |
| 6 | TokenGS (feed-forward) | 24.09 | 0.776 | 0.297 | 0.22 |
| 6 | TokenGS (token tuning) | 25.85 | 0.831 | 0.250 | 19.93 |
| 6 | Ours (E=0) | 27.10 | 0.864 | 0.140 | 0.16 |
| 6 | Ours (E=1) | 28.04 | 0.884 | 0.127 | 0.25 |
| 6 | Ours (E=16) | 29.68 | 0.913 | 0.097 | 0.48 |
The authors clarified how their anchors differ from AnchorSplat’s, but I still have a novelty concern about the decoder. Both methods place Gaussians by predicting offsets from a 3D anchor. The way SplatFoundry builds each voxel feature is also close to VolSplat, which pools image features from different views in the same voxel.
The most original part is learning the initial Adam state. However, ReSplat uses a network to predict each update, while SplatFoundry runs Adam to refine the Gaussians. The single-pass feedforward result still falls short at 64 views on DL3DV: 25.44 dB versus 26.44 dB for ReSplat with similar Gaussian budgets, and lower results at 32/16 views but with more Gaussians. The MVP results are also higher at 27.44 dB, though it uses more Gaussians.
W1. Feed-forward results without refinement are weaker than MVP's in the dense-view setting
We agree that feed-forward quality is not our main strength (higher PSNR than MVP's at 16 views, but lower at 32/64 views). The main value of our work is its compact Gaussian representation and sublinear growth in Gaussian counts (on DL3DV, MVP uses 7x and 15x more Gaussians than ours at 16 and 64 views, respectively). We address this quality gap with learned post optimization, which improves PSNR efficiently and outperforms the baselines within a few refinement epochs. We have tried increasing the voxel budget (Appendix Table 4), but the gains are marginal. We suspect that transferring information from patch tokens to voxel tokens introduces an additional challenge that direct patch-based Gaussian prediction avoids. We see this as a direction for future work. MVP's efficient context runtime benefits from its attention design, which we view as a complementary direction for future integration.
W2. The independent contribution of learned Adam needs stronger controls
We added the suggested ablations: data-driven and globally tuned positive v0, both with tuned attribute-wise learning rates and identical initial Gaussians. Both use LR multipliers (3, 0.3, 3, 0.3, 1) for position/scale/rotation/opacity/color, relative to the default 3DGS rates. Our full-state prediction exceeds the stronger of these controls by 0.37/0.63/0.86/0.92 dB at E=1/2/4/8.
Optimizer-initialization ablations. DL3DV, 32 views, all 140 scenes. Entries are PSNR (dB). Best PSNR is bold.
We discuss the two related works as follows:
Forward and Reverse Gradient-Based Hyperparameter Optimization (ICML 2017) establishes forward-mode Jacobian accumulation. Our AAJ's specific contribution is the explicit Adam-state derivative formulation and its efficient application to 3DGS reconstruction. We detach rendering gradients as a first-order approximation, as in first-order MAML in Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (ICML 2017). Under this approximation, AAJ derives compact endpoint derivatives with respect to the initial Gaussians, moments and learning rates, using O(G) derivative storage for G scalar Gaussian parameters, independent of refinement length.
Revisiting the Initial Steps in Adaptive Gradient Descent Optimization (CPAL 2025) initializes v0 using measured squared-gradient statistics or random positive values. The suggested ablations above show smaller improvements over cold-start than our full-state prediction, supporting our design.
We thank the reviewer for pointing out these relevant prior works. We have reframed AAJ's contribution in the revision by citing these foundations and emphasizing our Adam-state formulation and 3DGS application.