acceptodds
Under review as a conference paper at ICLR 2027

SplatFoundry: Voxel-Adaptive Gaussian Decoding with Learned Adam Refinement

Abstract

Feed-forward 3D Gaussian reconstruction replaces costly per-scene fitting with a single network evaluation. However, existing primitive allocations impose an unfavorable choice: pixel-aligned predictors replicate Gaussians across views, while a fixed latent codebook cannot grow with scene extent and complexity. We introduce SplatFoundry, a two-stage framework that combines adaptive 3D allocation with learned Adam refinement. First, we aggregate points and features from a frozen geometric foundation model into depth-adaptive voxel tokens. A Transformer decoder applies self-attention to the concatenation of image-patch and voxel tokens, then decodes a constant number of Gaussians from each occupied voxel token. The resulting primitive count adapts to the occupied 3D space rather than scaling with the number of patch tokens or being capped by a global budget. Second, a state estimator predicts per-Gaussian, per-attribute Adam moments and learning rates from voxel tokens. The differentiable post-optimization fits the Gaussians to the context views, while the refined Gaussians are supervised end-to-end by the standard novel-view objective. Unlike cold-start Adam, this learned initialization produces substantial improvement after only a few refinement epochs. To reduce training memory, we introduce the Accumulated Adam Jacobian (AAJ), whose backward-pass memory is independent of the number of Adam refinement iterations. Across DL3DV and three out-of-distribution datasets, our method improves PSNR by dB and uses fewer Gaussians on average than the strongest prior method in each setting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Feed-forward 3DGS, Novel-view synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yFFLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies static multi-view Gaussian reconstruction. It uses a frozen geometry backbone to lift image observations into depth-adaptive voxel tokens, jointly processes image and voxel tokens, and decodes 32 Gaussians per occupied voxel. It further predicts per-Gaussian Adam states and learning rates for context-view test-time refinement. An Accumulated Adam Jacobian (AAJ) is proposed to reduce training memory when differentiating through this refinement. The method is evaluated on DL3DV and three out-of-distribution datasets, primarily with calibrated cameras.

Critical strengths
  1. Adaptive allocation gives a useful quality–compactness trade-off. Rather than fixing the number of output tokens or producing Gaussians at every input location, the method allocates them at occupied 3D voxels. On DL3DV with 2/4/6 input views, its feed-forward results reach 22.38/25.84/27.10 dB with 0.12/0.17/0.20M Gaussians (Table 1). The controlled allocation ablations further support this design choice.

  2. The refinement mechanism is effective and carefully implemented. Predicting the initial Adam state improves over cold-start Adam on the same representation (Fig. 7). AAJ also reduces measured peak memory for the outer backward pass from 29.44 to 8.75 GiB in the reported setting (supplementary Table 2). These are substantive contributions beyond the initial Gaussian prediction.

Critical weaknesses
  1. The feed-forward result is weaker in the many-view setting. On DL3DV with 32/64 input views, the E=0 results are 24.94/25.44 dB, compared with MVP’s 25.73/27.44 dB. At 64 views, context processing also takes 6.4 s versus 0.4 s. The strongest many-view results therefore depend on refinement. Could finer voxels or a larger voxel-token budget improve the single-pass prediction, and what limits its current quality?

  2. The independent contribution of learned Adam needs stronger controls. Predicting optimizer state from voxel features is a coherent design, and Fig. 7 shows gains over cold-start Adam. However, nonzero second-moment initialization and online Jacobian accumulation have precedents, while Fig. 6 shows that learned Adam benefits other allocation schemes as well. Comparisons with fixed or data-driven nonzero v0v_0 and carefully tuned attribute-wise learning rates would better isolate the value of predicting the full state and its interaction with voxel tokens.

  3. The choice of gradient-based refinement needs clarification. The E>0 results require explicit rendering gradients and per-scene Adam steps. ReSplat also refines iteratively, but learns Gaussian updates from rendering residuals without test-time gradient computation. Why is predicting Adam’s initial state preferable to a voxel-conditioned network that processes rendering errors or gradients and predicts updates directly? A comparison using the same initializer and a matched wall-clock budget would clarify this choice. E>0 results should be distinguished clearly from single-pass feed-forward results.

  4. Positioning against closely related methods could be stronger. Please discuss how the voxel decoder and refinement differ from AnchorSplat, and clarify the comparison with TokenGS’s test-time token optimization, since the main table reports TokenGS’s feed-forward result. A method-level discussion would already help where a controlled numerical comparison is impractical.

Questions for authors

See weakness.

3 comments

W1. Feed-forward results without refinement are weaker than MVP's in the dense-view setting

We agree that feed-forward quality is not our main strength (higher PSNR than MVP's at 16 views, but lower at 32/64 views). The main value of our work is its compact Gaussian representation and sublinear growth in Gaussian counts (on DL3DV, MVP uses 7x and 15x more Gaussians than ours at 16 and 64 views, respectively). We address this quality gap with learned post optimization, which improves PSNR efficiently and outperforms the baselines within a few refinement epochs. We have tried increasing the voxel budget (Appendix Table 4), but the gains are marginal. We suspect that transferring information from patch tokens to voxel tokens introduces an additional challenge that direct patch-based Gaussian prediction avoids. We see this as a direction for future work. MVP's efficient context runtime benefits from its attention design, which we view as a complementary direction for future integration.

W2. The independent contribution of learned Adam needs stronger controls

We added the suggested ablations: data-driven and globally tuned positive v0v_0, both with tuned attribute-wise learning rates and identical initial Gaussians. Both use LR multipliers (3, 0.3, 3, 0.3, 1) for position/scale/rotation/opacity/color, relative to the default 3DGS rates. Our full-state prediction exceeds the stronger of these controls by 0.37/0.63/0.86/0.92 dB at E=1/2/4/8.

Optimizer-initialization ablations. DL3DV, 32 views, all 140 scenes. Entries are PSNR (dB). Best PSNR is bold.

InitializationE=1E=2E=4E=8
Cold-start Adam (Fig. 7a)23.5423.9824.4424.93
Empirical prior (context gradients)23.6223.9824.4324.97
Global prior (shared per attribute group)23.8124.2424.6024.94
Ours: predicted v0,ηv_0,\eta (Fig. 7a)24.2024.8425.3925.83
Ours: predicted m0,v0,ηm_0,v_0,\eta (Fig. 7a)24.1824.8725.4625.89

We discuss the two related works as follows:

  • Forward and Reverse Gradient-Based Hyperparameter Optimization (ICML 2017) establishes forward-mode Jacobian accumulation. Our AAJ's specific contribution is the explicit Adam-state derivative formulation and its efficient application to 3DGS reconstruction. We detach rendering gradients as a first-order approximation, as in first-order MAML in Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (ICML 2017). Under this approximation, AAJ derives compact endpoint derivatives with respect to the initial Gaussians, moments and learning rates, using O(G)O(G) derivative storage for GG scalar Gaussian parameters, independent of refinement length.

  • Revisiting the Initial Steps in Adaptive Gradient Descent Optimization (CPAL 2025) initializes v0v_0 using measured squared-gradient statistics or random positive values. The suggested ablations above show smaller improvements over cold-start than our full-state prediction, supporting our design.

We thank the reviewer for pointing out these relevant prior works. We have reframed AAJ's contribution in the revision by citing these foundations and emphasizing our Adam-state formulation and 3DGS application.

W3. Predicting Adam's initial state versus predicting updates directly

We trained an MLP Gaussian refiner using the same frozen initializer. Rendering residuals are in image space, while the refiner operates on voxel-grouped Gaussians. We therefore backpropagate the rendering loss and feed the resulting Gaussian-attribute gradients to the MLP instead of image residuals, unlike the gradient-free updates in ReSplat: Learning Recurrent Gaussian Splatting (ECCV 2026). Both our learned Adam and the MLP refiner are supervised at E=1 on the training scenes. When evaluated on test scenes, they achieve similar PSNR at E=1. However, our method outperforms the MLP refiner by 0.53 dB at E=8, suggesting better refinement-step generalization than the tested MLP refiner. Our learned Adam also requires less refinement training time and substantially less peak memory. Due to time constraints, we have not tested a more capable attention-based refiner, as used in the concurrent ReSplat. We do not expect attention alone to resolve this refinement-step generalization issue. To some extent, rendering gradients can also provide non-local information, since each Gaussian's gradient aggregates contributions from multiple pixels and reflects interactions with other Gaussians during rendering.

PSNR comparison. Entries are PSNR (dB). The MLP is selected by its E=8 PSNR.

MethodE=1E=2E=4E=8
Learned Adam24.1824.8725.4625.89
MLP24.1624.6325.0625.36

Refinement cost breakdown (E=1). Measurements cover only the post-refinement stage. Forward includes rendering-gradient computation, and backward is the outer training backward pass.

MethodTraining forward (ms) ↓Training backward (ms) ↓Inference refinement (ms) ↓Training peak (GiB) ↓
Learned Adam (AAJ)628.116.05533.825.080
MLP646.67160.21566.5018.788

W4a. How the voxel decoder and refinement differ from AnchorSplat

AnchorSplat: Feed-Forward 3D Gaussian Splatting with 3D Geometric Priors (CVPR 2026) uses uniform-scale voxelization to determine the anchor count and farthest point sampling to construct its decoder input tokens. We use depth-adaptive voxelization tied to the projected image footprint, followed by joint patch–voxel attention for voxel-grouped Gaussian prediction. AnchorSplat uses a rendering-error-conditioned neural refiner. We predict Adam moments and learning rates for explicit Gaussian optimization.

W4b. TokenGS's test-time token optimization

We now report TokenGS with test-time token optimization in Table 1, following TokenGS's official setup. It improves PSNR by 1.16/1.76 dB at 4/6 views. At 2 views, PSNR decreases by 0.03 dB, although SSIM and LPIPS improve. Our feed-forward (E=0) results already outperform their token-tuned results. Token tuning also takes more than 10 seconds in our tests, while our learned Adam completes 16 refinement epochs in less than half a second and achieves substantially higher final quality.

TokenGS and SplatFoundry on DL3DV. 256×448 evaluation. Our results are from Table 1. E denotes our refinement epochs. Best values within each view count are bold.

ViewsMethodPSNR ↑SSIM ↑LPIPS ↓Total context time (s) ↓
2TokenGS (feed-forward)20.070.6360.4200.14
2TokenGS (token tuning)20.030.6420.39612.44
2Ours (E=0)22.380.7250.2580.13
2Ours (E=1)22.690.7360.2510.15
2Ours (E=16)23.060.7500.2320.34
4TokenGS (feed-forward)23.110.7460.3190.17
4TokenGS (token tuning)24.270.7880.28015.80
4Ours (E=0)25.840.8350.1650.15
4Ours (E=1)26.500.8510.1550.17
4Ours (E=16)27.560.8760.1270.34
6TokenGS (feed-forward)24.090.7760.2970.22
6TokenGS (token tuning)25.850.8310.25019.93
6Ours (E=0)27.100.8640.1400.16
6Ours (E=1)28.040.8840.1270.25
6Ours (E=16)29.680.9130.0970.48

The authors clarified how their anchors differ from AnchorSplat’s, but I still have a novelty concern about the decoder. Both methods place Gaussians by predicting offsets from a 3D anchor. The way SplatFoundry builds each voxel feature is also close to VolSplat, which pools image features from different views in the same voxel.

The most original part is learning the initial Adam state. However, ReSplat uses a network to predict each update, while SplatFoundry runs Adam to refine the Gaussians. The single-pass feedforward result still falls short at 64 views on DL3DV: 25.44 dB versus 26.44 dB for ReSplat with similar Gaussian budgets, and lower results at 32/16 views but with more Gaussians. The MVP results are also higher at 27.44 dB, though it uses more Gaussians.

Reviewer_pN7iOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SplatFoundry, which allocates Gaussians based on the observed 3D space and refines them using a learned Adam initialization. Depth and features from a pretrained geometry model are used to construct voxels whose sizes vary with depth. Image patch tokens and voxel tokens are processed jointly to predict a fixed number of Gaussians per voxel. The model also predicts initial Adam moments and learning rates for each Gaussian attribute, which are used to refine the reconstruction against the input views. The proposed Accumulated Adam Jacobian (AAJ) reduces the memory required to train through this refinement process. Experiments on DL3DV and three additional datasets report reconstruction quality, Gaussian counts, and processing time.

Critical strengths
  • Voxel-based allocation consolidates redundant observations while allowing representation capacity to grow as new regions are observed. Figure 6 compares voxel, pixel, and codebook allocation within the same implementation, with learned Adam applied to all three. This provides useful evidence of their quality and runtime trade-offs. Evaluations with 2 to 64 input views and on three additional datasets further support the approach.

  • The learned Adam component is well studied. Figure 7 examines the contributions of the predicted moments and learning rates, and compares per-Gaussian predictions with shared optimizer states. AAJ is supported by both a derivation and empirical measurements. In the setting reported in Table 2, it reduces peak memory for the optimization stage from 29.44 GiB to 8.75 GiB.

Critical weaknesses
  • The experiments do not isolate the contribution of depth-dependent voxel sizing. As discussed in the paper, VolSplat also aggregates information from multiple views into voxels to predict Gaussians. SplatFoundry adjusts voxel sizes with depth to maintain a target image-space footprint, but the benefit of this adjustment remains unclear. Figure 6 compares different allocation strategies, while Table 3 varies the target footprint within the adaptive scheme. A comparison with fixed-size voxels using the same backbone and decoder would help distinguish the benefit of adaptive sizing from that of voxel-based allocation itself.

  • The setup of the learned Adam and cold-start Adam comparison in Figure 7 is unclear. Do they start from the same Gaussian predictions, or is the Gaussian prediction model trained separately for each setting? Since the Gaussian initialization and Adam state are learned jointly, this distinction matters when interpreting the gains. The current description leaves open whether they reflect the predicted Adam state alone or also differences in the initial Gaussians

Questions for authors
  1. Have you compared fixed-size and adaptive voxels using the same backbone and decoder? If results are already available, please report reconstruction quality, Gaussian counts, and processing time.

  2. Do learned Adam and cold-start Adam in Figure 7 use the same initial Gaussians? If the models were trained separately, which training conditions were held fixed and which differed?

1 comment

Q1. Comparison with uniformly sized voxels

Depth-adaptive sizing gives higher PSNR than two uniform-grid controls in our same-backbone/decoder ablation on DL3DV scenes. To compare against uniform voxel sizing at similar Gaussian counts, the fixed-scale strategy searches for an absolute voxel size, while the scene-scaled strategy searches for a multiplier of the scene's median depth to determine voxel size. Both uniform-grid variants were trained with settings aligned with those of the depth-adaptive variant. The results show the effectiveness of our depth-adaptive voxelization over uniform voxelization.

Comparison with uniform voxels. Entries report PSNR (dB), Gaussians (M), and context-processing time (s) without post optimization.

ViewsAllocationPSNR (dB)#GS (M)Context time (s)
16Depth-adaptive24.321.1741.08
16Uniform, fixed scale23.121.0800.99
16Uniform, scene-scaled22.201.0870.97
32Depth-adaptive25.301.6672.40
32Uniform, fixed scale23.781.5872.41
32Uniform, scene-scaled22.781.6182.40
64Depth-adaptive25.822.1996.49
64Uniform, fixed scale24.042.2466.67
64Uniform, scene-scaled22.982.3356.64

Q2. Do learned Adam and cold-start Adam use the same initial Gaussians?

Yes. All optimizer variants in both panels (a) and (b) of Figure 7 start from identical initial Gaussians within each setting. The revised caption and ablation text state this explicitly.

Reviewer_KkWXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPLATFOUNDRY, a two-stage framework that combines adaptive 3D allocation with learned Adam refinement. SPLATFOUNDRY combines adaptive Gaussian allocation and learning based Adam optimization. The voxel size is adjusted based on depth and projection scale, and subsequent scene level optimization is accelerated by learning Adam's initial first-order and second-order moments and learning rate. A good balance between reconstruction quality and Gaussian quantity is achieved in 3D Gaussian reconstruction.

Critical strengths
  1. The paper accelerates the convergence of scene level optimization by learning Adam's initial state and learning rate, which has a certain degree of innovation.
  2. The writing ideas of the paper are relatively clear and the logic is smooth.
Critical weaknesses

This article proposes a deep adaptive voxel allocation strategy that dynamically adjusts voxel resolution based on depth and projection size. However, depth cannot fully reflect the local geometric complexity, and fine structures in the distance may still be lost due to coarser voxel divisions. Although the author conducted ablation experiments in Figure 6 and Table 3, there is a lack of direct comparison with fixed scale voxels and other adaptive strategies under the same Gaussian budget. Considering that existing adaptive methods have also achieved competitive results, the independent advantages of this strategy itself are still not clear enough.

Questions for authors
  1. Can the author further analyze the reconstruction performance of this method under long-distance fine structures, especially the impact of deep adaptive voxel partitioning on reconstruction quality?
  2. Can we compare the proposed allocation strategy with fixed scale voxels and adaptive allocation methods based on local complexity or reconstruction uncertainty under the same backbone and similar Gaussian numbers, in order to further demonstrate the independent benefits of the proposed allocation strategy?
1 comment

W1. Depth cannot fully reflect the local geometric complexity

We agree that depth alone does not fully capture local geometric complexity. Our design is grounded in the sampling principle of commonly used 2D patchification: a nearby thin structure spans more patches, while the same structure farther away spans fewer patches and can lose detail. Our strategy of tying voxel size to its projected image footprint (Sec. 3.2, Appendix B.4) extends this established principle to 3D but inherits the same sampling limitation. Content-dependent voxel sizing is a promising extension, analogous to Accelerating Vision Transformers with Adaptive Patch Sizes (ICLR 2026), which assigns smaller patches to detailed regions and larger patches to homogeneous regions. We address the detailed questions below.

Q1. Long-distance fine structures

We analyze how geometric complexity and depth-dependent voxel scale jointly relate to rendering error on DL3DV scenes (Appendix C.1, Fig. 9). We use within-voxel normal variation as a proxy for geometric complexity and group voxels by size, which reflects source-view distance under our allocation rule. Within each voxel-size group, greater normal variation is generally associated with higher mass-weighted novel-view MSE. At comparable normal variation, larger-voxel groups generally have lower error. One possible explanation is that these distant voxels are also farther from the novel-view camera, so the corresponding image regions contain less resolvable detail and are easier to predict.

Q2a. Comparison with uniformly sized voxels

Depth-adaptive sizing gives higher PSNR than two uniform-grid controls in our same-backbone/decoder ablation on DL3DV scenes. To compare against uniform voxel sizing at similar Gaussian counts, the fixed-scale strategy searches for an absolute voxel size, while the scene-scaled strategy searches for a multiplier of the scene's median depth to determine voxel size. Both uniform-grid variants were trained with settings aligned with those of the depth-adaptive variant. The results show the effectiveness of our depth-adaptive voxelization over uniform voxelization.

Comparison with uniform voxels. Entries report PSNR (dB), Gaussians (M), and context-processing time (s) without post optimization.

ViewsAllocationPSNR (dB)#GS (M)Context time (s)
16Depth-adaptive24.321.1741.08
16Uniform, fixed scale23.121.0800.99
16Uniform, scene-scaled22.201.0870.97
32Depth-adaptive25.301.6672.40
32Uniform, fixed scale23.781.5872.41
32Uniform, scene-scaled22.781.6182.40
64Depth-adaptive25.822.1996.49
64Uniform, fixed scale24.042.2466.67
64Uniform, scene-scaled22.982.3356.64

Q2b. Adaptive allocation based on local complexity

We have made some initial attempts to incorporate local complexity into our adaptive voxelization strategy. We score voxels with local entropy, Sobel energy or depth-derived normal variation, then split high-scoring cells (top 10%) and merge low-scoring cells to maintain similar Gaussian counts. The variants use training settings aligned with our main depth-adaptive model. Our depth-adaptive design achieves the best PSNR among these variants at similar Gaussian counts.

NVS with local-complexity allocation. Entries report PSNR (dB) / Gaussians (M) without post optimization.

Allocation16 views32 views64 views
Depth-adaptive (reference)24.32 / 1.17425.30 / 1.66725.82 / 2.199
Local entropy24.09 / 1.14525.02 / 1.64125.49 / 2.198
Sobel energy24.10 / 1.14725.03 / 1.64325.50 / 2.204
Normal variation23.66 / 1.13224.52 / 1.63124.94 / 2.187

In 2D, APT (Accelerating Vision Transformers with Adaptive Patch Sizes, ICLR 2026) improves efficiency while maintaining the performance of conventional fixed-size patchification. Our current design targets a fixed image footprint (Sec. 3.2, Appendix B.4), following the same principle as conventional 2D patchification. Future work may explore content-aware adaptive voxelization for greater efficiency while preserving rendering quality.

Comments on the paper

1 comment

We thank the reviewers for their time and constructive feedback. We appreciate the recognition of SplatFoundry's compact Gaussian estimation and effective learned-state refinement.

SplatFoundry contributes (1) depth-adaptive voxelization that ties voxel size to its projected image footprint, extending the widely used 2D patchification principle to 3D, (2) learned per-Gaussian Adam states for rapid refinement, (3) an Adam-specific Jacobian formulation (AAJ) for memory-efficient training, and (4) a resolution-decoupling strategy that caps Transformer resolution while using high-resolution refinement feedback. After 16 epochs of learned refinement, it improves PSNR over the strongest non-concurrent baseline in each setting by 3.0–3.8 dB on sparse-view DL3DV, 0.9–2.0 dB on full-sequence DL3DV, and 1.2–2.8 dB across OOD datasets, while using approximately 23–93% fewer Gaussians (Table 1).

We have strengthened the evaluation in response to the reviews (the results are also included directly in the relevant reviewer responses for ease of discussion):

  • Added uniform-grid and local-complexity allocation controls (Appendices C.3–C.4). Depth-adaptive allocation achieves the best PSNR among the tested variants at similar Gaussian counts.

  • Added a joint voxel-scale/normal-variation analysis (Appendix C.1, Fig. 9) to examine how distance-related voxel scale and geometric complexity relate to rendering error.

  • Added empirical and globally tuned nonzero Adam second-moment priors with tuned attribute-wise learning rates (Fig. 7a). Full-state prediction outperforms the stronger control by 0.37–0.92 dB across the evaluated refinement horizons.

  • Added an MLP Gaussian-refiner comparison and refinement-cost breakdown. Learned Adam achieves 0.53 dB higher PSNR at eight epochs. In the separate cost benchmark, learned Adam with AAJ has lower peak training memory and backward cost than the MLP.

  • Added TokenGS with test-time token optimization to Table 1. Our feed-forward results already exceed its token-tuned quality.

We have also clarified the shared initialization in Fig. 7 and AAJ's connection to prior work. Below, we address each reviewer's comments with the corresponding evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.