acceptodds
Withdrawn from ICLR 2027

AllocSplat: Spatial Allocation for Budget-Adaptive Feed-Forward Gaussian Splatting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Feed-forward 3D Gaussian Splatting reconstructs renderable scenes from a few images with known camera poses. Different storage and rendering limits call for different numbers of Gaussians. The challenge is to distribute a limited number of Gaussians across the scene while preserving local visual detail. We introduce AllocSplat, which uses 3D anchor points to determine where Gaussian groups are placed and how their spatial scales are set. A joint geometry–appearance rule selects these anchors from depth-based 3D candidates, considering both scene coverage and differences in visual content. A density-adaptive local decoder generates a fixed number of Gaussians per anchor, so the anchor count sets the total Gaussian count. It adapts each group's spatial scale to the local anchor density, using distances to nearby selected anchors to limit Gaussian sizes and offsets from the anchor. When the requested budget changes, the model selects the corresponding number of anchors and recomputes local scales and information shared across anchors before decoding. The same anchor point can thus generate different Gaussian groups at different budgets without changing the network weights. Experiments on DL3DV-140 show improved quality–count trade-offs across multiple Gaussian budgets. With four input views, AllocSplat achieves 0.99 dB higher PSNR than TokenGS using 85.9% fewer Gaussians, and exceeds ZipSplat by 0.29–1.65 dB at matched primitive counts.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Feed-Forward Reconstruction, Novel View Synthesis, Spatial Anchor Allocation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B3s4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AllocSplat selects 3D anchors using explicit geometry and appearance, then adjusts local offset and scale bounds using anchor spacing. It re-decodes Gaussian groups when the budget changes. DL3DV-140 and transfer experiments evaluate quality versus primitive count, with ablations of selection, scale adaptation, and re-decoding.

Critical strengths
  1. The connection between spatial allocation and budget-dependent decoding is well motivated. The method changes not just which primitives are retained but how the retained support is represented at the requested budget.

  2. The component studies address relevant confounders. Table 3 separates retaining previously decoded groups, re-decoding at frozen scales, and updating the scales. Selection and scale-rule variants are trained separately, and matched-count comparisons with ZipSplat are provided.

  3. The evaluation takes care over common inputs, target cameras, and non-input target views. The cross-dataset results provide evidence beyond a single benchmark. The paper also reports limitations, including cases where smaller Gaussian groups are more efficient at matched counts.

Critical weaknesses
  1. Independence from backbone pre-training is unresolved. Section 4.1 and Appendix A.1 state that DA3-Giant-1.1 was trained on DL3DV but do not establish exclusion of the 140 test scenes. Excluding them from AllocSplat training alone is insufficient. This is uncertainty, not evidence of leakage, but it limits claims about unseen-scene generalization.

  2. Allocation gains need separating from system-level gains. External baselines differ in backbone and training. At 1024 anchors, the joint selector gains about 0.14 dB over spatial FPS without achieving the best LPIPS, while its low-budget advantage is larger. The conclusions should reflect this budget dependence.

Questions for authors
  1. Can existing training records establish whether DL3DV-140 was excluded? If not, which existing evaluation scenes are demonstrably unseen by the backbone? Retraining the backbone is not requested.

  2. Please summarize selection, scale adaptation, and re-decoding gains at low and default budgets using existing ablations, including the default-budget PSNR/LPIPS trade-off.

  3. Can you provide a focused matched-device latency/memory comparison with the main budget-adaptive baseline, using the same number of target renders and stating whether the full geometry backbone is included?

P.S. I encourage the authors to address these questions carefully. I would be willing to raise my score if the responses adequately resolve my concerns.

Reviewer_sw6AOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The AllocSplat paper proposes a new way of doing feedforward 3D GS few-view novel view synthesis. It uses a DepthAnything V3 backbone to determine pixel depths and extract feature representations, then it projects these pixels to 3D as spatial anchors, and then a learned transformer aggregates the feature representations and decodes each spatial anchor into a fixed-size group of 3D Gaussian primitives to complete the feedforward 3D GS scene. In this design, the authors identify two important design questions: how to allocate the spatial anchors given a fixed total primitive budget, and how to constrain the Gaussian center offsets when decoding a group of Gaussian primitives from each spatial anchor. To answer those questions, they propose a geometry-appearance anchor allocation scheme and an adaptive spatial scale constraint based on nearest-neighbor anchor spacing to constrain the decoded Gaussian centers. Ablations show the effectiveness of the proposed anchor allocation and adaptive scale techniques over other alternatives, as well as the whole framework having superior generalization to other datasets compared to concurrent approaches.

Critical strengths

1. Strong empirical results. Table 1 and Table 5 show AllocSplat having better PSNR on held-out views compared to concurrent approaches while simultaneously using fewer Gaussian primitives.

2. The two core AllocSplat techniques, namely geometry-appearance anchor allocation and the density-adaptive decoder both have intuitive explanations behind their mechanism, and the paper does a good job of explaining them.

Critical weaknesses

1. It's hard to separate the generalizability of the proposed framework from the using the large DA3-Giant (Lin et al., 2026) model as the depth and representation backbone. All ablations hold the choice of the backbone fixed as DA3-Giant. While I think using a good pre-trained visual geometry model is generally a good idea, the paper leaves me wondering if the joint geometry-appearance allocation and the density-adaptive local decoder, which the paper claims as its main contributions, remain effective when different visual-geometry backbones are used. E.g., other popular models such as DA3 of different sizes, DA2, or VGGT-Omega. The use of a large backbone model for depths and representations also confounds the interpretation of the positive results in Section 4.5 Cross-dataset generalization. Is AllocSplat better because of its smart modeling decisions, or simply because it uses a powerful feature extractor that is trained on Internet-scale data, which may very well contain the training and held-out datasets?

Minor issues:

i. I find the ablations on "Budget-specific re-decoding" in L405-410 and Table 3 to be not that helpful. If I were to construct the scene with a different "M" setting (number of anchors) from the start, I wouldn't even have anything to cache, thus the scales and decoding are automatically computed under the new setting. It is not surprising to me that using the scales computed from a stale "M" number or not re-decoding when changing "M" are suboptimal. It's asking the model to do something it was not trained to do in the first place. With that said, I still believe the experiments are informative about whether we can reuse representations and scales computed for a bigger "M", but I just do not believe they are too surprising. The introduction lists this result as a main contribution ("We show that decoding Gaussian groups for the requested allocation improves reconstruction over retaining groups from a larger allocation, with further gains from updating local scales."), but this statement seems unsurprising to me, as intuition already suggests retaining groups and features from a different, larger allocation budget is probably suboptimal.

Questions for authors

L431 says "Smaller groups use primitives more efficiently, while larger groups give a higher maximum capacity." Is "larger groups give a higher maximum capacity" referring to when the compute and memory budget are held constant, as smaller groups lead to more groups that participate in global attention, which is expensive? Can the authors clarify why larger groups give a higher maximum capacity?

In eq. 8, the authors define the local spatial scale as the mean L2 distance to an anchor's neighbors. Aside from the two other rules tested in Table 4, I am curious about a per-anchor learned scale parameter can also be effective? Additionally, is a hard constraint (eq. 9) on the center offset really beneficial? Perhaps a soft constraint can also be effective?

The idea of spatially anchoring Gaussians here feels related to the concept of anchoring done in Scaffold GS and its related works, like Octree-GS. Beyond the brief discussions included in the paper, can the authors elaborate on how the idea of "anchoring" differs in this specific setting?

Reviewer_WJEnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AllocSplat, a budget-adaptive feed-forward 3D Gaussian Splatting (3DGS) framework for sparse-view reconstruction. The framework dynamically adjusts the number of generated Gaussian primitives according to a specified anchor budget at inference time, without retraining or modifying the network weights. Specifically, AllocSplat first leverages a pretrained visual geometry model to extract features and construct 3D anchor candidates, which are subsequently selected through a joint geometry–appearance allocation strategy. It then determines the local spatial scale of each anchor based on distances to its neighboring anchors, using this scale to constrain local Gaussian decoding. By adjusting the number of selected anchors, the framework controls the total Gaussian count and enables budget-adaptive reconstruction.

Critical strengths
  1. The paper constructs local Gaussian groups using explicit 3D anchors and regulates their decoding ranges according to the spatial distribution of these anchors. Compared with directly pruning Gaussian primitives or predicting Gaussian positions without explicit spatial constraints, this structured representation provides better geometric interpretability.

  2. The proposed method supports flexible output budgets without retraining the model. Moreover, the deterministic greedy allocation strategy guarantees that the anchor set at a smaller budget is a strict prefix of that at a larger budget, enabling the reuse of cached candidate rankings for inference at multiple budgets.

  3. AllocSplat achieves competitive PSNR and SSIM on DL3DV-140 while using substantially fewer Gaussian primitives than existing methods, demonstrating a favorable trade-off between reconstruction quality and representation size.

Critical weaknesses
  1. AllocSplat relies heavily on the depth predictions, confidence estimates, and visual features provided by the pretrained DA3 backbone. These outputs are not only used to initialize the reconstruction network but also directly determine the candidate anchor positions and subsequent spatial allocation. Consequently, errors or inconsistencies in the backbone predictions may propagate through the reconstruction pipeline, raising concerns about the method's robustness and dependence on the pretrained geometry model.

  2. The paper lacks direct comparisons with closely related methods, such as EcoSplat, AdaptiveSplat, and AnchorSplat. Without evaluating these approaches under consistent conditions, including the same input views, test scenes, and comparable Gaussian budgets, it is difficult to establish the advantages of the proposed budget-control mechanism over existing methods.

  3. As shown in Table 9(a), the optimal Gaussian group size varies with the primitive budget, whereas the proposed model adopts a fixed group size of K=36. Consequently, the framework can only generate Gaussian counts that are integer multiples of K, rather than supporting arbitrary target budgets. Furthermore, the maximum representation capacity is constrained by the fixed candidate anchor pool, introducing an inherent trade-off between primitive efficiency at smaller budgets and the maximum achievable capacity at larger budgets.

Questions for authors
  1. How sensitive is the proposed method to the choice of backbone? To what extent do the performance improvements originate from the pretrained geometry backbone rather than the proposed spatial allocation mechanism? Could the authors provide controlled experiments to disentangle these contributions?

  2. Could the authors include direct comparisons with additional budget-controllable Gaussian reconstruction methods? Moreover, Appendix C.6 indicates that the inference pipeline incurs substantial computational overhead. Could the authors report end-to-end inference latency and peak GPU memory usage for AllocSplat and the major baselines under identical hardware conditions?

  3. What motivates the choice of K=36 as the default Gaussian group size? Given the observations in Table 9(a), how could the framework jointly optimize the number of anchors and the number of Gaussians per anchor under a specified total Gaussian budget, rather than relying on a fixed group size?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.