acceptodds
Under review as a conference paper at ICLR 2027

FineSplat: Fine Patches, Local Groups, Detailed Splats

Abstract

Feed-forward 3D Gaussian Splatting (3DGS) promises instant scene reconstruction with high-quality novel-view synthesis. Yet, fine-grained geometry and appearance remain difficult to recover, motivating us to revisit how feed-forward models represent, aggregate, and refine detailed visual information. We introduce FineSplat, a feed-forward 3DGS framework built around fine-grained image patches. Since global multi-view attention becomes prohibitively expensive at such high token resolutions, we develop a dual-grouping strategy that combines two complementary forms of locality: view-wise grouping enables efficient interaction among neighboring views, while 3D spatial grouping reconnects geometrically related tokens that may never have interacted under view-wise grouping by lifting them into the 3D scene space. We further leverage reconstruction error as direct feedback, providing an explicit rendering-based signal to recover fine-grained structures and appearance, thereby overcoming the limitations of one-pass feed-forward prediction. Finally, a compact Gaussian generation scheme preserves the benefits of fine-grained features without unnecessarily increasing the number of primitives. Together, these designs enable FineSplat to recover richer geometry and appearance while remaining efficient and compact, achieving state-of-the-art results by a substantial margin, under the same or smaller Gaussian budgets than existing methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis, Feed-Forward 3D Gaussian Splatting, Pose-Free Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RWowOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Well-motivated dual-grouping design for fine-grained reconstruction: Combining view-wise grouping with 3D spatial grouping addresses a clear limitation of local multi-view attention: geometrically related tokens may be separated by view-group boundaries. FineSplat reconnects these tokens using their predicted 3D positions and incorporates rendering-error feedback to guide iterative correction. This provides a coherent mechanism for integrating fine-grained cross-view information.

  • Strong rendering quality with a compact Gaussian representation: FineSplat demonstrates a compelling balance between reconstruction quality and primitive count compared with pixel-aligned baselines. On DL3DV with 24 posed input views, it improves PSNR by 2.88 dB over 2Xplat while using approximately one-quarter as many Gaussians (Table 1). The same DL3DV-trained checkpoint also achieves strong results on RE10K and ScanNet++, providing useful evidence of cross-dataset generalization.

Critical strengths
  • Well-motivated dual-grouping design for fine-grained reconstruction: Combining view-wise grouping with 3D spatial grouping addresses a clear limitation of local multi-view attention: geometrically related tokens may be separated by view-group boundaries. FineSplat reconnects these tokens using their predicted 3D positions and incorporates rendering-error feedback to guide iterative correction. This provides a coherent mechanism for integrating fine-grained cross-view information.

  • Strong rendering quality with a compact Gaussian representation: FineSplat demonstrates a compelling balance between reconstruction quality and primitive count compared with pixel-aligned baselines. On DL3DV with 24 posed input views, it improves PSNR by 2.88 dB over 2Xplat while using approximately one-quarter as many Gaussians (Table 1). The same DL3DV-trained checkpoint also achieves strong results on RE10K and ScanNet++, providing useful evidence of cross-dataset generalization.

Critical weaknesses
  • Limited differentiation from prior work and insufficient isolation of the contributions: FineSplat builds on the MVP architecture and the two-expert formulation of 2Xplat, while its rendering-error-driven recurrent refinement is closely related to ReSplat. Although the proposed visibility-aware error aggregation and serialized refinement of fine-grained tokens introduce specific differences, their advantages are not sufficiently established through direct comparisons and controlled ablations. ReSplat is cited but not directly compared, and the existing ablations do not independently isolate the contribution of 3D spatial grouping from rendering-error feedback or additional refinement computation. In particular, Table 5 modifies the observations used for error aggregation rather than the spatial attention mechanism, while Table 8 lacks a variant without rendering-error inputs. Comparisons under matched initialization, Gaussian budget, and training conditions would clarify the contribution beyond the established components.

  • Incomplete scalability evaluation on the main benchmark and insufficient computational validation: Although DL3DV serves as the primary in-domain benchmark, its evaluation is restricted to 6, 12, and 24 input views. Many-view experiments with 32, 64, and 128 views are provided only on ScanNet++. While these results offer useful evidence of cross-dataset generalization, they leave unanswered whether the reported advantages persist beyond 24 input views on DL3DV itself. Evaluating larger input sets on the main benchmark would therefore provide more direct support for the claimed scalability. Furthermore, despite motivating the architecture through computational efficiency, the paper does not report reconstruction latency, rendering FPS, peak GPU memory, or FLOPs relative to baselines. This omission is particularly relevant given the overhead of repeated context rendering, error-feature extraction, and cross-view aggregation. Resolution scaling is also insufficiently explored: the main evaluation uses 224 × 224 images, with an additional 252 × 252 evaluation in Appendix C.4, but no quantitative evaluation at substantially higher resolutions such as 540 × 960.

Questions for authors
  • Many-view scalability on the main DL3DV benchmark: The evaluation on DL3DV is limited to 6, 12, and 24 input views, whereas experiments with 32, 64, and 128 views are provided only on ScanNet++. Could the authors extend the evaluation on DL3DV to larger input sets, such as 64 and 128 views, to establish whether the reported advantages persist on the primary in-domain benchmark?

  • Computational efficiency and high-resolution performance: Could the authors report end-to-end reconstruction latency, peak GPU memory, and FLOPs across input resolutions and view counts, with novel-view rendering FPS reported separately? A breakdown covering the DA3 geometry expert, Stage 1, and Stage 2—including repeated rendering, error-feature extraction, and cross-view aggregation—would clarify the practical computational benefits. Beyond the current 224 × 224 and 252 × 252 evaluations, could the authors also provide reconstruction-quality and efficiency comparisons at substantially higher input and rendering resolutions, such as 540 × 960?

  • Isolating the contributions of 3D grouping and rendering feedback: Could the authors compare the full model with (i) the same 3D refinement module without rendering-error inputs, retaining the image-context features, and (ii) a refinement module using the same error features and number of refinement steps but image-space or non-geometric grouping? Matching the initialization, Gaussian budget, and training protocol would help establish how much of the improvement comes specifically from the proposed 3D interactions, beyond rendering feedback and additional refinement computation.

  • Direct comparison with ReSplat: ReSplat is closely related because it also combines compact Gaussian initialization, rendering-error feedback, and recurrent 3D attention. Could the authors include a direct comparison under a common posed evaluation protocol and clarify the specific advantages of FineSplat’s visibility-aware error aggregation and serialized refinement of fine-grained tokens?

  • Gaussian-budget control and quality trade-off in Figure 1: What configurations produce the FineSplat results with 301K, 251K, and 213K Gaussians in Figure 1? Do these correspond to different opacity-pruning thresholds or other changes? Beyond the unpruned and default-pruned results already reported, could the authors provide a broader quality-versus-Gaussian-count curve, identify the settings used in the figure, and show where further compression begins to substantially degrade reconstruction quality?

  • Comprehensive Comparisons and Baseline Exclusions on RE10K: Could the authors provide a more thorough comparison on the RE10K dataset across a broader range of input view numbers to better demonstrate the framework's robustness and scalability under varying data sparseness? In the main comparison tables, a significant number of baseline methods are greyed out or omitted. Could the authors clarify the criteria for excluding these methods, and include comparisons against recent state-of-the-art feed-forward 3DGS methods under a unified and fair evaluation protocol?

Reviewer_M29eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FineSplat reconstructs a Gaussian scene from multiple images. A view-grouped pyramid makes an initial prediction from fine patch tokens. The model then places the tokens in 3D and updates them using nearby tokens and errors from rendered input views. Visibility checks select which views contribute to each error signal. Four refinement steps produce a compact Gaussian representation. The experiments cover posed and pose-free reconstruction, transfer, and longer input sequences.

Critical strengths

The results are strong. With 24 posed views on DL3DV, FineSplat reaches 27.23 dB compared with 24.35 dB for 2Xplat, using about four times fewer Gaussians.

The evaluation on RE10K and ScanNet++ is useful. The same checkpoint also works with 128 input views, beyond the number used during training.

Critical weaknesses

The paper reports Gaussian counts, but not total reconstruction time or peak GPU memory. The actual cost of the full pipeline is still unclear.

The ablations show that Stage 2 helps, but do not clearly separate the benefit of 3D grouping from the benefit of extra refinement.

The differences from ReSplat need more discussion, especially the use of rendering errors and local 3D refinement.

Questions for authors

Could you report total runtime and peak GPU memory, including pose prediction and all refinement steps?

Could you compare 3D grouping with view-wise refinement under a similar compute budget?

Could you clarify how the error gathering and 3D refinement differ from ReSplat? A comparison on one or two scenes would also help, if practical.

Reviewer_dkg3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FineSplat, which is a feed-forward 3DGS method designed to recover splats under a smaller budget. The authors propose a 2 stage approach. Firstly, two task specialists (geometry + appearance) recover the initial set of Gaussians, while also being designed to produce fine-grained features (thanks to a feature pyramid). Then a refinement step is performed. Firstly, Gaussians with features attached are spatially grouped in 3D, which allows spatial information sharing via attention. Further, the error map is computed to provide guidance for the token updates. The authors present results of testing their approach on in- and out-of-domain datasets used in training. The method is reported to outperform the benchmarked competitors.

Critical strengths

The big strength of the paper lies in the results. The quantitative tables show substantial improvements across the reported benchmarks. This holds as well on the out-of-distribution dataset. Similarly, the qualitative images show samples with visible improvements against competitors. Table 4 clearly shows the effectiveness of the iterative refinement approach.

Additionally, the idea of spatial grouping is very sound and well-motivated against view-grouping.

Critical weaknesses

I am not clear on the novelty of the paper. The authors cite 2Xplat (only on arXiv) as effectively the source of stage 1 in the pipeline. It seems that the architecture described in B.1 (DA3 + MVP) is pretty much the same as 2Xplat. This makes me believe that the results in Table 1 for 2Xplat should be similar to depth 0 in Table 4. The discrepancy is slightly worrying and should be explained. Similarly, ReSplat is mentioned as related work in the paper, stating their observations as the motivation for this work. Namely, ReSplat used the idea of iterative refinement based on the rendering error. This should be detailed more, and ReSplat should be included in the comparisons. Similarly, the paper does not highlight well that the compact representation comes from feature subsampling (not from spatial grouping), which makes the exact 4x reduction against pixel-aligned prediction (which would make it the exact same amount of Gaussians as ReSplat, which subsamples the depth map, one more reason to include in the comparison).

It's a shame not to include more challenging benchmarks with higher resolution (such as the 3DGS protocol including MipNeRF360, Tanks & Temples, and DeepBlending scenes), also with a notably higher number of Gaussians (1M+). This, in principle, should be a great example to showcase the benefits of being able to extract detail. On the note of efficiency, which is part of the paper (memory efficiency through the number of Gaussians), it would be great to compare the time cost of evaluating the benchmarked methods.

Questions for authors
  • I am not sure if posed/unposed classification applies to this work. Unposed in the literature typically refers to methods predicting the poses as part of the algorithm, as opposed to using an off-the-shelf estimator. I am fine with the comparison; I believe off-the-shelf estimator -> method is a valid setup; it should just be made clearer.
  • How does the method scale with the number of views?
  • Do similar tokens farther in space communicate (e.g. one would want all the tokens describing a wall to share information even though not necessarily close in space)?
  • Is the method at all resolution-dependent?
  • Is the method sensitive to the pose backbone (e.g. trying a couple of different ones)?
  • Could we do Table 8 without e_self and without e_agg? Would that be just the 3D grouping impact?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.