acceptodds
Withdrawn from ICLR 2027

ARGeoSplat: Autoregressive Geometry Generation and Refinement for Sparse-View 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A few posed photographs leave much of a scene unseen. Multiview generation can fill in missing views, but plausible RGB images alone do not specify a renderable 3D scene. We extend a next-scale autoregressive multiview generator to produce RGB images and camera-frame pointmaps together. Separate bitwise tokenizers and prediction heads represent appearance and geometry, while a shared transformer conditions each scale on both modalities at earlier scales. We train the geometry predictions with decoded XYZ supervision through a straight-through estimator and a ground-truth-guided cross-view consistency loss. The generated pointmaps then provide proposals for a feed-forward Gaussian predictor. A multiview cost volume searches near each proposed depth to correct local errors, and training on free-running generator outputs exposes the predictor to the errors it will see at inference. With two or three input views, our Gaussian renders exceed EscherNet in PSNR by 1.47–4.02 dB on GSO and OO3D and reduce cross-view photometric error by 0.16–0.24. In ablations, geometry supervision lowers XYZ L2 from 0.0194 to 0.0144, while local depth refinement raises render PSNR from 21.11 to 23.12.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
autoregressive geometry generation, next-scale autoregression, feedforward 3D gaussian splatting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c6ccOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ARGeoSplat, which extends next-scale autoregressive multiview generation to jointly predict RGB and pointmaps. A downstream Gaussian predictor takes in the predicted pointmaps and refines their depths to enable novel view synthesis. Experiments on GSO and OmniObject3D show that it improves NVS performance and cross-view photometric consistency given a low number of input views.

Critical strengths
  • The paper brings RGB + pointmap prediction into a joint processing in autoregressive view synthesis, which is also different from diffusion-based joint appearance-geometry generation.
  • Pointmaps can be generated for unobserved target areas.
Critical weaknesses
  • The paper doesn't provide comparisons with its closest prior work, such as NAMVIS or Bolt3D, where joint appearance-geometry generation followed by Gaussian prediction is performed. Also, other multi-view 3DGS reconstruction methods are only discussed without evaluation/comparison.
  • While the method can predict explicit pointmaps and receive explicit and direct geometric supervision, its geometric accuracy is not evaluated in the main experiment. It remains unclear whether those explicit pointmaps can be used for broad downstream tasks like other reconstruction models such as VGGT, or just included to serve as a starting point for further refinement, leading to marginal performance improvements as observed in Tab.1 and 2 but with additional training, parameter, and methodological complexity overhead.
  • Currently, the geometry head is trained through Lgen\mathcal L_{\mathrm{gen}}, while depth refinement and Gaussian rendering losses train the downstream reconstruction stage. It remains unclear whether the whole computational graph is end-to-end differentiable or not. If not, it could create a potential mismatch between (1) generated pointmaps and (2) pointmaps that support the best 3DGS and always require a refinement process to compensate for this mismatch.
  • The cost volume searches within 10% of the generated depth, excluding larger errors and leaving unsupported pixels unchanged. Currently, there is no analysis or discussion regarding how frequently those extreme cases remain unresolved with their errors preserved. Also, from Tab.4, the benefits after CV seem marginal. It remains unclear how important or sensitive this hyperparameter is and whether it can be generalized to different use cases/datasets.
Questions for authors

See weakness

Reviewer_xEsiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses object-level 3D reconstruction from a few posed images. It builds on NAMVIS, an autoregressive multi-view generator, to jointly generate novel views and their geometry in the form of pointmaps. The generated geometry is then refined with multi-view matching and converted into 3D Gaussians by a feed-forward network. Experiments on GSO and OmniObject3D compare the method with generative and feed-forward baselines.

Critical strengths

1. A reasonable and coherent design

  • Generating explicit geometry together with novel views is a reasonable idea. The method also accounts for errors in the generated geometry, and the overall design is coherent.

2. Fast inference and higher PSNR than EscherNet

  • The full pipeline takes 0.8 s per object, versus 2.0–3.2 s for the generative baselines (L401–403). With two or three input views, it achieves higher PSNR than EscherNet (Tables 1–2), and the ablations support the geometry supervision and depth refinement (Tables 3–4).
Critical weaknesses

1. The experimental comparisons could be more complete, and the consistency claim needs clarification

  • Baseline coverage is limited:
    • With two or three input views, EscherNet is the only baseline, and no explicit 3D reconstruction method is included.
    • An RGB-only control is missing, e.g., generating images with NAMVIS, estimating geometry with an off-the-shelf method, and then applying the same Gaussian predictor.
  • The consistency comparison is between different types of outputs. The abstract reports that cross-view photometric error is reduced by 0.16–0.24, but this compares 3DGS renders with EscherNet's generated images. Renders from a single 3D representation are inherently more consistent: in the single-view setting, for example, Splatter Image has the lowest consistency error, even though its PSNR and LPIPS are not among the best (Tables 1–2). When generated images are compared with generated images (Ours (generated) vs. EscherNet, Tables 1–2), the gap is much smaller.
  • As a result, it is difficult to tell how much of the improvement comes from the proposed design.

2. The presentation and some missing key details make the results harder to assess

  • Writing and figures:
    • The abstract and introduction focus mostly on implementation details, and do not explain why next-scale autoregression is chosen over diffusion (e.g., Bolt3D).
    • Some methods and datasets are not cited at their first mention, e.g., EscherNet and GSO in the introduction, and LFQ is not cited anywhere.
    • The experiments do not visualize the generated pointmaps or the effect of depth refinement.
  • Missing key details: the paper reports the optimizer, learning rates, loss weights, and inference timing setup (L270–291), but some key information is missing, for example:
    • whether the generator is initialized from pretrained NAMVIS weights;
    • the model size (including the newly added modules) and the training scale of the generator (training steps, batch size, and compute);
    • how the views are selected for evaluation.
Questions for authors
  1. RGB-only control. Could you report an RGB-only variant to validate the benefit of jointly generating geometry? For example, one could generate images with NAMVIS, estimate geometry with an off-the-shelf method, and then apply the same Gaussian predictor.
  2. Source of the consistency gains. How much of the consistency improvement comes from the explicit 3D representation itself? For example, EscherNet's generated images could also be reconstructed into 3DGS before comparison. In addition, could you report geometric metrics such as Chamfer distance?
  3. Implementation details. Is the generator initialized from pretrained NAMVIS weights? Could you report the model size (including the newly added modules), the training scale of the generator, and how the evaluation views are selected?
  4. Training on generated inputs. The abstract highlights training the Gaussian predictor on generator outputs, but this design is not ablated. Could you provide some experimental evidence for its benefit?
Reviewer_z5TzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ARGeoSplat, a novel approach for predicting 3D Gaussians from a few posed images. ARGeoSplat is built on top of NAMVIS. NAMVIS synthesizes novel views given a few posed images using next-scale prediction. ARGeoSplat adopts the same next-scale prediction strategy, but predicts pointmaps in addition to RGB images for novel views. Given the predicted novel-view RGB images and pointmaps (and target-view camera poses), a Gaussian predictor converts them into Gaussian primitives and refines their depths.

ARGeoSplat achieves overall decent results compared to baselines on GSO and OO3D in terms of novel-view synthesis and cross-view consistency. Ablation studies verify the effectiveness of the proposed component in ARGeoSplat.

Critical strengths

(1) The paper writing is good and easy to follow.

(2) The proposed method, ARGeoSplat, achieves competitive results compared with baselines. Concretely, the generated RGB for novel views achieves the best novel-view synthesis results, whereas the full Gaussian improves cross-view consistency over the "generated" with explicit 3D primitives.

(3) The ablation studies are comprehensive and verify the contribution of each component.

Critical weaknesses

(1) Insufficient comparison. The baseline method, NAMVIS, that the proposed method is built on, is not compared. "Ours (generated)" is a close version but not exactly, because it adds pointmap prediction. Also, the baselines seem a bit old (EscherNet), so can authors also add comparison with more recent methods, e.g., LagerNVS (CVPR'26)?

(2) The motivation for predicting explicit 3D Gaussians is not well justified. From Table 1 and Table 2, the best novel-view synthesis results are achieved with "Ours (generated)" (I assume this is close to the performance of the non-Gaussian baseline, NAMVIS). The full system improves over "Ours (generated)" only on cross-view consistency, at the cost of inference speed.

(3) The idea of predicting RGB and geometry simultaneously for novel views has been explored in Bolt3D, though they use continuous VAE latents for both RGB and geometry. I don't see any strong reason for why we should use next-scale prediction with discrete latents instead of denoising diffusion with continuous latents. Also, discrete latents may risk causing information loss.

Questions for authors

Minor Questions:

(1) How are target-view cameras selected during trining, and how many target views are synthesized for training and inference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.