acceptodds
Under review as a conference paper at ICLR 2027

R²: Residual-Guided Attention and Geometric Point Refinement for 3D Reconstruction

Abstract

Recent advances in geometric reconstruction have largely followed from feed-forward approaches that jointly estimate cameras, depth, and 3D point maps from images. Such models output these variables through a loosely parallel architecture despite their strong geometric dependencies. We propose R, an approach based on residual-guided camera attention and camera- and depth-conditioned point refinement. Starting from a pretrained VGGT representation, a frozen teacher first produces provisional camera parameters and geometry. We then construct a multi-view rigid-consistency residual and use it as a signed geometric prior for camera-query attention. The final camera and depth predictions are combined with the initial point map in a camera- and depth-conditioned refinement stage, which we refer to as geometric point refinement. Experiments show improvements in selected camera and depth metrics, while the proposed point refinement reduces median point error by 12.57% in a matched ablation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D reconstruction, hierarchical geometry estimation, residual attention, camera refinement, depth prediction, point cloud reconstruction, multi-view reconstruction, VGGT

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gdBBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents R$^2$, a VGGT-based framework for 3D reconstruction. The method derives a cross-view rigid-consistency prior from a frozen teacher to guide camera-token attention and introduces a camera- and depth-conditioned refinement module to improve the predicted 3D points. Experimental results show mixed improvements in camera pose and depth estimation over VGGT, while a controlled ablation demonstrates reduced point reconstruction errors with the proposed refinement module.

Critical strengths
  • Relevant research problem. The paper explores an important direction of incorporating explicit geometric consistency and coupling into feed-forward 3D reconstruction.
  • Efficient architectural design. The proposed modules largely preserve VGGT’s pretrained architecture and require only selective fine-tuning, offering a practical way to incorporate additional geometric constraints.
Critical weaknesses
  • The paper could benefit from further analysis of the motivation behind the proposed modifications. While the importance of explicit geometric consistency and coupling is well motivated, I would appreciate a clearer explanation of how the proposed modules (i.e., residual-guided camera attention and point refinement) specifically address the identified limitations. Given the mixed empirical improvements, a deeper analysis of the underlying failure modes and how each module helps mitigate them would make the methodological contributions more convincing.
  • The geometric prior is derived entirely from the frozen VGGT teacher’s predictions, and its additional value beyond the pretrained representation could be further clarified. While explicitly modeling geometric consistency may provide a useful inductive bias, I wonder whether similar improvements could be achieved through model adaptation alone. A controlled comparison and an analysis of sensitivity to teacher prediction errors would help better understand and justify the benefits of the proposed prior.
  • I would appreciate further clarification on several key technical details, including how cross-view correspondences are obtained and filtered, how correspondence uncertainty in Eq. (4) is computed, and how the 16-channel camera encoding in Eq. (9) is constructed. As these are central components of the proposed method, more detailed explanations would be valuable for reproducibility and for understanding the design choices. Without these details, I find it difficult to fully assess the technical soundness of the geometric prior and refinement module.
  • The experimental evidence is not sufficiently convincing. The proposed method does not consistently outperform strong baselines and shows lower performance than the original VGGT across all reported camera metrics on DyCheck. Additionally, I noticed that the highlighted 12.57% reduction in median point error comes from an ablation with a different training configuration from the final model. I wonder if the authors could further clarify this discrepancy and isolate the contribution of camera and depth conditioning from the general benefit of introducing a refinement network. These additional analyses would help better establish the effectiveness of the proposed components.
Questions for authors
  • Please see the concerns raised in the Weaknesses section.
  • I would appreciate further analysis or visualizations illustrating when and why geometric guidance helps or potentially hurts performance. A direct depth-unprojection baseline and an evaluation of the trade-off between accuracy and computational cost would also help better understand the practical benefits of the proposed design.
  • It would be helpful if the authors could use author–year citations consistently throughout the manuscript, distinguishing between narrative citations (e.g., “Smith et al. (2024) propose …”) and parenthetical citations (e.g., “… (Smith et al., 2024)”).
Reviewer_TA2xOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes R2R^2, an extension to the pretrained feed-forward 3D reconstruction model VGGT, aiming to address two issues: (1) the lack of explicit cross-view geometric verification during feature aggregation in dynamic scenes, and (2) the loosely parallel prediction of camera poses, depth maps, and 3D point maps despite their projective dependencies.

To tackle the first issue, the authors employ a two-pass teacher-student framework. A frozen VGGT teacher first predicts provisional camera parameters, depth maps, and patch features. By comparing observed cross-view feature correspondences with rigid reprojections, the method computes a rigid-consistency residual. This residual is normalized, ranked into percentiles q∈[0,1]q \in [0, 1], and converted into a signed patch-level bias B∈RV×PB \in \mathbb{R}^{V \times P} that modulates camera-token-to-patch-token attention in global blocks of a student VGGT, aided by lightweight query adapters AQℓA_Q^\ell.

To address the second issue, the authors append a 3-layer convolutional network at the end of the pipeline that takes concatenated camera encodings, log depth, and initial 3D points Href=[Φcam(C),log⁡D,P1]H_{\text{ref}} = [\Phi_{\text{cam}}(C), \log D, P_1] to predict a bounded residual offset Pfinal=P1+0.1tanh⁡(ΔP)P_{\text{final}} = P_1 + 0.1 \tanh(\Delta P).

Experiments are conducted on several dynamic-scene benchmarks (Sintel, TUM, Bonn, DyCheck).

Critical strengths
  1. Identifying that parallel prediction heads in feed-forward models (like VGGT) ignore projective dependencies, and that unverified cross-view feature aggregation struggles in dynamic scenes, is a valid and well-motivated starting point.
  2. Using reprojection residuals to measure geometric agreement rather than relying purely on semantic motion segmentation is conceptually appealing, as it naturally accounts for non-semantic outliers such as occlusions and correspondence failures.
  3. The authors are relatively transparent in the Discussion section about the limitations of their method, acknowledging that camera pose improvements are inconsistent across datasets and that qualitative point clouds still suffer from scattered peripheral noise.
Critical weaknesses
  1. Unconvincing Empirical Results:
  • On DyCheck (Tables 1 & 2), R2R^2 performs worse than the vanilla pretrained VGGT across all single-value camera and depth metrics. Results on Sintel and Bonn are also mixed and largely inferior to vanilla VGGT. Recent methods (VGGT-Ω\Omega, PAGE-4D) substantially outperform R2 across almost all benchmarks.
  • In Table 6, guiding blocks yields virtually identical results (e.g., Sintel ATE: 0.20680.2068, 0.20600.2060, 0.20690.2069; TUM ATE: 0.01210.0121, 0.01210.0121, 0.01190.0119). Claiming blocks 8--12 are optimal based on 10−410^{-4} fluctuations indicates that the prior BjB_j has negligible impact.
  1. Methodological Flaws & Ad-hoc Hyperparameters:
  • The residual rs→t,ir_{s\rightarrow t,i} relies on provisional teacher predictions (Ds,Es,EtD_s, E_s, E_t). When dynamic objects corrupt the teacher's initial pose, rigid reprojections fail globally, producing large residuals even on static backgrounds and misleading the student.
  • Eq. (5) hard-codes a within-view split at q=0.75q=0.75: b(q)={1−q0.75,q≤0.75,−q−0.750.25,q>0.75.b(q) = \begin{cases} 1 - \frac{q}{0.75}, & q \le 0.75, \\ -\frac{q - 0.75}{0.25}, & q > 0.75. \end{cases} This forces 25% of patches in every view to receive negative bias regardless of scene content—unjustly penalizing static patches in purely static scenes while assigning positive support to dynamic patches when moving objects exceed 25% of the view.
  • Bounding 3D offsets via Pfinal=P1+0.1tanh⁡(ΔP)P_{\text{final}} = P_1 + 0.1 \tanh(\Delta P) (Eq. 11) via a trivial 3-layer CNN ignores vast scale differences between indoor (TUM) and outdoor (Waymo, Forest) scenes.
  1. Running VGGT twice (Teacher + Student) plus cross-view matching and reprojection roughly doubles inference latency and memory overhead, yet the paper reports no runtime, FLOPs, or memory comparisons.

  2. Missing Promised Ablations & Flawed Setup:

  • Sec. 4.3 promises to "separate the contribution of the geometric residual from that of the additional trainable parameters," but this experiment is completely absent. It is unclear if any gains come from BjB_j or simply from fine-tuning blocks 8--12.
  • For Table 3, the backbone is frozen while VGGT's prediction heads are re-initialized from scratch and trained for only 20K steps. Training dense heads from scratch for 20K steps creates an under-trained baseline ("No refinement"), making the 12.57% gain unrepresentative of the pretrained pipeline. Table 3 also omits which dataset is evaluated.
  1. Severe Writing, Reproducibility, and Citation Irregularities:
  • Key terms lack mathematical definitions: how ms→t,im_{s\rightarrow t,i} is extracted, correspondence uncertainty us,iu_{s,i} in Eq. (4), the 16-channel camera encoding Φcam(C)\Phi_{\text{cam}}(C) in Eq. (9), and the exact loss terms in Eq. (12).
  • Training steps vary arbitrarily across experiments: 18,60018,600 (Sec. 4.1), 20,00020,000 (Sec. 5.3), 15,20015,200 (Fig. 2), and 1,0001,000 (Sec. A.4).
Questions for authors

Questions and Suggestions

  1. The authors aim to improve a geometric foundation model (VGGT) via reprojection residuals rs→t,ir_{s\rightarrow t,i}, yet the empirical results fail to support this claim (often degrading performance over the baseline). Combined with missing promised ablations (Sec. 4.3), undefined mathematical terms, this submission falls short of basic scientific rigor. Please treat future submissions with greater care.
  2. When experimental evidence consistently fails to validate a hypothesis (e.g., degraded metrics on DyCheck/Sintel and 10−410^{-4} variations in Table 6), it is necessary to question the hypothesis itself. Specifically, computing rigid reprojections from noisy provisional predictions (Ds,Es,Et)(D_s, E_s, E_t) in dynamic scenes may inherently do not work.
  3. Running a heavy foundation model like VGGT just to compute inter-frame residuals is computationally impractical. A much more sensible design would be to estimate initial poses and residuals using a lightweight network first, and then feed them into a heavier backbone for refinement.
Reviewer_hdFYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers the problem of 3D reconstruction and camera pose estimation from a set of input images, particularly in dynamic scenes. Building on an existing feedforward model (VGGT), it introduces a cross-view geometric consistency residual to guide camera attention, together with a lightweight module that refines the predicted 3D point maps using camera and depth estimates. The method is evaluated against the baseline and several recent approaches, achieving mixed results on camera pose and depth estimation, while demonstrating improvements in point reconstruction in a separate ablation.

Critical strengths
  • The idea of using intermediate geometric estimates to refine feed-forward reconstruction predictions is generally sensible. In particular, the proposed camera- and depth-conditioned point-refinement module reduces point reconstruction error in the reported ablation, providing preliminary evidence that such learned refinement could be useful.
Critical weaknesses
  • Limited quantitative improvements over existing baselines. Even among the methods included in the evaluation, the proposed approach does not establish state-of-the-art performance, with stronger alternatives frequently outperforming it across the considered camera pose and depth estimation benchmarks.

  • Limited comparison with relevant prior work. The evaluation omits several recent methods for feed-forward and dynamic-scene reconstruction, including CUT3R [E], V-DPM [C], and St4RTrack [D]. In addition, MASt3R-SfM [A] and GGPT [B] would be interesting methods to discuss and compare against, as they also explore improving initial reconstruction estimates through geometric reasoning and refinement.

  • Limited qualitative evaluation. The qualitative results in the main paper are restricted to a single figure showing one scene, which provides little insight into the practical advantages, limitations, or generalization capabilities of the proposed approach.

References

[A] Duisterhof et al., MASt3R-SfM: A Fully-Integrated Solution for Unconstrained Structure-from-Motion. 3DV 2025. https://arxiv.org/abs/2409.19152

[B] Chen et al., GGPT: Geometry-Grounded Point Transformer. CVPR 2026. https://arxiv.org/abs/2603.11174

[C] Sucar et al., V-DPM: 4D Video Reconstruction with Dynamic Point Maps. CVPR 2026. https://arxiv.org/abs/2601.09499

[D] Feng et al., St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World. ICCV 2025. https://arxiv.org/abs/2504.13152

[E] Wang et al., Continuous 3D Perception Model with Persistent State (CUT3R). CVPR 2025. https://arxiv.org/abs/2501.12387

Questions for authors

Rather than specific questions, I have a few suggestions for future revisions of the manuscript: broaden the comparison with relevant recent baselines, provide stronger experimental evidence for the proposed contributions, including more comprehensive ablations, and expand the qualitative evaluation to better illustrate the advantages and limitations of the approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.