acceptodds
Withdrawn from ICLR 2027

Visual Geometry Transformer in the Wild: Distractor-Free 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Current end-to-end multi-view 3D reconstruction methods achieve impressive results, but rely on a restrictive static assumption: the scene is entirely distractor-free with consistent cross-view geometry. This reliance on idealized inputs causes even the most advanced methods to fail in real-world settings, where transient distractors and occlusions are present. To address this, we propose Visual Geometry Transformer in the Wild (VGTW), an end-to-end framework for robust reconstruction from inconsistent views. At its core, we isolate and suppress distractor-affected regions while preserving the consistent components across views. Specifically, we introduce a Distractor-aware Training (DAT) strategy that separates clean features from distractor-contaminated ones in the attention mechanism while enforcing feature consistency across images. To enable this, we train the model with an auxiliary mask prediction head, using supervision from a new dataset we collected with pixel-level distractor masks. The resulting VGTW model is a feed-forward network that directly outputs clean, distractor-free point clouds. Remarkably, it requires no additional 3D supervision, remains computationally efficient, and is compatible with existing pipelines. Extensive experiments validate our approach, demonstrating state-of-the-art performance and robust generalization in diverse, real-world scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xXCsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VGTW (Visual geometry transformer in the wild), an end-to-end feed-forward framework for distractor-free 3d reconstruction from unordered multi-view image collections. It first observes that the attention of the static parts can be heavily focusing on the distractors. To address this, VGTW proposes a distractor-aware training strategy, consisting of the distractor supression loss, cross-view consistency loss and a distractor mask head. The method is based on VGGT and pi3. After being finetuned with the proposed training strategy, both VGTW(VGGT) and VGTW(pi3) achieves better resutls on point map estimation and depth map estimation.

Critical strengths

The proposed method effectively results in a cleaner point cloud of the static scene without requiring 3D ground truth. It finetunes large feed-forward models via LoRA so it's also computationally efficient. Besides, the ablation cleanly isolates each proposed components of the method and provides a thorough analysis.

Critical weaknesses
  1. The ground truth of the clean point clouds is generated using a pretrained pi3 model. This can potentially lead to a systematic bias towards pi3 results compared to other baselines like DUSt3R and MASt3R.

  2. The whole model is only trained on RobustNeRF-Mask (~1,000 images across a few scenes) and NeRF-OSR, and the only unseen data being evaluated on is the NeRF on-the-go data consisting of 6 test scenes. Besides NeRF on-the-go is pretty similar to RobustNeRF in terms of capture styles, camera trajectories, resolution, and distractor dynamics.

Questions for authors
  1. How do the results compare if evaluated against standard COLMAP point clouds / mesh reconstructions reconstructed solely from the clean/distractor-free frames, rather than relying on pseudo-ground-truth predictions from pi3?

  2. Can the proposed method and the trained model be generalized to other datasets with less similarities?

Reviewer_McQMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses VGGT and π3 to reconstruct static 3D scenes from unordered images containing moving or temporary objects. The main idea is to suppress these distractors during training and remove their points using a predicted mask. It trains LoRA adapters and a mask head with 2D masks from RobustNeRF-Mask and NeRFOSR, while encouraging features of the static scene to be consistent across views. The results show better reconstruction and depth scores on RobustNeRF and unseen NeRF-on-the-go scenes, with small inference overhead. It also shows qualitative results on DAVIS and DTU.

Critical strengths
  1. The method is practical and works with both VGGT and π3. Training uses 2D distractor masks, and the reported runtime for 10 views increases from 0.65 s to 0.70 s (Table 14). This makes the approach useful for improving an existing reconstruction model at a small inference cost.

  2. The results on unseen NeRF-on-the-go scenes support transfer beyond the training data. For the π3 model, average accuracy and completeness errors decrease from 0.051/0.074 to 0.027/0.060 (Table 1). The qualitative examples also show fewer distractor points and cleaner static geometry.

Critical weaknesses
  1. The evaluation needs clearer validation and reporting. Section 5.2 uses π3 predictions from clean views as the point-cloud reference. This provides a useful proxy, but an independent reference would strengthen the evidence for geometric accuracy. For the depth results, need to clarify whether all methods are evaluated on the same static pixels and report the retained coverage.

  2. The ablation does not fully isolate the benefit of the feature losses. A mask-head-only baseline with the same supervision and output filtering would help clarify how much these losses add. Table 5 also shows a tradeoff under high occlusion: Overall error is 0.101 with suppression alone, 0.148 after adding consistency, and 0.126 for the full method, although the full method has the best normal consistency.

Questions for authors
  1. Could you clarify which static pixels are used for the depth comparison and how retained coverage is handled across methods?
  2. How does a mask-head-only baseline compare with the full method under the same supervision and output filtering?
Reviewer_RpeTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes distractor-free feed-forward geometry model for consistent predictions in real-world settings. Method first starts from the analysis and the authors report simply masking the distractor region such as transient objects and occluded area help robust geometry predictions. The core idea lies in the training. The Distractor-aware Training ensures the model to attend to the distractor-free region by using the distractor mask labels. Experiments on in-house labeled RobustNeRF and unseen NeRF on-the-go show performance gains, built upon the two state-of-the-art feed-forward geometry models.

Critical strengths
  1. Analysis before any proposed method comes. Section 3.2 describes the core insights and hypothesis in their problem setting of this research. Then the method comes to resolve the challenges inside these insights and hypothesis. The experimental results further support the insights, which makes this paper a good example of academic research paper.
Critical weaknesses
  1. The insights, hypothesis, method proposal and experimental support is clear, but the novelty is not clear. As cited in the Related Works, MonST3R already adopts the binary dynamic mask by combining optical flow residuals with further SAM2 refinement, masking out moving pixels.
  2. Evaluation fairness. It is critical whether the comparison with baselines is fair enough. The proposed method is fine-tuned from pretrained baseline models with LoRA adaptor, especially on the RobustNeRF dataset with distractor mask. The compared methods like DUSt3R, MASt3R, Fast3R, VGGT-4D, VGGT and π3\pi^3 are never mentioned whether they are trained on the same dataset, and they should be fine-tuned in this dataset without the proposed DAT as the controlled experiments.
  3. The proposed two losses put the task into supervised 2D supervision, not unsupervised 3D geometric outlier (like RobustNeRF, NeRF on-the-go, or Easi3R). For the long-term research, unsupervised outlier rejection is more promising than 2D supervised semantic masking for feed-forward 3D reconstruction.
Questions for authors
  1. As stated in weakness 2, how are the experiments and evaluation fair? The compared models are not described how they are trained in the controlled experiments. It is important to state it in the main paper.
  2. Comparison with VGGT4D: VGGT4D reconstructs both static and dynamic scenes. In the paper's experiments, the GT (perhaps including the evaluation GT) is generated on the static images, using pretrained π3\pi^3 model. This GT already downgrades the VGGT4D's capability and makes the evaluation unfair. Figure 5 qualitatively show that VGGT4D fails dramatically, even worse than original VGGT on the distractor-present scenes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.