- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces VGTW (Visual geometry transformer in the wild), an end-to-end feed-forward framework for distractor-free 3d reconstruction from unordered multi-view image collections. It first observes that the attention of the static parts can be heavily focusing on the distractors. To address this, VGTW proposes a distractor-aware training strategy, consisting of the distractor supression loss, cross-view consistency loss and a distractor mask head. The method is based on VGGT and pi3. After being finetuned with the proposed training strategy, both VGTW(VGGT) and VGTW(pi3) achieves better resutls on point map estimation and depth map estimation.
Critical strengths
The proposed method effectively results in a cleaner point cloud of the static scene without requiring 3D ground truth. It finetunes large feed-forward models via LoRA so it's also computationally efficient. Besides, the ablation cleanly isolates each proposed components of the method and provides a thorough analysis.
Critical weaknesses
-
The ground truth of the clean point clouds is generated using a pretrained pi3 model. This can potentially lead to a systematic bias towards pi3 results compared to other baselines like DUSt3R and MASt3R.
-
The whole model is only trained on RobustNeRF-Mask (~1,000 images across a few scenes) and NeRF-OSR, and the only unseen data being evaluated on is the NeRF on-the-go data consisting of 6 test scenes. Besides NeRF on-the-go is pretty similar to RobustNeRF in terms of capture styles, camera trajectories, resolution, and distractor dynamics.
Questions for authors
-
How do the results compare if evaluated against standard COLMAP point clouds / mesh reconstructions reconstructed solely from the clean/distractor-free frames, rather than relying on pseudo-ground-truth predictions from pi3?
-
Can the proposed method and the trained model be generalized to other datasets with less similarities?