- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents OpenU3R, a feed-forward framework for joint 3D Gaussian reconstruction and scene understanding. A frozen VGGT-Ω backbone predicts geometry features that are converted into pixel-aligned 3D Gaussians for scene reconstruction. At the same time, CroCo appearance features are fused with VGGT geometric features. Mask2Former is used to produce object-centric representations and instance masks. The resulting object queries are projected into a shared CLIP embedding space. Object recognition is performed by matching query embeddings with text embeddings rather than using a fixed classifier. The model further distills confidence-filtered multi-view CLIP features into the object queries through a confidence-aware distillation objective. Finally, semantic and instance predictions are propagated to the corresponding 3D Gaussians.
Experiments demonstrate strong performance on ScanNet and large gains under zero-shot transfer to RealEstate10K-483. Compared with SIU3R, the method improves mIoU, PQ, and mAP on ScanNet while preserving reconstruction quality. On RealEstate10K-483, substantially larger improvements are reported.
Critical strengths
- Well-motivated object-centric open-vocabulary formulation
The paper presents a coherent object-centric formulation in which shared object queries serve as the representation for semantic understanding, while remaining tightly coupled to the reconstructed 3D Gaussians through a pixel-aligned design. Open-vocabulary recognition is achieved through query-text matching in a shared CLIP embedding space rather than a fixed classifier, enabling transfer beyond a predefined taxonomy. Furthermore, the confidence-aware multi-view CLIP distillation mechanism is well motivated. It aggregates object-level semantic cues across views while filtering unreliable observations, improving robustness to occlusions, viewpoint changes, and noisy single-view predictions.
- Strong empirical performance, particularly under domain shift
The experimental results are compelling, especially under the cross-dataset setting. It achieves clear improvements over prior methods on ScanNet. More importantly, the gains on RealEstate10K-483 are substantial across semantic, panoptic, and instance-level metrics, suggesting significantly improved robustness under domain shift.
Critical weaknesses
- Novelty is moderate
While the paper is well executed, its novelty appears incremental relative to the crowded literature on feed-forward reconstruction, semantic Gaussians, open-vocabulary understanding. The primary contribution appears to be extending an SIU3R-style framework with CLIP-aligned query representations and multi-view distillation. This is a meaningful contribution, but the conceptual advancement over the closest prior work feels moderate.
- Ablation evidence does not fully support the claims
The presented ablations indicate that Complementary Feature Fusion is responsible for the majority of the in-domain performance gains. In contrast, Open-Vocabulary Query Recognition and Confidence-aware Multi-View Distillation, which are core innovations of the paper, contribute modest improvements on ScanNet. Furthermore, the paper's strongest results are obtained on RealEstate10K-483, but no cross-dataset component ablations are provided. It remains unclear whether the reported zero-shot gains arise from the proposed open-vocabulary mechanisms or the Complementary Feature Fusion or the combination of several strong pretrained components such as VGGT-Ω, CroCo, CLIP, and Qwen3-VL. Stronger attribution analysis, particularly on RealEstate10K-483, would significantly strengthen the paper.
Questions for authors
-
Can the authors provide a component ablation on RealEstate10K-483 (e.g., w/o CFF, w/o OVQR, w/o CMD)? This would help establish which components are responsible for the significant cross-dataset improvements.
-
How does a simple baseline consisting of SIU3R with CLIP-based query-text matching perform relative to OpenU3R? This comparison would help clarify the contribution of the proposed distillation framework beyond replacing a classifier with CLIP retrieval.
-
How sensitive are the zero-shot results to the vocabulary proposed by Qwen3-VL? What performance is obtained when using a fixed vocabulary or an alternative vocabulary proposal mechanism?