acceptodds
Withdrawn from ICLR 2027

OpenU3R: Towards Zero-Shot Generalizable Feed-Forward Open-Vocabulary Understanding and 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Jointly performing 3D Gaussian reconstruction and scene understanding in real-time is essential for embodied agents operating in unseen environments, yet existing feed-forward methods are restricted to closed vocabularies or struggle to generalize across datasets. Thus, we present OpenU3R, an open-vocabulary feed-forward framework that leverages complementary multi-view cues to support high-quality 3D Gaussian reconstruction together with semantic and instance-level scene understanding. Instead of using a fixed closed-set classifier, OpenU3R projects object queries into a shared vision-language embedding space and matches them against text embeddings, thereby endowing our method with open-vocabulary semantic capability. We further propose confidence-aware multi-view CLIP distillation, which pools frozen CLIP features within object masks across views and injects them into the object queries, improving cross-dataset transfer. On ScanNet, OpenU3R outperforms the best-performing baseline by 7.86 percentage points in mIoU, 5.43 percentage points in PQ, and 3.36 percentage points in mAP while maintaining competitive novel-view synthesis quality. Under zero-shot evaluation on RealEstate10K, OpenU3R improves over the same baseline by 29.00 percentage points in mIoU, 32.93 percentage points in PQ, and 11.95 percentage points in mAP, demonstrating strong cross-dataset generalization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Understanding, Multi-View 3D Reconstruction, 3D Gaussian Splatting, Open-Vocabulary Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2s31OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OpenU3R, a feed-forward framework for joint 3D Gaussian reconstruction and scene understanding. A frozen VGGT-Ω backbone predicts geometry features that are converted into pixel-aligned 3D Gaussians for scene reconstruction. At the same time, CroCo appearance features are fused with VGGT geometric features. Mask2Former is used to produce object-centric representations and instance masks. The resulting object queries are projected into a shared CLIP embedding space. Object recognition is performed by matching query embeddings with text embeddings rather than using a fixed classifier. The model further distills confidence-filtered multi-view CLIP features into the object queries through a confidence-aware distillation objective. Finally, semantic and instance predictions are propagated to the corresponding 3D Gaussians.

Experiments demonstrate strong performance on ScanNet and large gains under zero-shot transfer to RealEstate10K-483. Compared with SIU3R, the method improves mIoU, PQ, and mAP on ScanNet while preserving reconstruction quality. On RealEstate10K-483, substantially larger improvements are reported.

Critical strengths
  1. Well-motivated object-centric open-vocabulary formulation

The paper presents a coherent object-centric formulation in which shared object queries serve as the representation for semantic understanding, while remaining tightly coupled to the reconstructed 3D Gaussians through a pixel-aligned design. Open-vocabulary recognition is achieved through query-text matching in a shared CLIP embedding space rather than a fixed classifier, enabling transfer beyond a predefined taxonomy. Furthermore, the confidence-aware multi-view CLIP distillation mechanism is well motivated. It aggregates object-level semantic cues across views while filtering unreliable observations, improving robustness to occlusions, viewpoint changes, and noisy single-view predictions.

  1. Strong empirical performance, particularly under domain shift

The experimental results are compelling, especially under the cross-dataset setting. It achieves clear improvements over prior methods on ScanNet. More importantly, the gains on RealEstate10K-483 are substantial across semantic, panoptic, and instance-level metrics, suggesting significantly improved robustness under domain shift.

Critical weaknesses
  1. Novelty is moderate

While the paper is well executed, its novelty appears incremental relative to the crowded literature on feed-forward reconstruction, semantic Gaussians, open-vocabulary understanding. The primary contribution appears to be extending an SIU3R-style framework with CLIP-aligned query representations and multi-view distillation. This is a meaningful contribution, but the conceptual advancement over the closest prior work feels moderate.

  1. Ablation evidence does not fully support the claims

The presented ablations indicate that Complementary Feature Fusion is responsible for the majority of the in-domain performance gains. In contrast, Open-Vocabulary Query Recognition and Confidence-aware Multi-View Distillation, which are core innovations of the paper, contribute modest improvements on ScanNet. Furthermore, the paper's strongest results are obtained on RealEstate10K-483, but no cross-dataset component ablations are provided. It remains unclear whether the reported zero-shot gains arise from the proposed open-vocabulary mechanisms or the Complementary Feature Fusion or the combination of several strong pretrained components such as VGGT-Ω, CroCo, CLIP, and Qwen3-VL. Stronger attribution analysis, particularly on RealEstate10K-483, would significantly strengthen the paper.

Questions for authors
  1. Can the authors provide a component ablation on RealEstate10K-483 (e.g., w/o CFF, w/o OVQR, w/o CMD)? This would help establish which components are responsible for the significant cross-dataset improvements.

  2. How does a simple baseline consisting of SIU3R with CLIP-based query-text matching perform relative to OpenU3R? This comparison would help clarify the contribution of the proposed distillation framework beyond replacing a classifier with CLIP retrieval.

  3. How sensitive are the zero-shot results to the vocabulary proposed by Qwen3-VL? What performance is obtained when using a fixed vocabulary or an alternative vocabulary proposal mechanism?

Reviewer_VRMeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OpenU3R, a feed-forward model for joint 3D Gaussian reconstruction and open-vocabulary scene understanding from two sparse views. It combines VGGT-Ω geometry features with CroCo appearance features, predicts shared object queries across views, and matches them with CLIP text embeddings. The paper also introduces RealEstate10K-483, an automatically annotated benchmark for cross-dataset evaluation.

Critical strengths
  • The paper studies an important problem: combining fast 3D reconstruction with semantic, instance, and panoptic understanding.
  • The overall design is coherent, and the geometry-appearance fusion leads to clear improvements in scene understanding.
  • The confidence-aware multi-view CLIP distillation is well motivated.
  • The reported results on ScanNet and RealEstate10K-483 are strong across reconstruction and understanding metrics.
Critical weaknesses
  1. The zero-shot evaluation is not sufficiently independent.
    RealEstate10K-483 uses VGGT-Ω for geometry and Qwen3-VL for its vocabulary, while OpenU3R relies on the same models. This may bias the evaluation toward OpenU3R. No human validation of the pseudo-GT is provided. Evaluation on an independent benchmark such as ScanNet++ or Replica would make the claim more convincing. Moreover, the quantitative evaluation maps the labels back to ScanNet-20, so performance on genuinely unseen categories remains unclear.

  2. An important baseline is missing.
    The annotation pipeline itself combines VGGT-Ω, Qwen3-VL, and SAM3. The full pipeline cannot be used directly because it sees target frames, but a version restricted to the same two context views would be a natural baseline. Without this comparison, it is unclear how much OpenU3R improves over a direct combination of existing foundation models.

  3. The method is less general than the presentation suggests.
    Although the formulation uses an arbitrary number of context views, the submitted implementation only supports exactly two. There is no experiment with more input views.

  4. The novelty and experimental evidence of some modules are limited.
    Shared multi-view queries, text-based query classification, and geometry-semantic fusion are closely related to SIU3R and PanSt3R. The exact fusion and distillation designs may be useful contributions, but their zero-shot benefits are not directly ablated on RealEstate10K-483.

  5. Some practical claims are not fully supported.
    The paper claims real-time inference but provides no runtime or memory comparison. Figure 1 also depicts deployment on a robot with stereo cameras, although no real-robot experiment is presented, which is potentially misleading.

Questions for authors

Refer to the weakness part.

Reviewer_KyTvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

OpenU3R combines VGGT-Ω geometry and CroCo appearance features with shared object queries for Gaussian reconstruction and semantic/instance prediction. Queries are projected into CLIP space and trained with multi-view teacher distillation; Qwen3-VL proposes a scene vocabulary at inference. The authors also automatically annotate RealEstate10K-483 for cross-dataset evaluation.

Critical strengths
  1. The unified representation is attractive. Pixel-aligned Gaussians and cross-view shared queries support RGB, semantic, and instance rendering. Replacing fixed classification with query–text matching is a reasonable open-vocabulary direction (pp. 4–6).
  2. The reported in-domain and cross-domain gains are substantial. ScanNet mIoU/PQ increase from SIU3R's 0.5920/0.6495 to 0.6706/0.7038, with substantial improvements also reported on RealEstate10K-483. Joint reporting of image, depth, and understanding metrics exposes trade-offs (pp. 7–9).
  3. The specification includes useful implementation detail. Zero-initialized residual fusion, teacher filtering, and the distillation schedule are described. Table 3 directly supports the usefulness of complementary geometry/appearance fusion.
Critical weaknesses
  1. The main experiments do not adequately test unseen-category recognition or its mechanism. Cross-dataset annotations are mapped to ScanNet-20 for metrics (p. 8), establishing transfer on a shared taxonomy rather than recognition of unseen categories among the 660 original concepts. The open-vocabulary examples on p. 17 help, but do not replace an explicit seen/unseen split with independently validated labels. Distillation and recognition ablations in Table 3 are only on ScanNet: full-model mAP is 0.3050, below 0.3364 without distillation and 0.3324 without open-vocabulary recognition. The modules provide small mIoU gains, but their principal cross-domain/unseen-category benefits remain untested directly.
  2. The custom evaluation and input conditions may favor the chosen foundation models. Pages 13–14 use VGGT-Ω for geometry/poses, Qwen3-VL for vocabulary, and SAM3 for semantic/instance labels. The evaluated system also uses VGGT-Ω and Qwen3-VL. Shared models do not invalidate automatic labels, but can create correlated errors, especially when evaluating depth against the same model family, without an independent audit. Moreover, Eq. (1) explicitly uses known camera intrinsics and poses, whereas the original SIU3R adopts unposed inputs. It is unclear whether all baselines receive identical camera information, or whether information estimated from complete evaluation clips enters model-visible inputs. These boundaries must be specified before attributing the transfer gains to the proposed semantic mechanisms.
  3. The central real-time claim lacks complete timing evidence. The manuscript does not report end-to-end latency, FPS, or memory covering VGGT-Ω, CroCo, query decoding, Qwen3-VL vocabulary generation, CLIP text encoding, and rendering. The appendix's less-than-five-minutes figure for 100 frames concerns annotation, not model inference. If vocabulary generation is amortized, both initial and cached costs should be reported. Feed-forward execution alone does not establish real-time performance.
Questions for authors
  1. What are the results and semantic-module ablations on independently validated categories explicitly disjoint from training? What is the recall of the Qwen-proposed vocabulary?
  2. Which intrinsics, poses, vocabularies, and context views does each method receive? How are correlated annotation errors audited, and can target-frame information enter model-visible inputs through preprocessing?
  3. What are complete initial/cached-vocabulary latency, throughput, and memory, with hardware, view count, resolution, and timing boundaries specified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.