acceptodds
Under review as a conference paper at ICLR 2027

Lightweight 3D Latent Semantics from Bayesian Inversion of 2D Foundation Models

31 views

Abstract

We present Casper3D, a lightweight probabilistic framework for converting noisy multi-view 2D foundation-model embeddings into a latent 3D semantic representation. We model view-level semantic features as noisy observations of an underlying 3D semantic state and infer this state with a set-based variational model that incorporates relative pose during multi-view reasoning. Casper3D is trained by predicting held-out semantic observations from novel viewpoints, while remaining aligned with visual and text semantic spaces for open-vocabulary 3D understanding. The framework is backbone-agnostic and applies to both language-aligned and self-supervised embeddings. We show that Casper3D produces more stable 3D semantics than multi-view pooling, especially in ambiguous and noisy settings.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
3D Representation Learning, Open-Vocabulary 3D Semantic Segmentation, 2D Foundation Models, Bayesian Inference, Multi-View Feature Fusion, Variational Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_99EPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Casper3D, a probabilistic framework for lifting multi-view 2D foundation model features into 3D semantic representations. It uses a pose-aware variational encoder to infer a view-invariant latent representation, trained through held-out view feature prediction. Experiments on ScanNet, ScanNet200, and Matterport3D demonstrate improved open-vocabulary 3D segmentation performance, particularly under sparse-view settings.

Critical strengths
  1. The paper presents an intuitive probabilistic perspective on multi-view 2D-to-3D feature lifting, an important problem in 3D vision with potential applications to various downstream tasks.
  2. The proposed framework, combining pose-aware attention, variational latent modeling, and held-out feature prediction, is well motivated and empirically effective.
Critical weaknesses
  1. Limited evaluation of generality. My main concern is that while the proposed framework has the potential to serve as a general approach to 3D representation learning, the evaluation focuses mainly on semantic segmentation. Exploring additional downstream tasks, such as 3D grounding or 3D VQA, would further demonstrate its generality and broader applicability.
  2. Related work (optional). The authors may consider discussing the LVSM series (LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias) and Spa3R (Predictive Spatial Field Modeling for 3D Visual Reasoning), which share related ideas of multi-view latent representation learning and held-out view predictive modeling.
Questions for authors

See weaknesses

Reviewer_akg5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work adopts a Bayesian‑style VAE for 3D understanding. Though the Bayesian perspective is interesting, its core motivation is questionable: it lacks essential Bayesian components and shows no fundamental advantage over deterministic encoders. Key 2026 SOTA baselines are omitted. Built largely on existing VAE paradigms, the contribution is incremental. Evaluations are restricted to indoor datasets. Large‑scale outdoor benchmarks such as nuScenes and SemanticKITTI are required to validate its claimed general‑purpose capability.

Critical strengths

(1)Applying Bayesian framework for the 3D understanding task is interesting.

Critical weaknesses

(1)The motivation constitutes the most critical concern. The so-called ``Bayesian inverse problem'' in this paper is modeling-wise equivalent to a noisy autoencoder trained with cosine loss and KL regularization. It lacks an explicit forward physical model, calibrated noise model, and domain-specific priors. Besides, the Bayesian nature of the posterior is not validated. From a Bayesian-modeling perspective, it shares no essential difference from deep deterministic regression, except replacing MSE with cosine loss, which brings no substantial gain in information content.

(2)It remains unclear why Bayesian inference is necessary instead of a deterministic encoder with hold‑out prediction. The paper argues that the absence of ground-truth 3D latent variables motivates its Bayesian formulation. Nevertheless, a deterministic encoder trained with identical objectives can also work without 3D supervision. Moreover, the main implementation is realized by a plain deep network and exhibits no perceivable difference.

(3)Several important baselines are missing. The paper mainly compares against segmentation works published in 2025, while relevant 2026 papers are neither compared nor discussed:

[1]GeoPurify: A Data-Efficient Geometric Distillation Framework for Open-Vocabulary 3D Segmentation, ICLR 2026

[2]OVSeg3R: Learn Open-vocabulary Instance Segmentation from 2D via 3D Reconstruction, ICLR 2026

[3]GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation, CVPR 2026

(4)This paper is essentially an application of the VAE framework without further substantial improvements, and only delivers incremental innovation.

(5)How does Casper3D perform on outdoor and large-scale benchmarks? Evaluations are primarily conducted on indoor RGB-D datasets. Given the claimed generality of the proposed method, evaluations on nuScenes and SemanticKITTI would strengthen its arguments. Although presented as a general-purpose framework, the paper does not demonstrate its capacity on downstream tasks such as scene understanding or map reconstruction.

Questions for authors

please refer to the weakness.

Reviewer_hRGwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to build 3D feature latents from 2D foundation models, where the main challenge is that no ground-truth 3D latent exists. To address this, the paper formulates the problem as a Bayesian inverse problem and infers the probability distribution of the 3D latent from the features observed in each view. It uses view-invariant modeling to remove view-dependent factors as much as possible. For the same purpose, the loss is computed on a held-out set rather than the context views, so that a 3D-consistency objective suppresses view-dependent factors.

Critical strengths
  • The motivation is clear. Building 3D features is difficult and requires lifting 2D features. I agree that 2D features contain view-dependent components due to occlusion, pose-dependent factors, and appearance, and I agree with the motivation that removing these components yields proper 3D features.

  • I also find the Bayesian inverse problem formulation well suited to this goal. 2D features can be viewed as being generated from a 3D latent conditioned on the camera viewpoint. Since only 2D features are available, it is reasonable to recover the 3D latent through the inverse process, and the paper turns this idea into a model well.

  • The paper validates the approach on 3D scene understanding tasks, and the results clearly show that it outperforms other models.

  • The paper is very well organized and easy to follow.

Critical weaknesses
  1. Novelty compared with feature lifting methods. My main concern is that feature lifting methods based on NeRF or 3DGS seem to share a similar concept [A,B,C]. Although they do not call it a Bayesian inverse problem, NVS-based training also applies the loss on held-out views in the same sense. It would be helpful to clarify the differences from feed-forward models that embed features into Gaussians, as well as from recent feed-forward NVS models [D}.

  2. Validation with a single foundation model. It is disappointing that the method is validated only with TIPSv2. The method seems general, yet it is shown to work with only one model. I am curious about results with other foundation models, especially models without explicit semantics, such as DINOv3 and SAM. If this is difficult, the authors could follow the setup of multi-view foundation models.

  3. Gaussian prior on the 3D latent. I wonder whether it is appropriate to model the 3D latent z with a Gaussian distribution. The paper makes p(z) follow a Gaussian, but similar to the feature-average regularization term, shouldn’t the latent distribution follow the distribution of the foundation model? It is also interesting that the model collapses without the KL regularization term. I would like to hear the authors’ thoughts on this.

  4. Necessity of the pose bias. Is the pose bias necessary? Since h_k already incorporates both semantic features and pose information, it is unclear whether adding pose information again provides additional benefit.

  5. Figure readability (minor). The figures are not very readable. While the text is easy to follow, the figures are difficult to understand.

[A] Fan, Zhiwen, et al. "Large spatial model: End-to-end unposed images to semantic 3d." Advances in neural information processing systems 37 (2024): 40212-40229.

[B] An, Honggyu, et al. "C3g: Learning compact 3d representations with 2k gaussians." arXiv preprint arXiv:2512.04021 (2025).

[C] Sun, Xiangyu, et al. "Uni3r: Unified 3d reconstruction and semantic understanding via generalizable gaussian splatting from unposed multi-view images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[D] Ryu, Kwonyoung, et al. "Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

Questions for authors

See weakness. I just curious about the clarification of contribution and generalizbility. Please focus on it and when the author resolves it well, I'll raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.