- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Casper3D, a probabilistic framework for lifting multi-view 2D foundation model features into 3D semantic representations. It uses a pose-aware variational encoder to infer a view-invariant latent representation, trained through held-out view feature prediction. Experiments on ScanNet, ScanNet200, and Matterport3D demonstrate improved open-vocabulary 3D segmentation performance, particularly under sparse-view settings.
Critical strengths
- The paper presents an intuitive probabilistic perspective on multi-view 2D-to-3D feature lifting, an important problem in 3D vision with potential applications to various downstream tasks.
- The proposed framework, combining pose-aware attention, variational latent modeling, and held-out feature prediction, is well motivated and empirically effective.
Critical weaknesses
- Limited evaluation of generality. My main concern is that while the proposed framework has the potential to serve as a general approach to 3D representation learning, the evaluation focuses mainly on semantic segmentation. Exploring additional downstream tasks, such as 3D grounding or 3D VQA, would further demonstrate its generality and broader applicability.
- Related work (optional). The authors may consider discussing the LVSM series (LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias) and Spa3R (Predictive Spatial Field Modeling for 3D Visual Reasoning), which share related ideas of multi-view latent representation learning and held-out view predictive modeling.
Questions for authors
See weaknesses