- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OpenPocket3D is a training-free framework for assigning class names to existing 3D instance masks without requiring a predefined category vocabulary. For each instance, it selects high-visibility camera views, generates candidate object names using a vision-language model, and filters these candidates using relative attention and projected-mask alignment. Final classification combines semantic similarity, candidate frequency, and spatial support. Experiments cover ScanNet200, ScanNet++, and Replica.
Critical strengths
the idea of query-free open-vocabulary segmentation makes sense and it shows state of the art results on the widely-used benchmarks on thi task
Critical weaknesses
The novelty is limited here. The general paradigm still follows the existing approach and does not demonstrate much contribution to this direction. The concept of being query-free makes sense, but whether this is beneficial in practical applications is questionable. Using a query is a natural way to interact with 3D scene. For example, given a scene, a user may want to select or identify a particular object under an open-vocabulary setting, they will typically provide a query.
The evaluation protocol is also unclear. In particular, how are the query-free predictions mapped to the predefined categories used for evaluation?
Questions for authors
see above
Response to Reviewer WuZ7
1. Novelty and Contributions
OpenPocket3D enables open-ended prediction from frozen 3D representations by constructing spatially verified, instance-specific pocket vocabularies.
Image-level names may describe neighboring objects rather than the target instance. OpenPocket3D explicitly evaluates target-specific support by contrasting candidate attention with competing classes and comparing support inside the projected mask with surrounding regions (Eqs. 1–2). The retained hypotheses form a pocket vocabulary scored using frozen 3D semantics and spatial evidence (Eqs. 3–5).
This training-free mechanism provides two concrete benefits: it avoids the additional OSM representation-and-decoding pipeline, and it improves class assignment beyond semantic matching over generated names.
Design advantages and supporting evidence.
The reported OpenPocket3D runtime assumes precomputed OV masks and features and excludes their construction. The AP comparison measures the complete class-assignment design.
Separately, Table 13 evaluates Candidate Expansion while keeping the other components fixed. Adding high-likelihood hypotheses and localizing candidates in complementary selected views jointly improves AP from 20.6 to 21.5.
Table 13. Candidate Expansion ablation on ScanNet200 with InternVL + Open3DIS (Appendix D.2). All other components are fixed.
2. Practical Benefits of Query-Free Prediction
Query-free prediction provides object names for unfamiliar scenes before users specify what to search for.
Query-based retrieval remains useful when the target is known. In an unfamiliar scene, OpenPocket3D automatically names detected 3D instances, providing a semantic inventory without requiring users to enumerate candidate categories. This inventory can guide subsequent queries.
Appendix F (Fig. 6) illustrates this with a wall-mounted device: the benchmark-query-based model predicts monitor, whereas OpenPocket3D outputs thermostat without a user-provided class query.
3. Evaluation Protocol
Per-scene predictions are finalized without benchmark-category queries; generated names are mapped to benchmark categories afterward for evaluation.
We follow the open-ended evaluation protocol of OE-3DIS and Any3DIS (Appendix A.2). Sentence-BERT cosine similarity maps each generated name to its nearest benchmark category; predictions with maximum similarity below 0.5 are discarded. The evaluator handles duplicates through scene-level linear sum assignment and assigns unit confidence to retained predictions.
These evaluation steps operate on finalized predictions and do not alter view selection, candidate generation, or per-scene class assignment.