acceptodds
Under review as a conference paper at ICLR 2027

OpenPocket3D: Open-Ended 3D Instance Segmentation via Instance-Wise Pocket Vocabulary

Abstract

Open-ended 3D scene understanding aims to predict object-level semantics without relying on a predefined query vocabulary, whereas standard open-vocabulary methods require textual queries at inference time. While recent open-ended approaches eliminate this dependency by generating class names directly from visual inputs, they often suffer from noisy evidence and high computational overhead due to extensive multi-view representation construction or exhaustive proposal-wise language querying. To overcome these bottlenecks, we propose OpenPocket3D, a training-free, plug-and-play framework that seamlessly converts frozen open-vocabulary 3D backbones into open-ended predictors. By decoupling semantic prediction from 3D representation construction, OpenPocket3D shifts the paradigm from building entirely new features to efficient vocabulary selection. Given precomputed 3D masks, our method selects high-visibility 2D views, localizes VLM-generated candidate classes using internal attention maps, and rigorously filters them by enforcing geometric consistency with projected 3D masks. This distills an instance-wise “pocket vocabulary” that reduces the search space for each instance, enabling final predictions based on semantic similarity and geometric validation. Extensive experiments on ScanNet200, ScanNet++, and Replica demonstrate broad applicability across datasets and state-of-the-art performance on established open-ended benchmarks. The project page is available at: openpocket3d.github.io.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-ended 3D instance segmentation, open-vocabulary 3D perception, vision-language models, 3D scene understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WuZ7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

OpenPocket3D is a training-free framework for assigning class names to existing 3D instance masks without requiring a predefined category vocabulary. For each instance, it selects high-visibility camera views, generates candidate object names using a vision-language model, and filters these candidates using relative attention and projected-mask alignment. Final classification combines semantic similarity, candidate frequency, and spatial support. Experiments cover ScanNet200, ScanNet++, and Replica.

Critical strengths

the idea of query-free open-vocabulary segmentation makes sense and it shows state of the art results on the widely-used benchmarks on thi task

Critical weaknesses

The novelty is limited here. The general paradigm still follows the existing approach and does not demonstrate much contribution to this direction. The concept of being query-free makes sense, but whether this is beneficial in practical applications is questionable. Using a query is a natural way to interact with 3D scene. For example, given a scene, a user may want to select or identify a particular object under an open-vocabulary setting, they will typically provide a query.

The evaluation protocol is also unclear. In particular, how are the query-free predictions mapped to the predefined categories used for evaluation?

Questions for authors

see above

1 comment
AuthorsOpenReview ↗

Response to Reviewer WuZ7

1. Novelty and Contributions

OpenPocket3D enables open-ended prediction from frozen 3D representations by constructing spatially verified, instance-specific pocket vocabularies.

Image-level names may describe neighboring objects rather than the target instance. OpenPocket3D explicitly evaluates target-specific support by contrasting candidate attention with competing classes and comparing support inside the projected mask with surrounding regions (Eqs. 1–2). The retained hypotheses form a pocket vocabulary scored using frozen 3D semantics and spatial evidence (Eqs. 3–5).

This training-free mechanism provides two concrete benefits: it avoids the additional OSM representation-and-decoding pipeline, and it improves class assignment beyond semantic matching over generated names.

Design advantages and supporting evidence.

Comparison basisExisting burden or limitationOpenPocket3D advantageSupporting evidence
OSM-based OE-3DIS / Any3DISAdditional OSM token construction, aggregation, and proposal-wise language decodingLower additional prediction cost when existing OV masks and features are availableReported runtime: 472.17 s/scene, versus 1259.29 for OE-3DIS and 910.54 for Any3DIS (Tables 1 and 8)
Generated-name semantic matching (matched baseline)Semantic similarity alone does not explicitly verify whether a name is spatially supported by the target instanceMore accurate instance-level class assignment through verified hypotheses and semantic–geometric scoring16.6 → 21.5 AP, with masks, features, selected views, VLM, and generated candidate pool fixed before verification (Table 5; ScanNet200, InternVL + Open3DIS)

The reported OpenPocket3D runtime assumes precomputed OV masks and features and excludes their construction. The AP comparison measures the complete class-assignment design.

Separately, Table 13 evaluates Candidate Expansion while keeping the other components fixed. Adding high-likelihood hypotheses and localizing candidates in complementary selected views jointly improves AP from 20.6 to 21.5.

Table 13. Candidate Expansion ablation on ScanNet200 with InternVL + Open3DIS (Appendix D.2). All other components are fixed.

High-likelihood CandidatesCross-view LocalizationAP
––20.6
✓–21.0
–✓20.8
✓✓21.5

2. Practical Benefits of Query-Free Prediction

Query-free prediction provides object names for unfamiliar scenes before users specify what to search for.

Query-based retrieval remains useful when the target is known. In an unfamiliar scene, OpenPocket3D automatically names detected 3D instances, providing a semantic inventory without requiring users to enumerate candidate categories. This inventory can guide subsequent queries.

Appendix F (Fig. 6) illustrates this with a wall-mounted device: the benchmark-query-based model predicts monitor, whereas OpenPocket3D outputs thermostat without a user-provided class query.

3. Evaluation Protocol

Per-scene predictions are finalized without benchmark-category queries; generated names are mapped to benchmark categories afterward for evaluation.

We follow the open-ended evaluation protocol of OE-3DIS and Any3DIS (Appendix A.2). Sentence-BERT cosine similarity maps each generated name to its nearest benchmark category; predictions with maximum similarity below 0.5 are discarded. The evaluator handles duplicates through scene-level linear sum assignment and assigns unit confidence to retained predictions.

These evaluation steps operate on finalized predictions and do not alter view selection, candidate generation, or per-scene class assignment.

Reviewer_j3AsOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes OpenPocket3D, a training-free framework that converts frozen open-vocabulary 3D backbones into open-ended predictors using a frozen VLM. It consists of three stages:

  • Candidate Localization: Selects high-visibility views, generates candidate classes with a VLM, and localizes them via Relative Attention Maps.
  • Pocket Vocabulary Scoring: Retains candidates consistent with the projected 3D mask and assigns each a semantic and a geometric score.
  • Class Assignment: Selects the candidate with the highest combined score.
Critical strengths
  • Training-free and plug-and-play, validated on three different backbones.
  • Reuses VLM attention maps for localization, removing the need for an extra 2D segmentor such as SAM.
  • Faster than prior open-ended methods (472 s/scene vs. 1259 s for OE-3DIS and 911 s for Any3DIS).
  • Extensively validated across three datasets and multiple VLM–backbone combinations.
Critical weaknesses
  • The candidate pool depends on the VLM and 2D view coverage. Classes the VLM fails to generate can never be predicted, and gains vary across VLMs (e.g., LLaVA + OpenMask3D in Tables 5 and 14).
  • A gap remains compared with the open-vocabulary setting (21.5 vs. 23.7 AP for Open3DIS on ScanNet200).
Questions for authors
  • Since results are reported on the same validation split, was α tuned on the evaluation data? If so, please report results with a single fixed α or with α tuned on a held-out split.
3 comments
AuthorsOpenReview ↗

Response to Reviewer j3As

1. Candidate Pool and View Coverage

Candidate Expansion broadens the initial decoded pool; classes absent from the final pool cannot be predicted by our method.

For autoregressive VLMs, high-likelihood alternatives add candidates beyond the initially decoded names. Cross-view localization also tests a candidate in a complementary selected view, even when it was not generated there (Sec. 3.1; Appendix A.1). Together, these operations improve downstream AP from 20.6 to 21.5 (Table 13). Dependence on observed views remains a limitation, as acknowledged in Sec. 5.

Table 13. Candidate Expansion ablation on ScanNet200 with InternVL + Open3DIS (Appendix D.2). All other components are fixed.

High-likelihood CandidatesCross-view LocalizationAP
––20.6
✓–21.0
–✓20.8
✓✓21.5

2. Performance Across VLMs

AP improves in 21 of 24 evaluated settings; the remaining three decrease by 0.3 AP (Tables 5 and 14).

The three decreases occur with LLaVA: OpenMask3D on ScanNet200 and ScanNet++, and OpenScene* on Replica, each by 0.3 AP. These results support applicability across the evaluated models, with configuration-dependent gains.

Table 14. Analysis of OpenPocket3D across VLMs and open-vocabulary backbones on ScanNet++ and Replica (Appendix E). Values report AP, with changes from the corresponding open-ended semantic-matching baselines in parentheses.

DatasetVLMOpenScene*OpenMask3DOpen3DIS
ScanNet++RAM++–13.4 (+0.7)18.8 (+1.3)
LLaVA–12.9 (-0.3)19.1 (+1.0)
InternVL–16.5 (+0.5)21.9 (+1.4)
ReplicaRAM++9.1 (+2.0)10.7 (+3.7)9.1 (+1.6)
LLaVA12.5 (-0.3)12.0 (+1.0)13.5 (+3.8)
InternVL15.2 (+1.2)17.7 (+3.5)18.0 (+6.6)

The gains in Tables 5 and 14 are obtained without changing the underlying VLM or 3D backbone and without additional training. Although the magnitude of improvement varies across VLM–backbone combinations, Tables 4 and 12 show that the full formulation outperforms both semantic-only and spatial-only variants for RAM++, LLaVA, and InternVL. These findings support the complementary use of existing semantic and spatial evidence for instance-wise class assignment across VLM sources.

3. Comparison with the Open-Vocabulary Setting

OpenPocket3D reaches 21.5 AP without benchmark-category queries, compared with the 23.7 AP query-based OV reference.

The OV reference directly matches 3D features to benchmark-category queries, whereas the OE setting generates class names and maps them to benchmark categories only during evaluation (Tables 2–3; Appendix A.2). Under matched open-ended conditions, OpenPocket3D improves AP from 16.6 to 21.5 using the same Open3DIS masks, features, views, and candidate set (Table 5).

4. Selection of the Fusion Weight

Yes, the reported α values were selected using evaluation-split AP; fixing α at 0.5 retains gains in 21 of 24 settings.

As reported in Appendix A.3, the original α values were chosen separately for each dataset–backbone–VLM combination on the validation split used for evaluation. To assess this tuning effect, Table A fixes α = 0.5 across all 24 settings. Mean AP decreases by 0.29 relative to the tuned results, while improvements over matched open-ended baselines remain in 21 settings. For InternVL + Open3DIS on ScanNet200, fixed-α AP is 20.4, compared with the 16.6 AP matched baseline.

Table A. AP with α = 0.5 fixed across all settings. Parentheses show changes relative to the originally reported tuned-α results.

DatasetVLMOpenScene*OpenMask3DOpen3DIS
ScanNet200RAM++10.4 (-0.2)12.7 (-0.1)14.0 (-0.2)
LLaVA11.0 (0.0)11.6 (-0.1)14.8 (0.0)
InternVL15.7 (-0.7)17.8 (0.0)20.4 (-1.1)
ScanNet++RAM++–13.2 (-0.2)18.5 (-0.3)
LLaVA–12.3 (-0.6)19.1 (0.0)
InternVL–16.4 (-0.1)21.9 (0.0)
ReplicaRAM++9.1 (0.0)10.7 (0.0)8.8 (-0.3)
LLaVA12.5 (0.0)11.7 (-0.3)12.6 (-0.9)
InternVL15.2 (0.0)15.9 (-1.8)18.0 (0.0)

Thank you for your detailed response. We observe that the dependence on observed views is a limitation of the method. The other answers resolve most of my questions. I will therefore keep my score and recommend a weak accept for this paper.

Thank you for considering our clarifications and for your positive assessment.

Candidate Expansion makes fuller use of the selected views through additional class hypotheses and complementary spatial evidence. Class prediction remains dependent on the evidence available in those views, and we will clarify this scope in the revised manuscript (Sec. 5).

Reviewer_FvhdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method for open-volabulary 3D segmentation. It builds on existing 3D segmentation backbones, projects the generated 3D masks into high-visibility viewpoints, then uses VLM to propose,verify and assign open-vocabulary labels to 2D masks, and finally fuses the labels into the 3D masks. The paper has a focused direction and shows performance improvements, with sufficient comparative analysis; however, overall it does not go beyond the framework of past methods.

Critical strengths
  1. This paper is clearly written, and the figures make the intent relatively easy to grasp. At the same time, the problem is well focused, addressing specifically the step of acquiring semantic labels for 3D masks.

  2. This paper has sufficient experiments, comparing different 3D pretrained segmentation models, and different types of VLMs. It also ablate serveral improvement technicals.

Critical weaknesses

The innovation of this paper is relatively limited. First, it still stays within the framework of "first obtaining 3D masks, then assigning labels to them." Its performance is largely constrained by the accuracy of the 3D mask generation in the first stage. In addition, using VLMs for label proposals is a widely used approach.

Questions for authors
  1. In Tab2 and Tab3, why does ours slightly reduce AP for Open3DIS? Can you analyze what is special about Open3DIS?

  2. Current VLMs are gradually equipped 2D grounding capabilities (e.g., outputting the bbox or point of an object in the image). Does this make relative attention map filtering in this paper unnecessary? Or does it still have some unique advantage?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Fvhd

1. Novelty and Use of VLM Label Proposals

OpenPocket3D introduces instance-specific pocket vocabularies that make spatial verification integral to open-ended class prediction over frozen 3D representations.

VLM-generated names can describe neighboring objects rather than the target instance. We address this ambiguity by contrasting candidate attention with competing classes and comparing support inside the projected mask with its surroundings (Eqs. 1–2). The verified hypotheses are then scored using spatial evidence and frozen 3D semantics (Eqs. 3–5).

This mechanism provides two concrete benefits: it avoids the additional OSM representation-and-decoding pipeline, and it improves class assignment beyond semantic matching over generated names.

Design advantages and supporting evidence.

Comparison basisExisting burden or limitationOpenPocket3D advantageSupporting evidence
OSM-based OE-3DIS / Any3DISAdditional OSM token construction, aggregation, and proposal-wise language decodingLower additional prediction cost through reuse of existing OV masks and featuresReported runtime: 472.17 s/scene, versus 1259.29 for OE-3DIS and 910.54 for Any3DIS (Tables 1 and 8)
Semantic matching over generated namesSemantic similarity alone does not explicitly verify whether a name is spatially supported by the target instanceMore accurate instance-level class assignment using verified hypotheses and semantic–geometric scoring16.6 → 21.5 AP, with masks, features, views, VLM, and generated candidate pool fixed (Table 5; ScanNet200, InternVL + Open3DIS)

2. Dependence on 3D Mask Quality

OpenPocket3D delivers recognition gains with fixed masks and can benefit from stronger proposal backbones.

The matched comparison above demonstrates a class-assignment gain with unchanged masks. Tables 5 and 14 evaluate this design across different backbones, VLMs, and datasets.

Our module operates on 3D instance masks and language-aligned instance features. Its application to OpenScene*, OpenMask3D, and Open3DIS supports this modularity. Improved proposal recall and mask accuracy can provide better instance coverage and spatial evidence for verification, alleviating the upstream limitations discussed in Sec. 5 without redesigning the class-assignment mechanism.

3. Open3DIS and the Open-Vocabulary Reference

The OV reference uses benchmark-category queries; under matched OE conditions, OpenPocket3D improves AP in all nine Open3DIS settings.

The 23.7 AP OV reference receives benchmark-category queries, whereas the 21.5 AP OE result generates class names before evaluation mapping. These results use different vocabulary and category-mapping conditions.

The changes below are measured under the same OE protocol using identical Open3DIS masks and features (Tables 5 and 14).

AP changes from matched open-ended semantic-matching baselines.

DatasetRAM++LLaVAInternVL
ScanNet200+2.8+0.9+4.9
ScanNet+++1.3+1.0+1.4
Replica+1.6+3.8+6.6

4. Relative Attention and Direct 2D Grounding

Relative Attention enables instance-level verification with dense, class-relative evidence from the candidate-generating VLM.

Point/BBox grounding can provide candidate locations. Relative Attention gives our verification criterion dense, class-relative responses over both the projected mask and its surrounding context (Eqs. 1–2). This allows candidate support to be evaluated against the existing 3D instance.

The evidence comes from the same frozen VLM that proposes the labels, requiring neither coordinate-generation capabilities nor an additional 2D segmentor during open-ended class assignment. Tables 4 and 12 show the effectiveness of combining spatial evidence with frozen 3D semantics in the complete design across RAM++, LLaVA, and InternVL.

Thank you for the clarification. I agree that the reduced AP on Open3DIS reference appear to be reasonable , the provided open-ended comparisons resolve my concern. As for my second question, the response explain why relative attention may be useful, but does not provide controlled comparison with direct 2D grounding. Its advantage on accuracy and efficiency therefore remain unverified.

Thank you for confirming that our response resolved your concern about the Open3DIS reference. We have now conducted the controlled comparison with direct 2D grounding that you requested.

Using the same InternVL2.5-8B model and candidate inputs, Relative Attention achieves higher downstream AP and 3.4×/2.7× speedups over candidate-wise/joint box grounding, respectively.

We evaluated all three variants on the ScanNet200 validation set, keeping the Open3DIS proposals and features, selected views, candidate names, and semantic scoring unchanged. For direct grounding, the same VLM was prompted to return normalized bounding-box coordinates for visible instances of the candidate classes. We tested one prompt per candidate (repeat) and one prompt containing all candidates in a view (multi-once). The predicted boxes supplied the spatial evidence for candidate verification and geometric scoring.

Localization methodAPAP50AP25Runtime (s/scene)
Direct Grounding (repeat)20.028.531.71,598.5
Direct Grounding (multi-once)18.325.428.71,260.5
Relative Attention (ours)21.528.732.8467.7

Relative Attention improves AP by 1.5 points over candidate-wise grounding and 3.2 points over joint grounding. Joint prompting reduces the cost of direct grounding, but still requires approximately 2.7 times the runtime of our method.

After candidate generation, our localization step extracts spatial maps for the candidate classes in a view through one packed forward pass, without autoregressive coordinate decoding. Both direct-grounding variants instead generate bounding-box coordinates autoregressively.

These results support the accuracy–efficiency advantage of Relative Attention for candidate verification within OpenPocket3D. We plan to include the experimental protocol and results in the appendix of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

Dear AC,

We thank you and the reviewers for the careful evaluation and constructive discussion. OpenPocket3D enables query-free prediction from frozen OV 3D representations through spatially verified, instance-specific pocket vocabularies. It verifies VLM-generated hypotheses against competing classes and projected-mask context, then combines spatial evidence with existing 3D semantics for class assignment.

Below we summarize the reviewer feedback, discussion outcomes, and corresponding manuscript updates.

(1) Reviewer feedback and discussion outcomes

  • [Fvhd] highlighted that "the problem is well focused, addressing specifically the step of acquiring semantic labels for 3D masks."
  • [j3As] recognized the "Training-free and plug-and-play" design and that the method is "Extensively validated across three datasets and multiple VLM–backbone combinations."
  • [WuZ7] noted that "the idea of query-free open-vocabulary segmentation makes sense" and that "it shows state of the art results".
  • [Fvhd, follow-up on the Open3DIS reference] confirmed that "the provided open-ended comparisons resolve my concern." In response to the reviewer's remaining request, we subsequently added a controlled comparison with direct 2D grounding.
  • [j3As, follow-up] stated that "The other answers resolve most of my questions" and confirmed: "I will therefore keep my score and recommend a weak accept for this paper."

(2) Rebuttal evidence and manuscript updates

Main Paper

  • [WuZ7, Fvhd] Secs. 1–3; Fig. 1; Sec. 4.2, Tables 4–5. We clarified how instance-specific verification and scoring differ from OSM-based representation construction and image-tag-based semantic matching. On ScanNet200 with InternVL + Open3DIS, the complete class-assignment design improves AP from 16.6 to 21.5, holding masks, features, selected views, VLM, and the generated candidate pool before verification fixed.
  • [WuZ7, j3As, Fvhd] Sec. 4; Tables 2–3. We distinguished benchmark-query-based OV references from open-ended prediction. Benchmark category names are excluded from per-scene candidate generation and class assignment; generated predictions are mapped to benchmark categories afterward for evaluation.
  • [j3As, Fvhd] Sec. 5. We made the prediction scope explicit: mask quality constrains the available instances, while selected views and the chosen VLM constrain the candidate classes. Candidate Expansion makes fuller use of the selected views; dependence on available observations remains explicit.

Appendix

  • [Fvhd] App. D.3, Table 15 — new direct-grounding comparison. We added a comparison of downstream AP and runtime on ScanNet200 using the same InternVL2.5-8B model, Open3DIS masks and features, selected views, and candidate classes. Relative Attention achieves 21.5 AP, versus 20.0 for candidate-wise box prompting and 18.3 for joint box prompting. In the same comparison, candidate-wise and joint box prompting take 3.4× and 2.7× as long as Relative Attention, respectively (1,598.5 and 1,260.5 vs. 467.7 s/scene).
  • [j3As] App. A.3; App. C.1, Table 11 — new fixed-weight evaluation. We disclosed validation-AP-based selection of the original α values and added results with α = 0.5 across all 24 settings. Improvements over matched baselines remain in 21/24 settings, with a mean reduction of 0.29 AP relative to tuned results. InternVL + Open3DIS on ScanNet200 retains 20.4 AP, versus the 16.6 AP baseline.
  • [WuZ7] App. A.2. We clarified post-prediction Sentence-BERT category mapping, the 0.5 similarity threshold, duplicate handling, and unit-confidence AP evaluation. These evaluation steps do not alter per-scene class prediction.
  • [WuZ7, j3As] App. A.1; App. D.2, Table 14; App. E, Table 16. We clarified Candidate Expansion and the interpretation of existing ablations: high-likelihood alternatives and complementary-view localization improve AP from 20.6 to 21.5 on ScanNet200 with InternVL + Open3DIS. The cross-model results show gains in 21/24 settings, with configuration-dependent improvements.
  • [WuZ7, Fvhd] App. B.1, Table 8. We clarified the runtime comparison's scope: OpenPocket3D reuses precomputed OV masks and features, avoiding additional open-ended representation construction.
  • [WuZ7] App. F, Fig. 6. We expanded the explanation of the existing thermostat example to illustrate how query-free prediction provides an object inventory before users specify class queries.

The discussion has helped us sharpen the contribution, clarify the evaluation, and add controlled evidence for both fusion-weight robustness and the choice of localization mechanism. Thank you for your time and consideration.

Best regards,

Authors of OpenPocket3D

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.