acceptodds
Withdrawn from ICLR 2027

OV-RASF: Open-Vocabulary 3D Instance Segmentation via Reliability-Aware Semantic Fields

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-Vocabulary 3D instance segmentation aims to localize objects and generate 3D instance masks from point clouds according to text queries. Existing methods struggle to balance efficiency and reliability: they either rely on foundation models for per-frame segmentation and classification, incurring high inference costs, or adopt bounding-box and multi-view label voting, which still suffers from in-box background interference, sparse semantic observations, and cross-view prediction inconsistency. The key issue is the lack of explicit modeling of instance-level semantic reliability. To this end, we propose OV-RASF, which, for the first time, introduces instance-level semantic reliability modeling into this task. The method projects and fuses multi-view vision-language features into 3D point space to construct an instance-queryable discrete 3D semantic field, and jointly quantifies reliability via semantic coverage and class response margin to adaptively adjust fusion weights. It leverages superpoint local consistency to constrain semantic propagation, completing masks and suppressing cross-instance leakage. Through class-residual gated fusion, it adaptively selects a more reliable prediction source for each class. The method requires no additional training and achieves more reliable 3D instance segmentation while exploiting the Open-Vocabulary recognition capability of 2D detectors. To validate the effectiveness of our method, we conduct experiments on two mainstream datasets (ScanNet200 and Replica). Results show that, compared with the current state-of-the-art methods, our method improves by 1.4% on ScanNet200 and 1.1% on Replica. The code will be made public later.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-vocabulary 3D instance segmentation, 3D semantic field

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3CucOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose a training-free framework for open-vocabulary 3D instance Seg. A few modules are combined in the proposed method. Experimental results show improvements on existing methods.

Critical strengths

The proposed framework is simple and requires no additional backbone training.

Critical weaknesses

Limited novelty. It is more like a combination of existing techniques.

The ablation study does not clearly isolate the contribution of each component.

Questions for authors
  1. Is the Residual Gated Fusion mechanism generalizable to unseen categories? Extra experiments are needed.
Reviewer_WZz8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes OV-RASF, an open-vocabulary 3D instance segmentation (OV-3DIS) method that improves object classification and mask completeness in a training-free manner. Starting from class-agnostic 3D proposals and multi-view 2D detections, it builds a point-aligned vision-language semantic field and weights its contribution according to each instance’s semantic coverage and class-score margin. It then refines masks through semantic similarity constrained by local superpoint connectivity. Experiments report 26.1 mAP on ScanNet200 and 25.1 mAP on Replica, improving over OpenYOLO3D by 1.4 percentage points on both datasets, with increased inference time.

Critical strengths

The method combine semantic coverage and class-score margin to control how strongly 3D semantic features influence each instance classification. This is a simple mechnism to limit unreliable evidence from sparse observations or ambiguous features in a training-free manner. Furthermore, the results on two indoor benchmarks show some improvement on the AP score, including a 2.5-point improvement in ScanNet200 tail-class AP.

Critical weaknesses

The paper gives a very little new insight to the ongoing field OV-3DIS. It simply leverages most techniques from the already established works. Below are two of my main concerns:

  • The paper appears to combine instance-level feature aggregation from Open3DIS, the overall pipeline and multi-view prediction aggregation of OpenYOLO3D, and the point-aligned semantic representation of OpenScene. It borrows the concept of instance-aware feature aggregation from Open3DIS, adapts the MVPDist component from OpenYOLO3D and uses the point-pixel correspondences feature from OpenScene. Although the author introduce reliability weighting, superpoint-constrained refinement, and class-wise gating, the paper does not sufficiently establish why these additions constitute a substantial methodological advance beyond combining existing techniques. Furthermore, ablation studies and detailed analysis are limited

  • Evaluating two indoor benchmarks ScanNet200 and Replica is very limited given that the performance improvement is modest compared to the recent state-of-the-art. Furthermore, relying on a pretrained 3D backbone would not allow the model to adapt to other datasets with different distributions like ScanNet++ where the number of instance classes is much greater 1k5+ and point distribution is much denser 4M points per scene.

Questions for authors
  • Could the authors elaborate what is the key methodological novelty beyond combining Open3DIS-style instance feature aggregation, OpenYOLO3D’s detection-based pipeline, and OpenScene-style feature lifting? Which experiment isolates the benefit of the proposed reliability modeling over this direct combination? The current paper does not showcase this behavior.

  • Is the model zero-shot transferrable to completely different datasets like ScanNet++? A zero-shot 50 validation scenes evaluation would be helpful to see if the method is adaptive to different point cloud distribution but still in the same indoor domain.

  • Does the reported runtime include image-feature extraction from OpenSeg, 3D semantic-field construction, superpoint generation, and both 3D and 2D prediction branches? Could the authors explicitly clarify the runtime for each module?

  • Could the authors clarify how the class-wise gate additional supervision affects comparability with the baselines and how the gate handles categories absent from calibration? From my understanding, conventional zero-shot OV-3DIS methods assign semantic labels without category-specific calibration. This is new, I’m not getting the motivation behind this. Can the model work zero-shot like the previous works?

  • It’s really hard for me to understand the configuration in Table 4. Could the authors clarify the configurations R1–R4 and distinguish the gains from adaptive reliability weighting, mask refinement, and class-wise gating? Why does the full model achieve 26.3 mAP in Table 4 but 26.1 in Table 1?

  • Since the method chanegs both instance masks and semantic predictions, could the authors report class-agnostic mask AP/AR before and after refinement, together with an oracle-label evaluation using ground-truth category assignments? These results would help distinguish improvements in mask quality from those in semantic classification. Evaluating the original and enhanced classifiers on identical masks would further isolate the classification gains.

  • There’s a font error in Table 1.

Reviewer_KLf1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OV-RASF proposes an inference-stage framework for open-vocabulary 3D instance segmentation. It combines coverage- and margin-based semantic fusion, superpoint-constrained mask refinement, and class-wise branch selection using calibration-set performance. Experiments are reported on ScanNet200 and Replica.

Critical strengths
  1. The motivation is clear: semantic evidence varies in coverage and ambiguity, making uniform fusion potentially unreliable.
  2. The framework requires no backbone retraining and offers a practical way to incorporate semantic reliability and local geometric constraints into existing inference pipelines.
Critical weaknesses
  1. Reporting inconsistencies reduce confidence in the quantitative evidence. The full model achieves 26.1 mAP in Table 1 but 26.3 in Table 4. OpenMask3D and Open3DIS have identical mAP, mAP50, and mAP25 values across ScanNet200 and Replica. Table 4 also mixes fractional and percentage mAP values, and R1–R4 lack explicit configuration definitions. These issues may be correctable reporting errors, but they need resolution before the claimed improvements can be assessed reliably.

  2. The contribution of reliability modeling and the calibration protocol remain insufficiently established. The fusion-weight sweep does not isolate adaptive reliability weighting from simpler fixed-weight, coverage-only, or margin-only alternatives. Class-wise gating additionally uses labeled calibration data. Although the paper states that calibration and evaluation are independent, it does not specify the splits or whether Replica labels inform gating or hyperparameter selection. Consequently, the contribution of reliability modeling and the scope of cross-dataset generalization remain unclear.

Questions for authors
  1. Please reconcile Tables 1 and 4, verify the repeated baseline results across datasets, and explicitly define R1–R4 and their metric units.
  2. How does the method compare with fixed fusion, coverage-only weighting, and margin-only weighting under identical tuning conditions, preferably with class-wise gating disabled?
  3. Which scenes and labels are used for calibration and hyperparameter selection on each dataset? Are any Replica labels used? How are categories absent from the calibration set handled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.