acceptodds
Under review as a conference paper at ICLR 2027

3DISCO: 3D Open Vocab Instance Segmentation with Clean Objects via Gaussian Splatting

Abstract

Segmenting 3D scenes into semantically meaningful instances is central to embodied perception. Leveraging high-quality 2D segmentations across multiple views is promising, but maintaining consistent identities across views, constraining Gaussians to lie within one object's boundaries, and assigning semantics based on highly informative views are challenging. We introduce 3DISCO, a 3D segmentation method based on Gaussian splatting that identifies objects using instance features learned via uncertainty-aware contrastive training. 3DISCO treats 2D masks as noisy labels, using predicted uncertainty to down-weight unreliable regions and guide the recovery of erroneous or missing annotations. To enable cleaner boundary delineation, 3DISCO prunes Gaussians that exhibit feature similarity to an instance but are inconsistent with that instance's 2D masks. 3DISCO then assigns semantics to each instance using features from only its most informative views, selected via a multimodal reranker, producing language-aligned 3D representations. Experiments on LERF and ScanNet show substantial improvements in semantic accuracy and boundary coherence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open vocabulary, 3d segmentation, gaussian splatting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Sk48OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

3DISCO addresses open-vocabulary 3D instance segmentation from posed images. Qwen3-VL and SAM3 provide initial object masks. The method learns Gaussian instance features through a contrastive objective with uncertainty weighting. The resulting representation guides subsequent mask correction. A prototype-guided mask consistency objective suppresses Gaussians whose projections conflict with their assigned masks. Clustering extracts instances for semantic labeling. Recognition combines isolated object renders with contextual crops through view selection and multimodal reranking. The authors report improved retrieval and semantic segmentation on LERF-OVS and ScanNet++. Component ablations examine the proposed mechanisms. Scene editing and robot manipulation demonstrations illustrate downstream applications.

Critical strengths
  1. The PMC objective directly connects instance assignments with projected mask consistency. Table 3 supports its contribution to retrieval performance. Figure 3 also shows cleaner object movement with fewer floating artifacts. These results provide meaningful evidence that refining the geometry associated with an instance benefits recognition and editing. This is practically consequential because extracted objects must retain coherent geometry when manipulated independently.
  2. Uncertainty estimates influence both feature learning and mask correction. The learned representation guides new SAM prompts while replacement masks provide fresh supervision. The ablations in Table 3 support the usefulness of this adaptive process. The design offers a practical way to revise imperfect image masks using information accumulated across views. It addresses a relevant limitation of learning 3D instances from a fixed set of noisy masks.
Critical weaknesses
  1. The paper presents independent 3D instance recovery as a central contribution. The evaluation in Section 4.1 mainly measures language retrieval and semantic segmentation. The ScanNet++ metric aggregates predictions by class. Merging objects of the same class or splitting one object can therefore remain unpenalized when their combined region is unchanged. The editing examples demonstrate instance separation in selected cases. They do not establish overall instance quality. This leaves the central instance modeling claim insufficiently validated.
  2. Uncertainty is used to address mask noise. The subsequent PMC stage freezes instance features and suppresses Gaussians that conflict with the masks. Such conflicts can reflect incorrect instance assignments or residual supervision errors. The objective in Section 3.3 does not explicitly distinguish these sources. Section 4.2 reports surface loss when an alternative mask source is used. It remains unclear how reliably the preceding correction stage protects valid geometry during pruning. This limits the evidence for robustness to remaining mask errors.
  3. The semantic module presents view selection and label reranking as important contributions. The analysis in Table 3 supports the complementary value of isolated renders and contextual crops. It does not separately establish the contributions of view selection and reranking. The extent to which the semantic gains arise from these proposed mechanisms therefore remains unclear. This limits the evidence for the independent value of the retrieval and reranking strategy.
  4. The introduction claims improved overall processing efficiency. Its argument emphasizes fewer candidate masks and fewer views used for recognition. These savings concern individual stages of a pipeline that also requires multiple model calls and optimization stages. The paper reports an overall runtime but does not establish a comparable cost advantage over existing methods. The relationship between complete processing cost and performance gains therefore remains insufficiently characterized.
Questions for authors

None.

3 comments

We thank the reviewer for the detailed and constructive review. We address each point in order below. Tables R1–R5 are numbered consistently across our responses to all three reviewers, so the same number always refers to the same table.

Instance segmentation on ScanNet++

We did not originally report instance segmentation metrics because LERF-OVS, our main benchmark, contains no instance ground truth, and we evaluated ScanNet++ with the same class-level metrics for consistency. We agree with the reviewer that class-aggregated metrics can leave merged or split instances unpenalized and that instance AP is a better measure of instance quality. As requested, we have now included 3D instance AP, AP50, and AP25 for our method and OpenSplat3D (Table R1). OpenSplat3D uses its original masks and encoder. Besides open-vocabulary AP, we report class-agnostic AP, which ignores class labels and so directly penalizes merged or split instances.

In the original submission's semantic segmentation comparison on ScanNet++, all methods were run on the same 10 scenes with an identical evaluation protocol, and each scene was queried only with the classes annotated in it, following the LERF-OVS convention. However, Instance AP counts predictions of classes absent from a scene as false positives, which the original protocol cannot produce. In the revised paper (Sec. 4.2), we have updated both the semantic and the instance segmentation results to query every scene with the full class list of the official ScanNet++ instance benchmark, including classes that do not appear in the scene. We consider 83 of the top-100 semantic classes: the benchmark itself excludes 16 classes (e.g., wall, floor, ceiling), and we additionally remove "window" to focus the evaluation on objects. We compute class-agnostic AP in the same way as OpenSplat3D.

Table R1. 3D instance segmentation on 10 ScanNet++ scenes, with every scene queried with all 83 classes (OV = open-vocabulary, CA = class-agnostic).

MethodOV APOV AP50OV AP25CA APCA AP50CA AP25
OpenSplat3D13.7427.8041.3510.0823.8341.67
Ours19.2636.5248.8612.9027.2146.60

Our method outperforms OpenSplat3D on every metric, including class-agnostic AP, so our instances are more accurate even when class labels are ignored.

We added these results to Sec. 4.2 and describe the protocol in App. B. We are currently evaluating on the 50 ScanNet++ validation scenes used by OpenSplat3D and would be happy to include these results in the camera-ready version, if the reviewers agree.

Residual mask errors and PMC

The PMC objective alone does not distinguish incorrect instance assignments from residual mask errors. Our pipeline handles residual mask errors before PMC through uncertainty weighting and mask correction, and Table R4 reports the corresponding ablations on LERF-OVS. These rows were already in the ablation table of the original submission, and the revised paper (Sec. 4.2) updates them to held-out calibration.

Table R4. LERF-OVS mIoU of our 2DGS variant with per-scene held-out calibration for ablations of PMC and the stages before it, and with LangSplat's SAM masks in place of ours.

SettingmIoU
No PMC47.70
No uncertainty62.86
No mask correction62.01
No uncertainty, no mask correction64.06
LangSplat (SAM level-3) masks64.42
Full69.93

Uncertainty weighting and mask correction work as a pair. Removing either one lowers mIoU, and using either one without the other is worse than using neither, so their gain comes from the combination. Uncertainty weighting downweights the instance supervision in uncertain regions, so without mask correction these regions receive only a weak training signal. Mask correction replaces the masks in these regions and gives them a stronger signal, but it relies on the uncertainty estimates to detect missed objects, and without them it discards valid masks and re-queries spurious regions. With both components in place, adding PMC raises mIoU substantially, which indicates that the Gaussians it suppresses are mostly erroneous rather than valid surfaces.

The surface loss in Sec. 4.2 shows the limit of what these stages can repair. That experiment runs our full pipeline, including uncertainty weighting and mask correction, with LangSplat's SAM masks, which are inconsistent across views. When the input masks disagree this strongly, the correction stage cannot restore enough consistency, PMC suppresses some valid surfaces, and accuracy drops (Table R4). This is why we state in Sec. 4.2 that our method needs moderately view-consistent masks. With such masks, the ablations above and the occlusion experiment show that valid geometry is largely preserved.

Occlusion handling in PMC

Rendering an instance in isolation, with the opacity of all other Gaussians set to zero (App. A.4), can expose surfaces that are hidden in a given view and therefore absent from its visible SAM mask. Our PMC objective does not explicitly reason about visibility, and instead relies on multi-view aggregation to prevent individual occlusions from distorting an object's geometry. A surface that is hidden in some views still lies inside the object's mask in the views where it is visible, and in those views PMC's mask term rewards its opacity. Therefore, the supporting views protect the surface against the conflicting views in which it is occluded. Erroneous geometry does not receive this support. Floaters and Gaussians assigned to the wrong instance fall outside the instance's mask in most of the views where they are visible, so PMC suppresses them. Therefore, the difference between occluded surfaces and erroneous geometry lies in how many views support them.

To test this, we built a simulated scene with two objects, one partially occluding the other, and measured how well PMC preserves the occluded object's full geometry as the amount of occlusion in the training views increases. Starting from a fixed set of unoccluded views that covers the whole object, we add a growing number of views in which 30 to 50% of the object is hidden, raising the fraction of occluded training views from 0% to 70%. We then compare the object's isolated render against its ground-truth mask in unoccluded views. mIoU stays above 0.96 even with 30% occluded views, and over the rest of the range it declines only to around 0.86. Therefore, PMC preserves occluded surfaces even when most training views are occluded, although frequent occlusion costs a moderate amount of geometry. The revised paper adds this discussion and the occlusion experiment in App. C.1.

View selection and reranking

View selection was designed primarily for efficiency. Without it, label reranking has to run on every view of every object, which is expensive, whereas view selection restricts it to the top TT most informative views of each object. Therefore, the paper assessed the accuracy of the semantic module through its two view types, isolated renders and contextual crops (paper Tab. 3), rather than ablating view selection separately. We agree with the reviewer that separating view selection from reranking is valuable, and we have now ablated both on LERF-OVS (Table R3).

Table R3. LERF-OVS mIoU of our 2DGS variant with per-scene held-out calibration when removing view selection, reranking, or both.

SettingmIoU
No view selection67.89
No reranking63.75
Neither62.07
Full69.93

Removing either component lowers mIoU, with reranking having the larger effect, and removing both gives the lowest mIoU, so each component contributes to the final accuracy. Beyond reducing the number of views used for recognition, view selection also improves accuracy, since it restricts the job of the reranker. The revised paper adds these ablations to Sec. 4.2 and App. C.3.

End-to-end processing cost

The introduction based its efficiency claim on two stages of our pipeline: the smaller number of candidate masks and the smaller number of views used for recognition. We agree that the complete cost is the relevant comparison, so Table R5 reports the average per-scene runtime of each full pipeline, including all model calls—we now include this in App. A.9 of the revised paper. The table separates training-based methods, which optimize per-scene features, from training-free methods.

Table R5. Average per-scene runtime in minutes, grouped by whether a method trains per-scene features. Baseline masks use LangSplat's SAM+CLIP preprocessing.

MethodTypeMasksTrain + EvalTotal
LangSplatTraining-based148.8875.48224.36
OpenGaussianTraining-based148.8853.78202.66
OursTraining-based3.4730.6034.07
Dr. SplatTraining-free148.8821.10169.98
VALATraining-free148.889.05157.93

Among training-based methods, ours has the fastest training and evaluation, so it would remain the fastest training-based pipeline end to end even if the baselines used our masks. The training-free methods Dr. Splat and VALA are faster in training and evaluation because they assign 2D features to Gaussians by weighted multi-view aggregation, with no gradient-based optimization and no instance learning. Dr. Splat additionally relies on a product-quantization codebook that is built once offline from CLIP embeddings of LVIS object crops. With the SAM+CLIP preprocessing used by the baselines, our full pipeline has the lowest total runtime of all compared methods. If the training-free methods used our masks, they would also be faster than ours end to end, but this speed comes at a cost in accuracy: both score clearly below ours in LERF-OVS mIoU with every combination of masks and encoder (Table R2, in our responses to Reviewers nT8i and daYP).

We hope these clarifications and results address the reviewer's concerns.

Reviewer_daYPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents 3DISCO, a Gaussian-based framework for open-vocabulary 3D instance segmentation. It combines uncertainty-aware instance feature learning, mask correction, prototype-guided mask consistency (PMC), and semantic retrieval from selected isolated and contextual views. Experiments are conducted on LERF-OVS and ten ScanNet++ scenes, with additional demonstrations in scene editing and robotic manipulation.

Critical strengths

The paper addresses an important limitation of Gaussian-based instance segmentation: contamination by unrelated geometry. Table 3 demonstrates a substantial improvement from PMC, increasing mIoU from 52.56 to 70.67. The scene-editing examples further illustrate the practical benefits of cleaner instance extraction.

Critical weaknesses
    1. Potential inconsistency between isolated rendering and visible-mask supervision under occlusion.

Equation (8) aligns an isolated instance rendering with the visible SAM mask, while Appendix A.4 explicitly sets the opacity of non-target Gaussians to zero. Consequently, valid surfaces hidden behind other objects may become visible in the isolated rendering but remain absent from the supervision mask, potentially receiving incorrect penalties. Section 4.2 acknowledges that inconsistent LangSplat masks cause PMC to suppress valid surface Gaussians, illustrating its sensitivity to imperfect supervision. Although this does not directly establish the occlusion-related failure, it raises concerns about geometric completeness. Furthermore, no qualitative ScanNet++ results are provided in the main paper or supplementary material. Given the substantial occlusion and scene complexity in ScanNet++, additional qualitative evaluations would be valuable.

    1. Unclear fairness of evaluation and threshold calibration.

Appendix A.7 states that both retrieval thresholds are optimized separately for each LERF-OVS scene using three ground-truth masks. However, it is unclear whether these calibration masks are excluded from the final evaluation or whether the compared methods receive equivalent calibration access. Since the baseline results are imported from previous publications, this raises concerns about the fairness of the reported comparisons. Results using fixed thresholds or strictly held-out calibration masks would strengthen the evaluation.

    1. Insufficient isolation of improvements from stronger 2D foundation models.

The framework incorporates Qwen3-VL, SAM3, Perception Encoder, and a multimodal reranker, whereas several earlier baselines rely on SAM and CLIP. Therefore, the reported improvements may partly originate from stronger 2D segmentation and semantic recognition rather than the proposed 3D components. More controlled comparisons using identical 2D masks and recognition modules are necessary. In particular, how much of the improvement over OpenSplat3D would remain under matched 2D supervision and semantic retrieval? The authors also acknowledge in Section 4.2 that the method requires moderately view-consistent masks, further highlighting its dependence on 2D segmentation quality.

    1. Insufficient evaluation of 3D instance segmentation.

Although the paper targets open-vocabulary 3D instance segmentation, its ScanNet++ experiments evaluate only class-level semantic segmentation using mIoU and mAcc@25 across ten scenes. Standard instance-level metrics, such as class-agnostic AP and open-vocabulary instance AP, are missing. Furthermore, Appendix B indicates that each scene is queried using its ground-truth annotated classes, which simplifies the open-vocabulary recognition setting. While OpenSplat3D is included in the LERF-OVS comparison, an instance-level comparison on ScanNet++ under a consistent evaluation protocol is needed to substantiate the paper's central claims.

Questions for authors
  • How does PMC prevent valid surfaces occluded by other instances from being incorrectly penalized when rendering instances in isolation, as described in Appendix A.4?

  • Were the three ground-truth masks used for threshold calibration excluded from evaluation? Did all methods in Table 1 receive equivalent calibration access?

  • Could the authors provide controlled comparisons using the same 2D segmentation and semantic recognition models to isolate the contributions of the proposed 3D components?

  • Could the authors report standard 3D instance segmentation metrics on ScanNet++, including a comparison with OpenSplat3D under the same protocol?

4 comments

We thank the reviewer for the detailed and constructive review. We address each point in order below. Tables R1–R5 are numbered consistently across our responses to all three reviewers, so the same number always refers to the same table.

Occlusion handling in PMC

Rendering an instance in isolation, with the opacity of all other Gaussians set to zero (App. A.4), can expose surfaces that are hidden in a given view and therefore absent from its visible SAM mask. Our PMC objective does not explicitly reason about visibility, and instead relies on multi-view aggregation to prevent individual occlusions from distorting an object's geometry. A surface that is hidden in some views still lies inside the object's mask in the views where it is visible, and in those views PMC's mask term rewards its opacity. Therefore, the supporting views protect the surface against the conflicting views in which it is occluded. Erroneous geometry does not receive this support. Floaters and Gaussians assigned to the wrong instance fall outside the instance's mask in most of the views where they are visible, so PMC suppresses them. Therefore, the difference between occluded surfaces and erroneous geometry lies in how many views support them.

To test this, we built a simulated scene with two objects, one partially occluding the other, and measured how well PMC preserves the occluded object's full geometry as the amount of occlusion in the training views increases. Starting from a fixed set of unoccluded views that covers the whole object, we add a growing number of views in which 30 to 50% of the object is hidden, raising the fraction of occluded training views from 0% to 70%. We then compare the object's isolated render against its ground-truth mask in unoccluded views. mIoU stays above 0.96 even with 30% occluded views, and over the rest of the range it declines only to around 0.86. Therefore, PMC preserves occluded surfaces even when most training views are occluded, although frequent occlusion costs a moderate amount of geometry. The revised paper adds this discussion and the occlusion experiment in App. C.1.

Residual mask errors and PMC

The PMC objective alone does not distinguish incorrect instance assignments from residual mask errors. Our pipeline handles residual mask errors before PMC through uncertainty weighting and mask correction, and Table R4 reports the corresponding ablations on LERF-OVS. These rows were already in the ablation table of the original submission, and the revised paper (Sec. 4.2) updates them to held-out calibration.

Table R4. LERF-OVS mIoU of our 2DGS variant with per-scene held-out calibration for ablations of PMC and the stages before it, and with LangSplat's SAM masks in place of ours.

SettingmIoU
No PMC47.70
No uncertainty62.86
No mask correction62.01
No uncertainty, no mask correction64.06
LangSplat (SAM level-3) masks64.42
Full69.93

Uncertainty weighting and mask correction work as a pair. Removing either one lowers mIoU, and using either one without the other is worse than using neither, so their gain comes from the combination. Uncertainty weighting downweights the instance supervision in uncertain regions, so without mask correction these regions receive only a weak training signal. Mask correction replaces the masks in these regions and gives them a stronger signal, but it relies on the uncertainty estimates to detect missed objects, and without them it discards valid masks and re-queries spurious regions. With both components in place, adding PMC raises mIoU substantially, which indicates that the Gaussians it suppresses are mostly erroneous rather than valid surfaces.

The surface loss in Sec. 4.2 shows the limit of what these stages can repair. That experiment runs our full pipeline, including uncertainty weighting and mask correction, with LangSplat's SAM masks, which are inconsistent across views. When the input masks disagree this strongly, the correction stage cannot restore enough consistency, PMC suppresses some valid surfaces, and accuracy drops (Table R4). This is why we state in Sec. 4.2 that our method needs moderately view-consistent masks. With such masks, the ablations above and the occlusion experiment show that valid geometry is largely preserved.

Qualitative results on ScanNet++

We agree with the reviewer that qualitative ScanNet++ results are valuable, given the substantial occlusion and scene complexity in ScanNet++. App. D.3 of the revised supplement adds qualitative results in the semantic segmentation setting, showing six different classes from the evaluated scenes, each rendered in isolation from the Gaussians that our method retrieves for one open-vocabulary query. All renders show little-to-no artifacts and no noticeable distorted geometry due to the effect of occlusions on the PMC loss.

Threshold calibration

LERF-OVS provides no validation set with ground-truth masks, so in the original submission the three ground-truth masks per object class used for calibration in App. A.7 came from the evaluation frames—they were not kept separate from the final evaluation. Moreover, we obtained the results in Tab. 1 from published works directly to retain their reported implementation and hyperparameters, and so we did not claibrate them. We agree with the reviewer that held-out calibration across baselines gives fairer comparison, and we have rerun calibration accordingly.

We now created our own validation set using our method + SAM to automate the process of identifying objects. We first ran training for our method to obtain instance clusters, without retrieval or semantic association. We select calibration views greedily as the training frames that are not evaluation views and in which the most object clusters are visible, so that a few views cover many objects. In these views, we prompt SAM with the bounding boxes of the instances identified by our method and manually annotate each resulting mask with its class name, omitting clusters that do not correspond to a dataset label. Using three masks per object class, we grid-search our two retrieval thresholds, τmin⁡\tau_{\min} and τconf\tau_{\mathrm{conf}}, to maximize the IoU between these masks and the masks predicted by our retrieval and reranking procedure (Sec. 3.4). Because the box prompts only locate objects that our method detects, objects missed by our method receive no calibration mask. However, the mask boundaries come from SAM rather than from any method's segmentation, so we use the same masks unchanged to calibrate the retrieval thresholds of every baseline that we run. Therefore, in Table R2 (below) all methods receive the same calibration budget, and all new LERF-OVS results in this rebuttal use these held-out masks.

We evaluate all methods in two settings. Per-scene calibrates each method's thresholds separately for each scene, and Shared uses a single threshold per method, shared across all scenes. Both settings choose the thresholds on the same held-out masks. Both of our variants remain the strongest with per-scene calibration, and even with a shared threshold they outperform every baseline configuration calibrated per scene in mIoU. The revised paper describes this procedure in Sec. 4.1 and App. A.7.

Controlled comparison with matched 2D models

The baseline results in paper Tab. 1 are taken from prior work, so each baseline uses its original masks, encoder, and threshold selection. We agree with the reviewer that matching the 2D models is a valuable ablation for separating our 3D contributions from the gains of stronger 2D models, and the revised paper includes it in Sec. 4.2 and App. C.2. We ran select baselines on LERF-OVS under the four combinations of its original or our Qwen3-VL+SAM3 masks and its original encoder or the Perception Encoder (PE). The only exception to the encoder swap is Dr. Splat, which keeps CLIP because its product-quantization codebook is pre-built on OpenCLIP ViT-B/16 embeddings of LVIS crops and does not transfer to another encoder's feature space. We report our method with both 2DGS and 3DGS. We run each baseline from its public codebase with its released default settings, and evaluate all methods, including ours, with the same evaluation code and held-out calibration. Therefore, the resulting numbers differ slightly from those reported in the original papers, but all entries in Table R2 are directly comparable.

Table R2. LERF-OVS mIoU and [email protected] with select baseline run by us under all combinations of its original or our masks and its original encoder or PE. The last row of each baseline uses its original masks and encoder. LangSplat SAM denotes the SAM masks from LangSplat's preprocessing, which VALA, OpenGaussian, LangSplat, and Dr. Splat originally use. Dr. Splat keeps CLIP because its codebook is built in CLIP space, and with our masks LangSplat uses a single feature level because they have no three-level SAM hierarchy. Per-scene calibrates each method's thresholds separately for each scene, and Shared uses one threshold per method across all scenes, both chosen on the same held-out masks. Bold marks the best result in each column.

MethodEncoderMasksPer-scene mIoUPer-scene [email protected]Shared mIoUShared [email protected]
Ours (2DGS)PEOurs69.9385.8469.0385.40
Ours (3DGS)PEOurs68.2485.2964.8582.20
VALAPEOurs57.6881.7357.2480.92
VALAPELangSplat SAM58.5384.5657.9087.30
VALACLIPOurs54.2478.2554.2478.18
VALACLIPLangSplat SAM53.5779.2353.1078.18
OpenSplat3DPEOurs53.0871.4851.1071.48
OpenSplat3DPEOpenSplat3D SAM52.0372.7451.3175.02
OpenSplat3DMasQCLIPOurs59.5382.2657.0681.07
OpenSplat3DMasQCLIPOpenSplat3D SAM59.3382.2354.0773.46
OpenGaussian†PEOurs49.4768.8049.4768.80
OpenGaussian†PELangSplat SAM47.6065.5947.6065.59
OpenGaussian†CLIPOurs46.8668.3246.8668.32
OpenGaussian†CLIPLangSplat SAM40.3254.0640.3254.06
LangSplatPEOurs47.8573.1846.2769.77
LangSplatPELangSplat SAM44.5969.4943.1968.50
LangSplatCLIPOurs42.0265.7041.8862.51
LangSplatCLIPLangSplat SAM41.8563.1541.8062.38
Dr. SplatCLIPOurs43.0862.3742.4761.44
Dr. SplatCLIPLangSplat SAM44.7864.2844.8667.27
LaGaPEOurs53.7577.7451.8474.78
LaGaPELangSplat SAM51.7981.7652.6781.24
LaGaCLIPOurs49.4773.9448.4775.07
LaGaCLIPLangSplat SAM47.1271.8246.2468.50

† OpenGaussian's inference has no language threshold, so both settings give the same result.

No combination of masks and encoder closes the gap to our method on mIoU. The strongest baseline configuration, OpenSplat3D with MasQCLIP and our masks, remains well below both of our variants. Conversely, with LangSplat's SAM masks in place of ours, our method still scores above every baseline configuration (Table R4, in our responses to Reviewers daYP and Sk48).

With a shared threshold, VALA with PE and LangSplat's masks reaches a slightly higher [email protected] than our method, but a substantially lower mIoU.

The masks and the encoder affect the baselines differently. Our masks change baseline mIoU only slightly in most settings, yet they clearly improve our own method over LangSplat's SAM masks, which indicates that the gain from our masks comes from how our 3D components use them. PE raises the mIoU of VALA, OpenGaussian, and LangSplat over CLIP, but lowers that of OpenSplat3D relative to MasQCLIP.

OpenSplat3D is the only baseline hurt by PE, with either set of masks. OpenSplat3D embeds each instance with MasQCLIP, which conditions the embedding on the instance mask, and feeds the encoder multi-scale crops whose wider zoom levels include neighboring objects. With MasQCLIP, the mask tells the encoder which object to embed. PE, like any mask-agnostic encoder, sees only the crop, so the embedding also absorbs the surrounding objects. The other baselines originally use plain CLIP, for which PE is a like-for-like upgrade, and our method avoids the issue because its isolated renders contain only the instance's Gaussians. Therefore, we measure our margin over OpenSplat3D against its MasQCLIP configurations, which remain well below both of our variants with either set of masks.

Note that LangSplat scores considerably higher in Table R2 than in paper Tab. 1 because the two results use different 3D retrieval procedures. LangSplat only produces 2D relevancy maps, so in our re-run we follow VALA's inference procedure for 3D retrieval, which keeps the Gaussians whose relevancy exceeds a threshold, and both of our settings choose this threshold on the held-out masks. The result in paper Tab. 1, taken from prior work, does not use this procedure. Our re-run with LangSplat's original CLIP encoder and SAM masks already shows most of this improvement, so it comes mainly from the retrieval procedure rather than from the 2D models.

Our results in Table R2 include our multimodal reranker, so we also ablated it together with view selection (Table R3). Without either component, our method still scores above every baseline configuration in Table R2.

Table R3. LERF-OVS mIoU of our 2DGS variant with per-scene held-out calibration when removing view selection, reranking, or both.

SettingmIoU
No view selection67.89
No reranking63.75
Neither62.07
Full69.93

Removing either component lowers mIoU, with reranking having the larger effect, and removing both gives the lowest mIoU, so each component contributes to the final accuracy. The revised paper adds these ablations to Sec. 4.2 and App. C.3.

Instance segmentation on ScanNet++

We did not originally report instance segmentation metrics because LERF-OVS, our main benchmark, contains no instance ground truth, and we evaluated ScanNet++ with the same class-level metrics for consistency. We agree with the reviewer that class-aggregated metrics can leave merged or split instances unpenalized and that instance AP is a better measure of instance quality. As requested, we have now included 3D instance AP, AP50, and AP25 for our method and OpenSplat3D (Table R1). OpenSplat3D uses its original masks and encoder. Besides open-vocabulary AP, we report class-agnostic AP, which ignores class labels and so directly penalizes merged or split instances.

In the original submission's semantic segmentation comparison on ScanNet++, all methods were run on the same 10 scenes with an identical evaluation protocol, and each scene was queried only with the classes annotated in it, following the LERF-OVS convention. However, Instance AP counts predictions of classes absent from a scene as false positives, which the original protocol cannot produce. In the revised paper (Sec. 4.2), we have updated both the semantic and the instance segmentation results to query every scene with the full class list of the official ScanNet++ instance benchmark, including classes that do not appear in the scene. We consider 83 of the top-100 semantic classes: the benchmark itself excludes 16 classes (e.g., wall, floor, ceiling), and we additionally remove "window" to focus the evaluation on objects. We compute class-agnostic AP in the same way as OpenSplat3D.

Table R1. 3D instance segmentation on 10 ScanNet++ scenes, with every scene queried with all 83 classes (OV = open-vocabulary, CA = class-agnostic).

MethodOV APOV AP50OV AP25CA APCA AP50CA AP25
OpenSplat3D13.7427.8041.3510.0823.8341.67
Ours19.2636.5248.8612.9027.2146.60

Our method outperforms OpenSplat3D on every metric, including class-agnostic AP, so our instances are more accurate even when class labels are ignored.

We added these results to Sec. 4.2 and describe the protocol in App. B. We are currently evaluating on the 50 ScanNet++ validation scenes used by OpenSplat3D and would be happy to include these results in the camera-ready version, if the reviewers agree.

We hope these clarifications and results address the reviewer's concerns.

Reviewer_nT8iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

3DISCO is a Gaussian splatting framework for open-vocabulary 3D instance segmentation from posed multi-view images. It combines uncertainty-aware instance learning and mask correction with a prototype-guided mask consistency objective to prune inconsistent Gaussians. Semantic labels are retrieved using isolated object renders, contextual crops, and multimodal reranking. Evaluation on LERF-OVS and 10 ScanNet++ scenes reports improvements over the compared methods, supported by component ablations and demonstrations of scene editing and robot manipulation.

Critical strengths
  1. Coherent and well-motivated methodology.
    The method addresses complementary sources of error when lifting noisy 2D masks into 3D instances. Its staged refinement process combines uncertainty-aware instance learning, mask correction, and geometric cleanup to recover missed objects and reduce inconsistent instance assignments. Integrating these stages with isolated and contextual views provides a coherent framework for open-vocabulary 3D understanding.

  2. Strong reported performance on LERF-OVS.
    The full model achieves 70.67% mIoU, exceeding the strongest baseline listed in Table 1 by 6.67 percentage points. The component ablations also show substantial performance drops when PMC or isolated object views are removed, supporting their effectiveness within the proposed pipeline.

Critical weaknesses
  1. Potential sensitivity of PMC to inter-object occlusion. PMC independently renders each instance and penalizes opacity outside its corresponding 2D mask. Under inter-object occlusion, however, the isolated rendering may expose genuine surfaces that are absent from the visible-region SAM mask. These surfaces could therefore receive the same penalty as incorrectly assigned Gaussians or floating artifacts. Positive supervision from other views may counteract this effect, so this concern does not establish that PMC necessarily removes valid geometry. Nevertheless, the paper does not sufficiently explain how it distinguishes occluded surfaces from erroneous geometry. An explicit discussion of occlusion handling, supported by a targeted evaluation under varying occlusion levels, would help establish the robustness of this central component.

  2. Incomplete validation of instance-level performance and insufficiently controlled comparisons. The reported improvements are substantial, but the experimental evidence leaves several important questions unresolved. Despite the claim of panoptic segmentation, the ScanNet++ evaluation covers only 10 scenes and reports class-level metrics, which do not adequately capture instance merging or splitting. An instance-level evaluation comparable to OpenSplat3D’s evaluation on 50 ScanNet++ validation scenes would provide stronger support for this claim. Furthermore, the pipeline relies on powerful pretrained components, including Qwen3-VL-assisted mask generation and multimodal reranking. Such reliance is not inherently problematic, but the current comparisons do not sufficiently disentangle improvements from these components from those attributable to the proposed instance-learning and geometric refinement methods. Finally, Appendix A.7 states that the retrieval thresholds on LERF-OVS are optimized separately for each scene using three ground-truth masks. The paper does not clarify whether these masks are excluded from the final evaluation or whether the baselines receive an equivalent calibration budget. This raises a material concern about comparability, although it is insufficient to establish test leakage. The calibration/test split and baseline calibration procedures should be explicitly documented.

Questions for authors
  1. Clarification: Evaluation fairness.
    How are the three ground-truth masks used for per-scene threshold calibration on LERF-OVS selected, and are they kept separate from the final evaluation? Do the baselines receive an equivalent calibration budget? Please also clarify whether all ScanNet++ baselines use the same scenes, candidate classes, and evaluation protocol.

  2. Clarification: Occlusion handling in PMC.
    Isolated rendering may expose genuine surfaces that are occluded in the original image and therefore absent from the SAM mask. How does PMC distinguish these surfaces from erroneous geometry outside the mask? Does it explicitly account for visibility, or rely on positive supervision from other views?

  3. Additional evidence supporting the current claims: Instance segmentation.
    Could you provide instance-level evaluation on ScanNet++, reporting AP, AP50, and AP25, with an OpenSplat3D comparison under matched settings? Results on the same 10 evaluated scenes would help determine whether the improvements extend to instance separation beyond class-level segmentation.

My assessment and score may change based on the authors’ clarifications and supporting evidence.

5 comments

We thank the reviewer for the detailed and constructive review. We address each point in order below. Tables R1–R5 are numbered consistently across our responses to all three reviewers, so the same number always refers to the same table.

Threshold calibration

LERF-OVS provides no validation set with ground-truth masks, so in the original submission the three ground-truth masks per object class used for calibration in App. A.7 came from the evaluation frames—they were not kept separate from the final evaluation. Moreover, we obtained the results in Tab. 1 from published works directly to retain their reported implementation and hyperparameters, and so we did not claibrate them. We agree with the reviewer that held-out calibration across baselines gives fairer comparison, and we have rerun calibration accordingly.

We now created our own validation set using our method + SAM to automate the process of identifying objects. We first run training for our method to obtain instance clusters, without retrieval or semantic association. We select calibration views greedily as the training frames that are not evaluation views and in which the most object clusters are visible, so that a few views cover many objects. In these views, we prompt SAM with the bounding boxes of the instances identified by our method and manually annotate each resulting mask with its class name, omitting clusters that do not correspond to a dataset label. Using three masks per object class, we grid-search our two retrieval thresholds, τmin⁡\tau_{\min} and τconf\tau_{\mathrm{conf}}, to maximize the IoU between these masks and the masks predicted by our retrieval and reranking procedure (Sec. 3.4). Because the box prompts only locate objects that our method detects, objects missed by our method receive no calibration mask. However, the mask boundaries come from SAM rather than from any method's segmentation, so we use the same masks unchanged to calibrate the retrieval thresholds of every baseline that we run. Therefore, in Table R2 (below) all methods receive the same calibration budget, and all new LERF-OVS results in this rebuttal use these held-out masks.

We evaluate all methods in two settings. Per-scene calibrates each method's thresholds separately for each scene, and Shared uses a single threshold per method, shared across all scenes. Both settings choose the thresholds on the same held-out masks. Both of our variants remain the strongest with per-scene calibration, and even with a shared threshold they outperform every baseline configuration calibrated per scene in mIoU. The revised paper describes this procedure in Sec. 4.1 and App. A.7.

Occlusion handling in PMC

Rendering an instance in isolation, with the opacity of all other Gaussians set to zero (App. A.4), can expose surfaces that are hidden in a given view and therefore absent from its visible SAM mask. Our PMC objective does not explicitly reason about visibility, and instead relies on multi-view aggregation to prevent individual occlusions from distorting an object's geometry. A surface that is hidden in some views still lies inside the object's mask in the views where it is visible, and in those views PMC's mask term rewards its opacity. Therefore, the supporting views protect the surface against the conflicting views in which it is occluded. Erroneous geometry does not receive this support. Floaters and Gaussians assigned to the wrong instance fall outside the instance's mask in most of the views where they are visible, so PMC suppresses them. Therefore, the difference between occluded surfaces and erroneous geometry lies in how many views support them.

To test this, we built a simulated scene with two objects, one partially occluding the other, and measured how well PMC preserves the occluded object's full geometry as the amount of occlusion in the training views increases. Starting from a fixed set of unoccluded views that covers the whole object, we add a growing number of views in which 30 to 50% of the object is hidden, raising the fraction of occluded training views from 0% to 70%. We then compare the object's isolated render against its ground-truth mask in unoccluded views. mIoU stays above 0.96 even with 30% occluded views, and over the rest of the range it declines only to around 0.86. Therefore, PMC preserves occluded surfaces even when most training views are occluded, although frequent occlusion costs a moderate amount of geometry. The revised paper adds this discussion and the occlusion experiment in App. C.1.

Instance segmentation on ScanNet++

We did not originally report instance segmentation metrics because LERF-OVS, our main benchmark, contains no instance ground truth, and we evaluated ScanNet++ with the same class-level metrics for consistency. We agree with the reviewer that class-aggregated metrics can leave merged or split instances unpenalized and that instance AP is a better measure of instance quality. As requested, we have now included 3D instance AP, AP50, and AP25 for our method and OpenSplat3D (Table R1). OpenSplat3D uses its original masks and encoder. Besides open-vocabulary AP, we report class-agnostic AP, which ignores class labels and so directly penalizes merged or split instances.

In the paper's semantic segmentation comparison on ScanNet++, all methods were run on the same 10 scenes with an identical evaluation protocol, and each scene was queried only with the classes annotated in it, following the LERF-OVS convention. However, Instance AP counts predictions of classes absent from a scene as false positives, which the original protocol cannot produce. In the revised paper (Sec. 4.2), we have updated both the semantic and the instance segmentation results to query every scene with the full class list of the official ScanNet++ instance benchmark, including classes that do not appear in the scene. We consider 83 of the top-100 semantic classes: the benchmakr itself excludes 16 classes (e.g., wall, floor, ceiling), and we additionally remove "window" to focus the evaluation on objects. We compute class-agnostic AP in the same way as OpenSplat3D.

Table R1. 3D instance segmentation on 10 ScanNet++ scenes, with every scene queried with all 83 classes (OV = open-vocabulary, CA = class-agnostic).

MethodOV APOV AP50OV AP25CA APCA AP50CA AP25
OpenSplat3D13.7427.8041.3510.0823.8341.67
Ours19.2636.5248.8612.9027.2146.60

Our method outperforms OpenSplat3D on every metric, including class-agnostic AP, so our instances are more accurate even when class labels are ignored.

We added these results to Sec. 4.2 and describe the protocol in App. B. We are currently evaluating on the 50 ScanNet++ validation scenes used by OpenSplat3D and would be happy to include these results in the camera-ready version, if the reviewers agree.

Controlled comparison with matched 2D models

The baseline results in paper Tab. 1 are taken from prior work, so each baseline uses its original masks, encoder, and threshold selection. We agree with the reviewer that matching the 2D models is a valuable ablation for separating our 3D contributions from the gains of stronger 2D models, and the revised paper includes it in Sec. 4.2 and App. C.2. We ran select baselines on LERF-OVS under the four combinations of its original or our Qwen3-VL+SAM3 masks and its original encoder or the Perception Encoder (PE). The only exception to the encoder swap is Dr. Splat, which keeps CLIP because its product-quantization codebook is pre-built on OpenCLIP ViT-B/16 embeddings of LVIS crops and does not transfer to another encoder's feature space. We report our method with both 2DGS and 3DGS. We run each baseline from its public codebase with its released default settings, and evaluate all methods, including ours, with the same evaluation code and held-out calibration. Therefore, the resulting numbers differ slightly from those reported in the original papers, but all entries in Table R2 are directly comparable.

Table R2. LERF-OVS mIoU and [email protected] with select baselines run by us under all combinations of its original or our masks and its original encoder or PE. The last row of each baseline uses its original masks and encoder. LangSplat SAM denotes the SAM masks from LangSplat's preprocessing, which VALA, OpenGaussian, LangSplat, and Dr. Splat originally use. Dr. Splat keeps CLIP because its codebook is built in CLIP space, and with our masks LangSplat uses a single feature level because they have no three-level SAM hierarchy. Per-scene calibrates each method's thresholds separately for each scene, and Shared uses one threshold per method across all scenes, both chosen on the same held-out masks. Bold marks the best result in each column.

MethodEncoderMasksPer-scene mIoUPer-scene [email protected]Shared mIoUShared [email protected]
Ours (2DGS)PEOurs69.9385.8469.0385.40
Ours (3DGS)PEOurs68.2485.2964.8582.20
VALAPEOurs57.6881.7357.2480.92
VALAPELangSplat SAM58.5384.5657.9087.30
VALACLIPOurs54.2478.2554.2478.18
VALACLIPLangSplat SAM53.5779.2353.1078.18
OpenSplat3DPEOurs53.0871.4851.1071.48
OpenSplat3DPEOpenSplat3D SAM52.0372.7451.3175.02
OpenSplat3DMasQCLIPOurs59.5382.2657.0681.07
OpenSplat3DMasQCLIPOpenSplat3D SAM59.3382.2354.0773.46
OpenGaussian†PEOurs49.4768.8049.4768.80
OpenGaussian†PELangSplat SAM47.6065.5947.6065.59
OpenGaussian†CLIPOurs46.8668.3246.8668.32
OpenGaussian†CLIPLangSplat SAM40.3254.0640.3254.06
LangSplatPEOurs47.8573.1846.2769.77
LangSplatPELangSplat SAM44.5969.4943.1968.50
LangSplatCLIPOurs42.0265.7041.8862.51
LangSplatCLIPLangSplat SAM41.8563.1541.8062.38
Dr. SplatCLIPOurs43.0862.3742.4761.44
Dr. SplatCLIPLangSplat SAM44.7864.2844.8667.27
LaGaPEOurs53.7577.7451.8474.78
LaGaPELangSplat SAM51.7981.7652.6781.24
LaGaCLIPOurs49.4773.9448.4775.07
LaGaCLIPLangSplat SAM47.1271.8246.2468.50

Comments on the paper

1 comment

We thank all reviewers for their detailed and constructive feedback. Several concerns were shared across reviews, and we have addressed them with new experiments and a revised paper, in which all additions and modifications are marked in blue. Tables R1–R5 are numbered consistently across our individual responses, so the same number always refers to the same table.

Threshold calibration (nT8i, daYP)

LERF-OVS provides no validation set with ground-truth masks, so our original submission calibrated the retrieval thresholds on masks from the evaluation frames. We now calibrate using SAM masks on held-out training frames excluded from evaluation, and we use the same masks to calibrate select baselines that we run. We report two settings: Per-scene, which calibrates each method's thresholds separately for each scene, and Shared, which uses a single threshold per method across all scenes. With held-out per-scene calibration, our 2DGS variant reaches 69.93 mIoU on LERF-OVS, and even with a shared threshold, both of our variants (3DGS and 2DGS) outperform every rerun baseline configuration calibrated per scene in mIoU (Table R2; Sec. 4.1, App. A.7).

Contribution of stronger 2D models (nT8i, daYP)

The baseline results in paper Tab. 1 are taken from prior work. We have now run select baselines ourselves on LERF-OVS under the four combinations of its original or our Qwen3-VL+SAM3 masks and its original encoder or the Perception Encoder—except Dr. Splat, which keeps CLIP because its codebook is built in CLIP space (Table R2; Sec. 4.2, App. C.2). We ran baselines from their public codebases, and all methods, including ours, use the same evaluation code and held-out calibration, so their numbers differ slightly from those originally reported but are directly comparable. No combination closes the gap: the strongest baseline configuration, OpenSplat3D with MasQCLIP and our masks, reaches 59.53 mIoU, well below both of our variants (69.93 with 2DGS, 68.24 with 3DGS). Our method also stays above every baseline configuration with LangSplat's SAM masks in place of ours (Table R4) and without view selection and label reranking (62.07 mIoU, Table R3).

Instance-level evaluation (nT8i, daYP, Sk48)

As requested by the reviewers, we now report 3D instance AP, AP50, and AP25 on the 10 ScanNet++ scenes compared to OpenSplat3D, both open-vocabulary and class-agnostic (Table R1; Sec. 4.2, App. B). Our method outperforms OpenSplat3D on every metric, including class-agnostic AP, which ignores class labels and directly penalizes merged or split instances. For both semantic and instance segmentation, every scene is now queried with the 83 object classes of the ScanNet++ top-100 instance benchmark (all except window), including classes that do not appear in the scene, so that predictions of absent classes count as false positives. We are currently evaluating on the 50 ScanNet++ validation scenes used by OpenSplat3D and would be happy to include these results in the camera-ready version, if the reviewers agree.

Robustness of PMC to occlusion and mask errors (nT8i, daYP, Sk48)

PMC relies on multi-view aggregation rather than explicit visibility reasoning: an occluded surface is supported by the views in which it is visible, whereas floaters and misassigned Gaussians fall outside the instance's mask in most views. In a controlled occlusion experiment, mIoU stays above 0.96 with 30% occluded training views and declines only to around 0.86 as the fraction of occluded views grows to 70% (App. C.1). Ablations of PMC, uncertainty weighting, and mask correction show that, with moderately view-consistent masks, the Gaussians PMC suppresses are mostly erroneous rather than valid surfaces (Table R4; Sec. 4.2).

Additional results

  • View selection and reranking (Sk48): both components contribute, with label reranking having the larger effect (Table R3; Sec. 4.2, App. C.3).
  • End-to-end runtime (Sk48): our pipeline is the fastest training-based method, and with the baselines' SAM+CLIP preprocessing it has the lowest total runtime of all compared methods (Table R5; App. A.9).
  • Qualitative ScanNet++ results (daYP): App. D.3 shows open-vocabulary segmentation renders of six different objects from the evaluated scenes.

We hope these results address the reviewers' concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.