- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
3DISCO addresses open-vocabulary 3D instance segmentation from posed images. Qwen3-VL and SAM3 provide initial object masks. The method learns Gaussian instance features through a contrastive objective with uncertainty weighting. The resulting representation guides subsequent mask correction. A prototype-guided mask consistency objective suppresses Gaussians whose projections conflict with their assigned masks. Clustering extracts instances for semantic labeling. Recognition combines isolated object renders with contextual crops through view selection and multimodal reranking. The authors report improved retrieval and semantic segmentation on LERF-OVS and ScanNet++. Component ablations examine the proposed mechanisms. Scene editing and robot manipulation demonstrations illustrate downstream applications.
Critical strengths
- The PMC objective directly connects instance assignments with projected mask consistency. Table 3 supports its contribution to retrieval performance. Figure 3 also shows cleaner object movement with fewer floating artifacts. These results provide meaningful evidence that refining the geometry associated with an instance benefits recognition and editing. This is practically consequential because extracted objects must retain coherent geometry when manipulated independently.
- Uncertainty estimates influence both feature learning and mask correction. The learned representation guides new SAM prompts while replacement masks provide fresh supervision. The ablations in Table 3 support the usefulness of this adaptive process. The design offers a practical way to revise imperfect image masks using information accumulated across views. It addresses a relevant limitation of learning 3D instances from a fixed set of noisy masks.
Critical weaknesses
- The paper presents independent 3D instance recovery as a central contribution. The evaluation in Section 4.1 mainly measures language retrieval and semantic segmentation. The ScanNet++ metric aggregates predictions by class. Merging objects of the same class or splitting one object can therefore remain unpenalized when their combined region is unchanged. The editing examples demonstrate instance separation in selected cases. They do not establish overall instance quality. This leaves the central instance modeling claim insufficiently validated.
- Uncertainty is used to address mask noise. The subsequent PMC stage freezes instance features and suppresses Gaussians that conflict with the masks. Such conflicts can reflect incorrect instance assignments or residual supervision errors. The objective in Section 3.3 does not explicitly distinguish these sources. Section 4.2 reports surface loss when an alternative mask source is used. It remains unclear how reliably the preceding correction stage protects valid geometry during pruning. This limits the evidence for robustness to remaining mask errors.
- The semantic module presents view selection and label reranking as important contributions. The analysis in Table 3 supports the complementary value of isolated renders and contextual crops. It does not separately establish the contributions of view selection and reranking. The extent to which the semantic gains arise from these proposed mechanisms therefore remains unclear. This limits the evidence for the independent value of the retrieval and reranking strategy.
- The introduction claims improved overall processing efficiency. Its argument emphasizes fewer candidate masks and fewer views used for recognition. These savings concern individual stages of a pipeline that also requires multiple model calls and optimization stages. The paper reports an overall runtime but does not establish a comparable cost advantage over existing methods. The relationship between complete processing cost and performance gains therefore remains insufficiently characterized.
Questions for authors
None.
3 comments
Occlusion handling in PMC
Rendering an instance in isolation, with the opacity of all other Gaussians set to zero (App. A.4), can expose surfaces that are hidden in a given view and therefore absent from its visible SAM mask. Our PMC objective does not explicitly reason about visibility, and instead relies on multi-view aggregation to prevent individual occlusions from distorting an object's geometry. A surface that is hidden in some views still lies inside the object's mask in the views where it is visible, and in those views PMC's mask term rewards its opacity. Therefore, the supporting views protect the surface against the conflicting views in which it is occluded. Erroneous geometry does not receive this support. Floaters and Gaussians assigned to the wrong instance fall outside the instance's mask in most of the views where they are visible, so PMC suppresses them. Therefore, the difference between occluded surfaces and erroneous geometry lies in how many views support them.
To test this, we built a simulated scene with two objects, one partially occluding the other, and measured how well PMC preserves the occluded object's full geometry as the amount of occlusion in the training views increases. Starting from a fixed set of unoccluded views that covers the whole object, we add a growing number of views in which 30 to 50% of the object is hidden, raising the fraction of occluded training views from 0% to 70%. We then compare the object's isolated render against its ground-truth mask in unoccluded views. mIoU stays above 0.96 even with 30% occluded views, and over the rest of the range it declines only to around 0.86. Therefore, PMC preserves occluded surfaces even when most training views are occluded, although frequent occlusion costs a moderate amount of geometry. The revised paper adds this discussion and the occlusion experiment in App. C.1.
View selection and reranking
View selection was designed primarily for efficiency. Without it, label reranking has to run on every view of every object, which is expensive, whereas view selection restricts it to the top most informative views of each object. Therefore, the paper assessed the accuracy of the semantic module through its two view types, isolated renders and contextual crops (paper Tab. 3), rather than ablating view selection separately. We agree with the reviewer that separating view selection from reranking is valuable, and we have now ablated both on LERF-OVS (Table R3).
Table R3. LERF-OVS mIoU of our 2DGS variant with per-scene held-out calibration when removing view selection, reranking, or both.
| Setting | mIoU |
|---|---|
| No view selection | 67.89 |
| No reranking | 63.75 |
| Neither | 62.07 |
| Full | 69.93 |
Removing either component lowers mIoU, with reranking having the larger effect, and removing both gives the lowest mIoU, so each component contributes to the final accuracy. Beyond reducing the number of views used for recognition, view selection also improves accuracy, since it restricts the job of the reranker. The revised paper adds these ablations to Sec. 4.2 and App. C.3.
End-to-end processing cost
The introduction based its efficiency claim on two stages of our pipeline: the smaller number of candidate masks and the smaller number of views used for recognition. We agree that the complete cost is the relevant comparison, so Table R5 reports the average per-scene runtime of each full pipeline, including all model calls—we now include this in App. A.9 of the revised paper. The table separates training-based methods, which optimize per-scene features, from training-free methods.
Table R5. Average per-scene runtime in minutes, grouped by whether a method trains per-scene features. Baseline masks use LangSplat's SAM+CLIP preprocessing.
| Method | Type | Masks | Train + Eval | Total |
|---|---|---|---|---|
| LangSplat | Training-based | 148.88 | 75.48 | 224.36 |
| OpenGaussian | Training-based | 148.88 | 53.78 | 202.66 |
| Ours | Training-based | 3.47 | 30.60 | 34.07 |
| Dr. Splat | Training-free | 148.88 | 21.10 | 169.98 |
| VALA | Training-free | 148.88 | 9.05 | 157.93 |
Among training-based methods, ours has the fastest training and evaluation, so it would remain the fastest training-based pipeline end to end even if the baselines used our masks. The training-free methods Dr. Splat and VALA are faster in training and evaluation because they assign 2D features to Gaussians by weighted multi-view aggregation, with no gradient-based optimization and no instance learning. Dr. Splat additionally relies on a product-quantization codebook that is built once offline from CLIP embeddings of LVIS object crops. With the SAM+CLIP preprocessing used by the baselines, our full pipeline has the lowest total runtime of all compared methods. If the training-free methods used our masks, they would also be faster than ours end to end, but this speed comes at a cost in accuracy: both score clearly below ours in LERF-OVS mIoU with every combination of masks and encoder (Table R2, in our responses to Reviewers nT8i and daYP).
We hope these clarifications and results address the reviewer's concerns.
We thank the reviewer for the detailed and constructive review. We address each point in order below. Tables R1–R5 are numbered consistently across our responses to all three reviewers, so the same number always refers to the same table.
Instance segmentation on ScanNet++
We did not originally report instance segmentation metrics because LERF-OVS, our main benchmark, contains no instance ground truth, and we evaluated ScanNet++ with the same class-level metrics for consistency. We agree with the reviewer that class-aggregated metrics can leave merged or split instances unpenalized and that instance AP is a better measure of instance quality. As requested, we have now included 3D instance AP, AP50, and AP25 for our method and OpenSplat3D (Table R1). OpenSplat3D uses its original masks and encoder. Besides open-vocabulary AP, we report class-agnostic AP, which ignores class labels and so directly penalizes merged or split instances.
In the original submission's semantic segmentation comparison on ScanNet++, all methods were run on the same 10 scenes with an identical evaluation protocol, and each scene was queried only with the classes annotated in it, following the LERF-OVS convention. However, Instance AP counts predictions of classes absent from a scene as false positives, which the original protocol cannot produce. In the revised paper (Sec. 4.2), we have updated both the semantic and the instance segmentation results to query every scene with the full class list of the official ScanNet++ instance benchmark, including classes that do not appear in the scene. We consider 83 of the top-100 semantic classes: the benchmark itself excludes 16 classes (e.g., wall, floor, ceiling), and we additionally remove "window" to focus the evaluation on objects. We compute class-agnostic AP in the same way as OpenSplat3D.
Table R1. 3D instance segmentation on 10 ScanNet++ scenes, with every scene queried with all 83 classes (OV = open-vocabulary, CA = class-agnostic).
Our method outperforms OpenSplat3D on every metric, including class-agnostic AP, so our instances are more accurate even when class labels are ignored.
We added these results to Sec. 4.2 and describe the protocol in App. B. We are currently evaluating on the 50 ScanNet++ validation scenes used by OpenSplat3D and would be happy to include these results in the camera-ready version, if the reviewers agree.
Residual mask errors and PMC
The PMC objective alone does not distinguish incorrect instance assignments from residual mask errors. Our pipeline handles residual mask errors before PMC through uncertainty weighting and mask correction, and Table R4 reports the corresponding ablations on LERF-OVS. These rows were already in the ablation table of the original submission, and the revised paper (Sec. 4.2) updates them to held-out calibration.
Table R4. LERF-OVS mIoU of our 2DGS variant with per-scene held-out calibration for ablations of PMC and the stages before it, and with LangSplat's SAM masks in place of ours.
Uncertainty weighting and mask correction work as a pair. Removing either one lowers mIoU, and using either one without the other is worse than using neither, so their gain comes from the combination. Uncertainty weighting downweights the instance supervision in uncertain regions, so without mask correction these regions receive only a weak training signal. Mask correction replaces the masks in these regions and gives them a stronger signal, but it relies on the uncertainty estimates to detect missed objects, and without them it discards valid masks and re-queries spurious regions. With both components in place, adding PMC raises mIoU substantially, which indicates that the Gaussians it suppresses are mostly erroneous rather than valid surfaces.
The surface loss in Sec. 4.2 shows the limit of what these stages can repair. That experiment runs our full pipeline, including uncertainty weighting and mask correction, with LangSplat's SAM masks, which are inconsistent across views. When the input masks disagree this strongly, the correction stage cannot restore enough consistency, PMC suppresses some valid surfaces, and accuracy drops (Table R4). This is why we state in Sec. 4.2 that our method needs moderately view-consistent masks. With such masks, the ablations above and the occlusion experiment show that valid geometry is largely preserved.