- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces 3D-GIA, a framework for open-vocabulary 3D segmentation with 3D Gaussian Splatting. It proposes a conflict-constrained association framework that generates persistent scene-global identities from unordered masks, together with a hierarchy-aware design that separates whole objects from parts. The resulting identities are used to supervise identity and language learning and to guide geometry refinement, aiming to improve cross-view mask consistency and fine-grained part segmentation.
Critical strengths
The paper attributes the main bottleneck of 3DGS-based segmentation to the lack of a persistent cross-view unit of correspondence, and proposes a conflict-constrained association framework to generate persistent scene-global identities from unordered SAM2 masks. This design is reasonable.
Critical weaknesses
- The proposed pipeline is quite complicated and relies on a large number of hand-crafted rules and hyper-parameters, the sensitivity analysis of the hyper-parameter selection is limited. And according to Appendix A.7 some parameters need to be tuned on held-out training views. Therefore, it is unclear if these settings and parameter selections can be transfer to new datasets or scenes without re-tuning.
- The proposed method includes hierarchy-aware designs and claims better handling of fine-grained parts. However, the evaluated benchmarks, i.e. LERF-Mask, 3DOVS and ScanNet, contain almost no object part-level queries. It would be better to evaluate the proposed method on LERF-OVS dataset (used by LangSplat), which contains more object part-level queries, and report whole-object and part queries separately.
- In ablation study table 5, w/o hierarchy module cause the mIoU performance on LERF-Mask drops to 68.8, which is a bit surprising because LERF-Mask almost contains no part queries. The queries that could possibly be affected could be eggs/pork belly/wavy noodles inside the ramen. It would be better if the authors can provide a per-query breakdown of this ablation. Moreover, 68.8 is lower than gaussian-grouping (72.8), which does not use any hierarchy and simply rely on DEVA to do the association, without conflict-constrained association. This post question on the effectiveness of the proposed conflict-constrained association, as gaussian-grouping (no hierarchy + DEVA) performs better than the ablated variant (no hierarchy + conflict-constrained association). A controlled comparison using the same masks for both association strategies would help clarify this.
Questions for authors
- See weakness 2. Can the authors provide comparison on LERF-OVS benchmark and report whole-object and part queries separately?
- See weakness 3. Can the authors provide a controlled comparison using the same masks (i.e. the one used by gaussian-grouping) and compare different association strategies.
5 comments
Part 2: Responses to Critical Weaknesses
Response to W1
Preset development and cross-scene reuse are separate stages. The LERF-Mask SAM2 preset is developed on figurines training views and then reused for ramen and teatime. 3D-OVS uses another preset shared across its five scenes. This reuse is clarified in Sec. 4.1 (Implementation Details). Appendix A.8 describes the training-view development protocol and candidate settings; Appendix A.9/Table 10 lists the final coarse/fine SAM2 parameters. Once selected, the presets and association settings are fixed across scenes within each benchmark.
Across seven ScanNet scenes, we vary each of four association thresholds while fixing the other three. B3-F1 changes by only -0.48 to +0.26 points from the fixed setting's 76.94% (Appendix C.1/Table 12). This checks threshold sensitivity without changing masks or the other association settings.
Camera extent and neighborhood statistics provide scale and bandwidth adaptation (Appendix A.6), allowing common settings to accommodate different scene scales. These results support fixed-setting reuse across scenes and stability within the tested threshold ranges.
Response to W2 See Response to Q1 on LERF-OVS whole-object/part evaluation.
Response to W3
Hierarchy improves observations, association, and learning before a query is issued, so whole-object queries also benefit. Secs. 3.2–3.4 use fine regions and parent residuals to learn distinct, consistent identities for whole-object coverage and separation from neighboring instances; this does not require the query to name a part. The original w/o hierarchy module replaces hierarchical SAM2 inputs with single-level masks and removes hierarchy processing and coarse/fine supervision together. Thus, the drop from 88.0 to 68.8 measures a joint change to the hierarchy pipeline, rather than removal of a part-query function alone.
Following suggestions from the other reviewers, we have added these finer-grained ablations to Sec. 4.4/Table 5, with scene-wise results and factor definitions in Appendix G.1/Table 15. Hierarchical-input rows retain the same raw SAM2 coarse/fine masks while changing pre-association processing, parent-child association, or the whole hierarchy loss:
| Setting | Masks | ramen | figurines | teatime | Mean (mIoU/mBIoU, %) |
|---|---|---|---|---|---|
| Full model | Hierarchical | 88.8/78.5 | 89.4/87.1 | 85.9/80.7 | 88.0/82.1 |
| w/o decomposition | Hierarchical | 88.3/78.5 | 86.1/84.0 | 80.8/75.8 | 85.1/79.4 |
| w/o parent-child constraints | Hierarchical | 86.7/77.1 | 72.6/70.8 | 74.3/68.5 | 77.9/72.1 |
| w/o hierarchy losses | Hierarchical | 87.3/77.1 | 86.1/84.0 | 81.1/75.7 | 84.8/79.0 |
| w/o hierarchy module (Table 5) | Single-level | 79.5/69.0 | 59.3/56.8 | 67.6/60.0 | 68.8/61.9 |
Decomposition separates parent residuals from child observations; parent-child association uses the retained relations to guide matches and prevent incompatible merges. The loss ablation removes the whole L_hier. Each single-factor ablation lowers mean performance, showing benefits from hierarchy processing and supervision beyond mask inputs.
The complete per-query comparison is below and in Appendix E/Table 14.
| Scene | Query | Full IoU (%) | w/o hierarchy module IoU (%) |
|---|---|---|---|
| ramen | chopsticks | 77.16 | 61.52 |
| ramen | egg | 95.94 | 90.87 |
| ramen | glass of water | 83.33 | 42.89 |
| ramen | pork belly | 95.14 | 96.83 |
| ramen | wavy noodles in bowl | 85.78 | 91.01 |
| ramen | yellow bowl | 95.26 | 94.09 |
| figurines | green apple | 91.99 | 95.31 |
| figurines | green toy chair | 91.32 | 0.54 |
| figurines | old camera | 84.84 | 71.33 |
| figurines | porcelain hand | 84.42 | 58.61 |
| figurines | red apple | 90.68 | 94.88 |
| figurines | red toy chair | 89.64 | 0.10 |
| figurines | rubber duck with red hat | 93.02 | 93.98 |
| teatime | apple | 90.97 | 94.07 |
| teatime | bag of cookies | 94.58 | 87.86 |
| teatime | coffee mug | 77.15 | 72.34 |
| teatime | cookies on a plate | 89.69 | 51.23 |
| teatime | paper napkin | 91.42 | 64.89 |
| teatime | plate | 89.81 | 89.14 |
| teatime | sheep | 88.80 | 72.83 |
| teatime | spoon handle | 53.35 | 0.00 |
| teatime | stuffed bear | 90.46 | 60.54 |
| teatime | tea in a glass | 93.68 | 83.02 |
The per-query results show that the hierarchy ablation affects whole-object queries as well as parts. Together with the fixed-mask controls, this supports the hierarchy's role in whole-object coverage and identity separation, explaining why its removal can cause a substantial drop even on a benchmark with few part queries.
For the same-mask association comparison, see Response to Q2.
Dear Reviewer Sghb,
We hope the above clarifications and the additional experiments in the revised draft sufficiently addressed your concerns. If you are satisfied, we kindly request you to consider updating the score to reflect the newly added results and discussion. We remain committed to addressing any remaining points you may have during the discussion phase.
Regards from the authors.
Part 1: Responses to Questions for Authors
Thank you for recognizing the motivation for persistent cross-view correspondence. We report new whole-object/part evaluation results and clarify the hierarchy and parameter-development protocol.
Response to Q1
We evaluated 3D-GIA on all four LERF-OVS scenes (67 scene-query entries; 208 annotated query-view pairs). For this additional evaluation, we freeze the trained Gaussian and identity fields and fuse text-relevance maps from the learned language field, training-view CLIP ViT-B/16 identity prototypes, and training-view CLIP ViT-L/14@336 identity prototypes. The three maps are min-max normalized individually and averaged equally. We aggregate 100 SAM candidates per query-view, each prompted with three positive points, using SAM-predicted IoU weights and Otsu binarization. This text-guided semantic-fusion/SAM inference rule is shared across all scenes and queries, without Grounded-SAM identity selection or optional hierarchy expansion.
Since the dataset does not provide whole-object/part labels, we manually annotate 55 whole-object and 12 part queries as a supplementary split, shared by all four methods. Parts include named components, attached accessories, and dish ingredients.
Scores average query-view IoUs within each scene/group, then average eligible scenes equally. Overall and whole-object results cover all four scenes; part results cover ramen, figurines, and teatime. Waldo Kitchen has no part annotations and is not assigned a zero part score. These results are included in Sec. 4.6/Table 7, with inference and grouping details in Appendix G.3.
Response to Q2
We ran the requested control using shared pre-association SAM observations from GG's native pipeline. Both branches share the 8K Gaussian initialization, applicable downstream losses, unknown-pixel policy, training schedule, and frozen evaluation; DEVA retains its native sequence processing. Unobserved pixels are ignored rather than supervised as background, so sparse detections do not become artificial negative supervision. Single-level masks provide no parent-child relations, so neither branch uses hierarchy-specific association or supervision.
This common-input control tests association without hierarchical inputs. We report it in Sec. 4.7/Table 8(b), with the protocol in Appendix G.2.