acceptodds
Under review as a conference paper at ICLR 2027

NestSplat: Hierarchical Relational Reasoning Segmentation in 3D Gaussian Splatting

Abstract

Language-guided segmentation in 3D Gaussian Splatting (3DGS) must resolve spatial and relational constraints, yet existing benchmarks emphasize single targets and shallow relations. We introduce Nest-LERF and Nest-ScanNet under zero-target, single-target, and grouped-multiple-target settings. Nest-LERF covers nesting, stacking, part–whole, and mixed relations, while room-scale Nest-ScanNet covers nesting, part–whole, spatial-directional, and mixed relations. We also propose NestSplat, a training-free framework that derives a scene vocabulary from training views, lifts 2D masks into language-anchored 3D supports, and selects query targets through bounded vision–language decisions with geometric context. Without per-scene training, NestSplat achieves 34.16% and 27.42% mIoU on Nest-LERF and Nest-ScanNet, respectively, and transfers strongly to Ref-LERF and Causal-LERF.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Language-Guided Reasoning Segmentation, Hierarchical Instance Graph, Training-Free, Zero-Shot Language Grounding, Geometric Instance Registry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wUNYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NestSplat addresses segmentation of reconstructed 3D Gaussian scenes from natural language instructions. The paper introduces Nest-LERF and Nest-ScanNet to evaluate relational queries with absent targets and single or grouped targets. The method builds an object vocabulary from scene observations without accessing evaluation queries. It lifts 2D masks into fixed Gaussian supports and computes geometric relations using connected components. A multimodal language model performs target selection with bounded calls. Each query returns no support or one support that may contain multiple matching instances. The method requires no additional semantic training for each scene. The authors report improved segmentation on the proposed benchmarks and additional validation on existing referring and reasoning datasets.

Critical strengths
  1. The benchmarks explicitly include absent targets and groups of matching objects. This extends mask evaluation beyond the assumption of one referred object. The annotation protocol separates relation categories from target cardinality and includes independent verification and adjudication (Appendix A). These choices provide a useful setting for studying target existence and relational disambiguation.
  2. Scene preparation produces reusable supports without accessing evaluation queries or learning additional semantic features. Representing connected components separately addresses the geometric distortion caused by a single enclosing box. The controlled analysis supports the usefulness of this design on the evaluated scene (Table 9). Together with bounded model calls, these choices provide a practical pipeline for grounding language in reconstructed scenes.
Critical weaknesses
  1. The router selects a complete precomputed support and cannot select its geometric components independently (Sec. 3.3). If several instances share a support and no finer candidate exists, then relational reasoning cannot isolate the required instance. This constrains the stated task of single instance disambiguation. Its practical impact depends on whether support construction preserves the distinctions required by the queries. That coverage remains unclear from the provided materials.
  2. Geometric hints summarize the immediate neighborhood of a single landmark (Sec. 3.3). The inventory selector retains access to all nodes and the results of visual parsing. It remains unclear how the local geometric summary supplies the evidence needed for the full constraints in nested and mixed queries. The component analysis supports the usefulness of geometric hints but does not fully explain their role in composing relations. This leaves the connection between bounded routing and hierarchical reasoning insufficiently established.
  3. The fallback rule can restore an earlier candidate when inventory selection returns no valid node. Empty masks can also result from insufficient projection coverage (Appendices B.4 and A.5). The provided description does not clearly explain how intentional absence decisions interact with fallback. It is therefore difficult to distinguish deliberate rejection from other causes of empty predictions. This limits their interpretation as evidence that the system recognized the absence of a valid target.
  4. The benchmark includes appearance attributes and functional cues alongside relational constraints (Appendix A.1). The annotation protocol checks target validity and geometric relations. However, the provided description does not establish how often relational composition is necessary to distinguish the target. Final mask accuracy and scores grouped by relation type cannot resolve this distinction on their own. This limits how specifically the results support the benchmark’s motivation of evaluating hierarchical reasoning.
Questions for authors

None.

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal to Reviewer wUNY

Dear Reviewer wUNY,

Thank you for the time and care you devoted to reviewing our paper and for your valuable feedback. As the discussion period will close on November 18, we would be grateful if you could take a moment to review our rebuttal and follow-up comments.

Please let us know if any concerns remain unresolved or if any part of our response would benefit from further clarification. We would greatly appreciate your feedback and would be happy to address any remaining questions before the discussion closes.

Thank you again for your time and consideration.

Reviewer_8vh7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a training-free framework for language-guided 3D Gaussian segmentation. The main idea is to construct a geometry-aware scene graph that captures hierarchical and spatial relationships between objects, enabling more accurate relational reasoning and target segmentation. The authors also introduce two benchmarks for evaluating such capabilities. Experiments show clear improvements over existing baselines, and the ablation studies provide some evidence for the usefulness of the graph-based design.

Critical strengths
  1. The problem itself is well-motivated. Language-guided 3D segmentation requires resolving containment, support, and part–whole relations, including cases where no object satisfies the query. The authors introduce two benchmarks to provide the useful evaluation setting for this direction.
  2. The idea of component-aware geometry is interesting. It avoids misleading spatial relations caused by enclosing disconnected regions in a single bounding box, and the ablations show that it improves segmentation performance.
Critical weaknesses

The ablation studies suggest that the relational graph and component-aware geometry are useful, but they are only evaluated on the figurines scene. Since these are the main technical contributions of the paper, I would like to know whether the same improvements hold across different scenes and relation types, particularly on Nest-ScanNet, where the overall performance gains are substantial.

Questions for authors
  1. The proposed component-aware representation is interesting, but how much does it help across different scenes, beyond the figurines scene used in the ablation?

  2. How do the authors ensure that the mixed-relation queries actually require reasoning over multiple relationships, rather than being solvable using simpler object or spatial cues?

  3. Do the benchmarks include queries targeting only some instances within one support? For example, two cups share a support, but only the cup to the left of a plate is requested.If so, how does the single-node output handle this.

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal to Reviewer 8vh7

Dear Reviewer 8vh7,

Thank you for the time and care you devoted to reviewing our paper and for your valuable feedback. As the discussion period will close on November 18, we would be grateful if you could take a moment to review our rebuttal and follow-up comments.

Please let us know if any concerns remain unresolved or if any part of our response would benefit from further clarification. We would greatly appreciate your feedback and would be happy to address any remaining questions before the discussion closes.

Thank you again for your time and consideration.

Reviewer_CXkmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NestSplat combines scene nouns, SAM 3-mask elevation, component geometry and bounded VLM invocations for relational command segmentation in 3D Gaussian scenes. The authors propose two new benchmarks, reporting mIoU values of 34.16 per cent and 27.42 per cent respectively. The system’s results are valuable, but the contribution of the geometric mechanism and the limits of its representational capacity require further explanation.

Critical strengths
  1. Covers zero-target, single-target and group-by-description targets, combining 2D pixel and 3D instance evaluation, and outlines the annotation review and aggregation rules.
  2. Includes three repeated ablation studies using a shared front-end, as well as a comparison with an updated version of the CausalSplat query model.
  3. Negative results are disclosed candidly. The authors explicitly report that only 7 out of 26 zero-target experiments were successful, and they limit the scope of evidence presented for mobile scenarios.
Critical weaknesses
  1. Core ablation covers only 46 queries on ‘figurines’: 33.34 per cent for the full model and 31.27 per cent for the image-free model. This supports local gains, but is insufficient to prove that gains in other scenarios and on ScanNet also stem from the geometric design. We recommend introducing limited matching control or narrowing the conclusions regarding the mechanism. (Table 9, A.7–A.10)
  2. There is a lack of complete prompt and response structures, rules for binding names to landmarks, and explanations regarding the LERF metric and vertical alignment. Details regarding existing models, thresholds and caching should be acknowledged, but the workflow cannot yet be uniquely determined. (pp. 3–6, 17–18)
  3. When the selector fails to return a valid node, it may fall back to earlier matches; a distinction has not yet been made between active ‘no target’ cases and parsing failures. State priorities and logs should be supplemented; it cannot be concluded that backtracking caused all 19 zero-target failures. (Page 18, B.4)
  4. InstanceGaussian achieves 6.1373 per cent in Table A.2 when weighted by query count, yet 3.55 per cent in Table A.4. This cannot be explained by macro-average discrepancies or rounding; it should be regenerated from the same query-by-query records; this alone is insufficient to overturn the main ranking.
Questions for authors

Details in Weaknesses

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal to Reviewer CXkm

Dear Reviewer CXkm,

Thank you for the time and care you devoted to reviewing our paper and for your valuable feedback. As the discussion period will close on November 18, we would be grateful if you could take a moment to review our rebuttal and follow-up comments.

Please let us know if any concerns remain unresolved or if any part of our response would benefit from further clarification. We would greatly appreciate your feedback and would be happy to address any remaining questions before the discussion closes.

Thank you again for your time and consideration.

Comments on the paper

1 comment
Area_Chair_k87nOpenReview ↗

Authors' rebuttals

Dear Reviewers,

Authors have worked hard to address your concerns. Please check the rebuttals, and I encourage you to discuss with the authors to address any open points.

Regards, AC for this article.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.