acceptodds
Under review as a conference paper at ICLR 2027

Gaussians All the Way Down: Hierarchical Gaussian Mixture Models for Open-Vocab Gaussian Splatting

Abstract

3D Gaussian Splatting (3DGS) has revolutionized high-fidelity scene synthesis. Yet, seamlessly integrating semantic knowledge in 3DGS remains a computational challenge. Existing approaches either rely on expensive, per-scene learned compression for real-time inference or avoid training entirely and work with computationally expensive scene-wide unordered feature sets. We present a framework that directly lifts 2D features into 3D Gaussians and organizes them via hierarchical Gaussian Mixture Models (GMM), clustering jointly over spatial and semantic dimensions through parallelized expectation-maximization (EM). This produces a traversable tree whose cluster centroids act as a lightweight, non-learned codebook: lifted and generated in under 10 minutes, using on average 23 less memory than storing raw features per-Gaussian, and enabling sub-millisecond semantic queries. This method is demonstrated across four semantic datasets ranging from small objects to large-scale rooms, achieving state-of-the-art semantic segmentation without any learned feature compression or per-scene training. We demonstrate the robustness of the pipeline with an extensive ablation across six different text-aligned feature backbones, multiple clustering backends, covariance constraints, and query methodologies. Combined we present an efficient hierarchical GMM data structure for joint spatial-semantic clustering; state-of-the-art open-vocabulary segmentation with an order-of-magnitude smaller on-disk footprint than raw-feature methods; and sub-millisecond query times on par with learned, per-scene optimized methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Semantic Segmentation, Open-Vocab

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1nRLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper lifts pretrained 2D semantic features onto an existing Gaussian scene and recursively clusters joint spatial and semantic representations. Cluster centroids provide a compact semantic representation for open-vocabulary queries.

Critical strengths
  • Compressing lifted features into spatial-semantic cluster representatives addresses a practical bottleneck. The paper reports substantial storage savings and evaluates several scene scales.
  • The main claimed benefit is a favorable combination of segmentation quality, compact storage and fast queries without a learned per-scene feature compressor.
Critical weaknesses
  • The adopted 2D-to-3D optimization-based lifting pipeline has been extensively explored in prior work, limiting the overall contribution.
  • It would be helpful to include comparisons with more recent works [1, 2]. [1] Cos3D: Collaborative Open-Vocabulary 3D Segmentation. NeurIPS 2025. [2] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR 2025.
Questions for authors

Please refer to the weakness part.

Reviewer_yf5eOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a pipeline for open-vocabulary segmentation with 3D Gaussian Splatting. The method directly lifts CLIP features onto 3D Gaussians and then clusters them. Unlike prior work, it organizes the Gaussians into a tree structure using hierarchical Gaussian Mixture Models (GMMs).

Critical strengths
  • The paper is easy to read.
  • The method achieves higher mIoU and mAcc than prior methods.
  • The proposed tree structure distinguishes the method from existing approaches.
Critical weaknesses
  • Unexplained CLIP vs. CLIP+SAM result. In Table 6, CLIP alone outperforms CLIP+SAM on LERF-OVS, but the paper does not explain why. Prior work (e.g., LangSplat) showed that SAM is a key component, so it is confusing that this method reaches 76.41 mIoU without SAM, far higher than prior methods. The authors should provide reasoning for this result.
  • Contribution already present in prior work. The paper claims that clustering enables memory compression, but this technique already exists in prior work [1, 2]. Table 5 does not include comparisons with these fast and memory-efficient methods.
  • Marginal benefit of the GMM. In Table 10, the mIoU gap between k-means and GMM is marginal (1.1 percentage points), even though the GMM is the main component of the paper.
  • Lack of qualitative comparison. The paper does not include qualitative comparisons with baselines. Such comparisons are important for understanding in which aspects the method achieves higher mIoU.

[1] Training-Free Hierarchical Scene Understanding for Gaussian Splatting with Superpoint Graphs (ACM MM 2025)

[2] LightSplat: Fast and Memory-Efficient Open-Vocabulary 3D Scene Understanding in Five Seconds (CVPR 2026)

Questions for authors
  1. What is the setting for the CLIP-only variant in Table 6? Are the features obtained by cropping the image? Why does CLIP alone outperform SAM+CLIP on LERF-OVS?
  2. How does the method compare qualitatively with other methods? In what aspects does it achieve higher performance?
  3. Were the hyperparameters tuned on a separate validation split? If they were tuned on the full test data, could the authors report results where hyperparameters are tuned on a validation split and then evaluated on a held-out test split?
Reviewer_bWSoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free method for semantic 3D Gaussian Splatting. It uses hierarchical Gaussian mixtures to group Gaussians by spatial and semantic similarity, allowing fast open-vocabulary queries without learned feature compression.

Experiments across four datasets show strong segmentation results, around 23X less feature storage, and sub-millisecond query times. The authors also provide extensive ablations on clustering and feature representations.

Critical strengths
  1. I appreciate the comparisons with simple clustering methods in Table 9 and the detailed ablations. They help show that the hierarchical structure contributes to the results.

  2. The segmentation results are impressive across different scene types. Achieving this without per-scene training, while substantially reducing storage and query time, is a meaningful contribution.

Critical weaknesses
  1. Table 8 reports 76.87% mIoU for the global recipe and 82.49% for per-scene tuning. However, averaging the results in Tables 1–3 across their 13 scenes gives roughly 82.92%, even higher than the reported per-scene ceiling. I also noticed that Appendix A.1.1 describes dynamic thresholding as the default, while Table 7 and Appendix A.2.9 indicate Otsu is the default. These may be reporting or configuration differences, but they need clarification. Otherwise, it's difficult to know exactly which results correspond to the claimed default setup.

  2. The reported 23X storage reduction counts only leaf assignments and leaf centroids, dropping internal-node information. However, Section 3.6 describes querying through the hierarchy using internal cluster centroids. I'm wondering whether the 0.17 ms query time uses exactly the representation counted in the storage comparison. If internal nodes must be stored or reconstructed, their cost should be included. Also, Table 5 reports 74 minutes of preprocessing, so the distinction between fast feature mapping and full pipeline time could be clearer.

Questions for authors

Please refer to Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.