acceptodds
Withdrawn from ICLR 2027

Rethinking Representation Granularity in Open-Vocabulary 3DGS: Endogenous Semantics via Sparse Anchors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-vocabulary semantic understanding with 3D Gaussian Splatting (3DGS) has long been constrained by two untested representation premises, namely that semantics rely on dense per-primitive representations over the full set of primitives, and that semantics must take the form of independent extrinsic high-dimensional parameters, rather than being intrinsic properties of geometric primitives. From the perspective of representation learning, this paper systematically revisits representation granularity and semantic sources using structured sparse anchors as an experimental platform. We propose Endogenous Semantic Mapping: it discards traditional extrinsically attached semantic vectors, takes only the intrinsic geometric and appearance features of sparse anchors as input, and directly generates semantics through a globally shared network. Specifically, semantic vectors are embedded as native rendering attributes of 3D Gaussians, and pixel-wise semantic features are directly output through the native rasterization pipeline, rather than being predicted by an external network and attached as auxiliary properties. This method achieves zero scene-level semantic parameter storage and can directly reuse pre-trained geometric models. Controlled experiments on multiple public benchmarks reveal a key finding that dense Gaussian representations exhibit significant over-parameterization for semantic tasks. When semantic representation units are compressed to approximately 10% of the full dense primitives, no significant degradation in segmentation accuracy is observed, demonstrating a fundamental mismatch between the optimal representation granularities required for rendering and semantics. At this extremely sparse granularity, the proposed method not only significantly reduces storage and VRAM overhead, but also achieves competitive segmentation accuracy against mainstream methods that adopt full dense representations with extrinsically attached semantics on public 3D semantic benchmarks. This study empirically confirms that native structured scene features possess strong semantic encoding capacity, and establishes a new design paradigm for lightweight 3D semantic representation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting (3DGS), open-vocabulary 3D semantics, representation redundancy, structured sparse anchors, endogenous semantic mapping

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mz6vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an Endogenous Semantic Mapping framework based on Scaffold-GS Sparse Anchors, aiming to explore Semantic Representation Redundancy in 3DGS. The authors argue that existing methods, which rely on dense per-Gaussian semantic embeddings, suffer from overparameterization, whereas the geometric and appearance features in pre-trained scenes may already contain sufficient semantic information. To this end, the authors map the 35D native features of the anchors through a shared multilayer perceptron (MLP) to 16D semantic features, and then, via Gaussian rasterization and dimension alignment, obtain pixel-level.

Critical strengths
  1. This paper re-examines open-vocabulary 3D Gaussian Splatting from the perspectives of representation granularity and semantic source, aiming to break away from the conventional design that requires semantic representations to maintain the same granularity as rendering primitives and necessitates the independent addition of semantic parameters. This research direction is of practical significance and also offers a new perspective for lightweight 3D semantic representation.
  2. The authors directly leverage the pre-trained Anchor Features from Scaffold-GS and generate semantic representations via a lightweight Shared Semantic MLP, eliminating the need to maintain separate trainable semantic vectors for each Gaussian or anchor. At the same time, they reuse the existing Gaussian rasterization pipeline to achieve spatial alignment between geometry and semantics.
  3. This method achieves an average mIoU of 69.8% on LERF-OVS, outperforming the existing baseline methods listed in the paper. The ScanNet experiment in the Appendix also yields promising results. Furthermore, the authors report lower semantic storage overhead and GPU memory usage, highlighting the framework’s potential in terms of semantic representation efficiency.
Critical weaknesses
  1. The relationship between Sparse Anchor distribution and Semantic Redundancy lacks sufficient evidence. In Section 3.1, the authors argue that the anchors of Scaffold-GS tend to be distributed at locations with significant geometric variations, which happen to possess higher semantic discriminative power. However, the Anchor Generation in Scaffold-GS is primarily driven by optimization signals from the geometric reconstruction process and does not directly ensure that its distribution aligns with Semantic Boundaries or Semantic Importance. Therefore, the authors attribute the effectiveness of Anchor-level Representation to the redundancy inherent in Dense Gaussian Semantics, thereby relying on assumptions that have not been sufficiently validated. Current experiments have not directly analyzed the relationship between anchor spatial distribution and semantic importance.
  2. Sparsity sensitivity analysis is not yet sufficient to rigorously support the core conclusions. In Section 4.2, by adjusting the number of child Gaussians (k) generated for each anchor, we reduced the semantic representation ratio from 50% to 5%, and observed that the mIoU remained essentially stable. However, according to Appendix A.4, the variation in the number of anchors during the experiment is relatively limited; the significant changes in the Semantic Representation Ratio are primarily attributable to an increase in the number of Child Gaussians, rather than a substantial reduction in the number of independent Semantic Representation Units. Therefore, this experiment more directly demonstrates that Anchor-level Semantic Mapping exhibits a certain degree of stability across different Child Gaussian configurations, but it does not sufficiently substantiate the extent of redundancy in Dense Semantic Primitives as claimed by the authors.
  3. The implementation conditions for Global Shared Semantic Mapping and Zero Scene-level Semantic Storage remain unclear. The authors claim that the Semantic MLP is shared across different scenes, thereby achieving zero scene-level semantic storage. However, the anchor features of Scaffold-GS are independently reconstructed for each scene and may not share a consistent feature space across different scenes. At the same time, the paper reports the semantic training time for each scenario separately, but does not clearly specify whether all scenarios use the same fixed semantic MLP. If the new scene still requires separate training and storage of the Semantic Mapping Network, then strictly speaking, only zero per-anchor semantic embedding storage can be achieved, but not zero-scene-specific semantic parameter storage.
  4. Existing experiments have not yet sufficiently disentangled the contributions of the representation design and the training strategy. Appendix A.3 shows that when Embedding Loss alone is used, the average mIoU of LERF-OVS is only 56.35%; after BCE and Dice Loss are incorporated in the second stage, it improves to 69.81%. This indicates that the final performance largely depends on additional segmentation supervision. Although the authors claim that their main contribution lies in Representation Design, the existing experiments do not sufficiently compare the performance of different representation granularities under the same backbone, pseudo-labels, and training objectives. Additionally, the primary baseline results are derived from existing papers, and the comparability of the associated supervision and training protocols still requires further clarification.
Questions for authors
  1. Can the authors provide direct evidence demonstrating a strong correlation between the Anchor Spatial Distribution of Scaffold-GS and regions of geometric variation or semantic boundaries?
  2. Under fixed Geometry, Child Gaussian Configuration, and Training Settings, can the authors directly reduce the number of anchors involved in semantic mapping and evaluate the resulting changes in mIoU? This would provide more direct support for the core conclusion regarding semantic representation redundancy.
  3. Please clarify whether the Semantic Mapping Network uses identical parameters across all scenarios. If the network indeed shares representations across different scenes, could the authors provide experiments demonstrating training on a subset of scenes followed by direct application to unseen scenes? If a Semantic MLP needs to be optimized independently for each scene, please clarify the precise meaning and statistical scope of “Zero Scene-level Semantic Storage.”
  4. In the second stage, the SFNS Loss led to a significant performance improvement. Can the authors compare Dense Per-Gaussian Mapping with Sparse Anchor Mapping under a unified framework that incorporates Backbone, Semantic Supervision, and Loss Functions? Additionally, please clarify whether the supervision and post-processing protocols of each baseline are comparable, so as to further isolate the contribution of the Representation Design itself.
Reviewer_dHPeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies representation granularity in open-vocabulary 3D Gaussian Splatting. It argues that the density needed for rendering is far higher than what semantic representation needs. Based on this, it proposes an endogenous semantic mapping framework built on sparse Scaffold-GS anchors. Instead of storing a semantic vector per Gaussian or per anchor, a shared MLP maps each anchor’s native attributes to semantic features. The mapping is trained with pseudo-labels from multimodal foundation models (SAM and OpenCLIP). The method is mainly evaluated on four indoor LERF-OVS scenes, with a complementary ScanNet experiment that uses native Gaussian representations. It reports competitive open-vocabulary segmentation, mIoU that stays relatively stable as the number of child Gaussians per anchor (k) changes, and lower semantic storage and GPU memory than prior methods.

Critical strengths
  1. A new, storage-efficient way to represent semantics. The model derives semantic features from native Scaffold-GS anchor attributes through a shared MLP, so it doesn’t store explicit per-primitive semantic vectors. This differs from the dominant feature-distillation approach. It could substantially cut per-scene semantic storage, which matters for scaling open-vocabulary 3DGS.
  2. A worthwhile question about granularity. The paper asks whether semantic representation needs the same density as rendering. This is a useful question for the community. The initial evidence, with mIoU staying relatively stable as k varies, suggests the question deserves further study, even though the current experiments don’t settle it (see Weaknesses).
Critical weaknesses

1.Scaffold-GS anchor features are optimized for reconstructing each scene and may vary across independently reconstructed scenes. So it is unclear how a single shared MLP can consistently map them to semantic concepts. The paper doesn’t say whether the mapping is trained jointly across scenes or separately for each scene. It also doesn’t evaluate generalization to held-out scenes without scene-specific fine-tuning. Without this, the claims of “globally shared semantic knowledge” and “zero scene-level semantic storage” are not supported. In addition, the mapping is trained with pseudo-labels from SAM and OpenCLIP. This means the semantics are learned through external supervision rather than arising solely from native geometric and appearance features. Removing per-anchor semantic vectors is a useful storage advantage, but by itself it does not show that semantics are intrinsically encoded in the native representation. Since these claims are the paper’s main conceptual contribution, this gap is consequential. 2. Varying k changes the number of child Gaussians per anchor, but anchor counts also change across configurations. The relatively stable mIoU may therefore reflect properties of Scaffold-GS rather than a general mismatch between rendering granularity and semantic granularity. The paper doesn’t include controlled experiments that separate semantic density from changes in the underlying representation, nor per-scene results or variability estimates. On top of that, the main evaluation uses only four indoor LERF-OVS scenes. The ScanNet experiment uses native Gaussians rather than the proposed anchor structure, so it doesn’t directly test the claimed sparsity benefits. The authors also acknowledge that the redundancy analysis has not been repeated on other representations or on outdoor and large-scale scenes. Because the paper presents this as a general paradigm, the narrow evidence undermines its central claim.

Questions for authors

1.Is the shared semantic MLP trained jointly across scenes or separately for each scene? If it is trained across scenes, how does it perform on held-out scenes without scene-specific fine-tuning? If it is trained per scene, the MLP is effectively per-scene semantic storage. How does that fit with the “zero scene-level semantic storage” claim? 2.In the sparsity experiments, how do anchor counts change with k? Can you show results where the semantic representation density varies while the underlying representation is held fixed (or the reverse)? Please also give per-scene mIoU and variability across runs. 3.The baseline results are mostly taken from prior publications. Are the scene selection, text queries, thresholds and post-processing the same as in your evaluation? Can you reproduce the strongest baselines under identical conditions? 4.The memory figures in Table 2 also come from different papers. Were they measured with the same hardware, rendering resolution and software setup? Please report per-scene storage separately from the shared model’s overhead (the shared MLP and dimension-alignment layers), and give controlled measurements of GPU memory and inference latency.

Reviewer_TDmHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether open-vocabulary 3DGS requires dense semantic representations. The authors propose to use sparse anchors from Scaffold-GS and predict semantic features from their geometric and appearance attributes through a shared MLP, without storing additional per-anchor semantic embeddings. The method achieves 69.81% mIoU on LERF-OVS while reducing semantic representation and storage overhead.

Critical strengths
  1. The paper explores an interesting question about the different representation granularities needed for rendering and semantic understanding. The motivation is clear and relevant to efficient 3D scene understanding.

  2. The method is simple and practical. It can reuse pretrained Scaffold-GS models without modifying the geometry, and the results on LERF-OVS are competitive. The ablations also provide useful observations about representation density and network capacity.

Critical weaknesses

1. The novelty over existing work is somewhat limited.

GALA (3DV 2026) has already studied the effect of Gaussian density on semantic segmentation using Scaffold-GS. Its supplementary material reports a similar observation that reducing the number of Gaussians can improve semantic performance. OpenGaFF (arXiv:2605.06088) also predicts semantic features from Gaussian geometry and appearance using a shared network. Although the proposed method differs in using frozen sparse anchors, the paper does not sufficiently clarify its contribution beyond these related approaches.

2. The experiments do not fully support the central claims.

The sparsity analysis mainly varies the number of child Gaussians per anchor through k. However, both the anchor counts and rendered Gaussian counts change across configurations. It is therefore difficult to determine whether the stable mIoU actually comes from semantic redundancy or other changes in the representation.

Also, the second training stage improves mIoU from 56.35% to 69.81%, while the baseline results are taken from previous papers. A comparison under the same supervision and training settings would help clarify how much of the improvement comes from the proposed representation.

Questions for authors
  1. Could the authors provide a controlled experiment that keeps the geometry fixed while varying the number of semantic representation units?
  2. How does the proposed method differ from the observations in GALA and the shared semantic mapping in OpenGaFF?
  3. Is the semantic MLP trained separately for each scene, or can the same network generalize to unseen scenes without retraining?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.