- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an Endogenous Semantic Mapping framework based on Scaffold-GS Sparse Anchors, aiming to explore Semantic Representation Redundancy in 3DGS. The authors argue that existing methods, which rely on dense per-Gaussian semantic embeddings, suffer from overparameterization, whereas the geometric and appearance features in pre-trained scenes may already contain sufficient semantic information. To this end, the authors map the 35D native features of the anchors through a shared multilayer perceptron (MLP) to 16D semantic features, and then, via Gaussian rasterization and dimension alignment, obtain pixel-level.
Critical strengths
- This paper re-examines open-vocabulary 3D Gaussian Splatting from the perspectives of representation granularity and semantic source, aiming to break away from the conventional design that requires semantic representations to maintain the same granularity as rendering primitives and necessitates the independent addition of semantic parameters. This research direction is of practical significance and also offers a new perspective for lightweight 3D semantic representation.
- The authors directly leverage the pre-trained Anchor Features from Scaffold-GS and generate semantic representations via a lightweight Shared Semantic MLP, eliminating the need to maintain separate trainable semantic vectors for each Gaussian or anchor. At the same time, they reuse the existing Gaussian rasterization pipeline to achieve spatial alignment between geometry and semantics.
- This method achieves an average mIoU of 69.8% on LERF-OVS, outperforming the existing baseline methods listed in the paper. The ScanNet experiment in the Appendix also yields promising results. Furthermore, the authors report lower semantic storage overhead and GPU memory usage, highlighting the framework’s potential in terms of semantic representation efficiency.
Critical weaknesses
- The relationship between Sparse Anchor distribution and Semantic Redundancy lacks sufficient evidence. In Section 3.1, the authors argue that the anchors of Scaffold-GS tend to be distributed at locations with significant geometric variations, which happen to possess higher semantic discriminative power. However, the Anchor Generation in Scaffold-GS is primarily driven by optimization signals from the geometric reconstruction process and does not directly ensure that its distribution aligns with Semantic Boundaries or Semantic Importance. Therefore, the authors attribute the effectiveness of Anchor-level Representation to the redundancy inherent in Dense Gaussian Semantics, thereby relying on assumptions that have not been sufficiently validated. Current experiments have not directly analyzed the relationship between anchor spatial distribution and semantic importance.
- Sparsity sensitivity analysis is not yet sufficient to rigorously support the core conclusions. In Section 4.2, by adjusting the number of child Gaussians (k) generated for each anchor, we reduced the semantic representation ratio from 50% to 5%, and observed that the mIoU remained essentially stable. However, according to Appendix A.4, the variation in the number of anchors during the experiment is relatively limited; the significant changes in the Semantic Representation Ratio are primarily attributable to an increase in the number of Child Gaussians, rather than a substantial reduction in the number of independent Semantic Representation Units. Therefore, this experiment more directly demonstrates that Anchor-level Semantic Mapping exhibits a certain degree of stability across different Child Gaussian configurations, but it does not sufficiently substantiate the extent of redundancy in Dense Semantic Primitives as claimed by the authors.
- The implementation conditions for Global Shared Semantic Mapping and Zero Scene-level Semantic Storage remain unclear. The authors claim that the Semantic MLP is shared across different scenes, thereby achieving zero scene-level semantic storage. However, the anchor features of Scaffold-GS are independently reconstructed for each scene and may not share a consistent feature space across different scenes. At the same time, the paper reports the semantic training time for each scenario separately, but does not clearly specify whether all scenarios use the same fixed semantic MLP. If the new scene still requires separate training and storage of the Semantic Mapping Network, then strictly speaking, only zero per-anchor semantic embedding storage can be achieved, but not zero-scene-specific semantic parameter storage.
- Existing experiments have not yet sufficiently disentangled the contributions of the representation design and the training strategy. Appendix A.3 shows that when Embedding Loss alone is used, the average mIoU of LERF-OVS is only 56.35%; after BCE and Dice Loss are incorporated in the second stage, it improves to 69.81%. This indicates that the final performance largely depends on additional segmentation supervision. Although the authors claim that their main contribution lies in Representation Design, the existing experiments do not sufficiently compare the performance of different representation granularities under the same backbone, pseudo-labels, and training objectives. Additionally, the primary baseline results are derived from existing papers, and the comparability of the associated supervision and training protocols still requires further clarification.
Questions for authors
- Can the authors provide direct evidence demonstrating a strong correlation between the Anchor Spatial Distribution of Scaffold-GS and regions of geometric variation or semantic boundaries?
- Under fixed Geometry, Child Gaussian Configuration, and Training Settings, can the authors directly reduce the number of anchors involved in semantic mapping and evaluate the resulting changes in mIoU? This would provide more direct support for the core conclusion regarding semantic representation redundancy.
- Please clarify whether the Semantic Mapping Network uses identical parameters across all scenarios. If the network indeed shares representations across different scenes, could the authors provide experiments demonstrating training on a subset of scenes followed by direct application to unseen scenes? If a Semantic MLP needs to be optimized independently for each scene, please clarify the precise meaning and statistical scope of “Zero Scene-level Semantic Storage.”
- In the second stage, the SFNS Loss led to a significant performance improvement. Can the authors compare Dense Per-Gaussian Mapping with Sparse Anchor Mapping under a unified framework that incorporates Backbone, Semantic Supervision, and Loss Functions? Additionally, please clarify whether the supervision and post-processing protocols of each baseline are comparable, so as to further isolate the contribution of the Representation Design itself.