acceptodds
Under review as a conference paper at ICLR 2027

TOFUGS: TOPOLOGY BEFORE FUSION FOR OPEN-VOCABULARY GAUSSIAN SPLATTING

Abstract

Compact open-vocabulary 3D representations must consolidate multiview, multiscale semantics into one deployable descriptor per primitive. A common Gaussian splatting pipeline fuses these observations before semantic refinement, implicitly treating fusion as an independent step of refinement. However, this ordering matters because mask granularity can change feature directions as well as the induced refinement topology: fusion before topology constructs a single topology from mixed features and removes branch-specific information, whereas topology before fusion preserves the complementary partitions and mask registrations induced by each branch. To address this problem, we introduce ToFuGS (Topology-before-Fusion Gaussian Splatting), which independently refines branch-specific mask-derived fields first and uses training-view evidence to fuse them into a single deployable descriptor per Gaussian. On LERF, ToFuGS improves its underlying refinement pipeline by 7.34 mIoU points and establishes a new state of the art. Without retuning, it improves results on 3D-OVS, and an order control with a second refinement backend likewise favors refine-then-fuse.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Open-Vocabulary 3D Scene Understanding, Vision-Language Models, Semantic Feature Fields, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FkYROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work builds on the previously published SFS and proposes performing feature refinement before fusion for feature lifting from 2D multi-scale SAM masks and OpenCLIP descriptors to 3DGS. The proposed approach utilizes all levels of SAM masks as a "Uniform" branch and the middle-level features as a "Mid" branch. In this pipeline, it first lifts the features from each mask level and combines them into the Uniform branch while retaining the middle level as the Mid branch from the lifted 3DGS. After the initial round of lifting features to 3DGS separately to form the Uniform and Mid branches, the method evaluates and refines the quality of the lifted 3D features by computing the IoU between SAM masks and rendered feature clusters from 3DGS across training image views. The authors then define a score to assess feature stability across views and ultimately determine how to fuse the Uniform and Mid branches into the final 512-dimensional feature vector for each Gaussian. The authors compare the proposed approach with other state-of-the-art methods on LERF and 3D-OVS.

Critical strengths

The proposed approach is practical; it retains the footprint of SFS, as discussed by the authors, and should be straightforward to integrate into existing processing pipelines where SFS is already adopted (I would personally test it in our own pipeline).

Critical weaknesses
  1. The contribution of this work appears to be largely incremental over SFS (published at ICLR last year). SFS's primary contribution was providing a closed-form solution for feature lifting and proving an upper bound for it, with open-vocabulary segmentation serving as a validation method and post-lifting aggregation presented as an additional contribution. From my understanding, this paper applies post-lifting aggregation from SFS separately to the Uniform and Mid branches and introduces a fusion process at the end. Consequently, the reviewer is not convinced that the claimed contributions are substantial enough for ICLR.

  2. The paper claims that "Refine →\rightarrow Fuse" outperforms "Fuse →\rightarrow Refine." While this trend holds across all four scenes in Table 8, the gain on the Ramen scene with the SFS backend is only 0.14. Such a marginal improvement could be attributed to variations in 3D reconstruction quality. Given that the authors already evaluated three different reconstructions in Table 1, it would be beneficial to extend the results in Table 8 across all three reconstructions.

  3. Comparing Table 1 with Table 1(a) of the original SFS paper, the reported SFS baseline on LERF is 61.22, whereas the original SFS paper reported 65.1. The authors do not clarify the cause of this substantial discrepancy in their reproduction of SFS. This gap makes the claimed 7.34 gain of the proposed method less convincing.

  4. The authors' choice of the Mid branch setting (q=2q = 2) is not adequately justified with an ablation study.

  5. The paper misses several relevant related works, such as GAGS [1], THGS [2], and N2F2 [3], which also explore combining multi-granularity SAM masks in 2D/3D GS:

[1] Peng, Yuning, et al. "Gags: Granularity-aware feature distillation for language gaussian splatting." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 10. 2026.

[2] Dai, Shaohui, et al. "Training-free hierarchical scene understanding for gaussian splatting with superpoint graphs." Proceedings of the 33rd ACM International Conference on Multimedia. 2025.

[3] Bhalgat, Yash, et al. "N2f2: Hierarchical scene understanding with nested neural feature fields." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

Questions for authors
  1. Please provide experimental results and clarifications addressing the concerns raised in Weaknesses 2 and 4, and clarification for 3.

  2. Please clarify and further articulate the primary novel contributions of this work relative to SFS.

  3. Please incorporate and discuss the missing related works noted in Weakness 5.

Reviewer_CVMuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ToFuGS asks when multiscale semantic fields should be fused in open-vocabulary Gaussian splatting. The method builds a cross-scale Uniform field and a medium-scale Mid field, refines each through its own clustering and mask registration, and combines their descriptors only afterwards. Fusion weights come from agreement between registered mask descriptors across training-view halves and from persistence of registration support, and a stability gate chooses between continuous blending and a single branch. The deployed representation keeps one descriptor per Gaussian. Experiments use LERF and 3D-OVS, with an operator-order control on Dr. Splat.

Critical strengths
  1. The order claim is tested with matched controls on two backends. Table 2d holds the fusion weights at equal values and changes only the operator order: 64.19 to 66.64 for SFS, and 49.56 to 52.34 for Dr. Splat. Appendix F reports per-scene differences that are positive in all four scenes for both backends. This is direct evidence for the central claim, and it does not depend on the adaptive weighting scheme.

  2. The complete method improves on its base pipeline under a stated deployment constraint. Across three independently trained reconstructions, Table 1 raises LERF mIoU from 61.22 (SFS) to 68.56, above the stronger standalone Mid branch at 65.50. The same configuration transfers to 3D-OVS without retuning (Table 2c, 69.69 to 74.03). The deployed field keeps one descriptor per Gaussian at 970.73 MiB (Section 4.5).

Critical weaknesses
  1. The comparative evaluation is harder to interpret than the internal SFS comparison. Under the common readout of Table 2a, several baselines sit well below their originally reported numbers — CCL-LGS 50.61 against 65.6 in Table 9, LangSplatV2 46.43 against 59.9 — and the manuscript does not document how each baseline was adapted or what drives these shifts. The margin over the strongest external method is thin: 68.56 against LEGO's 67.18, with no paired uncertainty reported. The readout itself selects a histogram peak using the annotated instance count (Section 4.1; Appendix A). This is inherited from SFS and applied uniformly to all six methods, but it narrows the query-time segmentation claim, and no annotation-free results are given. On 3D-OVS the comparison covers only SFS and the paper's own variants (Table 2c), so the second dataset supports transfer of the ordering claim rather than an external ranking.

  2. The topology-sharing ablation does not isolate the contribution of independent partitions. Sharing partitions also makes the branch evidence identical and selects Mid in every scene (Table 2b; Appendix F), so the 68.14 score against 64.60 and 66.20 mixes a change in refinement structure with a change in fusion behavior. The paper discloses this coupling, and the equal-weight control in Table 2d remains valid for the ordering claim itself; what is unsettled is how much of the full gain belongs to topology diversity as such.

Questions for authors
  1. Do any existing runs compare independent and shared partitions under one fixed blending rule? If not, how narrowly should Table 2b be read?
  2. How was each Table 2a baseline reconstructed and adapted to the common readout, and what accounts for the shifts relative to Table 9?
  3. Are any results available with an annotation-free readout? If not, please state the intended scope of the query-time segmentation claim.
Reviewer_tzL6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the ordering of semantic fusion and refinement in open-vocabulary Gaussian Splatting. ToFuGS independently refines a cross-scale Uniform branch and an intermediate-scale Mid branch, then adaptively fuses them using support evidence from training views. The final representation retains a single descriptor per Gaussian. The authors report improvements on LERF and 3D-OVS and evaluate the design through operator-order comparisons, fusion variants, and an additional backend.

Critical strengths

The paper addresses a concrete and practically relevant issue: fusion can change the grouping structure on which subsequent refinement operates. The direct comparison between early and late fusion supports the design motivation. The main results cover three independently trained reconstructions, and the same configuration also improves performance on 3D-OVS. Retaining a single descriptor per Gaussian at deployment is another practical advantage.

Critical weaknesses
  1. The method provides a useful practical improvement, but the scope of its central design conclusion remains insufficiently established. The new contributions primarily concern branch construction, operator ordering, and evidence-guided fusion, while feature lifting and refinement largely follow existing methods. The cross-dataset and cross-backend experiments provide positive evidence, but the order comparisons primarily use the fixed Uniform–Mid pair. The hierarchy-routing alternative in the appendix has different storage and runtime costs and does not establish whether the ordering advantage is stable across granularity choices under a comparable two-branch budget. Thus, the evidence more strongly supports this particular configuration than a broader fusion design principle. Using the already available scale-specific features, a small set of early-versus-late fusion comparisons with pairs such as Uniform–Small and Uniform–Large would help establish the conditions under which the advantage holds.

  2. The connection between the reliability measure and actual segmentation correctness is still supported mainly indirectly. Table 6 tests whether a branch selected using several partitions continues to receive higher support under another partition. This demonstrates stability of the internal preference, but does not directly establish that the better-supported branch is semantically more accurate. The authors also acknowledge that consistent mask or descriptor errors can receive strong support. The fusion ablations and aggregate improvements demonstrate that the strategy is useful, but do not fully explain when it improves or harms predictions. An analysis on the existing evaluation views linking reliability to actual segmentation error would strengthen this aspect. In particular, the authors could report fusion gains relative to the two branches across reliability intervals and include representative cases with high reliability but incorrect predictions.

Questions for authors
  1. Does refine-then-fuse continue to outperform early fusion with other two-branch granularity combinations?
  2. Does the reliability score correlate with actual segmentation gains? How frequently does fusion degrade predictions despite high reliability, and what are the main causes?
  3. Do the main relative advantages persist when retaining the competing methods’ native readouts or removing annotated-instance-count assistance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.