acceptodds
Withdrawn from ICLR 2027

GALE: Learning Where to Distill for Sparse-View 3D Gaussian Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse-view 3D Gaussian segmentation aims to recover scene geometry and consistent object identities from only a few images. Existing methods use geometry and segmentation foundation models for this task, but do not fully resolve which Gaussians each 2D mask should supervise. This ambiguity arises because Gaussians on different surfaces can project into the same mask. In this paper, we present GALE (Geometry-Aware Lifting of Evidence), a lightweight framework that learns where to distill through geometry-conditioned sparse attention. Built on a VGGT-initialized reconstruction, GALE combines predicted depth and confidence with Gaussian visibility to automatically select and weight the 3D support of each SAM2 mask. These uncertainty-aware assignments focus supervision on compatible Gaussians, guiding knowledge transfer in both scene-specific optimization and cross-scene prediction. Experiments show that GALE outperforms prior distillation and grouping methods on views with held-out annotations. When generalizing to unseen scenes, it surpasses other methods on most metrics using only 1.51M task-specific trainable parameters, approximately 1/40 of the state-of-the-art method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
3D Gaussian Splatting, 3D Gaussian segmentation, panoptic segmentation, VGGT, SAM2, geometry-aware distillation, sparse attention, cross-scene generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1bfZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes GALE, a framework for sparse-view 3D Gaussian segmentation. It introduces geometry-aware sparse attention to assign 2D mask supervision to 3D Gaussians, combining geometric cues, visibility, and learned feature affinity. The resulting weights guide region-level feature distillation. The method is evaluated in both scene-specific optimization and cross-scene generalization settings, showing improvements over several existing approaches.

Critical strengths
  1. The motivation of this work is reasonable: determining which 3D Gaussians should receive supervision from 2D segmentation masks, especially under sparse-view settings where geometric ambiguity is involved.
Critical weaknesses
  1. Limited novelty. The proposed framework relies heavily on a combination of existing techniques and manually designed geometric heuristics. While the design is reasonable, I find the technical novelty rather limited.
  2. Poor presentation. The paper lacks a clear narrative. The architecture figure is fragmented, with insufficient emphasis on the key components and their relationships, making the overall method difficult to follow.
  3. Insufficient ablations. The method relies on several geometric cues, including depth consistency, visibility, and confidence, but their individual contributions are not evaluated. Such ablations are necessary to better understand the proposed attribution mechanism.
  4. Unfair comparisons. Uni3R uses a released two-view checkpoint for the 6-view and 10-view evaluations, while LSM only supports two input views. In contrast, GALE is trained with 2–24 views.
  5. Marginal Improvement. Under the two-view setting, GALE also shows only a marginal improvement over Uni3R.
Questions for authors

See Weaknesses 1,3,5

Reviewer_aN8uOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that when scene understanding features such as segmentation are naively lifted to 3D Gaussians, the Gaussians may not receive proper loss signals due to issues such as occlusion and ambiguous boundaries. To address this, it proposes a new method for lifting features to Gaussians. It uses information from VGGT to perform weighted lifting of per-view information onto each Gaussian, and then assigns labels to the Gaussians through voting based on this lifting.

Critical strengths
  • The motivation is clear. For 3D segmentation, it is reasonable to leverage geometric information, and lifting 2D features should make use of 3D knowledge. This paper makes the most of VGGT’s attributes to perform this lifting.

  • The paper conducts experiments in both per-scene optimization and generalizable settings, and shows through extensive experiments that the proposed module works well across different settings. The ablation studies on each component are also well presented.

Critical weaknesses
  1. Lack of justification for the formulation. My biggest concern is that the equations lack sufficient justification. When defining the descriptor in the geometry-aware attribution, it seems that all available information is simply included. It is also unclear why these terms are summed in Eq. 3. Similarly, in Eq. 4, the weight is defined by multiplying the attention weight, confidence, opacity, and depth gap. However, there is no explanation of why these are multiplied, whether all of them are necessary, or why an exponential is used for the depth gap. An ablation study on these choices, or an explanation of the reasoning behind the formulation, would be helpful. Personally, I find the overall design reasonable: computing queries and keys, then using VGGT features as conditions to output the weights. However, I cannot see the rationale behind the specific equations within these modules, and I would like to hear the authors’ reasoning.

  2. Possible trivial solution in Eq. 5 and 6. This is more of a question. If s takes the same value within a mask (a trivial solution), there seems to be no reason for the weights to be learned. It would be helpful to clarify this.

  3. Lack of analysis on the learned weights. As an extension of W1, there is no analysis of whether w is learned as the authors intend. Given the limited justification for the formulation, it would be helpful to at least provide visualizations or plots showing how w behaves after training. Without such analysis, it is unclear whether w is learned as intended.

  4. Dense description and the contribution of sparse voting. It is unclear how the dense description is obtained. There is no precise explanation of whether the input-view masks are simply projected to 3D through the rasterizer, which makes it difficult to understand. Moreover, Table 3 and Figure 7 suggest that the dense description fills in most regions and also corrects the mispredictions from sparse voting. Even the w/o sparse voting variant performs quite well. It is therefore disappointing that the main contribution of the paper appears to provide the smallest performance gain.

  5. Additional baselines for per-scene optimization. In the per-scene optimization setting, beyond comparing with SAM-based methods, it would be good to also compare with models that distill SAM + CLIP features, such as Feature 3DGS, LangSplat, and CF3 [A,B,C].

  6. Comparison with C3G in the generalizable setting. In the generalizable setting, C3G[D] reuses the attention weights obtained for predicting geometry Gaussians to compute a weighted sum of features. A comparison with C3G would be helpful. Since this appears to work similarly to what this paper aims to achieve, I would also like to hear the authors’ thoughts on it.

[A] Zhou, Shijie, et al. "Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[B] Qin, Minghan, et al. "Langsplat: 3d language gaussian splatting." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[C] Lee, Hyunjoon, Joonkyu Min, and Jaesik Park. "CF 3: Compact and Fast 3D Feature Fields." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[D] An, Honggyu, et al. "Learning Compact 3D Representations from Feed-Forward Novel View Synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Questions for authors

.

Reviewer_DwESOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GALE, a geometry-aware framework for transferring 2D segmentation information into a 3D Gaussian representation. The central contribution is a learned sparse attribution mechanism that determines which Gaussians should receive supervision from a given 2D mask by combining geometric evidence, visibility cues, and learned feature affinity. The approach is applied both in a scene-specific optimization setting and a cross-scene feed-forward setting, with experiments demonstrating improvements in semantic segmentation metrics over a few recent 3D Gaussian segmentation methods.

Critical strengths
  • Strong quantitative semantic segmentation results. The reported semantic mIoU improvements are consistent across multiple benchmarks and view regimes.

  • Good engineering and extensive experimentation. The paper contains ablations and evaluations spanning both scene-specific and cross-scene settings.

Critical weaknesses
  1. Unclear necessity of the proposed complexity. The most significant concern is whether the proposed multi-stage pipeline is actually needed. The method introduces a substantial amount of machinery: VGGT geometry prediction, SAM2 masks, geometry descriptors, sparse attribution networks, voting mechanisms, feature aggregation modules, and segmentation heads. Yet the empirical gains over simpler alternatives often appear incremental given the complexity increase.

In particular, learning-free methods such as LUDVIG already demonstrate that strong 3D semantic grouping can emerge from foundation-model priors with minimal additional learning. The paper does not convincingly establish why a learned attribution model is necessary, rather than a carefully designed geometric association heuristic. Missing are comparisons that isolate the benefit per unit of complexity, inference cost, or implementation overhead.

  1. Attribution module is not convincingly isolated. The central claim is that geometry-aware attribution is the key innovation. However, there is no convincing evidence showing that attribution itself is responsible for the gains. The reported improvements could stem from: Better geometry from VGGT, Better masks from SAM2, Additional learned feature aggregation, Cross-view voting mechanisms, or Auxiliary supervision signals. Without stronger controlled baselines, it remains difficult to conclude that the proposed attribution strategy is the primary reason for improved performance.
Questions for authors
  1. Can the authors provide qualitative examples where SAM2 or VGGT make substantial errors? Does GALE correct these errors or simply inherit them?

  2. What is the performance of a purely geometric attribution baseline without any learned attribution network?

  3. How does performance change when VGGT geometry quality is artificially degraded? This would help evaluate robustness and dependence on foundation-model priors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.