acceptodds
Under review as a conference paper at ICLR 2027

Sparse Code Uplifting for Efficient 3D Language Gaussian Splatting

Abstract

3D Language Gaussian Splatting (3DLGS) augments 3D Gaussian Splatting with language-aligned feature representations for open-vocabulary 3D scene understanding. A core challenge is efficiently associating high-dimensional vision-language embeddings with millions of 3D Gaussians while preserving efficient feature rendering for text-based querying. Existing methods either store dense features directly on Gaussians, causing high storage costs and slow rendering, or learn compact representations through expensive per-scene optimization with repeated feature rasterization. No existing method simultaneously achieves fast 3D semantic reconstruction, efficient storage, and fast rendering. We propose SCOUP (Sparse COde UPlifting), which addresses all three by decoupling language representation learning from 3D Gaussian optimization. Rather than working directly in 3D, we learn sparse codebook-based representations entirely using features associated with 2D image regions, associating each region with a sparse set of codebook coefficients. We then uplift these coefficients to 3D Gaussians with our weighted sparse aggregation using Gaussian-to-pixel associations, where each Gaussian accumulates coefficients over codebook atoms across views. Top- filtering then extracts the most dominant multi-view coefficients per Gaussian, enabling efficient storage and fast rendering. Our method achieves more than 400× training speedup while being 3× more memory efficient during training compared to the state-of-the-art in rendering speed. Across multiple benchmarks, SCOUP matches or outperforms existing methods in open-vocabulary querying accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Language Gaussian Splatting, 3D Scene Understanding, Sparse Coding

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_gsVwOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SCOUP, a framework for 3D Language Gaussian Splatting (3DLGS) that aims to simultaneously achieve fast semantic reconstruction, compact storage, and fast rendering. The key idea is to shift sparse semantic representation learning from 3D into 2D. More specifically, SCOUP learns a scene-specific sparse codebook and sparse coefficients over SAM regions in 2D. It then uplifts the learned coefficients to 3D Gaussians through weighted sparse aggregation followed by Top-K consensus filtering.

A central claim of the paper is that this reformulation is not merely computationally efficient. Learning sparse coefficients at the region level avoids object-scale imbalance inherent in Gaussian-level optimization, where large structures contribute many training samples. The uplifted sparse coefficients are subsequently aggregated across views, and Top-K filtering acts as a voting mechanism that suppresses inconsistent semantic evidence from multiple views.

SCOUP achieves faster semantic reconstruction than LangSplatV2, lower training memory usage, comparable rendering speed and storage efficiency, and improved open-vocabulary localization and segmentation accuracy.

Critical strengths
  1. Novel reformulation of semantic learning for 3DLGS

The key observation of the paper is that sparse semantic learning should be performed in the 2D region domain rather than in 3D Gaussian space (often millions). This reformulation is conceptually clean, addresses a fundamental bottleneck of prior approaches. Additional experiments also demonstrate that 2D sparse learning produces more balanced codebook representations across object scales than prior Gaussian-level optimization. This contribution changes both the computational structure and the statistical properties of semantic learning in 3DLGS.

  1. Strong empirical results

The paper demonstrates a good combination of efficiency and accuracy over strong baselines such as LangSplatV2. These gains are supported by extensive analyses. The extensive ablations show that SCOUP achieves efficient semantic reconstruction without sacrificing semantic fidelity, improves cross-view consistency, and maintains robustness to noisy associations and incomplete 3DGS reconstructions.

Critical weaknesses
  1. Strong dependence on region quality

The framework fundamentally relies on region-level semantic representations. As acknowledged by the authors, performance is sensitive to the quality of the underlying segmentation masks. Failure cases can stem from region decomposition rather than the sparse uplifting framework itself. Although the method is not tied to SAM specifically, its practical performance remains closely coupled to the quality of the region proposals.

Questions for authors
  1. A central argument of the paper is that learning sparse representations on SAM regions produces more balanced codebooks than learning directly on 3D Gaussians, particularly benefiting small objects. Could the authors provide a quantitative breakdown of localization/segmentation performance by object size (e.g., small, medium, large objects) to directly validate this claim?
Reviewer_53QbOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SCOUP, an efficient method for constructing 3D Gaussian semantic representations based on CLIP priors. Rather than following conventional wisdom to derive solutions, SCOUP adopts a reverse perspective and proposes a 2D uplifting paradigm that is fundamentally distinct from 3D feature rasterization. It learns the codebook and Top−KTop-K sparse coefficients directly on 2D feature maps, then fixes the codebook and uplifts the sparse coefficients onto 3D Gaussian primitives, enabling efficient semantic transfer from CLIP priors to 3D Gaussian primitives. The method delivers strong semantic reconstruction performance while ensuring high learning efficiency.

Critical strengths
  1. The proposed method is straightforward and practical. Learning the codebook and coefficient parameters directly on 2D feature maps eliminates the highly time-consuming repeated rasterization during iterative learning, a major bottleneck in prior methods that directly construct 3D representations. Once 2D learning is completed, high-quality semantic representation of 3D Gaussian primitives can be built solely by aggregating Top−KTop-K coefficient parameters.
  2. Experimental results demonstrate remarkable efficiency gains. On the LERF-OVS benchmark, as shown in Table 1, semantic construction time drops from 300 minutes (LangSplatV2) to 0.7 minutes, and peak GPU memory decreases from 16.4 GB to 5.2 GB, while maintaining the reported rendering speed and language representation storage footprint.
Critical weaknesses
  1. My primary concern is the insufficient theoretical justification and experimental support for the feature semantic fidelity of the proposed 2D-3D-2D transfer pipeline. I understand that Equation 4 fits a set of sparse coefficients and a codebook via learnable parameters to make the reconstructed 2D features approximately match the CLIP priors. However, during reconstruction onto 3D primitives, Top−KTop-K truncation of the sparse coefficients inevitably causes semantic information loss when 3D features are rasterized back to 2D pixel features from arbitrary views. Appendix G only reports cross-view similarity of decoded 2D features, rather than measuring reconstruction fidelity through the full 2D-3D-2D pipeline. For this reason, the paper’s claim of "nearly lossless" fidelity lacks both empirical and theoretical backing.
  2. It remains unclear under what conditions Top−KTop-K selection represents a reliable semantic consensus. The Top−KTop-K truncation rule retains elements with the largest cumulative rendering-weighted coefficients, but low-weight components may still carry useful or even critical information. The controlled corruption experiment demonstrates robustness when coefficients are randomly permuted among image regions. However, whether similar benefits extend to systematic segmentation errors or persistent cross-view semantic inconsistencies remains unclear.
  3. The scope of complexity and acceleration claims in the method section should be stated more precisely. The D/KD/K factor describes the arithmetic reduction of feature dimensions during coefficient accumulation, not the entire semantic construction pipeline. Algorithm 1 also includes accumulators of length LL and per-Gaussian Top−KTop-K selection. Appendix E does account for 2D sparse encoding time, so this does not mean the authors omitted the first stage. Nonetheless, aggregation complexity, total semantic construction time, and the full pipeline including RGB 3DGS reconstruction and SAM/CLIP preprocessing should be more clearly delineated.
Questions for authors
  1. Since the task objective is to learn 3D semantic representations from CLIP priors, could you quantify the feature reconstruction fidelity across the entire pipeline, instead of only indirectly supporting the "nearly lossless" claim via segmentation performance? If possible, could you provide the theoretical rationale for directly rasterizing the sparse coefficients of 3D Gaussian primitives to 2D pixels after Top−KTop-K truncation?
  2. When does Top−KTop-K selection discard useful information rather than noise? Could you report how the quality of retained coefficients and filtering effects impact semantic boundaries, sparsely observed regions, or well-documented 3D-OVS failure cases?
  3. What exactly is included in the efficiency measurements? Please confirm whether Table 1 covers codebook initialization, 2D optimization, association computation, uplifting, and Top−KTop-K filtering, and whether the reported rendering time includes codebook decoding. A comprehensive breakdown of these operations, separated from shared preprocessing steps, would help clarify the reported speedup.
Reviewer_NU4fOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCOUP, a 3D Language Gaussian Splatting method that targets fast semantic reconstruction, compact storage and fast rendering. SCOUP learns a sparse codebook and per-SAM region coefficients from 2D, rather than in 3D as LangSplatV2 does, then lifts the coefficients instead of full-dimensional CLIP features onto pretrained 3DGS, with LUDVIG-style aggregation. A Top-K filter on the aggregated coefficients is then applied, intended as multi-view consensus voting. Compared to LangSplatV2, SCOUP reports much faster reconstruction with the same rendering speed and storage. Compared with lifting-based methods such as VALA, it reports comparable reconstruction time, smaller storage, faster rendering and higher accuracy on LERF-OVS. Validation consists of 2D open-vocabulary querying on LERF-OVS, Mip-NeRF360 and 3D-OVS, plus ablations on the proposed sparse uplifting and Top-K filtering.

Critical strengths
  1. The paper offers a practical solution that achieves fast semantic reconstruction, compact storage and fast rendering at one time. It combines the sparse codebook representation of LangSplatV2 with the fast 2D-to-3D aggregation of lifting-based methods, and inherits the advantages of both lines. As shown in Table 1, it is the only method that is competitive on all four efficiency metrics, including reconstruction time, training memory, rendering speed and storage.

  2. The exploration on developing 2D codebook instead of 3D is interesting, since it is against the previous argument (from LangSplatV2) that a 2D codebook is suboptimal because it ignores 3D priors. The results from this paper suggest that learning sparse codes on 2D region features is not only much cheaper, since there are far fewer regions than Gaussians and no rasterization is needed, but can also be competitive in accuracy. Decoupling semantic compression from 3DGS this way is simple and could be reusable for furture 3DLGS researches. Though I will expect deeper analysis on how much it effects performance under 3D evaluation protocol, but the direction itself seems valuable.

Critical weaknesses
  1. The evaluation is not fully convincing, mainly: a) Lack 3D evaluation [Resolved]. My major concern is that all accuracy results are measured and reported on rendered 2D maps. Note that the “3D object localization” and “3D semantic segmentation” metrics in Tables 2–4 are computed on rendering but not in Gaussian space. However, the closest lifting-based works (e.g. VALA, Dr. Splat) also evaluate querying directly on the Gaussian space. Direct 3D search capability is important to discuss as it is what downstream applications in this paper's own motivation need. Furthermore, the codebook is learned purely in 2D, and SCOUP’s Top-K works over coefficients rather than over Gaussians along a ray. Occluded or barely visible Gaussians are therefore not filtered out as in VALA or Dr.Splat. The 2D-rendering based protocol will hide this potential cases. I am not assuming SCOUP would do badly in 3D, but without such an evaluation the pros and cons of the 2D codebook cannot be fully judged.

b) The baseline coverage is uneven. Relatively complete accuracy numbers of all baselines only on LERF-OVS, although they were run on all three datasets for Table 1 (VALA results not appear for 3D-OVS and MipNeRF360, OccamLGS not appear for MipNeRF360.)

c) On 3D-OVS, SCOUP is below LangSplatV2, and the SAM-based explanation does not quite hold. If the baselines comparison use the same SAM masks, SAM’s failure modes cannot explain the gap. If they do not, the comparison is not controlled. So for now the claim of matching or outperforming existing methods is currently supported mainly by 2D results on LERF-OVS.

  1. [Not fully convince but I appreciate the pratical performance and value] Apart from the 2D codebook, the technical contribution looks not sufficient. The representation, the softmax/top-K parameterization and the renderer come from LangSplatV2, and the aggregation comes from LUDVIG. From the ablation, the sparse uplifting is mainly a speed-up. The accuracy gain mainly comes from the Top-K step, which is essentially LangSplatV2’s sparsification applied after aggregation. So for me the contribution beyond prior work is mainly on the 2D codebook, which is promising but not yet fully validated (see W1).
Questions for authors

Major

Q1 (W1a, 3D evaluation). Can the authors report the 3D evaluation results following VALA or Dr.Splat's evaluation protocol, compared with VALA, (Dr. Splat), (LUDVIG) and Occam’s LGS? If SCOUP turns out weaker in 3D, I would still find it valuable to see this discussed as a trade-off of the 2D codebook. Relatedly, the “3D localization/segmentation” metrics in Tables 2–4 are computed on rendered views, so it would help to name them as 2D evaluation.

Q2 (W1b, baseline coverage). VALA and Occam’s LGS were already run on Mip-NeRF360 and 3D-OVS for Table 1. Could you add their accuracy on these datasets?

Q3 (W1c, 3D-OVS and protocol). Do SCOUP and the baselines use the same SAM/CLIP preprocessing? If they do, what in SCOUP causes the drop on Bed and Sofa relative to LangSplatV2? Also, the mask thresholds and pooling kernel differ from LangSplatV2’s protocol. How were they chosen, and how does SCOUP perform under the original settings?

Q4 How is FPS defined in Table 1? LangSplatV2 reports 476 FPS for feature rendering. A clear definition in the core efficency metrics will reduce the confusion.

Minor "In-the-wild” may not be the best description of the LERF-OVS scenes in the dataset section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.