acceptodds
Under review as a conference paper at ICLR 2027

SpecGS: Frequency–Spatial Support Coupled Gaussian Splatting for Memory-Efficient Large-Scale Scene Reconstruction

Abstract

Despite the success of 3D Gaussian Splatting (3DGS) in real-time rendering, reconstructing large-scale urban scenes remains challenging. Due to the inherent low-pass nature of Gaussian primitives, capturing high-frequency details (e.g., sharp edges and intricate textures) typically requires excessive primitive densification. Consequently, many existing methods rely on multi-GPU training or scene partitioning to scale reconstruction. We propose SpecGS, a 3DGS framework that improves scalability through frequency-guided Gaussian adaptation. SpecGS estimates Gaussian-level high-frequency evidence from image-space frequency responses and couples it with spatial-support regulation and densification prioritization, enabling representation capacity to be preferentially allocated to regions requiring fine-detail reconstruction. Moreover, to ensure stable optimization at scale, we introduce a three-stage optimization strategy with an Adaptive Convergence Trigger (ACT) that automatically schedules coarse-to-fine resolution progression based on optimization saturation. At the final stage, Differenceof-Gaussian (DoG) supervision further emphasizes high-frequency reconstruction while providing the frequency evidence used for Gaussian adaptation. Experiments on multiple urban datasets demonstrate that SpecGS achieves a favorable quality–efficiency trade-off with significantly fewer primitives and reduced memory consumption, enabling practical city-scale reconstruction on a single GPU.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
3DGS

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_e7yxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpecGS is a single-GPU large-scale 3D Gaussian Splatting framework that allocates Gaussian capacity using image-space frequency evidence. It controls training resolution according to the Adaptive Convergence Trigger (ACT), and enhances high-frequency rendering quality with DoG supervision in the final stage. On four main urban scenes, SpecGS achieves competitive results with significantly less Gaussian count and GPU costs.

Critical strengths
  1. Favorable quality–memory trade-off. Against the common Table 2 baselines (CityGaussian, 3DGS, and CityGS-X), SpecGS reduces the four-scene average Gaussian count by 43.0% (8.56M→4.88M) and peak memory by at least 47.2% ($\geq$20.68→10.92 GB), while improving average PSNR by 0.36 dB, SSIM by 2.84%, and LPIPS by 12.65%. The considerable memory reduction allows city-scale optimization on one A5000, where several baselines exceed 24 GB.
  2. The components are meaningfully ablated. Table 4, Table 5, and Figure 6 provide detailed quantitative and qualitative validation of each module, which sufficiently support the model design choices.
Critical weaknesses
  1. The method design is heuristic and involves complex hyperparameter settings, involving γ of Frequency–Spatial Support Coupling, η of Frequency-Guided Densification, intermediate training resolution, ACT saturation threshold, and so on. It deserves a discussion of generalization ability and how to adjust the hyperparameters when applied to a custom dataset.
Questions for authors
  1. Why does SpecGS underperform standard 3DGS by 1.51 dB PSNR on Sci-Art despite all core ablations being conducted on that scene? Which content triggers failure, and is it related to misleading high-frequency responses?
1 comment

Thanks for the authors'effort in resolving my concern. I do appreciate additional experiments regarding robustness to hyperparameter settings and enhanced ablation settings. I would like to improve the rating slightly, yet no intermediate score exists. Thus, I still recommend weak acceptance.

Reviewer_C9tCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SpecGS, a framework for memory-efficient large-scale scene reconstruction based on 3D Gaussian Splatting. It addresses the computational and memory costs of representing large scenes by allocating Gaussian capacity according to local image detail. The method extracts Difference-of-Gaussians responses from training images and aggregates them into per-Gaussian frequency scores. These scores guide spatial-support regularization and densification priorities, while a frequency-weighted reconstruction loss emphasizes detailed regions. The framework also incorporates a coarse-to-fine training schedule with an adaptive convergence trigger that controls transitions between rendering resolutions. The underlying Gaussian representation and rendering procedure remain compatible with standard 3DGS. Experiments on urban scenes and additional benchmarks report fewer Gaussian primitives and lower GPU memory consumption, with competitive rendering quality and reduced training time in the evaluated settings.

Critical strengths
  1. Could the authors provide effective Gaussian scale distributions and normalized Gaussian densities for high- and low-frequency regions, comparing the full method with the variant without frequency-guided densification? This would help verify whether the proposed mechanism allocates compact Gaussians to detailed regions while avoiding redundant allocation in smooth regions.

  2. Could the authors quantitatively evaluate geometric reconstruction quality, for example by reporting F-score on datasets with reference geometry under a consistent evaluation protocol? Comparisons with relevant baselines would help establish whether the sharper rendered depth maps correspond to improved geometric accuracy and completeness.

Critical weaknesses
  1. Insufficient validation of frequency-dependent Gaussian allocation. The paper attributes its efficiency to allocating Gaussian capacity according to local frequency. However, the overall support distribution in Fig. 9(c) does not establish how Gaussian scales and densities differ between high- and low-frequency regions. Reporting effective scale distributions, normalized Gaussian densities, and densification rates across frequency groups would help verify the proposed mechanism. Comparisons with standard 3DGS and the variant without frequency-guided densification are needed to determine whether the proposed guidance improves this allocation beyond conventional optimization. Regional reconstruction errors would further connect these changes to improved detail recovery.

  2. Incomplete training-time comparisons and missing timing ablations. Table 3 covers only a subset of the baselines. In particular, DoGaussian and CityGS-X are included in the main reconstruction-quality comparison, but their training times are not reported, leaving the practical efficiency comparison incomplete. The MixGS comparison additionally uses different hardware and achieves different reconstruction quality. A unified comparison of training time, quality, and peak GPU memory under consistent settings would better substantiate the efficiency claims. Furthermore, Tables 4 and 5 omit training times for the ablation variants, making it difficult to separate the acceleration attributable to frequency-guided allocation from that attributable to coarse-to-fine training and ACT.

  3. Lack of quantitative evaluation of geometric reconstruction quality. Fig. 1(c) highlights sharper rendered depth maps, and the training objective includes depth supervision. However, the evaluation primarily reports image-rendering metrics, without quantitative evidence that the reconstructed geometry is more accurate. Visually sharper depth boundaries alone do not establish geometric accuracy or completeness. On datasets with reference geometry, reporting F-score under a consistent reconstruction and evaluation protocol, together with comparisons against the relevant baselines, would help substantiate the claimed improvements in geometric detail.

Questions for authors
  1. Could the authors provide effective Gaussian scale distributions and normalized Gaussian densities for high- and low-frequency regions, comparing the full method with the variant without frequency-guided densification? This would help verify whether the proposed mechanism allocates compact Gaussians to detailed regions while avoiding redundant allocation in smooth regions.

  2. Could the authors quantitatively evaluate geometric reconstruction quality, for example by reporting F-score on datasets with reference geometry under a consistent evaluation protocol? Comparisons with relevant baselines would help establish whether the sharper rendered depth maps correspond to improved geometric accuracy and completeness.

Reviewer_TTi1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SpecGS studies memory-efficient Gaussian splatting for large urban scenes. It aggregates image-space Difference-of-Gaussians responses into per-Gaussian frequency statistics, uses these statistics to regularize a multiplicative support variable and lower densification thresholds in detail-rich regions, and adds a DoG-weighted reconstruction loss. A loss-based Adaptive Convergence Trigger (ACT) controls a three-stage resolution curriculum. The representation retains ordinary Gaussian rendering primitives. Experiments cover four primary urban scenes, additional GauU-Scene scenes and standard benchmarks, with supplementary comparisons against frequency-aware and distributed methods.

Critical strengths
  1. A practically relevant memory–quality trade-off. Table 2 reports 9.31–12.09 GB memory for the four main scenes and substantially fewer Gaussians than 3DGS and CityGaussian. The gains are not uniformly in PSNR, but several SSIM/LPIPS results improve. Additional GauU-Scene results in Table 12 support applicability beyond the primary scenes.

  2. Broader evaluation than the main tables alone suggest. The supplement includes Mip-Splatting (Table 8), FDS-GS (Table 13), standard scene benchmarks (Table 14), and controlled CityGS integrations with SSAGS (Tables 9–10). The paper also acknowledges MixGS's higher average PSNR and the fact that the adaptive threshold alone does not reduce Gaussian counts.

  3. Useful component analyses and transparent qualifications. Tables 4–7 examine architectural components, stage scheduling, bandwidth transition timing, and longer training. The authors correctly distinguish the frequency statistic from a learned frequency parameter and explicitly state that the oscillatory 1D basis is not the actual 3D primitive.

Critical weaknesses
  1. The novelty and relevant baseline comparison are incomplete. Frequency-guided densification and coarse-to-fine frequency regularization already occur in FreGS (CVPR 2024). FDS-GS (AAAI 2025) already links scale/density control with frequency-aware allocation. SpecGS does compare with FDS-GS on Rubble, which is valuable, but a single-scene numerical comparison does not replace a clear methodological distinction or wider controlled evidence. FreGS is not discussed or compared. Further, Table 14 evaluates only quality-preserving components of Taming-3DGS, omitting the resource-aware allocation central to the full method. A full budgeted Taming baseline is especially relevant to the central memory claim. These omissions limit the evidence for an advance beyond combining known frequency guidance, scale modulation, and progressive optimization.

  2. Bounded modulation does not itself bound effective Gaussian support. Equations 6–8 define effective scale as σ~k=skσk\widetilde{\sigma}_k=s_k\sigma_k, with an optimized original scale and 0<sk<20<s_k<2, while the regularizer penalizes only sk>1s_k>1. If both factors remain free, any positive effective scale can be represented at sk=1s_k=1, where this regularizer is zero; increasing the original scale can also compensate for reducing the multiplier. The bounded derivative in Appendix C therefore bounds sensitivity of the multiplier, not the effective support. An optimization benefit is possible, but the manuscript does not isolate it from ordinary scale reparameterization or directly regularizing effective scale. This is particularly consequential given the large memory change when support modulation is removed in Table 4.

  3. The experiments do not sufficiently isolate the origin of memory and speed improvements. The method adds monocular depth supervision, progressive resolutions, a different training schedule, and frequency adaptation simultaneously, but there is no depth-only baseline or comparison against an ordinary Gaussian model with the same depth supervision and resolution curriculum. Most architectural ablations are restricted to Sci-Art and omit final/peak primitive counts and time. Moreover, Eq. 9 lowers thresholds relative to standard densification for nonnegative frequency scores, whereas removing this rule increases memory from 10.82 to 20.33 GB (Table 4). Appendix D correctly identifies this as an emergent interaction, but densification/pruning trajectories and controlled ablations are needed to explain it. The ACT comparison also needs an explicitly specified fixed-stage schedule and matched compute budget: removing ACT changes PSNR by 2.42 dB (Table 5), yet the alternative timing is not described. A small sweep of the DoG midpoint does not address these uncertainties.

  4. Baseline protocols and reproducibility details leave material uncertainty. Section 4.1 defaults to one A5000, but Table 8 reports Mip-Splatting memory above 40–60 GB, Table 13 reports FDS-GS above 30 GB, and Table 4 reports a support-removal variant exceeding 24 GB while still assigning it a final PSNR. The hardware and completion protocol for these experiments need to be stated. The VastGaussian comparison disables appearance encoding without giving the full variant's performance. Training time comparisons mix reported RTX3090 and A5000 results, appropriately labeled but insufficient for a hardware-controlled speed claim; the matched Rubble comparison is stronger but covers one scene. Appendix A also omits numerical values for the frequency EMA rate and guidance strength, ACT window lengths and stage caps, DoG bandwidth endpoints, and the monocular depth model/loss implementation. Without these, reproducing the central trade-off or auditing baseline parity is difficult.

Questions for authors
  1. Are original Gaussian scales optimized freely after the support variable is introduced? What prevents σk\sigma_k from compensating for sks_k, and what happens with fixed sk=1s_k=1, a frequency-independent multiplier, and direct regularization of effective scale under the same optimization schedule?

  2. Please provide a controlled comparison with FreGS, the full budgeted Taming-3DGS, and FDS-GS across more than Rubble, using the same inputs, depth information, hardware, and comparable resource budgets. How much improvement remains when the baseline receives the same depth supervision and resolution curriculum?

  3. What fixed-stage schedule is used for “w/o ACT”? Please report actual transition iterations, densification/pruning intervals, final and peak Gaussian counts, memory trajectories, and training times for the principal ablations, preferably on another urban scene as well.

  4. Which hardware completed the experiments exceeding 24 GB, and how was the Table 4 overflow variant's PSNR obtained? Please specify whether memory is allocated/reserved, training peak or rendering memory, and per-device or aggregate for distributed methods. Please also supply the missing ACT, DoG, EMA, densification, and depth-supervision settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.