acceptodds
Under review as a conference paper at ICLR 2027

OccamSplat: Learning Appearance Capacity for Dynamic Gaussian Splatting

Abstract

Capturing view- and time-dependent appearance with explicit Gaussian models can be costly when every primitive receives a full angular–temporal basis. Gaussians contribute unequally to rendered images, and their needs for directional and temporal variation differ. Uniform allocation therefore spends storage and computation on capacity that may offer little reconstruction benefit. The challenge is to identify useful extensions before fitting them: motion and residual magnitude are indirect cues, and updates to different Gaussians can interact. We introduce OccamSplat, which selects appearance capacity by verifying the rendering benefit of a concrete joint update before activation. Starting from base color, we probe candidate four-dimensional spherical-harmonic (4DSH) bands with zero coefficients, preserving the current rendering. Cross-observation gradient agreement yields bounded coefficient updates, which are accepted only when joint rendering reduces the loss on two disjoint sampled observation sets not used to construct them. We evaluate on Neural 3D Video, MeetRoom, and Technicolor light-field. On Neural 3D Video, under the same backbone and densification configuration, our method improves PSNR over Uniform 4DSH by 0.60 dB with 12.8% of non-DC coefficient rows active, while rendering at 257 FPS.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic Gaussian Splatting, Novel View Synthesis, Efficient Rendering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bCnXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how dynamic Gaussian splatting can accurately and efficiently model appearance that varies with time and viewpoint. Existing 4D Spherical Harmonics (4DSH) methods typically assign a full set of spherical harmonic coefficients to each Gaussian particle. However, Gaussians differ substantially in their contributions to the rendered image and the complexity of the appearance that they need to represent, leading to redundancy in the allocated coefficients.

To address this issue, this paper proposes OccamSplat, which core idea is to adaptively determine the appearance representation complexity required by each Gaussian based on its actual rendering benefits. Specifically, the method follows an “evaluate, select, and validate” strategy. It first computes gradients for candidate 4DSH bands, then selects updates based on the consistency of image loss gradients across views, and finally uses joint rendering validation to decide whether to activate new bands.

The authors conducted experiments on N3DV, MeetRoom, and Technicolor. On N3DV, OccamSplat increases rendering speed by 157% and reduces model size by approximately 19.4% compared with Swift4D, while also achieving a slight PSNR improvement of 0.09dB. These results demonstrate efficiency gains without sacrificing reconstruction quality.

Critical strengths

Traditional SH capacity allocation methods based on rendering residuals have difficulty directly determining whether additional appearance coefficients will improve reconstruction quality. In contrast, OccamSplat directly links 4DSH capacity allocation to the reconstruction benefits of specific parameter updates, rather than relying solely on the current rendering error. The most compelling evidence comes from Table 3. With the same 4DGS backbone and matched training time and appearance parameter budgets, OccamSplat achieves a PSNR of 32.20dB, outperforming residual based SH growth (31.78dB) and learnable band masks (31.85dB) by 0.42dB and 0.35dB, respectively. This further demonstrates the effectiveness of the allocation strategy.

Critical weaknesses
  1. OccamSplat does not show a substantial reconstruction quality advantage across datasets. On MeetRoom (Table 6), its average PSNR is 31.60dB, below Swift4D’s 32.05dB. On Technicolor (Table 7), its average PSNR is 33.77dB, below 4DScaffold-GS (34.11dB) and CEM-4DGS (34.04dB). More importantly, the additional MeetRoom and Technicolor experiments in the appendix report only PSNR, but not report FPS, model size,n or training time on these datasets. It is therefore unclear whether sufficient efficiency gains compensate for the lower reconstruction quality.

  2. The paper provides limited evidence of how well the appearance capacity allocation strategy transfers across backbones. The main experiments on N3DV, MeetRoom, and Technicolor use real multiview videos. However, the transfer to the SC-GS backbone in Sec. 5.5 is demonstrated using only a single failure case from D-NeRF, a monocular synthetic video dataset, and the speedup is limited. A comprehensive quantitative evaluation using data consistent with the main experiments is missing. The paper also does not examine whether incorporating the same allocation strategy into improved 4DGS methods, such as Ex4DGS, yields consistent gains. I notice that the code submitted in Supplemental includes an attempt to apply the strategy to Ex4DGS, can the authors show the comparison result on Ex4DGS backbone with the proposed strategy? It is difficult to assess the gains in reconstruction quality, model capacity, and efficiency on other backbones, or to determine how much the strategy’s effectiveness depends on the specific 4DGS implementation.

Questions for authors
  1. This paper uses joint rendering validation to decide whether to add 4DSH bands. In Sec. 4.4, updates to a group of candidate Gaussians are applied simultaneously, and the decision to activate the corresponding parameters is based on the change in the overall reconstruction loss, rather than an assessment of each Gaussian individually. Could the authors investigate whether Gaussians with small or even negative contributions can receive additional parameters when only some Gaussians in the group produce substantial gains? If this occurs, could the authors evaluate its impact on capacity allocation and reconstruction quality?

  2. Table 2 shows that removing the validation mechanism also increases the proportion of activated coefficients from 12.8% to 25.0% and 58.6%. I would like to understand whether the reported performance gains come from the joint validation criterion itself or from the stricter capacity constraint.

  3. Prior work has already explored appearance capacity allocation for individual Gaussians from several perspectives. The related work section discusses EfficientGS [1] and RDO-Gaussian [2], which adjust SH representations to accommodate differences in appearance complexity across Gaussians. RD4DGS [3] further extends adaptive SH allocation to dynamic scenes, and Table 3 includes comparisons with these approaches. However, the closely relevant GauRep [4], which is not cited in the paper, already uses loss gradients with respect to SH coefficients to adaptively determine whether to increase the SH order of each Gaussian. OccamSplat extends this direction to dynamic reconstruction and uses gradient consistency across observations and joint rendering validation to decide whether to activate new 4DSH bands. What additional benefits does this validation mechanism provide over existing methods, particularly for modeling dynamic appearance? A more detailed discussion of the differences from gradient driven SH order adaptation methods such as GauRep would help clarify the novelty of this work.

I would consider raising my recommended score if the authors can address my concerns appropriately.

References:

[1] "Efficientgs: Streamlining gaussian splatting for large-scale high-resolution scene representation." IEEE MultiMedia, 2025.

[2] "End-to-end rate-distortion optimized 3d gaussian representation." European Conference on Computer Vision. 2024.

[3] "Temporal smoothness-aware rate-distortion optimized 4D Gaussian splatting." Advances in Neural Information Processing Systems, 2026.

[4] "Effective Gaussian Management for High-fidelity Scene Reconstruction." IEEE Transactions on Visualization and Computer Graphics, 2026.


Post-rebuttal Comments

After reading the authors' rebuttal, I am convinced by the new experiments that the method works on various datasets, with good efficiency and performance trade-off. The authors’ explanation of how their approach differs from GauRep also helps clear up my concern. As such, I would like to raise my rating from Weak Rejection to Weak Acceptance.

Reviewer_vnpWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces OccamSplat, a method for adaptively allocating per-Gaussian appearance capacity in dynamic Gaussian splatting. Starting from DC-only color, the method progressively adds angular and temporal 4DSH bands. It probes candidate bands at zero coefficients and uses gradient agreement across training observations to select candidates and construct bounded coefficient updates. A joint proposal is activated only when it reduces rendering loss on two disjoint sets of training observations not used to construct it. Experiments demonstrate the benefits of adaptive capacity allocation in rendering quality, rendering speed, and training time. Component ablations and comparisons with alternative allocation strategies further support the effectiveness of the proposed design.

Critical strengths
  1. The paper addresses appearance modeling, an important aspect of novel view synthesis, by asking where additional expressive capacity is actually needed. The proposed allocation procedure is well-motivated and carefully designed: gradient agreement identifies promising updates across observations, and joint rendering verification checks their actual benefit before activating additional capacity.

  2. The experimental evaluation convincingly supports the design. Beyond improvements in reconstruction quality, rendering speed, and training time over the matched Uniform 4DSH baseline, the controlled capacity-reassignment experiment shows that the proposed allocation is more effective than random or visibility-based allocation at the same capacity. Component ablations further substantiate the roles of the main design choices.

Critical weaknesses
  1. The robustness of the allocation decisions to the chosen settings is not fully characterized. In particular, it remains unclear how changes in observation sampling, proposal step size, and verification threshold affect the resulting allocation and reconstruction quality. The reported results establish effectiveness under the chosen configuration, but offer limited guidance on sensitivity to these choices.

  2. The allocation comparisons and ablations support the effectiveness of the method, but provide less direct insight into its selection process during training. For example, the paper does not report proposal acceptance rates at the two verification stages or examine how the gradient-agreement criterion relates to subsequent joint rendering gains. Such diagnostics would make it easier to understand when and how the proposed selection and verification mechanisms are effective.

Questions for authors
  1. The SC-GS experiment provides some insight into applicability beyond the main backbone. Could the authors clarify what changes would be needed to apply the allocation procedure to angular SH bands in static 3D Gaussian splatting, and which assumptions or implementation requirements might restrict its applicability?

  2. Could the authors clarify how the renderer handles Gaussians with different active band counts? In particular, what changes are required relative to the standard 4DGS implementation to support compact storage and evaluation, and does this introduce additional indexing or branching overhead? Since OccamSplat also produces fewer Gaussians than Uniform 4DSH, it would be helpful to clarify the respective contributions of reduced appearance computation and reduced Gaussian count to the reported rendering speedup.

Reviewer_B9RzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper, OccamSplat, builds on 4D Gaussian Splatting (Yang et al., ICLR 2024) and proposes a way to selectively allocate the Gaussians' 4D spherical harmonics (4D SH) components, which modulate view-dependent and time-dependent appearances, rather than setting all Gaussians to have the same 4D SH order as in the default practice. During training, the proposed OccamSplat technique finds candidate Gaussians whose 4D SH order can be raised (in groups the authors call "bands") to improve image quality. To identify those candidates, the paper uses a zero-coefficient gradient probing technique to get the gradients for the higher-order SH coefficients, which are set to 0, and takes Gaussians whose loss gradients correlate positively in sign across training view-time observations as the candidates. Then, it validates that the these candidates would actually help with image quality when given higher-order SH coefficients by taking a small, bounded gradient update for the candidate Gaussians, and checking how the rendering loss actually changes. Results demonstrate that this selective 4D SH coefficient allocation technique improves image quality while having reduced storage size and increased rendering speed.

Critical strengths

The paper's main idea is intuitive and explained very well: we wish to selectively allocate higher-order 4D spherical harmonics coefficients to reduce size and accelerate training and inference, so the authors develop a smart technique to efficiently perform "guess and check" updates to subsets of Gaussians to see if giving them higher-order 4D HS coefficients can likely lead to worthwhile improvements in image quality. I find this idea neat and well-motivated.

Quantitative data with very informative baselines (namely the vanilla Uniform 4DHS variant and other concurrent works) show real PSNR improvement with either reduced size and/or improved rendering speed.

The experiments investigating visibility, motion, and appearance complexity provide good insight into why uniform allocation (Uniform 4DHS) is inefficient. Furthermore, the results suggest that using simply using visibility or motion heuristics cannot accurately predict the need for additional angular and temporal appearance capacity, which is the paper addresses with its proposed technique.

Critical weaknesses

Given that the idea is intuitive and straightforward (guessing and checking using gradients for proposals and finite differences for admission), I think the paper should go beyond showing that the formulation is effective, but further include detailed experiments of reasonable, alternative design choices to show to what extend the paper's design choices are necessary, not just sufficient, for good performance.

(1). For instance, I agree that the finite-step verification seems to be extremely important, however, I do not think its ablation is very solid. The authors ablate skipping verification with V in table 2, but that also reduces the number of total data samples used for validation (from 4 to 2). Does the usefulness of including V come from simply having two more views, or from the two-step verification procedure?

(2). Additionally, also on the topic of the verification step, I have concerns that randomly selecting views might include nearly redundant gradient information if the selected views are "close" in space and time. What are some worst-case scenarios if nearby views are chosen? Can views be chosen in a way that minimizes redundancy, and does that make verification more reliable?

(3). Also, if the goal of the finite-step verification is to approximate the gains from actually continuing the update with more 4D HS coefficients, then can the authors show how well the finite-step gains correlate with the gains from if normal optimization had actually been run for a few steps (or even longer) on the same candidates, with the same 4D HS coefficients enabled? Basically, how well do the one-step finite difference gains predict those from the counterfactual trajectory where the actual optimization had been carried out?

Questions for authors

The questions are the same as those included in my "weaknesses" assessment.

(1). I agree that the finite-step verification seems to be extremely important, however, I do not think its ablation is very solid. The authors ablate skipping verification with V in table 2, but that also reduces the number of total data samples used for validation (from 4 to 2). Does the usefulness of including V come from simply having two more views, or from the two-step verification procedure?

(2). Additionally, also on the topic of the verification step, I have concerns that randomly selecting views might include nearly redundant gradient information if the selected views are "close" in space and time. What are some worst-case scenarios if nearby views are chosen? Can views be chosen in a way that minimizes redundancy, and does that make verification more reliable?

(3). Also, if the goal of the finite-step verification is to approximate the gains from actually continuing the update with more 4D HS coefficients, then can the authors show how well the finite-step gains correlate with the gains from if normal optimization had actually been run for a few steps (or even longer) on the same candidates, with the same 4D HS coefficients enabled? Basically, how well do the one-step finite difference gains predict those from the counterfactual trajectory where the actual optimization had been carried out?

Minor questions: i. Why not allow arbitrary subsets of 4D SH coefficients instead of requiring a fixed progression order? Is it for the sake of tractability? ii. Can you include "#GS" for experiments in Table 2, Figure 7, and Table 4? It is hard to know how comparable different experiment variants are without knowing the number of Gaussian primitives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.