acceptodds
Withdrawn from ICLR 2027

Learning 4D Splatting Kernels

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present a differentiable framework for learning spatial and temporal kernels directly from data in a 4D splatting-based pipeline for dynamic novel-view synthesis. Each primitive is equipped with spatial and temporal latents. The spatial latent and time-conditioned 3D attributes are mapped to a radially symmetric 2D kernel through neural projection and decoding, while a separate decoder maps the temporal latent to a temporal kernel. The networks are jointly optimized with 4D primitive attributes, including the latents. The learned kernels adapt to scene structure and dynamics, exhibiting diverse profiles beyond Gaussian forms. The effectiveness of our approach is demonstrated on standard dynamic novel-view synthesis benchmarks, comparing favorably against state-of-the-art techniques.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic scene reconstruction, Neural Splatting Kernel

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ak4ZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The prior work, LVDSK, proposes learning view-dependent splatting kernels using neural networks instead of fixed Gaussian kernels. This paper extends LVDSK to dynamic scene reconstruction using native 4D Gaussian primitives. It applies spatial kernel learning to time-conditioned 3D geometry and introduces a center-anchored temporal kernel. Experiments on N3V and D-NeRF show improved reconstruction quality over 4DGS and UBS.

Critical strengths
  1. Well-motivated by LVDSK's improvements on static scenes, this work presents a useful representation for dynamic scene reconstruction.
  2. The method achieves better reconstruction quality at similar memory footprints.

While I acknowledge these strengths, my initial score is primarily driven by the concerns below. I am willing to revise my recommendation toward acceptance if the authors adequately address my concerns during the rebuttal period.

Critical weaknesses
  1. While effective, the contribution appears incremental relative to LVDSK rather than a fundamentally new contribution.
    • As native 4DGS produces time-conditioned 3D primitives for rendering, applying LVDSK’s latent-based neural projection and radial decoding to these primitives is straightforward.
    • The temporal decoder closely follows LVDSK’s design.
    • The pre-training approach is also inherited from LVDSK.
    • On N3V, the temporal kernel yields a modest improvement of 0.18 dB, whereas the spatial kernel achieves a larger gain of 0.29 dB.
  2. The experiments do not sufficiently distinguish the proposed method from other alternative extensions.
    • The motivation for more flexible kernels is reasonable, but comparisons are restricted to 4D kernels (see Q1 for more detail).
Questions for authors
  1. Why should the learnable kernel be designed as 4D for dynamic scene reconstruction? LVDSK could be extended to alternative approaches, such as:
    • (a) Online per-frame 3DGS reconstruction (e.g., IGS [1] or more recent methods),
    • (b) Deforming canonical 3DGS over time (e.g., E-D3DGS [2] or more recent methods).
    • These comparisons would provide a more meaningful insight beyond simply extending LVDSK to dynamic scenes.
  2. How robust is the proposed method to varying lengths of the input video sequence? Specifically, does the method require hyperparameter re-tuning when scaling to significantly shorter or longer video sequences?

[1] Yan et al. "Instant Gaussian stream: Fast and generalizable streaming of dynamic scene reconstruction via Gaussian splatting." CVPR 2025.

[2] Bae et al. "Per-Gaussian embedding-based deformation for deformable 3D Gaussian splatting." ECCV 2024.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ak4Z

Thank you for your constructive comments and suggestions.

Response C1. (W1) Novelty beyond LVDSK.

Please refer to Response A1.

 

Response C2. (W2, Q1) Choice of 4D representation.

We adopt 4DGS because its unified representation provides a natural foundation for the joint kernels learning. Our framework could also be incorporated into other dynamic representations.

 

Response C3. (Q2) Robustness to sequence length.

Our method does not assume a fixed number of frames. We normalize sequence timestamps to a fixed range and evaluate temporal kernels continuously using normalized relative time instead of frame indices, without requiring changes to the representation or network architecture for different sequence lengths.

We evaluated our method on three real-world datasets (N3V, Technicolor, and Google Immersive) with varying sequence lengths. Our method achieves high reconstruction quality without sequence length specific hyperparameter tuning.

Reviewer_bJCHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Extending native 4D Gaussian splatting, this work incorporates learned spatial and temporal kernels. Specifically, a projection network maps per-primitive spatial latents and time-conditioned geometry to view-dependent radial spatial kernels, while a separate decoder models temporal responses through a center-anchored energy parameterization. The kernel networks, primitive geometry, and latent attributes are jointly optimized. Experiments on Neural 3D Video and D-NeRF show that the proposed approach achieves superior reconstruction quality over evaluated 4DGS and UBS implementations, while maintaining competitive performance at comparable serialized model sizes.

Critical strengths
  1. The proposed method directly embeds learnable kernels into native 4D geometry, avoiding the overhead of a heavy, standalone deformation network. In addition, the center-anchored temporal parameterization is well-motivated, and the paper includes targeted component ablations that support several design choices, although the overall contribution remains partly entangled with optimization changes.
  2. The approach achieves consistent reconstruction gains on Neural 3D Video, boosting the average PSNR from 31.92 dB to 32.53 dB over the evaluated 4DGS baseline alongside corresponding improvements in SSIM and LPIPS. Furthermore, accounting for both primitive attributes and network weights when comparing serialized model sizes provides a rigorous control for evaluating representation efficiency.
Critical weaknesses
  1. It is hard to tell how much of the overall gain over 4DGS actually comes from the learned kernel formulation versus the accompanying training heuristics. Because the spatial projection and decoder closely follow LVDSK, the main novelty rests on the temporal extension and its joint optimization with 4D primitives. Yet in the ablations, replacing the learned temporal kernel with a standard Gaussian drops PSNR by only 0.18 dB (from 32.53 to 32.35 dB). Meanwhile, the full pipeline also modifies opacity resetting, regularization, and initialization, without testing a baseline that uses standard Gaussian kernels for both space and time under those exact same optimization tricks. In addition, the default rasterizer uses linear interpolation between only two uniformly spaced radial samples. If those samples sit at the interval endpoints, the rendered radial profile is simply linear within its support—making it unclear why a full neural radial decoder is needed instead of directly predicting the two endpoint values.

  2. The submitted code provides per-scene configurations, kernel-pretraining commands, optimization settings, and data-split conventions, which substantially improves reproducibility. My remaining concern is the configuration of the UBS comparison: the reported results differ from the original UBS-7D numbers, but those numbers may reflect different dimensionality, appearance models, primitive budgets, or preprocessing. Please identify the exact UBS variant and configuration and explain the relevant protocol differences. The numerical discrepancy alone does not establish an unfair reproduction.

Questions for authors
  1. What PSNR/SSIM does the model achieve if both spatial and temporal kernels are fixed to standard Gaussians while keeping the exact same opacity reset policy, regularization, and initialization as your full method?

  2. The submitted rasterizer samples the spatial profile at radii 0 and 1 and interpolates linearly by default. What benefit does the full radial decoder provide over directly predicting the two responses, and how sensitive are quality and runtime to a larger sampling budget?

  3. Which UBS variant and configuration were evaluated, and which protocol differences explain the discrepancy from the original UBS-7D results?

1 comment
AuthorsOpenReview ↗

Response to Reviewer bJCH

Thank you for your constructive comments and suggestions.

Response B1. (W1, Q1) Contribution of learned kernels.

We replaced both learned spatial and temporal kernels with Gaussian functions in our full implementation, keeping all other settings unchanged. On the N3V dataset, this ablation achieves 32.11 dB PSNR, compared with 32.53 dB for our full method. Our learned kernels achieve a 0.42 dB PSNR improvement under identical training settings.

 

Response B2. (W1, Q2) Spatial decoder and sampling.

Please refer to Response A4 for the spatial sampling strategy and its efficiency trade-off.

Our full radial decoder represents the spatial kernel as a continuous radial function, while S=2S=2 specifies the number of samples used during rasterization. This allows us to change the number of samples without modifying the network architecture.

 

Response B3. (W2, Q3) UBS reproduction and evaluation settings.

We use the official implementation of UBS. For D-NeRF, the main difference is image resolution. The original UBS paper evaluates at half resolution (r=2r=2), while we use the original resolution (r=1r=1). Using the official code at r=2r=2, we can closely reproduce the results reported in the original paper.

For N3V, although the original paper reports quantitative results, the released code does not provide the specific experimental configuration used to obtain them. We therefore could not reproduce the exact reported performance using the available official implementation.

Reviewer_PqYpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper learns primitive-specific spatial and temporal kernels directly from data in a 4D Gaussian Splatting pipeline for dynamic novel-view synthesis, replacing the fixed Gaussian kernels of 4DGS. Its main components are:

  1. A neural projection network uses spatial latent features and time-conditioned 3D primitive attributes to produce a view-dependent, radially symmetric 2D kernel. 2)A separate decoder predicts primitive-specific temporal responses using normalized relative time and temporal latent features. 3)Efficient rendering: Spatial responses are pre-sampled into a lookup profile, while temporal responses are evaluated once per primitive per query timestamp.
  2. Joint optimization: Primitive parameters, latent features, and shared neural networks are optimized using image reconstruction losses. The experiments use six scenes from Neural 3D Video (N3V) and eight scenes from D-NeRF, with comparisons against 4DGS and Universal Beta Splatting (UBS). The reported average results are promising.
Critical strengths

The paper identifies a reasonable limitation of existing Gaussian splatting approaches: fixed Gaussian functional forms restrict the flexibility of primitive-level spatial and temporal responses. Learning these responses from data is a natural research direction, and integrating them into a native 4D representation is technically relevant to dynamic novel-view synthesis. The paper's motivation is easy to follow from the introduction through the method section. The method retains the 4D Gaussian representation and its time-conditioning mechanism rather than replacing the entire reconstruction pipeline. The division into spatial and temporal decoders is understandable, and the resulting responses are integrated into the existing alpha-compositing formulation. The temporal center anchoring is a particularly useful design choice. Fixing the response at the temporal center removes one degree of freedom that would otherwise be entangled with primitive opacity.

Critical weaknesses
  1. The spatial kernel is a direct port of LVDSK. The only change is swapping the geometric inputs for the time-conditioned covariance. The paper says "Following LVDSK" but never states the improvement explicitly. The paper explicitly builds its spatial-kernel formulation on Learning View-Dependent Splatting Kernels (LVDSK), while extending the setting to dynamic 4D representations and adding learned temporal kernels.
  2. UBS (Liu et al., 2026) explicitly models temporal dependency within a unified spatial/temporal/angular kernel family, and 4C4D (Zhou et al., 2026) already learns a neural decaying opacity function over time within native 4D Gaussians. Both are cited in Related Work, but neither is discussed as a direct prior for the temporal mechanism.
  3. The experimental comparison is too narrow for the strength of the claims. Related Work itself lists 4D-RotorGS, 4C4D, etc. However, none appear in Table 1.
  4. The claim "maintaining competitive rendering speed" is misleading. Does the spatial kernel remain sufficiently expressive during actual rendering?
  5. The authors disable periodic opacity resetting and introduce continuous base-opacity regularization. They explain that the original opacity-reset mechanism can suppress primitives whose learned kernel responses are small. This is a plausible implementation decision, but it also changes the optimization procedure independently of the kernel representation. The principal comparison against 4DGS therefore does not fully isolate the effect of learned kernels unless the optimization differences are controlled. The existing ablations test individual kernel substitutions, but the paper does not clearly report a matched all-Gaussian baseline that retains the proposed training modifications while replacing both learned kernels with Gaussian responses.
Questions for authors
  1. What exactly does S=2 mean in the spatial lookup table? Does the rasterizer use only two radial sample values per primitive? If so, can the authors report a sample-count ablation and show the actual interpolated kernels used during rendering?

  2. How much of the gain comes from the learned kernels rather than the revised training procedure? Can the authors compare against an all-Gaussian 4DGS variant with the same opacity-reset policy, opacity regularization, and remaining optimization settings?

  3. What is the practical trade-off between reconstruction quality and rendering speed? Can the authors explain the approximate slowdown relative to 4DGS and report comparable timing measurements with full implementation details?

  4. What is the essential novelty beyond LVDSK-style spatial kernels and a learned temporal response? Can the authors demonstrate specific dynamic scene cases that the proposed formulation models better than existing analytical or learned-kernel alternatives?

1 comment
AuthorsOpenReview ↗

Response to Reviewer PqYp

Thank you for your constructive comments and suggestions.

Response A1. (W1, Q4) Novelty beyond LVDSK.

We extend LVDSK's learned spatial kernel formulation to a unified 4D Gaussian representation and introduce a learned temporal kernel for each primitive. For the temporal kernel, we employ scale-normalized relative time and center anchoring to model temporal variations while reducing ambiguity with base opacity. Figure 3 demonstrates improved reconstruction of dynamic details, including flames and moving objects.

 

Response A2. (W2) Relationship to UBS and 4C4D.

UBS uses analytical Beta kernels with learnable parameters. In contrast, we learn spatial and temporal kernel profiles directly from data instead of hand-crafted functional forms.

4C4D uses a neural function to modulate the amplitude of a Gaussian temporal kernel while preserving its shape. In contrast, we directly learn the temporal kernel profile itself.

 

Response A3. (W3) More comparisons.

To broaden the experimental evaluation, we included additional methods and datasets in our comparisons. The following table reports the dataset averages. Dataset abbreviations: DN = D-NeRF, TC = Technicolor, and GI = Google Immersive. Metrics: P = PSNR↑, S = SSIM↑, L = LPIPS↓.

N3V·PN3V·SN3V·LDN·PDN·SDN·LTC·PTC·STC·LGI·PGI·SGI·L
4DGS31.920.9460.10331.860.9610.04532.570.9210.13226.760.8500.292
Ex4DGS31.980.9530.09529.310.9650.04235.340.9570.08326.830.9010.180
STG31.900.9510.10723.760.9140.09535.290.9550.07528.550.9170.156
UBS30.620.9390.11732.250.9670.04235.430.9570.05929.210.9400.110
Ours32.530.9540.09132.980.9670.04135.640.9600.07229.520.9450.117

 

Response A4. (W4, Q1, Q3) Spatial sampling and rendering efficiency.

With S=2S=2, we pre-sample the 1D radial profile at r~=0\tilde r=0 and 11, and compute the final spatial kernel via linear interpolation during rasterization. LVDSK provides a sample-count ablation (S=2,4,8S=2,4,8), showing that more samples can improve reconstruction quality, especially at smaller memory budgets, but reduce rendering speed. It also demonstrates that S=2S=2 suffices to produce results with high quality.

On the N3V dataset, our full implementation achieves real-time rendering at approximately one-fifth the speed of 4DGS and twice the speed of UBS. The main computational overhead comes from neural projection and kernel decoding. We have not optimized these networks for inference in this work, while network distillation offers a promising direction for further acceleration.

 

Response A5. (W5, Q2) Contribution of learned kernels.

We applied our modified training procedure to the original 4DGS implementation. On the N3V dataset, the average PSNR changes from 31.92 to 31.95 dB. This indicates that the training modifications are primarily intended to accommodate our learned kernels, and that our performance gains mainly come from the learned kernels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.