- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Dynamical State-Space Diffusion (DSSD), an extension of the paper Dynamical Diffusion (DyDiff). The authors regard the exponential moving average process in DyDiff as a special state-space operator and introduce a learnable operator with a small number of tunable parameters. The operator kernel is defined by damped modes to combine decay rates and oscillations, and can be obtained via Cholesky decomposition. Experiments on field prediction, video prediction and multivariate time-series forecasting show the effectiveness of the method.
Critical strengths
- DSSD proposes a unified framework to generalize the temporal mixing in diffusion prediction models. It introduces the operator framework to explain why DyDiff's temporal mixing and provides a more general learnable form to improve it with a small parameter overhead.
- The paper provides extensive experiments on multiple prediction tasks, supporting the effectiveness of the method. The visualizations of temporal modes are interesting, inspiring further investigations into powerful temporal modes.
Critical weaknesses
- The paper lacks of necessary ablation studies on key designs, such as the number of temporal modes, fixed versus learnable parameters, and oscillatory versus non-oscillatory modes. More ablation studies would help readers clearly find out which factors actually make a difference.
- The spatio-temproral experiments are limited to low resolutions and short prediction horizons, leaving the method’s performance at higher resolutions and longer horizons unclear. This limits the evidence for the effectiveness.
Questions for authors
- Does the proposed framework also apply to a flow-matching setup? Recently, there are more world models with flow matching setup.
- For for long-horizon prediction, how does DSSD’s practical computational cost compare with original diffusion? Does Cholesky decomposition remain computationally feasible?
5 comments
Response to W2.
W2. To evaluate longer horizons and longer context windows within the rebuttal period, we ran additional experiments on the Turbulence dataset varying the prediction horizon (with fixed context ) and the context length (with fixed horizon ). All runs use the same architecture, training budget, and sampling schedule as the main experiments.
| Method | CRPS raw ↓ | CRPS w8avg ↓ | CRPS w8max ↓ | CSI w5 ↑ | Temporal variogram ↓ | Energy score ↓ | RMSE ↓ | |
|---|---|---|---|---|---|---|---|---|
| (paper) | DPM | 0.01148 | 0.00751 | 0.01065 | 0.9641 | 0.00262 | 0.01920 | 0.02600 |
| DyDiff | 0.01100 | 0.00752 | 0.01123 | 0.9674 | 0.00260 | 0.01791 | 0.02427 | |
| DSSD (Ours) | 0.00756 | 0.00440 | 0.00757 | 0.9779 | 0.00147 | 0.01178 | 0.01582 | |
| DPM | 0.01504 | 0.01268 | 0.01612 | 0.9682 | 0.00323 | 0.02399 | 0.03342 | |
| DyDiff | 0.01101 | 0.00798 | 0.01145 | 0.9761 | 0.00251 | 0.01771 | 0.02406 | |
| DSSD (Ours) | 0.00691 | 0.00421 | 0.00640 | 0.9847 | 0.00110 | 0.01078 | 0.01460 | |
| DPM | 0.01782 | 0.01407 | 0.01820 | 0.9618 | 0.00430 | 0.02955 | 0.04155 | |
| DyDiff | 0.01184 | 0.00824 | 0.01136 | 0.9748 | 0.00264 | 0.01896 | 0.02624 | |
| DSSD (Ours) | 0.00880 | 0.00593 | 0.00825 | 0.9816 | 0.00161 | 0.01419 | 0.02088 |
Prediction horizon scaling on Turbulence ().
| Method | CRPS raw ↓ | CRPS w8avg ↓ | CRPS w8max ↓ | CSI w5 ↑ | Temporal variogram ↓ | Energy score ↓ | RMSE ↓ | |
|---|---|---|---|---|---|---|---|---|
| (paper) | DPM | 0.01148 | 0.00751 | 0.01065 | 0.9641 | 0.00262 | 0.01920 | 0.02600 |
| DyDiff | 0.01100 | 0.00752 | 0.01123 | 0.9674 | 0.00260 | 0.01791 | 0.02427 | |
| DSSD (Ours) | 0.00756 | 0.00440 | 0.00757 | 0.9779 | 0.00147 | 0.01178 | 0.01582 | |
| DPM | 0.01384 | 0.01116 | 0.01430 | 0.9717 | 0.00283 | 0.02230 | 0.03016 | |
| DyDiff | 0.01144 | 0.00761 | 0.01115 | 0.9780 | 0.00232 | 0.01825 | 0.02435 | |
| DSSD (Ours) | 0.00668 | 0.00387 | 0.00701 | 0.9858 | 0.00119 | 0.01044 | 0.01394 | |
| DPM | 0.01152 | 0.00922 | 0.01305 | 0.9758 | 0.00238 | 0.01836 | 0.02466 | |
| DyDiff | 0.00989 | 0.00599 | 0.00940 | 0.9788 | 0.00224 | 0.01601 | 0.02154 | |
| DSSD (Ours) | 0.00610 | 0.00329 | 0.00636 | 0.9867 | 0.00106 | 0.00953 | 0.01274 |
Context length scaling on Turbulence ().
Across all horizon and context configurations, DSSD outperforms both DPM and DyDiff on all seven metrics. Notably, as the prediction horizon doubles to , DPM's error grows sharply (RMSE ), while DSSD remains the best at every horizon, with the lowest error (RMSE vs. for DyDiff) and the best temporal consistency (temporal variogram vs. for DyDiff and for DPM). Furthermore, DSSD effectively exploits longer context windows, monotonically improving all metrics as increases from 4 to 12. While training high-resolution backbones was not feasible within the rebuttal window, we note that DSSD's temporal operator acts purely along the time axis () independently of spatial resolution, making it directly compatible with higher-resolution architectures.
Response to Q1 and Q2.
Q1. Thank you for this question. DSSD can work with flow matching. The DSSD forward process is a linear interpolant between the data and the noise, , where the temporal operator enters through and . Proposition 4 already shows that our sampler follows a velocity field that transports the marginals of this interpolant. To move to flow matching, one can replace the variance-preserving path with a flow-matching path, such as the linear path of rectified flow, and tie the dynamics schedule to that path in the same way. The model is then trained to predict the velocity . The only difference from standard flow matching is that the velocity includes the time derivative of the temporal operator. The conditions that make the operator valid depend only on the dynamics schedule, not on the choice of path.
We also see world models as a promising application. Our operator is causal and can be extended one frame at a time: adding a frame does not change how earlier frames are corrupted, and the operator can be computed by a state-space recurrence. The corruption of each frame can therefore be defined without access to future frames, which fits world models that generate frames sequentially, often conditioned on actions. In our current experiments, the denoiser generates the future window jointly. Combining DSSD with frame-by-frame generation, as well as with flow matching, is future work.
Q2. The Cholesky decomposition remains feasible, and the overhead of DSSD over standard diffusion is small. In end-to-end terms, DSSD costs the same as standard diffusion: in Table 8 (one A100 GPU), the median time per training update of DSSD is within 5% of DPM on every dataset (e.g., 2.22 s vs. 2.22 s on SEVIR and 1.02 s vs. 1.06 s on Turbulence), and the reserved memory is identical. DSSD uses the same denoiser as standard diffusion and only adds the construction of the temporal operator. The operator is an matrix along the time axis of the window, shared across all spatial locations and channels, so its cost does not depend on the spatial resolution. We measured the time to build the operator (mode computation and Cholesky factorization) together with its backward pass, for a batch of 16 noise levels on a single GPU (ms):
| Window length | Mode computation | Cholesky | Cholesky (forward + backward) | Total (forward + backward) |
|---|---|---|---|---|
| 15 | 2.84 | 0.06 | 0.45 | 8.41 |
| 256 | 2.84 | 0.52 | 1.99 | 8.89 |
| 512 | 2.96 | 1.79 | 10.29 | 15.06 |
| 1024 | 3.21 | 7.78 | 69.23 | 77.02 |
Up to , the total cost stays nearly constant (8.4–8.9 ms), because it is dominated by the fixed cost of computing the modes. The Cholesky factorization grows as and becomes the main cost only beyond , reaching 77 ms at . Part of this cost comes from computing the factorization in float64 for numerical stability, which is slow on consumer GPUs. In float32, the Cholesky factorization takes 1.7 ms at . For the window lengths in our experiments (), building the operator takes 8–11 ms, about 1% of a training step (0.8–2.2 s). Since the learned parameters are fixed at sampling time and the operator depends only on the noise level, we cache it for each noise level on the sampling grid, so it is computed only once per noise level and shared across all samples in a batch. The covariance is also well conditioned: its condition number stays below 75 up to and decreases after training (34 at ). For even longer windows, the state-space recurrence of Proposition 2 applies the operator at a cost linear in without forming the matrix. We will add these measurements to the appendix.
Response to W1.
W1. We ran the three ablations. We ran the additional experiments on the two datasets (Turbulence and SEVIR) we could cover within the rebuttal period. All runs use the setting of the main experiments and change only the operator.
Number of temporal modes (Turbulence). We trained DSSD with P∈{1,2,6,12} modes; P=6 is the setting of the paper.
A single mode is the worst on all seven metrics, and the scores saturate from P=6: the differences between P=6 and P=12 are below 3% on every metric.
Fixed multimode operator and learned non-oscillatory modes. We ran the two variants on SEVIR: the six-mode operator frozen at its initialization, and a learned operator whose modes are constrained to be non-oscillatory (frequencies fixed to zero).
The learned oscillatory operator is better than both variants on all seven metrics. Removing the oscillatory modes increases CRPS from 3.975 to 5.691 and RMSE from 14.59 to 36.69. Freezing the operator degrades every metric, most strongly the temporal variogram (2.129 vs. 3.440) and RMSE (14.59 vs. 19.98).