acceptodds
Under review as a conference paper at ICLR 2027

SpecKCast: Spectral Kalman Forecasting for Diffusion Acceleration

Abstract

Diffusion Transformers (DiTs) achieve high-fidelity image and video generation, but they suffer from substantial inference costs due to repeated transformer evaluations across many denoising steps. Feature caching reduces this cost by reusing features across denoising steps, and recent forecasting methods extend it by extrapolating features to skipped steps. However, these methods forecast the signal uniformly, even though its spatial-frequency components evolve at different stages of denoising. They also overlook that denoising progresses nonuniformly, so equal step intervals can correspond to very different amounts of change. These mismatches amplify forecast errors as more steps are skipped, limiting the achievable acceleration. We propose SpecKCast, a training-free method that forecasts the model output in the frequency domain. SpecKCast applies a 2D discrete cosine transform (DCT) to the output and tracks each coefficient with a lightweight Kalman filter along a band-specific time coordinate derived from the noise schedule. The filter maintains each coefficient's value and rate of change, and its forecast uncertainty determines when to evaluate the transformer. Experiments on FLUX.1-dev, Qwen-Image, Wan2.1-14B, and HunyuanVideo show that SpecKCast achieves a better trade-off between acceleration and fidelity than existing caching and forecasting methods, reaching up to speedup with negligible memory overhead.

Primary area:
generative models
Keywords:
Diffusion Models, Sampling Acceleration, Diffusion Forecasting, Training-free Inference, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BF7LOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpecKCast, a diffusion cache-then-forcast framework for training-free diffusion sampling acceleration. The paper builds upon the existing diffusion caching framework like TaylorSeer and Spectrum, while applying Kalman filtering to the spectral decomposition of the diffusion trajectory. Experiments have been performed on text-to-image and text-to-video diffusion models, showing that the proposed approach achieve decent speedup while preserving a low cost in inference memory overhead.

Critical strengths
  1. The paper is well motivated by the observation that different spatial-frequency components demonstrates different temporal dynamics in the diffusion trajectory. The proposed method leverages such an inductive bias to design frequency-dependent time coordinates.

  2. The proposed approach reduces the inference-time memory overhead in the previous approaches that require to store the intermediate activations from the previous steps to fit the forecaster.

  3. Experiments are performed across a suite of t2i and t2v models, showing the generality of the approach.

Critical weaknesses
  1. The components proposed in the paper is not novel independently. Frequency decomposition, Kalman filtering, and adaptive scheduling are all either existing techniques or ideas previously discussed/touched in literature. While putting them together into a framework should be considered a contribution, the overall novelty is limited.

  2. The ablation study suggests that the performance gain depends heavily on the empirical techniques like the time parameteraization and scheduling, instead of the Kalman forecaster itself. Table 3 shows that under the same frequency decomposition and scheduling, the Kalman and Taylor forecasters perform very similarly. However, enabling adaptive scheduling significantly improves the performance, which is not the key contribution endorsed by the present paper.

Questions for authors

Q1. Could the authors quantify more clearly whether the main performance improvement comes from the Kalman forecaster rather than adaptive scheduling? Currently it appears to be the latter.

Q2. What is the relationship between the predicted scheduling risk/uncertainty and the actual forecast error along the diffusion trajectory?

Q3. Could the authors provide an ablation on the size of the calibration set to see whether the offline calibration procedure is sensitive to the choice of the set or not?

As of now, I am giving a score of 2. However, I am willing to raise the score if the concerns are fully addressed.

Reviewer_3hYJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpecKCast, which accelerates diffusion transformers by tracking 2D DCT output coefficients with a band-specific Kalman filter and scheduling full evaluations via forecast uncertainty, outperforming prior caching and forecasting methods on FLUX.1-dev, Qwen-Image, Wan2.1-14B, and HunyuanVideo.

Critical strengths
  1. Interestingly combines frequency with Kalman filtering for prediction and achieves a high acceleration ratio.
  2. This paper analyzes the changes in high- and low-frequency trajectories (in Figure 2), providing good visualization.
Critical weaknesses
  1. Lack of Novelty: The frequency-based method is not novel, like FreqCa.

  2. Insufficient ablation: Table 3 only compares the predictor, scheduler, time coordinate, and frequency decomposition on FLUX 14 NFE; LL, τ\tau, TwT_w, β\beta, and the QscQ^{sc}, γ\gamma, ε\varepsilon calibrated by model and budget do not have sensitivity analysis, so these gains cannot be separated from hyperparameter selection.

Questions for authors
  1. The author needs to clarify the differences and improvements from the existing frequency-based methods.
  2. Analyze more hyperparameters to determine the actual points of innovation that lead to improvement.
Reviewer_ud2JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpecKCast is a training-free approach that speeds up diffusion transformers by predicting outputs at skipped denoising steps using DCT-based frequency tracking. It achieves up to ~6× speedup in image tasks with minimal extra memory.

Critical strengths
  1. SpecKCast is train-free and avoids storing large histories of intermediate features.
  2. The appendix explains the Kalman formulation and band-specific time coordinates in detail. It makes the method easier to understand.
Critical weaknesses
  1. Table 3 shows better results with warped log SNR than raw log SNR under adaptive scheduling. However, it is unclear whether the gains come from more accurate forecasts or better placement of transformer evaluations.
  2. The paper does not directly validate whether low risk predicts small forecast errors at skipped steps. Figure 12 is schematic, and online correction observes errors only at computed steps.
  3. Table 3 retains warped log SNR when frequency decomposition is removed, but the time coordinate is defined separately for each band. Please explain how this variant handles time coordinates.
  4. Appendix C gives the controller equations, but refers to B.4 for noise fitting, perceptual weight calibration, and threshold selection that B.4 does not describe. Please provide the missing calibration details.
  5. At about 14 NFE on Qwen-Image, the reported differences between SpecKCast and Spectrum are small: 0.001 each in SSIM and LPIPS and 0.01 each in ImageReward and CLIP Score.
Questions for authors

Please see Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.