- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SpecKCast, a diffusion cache-then-forcast framework for training-free diffusion sampling acceleration. The paper builds upon the existing diffusion caching framework like TaylorSeer and Spectrum, while applying Kalman filtering to the spectral decomposition of the diffusion trajectory. Experiments have been performed on text-to-image and text-to-video diffusion models, showing that the proposed approach achieve decent speedup while preserving a low cost in inference memory overhead.
Critical strengths
-
The paper is well motivated by the observation that different spatial-frequency components demonstrates different temporal dynamics in the diffusion trajectory. The proposed method leverages such an inductive bias to design frequency-dependent time coordinates.
-
The proposed approach reduces the inference-time memory overhead in the previous approaches that require to store the intermediate activations from the previous steps to fit the forecaster.
-
Experiments are performed across a suite of t2i and t2v models, showing the generality of the approach.
Critical weaknesses
-
The components proposed in the paper is not novel independently. Frequency decomposition, Kalman filtering, and adaptive scheduling are all either existing techniques or ideas previously discussed/touched in literature. While putting them together into a framework should be considered a contribution, the overall novelty is limited.
-
The ablation study suggests that the performance gain depends heavily on the empirical techniques like the time parameteraization and scheduling, instead of the Kalman forecaster itself. Table 3 shows that under the same frequency decomposition and scheduling, the Kalman and Taylor forecasters perform very similarly. However, enabling adaptive scheduling significantly improves the performance, which is not the key contribution endorsed by the present paper.
Questions for authors
Q1. Could the authors quantify more clearly whether the main performance improvement comes from the Kalman forecaster rather than adaptive scheduling? Currently it appears to be the latter.
Q2. What is the relationship between the predicted scheduling risk/uncertainty and the actual forecast error along the diffusion trajectory?
Q3. Could the authors provide an ablation on the size of the calibration set to see whether the offline calibration procedure is sensitive to the choice of the set or not?
As of now, I am giving a score of 2. However, I am willing to raise the score if the concerns are fully addressed.