- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CALM, an offline calibrated layer-head allocation method for timing KV-cache merging in autoregressive video diffusion models. CALM first evaluates the sensitivity of different layers/heads, then dynamically programs non-uniform allocation under a fixed representative-token budget. In the self-forcing experiment of Wan2.1-T2V-1.3B, at (R=8), CALM reduced the full-clip LPIPS from 0.149 to 0.117 over 400 prompts, with even more pronounced improvements in later frames. The paper also analyzes allocation sensitivity, proxy effectiveness, and cross-model generalization.
Critical strengths
The experiments were well controlled. With the same logical cache budget, CALM consistently improves perceived quality and reconstruction metrics compared to uniform allocation, especially enhancing the quality of later frames.
The sensitivity analysis is quite valuable. The paper shows clear layer-head heterogeneity and validates the effectiveness of non-uniform allocation through ablation and cross-model experiments.
Critical weaknesses
The actual efficiency gains haven't been proven yet. The 5.6× compression reported in the paper only refers to logical cache reduction; the actual KV memory is still 12.9 GB, and CALM is slower than uniform merging. So the real memory and speed benefits haven't shown up yet.
It also lacks important recent baselines. The paper doesn't compare with recent AR-video KV-cache compression methods, so it's hard to judge CALM's actual quality-efficiency advantage.
Questions for authors
Is it possible to compare CALM with recent AR-video KV-cache compression methods under the same actual memory usage, and report on quality, latency, and peak memory?
How robust is the additive proxy objective? It's recommended to verify whether the proxy score can reliably predict the final generation quality on more held-out prompts and across more diverse allocations.
Thank you for your response. However, my concerns about novelty have not been fully addressed. Voita et al. (2019) [1], which you cite, already establishes that attention heads differ substantially in importance and can be selectively pruned. DiTFastAttn (NeurIPS 2024) [2] uses model output error to guide the selection of compression strategies across layers and timesteps, including for video generation. DiTFastAttnV2 (ICCV 2025) [3] goes further by using per-head attention-output error to calibrate and optimize compression configurations for visual generation. These works establish concrete precedents for sensitivity-aware compression, error-guided optimization, and head-wise compression configuration.
I appreciate the additional analyses and the gains over Uniform, and acknowledge that applying this approach to temporal KV merging extends this line of work. However, current results do not fully resolve my concern about the contribution of using attention-output error as a proxy to guide token allocation.
Thank you again for the additional comments and analyses. I welcome further discussion.
[1] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, Italy. Association for Computational Linguistics.
[2] Yuan Z, Zhang H, Lu P, et al. Ditfastattn: Attention compression for diffusion transformer models[J]. Advances in Neural Information Processing Systems, 2024, 37: 1196-1219.
[3] Zhang H, Su R, Yuan Z, et al. Ditfastattnv2: Head-wise attention compression for multi-modality diffusion transformers[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 16399-16409.