acceptodds
Withdrawn from ICLR 2027

Offline Makes Online Better: Calibrated Cache Compression for Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Token merging is an effective approach to reducing KV-cache memory consumption in autoregressive video generation by clustering temporally redundant tokens and retaining only a small set of representatives. Existing methods, however, typically apply a uniform merging ratio across all attention layers and heads. We observe that different layers and heads exhibit substantially different yet jointly correlated sensitivities to token merging, and that these sensitivity patterns remain consistent across prompts. Based on this observation, we introduce CALM, a Calibrated Allocation for Layer-Head Merging that uses offline calibration to improve online cache compression. Exhaustively evaluating all possible layer-head allocations is prohibitively expensive. To enable efficient calibration, we show that attention-output error strongly correlates with end-to-end video quality and can therefore serve as a lightweight proxy for allocation risk assessment. We evaluate different allocation strategies using this proxy and construct a fixed layer-head risk map offline. Given a cache budget at inference time, CALM uses dynamic programming to determine the best allocation of merging ratios. Experiments across three different autoregressive video models demonstrate that CALM consistently outperforms uniform token merging under the same memory budget, by reducing the LPIPS to original videos by around with higher video-quality scores.

Primary area:
generative models
Keywords:
Compression Algorithm, Video Generation, Token Merge

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DHCBOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CALM, an offline calibrated layer-head allocation method for timing KV-cache merging in autoregressive video diffusion models. CALM first evaluates the sensitivity of different layers/heads, then dynamically programs non-uniform allocation under a fixed representative-token budget. In the self-forcing experiment of Wan2.1-T2V-1.3B, at (R=8), CALM reduced the full-clip LPIPS from 0.149 to 0.117 over 400 prompts, with even more pronounced improvements in later frames. The paper also analyzes allocation sensitivity, proxy effectiveness, and cross-model generalization.

Critical strengths

The experiments were well controlled. With the same logical cache budget, CALM consistently improves perceived quality and reconstruction metrics compared to uniform allocation, especially enhancing the quality of later frames.

The sensitivity analysis is quite valuable. The paper shows clear layer-head heterogeneity and validates the effectiveness of non-uniform allocation through ablation and cross-model experiments.

Critical weaknesses

The actual efficiency gains haven't been proven yet. The 5.6× compression reported in the paper only refers to logical cache reduction; the actual KV memory is still 12.9 GB, and CALM is slower than uniform merging. So the real memory and speed benefits haven't shown up yet.

It also lacks important recent baselines. The paper doesn't compare with recent AR-video KV-cache compression methods, so it's hard to judge CALM's actual quality-efficiency advantage.

Questions for authors

Is it possible to compare CALM with recent AR-video KV-cache compression methods under the same actual memory usage, and report on quality, latency, and peak memory?

How robust is the additive proxy objective? It's recommended to verify whether the proxy score can reliably predict the final generation quality on more held-out prompts and across more diverse allocations.

Reviewer_QwhiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Calibrated Allocation for Layer-Head Merging (CALM), which uses offline calibration to improve online cache compression, with attention-output error serving as a lightweight proxy for allocation assessment. Several ablation studies demonstrate the effectiveness of the CALM strategy.

Critical strengths
  1. Token compression is an important task for video models.
  2. Offline token compression methods are efficient and have practical value.
Critical weaknesses
  1. The contribution is limited. The main contribution is using attention-output error as a proxy correlated with video quality, measured by LPIPS, and then applying token allocation algorithms. However, the correlation between attention-output error and video quality is intuitive.
  2. The claim in L200 that “The patterns are consistent across prompts” is insufficiently supported. The reported head-level correlations of 0.69 and 0.41 do not indicate particularly strong correlations. How were the 100 prompts selected? Are they diverse, or do they contain highly similar information that leads to consistent patterns? How sensitive are the results to prompt selection?
  3. What are the computational efficiency and resource requirements of the dynamic programming allocation method? Could its runtime grow prohibitively as video models scale, with more layers, more heads, or more tokens to allocate?
  4. Section 6.1 should specify which model was used for the main experiments.
  5. The paper should report more comparisons with the most relevant prior methods, particularly those discussed in the related work.
  6. In Table 3, CALM provides no significant gains or even performs worse for CausVid at R = 5 and 6, Causal Forcing at R = 2 and 3, and SGF at R = 2 and 3. Moreover, the transfer experiments in Table 7 show almost entirely negative gains. These results suggest limited applicability and transferability: the method works only under specific settings and requires recalibration for each model.
  7. Table 1 should include arrows indicating whether higher or lower values are better.
  8. The value mentioned in L456 does not appear in Table 4.
Questions for authors

Please see weaknesses, especially W1,2,3,5,6.

2 comments

Thank you for your response. However, my concerns about novelty have not been fully addressed. Voita et al. (2019) [1], which you cite, already establishes that attention heads differ substantially in importance and can be selectively pruned. DiTFastAttn (NeurIPS 2024) [2] uses model output error to guide the selection of compression strategies across layers and timesteps, including for video generation. DiTFastAttnV2 (ICCV 2025) [3] goes further by using per-head attention-output error to calibrate and optimize compression configurations for visual generation. These works establish concrete precedents for sensitivity-aware compression, error-guided optimization, and head-wise compression configuration.

I appreciate the additional analyses and the gains over Uniform, and acknowledge that applying this approach to temporal KV merging extends this line of work. However, current results do not fully resolve my concern about the contribution of using attention-output error as a proxy to guide token allocation.

Thank you again for the additional comments and analyses. I welcome further discussion.

[1] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, Italy. Association for Computational Linguistics.

[2] Yuan Z, Zhang H, Lu P, et al. Ditfastattn: Attention compression for diffusion transformer models[J]. Advances in Neural Information Processing Systems, 2024, 37: 1196-1219.

[3] Zhang H, Su R, Yuan Z, et al. Ditfastattnv2: Head-wise attention compression for multi-modality diffusion transformers[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 16399-16409.

AuthorsOpenReview ↗

Response to concerns about novelty

Thank you and this is a fair push. In short: we share a high-level principle with these works, not a method. They select per-layer configurations under per-layer error thresholds to cut attention FLOPs in bidirectional diffusion. We jointly allocate one temporal-KV budget across all layer–head pairs in autoregressive video. Our Layer Pyramid ablation shows the gain comes from this calibrated allocation, not from non-uniformity alone.

We agree that the underlying principle is established, and we build on it: attention units differ in sensitivity (Voita et al.), and output error can guide compression (DiTFastAttn/V2). None of the three works operates on token merging (Voita et al. prune heads; DiTFastAttn/V2 sparsify or reuse attention), so what we share with them is the high-level principle. Transferring such principles across settings is a common and productive route, and we see this pattern in many papers, with ideas moving from LLMs and ViTs to diffusion models and then to autoregressive video: token merging moved from ViTs (ToMe) to Stable Diffusion (ToMe for SD) and then to video (VidToMe), and KV-cache compression developed for LLMs (e.g., StreamingLLM, H2O) now has autoregressive-video counterparts (e.g., Forcing-KV). We believe a contribution in this line is best judged by whether the specific instantiation is non-trivial and yields non-obvious findings.

On the setting: we compress the growing temporal KV cache of a causal, autoregressive video generator on the number of temporal key–value representatives kept per head rather than attention FLOPs in bidirectional diffusion transformers.

On the claim: it is narrower than “error can guide compression,” and concerns what the calibration produces. Our proxy drives a single joint allocation: one shared representative budget distributed across all layer–head pairs to minimize the summed calibrated error. We solve it exactly by dynamic programming, though a global greedy solver reaches nearly the same map (Table 1), so the point is the joint formulation rather than the solver. DiTFastAttn/V2 instead decide layer by layer under per-layer error thresholds: DiTFastAttn greedily selects a strategy per (layer, step), and DiTFastAttnV2 solves a small integer program over each layer’s heads, progressing through timesteps and layers sequentially; neither allocates one budget jointly across layers. Because each (layer, head, count) is probed once and the budget enters only at allocation time, the same distortion table re-allocates to other budgets without re-measuring.

The Layer Pyramid ablation also speaks directly to this concern. A budget-matched, hand-designed non-uniform schedule, which already encodes the prior that earlier layers matter more, which ties uniform merging on 24 held-out prompts (LPIPS 0.192 vs 0.193, not significant), while our calibrated allocation reaches 0.156 (Δ = −0.036 vs. the pyramid, 95% CI [−0.051, −0.023]). The gain comes from calibration, not from non-uniformity per se i.e., from the error-proxy-guided allocation in question, not from a hand-set prior. We will add DiTFastAttn/V2 and make this delineation explicit in the revision, and are glad to run any further head-wise comparison you would find convincing.

Reviewer_bhHDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CALM calibrates layer–head sensitivity to temporal KV merging using dense-generation queries and recursively merged shadow caches. A dynamic program allocates a fixed total number of representatives using the resulting distortion curves, and the allocation is reused across prompts. The main evaluation compares allocations under the same FPS merge operator; additional experiments examine other generators and limited transfer. The demonstrated compression budget is logical: the implementation retains a full resident KV buffer.

Critical strengths
  1. The main experiment isolates allocation and demonstrates meaningful fidelity gains. Under the same operator and representative budget, Table 1 reduces full-video LPIPS from 0.149 to 0.117 over 400 prompts, approximately 21% relatively, with improvements reported on 356 prompts. Early LPIPS remains 0.015, and Table 2’s quality/consistency dimensions provide complementary evidence. These results establish a useful allocation effect even though they do not establish realized memory savings.

  2. The proxy has relevant equal-budget validation and informative controls. Appendix D/Table 5 compares seven fixed allocation maps at B=2880 on 102 common completed prompts, reporting Spearman correlation 0.96. Table 1 also compares Random, Head-only, Layer-only, Sensitivity, and Greedy allocations. Greedy’s LPIPS of 0.118 versus DP’s 0.117 supports the value of calibration rather than a large practical advantage from exact DP. Cross-model and operator-transfer experiments provide useful evidence about applicability boundaries.

Critical weaknesses
  1. The practical resource benefit remains unverified. Table 9 reports 12.9 GB resident KV buffers for all methods and peak GPU memory of 30.0/30.1/30.1 GB for Dense/Uniform/CALM. CALM takes 34.7 s per video, versus 30.2 s for Uniform and 31.0 s for Dense. Thus, 5.6× describes an ideal logical-capacity ratio. The paper acknowledges this in Sections 6.1, 7, and Appendix H, but the abstract’s memory-budget wording and Table 2’s “VRAM save” label can still be misleading. The equal-representative-budget fidelity gain is valid; what remains unclear is how it translates into a useful memory–quality trade-off, especially with per-layer padding to the widest head.

  2. The negative cross-model result should not be discounted on the stated grounds. For Causal Forcing at R=2, Table 3 reports a CALM−Uniform LPIPS difference of +0.037, with a paired 95% confidence interval of [+0.012,+0.065]. Sections 6.2 and F discount this result because Uniform is non-monotonic in R. Neither recursive heuristic merging nor nonlinear generation guarantees monotonic LPIPS, so this observation does not invalidate the fixed-budget comparison. Unless an evaluation error is found, it should be treated as a regression under this checkpoint and protocol. The significant positive results at other settings remain useful, but the abstract’s claim of consistent improvement across models is too broad.

Questions for authors
  1. Please align the abstract and Table 2 with the logical-budget qualification in Appendix H. Are compact-cache or padding-aware measurements already available that show the associated fidelity, memory, and latency trade-off?
  2. Is there a verified evaluation issue behind the Causal Forcing R=2 result? Otherwise, please retain it as a negative result and qualify the applicability claim. A repeated-seed check could help assess its robustness.
  3. For Table 5, how many prompts were attempted and completed for each map, why did only 102 enter the common set, and how does this set relate to the main 400-prompt evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.