acceptodds
Under review as a conference paper at ICLR 2027

Positional Specialization for Autoregressive Video KV-Cache Compression

Abstract

KV-cache compression for autoregressive video is usually framed as deciding how much of the past to preserve. Using persistent, position-matched interventions, we show instead that different temporal positions require different kinds of information. Distant states, including pinned and mid-lag positions, can be spatially collapsed from 1,560 tokens to a single token with negligible loss, whereas the same collapse on recent states causes severe frame-to-frame flicker. Spatial compressibility, however, does not imply dispensability: evicting distant states degrades image quality and nearly halves motion, whereas retaining each as a single token does not, and replacing them with another video's cache shifts output semantics increasingly with semantic distance. These findings motivate AxisKV, a two-axis allocation that spatially compresses distant states while quantizing recent states. At cache compression on Self-Forcing, AxisKV outperforms matched-memory uniform pooling and, against uniform quantization, trades a loss in aesthetic quality for higher image quality; it also remains usable at , where uniform quantization has no valid operating point. The positional specialization transfers across related autoregressive video generators.

Primary area:
generative models
Keywords:
Video generation, diffusion model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_58kAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the different roles of KV-cache positions in autoregressive video generation. Based on causal analysis, the authors propose AxisKV, which applies spatial compression to distant frames and quantization to recent frames. Experiments show that AxisKV reduces KV-cache memory while maintaining reasonable generation quality.

Critical strengths

The paper provides an interesting analysis of how different KV-cache positions affect video generation. The proposed method is simple and training-free, and achieves good memory compression with limited quality degradation on self-forcing.

Critical weaknesses
  1. The proposed method mainly combines existing spatial pooling and quantization techniques based on temporal positions. Although the analysis provides some useful insights, the compression strategy itself is relatively simple and does not introduce new compression techniques.

  2. The paper only reports calculated KV-cache memory reduction, without measuring actual peak GPU memory usage, inference latency, or FPS.

  3. The generalization is not fully convincing. AxisKV shows noticeable quality degradation on CausVid, especially in aesthetic and imaging quality.

Questions for authors
  1. Could the authors provide actual GPU memory usage and inference latency or FPS comparisons?

  2. Why does AxisKV perform worse on CausVid? Can the authors explain when the proposed compression strategy may not work well?

  3. Since all four evaluated models are based on Wan2.1, could the authors provide experiments on other video generation models with different backbones?

5 comments

We thank the reviewer for recognizing the value of our causal analysis, the simplicity of AxisKV, and its memory compression effectiveness. We address the concerns below with additional system-level measurements and clarification of our contributions and limitations. Changes in the revised manuscript are highlighted in blue for ease of reference.

W1: Novelty of the Compression Strategy

We agree that the spatial pooling and quantization operators used in AxisKV are not themselves new. This is an intentional design choice, as stated in Introduction. Our contribution lies in identifying position-dependent information requirements through causal analysis and translating these findings into position-specific compression-axis selection**, rather than proposing new compression operators.

Specifically, our persistent, position-matched interventions reveal two key findings:

  1. Spatial compressibility does not imply dispensability. Pinned states can tolerate compression to a single spatial token per frame, whereas evicting them reduces dynamic degree by 7.38 points and increases chunk discontinuity by 0.404.
  2. Spatial resolution and numerical precision have different positional requirements. Collapsing recent states to one spatial token reduces subject consistency by 2.45 points, while 2-bit quantization causes a substantially smaller decrease of 0.32 points.

These findings directly motivate AxisKV's allocation. Furthermore, under the same 8× historical KV payload budget and implementation, uniform pooling and uniform quantization degrade all three primary consistency dimensions, whereas AxisKV shows no statistically significant differences from the uncompressed baseline on these dimensions.

The novelty is therefore the causal and experimental connection between positional information requirements and compression-axis selection. Using established compression operators allows us to isolate the contribution of the allocation itself without introducing additional factors.

W2/Q1: Actual GPU Memory Usage, Latency, and FPS

We thank the reviewer for emphasizing the importance of system-level measurements. To address this concern, we additionally implemented write-time compressed KV storage and measured GPU memory usage, generation latency, and FPS on a single GH200 GPU. Unlike the read-time compression used for controlled quality analysis in the original paper, this implementation physically stores historical KV states in compressed form.

For the nominal 12× AxisKV configuration, we obtain:

MetricUncompressedAxisKV (Write-time)
Historical KV memory2,468 MiB258.7 MiB
Peak GPU memory23,810 MiB21,710 MiB
Generation time32.4 s53.6 s
FPS14.728.90

Actual historical KV storage is reduced by 9.54×, closely matching the approximately 9.6× reduction predicted in Appendix J after accounting for metadata. End-to-end peak GPU memory decreases by approximately 2.1 GiB (8.8%). The smaller reduction in peak memory is expected because model weights, activations, and other buffers are not compressed alongside the historical KV cache.

This memory saving comes with additional inference overhead: generation time increases from 32.4 s to 53.6 s. The current implementation reconstructs compressed KV states during reading while retaining the original attention sequence length. Thus, AxisKV provides a memory–latency trade-off rather than inference acceleration.

We have added a brief summary of these system-level results in Section 4 and included the detailed GPU memory, latency, and FPS measurements, along with implementation details and limitations, in the revised Appendix J.

W3/Q2: Performance Degradation on CausVid and Applicability

We agree that AxisKV exhibits noticeable quality degradation on CausVid. As discussed in Appendix F, independent pinned-state spatial-collapse probes reveal different compression sensitivities across generators. CausVid exhibits the largest semantic probe tax (−0.111), compared with Self-Forcing (−0.059), Rolling Forcing (−0.020), and LongLive (−0.018), consistent with its larger quality degradation under AxisKV. To further investigate this sensitivity, we conducted an additional pinned-state resolution ablation on CausVid, retaining 1, 28, or 104 spatial tokens per pinned state. Increasing the resolution from 1 to 104 tokens recovers only 17% of the aesthetic-quality loss and approximately 10% of the semantic-consistency loss, while all tested configurations remain substantially behind uniform 2-bit quantization.

AxisKV's fixed allocation is derived from causal interventions on Self-Forcing, while its application to other generators evaluates the transferability of this allocation. The CausVid results highlight the broader value of our causal probing approach: it characterizes position-dependent information requirements and helps identify generator-specific compression sensitivity. These findings can inform the design of model-specific compression allocations, rather than assuming that a fixed configuration is universally optimal.

We have further highlighted the role of causal probing in assessing model-dependent compression sensitivity in Section 4 and included the additional CausVid ablation in the revised Appendix F.

Q3: Generalization Beyond Wan2.1

We thank the reviewer for raising this important question. Our evaluation covers four representative autoregressive video generators, Self-Forcing, CausVid, Rolling Forcing, and LongLive, all based on Wan2.1. This backbone is also widely adopted in recent autoregressive video KV-cache management works, including Forcing-KV, Dummy Forcing, Deep Forcing, Relax Forcing, and Pyramid Forcing. We focus on this established setting to facilitate comparisons with existing methods and investigate how positional information requirements vary across generators under a shared backbone and cache architecture.

We acknowledge that generalization to different video generation backbones remains an open question. Extending AxisKV to other architectures would require examining their positional compression sensitivity and adapting the allocation accordingly, rather than directly transferring a fixed configuration. We view this as a meaningful direction for future work. We have clarified the Wan2.1-based scope of our evaluation in the revised Introduction.

Thank you for your detailed response. I think this is an interesting attempt, but there is still room for improvement. AxisKV reduces peak GPU memory usage by 8.8%, but the inference FPS drops from 14.72 to 8.90, which I don't think is an ideal trade-off.

Furthermore, the authors' explanation seems to be discussing a general property of autoregressive video generators, but additional CausVid experiments show that the proposed method does not generalize consistently across different generators. And the current experiments are all based on Wan2.1, without evaluation on newer backbones such as Wan2.2.

This doesn't change the fact that I think the motivation of this work is interesting. I hope the authors can propose a more effective or generalizable method based on the findings. However, I still lean towards rejection unless the authors can further address my concerns.

By the way, if the text in the figures could be selected and copied, it would greatly increase the readability of the paper.

Reviewer_8iEmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies that different temporal positions in autoregressive video KV caches require different types of information, rather than simply differing in importance. To this end, the authors propose a training-free, position-dependent KV cache compression strategy to systematically investigate spatial detail, numerical precision, content identity, and slot presence. These experiments provide a clear rationale for the proposed position-dependent compression strategy. Based on these findings, the authors propose AxisKV, which spatially compresses distant cache states while preserving the spatial resolution of recent states and reducing their numerical precision. This design is simple and training-free, and effectively exploits the different information requirements of cache positions. Experiments demonstrate the effectiveness of AxisKV under 8× and 12× compression, while the ablation studies further validate the importance of retaining distant cache states rather than simply evicting them.

Critical strengths

Strengths

  • The paper is well-written and easy to follow.

  • The method proposed in this paper is simple yet performs exceptionally well, saving up to 12 times the memory. Furthermore, AxisKV requires no training and can be easily applied to other video generation models.

  • The authors are the first to observe video generation models from the perspective of the location of key-value (KV) pairs, and through observational experiments, they discovered the relationship between location and KV pairs. This motivation and finding are reasonable and convincing.

Critical weaknesses

Weaknesses

  • Limited generalization. Although AxisKV is evaluated on four autoregressive video generators, all four are based on the same Wan2.1 prior. Moreover, AxisKV suffers substantial aesthetic and imaging quality degradation on CausVid, suggesting that the proposed positional allocation may not generalize well across different generation architectures.

  • Fixed positional allocation. AxisKV relies on a predefined positional allocation derived from probing Self-Forcing. Although the authors evaluate the method on multiple generators, the substantial quality degradation on CausVid suggests that the optimal compression allocation may be model-dependent. It would be valuable to investigate whether the allocation can be adapted to different generators based on their cache requirements.

Questions for authors
  1. Position-based kv compression strategies have been a long-standing issue in large language models or image generation models, such as streaming-llm. Since models like self-forcing are AR-like models, could the authors clarify the differences and connections between AxisKV and these other works?
  2. Fixed positional allocation. AxisKV relies on a predefined positional allocation derived from probing Self-Forcing. Although the authors evaluate the method on multiple generators, the substantial quality degradation on CausVid suggests that the optimal compression allocation may be model-dependent. It would be valuable to investigate whether the allocation can be adapted to different generators based on their cache requirements.
  3. As shown in Table 2, AxisKV still achieves results similar to Full KV under 8x and 12x compression. However, since AxisKV only compresses the KV or reduces precision, and even directly compresses distant history into a single token, could the authors explain why AxisKV performs better on certain metrics?
  4. Could the authors investigate whether the optimal compression strategy depends on video content or generation state, rather than solely on cache position? For example, distant cache slots may primarily preserve scene context in videos with static backgrounds, but may still contain important semantic information in videos with frequent scene changes or changing subjects.
4 comments

We thank the reviewer for the positive assessment of our work and the constructive comments. We address the concerns through additional analysis and clarifications. All changes in the revised manuscript are highlighted in blue for ease of reference.

W1/W2/Q2: Generalization and Model-Dependent Allocation

We thank the reviewer for raising these questions. Our evaluation covers four representative autoregressive video generators—Self-Forcing, CausVid, Rolling Forcing, and LongLive—all based on Wan2.1. This backbone is also widely adopted in recent autoregressive video KV-cache management works, including Forcing-KV, Dummy Forcing, Deep Forcing, Relax Forcing, and Pyramid Forcing. We focus on this established setting to facilitate comparisons with existing methods and examine how positional information requirements vary across generators under a shared backbone and cache architecture.

Regarding fixed allocation, AxisKV's compression configuration is derived from causal interventions on Self-Forcing, while its application to the other generators evaluates the transferability of this allocation. As discussed in Appendix F, independent pinned-state spatial-collapse probes reveal different sensitivities across generators. CausVid exhibits the largest semantic probe tax (−0.111), compared with Self-Forcing (−0.059), Rolling Forcing (−0.020), and LongLive (−0.018), consistent with its larger quality degradation under AxisKV. We further conducted a pinned-state resolution ablation on CausVid, where increasing the retained tokens from 1 to 104 recovers only 17% of the aesthetic-quality loss and approximately 10% of the semantic-consistency loss.

These findings highlight the broader value of our causal probing approach: it characterizes position-dependent information requirements and helps identify generator-specific compression sensitivity, rather than assuming that a fixed allocation transfers universally. AxisKV is one concrete allocation derived from these findings, and adapting it to other generators would require accounting for their measured sensitivities.

We have clarified the Wan2.1-based evaluation scope in the Introduction and further highlighted model-dependent compression sensitivity and the role of causal probing in Section 4 and Appendix F.

Q1: Connection to StreamingLLM and Other Position-Based KV-Cache Methods

We thank the reviewer for highlighting this connection. StreamingLLM and related methods demonstrate the importance of retaining initial attention sinks and recent context for long-context generation. Our pinned and sliding-window cache layout follows a similar principle. Our contribution is not the introduction of position-dependent cache retention, but the characterization of what information different cache positions need to preserve. In autoregressive video generation, each cached latent frame contains a spatial grid of 1,560 tokens, allowing spatial resolution and numerical precision to be investigated as distinct compression axes.

Our causal interventions reveal two key findings:

  1. Spatial compressibility does not imply dispensability. On Self-Forcing, pinned states can tolerate extreme spatial compression, whereas evicting them reduces dynamic degree by 7.38 points and increases chunk discontinuity by 0.404.
  2. Different positions favor different compression axes. Collapsing recent states to one spatial token reduces subject consistency by 2.45 points, while 2-bit quantization causes a much smaller decrease of 0.32 points.

These findings extend the perspective of position-dependent cache retention to position-dependent information requirements, which directly motivates AxisKV's spatial-versus-precision compression allocation.

In the revised version, we clarified this in the Related Work section.

Q3: Why Does AxisKV Improve Certain Quality Metrics?

We thank the reviewer for highlighting this observation.

One possible explanation is that spatial compression of distant KV states reduces redundant historical details and changes how the model balances distant and recent context, potentially improving certain quality metrics. Furthermore, automatic metrics such as VBench-Long imaging quality are imperfect proxies for human perceptual quality, and an improvement in a single metric does not necessarily imply better overall visual quality. While the exact mechanism remains to be established, these observations suggest that compression can affect different aspects of generation quality differently.

Q4: Does Compression Sensitivity Depend on Video Content?

We thank the reviewer for this insightful suggestion. To investigate content-dependent compression sensitivity, we conducted a per-prompt analysis on 100 paired 15-second Self-Forcing rollouts. We measure motion magnitude using average RAFT optical flow from uncompressed videos and divide the videos into three approximately equal-sized groups (low, medium, and high motion) based on their motion magnitude. We then evaluate the semantic consistency change caused by collapsing pinned states to a single spatial token.

Motion groupAverage flowSemantic change
Low motion0.32−0.0351 ± 0.0042
Medium motion1.33−0.0270 ± 0.0046
High motion7.46−0.0151 ± 0.0057

We observe a significant correlation between motion magnitude and semantic consistency change (Pearson r = +0.339). Low-motion videos exhibit approximately 2.3× greater semantic loss than high-motion videos. These results provide initial evidence that pinned-state compression sensitivity varies with video motion characteristics, beyond temporal position. While this does not directly identify an optimal content-adaptive allocation, it suggests that motion characteristics may help inform future adaptive compression policies.

While this analysis does not directly examine scene changes or generation-state-dependent allocation, it provides initial evidence that compression sensitivity varies with video characteristics beyond temporal position.

The analysis has been included in the revised Appendix F.

Thanks for the reply. Most of my questions have been resolved, and I will keep my score unchanged.

However, after carefully re-reading the manuscript, I believe there is still room for improvement, and I would encourage the authors to consider the following points.

1. More insightful Figures 1 and 2

As the authors noted, this work characterizes the KV Cache in autoregressive video generation models rather than compressing purely from a position-based perspective. It would therefore be valuable to present more insightful versions of Figures 1 and 2 — for example, from the perspective of KV Cache and video content

2. Clearer diagrams and analysis of compression strategies

The authors could also add clearer diagrams and a dedicated analysis section on how the different strategies compress the KV Cache along the spatial-resolution dimension. In particular, this section should make explicit which KV entries are preserved at each resolution level, and how that preservation relates to the video content being generated.

3. Applicability of AxisKV across models and architectures

It would be helpful to discuss how AxisKV is applied, or needs to be adapted, across different models, architectures, and generators. This would further improve both the readability and the generalizability of the paper.

4. Vector graphics for Figure 3

As also pointed out by Reviewer 58kA, rendering Figure 3 as a vector graphic would make the figure more credible and legible. The other figures located in the appendix also have similar issues.

5. A dedicated limitations section

Finally, adding a limitations section would considerably strengthen the credibility of the paper.

In summary, my recommendation lies between weak reject and weak accept, and the final decision depends on AC

We thank the reviewer for the thoughtful follow-up and are glad that most of the earlier concerns have been resolved. We appreciate the additional suggestions and agree that the presentation and discussion of our findings can be further improved.

We will refine the figures and explanations to better illustrate how different compression strategies affect KV-cache representations and relate to generated video content. We will also improve figure readability by using vector graphics with selectable text. Regarding generalizability and limitations, we will consolidate the existing discussions in Appendix into a dedicated limitations section in the main paper.

We appreciate the reviewer's constructive feedback, which will help improve the clarity and completeness of the manuscript.

Reviewer_i7WsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies what information different temporal positions in an autoregressive video KV cache must preserve. Persistent, position-matched interventions reveal that distant states tolerate substantial spatial compression, while recent states are sensitive to spatial collapse but tolerate reduced numerical precision. The experiments also distinguish retaining compact pinned states from removing them entirely. These findings motivate AxisKV, a training-free policy combining spatial pooling of distant states with quantization of recent states. Experiments demonstrate favorable quality–memory trade-offs on Self-Forcing and evaluate transfer to three related generators.

Critical strengths
  • Well-controlled interventions and empirical findings. Persistent perturbations capture autoregressive compounding, while position-matched controls help distinguish content replacement from freezing effects. The distinction between spatial compressibility and dispensability provides a concrete rationale for retaining distant states as compact summaries.

  • Useful compression principle. Choosing the compression axis according to temporal position follows directly from the findings. Matched-memory comparisons within the same implementation provide convincing evidence against uniform pooling and two-bit quantization on Self-Forcing.

  • Convincing matched-memory results. Comparisons within the same implementation isolate the allocation policy. On Self-Forcing, AxisKV substantially outperforms uniform pooling and improves several quality measures over uniform two-bit quantization at 8× compression.

Critical weaknesses
  • Model-dependent allocation. The fixed allocation does not transfer reliably to every evaluated generator. On CausVid, AxisKV loses 5.38 aesthetic-quality points and 2.86 imaging-quality points relative to the uncompressed cache, exceeding the losses from uniform two-bit quantization. The evidence therefore supports the recent-state spatial requirement more consistently than aggressive spatial collapse of all distant states.

  • Incomplete comparison with related policies. PackForcing, Relax Forcing, and Pyramid Forcing already allocate cache resources differently across temporal regions, but are absent from the main comparisons. The uniform baselines establish the value of heterogeneous allocation, while leaving AxisKV’s practical advantage over these existing position-dependent approaches unclear.

Questions for authors
  • On CausVid, would preserving a small spatial grid for pinned states recover the quality losses at a comparable memory budget? A focused control would help identify whether the failure comes from collapsing these states too aggressively.

  • Are matched-budget results available against the position-dependent policies discussed in related work? Such evidence would help assess the practical contribution separately from the intervention-based findings.

3 comments

We thank the reviewer for recognizing the strengths of our causal intervention design, position-dependent compression principle, and matched-budget evaluations. We address the concerns through additional experiments and analysis. All changes in the revised manuscript are highlighted in blue for ease of reference.

W1/Q1: Model-Dependent Allocation and Compression Strategy on CausVid

We agree that aggressive distant-state compression is not equally effective across generators. As discussed in Appendix F, we measured each generator's sensitivity to pinned-state spatial collapse through an independent causal probe (probe tax). CausVid exhibits the largest semantic probe tax (−0.111), compared with Self-Forcing (−0.059), Rolling Forcing (−0.020), and LongLive (−0.018), consistent with its larger quality degradation under AxisKV.

Following the reviewer's suggestion of preserving a small spatial grid, we conducted an additional pinned-state resolution ablation on CausVid, retaining 1, 28, or 104 spatial tokens per pinned state while keeping mid-lag states at 1 token and recent states at 6-bit precision.

Pinned tokensAesthetic ΔImaging ΔSemantic ΔKV memory
1−5.38−2.86−0.0768310 MiB
28−4.72−2.62−0.0699324 MiB
104−4.44−2.96−0.0689364 MiB
Uniform 2-bit−0.90−1.47+0.0005308 MiB

Increasing pinned-state resolution from 1 to 104 tokens raises KV memory by 17.4%, yet recovers only 17% of the aesthetic-quality loss and 10% of the semantic-consistency loss, with no consistent improvement in imaging quality. All three settings remain substantially behind uniform 2-bit quantization. These results indicate that preserving a small spatial grid is insufficient to resolve CausVid's quality degradation at the tested compression rates. Together with the probe-tax analysis in Appendix F, they reinforce that tolerance to aggressive distant-state spatial compression is generator-dependent.

In the revised version, We have included the new pinned-state resolution ablation and analysis in Appendix F.

W2/Q2: Comparison with Existing Position-Dependent Cache Policies

We thank the reviewer for highlighting these relevant methods. Deep Forcing, which also employs position-dependent cache management, was already included in our original Table 2. To provide a more comprehensive comparison, we additionally evaluated Relax Forcing and Pyramid Forcing using their released implementations under the same checkpoint, prompts, and evaluation protocol, and included the results in the revised Table 2. PackForcing was not included because its implementation is not publicly available.

Relax Forcing primarily reduces attention computation by selecting a limited number of historical frames for each attention operation, rather than compressing the stored KV representations. Its released configuration maintains a candidate KV cache from which historical frames are selected. Consequently, its physical KV storage grows with generation length until the configured cache limit is reached. At 30 seconds, Relax Forcing retains approximately 31.33 GiB of historical KV, compared with 308 MiB for AxisKV at 8× compression. Thus, Relax Forcing and AxisKV optimize different aspects of autoregressive video generation: attention computation and historical KV storage, respectively.

Pyramid Forcing adopts a head-wise temporal allocation strategy, retaining attention sinks, recent frames, and selected intermediate historical frames. Under its released configuration, it retains approximately 1,889 MiB of historical KV, compared with 308 MiB for AxisKV at 8× compression. Pyramid Forcing achieves slightly higher scores on several generation-quality metrics, but operates at a larger historical KV budget. Unlike AxisKV, which compresses spatial resolution and numerical precision according to temporal position, Pyramid Forcing primarily optimizes which historical information is retained.

Regarding matched-budget comparisons, AxisKV's 308 MiB historical KV budget is equivalent to only 1.12 full-resolution KV frames. Matching this budget would substantially constrain the temporal allocation mechanisms of other position-dependent methods. Relax Forcing maintains a candidate history for selecting relevant frames; matching 308 MiB would allow storage equivalent to fewer than two full-resolution frames, severely restricting its history-selection mechanism. Pyramid Forcing distributes historical KV across attention heads and temporal regions (sink, intermediate, and recent), and matching this budget would require substantially reducing these allocations from its released configuration of 1,889 MiB. We therefore evaluate these methods under their released configurations in Table 2. For controlled matched-budget comparisons, we focus on compression strategies whose storage budgets can be directly matched without redesigning their underlying cache-management mechanisms, as reported in Table 3. These comparisons isolate the contribution of AxisKV's position-dependent compression-axis allocation.

We have updated the results in Table 2 and additional analysis.

Thank you for the reply. My concerns have been addressed and I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.