- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PaFu-KV, a learned KV cache policy for autoregressive video diffusion. It derives token salience targets from a bidirectional teacher by aggregating past, present, and future attention, and distills them into a lightweight prediction head. During inference, predicted salience guides cache retention and eviction. Experiments on VBench and VBench-Long report improved throughput and temporal consistency under a limited cache budget.
Critical strengths
- The bidirectional teacher provides past- and future-informed salience supervision, offering a well-motivated way to learn token importance for cache retention under causal generation constraints.
- The salience estimator is lightweight and runs only at the final Transformer layer and final denoising stage. Sharing the selected token indices across layers further limits inference overhead.
Critical weaknesses
- The main comparisons involve different complete generation systems, making it difficult to isolate the contribution of PaFu-KV from training and configuration differences. Controlled comparisons against FIFO and other training-free KV cache eviction methods are rrecommended.
- The claim of negligible quality degradation is not fully supported. In Table 1, the semantic score drops to 71.36, compared with 81.28 for Self Forcing and 76.47 for LongLive, despite the higher total score. Detailed scores across all VBench evaluation dimensions should be provided to clarify which aspects deteriorate and better characterize the quality–efficiency trade-off.
Questions for authors
Please refer to the weaknesses listed above.
5 comments
W2: Full VBench breakdown and semantic degradation
W2.1: Full VBench breakdown and Clarification of Semantic Drop
To explain the semantic-score drop, we first clarify that VBench Semantic Score comprises nine dimensions: Object Class, Multiple Objects and so on. These cover diverse aspects of text-video alignment, including content, attributes, spatial relationships, scene understanding, and stylistic consistency. These dimensions also use different evaluators and criteria. For example, Object Class and Color rely on GRiT for object detection and color-caption comparison, respectively. Appearance Style and Temporal Style use ViCLIP-based video-text similarity. Therefore, a video may correctly depict the requested objects but still receive lower scores if its appearance or temporal style differs from the textual description. Consequently, accurate depiction of the main content can coexist with a lower aggregate Semantic Score. We follow the official VBench aggregation protocol, which assigns a 4:1 weighting to Quality and Semantic, while reporting both scores separately for transparency. We further provide a dimension-wise breakdown of the Semantic Score to better understand the source of the performance difference.
| Baselines | Object Class ↑ | Multiple Objects ↑ | Human Action ↑ | Color ↑ | Spatial Relationship ↑ | Scene ↑ | Appearance Style ↑ | Temporal Style ↑ | Overall Consistency ↑ | Semantic Score ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| Wan2.1-1.3B | 90.8 | 73.1 | 94.6 | 84.8 | 77.6 | 56.1 | 24.2 | 26.9 | 26.6 | 80.1 |
| Self-Forcing | 94.9 | 88.1 | 95.6 | 85.7 | 84.2 | 53.6 | 21.2 | 24.6 | 27.7 | 81.3 |
| LongLive | 91.5 | 84.2 | 93.4 | 82.2 | 78.6 | 50.2 | 18.8 | 23.0 | 24.9 | 76.5 |
| PaFu-KV | 92.3 | 81.6 | 91.6 | 80.1 | 80.2 | 50.1 | 15.2 | 17.2 | 20.1 | 71.4 |
The dimension-wise results suggest that PaFu-KV maintains relatively strong performance in object categories and spatial relationships. More pronounced degradation is observed in Appearance and Temporal Style. This pattern suggests that the semantic-score drop mainly reflects weaker stylistic, instead of failures in generating the objects or actions. This is consistent with PaFu-KV's design. Our salience estimation measures token contributions through past, current, and future attention interactions. Under a limited KV Cache budget, salience-based selection may preserve essential visual structures and temporal continuity while discarding less salient tokens encoding fine-grained attributes or stylistic information. Consequently, visual consistency can be maintained even when some aspects of prompt adherence are weakened.
We also provide a breakdown of the Quality Score across its seven constituent dimensions. As shown below, PaFu-KV improves several aspect, while exhibiting a slight decrease in Motion Smoothness.
| Method | Subject Consistency ↑ | Background Consistency ↑ | Temporal Flickering ↑ | Motion Smoothness ↑ | Dynamic Degree ↑ | Aesthetic Quality ↑ | Imaging Quality ↑ | Quality |
|---|---|---|---|---|---|---|---|---|
| Wan2.1-1.3B | 96.7 | 95.5 | 99.3 | 98.3 | 69.7 | 68.5 | 68.1 | 85.3 |
| Self-Forcing | 95.5 | 96.6 | 98.8 | 98.4 | 65.6 | 67.7 | 70.2 | 85.1 |
| LongLive | 98.6 | 97.9 | 99.4 | 98.8 | 69.3 | 68.9 | 71.1 | 87.0 |
| PaFu-KV | 98.9 | 98.1 | 99.5 | 98.1 | 73.6 | 74.7 | 75.0 | 88.6 |
The Quality improvement alongside the Semantic decrease reflects a trade-off between visual generation quality and fine-grained prompt adherence. PaFu-KV uses salience-guided historical KV selection to preserve informative contextual tokens and reduce interference from drifting representations, improving long-horizon visual consistency and generation quality. However, attention-based salience does not necessarily capture every prompt-specific semantic constraint. Under a fixed cache budget, tokens encoding fine-grained attributes or stylistic details may be discarded.
W2.2: Extenting Stage 1 training for better Semantic Score.
We thank the reviewer for highlighting the semantic-score drop and suggesting an alternative operating point. We attribute the degradation partly to insufficient Stage-1 training. Our training consists of two stages: Stage 1 jointly trains the AR generator and SEH without KV cache eviction, allowing SEH to learn reliable token salience before cache eviction. Stage 2 enables salience-based eviction. We hypothesize that the original 500 Stage-1 iterations were insufficient for SEH convergence, potentially underestimating the salience of semantically important tokens. Consequently, prematurely evicting these tokens during Stage 2 may increase training difficulty and weaken semantic adherence. To investigate this hypothesis, we increase Stage-1 training from 500 to 1,500 iterations while keeping Stage 2 at 2,500 iterations. Evaluations on VBench are shown below:
| Method | Total ↑ | Quality ↑ | Semantic ↑ |
|---|---|---|---|
| LongLive | 84.87 | 86.97 | 76.47 |
| PaFu-KV (Original) | 85.12 | 88.56 | 71.36 |
| PaFu-KV (Extended Stage 1) | 86.55 | 88.74 | 77.81 |
Evaluations on VBench-Long are shown below:
| Method | Subj. ↑ | Back. ↑ | Mot. ↑ | Aes. ↑ | Imag. ↑ | Drift ↓ |
|---|---|---|---|---|---|---|
| LongLive | 90.12 | 91.42 | 99.07 | 62.94 | 69.46 | 0.59 |
| PaFu-KV (Original) | 93.91 | 94.14 | 98.14 | 62.48 | 71.23 | 0.29 |
| PaFu-KV (Extended Stage 1) | 93.87 | 94.28 | 98.79 | 62.76 | 71.15 | 0.21 |
The extended Stage-1 training improves semantic adherence, supporting our hypothesis. We will incorporate the updated results and clarify the two-stage training rationale in the revised manuscript.
Response To Reviewer Thzy:
Thanks for your valuable suggestions. We will incorporate the relevant explanations into the final version for better understanding.
W1: Comparisons with FIFO and Training-Free KV Cache Eviction Methods
W1.1: Comparisons with FIFO.
To address this concern, we conduct three ablation experiments: Past-Current-only (PCo), Current-Future-only (CFo) and Current-only (Co). All variants are implemented by modifying the salience scoring criterion in the Real Score Network, while keeping the training implementaion unchanged. Specifically, we modify Eq. (5) to construct different salience supervision targets, allowing us to isolate the contributions of past, current, and future interactions. For PCo, we remove the upper-triangular component representing future interactions. Eq. (5) is modified as:
\begin{cases} \frac{\mathrm{diag}_{b,j}+\mathrm{low}_{b,j}}{2}, 0 \leq j < L-L_B,\\\\ \mathrm{diag}_{b,j}, L-L_B \leq j < L. \end{cases}$$ For CFo, we remove the lower-triangular component representing past interactions. Eq. (5) is modified as: $$s_{b,j}^{\text{w/o Lower}}= \begin{cases} \mathrm{diag}_{b,j}, 0\leq j<L_B, \\\\ \frac{\mathrm{up}_{b,j}+\mathrm{diag}_{b,j}}{2}, L_B\leq j<L. \end{cases}$$ For the setting without both PCo and CFo, we remove both the upper- and lower-triangular components. Eq. (5) is modified as: $$s_{b,j}^{\text{w/o Upper,Lower}}=\mathrm{diag}_{b,j}, 0\leq j<L.$$ Furthermore, we include a FIFO control, where cache retention and eviction depend solely on token arrival order. Thus, SEH predictions are never used for cache selection. Retaining SEH and its salience loss would introduce an auxiliary objective with no role in inference-time cache management. We therefore disable both. Without SEH-based salience prediction and top-k selection, the method reduces to a LongLive-style autoregressive framework with FIFO KV-cache management. Hence, we directly use LongLive as the FIFO baseline. All ablation use same training configurations, with only the salience scoring targets or cache eviction policy changed. The results are reported below: | Variant | Subj. ↑ | Back. ↑ | Mot. ↑ | Aes. ↑ | Imag. ↑ | Drift ↓ | |-|-:|-:|-:|-:|-:|-:| | FIFO (LongLive) | 90.12 | 91.42 | **99.07** | **62.94** | 69.46 | 0.59 | | PCo | 91.40 | 92.16 | 97.62 | 62.12 | 68.72 | 0.65 | | CFo | 89.84 | 90.68 | 96.75 | 61.61 | 69.76 | 0.83 | | Co | 85.96 | 86.84 | 90.80 | 57.26 | 65.20 | 2.37 | | **Full PaFu-KV** | **93.91** | **94.14** | 98.14 | 62.48 | **71.23** | **0.29** | These results show that past, current, and future interactions provide complementary information for salience estimation. Removing either component degrades performance, with current-only interactions causing the largest drop. Compared with the FIFO (LongLive) baseline, PaFu-KV better preserves long-term visual consistency and mitigates quality degradation, demonstrating its advantage over heuristic FIFO eviction. ## W1.2: Comparisons with Training-Free KV Cache Eviction Methods Additionally, we include three representative methods specifically designed for autoregressive video diffusion: (1) Deep Forcing [1], which combines Deep Sink and Participative Compression to preserve important historical tokens based on recent attention; (2) Dummy Forcing [2], which identifies redundant attention heads and assigns heterogeneous context budgets; and (3) FIFO (LongLive), which serves as a heuristic cache eviction baseline. All methods are evaluated using the same VBench-Long protocol under a matched KV Cache budget. [1] Yi J, Jang W, Cho P H, et al. Deep forcing: Training-free long video generation with deep sink and participative compression[J]. arXiv preprint arXiv:2512.05081, 2025. [2] Guo H, Jia Z, Li J, et al. Efficient autoregressive video diffusion with dummy head[J]. arXiv preprint arXiv:2601.20499, 2026. | Method | Subj. ↑ | Back. ↑ | Mot. ↑ | Aes. ↑ | Imag. ↑ | Drift ↓ | |-|-:|-:|-:|-:|-:|-:| | FIFO (LongLive) | 90.12 | 91.42 | **99.07** | **62.94** | 69.46 | 0.59 | | Deep Forcing | 91.28 | 92.17 | 98.47 | 62.12 | 70.46 | 0.57 | | Dummy Forcing | 93.42 | 93.58 | 98.26 | 61.82 | **71.98** | 0.35 | | **PaFu-KV (Ours)** | **93.91** | **94.14** | 98.14 | 62.48 | 71.23 | **0.29** | Under the matched KV Cache budget, PaFu-KV achieves the best subject consistency background consistency and quality drift among all compared methods. These results demonstrate the effectiveness of salience-aware cache management in preserving long-term visual consistency and mitigating error accumulation.