acceptodds
Under review as a conference paper at ICLR 2027

Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusion

Abstract

Video generation is pivotal to digital media creation, and recent advances in autoregressive video generation have markedly enhanced the efficiency of real-time video synthesis. However, existing approaches generally rely on heuristic KV Cache policies, which ignore differences in token importance in long-term video generation. This leads to the loss of critical spatiotemporal information and the accumulation of redundant, invalid cache, thereby degrading video generation quality and efficiency. To address this limitation, we first observe that token contributions to video generation are highly time-heterogeneous and accordingly propose a novel Past- and Future-Informed KV Cache Policy (PaFu-KV). Specifically, PaFu-KV introduces a lightweight Salience Estimation Head distilled from a bidirectional teacher to estimate salience scores, allowing the KV cache to retain informative tokens while discarding less relevant ones. This policy yields a better quality-efficiency trade-off by shrinking KV cache capacity and reducing memory footprint at inference time. Extensive experiments on benchmarks demonstrate that our method preserves high-fidelity video generation quality while enables accelerated inference, thereby enabling more efficient long-horizon video generation. Our code will be released upon paper acceptance.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Video Diffusion Models, KV Cache

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ThzyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PaFu-KV, a learned KV cache policy for autoregressive video diffusion. It derives token salience targets from a bidirectional teacher by aggregating past, present, and future attention, and distills them into a lightweight prediction head. During inference, predicted salience guides cache retention and eviction. Experiments on VBench and VBench-Long report improved throughput and temporal consistency under a limited cache budget.

Critical strengths
  1. The bidirectional teacher provides past- and future-informed salience supervision, offering a well-motivated way to learn token importance for cache retention under causal generation constraints.
  2. The salience estimator is lightweight and runs only at the final Transformer layer and final denoising stage. Sharing the selected token indices across layers further limits inference overhead.
Critical weaknesses
  1. The main comparisons involve different complete generation systems, making it difficult to isolate the contribution of PaFu-KV from training and configuration differences. Controlled comparisons against FIFO and other training-free KV cache eviction methods are rrecommended.
  2. The claim of negligible quality degradation is not fully supported. In Table 1, the semantic score drops to 71.36, compared with 81.28 for Self Forcing and 76.47 for LongLive, despite the higher total score. Detailed scores across all VBench evaluation dimensions should be provided to clarify which aspects deteriorate and better characterize the quality–efficiency trade-off.
Questions for authors

Please refer to the weaknesses listed above.

5 comments

Response To Reviewer Thzy:

Thanks for your valuable suggestions. We will incorporate the relevant explanations into the final version for better understanding.

W1: Comparisons with FIFO and Training-Free KV Cache Eviction Methods

W1.1: Comparisons with FIFO.

To address this concern, we conduct three ablation experiments: Past-Current-only (PCo), Current-Future-only (CFo) and Current-only (Co). All variants are implemented by modifying the salience scoring criterion in the Real Score Network, while keeping the training implementaion unchanged. Specifically, we modify Eq. (5) to construct different salience supervision targets, allowing us to isolate the contributions of past, current, and future interactions. For PCo, we remove the upper-triangular component representing future interactions. Eq. (5) is modified as:

\begin{cases} \frac{\mathrm{diag}_{b,j}+\mathrm{low}_{b,j}}{2}, 0 \leq j < L-L_B,\\\\ \mathrm{diag}_{b,j}, L-L_B \leq j < L. \end{cases}$$ For CFo, we remove the lower-triangular component representing past interactions. Eq. (5) is modified as: $$s_{b,j}^{\text{w/o Lower}}= \begin{cases} \mathrm{diag}_{b,j}, 0\leq j<L_B, \\\\ \frac{\mathrm{up}_{b,j}+\mathrm{diag}_{b,j}}{2}, L_B\leq j<L. \end{cases}$$ For the setting without both PCo and CFo, we remove both the upper- and lower-triangular components. Eq. (5) is modified as: $$s_{b,j}^{\text{w/o Upper,Lower}}=\mathrm{diag}_{b,j}, 0\leq j<L.$$ Furthermore, we include a FIFO control, where cache retention and eviction depend solely on token arrival order. Thus, SEH predictions are never used for cache selection. Retaining SEH and its salience loss would introduce an auxiliary objective with no role in inference-time cache management. We therefore disable both. Without SEH-based salience prediction and top-k selection, the method reduces to a LongLive-style autoregressive framework with FIFO KV-cache management. Hence, we directly use LongLive as the FIFO baseline. All ablation use same training configurations, with only the salience scoring targets or cache eviction policy changed. The results are reported below: | Variant | Subj. ↑ | Back. ↑ | Mot. ↑ | Aes. ↑ | Imag. ↑ | Drift ↓ | |-|-:|-:|-:|-:|-:|-:| | FIFO (LongLive) | 90.12 | 91.42 | **99.07** | **62.94** | 69.46 | 0.59 | | PCo | 91.40 | 92.16 | 97.62 | 62.12 | 68.72 | 0.65 | | CFo | 89.84 | 90.68 | 96.75 | 61.61 | 69.76 | 0.83 | | Co | 85.96 | 86.84 | 90.80 | 57.26 | 65.20 | 2.37 | | **Full PaFu-KV** | **93.91** | **94.14** | 98.14 | 62.48 | **71.23** | **0.29** | These results show that past, current, and future interactions provide complementary information for salience estimation. Removing either component degrades performance, with current-only interactions causing the largest drop. Compared with the FIFO (LongLive) baseline, PaFu-KV better preserves long-term visual consistency and mitigates quality degradation, demonstrating its advantage over heuristic FIFO eviction. ## W1.2: Comparisons with Training-Free KV Cache Eviction Methods Additionally, we include three representative methods specifically designed for autoregressive video diffusion: (1) Deep Forcing [1], which combines Deep Sink and Participative Compression to preserve important historical tokens based on recent attention; (2) Dummy Forcing [2], which identifies redundant attention heads and assigns heterogeneous context budgets; and (3) FIFO (LongLive), which serves as a heuristic cache eviction baseline. All methods are evaluated using the same VBench-Long protocol under a matched KV Cache budget. [1] Yi J, Jang W, Cho P H, et al. Deep forcing: Training-free long video generation with deep sink and participative compression[J]. arXiv preprint arXiv:2512.05081, 2025. [2] Guo H, Jia Z, Li J, et al. Efficient autoregressive video diffusion with dummy head[J]. arXiv preprint arXiv:2601.20499, 2026. | Method | Subj. ↑ | Back. ↑ | Mot. ↑ | Aes. ↑ | Imag. ↑ | Drift ↓ | |-|-:|-:|-:|-:|-:|-:| | FIFO (LongLive) | 90.12 | 91.42 | **99.07** | **62.94** | 69.46 | 0.59 | | Deep Forcing | 91.28 | 92.17 | 98.47 | 62.12 | 70.46 | 0.57 | | Dummy Forcing | 93.42 | 93.58 | 98.26 | 61.82 | **71.98** | 0.35 | | **PaFu-KV (Ours)** | **93.91** | **94.14** | 98.14 | 62.48 | 71.23 | **0.29** | Under the matched KV Cache budget, PaFu-KV achieves the best subject consistency background consistency and quality drift among all compared methods. These results demonstrate the effectiveness of salience-aware cache management in preserving long-term visual consistency and mitigating error accumulation.

W2: Full VBench breakdown and semantic degradation

W2.1: Full VBench breakdown and Clarification of Semantic Drop

To explain the semantic-score drop, we first clarify that VBench Semantic Score comprises nine dimensions: Object Class, Multiple Objects and so on. These cover diverse aspects of text-video alignment, including content, attributes, spatial relationships, scene understanding, and stylistic consistency. These dimensions also use different evaluators and criteria. For example, Object Class and Color rely on GRiT for object detection and color-caption comparison, respectively. Appearance Style and Temporal Style use ViCLIP-based video-text similarity. Therefore, a video may correctly depict the requested objects but still receive lower scores if its appearance or temporal style differs from the textual description. Consequently, accurate depiction of the main content can coexist with a lower aggregate Semantic Score. We follow the official VBench aggregation protocol, which assigns a 4:1 weighting to Quality and Semantic, while reporting both scores separately for transparency. We further provide a dimension-wise breakdown of the Semantic Score to better understand the source of the performance difference.

BaselinesObject Class ↑Multiple Objects ↑Human Action ↑Color ↑Spatial Relationship ↑Scene ↑Appearance Style ↑Temporal Style ↑Overall Consistency ↑Semantic Score ↑
Wan2.1-1.3B90.873.194.684.877.656.124.226.926.680.1
Self-Forcing94.988.195.685.784.253.621.224.627.781.3
LongLive91.584.293.482.278.650.218.823.024.976.5
PaFu-KV92.381.691.680.180.250.115.217.220.171.4

The dimension-wise results suggest that PaFu-KV maintains relatively strong performance in object categories and spatial relationships. More pronounced degradation is observed in Appearance and Temporal Style. This pattern suggests that the semantic-score drop mainly reflects weaker stylistic, instead of failures in generating the objects or actions. This is consistent with PaFu-KV's design. Our salience estimation measures token contributions through past, current, and future attention interactions. Under a limited KV Cache budget, salience-based selection may preserve essential visual structures and temporal continuity while discarding less salient tokens encoding fine-grained attributes or stylistic information. Consequently, visual consistency can be maintained even when some aspects of prompt adherence are weakened.

We also provide a breakdown of the Quality Score across its seven constituent dimensions. As shown below, PaFu-KV improves several aspect, while exhibiting a slight decrease in Motion Smoothness.

MethodSubject Consistency ↑Background Consistency ↑Temporal Flickering ↑Motion Smoothness ↑Dynamic Degree ↑Aesthetic Quality ↑Imaging Quality ↑Quality
Wan2.1-1.3B96.795.599.398.369.768.568.185.3
Self-Forcing95.596.698.898.465.667.770.285.1
LongLive98.697.999.498.869.368.971.187.0
PaFu-KV98.998.199.598.173.674.775.088.6

The Quality improvement alongside the Semantic decrease reflects a trade-off between visual generation quality and fine-grained prompt adherence. PaFu-KV uses salience-guided historical KV selection to preserve informative contextual tokens and reduce interference from drifting representations, improving long-horizon visual consistency and generation quality. However, attention-based salience does not necessarily capture every prompt-specific semantic constraint. Under a fixed cache budget, tokens encoding fine-grained attributes or stylistic details may be discarded.

W2.2: Extenting Stage 1 training for better Semantic Score.

We thank the reviewer for highlighting the semantic-score drop and suggesting an alternative operating point. We attribute the degradation partly to insufficient Stage-1 training. Our training consists of two stages: Stage 1 jointly trains the AR generator and SEH without KV cache eviction, allowing SEH to learn reliable token salience before cache eviction. Stage 2 enables salience-based eviction. We hypothesize that the original 500 Stage-1 iterations were insufficient for SEH convergence, potentially underestimating the salience of semantically important tokens. Consequently, prematurely evicting these tokens during Stage 2 may increase training difficulty and weaken semantic adherence. To investigate this hypothesis, we increase Stage-1 training from 500 to 1,500 iterations while keeping Stage 2 at 2,500 iterations. Evaluations on VBench are shown below:

MethodTotal ↑Quality ↑Semantic ↑
LongLive84.8786.9776.47
PaFu-KV (Original)85.1288.5671.36
PaFu-KV (Extended Stage 1)86.5588.7477.81

Evaluations on VBench-Long are shown below:

MethodSubj. ↑Back. ↑Mot. ↑Aes. ↑Imag. ↑Drift ↓
LongLive90.1291.4299.0762.9469.460.59
PaFu-KV (Original)93.9194.1498.1462.4871.230.29
PaFu-KV (Extended Stage 1)93.8794.2898.7962.7671.150.21

The extended Stage-1 training improves semantic adherence, supporting our hypothesis. We will incorporate the updated results and clarify the two-stage training rationale in the revised manuscript.

Reviewer_jJcqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PaFu-KV proposes token retaining method for KV cache of autoregressive video diffusion models. They use Wan2.1-14B as the teacher to verify and evaluate the importance of each token depending on the attention scores of that with Past/Current/Future token queries. They propose to separate the attention scores within each groups’ head, rather than letting only one group who dominate a global maximum. The generator and prediction head are trained together using DMD and a salience-regression loss, with LongLive-style streaming long tuning. During inference, retained history and new tokens are ranked by predicted importance scores to maintain a fixed-size compact cache. The reported results improve long-video subject/background consistency and throughput, but trade off some short-video semantic alignment.

Critical strengths
  1. The teacher–student design addresses a real limitation of causal cache management. Future interactions can be observed during training and used to supervise a small prediction head, while inference remains causal.
  2. The long-video consistency results are impressive. Table 2 reports subject/background consistency of 93.91/94.14, improvements of 3.79/2.72 points over LongLive. Removing the frame sink retains most of this result (93.12/93.09, Table 3), suggesting that the gains are not mainly explained by the sink mechanism.
Critical weaknesses
  1. The central method is understandable. Reconstructing the implementation is harder: Algorithm 3 omits the softmax used in Eq. (4) and reverses the order of query maximization and head averaging. Algorithm 2 also differs from the text about when new tokens enter the selection.
  2. Table 3's max and average ablations test how bidirectional teacher attention is aggregated, but they do not remove future information. A matched-capacity FIFO baseline and a past/current-only teacher target, trained with the same recipe, are needed to determine whether learned retention and specifically future-informed supervision account for the gains.
  3. The reported gains come with quality and cache-capacity trade-offs that the negligible-degradation claim understates. Relative to LongLive, Table 1 improves throughput by about 14% and total VBench by only 0.25 points, while the semantic score falls by 5.11 points (76.47 to 71.36).
Questions for authors
  1. With the same backbone, training recipe and budget, sink setting, and retained-token capacity, how does the full method compare with (a) FIFO retention and (b) a salience target using only past/current queries? The latter should retain the same teacher and other settings.
  2. Please reconcile Eq. (4) with Algorithm 3's normalization and reduction order, and Algorithm 2 with the joint current/history selection in Section 3.4. Specify how current tokens and sink tokens count toward the 4,680-token budget.
8 comments

Response To Reviewer jJcq:

Thanks for your valuable suggestions. We will incorporate the relevant explanations into the final version for better understanding.

W1: Consistency between the equations, algorithms, and implementation

Thank you for pointing out these inconsistencies. Algorithm 3 should follow the salience definition in the main text. Specifically, the scaled query–key products should first be normalized using softmax over the key dimension. For each temporal region, query maximization should then be performed independently within each attention head, followed by averaging across heads, as specified in Eq. (4). In the block-wise computation, this requires maintaining per-head running maxima across query chunks before head averaging. The resulting regional scores are then combined according to Eq. (5). We will correct Algorithm 3 accordingly and revise the relevant descriptions in the main text to make the normalization, aggregation order, and correspondence between the equations and pseudocode explicit and consistent.

We have highlighted the revisions to the algorithm on Pages 18–19 of the Appendix in red.

W2: Isolating the Contributions of Learned Retention and Future-Informed Supervision

To address this concern, we conduct three ablation experiments: Past-Current-only (PCo), Current-Future-only (CFo) and Current-only (Co). All variants are implemented by modifying the salience scoring criterion in the Real Score Network, while keeping the training implementaion unchanged. Specifically, we modify Eq. (5) to construct different salience supervision targets, allowing us to isolate the contributions of past, current, and future interactions. For PCo, we remove the upper-triangular component representing future interactions. Eq. (5) is modified as:

\begin{cases} \frac{\mathrm{diag}_{b,j}+\mathrm{low}_{b,j}}{2}, 0 \leq j < L-L_B,\\\\ \mathrm{diag}_{b,j}, L-L_B \leq j < L. \end{cases}$$ For CFo, we remove the lower-triangular component representing past interactions. Eq. (5) is modified as: $$s_{b,j}^{\text{w/o Lower}}= \begin{cases} \mathrm{diag}_{b,j}, 0\leq j<L_B, \\\\ \frac{\mathrm{up}_{b,j}+\mathrm{diag}_{b,j}}{2}, L_B\leq j<L. \end{cases}$$ For the setting without both PCo and CFo, we remove both the upper- and lower-triangular components. Eq. (5) is modified as: $$s_{b,j}^{\text{w/o Upper,Lower}}=\mathrm{diag}_{b,j}, 0\leq j<L.$$ Furthermore, we include a FIFO control, where cache retention and eviction depend solely on token arrival order. Thus, SEH predictions are never used for cache selection. Retaining SEH and its salience loss would introduce an auxiliary objective with no role in inference-time cache management. We therefore disable both. Without SEH-based salience prediction and top-k selection, the method reduces to a LongLive-style autoregressive framework with FIFO KV-cache management. Hence, we directly use LongLive as the FIFO baseline. All ablation use same training configurations, with only the salience scoring targets or cache eviction policy changed. The results are reported below: | Variant | Subj. ↑ | Back. ↑ | Mot. ↑ | Aes. ↑ | Imag. ↑ | Drift ↓ | |-|-:|-:|-:|-:|-:|-:| | FIFO (LongLive) | 90.12 | 91.42 | **99.07** | **62.94** | 69.46 | 0.59 | | PCo | 91.40 | 92.16 | 97.62 | 62.12 | 68.72 | 0.65 | | CFo | 89.84 | 90.68 | 96.75 | 61.61 | 69.76 | 0.83 | | Co | 85.96 | 86.84 | 90.80 | 57.26 | 65.20 | 2.37 | | **Full PaFu-KV** | **93.91** | **94.14** | 98.14 | 62.48 | **71.23** | **0.29** | These results show that past, current, and future interactions provide complementary information for salience estimation. Removing either component degrades performance, with current-only interactions causing the largest drop. Compared with the FIFO (LongLive) baseline, PaFu-KV better preserves long-term visual consistency and mitigates quality degradation, demonstrating its advantage over heuristic FIFO eviction.

W3: The magnitude of the quality–efficiency trade-off

W3.1: Clarification of Semantic Drop

To explain the semantic-score drop, we first clarify that VBench Semantic Score comprises nine dimensions: Object Class, Multiple Objects, Human Action, Color, Spatial Relationship, Scene, Appearance Style, Temporal Style, and Overall Consistency. These cover diverse aspects of text-video alignment, including content, attributes, spatial relationships, scene understanding, and stylistic consistency. These dimensions also use different evaluators and criteria. For example, Object Class and Color rely on GRiT for object detection and color-caption comparison, respectively. Appearance Style and Temporal Style use ViCLIP-based video-text similarity. Therefore, a video may correctly depict the requested objects but still receive lower scores if its appearance or temporal style differs from the textual description. Consequently, accurate depiction of the main content can coexist with a lower aggregate Semantic Score. We follow the official VBench aggregation protocol, which assigns a 4:1 weighting to Quality and Semantic, while reporting both scores separately for transparency. We further provide a dimension-wise breakdown of the Semantic Score to better understand the source of the performance difference.

BaselinesObject Class ↑Multiple Objects ↑Human Action ↑Color ↑Spatial Relationship ↑Scene ↑Appearance Style ↑Temporal Style ↑Overall Consistency ↑Semantic Score ↑
Wan2.1-1.3B90.873.194.684.877.656.124.226.926.680.1
Self-Forcing94.988.195.685.784.253.621.224.627.781.3
LongLive91.584.293.482.278.650.218.823.024.976.5
PaFu-KV92.381.691.680.180.250.115.217.220.171.4

The dimension-wise results suggest that PaFu-KV maintains relatively strong performance in object categories and spatial relationships. More pronounced degradation is observed in Appearance and Temporal Style. This pattern suggests that the semantic-score drop mainly reflects weaker stylistic, instead of failures in generating the objects or actions. This is consistent with PaFu-KV's design. Our salience estimation measures token contributions through past, current, and future attention interactions. Under a limited KV Cache budget, salience-based selection may preserve essential visual structures and temporal continuity while discarding less salient tokens encoding fine-grained attributes or stylistic information. Consequently, visual consistency can be maintained even when some aspects of prompt adherence are weakened.

To further contextualize this observation, we also provide a breakdown of the Quality Score across its seven constituent dimensions. As shown below, PaFu-KV improves several aspect, while exhibiting a slight decrease in Motion Smoothness.

MethodSubject Consistency ↑Background Consistency ↑Temporal Flickering ↑Motion Smoothness ↑Dynamic Degree ↑Aesthetic Quality ↑Imaging Quality ↑Quality
Wan2.1-1.3B96.795.599.398.369.768.568.185.3
Self-Forcing95.596.698.898.465.667.770.285.1
LongLive98.697.999.498.869.368.971.187.0
PaFu-KV98.998.199.598.173.674.775.088.6

The Quality improvement alongside the Semantic decrease reflects a trade-off between visual generation quality and fine-grained prompt adherence. PaFu-KV uses salience-guided historical KV selection to preserve informative contextual tokens and reduce interference from drifting representations, improving long-horizon visual consistency and generation quality. However, attention-based salience does not necessarily capture every prompt-specific semantic constraint. Under a fixed cache budget, tokens encoding fine-grained attributes or stylistic details may be discarded.

W3.2: Extenting Stage 1 training for better Semantic Score.

We thank the reviewer for highlighting the semantic-score drop. Moreover, we attribute the degradation partly to insufficient Stage-1 training. Our training consists of two stages: Stage 1 jointly trains the AR generator and SEH without KV cache eviction, allowing SEH to learn reliable token salience before cache eviction. Stage 2 enables salience-based eviction. We hypothesize that the original 500 Stage-1 iterations were insufficient for SEH convergence, potentially underestimating the salience of semantically important tokens. Consequently, prematurely evicting these tokens during Stage 2 may increase training difficulty and weaken semantic adherence. To investigate this hypothesis, we increase Stage-1 training from 500 to 1,500 iterations while keeping Stage 2 at 2,500 iterations. Evaluations on VBench are shown below:

MethodTotal ↑Quality ↑Semantic ↑
LongLive84.8786.9776.47
PaFu-KV (Original)85.1288.5671.36
PaFu-KV (Extended Stage 1)86.5588.7477.81

Evaluations on VBench-Long are shown below:

MethodSubj. ↑Back. ↑Mot. ↑Aes. ↑Imag. ↑Drift ↓
LongLive90.1291.4299.0762.9469.460.59
PaFu-KV (Original)93.9194.1498.1462.4871.230.29
PaFu-KV (Extended Stage 1)93.8794.2898.7962.7671.150.21

The extended Stage-1 training improves semantic adherence, supporting our hypothesis. We will incorporate the updated results and clarify the two-stage training rationale in the revised manuscript.

Q1: Comparison with matched FIFO and past/current-query targets

Please refer to our response to W2.

Q2: Reconciled operators, selection timing, and the 4,680-token budget

Thank you for identifying these inconsistencies. Eq. (4) defines salience from softmax-normalized attention, with the maximum taken over the relevant queries separately for each head, followed by averaging across heads. Algorithm 3 should therefore apply softmax over the full key dimension, accumulate the query maxima for each head and temporal region across computation chunks, and only then average across heads and fuse the regions using Eq. (5). Averaging across heads before taking the maximum is generally not equivalent.

To match the joint selection in Section 3.4, Algorithm 2 should concatenate the current chunk’s non-sink KV entries and scores with those of the retained non-sink history before selection. With a selectable-cache capacity of Bsel=4,680B_{\mathrm{sel}}=4,680. The same indices select the corresponding keys, values, scores, and positional identifiers across layers. Thus, current tokens compete with historical tokens for retention. The reported 4,680-token capacity refers to the selectable non-sink cache. Protected sink tokens are retained separately and do not consume these slots. If SsinkS_{\mathrm{sink}} sink tokens are retained, the total retained cache is bounded by 4,680+Ssink4,680+S_{\mathrm{sink}} token positions. During generation, the complete current chunk is processed before its final salience scores are available; after generation, only its selected non-sink tokens enter the same 4,680-slot pool as the retained history.

We have highlighted the revisions to the algorithm on Pages 18–19 of the Appendix in red.

Reviewer_KKpKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PaFu-KV learns a cache-retention policy for autoregressive video diffusion. A bidirectional teacher provides salience targets that aggregate attention from past, current, and future temporal blocks. A lightweight head predicts these targets from the student's final-layer QKV features, enabling top-k cache retention during causal inference. The generator and salience head are jointly trained with distribution matching distillation and cache eviction, alongside streaming long tuning and a frame sink. Experiments report higher throughput and improved subject/background consistency on 30-second videos, together with short-video VBench results.

Critical strengths
  1. The design offers a practical route to anticipating future token utility. A bidirectional teacher supplies global salience targets, while a lightweight head estimates retention scores under causal inference constraints. The separation between teacher-side target construction and inexpensive inference-time prediction is well motivated and could make future-aware cache management practical.
  2. The system-level gains are relevant to streaming generation. PaFu-KV achieves 23.6 FPS versus 20.7 for LongLive on the stated H100 setup. For 30-second generation, subject consistency improves from 90.12 to 93.91 and background consistency from 91.42 to 94.14 (Tables 1–2). These are useful outcomes, and the cache-size and frame-sink ablations help characterize the system's operating point.
Critical weaknesses
  1. The experiments do not establish the benefit of future-informed supervision. Table 3 varies the head, aggregation, cache size, and frame sink, but lacks a comparison with an otherwise identical policy trained without future-informed salience targets. Cross-system comparisons also change generator training. The gains could therefore come from learned retention or additional distillation and eviction-aware training, without depending on future information.

  2. Table 1 reports a semantic score of 71.36, below LongLive's 76.47 and Self-Forcing's 81.28, despite better visual-quality and total scores. The long-video results emphasize consistency without resolving this semantic loss. This weakens the claim of negligible quality degradation and matters when judging the speedup.

Questions for authors
  1. Please compare full salience supervision with a past-and-current-only target, keeping student architecture and initialization, DMD supervision, streaming/eviction training, optimization budget, cache capacity, and frame sink fixed. Does removing future-query aggregation leave future context in the bidirectional teacher features? Distinguish this control from a strictly future-free target, and report consistency, semantic adherence, and throughput.

  2. What explains the semantic-score drop in Table 1? Can a different operating point recover semantic adherence while retaining a useful speedup? Please report consistency, throughput, and peak GPU memory; Appendix F already acknowledges limited actual memory savings.

  3. Does Algorithm 3 apply the softmax in Section 3.2? Are maxima computed per head before averaging, as in Eq. 4, or after averaging heads, as the pseudocode suggests? Please specify which target was used for training.

5 comments

Response To Reviewer KKpK:

Thanks for your valuable suggestions. We will incorporate the relevant explanations into the final version for better understanding.

W1: Establishing the contribution of future-informed supervision

To address this concern, we conduct three ablation experiments: Past-Current-only (PCo), Current-Future-only (CFo) and Current-only (Co). All variants are implemented by modifying the salience scoring criterion in the Real Score Network, while keeping the training implementaion unchanged. Specifically, we modify Eq. (5) to construct different salience supervision targets, allowing us to isolate the contributions of past, current, and future interactions. For PCo, we remove the upper-triangular component representing future interactions. Eq. (5) is modified as:

\begin{cases} \frac{\mathrm{diag}_{b,j}+\mathrm{low}_{b,j}}{2}, 0 \leq j < L-L_B,\\\\ \mathrm{diag}_{b,j}, L-L_B \leq j < L. \end{cases}$$ For CFo, we remove the lower-triangular component representing past interactions. Eq. (5) is modified as: $$s_{b,j}^{\text{w/o Lower}}= \begin{cases} \mathrm{diag}_{b,j}, 0\leq j<L_B, \\\\ \frac{\mathrm{up}_{b,j}+\mathrm{diag}_{b,j}}{2}, L_B\leq j<L. \end{cases}$$ For the setting without both PCo and CFo, we remove both the upper- and lower-triangular components. Eq. (5) is modified as: $$s_{b,j}^{\text{w/o Upper,Lower}}=\mathrm{diag}_{b,j}, 0\leq j<L.$$ Furthermore, we include a FIFO control, where cache retention and eviction depend solely on token arrival order. Thus, SEH predictions are never used for cache selection. Retaining SEH and its salience loss would introduce an auxiliary objective with no role in inference-time cache management. We therefore disable both. Without SEH-based salience prediction and top-k selection, the method reduces to a LongLive-style autoregressive framework with FIFO KV-cache management. Hence, we directly use LongLive as the FIFO baseline. All ablation use same training configurations, with only the salience scoring targets or cache eviction policy changed. The results are reported below: | Variant | Subj. ↑ | Back. ↑ | Mot. ↑ | Aes. ↑ | Imag. ↑ | Drift ↓ | |-|-:|-:|-:|-:|-:|-:| | FIFO (LongLive) | 90.12 | 91.42 | **99.07** | **62.94** | 69.46 | 0.59 | | PCo | 91.40 | 92.16 | 97.62 | 62.12 | 68.72 | 0.65 | | CFo | 89.84 | 90.68 | 96.75 | 61.61 | 69.76 | 0.83 | | Co | 85.96 | 86.84 | 90.80 | 57.26 | 65.20 | 2.37 | | **Full PaFu-KV** | **93.91** | **94.14** | 98.14 | 62.48 | **71.23** | **0.29** | These results show that past, current, and future interactions provide complementary information for salience estimation. Removing either component degrades performance, with current-only interactions causing the largest drop. Compared with the FIFO (LongLive) baseline, PaFu-KV better preserves long-term visual consistency and mitigates quality degradation, demonstrating its advantage over heuristic FIFO eviction.

W2: Semantic Degradation Despite Higher Visual Quality and Total Scores

To explain the semantic-score drop, we first clarify that VBench Semantic Score comprises nine dimensions: Object Class, Multiple Objects, Human Action, Color, Spatial Relationship, Scene, Appearance Style, Temporal Style, and Overall Consistency. These cover diverse aspects of text-video alignment, including content, attributes, spatial relationships, scene understanding, and stylistic consistency. These dimensions also use different evaluators and criteria. For example, Object Class and Color rely on GRiT for object detection and color-caption comparison, respectively. Appearance Style and Temporal Style use ViCLIP-based video-text similarity. Therefore, a video may correctly depict the requested objects but still receive lower scores if its appearance or temporal style differs from the textual description. Consequently, accurate depiction of the main content can coexist with a lower aggregate Semantic Score. We follow the official VBench aggregation protocol, which assigns a 4:1 weighting to Quality and Semantic, while reporting both scores separately for transparency. We further provide a dimension-wise breakdown of the Semantic Score to better understand the source of the performance difference.

BaselinesObject Class ↑Multiple Objects ↑Human Action ↑Color ↑Spatial Relationship ↑Scene ↑Appearance Style ↑Temporal Style ↑Overall Consistency ↑Semantic Score ↑
Wan2.1-1.3B90.873.194.684.877.656.124.226.926.680.1
Self-Forcing94.988.195.685.784.253.621.224.627.781.3
LongLive91.584.293.482.278.650.218.823.024.976.5
PaFu-KV92.381.691.680.180.250.115.217.220.171.4

The dimension-wise results suggest that PaFu-KV maintains relatively strong performance in object categories and spatial relationships. More pronounced degradation is observed in Appearance and Temporal Style. This pattern suggests that the semantic-score drop mainly reflects weaker stylistic, instead of failures in generating the objects or actions. This is consistent with PaFu-KV's design. Our salience estimation measures token contributions through past, current, and future attention interactions. Under a limited KV Cache budget, salience-based selection may preserve essential visual structures and temporal continuity while discarding less salient tokens encoding fine-grained attributes or stylistic information. Consequently, visual consistency can be maintained even when some aspects of prompt adherence are weakened.

To further contextualize this observation, we also provide a breakdown of the Quality Score across its seven constituent dimensions. As shown below, PaFu-KV improves several aspect, while exhibiting a slight decrease in Motion Smoothness.

MethodSubject Consistency ↑Background Consistency ↑Temporal Flickering ↑Motion Smoothness ↑Dynamic Degree ↑Aesthetic Quality ↑Imaging Quality ↑Quality
Wan2.1-1.3B96.795.599.398.369.768.568.185.3
Self-Forcing95.596.698.898.465.667.770.285.1
LongLive98.697.999.498.869.368.971.187.0
PaFu-KV98.998.199.598.173.674.775.088.6

The Quality improvement alongside the Semantic decrease reflects a trade-off between visual generation quality and fine-grained prompt adherence. PaFu-KV uses salience-guided historical KV selection to preserve informative contextual tokens and reduce interference from drifting representations, improving long-horizon visual consistency and generation quality. However, attention-based salience does not necessarily capture every prompt-specific semantic constraint. Under a fixed cache budget, tokens encoding fine-grained attributes or stylistic details may be discarded.

Q1: Matched full, past/current, and strictly future-free supervision

Please refer to our response to Reviewer KKpK (W1).

Q2: Semantic loss, alternative operating points, and actual memory savings

Q2.1: Semantic loss

Please refer to our response to Reviewer KKpK (W2).

Q2.2: Alternative Operating Point

We thank the reviewer for highlighting the semantic-score drop and suggesting an alternative operating point. We attribute the degradation partly to insufficient Stage-1 training. Our training consists of two stages: Stage 1 jointly trains the AR generator and SEH without KV cache eviction, allowing SEH to learn reliable token salience before cache eviction. Stage 2 enables salience-based eviction. We hypothesize that the original 500 Stage-1 iterations were insufficient for SEH convergence, potentially underestimating the salience of semantically important tokens. Consequently, prematurely evicting these tokens during Stage 2 may increase training difficulty and weaken semantic adherence. To investigate this hypothesis, we increase Stage-1 training from 500 to 1,500 iterations while keeping Stage 2 at 2,500 iterations. Evaluations on VBench are shown below:

MethodTotal ↑Quality ↑Semantic ↑
LongLive84.8786.9776.47
PaFu-KV (Original)85.1288.5671.36
PaFu-KV (Extended Stage 1)86.5588.7477.81

Evaluations on VBench-Long are shown below:

MethodSubj. ↑Back. ↑Mot. ↑Aes. ↑Imag. ↑Drift ↓
LongLive90.1291.4299.0762.9469.460.59
PaFu-KV (Original)93.9194.1498.1462.4871.230.29
PaFu-KV (Extended Stage 1)93.8794.2898.7962.7671.150.21

The extended Stage-1 training improves semantic adherence, supporting our hypothesis. We will incorporate the updated results and clarify the two-stage training rationale in the revised manuscript.

Q2.3: Actual Memory Savings

We add results measuring throughput and memory across cache sizes. Throughput scales roughly linearly with KV cache size.

SizeThroughput (FPS)GPU Memory (GB)
936019.0227.02
624021.3825.79
4680 (default)23.6024.63
312024.4824.08
156025.7923.48

Q3: Softmax, reduction order, and the target used for training

Thank you for pointing out this ambiguity. We clarify that the salience estimation follows the formulation in Sec. 3.2 and Eq. (4). Specifically, we first compute the attention probabilities using Softmax over the full key dimension. We then compute the maximum attention weight over the query dimension independently for each attention head, separately for the lower-triangular, diagonal, and upper-triangular regions. Finally, we average the resulting per-head maxima across attention heads and fuse the three components according to Eq. (5). Therefore, the training target sreals_{\mathrm{real}} is the resulting spatialtemporal-balanced salience score extracted from the bidirectional teacher. This target supervises the SEH through the SmoothL1 loss in Eq. (6).

We have highlighted the revisions to the algorithm on Pages 18–19 of the Appendix in red.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback. We appreciate their recognition of the following strengths of PaFu-KV:

  • A real limitation of causal cache management. The reviewers noted that an autoregressive generator must decide which tokens to retain before it can observe which ones later chunks will need. This makes future token utility a relevant signal for learning a retention policy, even though it is unavailable at inference (KKpK, jJcq, Thzy).
  • A well-motivated salience target for video. The reviewers appreciated the use of a bidirectional teacher to supervise token retention. PaFu-KV aggregates past, current, and future attention separately, so cross-chunk interactions can contribute to a token's salience even when within-chunk attention is stronger (KKpK, jJcq, Thzy).
  • Lightweight inference without access to future frames. The teacher is used only during training. At inference, the salience head uses the student's available QKV features and runs only at the final Transformer layer and final denoising step. The selected token indices are reused across layers, limiting the overhead of cache selection (KKpK, Thzy).
  • Low inference overhead. The salience head runs only at the final Transformer layer and final denoising step, and the selected token indices are reused across layers (Thzy).
  • Practical gains for streaming video. The reviewers highlighted the throughput improvement over LongLive from 20.7 to 23.6 FPS on an H100, as well as the gains in subject consistency and background consistency on 30-second videos (KKpK, jJcq).

In response to the reviewers' constructive suggestions, we acknowledge that our work can be further improved and have addressed all concerns as follows:

  • FIFO and training-free comparisons (KKpK, jJcq, Thzy). We report matched-cache-budget comparisons on VBench-Long against LongLive (FIFO), Deep Forcing, and Dummy Forcing. PaFu-KV achieves the highest subject and background consistency and the lowest drift among these methods.
  • The effect of explicitly aggregating future queries (KKpK, jJcq). We compare the full target with past/current-only, current/future-only, and current-only targets while keeping the bidirectional teacher and training recipe fixed. Removing future-query aggregation lowers subject/background consistency. The current-only target performs worst among these variants.
  • A strictly future-free teacher control (KKpK). Removing future queries from the aggregation does not prevent the bidirectional teacher's representations from encoding future context. We replaced the bidirectional teacher with an autoregressive video generation model. We adopted the Stage-2 Causal ODE Distillation procedure described in Causal Forcing [1] to eliminate access to future information. Experiment results show removing the future component degrades performance.
  • An alternative training schedule for semantic adherence (KKpK, jJcq, Thzy). We extended the pre-eviction training stage from 500 to 1,500 iterations, keeping Stage 2 fixed at 2,500. This raises the VBench semantic score from 71.36 to 77.81, above LongLive's 76.47. The result suggests that training longer before enabling eviction can improve semantic adherence without changing the inference procedure.
  • Throughput and measured GPU memory across cache sizes (KKpK, jJcq). Reducing the cache budget from 9,360 to 1,560 tokens increases throughput from 19.02 to 25.79 FPS, while measured GPU memory falls from 27.02 to 23.48 GB. The default 4,680-slot setting measures 23.60 FPS and 24.63 GB. These measurements make clear that shrinking the cache yields a more pronounced throughput benefit than an end-to-end memory benefit.

We have uploaded a revised manuscript, with all changes highlighted in red. The main revisions are as follows:

  1. We revised Algorithm 2 (PaFu-KV Inference) and Algorithm 3 (Block-wise Salience Score Computation) in Appendix D (page 18–19).
  2. We Added additional ablation study and analysis in Sec. 4.4 with Past-Current-only (PCo), Current-Future-only (CFo), Current-only (Co), PCo with Causal Forcing and LongLive with 4,680 non-sink slots.

We thank the reviewers again for their constructive comments. We hope that the additional evidence and revisions clarify the scope of our contributions and address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.