acceptodds
Under review as a conference paper at ICLR 2027

Semantically Aligned KV Cache Compression for Efficient Agentic Autoregressive Video Generation

Abstract

Agentic autoregressive video generation offers a promising approach to controllable video synthesis through semantic planning, sequential generation, and visual verification. However, growing visual histories enlarge the key-value (KV) cache, increasing memory use and attention costs across candidate rollouts. KV cache compression reduces this overhead, but existing methods primarily optimize individual generators without explicitly accounting for the planner's semantic constraints. Applying them directly can assign low importance to states associated with particular requested attributes or actions, potentially weakening semantic fidelity. In this paper, we introduce **SemanticKV**, a training-free method for semantically aligned KV cache compression. To our knowledge, **SemanticKV** is the first KV cache compression method for agentic autoregressive video generation that explicitly uses planner-extracted semantic constraints to guide retention. Text cross-attention identifies constraint-related video positions, whose self-attention queries are collected into separate groups. Attention weights are averaged within each group and maximized across groups, giving each cached entry a high score when it is strongly used by any represented constraint. A single top- selection retains high-scoring entries under the memory budget, leaving their keys, values, and positions unchanged. For fixed query groups and equal entry costs, this selection minimizes an additive upper bound on worst-group discarded attention. The method operates independently within each candidate without retraining or additional video generations. Our evaluation spans four AR video backbones and three benchmarks, covering 5-, 30-, and 60-second generation; our method achieves strong performance across these settings. On Causal Forcing at a compression ratio of , **SemanticKV** obtains a VBench Total Score of , compared with for the strongest compressed baseline and for Full KV. Meanwhile, it achieves a speedup of over Full KV and reduces KV-cache memory from GB to GB, corresponding to a reduction.

Primary area:
generative models
Keywords:
Agentic Autoregressive Video Generation, KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PGToOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SemanticKV, a training-free method for KV Cache compression in autoregressive video generation. It compresses the KV cache by discarding entries that are not strongly attended by the selected keyword text tokens. The authors provided comparisons with other baseline methods on Causal Forcing, Self Forcing, Longlive, and Longlive 2.0 to demonstrate the effectiveness of SemanticKV.

Critical strengths
  1. The paper addresses a meaningful problem in AR video generation: KV cache compression. As the video length grows, the KV cache should be sensibly managed to control the computational budget.
  2. Core code is provided in the supplementary material to enhance reproducibility.
Critical weaknesses
  1. Inaccurate and incremental contributions: a) The paper spent too much space describing the agentic AR video generation framework in Section 3. However, it is mostly unrelated to the methodology section, except for the extraction of semantic constraints R. The authors should consider substantially reorganizing the paper's framing to highlight the real contribution. b) The paper's method can be described as retaining the most attended KV entries by the selected keyword tokens, which has been explored by previous works. In addition, the method also has no significant relation to agentic generation.
  2. Insufficient experimental results: a) The authors did not provide any video results. In addition, the qualitative results in the appendix remain limited. It is rather hard to assess the real performance of video generation methods without substantial video results. b) Agentic workflows play a more important role in long video generation than in generating short clips. However, there is no comparison with other baseline methods except Full KV, which is critical in the experiments.
Questions for authors

Did the authors apply the agentic AR video generation framework to all baseline methods? This should be explicitly stated to ensure fair comparisons, since the original prompts seem to be very short in the appendix.

2 comments
AuthorsOpenReview ↗

Response to Reviewer PGTo (Part 1/2)

Thank you for asking us to clarify the contribution of the cache policy, the comparison protocol, and the qualitative evidence for long-video generation.

W1(a) Inaccurate and incremental contributions: a) The paper spent too much space describing the agentic AR video generation framework in Section 3. However, it is mostly unrelated to the methodology section, except for the extraction of semantic constraints R. The authors should consider substantially reorganizing the paper's framing to highlight the real contribution.

We agree that the link from Section 3 to the cache policy should be explicit. The revised opening of Section 3 states that the planner, generator, and verifier define the shared workflow and points directly to the constraint-conditioned query selection in Section 4.1. The planner produces constraints and a rewritten prompt. The generator uses the prompt for synthesis, and SemanticKV uses the constraints to determine which visual queries guide cache retention. Section 5.5 evaluates that cache policy under the common workflow.

W1(b) b) The paper's method can be described as retaining the most attended KV entries by the selected keyword tokens, which has been explored by previous works. In addition, the method also has no significant relation to agentic generation.

Thank you for pointing out the connection between SemanticKV and prior attention-based eviction methods. We agree that using observed attention to identify history worth retaining has precedents. Two distinctions are relevant here.

  1. Which queries score the KV entries? SnapKV, for example, uses queries from a window at the end of the text prompt to identify important historical positions. In SemanticKV, keywords serve a localization role. We match each planner constraint to the corresponding words in the prompt, then use text cross-attention to locate visual positions associated with those words. The visual self-attention queries at those positions then score historical video KV entries. Different constraints in the same prompt can therefore supply different scoring queries.
  2. How does the planner affect cache retention? Each constraint supplied by the planner defines a query group. Within a head, SemanticKV averages attention to each cache entry separately for each group and takes the highest group average. An entry associated with a requested object can thus receive high retention priority even if only one group attends to it. We tested this choice by keeping the planner, queries, and cache budget fixed and comparing constraint-wise scoring with pooling the same queries. On prompts with at least three constraints, scoring the groups separately improves Semantic Score by 0.96 points (95% CI [+0.08, +2.01]). This comparison directly tests the value of preserving the planner’s constraint groups during cache scoring (Appendix I).

Section 2 of the revision discusses the connection to prior work. Sections 3 and 4.1 explain how the planner’s output determines query selection and grouping.

W2(a) Insufficient experimental results: a) The authors did not provide any video results. In addition, the qualitative results in the appendix remain limited. It is rather hard to assess the real performance of video generation methods without substantial video results.

Thank you for pointing out the limited qualitative examples in the original submission. We added two matched frame comparisons in Appendix I. Figure 13 shows frames from 30-second videos generated by five methods, and Figure 14 shows their 60-second results for the same scene. Both figures use the same original prompt and seed 42. We selected the first MovieGen prompt in hash order among those with at least three constraints, independently of generation scores.

The prompt describes a young woman driving a car, with a blurred green forest visible through a rainy window. Each row shows one method at five equally spaced times, with the columns aligned across methods. Reading across a row shows how the subject and background appear at successive sampled times. Reading down a column lets the reader compare how the methods depict the same scene requirements.

AuthorsOpenReview ↗

Response to Reviewer PGTo (Part 2/2)

W2(b) b) Agentic workflows play a more important role in long video generation than in generating short clips. However, there is no comparison with other baseline methods except Full KV, which is critical in the experiments.

We agree that evaluating long-video generation in an agentic workflow calls for comparing cache policies within that same workflow. We therefore added 30- and 60-second comparisons with Dummy Forcing and Forcing-KV on LongLive 2.0. All three methods share the planner and critic, use the same 128 MovieGen prompts and five seeds, and retain 8,448 KV slots per layer.

Under these conditions, SemanticKV preserves the subject’s appearance across successive clips more consistently than Forcing-KV, with a 3.97-point advantage at 60 seconds. We also measured within-clip quality by aggregating six VBench dimensions on two-second clips. SemanticKV exceeds Dummy Forcing on this measure by more than one point at both durations.

MetricDurationSemanticKVComparatorGain (points)
Cross-clip subject consistency30 s83.64Forcing-KV (80.10)+3.54
Cross-clip subject consistency60 s78.16Forcing-KV (74.18)+3.97
Within-clip score30 s79.66Dummy Forcing (78.31)+1.35
Within-clip score60 s78.84Dummy Forcing (77.81)+1.04

Section 5.5 reports the paired within-clip score differences against both compressed baselines and Full KV, with 95% confidence intervals. Appendix I gives the scores before differencing.

Q1 Did the authors apply the agentic AR video generation framework to all baseline methods? This should be explicitly stated to ensure fair comparisons, since the original prompts seem to be very short in the appendix.

Yes. Within each backbone, all five policies use the same cached plan, original and rewritten prompts, two-candidate generation protocol, and critic selection. We used this common plan to regenerate candidates whose saved plans differed and replayed the affected selections. The short captions show the original benchmark prompts. For example, Appendix I records both the original input “a white car” and its saved rewrite and exclusions. The same planner fields are used for every method.

For SemanticKV, a seed-matched rewrite raises independent constraint satisfaction by 3.94/3.37 points at 30/60 seconds, and critic selection adds 2.44 points at both lengths over a uniform choice from the same pair. All five methods benefit from selection. Section 5.5 and Appendices B.2 and I make the shared workflow and the cache-policy comparisons explicit.

Reviewer_76VHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SemanticKV, a training-free KV cache compression method for agentic autoregressive video generation. It uses planner-extracted semantic constraints to identify relevant self-attention queries, aggregates attention scores within and across constraint groups, and retains important KV entries under a fixed budget. The paper provides a theoretical justification for the scoring strategy and evaluates quality and efficiency across four video backbones and different compression settings.

Critical strengths

The main idea of using semantic constraints to guide KV cache compression is interesting and fits the agentic generation setting. Standard attention-based compression may discard information that is important for a specific object or action but receives relatively low overall attention. SemanticKV addresses this by considering different semantic constraints separately. The groupwise maximum aggregation is straightforward, and the method does not require retraining or modifications to the generator.

The theoretical result in Section 4 and Appendix D is also useful, although it applies to the proposed attention-based objective rather than directly guaranteeing video quality. Experimentally, the results are encouraging under aggressive compression. For instance, at a compression ratio of 0.9 on LongLive, SemanticKV achieves an Object Class score of 91.60, compared with 88.78 for Dummy Forcing and 87.91 for Forcing-KV. The additional statistical analyses in Appendix G are helpful for interpreting the results. Overall, I think the proposed direction is reasonable, particularly for generation tasks involving multiple semantic constraints.

Critical weaknesses

My main concern is that the improvements over existing compressed baselines are often quite small. In Table 1, the Total Score improvements over the strongest compressed baseline are only 0.13 on Self Forcing and 0.05 on Causal Forcing. It is difficult to judge the practical significance of these differences. The ablation results in Table 7 raise a similar question. The complete method achieves a Semantic Score of 77.18, compared with 76.94 using uniform queries and 76.78 using pooled constraints. These results are consistent with the proposed design, but the margins are narrow. Appendix G provides useful statistical analyses of the main comparisons, although most individual dimension-level confidence intervals include zero. More importantly, the reported analyses do not directly establish whether the improvements from the individual components in Table 7 are statistically reliable.

I am also not fully convinced by the long-horizon evaluation. Table 5 reports results for 30- and 60-second generation, but the comparison is limited to Full KV. The reported speedups are useful, yet they do not show whether SemanticKV performs better than other compression methods as the generation history grows. Since preserving important historical information is a central motivation of the paper, I would expect at least some long-horizon comparisons with compressed baselines under matched memory budgets.

Another issue is the clarity of the baseline implementations. Appendix B.2 does not provide sufficiently concrete details about some of the adapted methods. In particular, the exact attention scoring procedures, head classification settings, and retention schedules are not entirely clear to me. Given the relatively small performance differences, these implementation choices could affect the comparisons. Clarifying them would make the experimental evidence more convincing.

Questions for authors
  1. Could the authors provide paired confidence intervals or repeated-run statistics for the component ablations in Table 7? In particular, are the gains from semantic query selection and groupwise aggregation consistent across prompts and random seeds?
  2. How does SemanticKV compare with existing compressed baselines in the 30- and 60-second generation settings? A comparison under matched cache budgets would better establish its advantage in long-horizon generation.
  3. Could the authors clarify the actual baseline adaptation settings in Appendix B.2, particularly the attention importance scoring, head classification, and retention schedules?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 76VH (Part 1/2)

Thank you for requesting paired component estimates, matched long-video comparisons, and concrete details about the baseline adapters.

W1 & Q1 My main concern is that the improvements over existing compressed baselines are often quite small. In Table 1, the Total Score improvements over the strongest compressed baseline are only 0.13 on Self Forcing and 0.05 on Causal Forcing. It is difficult to judge the practical significance of these differences. The ablation results in Table 7 raise a similar question. The complete method achieves a Semantic Score of 77.18, compared with 76.94 using uniform queries and 76.78 using pooled constraints. These results are consistent with the proposed design, but the margins are narrow. Appendix G provides useful statistical analyses of the main comparisons, although most individual dimension-level confidence intervals include zero. More importantly, the reported analyses do not directly establish whether the improvements from the individual components in Table 7 are statistically reliable. Could the authors provide paired confidence intervals or repeated-run statistics for the component ablations in Table 7? In particular, are the gains from semantic query selection and groupwise aggregation consistent across prompts and random seeds?

  1. Practical significance of the Table 1 gains. We agree that the Total Score margins over the strongest compressed baseline are modest. At 70% cache eviction, all compressed methods score within 0.32 points of Full KV on both backbones, leaving little room for further improvement. Even so, SemanticKV achieves the highest compressed Total Score on Self Forcing (83.03), leading the strongest baseline, R-KV, by 0.12 points and Forcing-KV by 0.39 points (95% CI [+0.18, +0.61]). SemanticKV also reaches 1.17× and 1.23× the DiT throughput of Full KV on Self Forcing and Causal Forcing, respectively. The practical benefit is preserving video quality with a substantially smaller cache.

  2. Contribution of each component in Table 7. Following your suggestion, we replaced one component at a time, matched outputs by prompt and seed, and drew 10,000 paired bootstrap resamples at the prompt level. Semantic query selection improves mean Semantic Score by 0.21 points over uniform queries (95% CI [−0.31, +0.74]). Groupwise scoring improves it by 0.38 points over pooled constraints ([−0.12, +0.86]). Both estimates agree in direction with Table 7, although both intervals include zero across the full prompt set.

  3. Where are the gains clearest? Semantic query selection improves Semantic Score in 4 of 5 seeds, and groupwise scoring improves it in 3 of 5. The clearest evidence for groupwise scoring comes from prompts with multiple constraints. On prompts with at least three requirements, scoring each constraint separately improves Semantic Score by 0.96 points over pooling (95% CI [+0.08, +2.01]). This comparison directly tests the setting that motivated the design: when a video must satisfy several requirements at once, keeping their scores separate improves semantic performance on average (Appendix I).

AuthorsOpenReview ↗

Response to Reviewer 76VH (Part 2/2)

W2 & Q2 I am also not fully convinced by the long-horizon evaluation. Table 5 reports results for 30- and 60-second generation, but the comparison is limited to Full KV. The reported speedups are useful, yet they do not show whether SemanticKV performs better than other compression methods as the generation history grows. Since preserving important historical information is a central motivation of the paper, I would expect at least some long-horizon comparisons with compressed baselines under matched memory budgets. How does SemanticKV compare with existing compressed baselines in the 30- and 60-second generation settings? A comparison under matched cache budgets would better establish its advantage in long-horizon generation.

We agree that assessing how well a method preserves history as generation progresses calls for a direct comparison at the same cache budget. We compared SemanticKV, Dummy Forcing, and Forcing-KV on LongLive 2.0 at 30 and 60 seconds. Each method retained 8,448 KV slots per layer and used the same 128 MovieGen prompts and five seeds.

To address your question about historical information, we measured how consistently the same subject appears across successive clips. SemanticKV exceeds Forcing-KV on this measure by 3.54 points at 30 seconds and 3.97 points at 60 seconds. We also assessed within-clip quality by aggregating six VBench dimensions on two-second clips. On this measure, SemanticKV scores higher than Dummy Forcing at both durations.

MetricDurationSemanticKVComparatorGain (points)
Cross-clip subject consistency30 s83.64Forcing-KV (80.10)+3.54
Cross-clip subject consistency60 s78.16Forcing-KV (74.18)+3.97
Within-clip score30 s79.66Dummy Forcing (78.31)+1.35
Within-clip score60 s78.84Dummy Forcing (77.81)+1.04

Section 5.5 reports the paired within-clip score differences against both compressed baselines and Full KV, with 95% confidence intervals. Appendix I gives the scores before differencing.

W3 & Q3 Another issue is the clarity of the baseline implementations. Appendix B.2 does not provide sufficiently concrete details about some of the adapted methods. In particular, the exact attention scoring procedures, head classification settings, and retention schedules are not entirely clear to me. Given the relatively small performance differences, these implementation choices could affect the comparisons. Clarifying them would make the experimental evidence more convincing. Could the authors clarify the actual baseline adaptation settings in Appendix B.2, particularly the attention importance scoring, head classification, and retention schedules?

We replaced the general baseline descriptions in Appendix B.2 with the settings used for evaluation.

  • R-KV averages attention from 64 current queries per head on the short-video backbones and combines importance and redundancy using a 0.07 mixing coefficient and a 0.5 similarity threshold. The LongLive 2.0 adapter uses 16 observations and scores 64-token groups.
  • Dummy Forcing classifies LongLive 2.0 heads at the first compression of each candidate, using attention from 16 queries to the first, middle, and most recent history. Forcing-KV transfers the released LongLive head profile to LongLive 2.0, with recent-history retention for static heads and adjacent-group cosine scoring for dynamic heads. R-KV and Forcing-KV select after each clean cache update.
  • The four compressed methods on LongLive 2.0 protect the most recent 7,040 slots and retain 1,408 additional history slots per layer. The logged KV allocation is 6.229 GB per compressed candidate.
Reviewer_sdFTOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets KV-cache cost in an agentic autoregressive video pipeline. The authors argue that existing KV compression methods rank cached entries by globally pooled importance and may discard history that matters for a specific requested object, attribute or action. The proposed SemanticKV is training-free. It matches each planner-extracted constraint to its words in the prompt, uses text cross-attention to find the video positions that attend most to those words, and collects their self-attention queries into one group per constraint. Each cached entry is scored by its mean attention within each group and then the maximum over groups, and a single top-k selection keeps the highest-scoring entries under the memory budget. The authors show that this score gives the tightest additive upper bound on the worst-group discarded attention. Experiments use Self Forcing and Causal Forcing, LongLive and LongLive 2.0, compared with Full KV, R-KV, Dummy Forcing and Forcing-KV, plus paired per-prompt analyses and qualitative examples.

Critical strengths
  1. Using the semantic constraints already produced by the planner to guide cache retention is intuitive, and the method is training-free and keeps keys, values and positions unchanged.
  2. The mean–max aggregation is simple and well motivated against pooled scoring, and both components are ablated.
  3. The evaluation covers four backbones, three benchmarks and several compression ratios, with paired prompt-level analyses.
Critical weaknesses
  1. The Total Score gains are marginal: +0.13 and +0.05 over the best baseline, and all compressed methods lie within 0.34/0.10 of each other (Table 1). On these two backbones SemanticKV is also slower than Dummy Forcing.
  2. Long videos are where KV compression matters most, yet the 30/60-s results (Table 5) only compare against Full KV, with no compressed baseline.
  3. Replacing constraint-conditioned queries with uniform ones costs only 0.24 Semantic Score and gives a higher Total Score (81.73 by the VBench formula vs. 81.68, Table 7). On one-constraint prompts the gains are near zero (App. G.2), yet Figs. 8 and 11 use such prompts. What does the agentic setting actually add?
  4. SemanticKV beats Full KV on semantic dimensions (Table 9; +3.83 on 3+-constraint prompts in App. G.2; Figs. 8–12). Theorem 4.1 measures deviation from Full KV. How does "preserving constraint-related states" explain doing better than the full cache?
  5. Technical novelty is limited: scoring cache entries by the attention of selected queries follows standard attention-based eviction (H2O [2], SnapKV [1], not cited); max aggregation over groups appears in OmniKV [3] (over heads), and the bound with top-k optimality follows Ada-KV, as App. D.3 itself acknowledges.

[1] SnapKV: LLM Knows What You are Looking for Before Generation

[2] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

[3] OmniKV: Dynamic Context Selection for Efficient Long-Context LLMs

Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer sdFT (Part 1/2)

Thank you for asking us to make the empirical claims more precise and to clarify the scope and novelty of the method.

W1 The Total Score gains are marginal: +0.13 and +0.05 over the best baseline, and all compressed methods lie within 0.34/0.10 of each other (Table 1). On these two backbones SemanticKV is also slower than Dummy Forcing.

Video quality. We agree that the gains in Total Score are modest. In the five-second experiments, even after 70% cache eviction, every compressed method scores within 0.32 points of Full KV on both backbones. This leaves little room to improve on the strongest baseline. Even so, on Self Forcing, SemanticKV achieves the highest Total Score among the compressed methods (83.03), outperforming the strongest baseline, R-KV, by 0.12 points and Forcing-KV by 0.39 points (Section 5.5 and Appendix I).

Speed. We agree that Dummy Forcing is faster on both backbones, but the gap in DiT throughput is small. On Self Forcing, SemanticKV’s throughput is 2.8% lower, yet its Total Score is 0.19 points higher (83.03 vs. 82.84). On Causal Forcing, the throughput gap is 3.4%. Relative to Full KV, SemanticKV still delivers 1.17× and 1.23× the DiT throughput on Self Forcing and Causal Forcing, respectively (Section 5.2 and Appendix I).

W2 Long videos are where KV compression matters most, yet the 30/60-s results (Table 5) only compare against Full KV, with no compressed baseline.

We agree that long-video evaluation calls for a direct comparison with compressed baselines. We therefore compare SemanticKV with Dummy Forcing and Forcing-KV on LongLive 2.0 at 30 and 60 seconds. All three methods retain 8,448 KV slots per layer and use the same 128 MovieGen prompts and five seeds.

At both durations, SemanticKV scores higher than Dummy Forcing on within-clip quality, measured by aggregating six VBench dimensions on two-second clips. Compared with Forcing-KV, SemanticKV better preserves the subject’s appearance across successive clips. The table highlights these two gains.

MetricDurationSemanticKVComparatorGain (points)
Within-clip score30 s79.66Dummy Forcing (78.31)+1.35
Within-clip score60 s78.84Dummy Forcing (77.81)+1.04
Cross-clip subject consistency30 s83.64Forcing-KV (80.10)+3.54
Cross-clip subject consistency60 s78.16Forcing-KV (74.18)+3.97

Section 5.5 reports the paired within-clip score differences against both compressed baselines and Full KV, with 95% confidence intervals. Appendix I gives the scores before differencing.

W3 Replacing constraint-conditioned queries with uniform ones costs only 0.24 Semantic Score and gives a higher Total Score (81.73 by the VBench formula vs. 81.68, Table 7). On one-constraint prompts the gains are near zero (App. G.2), yet Figs. 8 and 11 use such prompts. What does the agentic setting actually add?

Your reading of Table 7 is correct. Uniform queries give a slightly higher Total Score than SemanticKV (81.73 vs. 81.68), while SemanticKV scores 0.24 points higher on Semantic Score (77.18 vs. 76.94). We also agree that the gains on single-constraint prompts are small and that the examples in Figures 8 and 11 cannot establish the benefit of grouping multiple constraints.

What does the shared workflow contribute? We evaluated prompt rewriting and candidate selection separately. Holding the generation seed fixed, rewriting the prompt improves independently judged constraint satisfaction for SemanticKV by 3.94 percentage points at 30 seconds and 3.37 at 60 seconds. Selecting from the same two candidates with the critic improves constraint satisfaction by 2.44 percentage points at both durations over random selection (Section 5.5). These comparisons measure how SemanticKV benefits from the workflow shared by all methods.

How does SemanticKV use the planner’s constraints? SemanticKV groups queries by constraint. For each cache entry, it averages the attention weights within each group and takes the highest of these group averages. With one constraint, this reduces to the mean of a single group. With several constraints, an entry can receive a high retention score even if it matters to only one of them. On two-constraint prompts, SemanticKV exceeds Dummy Forcing by 3.26 points and Forcing-KV by 7.24 points, averaged across six semantic dimensions (Section 5.4 and Appendix G.2). Appendix I also shows matched frames at both durations for one prompt with at least three constraints, using the same seed across methods.

AuthorsOpenReview ↗

Response to Reviewer sdFT (Part 2/2)

W4 SemanticKV beats Full KV on semantic dimensions (Table 9; +3.83 on 3+-constraint prompts in App. G.2; Figs. 8–12). Theorem 4.1 measures deviation from Full KV. How does "preserving constraint-related states" explain doing better than the full cache?

Your reading of Theorem 4.1 is correct. For fixed queries and a fixed pre-eviction cache, discarding less attention mass from a constraint group gives a tighter upper bound on that group’s mean attention-output error. This explains why we prioritize entries that each constraint group relies on. The bound alone cannot predict a higher semantic score for the generated video. We clarify this scope in Section 4.2.

Why, then, can a video score higher than Full KV? Full KV provides the reference attention output in the theorem, but its generated video can still miss a prompt requirement. SemanticKV prioritizes entries that receive high attention from at least one constraint group. After eviction, attention is renormalized over the remaining entries. This changes subsequent generation and can yield a video that better matches the prompt. For example, given “a snowboard,” Full KV generates a skateboard, while SemanticKV generates a snowboard being ridden on snow. Their Object Class scores are 0 and 1, respectively. This illustrates how the scores can differ, although the example cannot identify which cache entry caused the change.

On prompts with at least three constraints, the 3.83-point gain you highlight is the average difference between SemanticKV and Full KV across six semantic dimensions (95% CI [+1.17, +6.98], Appendix G.2). This result describes how well the generated videos match the requested content on those dimensions.

W5 Technical novelty is limited: scoring cache entries by the attention of selected queries follows standard attention-based eviction (H2O [2], SnapKV [1], not cited); max aggregation over groups appears in OmniKV [3] (over heads), and the bound with top-k optimality follows Ada-KV, as App. D.3 itself acknowledges. [1] SnapKV: LLM Knows What You are Looking for Before Generation [2] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models [3] OmniKV: Dynamic Context Selection for Efficient Long-Context LLMs

Thank you for pointing out the connections between SemanticKV and prior work on KV-cache eviction. We agree that attention-based ranking, maximum aggregation, and top-k selection under a bound all have precedents. We build on these ideas to address a specific question: when a video prompt contains several requirements, how can a fixed cache budget preserve visual history that matters to any one of them?

  1. Which queries score the cache. SemanticKV, H2O, and SnapKV are similar in that they use observed attention to identify history worth retaining. They differ in which observations determine an entry’s score. H2O accumulates past attention, and SnapKV uses queries from a window at the end of the text prompt. In SemanticKV, each planner-extracted constraint determines where to look. Text cross-attention first locates visual positions associated with that constraint. The visual self-attention queries at those positions then score historical video KV entries. Different requirements in the same prompt can therefore supply different scoring queries.
  2. Which groups enter the maximum. SemanticKV and OmniKV are similar in that both aggregate attention scores using a maximum. They differ in what the maximum ranges over. OmniKV maximizes across attention heads. SemanticKV stays within one head, averages scores separately for each constraint group, and takes the largest group average. For example, if only a few queries attend to a requested object, pooling all queries can dilute the score of an entry those queries rely on. With separate group averages, that entry can still receive high priority even when other groups contain many more queries.
  3. What top-k selection optimizes. SemanticKV and Ada-KV are similar in that both relate discarded attention mass to local output changes and analyze top-k selection through a bound. They differ in the objective. We ask how much attention is lost by the worst-affected observed constraint group. We show that, among fixed nonnegative entry weights whose sum over evicted entries bounds this loss for every eviction choice, the smallest valid weight for each entry is its highest mean score across constraint groups. With fixed query groups and equal entry costs, retaining the k highest-scoring optional entries therefore minimizes this additive bound. This gives a theoretical reason for taking the maximum across constraints.

Section 2 of the revision adds the relevant citations. Section 4.2 and Appendix D explain the constraint-group score and its theoretical objective.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank the reviewers and the area chair for their careful reading and constructive feedback. The reviews asked how the planner influences cache retention, whether small quality differences are reliable, and how SemanticKV compares with other compressed methods on long videos. We address these questions in the individual responses below.

Main questions addressed

  • Novelty and the role of the agentic workflow (sdFT and PGTo). We compare SemanticKV with H2O, SnapKV, OmniKV, and Ada-KV. We explain how planner constraints identify groups of visual queries and how these groups enter the cache score. We also distinguish the effects of prompt rewriting and candidate selection, which all methods share, from the effect of changing cache retention.
  • Experimental comparisons and their scope (all reviewers). We report paired short-video results, matched-budget comparisons at 30 and 60 seconds, and prompt-cluster intervals for both component ablations. We clarify which claims concern local attention error, overall video quality, appearance consistency, and constraint satisfaction.

Changes in the revised manuscript

  • Clarified the method and its relation to prior work (Sections 2–4 and Appendix D). We cite the relevant cache-selection precedents, identify where the planner constraints enter the retention rule, and state the fixed-query and fixed-cache scope of the theoretical result.
  • Updated the short-video summary and added matched experiments (Sections 5.2–5.5 and Appendix I). At 90% eviction on LongLive, controlled DiT throughput reaches 1.62× Full KV and exceeds Dummy Forcing by 14.6%. The new section reports controlled timing, 30- and 60-second compressed baselines, the roles of rewriting and selection, and uncertainty in the cache ablations.
  • Documented the baseline adapters and provided matched qualitative examples (Appendices B.2 and I). We specify scoring, head assignment, update schedules, and realized cache capacity. We add an actual saved plan and matched frames at both long-video durations.

New and revised manuscript text, updated values, and captions of added figures are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.