acceptodds
Under review as a conference paper at ICLR 2027

Entropy Forcing: Entropy-Routed Subset Selection for KV Cache in Streaming Video Generation

Abstract

Autoregressive (AR) video diffusion has emerged as a promising paradigm for streaming and open-ended long-video synthesis, with generation quality depending heavily on effective KV cache management under a fixed budget. Existing methods typically rely on attention-based importance, selecting KV entries with the highest attention scores for subsequent generation. However, the reliability of this importance signal depends on how distinctly the attention scores separate different KV entries. Specifically, well-separated scores yield clear rankings, whereas clustered scores provide limited discrimination and make selection less reliable. In such cases, existing methods lack an appropriate criterion for selecting a suitable subset of KV entries. To address these limitations, we propose **Entropy Forcing**, a training-free KV cache selection method for AR video diffusion. First, instead of scoring KV entries independently, we introduce a coverage-aware objective that explicitly accounts for redundancy among retained historical representations. Second, we use normalized Rényi-2 entropy as a global routing signal to adaptively determine how much the selection should rely on attention-based importance versus representation coverage. Finally, we exhaustively score and rank candidate KV subsets under the given budget, with parallel evaluation enabling efficient inference. Experiments on long-video generation show that Entropy Forcing improves motion dynamics while maintaining competitive visual quality and temporal consistency compared with recent state-of-the-art methods. Notably, it consistently achieves the best overall average performance across the evaluated metrics.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, KV Cache Management, Long-Horizon Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w87nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an entropy-guided KV cache selection method that improves visual quality and temporal consistency without retraining the entire video model.

Critical strengths
  1. The paper is easy to follow, with clear motivation and a novel perspective on addressing the problem.
  2. Compared with many existing KV cache modification approaches, the proposed method is simple and effective. Using entropy to guide cache selection is intuitive and draws on a concept commonly used in LLMs.
Critical weaknesses
  1. Are there any failure cases or potential drawbacks associated with the proposed method? Although entropy-based cache selection is simple and effective, it would be helpful to discuss scenarios in which it may negatively affect visual quality or temporal consistency.
  2. How does the proposed method perform in long-video generation, particularly for videos longer than one minute?
Questions for authors

See weaknesses.

3 comments

Thank you for highlighting the clear motivation and the intuitive use of entropy to guide KV cache selection. We appreciate your assessment that the approach is simple and effective.

[W1] Failure-case analysis

We added two failure cases from 120-second Entropy Forcing generations to Figure 18 in Appendix G, with 16 timestamped frames per video.

  1. Inconsistent character appearance (Figure 18a). At 20 s, the yellow-orange fairy and the blue protagonist look distinct. At 23–24 s, the fairy develops long ears like the protagonist's. By 25 s, its face and fur also resemble the protagonist's, and the two characters look nearly identical in the sampled frames through 38 s. At 39 s, only one blue character is visible again. A possible explanation is that the model mixes the two characters' features as it generates successive chunks and carries the changed appearance into later frames.

  2. Prompt-following failure (Figure 18b). The prompt asks for a wide aerial view showing Santorini's town and bay together, as seen at 2 s. In the 32–41 s samples, a nearby terrace hides most of the town. In the 79–83 s samples, the walls and windows of a single building fill most of the frame. The wide view returns at 84–85 s, and a nearby building again dominates the foreground at 119 s. One possible reason is that the model struggles to maintain a camera view specified only through text over a long video.

[W2] Performance on videos longer than one minute

We have added a VBench-Long comparison on 120-second videos. The quantitative results are presented below and included in Section 5.1 of the revised manuscript, with qualitative visual comparisons provided in Appendix G(Figure 15).

MethodDynamic Degree ↑Motion Smoothness ↑Overall Consistency ↑Imaging Quality ↑Aesthetic Quality ↑Subject Consistency ↑Background Consistency ↑Avg. ↑
Self Forcing36.7597.8720.5968.0356.4997.7696.5867.72
+ Deep Forcing40.7597.8622.7870.2761.2297.5196.2869.52
+ Entropy Forcing (ours)44.2597.9922.7171.8962.5798.2397.2570.70

As shown, Entropy Forcing outperforms Deep Forcing on six of the seven metrics at 120 seconds, with gains of 3.50 points in Dynamic Degree and 1.18 points in the average score. These results demonstrate the effectiveness of our approach for video generation beyond one minute.

Thank you again for the constructive feedback.

Thank you again for your constructive feedback. We hope our rebuttal and additional experiments have adequately addressed your concerns. We would appreciate it if you could let us know whether any concerns remain or if further clarification would be helpful.

The authors' response has addressed my concerns, and I will maintain my original score.

Reviewer_TvnbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Entropy Forcing, a training-free KV cache selection method for autoregressive video diffusion. It addresses two limitations of conventional attention-based selection: unreliable importance rankings under flat attention distributions and redundancy among independently selected historical frames. The method introduces a facility-location-style representation coverage objective and uses normalized Rényi-2 entropy to adaptively balance attention relevance and coverage. Exhaustive subset selection identifies the optimal retained memory under a fixed budget. Experiments on Wan2.1-T2V-1.3B demonstrate improved motion dynamics and visual quality with negligible selection overhead.

Critical strengths
  1. Well-motivated problem formulation. The paper identifies meaningful limitations of attention-based KV selection, supported by attention-score analysis, historical representation similarity, and a strong negative correlation between attention entropy and selection effectiveness (Pearson (r=-0.80)).

  2. Simple and effective training-free design. The coverage-aware objective explicitly considers redundancy among retained frames, while entropy-guided routing provides an adaptive selection mechanism without additional training. The method improves Dynamic Degree from 52.90 to 56.30 for 30-second videos and introduces less than 0.02% selection overhead.

Critical weaknesses
  1. Insufficient theoretical justification for entropy-guided routing. Although the paper establishes a mathematical relationship between Rényi-2 entropy and the uniformity of attention distributions, this does not justify the proposed linear weighting of attention relevance and representation coverage. In particular, a flatter attention distribution does not necessarily imply that representation coverage contributes more to generation quality. The choice of HH and 1−H1-H as fusion weights remains heuristic, without theoretical guarantees or comparisons against alternative weighting strategies.

  2. Potential loss of spatiotemporal information in representation coverage. The proposed coverage objective relies on mean-pooled frame-level value representations, which may discard spatial structure, local motion, and fine-grained temporal differences. Frames with similar global representations may contain substantially different object positions or motion states. Consequently, high feature similarity does not necessarily indicate redundant historical information. The paper provides limited evidence that the proposed representation similarity reliably reflects the actual impact of KV eviction on generation quality.

  3. Limited experimental generalizability. All main experiments are conducted on Wan2.1-T2V-1.3B, without validation on other autoregressive video generation architectures, such as Causal Forcing, or larger models.

Questions for authors
  1. Could the authors explain why the fusion weights should be exactly (H) and (1-H), and provide comparisons against fixed-weight fusion and alternative concentration measures, such as Shannon entropy or attention-score variance?
  2. Could the authors report variance or confidence intervals for the ablation results and extend evaluation to additional video generation architectures?
  3. Given the combinatorial exhaustive search, how does selection overhead scale with larger candidate sets?
6 comments

Thank you for highlighting the motivation for our problem formulation and the empirical analyses of attention scores and historical redundancy that support it. We also appreciate your recognition that the training-free design improves motion dynamics while keeping selection overhead low.

[W1 & Q1] Justification for entropy-guided routing mechanism

Thank you for raising this concern. Our use of HH is motivated by a clear empirical trade-off: the relative effectiveness of attention-based relevance versus representation coverage shifts systematically with the flatness of the attention distribution. Specifically, normalized Rényi-2 entropy HH measures attention flatness. As HH increases, the reconstruction advantage of Attention Top-BB over random selection drops sharply (r=−0.80r = -0.80, Figure 5), confirming that attention-based ranking loses discriminative power when scores become uniform. Our experiments further confirm the complementary role of coverage across entropy regimes (Appendix A.2, Table 4):

  • High-entropy regime: Coverage-only outperforms Attention-only by 0.68 dB in mean local PSNR.
  • Low-entropy regime: Attention-only leads Coverage-only by 0.85 dB.

Together, these results directly justify using HH to adaptively govern the relevance–coverage trade-off—prioritizing coverage when attention is flat (high HH) and relying on attention ranking when it is concentrated (low HH).

Below, we present three comparative experiments demonstrating that our entropy-guided routing design is not an arbitrary heuristic, but an empirically grounded, simple, and effective approach.

1. Comparison with other combination strategies.

We compare our weighting scheme, which assigns H to coverage and 1-H to attention-based relevance, with nonlinear alternatives based on HsH^{s} and 1−Hs1-H^{s}. The results are presented below and reported in Appendix D.2 of the revised manuscript.

Mappingd(α)d(\alpha)CLIP-Text Similarity ↑Temporal Consistency ↑Overall Consistency ↑Subject Consistency ↑Background Consistency ↑Motion Smoothness ↑Avg. Score ↑
H1/8H^{1/8}0.38934.33498.81123.44596.60296.93097.82474.658
H1/4H^{1/4}0.30034.46898.89023.58296.56697.03797.87874.737
H1/2H^{1/2}0.16734.62798.99523.86796.86397.12998.20374.947
H2H^{2}0.16734.66799.02523.85097.00997.28898.08574.987
H4H^{4}0.30034.54199.00723.75996.90097.12297.82674.859
H8H^{8}0.38934.55498.99223.42896.81997.15198.20174.858

Across the tested settings, performance improves as ss approaches 1, where the alternatives recover our linear weighting scheme. These results provide empirical justification for linear entropy-based weighting as a simple and effective design choice.

2. Comparison with fixed-weight fusion strategies.

We compare our method with five fixed-weight variants. The results are presented below, with a corresponding visualization provided in Figure 6 of the revised manuscript.

Selection strategyCLIP-TTemporal ConsistencyOverall ConsistencySubject ConsistencyBackground ConsistencyMotion Smoothness
Attention only (λ=0\lambda=0)33.40 ± 0.5898.70 ± 0.0422.92 ± 0.9696.27 ± 0.1496.53 ± 0.3698.04 ± 0.09
Fixed λ=0.25\lambda=0.2533.33 ± 0.11998.78 ± 0.2623.27 ± 1.3796.33 ± 0.5296.79 ± 0.2798.01 ± 0.08
Fixed λ=0.50\lambda=0.5033.18 ± 0.5098.79 ± 0.3523.21 ± 0.7596.30 ± 0.4896.69 ± 0.4398.04 ± 0.30
Fixed λ=0.75\lambda=0.7533.03 ± 0.5198.81 ± 0.0323.17 ± 0.6696.40 ± 0.2496.53 ± 0.1597.97 ± 0.09
Coverage only (λ=1\lambda=1)33.40 ± 1.1598.81 ± 0.1123.28 ± 0.5696.38 ± 0.2096.66 ± 0.3097.80 ± 0.12
Rényi-2 routing (ours)35.49 ± 0.4399.12 ± 0.1823.99 ± 0.3697.67 ± 0.1397.68 ± 0.2498.34 ± 0.27

Our adaptive routing method achieves the highest scores across all six metrics. Its gains over the best of the five fixed-weight mixtures for each metric range from 0.30 to 2.09 points. These results support the benefit of adaptive routing over the evaluated fixed-weight alternatives.

(continuing...)

3. Comparison with alternative concentration measures.

We compare our weighting based on normalized Rényi-2 entropy with alternatives based on normalized Shannon entropy and normalized attention variance. The results are presented below, with a corresponding visualization provided in Figure 7 of the revised manuscript.

Routing statisticCLIP-TTemporal ConsistencyOverall ConsistencySubject ConsistencyBackground ConsistencyMotion Smoothness
Normalized Shannon entropy34.5499.0023.0897.1697.4398.05
Normalized attention variance34.1198.9123.3797.3997.4297.93
Rényi-2 entropy (ours)35.4999.1223.9997.6797.6898.34

As observed, of the three measures of attention concentration, Rényi-2 entropy yields the highest scores across all six evaluation metrics, supporting its use for adaptive weighting.

[W2] Global mean pooling for representation coverage

Thank you for raising this question. To examine our choice of global mean pooling, we compare it with spatial pooling, which groups tokens by their spatial positions and averages tokens within each group. We evaluate five pooling granularities (P=1,2,4,8,16), where P=1 corresponds to global mean pooling.

  1. Consistent frame-similarity patterns across spatial pooling levels. As shown in Figure 10 (Appendix D, p. 18), the frame-level similarity matrices exhibit highly similar patterns across all five settings, supporting global mean pooling as a compact representation for coverage estimation.
  2. Comparable generation quality across spatial pooling levels. The generation results, presented below and detailed in Apendix D , differ from those of global mean pooling (P=1) by less than 0.005 points (less than 0.6%) across the reported metrics.
PPCLIP-Text Similarity ↑Temporal Consistency ↑Overall Consistency ↑Subject Consistency ↑Background Consistency ↑Motion Smoothness ↑Avg. Score ↑Δ\Delta Avg. Score (Compared with P=1P=1)
1 (default)0.351150.991430.237110.975400.974870.979950.75165-
20.342270.989260.227060.972180.970380.982220.74723−0.00442(−0.59%)
40.349270.990570.238640.974520.973960.985060.75200+0.00035(+0.05%)
80.347650.991330.230780.980450.976140.982870.75154−0.00012(−0.02%)
160.359210.986120.231960.980430.970280.982200.75170+0.00005(+0.01%)

These results support global mean pooling as a simple default that avoids tuning the spatial pooling granularity. We have added these experiments and clarified this design choice in Appendix D of the revised manuscript.

[W3 & Q2.2] Generalization to other generation architectures

To evaluate the generalization ability of our method, we compare Entropy Forcing with Deep Forcing on the Causal Forcing backbone under matched cache and generation settings. The full VBench-Long results are presented below. We have also added the experimental settings and results to Section 5.1 of the revised manuscript.

MethodDynamic Degree ↑Motion Smoothness ↑Overall Consistency ↑Imaging Quality ↑Aesthetic Quality ↑Subject Consistency ↑Background Consistency ↑Avg. ↑
30 seconds
Causal Forcing66.8397.5922.0272.4359.7696.1095.5672.90
+ Deep Forcing61.6796.9421.4170.8359.6795.5095.3271.62
+ Entropy Forcing (ours)65.3397.9321.8472.3960.4396.6895.9172.93
60 seconds
Causal Forcing69.7596.7420.6070.5656.2695.1594.9872.01
+ Deep Forcing62.8396.5819.5369.6157.4595.0295.1970.89
+ Entropy Forcing (ours)66.1497.7620.6271.9558.2696.3195.5472.37

Entropy Forcing improves all seven metrics over the matched Deep Forcing baseline at both durations. The average score improves by 1.31 and 1.48 points at 30 and 60 seconds, respectively. These results justify the effectiveness and generalizability of our approach across different generation architectures.

[Q2.1] Confidence intervals in the ablation results

Thank you for your suggestion. We now report standard deviations for the ablation results in Figure 6 of the revised manuscript.

[Q3] Selection overhead on larger candidate sets

Exhaustive selection evaluates (NB)\binom{N}{B} subsets per layer and cache update, where N is the number of candidates and B is the Memory retention budget. Once the cache is full, each update adds c frames to the selectable pool, giving N=B+c, while Sink and Recent frames remain protected. We measured the time required for one exhaustive Memory selection while varying N and fixing B=8. Each reported time is the median of five runs on an NVIDIA RTX 4090 and includes subset index construction.

Candidates NNSubsets (N8)\binom{N}{8}Selection time (ms)
990.973
10450.973
111650.973
124950.965
143,0030.969
1612,8700.967
1843,7580.982
20125,9701.111
22319,7701.560
24735,4712.629

As reproted, selection time increases with the number of candidate subsets. In our standard 4 Sink / 8 Memory / 4 Recent configuration, the search includes only the eight retained Memory frames and three incoming candidates. The candidate pool therefore remains fixed at N=11, corresponding to 165 subsets and 0.973 ms per selection. Longer videos increase the number of cache updates while leaving the search size per update unchanged.

It should be noticed that, larger candidate pools (bigger than 20) in typical autoregressive video generation is not a common setting. For example, Self Forcing and Causal Forcing use a default attention cap equivalent to 21 latent frames, which leads to a candidate set much less than 21 given that the sink frames and recent frames are protected from selection. These results support the practicality of exhaustive selection for our bounded Memory pool.

Reviewer_CCtZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free KV cache selection method for autoregressive video generation. It combines attention-based importance with value-space representation coverage, using normalized Rényi-2 attention entropy to adaptively balance the two criteria. The paper evaluates feasible subsets within a bounded candidate pool and reports improvements in long-video generation.

Critical strengths
  • The paper focuses on retaining useful historical information under a fixed KV cache budget, which is an important problem in autoregressive video generation.

  • Considering representation redundancy alongside attention-based relevance is a sensible design and the reported subset-selection overhead is small in the evaluated setting.

Critical weaknesses
  • The component-level evaluation is limited. The ablation uses only 20 prompts, evaluates performance at the 45th chunk, and reports modest numerical gains. It also omits the main VBench-Long metrics. Moreover, Table 2 compares the proposed method only against attention-only and coverage-only selection, without a fixed-weight combination. The results do not isolate the benefit of entropy-based adaptation from that of simply combining the two criteria.

  • The comparison with Deep Forcing may be confounded by differences in cache allocation. Section 5.2.2 states that the proposed method uses 4 sink, 4 recent, and 8 memory frames, whereas Deep Forcing uses 10, 4, and 2, respectively. Although the total budget is the same, the selectable memory capacity differs substantially.

  • Another concern is the technical novelty of this paper. The coverage objective is an established facility-location formulation, and the entropy-based fusion is a linear interpolation of two existing criteria. The distinctive contribution is applying this combination to KV cache selection for autoregressive video diffusion, rather than proposing fundamentally new optimization machinery.

Questions for authors
  • Were the reproduced Deep Forcing results in Table 1 obtained using the 10/4/2 allocation described in Section 5.2.2? Does the advantage of Entropy Forcing persist when both methods use the same sink/recent/memory allocation?

  • How does the proposed weighting based on Rényi-2 entropy compare with a fixed-weight combination or weighting based on other measures of attention-score concentration?

6 comments

Thank you for recognizing the importance of retaining useful historical information under a fixed KV cache budget. We appreciate your assessment that accounting for representation redundancy alongside attention-based relevance is a sensible design, and your recognition of the low measured overhead of subset selection.

[W1.1] Expanded ablation settings

Thank you for your constructive suggestion. We have expanded the ablation studies in three ways:

  1. Increased the number of prompts from 20 to 60.
  2. Extended the evaluation from a single chunk (chunk 45) to multiple chunks spanning 36–60.
  3. Added four metrics from the main VBench-Long evaluation (Overall Consistency, Subject Consistency, Background Consistency, and Motion Smoothness) alongside CLIP-T and Temporal Consistency.

We have updated Figure 6 in the revised manuscript to reflect these changes. The additional ablation studies use the same expanded evaluation settings.

[W1.2 & Q2.1] Comparison with fixed weight combination

We compare our method with five fixed-weight variants. The results are presented below, with a corresponding visualization provided in Figure 6 of the revised manuscript.

Selection strategyCLIP-TTemporal ConsistencyOverall ConsistencySubject ConsistencyBackground ConsistencyMotion Smoothness
Attention only (λ=0\lambda=0)33.40 ± 0.5898.70 ± 0.0422.92 ± 0.9696.27 ± 0.1496.53 ± 0.3698.04 ± 0.09
Fixed λ=0.25\lambda=0.2533.33 ± 0.11998.78 ± 0.2623.27 ± 1.3796.33 ± 0.5296.79 ± 0.2798.01 ± 0.08
Fixed λ=0.50\lambda=0.5033.18 ± 0.5098.79 ± 0.3523.21 ± 0.7596.30 ± 0.4896.69 ± 0.4398.04 ± 0.30
Fixed λ=0.75\lambda=0.7533.03 ± 0.5198.81 ± 0.0323.17 ± 0.6696.40 ± 0.2496.53 ± 0.1597.97 ± 0.09
Coverage only (λ=1\lambda=1)33.40 ± 1.1598.81 ± 0.1123.28 ± 0.5696.38 ± 0.2096.66 ± 0.3097.80 ± 0.12
Rényi-2 routing (ours)35.49 ± 0.4399.12 ± 0.1823.99 ± 0.3697.67 ± 0.1397.68 ± 0.2498.34 ± 0.27

Our adaptive routing method achieves the highest scores across all six metrics. Its gains over the best of the five fixed-weight mixtures for each metric range from 0.30 to 2.09 points. These results support the benefit of adaptive routing over the evaluated fixed-weight alternatives.

[W2 & Q1] Fairness of the comparison under matched cache allocations

First, we clarify that the Deep Forcing results in our main experiments (Table 1) were reproduced using the same cache allocation as Entropy Forcing (4 Sink, 8 Memory, and 4 Recent frames) for a fair comparison. We have made this shared allocation explicit in Section 5.1 and the caption of Table 1 in the revised manuscript.

To further evaluate our approach, we compare it with Deep Forcing using Deep Forcing’s original cache allocation of 10 Sink, 2 Memory, and 4 Recent frames for both methods. The results are presented below. We have added the experimental settings and results to Section 5.1 and panel (a) of Table 2 of the revised manuscript.

MethodDynamic Degree ↑Motion Smoothness ↑Overall Consistency ↑Imaging Quality ↑Aesthetic Quality ↑Subject Consistency ↑Background Consistency ↑Avg. ↑
30 seconds
+ Deep Forcing46.4898.4325.1068.1261.7797.4596.9370.61
+ Entropy Forcing (ours)53.4898.6725.2670.4362.0397.8197.3772.15
60 seconds
+ Deep Forcing43.1098.3524.4868.4360.4897.3296.3069.78
+ Entropy Forcing (ours)51.2498.1424.6769.3761.3197.4397.6171.40

The gains persist when both methods have only two selectable Memory frames. Entropy Forcing improves Dynamic Degree by 7.00 and 8.14 points and the average VBench-Long score by 1.54 and 1.62 points at 30 and 60 seconds, respectively. These results demonstrate consistent gains of our approach across the tested cache allocations.

[W3] Methodological Novelty Beyond Prior Work

We appreciate the reviewer’s feedback. The central novelty of our work does not lie in an incremental heuristic, but in identifying a fundamental failure mode of conventional KV cache selection and re-architecting cache management from heuristic entry-wise ranking to principled, entropy-routed subset-level optimization. Below, we clarify our methodological distinction from prior works, followed by our theoretical and empirical contributions.

1. Distinction from Prior Work.

Prior KV cache selection methods (e.g., H2O, SnapKV, Deep Forcing) operate under two implicit, unexamined assumptions: (1) attention scores are always reliable indicators of importance, and (2) candidate entries can be scored independently. We show both assumptions fail in long-video autoregressive diffusion, and we introduce a unified methodology connecting representation, routing, and optimization:

  • Objective: From independent entry scoring to joint subset coverage.
    • Prior Work: Traditional methods evaluate entries independently and retain the top-BB candidates. Under this additive objective (R(S)=∑m∈SpmR(S) = \sum_{m \in S} p_m), the marginal gain of adding a candidate is constant: Δ(m∣S)=pm\Delta(m \mid S) = p_m. It cannot distinguish between complementary historical context and redundant representations, allowing multiple visually identical frames to waste limited cache budget.
    • Entropy Forcing: We formulate cache retention as a coverage-aware facility location problem on cached value representations (Eq. 9). The marginal gain ΔC(m∣S)=∑j∈Mpjmax⁡(0,sjm−max⁡i∈Ssji)\Delta_C(m \mid S) = \sum_{j \in \mathcal{M}} p_j \max(0, s_{jm} - \max_{i \in S} s_{ji}) explicitly depends on what is already covered. If a visual state is already represented in SS, redundant candidates yield zero marginal utility, natively guaranteeing diverse representation coverage.
  • Selection Criterion: From static attention reliance to entropy-governed adaptive routing.
    • Prior Work: Prior methods unconditionally rely on attention rankings (or fixed heuristic weights), ignoring whether the attention distribution is actually discriminative.
    • Entropy Forcing: We discover that attention distribution flatness dictates the reliability of rank-based selection. We introduce the normalized Rényi-2 entropy as a continuous, parameter-free routing signal (Eq. 13) that dynamically balances attention relevance (1−H)(1 - H) and value coverage (H)(H). This allows the model to trust attention when it is sharp/peaked, and automatically retreat to coverage when attention is flat/ambiguous.
  • Optimization: From greedy/approximate eviction to exact parallel subset optimization.
    • Prior Work: Subset selection problems are typically avoided due to combinatorial complexity and replaced by heuristic greedy pruning.
    • Entropy Forcing: Taking advantage of the bounded memory candidate pool in streaming video diffusion (N=B+cN=B+c), we show that evaluating all feasible subsets ((113)=165\binom{11}{3} = 165 subsets under the standard 4/8/4 setting) can be executed in parallel with exact global optimality (Algorithm 1). Because pairwise similarities and attention scores are precomputed once, exact subset evaluation introduces virtually zero overhead—accounting for only 0.0150%–0.0155% of total generation time (Appendix E, Table 6).

2. Theoretical Foundation.

We establish a theoretical proof in Appendix B.1 explaining why flat attention fundamentally degrades rank-based Top-BB selection. By proving the exact geometric identity linking normalized Rényi-2 entropy HH to the ℓ2\ell_2 distance from uniformity (∣p−u∣22=N−H−1/N\\|p - u\\|_2^2 = N^{-H} - 1/N), we derive an analytical upper bound on the advantage of Top-BB over uniform random selection: 0≤R(Stop)−E[R(Srand)]≤B(1−BN)(N−H−1N)0 \le R(S_{\text{top}}) - \mathbb{E}[R(S_{\text{rand}})] \le \sqrt{B\left(1 - \frac{B}{N}\right)\left(N^{-H} - \frac{1}{N}\right)} This bound proves that as H→1H \to 1, the theoretical advantage of attention ranking strictly vanishes to zero, providing the first mathematical justification for entropy-based routing in KV cache management.

(continuing...)

3. Empirical Validation of the Core Mechanism.

  • Correlation verification: We empirically confirm the theoretical bound on actual generation states (Figure 5), observing a strong negative Pearson correlation (r=−0.80r = -0.80, p<0.001p < 0.001) between attention entropy and Top-BB advantage.
  • Regime-shift validation: In controlled matched-state experiments (Table 3), Coverage-only outperforms Attention-only by +0.68 dB PSNR under high entropy (H≥0.75H \ge 0.75), whereas Attention-only leads by 0.85 dB under lower entropy, directly validating the necessity of dynamic entropy routing.
  • Superior generation performance: On VBench-Long across 30s, 60s, and 120s generation, Entropy Forcing achieves the highest average scores and substantially improves Dynamic Degree (+7.00 to +8.14 points over Deep Forcing), effectively resolving the motion degradation and flickering common to long streaming generation.

[Q2.2] Comparison with weighting based on alternative measures of attention concentration

Thank you for your constructive suggestion. We compare our weighting based on normalized Rényi-2 entropy with alternatives based on normalized Shannon entropy and normalized attention variance. The results are presented below, with a corresponding visualization provided in Figure 7 of the revised manuscript.

Routing statisticCLIP-Text Similarity ↑Temporal Consistency ↑Overall Consistency ↑Subject Consistency ↑Background Consistency ↑Motion Smoothness ↑
Rényi-2 entropy35.4999.1223.9997.6797.6898.34
Shannon entropy34.5499.0023.0897.1697.4398.05
Normalized attention variance34.1198.9123.3797.3997.4297.93

As observed, of the three measures of attention concentration, Rényi-2 entropy yields the highest scores across all six evaluation metrics, supporting its use for adaptive weighting.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair for their time, and Reviewers CCtZ, Tvnb, and w87n for their constructive feedback. Below is a concise summary of the core strengths acknowledged by the reviewers and the comprehensive enhancements made in our revised manuscript.


1. Reviewer Recognition & Core Strengths

  • Well-Motivated & Timely Problem (CCtZ, Tvnb, w87n): Acknowledged as a critical challenge in autoregressive video generation (CCtZ), presented with a clear motivation and novel perspective (w87n). Reviewer Tvnb specifically highlighted our rigorous attention-score analysis and the strong negative correlation (r=−0.80r = -0.80) between attention entropy and selection effectiveness.
  • Intuitive & Training-Free Design (CCtZ, Tvnb, w87n): Praised as a sensible, simple, and effective joint formulation of redundancy and relevance (CCtZ, Tvnb, w87n), with intuitive entropy-guided routing requiring zero retraining (Tvnb, w87n).
  • Strong Empirical Gains with Minimal Overhead (CCtZ, Tvnb): Recognized for substantial improvements in motion dynamics (e.g., boosting Dynamic Degree on 30s videos) with negligible subset-selection overhead (<0.02%, CCtZ, Tvnb).

2. Summary of Major Revisions & Responses

(1) Elevating Methodological Novelty (Addressing CCtZ [W3])

We clarified our paradigm shift from heuristic entry-wise ranking to principled, entropy-routed subset-level optimization:

  • Joint Subset Coverage: Formulated via a coverage-aware facility location objective (Eq. 9) that penalizes visual redundancy, replacing additive independent scoring.
  • Theoretically Grounded Routing: Introduced normalized Rényi-2 entropy (HH, Eq. 13) to handle flat attention. We proved an analytical upper bound (Appendix B.1) showing attention-ranking advantage vanishes as H→1H \to 1, directly corroborated by empirical correlation (r=−0.80r = -0.80, Figure 5) and regime-shift validation (+0.68 dB PSNR under high entropy; +0.85 dB under low entropy).
  • Exact Parallel Optimization: Exploiting the bounded memory pool (N=B+cN=B+c), all 165 subsets are evaluated in parallel globally in 0.973 ms (only 0.015% of total generation time, Appendix E).

(2) Comprehensive Justification of Linear Routing (H,1−H)(H, 1-H) (Addressing Tvnb [W1, Q1])

To address concerns regarding the linear weighting form, we provided exhaustive empirical verification demonstrating that our adaptive routing is not an arbitrary heuristic:

  • Vs. Fixed-Weight Mixtures (Figure 6): Adaptive Rényi-2 routing outperforms Attention-only, Coverage-only, and 5 fixed-weight variants (λ∈{0.25,0.50,0.75}\lambda \in \{0.25, 0.50, 0.75\}) across all six metrics by +0.30 to +2.09 points.
  • Vs. Nonlinear Mappings (Appendix D.2): Testing HsH^s across six exponents (s∈{1/8,…,8}s \in \{1/8, \dots, 8\}) confirms performance peaks as s→1s \to 1, recovering our linear design.
  • Vs. Alternative Measures (Figure 7): Normalized Rényi-2 entropy consistently outperforms normalized Shannon entropy and attention variance.

(3) Rigorous Generalization, Fairness, and Scalability (Addressing CCtZ, Tvnb, w87n)

  • Fair & Matched Allocations (Section 5.1): Confirmed main results use matched budgets (4/8/4). Under Deep Forcing's original allocation (10/2/4 with only 2 selectable memory frames), Entropy Forcing still boosts Dynamic Degree by +7.00 (30s) and +8.14 (60s), and average VBench-Long score by +1.54 / +1.62.
  • Cross-Architecture & Long-Horizon Generalization (Section 5.1, Appendix G): Demonstrated seamless transfer to the Causal Forcing backbone (improving all 7 metrics at 30s and 60s) and scaling to 120-second videos (outperforming Deep Forcing on 6 of 7 metrics, with +3.50 Dynamic Degree). Transparent failure cases (appearance drift, camera views) were added to Appendix G.
  • Statistical Rigor & Spatial Pooling (Figure 6, Appendix D): Scaled ablations to 60 prompts (chunks 36–60) with standard deviations; proved spatial pooling granularities (P=1P=1 to 1616) alter results by <0.6%, validating global mean pooling (P=1P=1) as a robust, tuning-free default.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.