acceptodds
Under review as a conference paper at ICLR 2027

HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation

Abstract

Autoregressive (AR) video diffusion models have become a promising paradigm for long and streaming video synthesis, but the continuously growing Key-Value (KV) cache makes attention the dominant inference cost. This cost is governed by how many tokens sit in the cache, so it becomes the bottleneck precisely in the regime the field is moving toward: high resolution, where each frame contributes many tokens, and long horizons, where frames accumulate. Existing remedies either evict the cache with coarse heuristics that cause inter-frame flickering, or require model re-training. We show that the attention heads of a pre-trained AR video DiT possess a stable structural property: each head's spatio-temporal receptive field is reproducible from a small, fixed sub-context, and which sub-context suffices can be identified from a single forward pass. We formalize this as four archetypes – Sink, Dummy, Spatial, and Global – and build HeadCast, a training-free, plug-and-play framework that classifies every head once at the maximum-noise step and restructures the monolithic KV cache into head-specific pathways. Crucially, it retains the Global heads that preserve the long-range temporal consistency aggressive eviction destroys. Because the Spatial pathway operates on a fixed-size grid, its savings grow monotonically with the token count: across state-of-the-art AR models, end-to-end speedup rises from 1.02-1.08x at 480P to 1.62x at 720P and 1.95x at 1080P, entirely without training, while keeping VBench quality within 0.2 points of full attention and largely flicker-free. A single threshold set transfers unchanged across four backbones, three resolutions and both clip lengths (5 s and 30 s), indicating that the taxonomy reflects model-independent structure rather than per-checkpoint calibration.

Primary area:
generative models
Keywords:
autoregressive video generation, video diffusion model, efficient inference, attention head specialization, KV cache compression, sparse attention, training-free acceleration, long video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qrTVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HeadCast, a training-free method for accelerating autoregressive video diffusion through head-specific attention and KV-cache management. Its central observation is that attention heads exhibit relatively stable dependencies that can be categorized as Sink, Dummy, Spatial, or Global. After a warm-up, heads are classified once at the maximum-noise timestep by comparing restricted-context and full-context outputs. Each category then uses an appropriate cache and attention pathway, with spatial attention implemented through local grids batched using FlashAttention and Global heads retaining the baseline sliding-window context.

Critical strengths
  1. Identification of four attention-head types. The paper identifies Sink, Dummy, Spatial, and Global heads, revealing distinct spatiotemporal dependencies in pretrained autoregressive video models.
  2. Tailored processing for different head types. HeadCast assigns each type dedicated KV-cache management and attention operations, reducing computational cost while retaining the context needed for temporal consistency.
Critical weaknesses

Limited novelty relative to Dummy Forcing. Dummy Forcing already introduces head classification, heterogeneous KV-cache management, and packed attention execution. HeadCast primarily extends this framework with Spatial heads and output-similarity-based classification. While these additions are useful, the overall contribution appears incremental, and the paper does not sufficiently establish a substantial conceptual advance over the existing approach.

Questions for authors
  1. In the 30-second LongLive setting in Table 1, Dummy Forcing achieves both higher speedup and higher VBench scores than HeadCast. Could the authors provide the complete per-dimension VBench-Long results to explain these differences? Please also report the complete VBench/VBench-Long breakdowns for Table 2 to substantiate the quality-preservation claims.

  2. Could the authors clarify the technical novelty of HeadCast relative to Dummy Forcing, particularly regarding head classification and head-specific processing, and distinguish the new contributions from extensions of the existing framework?

7 comments
AuthorsOpenReview ↗

Response to Reviewer qrTV (1/4): Q1 - per-dimension VBench results

We thank the reviewer for the careful reading. We first give the requested per-dimension results, then separate what HeadCast inherits from Dummy Forcing from what is new.

Short answers

  • LongLive 480P: Dummy Forcing's Total is +0.22 above the full-attention model itself. This comes from Imaging Quality, while all four of its temporal-consistency dimensions are below full attention; HeadCast's −0.14 comes from Dynamic Degree (Q1). At 480P attention is a small share of the runtime, so both speedups are small (1.08× vs 1.16×).
  • Novelty: Dummy Forcing restricts every head along the temporal axis and fills its dummy class by a fixed budget; on 30 s rollouts this flickers, also with its released code (BBD 1.91 against 1.29 for full attention). HeadCast adds an uncompressed class (Global), a test instead of a budget, a second sparsity axis (Spatial) that brings the speed back, and an execution scheme for it (Q2).

Q1. Complete per-dimension VBench / VBench-Long results

Every cell is Full attention / Dummy Forcing / HeadCast (×100).

LongLive, 30 s, VBench-Long (480P: standard 16-dimension suite; 720P/1080P: MovieGen prompts in custom mode, which supports the eight dimensions listed)

Dimension480P (Table 1)720P (Table 2)1080P (Table 2)
Subject consistency98.27 / 98.22 / 98.4097.37 / 95.97 / 97.7196.71 / 95.10 / 97.27
Background consistency97.27 / 96.86 / 97.2297.42 / 96.63 / 97.6197.01 / 96.19 / 97.28
Temporal flickering99.47 / 99.12 / 99.3798.13 / 97.04 / 98.2598.29 / 97.48 / 98.42
Motion smoothness99.06 / 98.63 / 99.0198.95 / 98.45 / 98.9998.90 / 98.51 / 98.98
Dynamic degree35.59 / 34.81 / 29.8431.35 / 44.53 / 27.0914.71 / 33.67 / 13.87
Aesthetic quality63.84 / 64.57 / 63.8662.55 / 62.05 / 62.6259.41 / 59.23 / 60.06
Imaging quality65.97 / 69.65 / 66.8168.45 / 69.54 / 68.9466.69 / 68.49 / 67.21
Object class95.54 / 94.29 / 95.67––
Multiple objects85.02 / 85.85 / 86.88––
Human action95.55 / 95.50 / 96.00––
Color86.97 / 90.16 / 88.66––
Spatial relationship79.97 / 78.11 / 78.96––
Scene55.61 / 58.86 / 58.09––
Appearance style20.54 / 20.62 / 20.55––
Temporal style24.16 / 24.16 / 24.26––
Overall consistency26.46 / 26.29 / 26.5724.68 / 24.31 / 24.7323.80 / 23.40 / 24.04
Total82.26 / 82.48 / 82.1279.07 / 78.85 / 79.0576.84 / 77.18 / 77.26

Self-Forcing, 5 s, VBench (Table 2)

Dimension720P1080P
Subject consistency94.86 / 95.36 / 94.8893.82 / 94.43 / 93.90
Background consistency96.41 / 96.78 / 96.4896.39 / 96.76 / 96.44
Temporal flickering98.92 / 99.31 / 98.9798.92 / 99.25 / 98.93
Motion smoothness98.21 / 98.25 / 98.2198.59 / 98.61 / 98.60
Dynamic degree65.00 / 62.50 / 65.2860.00 / 53.89 / 60.83
Aesthetic quality66.46 / 66.53 / 66.4765.13 / 65.40 / 65.23
Imaging quality69.50 / 69.43 / 69.4867.98 / 67.39 / 67.94
Object class93.35 / 92.71 / 93.4394.16 / 94.02 / 94.35
Multiple objects88.49 / 88.69 / 88.4983.73 / 83.99 / 83.95
Human action96.40 / 95.80 / 96.2095.80 / 96.60 / 95.80
Color89.26 / 88.83 / 89.4488.49 / 88.39 / 88.93
Spatial relationship81.98 / 82.92 / 82.0080.64 / 80.01 / 80.52
Scene55.20 / 55.23 / 55.4450.55 / 50.86 / 50.51
Appearance style19.90 / 20.01 / 19.8919.82 / 19.81 / 19.82
Temporal style24.63 / 24.69 / 24.6224.09 / 24.00 / 24.08
Overall consistency26.75 / 26.81 / 26.7326.55 / 26.55 / 26.54
Total83.78 / 83.91 / 83.8382.80 / 82.66 / 82.90
AuthorsOpenReview ↗

Response to Reviewer qrTV (2/4): Q1 - what the breakdown explains

Q1 (continued). What the breakdown explains

  1. LongLive 480P (the row the reviewer asks about). The three Totals lie within 0.36 points, for different reasons. Dummy Forcing scores +0.22 above the full-attention model it approximates: with VBench's own normalisation and weights, this is +0.45 from Imaging Quality (65.97 → 69.65), −0.37 from its four temporal-consistency dimensions, which are all below full attention, and +0.14 from the rest. HeadCast's −0.14 is −0.35 from Dynamic Degree (flow magnitude 0.92× that of full attention; see our reply to Reviewer 1st9) and +0.21 from all other dimensions; its temporal-consistency dimensions are within 0.15 of full attention. A Total above the full-attention reference is therefore not evidence that the reference is better preserved: on the 30 s clips of App. G, BBD is 1.78 for Dummy Forcing against 1.30 for full attention and 1.26 for HeadCast.
  2. LongLive 720P / 1080P. Dummy Forcing is below full attention on all four temporal-consistency dimensions (−0.91 / −0.80 Total points). Its Total is held up by Dynamic Degree rising far above the full model (31.4 → 44.5, 14.7 → 33.7; +0.81 / +1.17 Total points). This extra "motion" is block-boundary jumps and drift: on the 720P videos its optical-flow magnitude is 1.54× that of full attention (95 % CI 1.38–1.72), and 1.87× on frame pairs that straddle a block boundary. HeadCast is at or above full attention on all four temporal-consistency dimensions, with Total 79.05 vs 79.07 and 77.26 vs 76.84.
  3. Self-Forcing 5 s. HeadCast is within 0.3 points of full attention on every dimension at 720P, and within 0.45 at 1080P except Dynamic Degree (+0.83). Dummy Forcing loses Dynamic Degree (65.00 → 62.50, 60.00 → 53.89): its clips become more static, which raises its flickering / smoothness scores.
  4. Speed. Dummy Forcing is the faster method on every LongLive row (1.16× / 1.74× / 2.16× against 1.08× / 1.62× / 1.95×), at lower fidelity to the full model: at 480P its PSNR is 13.6 dB against 16.3 dB for HeadCast (Table 1, corrected in the revision; see the general response). At matched throughput on Self-Forcing 720P (Fig. 4: 9.41 vs 9.60 FPS) HeadCast keeps 25.32 dB against 18.81 dB.
AuthorsOpenReview ↗

Response to Reviewer qrTV (3/4): Q2 - what is new relative to Dummy Forcing

Q2. What is new relative to Dummy Forcing

HeadCast builds on Dummy Forcing, and Sec. 3.2 says so. Two things are inherited: the observation that some heads are temporally sparse (our Sink and Dummy types), and serving those heads with one fused kernel call. The rest came out of asking why Dummy Forcing breaks on long rollouts.

(1) The failure and its two causes. Dummy Forcing restricts every head. In the released implementation sink heads read {first frame, current block}, neighbor heads {previous block, current block} and dummy heads {last frame, current block}; half of the heads are made dummy by a fixed budget, ranked by attention mass. On 30 s rollouts the output flickers at block boundaries (Fig. 3, App. G), and 90.9 % of the user-study votes prefer HeadCast (Table 3). In the paper Dummy Forcing runs inside our pipeline. As a cross-check we ran its released code with its released LongLive configuration on the 40 prompts of App. G (30 s, 480P) and then changed one thing at a time (App. J.7).

Released Dummy Forcing code. Reference: the same code without head programming (BBD 1.29; 2 of 40 videos above 1.5).

VariantHeads restrictedBBD − Full (95 % CI)Videos with BBD > 1.5
Released configurationall 360 (≈120 sink, ≈60 neighbor, 180 dummy)+0.62 [+0.49, +0.76]33 of 40
Neighbor heads read the 9 most recent framessink + dummy (≈300)+0.50 [+0.39, +0.62]30
Only its sink heads restricted≈120+0.20 [+0.12, +0.28]16
Only its dummy heads restricted180−0.07 [−0.14, +0.00]3

(A group that is not restricted reads the 9 most recent frames. At 720P the released configuration gives +0.28 [+0.18, +0.39], with 20 of 40 videos above 1.5.)

Our code. The N best candidates become Dummy heads; all other heads keep the full window (Global).

VariantPSNR (dB)LPIPSPSNR − HeadCast (95 % CI)
N = 9016.020.276−0.59 [−1.10, −0.11]
N = 18012.940.392−3.67 [−4.63, −2.76]
N = 27012.060.429−4.55 [−5.68, −3.49]
HeadCast (94 Dummy heads, by its test)16.610.263–

(Fidelity to the full-attention output of the same code on the 40 prompts of App. G. These are new generations with the supplementary code, so the HeadCast row differs slightly from Table 1, which uses the videos of App. G: 16.61 against 16.30 dB.)

  • Restricting the dummy heads alone does not raise BBD; restricting the sink heads does (+0.20). On average only 2 of these ≈120 sink heads reproduce their output from {first frame, current block}. The increase appears once heads that depend on the history lose it. The released design has no class for such heads; in HeadCast a head that fails its test keeps the window (Global).
  • A budget instead of a test costs fidelity. On these prompts HeadCast's test makes only 94 ± 11 of the 360 heads Dummy heads. Ranked by the cosine between the output from the current block alone and the full-window output, the 90th head is at 0.96 and the 180th at 0.82, so a budget of 180 includes about 80 heads that use the history: N = 180 is 3.7 dB below HeadCast, N = 90 only 0.6 dB.
Reviewer_1st9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces HeadCast, a training-free framework that reduces KV-cache attention costs in autoregressive video diffusion. A single forward pass at maximum noise classifies heads into four types: Sink, Dummy, Spatial, and Global. Each assigned a dedicated cache and attention path. Global heads retain their full cache to prevent eviction-induced flickering. Across four backbones, HeadCast achieves up to 1.95× end-to-end speedup at 1080p while remaining within 0.2 VBench points of full attention.

Critical strengths
  1. Paper is easy to read and follow.

  2. The idea is interesting with high potential for efficient autoregressive video diffusion generation.

  3. The approach is training-free, plug-and-play, realized with hardware-efficient FlashAttention batching so nominal sparsity becomes real wall-clock speedup.

Critical weaknesses
  1. Limited baseline coverage. Dummy Forcing is the only direct comparator. Although exclusions are individually justified, relevant training-free methods such as FAST-AR, FpcusedForcing, SCOPE and more are neither cited nor evaluated.

  2. Limited quality evidence. PSNR/LPIPS measure fidelity to full-attention outputs rather than quality, while VBench reportedly misses flickering. The user study therefore carries much of the perceptual evidence, with only 39.6% preferring HeadCast over full attention.

  3. Generalization and stability concerns. All backbones are Wan-derived, limiting the threshold-transfer claim. The impact of 79.8% cross-prompt classification stability remains unclear, and LongLive’s lower Dynamic Degree (29.8 vs. 35.6) suggests possible motion suppression. A limitations section is also missing.

Questions for authors

My main concerns are limited baseline coverage and generalization across backbones.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1st9 (1/4): W1 - baseline coverage

e thank the reviewer for the constructive assessment.

Short answers

  • Baselines: FAST-AR and Focused Forcing have released no code and cannot be run; SCOPE is public but accelerates 50- and 64-step samplers, whereas our backbones take 4 steps. All three are now discussed in Sec. 2; they make a different unit sparse and are complementary to head-level routing.
  • Generalization: two backbones are added. Krea Realtime 14B (Wan family, 1,600 heads): the complete rule is applied unchanged and finds all four types; we report classification and fidelity. MiniMax-H3 33B (not Wan, 2,800 heads): both thresholds are unchanged, with the scale-dependent MSE term of the Spatial score omitted; 30 % of the heads are Dummy and 18 % Spatial, and the labels are as consistent across clips as on Wan. For H3 we report the classification only.
  • Motion: the reviewer's reading of Dynamic Degree is right for LongLive's 30 s rollouts: the flow magnitude is 0.92× / 0.86× that of full attention at 480P / 720P. We now state this as a limitation.
  • Quality: we add paired, reference-free flicker statistics and the per-dimension VBench results.

W1. Baseline coverage (FAST-AR, Focused Forcing, SCOPE)

We agree these works belong in the paper; the revision discusses all three (Sec. 2.3, App. J.8). They could not serve as baselines, for two reasons.

Two of the three cannot be run. Neither FAST-AR nor Focused Forcing has released code. FAST-AR's project page and its ICML page link no implementation, and the repository named in the Focused Forcing paper holds a single commit (late October, after the ICLR deadline) with a one-line README. A comparison would rest on our own re-implementation, with no way to check it against the reported numbers. The submission applies the same criterion to Forcing-KV (Sec. 2.3 and 5.1).

All three act on a different unit, and SCOPE and FAST-AR in a different regime.

MethodUnit made sparseModels / regime in its paperVenueCode
SCOPEdenoising step (cache / predict / recompute)SkyReels-V2 (50 steps), MAGI-1 (64 steps)ACM MM 2026public
FAST-ARtoken (merge near-duplicate keys; ANN-selected keys per query)Rolling-Forcing, LongVie2; 3000-frame streams, unbounded cacheICML 2026none released
Focused Forcingframe (history frames per generated frame; per-head budgets from an offline pass)Self-Forcing family, 832×480NeurIPS 2026none released (repository holds only a README)
Dummy Forcinghead, temporal onlySelf-Forcing, LongLive, 480P–1080P–public
HeadCasthead, temporal and spatial; Global heads untouchedfour backbones, 480P–1080P, 5–30 s–supplementary
  • SCOPE skips or extrapolates denoising steps and evaluates only the frames still being denoised; it does not touch attention or the KV cache, so it is complementary to HeadCast. Our backbones are 4-step distilled, where step-level redundancy is largely gone: the Dummy Forcing paper already tests the step-skipping method TeaCache on Self-Forcing and LongLive and finds a smaller speedup than its own, with a VBench drop. We have not run SCOPE and HeadCast together.
  • FAST-AR targets an unbounded cache: its experiments "generate streams of 3000 frames without imposing a context-window bound on the KV cache for any method", and its 5–10× is measured against dense attention over that growing cache. Our backbones are used as released, with a bounded window (4 or 7 blocks plus the sink), where near-duplicate keys accumulated over thousands of frames are mostly absent.
  • Focused Forcing is the closest, and the one that shares our backbones. Besides having no code, its per-head budgets come from an offline pass that needs the real and fake score networks of the distillation. Like Dummy Forcing it acts on the temporal axis only. HeadCast adds the Spatial type and the uncompressed Global type, and classifies online from the generator alone. Its user study independently reports the problem that motivates our work: in a four-way choice among accelerated methods (160 votes), Dummy Forcing receives 4.4 % (MonarchRT 18.1 %, TaylorSeer 20.0 %, Focused Forcing 57.5 %).

Dummy Forcing is the runnable baseline that shares our setting (training-free, head-level, bounded window, the same backbones). Its paper compares against step skipping (TeaCache) and token-level KV pruning (R-KV, InfiniPot-V) on Self-Forcing and LongLive and outperforms them.

AuthorsOpenReview ↗

Response to Reviewer 1st9 (2/4): W2 - quality evidence

W2. Quality evidence

  1. Fidelity to the unaccelerated model is the standard primary metric for training-free acceleration. The works above use it as well (FAST-AR: "fidelity to the dense-attention baseline using PSNR, SSIM, and LPIPS … under matched prompts and random seeds"; SCOPE: LPIPS / SSIM / PSNR "against the Original output as reference").
  2. A reference-free temporal measurement is in the paper (App. G, Table 8): the block-boundary discontinuity (BBD). We add paired statistics over the same prompts (30 s LongLive clips; difference to full attention, 95 % bootstrap CI):
ResolutionHeadCast − FullDummy F. − FullVideos where Dummy F. > FullVideos with BBD > 1.5 (Full / HeadCast / Dummy F.)
480P−0.044 [−0.075, −0.010]+0.472 [+0.400, +0.543]95 %3 / 4 / 33 of 40
720P−0.003 [−0.036, +0.032]+0.290 [+0.221, +0.361]90 %3 / 3 / 19 of 40
1080P−0.061 [−0.106, −0.018]+0.175 [+0.119, +0.232]89 %4 / 2 / 7 of 36

(1080P: the 36 prompts for which the videos of all three methods are available.) The released Dummy Forcing code and configuration give the same picture: +0.62 [+0.49, +0.76] at 480P and +0.28 [+0.18, +0.39] at 720P (reply to Reviewer qrTV, Q2). 3. VBench registers the flicker per dimension; the aggregate hides it. On LongLive 30 s at 720P / 1080P, Dummy Forcing is below full attention on all four temporal-consistency dimensions (subject consistency −1.4 / −1.6, temporal flickering −1.1 / −0.8), while HeadCast is at or above full attention on all four (full tables in our reply to Reviewer qrTV). 4. Reading the 39.6 %. The study is a two-alternative forced choice against the unaccelerated model with no tie option, so parity would be 50 %. HeadCast receives 39.6 %: raters prefer the full model more often than not, and we do not claim perceptual parity (Sec. 5.4). In the same protocol it receives 90.9 % against Dummy Forcing.

AuthorsOpenReview ↗

Response to Reviewer 1st9 (3/4): W3 - other backbones; cross-prompt stability

W3. Generalization to other backbones

The reviewer is right that the four backbones are distillations of the same Wan2.1-1.3B model. We therefore ran the classifier, with the thresholds unchanged, on a model of the same family at ten times the size and on one with a different architecture.

Table B. Head archetypes on the added backbones (mean ± std; θ = 0.95, θ_sc = 0.755). "Majority label": share of (head, clip) pairs in which a head receives its majority type; 87–89 % on the 1.3B models.

BackboneHeadsSinkDummySpatialGlobalMajority label
Krea Realtime 14B (Wan2.1-14B family; 20 prompts)1,60016 ± 10374 ± 41 (23 %)231 ± 83 (14 %)980 ± 5789.2 %
MiniMax-H3 33B (SolarWM; not Wan; 20 clips) (†)2,80019 ± 5837 ± 70 (30 %)507 ± 141 (18 %)1438 ± 10788.4 %
  • Krea Realtime 14B (40 layers × 40 heads; 480P, 10 s), with the complete rule of the paper unchanged: all four types are found. PSNR to full attention is 27.8 dB over the first 5 s (the protocol of Sec. 5.1) and 18.9 dB (LPIPS 0.137) over 10 s as the two rollouts drift apart; BBD stays at the level of full attention (1.290 vs 1.265). We report no speed.
  • MiniMax-H3 33B. SolarWM (arXiv:2609.02886) releases a 4-step autoregressive world model on the MiniMax-H3 omni transformer, which is not derived from Wan: 50 layers × 56 heads, one self-attention over a joint sequence of caption, first-frame anchor, audio and video tokens, its own VAE and a Qwen-based text encoder; it caches the KV of the five preceding chunks. We map the proxies of Sec. 4.2 onto its key sequence (Sink: the conditioning prefix; Dummy: the current chunk; Spatial: the query's own cell of a non-overlapping grid, as in Sec. 4.4, in every cached chunk; Global: all keys) and classify once, at the first chunk with a full window and the first denoising step, on 20 clips of the public SolarWM test set.
    • (†) The Spatial test on H3. Both thresholds are unchanged (mean cosine ≥ 0.95 and 5th-percentile cosine ≥ 0.755), but the MSE penalty of Eq. 3 is omitted: it is measured in activation units, and H3's attention outputs are about 170× larger in mean square than Wan's (applied as is, it leaves 2 Spatial heads). This test is looser than the complete rule — on Self-Forcing the omitted term rejects about a quarter of such heads — so the Spatial count on H3 is not directly comparable with those on Wan.
    • With this test all four types are found. 30 % of the heads reproduce their output from the current chunk alone (727–993 of 2,800 over the clips), a share in the range of the Wan models, and a further 18 % from the query's own grid cell; 49 % pass one of the three tests. The labels are as consistent across inputs as on Wan (88.4 % against 87–89 %), both types are spread over the depth, and the nestedness over denoising steps (reply to Reviewer NrtA, W5) holds: 94.8 / 94.4 / 94.7 % against 81.9 / 74.1 / 66.8 %. Re-classifying two and four chunks later reproduces 89.6 % and 87.2 % of the labels.
    • Scope. The experiment shows that temporally and spatially sparse heads, as measured by our tests, are not specific to the Wan family. We have no optimised implementation for H3 and claim neither quality nor speed for it; the Limitations say so.

W3 (continued). Stability: the 79.8 % cross-prompt figure

The 79.8 % is the agreement between the classifications of two different prompts (App. B). HeadCast never transfers a classification across videos, so it is not an error rate of the method. Doing the transfer on purpose is Table A in our reply to Reviewer NrtA: a table voted over 40 other prompts is on par on 5 s clips (+0.07 dB, CI [−0.39, +0.49]), so the archetypes are largely a property of the model, but it lowers the motion of 30 s rollouts from 0.96× to 0.85× that of full attention. Within a video the labels are stable: 92–94 % hold over the next three AR steps, and 88–90 % up to 34 blocks later in a 30 s rollout.

Reviewer_NrtAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes to divide the attention heads into different types, each of which see different ranges of tokens: anchor heads for tokens from the first frame, dummy heads for tokens from the most recent block, spatial heads for tokens from a spatial neighborhood and global heads for all tokens. This is classified at the first step of denoising. With reduced token numbers on certain heads, it brings speedups in autoregressive video generation.

Overall, it is essentially a kind of sparse attention for different attention heads.

Critical strengths

1, It shows the attention head-level locality and its one-shot-identifiable property in autoregressive video generation.

2, It proposes a training-free framework to classify the KV caches and deal with them with different heads. This brings 1.95x speedup on 1080p videos.

Critical weaknesses

1, Was Fig. 1 computed on real videos or just an illustration? It looks so flawless. Can we observe the same patterns for most videos? Does it work for most layers?

2, As claimed at L196, if most heads keep their archetype across AR steps, which means it is not related to the video contents, why can't we use the same head classification for all videos?

3, The observations in Sec. 3.2 have no statistical supports. We don't know whether it comes from one video or from a huge dataset.

3, I think fig.2 does not show your key idea well. Why do both spatial heads and global heads see 4 caches? Besides, putting the legend of "cache of past frames" on top-left makes people confused. It looks like different colors are for different frames/chunks.

4, The thresholds in Eq. 4 are hand-crafted.

5, From Table 5, since selecting different timesteps for classification has a significant impact, why can we use the same classification for different time steps?

6, How did you implement it? After each attention step, we guess we need to accumulate the results from different heads. The overall runtime is still dependent on the slowest head (the full attention).

7, It still keeps the full KV cache, which has no memory benefits.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NrtA (1/3): W1, W3 - Fig. 1 and the statistics behind the observations

We thank the reviewer for the detailed questions. Some are answered by material in the appendix that should have been in the main text; for the others we ran new experiments.

Short answers

  • Fig. 1 is measured on real generations; we add statistics over 280 videos (W1, W3).
  • We ran the suggested fixed-classification experiment: a table voted over 40 prompts comes close on average but loses motion on 30 s rollouts, which is why we classify online (W2). The first-step labels stay valid at the later denoising steps (W5).
  • Two factual points: HeadCast does not keep the full KV cache (it is about 33 % smaller on Self-Forcing, App. E; W7), and its runtime is not bounded by the full-attention heads, because attention cost adds up over heads (Table 4; W6).

W1 and W3. Is Fig. 1 real? Most videos, most layers? Statistical support for Sec. 3.2

Fig. 1 is measured, not drawn. Each panel is the attention of one specific head recorded during a real generation (Self-Forcing, 480P; the 4,680 query tokens of the current block against 14,040 cached key tokens). Panels (a)–(c) look clean because they show the aggregated views of Sec. 3.2 (attention mass per frame and per spatial position); App. A (Figs. 5–6) shows the raw maps of other heads at two AR steps.

Population statistics. The classifier is itself a measurement over all 360 heads: it tests whether a head's output is reproduced (cosine ≥ 0.95) when its keys are restricted to the context of an archetype. We ran it on 40 prompts for each of seven settings (Self-Forcing, Causal Forcing and Reward Forcing at 480P; Self-Forcing at 720P; LongLive at 480P / 720P / 1080P), i.e. on 280 videos.

  • Counts. Dummy heads: 73–143 per video; Sink heads: 0–51; Spatial heads: at least 20 in every video; the rest are Global. No type is a property of a few prompts.
  • Most videos. In every setting a head receives its majority type in 87–89 % of the videos. Two arbitrary prompts agree on 82–83 % of the heads, in line with the 79.8 % of App. B.
  • Most layers. On Self-Forcing 720P the first / middle / last ten layers hold 35 / 31 / 34 % of the Dummy heads, 28 / 33 / 39 % of the Spatial heads and 38 / 34 / 28 % of the Global heads.

The revision states the sample sizes and these statistics in Sec. 3.2 and App. J.1.

AuthorsOpenReview ↗

Response to Reviewer NrtA (2/3): W2 - fixed vs. online classification; Fig. 2

W2. If archetypes are stable, why not one classification for all videos?

The stability in L196 is within one video: 90.3 % of the heads keep their label between two AR steps. We re-measured it on 40 prompts: 92–94 % over the next three AR steps (Self-Forcing) and 88–90 % at 7 to 34 blocks after the classification in a 30 s LongLive rollout, with no downward trend. Across prompts the agreement is lower (79.8 %, App. B).

We ran the experiment the reviewer suggests: a fixed table built by per-head majority vote over 40 calibration prompts and applied to 40 held-out prompts, and, as a second variant, the classification of a single calibration video reused for all videos (three different videos).

Table A. Fixed vs online classification on 40 held-out prompts (same prompts and initial noise in all rows). ΔPSNR / ΔLPIPS: fidelity to full attention, as paired difference to the online classification (95 % bootstrap CI). BBD: block-boundary discontinuity (App. G). Motion: optical-flow magnitude relative to full attention. Last column: heads per video that the table restricts although their own test fails on that video.

SettingClassificationΔPSNR (dB)ΔLPIPSBBD − FullMotion / FullRestricted against own test
Self-Forcing 5 s, 720POnline (ours)ref.ref.+0.0020.960
Fixed, majority of 40 prompts+0.07 [−0.39, +0.49]+0.003 [−0.002, +0.007]−0.0030.9624.6
Fixed, single video (3 tables)−0.77 / −0.49 / +0.30+0.011 / +0.007 / +0.001+0.004 / +0.004 / −0.0010.92 / 0.95 / 0.9633.0 / 29.6 / 22.4
LongLive 30 s, 480POnline (ours)ref.ref.−0.0430.960
Fixed, majority of 40 prompts+0.44 [+0.20, +0.71]−0.015 [−0.025, −0.006]−0.0180.8522.1
Fixed, single video (3 tables)+0.02 / +0.44 / +0.33−0.008 / −0.012 / −0.010−0.017 / −0.040 / +0.0020.93 / 0.86 / 0.7421.4 / 34.8 / 30.4
  • A fixed table comes close on average. With the majority table, 5 s clips do not differ significantly from the online classification (+0.07 dB, CI [−0.39, +0.49]). On 30 s rollouts it is slightly closer to full attention (+0.44 dB, CI [+0.20, +0.71]), and its BBD stays below that of full attention (−0.018; online −0.043). This supports Sec. 3.2: archetypes are largely a property of the model.
  • The transfer is not free. Every fixed table restricts 21–35 heads per video whose own test fails on that video. (a) On 30 s rollouts all four fixed tables move less than the online classification: 0.74–0.93× the flow magnitude of full attention against 0.96× (majority table vs online: ratio 0.89, CI [0.82, 0.97], p = 0.004). (b) A table taken from a single video is unreliable on individual videos. On 5 s clips the three tables give −0.77 to +0.30 dB. On 30 s rollouts their means are not lower (+0.02 / +0.44 / +0.33 dB), but per video they differ from the online classification by −4.4 to +4.7 dB (majority table: −1.1 to +3.4 dB). Inspecting the videos, we found a rollout in which such a table makes a second animal appear after about 9 s that the full model never generates; it stays until the end (last 10 s: 17.4 against 20.9 dB).

We therefore keep the online classification: it stays closest to the motion of the full model, restricts no head that fails its own test on the video being generated, needs no calibration set for a new backbone, resolution or window length, and costs under 1.5 % of a 30 s generation (Table 7). A majority-voted table is a reasonable option for short clips. (Summary in Sec. 5.5, full results in App. J.2.)

W3 (second). Fig. 2

Each row is a head and each cylinder the cached KV of one past block; colour encodes the head type, not the frame. Spatial and Global heads both keep all blocks by design: a Spatial head is sparse in space, not in time (Observation 2). The difference lies inside each block: a Spatial head reads only the keys of its own grid cell (the half-filled cylinders; App. C, Fig. 9), a Global head the whole block. We agree this was too subtle; the revised caption states how to read the diagram.

AuthorsOpenReview ↗

Response to Reviewer NrtA (3/3): W4-W7 - thresholds, denoising steps, implementation, memory

W4. Hand-crafted thresholds

  • θ is a tolerance with a direct meaning: a head is restricted only if the restricted context reproduces its output with cosine ≥ 0.95. Dummy Forcing instead fixes a budget (180 of the 360 heads are dummy), independent of the video. Such a budget is costly: making the 180 best candidates Dummy heads lowers PSNR on 30 s rollouts by 3.7 dB relative to HeadCast, whose test selects 94 on the same videos (reply to Reviewer qrTV, Q2).
  • The same two values (θ, θ_sc) are used for every backbone, resolution and clip length. This includes Krea Realtime 14B, added in this rebuttal with the complete rule unchanged (23 % Dummy heads, against 20–40 % on the 1.3B models). On the non-Wan MiniMax-H3 33B both values are unchanged as well, with one term of the Spatial score omitted because it is not scale-free (30 % Dummy heads; Table B in our reply to Reviewer 1st9).
  • Sensitivity (App. B, Fig. 7): both sweeps are smooth and monotone. Moving θ from 0.91 to 0.99 changes PSNR from 24.51 to 26.64 dB and FPS from 9.51 to 7.24; θ_sc over its range changes ±0.52 FPS and ∓0.79 dB. The threshold is a speed–fidelity dial (Fig. 4).

W5. Table 5 shows the timestep matters — why reuse one classification for all timesteps?

Table 5 varies the step at which the single classification is taken: the lower the noise, the more heads look restrictable (Dummy 136 → 191, Global 110 → 66). The restricted sets are approximately nested. We verified this on 40 prompts (Self-Forcing 720P) by re-running the test at each denoising step of the same video and AR step:

Step t′Heads restricted at t = 1000 that still pass their test at t′Heads restricted by a classification at t′ that pass the same test at t = 1000
75097.7 %84.7 %
50097.5 %76.6 %
25098.0 %68.0 %

(LongLive 480P: 97.2 / 96.3 / 96.5 % against 82.0 / 74.2 / 67.5 %.) The 2–3 % that fail later are about 5 heads per video. Conversely, of the heads restricted by a classification at t = 750 / 500 / 250, 30 / 45 / 59 per video are Global at t = 1000 (the other failing heads pass the test of another restricted type). So t = 1000 is the conservative choice whose labels hold at all four steps, while a later step over-restricts the high-noise steps, which is the fidelity loss in Table 5 (25.87 → 24.52 dB). A per-timestep assignment is not available in a KV-cached model: the cache is written once per block and read by all denoising steps, so its head-wise layout must be shared. (Summary in Sec. 5.5, full results in App. J.3.)

W6. Implementation; is the runtime bounded by the slowest (full-attention) heads?

No. Heads are not parallel workers whose latency is the maximum: the cost of an attention layer is the sum over heads of (queries × keys). The query tensor is sliced along the head dimension into three groups, each served by one FlashAttention call over its own cache: (i) Sink and Dummy heads fused (keys of two blocks), (ii) Spatial heads (cells stacked on the batch dimension), (iii) Global heads (full window). The outputs are written back with one indexed copy (Sec. 4.4; code in the supplementary material). The Global call costs N_global / N of the original attention, and the other two are far cheaper per head (75 % fewer keys for Sink / Dummy heads, about 94 % fewer for Spatial heads at 480P; App. D). The savings add up: Spatial only 1.19×, Sink + Dummy only 1.04×, both 1.31× (Table 4), and up to 1.95× end-to-end at 1080P (Table 2).

W7. "It still keeps the full KV cache, which has no memory benefits"

It does not. After classification the monolithic cache is split into three head-specific buffers and freed (Sec. 4.3): Sink and Dummy heads store two blocks instead of the window (−75 % for those heads); only Spatial and Global heads keep the window. The measured steady-state KV cache is about 33 % smaller than with full attention on Self-Forcing (App. E, Fig. 10). The revised Sec. 4.4 points to App. D–E from the main text.

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank the reviewers for their concrete questions. Each reviewer receives an individual reply; this note lists what is new. Unless noted, new experiments use the supplementary code with the paper's thresholds (θ = 0.95, θ_sc = 0.755) on 40 prompts per setting.

  1. Statistics behind the observations (NrtA): seven backbone / resolution settings × 40 prompts; a head keeps its majority archetype in 87–89 % of the videos.
  2. Requested experiments (NrtA, 1st9): a fixed classification table against the online classification, and the validity of the first-step labels at the later denoising steps.
  3. Other backbones (1st9): Krea Realtime 14B (Wan family, 1,600 heads; complete rule unchanged) and the non-Wan MiniMax-H3 33B (2,800 heads; both thresholds unchanged, the MSE term of the Spatial score omitted). 23 % and 30 % of their heads are Dummy heads; on H3 a further 18 % are Spatial heads. For H3 we report the classification only.
  4. Quality evidence (1st9, qrTV): per-dimension VBench tables, paired flicker statistics on 30 s videos, and an optical-flow analysis of Dynamic Degree.
  5. Positioning (1st9, qrTV): SCOPE, FAST-AR and Focused Forcing are discussed (the latter two have released no code); what HeadCast inherits from Dummy Forcing and what is new; controlled experiments, also with the released Dummy Forcing code, on why it flickers on long rollouts.

Revised PDF. New or rewritten text is in blue: Sec. 2.3, the captions of Fig. 1–2, one paragraph each in Sec. 3.2, 4.4 and 5.5, a Limitations paragraph (Sec. 6) and a new Appendix J. The numbering is unchanged. We also corrected an error that we found while preparing this response (Table 1, LongLive 480P): PSNR / LPIPS had been computed on videos that did not share the initial noise with the reference. On the 40 same-noise prompts of App. G they are 16.30 dB / 0.2608 for HeadCast and 13.61 dB / 0.3750 for Dummy Forcing (previously 11.02 / 0.5779 and 10.27 / 0.6161); the ordering is unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.