- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HeadCast, a training-free method for accelerating autoregressive video diffusion through head-specific attention and KV-cache management. Its central observation is that attention heads exhibit relatively stable dependencies that can be categorized as Sink, Dummy, Spatial, or Global. After a warm-up, heads are classified once at the maximum-noise timestep by comparing restricted-context and full-context outputs. Each category then uses an appropriate cache and attention pathway, with spatial attention implemented through local grids batched using FlashAttention and Global heads retaining the baseline sliding-window context.
Critical strengths
- Identification of four attention-head types. The paper identifies Sink, Dummy, Spatial, and Global heads, revealing distinct spatiotemporal dependencies in pretrained autoregressive video models.
- Tailored processing for different head types. HeadCast assigns each type dedicated KV-cache management and attention operations, reducing computational cost while retaining the context needed for temporal consistency.
Critical weaknesses
Limited novelty relative to Dummy Forcing. Dummy Forcing already introduces head classification, heterogeneous KV-cache management, and packed attention execution. HeadCast primarily extends this framework with Spatial heads and output-similarity-based classification. While these additions are useful, the overall contribution appears incremental, and the paper does not sufficiently establish a substantial conceptual advance over the existing approach.
Questions for authors
-
In the 30-second LongLive setting in Table 1, Dummy Forcing achieves both higher speedup and higher VBench scores than HeadCast. Could the authors provide the complete per-dimension VBench-Long results to explain these differences? Please also report the complete VBench/VBench-Long breakdowns for Table 2 to substantiate the quality-preservation claims.
-
Could the authors clarify the technical novelty of HeadCast relative to Dummy Forcing, particularly regarding head classification and head-specific processing, and distinguish the new contributions from extensions of the existing framework?
7 comments
Response to Reviewer qrTV (2/4): Q1 - what the breakdown explains
Q1 (continued). What the breakdown explains
- LongLive 480P (the row the reviewer asks about). The three Totals lie within 0.36 points, for different reasons. Dummy Forcing scores +0.22 above the full-attention model it approximates: with VBench's own normalisation and weights, this is +0.45 from Imaging Quality (65.97 → 69.65), −0.37 from its four temporal-consistency dimensions, which are all below full attention, and +0.14 from the rest. HeadCast's −0.14 is −0.35 from Dynamic Degree (flow magnitude 0.92× that of full attention; see our reply to Reviewer 1st9) and +0.21 from all other dimensions; its temporal-consistency dimensions are within 0.15 of full attention. A Total above the full-attention reference is therefore not evidence that the reference is better preserved: on the 30 s clips of App. G, BBD is 1.78 for Dummy Forcing against 1.30 for full attention and 1.26 for HeadCast.
- LongLive 720P / 1080P. Dummy Forcing is below full attention on all four temporal-consistency dimensions (−0.91 / −0.80 Total points). Its Total is held up by Dynamic Degree rising far above the full model (31.4 → 44.5, 14.7 → 33.7; +0.81 / +1.17 Total points). This extra "motion" is block-boundary jumps and drift: on the 720P videos its optical-flow magnitude is 1.54× that of full attention (95 % CI 1.38–1.72), and 1.87× on frame pairs that straddle a block boundary. HeadCast is at or above full attention on all four temporal-consistency dimensions, with Total 79.05 vs 79.07 and 77.26 vs 76.84.
- Self-Forcing 5 s. HeadCast is within 0.3 points of full attention on every dimension at 720P, and within 0.45 at 1080P except Dynamic Degree (+0.83). Dummy Forcing loses Dynamic Degree (65.00 → 62.50, 60.00 → 53.89): its clips become more static, which raises its flickering / smoothness scores.
- Speed. Dummy Forcing is the faster method on every LongLive row (1.16× / 1.74× / 2.16× against 1.08× / 1.62× / 1.95×), at lower fidelity to the full model: at 480P its PSNR is 13.6 dB against 16.3 dB for HeadCast (Table 1, corrected in the revision; see the general response). At matched throughput on Self-Forcing 720P (Fig. 4: 9.41 vs 9.60 FPS) HeadCast keeps 25.32 dB against 18.81 dB.
Response to Reviewer qrTV (3/4): Q2 - what is new relative to Dummy Forcing
Q2. What is new relative to Dummy Forcing
HeadCast builds on Dummy Forcing, and Sec. 3.2 says so. Two things are inherited: the observation that some heads are temporally sparse (our Sink and Dummy types), and serving those heads with one fused kernel call. The rest came out of asking why Dummy Forcing breaks on long rollouts.
(1) The failure and its two causes. Dummy Forcing restricts every head. In the released implementation sink heads read {first frame, current block}, neighbor heads {previous block, current block} and dummy heads {last frame, current block}; half of the heads are made dummy by a fixed budget, ranked by attention mass. On 30 s rollouts the output flickers at block boundaries (Fig. 3, App. G), and 90.9 % of the user-study votes prefer HeadCast (Table 3). In the paper Dummy Forcing runs inside our pipeline. As a cross-check we ran its released code with its released LongLive configuration on the 40 prompts of App. G (30 s, 480P) and then changed one thing at a time (App. J.7).
Released Dummy Forcing code. Reference: the same code without head programming (BBD 1.29; 2 of 40 videos above 1.5).
| Variant | Heads restricted | BBD − Full (95 % CI) | Videos with BBD > 1.5 |
|---|---|---|---|
| Released configuration | all 360 (≈120 sink, ≈60 neighbor, 180 dummy) | +0.62 [+0.49, +0.76] | 33 of 40 |
| Neighbor heads read the 9 most recent frames | sink + dummy (≈300) | +0.50 [+0.39, +0.62] | 30 |
| Only its sink heads restricted | ≈120 | +0.20 [+0.12, +0.28] | 16 |
| Only its dummy heads restricted | 180 | −0.07 [−0.14, +0.00] | 3 |
(A group that is not restricted reads the 9 most recent frames. At 720P the released configuration gives +0.28 [+0.18, +0.39], with 20 of 40 videos above 1.5.)
Our code. The N best candidates become Dummy heads; all other heads keep the full window (Global).
| Variant | PSNR (dB) | LPIPS | PSNR − HeadCast (95 % CI) |
|---|---|---|---|
| N = 90 | 16.02 | 0.276 | −0.59 [−1.10, −0.11] |
| N = 180 | 12.94 | 0.392 | −3.67 [−4.63, −2.76] |
| N = 270 | 12.06 | 0.429 | −4.55 [−5.68, −3.49] |
| HeadCast (94 Dummy heads, by its test) | 16.61 | 0.263 | – |
(Fidelity to the full-attention output of the same code on the 40 prompts of App. G. These are new generations with the supplementary code, so the HeadCast row differs slightly from Table 1, which uses the videos of App. G: 16.61 against 16.30 dB.)
- Restricting the dummy heads alone does not raise BBD; restricting the sink heads does (+0.20). On average only 2 of these ≈120 sink heads reproduce their output from {first frame, current block}. The increase appears once heads that depend on the history lose it. The released design has no class for such heads; in HeadCast a head that fails its test keeps the window (Global).
- A budget instead of a test costs fidelity. On these prompts HeadCast's test makes only 94 ± 11 of the 360 heads Dummy heads. Ranked by the cosine between the output from the current block alone and the full-window output, the 90th head is at 0.96 and the 180th at 0.82, so a budget of 180 includes about 80 heads that use the history: N = 180 is 3.7 dB below HeadCast, N = 90 only 0.6 dB.
Response to Reviewer qrTV (1/4): Q1 - per-dimension VBench results
We thank the reviewer for the careful reading. We first give the requested per-dimension results, then separate what HeadCast inherits from Dummy Forcing from what is new.
Short answers
Q1. Complete per-dimension VBench / VBench-Long results
Every cell is Full attention / Dummy Forcing / HeadCast (×100).
LongLive, 30 s, VBench-Long (480P: standard 16-dimension suite; 720P/1080P: MovieGen prompts in custom mode, which supports the eight dimensions listed)
Self-Forcing, 5 s, VBench (Table 2)