acceptodds
Under review as a conference paper at ICLR 2027

HorizonKV: An Overlap-Aware Sparse Attention System for Long-Horizon Autoregressive Video Diffusion

10 views

Abstract

Autoregressive diffusion transformers (AR-DiTs) generate video chunk by chunk and can in principle extend generation to arbitrary video length. However, as KV caches grow, existing models are forced to retain only a short window of local frames, causing catastrophic forgetting and drift. Retrieval-based sparse attention retrieves important historical frames, but GPU capacity limits the range of accessible history. We observe that access to the full KV history substantially improves generation quality. However, realizing this benefit raises two challenges. The first is insufficient memory capacity. Recent work addresses this by offloading KV to the CPU, but retrieval overhead slows generation. The second is increased compute FLOPs, as longer histories contain more important KV entries and thus require more attention FLOPs. To address these challenges, we present HorizonKV, a training-free sparse attention system for long-horizon autoregressive video diffusion. To address insufficient memory capacity, we introduce Overlap-Aware Execution, which uses tiered storage and selectively overlaps KV transfers with attention computation to hide transfer latency. To address increased compute FLOPs, we introduce Residency-Aware Sparse Attention, which adaptively selects two-level sparsity configurations based on KV cache residency on the GPU. It maintains quality while hiding as much communication overhead as possible without increasing FLOPs. Experiments show HorizonKV advances the Pareto frontier over all baselines, with up to 1.486x end-to-end speedups.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
AR-DiT, Video Generation, Single-GPU Acceleration, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_28kkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HorizonKV is a training-free system for long-horizon autoregressive video diffusion. It combines CPU–GPU KV storage, overlapping data transfers with computation, and residency-aware sparse attention to retain access to the full history while improving inference efficiency.

Critical strengths
  1. The paper addresses a valuable problem: retaining access to the full KV history in autoregressive video generation through joint algorithm and system design.
  2. The analyses of memory growth, transfer overhead, and execution costs provide practical insights for deployment (Sections 3 and 5.4, Appendix E).
  3. The paper is well written and easy to follow, with clear motivation and an accessible presentation of the system.
Critical weaknesses
  1. The proposed efficient attention execution assumes that all queries within a chunk share the same selected historical frames within each head. Has this assumption been evaluated through comparative experiments? Would allowing different queries to select different historical frames improve generation quality?
  2. The frame-then-block sparse attention design closely resembles Light Forcing [1], which also retrieves relevant frames before selecting important blocks. The paper should clarify the algorithmic novelty beyond this existing hierarchy, distinguishing it from the contributions in offloading, overlap scheduling, and residency-aware adaptation (Section 4, Appendix E).
  3. The qualitative examples focus on continuous single-shot videos (Appendix D, Figures 15–17), and no dedicated multi-shot evaluation is reported. How does HorizonKV perform across shot changes or prompt switches, where relevant history and cache locality may change substantially?
  4. The reliability of the 120-second Self Forcing results in Table 1 needs further justification, as the original model exhibits quality degradation when extrapolating substantially beyond its training duration. Furthermore, quantitative quality evaluation relies solely on VBench. Could the authors report additional assessments, such as VisionReward [2] scores or human preference results, to better substantiate the claimed long-video generation quality?

[1] Chengtao Lv et al. Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention. arXiv preprint arXiv:2602.04789, 2026.

[2] Jiazheng Xu et al. VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(13): 11269–11277, 2026.

Questions for authors

See Weaknesses

Reviewer_DK8jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper deals with speeding up AR video diffusion models while keeping good quality over long videos. The authors propose a training-free method that keeps the full KV history on the CPU and a smaller cache on the GPU. For each attention call, they score historical frames and select the top k. If some KV entries are missing from the GPU, they can transfer them while calculating attention on the cached ones. They also adjust k and attention density based on what is cached, choosing between settings tested beforehand to give similar quality. They evaluate on Self Forcing and LongLive, mainly with two-minute videos, and report better speed and overall quality than the competing methods. Ablations test the adaptive settings, overlapping transfers, and keeping CPU history versus using only the GPU cache.

Critical strengths

The main strength is how they combine historical retrieval with faster attention. They do not just keep more history; they adjust how many frames to use and how densely to attend to them based on what is already on the GPU, while overlapping transfers when this saves time. I think this is a useful contribution, even if some individual parts already exist, because it addresses both speed and access to older information without retraining the model.

The experiments give good support for the method. They compare against both acceleration and historical-memory methods on two backbones, and report better speed and overall quality. They also test adaptive versus fixed settings, execution overlap, and CPU history versus GPU-only caching. These ablations help show where the gains come from, which makes the results more convincing.

Critical weaknesses

I could not find full videos in the available material, so it is hard to see whether the outputs actually look good and stay consistent over time. The screenshots and scores do not show what happens when an object disappears and later returns, which seems important for a method that keeps long-term history. Such an example would help show whether the score-based retrieval actually helps preserve the object’s identity in difficult scenarios.

The authors do not compare against Forcing-KV and MemFlow, although they mention both in the paper. These methods also deal with speed and historical memory. Comparing against them, or explaining why the comparison is not suitable, would give a clearer picture of the method’s advantages.

Questions for authors

What explains the quality improvement over LongLive-RAG and OmniMem, given that they also retrieve historical information? Is it mainly the frame selection, the within-frame sparsity, or the adaptive choice of configurations?

Do you have examples or analysis showing that the method retrieves the relevant old frames when an object disappears and later returns? This would help distinguish remembering a specific object from improving general video quality.

Reviewer_L5nMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-horizon autoregressive video diffusion and argues that retrieving from the full KV history improves generation quality, but the history quickly exceeds GPU memory and CPU offloading places KV transfers on the critical path. The paper presents HorizonKV, a training-free sparse attention system that keeps the full KV history in host CPU memory with a bounded GPU cache. Overlap-Aware Execution splits attention into resident and missing frames and overlaps transfers with resident computation when a profiled cost model favors it. Residency-Aware Sparse Attention selects the frame count and within-frame density from an offline-calibrated quality plateau so that transfers can be hidden. Built on Self Forcing and LongLive and evaluated on 2-minute videos with VBench-Long, HorizonKV improves the total score with up to a 1.486x speedup.

Critical strengths
  1. Training-free method. HorizonKV requires no retraining or fine-tuning and is applied to both Self Forcing and LongLive.
  2. Offloading overhead is almost hidden. On Self Forcing, overlapping KV transfers with attention on resident KV yields a 1.21× speedup over serial transfers with the same selection, hiding nearly all transfer latency. Since the split attention outputs are merged exactly, this overlap comes at no quality cost.
Critical weaknesses
  1. The quality benefit of full-history retrieval is small. Compared with HorizonKV-GPU, retrieval from the full history raises the total score only from 0.8176 to 0.8248 on Self Forcing and from 0.7932 to 0.8007 on LongLive, and on LongLive the GPU-only variant even achieves a higher semantic score. Similarly, the motivating result in Figure 3a shows only about a 0.01 gain on a narrow y-axis (0.812 to 0.824). Gains of this size could be within noise.
  2. Missing comparisons with training-free long-video generation methods. One of the main claims is that retaining and retrieving from the full history is necessary for long-video quality, yet the paper does not compare with training-free long-horizon KV cache management methods that operate under a bounded KV cache, such as Infinity-RoPE [1], Deep Forcing [2], and MemRoPE [3].
  3. Narrow analysis and hardware scope. All motivating observations are measured only on Self Forcing with 60-second videos, and both evaluated models share the same Wan 1.3B backbone. Efficiency is measured only on a single A100 with PCIe, although whether transfers can be hidden depends on the ratio between attention throughput and host-device bandwidth, which differs substantially on newer GPUs (H200, B200).

[1]: Yesiltepe et al., “Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rollout”, CVPR 2026

[2]: Yi et al., “Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression”, ICML 2026

[3]: Kim et al., “MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens”, ECCV 2026

Questions for authors
  1. Could the authors verify whether Observations 1 to 3 also hold for other AR-DiTs, such as Causal Forcing [4] and Rolling Forcing [5]?
  2. Could the authors compare HorizonKV with training-free long-video generation methods such as Infinity-RoPE [1], Deep Forcing [2], and MemRoPE [3]?
  3. Could the authors clarify how the frame scoring in Eq. 2 differs from the Participative Compression of Deep Forcing [2], which also scores historical KV with the current denoising queries?
  4. Could the authors report the temporal distance distribution of the selected frames and the GPU cache hit rate as video length increases? Since RASA favors resident frames and selected frames remain resident under LRU, selection may drift toward recently accessed history, which would conflict with the goal of full-history retrieval.
  5. Could the authors evaluate HorizonKV on newer GPUs such as H200 or B200 and report the fraction of calls in which transfers cannot be fully hidden?

[4]: Zhu et al., “Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation”, ICML 2026

[5]: Liu et al., “Rolling Forcing: Autoregressive Long Video Diffusion in Real Time”, ICLR 2026

1 comment

Thank you for the detailed rebuttal, which addresses most of my concerns. However, my main concern about the benefit of full-history retrieval remains.

In Table 1, HorizonKV-GPU already achieves most of the quality gain and a slightly higher speedup, while full-history retrieval adds only 0.72–0.75 VBench points at a small latency cost. Figure 13 shows that this gap remains below one point even at 180 s. Although the new quantitative multi-shot results are encouraging, they lack uncertainty estimates and comparisons with other baselines, and are evaluated on only one backbone. Furthermore, the 14B experiment, where offloading should matter most, does not include HorizonKV-GPU, making it difficult to isolate the benefit of full-history retrieval.

While the revision strengthens the paper, I remain unconvinced that the magnitude and generality of the benefit from full-history retrieval have been sufficiently established. I will therefore maintain my score.

Reviewer_jBWpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HorizonKV, a training-free sparse attention system for long-horizon autoregressive video diffusion. The authors observe that restricting KV retrieval to a local window can miss content-dependent attention hotspots in distant history, while keeping the full KV history exceeds GPU capacity. To address this, HorizonKV stores the full history in host memory and introduces two mechanisms: Overlap-Aware Execution, which overlaps KV transfers with attention computation on GPU-resident KV and chooses between one-shot and split attention, and Residency-Aware Sparse Attention, which selects among quality-comparable sparsity configurations based on KV residency and estimated transfer and compute times. Experiments are conducted on Self Forcing and LongLive, reporting VBench quality gains and up to about 1.486× end-to-end speedup.

Critical strengths

1.The problem is well motivated. The analysis of KV cache growth and attention hotspot distribution in long video generation gives a useful empirical argument for why relying only on a local window is insufficient and why retrieving from a broader history is worthwhile. 2.The system design is internally coherent. Overlap-Aware Execution provides an explicit basis for deciding when split attention is beneficial, and Residency-Aware Sparse Attention connects sparsity selection with the ability to hide transfer overhead, rather than treating sparsification and offloading as separate problems.

Critical weaknesses

1.My first concern is whether the baseline comparison is fair. Section 5.1 lists LightForcing, LongLive-RAG, OmniMem, and HorizonKV-GPU, and Table 1 compares their quality and latency. However, the paper does not clearly report whether these methods were run with the same attention budget, GPU cache size, retrieval frame count, and offloading setup. The authors also say that LongLive-RAG and OmniMem are not designed for acceleration, yet their latencies are still used in the speedup and Pareto comparison. Without this information, I cannot tell whether the reported speedup and quality advantage are apples-to-apples. 2.The quality gain appears small and is not supported by variance or significance tests. Section 5.2 reports VBench total gains of about 0.02, but the paper does not report results over multiple seeds or prompts, confidence intervals, or paired tests. Since VBench scores can move with prompt and seed, a gain of this size needs statistical support before the paper can claim a substantial quality improvement over the strongest baseline. 3.The calibration used by the selection mechanism may overlap with the evaluation benchmark. Section 4.2 says the quality plateau and configuration scores come from offline calibration, and Section 5.1 says the main evaluation uses VBench-Long. Appendix E only checks transfer across two prompt subsets. The paper should state clearly whether the calibration prompts are disjoint from the main evaluation prompts. If they are not, the quality advantage may partly reflect tuning on the evaluation benchmark rather than the proposed system design.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.