acceptodds
Under review as a conference paper at ICLR 2027

LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation

Abstract

Autoregressive (AR) video diffusion enables variable-length synthesis, but long-horizon generation often suffers from accumulated errors and identity drift. For efficiency, existing methods commonly adopt sliding-window attention during generation. This creates an irreversible generation trajectory: once the active window accumulates appearance errors, subsequent generations can only condition on this degraded trajectory and drift further away. We address this limitation by formulating long video generation as a retrieval-augmented generation (RAG) problem. Rather than relying solely on the recent window, we treat previously generated latents as a dynamic, searchable history. We propose LongLive-RAG, a general retrieval framework for AR video generation. At each new block, LongLive-RAG uses a query embedding to retrieve relevant historical latents. This lightweight retrieval step adds only a small overhead relative to generation and lets the generator condition on non-local context instead of only the recent window. To make retrieval more discriminative, we introduce the Window Temporal Delta Loss that suppresses redundant local similarity and encourages embeddings to capture meaningful temporal changes. Together, these components help reduce error accumulation caused by sliding-window attention. Experiments across multiple AR backbones and generation lengths show improved long-video quality and the best average VBench-Long rank. To our knowledge, among open-ended AR long video generation methods, LongLive-RAG is the first to formulate self-generated latent history as content-addressable retrieval memory.

Primary area:
generative models
Keywords:
Long Video Generation, Autoregressive Video Diffusion, Latent Retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7AjfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a retrieval-augmented generation (RAG) method to augment context tokens, along with the existing sink and recent tokens. To generate these retrieved latents, the paper proposes the window temporal delta loss with some smoothing penalty to retrieve the embeddings that discourages duplicated latents. The used RAG adds little overhead to the existing pipelines.

Critical strengths

The paper leverages RAG-based method to augment the context tokens in long video generation.The paper's motivation is clear, and the proposed mechanism is logically feasible and appears effective.

To learn the embeddings that are suitable for the retrieval task, the paper presents the window temporal delta loss, which discourages excessive similarity within the given temporal window. However, this loss alone is not sufficient as it will retrieve some irrelevant information. Therefore, the paper also proposes a smoothing loss item to learn a more stable trajectory.

In summary, both the RAG-based retrieval and the associated loss functions are considered to be the main strengths of this paper.

Critical weaknesses

The paper deals with the long video generation. The proposed RAG-based retrieval mechanism is straightforward and should well in practice. There are similar works that have explored this idea before in other video analysis tasks. These similar RAGs have been widely explored. What are the advantages of the proposed RAG mechanism compared to other similar works?

VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos

The paper mentions the MemFlow work, which seems to be the lonely work cited that also leverages RAG for video generation. Since this use of RAG is quite straightforward, what bottlenecks prevent the existing works from employing such an effective mechanism? The computational burden also seems to be minimal.

The paper does not seem to include any video demo results. I have tried to find whether any supplementary or appendix included, but no such files are available. This is a significant weakness of the proposed work, as the video generation relies heavily on the video demos to compare its qualitative results. This missing inclusion of the video demos clearly diminishes the quality of this paper.

The proposed loss functions are very critical to ensure the success retrieval of the relevant tokens, not just the similar tokens. Both the window temporal delta loss and the smoothing loss function items play a critical role. But there is no ablation study to demonstrate the contributions of these loss function items.

When checking the experimental results presented in Table 2, it appears that the performance of the proposed LongLive-RAG also varies, across the 30s, 60s, and 120s generations.

In Figure 2, the paper shows the trajectory that is fixed by the proposed RAG method. Along with that, the flowchart also presents the token selection analysis. Do they correspond to the true token selections? Are there more vivid results to demonstrate that? There is a conflict on how to choose the tokens. On one hand, we do not want tokens to be duplicated. On the other hand, we also do not want tokens to be irrelevant. How to maintain such a balance is intriguing. The paper does not clearly offer these insights.

One minor suggestion, please do not use the reddit post as the reference, which does not seem to be a reliable source for references.

Questions for authors

Please present the video results, both in qualitative comparisons and ablation studies.

Please explain how to ensure the selection of tokens is relevant while not being duplicated.

Please explain the advantages of the proposed RAG mechanism compared to other similar works.

Please provide the ablation study results for the different loss function items.

These are some of the highlighted questions.

3 comments

Thank you for the detailed review and constructive questions. We answer each point below.

Weaknesses and Questions

“Novelty relative to prior video retrieval”

Answer:

The task and retrieved representation are different. VideoRAG [1] retrieves video clips and uses their frames for understanding; LongLive-RAG retrieves previously generated VAE latents to guide the next generation block. Our contribution is the temporal retrieval objective and reuse of the corresponding native context with frozen AR generators under a fixed attention budget. One retrieval encoder is used across all three backbones.

“Comparison with MemFlow”

Answer:

MemFlow [2] trains its memory-augmented generator through DMD and streaming long-tuning. We train only the retrieval head and reuse it unchanged with frozen Causal-Forcing, Self-Forcing, and LongLive checkpoints in their shared latent space. This adds retrieval to existing AR backbones without AR generator retraining. MemFlow does not evaluate this cross-backbone reuse with frozen generators.

“Video demonstration”

Answer:

Thank you for the suggestion. We have provided video examples for visual comparison.

“Loss ablation”

Answer:

Table 4 already ablates the embedding variants. We add AE + Smooth below to test whether SeqDelta helps beyond smoothing, and reproduce the full 30s Causal-Forcing table:

SettingSubject Cons. ↑Background Cons. ↑Motion Smooth. ↑Imaging Quality ↑
Random retrieval94.5494.3296.8168.79
Avg-pool desc.94.7794.4996.7669.11
AE only94.8294.4996.8769.48
AE + SeqDelta94.7694.5497.0469.14
AE + Smooth94.8194.4896.9869.42
AE + SeqDelta + Smooth95.4394.7997.1670.07

Full gains 0.62, 0.31, 0.18, and 0.65 points over AE + Smooth across the four metrics, respectively. Removing smoothing (AE + SeqDelta) also lowers all four scores. Both terms help in this comparison.

“Performance across video durations”

Answer:

We summarize the Causal-Forcing imaging-quality results across durations below:

Method30s60s120s
Base65.5362.3058.23
LongLive-RAG70.0768.1668.23
Gain+4.54+5.86+10.00

The gain grows with duration for Causal-Forcing imaging quality. It varies by backbone and metric: LongLive 120s imaging quality ties Base, and Causal-Forcing 120s background consistency falls from 94.66 to 94.08.

“Figure 2 and actual retrieval selections”

Answer:

Figure 2 is a schematic. The added decoded visualization shows actual retrieval selections, with history indices and generated outputs. Each row shows its own rollout.

“Retrieval relevance and redundancy”

Answer:

Redundancy also arises in local-window context: adjacent frames often repeat nearly identical content. The visualization shows this repetition in the baseline. Retrieval gives access to earlier references when recent context drifts; Table 3 and the actual selections show cleaner context for LongLive-RAG under the evaluated protocol.

Similarity scoring favors relevant history. SeqDelta discourages excessive similarity between neighboring entries, and smoothing keeps the embeddings smooth over time.

“Reddit citation and formal references”

Answer:

We agree that the technical discussion needs formal references. The Reddit citation acknowledges the original NTK-aware RoPE proposal, also cited by StreamingLLM [3] and YaRN [4]. We will add the formal references while keeping this attribution.

References

[1] VideoRAG.

[2] MemFlow, Sections 3 and 3.1.

[3] Xiao et al. Efficient Streaming Language Models with Attention Sinks, 2023.

[4] Peng et al. YaRN: Efficient Context Window Extension of Large Language Models, 2023.

Reviewer_7AjfOpenReview ↗

Additional comments

Thanks for your rebuttal. I have some additional questions.

Regarding the novelty of this work, the previously generated latents were used to build compact embeddings for search. This was indeed different from existing works where RAG was used, from an application perspective. But I am also wondering how this retrieval from the previously generated latents is significantly different compared to the existing works that utilize historical context as memory for video generation. Here, the previously generated latents can be treated as memory. Then, the difference is on how to effectively retrieve the latents from the stored memory. Could you offer more comments on the uniqueness of the proposed work? The Introduction does not clearly demonstrate this, and only related works mention these memory-based works in Section 2.

Regarding the loss ablation, I am not referring to the ablation study on the embedding variants. Table 4 results are confusing. Are there any separate/unmixed results which are specifically looking at the loss function ablation studies?

Regarding the performance variations across 30s, 60s, and 120s generations, I would prefer there are more explanations on why the method is not consistently performing better, as compared to the ∞-RoPE and Deep Forcing, not just the base.

We thank the reviewer for the follow-up questions. We respond to each point below.

“Novelty relative to existing memory-based methods”

Answer:

Thank you for the insightful question. LongLive-RAG follows the context-as-memory paradigm, where previously generated latents serve as historical memory. Our main contribution is a general retrieval-augmented framework for long video generation that decouples historical retrieval from the AR generation process.

Existing memory-based methods differ from ours in several important ways:

  • Deep Forcing makes long-term memory retention decisions based on short-term attention patterns, potentially discarding information that may become useful later.
  • MemFlow introduces adaptive memory retrieval, but its framework involves additional AR generator training. It has not demonstrated cross-backbone reuse of its retrieval mechanism without additional fine-tuning.
  • Context as Memory retrieves historical frames based on camera poses and field-of-view overlap, requiring geometric information that is generally unavailable in open-ended video generation.

In contrast, LongLive-RAG learns an independent retrieval module while keeping the AR generator frozen. The same retrieval encoder is trained once and reused across three different AR backbones without additional fine-tuning.

We have discussed these distinctions in the Appendix, and we will further clarify them in both the Introduction and Related Work sections to better highlight our motivation and contributions.

“Separate loss ablation”

Answer:

Our retrieval module is trained as an autoencoder (AE), where the reconstruction loss serves as the fundamental objective for preserving meaningful information from the original video latents. SeqDelta and Smooth are auxiliary losses designed to improve the learned retrieval space.

To further address the reviewer's question, we additionally trained models using only SeqDelta or Smooth, without the reconstruction loss. Since neither objective provides reconstruction supervision, we can only train the encoder.

We analyze the resulting embedding spaces using the average pairwise cosine similarity on the validation set:

Training objectiveAvg. pairwise cosine similarity
SeqDelta only0.152
Smooth only0.999

The results reveal two degenerate behaviors:

  • SeqDelta only: The embeddings become overly dispersed, without preserving the original visual information needed for meaningful retrieval.
  • Smooth only: The embeddings collapse to nearly identical representations, providing almost no discriminative ability for retrieval.

Neither objective alone produces a meaningful retrieval space.

“Performance variations across generation durations”

Answer:

Thank you for the follow-up question. We would like to clarify that LongLive-RAG performs consistently well on the metrics most directly related to our goal of reducing accumulated visual errors.

As shown in Table 2, LongLive-RAG achieves the highest Subject Consistency in all 9 settings (3 backbones × 3 durations) and the highest or tied-highest Imaging Quality in all 9 settings. These results hold across 30s, 60s, and 120s generations when compared directly with ∞-RoPE and Deep Forcing, not just the base models.

The variations mainly occur in other metrics, such as Dynamic Degree and Motion Smoothness. One possible explanation is that retrieving earlier visual context introduces a natural balance between long-range consistency and local temporal dynamics under a fixed attention budget. Different context selection strategies may therefore favor different aspects of video generation.

We will clarify these metric-level differences and provide a more detailed discussion of the underlying trade-offs in the revised paper.

Reviewer_nrFQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LongLive-RAG, a retrieval-based context-selection framework for autoregressive video diffusion. The method maintains compact embeddings of previously generated clean latents together with their corresponding native generator context. Before generating each block, it uses the latest completed latent as a query, retrieves historical entries by cosine similarity, and combines them with sink and recent context under a fixed attention budget. The retrieval encoder is trained offline using reconstruction, a local temporal similarity penalty, and second-order temporal smoothing, while the video generator remains frozen. The paper evaluates one shared retrieval encoder across Causal-Forcing, Self-Forcing, and LongLive on 128 MovieGenBench prompts at 30, 60, and 120 seconds. LongLive-RAG achieves the best average rank across the six reported VBench-Long metrics in all nine backbone-duration settings. Additional evidence includes embedding and retrieval-budget ablations, context-quality measurements, an auxiliary exposure evaluation, and multi-prompt/return-to-scene experiments in Appendix H.

Critical strengths

Table 2 evaluates three backbones at three generation lengths, with a total attention window of 12 for all compared methods. LongLive-RAG obtains the best average rank in every setting. Some improvements are substantial: for 120s Causal-Forcing, imaging quality increases from 58.23 to 68.23, alongside improvements in subject consistency and motion smoothness. This breadth makes the framework more convincing than a demonstration on a single generator.

Critical weaknesses

Algorithm 1 continually adds native context entries to the historical bank without specifying a capacity or eviction policy. A fixed active attention window therefore does not imply bounded historical storage. Table 1 measures encoding and search, but does not establish end-to-end latency or CPU/GPU memory costs, including historical context storage and movement. Separately, no uncertainty estimates or repeated-seed results are reported, although some gains are small—for example, LongLive subject consistency changes from 97.35 to 97.53 at 30s, and the full multi-prompt test’s mean CLIP score changes from 27.49 to 27.52. The average-rank summary obscures effect sizes, and Self-Forcing’s 30s dynamic degree decreases from 52.03 to 42.24. These omissions limit confidence in both the practical scalability claim and the extent of the overall improvement.

Questions for authors

1.Contribution of the loss: Can you provide an AE-plus-smoothing ablation under the same retrieval budget and recency guard? Does SeqDelta improve results across seeds or longer rollouts, and how does its benefit change when the recency guard is varied? This would clarify whether the proposed delta loss contributes beyond smoothing and candidate exclusion.

2.Implementation and resource use: What exactly is stored in each historical context entry—latents, per-layer KV tensors, or another representation? How are temporal positions handled when noncontiguous entries are retrieved? Please report historical CPU/GPU memory and end-to-end runtime at 30s, 60s, and 120s, including any transfer or context reconstruction.

3.Mechanism: For Table 3, are context entries compared along matched generation trajectories or along each method’s own rollout? The latter would conflate retrieval quality with differences in already-generated video quality. Can a matched-trajectory analysis show that retrieval selects cleaner context and improves subsequent generation?

2 comments

We sincerely thank you for your careful review and constructive suggestions. We respond to each point below.

Weaknesses and Questions

“Runtime and memory usage”

Answer:

Our contribution is a general retrieval framework for long-video generation. KV quantization and compression can be used alongside it. Paging and KV quantization are well studied in LLM serving [1,2]; here we test INT8 KV storage and KV selection/merge [3] on 30s Causal-Forcing using an NVIDIA RTX A6000:

SettingBank storage ↓Peak GPU memory ↓Peak CPU memory ↓Runtime ↓
Base without RAG0 GB~22 GB~15 GB~2 min 50 s
LongLive-RAG~31 GB~25 GB~45 GB~3 min 30 s
+ INT8 KV storage~15.5 GB~25 GB~30 GB~3 min 15 s
+ KV selection/merge [3]~4.7 GB~25 GB~20 GB~3 min

INT8 storage cuts the bank by ~50%; KV selection/merge cuts it by ~85%. They are applied separately, and neither changes the retrieval scoring rule. The archive still grows with video length; a fixed capacity or eviction policy remains future work.

“Variability across generation seeds”

Answer:

We ran Full on all 128 evaluation prompts with three generation seeds at 30s Causal-Forcing. The table gives the mean ± sample SD of the three benchmark averages:

SettingSubject Cons. ↑Background Cons. ↑Motion Smooth. ↑Dynamic Degree ↑Aesthetic Quality ↑Imaging Quality ↑
LongLive-RAG95.18 ± 0.3594.84 ± 0.1697.14 ± 0.1181.12 ± 1.6757.20 ± 0.5370.46 ± 0.89

Repeating every baseline and ablation requires regenerating long videos, so here we report three seeds for Full. Self Forcing [4] and FIFO-Diffusion [5] also cite compute cost for not reporting error bars. LongLive [6], CausVid [7], and Self-Forcing++ [8] report representative ablations without seed-wise error bars; Cosmos 3 uses one seed per prompt on RBench [9].

“Absolute gains and average rank”

Answer:

We agree that average rank alone does not show the size of the gain. On Causal-Forcing 120s, imaging quality rises from 58.23 to 68.23: +10.00 over Base and +1.42 over infinity-RoPE. Other settings have smaller gains or ties, including LongLive 120s imaging quality.

“Decrease in Dynamic Degree”

Answer:

We agree about Self-Forcing at 30s and 60s: Dynamic Degree falls, while imaging quality rises. Across the nine settings in the main table, Dynamic Degree increases in seven (Base → LongLive-RAG):

Backbone30s60s120s
Self-Forcing52.03 → 42.2451.72 → 44.6943.39 → 44.10
LongLive44.74 → 44.8444.56 → 49.9047.12 → 50.25
Causal-Forcing73.96 → 82.2972.32 → 88.2063.79 → 90.21

The same comparison for the other evaluated training-free methods on Self-Forcing is below. Parentheses show changes from Base:

Self-Forcing settingDynamic Degree, 30s ↑Imaging Quality, 30s ↑Dynamic Degree, 60s ↑Imaging Quality, 60s ↑
Base52.0363.3151.7262.22
+ infinity-RoPE46.82 (-5.21)63.9346.64 (-5.08)63.28
+ Deep Forcing38.85 (-13.18)61.9141.44 (-10.28)60.81
+ LongLive-RAG42.24 (-9.79)65.4344.69 (-7.03)64.97

LongLive-RAG is higher than Deep Forcing on both metrics, while infinity-RoPE has higher Dynamic Degree. On Causal-Forcing 120s, both metrics improve over Base. Dynamic Degree measures thresholded optical flow [10]. Its score can be sensitive to noise and visual artifacts that affect the flow estimate [11]. We will discuss it together with imaging quality and state the trade-off on Self-Forcing.

“AE + Smooth ablation”

Answer:

We have added AE + Smooth, which keeps smoothing and removes SeqDelta. The full 30s Causal-Forcing ablation is below:

SettingSubject Cons. ↑Background Cons. ↑Motion Smooth. ↑Imaging Quality ↑
Random retrieval94.5494.3296.8168.79
Avg-pool desc.94.7794.4996.7669.11
AE only94.8294.4996.8769.48
AE + SeqDelta94.7694.5497.0469.14
AE + Smooth94.8194.4896.9869.42
AE + SeqDelta + Smooth95.4394.7997.1670.07

Full gains 0.62, 0.31, 0.18, and 0.65 points over AE + Smooth across the four metrics, respectively. Removing smoothing (AE + SeqDelta) also lowers all four scores. Both terms help in this comparison.

“Ablation on longer videos”

Answer:

We ran this comparison at 60s on Causal-Forcing, using all 128 evaluation prompts:

SettingSubject Cons. ↑Background Cons. ↑Motion Smooth. ↑Imaging Quality ↑
AE + Smooth93.9393.9796.3267.67
Full94.2994.2496.4868.16

Full improves all four metrics over AE + Smooth at 60s, as at 30s.

“Ablation on recency guard”

Answer:

We added the recency-guard ablation. R=0 removes the guard, and we test R from 0 to 20:

Recency guard RSubject Cons. ↑Background Cons. ↑Motion Smooth. ↑Dynamic Degree ↑Aesthetic Quality ↑Imaging Quality ↑
095.3594.6897.2282.9257.6770.25
295.2994.6997.2081.1557.3670.25
5 (default)95.4394.7997.1682.2957.3170.07
1095.2994.6797.1980.8357.6670.13
2095.1094.5497.1981.4657.5470.07

The scores change little across these R values, including R=0. The loss ablation measures SeqDelta's gain separately at the default R=5.

“Historical KV storage”

Answer:

H_ctx stores the generator's native per-layer K/V tensors on CPU, with keys saved before RoPE. The 1024-dimensional embeddings form a separate search bank; the retrieval decoder is used only during training.

“Temporal indexing of retrieved context”

Answer:

We assign retrieved frames temporal index 0 and keep their spatial coordinates. Their pre-RoPE keys are rotated when reused; local/current tokens use rolling-cache positions. The context is [sink | similarity-ranked retrieval | recent context | current block].

“History comparability and context quality”

Answer:

The methods already share an initial history: before the context window is exceeded, they use the same context [4]. Their KV cache management then differs, and later histories may diverge. Table 3 measures context quality under this protocol:

ContextVAE cycle loss ↓CLIP-IQA ↑MUSIQ ↑
Base0.5580.36254.0
infinity-RoPE0.5290.34853.9
Deep Forcing0.5320.34752.9
LongLive-RAG0.4450.42158.9

The actual retrieval visualization gives a visual example: the retrieved frames are cleaner, with clearer fish and coral structures, while the local-window baseline shows color distortion and repetitive streak-like patterns.

References

[1] Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.

[2] Liu et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.

[3] Luo et al. Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation, 2026.

[4] Huang et al. Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, NeurIPS 2025. Section 3.4, Algorithm 2, and Checklist Q7.

[5] Kim et al. FIFO-Diffusion: Generating Infinite Videos from Text without Training, NeurIPS 2024. Checklist Q7.

[6] Yang et al. LongLive: Real-time Interactive Long Video Generation, ICLR 2026. Table 4.

[7] Yin et al. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CausVid), CVPR 2025. Table 4.

[8] Cui et al. Self-Forcing++: Towards Minute-Scale High-Quality Video Generation, ICLR 2026. Table 3 in the arXiv version.

[9] NVIDIA. Cosmos 3: Omnimodal World Models for Physical AI, 2026. Section 6.2.2, RBench evaluation.

[10] VBench official Dynamic Degree implementation.

[11] Oei et al. RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo, ICLR 2026.

Reviewer_ipT2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LongLive-RAG to reduce visual drift in long autoregressive videos. A learned encoder retrieves relevant past latents and combines their context with recent context, without updating the video generator. The encoder uses reconstruction, temporal-delta and smoothing losses to preserve visual information and encourage more useful, stable retrieval. A single encoder is evaluated on three backbones for 30s, 60s and 120s generation under a fixed attention budget. The results show improvements in consistency and image quality, with the best average VBench-Long rank across all nine settings. Ablations and return-to-scene tests further examine the role of retrieval.

Critical strengths
  1. One retrieval encoder trained on LongLive transfers to all three frozen backbones, with a fixed 12-slot attention budget. This makes the approach practical to reuse without retraining each generator.

  2. The method achieves the best average rank in all nine backbone/duration settings. For 120s Causal-Forcing, imaging quality improves from 58.23 to 68.23 (Table 2). The return-to-scene results in Appendix H also support the value of historical recall.

Critical weaknesses
  1. The temporal-delta loss is not fully justified by the ablation. Adding SeqDelta to AE slightly lowers subject consistency and imaging quality (Table 4), while AE+smoothing is missing. The full objective's gains therefore do not isolate the contribution of SeqDelta.

  2. Table 1 only times encoding and search. The historical context bank grows with the rollout, but its memory footprint and the cost of reusing old context are not reported. This leaves the practicality of open-ended generation unclear.

Questions for authors
  1. Could you add AE+smoothing without SeqDelta to Table 4, using the same seeds and context budget? This would clarify whether the temporal-delta term adds value beyond smoothing.

  2. Does H_ctx store latents or per-layer KV caches, and how are temporal positions assigned to retrieved entries? Please report peak GPU/CPU memory and end-to-end runtime for the 30s, 60s and 120s settings.

1 comment

Thank you for the careful review and helpful questions. We answer each point below.

Weaknesses and Questions

“Contributions of SeqDelta and smoothing”

Answer:

We have added AE + Smooth, which keeps smoothing and removes SeqDelta. The full 30s Causal-Forcing ablation is below:

SettingSubject Cons. ↑Background Cons. ↑Motion Smooth. ↑Imaging Quality ↑
Random retrieval94.5494.3296.8168.79
Avg-pool desc.94.7794.4996.7669.11
AE only94.8294.4996.8769.48
AE + SeqDelta94.7694.5497.0469.14
AE + Smooth94.8194.4896.9869.42
AE + SeqDelta + Smooth95.4394.7997.1670.07

Full gains 0.62, 0.31, 0.18, and 0.65 points over AE + Smooth across the four metrics, respectively. Removing smoothing (AE + SeqDelta) also lowers all four scores. Both terms help in this comparison.

“Runtime and memory usage”

Answer:

Our contribution is a general retrieval framework for long-video generation. KV quantization and compression can be used alongside it. Paging and KV quantization are well studied in LLM serving [1,2]; here we test INT8 KV storage and KV selection/merge [3] on 30s Causal-Forcing using an NVIDIA RTX A6000:

SettingBank storage ↓Peak GPU memory ↓Peak CPU memory ↓Runtime ↓
Base without RAG0 GB~22 GB~15 GB~2 min 50 s
LongLive-RAG~31 GB~25 GB~45 GB~3 min 30 s
+ INT8 KV storage~15.5 GB~25 GB~30 GB~3 min 15 s
+ KV selection/merge [3]~4.7 GB~25 GB~20 GB~3 min

INT8 storage cuts the bank by ~50%; KV selection/merge cuts it by ~85%. They are applied separately, and neither changes the retrieval scoring rule. The archive still grows with video length; a fixed capacity or eviction policy remains future work.

“Historical KV storage”

Answer:

H_ctx stores the generator's native per-layer K/V tensors on CPU, with keys saved before RoPE. The 1024-dimensional embeddings form a separate search bank; the retrieval decoder is used only during training.

“Temporal indexing of retrieved context”

Answer:

We assign retrieved frames temporal index 0 and keep their spatial coordinates. Their pre-RoPE keys are rotated when reused; local/current tokens use rolling-cache positions. The context is [sink | similarity-ranked retrieval | recent context | current block].

References

[1] Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.

[2] Liu et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.

[3] Luo et al. Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.