- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a retrieval-augmented generation (RAG) method to augment context tokens, along with the existing sink and recent tokens. To generate these retrieved latents, the paper proposes the window temporal delta loss with some smoothing penalty to retrieve the embeddings that discourages duplicated latents. The used RAG adds little overhead to the existing pipelines.
Critical strengths
The paper leverages RAG-based method to augment the context tokens in long video generation.The paper's motivation is clear, and the proposed mechanism is logically feasible and appears effective.
To learn the embeddings that are suitable for the retrieval task, the paper presents the window temporal delta loss, which discourages excessive similarity within the given temporal window. However, this loss alone is not sufficient as it will retrieve some irrelevant information. Therefore, the paper also proposes a smoothing loss item to learn a more stable trajectory.
In summary, both the RAG-based retrieval and the associated loss functions are considered to be the main strengths of this paper.
Critical weaknesses
The paper deals with the long video generation. The proposed RAG-based retrieval mechanism is straightforward and should well in practice. There are similar works that have explored this idea before in other video analysis tasks. These similar RAGs have been widely explored. What are the advantages of the proposed RAG mechanism compared to other similar works?
VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos
The paper mentions the MemFlow work, which seems to be the lonely work cited that also leverages RAG for video generation. Since this use of RAG is quite straightforward, what bottlenecks prevent the existing works from employing such an effective mechanism? The computational burden also seems to be minimal.
The paper does not seem to include any video demo results. I have tried to find whether any supplementary or appendix included, but no such files are available. This is a significant weakness of the proposed work, as the video generation relies heavily on the video demos to compare its qualitative results. This missing inclusion of the video demos clearly diminishes the quality of this paper.
The proposed loss functions are very critical to ensure the success retrieval of the relevant tokens, not just the similar tokens. Both the window temporal delta loss and the smoothing loss function items play a critical role. But there is no ablation study to demonstrate the contributions of these loss function items.
When checking the experimental results presented in Table 2, it appears that the performance of the proposed LongLive-RAG also varies, across the 30s, 60s, and 120s generations.
In Figure 2, the paper shows the trajectory that is fixed by the proposed RAG method. Along with that, the flowchart also presents the token selection analysis. Do they correspond to the true token selections? Are there more vivid results to demonstrate that? There is a conflict on how to choose the tokens. On one hand, we do not want tokens to be duplicated. On the other hand, we also do not want tokens to be irrelevant. How to maintain such a balance is intriguing. The paper does not clearly offer these insights.
One minor suggestion, please do not use the reddit post as the reference, which does not seem to be a reliable source for references.
Questions for authors
Please present the video results, both in qualitative comparisons and ablation studies.
Please explain how to ensure the selection of tokens is relevant while not being duplicated.
Please explain the advantages of the proposed RAG mechanism compared to other similar works.
Please provide the ablation study results for the different loss function items.
These are some of the highlighted questions.
3 comments
Additional comments
Thanks for your rebuttal. I have some additional questions.
Regarding the novelty of this work, the previously generated latents were used to build compact embeddings for search. This was indeed different from existing works where RAG was used, from an application perspective. But I am also wondering how this retrieval from the previously generated latents is significantly different compared to the existing works that utilize historical context as memory for video generation. Here, the previously generated latents can be treated as memory. Then, the difference is on how to effectively retrieve the latents from the stored memory. Could you offer more comments on the uniqueness of the proposed work? The Introduction does not clearly demonstrate this, and only related works mention these memory-based works in Section 2.
Regarding the loss ablation, I am not referring to the ablation study on the embedding variants. Table 4 results are confusing. Are there any separate/unmixed results which are specifically looking at the loss function ablation studies?
Regarding the performance variations across 30s, 60s, and 120s generations, I would prefer there are more explanations on why the method is not consistently performing better, as compared to the ∞-RoPE and Deep Forcing, not just the base.
We thank the reviewer for the follow-up questions. We respond to each point below.
“Novelty relative to existing memory-based methods”
Answer:
Thank you for the insightful question. LongLive-RAG follows the context-as-memory paradigm, where previously generated latents serve as historical memory. Our main contribution is a general retrieval-augmented framework for long video generation that decouples historical retrieval from the AR generation process.
Existing memory-based methods differ from ours in several important ways:
- Deep Forcing makes long-term memory retention decisions based on short-term attention patterns, potentially discarding information that may become useful later.
- MemFlow introduces adaptive memory retrieval, but its framework involves additional AR generator training. It has not demonstrated cross-backbone reuse of its retrieval mechanism without additional fine-tuning.
- Context as Memory retrieves historical frames based on camera poses and field-of-view overlap, requiring geometric information that is generally unavailable in open-ended video generation.
In contrast, LongLive-RAG learns an independent retrieval module while keeping the AR generator frozen. The same retrieval encoder is trained once and reused across three different AR backbones without additional fine-tuning.
We have discussed these distinctions in the Appendix, and we will further clarify them in both the Introduction and Related Work sections to better highlight our motivation and contributions.
“Separate loss ablation”
Answer:
Our retrieval module is trained as an autoencoder (AE), where the reconstruction loss serves as the fundamental objective for preserving meaningful information from the original video latents. SeqDelta and Smooth are auxiliary losses designed to improve the learned retrieval space.
To further address the reviewer's question, we additionally trained models using only SeqDelta or Smooth, without the reconstruction loss. Since neither objective provides reconstruction supervision, we can only train the encoder.
We analyze the resulting embedding spaces using the average pairwise cosine similarity on the validation set:
| Training objective | Avg. pairwise cosine similarity |
|---|---|
| SeqDelta only | 0.152 |
| Smooth only | 0.999 |
The results reveal two degenerate behaviors:
- SeqDelta only: The embeddings become overly dispersed, without preserving the original visual information needed for meaningful retrieval.
- Smooth only: The embeddings collapse to nearly identical representations, providing almost no discriminative ability for retrieval.
Neither objective alone produces a meaningful retrieval space.
“Performance variations across generation durations”
Answer:
Thank you for the follow-up question. We would like to clarify that LongLive-RAG performs consistently well on the metrics most directly related to our goal of reducing accumulated visual errors.
As shown in Table 2, LongLive-RAG achieves the highest Subject Consistency in all 9 settings (3 backbones × 3 durations) and the highest or tied-highest Imaging Quality in all 9 settings. These results hold across 30s, 60s, and 120s generations when compared directly with ∞-RoPE and Deep Forcing, not just the base models.
The variations mainly occur in other metrics, such as Dynamic Degree and Motion Smoothness. One possible explanation is that retrieving earlier visual context introduces a natural balance between long-range consistency and local temporal dynamics under a fixed attention budget. Different context selection strategies may therefore favor different aspects of video generation.
We will clarify these metric-level differences and provide a more detailed discussion of the underlying trade-offs in the revised paper.
Thank you for the detailed review and constructive questions. We answer each point below.
Weaknesses and Questions
“Novelty relative to prior video retrieval”
Answer:
The task and retrieved representation are different. VideoRAG [1] retrieves video clips and uses their frames for understanding; LongLive-RAG retrieves previously generated VAE latents to guide the next generation block. Our contribution is the temporal retrieval objective and reuse of the corresponding native context with frozen AR generators under a fixed attention budget. One retrieval encoder is used across all three backbones.
“Comparison with MemFlow”
Answer:
MemFlow [2] trains its memory-augmented generator through DMD and streaming long-tuning. We train only the retrieval head and reuse it unchanged with frozen Causal-Forcing, Self-Forcing, and LongLive checkpoints in their shared latent space. This adds retrieval to existing AR backbones without AR generator retraining. MemFlow does not evaluate this cross-backbone reuse with frozen generators.
“Video demonstration”
Answer:
Thank you for the suggestion. We have provided video examples for visual comparison.
“Loss ablation”
Answer:
Table 4 already ablates the embedding variants. We add AE + Smooth below to test whether SeqDelta helps beyond smoothing, and reproduce the full 30s Causal-Forcing table:
Full gains 0.62, 0.31, 0.18, and 0.65 points over AE + Smooth across the four metrics, respectively. Removing smoothing (AE + SeqDelta) also lowers all four scores. Both terms help in this comparison.
“Performance across video durations”
Answer:
We summarize the Causal-Forcing imaging-quality results across durations below:
The gain grows with duration for Causal-Forcing imaging quality. It varies by backbone and metric: LongLive 120s imaging quality ties Base, and Causal-Forcing 120s background consistency falls from 94.66 to 94.08.
“Figure 2 and actual retrieval selections”
Answer:
Figure 2 is a schematic. The added decoded visualization shows actual retrieval selections, with history indices and generated outputs. Each row shows its own rollout.
“Retrieval relevance and redundancy”
Answer:
Redundancy also arises in local-window context: adjacent frames often repeat nearly identical content. The visualization shows this repetition in the baseline. Retrieval gives access to earlier references when recent context drifts; Table 3 and the actual selections show cleaner context for LongLive-RAG under the evaluated protocol.
Similarity scoring favors relevant history. SeqDelta discourages excessive similarity between neighboring entries, and smoothing keeps the embeddings smooth over time.
“Reddit citation and formal references”
Answer:
We agree that the technical discussion needs formal references. The Reddit citation acknowledges the original NTK-aware RoPE proposal, also cited by StreamingLLM [3] and YaRN [4]. We will add the formal references while keeping this attribution.
References
[1] VideoRAG.
[2] MemFlow, Sections 3 and 3.1.
[3] Xiao et al. Efficient Streaming Language Models with Attention Sinks, 2023.
[4] Peng et al. YaRN: Efficient Context Window Extension of Large Language Models, 2023.