acceptodds
Withdrawn from ICLR 2027

VisaRAG: Retrieval-Augmented Dynamics Memory for Future Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Future video generation is ambiguous: visually similar initial states can evolve toward different outcomes, even under the same broad action description. Recent retrieval-augmented methods improve motion realism, physical plausibility, and temporal consistency, but using retrieved experiences to distinguish between such outcomes remains less explored. We propose VisaRAG, a framework that uses observed action-outcome transitions as external dynamics memory for generating video continuations. Given an observed video prefix, VisaRAG retrieves relevant experiences, encodes their temporal changes while suppressing static appearance, and injects the resulting memories through lightweight motion branches into a frozen video generator. A counterfactual learning objective trains the memory-conditioning components to favor memories consistent with the target outcome over alternatives, with the target video and all other generation conditions held fixed. To measure whether retrieved dynamics steer the generated outcome, we further introduce VisaBench, a benchmark that pairs visually similar interactions leading to different outcomes with same-outcome controls, and that conditions every sample on shared, outcome-neutral text so the prompt cannot reveal the target outcome. Across multiple benchmarks, VisaRAG consistently improves both action correctness and the quality of the resulting motion, demonstrating more accurate and realistic future video generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation, Future Prediction, Retrieval-Augmented Generation, Video Retrieval, Future Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VuixOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VisaRAG, a retrieval augmented framework for future video generation that treats previously observed action outcome transitions as external dynamics memory. It introduces STR for future directed retrieval, CAST for cross source dynamics transfer, and CMI for conditioning a frozen video generator on retrieved dynamics. The paper also introduces VisaBench, which contains visually similar interactions with different outcomes and uses outcome neutral conditions to evaluate whether retrieved experience can steer future generation.

Critical strengths
  1. The paper goes beyond using retrieval as a motion or appearance prior and instead treats retrieved videos as evidence for which future outcome should occur.
  2. VisaBench uses minimal outcome pairs, same outcome controls, and outcome neutral text, which provides a meaningful setting for evaluating future specific retrieval.
  3. VisaRAG is evaluated across multiple video generators and is compared with retrieval based baselines using matched backbones when possible. It consistently improves action correctness and video quality.
  4. Counterfactual memory learning is a useful idea. Holding the query, target, noise, and other conditions fixed while changing only the memory provides a principled way to encourage the generator to actually use retrieved dynamics.
Critical weaknesses
  1. Figure 4 and Table 2 present CAST as part of the VisaRAG pipeline and attribute performance gains to it. However, the method section explicitly states that CAST is not used in any reported generation experiment and that CMI directly receives the retrieved dynamics. This makes the component attribution in the main ablation difficult to interpret.
  2. CMI has a notable training and inference mismatch. During training, the positive memory is constructed from the target future itself, while at inference the model receives dynamics from a retrieved external video. It is unclear whether the gains would remain if CMI were trained using cross video compatible memories that better match the inference setting.
  3. The benchmark and generated videos are evaluated by ten university students, but the paper does not clearly report inter rater agreement or uncertainty for these judgments. This is important because action compatibility and future suitability are central evaluation criteria.
Questions for authors
  1. Could the authors clarify the apparent inconsistency regarding CAST? The paper states that CAST is not used in the reported end to end pipeline, while Table 2 presents performance gains from adding CAST and labels the full VisaRAG model as including it.
  2. Have the authors trained CMI using compatible memories from different videos rather than using the target future itself as the positive memory? How much does performance change under this more inference matched training setting?
  3. Could the authors report inter rater agreement and uncertainty for the human evaluations used in VisaBench and the generated video assessment?
Reviewer_n7Q8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VisaRAG retrieves complete video experiences from an observed prefix using a learned future-directed transition representation. Temporal feature differences from the retrieved experience condition a frozen generator through gated attention branches, trained with compatible and incompatible memories under otherwise matched conditions. The paper also studies a cross-source transfer module, CAST, and introduces VisaBench for action-outcome evaluation across physics, robotics, and driving. Results are reported on several generator backbones and broader physical/perceptual benchmarks. However, the submission gives conflicting accounts of the executed pipeline and evaluation conditions, which affects how these results can be interpreted.

Critical strengths
  1. Separating future-outcome correctness from generic motion realism is a worthwhile research question. The intended VisaBench design includes cross-source query and memory sets, explicit temporal cuts, and outcome-sensitive negatives. Appendix B also distinguishes retrieval queries from the stricter generation-evaluation subset. These are useful considerations for evaluating retrieval beyond appearance matching.

  2. The matched-backbone comparisons in Table 1 are potentially valuable, and the counterfactual training objective holds the target, noise, timestep, and other conditions fixed. This is a sensible way to test the role of memory, provided the corresponding intervention results support that role. I also appreciate that the appendix discusses limitations of transfer and retrieval utility rather than presenting every component as successful.

Critical weaknesses
  1. The central claim that memory content steers the outcome conflicts with the paper's own Appendix F (p. 38). That section states that the injected rows do not realize the retrieved outcome more often, and that an example produces the same video with any of five retrieved memories, attributing improvements to a payload-independent correction. This directly challenges the contribution stated on p. 2. There is also a pipeline inconsistency: Sections 4.1/4.3 and Appendix C repeatedly state that CAST is excluded from all reported end-to-end generation, while Table 2(a) and its discussion attribute progressive generation gains to adding CAST. These are not peripheral presentation issues; they prevent attribution of the reported gains to the proposed retrieval-transfer mechanism.

  2. The outcome-neutral evaluation protocol is not consistently documented. Appendix F says every system receives a caption followed by the intended event, and Figure 5 on p. 35 includes an unintentional-action text describing the eventual fall and landing. This is difficult to reconcile with the claim that the conditioning text cannot reveal the target outcome. If those descriptions were generation inputs, the experiment would test event-conditioned synthesis rather than selection of an unresolved future. If they were only evaluation annotations, that distinction needs to be explicit. Until the actual inputs are established, the action-correctness gains do not substantiate the stated future-prediction task.

Questions for authors
  1. Which pipeline produced Tables 1 and 2: raw retrieved dynamics or CAST outputs, and how many retrieved memories were used? Please reconcile these tables with Appendix C and Appendix F, and provide matched-noise outcome-swap results comparing compatible, incompatible, random, and null memories. In particular, what evidence shows an effect of memory content beyond the trained adapter itself?

  2. Please provide the exact generation prompts and observed frames for the evaluated subsets, clearly separated from judge descriptions and held-out labels. Were the intended event and outcome descriptions mentioned in Appendix F supplied to the generators? If so, can the main results be reported under the claimed shared, outcome-neutral condition?

My recommendation is clear rejection in the current form. The motivation is interesting, but the conflicting experimental accounts concern the main scientific claim, and I cannot resolve them from the submission.

Reviewer_HxZpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VisaRAG, a retrieval-augmented method for video continuation. Given an observed video prefix, the method retrieves relevant video as an evidence for future unfold using a learned STR module, extracts temporal feature differences, and injects them into a frozen video generator through trainable cross-attention. The authors propose a counterfactual objective to encourage the generator to distinguish memories with compatible and incompatible outcomes. The paper also introduces VisaBench for evaluating future-outcome correctness.

Critical strengths
  1. The idea is simple and generally makes sense .
  2. Using previous video experiences to help generate future interactions sounds interesting to me.
  3. The proposed benchmark is useful to evaluate the defined problem introduced in this paper.
Critical weaknesses

I have several major concerns:

  1. The applicability of the method depends on the coverage of the external video database. Although the experiments cover three domains and include cross-source retrieval, it is still unclear whether the memory bank can support more diverse video continuation cases, especially interactions or scenes that are not well represented in the database. I feel it is also necessary to provide an experiment evaluating how the quality of the video database affects generation quality.
  2. While this paper focuses on video generation, neither generated videos nor video samples from the benchmark are provided. This makes it difficult to evaluate the model’s actual performance and understand the effect of each component.
  3. Table 2(b) shows that STR achieves a retrieval accuracy of only 60%. It would be helpful to analyze the cases in which retrieval fails and evaluate how robust the model is to incorrectly retrieved memories.
  4. There are many typos. For example, on page 38, what do “Appendix ??” and “Proposition ??” refer to?
Questions for authors

See the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.