- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VisaRAG, a retrieval augmented framework for future video generation that treats previously observed action outcome transitions as external dynamics memory. It introduces STR for future directed retrieval, CAST for cross source dynamics transfer, and CMI for conditioning a frozen video generator on retrieved dynamics. The paper also introduces VisaBench, which contains visually similar interactions with different outcomes and uses outcome neutral conditions to evaluate whether retrieved experience can steer future generation.
Critical strengths
- The paper goes beyond using retrieval as a motion or appearance prior and instead treats retrieved videos as evidence for which future outcome should occur.
- VisaBench uses minimal outcome pairs, same outcome controls, and outcome neutral text, which provides a meaningful setting for evaluating future specific retrieval.
- VisaRAG is evaluated across multiple video generators and is compared with retrieval based baselines using matched backbones when possible. It consistently improves action correctness and video quality.
- Counterfactual memory learning is a useful idea. Holding the query, target, noise, and other conditions fixed while changing only the memory provides a principled way to encourage the generator to actually use retrieved dynamics.
Critical weaknesses
- Figure 4 and Table 2 present CAST as part of the VisaRAG pipeline and attribute performance gains to it. However, the method section explicitly states that CAST is not used in any reported generation experiment and that CMI directly receives the retrieved dynamics. This makes the component attribution in the main ablation difficult to interpret.
- CMI has a notable training and inference mismatch. During training, the positive memory is constructed from the target future itself, while at inference the model receives dynamics from a retrieved external video. It is unclear whether the gains would remain if CMI were trained using cross video compatible memories that better match the inference setting.
- The benchmark and generated videos are evaluated by ten university students, but the paper does not clearly report inter rater agreement or uncertainty for these judgments. This is important because action compatibility and future suitability are central evaluation criteria.
Questions for authors
- Could the authors clarify the apparent inconsistency regarding CAST? The paper states that CAST is not used in the reported end to end pipeline, while Table 2 presents performance gains from adding CAST and labels the full VisaRAG model as including it.
- Have the authors trained CMI using compatible memories from different videos rather than using the target future itself as the positive memory? How much does performance change under this more inference matched training setting?
- Could the authors report inter rater agreement and uncertainty for the human evaluations used in VisaBench and the generated video assessment?