- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes PhyMem, a memory module to improve the physical plausibility of the generated videos for existing video diffusion models. PhyMem retrieves the top-K videos from a video bank based on the text prompt and feeds them into the V-JEPA2 encoder to extract features and compress the memory tokens. These tokens will serve as additional key/value and be injected into the Wan2,1 self-attention. Experiments demonstrate improvement over the backbone model on two benchmarks: PhyGenBench and VideoPhy2.
Critical strengths
- The prompt cannot sufficiently describe the physical information, and retrieving real physical dynamics as an external condition is a rational solution.
- The explanation of the method and experiments is clear and easy to follow.
- The ablation study on shuffled retrieval, no-reference, and retrieval budget is convincing and provides informative insight.
Critical weaknesses
- The method is only applied to Wan2.1-1.3B, which cannot show the generalization ability of this method. Also, usually, large models, such as 14B Wan2.1 or Wan2.2 and Minimax H3, themselves already possess physical following capability. Applying this method to these models will make this work more meaningful.
- The comparison is out of date. Recently, there are more and more strong video generation models and some works specifically designed for this task. All of them should be included for comparison.
- Matched retrieval is better than shuffled retrieval can only demonstrate that the videos relative to the prompt are helpful but cannot show that the model leverages the physical dynamics. Also, V-JEPA2 memory shows better performance than Wan VAE memory can only show that V-JEPA2 representation is better.
- Compared with DiT-Mem, the method's novelty is limited. PhyMem and DiT-Mem share the major architecture.
Questions for authors
- Have the training target videos been removed from the retrieval bank?
- How many samples are generated for each prompt in Table 2?
7 comments
Thank you again for your constructive feedback. The revised manuscript now includes the Wan2.2 extension and VideoREPA comparison (Appendix D), together with reference-dynamics controls (Appendix E). Our response also clarifies the contribution beyond DiT-Mem. We would welcome your thoughts and would be happy to clarify any remaining concerns.
Thank you again for suggesting that we evaluate PhyMem on a larger model. As the discussion draws to a close, we would be glad to address any remaining questions about the Wan2.2 results or the reference-dynamics experiments (Appendices D and E).
Response to Reviewer r5A7
Thank you for the careful review and for recognizing the value of the retrieval ablations. We tested PhyMem on a larger generator, added a VideoREPA comparison, and examined reference dynamics through two complementary controls.
1. Further gains on a larger generator. The key question is whether PhyMem can further improve a larger generator that already has stronger native capabilities. We directly test this by extending PhyMem to Wan2.2-TI2V-5B, keeping the pretrained V-JEPA2 encoder and generator frozen and training the memory interface for the new generator. On all 160 PhyGenBench prompts:
Gains appear across all three evaluation stages. Predictive memory complements the capabilities of the larger Wan2.2 generator and provides additional improvements in physical plausibility. Appendix D gives the settings and full domain and stage results.
2. Comparison with another enhancement method. We evaluated VideoREPA 5B using the authors' publicly released LoRA weights on CogVideoX-5B, on the same 160 prompts with a common output protocol and Kimi evaluator.
The Wan2.2 pair provides a within-backbone comparison; VideoREPA adds a comparison with another physics-oriented enhancement system, using its own backbone and sampler. Appendix D reports the complete settings and results.
3. Does memory guidance use reference dynamics? For each of 40 prompts, we carefully selected and trimmed three online reference videos for each of the two conditions, A and B, retaining the relevant interaction or motion segments. We then construct three reference conditions:
All conditions use the same V-JEPA2 encoder, adapter, Wan2.1 + PhyMem generator, and sampling settings. Five raters score all 120 outputs across 40 prompts, with condition labels hidden and video positions randomized. Human physical-plausibility (PC) ratings use a 1–4 scale. Relative score change (%) = paired gain / comparison mean × 100.
A versus C tests temporal variation: The same reference sources are used, with temporal variation removed in C. A versus B tests target-compatible dynamics beyond semantic similarity: A and B have nearly identical semantic-adherence scores and identical mean interaction-completion and subject-motion scores. Together, these comparisons support the contribution of temporal content and dynamics aligned with the target interaction.
Appendix E, “Human Evaluation of Reference Dynamics,” gives the reference-construction procedure, concrete A/B examples, and human-evaluation protocol. Its mean-score and paired-PC comparison tables provide the full ratings and paired results.
4. Contribution beyond DiT-Mem. The central contribution is transferring latent-predictive video representations into an independently pretrained video generator. A compact, gated memory interface connects the frozen V-JEPA2 encoder and generator.
Beyond this cross-model connection, the training design and the study of memory mechanisms are also important parts of the contribution. For training, we combine learnable memory strength, reference dropout, and soft gating to calibrate the transfer of external predictive features. Section 3 and Appendix C describe the interface and training recipe.
To understand what makes the transferred memory effective, we examine representation source, reference alignment, and memory budget. The new reference interventions hold the encoder and generator fixed while changing the reference content, testing the roles of temporal variation and target-compatible dynamics (Appendix E).
Together with the 33.9M-parameter interface trained on 792 videos and the Wan2.2 extension, these contributions provide a practical, lightweight, plug-and-play approach to reusing predictive representations for generation.
Q1. Were training targets removed from the retrieval bank? We exclude each target video's own ID before selecting references. The saved training lists contain 0/792 exact self-ID matches. Appendix C.1 describes the per-query exclusion.
Q2. How many samples per prompt in Table 2? Table 2 reports one video per prompt and method, across 160 PhyGenBench and 120 VideoPhy2 prompts.