acceptodds
Under review as a conference paper at ICLR 2027

PhyMem: Retrieved Predictive Memories for Physically Plausible Video Generation

Abstract

Text-to-video diffusion models can synthesize visually compelling clips, yet they remain unreliable when prompts require coherent motion, causal interactions, or physically plausible temporal evolution. A key limitation is that text specifies what should happen but provides little evidence about how the scene should physically unfold. We study whether predictive video representations retrieved from real videos can provide external guidance for physically plausible generation with a frozen Transformer-based latent video diffusion backbone. We introduce PhyMem, a plug-in memory framework that (i) retrieves prompt-relevant reference clips, (ii) instantiates world-model memories with predictive representations extracted by a frozen V-JEPA2 encoder, (iii) compresses the resulting spatiotemporal tokens into compact memory tokens, and (iv) injects them into the self-attention layers of a frozen video diffusion backbone through a lightweight trainable memory module, without finetuning the backbone. Training updates only a lightweight memory interface, with M trainable parameters and approximately K training videos, while the video diffusion backbone, text encoder, and generator VAE remain frozen. Across physics-focused evaluations, PhyMem improves physical plausibility and physical commonsense under matched inference settings while largely preserving semantic adherence. Diagnostic ablations further indicate that the gains depend on representation source, retrieval relevance, and retrieval budget: V-JEPA2 memory outperforms native VAE memory, prompt-relevant references are more effective than shuffled ones, and retrieval performance peaks at an intermediate budget. These results suggest that retrieved predictive memories offer a parameter-efficient route toward improving physical plausibility in video generation.

Primary area:
generative models
Keywords:
video generation, video diffusion models, world models, physical plausibility, retrieval-augmented generation, memory-augmented generation, predictive video representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_r5A7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PhyMem, a memory module to improve the physical plausibility of the generated videos for existing video diffusion models. PhyMem retrieves the top-K videos from a video bank based on the text prompt and feeds them into the V-JEPA2 encoder to extract features and compress the memory tokens. These tokens will serve as additional key/value and be injected into the Wan2,1 self-attention. Experiments demonstrate improvement over the backbone model on two benchmarks: PhyGenBench and VideoPhy2.

Critical strengths
  1. The prompt cannot sufficiently describe the physical information, and retrieving real physical dynamics as an external condition is a rational solution.
  2. The explanation of the method and experiments is clear and easy to follow.
  3. The ablation study on shuffled retrieval, no-reference, and retrieval budget is convincing and provides informative insight.
Critical weaknesses
  1. The method is only applied to Wan2.1-1.3B, which cannot show the generalization ability of this method. Also, usually, large models, such as 14B Wan2.1 or Wan2.2 and Minimax H3, themselves already possess physical following capability. Applying this method to these models will make this work more meaningful.
  2. The comparison is out of date. Recently, there are more and more strong video generation models and some works specifically designed for this task. All of them should be included for comparison.
  3. Matched retrieval is better than shuffled retrieval can only demonstrate that the videos relative to the prompt are helpful but cannot show that the model leverages the physical dynamics. Also, V-JEPA2 memory shows better performance than Wan VAE memory can only show that V-JEPA2 representation is better.
  4. Compared with DiT-Mem, the method's novelty is limited. PhyMem and DiT-Mem share the major architecture.
Questions for authors
  1. Have the training target videos been removed from the retrieval bank?
  2. How many samples are generated for each prompt in Table 2?
7 comments
AuthorsOpenReview ↗

Response to Reviewer r5A7

Thank you for the careful review and for recognizing the value of the retrieval ablations. We tested PhyMem on a larger generator, added a VideoREPA comparison, and examined reference dynamics through two complementary controls.

1. Further gains on a larger generator. The key question is whether PhyMem can further improve a larger generator that already has stronger native capabilities. We directly test this by extending PhyMem to Wan2.2-TI2V-5B, keeping the pretrained V-JEPA2 encoder and generator frozen and training the memory interface for the new generator. On all 160 PhyGenBench prompts:

MetricWan2.2Wan2.2 + PhyMem (relative gain)
Combined0.39790.4229 (+6.3%)
Four-domain macro0.37370.4042 (+8.2%)

Gains appear across all three evaluation stages. Predictive memory complements the capabilities of the larger Wan2.2 generator and provides additional improvements in physical plausibility. Appendix D gives the settings and full domain and stage results.

2. Comparison with another enhancement method. We evaluated VideoREPA 5B using the authors' publicly released LoRA weights on CogVideoX-5B, on the same 160 prompts with a common output protocol and Kimi evaluator.

SystemCombined score
VideoREPA 5B0.3250
Wan2.2 + PhyMem0.4229 (+30.1% relative)

The Wan2.2 pair provides a within-backbone comparison; VideoREPA adds a comparison with another physics-oriented enhancement system, using its own backbone and sampler. Appendix D reports the complete settings and results.

3. Does memory guidance use reference dynamics? For each of 40 prompts, we carefully selected and trimmed three online reference videos for each of the two conditions, A and B, retaining the relevant interaction or motion segments. We then construct three reference conditions:

  • A: the three selected videos depict the target interaction with compatible dynamics.
  • B: the three other selected videos share similar objects or scene context but depict a different interaction or motion pattern.
  • C: three static clips derived directly from A, each repeating one fixed frame from its corresponding source video.

All conditions use the same V-JEPA2 encoder, adapter, Wan2.1 + PhyMem generator, and sampling settings. Five raters score all 120 outputs across 40 prompts, with condition labels hidden and video positions randomized. Human physical-plausibility (PC) ratings use a 1–4 scale. Relative score change (%) = paired gain / comparison mean × 100.

ContrastPC scoresPaired gain (points)Relative score change
A/B2.875/2.560+0.315+12.3%
A/C2.875/2.400+0.475+19.8%

A versus C tests temporal variation: The same reference sources are used, with temporal variation removed in C. A versus B tests target-compatible dynamics beyond semantic similarity: A and B have nearly identical semantic-adherence scores and identical mean interaction-completion and subject-motion scores. Together, these comparisons support the contribution of temporal content and dynamics aligned with the target interaction.

Appendix E, “Human Evaluation of Reference Dynamics,” gives the reference-construction procedure, concrete A/B examples, and human-evaluation protocol. Its mean-score and paired-PC comparison tables provide the full ratings and paired results.

4. Contribution beyond DiT-Mem. The central contribution is transferring latent-predictive video representations into an independently pretrained video generator. A compact, gated memory interface connects the frozen V-JEPA2 encoder and generator.

Beyond this cross-model connection, the training design and the study of memory mechanisms are also important parts of the contribution. For training, we combine learnable memory strength, reference dropout, and soft gating to calibrate the transfer of external predictive features. Section 3 and Appendix C describe the interface and training recipe.

To understand what makes the transferred memory effective, we examine representation source, reference alignment, and memory budget. The new reference interventions hold the encoder and generator fixed while changing the reference content, testing the roles of temporal variation and target-compatible dynamics (Appendix E).

Together with the 33.9M-parameter interface trained on 792 videos and the Wan2.2 extension, these contributions provide a practical, lightweight, plug-and-play approach to reusing predictive representations for generation.

Q1. Were training targets removed from the retrieval bank? We exclude each target video's own ID before selecting references. The saved training lists contain 0/792 exact self-ID matches. Appendix C.1 describes the per-query exclusion.

Q2. How many samples per prompt in Table 2? Table 2 reports one video per prompt and method, across 160 PhyGenBench and 120 VideoPhy2 prompts.

Thank you again for your constructive feedback. The revised manuscript now includes the Wan2.2 extension and VideoREPA comparison (Appendix D), together with reference-dynamics controls (Appendix E). Our response also clarifies the contribution beyond DiT-Mem. We would welcome your thoughts and would be happy to clarify any remaining concerns.

Thank you again for suggesting that we evaluate PhyMem on a larger model. As the discussion draws to a close, we would be glad to address any remaining questions about the Wan2.2 results or the reference-dynamics experiments (Appendices D and E).

Reviewer_MPRyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhyMem retrieves caption-matched real videos, extracts frozen V-JEPA2 features, compresses them into memory tokens, and injects them as additional keys and values into a frozen Wan2.1 generator. Only a 33.9M-parameter interface is trained on approximately 800 videos. Experiments on PhyGenBench and VideoPhy2 report improved physical plausibility, supported by memory-source, retrieval-relevance, and retrieval-budget ablations.

Critical strengths

S1. Parameter-efficient adaptation. The backbone, text encoder, VAE, and V-JEPA2 encoder remain frozen; only the 33.9M-parameter memory interface is trained. This offers a practical way to reuse pretrained representations with limited task-specific training data.

S2. Meaningful controlled improvements. Under matched inference settings, PhyGenBench Final improves from 0.48 to 0.55, while VideoPhy2 PC pass rate rises from 51.7% to 61.7% and Joint from 15.8% to 20.8%, with unchanged semantic-adherence pass rate.

S3. Useful diagnostic controls. The VAE-memory, shuffled-reference, and retrieval-budget experiments test distinct design choices. In particular, matched references outperform shuffled references at the same memory budget (0.55 versus 0.47 category macro).

S4. Careful reporting and interpretation. The paper documents inference settings, caching, masking, and human-study counts, reports failures and category regressions, and appropriately limits claims about physical reasoning, cross-backbone generality, and computational efficiency.

Critical weaknesses

W1. Limited methodological novelty. DiT-Mem already injects reference-video memories into a frozen generator. PhyMem mainly changes the feature source and interface, while the comparison lacks matched training data and budgets. Related physics-oriented retrieval methods such as PhysRAG also deserve discussion.

W2. The source of the feature advantage is unclear. V-JEPA2 and VAE memories differ in architecture, pretraining, and input projection. Their comparison therefore does not isolate predictive temporal information. Temporal-order or repeated-frame controls would help distinguish dynamics information from generic semantic features.

W3. Target–reference separation is unspecified. Targets and the retrieval bank both come from OpenVidHD, but target-ID and near-duplicate exclusions are not documented. Since training always retains the top-ranked reference, possible self-retrieval could change the learning problem. Actual valid-reference counts after cache misses are also unreported.

W4. Robustness and human support remain limited. Results use one seed without uncertainty estimates; VideoPhy2 Joint rises from 19 to 25 passing videos among 120 prompts. Human pair-majority outcomes are only 10 wins, 7 losses, and 13 ties, while hard-pair votes favor Wan2.1, 16 to 10. These results leave the consistency of the benefit uncertain.

W5. Practical overhead is unquantified. Fewer trainable parameters do not capture retrieval, reference encoding, caching, and extra attention costs. Although the authors appropriately avoid speed claims, latency, peak memory, and storage measurements are needed to assess the practical trade-off.

Questions for authors
  1. Were target videos and near duplicates excluded from retrieval for each split? Please clarify this protocol and report actual valid-reference counts after cache misses.

  2. Beyond the VAE comparison, what evidence distinguishes the benefit of predictive temporal information from stronger generic pretrained video features?

4 comments
AuthorsOpenReview ↗

Response to Reviewer MPRy (1/2)

Thank you for the detailed review and for recognizing the value of the controlled comparisons and diagnostic experiments. Your questions prompted us to add reference-dynamics controls, expand the human study, and document the retrieval protocol and practical costs more fully.

W1. Contribution and comparison. Our central contribution is transferring latent-predictive video representations into an independently pretrained video generator. A compact, gated memory interface connects the two representation spaces while keeping both pretrained models frozen.

The training design and the investigation of memory mechanisms are also important parts of this contribution. For training, we combine learnable memory strength, reference dropout, and soft gating to regulate the transfer of external predictive features. Section 3 and Appendix C describe the interface and training recipe.

To investigate what makes this memory effective, we examine representation source, reference alignment, and memory budget. The new reference interventions additionally hold the encoder and generator fixed while changing the reference content, testing the role of temporal variation and compatibility with the target dynamics (Appendix E).

The resource comparison is particularly relevant here. On the same 80 PhyGenBench prompts and Wan2.1-1.3B backbone:

MeasureDiT-MemPhyMemChange
Final score0.53330.5500+3.1% relative
Task-specific videos10K0.8K12.5× fewer (−92.0%)
Trainable memory parameters78.8M33.9M2.3× fewer (−57.0%)

Thus, PhyMem's performance does not rely on a larger task-specific dataset or a larger trainable module. These results highlight the practical value of transferring frozen predictive representations with limited task-specific adaptation. DiT-Mem reports training on eight A100 GPUs in its Appendix B; we also document our measured training and inference costs in Appendix C.4. Section 4.2.2 and the accompanying efficiency figure present the performance, data, and parameter comparison using each checkpoint's original training recipe.

We also examined PhysRAG as suggested, but the public checkpoint we checked lacked complete adapter/injection weights. We therefore evaluated another physics-oriented enhancement method, VideoREPA 5B, using the authors' publicly released LoRA weights on CogVideoX-5B. On all 160 PhyGenBench prompts under a common output and evaluation protocol:

SystemCombined score
Wan2.20.3979
Wan2.2 + PhyMem0.4229
VideoREPA 5B0.3250

Appendix D gives the settings and full results for this cross-system comparison.

W2 / Q2. Temporal content versus generic features. For each of 40 prompts, we carefully selected and trimmed three online reference videos for each of the two conditions, A and B, retaining the relevant interaction or motion segments. A depicts the target interaction with compatible dynamics. B shares similar objects or scene context but depicts a different interaction or motion pattern. C is constructed directly from A: we take one fixed frame from each of its three videos and repeat it to make three static clips.

This construction gives two complementary reference-control comparisons: dynamic versus repeated-frame references (A/C), and dynamically matched versus semantically similar but dynamically mismatched references (A/B). They share the same matched-reference condition A and are reported in the newly added Appendix E, “Human Evaluation of Reference Dynamics.” All conditions use the same V-JEPA2 encoder, adapter, Wan2.1 + PhyMem generator, and sampling settings.

Five raters scored the A/B/C outputs for each prompt, with condition labels hidden and video positions randomized. Physical plausibility (PC) was rated on a 1–4 scale. Relative score change (%) = paired gain / comparison mean × 100.

ComparisonPC scoresPaired gain (points)Relative score change
A vs. B2.875 / 2.560+0.315+12.3%
A vs. C2.875 / 2.400+0.475+19.8%

1. Does temporal variation help? A versus C. Because C is derived from the same reference sources as A, this comparison tests the benefit of time-varying content over fixed snapshots.

2. Is semantic similarity sufficient? A versus B. This comparison retains similar objects or scene context while changing the reference dynamics. The higher PC for A supports the value of dynamics that match the target interaction.

Together, the comparisons support the contribution of temporal variation and target-compatible dynamics within the transferred V-JEPA2 representations. Appendix E details the reference construction and human-evaluation protocol. Its mean-score table reports the four rating dimensions; its paired-PC comparison table reports PC gains and prompt-level wins/ties/losses.

We address W3–W5 in Part 2.

AuthorsOpenReview ↗

Response to Reviewer MPRy (2/2)

Continuing Part 1, we address W3–W5 below.

W3 / Q1. Target exclusion and valid-reference counts. We exclude each target video's own ID before selecting references. Checking the saved retrieval lists confirms:

Target listExact self-ID matches
Effective training targets0 / 792
Full target list0 / 1,000

This is an exact-ID exclusion; distinct clips with similar visual content can remain.

After accounting for missing cache files and before stochastic reference dropout:

Valid references per training targetNumber of targets
Five782
Four10

Missing slots are zero-filled and marked invalid. Inference is configured to use three references. Appendices C.1 and C.8 describe the target-exclusion and cache-handling rules.

W4. Robustness and human evidence. The submitted main benchmark results use one generation seed, with one evaluated video per prompt and method.

To strengthen the human evidence, we expanded the original 30-pair preference test to all 120 VideoPhy2 prompt pairs, including 60 hard prompts, evaluated by five raters. The new study scores physical plausibility, semantic adherence, target-interaction completion, and subject-motion intensity separately, rather than relying solely on an overall preference judgment. This distinguishes physical correctness from prompt matching, whether the requested interaction occurs, and the amount of motion.

Relative score change (%) = paired gain / comparison mean × 100.

PC (1–4)Wan2.1Wan2.1 + PhyMemPaired gain (points)Relative score change
All prompts2.4332.722+0.288+11.8%
Hard prompts2.3432.577+0.233+10.0%

The expanded evaluation therefore provides stronger human evidence of physical improvements, including on difficult interactions. Appendix B.5 provides the scoring rubric, results for all four dimensions, and paired statistical analysis. Robustness across generation seeds remains to be evaluated.

W5. Measured overhead. We have added matched inference-latency and peak GPU-memory measurements for native Wan and PhyMem, covering retrieval, token loading, memory adaptation, and memory-augmented denoising in the cached-reference-token path. We also report offline reference-encoding costs, training resources, and storage requirements. Reference features can be precomputed and reused, avoiding repeated online encoding. Appendix C.4 provides the measurement protocol, comparisons of the cache paths, and detailed cost tables.

Thanks for the response. My concerns have been addressed and I'll keep my rating.

Thank you for reviewing our response. We appreciate your constructive feedback and continued support!

Reviewer_kiuqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies physically plausible video generation by retrieving real videos, encoding them with V-JEPA2, and using their features as memory for a frozen video diffusion model. I find the topic interesting, the method intuitive, and the results encouraging. My main concerns are the limited backbone coverage, comparisons with existing methods, and the reliability of the physical evaluation.

Critical strengths
  1. The paper addresses a timely and interesting topic: improving the physical plausibility of generated videos.

  2. The writing is clear and easy to follow. The motivation and method are straightforward to understand.

  3. The method is intuitive, and keeping the generator frozen while training only a small memory interface is a practical design.

  4. The reported improvements on two physics-focused benchmarks are encouraging, with little change in semantic adherence. The matched Wan comparison and the ablations on memory source, reference relevance, and retrieval budget provide useful support.

Critical weaknesses
  1. The method is evaluated only on Wan2.1-T2V-1.3B. Although the interface is intended for compatible video diffusion Transformers, its effectiveness on other backbones or model scales remains untested. additional backbones would help establish the generality of the approach.

  2. Comparisons with other methods for improving physical plausibility are limited. The related work discusses several such approaches, but the experiments mainly compare with the native Wan baselines. Comparisons under compatible settings, or an explanation of why they are not applicable, would help assess the contribution beyond existing approaches.

  3. The physical improvements rely mainly on VLM-based evaluation. I would like to see whether the gains persist when controlling for motion dynamic degree. I appreciate the discussion of evaluation limitations and the inclusion of a human study, but it covers only 30 pairs: 10 majority wins for PhyMem, 7 for Wan, and 13 ties. The question also mixes action fidelity, physical plausibility, and realism. A larger human study focused on physical correctness would make the results more convincing.

Questions for authors
  1. Does the method also improve physical plausibility on another backbone or model scale? What changes would be needed to use the memory interface with a different generator?

  2. Beyond the released DiT-Mem checkpoint, could the author compare with other relevant methods for improving physical plausibility under compatible settings? If these comparisons are not applicable, could you explain why?

  3. Do the gains persist when controlling for motion dynamic degree? For example reporting dynamic degree metric or a more rigorous human study would help.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kiuq

Thank you for your encouraging assessment and helpful suggestions on backbone coverage, comparisons, and evaluation. We have addressed these points with a Wan2.2 extension, a VideoREPA comparison, and an expanded human study that rates physical plausibility (PC, on a 1–4 scale), semantic adherence, interaction completion, and motion intensity separately.

1. Another backbone and model scale. We extended PhyMem from Wan2.1-T2V-1.3B to Wan2.2-TI2V-5B. The extension retains the frozen V-JEPA2 encoder and the same memory-injection principle, adapting the interface to the new generator's feature dimensions and attention implementation. The Wan2.2 memory adapter is trained while the pretrained encoder and generator remain frozen.

On all 160 PhyGenBench prompts:

MetricWan2.2Wan2.2 + PhyMemRelative gain
Combined0.39790.4229+6.3%
Four-domain macro0.37370.4042+8.2%

Improvements appear in all three evaluation stages and in Mechanics, Optics, and Thermal. These results extend PhyMem to a second, larger generator. Appendix D gives the adaptation, generation, and evaluation protocol.

2. Other enhancement methods. We evaluated VideoREPA 5B using the authors' publicly released LoRA weights on CogVideoX-5B, on the same 160 prompts and with matched output resolution, frame count, generation seed, sampling-step count, and fixed guidance scale. All outputs were scored using the same Kimi backend and evaluation procedure.

MetricVideoREPA 5BWan2.2 + PhyMemRelative gain
Combined0.32500.4229+30.1%
Four-domain macro0.31320.4042+29.1%

VideoREPA uses CogVideoX-5B and its own sampler, so this is a comparison between enhancement systems with different backbones. Appendix D provides the complete settings and results.

3. Physical correctness versus motion degree. To examine whether higher physical scores accompany a change in motion intensity, we expanded the original 30-pair preference test to all 120 VideoPhy2 prompt pairs, including 60 hard prompts. Five raters assess the native Wan2.1 and Wan2.1 + PhyMem videos on four separate dimensions: physical plausibility, semantic adherence, target-interaction completion, and subject-motion intensity. This separates the physical judgment from prompt matching, action completion, and the amount of motion.

Relative score change (%) = paired gain / comparison mean × 100.

Human ratingWan2.1Wan2.1 + PhyMemPaired change (points)Relative score change
PC (1–4)2.4332.722+0.288+11.8%
Subject motion (1–4)3.7933.792−0.001−0.02%
Interaction (0–2)1.9471.945−0.002−0.09%

The PC gain therefore accompanies similar rated motion and action completion. Appendix B.5 provides the rubric and full results.

We also performed a separate five-rater reference-intervention study on 40 prompts. For each prompt, we carefully selected and trimmed three online reference videos for each of the two conditions, A and B, retaining the relevant interaction or motion segments. A matches the target dynamics, B has similar objects/settings but different dynamics, and C repeats a fixed frame from each of the three A references. All outputs use Wan2.1 + PhyMem.

ConditionPCSAInteractionMotion
A2.8752.6401.4753.080
B2.5602.6351.4753.080
C2.4002.5951.4203.085

PC, SA, and motion use 1–4 scales; interaction uses 0–2.

ComparisonPC scoresPaired gain (points)Relative score change
A vs. B2.875 / 2.560+0.315+12.3%
A vs. C2.875 / 2.400+0.475+19.8%

This tests how the reference dynamics affect physical plausibility while tracking motion intensity as a separate human-rating dimension. Appendix E gives the reference examples and paired results.

Thank you for the detailed response and additional experiments. The Wan2.2 extension and expanded human study strengthen the evaluation and address most of my concerns. I remain supportive of acceptance and will keep my rating of 3.

Thank you so much for reviewing our additional experiments and for your continued support. Your constructive suggestions helped us strengthen the evaluation and improve the paper!

Reviewer_4SwuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhyMem retrieves reference videos, encodes them with frozen V-JEPA2 features, and injects compressed memory tokens into a frozen Wan2.1 backbone through a small trainable interface. It improves automatic physics scores on PhyGenBench and VideoPhy2 under matched inference settings. Ablations examine the representation source, reference relevance, and number of retrieved videos.

Critical strengths
  1. Using real videos as examples of how a scene evolves is a reasonable complement to text conditioning. Improving a frozen generator with a 33.9M-parameter interface and approximately 800 task-specific training videos is a useful practical result.
  2. The reference and representation comparisons are informative. Relevant references outperform shuffled references, and V-JEPA2 features outperform the VAE alternative. These results support the usefulness of both retrieved content and the choice of video representation.
Critical weaknesses
  1. The methodological advance over DiT-Mem is limited in its current form: retrieving video memories and injecting them into a frozen diffusion backbone are already established. V-JEPA2 is a sensible new memory source, but the existing comparison does not show a substantial advantage over the prior method under comparable training conditions. The evidence is stronger for a useful feature choice than for a significant advance in physics-aware generation.
  2. The physical benefit needs stronger confirmation on difficult interactions. Automatic scores improve, but the small human study has many ties and does not show a clear benefit on its harder prompts. It remains uncertain whether the retrieved memories consistently improve difficult physical behavior beyond helping the generator produce familiar, plausible-looking events.
  3. The retrieval experiments do not directly test semantically similar references with incompatible physical dynamics. The shuffled-reference control and retrieval-budget study are useful, but leave unclear how the method behaves when retrieval matches the objects or setting while supplying the wrong interaction. This is an important practical failure mode for a system that relies on retrieved physical examples.
Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4Swu

Thank you for the thoughtful review and for highlighting the need for stronger evidence on difficult interactions and reference dynamics. In response, we have expanded the human study, added targeted reference controls, and extended PhyMem to Wan2.2-TI2V-5B with a VideoREPA comparison.

1. Contribution beyond DiT-Mem. The central contribution is transferring latent-predictive video representations into an independently pretrained video generator. A compact, gated memory interface connects the frozen V-JEPA2 encoder and generator.

Beyond this cross-model connection, we view the training design and the study of memory mechanisms as equally important parts of the contribution. For training, our recipe combines learnable memory strength, reference dropout, and soft gating to calibrate the transfer of external predictive features. Section 3 and Appendix C describe the interface and training recipe.

To understand the mechanisms behind the improvement, we examine how representation source, reference alignment, budget, and reference dynamics affect generation. The new interventions hold the encoder and generator fixed while changing the reference content, providing direct evidence about what makes predictive memory useful.

This transfer is also lightweight. On the same 80 PhyGenBench prompts and Wan2.1-1.3B backbone:

MeasureDiT-MemPhyMemChange
Final score0.53330.5500+3.1% relative
Task-specific videos10K0.8K12.5× fewer (−92.0%)
Trainable memory parameters78.8M33.9M2.3× fewer (−57.0%)

This comparison uses each checkpoint's original training recipe and highlights the practical value of reusing frozen predictive representations with limited task-specific adaptation. Section 4.2.2 and the accompanying efficiency figure present this comparison. DiT-Mem reports training on eight A100 GPUs in its Appendix B; we document our measured training and inference costs in Appendix C.4.

2. Difficult physical interactions. We expanded the original 30-pair preference test to all 120 VideoPhy2 prompt pairs, including 60 hard prompts, and introduced separate video-level ratings. Five raters assess physical plausibility (PC, rated on a 1–4 scale), semantic adherence, target-interaction completion, and subject motion as distinct dimensions. This separates physical correctness from prompt matching and whether the requested interaction occurs, providing a more targeted evaluation of difficult physical behavior. Relative score change (%) = paired gain / comparison mean × 100.

PC (1–4)Wan2.1Wan2.1 + PhyMemPaired gain (points)Relative score change
All prompts2.4332.722+0.288+11.8%
Hard prompts2.3432.577+0.233+10.0%

The PC gain accompanies nearly unchanged interaction completion and rated motion, supporting improved physical consistency within the depicted interactions. Rater comments also identify concrete differences, such as contact preceding a toy car’s movement and visible securing straps during a crane lift, rather than only a more plausible overall appearance. The rubric and full results are in Appendix B.5.

3. Semantically similar but incompatible dynamics. For each of 40 prompts, we carefully selected and trimmed three online reference videos for each of the two conditions, A and B, retaining the relevant interaction or motion segments. A depicts the target interaction with compatible dynamics; B shares similar objects/settings but depicts a different interaction or motion pattern. C is derived directly from A by repeating one fixed frame from each of its three reference videos. These conditions provide two complementary controls: A/B tests target-compatible dynamics beyond semantic similarity, and A/C tests time-varying content against fixed snapshots.

Five raters evaluated all 120 outputs with condition labels hidden and video positions randomized.

ComparisonPC scoresPaired gain (points)Relative score change
A vs. B2.875 / 2.560+0.315+12.3%
A vs. C2.875 / 2.400+0.475+19.8%

A and B have nearly identical semantic adherence and identical mean interaction completion and rated motion. These results support the importance of matching the target interaction and retaining time-varying reference content. Appendix E gives the reference-construction procedure, concrete A/B examples, and full results.

Thank you again for your constructive feedback. We have expanded the human evaluation of difficult interactions (Appendix B.5) and added reference-dynamics controls (Appendix E) in the revised manuscript, with changes marked in blue. We would welcome your thoughts on these additions and would be happy to clarify any remaining concerns.

Thank you again for your thoughtful comments on difficult physical interactions. As the discussion draws to a close, we would be happy to clarify any remaining questions about the expanded human study or the reference-dynamics controls (Appendices B.5 and E).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.