acceptodds
Under review as a conference paper at ICLR 2027

DiPhy: Distilling Transferable Physical Knowledge for Physics-Aware Video Generation

Abstract

Recent generative video models have achieved impressive visual quality and increasingly realistic video synthesis, yet they still struggle to generate videos that obey plausible physical dynamics. Existing approaches typically incorporate physical information through textual descriptions, trajectory constraints, or additional conditioning signals, whereas how physical knowledge can be effectively mined and transferred to video generation has not yet been sufficiently investigated. Facing this challenge, we propose DiPhy, which distills transferable physical knowledge into a diffusion model for physically plausible video generation. Specifically, a JEPA-based teacher first mines physics-related cues from videos under semantic guidance. These cues are organized into a structured physical codebook that serves as a compact shared space for transferable physical knowledge, which is then distilled into a student module for text-only physical reasoning without video access. The inferred physical codes are subsequently injected into the diffusion backbone through a lightweight physical branch, providing explicit physical guidance during generation. Extensive experiments on physics-oriented video benchmarks demonstrate that our DiPhy improves physical consistency while preserving semantic alignment and overall video quality, with physical commonsense scores improving by approximately 9% and Joint performance by over 10% on VideoPhy2. Moreover, our DiPhy enables efficient video generation with relatively short inference time.

Primary area:
generative models
Keywords:
video generation, physical consistency, physics-informed generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ePYAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiPhy aims to improve the physical plausibility of text-conditioned video generation by transferring video-derived representations through a shared codebook. A teacher combines frozen V-JEPA2 video features with VLM-derived semantic guidance and learns a 64-entry codebook using video-feature and semantic-feature reconstruction. A student predicts routing probabilities over the frozen codebook from text and VLM semantic conditions, using global-token alignment and routing-distribution distillation. Eight selected code vectors condition a lightweight physical branch that injects information into a frozen video diffusion backbone; a semantic adapter augments the text condition.

Training uses 10K videos sampled from WISA-80K. On VideoPhy2, DiPhy improves physical commonsense (PC) from 56.2 to 61.8 on CogVideoX-2B and from 58.0 to 63.3 on Wan2.1-1.3B. Joint semantic-and-physical pass rates improve from 23.3 to 25.8 and from 21.7 to 24.7, respectively. VBench and component, teacher-design, and reconstruction-weight ablations provide additional evaluation.

Critical strengths

S1. Coherent design for transferring video-derived priors. The shared codebook connects a video-conditioned teacher to a student using generation conditions. Semantic reconstruction helps make the teacher's representation predictable by the student.

S2. Practical integration with pretrained generators. A lightweight conditioning branch and semantic adapter augment a frozen generation backbone, and inference requires no reference video.

S3. Consistent reported gains across two backbones. DiPhy improves both physical commonsense and Joint performance on CogVideoX and Wan, with Joint increasing by 2.5 and 3.0 percentage points, respectively. Aggregate VBench quality also improves.

S4. Informative analysis of the teacher objective. Adding semantic reconstruction improves reported teacher–student alignment from 0.79 to 0.93 (Table 5), alongside higher generation scores (Table 6). This illustrates a useful distinction between reconstructing video features and making them predictable by the student.

Critical weaknesses

W1. Complex pipeline. Three training stages, VLM/first-frame preprocessing, codebook routing, and separate physical/semantic branches add implementation and training complexity. Timings with cached features leave the full preprocessing cost unclear.

W2. Insufficient motivation for the codebook. The paper does not establish why discrete codes are needed over continuous-feature distillation. Reconstruction and routing correlations do not isolate physical dynamics from appearance or semantics.

W3. Limited discrimination in some ablations. The semantic adapter and cross-attention improve Joint by only 0.8 and 0.6 points. Their necessity remains unclear without uncertainty estimates, although physical-branch and teacher ablations show larger PC drops.

W4. No human evaluation. Physical plausibility is assessed through VideoPhy2's automatic judge and selected qualitative examples. No human study independently validates whether the reported gains yield perceptible physical improvements.

W5. Unclear incremental novelty. VideoREPA already distills video representations. PhysAlign (Li et al., ECCV 2026) uses a physical codebook and predicts physical slots from text and a first frame. DiPhy's semantic-guided teacher and routing distillation need clearer differentiation.

W6. Modest advantages over existing methods. PC is below VideoREPA on CogVideoX (61.8 versus 62.7) and only 0.8 points above PhysVid on Wan. Joint advantages are 2.6 and 1.9 points, without uncertainty estimates.

Questions for authors
  1. What does the routed codebook add over a continuous-feature predictor under matched supervision and conditioning, and how does this contribution differ from PhysAlign?

  2. Under matched training budgets and conditioning, are the baseline and ablation gains consistent across seeds or supported by paired uncertainty estimates?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ePYA [1/4]

Thank you very much for your careful and constructive comments. We appreciate the opportunity to clarify the design of DiPhy and provide additional experiments addressing the concerns about the physical codebook, evaluation, and experimental reliability.

W1. Overall Pipeline

W1. Complex pipeline. Three training stages, VLM/first-frame preprocessing, codebook routing, and separate physical/semantic branches add implementation and training complexity. Timings with cached features leave the full preprocessing cost unclear.

Thanks! The three stages serve complementary purposes: Stage A mines and organizes reusable physical knowledge into a codebook; Stage B distills this knowledge into a student that predicts physical codes from image and text conditions; and Stage C injects the predicted codes into the frozen video generator through a lightweight physical branch. Overall, this staged design enables reusable physical knowledge to be learned and subsequently transferred to video generation.

Before training, a frozen V-JEPA2 encoder extracts video features offline for teacher-side physical knowledge mining, and this is its only use in the pipeline. At inference, Qwen3-VL processes the initial frame and text prompt to produce physical descriptions and semantic features; its measured additional cost is 3.99 seconds per sample. FLUX is only used when a benchmark provides text prompts without initial frames, and is not required when an initial image is already available. The results on Wan2.1-1.3B, which generates a 6-second video, are reported below:

MethodsParamsPeak MemoryOverall LatencyQwen3-VL Additional CostPC ↑
Wan2.1-1.3B1.418B18.39GB872.33s--58.0
DiPhy1.444B20.26GB934.09s3.99s63.3
Overhead+1.83%+1.87GB+7.08%----

Notably, the overall latency accounts for the complete inference pipeline, including Qwen3-VL processing, stage b(Distilling Transferable Physical Knowledge) and stage c(Physical Knowledge Injection). These results demonstrate that our DiPhy achieves improved physical consistency with relatively modest additional computational overhead. We will report these efficiency measurements in the revised manuscript.

W3. Limited Discrimination in Some Ablations

W3. Limited discrimination in some ablations. The semantic adapter and cross-attention improve Joint by only 0.8 and 0.6 points. Their necessity remains unclear without uncertainty estimates, although physical-branch and teacher ablations show larger PC drops.

Thank you for this observation. The physical branch is essential for transferring physical knowledge into the DiT backbone, as removing it causes a substantial drop in physical consistency. The semantic adapter provides complementary semantic guidance, while physical cross-attention enables fine-grained interactions between physical codes and hidden states of DiT Blocks, offering more effective physical guidance than global injection.

To further validate the contributions of these two components, We conduct 10,000 paired bootstrap iterations over the 600 VideoPhy2 prompts, sampling 600 prompts with replacement and evaluating the full model and each variant on identical samples. The improvements of the full DiPhy over the corresponding variants, together with their 95% confidence intervals, are reported below:

Contribution of the Semantic Adapter

MetricImprovement95% CI
PC ↑+0.5[0.3, 0.8]
SA ↑+1.1[0.9, 1.2]
Joint ↑+0.9[0.4, 1.0]

Physical Cross-Attention vs. Global Injection

MetricImprovement95% CI
PC ↑+1.2[0.8, 1.5]
SA ↑+0.7[0.4, 1.1]
Joint ↑+0.5[0.3, 0.7]

All six confidence intervals are strictly above zero, providing convincing evidence for the positive contributions of both components. In particular, the semantic adapter brings a more pronounced improvement in semantic alignment, while physical cross-attention provides a larger gain in physical consistency compared with global injection. These findings further support the complementary roles of these components in the overall framework of DiPhy.

AuthorsOpenReview ↗

Response to Reviewer ePYA [2/4]

W2. Motivation for Discrete Codebook

W2. Insufficient motivation for the codebook. The paper does not establish why discrete codes are needed over continuous-feature distillation. Reconstruction and routing correlations do not isolate physical dynamics from appearance or semantics.

Thank you for raising these important concerns. Discrete codes can organize physical knowledge into distinct and reusable prototypes. Similar physical phenomena can be represented by related prototypes, while complex scenarios involving multiple physical phenomena can be represented through weighted combinations of different codes. This provides a structured and compositional representation of physical knowledge. In contrast, continuous feature representations lack explicit physical prototypes, making their underlying physical semantics less interpretable and their knowledge organization less structured.

First of all, the code selection distributions in Fig. 5 show distinct patterns across physical categories: Melting and Solidification show a relatively high correlation (r = 0.92), while Elastic Motion and Melting exhibit a substantially lower correlation (r = 0.14). These observations indicate that the learned codebook exhibits category-dependent routing patterns: Different phenomena activate different distributions of physical codes, while similar categories share more similar code usage. This results provides evidence that our codebook organizes physics-relevant information rather than merely reconstructing general visual or semantic features.

Moreover, to further substantiate the information codebook preserved, we conduct a lightweight linear-probing experiment on IntPhys2. We extract four types of representations: pooled V-JEPA2 features, pooled Qwen3-VL semantic features, their concatenation, and the frozen teacher codebook representation. For each representation, we train the same lightweight linear classifier to distinguish physically plausible from implausible videos, using 204 videos (25%) for testing:

RepresentationAccuracyCorrect / Total
Pooled V-JEPA255.39%113 / 204
Pooled Qwen3-VL-8B52.94%108 / 204
Concatenated Features58.82%120 / 204
Teacher Physical Representation (DiPhy)61.76%126 / 204

The teacher physical representation outperforms pooled V-JEPA2 features, pooled semantic features, and their concatenation by 6.37, 8.82, and 2.94 percentage points. These results further indicate that the codebook-based representation retains more physics-discriminative information, rather than merely encoding features and reconstructing appearance or semantic information.

Meanwhile, compared with continuous representations, discrete representations can compress information into compact and reusable prototypes, facilitating the sharing of structured knowledge across diverse scenarios. To extensively support the above analysis, we conduct an additional comparison between continuous-feature prediction and the proposed discrete codebook representation. In the continuous setting, an MLP-style neural network, with a parameter count comparable to the original router and codebook, directly predicts the physical representation. In the discrete setting, the student router selects and aggregates physical codes from the codebook to construct the representation. For a fair comparison, both variants use the same teacher, the same physical and semantic decoders, and the same teacher-to-student distillation. They also share the same stage c) for knowledge injection. Experimental results are reported as below:

RepresentationPC ↑SA ↑Joint ↑
Continuous60.828.524.2
Discrete (DiPhy)61.828.725.8

Compared with continuous-feature prediction, DiPhy improves PC, SA, and Joint by 1.0, 0.2, and 1.6 points, respectively. These results suggest that organizing physical information into discrete codes provides benefits beyond directly regressing continuous features. Meanwhile, discrete codes provide more interpretable and identifiable physical prototypes than implicit continuous representations. Therefore, we designed a discrete physical codebook including a set of reusable codes.

In conclusion, these analyses demonstrate that DiPhy constructs a reusable and physics-oriented representation space through its physical codebook, organizing physical knowledge into interpretable prototypes rather than merely compressing or reconstructing existing visual and semantic features.

AuthorsOpenReview ↗

Response to Reviewer ePYA [3/4]

W4. Human Evaluation

W4. No human evaluation. Physical plausibility is assessed through VideoPhy2's automatic judge and selected qualitative examples. No human study independently validates whether the reported gains yield perceptible physical improvements.

Thank you for this suggestion. To assess whether DiPhy's improvements are perceptible to human observers, we conduct a human evaluation with 20 participants. Each participant ranks videos from three methods across 10 randomly selected prompts, assigning non-repeating scores of 1–3 (3 indicating the best performance) on Physical Plausibility, Temporal Consistency, and Visual Quality. The average scores across participants and prompts are reported below:

MethodPhysical Plausibility ↑Temporal Consistency ↑Visual Quality ↑
CogVideoX-2B1.451.551.85
VideoREPA1.951.902.00
DiPhy2.602.552.15

DiPhy achieves the highest average scores across all three dimensions, outperforming CogVideoX-2B and VideoREPA by 1.15/0.65 points in Physical Plausibility, 1.00/0.65 points in Temporal Consistency, and 0.30/0.15 points in Visual Quality, respectively. These results demonstrate that DiPhy is consistently preferred by human observers, complementing the automatic evaluation. We will include the human evaluation protocol and results in the revised manuscript.

W5 & Q1. Clarification of DiPhy's Contributions

W5. Unclear incremental novelty. VideoREPA (https://arxiv.org/html/2505.23656) already distills video representations. PhysAlign (https://link.springer.com/chapter/10.1007/978-3-032-37464-6_28) uses a physical codebook and predicts physical slots from text and a first frame (https://www.sci-hub.vg/10.1007/978-3-032-37464-6_28). DiPhy's semantic-guided teacher and routing distillation need clearer differentiation.

Q1. What does the routed codebook add over a continuous-feature predictor under matched supervision and conditioning, and how does this contribution differ from PhysAlign?

Thank you for raising these important questions and we will cite and discuss PhysAlign in the revised manuscript. We first briefly discuss the two related works:

VideoREPA transfers video foundation model knowledge to video diffusion models through Token Relation Distillation, aligning spatial and temporal token relations. However, video foundation models encode diverse information, which may not all be necessary for video generation. Extracting and distilling the most relevant and non-redundant physical knowledge remains an important direction for further research.

PhysAlign uses a Physical Dynamics Extractor and Universal Physical Codebook, together with a Physics Mapper and event-aware temporal gating, to predict and inject event-conditioned physical features. Although both PhysAlign and DiPhy employ discrete physical codebooks, their differences lie in two aspects:

(1) Physical Codebook Learning. PhysAlign learns its codebook by aligning extracted physical slots with physical features under generation objectives. In contrast, DiPhy mines physics-relevant information from V-JEPA2 features and uses physical and semantic reconstruction objectives to learn an independent and dedicated representation space.

(2) Physical Knowledge Injection. PhysAlign predicts physical features through its Physics Mapper and injects the feature representations into DiT. In contrast, DiPhy employs a Student Router to select and aggregate a sparse subset of reusable physical codes. These sparse physical tokens are directly injected into DiT as compact physical conditioning, enabling scene-dependent selection and composition of physical knowledge.

Based on these differences, we resummarize the distinct advantage of DiPhy:

  • Knowledge Mining: Learning an independent physics-oriented representation space with reusable codes, distinct from existing feature spaces.

  • Knowledge Distillation: Transferring physical knowledge to a student module that predicts relevant physical codes from text and first frame.

  • Sparse Knowledge Injection: Injecting selected codes into DiT through a lightweight branch for physical guidance with low parameter and memory overhead.

Meanwhile, the motivation for adopting discrete rather than continuous representations has already been discussed in our response to W2. Overall, DiPhy provides an integrated framework for mining, transferring, and injecting reusable and compact physical knowledge, distinguishing it from existing physics-informed methods.

Reviewer_57yrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiPhy leverages V-JEPA2 to extract the training video features and learns a 64-entry physical codebook under the guidance of semantic or physical description provided by Qwen3-VL. Then, the student module predicts the physical code based on the prompt and the VLM features. The physical code will be injected into the video diffusion model via lightweight cross-attention. This paper reports improvements in physical consistency and overall quality on benchmarks such as VideoPhy2 and VBench.

Critical strengths
  1. This paper addresses an important problem. Improving the physical plausibility of video generation is still a challenge for current diffusion models.
  2. Transferring physical information into a codebook and injecting it into the diffusion models is interesting and technically sound.
  3. The method is combined with two distinct video generation models, and both show improvements in physical commonsense.
  4. The experimental results suggest that the physical conditioning does not substantially degrade general video quality.
Critical weaknesses
  1. The selected backbone models are too small (<= 2B). Recent large video models have improved physical-following capability. Applying this method to these models and showing improvement will make this work more convincing.
  2. Only one baseline method is compared. Existing state-of-the-art methods, such as PhyGDPO [1] and PhysRAG [2], should be included for comparison.
  3. The student does not appear to operate from text alone. At inference time, the method first generates an initial frame using FLUX and then processes this frame together with the text using Qwen3-VL, which will introduce additional computation.
  4. Evaluation on two backbones demonstrates architectural portability, but it does not necessarily demonstrate that the learned knowledge transfers across physical phenomena, semantic domains, or datasets.

[1] Cai, Yuanhao, et al. "Phygdpo: Physics-aware groupwise direct preference optimization for physically consistent text-to-video generation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[2] Cheng, Kexu, et al. "PhysRAG: Enhancing physics-awareness in video generation via retrieval-augmented generation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

Questions for authors
  1. Does inference always require a FLUX-generated frame and Qwen3-VL? What is the end-to-end latency and compute cost when these components are included?
  2. Can code intervention or code swapping demonstrate that individual codes have predictable effects on physical behavior?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 57yr [1/3]

Thank you very much for your valuable and constructive comments! Below, we have provided detailed answers to your concerns.

W1. Evaluation on Larger Backbone Models

The selected backbone models are too small (≤ 2B). Recent large video models have improved physical-following capability. Applying this method to these models and showing improvement will make this work more convincing.

Thank you for this valuable suggestion. We initially selected CogVideoX-2B and Wan2.1-1.3B due to limited computational resources. Nevertheless, since our model and these compared methods are on the same model scale, the comparison is conducted under completely fair settings, and in theory, consistent results would be obtained if the model’s scale is expanded. To further examine the effectiveness of DiPhy on larger backbones, we conduct additional experiments using CogVideoX-5B and Wan2.2-5B. The results on VideoPhy2 are summarized below:

MethodsPC ↑SA ↑Joint ↑
CogVideoX-5B63.227.523.7
+ DiPhy69.029.726.2
Wan2.2-5B61.327.824.0
+ DiPhy66.829.226.7

These additional experimental results show that DiPhy consistently improves all three metrics on both backbones. Specifically, it improves PC, SA, and Joint by 5.8, 2.2, and 2.5 points on CogVideoX-5B, and by 5.5, 1.4, and 2.7 points on Wan2.2-5B, respectively. These results demonstrate that DiPhy remains effective when applied to larger video generation backbones.

W2. Comparison with More State-of-the-Art Methods

Only one baseline method is compared. Existing state-of-the-art methods, such as PhyGDPO [1] and PhysRAG [2], should be included for comparison.

Thank you for this suggestion. We conduct additional comparisons with PhyGDPO and PhysRAG using the corresponding backbones: the publicly released Wan2.1-1.3B version of PhyGDPO and Wan2.2-5B for PhysRAG. We evaluate the methods on VideoPhy2 and PhyGenBench, respectively.

Comparison on VideoPhy2. Using Wan2.1-1.3B, we compare DiPhy with the baseline and PhyGDPO:

MethodsPC ↑SA ↑Joint ↑
Wan2.1-1.3B58.026.321.7
+ PhyGDPO61.827.823.0
+ DiPhy63.328.324.7

Compared with PhyGDPO, DiPhy improves PC, SA, and Joint by 1.5, 0.5, and 1.7 points respectively. PhyGDPO is a preference optimization framework that constructs physics-rich training data and uses groupwise DPO with VLM-based rewards to improve physical consistency. However, the model may learn to optimize visual patterns favored by the reward model rather than genuinely capturing the underlying physical principles. In contrast, DiPhy explicitly learns and transfers physical knowledge through a structured codebook, providing physical guidance without relying on reward-driven optimization during training. We will also cite and discuss these methods in the Related Work section, to better clarify the relationship between DiPhy and existing physics-aware methods, such as PhyGDPO.

Comparison with PhysRAG on PhyGenBench. Using Wan2.2-5B, we compare DiPhy with the baseline and PhysRAG across four physical dimensions. The results are shown below:

MethodsMechanics ↑Optics ↑Thermal ↑Material ↑Average ↑
Wan2.2-5B0.560.620.490.500.54
+ PhysRAG0.560.680.520.530.57
+ DiPhy0.590.670.540.550.59

DiPhy achieves the best results in Mechanics, Thermal, Material, and Average, while PhysRAG performs slightly better in Optics. These improvements may be attributed to DiPhy's distinctive approach to organizing and utilizing physical knowledge. PhysRAG relies on retrieving relevant videos and extracting physical priors through learnable queries. In contrast, DiPhy learns a shared and compact physical codebook from diverse video representations and distills knowledge into a student router, which predicts weighted combinations of physical codes conditioned on the target scene. This design reduces reliance on individual retrieved references and enables compositional representations of diverse physical patterns. We believe these properties contribute to DiPhy's improvements across physical dimensions on PhyGenBench. Overall, these additional comparisons show that DiPhy achieves competitive results compared with existing physics-aware methods, providing convincing evidence of DiPhy's effectiveness in improving physical consistency in video generation.

AuthorsOpenReview ↗

Response to Reviewer 57yr [2/3]

W3. Clarification of Student's Inputs

The student does not appear to operate from text alone. At inference time, the method first generates an initial frame using FLUX and then processes this frame together with the text using Qwen3-VL, which will introduce additional computation.

Thank you for pointing out this imprecise description! Using "text-only inference", we intended to emphasize that the student module no longer requires real videos as input. Instead, it infers physical codes based on image, text and knowledge transferred from teacher. Specifically, the first frame and text prompt are processed together by Qwen3-VL to obtain physical descriptions and semantic features. Guided by the semantic features, the student module predicts the codes relevant to specific physical phenomenon.

In the revision, we will remove the description "text-only inference", and clarify that the student predicts physical codes based on the available first frame and text prompt, as future video frames are unavailable during inference. We will also explicitly clarify that the initial frame and text prompt are processed by Qwen3-VL to obtain physical descriptions and semantic features, which guide the student module in predicting the corresponding physical codes.

W4. Transfer of Physical Knowledge

Evaluation on two backbones demonstrates architectural portability, but it does not necessarily demonstrate that the learned knowledge transfers across physical phenomena, semantic domains, or datasets.

Thank you for this important question. In the natural world, visually different scenarios can share underlying physical mechanisms and be described using related physical prototypes. For example, conical pendulum motion and projectile motion exhibit different visual trajectories, but both can be analyzed using classical mechanics. We therefore design the teacher module, to compactly encode and organize recurring physical patterns into a structured and reusable codebook, facilitating physical knowledge transfer across diverse scenarios while compressing redundant information. Accordingly, videos involving related physical patterns can activate overlapping subsets of codes, while complex scenarios involving multiple physical phenomena can be represented through weighted combinations of different codes.

Meanwhile, the code selection distributions in Fig. 5 provide category-level evidence for this organization. Melting and Solidification exhibit a relatively high correlation in their code selection distributions(r = 0.92). In contrast, Elastic Motion and Melting exhibit a much lower correlation (r = 0.14). These results suggest that code usage reflects similarities and differences among physical categories, rather than following a uniform selection pattern.

Regarding generalization and knowledge transfer across diverse datasets, since DiPhy is trained on the WISA-80K, which including 17 physical phenomena across three physics categories, it may still struggling to represent entirely unseen physical categories, such as electromagnetism, when the training data contain neither relevant concepts nor visual examples. Nevertheless, our physical codebook provides an extensible mechanism for incorporating additional physical knowledge. When new physical phenomena are introduced, the codebook can be expanded and updated with additional codes, the router can be retrained using videos containing the new phenomena, and the student can then be updated to predict the expanded physical representation. This design provides a potential path to incorporating new physical knowledge without redesigning the entire generation backbone. We will incorporate this discussion into the Limitations section of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 57yr [3/3]

Q1. Inference-time Dependencies and Computational Cost

Does inference always require a FLUX-generated frame and Qwen3-VL? What is the end-to-end latency and compute cost when these components are included?

Thank you for raising this detail during inference!

Inference-time Dependencies. FLUX is not an essential component of DiPhy's inference pipeline. We use FLUX only because benchmarks such as VideoPhy2 provide text prompts without corresponding initial images, whereas DiPhy requires a first frame as an input condition. In practical image-conditioned video generation scenarios, where the initial frame is provided, FLUX is not required. Qwen3-VL, on the other hand, generates physical descriptions based on the first frame and text prompt. These descriptions are subsequently encoded by Qwen3-VL's own encoder to obtain semantic features, which guide the student module in predicting the corresponding physical codes.

Computational Cost. To comprehensively evaluate inference efficiency, we measure the model parameters, peak GPU memory, and end-to-end latency on a single V100 32GB GPU. Note that the V100 GPU used in our experiments has relatively limited computational capabilities and does not support standard FlashAttention-2 kernels, resulting in relatively longer inference times. Faster GPUs, such as the RTX 4090 and A100, as well as multi-GPU parallel inference, can further reduce the overall latency. Therefore, our primary focus is on quantifying the additional overhead introduced by DiPhy compared with the baseline under the same hardware and inference settings. The results on Wan2.1-1.3B, which generates a 6-second video, are reported below:

MethodsParamsPeak MemoryOverall LatencyQwen3-VL Additional CostPC ↑
Wan2.1-1.3B1.418B18.39GB872.33s--58.0
DiPhy1.444B20.26GB934.09s3.99s63.3
Overhead+1.83%+1.87GB+7.08%----

Notably, the overall latency accounts for the complete inference pipeline, including Qwen3-VL processing, stage b(Distilling Transferable Physical Knowledge) and stage c(Physical Knowledge Injection), and the Qwen3-VL additional cost consists of two steps: generating a physical description conditioned on the first frame and text prompt, and encoding the description with its own encoder to obtain semantic features. These results demonstrate that DiPhy achieves improved physical consistency with relatively modest additional computational overhead. We will clarify the role of FLUX and Qwen3-VL and report these efficiency measurements in the revised manuscript.

Q2. Code Swapping and Physical Behavior

Can code intervention or code swapping demonstrate that individual codes have predictable effects on physical behavior?

To investigate whether category-associated physical codes influence the physical behavior, we select 50 samples from each of the Collision and Melting categories(based on the category labels provided by WISA-80K) to conduct a code-swapping experiment. Specifically, we exchange the Top-8 codes associated with the two categories, and evaluate the generated videos before and after swapping using PC and SA on VideoPhy2. The results are presented below:

PC ↑SA ↑
Collision62.028.0
Collision(after swap)54.020.0
Melting66.026.0
Melting(after swap)54.022.0

After swapping the Top-8 codes, both categories show decreases in physical consistency and semantic adherence. For Collision, PC and SA decrease by 8.0 points each; for Melting, PC decreases by 12.0 points and SA by 4.0 points. These results provide evidence for the correlation between codes and their corresponding physical phenomena, demonstrating that such category-aware code representations play an important role in improving physical consistency in video generation.

We will incorporate these clarifications and refinements in the revised version. Thank you again for your valuable feedback!

Reviewer_hbZcOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

DiPhy transfers physics-related information from pretrained video representations to video generation. A teacher learns a physical codebook from V-JEPA2 features under VLM-derived semantic guidance. A student predicts codes from text and semantic features, which are injected into a frozen diffusion backbone alongside a semantic adapter. Experiments on CogVideoX-2B and Wan2.1-1.3B report improvements on VideoPhy2 and evaluate overall generation quality on VBench.

Critical strengths
  1. Transferring physics-related video representations into a frozen video generator is a promising direction. The framework improves physical consistency on two generation backbones while maintaining overall video quality, supporting its practical potential.
  2. The component ablations and supplementary teacher-design experiments provide useful evidence that video-derived supervision and the physical branch contribute to the improvements. These experiments go beyond reporting only end-to-end gains.
Critical weaknesses
  1. The motivation for the physical codebook is insufficiently established. The introduction motivates learning physical information from videos, but does not explain why this requires a codebook rather than a compact continuous representation. Since the codebook is a central contribution, the absence of a direct comparison with a simpler continuous-feature distillation alternative makes its specific value difficult to assess.
  2. The evidence that the codes capture physical understanding remains indirect. Reconstruction of pooled video and semantic features, code utilization, and category-level routing correlations do not clearly distinguish physical knowledge from general visual or semantic information. The generation improvements support the framework’s effectiveness, but leave unclear what physical information the codebook preserves beyond the semantic guidance already supplied by the VLM.
Questions for authors
  1. VideoREPA primarily uses a VideoMAE-based encoder, whereas DiPhy uses V-JEPA2. Would replacing VideoREPA’s encoder with the same V-JEPA2 encoder used by DiPhy improve its performance? Comparing against this baseline under matched training data and generation-backbone settings would help distinguish gains from the stronger encoder from those of the proposed codebook-based method.
  2. Could the authors provide a lightweight evaluation of the frozen codebook representation on a physical-understanding benchmark such as IntPhys 2, or an equivalent controlled set of physically plausible/implausible videos? For example, a simple probe comparing the teacher’s codebook representation against pooled V-JEPA2 and semantic features could help clarify what physical information is retained. A limited diagnostic would be sufficient; I am not requesting extensive benchmark-specific training.

I view this direction positively. Convincing clarifications and focused supporting results would make me likely to raise my score.

9 comments
AuthorsOpenReview ↗

Response to Reviewer hbZc [1/2]

Thank you very much for your valuable and insightful comments. Below, we have provided detailed answers to your concerns and questions.

W1. Motivation for the Physical Codebook

Thank you for raising this important question. Our motivation for introducing the codebook is to compactly encode and organize recurring physical patterns into a structured, reusable representation space, facilitating physical knowledge transfer across diverse scenarios. When people studying physics, they often simplify certain similar scenarios into a series of physical prototypes, such as inclined planes, conical pendulums, the melting of solids and solidification of liquids. Similar physical principles often underlie different visual scenarios, and complex scenarios can be representated as combinations of various physical prototypes.

Compared with continuous representations, discrete representations can compress information into compact and reusable prototypes, facilitating the sharing of structured knowledge across diverse scenarios. To further support the above analysis, we conduct an additional comparison between continuous-feature prediction and the proposed discrete codebook representation. In the continuous setting, an MLP-style neural network, with a parameter count comparable to the original router and codebook, directly predicts the physical representation. In the discrete setting, the student router selects and aggregates physical codes from the codebook to construct the representation. For a fair comparison, both variants use the same teacher, the same physical and semantic decoders, and the same teacher-to-student distillation. They also share the same stage c) for knowledge injection. Experimental results are reported as below:

RepresentationPC ↑SA ↑Joint ↑
Continuous60.828.524.2
Discrete (DiPhy)61.828.725.8

Compared with continuous-feature prediction, DiPhy improves PC, SA, and Joint by 1.0, 0.2, and 1.6 points, respectively. These results suggest that organizing physical information into reusable discrete codes provides benefits beyond directly regressing continuous features. Meanwhile, discrete codes provide more interpretable and identifiable physical prototypes than implicit continuous representations. Therefore, we designed a physical codebook including a set of learnable codes, which provides a collection of reusable physical prototypes. Videos involving similar physical mechanisms can activate overlapping subsets of codes, while scenarios involving multiple physical phenomena can be represented through weighted combinations of different codes.

AuthorsOpenReview ↗

Response to Reviewer hbZc [2/2]

W2 & Q2. Evidence of Physics-Relevant Information in the Codebook

Thank you for this critical question regarding physical information the codebook preserved.

First of all, the code selection distributions in Fig. 5 show distinct patterns across physical categories: Melting and Solidification show a relatively high correlation (r = 0.92), while Elastic Motion and Melting exhibit a substantially lower correlation (r = 0.14). These observations indicate that the learned codebook exhibits category-dependent routing patterns: Different phenomena activate different distributions of physical codes, while similar categories share more similar code usage. This results provides evidence that our codebook organizes physics-relevant information rather than merely reconstructing general visual or semantic features.

Meanwhile, for more comprehensive evaluation of the codebook, we further conduct a lightweight linear probing experiment on IntPhys2, extract four types of representations: pooled V-JEPA2 features, pooled Qwen3-VL semantic features, their concatenated features, and our frozen teacher codebook representation. For each representation, we train a lightweight linear classifier to distinguish physically plausible videos from implausible ones, and use 204 videos(25%) from IntPhys2 for test. The results are presented below:

RepresentationAccuracyCorrect / Total
Pooled V-JEPA255.39%113 / 204
Pooled Qwen3-VL-8B52.94%108 / 204
Concatenated Features58.82%120 / 204
Teacher Codebook Representation (Ours)61.76%126 / 204

As is shown in the table, our teacher codebook representation outperformed pooled V-JEPA2 features, pooled semantic features, and their simple concatenation by 6.37, 8.82, and 2.94 percentage points, respectively. These results suggest that the codebook-based representation retains more physics-discriminative information that is useful for distinguishing physically plausible from implausible videos.

Q1. Influence of Different Video Encoder

To investigate whether the improvements of DiPhy primarily originate from the stronger V-JEPA2 encoder, we replace the original VideoMAE V2 encoder in VideoREPA-2B with the same V-JEPA2 encoder used in DiPhy. The two methods are evaluated using matched training data, training budgets, and CogVideoX-2B generation backbones. The results on VideoPhy2 are summarized below:

MethodVideo EncoderPCSAJoint
VideoREPA-2BVideoMAE V262.727.023.2
VideoREPA-2BV-JEPA261.528.023.7
DiPhyV-JEPA261.828.725.8

Replacing VideoMAE V2 with V-JEPA2 in VideoREPA-2B leads to a decrease of 1.2 points in PC, despite slight improvements of 1.0 and 0.5 points in SA and Joint, respectively. Meanwhile, under the same V-JEPA2 encoder, DiPhy achieves improvements of 0.3, 0.7, and 2.1 points in PC, SA, and Joint respectively, compared with VideoREPA-2B, suggesting that the improvements of DiPhy cannot be attributed to the choice of video encoder. Instead, DiPhy improved the physical consistency of generative models through the codebook-based physical knowledge.

We will incorporate these clarifications and refinements into the revision. Thank you again for your detailed and thoughtful review and we truly appreciate your time and feedback.

AuthorsOpenReview ↗

Response to Reviewer hbZc

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we are eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Comments on the paper

1 comment
Area_Chair_UrxrOpenReview ↗

Please engage with the authors.

Dear reviewers,

The authors have submitted detailed responses to your reviews.

Could you please review their feedback, engage with the authors if any points remain unclear, and update your evaluation and score if their response has addressed your concerns?

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.