- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DiPhy aims to improve the physical plausibility of text-conditioned video generation by transferring video-derived representations through a shared codebook. A teacher combines frozen V-JEPA2 video features with VLM-derived semantic guidance and learns a 64-entry codebook using video-feature and semantic-feature reconstruction. A student predicts routing probabilities over the frozen codebook from text and VLM semantic conditions, using global-token alignment and routing-distribution distillation. Eight selected code vectors condition a lightweight physical branch that injects information into a frozen video diffusion backbone; a semantic adapter augments the text condition.
Training uses 10K videos sampled from WISA-80K. On VideoPhy2, DiPhy improves physical commonsense (PC) from 56.2 to 61.8 on CogVideoX-2B and from 58.0 to 63.3 on Wan2.1-1.3B. Joint semantic-and-physical pass rates improve from 23.3 to 25.8 and from 21.7 to 24.7, respectively. VBench and component, teacher-design, and reconstruction-weight ablations provide additional evaluation.
Critical strengths
S1. Coherent design for transferring video-derived priors. The shared codebook connects a video-conditioned teacher to a student using generation conditions. Semantic reconstruction helps make the teacher's representation predictable by the student.
S2. Practical integration with pretrained generators. A lightweight conditioning branch and semantic adapter augment a frozen generation backbone, and inference requires no reference video.
S3. Consistent reported gains across two backbones. DiPhy improves both physical commonsense and Joint performance on CogVideoX and Wan, with Joint increasing by 2.5 and 3.0 percentage points, respectively. Aggregate VBench quality also improves.
S4. Informative analysis of the teacher objective. Adding semantic reconstruction improves reported teacher–student alignment from 0.79 to 0.93 (Table 5), alongside higher generation scores (Table 6). This illustrates a useful distinction between reconstructing video features and making them predictable by the student.
Critical weaknesses
W1. Complex pipeline. Three training stages, VLM/first-frame preprocessing, codebook routing, and separate physical/semantic branches add implementation and training complexity. Timings with cached features leave the full preprocessing cost unclear.
W2. Insufficient motivation for the codebook. The paper does not establish why discrete codes are needed over continuous-feature distillation. Reconstruction and routing correlations do not isolate physical dynamics from appearance or semantics.
W3. Limited discrimination in some ablations. The semantic adapter and cross-attention improve Joint by only 0.8 and 0.6 points. Their necessity remains unclear without uncertainty estimates, although physical-branch and teacher ablations show larger PC drops.
W4. No human evaluation. Physical plausibility is assessed through VideoPhy2's automatic judge and selected qualitative examples. No human study independently validates whether the reported gains yield perceptible physical improvements.
W5. Unclear incremental novelty. VideoREPA already distills video representations. PhysAlign (Li et al., ECCV 2026) uses a physical codebook and predicts physical slots from text and a first frame. DiPhy's semantic-guided teacher and routing distillation need clearer differentiation.
W6. Modest advantages over existing methods. PC is below VideoREPA on CogVideoX (61.8 versus 62.7) and only 0.8 points above PhysVid on Wan. Joint advantages are 2.6 and 1.9 points, without uncertainty estimates.
Questions for authors
-
What does the routed codebook add over a continuous-feature predictor under matched supervision and conditioning, and how does this contribution differ from PhysAlign?
-
Under matched training budgets and conditioning, are the baseline and ablation gains consistent across seeds or supported by paired uncertainty estimates?
7 comments
Response to Reviewer ePYA [2/4]
W2. Motivation for Discrete Codebook
W2. Insufficient motivation for the codebook. The paper does not establish why discrete codes are needed over continuous-feature distillation. Reconstruction and routing correlations do not isolate physical dynamics from appearance or semantics.
Thank you for raising these important concerns. Discrete codes can organize physical knowledge into distinct and reusable prototypes. Similar physical phenomena can be represented by related prototypes, while complex scenarios involving multiple physical phenomena can be represented through weighted combinations of different codes. This provides a structured and compositional representation of physical knowledge. In contrast, continuous feature representations lack explicit physical prototypes, making their underlying physical semantics less interpretable and their knowledge organization less structured.
First of all, the code selection distributions in Fig. 5 show distinct patterns across physical categories: Melting and Solidification show a relatively high correlation (r = 0.92), while Elastic Motion and Melting exhibit a substantially lower correlation (r = 0.14). These observations indicate that the learned codebook exhibits category-dependent routing patterns: Different phenomena activate different distributions of physical codes, while similar categories share more similar code usage. This results provides evidence that our codebook organizes physics-relevant information rather than merely reconstructing general visual or semantic features.
Moreover, to further substantiate the information codebook preserved, we conduct a lightweight linear-probing experiment on IntPhys2. We extract four types of representations: pooled V-JEPA2 features, pooled Qwen3-VL semantic features, their concatenation, and the frozen teacher codebook representation. For each representation, we train the same lightweight linear classifier to distinguish physically plausible from implausible videos, using 204 videos (25%) for testing:
| Representation | Accuracy | Correct / Total |
|---|---|---|
| Pooled V-JEPA2 | 55.39% | 113 / 204 |
| Pooled Qwen3-VL-8B | 52.94% | 108 / 204 |
| Concatenated Features | 58.82% | 120 / 204 |
| Teacher Physical Representation (DiPhy) | 61.76% | 126 / 204 |
The teacher physical representation outperforms pooled V-JEPA2 features, pooled semantic features, and their concatenation by 6.37, 8.82, and 2.94 percentage points. These results further indicate that the codebook-based representation retains more physics-discriminative information, rather than merely encoding features and reconstructing appearance or semantic information.
Meanwhile, compared with continuous representations, discrete representations can compress information into compact and reusable prototypes, facilitating the sharing of structured knowledge across diverse scenarios. To extensively support the above analysis, we conduct an additional comparison between continuous-feature prediction and the proposed discrete codebook representation. In the continuous setting, an MLP-style neural network, with a parameter count comparable to the original router and codebook, directly predicts the physical representation. In the discrete setting, the student router selects and aggregates physical codes from the codebook to construct the representation. For a fair comparison, both variants use the same teacher, the same physical and semantic decoders, and the same teacher-to-student distillation. They also share the same stage c) for knowledge injection. Experimental results are reported as below:
| Representation | PC ↑ | SA ↑ | Joint ↑ |
|---|---|---|---|
| Continuous | 60.8 | 28.5 | 24.2 |
| Discrete (DiPhy) | 61.8 | 28.7 | 25.8 |
Compared with continuous-feature prediction, DiPhy improves PC, SA, and Joint by 1.0, 0.2, and 1.6 points, respectively. These results suggest that organizing physical information into discrete codes provides benefits beyond directly regressing continuous features. Meanwhile, discrete codes provide more interpretable and identifiable physical prototypes than implicit continuous representations. Therefore, we designed a discrete physical codebook including a set of reusable codes.
In conclusion, these analyses demonstrate that DiPhy constructs a reusable and physics-oriented representation space through its physical codebook, organizing physical knowledge into interpretable prototypes rather than merely compressing or reconstructing existing visual and semantic features.
Response to Reviewer ePYA [3/4]
W4. Human Evaluation
W4. No human evaluation. Physical plausibility is assessed through VideoPhy2's automatic judge and selected qualitative examples. No human study independently validates whether the reported gains yield perceptible physical improvements.
Thank you for this suggestion. To assess whether DiPhy's improvements are perceptible to human observers, we conduct a human evaluation with 20 participants. Each participant ranks videos from three methods across 10 randomly selected prompts, assigning non-repeating scores of 1–3 (3 indicating the best performance) on Physical Plausibility, Temporal Consistency, and Visual Quality. The average scores across participants and prompts are reported below:
| Method | Physical Plausibility ↑ | Temporal Consistency ↑ | Visual Quality ↑ |
|---|---|---|---|
| CogVideoX-2B | 1.45 | 1.55 | 1.85 |
| VideoREPA | 1.95 | 1.90 | 2.00 |
| DiPhy | 2.60 | 2.55 | 2.15 |
DiPhy achieves the highest average scores across all three dimensions, outperforming CogVideoX-2B and VideoREPA by 1.15/0.65 points in Physical Plausibility, 1.00/0.65 points in Temporal Consistency, and 0.30/0.15 points in Visual Quality, respectively. These results demonstrate that DiPhy is consistently preferred by human observers, complementing the automatic evaluation. We will include the human evaluation protocol and results in the revised manuscript.
W5 & Q1. Clarification of DiPhy's Contributions
W5. Unclear incremental novelty. VideoREPA (https://arxiv.org/html/2505.23656) already distills video representations. PhysAlign (https://link.springer.com/chapter/10.1007/978-3-032-37464-6_28) uses a physical codebook and predicts physical slots from text and a first frame (https://www.sci-hub.vg/10.1007/978-3-032-37464-6_28). DiPhy's semantic-guided teacher and routing distillation need clearer differentiation.
Q1. What does the routed codebook add over a continuous-feature predictor under matched supervision and conditioning, and how does this contribution differ from PhysAlign?
Thank you for raising these important questions and we will cite and discuss PhysAlign in the revised manuscript. We first briefly discuss the two related works:
VideoREPA transfers video foundation model knowledge to video diffusion models through Token Relation Distillation, aligning spatial and temporal token relations. However, video foundation models encode diverse information, which may not all be necessary for video generation. Extracting and distilling the most relevant and non-redundant physical knowledge remains an important direction for further research.
PhysAlign uses a Physical Dynamics Extractor and Universal Physical Codebook, together with a Physics Mapper and event-aware temporal gating, to predict and inject event-conditioned physical features. Although both PhysAlign and DiPhy employ discrete physical codebooks, their differences lie in two aspects:
(1) Physical Codebook Learning. PhysAlign learns its codebook by aligning extracted physical slots with physical features under generation objectives. In contrast, DiPhy mines physics-relevant information from V-JEPA2 features and uses physical and semantic reconstruction objectives to learn an independent and dedicated representation space.
(2) Physical Knowledge Injection. PhysAlign predicts physical features through its Physics Mapper and injects the feature representations into DiT. In contrast, DiPhy employs a Student Router to select and aggregate a sparse subset of reusable physical codes. These sparse physical tokens are directly injected into DiT as compact physical conditioning, enabling scene-dependent selection and composition of physical knowledge.
Based on these differences, we resummarize the distinct advantage of DiPhy:
-
Knowledge Mining: Learning an independent physics-oriented representation space with reusable codes, distinct from existing feature spaces.
-
Knowledge Distillation: Transferring physical knowledge to a student module that predicts relevant physical codes from text and first frame.
-
Sparse Knowledge Injection: Injecting selected codes into DiT through a lightweight branch for physical guidance with low parameter and memory overhead.
Meanwhile, the motivation for adopting discrete rather than continuous representations has already been discussed in our response to W2. Overall, DiPhy provides an integrated framework for mining, transferring, and injecting reusable and compact physical knowledge, distinguishing it from existing physics-informed methods.
Response to Reviewer ePYA [1/4]
Thank you very much for your careful and constructive comments. We appreciate the opportunity to clarify the design of DiPhy and provide additional experiments addressing the concerns about the physical codebook, evaluation, and experimental reliability.
W1. Overall Pipeline
Thanks! The three stages serve complementary purposes: Stage A mines and organizes reusable physical knowledge into a codebook; Stage B distills this knowledge into a student that predicts physical codes from image and text conditions; and Stage C injects the predicted codes into the frozen video generator through a lightweight physical branch. Overall, this staged design enables reusable physical knowledge to be learned and subsequently transferred to video generation.
Before training, a frozen V-JEPA2 encoder extracts video features offline for teacher-side physical knowledge mining, and this is its only use in the pipeline. At inference, Qwen3-VL processes the initial frame and text prompt to produce physical descriptions and semantic features; its measured additional cost is 3.99 seconds per sample. FLUX is only used when a benchmark provides text prompts without initial frames, and is not required when an initial image is already available. The results on Wan2.1-1.3B, which generates a 6-second video, are reported below:
Notably, the overall latency accounts for the complete inference pipeline, including Qwen3-VL processing, stage b(Distilling Transferable Physical Knowledge) and stage c(Physical Knowledge Injection). These results demonstrate that our DiPhy achieves improved physical consistency with relatively modest additional computational overhead. We will report these efficiency measurements in the revised manuscript.
W3. Limited Discrimination in Some Ablations
Thank you for this observation. The physical branch is essential for transferring physical knowledge into the DiT backbone, as removing it causes a substantial drop in physical consistency. The semantic adapter provides complementary semantic guidance, while physical cross-attention enables fine-grained interactions between physical codes and hidden states of DiT Blocks, offering more effective physical guidance than global injection.
To further validate the contributions of these two components, We conduct 10,000 paired bootstrap iterations over the 600 VideoPhy2 prompts, sampling 600 prompts with replacement and evaluating the full model and each variant on identical samples. The improvements of the full DiPhy over the corresponding variants, together with their 95% confidence intervals, are reported below:
Contribution of the Semantic Adapter
Physical Cross-Attention vs. Global Injection
All six confidence intervals are strictly above zero, providing convincing evidence for the positive contributions of both components. In particular, the semantic adapter brings a more pronounced improvement in semantic alignment, while physical cross-attention provides a larger gain in physical consistency compared with global injection. These findings further support the complementary roles of these components in the overall framework of DiPhy.