- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents PhysPilot, which enhances the physical plausibility of video generation by learning physical priors from the input image. It introduces a PhysEncoder to extract physics-relevant representations and employs a top-down DPO-based optimization strategy to improve both physical cue extraction and their utilization by the video generation model. Extensive experiments across specialized physical tasks and diverse open-world scenarios demonstrate the effectiveness and generalization of the proposed method.
Critical strengths
1、The proposed top-down reinforcement learning strategy is effective, and the ablation studies provide convincing evidence for the contribution of the different training stages.
2、The experimental evaluation is comprehensive, covering proxy physical tasks, related-task transfer, and diverse real-world scenarios, demonstrating the effectiveness and generalization of PhysPilot.
Critical weaknesses
1、The necessity of the trainable physical head is not fully justified. The proposed PhysEncoder already adopts a frozen pretrained DINOv2 encoder, while only the physical head is optimized. However, the experiments do not compare against directly using the frozen pretrained visual representation as the physical condition. Such a baseline would help clarify whether the improvements mainly come from the pretrained representation itself or from the proposed physical representation adaptation.
2、The choice of DPO as the reinforcement learning objective is not sufficiently justified. Although the proposed method is built upon a flow-based video diffusion model, the paper does not discuss why Flow-DPO is preferred over other reinforcement learning objectives for flow-based generation, such as Flow-GRPO. A comparison or discussion of alternative RL objectives would help justify the design choice.
3、Table 1 contains a formatting issue that affects its readability. The authors should carefully revise the table layout and ensure that all entries are properly aligned and formatted.
Questions for authors
see Weaknesses.
5 comments
Frozen-representation vs. no PhysEncoder. Directly using the frozen pretrained representation improves the trajectory metrics (L2 and CD) on average, and most of this gain is obtained on the Seen split of the evaluation set. It does not improve IoU on average, and on the Unseen split its effect is small and not consistent across the three metrics. The pretrained representation itself therefore accounts for a part of the improvement, mainly in trajectory accuracy on objects and backgrounds seen during training.
Ours vs. frozen-representation. With the physical head adapted by SFT and DPO, our model outperforms the frozen-representation baseline on every metric of both splits. On the Unseen split, the improvement brought by the adaptation is larger than the improvement brought by the frozen representation on all three metrics, and the improvement in average IoU comes entirely from the adaptation.
3. Representation-level evidence in the submission
The comparisons above evaluate the representations through the generated videos. The probes in Sec. 4.2 examine the representations themselves, without any training, and address the same question: whether the physical information is already present in the frozen pretrained feature ("DA dense"), or is acquired through adaptation. Each probe is evaluated on the frozen pretrained feature and on the adapted features, so that they can be compared directly.
Probe 1: physical grouping across semantic categories (Table 4 in the submission).This probe tests whether objects with the same physical behavior have similar features even when they belong to different semantic categories. We assign 185 objects from nine semantic categories to three physical groups by deformation behavior, and measure how well the feature similarity follows these physical groups. Pairs from the same semantic category are excluded, so semantic similarity cannot account for the result.
Table c: Physical grouping across semantic categories.
| Feature | Contrast (↑) | NN@5 (↑) | RSA (↑) |
|---|---|---|---|
| DA dense (frozen pretrained) | |||
| Stage I dense | |||
| Stage I latent | |||
| Stage III dense | |||
| Stage III latent |
The frozen pretrained feature follows the physical groups the least on all three metrics. The agreement increases after Stage I and increases further after Stage III. The physical grouping is therefore not inherited from the pretrained representation; it is formed by the adaptation.
Probe 2: sensitivity to physical versus appearance factors (Table 3 of the submission, with the frozen pretrained feature added). This probe changes one factor of a scene at a time and measures how much the feature changes. The factors are either physical (the support state of the object, i.e., whether it is in contact with the ground or in the air; its height) or related to appearance (background, lighting).
Table d: Sensitivity to physical and appearance factors.
| Changed factor | DA dense (frozen pretrained) | Stage III latent |
|---|---|---|
| Support state (air / ground) | ||
| Height | ||
| Rotation | ||
| Background | ||
| Lighting |
After Stage III, the feature changes about ten times more under a change of a physical factor (the support state) than under a replacement of the whole background, and it is almost unaffected by lighting. For the frozen pretrained feature, the responses to physical factors and to appearance factors are much closer. The adaptation therefore makes the feature sensitive to the physical factors of the scene rather than to its appearance.
Probe 3: separation of physics laws in open-world scenes. This probe repeats the test beyond the proxy task. Scenes from the WISA test set are labeled by their governing physics law (dynamics, thermodynamics, optics), and we measure how much more similar the features of scenes under the same law are than the features of scenes under different laws. This separation is the weakest for the frozen pretrained feature and increases after each stage of adaptation. After Stage III, it is more than three times that of the frozen pretrained feature. The conclusion of Probe 1 therefore also holds for general physical scenes.
Summary. These results answer the reviewer's question in three steps. (1) After SFT alone, using the frozen pretrained representation as the physical condition brings no stable gain. (2) After the full three-stage training (one SFT stage and two DPO stages), the frozen representation improves over the pipeline without any representation, but this improvement is smaller than the one obtained with the physical representation adaptation. (3) In the training-free probes, the frozen pretrained feature is clearly weaker than the adapted feature. The pretrained representation itself therefore explains only a part of the improvement, and the remaining part is obtained through our three-stage physical adaptation, which is why the physical head needs to be trainable.
R3-W2: Flow-DPO and comparison with Flow-GRPO
Below we explain why Flow-DPO is chosen, which follows from the form of supervision available in our setting, and report a direct comparison with Flow-GRPO on the proxy task, where an online reward is available.
Why Flow-DPO. Online RL objectives such as Flow-GRPO need a reward for every video sampled during training. Our pipeline targets the open-world setting, where such a reward is not available. We tried open-source VLMs as automatic judges, but they are weak at comparing the physical plausibility of a pair of videos (see also the Limitations), and optimizing the generator online against an unreliable judge would not reliably improve physical plausibility. Human annotation, in turn, cannot be provided for every video sampled during training. Physical plausibility is therefore judged by human annotators as pairwise preferences, which are collected before training. Flow-DPO can be trained directly on such a fixed set of preference pairs, without a reward model and without any reward query during training. We use the same form of supervision on the proxy task, where candidate videos are ranked against simulated ground truth by mask IoU, so one objective covers both settings. The proxy task also admits an automatic reward, which allows the direct comparison with Flow-GRPO below.
Comparison on the proxy task The proxy task does admit online RL: the same automatic protocol we use to label preference pairs (VAE decode → SAM 2 segmentation → IoU against simulated ground truth) can serve as a per-rollout reward. We therefore implemented Flow-GRPO (Liu et al., 2025) on our generator, following its ODE-to-SDE conversion for stochastic sampling and denoising-reduction strategy. The policy parameters are the same LoRA (r=64) trained by our Stage II, and both methods are trained for the same number of steps on top of the same Stage-I checkpoint:
Table e: Flow-DPO and Flow-GRPO on the proxy task in Stage II.
| Method (after Stage I) | Seen L2 (↓) | CD (↓) | IoU (↑) | Unseen L2 (↓) | CD (↓) | IoU (↑) | Avg L2 (↓) | CD (↓) | IoU (↑) |
|---|---|---|---|---|---|---|---|---|---|
| + DPO for (Stage II, ours) | |||||||||
| + Flow-GRPO for |
Under the same number of training steps, the two objectives improve different aspects. Flow-DPO gives the better trajectory metrics (L2 and CD) on both the Seen and Unseen splits of the evaluation set. Its improvement over the Stage-I checkpoint (Table 1, row 3) on these metrics is clearly larger than that of Flow-GRPO, which improves them only slightly. Flow-GRPO is slightly better on IoU on the Seen split and on average, and the two are equal on the Unseen split.
A plausible explanation for the better trajectory metrics (L2 and CD) of Flow-DPO in this comparison is the form of the optimization feedback. Both methods use the same IoU criterion. Flow-DPO converts it into fixed preference pairs and compares the preferred and the non-preferred video under the same noise and timestep. The loss therefore depends only on the difference between the two videos, and the gradient directly reflects what makes the preferred video better. Flow-GRPO instead estimates group-relative advantages from the IoU of the generated videos, and applies this video-level feedback to all stochastic denoising transitions. The reward indicates the overall quality of a video, but not which transitions caused a physical error, so the credit assignment is indirect. This makes the updates less effective at correcting the object motion, while Flow-GRPO remains competitive on IoU, the quantity it maximizes directly.
In conclusion, Flow-DPO is the more suitable objective for our pipeline. On the proxy task, it improves trajectory accuracy more than Flow-GRPO under the same number of training steps, with comparable IoU. It does not need to sample and score videos during training, and it also applies to the open-world setting, where no reliable online reward is available.
We thank the reviewer for the thoughtful suggestions on our baselines, training objective, and presentation.
R3-W1: Comparison of Frozen-representation baseline and the trainable physical head
We thank the reviewer for this suggestion. Following it, we add a baseline that directly uses the frozen pretrained representation as the physical condition. Its construction, the comparison with our pipeline, and the related representation-level evidence are given below.
1. Frozen-representation baseline
The frozen pretrained features (DINOv2-B tokens; the 64-channel dense output of the pretrained Depth Anything V2 head) do not match the 8-channel latent space of the video DiT, so "directly using the frozen pretrained visual representation as the physical condition" requires a dimensional mapping. We therefore instantiate the baseline with a non-learned mapping: Depth Anything V2's encoder and pretrained DPT head are kept frozen; their 64-channel dense features are projected to 8 channels by PCA, channel-wise affine-normalized to the latent statistics, and injected at exactly the same position as in our method (temporal concatenation). No part of this conditioning branch is trained: in all training stages, the DiT is the only learnable module. The baseline follows the same three-stage schedule and budget as our pipeline.
2. Comparison under the same schedule and budget
We compare three pipelines that differ only in the physical condition: no PhysEncoder, the frozen-representation baseline, and ours. All three follow the same three-stage schedule (SFT, followed by two DPO rounds) with the same budget. We report the results after SFT alone and after the full three stages.
After SFT alone.Table a: Physical-condition comparison after SFT alone.
After SFT alone, neither the frozen representation nor the jointly trained PhysEncoder brings a stable gain over the model without PhysEncoder. Both improve the trajectory metrics (L2 and CD) on the Unseen split of the evaluation set, but both are worse on the trajectory metrics of the Seen split, and neither improves the average results. Adding a physical condition through SFT alone, whether it is frozen or trainable, is therefore not sufficient.
After the full three stages.Table b: Physical-condition comparison after the full three-stage schedule.
The comparison between the first two rows measures the contribution of the pretrained representation itself, and the comparison between the last two rows measures the contribution of the physical representation adaptation.