- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces VIPER, a reference-guided image-to-video generation framework that treats a reference video as a visual demonstration of physical behavior rather than an appearance template. A multimodal large language model extracts reference-derived cues through learnable physics query tokens, whose hidden states are projected into conditioning tokens for a pretrained DiT-based video generator. A three-stage training strategy separates representation alignment, reference-target transfer learning, and lightweight generator adaptation. The authors also construct VIPER-19K, a curated dataset with material, trajectory, and physical-impact annotations and filtered reference-target pairs. Experiments report improved reference-video physical similarity and human preference while maintaining competitive general video quality.
Critical strengths
-
The problem formulation is compelling. Using a reference video to demonstrate material response, deformation, and interaction dynamics is a natural alternative to specifying these continuous and relational processes through detailed text prompts.
-
The framework is clearly motivated and presented. The separation between target appearance and reference behavior is intuitive, and the query-based conditioning interface and hierarchical training strategy are easy to follow.
-
The evaluation includes complementary perspectives. The paper reports VBench sub-metrics, VLM-based physical similarity, and human preference. The ablations also examine learnable queries and training stages.
Critical weaknesses
-
Physical similarity does not independently establish physical plausibility. The VLM-based metric evaluates similarity to the reference, which is relevant to the proposed task but does not independently establish adherence to physical principles. The human study includes physical plausibility, but the paper reports neither established physics-benchmark results nor task-specific motion or event measurements. Appropriately adapted benchmarks and appearance-invariant measurements would strengthen the claims. Such measurements should account for differences between reference and target scenes rather than rely on direct pixel-level optical-flow or IoU comparisons.
-
Comparisons with dedicated physics-aware methods are missing. Table 1 includes general video-generation and video-as-prompt methods, but no dedicated physics-aware training or adaptation methods. Relevant examples include DiffPhy, PILA, and PhysVid, although their conditioning interfaces and implementation availability may differ. Comparisons with compatible, available methods would strengthen the empirical positioning. For methods that cannot be compared directly, the paper should explain the input and task differences and discuss the corresponding limitations of its comparative claims.
-
Evaluation reliability and uncertainty are insufficiently documented. Section 5.2 specifies a scoring scale from 1 to 5 and a penalty for near-identical reference copying, but does not identify the judge model or report calibration, repeated-rating reliability, or quantitative agreement with human judgments. Evaluation uses 75 randomly sampled pairs from a 500-pair validation set. The human study recruits 25 volunteers and mentions 10 to 20 comparison groups, but does not clearly specify whether this number is per participant or overall, the total number of judgments, or uncertainty estimates. These omissions make the robustness of the reported differences difficult to assess.
-
The necessity of Stage 1 is not sufficiently supported by the reported ablation. Section 5.5 describes the hierarchical training strategy as essential for effective physical transfer. However, Table 2 reports physical similarity of 3.42 for the full model and 3.41 without Stage 1. The full model also performs better on the reported general video-quality metrics, but without variability estimates it is difficult to determine whether the small difference in physical similarity is reliable. This does not demonstrate that Stage 1 is ineffective, but the evidence is insufficient to establish its necessity for physical transfer.
Questions for authors
-
Physical plausibility evaluation. Could you provide task-appropriate physics-transfer measurements, such as normalized trajectories, contact-event ordering, or relative impact and deformation timing where correspondence is well-defined? Could you also evaluate on an appropriately adapted benchmark such as VideoPhy-2 or Physics-IQ Verified, with a clear protocol for reference construction and separation from held-out test outcomes?
-
Physics-aware baseline comparisons. Could you compare against compatible physics-aware methods such as DiffPhy, PILA, or PhysVid where implementations are available? For methods that cannot be evaluated directly, could you clarify their input and task differences and explain how these differences limit the comparative conclusions?
-
Evaluation protocol and reliability. Which VLM, checkpoint, and prompting protocol were used for physical-similarity evaluation? Is the judge from the same family as the Qwen3-VL-4B-Instruct encoder or Qwen3-VL-32B pair-filtering model? Could you provide calibration or reliability measurements, quantitative agreement with human judgments, the number of comparison groups per participant, the total number of judgments, the aggregation protocol, and uncertainty estimates?
-
Stage 1 ablation. Could you provide repeated-run variability or confidence intervals for the full model and the variant without Stage 1, particularly for the physical-similarity scores of 3.42 and 3.41? What evidence supports the claim that Stage 1 is necessary for effective physical transfer, rather than primarily improving general video quality or training stability?
4 comments
W2: Comparisons on Additional Baselines
PILA and DiffPhy is not open source, so we could not evaluation their performance. Consequently, we conducted additional comparisons PhysVid. To accommodate its text-to-video interfaces, we used Qwen3-VL-32B-Instruct to convert each target image into a text description of the subject and each reference video into a text description of the physical behavior. We then used Qwen3.5-35B-A3B to merge the two descriptions into a single generation prompt for each evaluation sample. The resulting prompt was supplied to PhysVid, while VIPER retained its native target-image, target-prompt, and reference-video inputs. The table below follows the evaluation metrics and column order of Table 1 in the paper.
| Method | Motion Smoothness ↑ | Temporal Flickering ↑ | Aesthetic Quality ↑ | Physical Similarity ↑ |
|---|---|---|---|---|
| PhysVid | 97.56 | 96.86 | 50.49 | 2.12 |
| VIPER (Ours) | 98.60 | 97.45 | 51.86 | 3.42 |
VIPER achieves the best overall performance among the compared methods under this evaluation protocol.
W3: Evaluation Reliability and Uncertainty
W3.1: Evaluation Reliability
Please refer to our response to W1.1
W3.2: Evaluation Uncertainty
In the evaluation, we randomly sampled 75 pairs from the 500-pair validation set. All baseline comparisons and ablation studies used the same 75 pairs. The selected pairs had balanced category distributions within the Material, Trajectory, and Impact annotation dimensions. For the human study, 25 volunteers each evaluated 10–20 comparison sets, totaling approximately 400 evaluations.
To strengthen our evaluation, we have expanded the experiments to all 500 validation pairs for VIPER and all baselines. The results are presented in the table below.
| Method | Motion Smoothness ↑ | Temporal Flickering ↑ | Aesthetic Quality ↑ | Physical Similarity ↑ | Preference Rank ↓ |
|---|---|---|---|---|---|
| Wan2.1-I2V-14B | 97.72 | 96.29 | 50.08 | 2.55 | 2 (31.6%) |
| Wan2.2-TI2V-5B | 97.51 | 96.72 | 49.14 | 2.13 | 2 (31.6%) |
| Wan2.2-I2V-14B | 98.25 | 96.08 | 50.42 | 2.71 | 2 (31.6%) |
| VAP-CogVideo | 98.34 | 97.21 | 50.13 | 2.75 | 3 (6.1%) |
| VAP-Wan | 97.32 | 95.91 | 48.58 | 2.76 | 3 (6.1%) |
| VACE | 98.62 | 97.59 | 49.26 | 2.06 | 4 (2.1%) |
| VIPER (ours) | 98.53 | 97.66 | 51.70 | 3.24 | 1 (60.2%) |
Additionally, we recruited 30 additional volunteers, increasing the human-study participant pool from 25 to 55, following the original human evaluation protocol.
| Criterion | VIPER (%) ↑ | Wan (%) ↑ | VAP (%) ↑ | VACE (%) ↑ |
|---|---|---|---|---|
| Reference-video physical similarity | 58.2 | 33.0 | 6.5 | 2.3 |
| Visual quality | 71.1 | 24.0 | 3.5 | 1.4 |
| Physical plausibility | 51.3 | 37.8 | 8.3 | 2.6 |
W4: The necessity of Stage 1
Stage 1 is designed to establish a usable conditioning interface between the frozen MLLM and the pretrained video DiT. Although the MLLM can represent visual dynamics, its query representations are not automatically compatible with the generator. Starting directly with cross-video pairs would require the learnable queries and connector to solve two problems simultaneously. Stage 1 separates these learning objectives, where the conditioning input and supervised target originate from the same video. This provides a correspondence between the observed dynamics and the sequence to be generated. This initialization allows Stage 2 to build on an established conditioning interface when learning cross-video transfer. To further validate our method's improvements, we evaluated VIPER and using three additional physics-related metrics. Please refer to W1.2 for details. As shown in the table below, the results further support the advantages of our approach.
| Method / Variant | Physical Similarity ↑ | VideoPhy-2 PC ↑ (1–5) | PhyGenEval ↑ (0–1) |
|---|---|---|---|
| w/o Stage 1 | 3.41 | 2.76 | 0.39 |
| VIPER | 3.42 | 3.57 | 0.49 |
Q1: Physical Plausibility Evaluation.
Q1.1: Physical Metrics
Please refer to our response to W1.2.
Q1.2: Evaluate on Physics-IQ Verified
Since VideoPhy-2 is designed for text-conditioned video generation, its input setting does not directly match our task. We select Physics-IQ Verified to our task. We use the switch frame at the third second as the target image and the 3–8-second segment of take-1 as the reference video. VIPER receives these inputs and generates a corresponding five-second video, which we evaluate using the Physics-IQ Verified metrics. As shown below, VIPER achieves promising results.
| Method | SP ↑ | ST ↑ | WS ↑ | MSE Score ↑ | Physics-IQ Verified ↑ |
|---|---|---|---|---|---|
| VIPER | 55.13 | 23.40 | 35.68 | 38.78 | 38.25 |
Q2: Physics-aware baseline comparisons.
Please refer to our response to W2.
Q3: Evaluation protocol and reliability
Please refer to our response to W1.
Q4: Stage 1 ablation
Please refer to our response to W4.
Thank you for the rebuttal and the added experiments. Two concerns remain; please respond directly within the limited discussion time.
1. Appearance-invariant physics evidence lacks baseline comparison. My original W1/Q1 asked for a measurement that is not VLM-judged, or a physics benchmark under a clear protocol. Of the three added metrics, VideoPhy-2 PC and PhyGenEval are themselves VLM-judged, so they do not replace the concern. Physics-IQ Verified is the only appearance-invariant benchmark, but numbers are reported only for VIPER (Q1.2) with no baseline under the same switch-frame protocol — so the table shows VIPER can be scored, not that it leads. Could the authors (a) run Wan2.2-I2V-14B and VACE (or VAP-CogVideo) on Physics-IQ Verified under the identical protocol, or (b) report one non-VLM scalar (e.g., contact-event ordering agreement, deformation-onset time error, or normalized trajectory IoU) across VIPER and the main baselines on the 500-pair set?
2. Stage 1 and main-table gaps are reported without uncertainty. The Stage 1 ablation (3.41 → 3.42 on Physical Similarity; wider on VideoPhy-2 PC and PhyGenEval) and the 500-pair main table both report single-run point estimates. Without seeds or paired tests, I cannot distinguish a real effect from run-to-run noise — especially for the 0.01-point primary-metric gap, but also for VIPER vs. VACE / Wan2.2-I2V-14B gaps of 0.5–1.3 on a 1–5 scale. Could the authors report a paired statistical test (per-pair sign test or bootstrap 95% CI) on the already-collected 500-pair per-sample scores? This is a post-hoc computation on existing data and should be feasible within the discussion window.
Based on the information currently available and any further clarification, I will finalize my recommendation during the reviewer–AC discussion phase.
Dear Reviewer 43rG:
We sincerely thank the reviewer for the constructive feedback and thoughtful consideration of our rebuttal. We are pleased that our responses have addressed your concerns, and we greatly appreciate your suggestions, which have helped us strengthen the manuscript.
We also thank you for raising these additional questions. We provide our clarifications below.
Baseline Comparisons for Appearance-Invariant Physics Evaluation
We thank the reviewer for identifying the remaining gap. We agree that VideoPhy-2 PC and PhyGenEval are VLM-based and therefore do not resolve the request for evidence independent of VLM judgment.
Following question (a), we have now evaluated Wan2.2-I2V-14B, VAP-Wan, VAP-CogVideo,VACE on Physics-IQ Verified. For Wan2.2-I2V-14B, We use the switch frame at the third second as the target image, and use verified prompt for physical conditioning. For VAP-Wan, VAP-CogVideo, VACE and VIPER, we use the switch frame at the third second as the target image and the 3–8-second segment of take-1 as the reference video. As shown below, VIPER achieves the highest score on Physics-IQ Verified owing to its strong ability to transfer physical dynamics from reference videos.
| Method | SP ↑ | ST ↑ | WS ↑ | MSE Score ↑ | Physics-IQ Verified ↑ |
|---|---|---|---|---|---|
| w/o Stage 1 | 50.12 | 19.46 | 26.61 | 27.19 | 29.25 |
| Wan2.2-I2V-14B | 51.22 | 20.73 | 28.97 | 29.32 | 32.56 |
| VAP-Wan | 52.34 | 21.91 | 30.45 | 31.67 | 34.09 |
| VAP-CogVideo | 52.15 | 20.52 | 30.24 | 31.12 | 33.51 |
| VACE | 53.68 | 22.30 | 32.46 | 34.79 | 35.81 |
| VIPER | 55.13 | 23.40 | 35.68 | 38.78 | 38.25 |
Following question (b), we additionally adopt the evaluation protocol of PISA [1] and report foreground mean Intersection-over-Union (mIoU) as a non-VLM scalar on the full 500-pair evaluation set.
For each validation pair, we use Qwen3-VL-32B-Instruct to identify the main object from the target video and obtain textual prompt. These prompt are fixed are shared across VIPER and all baselines. We then use SAM 3 to segment the corresponding objects, obtaining per-frame binary masks. Following PISA, we compute foreground mask IoU. We average the scores across frames and evaluated objects within each pair. Qwen is used solely for object identification and does not assign physical-correctness or quality scores. The final scalar is computed directly from binary masks, providing a geometric measurement of object-position and spatial-occupancy agreement evaluation. Consequently, PISA is a non-VLM metric. As shown below, VIPER achieves the highest score.
| Method | PISA foreground mIoU (%) ↑ |
|---|---|
| w/o Stage 1 | 25.04 |
| Wan2.2-I2V-14B | 18.82 |
| VAP-Wan | 23.34 |
| VAP-CogVideo | 19.03 |
| VACE | 12.41 |
| VIPER (Ours) | 36.97 |
[1] Li C, Michel O, Pan X, et al. Pisa experiments: Exploring physics post-training for video diffusion models by watching stuff drop[J].
Statistical Significance and Uncertainty of Reported Improvements
We conduct a post-hoc paired analysis using the retained per-sample scores for the Stage 1 ablation and baselines. We have added paired bootstrap analyses using the existing per-sample scores for both the Stage 1 ablation and the 500-pair main evaluation. For each sample, we additionally sampled five random seeds. For Physical Similarity and Physicas-IQ Verified, we align scores by test-pair ID and compute per-pair differences for the Stage 1 ablation and for VIPER against each baseline. For each comparison, we report the mean paired difference and its two-sided 95% percentile bootstrap confidence interval, obtained by resampling test pairs with replacement while preserving the pairing between methods. The results are summarized below:
| Comparison | Metric | Mean difference | 95% CI |
|---|---|---|---|
| With Stage 1 − without Stage 1 | Physical Similarity | +0.24 | [−0.22, 0.58] |
| VIPER − Wan2.2-I2V-14B | Physical Similarity | +0.65 | [0.56, 0.74] |
| VIPER − VAP-Wan | Physical Similarity | +0.75 | [0.66, 0.84] |
| VIPER − VAP-CogVideo | Physical Similarity | +0.70 | [0.61, 0.79] |
| VIPER − VACE | Physical Similarity | +1.48 | [1.39, 1.57] |
| With Stage 1 − without Stage 1 | Physics-IQ Verified | +9.00 | [7.68, 10.32] |
| VIPER − Wan2.2-I2V-14B | Physics-IQ Verified | +5.69 | [4.37, 7.01] |
| VIPER − VAP-Wan | Physics-IQ Verified | +4.16 | [2.84, 5.48] |
| VIPER − VAP-CogVideo | Physics-IQ Verified | +4.74 | [3.42, 6.06] |
| VIPER − VACE | Physics-IQ Verified | +2.44 | [1.12, 3.76] |
Finally, we sincerely hope that the additional evidence and clarifications have further strengthened the case for our approach. If there are any remaining concerns that may affect your assessment, we would be grateful for the opportunity to address them further. Thank you again for your thoughtful comments and constructive suggestions.
Response To Reviewer 43rG:
Thanks for your valuable suggestions. We will incorporate the relevant explanations into the final version for better understanding.
W1: Concern on Physical Similarity
W1.1: Reliability and prompt sensitivity of the VLM judge.
Our VLM-as-Judge score measures whether a generated video reproduces the material response, motion, and interactions demonstrated by the reference. The lenient prompt was designed to prevent imperfections in video generation from dominating this assessment of behavior transfer. During prompt development, we qualitatively observed prompt-induced harshness: instructions emphasizing strict error detection sometimes produced low scores for reference–generation pairs that humans judged to have similar physical. This observation motivates a controlled comparison of three variants. We vary the evaluation tolerance instructions and scoring anchors in Figure 9, while keeping other evaluation implementation unchanged.
To assess sensitivity to the judging prompt, we compare strict, neutral, and lenient prompts using Qwen3-VL-32B-Instruct on the same generated outputs, following our human-study protocol. Human volunteers provide 1–5 physical-similarity scores and select their preferred output for each comparison. We report human preference agreement and mean score bias. Human preference agreement measures the percentage of comparisons in which the judge's highest-scoring output matches the human votes. Mean score bias is the average difference between judge scores and the mean human ratings, negative values indicate under-scoring, and positive values indicate over-scoring. As shown below, Strict and Neutral prompts would assign lower scores than human assessors on average, whereas the Lenient prompt would achieve the highest human preference agreement with a mean score bias close to zero. This would support the lenient prompt as a way to align the assessment with human judgments.
To examine dependence on model, we evaluate the same generated outputs with Qwen3-VL-32B-Instruct, InternVL3.5-38B, and Gemma-3-27B-IT. As shown below, all judges achieve high agreement with human preferences, providing evidence that the assessment is less dependent on the preferences of a single judge.
W1.2: Additional Physical Metrics
We conduct physical evaluation with three additional metrics. VideoPhy-2 Physical Commonsense (PC) [1] takes a generated video as input and assesses its overall physical plausibility. PhyGenEval [2] takes a generated video, a scene description, physical laws, and evaluation questions as input to assess physical phenomena, event ordering, and overall naturalness.
[1] Bansal H, Peng C, et al. Videophy-2: A challenging action-centric physical commonsense evaluation in video generation
[2] Meng F, Liao J, et al. Towards world simulator: Crafting physical commonsense-based benchmark for video generation.
The results show that VIPER achieves the best performance, providing broader evidence of VIPER's effectiveness beyond temporal smoothness and visual quality.