acceptodds
Under review as a conference paper at ICLR 2027

VIPER: Visual In-Context Physics Reasoning for Physically Plausible Video Generation

Abstract

Modern video generation models can synthesize visually compelling and temporally coherent clips, yet controlling their physical behavior remains difficult with standard text and image conditions. The core challenge is a conditioning bottleneck: material response, contact interaction, deformation, and motion trajectory are continuous and relational physical cues that are hard to specify exhaustively in language but can be demonstrated naturally by video. We propose VIPER, a Visual In-Context Physics Reasoning framework for reference-guided image-to-video generation. Given a target image, a brief target prompt, and a reference video, VIPER treats the reference as a dense visual demonstration of the desired physical process rather than an appearance template. It uses a Multimodal Large Language Model (MLLM) to extract reference-derived physical cues and guide a pretrained image-to-video generator through a hierarchical training strategy, enabling physical behavior transfer while preserving the visual prior of the base generator. To support this setting, we construct VIPER-19K, a curated dataset with material, trajectory, and physical-impact annotations, together with filtered reference-target pairs. Experiments on an unseen validation set show that VIPER achieves stronger reference-video physical similarity and higher human preference than representative video generation and video-as-prompt baselines, while maintaining competitive general video quality. Qualitative results further demonstrate that VIPER can transfer reference-derived physical behavior to new target scenes without requiring carefully engineered prompts.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video generation, in-context learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_43rGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces VIPER, a reference-guided image-to-video generation framework that treats a reference video as a visual demonstration of physical behavior rather than an appearance template. A multimodal large language model extracts reference-derived cues through learnable physics query tokens, whose hidden states are projected into conditioning tokens for a pretrained DiT-based video generator. A three-stage training strategy separates representation alignment, reference-target transfer learning, and lightweight generator adaptation. The authors also construct VIPER-19K, a curated dataset with material, trajectory, and physical-impact annotations and filtered reference-target pairs. Experiments report improved reference-video physical similarity and human preference while maintaining competitive general video quality.

Critical strengths
  1. The problem formulation is compelling. Using a reference video to demonstrate material response, deformation, and interaction dynamics is a natural alternative to specifying these continuous and relational processes through detailed text prompts.

  2. The framework is clearly motivated and presented. The separation between target appearance and reference behavior is intuitive, and the query-based conditioning interface and hierarchical training strategy are easy to follow.

  3. The evaluation includes complementary perspectives. The paper reports VBench sub-metrics, VLM-based physical similarity, and human preference. The ablations also examine learnable queries and training stages.

Critical weaknesses
  1. Physical similarity does not independently establish physical plausibility. The VLM-based metric evaluates similarity to the reference, which is relevant to the proposed task but does not independently establish adherence to physical principles. The human study includes physical plausibility, but the paper reports neither established physics-benchmark results nor task-specific motion or event measurements. Appropriately adapted benchmarks and appearance-invariant measurements would strengthen the claims. Such measurements should account for differences between reference and target scenes rather than rely on direct pixel-level optical-flow or IoU comparisons.

  2. Comparisons with dedicated physics-aware methods are missing. Table 1 includes general video-generation and video-as-prompt methods, but no dedicated physics-aware training or adaptation methods. Relevant examples include DiffPhy, PILA, and PhysVid, although their conditioning interfaces and implementation availability may differ. Comparisons with compatible, available methods would strengthen the empirical positioning. For methods that cannot be compared directly, the paper should explain the input and task differences and discuss the corresponding limitations of its comparative claims.

  3. Evaluation reliability and uncertainty are insufficiently documented. Section 5.2 specifies a scoring scale from 1 to 5 and a penalty for near-identical reference copying, but does not identify the judge model or report calibration, repeated-rating reliability, or quantitative agreement with human judgments. Evaluation uses 75 randomly sampled pairs from a 500-pair validation set. The human study recruits 25 volunteers and mentions 10 to 20 comparison groups, but does not clearly specify whether this number is per participant or overall, the total number of judgments, or uncertainty estimates. These omissions make the robustness of the reported differences difficult to assess.

  4. The necessity of Stage 1 is not sufficiently supported by the reported ablation. Section 5.5 describes the hierarchical training strategy as essential for effective physical transfer. However, Table 2 reports physical similarity of 3.42 for the full model and 3.41 without Stage 1. The full model also performs better on the reported general video-quality metrics, but without variability estimates it is difficult to determine whether the small difference in physical similarity is reliable. This does not demonstrate that Stage 1 is ineffective, but the evidence is insufficient to establish its necessity for physical transfer.

Questions for authors
  1. Physical plausibility evaluation. Could you provide task-appropriate physics-transfer measurements, such as normalized trajectories, contact-event ordering, or relative impact and deformation timing where correspondence is well-defined? Could you also evaluate on an appropriately adapted benchmark such as VideoPhy-2 or Physics-IQ Verified, with a clear protocol for reference construction and separation from held-out test outcomes?

  2. Physics-aware baseline comparisons. Could you compare against compatible physics-aware methods such as DiffPhy, PILA, or PhysVid where implementations are available? For methods that cannot be evaluated directly, could you clarify their input and task differences and explain how these differences limit the comparative conclusions?

  3. Evaluation protocol and reliability. Which VLM, checkpoint, and prompting protocol were used for physical-similarity evaluation? Is the judge from the same family as the Qwen3-VL-4B-Instruct encoder or Qwen3-VL-32B pair-filtering model? Could you provide calibration or reliability measurements, quantitative agreement with human judgments, the number of comparison groups per participant, the total number of judgments, the aggregation protocol, and uncertainty estimates?

  4. Stage 1 ablation. Could you provide repeated-run variability or confidence intervals for the full model and the variant without Stage 1, particularly for the physical-similarity scores of 3.42 and 3.41? What evidence supports the claim that Stage 1 is necessary for effective physical transfer, rather than primarily improving general video quality or training stability?

4 comments

Response To Reviewer 43rG:

Thanks for your valuable suggestions. We will incorporate the relevant explanations into the final version for better understanding.

W1: Concern on Physical Similarity

W1.1: Reliability and prompt sensitivity of the VLM judge.

Our VLM-as-Judge score measures whether a generated video reproduces the material response, motion, and interactions demonstrated by the reference. The lenient prompt was designed to prevent imperfections in video generation from dominating this assessment of behavior transfer. During prompt development, we qualitatively observed prompt-induced harshness: instructions emphasizing strict error detection sometimes produced low scores for reference–generation pairs that humans judged to have similar physical. This observation motivates a controlled comparison of three variants. We vary the evaluation tolerance instructions and scoring anchors in Figure 9, while keeping other evaluation implementation unchanged.

PromptKey changes relative to Figure 9Core scoring logic
StrictReplace the instruction to be highly lenient and the positive-score default with strict scrutiny. Remove the instruction to overlook clipping, unnatural stretching, and disappearing pieces. Tighten the scoring anchors.Deduct for observable inconsistencies in material response, motion, contact, and causal timing, even when the intended event is recognizable. Reserve 5 for close agreement with the demonstrated dynamics.
NeutralRemove the positive-score default and blanket tolerance for physical errors. Distinguish rendering artifacts from errors that alter the physical event. Use evidence-based anchors across the full score range.Ignore flickering or appearance noise when the dynamics remain intact; penalize incorrect physical behavior in proportion to its effect on the transferred event.
Lenient (ours)Retain the original mindset, tolerance instructions, and scoring anchors.Prioritize recognizable transfer of the reference dynamics. Allow local approximations; give partial credit when the intended behavior remains identifiable, while penalizing major contradictions and literal clones.

To assess sensitivity to the judging prompt, we compare strict, neutral, and lenient prompts using Qwen3-VL-32B-Instruct on the same generated outputs, following our human-study protocol. Human volunteers provide 1–5 physical-similarity scores and select their preferred output for each comparison. We report human preference agreement and mean score bias. Human preference agreement measures the percentage of comparisons in which the judge's highest-scoring output matches the human votes. Mean score bias is the average difference between judge scores and the mean human ratings, negative values indicate under-scoring, and positive values indicate over-scoring. As shown below, Strict and Neutral prompts would assign lower scores than human assessors on average, whereas the Lenient prompt would achieve the highest human preference agreement with a mean score bias close to zero. This would support the lenient prompt as a way to align the assessment with human judgments.

Qwen3-VL-32B-Instruct promptHuman preference agreement (%) ↑Mean score bias (0 is ideal)
Strict64.0−1.54
Neutral76.0−1.01
Lenient (ours)84.0+0.24

To examine dependence on model, we evaluate the same generated outputs with Qwen3-VL-32B-Instruct, InternVL3.5-38B, and Gemma-3-27B-IT. As shown below, all judges achieve high agreement with human preferences, providing evidence that the assessment is less dependent on the preferences of a single judge.

JudgeHuman preference agreement (%) ↑VLM-as-Judge ↑
Qwen3-VL-32B-Instruct84.03.42
InternVL3.5-38B83.13.36
Gemma 3 27B IT80.43.31

W1.2: Additional Physical Metrics

We conduct physical evaluation with three additional metrics. VideoPhy-2 Physical Commonsense (PC) [1] takes a generated video as input and assesses its overall physical plausibility. PhyGenEval [2] takes a generated video, a scene description, physical laws, and evaluation questions as input to assess physical phenomena, event ordering, and overall naturalness.

[1] Bansal H, Peng C, et al. Videophy-2: A challenging action-centric physical commonsense evaluation in video generation

[2] Meng F, Liao J, et al. Towards world simulator: Crafting physical commonsense-based benchmark for video generation.

MethodPhysical Similarity ↑VideoPhy-2 PC ↑ (1–5)PhyGenEval ↑ (0–1)
Wan2.2-I2V-14B2.972.390.31
VAP-CogVideo2.922.160.34
VAP-Wan2.872.240.37
VACE2.142.150.29
VIPER (ours)3.423.570.49

The results show that VIPER achieves the best performance, providing broader evidence of VIPER's effectiveness beyond temporal smoothness and visual quality.

W2: Comparisons on Additional Baselines

PILA and DiffPhy is not open source, so we could not evaluation their performance. Consequently, we conducted additional comparisons PhysVid. To accommodate its text-to-video interfaces, we used Qwen3-VL-32B-Instruct to convert each target image into a text description of the subject and each reference video into a text description of the physical behavior. We then used Qwen3.5-35B-A3B to merge the two descriptions into a single generation prompt for each evaluation sample. The resulting prompt was supplied to PhysVid, while VIPER retained its native target-image, target-prompt, and reference-video inputs. The table below follows the evaluation metrics and column order of Table 1 in the paper.

MethodMotion Smoothness ↑Temporal Flickering ↑Aesthetic Quality ↑Physical Similarity ↑
PhysVid97.5696.8650.492.12
VIPER (Ours)98.6097.4551.863.42

VIPER achieves the best overall performance among the compared methods under this evaluation protocol.

W3: Evaluation Reliability and Uncertainty

W3.1: Evaluation Reliability

Please refer to our response to W1.1

W3.2: Evaluation Uncertainty

In the evaluation, we randomly sampled 75 pairs from the 500-pair validation set. All baseline comparisons and ablation studies used the same 75 pairs. The selected pairs had balanced category distributions within the Material, Trajectory, and Impact annotation dimensions. For the human study, 25 volunteers each evaluated 10–20 comparison sets, totaling approximately 400 evaluations.

To strengthen our evaluation, we have expanded the experiments to all 500 validation pairs for VIPER and all baselines. The results are presented in the table below.

MethodMotion Smoothness ↑Temporal Flickering ↑Aesthetic Quality ↑Physical Similarity ↑Preference Rank ↓
Wan2.1-I2V-14B97.7296.2950.082.552 (31.6%)
Wan2.2-TI2V-5B97.5196.7249.142.132 (31.6%)
Wan2.2-I2V-14B98.2596.0850.422.712 (31.6%)
VAP-CogVideo98.3497.2150.132.753 (6.1%)
VAP-Wan97.3295.9148.582.763 (6.1%)
VACE98.6297.5949.262.064 (2.1%)
VIPER (ours)98.5397.6651.703.241 (60.2%)

Additionally, we recruited 30 additional volunteers, increasing the human-study participant pool from 25 to 55, following the original human evaluation protocol.

CriterionVIPER (%) ↑Wan (%) ↑VAP (%) ↑VACE (%) ↑
Reference-video physical similarity58.233.06.52.3
Visual quality71.124.03.51.4
Physical plausibility51.337.88.32.6

W4: The necessity of Stage 1

Stage 1 is designed to establish a usable conditioning interface between the frozen MLLM and the pretrained video DiT. Although the MLLM can represent visual dynamics, its query representations are not automatically compatible with the generator. Starting directly with cross-video pairs would require the learnable queries and connector to solve two problems simultaneously. Stage 1 separates these learning objectives, where the conditioning input and supervised target originate from the same video. This provides a correspondence between the observed dynamics and the sequence to be generated. This initialization allows Stage 2 to build on an established conditioning interface when learning cross-video transfer. To further validate our method's improvements, we evaluated VIPER and using three additional physics-related metrics. Please refer to W1.2 for details. As shown in the table below, the results further support the advantages of our approach.

Method / VariantPhysical Similarity ↑VideoPhy-2 PC ↑ (1–5)PhyGenEval ↑ (0–1)
w/o Stage 13.412.760.39
VIPER3.423.570.49

Q1: Physical Plausibility Evaluation.

Q1.1: Physical Metrics

Please refer to our response to W1.2.

Q1.2: Evaluate on Physics-IQ Verified

Since VideoPhy-2 is designed for text-conditioned video generation, its input setting does not directly match our task. We select Physics-IQ Verified to our task. We use the switch frame at the third second as the target image and the 3–8-second segment of take-1 as the reference video. VIPER receives these inputs and generates a corresponding five-second video, which we evaluate using the Physics-IQ Verified metrics. As shown below, VIPER achieves promising results.

MethodSP ↑ST ↑WS ↑MSE Score ↑Physics-IQ Verified ↑
VIPER55.1323.4035.6838.7838.25

Q2: Physics-aware baseline comparisons.

Please refer to our response to W2.

Q3: Evaluation protocol and reliability

Please refer to our response to W1.

Q4: Stage 1 ablation

Please refer to our response to W4.

Thank you for the rebuttal and the added experiments. Two concerns remain; please respond directly within the limited discussion time.

1. Appearance-invariant physics evidence lacks baseline comparison. My original W1/Q1 asked for a measurement that is not VLM-judged, or a physics benchmark under a clear protocol. Of the three added metrics, VideoPhy-2 PC and PhyGenEval are themselves VLM-judged, so they do not replace the concern. Physics-IQ Verified is the only appearance-invariant benchmark, but numbers are reported only for VIPER (Q1.2) with no baseline under the same switch-frame protocol — so the table shows VIPER can be scored, not that it leads. Could the authors (a) run Wan2.2-I2V-14B and VACE (or VAP-CogVideo) on Physics-IQ Verified under the identical protocol, or (b) report one non-VLM scalar (e.g., contact-event ordering agreement, deformation-onset time error, or normalized trajectory IoU) across VIPER and the main baselines on the 500-pair set?

2. Stage 1 and main-table gaps are reported without uncertainty. The Stage 1 ablation (3.41 → 3.42 on Physical Similarity; wider on VideoPhy-2 PC and PhyGenEval) and the 500-pair main table both report single-run point estimates. Without seeds or paired tests, I cannot distinguish a real effect from run-to-run noise — especially for the 0.01-point primary-metric gap, but also for VIPER vs. VACE / Wan2.2-I2V-14B gaps of 0.5–1.3 on a 1–5 scale. Could the authors report a paired statistical test (per-pair sign test or bootstrap 95% CI) on the already-collected 500-pair per-sample scores? This is a post-hoc computation on existing data and should be feasible within the discussion window.

Based on the information currently available and any further clarification, I will finalize my recommendation during the reviewer–AC discussion phase.

Dear Reviewer 43rG:

We sincerely thank the reviewer for the constructive feedback and thoughtful consideration of our rebuttal. We are pleased that our responses have addressed your concerns, and we greatly appreciate your suggestions, which have helped us strengthen the manuscript.

We also thank you for raising these additional questions. We provide our clarifications below.

Baseline Comparisons for Appearance-Invariant Physics Evaluation

We thank the reviewer for identifying the remaining gap. We agree that VideoPhy-2 PC and PhyGenEval are VLM-based and therefore do not resolve the request for evidence independent of VLM judgment.

Following question (a), we have now evaluated Wan2.2-I2V-14B, VAP-Wan, VAP-CogVideo,VACE on Physics-IQ Verified. For Wan2.2-I2V-14B, We use the switch frame at the third second as the target image, and use verified prompt for physical conditioning. For VAP-Wan, VAP-CogVideo, VACE and VIPER, we use the switch frame at the third second as the target image and the 3–8-second segment of take-1 as the reference video. As shown below, VIPER achieves the highest score on Physics-IQ Verified owing to its strong ability to transfer physical dynamics from reference videos.

MethodSP ↑ST ↑WS ↑MSE Score ↑Physics-IQ Verified ↑
w/o Stage 150.1219.4626.6127.1929.25
Wan2.2-I2V-14B51.2220.7328.9729.3232.56
VAP-Wan52.3421.9130.4531.6734.09
VAP-CogVideo52.1520.5230.2431.1233.51
VACE53.6822.3032.4634.7935.81
VIPER55.1323.4035.6838.7838.25

Following question (b), we additionally adopt the evaluation protocol of PISA [1] and report foreground mean Intersection-over-Union (mIoU) as a non-VLM scalar on the full 500-pair evaluation set.

For each validation pair, we use Qwen3-VL-32B-Instruct to identify the main object from the target video and obtain textual prompt. These prompt are fixed are shared across VIPER and all baselines. We then use SAM 3 to segment the corresponding objects, obtaining per-frame binary masks. Following PISA, we compute foreground mask IoU. We average the scores across frames and evaluated objects within each pair. Qwen is used solely for object identification and does not assign physical-correctness or quality scores. The final scalar is computed directly from binary masks, providing a geometric measurement of object-position and spatial-occupancy agreement evaluation. Consequently, PISA is a non-VLM metric. As shown below, VIPER achieves the highest score.

MethodPISA foreground mIoU (%) ↑
w/o Stage 125.04
Wan2.2-I2V-14B18.82
VAP-Wan23.34
VAP-CogVideo19.03
VACE12.41
VIPER (Ours)36.97

[1] Li C, Michel O, Pan X, et al. Pisa experiments: Exploring physics post-training for video diffusion models by watching stuff drop[J].

Statistical Significance and Uncertainty of Reported Improvements

We conduct a post-hoc paired analysis using the retained per-sample scores for the Stage 1 ablation and baselines. We have added paired bootstrap analyses using the existing per-sample scores for both the Stage 1 ablation and the 500-pair main evaluation. For each sample, we additionally sampled five random seeds. For Physical Similarity and Physicas-IQ Verified, we align scores by test-pair ID and compute per-pair differences for the Stage 1 ablation and for VIPER against each baseline. For each comparison, we report the mean paired difference and its two-sided 95% percentile bootstrap confidence interval, obtained by resampling test pairs with replacement while preserving the pairing between methods. The results are summarized below:

ComparisonMetricMean difference95% CI
With Stage 1 − without Stage 1Physical Similarity+0.24[−0.22, 0.58]
VIPER − Wan2.2-I2V-14BPhysical Similarity+0.65[0.56, 0.74]
VIPER − VAP-WanPhysical Similarity+0.75[0.66, 0.84]
VIPER − VAP-CogVideoPhysical Similarity+0.70[0.61, 0.79]
VIPER − VACEPhysical Similarity+1.48[1.39, 1.57]
With Stage 1 − without Stage 1Physics-IQ Verified+9.00[7.68, 10.32]
VIPER − Wan2.2-I2V-14BPhysics-IQ Verified+5.69[4.37, 7.01]
VIPER − VAP-WanPhysics-IQ Verified+4.16[2.84, 5.48]
VIPER − VAP-CogVideoPhysics-IQ Verified+4.74[3.42, 6.06]
VIPER − VACEPhysics-IQ Verified+2.44[1.12, 3.76]

Finally, we sincerely hope that the additional evidence and clarifications have further strengthened the case for our approach. If there are any remaining concerns that may affect your assessment, we would be grateful for the opportunity to address them further. Thank you again for your thoughtful comments and constructive suggestions.

Reviewer_Fei5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VIPER studies reference-guided image-to-video generation, transferring physical behavior from a reference video to a target image and prompt. Learnable queries extract features from a frozen Qwen3-VL model; a connector conditions Wan2.2 through interface alignment, paired-video transfer training, and LoRA adaptation. The paper introduces VIPER-19K and reports improved reference-physics similarity and human preference over Wan, VAP, and VACE.

Critical strengths

S1. Relevant and clearly motivated control problem. Reference videos offer a useful way to specify material response, deformation, and interactions that are cumbersome to describe exhaustively in text. Separating target appearance from demonstrated behavior is a meaningful generation objective.

S2. Useful paired-data construction. VIPER-19K combines diverse video sources with material, trajectory, and impact annotations. Attribute-based pairing followed by MLLM filtering provides concrete supervision for transferring behavior across different visual conditions.

S3. Modular integration of pretrained models. Learnable queries and a connector establish the conditioning interface while both large models remain frozen; subsequent LoRA adaptation limits changes to the generator. This is a practical way to reuse existing understanding and generation models.

S4. Supporting automatic and human evidence. Reported physical similarity improves to 3.42 from 2.97 for the Wan2.2-I2V backbone and 2.92 for VAP-CogVideo. The 25-participant study also favors VIPER across the three criteria, providing evidence beyond automatic scores.

Critical weaknesses

W1. Limited differentiation from prior work. Video-as-Prompt already uses reference videos for in-context generation. VIPER adds physics-focused data and an MLLM query interface, but does not convincingly isolate the architectural advance over existing reference-conditioning approaches.

W2. Dependence on compatible reference videos limits applicability. Users must supply a reference demonstrating suitable behavior. Appendix F shows that mismatched physical contexts can cause implausible transfer when required forces or interactions are absent, limiting reliability beyond carefully selected reference-target pairs.

W3. Unexpected baseline behavior needs clarification. VACE and VAP obtain physical-similarity scores of 2.14 and 2.87–2.92, below Wan2.2-I2V's 2.97. This does not establish an implementation error, but checkpoints, reference-conditioning configurations, and adaptation settings need documentation to substantiate these comparisons. VACE remains stronger on temporal-quality metrics.

W4. Data and framework contributions are confounded. VIPER receives task-specific training, while equivalent VIPER-19K adaptation of VAP/VACE is not reported. Without a same-data Wan+LoRA or adapted-baseline control, the results cannot distinguish the framework's contribution from curated data and additional training.

W5. Some gains are marginal relative to the added complexity. Motion smoothness improves only from 98.58 to 98.60 over Wan2.2-I2V. Removing Stage 1 or Stage 3 changes physical similarity from 3.42 to 3.41 or 3.33. These small differences, without uncertainty estimates, do not establish the necessity of the full training schedule.

W6. Physics validation remains limited. Evaluation uses 75 held-out pairs, and Appendix D explicitly tolerates clipping, unrealistic stretching, and disappearing parts. The automatic score primarily measures broad behavioral agreement. The 25-participant study provides useful corroboration, but missing uncertainty estimates limit conclusions about reliable improvements in physical correctness.

Questions for authors
  1. What checkpoints and reference-conditioning settings were used for VAP/VACE, and were these models adapted on VIPER-19K?

  2. Is there an existing same-data Wan+LoRA or adapted-baseline comparison, with paired uncertainty estimates, that separates the framework's gains from data and training exposure?

5 comments

Response To Reviewer Fei5:

Thanks for your valuable suggestions. We will incorporate the relevant explanations into the final version for better understanding.

W1: Technical novelty of VIPER.

VIPER addresses a conditioning bottleneck: text or one image cannot fully specify continuous processes such as contact evolution, deformation onset, or material response. VIPER provides a complete reference-guided physical video generation pipeline: VIPER-19K reference-target pairs, MLLM extraction of transferable cues, lightweight MLLM-to-DiT injection, and multi-stage training. The lightweight connector deliberately preserves pretrained visual and physical priors rather than disrupting base-model capability with heavy modules, such a design might lead reviewers to mistakenly perceive it as a mere assembly of existing components.

W2: Applicability and Reliability of Reference-Based Physical Transfer

W2.1: Limited Applicability due to Reference Requirements

VIPER is designed to let users specify how a physical process should unfold through a visual demonstration. A reference video can convey physical details that are difficult to describe exhaustively in text. Users can provide the video itself without preparing depth maps, optical flow, or explicit force annotations. This offers a direct interface for controlling dynamics. Selecting a suitable reference nevertheless remains an additional user requirement.

W2.2: Unreliable transfer under mismatched physical contexts

We agree that Appendix F identifies a genuine limitation in adapting reference behavior to the target physical context. In Figure 12, the model transfers a deformation response without establishing the corresponding interaction in the target scene. Correctly recognizing the reference behavior is therefore insufficient to ensure physically plausible transfer.

The reference video serves as a control input that specifies the intended dynamics, and selecting a suitable reference remains an additional user requirement. Physical compatibility, however, does not require matching object identities or scene appearances. Section 5.4 and Appendix E provide qualitative examples across different target appearances and scenes, while Figure 11 demonstrates the use of synthetic or visually unrealistic references. These examples support flexibility in reference appearance and source, although they do not establish robustness to arbitrary reference–target combinations.

We will clarify this distinction in Section 4.1 and Appendix F. The current method supports reference-guided physical behavior transfer, while reliable adaptation when the target conditions lack the required interactions remains an unresolved limitation.

W3: Clarification of Baseline Performance and Experimental Settings

W3.1: Baselines Settings

All baselines use the target image and information from the reference video. For the three Wan baselines, by taking reference video as input, we use Qwen3-VL-4B-Instruct with the following prompt to generate a textual description of the reference video:

Describe the observed dynamics in temporal order: the initial physical configuration, visible interactions or contacts, resulting motion and material response, and the last observed state. Include only stages actually visible in the video.

Use generic physical roles, such as "a deformable body," "a contacting object," "a rigid surface," or "a fluid," rather than specific object identities. Preserve physically relevant geometry, support and contact conditions, camera-relative motion directions, changes in speed, deformation, recovery, fragmentation, and event timing when observable. Retain any interacting agent or boundary necessary to describe the physical process.

Distinguish visible interactions from unseen causes. If a driving cause is not visible, state this briefly without inventing forces, contacts, or mechanisms. Do not assume an eventual outcome beyond the observed sequence.

Exclude appearance details unrelated to physics, including color, texture, lighting, visual style, and decorative background content. Do not include numerical material constants, unsupported physical claims, evaluation scores, quality praise, method preferences, or reasoning commentary.

Output the physical description directly, without headings, field names, JSON, bullet points, or introductory phrases.

We directly append the resulting abstract description of the reference dynamics to the original generation prompt.

VACE and VAP use the target image and reference video as inputs, similar to VIPER. For VACE, we use the Wan-AI/Wan2.1-VACE-14B checkpoint, passing the target image to src_ref_images and the reference video to src_video. For VAP, we use the ByteDance/Video-As-Prompt-Wan2.1-14B and ByteDance/Video-As-Prompt-CogVideoX-5B checkpoints with WanImageToVideoMOTPipeline and CogVideoXImageToVideoMOTPipeline, respectively. The inference settings for all baselines are summarized below.

MethodSamplerStepsGuidance / CFGAdditional inference settings
Wan2.1-I2V-14B-480PUniPC405shift = 3
Wan2.2-TI2V-5BUniPC505shift = 5
Wan2.2-I2V-A14BUniPC40(3.5, 3.5)shift = 5; boundary = 0.9
VAP-CogVideoDDIM506dynamic CFG enabled
VAP-WanUniPC505flow shift = 3
VACEUniPC505shift = 16; context scale = 1

W3.2: Clarification of Baseline Performance

VACE's temporal advantage is consistent with its conditioning design. This structural guidance can constrain frame-to-frame evolution, providing a plausible explanation for smooth and visually stable outputs. However, a temporally coherent sequence can still exhibit an incorrect material response or interaction. Section 5.3 also reports copying-like VACE outputs, while our rubric penalizes near-identical reference reproduction. Such outputs can preserve temporal continuity yet fail at target adaptation.

Additionally, we conduct physical evaluation with three additional metrics. VideoPhy-2 Physical Commonsense (PC) [1] takes a generated video as input and assesses its overall physical plausibility. PhyGenEval [2] takes a generated video, a scene description, physical laws, and evaluation questions as input to assess physical phenomena, event ordering, and overall naturalness.

[1] Bansal H, Peng C, Bitton Y, et al. Videophy-2: A challenging action-centric physical commonsense evaluation in video generation[C]//International Conference on Learning Representations. 2026, 2026: 118456-118470.

[2] Meng F, Liao J, Tan X, et al. Towards world simulator: Crafting physical commonsense-based benchmark for video generation[J]. arXiv preprint arXiv:2410.05363, 2024.

MethodPhysical Similarity ↑VideoPhy-2 PC ↑ (1–5)PhyGenEval ↑ (0–1)
Wan2.2-I2V-14B2.972.390.31
VAP-CogVideo2.922.160.34
VAP-Wan2.872.240.37
VACE2.142.150.29
VIPER (ours)3.423.570.49

The results show that VIPER achieves the best performance, providing broader evidence of VIPER’s effectiveness beyond temporal smoothness and visual quality.

W4: Concern on Data–Framework Confounding

We conducted an additional same-data adaptation study on VIPER-19K, comparing VIPER with Wan2.2-I2V-14B+LoRA, VAP-CogVideo, VAP-Wan, and VACE. Each baseline was fine-tuned for 6,000 steps using LoRA with rank 64 and alpha 32.

Since Wan2.2 does not natively accept reference videos, we followed the evaluation protocol and converted each reference video into a textual description of its physical behavior, which was appended to the target prompt. The target image was provided through the original image-to-video pathway. VAP and VACE retained their native training pipelines and reference-conditioning interfaces. As shown below, our method achieves the best, indicating that the VIPER provides gains beyond those attributable to curated data.

ConfigurationMotion Smoothness ↑Temporal Flickering ↑Aesthetic Quality ↑Physical Similarity ↑
Wan2.2-I2V-14B + LoRA98.4897.1850.963.02
VAP-CogVideo + LoRA98.4997.3250.542.98
VAP-Wan + LoRA97.4296.0450.142.93
VACE + LoRA98.7897.5149.872.41
VIPER (ours)98.6097.4551.863.42

W5: Training-Schedule Necessity

To further validate our method's improvements in physical plausibility, we evaluated VIPER and the baselines using three additional physics-related metrics. Please refer to W3.2 for details. As shown in the table below, the results further support the advantages of our approach.

Method / VariantPhysical Similarity ↑VideoPhy-2 PC ↑ (1–5)PhyGenEval ↑ (0–1)
w/o Stage 13.412.760.39
w/o Stage 33.332.790.41
VIPER (full model)3.423.570.49

W6: Concerns on Limited Physics Validation

W6.1: Extended Evaluation

We have expanded the evaluation to all 500 pairs to evaluate VIPER and all baselines. The table below shows the results.

MethodMotion Smoothness ↑Temporal Flickering ↑Aesthetic Quality ↑Physical Similarity ↑Preference Rank ↓
Wan2.1-I2V-14B97.7296.2950.082.552 (31.6%)
Wan2.2-TI2V-5B97.5196.7249.142.132 (31.6%)
Wan2.2-I2V-14B98.2596.0850.422.712 (31.6%)
VAP-CogVideo98.3497.2150.132.753 (6.1%)
VAP-Wan97.3295.9148.582.763 (6.1%)
VACE98.6297.5949.262.064 (2.1%)
VIPER (ours)98.5397.6651.703.241 (60.2%)

Additionally, we recruited 30 additional volunteers, increasing the human-study participant pool from 25 to 55, following the original human evaluation protocol.

CriterionVIPER (%) ↑Wan (%) ↑VAP (%) ↑VACE (%) ↑
Reference-video physical similarity58.233.06.52.3
Visual quality71.124.03.51.4
Physical plausibility51.337.88.32.6

W6.2: Additional Physical Metrics.

To further validate our method, we evaluate its performance using three additional physical metrics. Please refer to our response to W3.2 for details.

Q1: VAP/VACE Configuration and Dataset Adaptation

Q1.1: Details of VAP/VACE

Please refer to our response to W3.1.

Q1.2: Train VAP/VACE on VIPER-19K

Please refer to our response to W4.

Q2: Same-data Wan+LoRA or adapted-baseline Comparison.

Please refer to our response to W4.

Reviewer_AZwcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VIPER, which is a reference-guided image-to-video framework that uses a reference video to condition the physical behavior of a generated video, while the target image and prompt specify its content. The method extracts cues from the reference with a multimodal language model, and inject them into a pretrained video generator, which is trained through a three-stage process. The work also present VIPER-19K, a dataset of annotated videos and reference-target pairs. Results on 75 held-out pairs show higher physical-similarity scores and stronger human preference for VIPER than the reported baselines, while general video-quality metrics remain competitive.

Critical strengths
  1. The paper discovers a practical limitation of text-conditioned video generation, which is that detailed physical behavior can be difficult to specify in texts. Using a video demonstration as a conditioning signal is an intuitive thought to this limitation.
  2. The ablation between target appearance and reference behavior is clear. The three-stage training setup is supported by ablations. In particular, removing the learnable queries substantially reduces the physical-similarity score.
  3. VIPER-19K combines real-world, generated, and simulated videos, which could inspire useful research on transferring physical behavior across scenes.
  4. The evaluation includes both automated measures and a human preference study. The qualitative example also help explain the intended behavior transfer.
Critical weaknesses
  1. The central physical-similarity result relies heavily on a VLM judge. The appendix's judging prompt explicitly presents to be highly lenient toward generation flaws, and that positive scores should be the default, which could inflate scores or makes comparisons depend on the judge's preferences. The manuscript does not report enough information to assess the judge's reliability or sensitivity to prompt wording.
  2. The evaluation is a bit small. 75 of the 500 validation pairs are sampled for the comparison experiments, and the manuscript does not provide confidence intervals. The human study has 25 participants, but the manuscript gives limited detail on vote counts and uncertainty. It is therefore a bit difficult to tell how stable the reported improvements are. I suggest the authors enhance this part significantly.
  3. The dataset construction needs stronger validation. Videos receive physical labels, and pairs are filtered with the MLLM, but the manuscript does not establish how reliable those labels and pair decisions are through human auditing, agreement measures, or error analysis at dataset scale. The manuscript also show that coarse labels can group videos with different motion direction or incompatible physical contexts.
  4. The baseline comparison is hard to fully interpret. For methods without native video conditioning, reference information is converted to texts, while other methods receive their own video-conditioning interfaces. More details on prompt construction, input settings, and tuning would help to establish how much of the gain come from VIPER's learned conditioning versus differences in the comparison protocol.
Questions for authors
  1. Which model provide the VLM-as-Judge scores, and how sensitive are the results to changes in the judging prompt/judge model? Could you include a human-validated subset or an evaluation with a less lenient rubric?
  2. Are the train/validation/test splits separated by original source video and source dataset before constructing pairs? How did the authors check for near-duplicate clips, or related reference-target pairs across splits?
  3. How are the physical annotations and pair-filtering decisions validated? It's better to report human audit results or agreement metrics, including the types of errors.
4 comments

Response To Reviewer AZwc:

Thanks for your valuable suggestions. We will incorporate the relevant explanations into the final version for better understanding.

W1. Reliability and prompt sensitivity of the VLM judge.

Our VLM-as-Judge score measures whether a generated video reproduces the material response, motion, and interactions demonstrated by the reference. The lenient prompt was designed to prevent imperfections in video generation from dominating this assessment of behavior transfer. During prompt development, we qualitatively observed prompt-induced harshness: instructions emphasizing strict error detection sometimes produced low scores for reference–generation pairs that humans judged to have similar physical. This observation motivates a controlled comparison of three variants. We vary the evaluation tolerance instructions and scoring anchors in Figure 9, while keeping other evaluation implementation unchanged.

PromptKey changes relative to Figure 9Core scoring logic
StrictReplace the instruction to be highly lenient and the positive-score default with strict scrutiny. Remove the instruction to overlook clipping, unnatural stretching, and disappearing pieces. Tighten the scoring anchors.Deduct for observable inconsistencies in material response, motion, contact, and causal timing, even when the intended event is recognizable. Reserve 5 for close agreement with the demonstrated dynamics.
NeutralRemove the positive-score default and blanket tolerance for physical errors. Distinguish rendering artifacts from errors that alter the physical event. Use evidence-based anchors across the full score range.Ignore flickering or appearance noise when the dynamics remain intact; penalize incorrect physical behavior in proportion to its effect on the transferred event.
Lenient (ours)Retain the original mindset, tolerance instructions, and scoring anchors.Prioritize recognizable transfer of the reference dynamics. Allow local approximations; give partial credit when the intended behavior remains identifiable, while penalizing major contradictions and literal clones.

To assess sensitivity to the judging prompt, we compare strict, neutral, and lenient prompts using Qwen3-VL-32B-Instruct on the same generated outputs, following our human-study protocol. Human volunteers provide 1–5 physical-similarity scores and select their preferred output for each comparison. We report human preference agreement and mean score bias. Human preference agreement measures the percentage of comparisons in which the judge’s highest-scoring output matches the human votes. Mean score bias is the average difference between judge scores and the mean human ratings, negative values indicate under-scoring, and positive values indicate over-scoring. As shown below, Strict and Neutral prompts would assign lower scores than human assessors on average, whereas the Lenient prompt would achieve the highest human preference agreement with a mean score bias close to zero. This would support the lenient prompt as a way to align the assessment with human judgments.

Qwen3-VL-32B-Instruct promptHuman preference agreement (%) ↑Mean score bias (0 is ideal)
Strict64.0−1.54
Neutral76.0−1.01
Lenient (ours)84.0+0.24

To examine dependence on model, we evaluate the same generated outputs with Qwen3-VL-32B-Instruct, InternVL3.5-38B, and Gemma-3-27B-IT. As shown below, all achieve high agreement with human preferences, providing evidence that the metric is less dependent on the preferences of a single judge.

JudgeHuman preference agreement (%) ↑VLM-as-Judge ↑
Qwen3-VL-32B-Instruct84.03.42
InternVL3.5-38B83.13.36
Gemma 3 27B IT80.43.31

W2. Evaluation scale, vote counts, and uncertainty.

We have expanded the evaluation to all 500 to evaluate VIPER and all baselines. The table below shows the results.

MethodMotion Smoothness ↑Temporal Flickering ↑Aesthetic Quality ↑Physical Similarity ↑Preference Rank ↓
Wan2.1-I2V-14B97.7296.2950.082.552 (31.6%)
Wan2.2-TI2V-5B97.5196.7249.142.132 (31.6%)
Wan2.2-I2V-14B98.2596.0850.422.712 (31.6%)
VAP-CogVideo98.3497.2150.132.753 (6.1%)
VAP-Wan97.3295.9148.582.763 (6.1%)
VACE98.6297.5949.262.064 (2.1%)
VIPER (ours)98.5397.6651.703.241 (60.2%)

Additionally, we recruited 30 additional volunteers, increasing the human-study participant pool from 25 to 55, following the original human evaluation protocol.

CriterionVIPER (%) ↑Wan (%) ↑VAP (%) ↑VACE (%) ↑
Reference-video physical similarity58.233.06.52.3
Visual quality71.124.03.51.4
Physical plausibility51.337.88.32.6

W3: Stronger Validation of Dataset Construction

Coarse material, trajectory, and impact labels are used to retrieve candidate pairs. Final selection additionally checks whether their material responses, motion directions, and interaction contexts support physical transfer, as shown in Section 3.2 and Appendix B. To validate both stages independently, we recruit five volunteers to audit 200 videos and 200 pairs, sampled across data sources and physical categories. Volunteers assign labels and judge pair compatibility without seeing the MLLM's decisions. We compare model annotations with adjudicated human labels and report Accuracy. The experimental results indicate that our dataset construction pipeline achieves a reasonably high level of accuracy.

Audit measureAccuracy
Material-label0.91
Trajectory-label0.92
Impact-label0.94

W4: Implementation Information for Baselines

All baselines use the target image and information from the reference video. For the three Wan baselines, by taking reference video as input, we use Qwen3-VL-4B-Instruct with the following prompt to generate a textual description of the reference video:

Describe the observed dynamics in temporal order: the initial physical configuration, visible interactions or contacts, resulting motion and material response, and the last observed state. Include only stages actually visible in the video.

Use generic physical roles, such as "a deformable body," "a contacting object," "a rigid surface," or "a fluid," rather than specific object identities. Preserve physically relevant geometry, support and contact conditions, camera-relative motion directions, changes in speed, deformation, recovery, fragmentation, and event timing when observable. Retain any interacting agent or boundary necessary to describe the physical process.

Distinguish visible interactions from unseen causes. If a driving cause is not visible, state this briefly without inventing forces, contacts, or mechanisms. Do not assume an eventual outcome beyond the observed sequence.

Exclude appearance details unrelated to physics, including color, texture, lighting, visual style, and decorative background content. Do not include numerical material constants, unsupported physical claims, evaluation scores, quality praise, method preferences, or reasoning commentary.

Output the physical description directly, without headings, field names, JSON, bullet points, or introductory phrases.

We directly append the resulting abstract description of the reference dynamics to the original generation prompt.

VACE and VAP use the target image and reference video as inputs, similar to VIPER. For VACE, we use the Wan-AI/Wan2.1-VACE-14B checkpoint, passing the target image to src_ref_images and the reference video to src_video. For VAP, we use the ByteDance/Video-As-Prompt-Wan2.1-14B and ByteDance/Video-As-Prompt-CogVideoX-5B checkpoints with WanImageToVideoMOTPipeline and CogVideoXImageToVideoMOTPipeline, respectively. The inference settings for all baselines are summarized below.

MethodSamplerStepsGuidance / CFGAdditional inference settings
Wan2.1-I2V-14B-480PUniPC405shift = 3
Wan2.2-TI2V-5BUniPC505shift = 5
Wan2.2-I2V-A14BUniPC40(3.5, 3.5)shift = 5; boundary = 0.9
VAP-CogVideoDDIM506dynamic CFG enabled
VAP-WanUniPC505flow shift = 3
VACEUniPC505shift = 16; context scale = 1

Q1: Reliability and prompt sensitivity of the VLM judge.

Please refer to our reponse to W1.

Q2: Dataset splitting and duplicate pairs

As described in Sections 3.1–3.2, we first collect videos from different sources and annotate their material, trajectory, and physical-impact attributes. We then group videos by compatible attributes, sample candidate pairs within each group, and use Qwen3-VL-32B to filter out pairs with incompatible physical behaviors. Dataset splitting is performed after the complete set of pairs has been constructed. During pair construction, (reference_video_id, target_video_id) is used as a unique identifier to ensure that each combination is retained only once. Consequently, an identical reference–target combination cannot appear in multiple splits.

Similarity in physical behavior does not imply duplicate samples. For example, two video pairs may both depict objects deforming under compression, but they are considered distinct samples if they use different reference–target combinations. Their shared physical properties are intentionally preserved, as they reflect the underlying physical regularities that the model is expected to learn from the dataset. Thus, our dataset construction process does not produce duplicate samples. Combining different videos increases the diversity of the resulting paired dataset.

Q3: Human Validation of Physical Annotations.

Please refer to our response to W3

Thanks for the rebuttal. I still have some further concerns regarding your responses:

W1 Given that both the vlm judge and the human volunteers assign 1–5 scores, the "human preference agreement" can be reported as correlation (e.g., Spearman's or Kendall's) rather than such ranking-based agreement in percents or averaged error. Currently, both the "human preference agreement" and "mean score bias" can discard score magnitudes and is ambiguous when scores tie.

W2 For human study, I think the following items are important to show the effectiveness of the research: questions distributed vs. responses collected, whether all 500 conditions are covered, any validity checks or exclusion criteria, whether output order was randomized per question, and per-question vote counts with inter-rater agreement. Also, Sec 5.2 says "10–20 comparison groups" but Appendix D says "15–20 questions" — please clarify.

Dear Reviewer AZwc:

We sincerely thank the reviewer for the constructive feedback and thoughtful consideration of our rebuttal. We are pleased that our responses have addressed your concerns, and we greatly appreciate your suggestions, which have helped us strengthen the manuscript.

We also thank you for raising these additional questions. We provide our clarifications below.

Detailed Analysis of VLM judge.

We agree that the two metrics originally reported have limitations. Human preference agreement considers only the top-ranked output and overlooks information about the remaining outputs, while mean score bias can mask discrepancies through the cancellation of positive and negative errors. We therefore additionally report Kendall’s and Spearman’s to assess human–VLM agreement. Each of the 55 volunteers was assigned 20 comparison groups, each containing videos generated by Wan, VAP-Wan, VACE, and VIPER. Volunteers rated the physical similarity of each output on a 1–5 scale. The same outputs were also scored by the VLM. We then computed Kendall’s and Spearman’s between the VLM scores and the mean human ratings. As shown below, the Lenient prompt achieves the highest correlation with human ratings under both metrics.

Qwen3-VL-32B-Instruct promptKendall’s ↑Spearman’s ↑
Strict0.430.58
Neutral0.610.72
Lenient (ours)0.740.86

W2. Details of the extended human study.

Thank you for highlighting the need for a more detailed study protocol. We recruited 30 additional volunteers, increasing the participant pool from 25 to 55. The extended study sampled comparison questions from the full validation set of 500 conditions. Each question presented outputs from VIPER, Wan2.2-I2V-14B, VAP-Wan, and VACE under the same condition. Participants select the best output on every evaluation dimension. The display order of the four outputs was independently randomized for each question. All 55 participants, including the original 25 volunteers, completed 20 questions each. These questions were independently and randomly sampled for each participant, yielding 1,100 assigned questions and 1,100 completed question-level responses. We applied two response-quality checks. First, responses submitted within 5 seconds were excluded under our minimum completion-time criterion. Second, responses exhibiting straight-lining, i.e., repeatedly selecting the same option across questions, were excluded. These checks removed 14 question-level responses, leaving 1,086 valid responses for analysis.

Clarification of Sec. 5.2 and Appendix D.

We apologize for the inconsistent question counts. In the initial human study, we adopted a staggered recruitment strategy to make the most of the time available for experiments and manuscript submission. We planned to recruit 25 volunteers. After issuing a call for volunteers, we immediately assigned 20 sample groups to each of the first 10 applicants and conducted the first round of evaluation. We then recruited a second batch of 10 volunteers. Since the original evaluation involved only 75 sample groups, and the first round of evaluation with 20 groups per volunteer had already covered the entire validation dataset, we assigned 15 sample groups to each volunteer in the second batch to expedite the human study. Similarly, volunteers in the third batch received only 10 sample groups each. Sec. 5.2 and Appendix D were written at different stages. We drafted Appendix D first, when recruitment had reached only the second batch, so it reported 15–20 sample groups. By the time we wrote Sec. 5.2, the entire human study had been completed, so it reported 10–20 sample groups. However, we forgot to update Appendix D accordingly, resulting in the inconsistency.

We sincerely apologize for the inconsistent descriptions in the appendix and the main text. We have revised the appendix by changing the lower bound from 15 to 10 to resolve this inconsistency. During the rebuttal phase, we standardized the number of questions, assigning exactly 20 sample groups to each volunteer. For the extended human study, we combined the original evaluation results with the extended evaluation results for analysis. Therefore, we reported 10–20 sample groups per volunteer in the rebuttal.

Finally, we sincerely hope that the additional evidence and clarifications have further strengthened the case for our approach. If there are any remaining concerns that may affect your assessment, we would be grateful for the opportunity to address them further. Thank you again for your thoughtful comments and constructive suggestions.

Comments on the paper

1 comment

We thank Reviewers AZwc, Fei5, and 43rG for their careful reading and constructive feedback. We appreciate their recognition of the following strengths:

  • Motivation (AZwc, Fei5, 43rG). VIPER provide a natural way to specify material response, deformation, and interactions that are difficult to fully describe in text.
  • Framework (AZwc, Fei5, 43rG). The reviewers appreciated the separation of target appearance from reference dynamics and the use of a modular, query-based conditioning interface. AZwc specifically highlighted the learnable-query ablation.
  • Dataset (AZwc, Fei5). VIPER-19K combines real, generated, and simulated videos with physical annotations and reference–target pairs that support learning across different visual settings.
  • Evaluation (AZwc, Fei5, 43rG). The reviewers valued the complementary use of automatic metrics and human judgments.

In response to the reviewers' constructive suggestions, we acknowledge that our work can be further improved and have addressed all concerns as follows:

  • Evaluation scale and human-study protocol (AZwc, Fei5, 43rG). We expanded the main comparison from 75 to all 500 validation pairs. In the extended human study, each of the 55 participants completed 20 comparison groups with randomized output order, yielding 1,086 valid group-level responses. We clarify that the original study's 10–20 groups referred to the number per participant.
  • Judge model sensitivity and agreement with humans (AZwc, 43rG). We used Qwen3-VL-32B-Instruct as our primary judge and compared three prompts, each implementing a different scoring rubric, to assess how closely its physical similarity scores align with human judgments. We also tested two additional VLM judges and observed comparable agreement with human preferences, suggesting that this agreement is not specific to a single judge model.
  • Physics benchmark evaluation (Fei5, 43rG). Alongside VideoPhy-2 PC and PhyGenEval, we evaluated VIPER and the baselines under an adapted Physics-IQ Verified protocol. Additionally, we report PISA-style foreground mask overlap on all 500 validation pairs. A VLM identifies which objects to segment, but the score is computed directly from the masks.
  • Spatial agreement without VLM scoring (43rG). This quantifies spatial agreement with the target video; it does not directly measure normalized trajectories, contact-event ordering, or physical correctness.
  • Baseline configuration and access to reference information (AZwc, Fei5). We document the checkpoints, prompts, conditioning inputs, and inference settings. In the main comparison, the Wan baselines receive textual descriptions of the reference dynamics, while VAP and VACE receive the reference video through their native interfaces. The comparison is therefore not between methods with and without access to reference information.
  • Same-data adaptation controls (Fei5). We also fine-tuned Wan, VAP, and VACE on VIPER-19K using LoRA. VIPER retained its advantage in physical similarity over these baselines trained on the same data, suggesting that its architecture and multistage training strategy contribute to the gains beyond what the dataset alone provides.
  • Technical contribution beyond video prompting (Fei5). VIPER’s contribution is not the use of a reference video itself. Its learnable physics queries and lightweight MLLM-to-DiT connector are designed to select transferable dynamics from the reference while leaving the target image and prompt to specify the output’s appearance.

We thank the reviewers again for their constructive comments. We hope that the additional evidence and revisions clarify the scope of our contributions and address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.