- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VidPhysOracle evaluates image-conditioned video generation against simulated reference futures. It provides 960 scenes across twelve scenario families covering rigid bodies, soft bodies, liquids, and gases, together with auxiliary annotations and 6,272 appearance-translated videos. The evaluation combines material-specific geometry, trajectory, deformation, and support-change measurements. Experiments benchmark seven models, examine resolution sensitivity and human agreement, and demonstrate soft-body adaptation through LoRA fine-tuning.
Critical strengths
The material coverage makes the benchmark potentially useful. The suite includes ropes, cloth, volumetric soft bodies, liquids, and smoke, with coupled interactions and simulation-derived annotations. This is a meaningful extension of rigid-body-focused simulation benchmarks and supports evaluating phenomena that require different physical observables.
Several evaluation choices address concrete failure modes: fixed spatial and temporal alignment retains position and timing errors. The temporal support term also penalizes frozen predictions that can retain substantial spatial overlap. The controlled examples in Table 12 demonstrate this complementary role. The resolution study provides useful evidence about robustness to spatial resampling (Sec. 4.5).
Critical weaknesses
The additional diagnostic value of the scoring design is insufficiently established: Reference-based evaluation is already established by Physics-IQ and RigidBench, while Sec. 4.3 provides a coverage comparison rather than an empirical comparison of evaluators. The human study uses one evaluator and one model pair, judging plausibility rather than the stated target of reference fidelity (Sec. 3.4; App. E). The liquid aggregate matches mean IoU at 98.0% agreement. The soft-body aggregate reaches 93.6%, compared with 95.7% for boundary Chamfer (Tables 9–10). The controlled tests are useful but cover only two liquid references (Table 12). These results do not establish that the added components produce more informative or reliable model rankings.
The main quantitative results are internally inconsistent (? Or maybe I am not understanding correctly? pls correct me if I am wrong.) Figure 2’s rigid-body scores disagree with Table 2, including the ordering of Wan2.2-TI2V-5B and Cosmos-Predict2.5-2B. The Wan2.2 soft-pad baseline also changes from 76.4 in Table 2 to 34.05 in Table 6. Section 5 specifies 128 soft-pad evaluation samples, but neither that section nor Appendix H explains evaluation differences that reconcile these values. These discrepancies affect both the benchmark comparison and the interpretation of the reported 55.71-point fine-tuning gain.
These are making my scores a bit lower at this time - I look forward to the authors' reply.
Questions for authors
What accounts for the discrepancies between Figure 2 and Table 2, and between the soft-pad baselines in Tables 2 and 6? Please specify any differences in evaluation instances, appearance variants, inference settings, or metric implementations.
Can you provide comparisons on shared model outputs showing what the composite scores detect beyond simple geometry metrics or existing reference-based evaluators? Evaluation across additional model pairs, using judgments of reference fidelity and reporting uncertainty, would materially affect my assessment.
5 comments
Responses to Reviewer dyrd (Part 2/2)
W1 & Q2: Additional diagnostic value of the composite scoring design.
Thank you for suggesting a broader reference-fidelity evaluation and a direct comparison with geometric baselines. We expanded the study to four models—Wan2.2-TI2V-5B, Cosmos-Predict2.5-2B, Cosmos3-Edge, and HunyuanVideo-1.5—with the present analysis using 10 annotators and 3,200 judgments on 640 comparisons from 320 reference instances across twelve scenario families. All six model pairings are covered. Annotators view the reference and anonymous predictions, judging reference fidelity rather than unrestricted physical plausibility.
As a supplementary analysis, we select comparisons with at least 80% of assigned annotators preferring the same prediction: 4/5 for every material category. Ties and unjudgeable responses do not support either direction and remain in the assigned-rater denominator. Selection uses human labels only, independently of metric predictions, retaining 247/640 comparisons (38.59%). Each retained comparison contributes one consensus label. All four metrics are compared on exactly the same consensus comparisons within each category:
| Material | Consensus comparisons / all | Composite | IoU | Chamfer | Centroid |
|---|---|---|---|---|---|
| Rigid | 76/160 | 75.00% | 64.47% | 73.68% | 69.74% |
| Soft | 64/160 | 82.81% | 84.38% | 81.25% | 84.38% |
| Liquid | 49/160 | 79.59% | 65.31% | 65.31% | 59.18% |
| Gas | 58/160 | 77.59% | 75.86% | 75.86% | 74.14% |
We report paired agreement differences between the composite and IoU, Chamfer, and Centroid below, in percentage points. Intervals use 10,000 scenario-stratified reference-cluster bootstrap replicates, retaining comparisons sharing a reference together and holding observed consensus labels fixed. These intervals are conditional on the participating annotators and observed consensus subset.
| Material | Composite−IoU [95% CI] | Composite−Chamfer [95% CI] | Composite−Centroid [95% CI] |
|---|---|---|---|
| Rigid | +10.53 [+2.53, +18.61] | +1.32 [-4.29, +7.32] | +5.26 [-1.45, +12.50] |
| Soft | -1.56 [-8.33, +5.26] | +1.56 [-3.33, +7.14] | -1.56 [-5.00, +0.00] |
| Liquid | +14.29 [-2.50, +32.08] | +14.29 [-2.13, +31.58] | +20.41 [+3.70, +38.64] |
| Gas | +1.72 [-9.38, +13.11] | +1.72 [-7.46, +10.77] | +3.45 [-12.50, +18.97] |
Complementary measurements expose errors that spatial overlap can miss.
The composite's diagnostic role is concrete. In the controlled tests in the submitted Table 12, freezing either liquid reference retains spatial agreement near 0.7 but produces zero temporal agreement and a zero total score. This detects missing evolution even when the occupied region still overlaps substantially. The expanded human study compares metrics against consensus judgments across multiple models and all material categories.
Physics-IQ and RigidBench already establish reference-based evaluation. VidPhysOracle extends this approach across material regimes and matches the observables to motion, deformation, and evolving fluid support. The controlled tests establish complementary error sensitivity, while the consensus-based comparisons quantify alignment with human reference-fidelity judgments.
Looking forward to your feedback
Dear Reviewer dyrd,
We would like to sincerely thank you again for your thoughtful feedback and valuable suggestions.
We have provided detailed responses and additional explanations addressing your concerns in our rebuttal. As the discussion deadline is approaching, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your concerns.
If you have any remaining concerns or further questions, we would be more than happy to discuss them and provide additional clarifications.
Thank you again for your time and consideration.
Best regards, Authors
Responses to Reviewer dyrd (Part 1/2)
Thank you for identifying these inconsistencies. We have corrected Figure 2 and rerun the paired Base/LoRA evaluation. The matched results retain substantial adaptation gains: +34.45 points on Raw inputs and +25.54 points on Transfer inputs.
Figure 2 was not synchronized with the updated tabulated results before submission. This was an oversight in manuscript preparation. We have corrected Figure 2 in the revised manuscript to make it consistent with Table 2.
The original fine-tuning evaluation was not fully aligned with the main benchmark configuration, and the relevant differences were insufficiently documented. We regenerated both Base and LoRA predictions under an explicitly matched protocol.
Both models use the same 128 soft-pad source instances, with two conditioning settings:
Base and LoRA share identical inputs, prompts, and paired generation seeds, yielding 512 predictions in total.
Both generate 49 frames at 1024×576 and 24 fps, using 30 sampling steps and CFG=5, matching these settings in the main benchmark. Flow shift is 5. LoRA uses the step-4500 checkpoint with rank=32, alpha=32, and scale=1.
Revised reporting uses matched gains and explicit evaluation populations. We replace the original Table 6 results and its 55.71-point gain with the matched results above. The revised description specifies the inputs, inference settings, and selected appearances. The fine-tuning evaluation uses one Transfer appearance per source instance, rather than all appearance variants in the full benchmark; its baseline is therefore a subset-specific estimate.
LoRA gains are calculated within each matched Base/LoRA experiment, rather than by comparing the fine-tuning result with the Table 2 baseline.