acceptodds
Under review as a conference paper at ICLR 2027

VidPhysOracle: Benchmarking Video World Models Against Physically Grounded Futures

Abstract

Recent video generation and world models can synthesize visually compelling futures, yet their ability to capture real-world physics remains unclear. Existing physics evaluations largely rely on hand-crafted rules or learned judges, requiring the evaluator to infer what should physically happen. We introduce VidPhysOracle, a benchmark and physically grounded video dataset that evaluates generated futures against oracle trajectories produced by physical simulation. Rather than asking whether a video merely looks physically plausible, VidPhysOracle measures how closely its dynamics match a known physical evolution.The dataset covers diverse rigid-body, deformable/soft-body, and fluid systems with controlled initial conditions and corresponding simulated futures. To reduce the simulation-to-real visual gap, we construct physics-preserving photorealistic videos through appearance transformation while retaining the underlying geometry and motion. We further introduce a reference-based unified physical consistency score to quantify dynamic agreement beyond pixel-level similarity, and benchmark seven representative open-source video and world models under identical initial conditions.Beyond evaluation, VidPhysOracle provides physically valid future trajectories for physics-aware fine-tuning and post-training, enabling existing video models to improve physical consistency. Overall, VidPhysOracle offers a unified framework for evaluating, diagnosing, and improving the physical capabilities of video world models.

Primary area:
datasets and benchmarks
Keywords:
Video Generation, Video Benchmark, Physical Simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dyrdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VidPhysOracle evaluates image-conditioned video generation against simulated reference futures. It provides 960 scenes across twelve scenario families covering rigid bodies, soft bodies, liquids, and gases, together with auxiliary annotations and 6,272 appearance-translated videos. The evaluation combines material-specific geometry, trajectory, deformation, and support-change measurements. Experiments benchmark seven models, examine resolution sensitivity and human agreement, and demonstrate soft-body adaptation through LoRA fine-tuning.

Critical strengths

The material coverage makes the benchmark potentially useful. The suite includes ropes, cloth, volumetric soft bodies, liquids, and smoke, with coupled interactions and simulation-derived annotations. This is a meaningful extension of rigid-body-focused simulation benchmarks and supports evaluating phenomena that require different physical observables.

Several evaluation choices address concrete failure modes: fixed spatial and temporal alignment retains position and timing errors. The temporal support term also penalizes frozen predictions that can retain substantial spatial overlap. The controlled examples in Table 12 demonstrate this complementary role. The resolution study provides useful evidence about robustness to spatial resampling (Sec. 4.5).

Critical weaknesses

The additional diagnostic value of the scoring design is insufficiently established: Reference-based evaluation is already established by Physics-IQ and RigidBench, while Sec. 4.3 provides a coverage comparison rather than an empirical comparison of evaluators. The human study uses one evaluator and one model pair, judging plausibility rather than the stated target of reference fidelity (Sec. 3.4; App. E). The liquid aggregate matches mean IoU at 98.0% agreement. The soft-body aggregate reaches 93.6%, compared with 95.7% for boundary Chamfer (Tables 9–10). The controlled tests are useful but cover only two liquid references (Table 12). These results do not establish that the added components produce more informative or reliable model rankings.

The main quantitative results are internally inconsistent (? Or maybe I am not understanding correctly? pls correct me if I am wrong.) Figure 2’s rigid-body scores disagree with Table 2, including the ordering of Wan2.2-TI2V-5B and Cosmos-Predict2.5-2B. The Wan2.2 soft-pad baseline also changes from 76.4 in Table 2 to 34.05 in Table 6. Section 5 specifies 128 soft-pad evaluation samples, but neither that section nor Appendix H explains evaluation differences that reconcile these values. These discrepancies affect both the benchmark comparison and the interpretation of the reported 55.71-point fine-tuning gain.

These are making my scores a bit lower at this time - I look forward to the authors' reply.

Questions for authors

What accounts for the discrepancies between Figure 2 and Table 2, and between the soft-pad baselines in Tables 2 and 6? Please specify any differences in evaluation instances, appearance variants, inference settings, or metric implementations.

Can you provide comparisons on shared model outputs showing what the composite scores detect beyond simple geometry metrics or existing reference-based evaluators? Evaluation across additional model pairs, using judgments of reference fidelity and reporting uncertainty, would materially affect my assessment.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer dyrd (Part 1/2)

W2 & Q1: Inconsistencies in the reported experimental results

Thank you for identifying these inconsistencies. We have corrected Figure 2 and rerun the paired Base/LoRA evaluation. The matched results retain substantial adaptation gains: +34.45 points on Raw inputs and +25.54 points on Transfer inputs.

(1) Figure 2 versus Table 2

Figure 2 was not synchronized with the updated tabulated results before submission. This was an oversight in manuscript preparation. We have corrected Figure 2 in the revised manuscript to make it consistent with Table 2.

(2) Table 2 versus Table 6

The original fine-tuning evaluation was not fully aligned with the main benchmark configuration, and the relevant differences were insufficiently documented. We regenerated both Base and LoRA predictions under an explicitly matched protocol.

Both models use the same 128 soft-pad source instances, with two conditioning settings:

  • Raw: the first frame of each simulation video.
  • Transfer: the first frame of one fixed appearance variant per instance, balanced across eight appearances with 16 instances each.

Base and LoRA share identical inputs, prompts, and paired generation seeds, yielding 512 predictions in total.

ConditioningBaseLoRAGain
Raw56.2190.66+34.45
Transfer64.0289.56+25.54

Both generate 49 frames at 1024×576 and 24 fps, using 30 sampling steps and CFG=5, matching these settings in the main benchmark. Flow shift is 5. LoRA uses the step-4500 checkpoint with rank=32, alpha=32, and scale=1.

Revised reporting uses matched gains and explicit evaluation populations. We replace the original Table 6 results and its 55.71-point gain with the matched results above. The revised description specifies the inputs, inference settings, and selected appearances. The fine-tuning evaluation uses one Transfer appearance per source instance, rather than all appearance variants in the full benchmark; its baseline is therefore a subset-specific estimate.

LoRA gains are calculated within each matched Base/LoRA experiment, rather than by comparing the fine-tuning result with the Table 2 baseline.

AuthorsOpenReview ↗

Responses to Reviewer dyrd (Part 2/2)

W1 & Q2: Additional diagnostic value of the composite scoring design.

Thank you for suggesting a broader reference-fidelity evaluation and a direct comparison with geometric baselines. We expanded the study to four models—Wan2.2-TI2V-5B, Cosmos-Predict2.5-2B, Cosmos3-Edge, and HunyuanVideo-1.5—with the present analysis using 10 annotators and 3,200 judgments on 640 comparisons from 320 reference instances across twelve scenario families. All six model pairings are covered. Annotators view the reference and anonymous predictions, judging reference fidelity rather than unrestricted physical plausibility.

As a supplementary analysis, we select comparisons with at least 80% of assigned annotators preferring the same prediction: 4/5 for every material category. Ties and unjudgeable responses do not support either direction and remain in the assigned-rater denominator. Selection uses human labels only, independently of metric predictions, retaining 247/640 comparisons (38.59%). Each retained comparison contributes one consensus label. All four metrics are compared on exactly the same consensus comparisons within each category:

MaterialConsensus comparisons / allCompositeIoUChamferCentroid
Rigid76/16075.00%64.47%73.68%69.74%
Soft64/16082.81%84.38%81.25%84.38%
Liquid49/16079.59%65.31%65.31%59.18%
Gas58/16077.59%75.86%75.86%74.14%

We report paired agreement differences between the composite and IoU, Chamfer, and Centroid below, in percentage points. Intervals use 10,000 scenario-stratified reference-cluster bootstrap replicates, retaining comparisons sharing a reference together and holding observed consensus labels fixed. These intervals are conditional on the participating annotators and observed consensus subset.

MaterialComposite−IoU [95% CI]Composite−Chamfer [95% CI]Composite−Centroid [95% CI]
Rigid+10.53 [+2.53, +18.61]+1.32 [-4.29, +7.32]+5.26 [-1.45, +12.50]
Soft-1.56 [-8.33, +5.26]+1.56 [-3.33, +7.14]-1.56 [-5.00, +0.00]
Liquid+14.29 [-2.50, +32.08]+14.29 [-2.13, +31.58]+20.41 [+3.70, +38.64]
Gas+1.72 [-9.38, +13.11]+1.72 [-7.46, +10.77]+3.45 [-12.50, +18.97]

Complementary measurements expose errors that spatial overlap can miss.

The composite's diagnostic role is concrete. In the controlled tests in the submitted Table 12, freezing either liquid reference retains spatial agreement near 0.7 but produces zero temporal agreement and a zero total score. This detects missing evolution even when the occupied region still overlaps substantially. The expanded human study compares metrics against consensus judgments across multiple models and all material categories.

Physics-IQ and RigidBench already establish reference-based evaluation. VidPhysOracle extends this approach across material regimes and matches the observables to motion, deformation, and evolving fluid support. The controlled tests establish complementary error sensitivity, while the consensus-based comparisons quantify alignment with human reference-fidelity judgments.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer dyrd,

We would like to sincerely thank you again for your thoughtful feedback and valuable suggestions.

We have provided detailed responses and additional explanations addressing your concerns in our rebuttal. As the discussion deadline is approaching, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your concerns.

If you have any remaining concerns or further questions, we would be more than happy to discuss them and provide additional clarifications.

Thank you again for your time and consideration.

Best regards, Authors

Reviewer_DBE9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VidPhysOracle introduces a simulation-grounded benchmark and dataset for evaluating video world models against reference physical trajectories. It contains 960 randomized source scenes across 12 rigid-body, soft-body, liquid, and gas scenarios, plus 6,272 appearance-translated variants. Material-specific metrics compare geometry, deformation, trajectories, occupancy, and temporal support under a unified protocol. The paper benchmarks seven open video models, validates selected metrics against human preferences, studies resolution sensitivity, and demonstrates LoRA fine-tuning on simulated soft-body videos.

Critical strengths
  • The benchmark substantially broadens physics evaluation across rigid, deformable, liquid, gas, and cross-material dynamics.
  • Material-specific metrics, detailed protocols, seven-model evaluation, and adaptation experiments make the resource unusually comprehensive.
Critical weaknesses
  • Reference fidelity is not equivalent to physical correctness when the future is underdetermined by the conditioning image and prompt. Randomized mass, friction, stiffness, and hidden 3D state may be unobservable, so a generated but physically valid alternative trajectory can be penalized. Moreover, simulator accuracy and numerical convergence are not validated against real experiments.

  • Experimental validation is too weak for a benchmark contribution. Human agreement uses one evaluator and an easy two-model pair, excludes ties and invalids, and reports neither label balance nor uncertainty. Seven-model rankings lack generation-seed variance, and Wan2.2’s soft-pad base score is unexplainedly 76.4 in Table 2 but 34.05 in Table 6.

Questions for authors
  • Which randomized physical parameters are explicitly exposed through the prompt or conditioning image, and which remain latent? How does the benchmark avoid penalizing alternative futures that are physically valid under unobserved parameter values?

  • What numerical-convergence, conservation, cross-solver, or real-world evidence supports treating each simulated rollout as an oracle? Are rankings stable under different solver settings and independently validated physical references?

  • For the human study, what are the valid, tied, and invalid counts, preference-label balance, exclusion reasons, and blinding protocol? Can multiple raters evaluate both plausibility and direct reference fidelity across several close model pairs, with confidence intervals and agreement statistics?

  • Why is Wan2.2’s soft-pad score 76.4 in Table 2 but 34.05 in Table 6? Please clarify differences in inputs, appearance variants, seeds, checkpoints, prompts, or scoring code, and report repeated-generation and repeated-training uncertainty.

  • Are geometry and motion quantitatively preserved by the appearance-translation pipeline? Do rankings remain stable with manual masks or alternative segmenters, trackers, metric weights, thresholds, and source-rendered versus translated inputs?

6 comments
AuthorsOpenReview ↗

Responses to Reviewer DBE9 (Part 1/4)

W1 & Q1: Hidden physical parameters may cause valid alternative futures to be penalized.

This concern does not, by itself, invalidate the benchmark’s stated objective: predicting a specified physical evolution. Our evaluation addresses this issue through its conditioning information, scenario design, and aggregate protocol.

Observable conditions and latent properties.

The image provides visible object shapes, counts, projected positions, orientations, and scene layout; the prompt specifies the intended interaction. Numerical properties such as density remain latent. A valid alternative may therefore score lower, but reference disagreement is not a verdict of physical impossibility, as distinguished in Sec. 3 and Appendix A.2.

Scenario design constrains the principal evolution.

We emphasize recognizable interactions whose main evolution is constrained by visible geometry and gravity:

  • Free fall: before contact, ideal gravitational acceleration is independent of object mass.
  • Domino toppling: the visible arrangement constrains the intended contact sequence.

Collision timing, rebound, and settling can still depend on latent properties; the design constrains the prediction problem without assuming a uniquely identifiable future.

Aggregate evaluation compares models under matched conditions.

All models receive the same conditioning information and reference instances across diverse randomized configurations. Averaging reduces dependence on individual configurations and measures expected reference consistency over the benchmark distribution. It does not eliminate latent ambiguity, but provides a common basis for comparing how accurately models predict observable motion, deformation, and flow.

W1 & Q2: Rationale of using simulation outputs as reference oracles

Our references are grounded in established numerical methods, documented simulation settings, and rollout-level quality control. The supporting evidence has two levels.

(1) Existing studies provide evidence for the underlying numerical methods.

  • Rigid bodies—MuJoCo: Erez et al. [1] quantitatively compare five physics engines, including their numerical consistency and speed–accuracy trade-offs.
  • Ropes—VBD: Chen et al. [2] analyze convergence and demonstrate stability for elastic-body simulation.
  • Soft volumes and liquids—MPM: Prior work establishes continuum formulations for deformable and fluid-like materials [3]. The APIC transfer used for our soft volumes additionally has an angular-momentum conservation property across particle–grid transfers [4].

(2) Our reference-generation pipeline includes explicit numerical and quality controls.

  • Documented numerical settings: Table 8 reports simulation rates, substeps, and solver iterations. For example, rigid bodies use 240 Hz with 200 solver iterations; ropes and cloth use 32 and 25 substeps, respectively. Settings remain fixed within each scenario family.
  • Instance-level screening: Initial configurations are checked for interpenetration, invalid parameters, and inadequate framing. Complete rollouts are then inspected to reject incorrect physical evolution and uninformative configurations (Appendix C.6).

“Oracle” denotes the reference evolution under the specified simulation model and configuration. The evidence above supports this construction; it does not establish exact agreement with real-world experiments or invariance to solver changes. Benchmark-specific convergence and conservation audits, cross-solver comparisons, and ranking stability under solver perturbations are not reported in the submission.

References

[1] Erez et al. Simulation Tools for Model-Based Robotics: Comparison of Bullet, Havok, MuJoCo, ODE and PhysX. ICRA, 2015.

[2] Chen et al. Vertex Block Descent. ACM TOG, 43(4), 2024.

[3] Stomakhin et al. Augmented MPM for Phase-Change and Varied Materials. ACM TOG, 33(4), 2014.

[4] Jiang et al. The Affine Particle-In-Cell Method. ACM TOG, 34(4), 2015.

AuthorsOpenReview ↗

Responses to Reviewer DBE9 (Part 2/4)

W2 & Q3: Completeness and statistical reliability of the human evaluation.

Thank you for requesting a more complete account of the human study, including label distributions, evaluation protocol, uncertainty, and inter-rater reliability. We have expanded the study to four models and an explicit reference-fidelity task; the present analysis uses 10 annotators and 3,200 judgments.

multiple annotators and model pairs & evaluation protocol

The study covers 320 reference instances across twelve scenario families and four material categories. Each reference appears in two distinct model-pair comparisons, yielding 640 comparisons across all six combinations of the four models. Each comparison receives five ratings, producing 3,200 judgments in the present analysis.

Annotators see the reference and two anonymous predictions; model identities and metric preferences are hidden. Left/right assignments are randomized in the task manifest, with a consistent recorded orientation for repeated ratings of the same item. Responses are left preferred, right preferred, tie, or unable to judge.

Consensus criterion and retained comparisons

We additionally evaluate agreement on comparisons with a clear directional human consensus, requiring at least 80% of all assigned annotators to prefer the same prediction: at least 4/5 for every material category. Ties and unjudgeable responses do not support either direction and remain in the assigned-rater denominator. This retains 247/640 comparisons (38.59%). Composite–consensus agreement is reported below:

MaterialConsensus comparisons / allMatching / evaluable comparisonsAgreement95% bootstrap interval
Rigid76/16057/7675.00%[63.51%, 85.06%]
Soft64/16053/6482.81%[72.31%, 92.06%]
Liquid49/16039/4979.59%[68.57%, 90.20%]
Gas58/16045/5877.59%[65.31%, 88.68%]

Intervals use 10,000 scenario-stratified reference-cluster bootstrap replicates, retaining comparisons sharing a reference together and holding observed consensus labels fixed.

Consistency between annotators

We separately compute nominal Krippendorff’s α on the full retained 10-annotator panel before consensus filtering, using left/right/tie responses and treating unjudgeable responses as missing:

MaterialKrippendorff’s α95% bootstrap intervalObserved pairwise agreement
Rigid0.441[0.357, 0.520]69.02%
Soft0.424[0.353, 0.491]66.12%
Liquid0.490[0.405, 0.564]66.74%
Gas0.331[0.245, 0.412]59.09%

The α intervals use scenario-stratified reference-cluster bootstrap resampling conditional on the retained annotators. 34 gas comparisons have fewer than two substantive ratings and therefore do not enter α. As a sensitivity analysis, treating “unable to judge” as a fourth response category gives Fleiss’ κ values of 0.337, 0.284, 0.351, 0.383, respectively.

AuthorsOpenReview ↗

Responses to Reviewer DBE9 (Part 3/4)

W2 & Q4: Inconsistent soft-pad baselines in Tables 2 and 6.

We have rerun Base and LoRA under matched conditions, confirming substantial improvements of +34.45 points on Raw inputs and +25.54 points on Transfer inputs. We clarify the evaluation differences and revised results below.

(1) The original evaluations were not fully aligned.

Table 2 reports the main benchmark results, whereas Table 6 used a separate fine-tuning evaluation whose configuration differences were insufficiently documented. The original baseline values were therefore presented without the information needed to distinguish their evaluation settings. We have addressed this by regenerating both Base and LoRA predictions under an explicitly matched protocol.

(2) Matched re-evaluation confirms the adaptation benefit.

Both models are evaluated on the same 128 soft-pad source instances, with two conditioning settings:

  • Raw: the first frame of each simulation video.
  • Transfer: the first frame of one fixed appearance variant per instance, balanced across eight appearances with 16 instances each.

Base and LoRA share identical conditioning images, prompts, and paired generation seeds, yielding 512 predictions in total.

ConditioningBaseLoRAGain
Raw56.2190.66+34.45
Transfer64.0289.56+25.54

Both models generate 49 frames at 1024×576 and 24 fps, using 30 sampling steps and CFG=5, matching these settings in the main benchmark. Flow shift is 5. LoRA uses the step-4500 checkpoint with rank=32, alpha=32, and scale=1.

(3) Revised reporting distinguishes the evaluation populations.

We replace the original Table 6 results with this matched evaluation and calculate gains only between Base and LoRA under identical conditions. The revised description explicitly specifies the inputs and inference settings.

The fine-tuning evaluation uses one selected Transfer appearance per source instance, rather than all appearance variants in the full benchmark. Its baseline is therefore a subset-specific estimate and should not be equated with the Table 2 average. The reported reruns use the same trained LoRA checkpoint and do not quantify variability across independent training runs.

Reviewer_DQAhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VidPhysOracle, a simulation-grounded benchmark for evaluating the physical consistency of video generation and world models. The benchmark comprises 960 randomized scenes across 12 scenario families spanning rigid bodies, soft bodies, liquids, and gases.

Critical strengths

The paper's core contribution is shifting physical evaluation from plausibility judgment to reference-grounded comparison.

Critical weaknesses
  1. While appearance transfer is central to reducing sim-to-real gap, the paper lacks quantitative validation of geometric and motion preservation. The authors acknowledge transfer failures for smoke and liquids (p.9) but provide only qualitative examples. Without metrics quantifying how transfer artifacts affect physical measurements, it remains unclear whether performance differences between synthetic and photorealistic variants reflect true physical reasoning deficits versus sensitivity to visual artifacts introduced by the transfer process itself.

  2. The paper explicitly states that "material-specific metrics are not calibrated to a common physical-error scale" (p.6), making cross-material comparison descriptive rather than physically meaningful. A 50-point rigid-body score versus a 50-point soft-body score may represent fundamentally different physical error magnitudes, limiting the benchmark's ability to diagnose whether models have systematic physical deficiencies or material-specific gaps.

  3. The human study (Section 3.4) compares only two models on 200 instances with a single evaluator judging plausibility rather than reference fidelity. This narrow validation provides insufficient evidence that the simulation-derived metrics reliably rank model performance on reference agreement across diverse models, scenarios, and physical regimes, weakening claims about the benchmark's validity as a general evaluation tool

Questions for authors

Weakness.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer DQAh (Part 1/2)

Q1: Lack of quantitative validation of transfer fidelity & the impact of transfer artifacts.

We now provide two complementary evaluations: direct transfer–simulation fidelity measurements and paired Raw/Transfer conditioning experiments. Appearance transfer was already applied selectively: Table 7 and Appendix C.5 include translations for the rigid-body and soft-body scenarios and the draining tank, but not for the water tank, U-connector, or gas scenarios.

(1) Quantitative validation of transfer fidelity

Evaluation Protocol.

To examine this issue, we evaluated 120 translated–simulation video pairs across all 12 scenario families, with 10 pairs per family. The table below distinguishes the selected scenarios from the additionally tested scenarios. Mixed-material scenes are evaluated separately by component, yielding 160 material-specific evaluations. We compare frames 1–47 using fixed spatial mapping and nominal timestamp alignment, without per-frame registration or temporal warping.

Table 1. Appearance-transfer fidelity across included and additionally tested scenarios. Entries summarize scenario/component means; ranges span the corresponding means. H denotes the reference image height.

GroupEvaluated componentsIoU ↑Centroid distance / H ↓Temporal-change FSS (0.10H) ↑
IncludedRigid components0.8301–0.91650.0022–0.0072—
IncludedSoft pad0.95490.0018—
IncludedCloth0.93360.0039—
IncludedSoft rope0.68850.0139—
IncludedDraining-tank liquid0.9437—0.7402
AdditionalWater-tank / smoke-impact rigid components0.4194–0.47940.0269–0.0306—
AdditionalWater-tank / U-connector liquid0.8865–0.9498—0.2976–0.6527
AdditionalSmoke0.7045–0.7717—0.2967–0.4332

Included rigid components have centroid deviations below 0.73% of image height, and soft-pad and cloth IoUs exceed 0.93. Thin-rope IoU is lower, but its boundary Chamfer distance is 0.0040H, providing a complementary measure of contour agreement.

Spatial agreement alone is insufficient for inclusion. Water-tank liquid has IoU 0.8865 but temporal-change FSS of only 0.2976. U-connector has IoU 0.9498 and temporal FSS 0.6527, yet visual inspection reveals apparent leakage through the pipe bottom. Its translations are excluded. These findings support combining spatial and temporal measurements with visual screening.

(2) Paired evaluation of Raw versus Transfer conditioning.

Paired Conditioning Experiment. We evaluate the unadapted Wan2.2-TI2V-5B on 332 matched source instances across the seven included scenario families. Each instance produces one Raw-conditioned and one Transfer-conditioned prediction, for 664 predictions. Each pair shares the prompt and generation seed, and both predictions are scored against the same Raw simulation reference.

Table 2. Model total scores under Raw versus Transfer first-frame conditioning. All scores are on a 0–100 scale, with higher values indicating greater reference agreement; material-specific score definitions differ and are not directly comparable across materials.

ScenePairsRawTransfer
Dominoes5036.3040.03
Falling objects5026.8528.97
Rolling objects5038.1043.91
Falling objects on soft pad*5026.3430.90
Gravity-draping flag5039.2848.61
Gravity-tightening rope*5024.4623.97
Draining tank3246.2652.70

Soft-pad and rope scores include the accompanying rigid objects; they are scene-level scores rather than isolated soft-pad or rope scores.

Results and Interpretation. These results suggest that appearance transfer does not systematically degrade the evaluated model’s reference-prediction performance: scores increase in six of seven scenarios, with only a small decrease of 0.50 points for rope.

Purpose of appearance transfer. Our use of appearance transfer serves two purposes. First, it provides a practical way to expand the dataset and increase visual diversity. Second, it aims to narrow the visual sim-to-real gap by translating synthetic renderings into more photorealistic appearances while retaining the simulation-derived physical reference.

Together, the two experiments quantify transfer fidelity and its effect on prediction scores separately. Transfer expands appearance diversity while simulation continues to define the physical target; segmentation and tracking uncertainty remains part of the measured correspondence.

AuthorsOpenReview ↗

Responses to Reviewer DQAh (Part 2/2)

Q2: Cross-material score comparability

Thank you for pointing out this limitation. We acknowledge that metrics across different material categories are not calibrated to a unified physical error scale, and their raw scores are therefore not directly comparable.

Interpretation of material-specific scores.

Our Benchmark primarily aims to compare the relative physical capabilities of different models within each material category, rather than establish a unified ranking across heterogeneous materials. Category-specific metrics are designed to capture the physical properties and behaviors relevant to each material.

Diagnostic value in coupled-material scenarios.

In multi-material scenarios, physical errors can propagate and interact across different materials. For example, inaccuracies in fluid simulation can affect the motion of coupled rigid bodies, while the resulting rigid-body motion can also reflect the accuracy of the fluid–solid interaction. Thus, evaluating one material's physical behavior can provide indirect insights into the accuracy of other materials and their interactions.

Relative benchmarking across categories.

For an overall comparison across material categories, we can use a reference model as a baseline and evaluate other models' relative performance against it within each category. This provides a consistent reference for aggregating category-specific results, while avoiding the need to interpret raw scores as sharing a unified physical scale. We also provide human-perception-based recommended metrics to support application-dependent evaluation.

Q3: Scope and validity of the human evaluation.

Thank you for highlighting the limited scope of the original human evaluation. We agree that a single-annotator plausibility study involving two models was insufficient to support broad claims about reference-fidelity ranking. We have therefore rerun the study with an explicit reference-fidelity task, broader model and scenario coverage, and multiple independent annotators.

Expanded model, scenario, and annotator coverage.

The new study evaluates four models—Wan2.2-TI2V-5B, Cosmos-Predict2.5-2B, Cosmos3-Edge, and HunyuanVideo-1.5—on 320 reference instances across twelve scenario families and four material categories, with 80 instances per category. Each reference is used in two distinct model-pair comparisons, yielding 640 comparisons covering all six model combinations.

Reference-fidelity task and annotators.

Annotators view a simulation reference and two anonymous predictions and select which prediction better reproduces the reference’s physical evolution. This directly evaluates correspondence to the reference, rather than asking whether a prediction is independently physically plausible. The analysis uses 10 annotators and 3,200 judgments, with five ratings per comparison in every material category. Ties and unjudgeable responses are recorded separately.

Agreement with human reference-fidelity judgments.

As a supplementary analysis, we select comparisons in which at least 80% of all assigned annotators prefer the same prediction: at least 4/5 for every material category. Ties and unjudgeable responses do not count toward either direction. This criterion retains 247/640 comparisons.

The following table reports composite–consensus agreement on the retained comparisons. Each comparison contributes one human-consensus label.

MaterialConsensus comparisons / allMatching / evaluable comparisonsAgreement95% bootstrap interval
Rigid76/16057/7675.00%[63.51%, 85.06%]
Soft64/16053/6482.81%[72.31%, 92.06%]
Liquid49/16039/4979.59%[68.57%, 90.20%]
Gas58/16045/5877.59%[65.31%, 88.68%]

Intervals use 10,000 scenario-stratified reference-cluster bootstrap replicates, retaining comparisons sharing the same reference together. Observed consensus labels are held fixed; intervals are conditional on the participating annotators.

Interpretation and scope.

These results support alignment with human reference-fidelity preferences on comparisons with clear directional consensus across the evaluated models and materials.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer DQAh,

We would like to sincerely thank you again for your thoughtful feedback and valuable suggestions.

We have provided detailed responses and additional explanations addressing your concerns in our rebuttal. As the discussion deadline is approaching, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your concerns.

If you have any remaining concerns or further questions, we would be more than happy to discuss them and provide additional clarifications.

Thank you again for your time and consideration.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers and area chairs for their careful assessment. We appreciate the reviewers' recognition of:

  • Reference-grounded physical evaluation: shifting evaluation from plausibility judgments to comparison with a specified physical evolution (DQAh).
  • Broad material coverage and a comprehensive resource: extending evaluation to rigid, deformable, liquid, gas, and coupled dynamics, with detailed protocols and adaptation experiments (DBE9, with the broader material coverage also recognized by dyrd).
  • Measurements addressing concrete failure modes: preserving position and timing errors through fixed alignment and detecting frozen predictions through temporal-support comparison (dyrd).

We have strengthened the submission through the following evaluations and corrections:

(1) Corrected results and matched adaptation experiments. We corrected Figure 2 to agree with Table 2 and reran Base/LoRA evaluation on the same 128 soft-pad instances with matched inputs, prompts, and generation seeds. At the main benchmark's reported resolution and sampling settings, LoRA improves the score by +34.45 points on Raw inputs and +25.54 points on Transfer inputs. The revised protocol explicitly distinguishes the fine-tuning subset from the full benchmark.

(2) Expanded human validation and direct metric comparisons. The new blinded study evaluates reference fidelity with four models, 10 annotators, 320 references, 640 comparisons, and 3,200 judgments. We compare the composite with IoU, Chamfer, and centroid metrics against multi-rater human consensus, reporting the consensus criterion, evaluation coverage, confidence intervals, and full-study inter-rater reliability. The composite–consensus agreement is 78.75% (95% CI: 73.32%–84.02%) across 247/640 eligible comparisons; paired baseline comparisons show higher agreement than IoU for rigid bodies (+10.53 percentage points, 95% CI: +2.53 to +18.61) and centroid metrics for liquids (+20.41 points, 95% CI: +3.70 to +38.64).

(3) Quantitative transfer fidelity and conditioning sensitivity. We evaluate 120 translated–simulation video pairs across all twelve scenario families, yielding 160 component-level evaluations. A separate 332-pair Raw/Transfer conditioning experiment evaluates the same unadapted model against matched simulation references. These analyses quantify transfer fidelity and conditioning effects separately, supporting selective appearance augmentation with explicit quality checks.

The revision PDF highlights changes in blue. The main updates are Figure 2, Section 5 / Table 6, and Appendix H.4 for corrected results and paired adaptation; Appendix E / Tables 10–12 for human evaluation and metric comparisons; and Appendix C.5 / Table 9 for transfer-fidelity validation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.