acceptodds
Under review as a conference paper at ICLR 2027

WorldOlympiad: Can Your World Model Survive a Triathlon?

Abstract

We introduce WorldOlympiad, a benchmark for diagnosing video-based world models across physical faithfulness, geometric consistency, and interaction fidelity. While existing benchmarks often focus on visual quality, semantic alignment, or short-term temporal coherence, they provide limited insight into whether generated videos obey physical rules, preserve coherent 3D structure, and sustain controllable interactions over long horizons. To address this gap, WorldOlympiad decomposes world-model evaluation into three complementary dimensions. The physical track uses object segmentation and MLLM-as-judge to assess whether generated videos follow interpretable rules in mechanics, thermal phenomena, and material properties. The geometry track reconstructs generated videos with Gaussian splatting and evaluates structural consistency, cross-view coherence, and camera-trajectory alignment. The interaction track assesses whether generated rollouts follow complex action prompts and maintain smooth, coherent transitions across consecutive video chunks. WorldOlympiad further covers three major downstream scenarios, including gaming, robotics, and general real-world videos, capturing diverse challenges from interactive control and embodied manipulation to open-domain motion and camera dynamics. Together, these tracks and scenarios form a scalable and interpretable evaluation suite that exposes failure modes beyond generic video quality. Experiments on state-of-the-art models reveal substantial gaps in physical reasoning, 3D consistency, and long-horizon interaction, underscoring the need for more structured evaluation protocols for generative world models.

Primary area:
datasets and benchmarks
Keywords:
Video Generation, Benchmark, World Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_e8uYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduced WorldOlympiad, a 1000 video benchmark evaluation the world modeling based on video from three main perspectives: physical faithfullness, geometric consistency and interaction fidelity. The collected videos are originally from other public benchmarks and contain three main origins: robotics, game and real-world domain. The evaluation part relies on MLLM judging, Gaussian-splat reconstruction, and camera related trajectory analysis. Experiment across 8 video generation pipelines has identified the low performance under the propose evaluation protocol.

Critical strengths

The most notable strength is the explicit involvement and discussion of thermodynamic and material change process, which discuss evaporation, solubility and hardness evolution. These phenomena are difficult to analyze and interpretate from video, and remain under-explored by prior works and benchmarks. This paper filled in this gap, in which the authors proposed to combine rule-based, relevance-filtered approaches to approach the physical sutble difference, going beyond just the object motion plausibility.

Critical weaknesses
  1. The rigid body motion (either single object, or multiple objects) remain as an important evaluation target from video, because visually evaluation has high accuracy rate for rigid body motion. The absence of typical rigid body physics tests including rotational dynamics, momentum conservation, and friction limits the benchmark’s ability to distinguish visually plausible object motion from physically correctness.

  2. Evaluation protocol heavily depends on pretrained models like MLLMs, SAM, as well as depth estimation, whose intrinsic errors and inaccuracy may propagate and aggregate into benchmark scores. Although in the Appendix, human preference rankings correlate strongly with aggregated scores, this protocol setting does not validate individual physical or geometric judgments. Component-level or instance-level ground-truth validation, more analysis on evaluator sensitivity are thus required to establish the reliability of measurements.

Questions for authors

See the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer e8uY(1/2)

We thank the reviewer for the constructive feedback on rigid-body physics coverage and evaluator error propagation.

R1. Rigid-body physics tests and quantitative motion evaluation

Our default MLLM rule judge evaluates qualitative visual plausibility and cannot by itself verify momentum conservation, rotational dynamics, or friction. To add quantitative rigid-body checks (collision momentum/energy conservation, sliding/rolling acceleration, and pendulum/spring invariants), we follow Morpheus [1] and apply its official trajectory-evaluation protocol to 48 videos per group (Sec. 4.4, Table 3; Appendix F.1, Table 16), using SAM to track object centers, areas, relative depth, distances, and angles frame by frame across the full sequence (eight-frame sampling applies only to the MLLM branch).

Following Morpheus [1], the invariant score measures temporal stability of task-prescribed quantities via s=1/(1+λd)s = 1 / (1 + \lambda d) for fluctuation dd:

Task familyQuantities checked, as prescribed by the task
Free fall, projectile motion, and bouncingEnergy, acceleration, and horizontal momentum or velocity
Sliding and rollingAcceleration
Simple pendulumEnergy, period, and rod length
Double pendulumEnergy and rod length
Spring motionHorizontal momentum or velocity, and period
CollisionsEnergy and horizontal momentum

Following Morpheus's PINN-based formulation [1], the dynamics score fits a physics-informed neural network (PINN) to both trajectory positions and governing equations of motion (free fall, pendulum, or damped oscillation), scored via normalized mean squared error as Sdyn=1−min⁡(NMSE,1)S_{\mathrm{dyn}} = 1 - \min(\mathrm{NMSE}, 1) without requiring frame-by-frame alignment to a reference video:

Video groupNumber of videosRule complianceInvariant scoreDynamics score
Hunyuan-WorldPlay480.6060.8890.520
LongLive480.7220.6630.713
LingBot-World480.8840.7080.863
Real reference videos480.8900.8880.951

LingBot-World leads the generated models on both rule compliance and PINN dynamics, yet all three generators remain clearly below real reference videos on dynamics; Hunyuan-WorldPlay scores high on invariants only because sliding-window selection ignores static segments. Since 2D-plus-depth trajectories cannot fully capture 3D rotational inertia or complex contact friction, we state this scope in Sec. 3.2.1, Sec. 4.4, and Appendix F.1, and leave quantitative physics metrics to future work (Sec. 5).

[1] Tragoudaras et al., Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems (ICML 2026)

AuthorsOpenReview ↗

Response to Reviewer e8uY(2/2)

R2. Error propagation and evaluator sensitivity

Our physical, geometric, and interaction tracks run independently, so errors from SAM segmentation, depth estimation, and MLLM judging do not accumulate across tracks. Within the physical track, SAM3 only highlights moving foreground objects to help the MLLM inspect the raw video frames instead of acting as a hard filter. Within the 3D geometry track, masking moving objects keeps foreground motion from distorting DA3 camera-pose estimation and static Gaussian-Splat scene reconstruction. To verify that our 3D scores are not driven by MLLM judging bias, we also replaced the MLLM reconstruction score with direct LPIPS perceptual similarity across all eight models and three domains, obtaining closely aligned rankings with Spearman correlations of 0.76 on gaming, 0.90 on robotics, and 0.80 on general videos in Sec. 4.4 and Appendix F.3, Table 17.

We also measured the end-to-end error of the automated pipeline directly on real videos, where physical laws naturally hold. In the 48-video comparison in Sec. 4.4, Table 3, real reference videos score 0.890 on SAM3+MLLM rule compliance, 0.888 on SAM-tracked invariants, and 0.951 on PINN dynamics. Because an ideal error-free pipeline would score 1.000 on real videos, the remaining gap of roughly 0.05 to 0.11 reflects the combined noise from segmentation, tracking, and judging. This pipeline noise is much smaller than the performance gaps among generated models, whose scores span 0.606 to 0.884 on rule compliance and 0.520 to 0.863 on dynamics.

Beyond the model-level human ranking study in Appendix E, Table 15, we conducted an additional audit of 50 individual physical rule–video pairs against human annotations. Automated judgments agree with human labels on 38 of the 50 cases, or 76.0%, while the 12 disagreements mainly involve visually ambiguous borderline clips and clear physical errors or clean passes match human labels. Varying the track and CLIP aggregation weights likewise keeps overall ranking correlations between 0.905 and 1.000. Together with the metric scope clarifications in Sec. 3.2, Sec. 4.4, and Sec. 5, these checks confirm that component noise does not drive our benchmark conclusions.

We thank you again for the constructive feedback and welcome any further questions.

Reviewer_vSjROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

WorldOlympiad is a benchmark for video-based world models organized as a "triathlon" of three tracks over three domains. It collects 1,000 long videos - 400 robotics (RoboCOIN), 400 gaming (GameGen-X), and 200 real-world (LVD-2M) - segments them into chunks with a three-stage Gemini-3-Pro-Preview captioning pipeline (action labels in W/A/S/D plus scene/action captions), and evaluates generated videos along: (i) physical faithfulness, using SAM3 object masks and an MLLM judge over mechanics, thermodynamics, and material rules; (ii) 3D geometric consistency, using Depth Anything 3 + Gaussian-splatting reconstruction, a meta-view render, and recovered-vs-reference camera trajectory agreement; and (iii) interaction fidelity, combining a CLIP frame-caption score with MLLM chunk-level, transition-level, and global judgments. Eight pipelines (Matrix-Game 2.0, LingBot-World, Cosmos-Predict-2.5, WoW, Rolling Forcing, LongLive, Yume-1.5, Hunyuan-WorldPlay) are evaluated under a unified protocol. LingBot-World ranks first (0.683), Cosmos-Predict-2.5 second (0.671), and geometry remains the weakest track for all models (0.25-0.42). A human preference study with 5 annotators, 20 prompts, 560 pairwise comparisons, and 2,800 labels reports rank correlation rho = 0.95 with the automatic rankings. Fine-grained results expose a mechanics-vs-thermodynamics gap, a geometry-interaction trade-off, and domain specialization effects.

Critical strengths
  1. The three-track design produces diagnostics that a single leaderboard cannot. Separating physical rules, reconstruction-based geometry, and interaction lets the benchmark expose concrete failure patterns - mechanics are nearly saturated for strong models (e.g., 0.983-1.000) while thermodynamics stays low, geometry is uniformly weak, and a model such as Hunyuan-WorldPlay trades geometry for interaction depending on domain. This decomposition is consequential because it turns "which model is best" into "which capability fails and where," which is what benchmark users actually need.

  2. Breadth and reproducibility. 1,000 long videos across gaming, robotics, and real-world domains, eight pipelines evaluated under one protocol, fixed prompts/thresholds, and a human preference study with 2,800 labels (rank correlation 0.95) form a substantial, reusable resource. The domain-wise breakdowns (Table 5) and the specialization-generalization analysis (WoW dropping from 0.502 on embodied to 0.368 on gaming) are useful findings that go beyond a static leaderboard.

Critical weaknesses
  1. The "interaction" track measures caption adherence and transition smoothness, not action-conditioned control. Models are driven by chunk captions mapped to their native generation configuration; the interaction score combines CLIP frame-caption similarity with MLLM judgments of chunk quality, boundary smoothness, and global coherence. Nothing in this metric requires the model to actually follow the W/A/S/D control signal, so a model that ignores player input but generates coherent, caption-consistent video can score well. Since the paper frames WorldOlympiad as evaluating interactive world models, this is a central construct-validity gap. The benchmark already has camera-action labels; reporting an explicit action-following accuracy (e.g., recovered trajectory vs. prescribed W/A/S/D, or human check of control adherence) would make the interaction claim credible.

  2. Judge applicability and calibration are not validated, with visible symptoms in the results. In the robotics subset, every model scores exactly 0.000 on thermodynamics (Table 6). Either the relevance judge is counting rules that are not observable in those videos as failures, or the judge systematically answers non-compliant; in both cases the physical score is not measuring model behavior on that dimension. At the same time, mechanics saturates at 1.000 for some models, so the track does not discriminate. The human study validates only the ranking of eight models on 20 prompts, not the per-rule judgments or per-dimension diagnostics that the paper's conclusions rely on. This is consequential because statements such as "thermodynamics and material questions remain fragile" are based on an instrument whose applicability and calibration have not been checked at the level where the claims are made.

Questions for authors
  1. Why do all eight models receive 0.000 on thermodynamics in the robotics subset? How many thermodynamics questions are judged relevant per robotics video, and are inapplicable questions excluded from the denominator? Please report per-dimension denominators alongside pass rates.

  2. Can you add an explicit action-adherence metric using the W/A/S/D labels (e.g., camera-direction accuracy or trajectory error) and report its correlation with the current interaction score? This would directly test whether "interaction fidelity" measures control.

  3. What fraction of the 1,000 videos constitutes the "same-scene subset" in Table 5, and how are generation prompts/reference images chosen? Are all models conditioned on exactly the same prompt and first frame?

  4. Does human judgment agree with the physical-rule judge at the case level, not only at the model-ranking level? A small audit (e.g., 100 rule-video pairs) would validate the diagnostic claims.

  5. The overall score weights physical, geometry, and interaction equally, and interaction itself is 90% MLLM + 10% CLIP. How sensitive is the final ranking to these choices (e.g., drop the CLIP term, or upweight geometry)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vSjR(1/3)

We thank the reviewer for the detailed evaluation and concrete questions.

R1. Robotics thermodynamics scores, relevance rate, and denominators (Q1)

All eight models score 0.000 on robotics thermodynamics because tabletop manipulation clips rarely involve phase transitions (such as melting, freezing, or vaporization): across the 400-video robotics subset, only 3 rule–video cases are judged relevant on the reference videos, and all eight models pass 0 of 3 cases.

Inapplicable questions are excluded from the denominator. Relevance is determined once from the original reference videos and shared across all evaluated pipelines, keeping denominators identical across models (631 for mechanics, 3 for thermodynamics, and 244 for materials on the shared robotics set):

ModelMechanics: score (passed/applicable)Thermodynamics: score (passed/applicable)Materials: score (passed/applicable)
Matrix-Game 2.00.366 (231/631)0.000 (0/3)0.373 (91/244)
LingBot-World0.960 (606/631)0.000 (0/3)0.959 (234/244)
Cosmos-Predict-2.50.940 (593/631)0.000 (0/3)0.967 (236/244)
WoW0.799 (504/631)0.000 (0/3)0.787 (192/244)
Rolling Forcing0.857 (541/631)0.000 (0/3)0.934 (228/244)
LongLive0.864 (545/631)0.000 (0/3)0.869 (212/244)
Yume-1.50.857 (541/631)0.000 (0/3)0.873 (213/244)
Hunyuan-WorldPlay0.577 (364/631)0.000 (0/3)0.811 (198/244)

Because the physical score averages over all applicable questions (Appendix B.3), each dimension's pass rate SdS_d is weighted by its applicable count NdN_d:

\frac{ N_{\mathrm{mech}} S_{\mathrm{mech}} + N_{\mathrm{thermo}} S_{\mathrm{thermo}} + N_{\mathrm{mat}} S_{\mathrm{mat}} }{ N_{\mathrm{mech}} + N_{\mathrm{thermo}} + N_{\mathrm{mat}} }.$$ Weighting by applicable counts keeps the overall physical score unaffected by the tiny thermodynamics count ($N=3$), which simply reflects that our robotics data distribution does not cover thermodynamic phenomena (thermodynamics is covered instead by the gaming and real-world subsets; domain and per-question pass rates are in Appendix C.3, Tables 8–9, formerly Tables 6–7). > ### R2. Explicit trajectory adherence versus the interaction score (Q2) The current Interaction score combines three MLLM submetrics (chunk instruction following Local, boundary smoothness Trans., and long-range coherence Global) with CLIP similarity; it measures semantic and visual state progression and does not directly measure trajectory execution. To directly test how the MLLM interaction judge relates to metric motion control, we added a **3D trajectory comparison** over 500 robotics cases per model. For each reference and generated RGB video, we segment and track the manipulated object, lift its motion into 3D via depth estimation and rigid fitting in a shared initial coordinate frame, and average its spatial path and elapsed-time velocity agreement into a single **Trajectory** score in $[0,1]$: | Model | Local ↑ | Trans. ↑ | Global ↑ | Interaction ↑ | Trajectory ↑ | | --- | ---: | ---: | ---: | ---: | ---: | | LingBot-VA | 0.463 | 0.549 | 0.546 | 0.510 | 0.203 | | Cosmos3 | 0.394 | 0.235 | 0.231 | 0.323 | 0.283 | LingBot-VA leads across all three MLLM submetrics and overall Interaction through stronger instruction following and long-horizon visual continuity, whereas Cosmos3 achieves a higher Trajectory score by matching the reference 3D path and execution timing more closely. A high MLLM interaction score therefore does not guarantee accurate trajectory execution, as the two metrics measure complementary semantic and kinematic aspects of control. We link this comparison in Sec. 3.2.3, report the 3D trajectory results and protocol in Sec. 4.4 (Table 4) and Appendix F.2, and leave fine-grained trajectory judges for interaction to future work (Sec. 5); direct W/A/S/D camera-control accuracy on the eight-model set remains a separate check.
AuthorsOpenReview ↗

Response to Reviewer vSjR(2/3)

R3. Proportion of the “same-scene subset” and conditioning inputs across models (Q3)

Table 7 in Appendix C.2 (Table 5 in the initial submission) covers all 1,000 benchmark videos grouped by domain (400 gaming, 400 robotics, and 200 general real-world videos; dataset composition in Appendix C.1, Table 6, formerly Table 4), with no additional matched-scene filtering. “Same-scene subset” was an imprecise label referring to these domain-grouped evaluations across the full benchmark.

For every benchmark video, the reference image is the initial frame of the source clip, and prompts come from our three-stage annotation pipeline (temporal chunking, per-chunk scene and W/A/S/D camera-action labeling, and full-video refinement). Models do not all receive the exact same prompt-and-first-frame input, because their native architectures accept different conditioning modalities, though every model's inputs are derived from the same source video and annotations:

  • Text-only long-video pipelines (LongLive and Rolling Forcing): Both build on the Wan2.1-T2V-1.3B text-to-video backbone and do not accept a first-frame image in their default inference paths. LongLive takes the ordered sequence of chunk captions together with frame switch indices, refreshing its causal KV cache at prompt-switch boundaries. Rolling Forcing (default DMD T2V path) does not support prompt-switch recaching or a single-frame initial latent; we merge the chunk descriptions into a long-horizon prompt for a single native rollout with rolling-window denoising, attention sinks, and a block-aligned 24-frame latent context window.
  • Image- and action-conditioned pipelines: Matrix-Game 2.0 takes the source initial frame and discrete navigation/camera actions (without text prompts); LingBot-World, Yume-1.5, and Hunyuan-WorldPlay take the initial frame, text prompt, and discrete action signals; Cosmos-Predict-2.5 and WoW take the initial frame and chunk text prompts via segment continuation.

Table 7 (formerly Table 5) therefore evaluates complete pipelines under their native conditioning configurations (Sec. 4.1).

R4. Instance-level human agreement with the physical judge (Q4)

Yes. Complementing our 20-prompt pairwise ranking study in Appendix E (Table 15, formerly Table 13), we conducted an additional 50-pair instance-level human audit of individual rule–video judgments during the rebuttal: automated physical predictions match human annotations on 38 of 50 cases (76.0%), with 12 cases (24.0%) showing disagreement.

Inspecting the 12 discordant pairs shows that mismatches occur mainly on visually borderline clips (such as brief occlusions, subtle motion artifacts near the pass/fail threshold, or ambiguous contact cues in sampled frames), whereas clear physical violations and clear passes align with human labels. A 76.0% instance-level agreement supports comparative evaluation across well-covered dimensions (mechanics N=631N=631, materials N=244N=244), whereas our robotics data distribution does not cover the thermodynamics domain (N=3N=3).

AuthorsOpenReview ↗

Response to Reviewer vSjR(3/3)

R5. Sensitivity of the final ranking to CLIP and track weights (Q5)

Our overall score averages the Physical, 3D Geometry, and Interaction tracks equally (1/3 each), with Interaction combining 90% MLLM and 10% CLIP by default. Below, we report overall scores and ranks under alternative CLIP and track weights (P/G/I percentages shown in headers):

ModelBaseline (1/3 each)CLIP 0% (1/3 each)CLIP 20% (1/3 each)50% Geom. (25/50/25)60% Geom. (20/60/20)
LingBot-World0.6830 (1)0.6891 (1)0.6769 (1)0.6055 (1)0.5590 (2)
Cosmos-Predict-2.50.6707 (2)0.6759 (2)0.6654 (2)0.6028 (2)0.5620 (1)
Rolling Forcing0.6100 (3)0.6100 (3)0.6100 (3)0.5378 (3)0.4944 (4)
Yume-1.50.6043 (4)0.6087 (4)0.5999 (4)0.5285 (5)0.4830 (5)
LongLive0.5840 (5)0.5807 (5)0.5873 (5)0.5288 (4)0.4956 (3)
Hunyuan-WorldPlay0.4773 (6)0.4724 (6)0.4823 (6)0.4640 (6)0.4560 (6)
WoW0.4343 (7)0.4347 (7)0.4340 (7)0.3882 (7)0.3606 (7)
Matrix-Game 2.00.2310 (8)0.2283 (8)0.2337 (8)0.2370 (8)0.2406 (8)
Spearman's ρ\rho vs. Baseline1.0001.0001.0000.9760.905

For the overall ranking, both removing CLIP (0%) and doubling CLIP to 20% leave all eight ranks unchanged (ρ=1.000\rho = 1.000); raising Geometry to 50% swaps only fourth and fifth places across a sub-0.0005 gap (ρ=0.976\rho = 0.976), while raising Geometry to 60% shifts the top three (ρ=0.905\rho = 0.905).

Because CLIP is only a subcomponent of the Interaction score, varying its weight between 0% and 20% has negligible impact on the three-track composite score, though the Interaction-only leaderboard itself depends slightly on the CLIP weight:

ModelCLIP 0%: MLLM only (rank)CLIP 10%: Baseline (rank)CLIP 20% (rank)
LingBot-World0.7522 (1)0.7340 (1)0.7158 (1)
Cosmos-Predict-2.50.7228 (2)0.7070 (2)0.6912 (2)
Yume-1.50.6622 (3)0.6490 (3)0.6358 (4†)
Rolling Forcing0.6361 (4)0.6360 (4)0.6359 (3†)
LongLive0.5161 (5)0.5260 (5)0.5359 (5)
WoW0.3461 (6)0.3450 (6)0.3439 (6)
Hunyuan-WorldPlay0.3011 (7)0.3160 (7)0.3309 (7)
Matrix-Game 2.00.1050 (8)0.1130 (8)0.1210 (8)

Within the Interaction track, removing CLIP (0%) keeps all eight ranks identical (ρ=1.000\rho = 1.000), while raising CLIP to 20% (ρ=0.976\rho = 0.976) preserves ranks 1–2 and 5–8 and swaps only Yume-1.5 and Rolling Forcing (†reweighted gap of 0.000111, smaller than three-decimal rounding precision and effectively a tie; four decimals are shown only to display the exact arithmetic).

We thank you again for the detailed comments and welcome any further feedback.

Reviewer_BtkLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper focuses on evaluating video world models beyond visual quality, especially their physical, geometric, and interaction consistency. It proposes WorldOlympiad, a benchmark with 1,000 videos across gaming, robotics, and real-world scenarios, using MLLMs and 3D reconstruction for evaluation. Experiments on eight world models show that LingBot-World achieves the best overall score, while geometric consistency remains a major weakness across models.

Critical strengths

S1. The paper focuses on an important problem. As world models continue to develop, having reliable ways to evaluate them is becoming increasingly important.

S2. Building a benchmark requires significant effort and resources. The authors have put considerable work into data collection, evaluation design, and experiments.

S3. The paper is well written and easy to follow. The overall structure is clear, and the methods and experiments are explained in a clear way.

Critical weaknesses

W1. The paper does not clearly explain the novelty of its evaluation methods compared to existing work. For example, MLLM-based physics evaluation has already been widely studied in WorldModelBench, WorldArena, and WBench. I am not sure what new insights or improvements this work brings.

W2. Several models achieve physics scores above 0.9, suggesting strong physical consistency. However, recent studies such as Morpheus, GAUGE, and World Models' Last Exam in Physics report significant limitations in physical fidelity. These findings seem quite different from your conclusions. Could you discuss why your results differ from these studies, and whether this difference comes from the evaluation methods?

W3. The geometry evaluation is very similar to World-R1, using DA3, 3D Gaussian Splatting, meta-view evaluation, and camera trajectory comparison. I am not sure what is new in this part or how the changes improve the evaluation.

Minor

M1: World-R1 uses LPIPS to evaluate reconstruction quality, while this paper uses MLLMs. Why is MLLM-based scoring more suitable than numerical metrics such as PSNR or LPIPS?

Questions for authors

Q1. Compared with existing benchmarks such as WorldModelBench, WorldArena, and WBench, what are the main new contributions of your evaluation methods, especially the MLLM-based physics evaluation?

Q2. Your physics scores are much higher than what recent studies such as Morpheus, GAUGE, and World Models' Last Exam in Physics might suggest. Could you discuss why your conclusions differ from these studies?

Q3. Your geometry evaluation is very similar to World-R1. What are the main differences, and how do they improve the evaluation?

Q4. Why do you use MLLMs to evaluate reconstruction quality instead of numerical metrics such as PSNR or LPIPS?

5 comments
AuthorsOpenReview ↗

Response to Reviewer BtkL(1/3)

We thank the reviewer for the constructive comments and suggestions.

R1. Differences from existing benchmarks (W1, Q1)

Compared with WorldModelBench and WBench, which feed raw video frames directly to an MLLM judge, our physical judge is SAM3-assisted: an MLLM first identifies the primary moving or deforming entities, SAM3 segments their foreground masks across frames, and the MLLM compliance judge inspects those localized object masks alongside the video to evaluate applicable mechanics, thermodynamics, and material rules. Compared with WorldArena, which restricts SAM3-assisted evaluation to embodied robotics, WorldOlympiad applies this SAM3-assisted physical evaluation across gaming, robotics, and general real-world videos:

BenchmarkGamingRoboticsReal-worldPhysical eval.SAM-assisted MLLM judge
WorldModelBench✓✓✓✓✗
WorldArena✗✓✗✓✓
WBench✓✓✓✓✗
WorldOlympiad (ours)✓✓✓✓✓

Highlighting moving foreground objects with SAM3 masks reduces background distraction when the MLLM checks contact, support, or deformation. In Sec. 2.3 and Sec. 3.2.1, we cite WorldModelBench, WorldArena, and WBench, clarify the visual scope of MLLM rule judging, and link to Sec. 4.4 and Appendix F.1.

AuthorsOpenReview ↗

Response to Reviewer BtkL (2/3)

R2. Why physical scores exceed 0.9 and comparison with quantitative benchmarks (W2, Q2)

Yes, this difference is expected and stems from two factors: (1) raw unnormalized aggregation where frequent basic rules dominate rarer, harder subcategories, and (2) orthogonal evaluation targets between visual compliance and strict physical laws.

(1) Raw unnormalized scores and subcategory frequency. WorldOlympiad reports the raw, unnormalized pass rate in [0,1][0,1] without post-hoc min–max stretching. Because rules are weighted by applicable counts (Appendix B.3), frequent basic mechanics and rigidity checks (which strong models pass at 0.85–1.00) account for most cases and pull the overall average up, whereas rarer thermodynamics cases carry smaller weight in the total yet remain far from saturated and clearly separate models (Appendix C.3, Table 8):

ModelOverall PhysicalGaming: ThermodynamicsGeneral: Thermodynamics
LingBot-World0.9420.4500.519
Cosmos-Predict-2.50.9060.4180.613
Rolling Forcing0.8730.4180.581
LongLive0.8630.3770.581
Yume-1.50.8630.3650.370
WoW0.7080.2260.300
Hunyuan-WorldPlay0.6920.4260.323
Matrix-Game 2.00.3250.1720.097

For example, LongLive and Yume-1.5 tie at 0.863 overall yet diverge sharply on general thermodynamics (0.581 vs. 0.370).

(2) Orthogonal and complementary evaluation targets. WorldOlympiad measures visual compliance with observable physical rules in open-domain, multi-object dynamic videos, where ground-truth physical parameters and closed-form equations of motion are unavailable. By contrast, Morpheus [1], GAUGE [2], and World Models' Last Exam in Physics [3] test strict Newtonian laws (exact conservation invariants and ODE/PINN trajectory fitting) in controlled setups with isolated phenomena. Both metrics are valid and complementary/orthogonal: a generated video can look visually plausible (>0.9 pass rate) while deviating from exact momentum conservation or acceleration curves, whereas open-domain world models also require scalable visual checks beyond controlled lab scenes.

Within our rebuttal compute budget, we verified this complementarity on 48 videos per group using the official Morpheus trajectory-evaluation protocol [1] on SAM-extracted object trajectories (eight-frame sampling applies only to the MLLM branch):

Video groupNumber of videosRule complianceInvariant scoreDynamics score
Hunyuan-WorldPlay480.6060.8890.520
LongLive480.7220.6630.713
LingBot-World480.8840.7080.863
Real reference videos480.8900.8880.951

LingBot-World approaches real videos in visual rule compliance yet remains clearly below real references on trajectory invariants and PINN dynamics, showing how trajectory metrics expose finer dynamical errors. Conversely, Hunyuan-WorldPlay gets a high invariant score only because sliding-window selection ignores static segments, while our visual rule judge and PINN dynamics both flag its poor physical behavior. Visual rule judging and quantitative trajectory mechanics thus serve complementary roles across open-domain and controlled settings. We clarify this scope in Sec. 3.2.1, report the comparison in Sec. 4.4 (Table 3) and Appendix F.1 (Table 16), and leave quantitative physics metrics to future work (Sec. 5).

[1] Tragoudaras et al., Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems (ICML 2026)
[2] Wang et al., GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models (arXiv:2608.05948, 2026)
[3] Gao et al., World Models' Last Exam in Physics (arXiv:2610.08791, 2026)

AuthorsOpenReview ↗

Response to Reviewer BtkL(3/3)

R3. Main differences from World-R1 and how masking improves evaluation (W3, Q3)

World-R1 evaluates static scenes under camera-control extrapolation, whereas WorldOlympiad evaluates general dynamic videos containing independently moving foreground objects across gaming, robotics, and real-world domains. Although both pipelines use standard 3D reconstruction tools (DA3 pose and depth estimation, 3D Gaussian splatting, meta-view rendering, and camera-trajectory comparison), applying static reconstruction directly to dynamic videos causes moving actors (such as robot arms, characters, vehicles, and manipulated objects) to violate multi-view static geometry assumptions, corrupting camera pose estimation and creating ghosting artifacts in held-out meta-views.

To resolve this, we mask dynamic foreground regions before reconstructing and scoring the static scene, preventing valid object motion from being mispenalized as 3D geometric inconsistency and allowing reconstruction-based 3D evaluation to scale to general non-static videos as one track of our benchmark. We cite World-R1 in Sec. 3.2.2.

R4. Why MLLM was used and comparison with LPIPS (M1, Q4)

We originally used an MLLM scorer to keep the evaluation pipeline unified across tracks and to inspect structural layout on low-motion or blurred clips, where blurred Gaussian splats can still match degraded input frames in feature space despite weak 3D geometry. At the same time, numerical perceptual metrics such as LPIPS are faster and avoid MLLM judging variance.

To test whether our 3D findings depend on MLLM scoring versus numerical metrics, we implemented an LPIPS variant replacing SreconS_{\mathrm{recon}} in Eq. (2) with SreconLPIPS=clip⁡(1−LPIPS‾(V^GS,Vˉ),0,1)S_{\mathrm{recon}}^{\mathrm{LPIPS}} = \operatorname{clip}(1 - \overline{\mathrm{LPIPS}}(\hat{V}_{\mathrm{GS}}, \bar{V}), 0, 1) and evaluated all eight models across the three domains (higher is better; Sec. 4.4; Appendix F.3, Table 17):

ModelGaming: LPIPS-based scoreGaming: MLLM scoreRobotics: LPIPS-based scoreRobotics: MLLM scoreGeneral: LPIPS-based scoreGeneral: MLLM score
Matrix-Game 2.00.2040.1890.3400.3380.2600.220
LingBot-World0.3280.3660.3900.3930.3230.335
Cosmos-Predict-2.50.3150.3610.5080.4790.3290.317
WoW0.2520.2230.3510.2720.3230.251
Rolling Forcing0.2780.2890.3920.3890.2950.285
LongLive0.2700.2920.5020.4720.3040.290
Yume-1.50.3410.3520.3200.2880.3250.302
Hunyuan-WorldPlay0.3670.3480.5380.6000.2430.219

LPIPS and MLLM scores produce closely aligned model rankings across all three domains, reaching Spearman's ρ=0.76\rho = 0.76 on gaming, ρ=0.90\rho = 0.90 on robotics, and ρ=0.80\rho = 0.80 on general scenes. We report these LPIPS results and transformation details in Sec. 4.4 and Appendix F.3 (Table 17), linked from Sec. 3.2.2.

We thank you again for the constructive suggestions and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.