- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduced WorldOlympiad, a 1000 video benchmark evaluation the world modeling based on video from three main perspectives: physical faithfullness, geometric consistency and interaction fidelity. The collected videos are originally from other public benchmarks and contain three main origins: robotics, game and real-world domain. The evaluation part relies on MLLM judging, Gaussian-splat reconstruction, and camera related trajectory analysis. Experiment across 8 video generation pipelines has identified the low performance under the propose evaluation protocol.
Critical strengths
The most notable strength is the explicit involvement and discussion of thermodynamic and material change process, which discuss evaporation, solubility and hardness evolution. These phenomena are difficult to analyze and interpretate from video, and remain under-explored by prior works and benchmarks. This paper filled in this gap, in which the authors proposed to combine rule-based, relevance-filtered approaches to approach the physical sutble difference, going beyond just the object motion plausibility.
Critical weaknesses
-
The rigid body motion (either single object, or multiple objects) remain as an important evaluation target from video, because visually evaluation has high accuracy rate for rigid body motion. The absence of typical rigid body physics tests including rotational dynamics, momentum conservation, and friction limits the benchmark’s ability to distinguish visually plausible object motion from physically correctness.
-
Evaluation protocol heavily depends on pretrained models like MLLMs, SAM, as well as depth estimation, whose intrinsic errors and inaccuracy may propagate and aggregate into benchmark scores. Although in the Appendix, human preference rankings correlate strongly with aggregated scores, this protocol setting does not validate individual physical or geometric judgments. Component-level or instance-level ground-truth validation, more analysis on evaluator sensitivity are thus required to establish the reliability of measurements.
Questions for authors
See the weakness section.
2 comments
Response to Reviewer e8uY(2/2)
R2. Error propagation and evaluator sensitivity
Our physical, geometric, and interaction tracks run independently, so errors from SAM segmentation, depth estimation, and MLLM judging do not accumulate across tracks. Within the physical track, SAM3 only highlights moving foreground objects to help the MLLM inspect the raw video frames instead of acting as a hard filter. Within the 3D geometry track, masking moving objects keeps foreground motion from distorting DA3 camera-pose estimation and static Gaussian-Splat scene reconstruction. To verify that our 3D scores are not driven by MLLM judging bias, we also replaced the MLLM reconstruction score with direct LPIPS perceptual similarity across all eight models and three domains, obtaining closely aligned rankings with Spearman correlations of 0.76 on gaming, 0.90 on robotics, and 0.80 on general videos in Sec. 4.4 and Appendix F.3, Table 17.
We also measured the end-to-end error of the automated pipeline directly on real videos, where physical laws naturally hold. In the 48-video comparison in Sec. 4.4, Table 3, real reference videos score 0.890 on SAM3+MLLM rule compliance, 0.888 on SAM-tracked invariants, and 0.951 on PINN dynamics. Because an ideal error-free pipeline would score 1.000 on real videos, the remaining gap of roughly 0.05 to 0.11 reflects the combined noise from segmentation, tracking, and judging. This pipeline noise is much smaller than the performance gaps among generated models, whose scores span 0.606 to 0.884 on rule compliance and 0.520 to 0.863 on dynamics.
Beyond the model-level human ranking study in Appendix E, Table 15, we conducted an additional audit of 50 individual physical rule–video pairs against human annotations. Automated judgments agree with human labels on 38 of the 50 cases, or 76.0%, while the 12 disagreements mainly involve visually ambiguous borderline clips and clear physical errors or clean passes match human labels. Varying the track and CLIP aggregation weights likewise keeps overall ranking correlations between 0.905 and 1.000. Together with the metric scope clarifications in Sec. 3.2, Sec. 4.4, and Sec. 5, these checks confirm that component noise does not drive our benchmark conclusions.
We thank you again for the constructive feedback and welcome any further questions.
Response to Reviewer e8uY(1/2)
We thank the reviewer for the constructive feedback on rigid-body physics coverage and evaluator error propagation.
Our default MLLM rule judge evaluates qualitative visual plausibility and cannot by itself verify momentum conservation, rotational dynamics, or friction. To add quantitative rigid-body checks (collision momentum/energy conservation, sliding/rolling acceleration, and pendulum/spring invariants), we follow Morpheus [1] and apply its official trajectory-evaluation protocol to 48 videos per group (Sec. 4.4, Table 3; Appendix F.1, Table 16), using SAM to track object centers, areas, relative depth, distances, and angles frame by frame across the full sequence (eight-frame sampling applies only to the MLLM branch).
Following Morpheus [1], the invariant score measures temporal stability of task-prescribed quantities via s=1/(1+λd) for fluctuation d:
Following Morpheus's PINN-based formulation [1], the dynamics score fits a physics-informed neural network (PINN) to both trajectory positions and governing equations of motion (free fall, pendulum, or damped oscillation), scored via normalized mean squared error as Sdyn=1−min(NMSE,1) without requiring frame-by-frame alignment to a reference video:
LingBot-World leads the generated models on both rule compliance and PINN dynamics, yet all three generators remain clearly below real reference videos on dynamics; Hunyuan-WorldPlay scores high on invariants only because sliding-window selection ignores static segments. Since 2D-plus-depth trajectories cannot fully capture 3D rotational inertia or complex contact friction, we state this scope in Sec. 3.2.1, Sec. 4.4, and Appendix F.1, and leave quantitative physics metrics to future work (Sec. 5).
[1] Tragoudaras et al., Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems (ICML 2026)