- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WorldRoamBench, a benchmark for evaluating the long-horizon stability of interactive world models under continuous action inputs. The benchmark evaluates models along four dimensions: action, vision, physics, and memory. The authors propose pose-based discrete action accuracy, sliding-window visual drift measurement, physical plausibility evaluation, and trajectory-aware scene and subject memory assessment. The benchmark contains over 1,000 test cases across diverse environments, including natural, urban, and indoor scenes. It evaluates 12 world models from both first-person and third-person perspectives. The results show that existing models still have clear limitations in sustained interaction, struggling to maintain action responsiveness, physical consistency, and memory fidelity simultaneously.
Critical strengths
1.Existing short-video benchmarks are limited in their ability to capture the accumulation of errors during extended interactions. WorldRoamBench jointly evaluates action following, visual quality, physical consistency, and memory across different environments, viewpoints, and interaction durations. This provides a more comprehensive assessment of current world models under sustained interaction. 2.The paper separates frame-level local action accuracy from global trajectory evaluation, reducing the risk that trajectory-level metrics mask individual action errors. For memory evaluation, the authors account for the actual motion trajectory, reducing the interference of action deviations when assessing memory during revisits.
Critical weaknesses
- I still have concerns about the incremental contribution of WorldRoamBench relative to WorldMark. WorldMark already evaluates action direction, response latency, and motion stability, and also accounts for motion deviations during revisits. Some of the limitations identified in this paper have already been considered in prior work. Although the proposed metrics use different implementations, the paper’s comparison mainly lists the metrics supported by each benchmark. It remains unclear what improvements the new metrics offer in practice.
- Some models obtain very similar scores in the main experiments. For example, the top two models in the FPV overall ranking differ by less than one point, while some Action Scores are also very close to each other or approach saturation. The paper does not establish whether these differences are stable. The authors conduct human validation for Physics and Subject Memory. However, these experiments primarily examine the agreement between automatic scores and human judgments for individual videos. They do not directly establish whether the overall model rankings align with human evaluations.
Questions for authors
Q1. Could the authors compare the proposed action and scene-memory metrics with the corresponding metrics of the most relevant existing benchmarks, particularly WorldMark, using the same generated videos? Can the proposed metrics identify failures that existing metrics miss, or achieve better agreement with human judgments? Q2. How reliable are the model rankings reported in the main experiments? Could the authors provide ranking stability analyses for the overall scores and Action Scores, particularly to determine whether the ordering of models with very similar scores is robust?
4 comments
Response to Reviewer GJ3v (2/2)
Q2. How reliable are the model rankings reported in the main experiments? Could the authors provide ranking stability analyses for the overall scores and Action Scores, particularly to determine whether the ordering of models with very similar scores is robust?
Answer. Thank you for asking about close rankings. We distinguish stability to threshold recalibration from stability to evaluation-scene sampling, alongside the dimension-wise comparison below.
(1) Similar Overall scores, different strengths. The 0.54-point Overall gap combines different strengths: Lyra 2.0 scores higher in Action and Vision; Genie 3 scores higher in Physics and Memory.
| Model | Overall | Action | Vision | Physics | Memory |
|---|---|---|---|---|---|
| Genie 3 | 70.15 | 76.31 | 66.21 | 66.67 | 71.40 |
| Lyra 2.0 | 69.61 | 90.82 | 69.61 | 51.52 | 66.49 |
FPV scores from Table 2(a), on a 0–100 scale.
(2) Stability to candidate recalibration. As reported in Reviewer TGB6 W3 and Reviewer Mfws Q1, we evaluate five scene splits, each using 864 calibration videos and 3,348 evaluation videos. On the same evaluation videos within each split, recalibrating the four shared candidates changes cross-model mean Strict Accuracy by -0.18 to 0.00 pp and preserves all 12 model ranks. These results show that, on matched evaluation videos, Strict Accuracy rankings remain stable when the four shared candidate thresholds are recalibrated.
(3) Action ranking stability under scene resampling. Using the existing 4,212 videos (351 per model), we perform 10,000 paired bootstrap resamples of the 39 source scenes, preserving the Nature/Urban/Indoor composition. All nine action–duration variants of a scene stay together, and all models share each resample. We recompute Action from Strict Accuracy, Partial Accuracy, and Trajectory scores.
| Model comparison | Δ Action (pp) | 95% CI (pp) | Order retained (%) |
|---|---|---|---|
| Lyra 2.0 − Genie 3 | +14.50 | [+8.90, +20.25] | 100.0 |
| Matrix-Game 3.0 − Lyra 2.0 | +0.52 | [−0.95, +1.98] | 75.0 |
| Lyra 2.0 − LingBot-World | +0.03 | [−2.17, +2.14] | 51.3 |
| LingBot-World − LingBot-World 2.0 | +0.04 | [−1.89, +1.74] | 53.6 |
Differences are computed before rounding. Order retained is the percentage of resamples preserving the listed ordering.
Across 10,000 scene resamples, Lyra 2.0 consistently outperforms Genie 3 in Action, whereas the four closely scored leaders change order. Thus, the Lyra–Genie Action difference is robust to scene resampling, but the fine ordering among these leaders is not. The recalibration result above concerns Strict Accuracy; these intervals concern the composite Action score.
(4) Interpreting close scores. Similar Overall scores can reflect different strengths across dimensions, as the Genie–Lyra comparison shows. The four leading models show similar measured Action performance on our benchmark, while their fine ordering changes under scene resampling.
We will add these confidence intervals and ranking-retention rates to the results discussion, distinguishing the robust Lyra–Genie Action difference from the uncertain ordering among the closely scored leaders.
I will raise my rating to Weak Accept
I appreciate the authors’ detailed response, which has resolved most of my concerns. I still believe the issue of ranking robustness is important. For a benchmark, improving discriminative power among different models remains a critical problem, and this aspect may require further enhancement in future work. Nevertheless, WorldRoamBench’s ability to separate and evaluate different capability dimensions is a valuable strength. I will raise my rating to Weak Accept.
Thank You for Your Endorsement!
Dear Reviewer GJ3v,
We sincerely thank you for the positive evaluation and thoughtful assessment of our revised manuscript. We greatly appreciate the recognition of our revisions and the increased rating.
We agree that WorldRoamBench has room to improve its ability to distinguish closely performing models. As our scene-resampling analysis showed, the ordering of models with similar Action scores can change with the choice of evaluation scenes. Improving the benchmark’s ability to distinguish these models and assessing the stability of their rankings will be priorities for future work.
We also appreciate your recognition of the value of evaluating Action, Vision, Physics, and Memory separately. These dimension-level results help readers see where models with similar Overall scores differ in their capabilities.
Sincerely,
Authors
Response to Reviewer GJ3v (1/2)
Thank you for recognizing the joint evaluation and the treatment of action deviations in memory assessment. We address the comparison with WorldMark and the reliability of close model rankings below.
Q1. Could the authors compare the proposed action and scene-memory metrics with the corresponding metrics of the most relevant existing benchmarks, particularly WorldMark, using the same generated videos? Can the proposed metrics identify failures that existing metrics miss, or achieve better agreement with human judgments?
Answer. Thank you for asking us to clarify the practical contribution beyond WorldMark. WorldRoamBench adds explicit diagnostics of which commands were executed, how much previously observed geometry is retained, and how much revisit geometry lacks support from prior observation. The following examples illustrate their practical value.
(1) Action: diagnosing complete and partial command execution. Our Action metric distinguishes complete from partial command execution by decoding camera motion into action labels and comparing their components with the input commands. WorldMark measures flow-derived direction accuracy and purity, response latency, and motion stability [1, Sec. 3.1]. For example, decoding
forward+look_rightfor alook_rightcommand passes partial matching but fails strict matching, explicitly identifying the extra translation.Figure 2(a) shows why this diagnosis complements trajectory geometry: the same Lyra 2.0 rollout scores 0.87 in TrajScore but only 0.53 in Action Accuracy, exposing local command errors despite a high trajectory score. Decoding and threshold validation are detailed in Reviewer Mfws Q1.
(2) Scene memory: separating retained and unsupported geometry. WorldMark’s Revisit Memory uses accumulated optical flow to pair outbound and return frames and compares DINOv2 features; its Global Memory also assesses cross-frame geometry. Our scene-memory protocol instead reconstructs and aligns the observation and revisit point clouds, then reports two directional measurements: Retention measures how closely previously observed geometry is recovered, while Anti-hallucination measures how consistently revisit geometry is supported by prior observation. These measurements distinguish missing earlier structure from unsupported revisit structure across multiple frames.
Figure 19 (Appendix M) illustrates the value of evaluating scene-level geometry. Even after action-aware transition localization and pose-aware frame pairing, the mine example has SSIM 0.168 and LPIPS 0.474 despite visibly preserved layout; its scene-memory score is 90.1/100. This example shows that image-level differences need not indicate scene-memory loss and motivates evaluating the retained scene geometry.
(3) Direct comparison and manuscript clarification. Because the metrics have different definitions, their raw scores cannot be directly compared to determine which evaluator is better. Our qualitative comparison with WorldMark therefore focuses on evaluation targets and metric definitions; Figures 2 and 19 illustrate the diagnostic value of our metrics against trajectory and frame-pair baselines. We will clarify Table 1’s discrete-action column as decoded action-set accuracy and expand Appendix D, using Figures 2 and 19 to explain the diagnostic value of command matching and bidirectional observation–revisit geometry.
[1] Xiaojie Xu et al. “WorldMark: A Unified Benchmark Suite for Interactive Video World Models.” arXiv:2604.21686v2, 2026.