acceptodds
Under review as a conference paper at ICLR 2027

WorldRoamBench: Evaluating Long-Horizon Stability of Interactive World Models

Abstract

Despite rapid progress in interactive world models (IWMs), their long-horizon stability under sustained action inputs remains insufficiently evaluated. We introduce WorldRoamBench, an open-world benchmark addressing this gap across four complementary dimensions: (i) Action: pose-based discrete-action accuracy reducing sensitivity to cross-model semantic scale disparity and exposing frame-level errors hidden by trajectory scores; (ii) Vision: sliding-window drift metric capturing non-monotonic mid-sequence collapse missed by start-vs-end comparisons; (iii) Physics: evaluation of physical plausibility across mechanics, optics, and 3D consistency, gated by camera-motion and subject-tracking checks; (iv) Memory: a trajectory-aware protocol reducing confounding from action-following errors, evaluating scene memory via transition-localized 3D point-cloud reconstruction and subject memory via tracking-plus-VLM reasoning. The benchmark comprises 1000+ test cases across Nature, Urban, and Indoor scenes in first/third-person views with WASD 10-60 s continuous interaction. Evaluating 12 open/closed-source models reveals none reliably satisfies all dimensions; even the best long-horizon first-person overall score is only 67.17/100 (Genie 3). Advances on WorldRoamBench are steps toward IWMs that are stable, physically grounded, memory-faithful, and deployable in real-world applications.

Primary area:
datasets and benchmarks
Keywords:
Interactive World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GJ3vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorldRoamBench, a benchmark for evaluating the long-horizon stability of interactive world models under continuous action inputs. The benchmark evaluates models along four dimensions: action, vision, physics, and memory. The authors propose pose-based discrete action accuracy, sliding-window visual drift measurement, physical plausibility evaluation, and trajectory-aware scene and subject memory assessment. The benchmark contains over 1,000 test cases across diverse environments, including natural, urban, and indoor scenes. It evaluates 12 world models from both first-person and third-person perspectives. The results show that existing models still have clear limitations in sustained interaction, struggling to maintain action responsiveness, physical consistency, and memory fidelity simultaneously.

Critical strengths

1.Existing short-video benchmarks are limited in their ability to capture the accumulation of errors during extended interactions. WorldRoamBench jointly evaluates action following, visual quality, physical consistency, and memory across different environments, viewpoints, and interaction durations. This provides a more comprehensive assessment of current world models under sustained interaction. 2.The paper separates frame-level local action accuracy from global trajectory evaluation, reducing the risk that trajectory-level metrics mask individual action errors. For memory evaluation, the authors account for the actual motion trajectory, reducing the interference of action deviations when assessing memory during revisits.

Critical weaknesses
  1. I still have concerns about the incremental contribution of WorldRoamBench relative to WorldMark. WorldMark already evaluates action direction, response latency, and motion stability, and also accounts for motion deviations during revisits. Some of the limitations identified in this paper have already been considered in prior work. Although the proposed metrics use different implementations, the paper’s comparison mainly lists the metrics supported by each benchmark. It remains unclear what improvements the new metrics offer in practice.
  2. Some models obtain very similar scores in the main experiments. For example, the top two models in the FPV overall ranking differ by less than one point, while some Action Scores are also very close to each other or approach saturation. The paper does not establish whether these differences are stable. The authors conduct human validation for Physics and Subject Memory. However, these experiments primarily examine the agreement between automatic scores and human judgments for individual videos. They do not directly establish whether the overall model rankings align with human evaluations.
Questions for authors

Q1. Could the authors compare the proposed action and scene-memory metrics with the corresponding metrics of the most relevant existing benchmarks, particularly WorldMark, using the same generated videos? Can the proposed metrics identify failures that existing metrics miss, or achieve better agreement with human judgments? Q2. How reliable are the model rankings reported in the main experiments? Could the authors provide ranking stability analyses for the overall scores and Action Scores, particularly to determine whether the ordering of models with very similar scores is robust?

4 comments
AuthorsOpenReview ↗

Response to Reviewer GJ3v (1/2)

Thank you for recognizing the joint evaluation and the treatment of action deviations in memory assessment. We address the comparison with WorldMark and the reliability of close model rankings below.

Q1. Could the authors compare the proposed action and scene-memory metrics with the corresponding metrics of the most relevant existing benchmarks, particularly WorldMark, using the same generated videos? Can the proposed metrics identify failures that existing metrics miss, or achieve better agreement with human judgments?

Answer. Thank you for asking us to clarify the practical contribution beyond WorldMark. WorldRoamBench adds explicit diagnostics of which commands were executed, how much previously observed geometry is retained, and how much revisit geometry lacks support from prior observation. The following examples illustrate their practical value.

(1) Action: diagnosing complete and partial command execution. Our Action metric distinguishes complete from partial command execution by decoding camera motion into action labels and comparing their components with the input commands. WorldMark measures flow-derived direction accuracy and purity, response latency, and motion stability [1, Sec. 3.1]. For example, decoding forward+look_right for a look_right command passes partial matching but fails strict matching, explicitly identifying the extra translation.

Figure 2(a) shows why this diagnosis complements trajectory geometry: the same Lyra 2.0 rollout scores 0.87 in TrajScore but only 0.53 in Action Accuracy, exposing local command errors despite a high trajectory score. Decoding and threshold validation are detailed in Reviewer Mfws Q1.

(2) Scene memory: separating retained and unsupported geometry. WorldMark’s Revisit Memory uses accumulated optical flow to pair outbound and return frames and compares DINOv2 features; its Global Memory also assesses cross-frame geometry. Our scene-memory protocol instead reconstructs and aligns the observation and revisit point clouds, then reports two directional measurements: Retention measures how closely previously observed geometry is recovered, while Anti-hallucination measures how consistently revisit geometry is supported by prior observation. These measurements distinguish missing earlier structure from unsupported revisit structure across multiple frames.

Figure 19 (Appendix M) illustrates the value of evaluating scene-level geometry. Even after action-aware transition localization and pose-aware frame pairing, the mine example has SSIM 0.168 and LPIPS 0.474 despite visibly preserved layout; its scene-memory score is 90.1/100. This example shows that image-level differences need not indicate scene-memory loss and motivates evaluating the retained scene geometry.

(3) Direct comparison and manuscript clarification. Because the metrics have different definitions, their raw scores cannot be directly compared to determine which evaluator is better. Our qualitative comparison with WorldMark therefore focuses on evaluation targets and metric definitions; Figures 2 and 19 illustrate the diagnostic value of our metrics against trajectory and frame-pair baselines. We will clarify Table 1’s discrete-action column as decoded action-set accuracy and expand Appendix D, using Figures 2 and 19 to explain the diagnostic value of command matching and bidirectional observation–revisit geometry.

[1] Xiaojie Xu et al. “WorldMark: A Unified Benchmark Suite for Interactive Video World Models.” arXiv:2604.21686v2, 2026.

AuthorsOpenReview ↗

Response to Reviewer GJ3v (2/2)

Q2. How reliable are the model rankings reported in the main experiments? Could the authors provide ranking stability analyses for the overall scores and Action Scores, particularly to determine whether the ordering of models with very similar scores is robust?

Answer. Thank you for asking about close rankings. We distinguish stability to threshold recalibration from stability to evaluation-scene sampling, alongside the dimension-wise comparison below.

(1) Similar Overall scores, different strengths. The 0.54-point Overall gap combines different strengths: Lyra 2.0 scores higher in Action and Vision; Genie 3 scores higher in Physics and Memory.

ModelOverallActionVisionPhysicsMemory
Genie 370.1576.3166.2166.6771.40
Lyra 2.069.6190.8269.6151.5266.49

FPV scores from Table 2(a), on a 0–100 scale.

(2) Stability to candidate recalibration. As reported in Reviewer TGB6 W3 and Reviewer Mfws Q1, we evaluate five scene splits, each using 864 calibration videos and 3,348 evaluation videos. On the same evaluation videos within each split, recalibrating the four shared candidates changes cross-model mean Strict Accuracy by -0.18 to 0.00 pp and preserves all 12 model ranks. These results show that, on matched evaluation videos, Strict Accuracy rankings remain stable when the four shared candidate thresholds are recalibrated.

(3) Action ranking stability under scene resampling. Using the existing 4,212 videos (351 per model), we perform 10,000 paired bootstrap resamples of the 39 source scenes, preserving the Nature/Urban/Indoor composition. All nine action–duration variants of a scene stay together, and all models share each resample. We recompute Action from Strict Accuracy, Partial Accuracy, and Trajectory scores.

Model comparisonΔ Action (pp)95% CI (pp)Order retained (%)
Lyra 2.0 − Genie 3+14.50[+8.90, +20.25]100.0
Matrix-Game 3.0 − Lyra 2.0+0.52[−0.95, +1.98]75.0
Lyra 2.0 − LingBot-World+0.03[−2.17, +2.14]51.3
LingBot-World − LingBot-World 2.0+0.04[−1.89, +1.74]53.6

Differences are computed before rounding. Order retained is the percentage of resamples preserving the listed ordering.

Across 10,000 scene resamples, Lyra 2.0 consistently outperforms Genie 3 in Action, whereas the four closely scored leaders change order. Thus, the Lyra–Genie Action difference is robust to scene resampling, but the fine ordering among these leaders is not. The recalibration result above concerns Strict Accuracy; these intervals concern the composite Action score.

(4) Interpreting close scores. Similar Overall scores can reflect different strengths across dimensions, as the Genie–Lyra comparison shows. The four leading models show similar measured Action performance on our benchmark, while their fine ordering changes under scene resampling.

We will add these confidence intervals and ranking-retention rates to the results discussion, distinguishing the robust Lyra–Genie Action difference from the uncertain ordering among the closely scored leaders.

Reviewer_GJ3vOpenReview ↗

I will raise my rating to Weak Accept

I appreciate the authors’ detailed response, which has resolved most of my concerns. I still believe the issue of ranking robustness is important. For a benchmark, improving discriminative power among different models remains a critical problem, and this aspect may require further enhancement in future work. Nevertheless, WorldRoamBench’s ability to separate and evaluate different capability dimensions is a valuable strength. I will raise my rating to Weak Accept.

AuthorsOpenReview ↗

Thank You for Your Endorsement!

Dear Reviewer GJ3v,

We sincerely thank you for the positive evaluation and thoughtful assessment of our revised manuscript. We greatly appreciate the recognition of our revisions and the increased rating.

We agree that WorldRoamBench has room to improve its ability to distinguish closely performing models. As our scene-resampling analysis showed, the ordering of models with similar Action scores can change with the choice of evaluation scenes. Improving the benchmark’s ability to distinguish these models and assessing the stability of their rankings will be priorities for future work.

We also appreciate your recognition of the value of evaluating Action, Vision, Physics, and Memory separately. These dimension-level results help readers see where models with similar Overall scores differ in their capabilities.

Sincerely,
Authors

Reviewer_MfwsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldRoamBench, a benchmark evaluating the long-horizon stability of interactive world models across four dimensions: action following, visual quality, interaction physics, and memory. It proposes several evaluation methods, including pose-based discrete-action accuracy, sliding-window visual drift, physics consistency assessment, and trajectory-aware scene memory evaluation. The benchmark comprises more than 1000 test cases covering diverse environments and interaction durations. Experiments on multiple models reveal substantial performance gaps, particularly in physical plausibility and long-term memory preservation.

Critical strengths
  • Important problem: Addresses the timely challenge of evaluating interactive world models under sustained action inputs rather than short video sequences.

  • Comprehensive evaluation framework: Covers multiple complementary dimensions, providing a more holistic assessment of interactive world models.

  • Broad experimental coverage: Evaluates 12 open- and closed-source models across diverse scenarios, including first- and third-person perspectives.

  • Interesting evaluation methods: Introduces trajectory-aware memory assessment and sliding-window visual drift metrics to address limitations of existing benchmarks.

Critical weaknesses
  • Potentially biased action evaluation: The action decoder selects the best translation threshold for each video by maximizing agreement with ground-truth actions, potentially inflating the reported accuracy. Additionally, assigning one action label to each four-frame window limits the claimed frame-level precision.

  • Insufficient independent metric validation: Several metrics rely on estimated camera poses, 3D reconstruction, and VLM judgments. The mechanics judge achieves only 73.2% agreement with human annotations on the same samples used for calibration, without held-out validation.

  • Questionable fairness of model comparisons: Different models use heterogeneous control interfaces and action adapters, potentially introducing systematic advantages. Furthermore, physics scores are calculated over model-dependent valid subsets, hence complicating direct comparisons.

  • Limited benchmark diversity and long-horizon evidence: The action cases are generated from only 72 initial images, and interactions last at most 60 seconds. All physics tests use 60-second rollouts, with identical physics scores reused across duration-specific comparisons.

Questions for authors
  1. How would the action accuracy and model rankings change if thresholds were fixed using independent calibration data rather than optimized separately for each test video? How does the four-frame aggregation affect accuracy near the action-switch boundaries?

  2. Can the authors provide held-out human validation of the physics and scene-memory metrics? How sensitive are these metrics and the resulting rankings to pose estimation, reconstruction, and VLM judgment errors?

  3. How do the authors ensure fair comparisons across models with different control interfaces and action adapters?

  4. Can the authors expand the benchmark with more independent starting scenes and longer interaction sequences? Could physics tests be conducted at multiple durations to directly measure horizon-dependent physical degradation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Mfws (1/4)

Thank you for recognizing our benchmark’s breadth and its trajectory-aware memory and sliding-window drift metrics.

Q1. How would the action accuracy and model rankings change if thresholds were fixed using independent calibration data rather than optimized separately for each test video? How does the four-frame aggregation affect accuracy near the action-switch boundaries?

Answer. Thank you for highlighting threshold selection and temporal resolution.

(1) Scale adaptation. See Reviewer TGB6 W3. Our multi-threshold design accommodates different displacements across models and scenes. We extend CompassReward [1] from three fixed translation candidates to four candidates calibrated on 4,212 FPV videos and shared across models and scenes. We follow the CompassReward code’s exact selection criterion [1]. For each video, we choose the candidate maximizing exact command agreement and keep it fixed throughout the rollout.

(2) Separate calibration and rankings. We stratified 39 scenes by Nature, Urban and Indoor into 8 calibration scenes (864 videos) and 31 evaluation scenes (3,348 videos). All models and variants of each scene stay together. With poses, K = 4 and 1° rotation fixed, a 31-value grid search on calibration videos yields one fixed threshold (0.025) or four candidates [0.0001, 0.015, 0.04, 0.3] with per-video selection.

ModelOriginal four Accuracy (%)Fixed single Accuracy (%)Calibrated four Accuracy (%)
LingBot-World90.77 (1)84.43 (1)90.54 (1)
Lyra 2.090.17 (2)83.56 (2)89.75 (2)
LingBot-World 2.089.00 (3)67.36 (6)88.72 (3)
Matrix-Game 3.088.40 (4)72.62 (5)88.38 (4)
Matrix-Game 2.087.56 (5)76.99 (3)87.31 (5)
Happy Oyster84.98 (6)74.76 (4)85.47 (6)
HY-World 1.583.10 (7)51.04 (7)83.10 (7)
SANA-WM-AR71.64 (8)45.67 (9)71.80 (8)
SANA-WM59.87 (9)22.95 (11)59.95 (9)
Genie 359.39 (10)51.03 (8)59.36 (10)
Yume 1.556.80 (11)38.21 (10)56.56 (11)
minWM46.69 (12)17.61 (12)46.69 (12)
Mean75.7057.1975.64

Ranks in parentheses. 279 evaluation videos per model.

Across five scene splits, calibrated-four mean Accuracy differs from original-four by −0.18 to 0.00 pp on matched evaluation sets, with all 12 Accuracy ranks unchanged. In the primary split, the fixed single threshold scores 18.45 pp lower than calibrated four and changes 8 of 12 ranks. A single global threshold cannot adapt to displacement magnitudes that vary across models and scenes. This motivates our multi-scale threshold design.

(3) Decoding near switches. Frame-level scoring does not imply independent per-frame action decoding. Appendix G (Fig. 15) compares K = 1 (per-frame), 2, 4 and 8, recalibrating thresholds for each K. To assess how K affects Accuracy near action-switch boundaries, we compare K = 1, 2, 4, 8 on 2,232 switched videos from the 31 evaluation scenes. Both threshold settings use the same 864 calibration videos.

All K values are scored on the same intervals covered by complete windows, without tail filling. Near-switch interval timestamp midpoints lie in [−1, +1) s of a commanded switch. Interior contains the other scored intervals. We average within videos, then within models, then equally across the 12 models.

SettingKNear-switch Acc. (%)Interior Acc. (%)Δ vs. K = 1 (pp)95% Confidence Interval (95% CI, pp)
Single threshold150.2556.23Reference—
251.4657.32+1.21[+0.91, +1.55]
451.3757.78+1.12[+0.65, +1.70]
850.4557.84+0.20[−0.38, +0.91]
Four fixed, shared candidate thresholds164.0572.51Reference—
265.4373.87+1.38[+1.13, +1.68]
465.5874.31+1.53[+1.08, +2.02]
864.8374.38+0.77[+0.35, +1.26]

Δ is near-switch Accuracy relative to K = 1 within each setting. We use 2,000 paired source-scene bootstrap draws. At the default offset, K = 4 gains 1.12 pp with a single threshold and 1.53 pp with four candidates. Both 95% confidence intervals lie above zero.

Window alignment. At K = 4, shifting the window start by 0–3 intervals changes mean near-switch Accuracy by about 1 pp. Video switch times stay fixed.

Window-start offsetSingle threshold (%)Four candidates (%)
0 (default)51.3765.58
151.3465.44
250.4264.37
350.6264.95
Max–min difference (pp)0.951.20

(4) Planned revisions. We will add the calibration splits, candidate sets, per-model scores and ranks across all five splits, and switch and offset comparisons to Appendix G. We will clarify that each video uses one of four shared candidates selected by exact command agreement for its full rollout and that one decoded label per four intervals is reused for frame-level scoring.

[1] Wang et al. WorldCompass, 2026. Paper · Official CompassReward code.

AuthorsOpenReview ↗

Response to Reviewer Mfws (2/4)

Q2. Can the authors provide held-out human validation of the physics and scene-memory metrics? How sensitive are these metrics and the resulting rankings to pose estimation, reconstruction, and VLM judgment errors?

Answer. Thank you for asking about independent validation and evaluator sensitivity. We clarify the benchmark’s evaluation scope, the existing evidence, and the interpretation of its scores below.

(1) Mechanics calibration within the benchmark. Our mechanics judge was calibrated on 100 generated videos, with 20 expert labels per video. Agreement with the human majority increased from 52.5% to 73.2%, averaged over ten models (Appendix O.2). This study targets accurate scoring within WorldRoamBench’s scenarios and protocols. The reported agreement is measured on the calibration videos; held-out validation would use videos from this same setting that were not used to refine the judge, and remains future work.

(2) Scene-memory measurement and safeguards. We estimate the observation–revisit transition from decoded actions, register the point clouds to reduce rigid misalignment from pose drift, and combine filtering with density-normalized distances to reduce reconstruction and sampling-density effects. Failed reconstructions and clouds with too few remaining points are excluded; Table 10 reports 99–103 valid FPV cases per model out of 103 candidates. Appendix M illustrates why viewpoint differences can confound frame-pair similarity. These controls and examples support the measurement design; independent human validation of scene-memory scores remains future work.

(3) Scope of the reported rankings. The reported rankings describe performance under the specified evaluation pipeline. We have not quantified how changes in pose estimation, reconstruction, or VLM judgments affect Physics and Memory scores or rankings. We will clarify this scope and avoid interpreting small score differences as established performance advantages.

AuthorsOpenReview ↗

Response to Reviewer Mfws (3/4)

Q3. How do the authors ensure fair comparisons across models with different control interfaces and action adapters?

Answer. Thank you for raising this question. To ensure fair comparisons, we keep test cases and scoring rules common across models and express the same intended actions through each model’s native interface. We apply three controls:

(1) Same test cases and action requirements. For each shared test case, models receive the same starting image and prescribed action requirements, including the action order and requested duration. The intended task is therefore fixed across models, independent of their input formats (Section 3.1).

(2) Native inference configurations. For example, keyboard-controlled models receive keyboard commands, pose-conditioned models receive camera-pose sequences derived from those commands, and Yume 1.5 receives motion captions for its native 32-frame generation chunks. These conversions provide the inputs required by each model’s existing configuration. No model is fine-tuned on the benchmark. Appendix E and Table 7 document the interfaces and conversion rules.

(3) Common scoring of generated outputs. Generated videos are evaluated using the same Action, Vision, Physics, and Memory pipelines. Within each view, metric definitions and scoring procedures are shared across models, whether their inputs are keyboard commands, camera poses, or text. Thus, the comparison uses common output criteria across the different native interfaces.

We will make these comparison controls more explicit in Section 3.1 and Appendix E, alongside the interface descriptions in Table 7. Beyond these shared controls, a key challenge for fair comparison is that identical actions can produce very different motion magnitudes across models. We address this variation using multi-scale thresholds for discrete-action accuracy and adaptive reference trajectories for trajectory evaluation (Appendices G–H).

AuthorsOpenReview ↗

Response to Reviewer Mfws (4/4)

Q4. Can the authors expand the benchmark with more independent starting scenes and longer interaction sequences? Could physics tests be conducted at multiple durations to directly measure horizon-dependent physical degradation?

Answer. Thank you for these suggestions. Our current scope balances controlled test design, diagnostic value, and the cost of evaluating multiple models. We explain why we consider this scope appropriate below.

(1) Expanding independent starting scenes. Adding scenes is feasible, but requires checking that each new image supports all nine balanced duration–switch conditions with minimal unintended physical interactions. Otherwise, correctly stopping at a wall could be mistaken for poor action following (Appendix B.5). Our 72 starting images yield 648 Action cases, comparing all nine conditions from each image. The full benchmark contains 1,006 cases; for scale, WorldMark [1] reports 500 cases from 25 photorealistic and 25 stylized reference images. Expansion therefore requires careful interaction design, not simply collecting more images.

(2) Longer rollouts at benchmark scale. We retain 10–60 s as the leaderboard standard because this range already reveals substantial degradation under sustained action input. Across 12 models, mean FPV Memory drops from 65.38 (Short) to 41.75 (Long). Mean Vision and Action scores also decline (Table 3). Among the benchmarks compared in Table 1, WorldRoamBench matches the longest evaluation duration and offers the broadest coverage of the listed capabilities, evaluating four dimensions in both views.

Longer rollouts are feasible. Our visual-drift study includes videos up to one hour (Appendix I). However, most current interactive world models are designed for rollouts lasting tens of seconds. Requiring several minutes of continuous generation would substantially reduce the number of models we could evaluate. At the current 10–60 s setting, evaluating one model’s 1,006 pre-generated videos across both views and all four dimensions, including pose estimation, already takes about a day on a single eight-GPU machine. A multi-minute standard would increase both generation and evaluation costs, which could discourage teams from submitting new models to the leaderboard.

(3) Physics duration and sustained compliance. Different from Action and Memory, which compare performance across interaction lengths, Physics tests focus on whether the model responds correctly when the target interaction occurs. A shorter video may end before a slow-moving model reaches an obstacle, leaving that response untested. We therefore use the benchmark’s longest duration, 60 seconds, for all Physics cases. Once a model passes through a solid obstacle, extending the video cannot change that failure. For that observed interaction, the verdict is independent of the video’s total duration. Future work could measure Physics degradation with duration, testing whether initially correct behavior remains physically consistent in longer videos.

We will add the following clarification to the captions of the horizon-specific tables:

“The horizon-specific Overall scores include a fixed Physics component evaluated on 60-second rollouts; they do not measure how Physics performance changes with duration.”

[1] Xiaojie Xu et al. “WorldMark: A Unified Benchmark Suite for Interactive Video World Models.” arXiv:2604.21686v2, 2026.

Reviewer_TGB6OpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

WorldRoamBench addresses an important and underexplored problem: evaluating the long-term stability of interactive world models. Its multi-dimensional design, covering action following, visual drift, physics, and memory, makes the evaluation more comprehensive than existing short-horizon benchmarks. The benchmark is useful, although the claims should be framed more carefully as evaluating observable long-horizon behavior rather than fully grounded physical or closed-loop world-model competence.

Critical strengths
  1. WorldRoamBench addresses an important and underexplored problem: evaluating the long-term stability of interactive world models. Its multi-dimensional design, covering action following, visual drift, physics, and memory, makes the evaluation more comprehensive than existing short-horizon benchmarks.

  2. The benchmark includes relatively long interaction sequences and diverse environments, which helps expose failure modes that are easily missed in frame-level or short-video evaluations.

  3. The combination of automated metrics, trajectory analysis, and VLM-based judgments is practical and provides a useful starting point for evaluating systems that are difficult to test in a fully simulated environment.

Critical weaknesses
  1. The benchmark is not truly closed-loop: the action sequence is fixed in advance, and the model does not repeatedly observe the environment and adapt its future actions. Therefore, the results measure action-conditioned video generation more directly than complete world-model interaction.

  2. Several evaluation components depend on estimated camera poses, heuristic validity checks, and VLM judgments. The paper provides some calibration results, but does not report sufficient held-out accuracy, false-positive/false-negative rates, or confidence intervals to clearly establish the reliability of the final scores.

  3. The action-decoding threshold is selected using agreement with the ground-truth actions for each video, which may introduce an optimistic bias. A fixed threshold determined on a separate calibration set would make the evaluation more convincing.

  4. Invalid or unevaluable cases appear to be excluded from parts of the physics evaluation, so the reported physics scores may be conditional on successful evaluation. The paper should also report coverage-adjusted scores, together with results across multiple random seeds, to make the benchmark rankings more robust.

Questions for authors

All questions have been present in the above weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer TGB6 (1/4)

Thank you for recognizing the value of evaluating sustained interaction jointly across action following, visual stability, physics, and memory. We address the scope of the benchmark and the reliability of its measurements separately below.

W1. The benchmark is not truly closed-loop: the action sequence is fixed in advance, and the model does not repeatedly observe the environment and adapt its future actions. Therefore, the results measure action-conditioned video generation more directly than complete world-model interaction.

Answer. Thank you for raising this point. The world model and the controller or user play different roles in an interactive system.

  • World model. Receives actions and generates how the environment evolves in response.

  • Controller or user. Observes the environment and selects subsequent actions.

Our benchmark evaluates the world model. In closed-loop interaction, different model outputs may lead even the same controller or user to issue different action sequences, so action-following accuracies may not be directly comparable under these different action conditions. We instead compare different world models using the same starting image and prescribed action sequence for each case; closed-loop evaluation is beyond the scope of this paper.

We clarify the distinction in three respects:

(1) Evaluation target. Our action sequences are fixed in advance and are not adjusted based on the generated observations. We evaluate whether the generated environment responds coherently and remains stable under sustained action inputs, rather than how a controller or user makes decisions based on observations.

(2) Purpose and design of predefined action sequences. Predefined actions provide controlled tests of world-model behavior. For each case, we fix both the starting image and intended action sequence. We tailor the sequence to the starting scene: Action and Memory paths use open space to reduce physics-driven confounds, Memory uses inverse actions for revisits, and Physics paths target the interaction being tested (Appendices B.2 and B.5).

(3) Scope of the conclusions. We agree that the evaluation scope should be explicit in the paper. We will add the following clarification to the Limitations and Future Work section:

“WorldRoamBench evaluates observable world-model stability under predefined action sequences. These sequences are not adapted to generated observations, and the results do not establish closed-loop task success.”

We thank the reviewer again for this constructive suggestion. As future work, we could develop an online platform where users or controllers choose actions based on each model’s generated observations, with user feedback complementing our benchmark through assessments of interactive experience.

AuthorsOpenReview ↗

Response to Reviewer TGB6 (2/4)

W2. Several evaluation components depend on estimated camera poses, heuristic validity checks, and VLM judgments. The paper provides some calibration results, but does not report sufficient held-out accuracy, false-positive/false-negative rates, or confidence intervals to clearly establish the reliability of the final scores.

Answer. Thank you for asking how we verify the automatic scores. We compare Mechanics and Subject Memory judgments directly with expert assessments and test how well different pose pipelines recover actions.

(1) Mechanics calibration. We sampled 100 videos from 630 mechanics rollouts spanning 63 test cases and 10 models. Twenty experts labeled each video, and we used the per-video majority vote as the reference. Refining the prompts and evidence-gathering procedure increased model-averaged agreement from 52.5% to 73.2% on the same calibration videos (Appendix O.2).

(2) Subject-memory agreement. Twenty experts independently rated each of 60 third-person videos from four models without seeing the VLM scores. Across 1,200 video–expert pairs, 88.3% of ratings were within one point of the VLM score on the 1–10 scale, with a mean absolute difference of 0.83.

(3) Pose estimation and validity checks. We compared six pose pipelines on 4,212 first-person Action rollouts. MegaSaM+DA-v3 achieved a mean action-recoverability ceiling of 0.795, compared with 0.770 for upstream MegaSaM, and led on 11 of 12 models. The validity checks require usable camera-motion evidence in first person and continuous subject tracking in third person. Table 5 shows how many third-person videos pass these checks for each model and test suite.

(4) Uncertainty and scope. Our paired scene bootstrap gives 95% confidence intervals for Action score differences, quantifying their sensitivity to the choice of evaluation scenes (Reviewer GJ3v Q2). The Mechanics study reports calibration-set agreement. Held-out Mechanics accuracy and separate false-positive and false-negative rates remain open. We will make the scope of these results explicit in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer TGB6 (3/4)

W3. The action-decoding threshold is selected using agreement with the ground-truth actions for each video, which may introduce an optimistic bias. A fixed threshold determined on a separate calibration set would make the evaluation more convincing.

Answer. Thank you for suggesting a separate calibration set. We separate shared candidate calibration from per-video selection and report the requested comparison, including a single threshold calibrated on separate scenes and fixed for all evaluation videos.

(1) Measurement purpose and precedent. The same action can yield different displacements across models, scenes, and reconstruction scales. High thresholds can miss small movements; low thresholds can admit pose noise. Following WorldCompass’s multi-threshold design for scene-dependent pose scale, we extend the three fixed translation candidates in its released CompassReward code [1] to four.

(2) Shared candidates and one choice per video. The four translation candidates were originally calibrated jointly on all 4,212 FPV videos and are shared across models and scenes. For each video, we select the candidate yielding the highest exact agreement with its input commands and use that single threshold throughout the rollout (using exactly the same candidate-selection criterion as the released CompassReward code; see Algorithm 2 of our paper). We neither choose thresholds frame by frame nor search outside this shared four-candidate set. This restricted choice is intended to accommodate motion-scale variation across scenes, as motivated in point (1). Indeed, using the video’s input commands for selection can introduce optimistic bias. In future work, we plan to develop and validate a selector that chooses among these four candidates.

(3) Separate calibration experiment. Figure 15 already compares a global single threshold, four shared candidates with per-video selection, and a per-video full-grid ceiling. To address the reviewer’s suggestion, we additionally compare global-single and four-candidate decoding with thresholds determined on separate calibration scenes.

We partition the existing 39 source scenes into 8 calibration scenes (864 videos) and 31 evaluation scenes (3,348 videos), stratified across Nature, Urban, and Indoor. All models and action variants of a scene remain in the same partition. With pose estimates, K = 4, and rotation threshold 1° fixed, we select a single threshold and a four-candidate set from the same 31-value grid using only calibration videos. On the same evaluation videos:

MethodTranslation thresholdsThreshold rule at evaluationAccuracy (%)Models with changed rank
Original four[0.0001, 0.015, 0.05, 0.3]For each video, choose one of four75.70Reference
Calibrated four[0.0001, 0.015, 0.04, 0.3]For each video, choose one of four75.640/12
Calibrated single0.025Use the same threshold for all videos57.198/12

Counts indicate changes in Accuracy rank relative to Original four on the same 3,348 evaluation videos.

Across five scene splits, recalibrating the four candidates changes mean accuracy by -0.18 to 0.00 percentage points relative to the original four on each split’s evaluation set; the 12-model accuracy ordering is unchanged within every split. This supports limited sensitivity to candidate recalibration on the tested splits.

Using one global threshold lowers primary-split accuracy by 18.45 percentage points and changes 8 of 12 Accuracy ranks relative to calibrated-four. A shared numerical cutoff cannot adapt to video-specific motion scales, motivating our multi-scale design.

(4) Planned manuscript revisions. We will revise Appendix G, “Threshold Scaling and Calibration,” to distinguish shared candidate-set calibration from per-video threshold selection, adding the following clarification:

“All models share four fixed translation candidates. For each video, we select the candidate whose decoded actions have the highest exact agreement with the input commands and apply it throughout the rollout. We neither choose thresholds frame by frame nor search outside this shared four-candidate set.”

We will also add the completed scene-disjoint calibration study to Appendix G, including the scene partitions, selected thresholds, and per-model accuracy and rankings for the original-four, calibrated-four, and calibrated-single settings across all five splits. The calibrated-single baseline uses one translation threshold selected solely on calibration scenes and applied unchanged to every evaluation video.

[1] Zehan Wang et al. “WorldCompass: Reinforcement Learning for Long-Horizon World Models.” arXiv:2602.09022, 2026. Official CompassReward implementation.

AuthorsOpenReview ↗

Response to Reviewer TGB6 (4/4)

W4. Invalid or unevaluable cases appear to be excluded from parts of the physics evaluation, so the reported physics scores may be conditional on successful evaluation. The paper should also report coverage-adjusted scores, together with results across multiple random seeds, to make the benchmark rankings more robust.

Answer. Thank you for highlighting coverage and generation variability. We clarify the scope of the reported scores and the available evidence below.

(1) Why we exclude unevaluable cases. Our Physics score measures physical plausibility among evaluable cases. The validity gate requires usable camera-motion evidence in FPV and continuous subject tracking in TPV; cases failing these checks are excluded before Physics scoring. Protocol-specific evidence checks can also mark cases unevaluable (Appendix J.1–J.2). We average valid scores within each domain and weight domains with valid cases equally, so the Physics score is conditional on each model’s evaluable subset.

(2) How controllability enters the TPV leaderboard. TPV Overall is multiplied by the Control Rate pooled across Action, Memory, and Physics (Sec. 2.6). Thus, inability to maintain a continuously trackable subject affects Overall, while Physics is assessed where usable evidence is available. Table 5 reports TPV gate counts, including 70/71 Physics cases for Happy Oyster and 28/71 for HY-World 1.5, before any later protocol-level exclusions. We retain this overall weighting and conditional Physics scoring; no separate coverage-adjusted Physics score is reported. We will clarify this distinction in Appendix J.

(3) Generation variability and preliminary observations. We have not conducted a systematic repeated-generation study. Generating the benchmark videos across 12 models took about a month, partly due to daily quotas on closed-source models; repetition would require substantial computation, access fees, and time. Visual checks of a few cases from open-source models showed broadly similar physical behavior across seeds. These qualitative observations are preliminary; systematic assessment of generation-seed effects on scores and rankings remains future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.