acceptodds
Under review as a conference paper at ICLR 2027

LongWM-Bench: Benchmarking Persistent and Navigable Worlds over Long-Horizon Interaction

Abstract

Short-term visual quality and action following offer limited evidence of whether an interactive world model remains reliable during extended use. We introduce LongWM-Bench, a benchmark of 300 tasks with target rollout durations of 60–180 seconds. Two task suites evaluate sustained control and revisit memory through combined inputs, repeated changes in control, and six families of trajectories with delayed and repeated revisits. We combine automatic visual and motion measurements with checklists tailored to each scene to assess visual quality, temporal consistency, revisit retention, physical and causal validity, and camera responses to actions. Evaluating 12 interactive world models reveals distinct capability profiles: models with similar appearance consistency differ substantially in revisit retention, while accurate camera responses can coexist with weak physical and causal validity. Coherent scene evolution remains a common weakness under our evaluation. These findings highlight the need to evaluate whether generated worlds remain controllable, retain previously observed content, and evolve coherently throughout interaction. Evaluation code and example data are included in the supplementary material.

Primary area:
generative models
Keywords:
World models, interactive video generation, long-horizon evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_knyzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LongWM-Bench. It benchmarks 300 tasks with longer target rollout durations of 60 to 180 seconds in order to evaluate sustained control, delays, and revisits in World Model explorations (With interactions). This benchmark combines visual and motion metrics with scene-specific judgements to assess controllability, perceptual quality, temporal consistency, revisit memory, and physical and causal validity. 12 major models were experimented on with a supplementary human-agreement study to mainly show that appearance consistency and accurate control do not necessarily imply scene retention or coherent world evolution. Main contributions are the extended evaluation horizons and systematic revisit task design.

Critical strengths
  1. The task designs for sustained interaction in this benchmark are outstanding. This work extends the evaluation to target duration of 60-180 seconds (targeted on delays, revisits, etc.), which is significantly impactful for future benchmark design in this area. Because it addresses a gap in evaluating world persistence during continued exploration. Test cases demonstrated in this work also are meaningful for developing more reliable interactive WM.

  2. Very useful empirical characterizations. The reported capability profiles give evidence of differences in different models in a way that reveals important mechanisms, this will help identify priorities for further investigations.

Critical weaknesses
  1. Some core metrics are somewhat unclear. This could undermine the quality of this benchmark. To my understanding, revisit similarity can be affected by navigation errors or limited movement. Although the appendix restricts evaluation to actual revisits, how successful departures and returns are verified, and how exclusions affect cross-model comparisons. Moreover, the 60° translation tolerance accepts a 45° error, so it means that executing only W when W+D is requested can count as correct. These issues weaken the interpretation of the reported memory and composite-control scores.
Questions for authors
  1. How do you verify that a model has actually left and returned to an observation anchor, and how are unsuccessful or unverifiable revisits handled? If there are clear verifications and they hold true on the identical effective samples then my proposed weakness will be undermined.

  2. Does the translation metric impose any component-wise requirements beyond the 60° direction tolerance? Under the stated rule, executing only W when W+D is requested would count as correct. If no additional constraints are applied, do the composite-control findings hold when such partial execution is treated as failure?

5 comments
AuthorsOpenReview ↗

Response to Reviewer knyz — Part 1/3

We thank the reviewer for the thoughtful assessment and constructive suggestions. We appreciate the recognition of our sustained-interaction task design and its value for evaluating world persistence during continued exploration, as well as the usefulness of the model capability profiles. We address the two metric concerns below through a departure-and-return audit on identical effective observation pairs and a stricter-threshold analysis of composite controls.

Q1: Verifying Departures and Returns in Revisits

R:

Thank you for this suggestion. We believe that memory evaluation in revisit tasks should build on reliable navigation control. A model needs to leave and return to a previously observed location so that content retention can be assessed from the corresponding viewpoint. We therefore interpret GMS/LMS as joint measures of navigation and visual retention, as now clarified in Appendix C.3. To address your questions about verification and sample consistency, we supplement the evaluation with a ViPE-based departure-and-return audit and recompute scores on identical observation pairs that pass the audit across all four models.

(1) Departure and return verification criteria.

Our tasks use symmetric out-and-back paths, closed loops, and nested revisits to specify the anchor associated with each observation. For example, in A→B→C→B→A, the later observations of B and A correspond to earlier observations of the same anchors. We use these correspondences to pair observations and further check the actual motion and return poses in the generated trajectories.

This analysis includes Lyra 2.0, Echo-WM, LingBot-World-v2, and Matrix-Game 2.0, each with cached ViPE trajectories for all 150 revisit tasks. GMS compares the initial observation with subsequent revisits, while LMS compares successive revisits to the same anchor. For each observation window, we select the valid pose nearest the actual scoring-frame timestamp and apply the following criteria:

  • Departure: between the two observations, the pose differs from the pair’s reference observation by more than 30° in rotation or 2 ViPE coordinate units in translation, sustained for at least 0.5 seconds and three consecutive valid pose samples.
  • Return: both observations satisfy rotation and translation tolerances relative to the same anchor’s initial pose. We test three return-tolerance settings: 45°/10 units, 60°/10 units, and 60°/15 units.

The departure criterion checks sustained pose changes, while the return criterion checks approximate anchor alignment within the observation windows. Translation thresholds are expressed in ViPE reconstruction units.

(2) Handling unsuccessful or unverifiable pairs and constructing identical effective samples.

If any model fails the departure or return check, or lacks usable poses, the observation pair is excluded for all four models in this supplementary analysis. For each metric and tolerance setting, we retain the intersection of observation pairs that pass the audit across all four models. Every comparison therefore uses exactly the same eligible observation pairs.

This filtering applies to the supplementary scores GMS′/LMS′; the original GMS/LMS remain joint measures computed at prescribed revisit checkpoints. When recomputing scores, we preserve the original visual similarities, elapsed-time weights, weighted aggregation within each anchor, and equal weighting across anchors and tasks.

To examine the effects of observation-pair filtering and changes in task coverage on model rankings, we compare against two baselines:

  • Full common baseline: all shared observation pairs with original scores for every model, comprising 425 GMS pairs from 150 tasks and 184 LMS pairs from 89 tasks.
  • Retained-task baseline: only tasks retained after the audit, but including all originally shared observation pairs within those tasks before filtering.
AuthorsOpenReview ↗

Response to Reviewer knyz — Part 2/3

(3) Model comparisons on jointly verified observation pairs.

Rotation toleranceTranslation toleranceMetricρ: full common baselineρ: retained-task baseline
45°10GMS′1.0001.000
45°10LMS′1.0001.000
60°10GMS′1.0001.000
60°10LMS′1.0001.000
60°15GMS′1.0001.000
60°15LMS′1.0001.000

Here, ρ is Spearman’s rank correlation across the four models.

Under the 60°/15-unit setting, we obtain 82 shared GMS observation pairs across 51 revisit tasks, with all four models satisfying the stated departure and approximate-return criteria for every retained pair. Recomputing scores preserves the four-model rankings of both baselines, with Spearman ρ=1.000. LMS retains 25 shared observation pairs from 21 tasks, also with a ranking correlation of 1.000. Across all three tolerance settings, both metrics preserve the rankings of both baselines.

LMS requires at least three observations of the same anchor to form a pair of successive revisits, resulting in fewer available and retained pairs than GMS. These results show that the original four-model ordering is preserved on identical effective observation pairs that jointly pass the departure-and-return audit.

(4) Supplementary sensitivity analysis of return-pose tolerances.

We additionally conduct a case-level return-pose sensitivity analysis on a subset of six models and 100 revisit tasks. A case contributes to GMS_cond/LMS_cond only when every revisit satisfies both rotation and translation tolerances relative to its corresponding initial anchor observation. Cases failing these checks or lacking usable poses are excluded; observation frames, visual features, and aggregation procedures remain unchanged.

This analysis filters cases separately for each model, so the retained cases and eligible models can vary across settings. Each row compares the original and conditional rankings over the same eligible models. It examines return-threshold sensitivity, while the shared-pair audit above directly tests model comparisons on identical effective samples.

Rotation toleranceTranslation toleranceρ (GMS)τ-b (GMS)ρ (LMS)τ-b (LMS)
45°10.9000.8000.9000.800
45°21.0001.0001.0001.000
45°51.0001.0001.0001.000
40°10.9000.8000.9000.800
40°20.7710.6001.0001.000
40°51.0001.0000.9430.867
35°20.9430.8671.0001.000
35°51.0001.0000.9430.867
30°20.9000.8001.0001.000
30°50.9430.8670.9430.867
25°20.9000.8001.0001.000
25°51.0001.0000.9000.800

Translation tolerances are positional-distance thresholds in ViPE reconstruction units. Across the twelve settings, Spearman correlations with the original rankings range from 0.771 to 1.000 for GMS_cond and 0.900 to 1.000 for LMS_cond, indicating high ranking agreement under the tested return-pose tolerances. The complete threshold results are reported in Appendix C.3, Table 5.

We have added the shared-pair departure-and-return audit to Appendix C.3, specifying the verification criteria, the exclusion of unsuccessful or unverifiable pairs across all four models, the resulting common-sample coverage, and the ranking comparisons. The supplementary case-level return-pose sensitivity results are reported in Table 5.

AuthorsOpenReview ↗

Response to Reviewer knyz — Part 3/3

Q2: Analysis of the 60° Tolerance

R:

Thank you for this suggestion. The current TR criterion applies no additional component-wise requirements, so executing only W for a W+D command does count as correct. We therefore tighten the directional tolerance to 30°, which rejects this partial execution, and re-examine composite-control performance. Across eleven models, TR remains lower for composite inputs than for single-key inputs. We detail the criterion and supplementary results below.

(1) The current TR criterion and partial execution.

TR sums the commanded translation vectors to obtain a target direction and checks whether the observed net displacement lies within a 60° tolerance of that direction. For a W+D command, executing only W produces a 45° directional deviation and therefore passes the current criterion.

We tighten the tolerance to 30°, so that this single-axis partial execution no longer passes, while keeping the displacement-direction calculation unchanged.

(2) Composite-control performance under the 30° criterion.

We recompute TR across eleven models, excluding Genie 3. For each model and input group, the 60° and 30° evaluations use the same evaluable action segments. We compare single-key and composite-key inputs and additionally report the subset of commands containing two translation directions.

ModelSingle-key TR, 60°Composite-key TR, 60°Single-key TR, 30°Composite-key TR, 30°Two-axis translation TR, 30°
HappyOyster82.378.071.153.554.8
Lyra 2.073.867.364.648.448.1
Echo-WM80.878.072.062.955.7
LingBot-World-v286.782.181.257.762.0
Matrix-Game 3.084.974.875.852.551.6
Matrix-Game 2.081.378.672.159.259.7
DreamX-World95.293.390.281.381.6
Alaya-EVOKE64.254.356.335.522.4
ABot-World63.552.848.326.342.0
Yume-1.569.548.161.627.614.4
Cosmos 329.035.918.217.314.7

Scores are percentages, with equal weighting across action segments within each input group. Single-key inputs contain only one translation key; composite-key inputs contain a translation key together with additional translation or rotation keys. Two-axis translation inputs contain two translation directions and may also include rotation.

Under the 30° criterion, all eleven models have lower TR for composite-key inputs than for single-key inputs, and the same pattern holds for the two-axis translation subset. For example, HappyOyster achieves 71.1%, 53.5%, and 54.8% on single-key, composite-key, and two-axis translation inputs, respectively; the corresponding scores for Echo-WM are 72.0%, 62.9%, and 55.7%.

These results show that the main finding of lower translation accuracy under composite inputs persists when the single-axis partial execution described by the reviewer is counted as failure.

(3) Model-ranking stability across directional tolerances.

We additionally sweep the tolerance from 60° to 10° in 5° steps, keeping each model’s evaluable action segments fixed across thresholds. We compute TR within each task and then average equally across tasks to obtain overall model scores. The table below compares the ranking at each threshold with the default 60° ranking.

Ranking correlation60°55°50°45°40°35°30°25°20°15°10°
Spearman ρ1.0000.9820.9730.9730.9730.9550.9730.9550.9730.9550.955
Kendall τ-b1.0000.9270.8910.8910.8910.8550.8910.8550.8910.8550.855

The Spearman correlation between the 30° and default 60° rankings is 0.973; across the entire sweep, the minimum correlation is 0.955 and the maximum rank displacement is two positions. Thus, stricter directional criteria preserve the main composite-control finding and maintain high agreement in the overall model rankings.

The TR criterion is defined in Appendix D.1.5. The partial-execution issue and stricter-threshold analyses are included in Appendix C.4, Figures 12–13.

Reviewer_BnyQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel benchmark, LongWM, that evaluates the reliability of interactive world models over extended interaction. covering complex control and six revisit trajectory families. It comprises 300 tasks with target durations of 60–180 seconds, divided equally between complex-control and revisit-memory tasks. The evaluation covers 5 dimensions: perceptual quality, temporal consistency, revisit memory, physical & causal validity, and action controllability. The experiments cover 12 world models, and the results suggest that appearance consistency, revisit retention, and control accuracy capture different aspects of performance, while coherent scene evolution remains challenging.

Critical strengths
  1. The benchmark operationalizes an important evaluation problem through sustained control and structured revisits. Testing combined controls, direction changes, and six revisit families provides a concrete way to examine whether generated environments remain usable beyond locally convincing frames. Delayed and repeated returns are particularly valuable because they expose failures that adjacent-frame consistency cannot capture.
  2. The evaluation provides informative capability profiles. The results of evaluated models and the separate reporting of control, memory, and physical–causal metrics reveal meaningful differences. The human study and sensitivity analyses also represent substantive validation efforts. These features make the benchmark potentially useful for diagnosing model weaknesses, even though the validity of several measurements requires stronger support.
Critical weaknesses
  1. Ambiguity in the definition of action control The paper does not clearly establish a consistent interpretation of movement commands across first- and third-person settings. TR and CR measure action controllability through camera motion, whereas Figures 12 and 14 illustrate first-person camera control and subject control in a third-person view, respectively. This ambiguity leaves unclear which form of interaction the benchmark aims to evaluate and whether the proposed metrics adequately capture the intended control behavior in both settings.
  2. Unclear control alignment across different world models. Although the paper describes converting benchmark actions into model-specific input formats, it remains unclear how these conversions are implemented and how equivalent intended control behaviors are ensured across models. This limits confidence in the fairness and comparability of the reported controllability results.
  3. Limited validation on real videos. The evaluation metrics are compared with human judgments on model-generated videos, but no corresponding assessment on real videos is reported. Including real videos as a reference would provide additional evidence of the evaluation framework’s validity, particularly its ability to recognize physically plausible and temporally coherent behavior. Although the human comparison is useful, relying solely on generated videos may leave potential blind spots shared by the evaluators and the evaluated models insufficiently examined.
Questions for authors
  1. What's the definition of action control? Specify whether movement commands control the camera, the subject, or both in first- and third-person settings. Explain how the behaviors illustrated in Figures 12 and 14 relate to the camera-based TR/CR metrics, and clarify what these metrics establish about third-person subject control.

  2. How to align the control input of different world models? Document how benchmark actions are converted into each model’s native inputs, including motion magnitudes, timing, and unsupported controls.

  3. What are the evaluation results of current metrics on the real videos?

  4. How to distinguish navigation failure from memory failure when computing GMS/LMS? Although the prescribed action sequences are designed to return to an earlier viewpoint, this does not establish that the generated camera trajectory actually does so. Is successful return verified before measuring revisit similarity? If not, should GMS/LMS be interpreted as joint measures of navigation and visual retention rather than memory retention alone?

9 comments
AuthorsOpenReview ↗

Response to Reviewer BnyQ — Part 1/4

We thank the reviewer for the detailed assessment and constructive suggestions. We appreciate the recognition of sustained control and structured revisits as valuable evaluation tasks, particularly the ability of delayed and repeated returns to expose failures beyond adjacent-frame consistency. We also appreciate the positive assessment of our model capability analyses. Below, we clarify the control protocol and metric definitions, and present additional real-video evaluation and pose-based revisit analyses.

Q1: About Action-Control Semantics

R:

TR/CR evaluate the camera’s response to navigation commands in both first- and third-person settings. We clarify the measurement target, its interpretation across viewpoints, and the role of the qualitative examples below.

(1) Measurement target of TR/CR.

Following the camera-based navigation formulation used in prior work such as WBench, we map the prescribed commands to each model’s native interface. Camera-conditioned models receive camera trajectories, while action-conditioned models receive native movement and view-control inputs. TR measures whether camera displacement follows the commanded translation direction, and CR measures whether yaw/pitch rotation follows the commanded direction with sufficient angular magnitude.

(2) Control semantics in first- and third-person settings.

In first-person settings, these responses directly describe navigation from the observer’s viewpoint. In third-person settings, the model’s native interface determines how subject and camera motion are coupled. When the camera follows a moving subject, TR/CR measure the resulting camera-navigation response. The quantitative control scores therefore assess camera navigation; subject behavior is examined qualitatively.

(3) Relationship between qualitative examples and quantitative control evaluation.

Figures 15 and 17 illustrate these complementary aspects. Figure 15 shows ongoing scene evolution during first-person navigation, while Figure 17 shows third-person behavior under combined inputs, including subject motion, identity changes, and scene drift. Frames were selected to illustrate characteristic model behaviors, and their timestamps differ across models; frames in the same column therefore do not necessarily correspond to the same action stage. TR/CR are computed from camera poses within each video’s corresponding instruction windows.

The camera-based definitions of TR/CR are stated in Section 3.3 and Appendix D.1.5. We have clarified the qualitative role of these examples and the frame-selection procedure in the captions of Figures 15 and 17 in Appendix H.

AuthorsOpenReview ↗

Response to Reviewer BnyQ — Part 2/4

Q2: Cross-Model Control Alignment and Equivalence

R:

Thank you for this suggestion. We represent each task as a shared, time-indexed action sequence specifying translation directions, view rotations, simultaneous inputs, idle intervals, and segment durations. Model-specific adapters convert this sequence into each model's native inputs. Models use the same initial image and task requirements, with text conditioning and rollout duration handled according to the interface capabilities documented in Appendix D, Table 9.

(1) Conversion to native inputs. For keyboard/mouse- and action-conditioned models, commands are converted into key states, mouse increments, or action codes. For camera-conditioned models, the adapters construct camera trajectories from the action sequence and encode them in the required representation, such as camera poses, Plücker embeddings, PRoPE, or UCPE. Where supported, simultaneous controls are encoded jointly within the same interval; idle intervals use the corresponding neutral inputs.

(2) Motion magnitudes and timing. Each adapter uses fixed translation steps, yaw/pitch increments, or action amplitudes appropriate to the model's native control interface. For models requiring explicit camera trajectories, the adapters construct per-frame poses using fixed motion parameters. The shared action sequence aligns the requested navigation behavior and its intended timing; the numerical control magnitudes and temporal discretization used by each model are documented separately in the appendix.

We have added a per-model control-mapping table to Appendix D (Table 10), reporting native control parameters and their units, timing conversions, and any generation-block quantization, output trimming, or temporal resampling. For example, Matrix-Game 2.0 and 3.0 both use native mouse increments of ±0.1, while a 0.5-second instruction is converted into six and eight control frames, respectively. Yume-1.5 quantizes instruction durations to native generation blocks, whereas ABot-World applies temporal resampling after generation. Idle inputs follow each model's native dynamics; for example, Echo-WM can retain motion that decays through inertia under neutral inputs.

(3) Unsupported controls and interface adaptations. In the evaluated Cosmos 3 configuration, yaw has a corresponding action, while pitch does not. Turning in place is approximated by a turn with forward motion, and lateral translation commands are mapped to curved forward motion in the corresponding direction. Genie 3 is evaluated through its official hosted interface, with a maximum evaluated rollout duration of 60 seconds; unavailable long-horizon metrics are marked as N/A.

(4) Comparison on commonly supported rotation controls.

The main CR evaluation includes pitch commands. To examine performance on commonly supported rotation controls, we additionally compare CR over all rotation controls with CR restricted to the yaw-control subset. The results for eleven models, excluding Genie 3, are shown below.

ModelCR: all rotations (%)CR: yaw subset (%)Difference (pp)
Matrix-Game 3.095.897.0+1.2
Lyra 2.095.796.5+0.8
HappyOyster93.693.7+0.1
Echo-WM92.796.6+3.9
Matrix-Game 2.090.191.7+1.6
DreamX-World90.092.9+2.9
LingBot-World-v287.887.9+0.1
ABot-World81.982.1+0.2
Alaya-EVOKE41.748.2+6.5
Yume-1.524.528.0+3.5
Cosmos 322.837.8+15.0

In this supplementary comparison, the yaw-subset ranking has a Spearman correlation of 0.955 with the all-rotation ranking across the eleven models. Cosmos 3's CR increases from 22.8% to 37.8%, a gain of 15.0 percentage points, consistent with its support for yaw but not pitch. The high ranking agreement indicates that relative model performance remains broadly consistent when evaluation is restricted to commonly supported yaw controls. The common-yaw comparison is reported above, and the control-adapter specifications are provided in Appendix D, Table 10.

To support reproducibility, we will release the inference code, model-specific adapters, and configurations used in our evaluation upon acceptance, including action mappings, motion parameters, timing conversions, and the handling of unsupported controls.

AuthorsOpenReview ↗

Response to Reviewer BnyQ — Part 3/4

Q3: Validation on Real Videos

R:

Thank you for this suggestion. We evaluated 80 real videos totaling approximately 93 minutes, comprising 50 egocentric videos from DL3DV-10K and 30 exocentric videos from Ego-Exo4D.

(1) Real-video data and evaluation setup.

The DL3DV-10K subset contains moving-camera recordings of diverse indoor and outdoor environments, with durations of 56–94 seconds. Reconstructed camera poses identify revisit candidates with similar positions and viewing directions and link them to timestamps for GMS/LMS pairing. The Ego-Exo4D subset contains fixed-camera recordings from 24 locations across seven activities, including basketball, soccer, climbing, cooking, and bicycle repair, with durations of 63–120 seconds.

For revisit-related evaluation, we use two-observation pairs. In the egocentric videos, reconstructed camera poses identify earlier and later observations with similar positions and viewing directions. In the exocentric videos, frames at approximately 10% and 90% of each recording provide a fixed-view comparison across time. We report GMS and the corresponding two-observation adaptation, LMS-2obs.

(2) Reference scores and comparison with generated videos.

The table below reports equal-weighted video-level means, with scores expressed on a 0–100 scale. TR/CR are not reported because these recordings do not include the prescribed control-command sequences required to evaluate action following.

Real-video setVQDDJPSCBCGMSLMS-2obs
Egocentric, 50 videos70.1943.4998.09—92.4385.3085.30
Exocentric, 30 videos68.9511.7597.9484.1095.7788.2488.24

Both real-video sets exceed the highest generated-video scores in Table 2 for visual quality, temporal continuity, and background consistency. Their VQ scores are 70.19/68.95, JP scores are 98.09/97.94, and BC scores are 92.43/95.77, compared with generated-video maxima of 68.61, 88.87, and 86.91, respectively. Exocentric SC reaches 84.10, also exceeding the generated-video maximum of 82.19. These results support the metrics’ ability to recognize the visual quality and consistency of real recordings. Real videos can nevertheless receive imperfect similarity scores because normal viewpoint changes, articulation, and occlusion also alter visual features.

(3) Metric-specific interpretation and adaptations.

DD depends on the motion present in the footage. The exocentric set has a DD of 11.75, primarily because its cameras are largely stationary and motion is concentrated in people and manipulated objects. Egocentric camera movement produces motion across much of the image, yielding a higher DD of 43.49. However, these recordings generally exhibit relatively slow, smooth camera movement, which supports stable multiview capture and camera-pose reconstruction while limiting motion intensity. DD therefore reflects capture conditions and scene dynamics, while real videos are not necessarily expected to maximize this metric.

For SC and BC, egocentric SC is omitted because the reference annotations do not designate persistent foreground subjects or provide corresponding masks. Exocentric SC uses masks of the main activity participants. Both metrics are sampled at approximately 1 FPS.

For GMS and LMS-2obs, egocentric pairs are pose-assisted geometric revisit candidates, whereas exocentric pairs measure appearance retention across time from a fixed viewpoint. The original LMS requires at least three observations—an initial observation followed by two returns—to assess retention across repeated revisits. Given the real-video constraints described above, LMS-2obs includes the comparison between the initial and first subsequent observations. With two observations, LMS-2obs equals GMS; the two columns therefore report the same pairwise appearance comparison. The real-video GMS scores of 85.30 and 88.24 are numerically higher than the generated-video maximum of 79.81.

(4) Supplementary checklist assessment.

We additionally conducted a supplementary visual assessment using scene-specific checklists for PC, CC, and EC. This assessment covers a total of 329 checklist questions across 80 real videos.

Real-video setPCCCEC
Egocentric, 50 videos100100100
Exocentric, 30 videos100100100

Scores are expressed on a 0–100 scale, with 100 corresponding to a rating of 5/5. Across the scored checklist items, no confirmed violations were identified in the sampled evidence for support relationships, material and shape properties, lighting relationships, and scene evolution. These evaluations provide real-video reference values for the assessed metrics.

The real-video reference scores, evaluation settings, and metric-specific analyses are included in Appendix I (Table 16), together with the supplementary PC/CC/EC assessment covering 329 checklist questions.

Reviewer_1oF7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LongWM-Bench, a benchmark for evaluating the long-horizon reliability of interactive world models. It consists of 300 tasks with 60–180-second interactions, focusing on sustained controllability and world persistence through complex action sequences and repeated revisits. The benchmark evaluates five dimensions using automatic metrics and VLM-based judgments. Experiments on 12 world models reveal substantial differences in their capabilities, particularly in revisit memory, action controllability, and physical/causal consistency, highlighting ongoing challenges in maintaining coherent and persistent interactive worlds.

Critical strengths
  1. The paper addresses an important and underexplored problem of evaluating long-horizon consistency, controllability, and memory in interactive world models.

  2. The benchmark covers diverse scenes and interaction patterns, with 300 tasks and six revisit trajectory families across five evaluation dimensions.

  3. The evaluation of 12 world models provides useful insights into their distinct capability profiles and failure modes, supported by detailed quantitative and qualitative analyses.

Critical weaknesses

W1. Insufficient validation of benchmark data reliability and potential construction biases.

The benchmark relies heavily on LLM-generated scene descriptions, action trajectories, and physical/causal evaluation checklists. Although the authors employ automated verification and human filtering, it remains unclear whether these procedures adequately ensure the correctness and representativeness of the resulting evaluation data. In particular, LLM-generated checklists may introduce ambiguous expectations about future scene evolution, while the selected scene and action distributions may systematically favor certain models.

W2. Limited construct validity of the proposed evaluation metrics.

Several core metrics may not faithfully capture their intended capabilities. GMS/LMS rely on visual feature similarity, potentially conflating world-memory failures with camera-pose and viewpoint differences. Similarly, TR/CR primarily evaluate directional agreement rather than accurate sustained control, overlooking factors such as motion magnitude and accumulated trajectory errors. These limitations raise concerns about whether the reported scores genuinely reflect the intended world-model capabilities.

W3. Insufficient evidence that the reported model comparisons are reliable and unbiased.

The paper does not sufficiently establish whether its evaluation results reflect genuine differences in model capabilities rather than artifacts of evaluation choices. Automatic metrics exhibit disagreement with human judgments (Table 3), and evaluator selection may influence physical/causal scores (Appendix G). Furthermore, heterogeneous control interfaces complicate cross-model comparisons. The current analyses do not adequately demonstrate the robustness of the reported relative model performances.

Questions for authors

Regarding W1 — Benchmark Data Reliability

Q1. How do the authors validate the correctness and unambiguity of LLM-generated physical/causal checklists, particularly when multiple future scene evolutions are plausible from the same initial observation?

Q2. How sensitive are the reported model comparisons to the benchmark's data construction choices, such as scene sources, viewpoint distributions?

Regarding W2 — Metric Construct Validity

Q3. Since GMS/LMS are based on visual feature similarity at predefined revisit checkpoints, how do the authors account for differences in the actual camera poses reached by different models? Can the observed score differences be attributed to world-memory consistency rather than navigation or viewpoint errors?

Q4. TR/CR evaluate per-segment directional agreement without explicitly measuring motion magnitude, response latency, or accumulated trajectory errors. Is there evidence that models with higher TR/CR scores also exhibit more reliable long-horizon control behavior?

Regarding W3 — Reliability of Model Comparisons

Q5. Do the relative model performances implied by the automatic metrics agree with independent human judgments? Can you examine model-ranking agreement beyond the score-level L1 differences reported in Table 3?

Q6. Do the conclusions remain consistent across alternative VLM judges and commonly supported control settings?

9 comments
AuthorsOpenReview ↗

Response to Reviewer 1oF7 — Part 1/6

We thank the reviewer for the careful assessment and constructive suggestions. We appreciate the recognition of the importance of long-horizon evaluation, the diversity of our scenes and interaction patterns, and the insights provided by the model capability and failure analyses. Below, we address the concerns about checklist validity, data composition, metric interpretation, and evaluation reliability with clarifications and supplementary analyses.

Q1: Checklist Correctness and Ambiguity under Multiple Plausible Futures

R:

Thank you for this suggestion. The checklists assess physical and causal constraints grounded in the scene and task, allowing multiple plausible evolutions that satisfy these constraints. We reduce ambiguity about future evolution through clearly defined checks and human filtering, and supplement this with real-video reference evaluation and agreement with independent human model rankings.

(1) Checklist scope and multiple plausible futures.

The checks cover observable phenomena such as support relations, material and shape properties, lighting relations, and scene evolution. They assess whether the observed evolution satisfies the relevant scene and task constraints; multiple plausible futures can meet these requirements.

For example, a support check asks whether a table remains physically supported. An evolution check is tied to the phenomenon described in the scene—for example, whether a balloon specified as rising shows visible progression over the evaluation interval. The check assesses whether the described ascent is reflected in the observations, allowing different plausible trajectories. Table 12 provides concrete PC, CC, and EC checklist examples illustrating what each dimension assesses.

(2) Checklist construction and human review.

During checklist construction, items are generated according to each dimension’s applicability to the input scene. Inapplicable dimensions are marked accordingly and have no items. Candidate items are produced by three LLMs and filtered by human reviewers, who remove redundant, speculative, ambiguous, and visually unverifiable items; approximately 78% are retained. The final checklists contain 1,405 items across 300 benchmark cases, averaging 4.68 items per case: 736 PC items, 314 CC items, and 355 EC items. We report this distribution in the dataset statistics (Sec. 3.2); checklist-extraction and scoring prompts are provided in App. J.2.

(3) Real-video references and human-ranking agreement.

We conducted a supplementary visual assessment on 80 real videos, comprising 50 egocentric videos from DL3DV-10K and 30 exocentric videos from Ego-Exo4D. The former contain moving-camera recordings of indoor and outdoor scenes, while the latter contain fixed-camera recordings of human activities. We construct scene-specific PC, CC, and EC checklists for these videos, covering 329 checklist questions in this supplementary assessment.

Real-video setPCCCEC
Egocentric, 50 videos100100100
Exocentric, 30 videos100100100

Scores are expressed on a 0–100 scale, with 100 corresponding to a rating of 5/5. Across the scored checklist items, no confirmed violations were identified in the sampled evidence for support relationships, material and shape properties, lighting relationships, and scene evolution. These results provide real-video reference values for the assessed metrics.

Across 10 open-source models, the checklist metrics also agree closely with independent human model rankings, with SRCC values of 0.976, 0.952, and 0.952 for PC, CC, and EC, respectively; the full alignment analysis is reported in our response to Q5. Human filtering, real-video references, and human-ranking agreement provide complementary evidence through item review, performance on real scenes, and model comparisons.

AuthorsOpenReview ↗

Response to Reviewer 1oF7 — Part 2/6

Q2: Sensitivity to Scene Sources and Viewpoint Distributions

R:

Thank you for this suggestion. We examine whether scene sources, viewpoints, and domains affect model comparisons through two analyses: subset analysis compares rankings within categories, while uniform reweighting tests rankings under changes in category proportions.

(1) Rankings within scene-source, viewpoint, and domain subsets.

We stratify cases by scene source (real / real-AI-edited / AI-generated), viewpoint (first / third person), and nine scene domains. Within each subset, we recompute model scores and compare the resulting rankings with the corresponding full-data rankings using Spearman’s ρ. Entries report the min/median correlation across subsets within each stratification.

StratificationVQDDJPSCBCGMSLMSPCCCECTRCR
Scene source0.97 / 0.980.97 / 0.980.93 / 0.960.93 / 0.950.97 / 0.980.99 / 0.990.99 / 1.000.92 / 0.950.97 / 0.970.90 / 0.940.97 / 0.980.97 / 0.98
Viewpoint0.96 / 0.980.97 / 0.980.95 / 0.960.91 / 0.930.92 / 0.950.97 / 0.980.99 / 1.000.97 / 0.970.95 / 0.960.96 / 0.970.83 / 0.870.92 / 0.96
Domain0.90 / 0.980.94 / 0.970.59 / 0.880.92 / 0.940.89 / 0.950.94 / 0.980.95 / 0.990.90 / 0.950.90 / 0.940.73 / 0.900.94 / 0.950.90 / 0.97

The subset analyses show broadly consistent model rankings across the tested stratifications. GMS/LMS are particularly stable, with minimum correlations of 0.94–0.99 and median correlations of 0.98–1.00. Greater variation occurs for TR across viewpoint subsets (minimum/median ρ = 0.83/0.87) and for JP and EC across domains (0.59/0.88 and 0.73/0.90, respectively). These results identify the metrics most sensitive to particular subsets while supporting the stability of the revisit-based model comparisons.

(2) Sensitivity to category proportions.

To directly test the effect of subset proportions, we additionally evaluate uniform weighting across source categories, viewpoints, and domains. Rankings under uniform and original weighting are compared using the same eligible case population and model set.

Weighting schemeVQDDJPSCBCGMSLMSPCCCECTRCR
Uniform scene source1.001.001.001.001.001.001.001.001.001.001.000.99
Uniform viewpoint0.990.990.990.991.001.000.991.001.001.001.001.00
Uniform domain1.000.990.970.991.000.990.991.001.000.970.991.00

Uniform reweighting across scene sources, viewpoints, and domains yields ranking correlations of 0.97–1.00 with the original weighting. This shows that the model comparisons remain stable under the tested changes in category proportions. See Appendix C.6, Tables 7–8 for the subset and uniform-reweighting analyses, respectively.

AuthorsOpenReview ↗

Response to Reviewer 1oF7 — Part 3/6

Q3: Separating World Memory from Camera-Pose and Viewpoint Errors

R:

GMS/LMS measure visual retention at prescribed revisit checkpoints, so their values reflect both the camera’s return accuracy and the consistency of the revisited content. We interpret them as joint measures of navigation and visual retention and use ViPE-based pose conditioning to examine how return-pose differences affect model comparisons.

(1) Return-pose conditioning and sample eligibility.

This analysis starts from six models evaluated on a subset of 100 revisit tasks. We filter cases under different rotation and translation tolerances and recompute GMS_cond/LMS_cond. A case contributes to the conditional scores only if every revisit observation satisfies both pose tolerances relative to its initial anchor observation; cases that fail these checks or lack usable poses are excluded. Observation frames, visual features, and aggregation procedures remain unchanged.

Filtering is performed separately for each model, so the retained cases and models meeting the coverage requirements can vary across tolerance settings. Within each row, the original and conditional rankings are compared over the same eligible models.

The translation tolerance is the maximum allowed Euclidean distance between the estimated camera positions at a revisit observation and its corresponding initial anchor observation. Distances are expressed in ViPE reconstruction units; smaller thresholds require closer positional agreement. Both rotation and translation errors must satisfy their respective tolerances.

(2) Ranking agreement across pose tolerances.

Rotation toleranceTranslation toleranceρ (GMS)τ-b (GMS)ρ (LMS)τ-b (LMS)
45°10.9000.8000.9000.800
45°21.0001.0001.0001.000
45°51.0001.0001.0001.000
40°10.9000.8000.9000.800
40°20.7710.6001.0001.000
40°51.0001.0000.9430.867
35°20.9430.8671.0001.000
35°51.0001.0000.9430.867
30°20.9000.8001.0001.000
30°50.9430.8670.9430.867
25°20.9000.8001.0001.000
25°51.0001.0000.9000.800

Across the twelve tolerance settings, Spearman correlations between the original and conditional rankings range from 0.771 to 1.000 for GMS and from 0.900 to 1.000 for LMS. The principal model comparisons thus remain broadly consistent when evaluation is restricted to cases meeting the tested pose criteria, supporting the robustness of the reported comparisons of revisit performance.

The original and conditional scores provide complementary perspectives: GMS/LMS assess navigation and visual retention jointly, while GMS_cond/LMS_cond assess visual retention under the specified pose conditions. We have clarified this interpretation and added the case-level return-pose conditioning rules and ranking results in Appendix C.3, Table 5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair and reviewers for their careful reading and constructive feedback. We appreciate their recognition of the importance of long-horizon evaluation, the value of sustained-control and structured-revisit tasks, and the insights provided by our model capability analyses. We have conducted supplementary experiments and clarified the evaluation protocols to address the main concerns, summarized below.

(1) Interpretation and validation of revisit metrics. (1oF7 Q3; BnyQ Q4; knyz Q1) We agree that reliable navigation provides a foundation for evaluating revisit memory and clarify that GMS/LMS jointly measure navigation and visual retention. To examine how navigation errors affect model comparisons, we supplement the evaluation with return-pose conditioning and a departure-and-return audit on identical effective observation pairs across four models. Under all three tested tolerance settings, the audited GMS/LMS rankings agree with both the full common baseline and the retained-task baseline (Spearman ρ=1.000), supporting the stability of the reported comparisons under these revisit conditions.

(2) Control protocols and long-horizon performance. (BnyQ Q1–Q2; knyz Q2; 1oF7 Q4, Q6) We clarify that TR/CR evaluate the camera’s response to navigation commands in both first- and third-person settings, and document per-model input mappings, control magnitudes, timing conversions, and interface adaptations in Table 10. Rankings on the commonly supported yaw subset closely agree with those over all rotation controls (ρ=0.955). Tightening the TR tolerance to 30° rejects partial execution such as executing only W for W+D; all eleven models still achieve lower TR on composite inputs than on single-key inputs. Across the 10°–60° threshold sweep, correlations with the default ranking remain at least 0.955. Across ten models, early-stage TR/CR also correlates positively with the rate of successfully executing all five final relevant instructions, linking directional responsiveness to long-horizon instruction following.

(3) Checklist construction and real-video reference evaluation. (1oF7 Q1; BnyQ Q3) The checklists assess observable physical and causal constraints grounded in each scene and task, allowing multiple plausible evolutions that satisfy those constraints. Candidate items from three LLMs undergo human review, with approximately 78% retained. We additionally evaluate 80 real videos from DL3DV-10K and Ego-Exo4D: both sets exceed the highest generated-video scores for temporal continuity and background consistency. A supplementary assessment covering 329 scene-specific PC/CC/EC checklist questions identifies no confirmed violations in the sampled evidence, providing real-video reference values for these metrics.

(4) Human agreement and evaluation robustness. (1oF7 Q2, Q5–Q6) On 40 stratified tasks covering ten open-source models and three independent human annotators, automatic metrics achieve a mean model-ranking SRCC of 0.945 across twelve metrics. Five VLM judge configurations from three model families also show strong agreement with human checklist rankings. We further examine scene-source, viewpoint, and domain subsets and apply uniform reweighting to test composition sensitivity. These analyses identify variation across subsets, while reweighted rankings remain highly correlated with the original rankings (ρ=0.97–1.00), supporting robustness to the tested changes in category proportions.

The revised manuscript includes additional evaluation details and experimental analyses. We provide detailed responses to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.