- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LongWM-Bench. It benchmarks 300 tasks with longer target rollout durations of 60 to 180 seconds in order to evaluate sustained control, delays, and revisits in World Model explorations (With interactions). This benchmark combines visual and motion metrics with scene-specific judgements to assess controllability, perceptual quality, temporal consistency, revisit memory, and physical and causal validity. 12 major models were experimented on with a supplementary human-agreement study to mainly show that appearance consistency and accurate control do not necessarily imply scene retention or coherent world evolution. Main contributions are the extended evaluation horizons and systematic revisit task design.
Critical strengths
-
The task designs for sustained interaction in this benchmark are outstanding. This work extends the evaluation to target duration of 60-180 seconds (targeted on delays, revisits, etc.), which is significantly impactful for future benchmark design in this area. Because it addresses a gap in evaluating world persistence during continued exploration. Test cases demonstrated in this work also are meaningful for developing more reliable interactive WM.
-
Very useful empirical characterizations. The reported capability profiles give evidence of differences in different models in a way that reveals important mechanisms, this will help identify priorities for further investigations.
Critical weaknesses
- Some core metrics are somewhat unclear. This could undermine the quality of this benchmark. To my understanding, revisit similarity can be affected by navigation errors or limited movement. Although the appendix restricts evaluation to actual revisits, how successful departures and returns are verified, and how exclusions affect cross-model comparisons. Moreover, the 60° translation tolerance accepts a 45° error, so it means that executing only W when W+D is requested can count as correct. These issues weaken the interpretation of the reported memory and composite-control scores.
Questions for authors
-
How do you verify that a model has actually left and returned to an observation anchor, and how are unsuccessful or unverifiable revisits handled? If there are clear verifications and they hold true on the identical effective samples then my proposed weakness will be undermined.
-
Does the translation metric impose any component-wise requirements beyond the 60° direction tolerance? Under the stated rule, executing only W when W+D is requested would count as correct. If no additional constraints are applied, do the composite-control findings hold when such partial execution is treated as failure?
5 comments
Response to Reviewer knyz — Part 2/3
(3) Model comparisons on jointly verified observation pairs.
| Rotation tolerance | Translation tolerance | Metric | ρ: full common baseline | ρ: retained-task baseline |
|---|---|---|---|---|
| 45° | 10 | GMS′ | 1.000 | 1.000 |
| 45° | 10 | LMS′ | 1.000 | 1.000 |
| 60° | 10 | GMS′ | 1.000 | 1.000 |
| 60° | 10 | LMS′ | 1.000 | 1.000 |
| 60° | 15 | GMS′ | 1.000 | 1.000 |
| 60° | 15 | LMS′ | 1.000 | 1.000 |
Here, ρ is Spearman’s rank correlation across the four models.
Under the 60°/15-unit setting, we obtain 82 shared GMS observation pairs across 51 revisit tasks, with all four models satisfying the stated departure and approximate-return criteria for every retained pair. Recomputing scores preserves the four-model rankings of both baselines, with Spearman ρ=1.000. LMS retains 25 shared observation pairs from 21 tasks, also with a ranking correlation of 1.000. Across all three tolerance settings, both metrics preserve the rankings of both baselines.
LMS requires at least three observations of the same anchor to form a pair of successive revisits, resulting in fewer available and retained pairs than GMS. These results show that the original four-model ordering is preserved on identical effective observation pairs that jointly pass the departure-and-return audit.
(4) Supplementary sensitivity analysis of return-pose tolerances.
We additionally conduct a case-level return-pose sensitivity analysis on a subset of six models and 100 revisit tasks. A case contributes to GMS_cond/LMS_cond only when every revisit satisfies both rotation and translation tolerances relative to its corresponding initial anchor observation. Cases failing these checks or lacking usable poses are excluded; observation frames, visual features, and aggregation procedures remain unchanged.
This analysis filters cases separately for each model, so the retained cases and eligible models can vary across settings. Each row compares the original and conditional rankings over the same eligible models. It examines return-threshold sensitivity, while the shared-pair audit above directly tests model comparisons on identical effective samples.
| Rotation tolerance | Translation tolerance | ρ (GMS) | τ-b (GMS) | ρ (LMS) | τ-b (LMS) |
|---|---|---|---|---|---|
| 45° | 1 | 0.900 | 0.800 | 0.900 | 0.800 |
| 45° | 2 | 1.000 | 1.000 | 1.000 | 1.000 |
| 45° | 5 | 1.000 | 1.000 | 1.000 | 1.000 |
| 40° | 1 | 0.900 | 0.800 | 0.900 | 0.800 |
| 40° | 2 | 0.771 | 0.600 | 1.000 | 1.000 |
| 40° | 5 | 1.000 | 1.000 | 0.943 | 0.867 |
| 35° | 2 | 0.943 | 0.867 | 1.000 | 1.000 |
| 35° | 5 | 1.000 | 1.000 | 0.943 | 0.867 |
| 30° | 2 | 0.900 | 0.800 | 1.000 | 1.000 |
| 30° | 5 | 0.943 | 0.867 | 0.943 | 0.867 |
| 25° | 2 | 0.900 | 0.800 | 1.000 | 1.000 |
| 25° | 5 | 1.000 | 1.000 | 0.900 | 0.800 |
Translation tolerances are positional-distance thresholds in ViPE reconstruction units. Across the twelve settings, Spearman correlations with the original rankings range from 0.771 to 1.000 for GMS_cond and 0.900 to 1.000 for LMS_cond, indicating high ranking agreement under the tested return-pose tolerances. The complete threshold results are reported in Appendix C.3, Table 5.
We have added the shared-pair departure-and-return audit to Appendix C.3, specifying the verification criteria, the exclusion of unsuccessful or unverifiable pairs across all four models, the resulting common-sample coverage, and the ranking comparisons. The supplementary case-level return-pose sensitivity results are reported in Table 5.
Response to Reviewer knyz — Part 3/3
Q2: Analysis of the 60° Tolerance
R:
Thank you for this suggestion. The current TR criterion applies no additional component-wise requirements, so executing only W for a W+D command does count as correct. We therefore tighten the directional tolerance to 30°, which rejects this partial execution, and re-examine composite-control performance. Across eleven models, TR remains lower for composite inputs than for single-key inputs. We detail the criterion and supplementary results below.
(1) The current TR criterion and partial execution.
TR sums the commanded translation vectors to obtain a target direction and checks whether the observed net displacement lies within a 60° tolerance of that direction. For a W+D command, executing only W produces a 45° directional deviation and therefore passes the current criterion.
We tighten the tolerance to 30°, so that this single-axis partial execution no longer passes, while keeping the displacement-direction calculation unchanged.
(2) Composite-control performance under the 30° criterion.
We recompute TR across eleven models, excluding Genie 3. For each model and input group, the 60° and 30° evaluations use the same evaluable action segments. We compare single-key and composite-key inputs and additionally report the subset of commands containing two translation directions.
| Model | Single-key TR, 60° | Composite-key TR, 60° | Single-key TR, 30° | Composite-key TR, 30° | Two-axis translation TR, 30° |
|---|---|---|---|---|---|
| HappyOyster | 82.3 | 78.0 | 71.1 | 53.5 | 54.8 |
| Lyra 2.0 | 73.8 | 67.3 | 64.6 | 48.4 | 48.1 |
| Echo-WM | 80.8 | 78.0 | 72.0 | 62.9 | 55.7 |
| LingBot-World-v2 | 86.7 | 82.1 | 81.2 | 57.7 | 62.0 |
| Matrix-Game 3.0 | 84.9 | 74.8 | 75.8 | 52.5 | 51.6 |
| Matrix-Game 2.0 | 81.3 | 78.6 | 72.1 | 59.2 | 59.7 |
| DreamX-World | 95.2 | 93.3 | 90.2 | 81.3 | 81.6 |
| Alaya-EVOKE | 64.2 | 54.3 | 56.3 | 35.5 | 22.4 |
| ABot-World | 63.5 | 52.8 | 48.3 | 26.3 | 42.0 |
| Yume-1.5 | 69.5 | 48.1 | 61.6 | 27.6 | 14.4 |
| Cosmos 3 | 29.0 | 35.9 | 18.2 | 17.3 | 14.7 |
Scores are percentages, with equal weighting across action segments within each input group. Single-key inputs contain only one translation key; composite-key inputs contain a translation key together with additional translation or rotation keys. Two-axis translation inputs contain two translation directions and may also include rotation.
Under the 30° criterion, all eleven models have lower TR for composite-key inputs than for single-key inputs, and the same pattern holds for the two-axis translation subset. For example, HappyOyster achieves 71.1%, 53.5%, and 54.8% on single-key, composite-key, and two-axis translation inputs, respectively; the corresponding scores for Echo-WM are 72.0%, 62.9%, and 55.7%.
These results show that the main finding of lower translation accuracy under composite inputs persists when the single-axis partial execution described by the reviewer is counted as failure.
(3) Model-ranking stability across directional tolerances.
We additionally sweep the tolerance from 60° to 10° in 5° steps, keeping each model’s evaluable action segments fixed across thresholds. We compute TR within each task and then average equally across tasks to obtain overall model scores. The table below compares the ranking at each threshold with the default 60° ranking.
| Ranking correlation | 60° | 55° | 50° | 45° | 40° | 35° | 30° | 25° | 20° | 15° | 10° |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Spearman ρ | 1.000 | 0.982 | 0.973 | 0.973 | 0.973 | 0.955 | 0.973 | 0.955 | 0.973 | 0.955 | 0.955 |
| Kendall τ-b | 1.000 | 0.927 | 0.891 | 0.891 | 0.891 | 0.855 | 0.891 | 0.855 | 0.891 | 0.855 | 0.855 |
The Spearman correlation between the 30° and default 60° rankings is 0.973; across the entire sweep, the minimum correlation is 0.955 and the maximum rank displacement is two positions. Thus, stricter directional criteria preserve the main composite-control finding and maintain high agreement in the overall model rankings.
The TR criterion is defined in Appendix D.1.5. The partial-execution issue and stricter-threshold analyses are included in Appendix C.4, Figures 12–13.
Response to Reviewer knyz — Part 1/3
We thank the reviewer for the thoughtful assessment and constructive suggestions. We appreciate the recognition of our sustained-interaction task design and its value for evaluating world persistence during continued exploration, as well as the usefulness of the model capability profiles. We address the two metric concerns below through a departure-and-return audit on identical effective observation pairs and a stricter-threshold analysis of composite controls.
Q1: Verifying Departures and Returns in Revisits
R:
Thank you for this suggestion. We believe that memory evaluation in revisit tasks should build on reliable navigation control. A model needs to leave and return to a previously observed location so that content retention can be assessed from the corresponding viewpoint. We therefore interpret GMS/LMS as joint measures of navigation and visual retention, as now clarified in Appendix C.3. To address your questions about verification and sample consistency, we supplement the evaluation with a ViPE-based departure-and-return audit and recompute scores on identical observation pairs that pass the audit across all four models.
(1) Departure and return verification criteria.
Our tasks use symmetric out-and-back paths, closed loops, and nested revisits to specify the anchor associated with each observation. For example, in A→B→C→B→A, the later observations of B and A correspond to earlier observations of the same anchors. We use these correspondences to pair observations and further check the actual motion and return poses in the generated trajectories.
This analysis includes Lyra 2.0, Echo-WM, LingBot-World-v2, and Matrix-Game 2.0, each with cached ViPE trajectories for all 150 revisit tasks. GMS compares the initial observation with subsequent revisits, while LMS compares successive revisits to the same anchor. For each observation window, we select the valid pose nearest the actual scoring-frame timestamp and apply the following criteria:
The departure criterion checks sustained pose changes, while the return criterion checks approximate anchor alignment within the observation windows. Translation thresholds are expressed in ViPE reconstruction units.
(2) Handling unsuccessful or unverifiable pairs and constructing identical effective samples.
If any model fails the departure or return check, or lacks usable poses, the observation pair is excluded for all four models in this supplementary analysis. For each metric and tolerance setting, we retain the intersection of observation pairs that pass the audit across all four models. Every comparison therefore uses exactly the same eligible observation pairs.
This filtering applies to the supplementary scores GMS′/LMS′; the original GMS/LMS remain joint measures computed at prescribed revisit checkpoints. When recomputing scores, we preserve the original visual similarities, elapsed-time weights, weighted aggregation within each anchor, and equal weighting across anchors and tasks.
To examine the effects of observation-pair filtering and changes in task coverage on model rankings, we compare against two baselines: