acceptodds
Under review as a conference paper at ICLR 2027

WorldMemBench:A Comprehensive Benchmark for Memory of Video World Models

Abstract

As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requires both preserving previously observed content and ensuring coherent world evolution over time. World memory is a critical capability to evaluate alongside visual fidelity and video duration. We introduce WorldMemBench, a benchmark for systematically evaluating world memory in video world models. Across 260 test cases, it assesses static memory through the retention of human identity and appearance, object appearance and geometry, and environmental content and geometry, and dynamic memory through event progression and state persistence. Using controlled camera departures and multi-revisit trajectories, the benchmark evaluates static content against fixed initial references and dynamic events against expected state transitions. It combines appearance and geometry metrics with event-specific assessments, and reports video quality and camera-following accuracy as complementary diagnostics. Evaluation of nine representative models shows that strong static retention does not imply reliable dynamic state evolution, while repeated revisits reveal memory degradation beyond the first return. Human studies support the alignment between our automatic evaluations and human judgments. WorldMemBench provides a systematic testbed for diagnosing memory failures and guiding the development of persistent, temporally coherent video worlds.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Video World Models, World Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_ub5YOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WorldMemBench, a benchmark for evaluating world memory in video world models. It divides memory into two types: static memory, which measures how well models preserve the identity, appearance, and geometry of people, objects, and environments, and dynamic memory, which evaluates whether events and states evolve consistently when out of view. The benchmark includes 260 cases across 23 categories and uses controlled departure-and-revisit camera trajectories to test memory consistency. Static memory is evaluated using visual similarity and geometric metrics, while dynamic memory is assessed through a VLM-based checklist.. Evaluations of nine models show that strong static memory does not guarantee reliable dynamic memory, and performance degrades with repeated revisits and changes in rotation axes. Human studies support the evaluation metrics, while training-free experiments on LingBot World v2 suggest that combining recent context with FoV-relevant history offers a good trade-off between memory and efficiency.

Critical strengths
  1. The paper clearly separates world memory into static memory (preserving content) and dynamic memory (tracking changes outside the camera view). This addresses an important gap in existing metrics like FID, FVD, and VBench.
  2. The evaluation is well designed. Repeated revisits help reveal how memory degrades over time, while comparing LRLR and LRUD helps isolate the effect of switching axes. The static metrics cover both appearance and geometry, and the event-gated checklist evaluates dynamic changes more meaningfully than pixel similarity.
  3. The paper evaluates nine recent models under the same protocol and also reports video quality and camera-following performance. The results reveal interesting differences between models, including memory degradation over time, sensitivity to axis changes, and the fact that better video quality does not necessarily mean better memory.
  4. The human evaluation supports the automatic metrics, while additional experiments on revisit accuracy and dynamic scoring help verify the robustness of the findings. The context-access experiments (FullKV, Sliding Windows, and FoV) also show that the benchmark can help identify model weaknesses and guide future model improvements, rather than simply ranking models.
Critical weaknesses
  1. The dynamic memory score seems to mix up action execution and memory. If an event fails to start, all eight questions get zero, even though this does not necessarily mean the model failed to remember it. For example, HY-WorldPlay scores 0.928 when the event starts, but its event-start rate is only 31.7%, bringing the overall score down to 0.294. The paper uses this result to argue that strong static memory does not imply strong dynamic memory, but it may actually say more about the model's difficulty in executing actions than its memory.

  2. Several factors unrelated to memory can also affect the scores. If a target is not detected during a revisit, it gets zero, but this could be because the target disappeared, the tracker failed, or the camera did not return to the right position. Depth Spearman and Object Geometry also depend heavily on viewpoint alignment, so small camera errors can lower the scores. In the cross-axis experiments, Rot. RPE increases noticeably, suggesting that the camera itself is less accurate.

  3. The dynamic memory evaluation only checks what the scene looks like when the camera returns. This makes it hard to tell whether the model actually keeps track of events while they are off-screen or simply generates a plausible final state when the camera looks back. These are quite different abilities, especially when evaluating world models, but the current benchmark cannot really distinguish between them.

Questions for authors

I am leaning toward acceptance, but I would like the authors to clarify the concerns raised in the above weaknesses.

4 comments
AuthorsOpenReview ↗

(1/2)We thank Reviewer ub5Y for the effort in reviewing our work and providing constructive comments. We address each concern or question below.

Dear Reviewer ub5Y,

Thank you for your positive inclination toward our paper and for recognizing our static/dynamic memory taxonomy, multi-revisit and cross-axis protocols. Below, we clarify each of your three concerns.

W1. Event occurrence and continuation in Dynamic Memory.

  1. Dynamic memory requires both dynamics and memory. The dynamics component requires the intended event to occur, while the memory component requires the model to retain what has happened and preserve its consequences across camera departures and returns (L352–364). Our Overall score evaluates these capabilities jointly. If chopping never begins, preserving untouched vegetables upon return demonstrates static retention but does not fulfill the dynamic-memory task. Evaluating models only on their respective successful-start subsets introduces selection bias by comparing different sets of cases: a model that predominantly generates static scenes and initiates actions in only a few cases could achieve a high conditional score despite low overall task completion. Assigning zero when an event fails to start keeps evaluation on the same complete set of 60 tasks.
  2. Why “strong static memory does not imply strong dynamic memory” holds: HY-WorldPlay’s conditional score of 0.928 demonstrates strong performance on the 19 cases where events start. Its 31.7% start rate, however, shows that this capability is not reliably realized across the complete set of 60 dynamic tasks, yielding an Overall of 0.294. These results are consistent with our conclusion: strong static memory does not imply strong dynamic memory, because preserving established content alone does not ensure that the model can generate and maintain an evolving world. We report initiation rates and conditional scores separately to identify the source of this gap, while Overall measures the combined capability (Table 5, L750–767). The event-start rate is also included in the revised main results (Table 2, L328 and L341, highlighted in red).

W2. Target observability and camera accuracy in static-memory evaluation.

  1. Comparing revisits with valid target observations and camera returns. For Human/Object, we retain videos only when every initially evaluated target has departure evidence and valid observations at every compared revisit. For Environment, DA3-estimated orientations must indicate a departure of at least 30° from the initial direction and a return orientation error of at most 30°. Each comparison uses fixed cases across the compared revisits and trajectories within each model, with effects averaged over cases and then equally over models (L1281–1292).

  2. Memory degradation persists across repeated revisits. All seven static-memory metrics retain a positive mean decrease from the first to the third revisit after filtering. For this analysis, all three returns under LRLRLR must satisfy the criteria above. The table compares the unfiltered and filtered decreases, R1 − R3 (Table 12, L1315–1327).

    MetricRetained model–case pairs / modelsUnfiltered decreaseFiltered decrease
    Human Identity125 / 80.2370.110
    Human Appearance93 / 70.1620.058
    Object Appearance155 / 90.2290.158
    Object Geometry155 / 90.2050.101
    Environment Visible431 / 90.1610.152
    Environment Depth431 / 90.1380.135
    Environment Reconstruction431 / 90.1960.139
  3. The cross-axis penalty persists after controlling return accuracy. For this comparison, both returns in both LRLR and LRUD must satisfy the same criteria. We measure the additional degradation from switching axes as

    Δaxis=(R1−R2)LRUD−(R1−R2)LRLR.\Delta_{\mathrm{axis}} = (R_1-R_2)_{\mathrm{LRUD}} - (R_1-R_2)_{\mathrm{LRLR}}.

    All seven metrics retain a positive mean cross-axis penalty after filtering, showing that switching from yaw to pitch remains harder on average among qualified returns (Table 12, L1328–1334).

    MetricRetained model–case pairs / modelsUnfiltered Δaxis\Delta_{\mathrm{axis}}Filtered Δaxis\Delta_{\mathrm{axis}}
    Human Identity135 / 80.0230.034
    Human Appearance109 / 80.0560.050
    Object Appearance141 / 80.1170.023
    Object Geometry141 / 80.1150.051
    Environment Visible281 / 90.1520.128
    Environment Depth281 / 90.1070.076
    Environment Reconstruction281 / 90.0680.056

    Both trends also persist under tighter return tolerances. With a 5° return-error threshold and the same 30° departure requirement, the mean multi-revisit decreases remain positive for Environment Visible, Depth, and Reconstruction (0.092, 0.057, and 0.116), as do the cross-axis penalties (0.094, 0.158, and 0.050) (Table 13, L1351–1359).

AuthorsOpenReview ↗

(2/2)We thank Reviewer ub5Y for the effort in reviewing our work and providing constructive comments. We address each concern or question below.

W3. Off-screen state consistency and internal mechanisms.

  1. We evaluate state changes across the visible sequence. The judge receives chronologically sampled frames spanning the entire rollout, together with the planned camera phases (L734–741). Return Progress compares the returned state with the pre-departure baseline; Persistence/Reset checks whether its consequences persist without reverting; Final State assesses the expected outcome; and Causal/Physics examines the visible ordering of events. The questions therefore use evidence before departure, upon return, and throughout subsequent observations (L788–797).
  2. Our contribution is a behavioral benchmark for dynamic memory. It evaluates whether generated worlds preserve event progression and consequences across interrupted observation. Investigating how these mechanisms support dynamic memory is a direction for our future work.

Sincerely,

WorldMemBench Authors

Thank you for the thoughtful and detailed response. Your explanations addressed most of my concerns. I will keep my positive score.

Dear Reviewer ub5Y,

We deeply appreciate your encouraging feedback and your support for our work! We are delighted to hear that our explanations and analyses addressed your concerns.

Best regards,

WorldMemBench Authors

Reviewer_AyxLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduced WorldMemBench, a benchmark for video world models with multiple cases on human object scene and environment, on both static and dynamic scenes. It contains the benchmark that go over the rotation and then come back to the original start view several times, and it scores with different aspect like ArcFace, dino, iou and depth. Dynamic memory is scored by Qwen model on different questions. Also authors frind that scores drop from the first to the third revisit.

Critical strengths
  1. Multi revisits are a novel term that previous benchmarks are not taking into count. It is essential and significant for the world video models to evaluate the consistency.
  2. Nine recent models with pose, keyboard and text control are evaluated, with detailed setting provided.
  3. This benchmark helps in diagnosing design choices. For LingBot World v2, the FoV strategy outperforms the sliding window approach across all eleven memory metrics, given the same budget.
Critical weaknesses
  1. Length is limited to 20-35s, others like WorldRoamBench is more diverse and longer spread on the time length.
  2. The gap between static and dynamic memory lays primarily from the event start gate, HY-WorldPlay achieves the highest conditional dynamic score but its event triggering rate is pretty low, resulting in a gated score of 0.294. Because this gate is evaluated before the camera moves away, it measures whether the model has introduced dynamic elements into the scene, rather than its actual memory capacity. The event triggering rate should be reported as a separate metric.
  3. The same Qwen3.8-27B both writes and answers the dynamic questions. It may biased.
Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

(1/2)We thank Reviewer AyxL for the effort in reviewing our work and providing constructive comments. We address each concern or question below.

Dear Reviewer AyxL,

Thank you for your positive assessment and for recognizing the novelty of our multi-revisit protocol, the breadth of our 9-model evaluation, and the diagnostic value of our context mechanism analysis. We address your three concerns below with extended-horizon experiments, reporting improvements, and cross-judge validations.

W1. Evaluation duration and longer-horizon coverage.

  1. Our benchmark evaluates memory across repeated interactions. WorldMemBench uses repeated and cross-axis revisits to assess whether previously observed content remains consistent across successive camera departures and returns. Comparing each revisit against a fixed initial reference reveals cumulative memory degradation beyond the first return.

  2. Longer rollouts introduce additional generation degradation and memory costs. In our extended experiments, we observe visual degradation in later segments and substantial GPU-memory requirements. For example, approximately 61-second LingBot FullKV runs reach sampled device-memory peaks of about 140 GiB on the busiest GPU in an eight-GPU configuration. The 20–35-second default therefore balances repeated memory testing with practical evaluation costs.

  3. Following your suggestion, we extended the evaluation to 40–63 seconds. We evaluated all nine models on extended LRLRLR trajectories of approximately 1,000 frames at their native frame rates, preserving all three revisits and the original rotation angles. In the reported longer-horizon results, HY-WorldPlay leads all seven static-memory metrics, and Evoke remains second across all four Human/Object metrics.

    ModelAesthetic ↑Imaging ↑Human ID ↑Human App. ↑Object App. ↑Object Geometry ↑Env. Visible ↑Env. Depth ↑Env. Recon. ↑Rotation RPE ↓
    LingBot-World0.53440.67070.37760.63790.21180.20470.66390.77280.29790.8684
    LingBot-v20.54750.67200.47730.53160.39480.39020.58550.50070.26350.5246
    HY-WorldPlay0.51940.65490.79690.85950.90540.87850.90600.89490.64070.5310
    Matrix-Game-30.49140.69240.27100.27790.51520.41620.67260.43700.31590.7248
    Matrix-Game-3.50.44700.60690.00000.02750.21260.21270.49540.54020.29972.8609
    SANA-WM0.45750.61500.12110.18640.09930.09800.32140.60320.16800.7258
    DreamX-World0.50670.65160.14780.54160.68190.68020.45720.19590.11411.1269
    YUME0.53170.67060.26040.35300.25060.23410.54340.66370.29260.6507
    Evoke0.54130.69070.52030.79970.77350.73510.74570.73930.37450.7788

    To further broaden the evaluation in our paper, we have added the longer-horizon findings to the main text and the experimental setup, results, and analysis to Appendix G.3 (L461–463; Table 15, L1406–1435, highlighted in red).

AuthorsOpenReview ↗

(2/2)We thank Reviewer AyxL for the effort in reviewing our work and providing constructive comments. We address each concern or question below.

W2. Event initiation and Dynamic Memory scoring.

  1. The event-start rate is now reported separately in the main results. Following your suggestion, we have added the event-start rate alongside Dynamic Overall in Table 2 (L328 and L341, highlighted in red). We also retain the decomposition into event-start rate, conditional score, and Overall (Table 5, L745–767), where Overall = event-start rate × conditional score. For HY-WorldPlay, its 31.7% start rate and 0.928 conditional score yield an Overall of 0.294. This helps readers distinguish the contributions of event initiation and subsequent checklist performance to the Overall score.
  2. Dynamic memory requires both dynamics and memory. The dynamics component requires the intended event to occur, while the memory component requires the model to retain what has happened and preserve its consequences across camera departures and returns. The event-start gate checks the first component; the subsequent checklist evaluates event progression and state consistency. Evaluating models only on their respective successful-start subsets introduces selection bias by comparing different sets of cases: a model that predominantly generates static scenes and initiates actions in only a few cases could achieve a high conditional score despite low overall task completion. Assigning zero when an event fails to start keeps the evaluation on the same complete set of 60 tasks.

W3. Potential bias from using the same model to generate and answer questions.

  1. Decoupled question generation and blind visual judging: In our protocol, Qwen3.8-27B is only used offline to draft candidate binary checklist questions from text prompts, from which we curate 8 factual questions per case and freeze the reference answers prior to evaluation. During scoring, the VLM judge only inspects the generated video frames to answer specific visual yes/no questions, without access to the generation prompt or reference answers (L354–359, L734–741, and L776–782).

  2. Cross-judge validation with Qwen3.6-27B: We re-evaluated all 540 videos (4,320 questions) by replacing the judge with Qwen3.6-27B under the exact same sampling and scoring protocol:

    JudgeDreamXMatrix-Game-3.5SANAYUMELingBot-WorldMatrix-Game-3EvokeLingBot-World-v2HY-WorldPlay
    Qwen3.80.56250.23960.41460.77500.80830.04580.70630.69580.2938
    Qwen3.60.58540.25210.48750.83130.85630.06460.68960.71040.2854

    Replacing the judge yields a near-perfect Overall Spearman rank correlation of 0.983 with the original evaluations, and subtask rank correlations of 0.979 (Return Progress), 1.000 (Persistence/Reset), 0.983 (Final State), and 0.996 (Causal/Physics). The mean absolute score difference is only 0.030, with 7 out of 9 model ranks remaining completely unchanged.

Sincerely,

WorldMemBench Authors

Thanks, the author's response solved my major concerns, I'd like to keep my rating.

Dear Reviewer AyxL,

Thank you very much for taking the time to read our response and for maintaining your positive rating! We are glad that our revisions resolved your concerns, and we really appreciate your insightful suggestions, which have helped us make the work much more rigorous.

Best regards,

WorldMemBench Authors

Reviewer_VhwBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper centers on whether a generated scene remains consistent with earlier observations after the camera moves away and returns. Prescribed camera paths are used to examine continuity in human identity, object properties, and environmental details, including changes that should persist over time. Video quality and compliance with camera instructions are assessed alongside these tests. The experiments compare models, vary the historical context available during generation, and examine the effects of returning to the same scene multiple times.

Critical strengths
  1. Testing several returns reveals weaknesses that a single return may miss.

    The extended sequences in §5.3 (p. 8) provide a useful test of longer-term consistency. By the third return, the average scores for object appearance and human identity have decreased by 0.229 and 0.217, respectively, relative to the first return. A convincing reconstruction on the initial revisit therefore offers limited evidence of sustained retention. Repeated returns make the evaluation more informative by exposing inconsistencies that shorter sequences may overlook.

  2. The context comparison clarifies the benefits and limitations of selective history.

    FoV-based selection produces the strongest results on eight of the eleven metrics in Table 3 (p. 9), while achieving a 1.23× speedup over the full-history configuration. These findings support its practical value in the evaluated setting. However, full history performs better on the remaining three metrics, all of which concern static information. The comparison thus identifies both the gains available from context selection and the aspects of scene consistency that still benefit from a more complete history.

Critical weaknesses
  1. The dynamic evaluation does not fully isolate retention from action execution.

    Poor performance can arise at different stages: the requested activity may never begin, or its effects may disappear after it has occurred. In the vegetable-chopping example, failing to show any chopping and restoring the vegetables to their earlier state after chopping represent distinct errors, although both reduce the overall score. The analyses in §4.5 and Appendix D.2 partly address this ambiguity. However, restricting the comparison to successful event initiation can leave each model evaluated on a different subset of examples. The extent to which the reported performance gaps reflect memory, rather than differences in initiating the requested actions, therefore remains uncertain.

  2. Human validation provides uneven support across evaluation dimensions.

    Agreement with human judgments varies considerably across the proposed measures. Table 6 reports model-ranking correlations of 0.301 for environment depth and 0.606 for object geometry, while the sample-level correlation for causality/physics is 0.401 in Table 7. These values warrant caution when interpreting comparisons on the corresponding dimensions, particularly environment depth and causality/physics. Further validation would help establish whether differences in these scores reliably correspond to differences perceived by human evaluators.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

We thank Reviewer VhwB for the effort in reviewing our work and providing constructive comments. We address each concern or question below.

Dear Reviewer VhwB,

Thank you for raising concerns about separating event initiation from subsequent retention, particularly when successful-start subsets differ across models. We also appreciate your request for stronger evidence that differences in individual metric scores correspond to differences perceived by human evaluators.

W1. Event initiation and state consistency in Dynamic Memory.

  1. Dynamic memory requires both dynamics and memory: The dynamics component requires the intended event to occur, while the memory component requires the model to retain what has happened and preserve its consequences across camera departures and returns (L352–364). Evaluating models only on their respective successful-start subsets introduces selection bias by comparing different sets of cases: a model that predominantly generates static scenes and initiates actions in only a few cases could achieve a high conditional score despite low overall task completion. Conversely, restricting evaluation to cases where all nine models initiate the event would substantially reduce coverage and make the evaluated subset dependent on every model’s initiation success. Assigning zero when an event fails to start keeps the evaluation on the same complete set of 60 tasks. We separately report the event-start rate and the conditional score, with Overall = event-start rate × conditional score (Table 5, L745–767).
  2. Comparing the same events after they start: To address your question of whether performance gaps genuinely reflect memory rather than action initiation, we compared three context policies—FoV, Sliding Windows, and FullKV—using the same LingBot-v2 backbone. We retained the same 50 cases judged to initiate the event under all three policies and recomputed their Dynamic Memory scores. On this matched subset, FoV outperforms Sliding Windows in Dynamic Overall (0.8575 vs. 0.8150). Because all three policies initiate the event on every retained case, this difference reflects subsequent checklist performance rather than differences in event initiation or evaluated cases. We have added this matched-case analysis to Appendix H (Table 16, L1474–1491, highlighted in red).

W2. Human alignment across evaluation dimensions.

Our automatic metrics reliably reflect human preferences at the video level, achieving high pairwise agreement (78.5%–81.8%) with decisive human-majority votes on Environment Depth and Object Geometry. The lower model-level Spearman correlations on these two metrics are driven by large rank shifts in a few models, which arise from distinct perceptual vs. algorithmic sensitivities. For Causal/Physics, we additionally measured agreement on individual questions.

MetricReported SpearmanAgreement with the human majorityMain rank change: automatic → human
Environment Depth0.30181.8% on decisive A/B video comparisonsMatrix-Game-3.5: tied 2nd → 8th
Object Geometry0.60678.5% on decisive A/B video comparisonsLingBot-World: 8th → 3rd
Causal/Physics0.40166.7% on individual checklist judgments—
  1. Environment Depth: The metric measures depth-order consistency at corresponding pixel locations, while human spatial judgments also depend on recognizable scene content. Changes to objects and occlusion boundaries can reduce perceived spatial consistency even when the coarse near–far layout remains similar. Conversely, recognizable content can support a favorable human judgment despite viewpoint shifts that alter pixelwise depth correspondence.

  2. Object Geometry: We include complex cases to broaden coverage, where human observers may overlook subtle changes in shape and proportions, resulting in favorable ratings despite these changes. Our metric compares normalized silhouettes and is sensitive to changes in object contours and aspect ratios. After a revisit, these changes may go unnoticed when the object’s overall appearance and recognizable structure remain largely intact, even though silhouette IoU decreases. Human judgments may therefore remain favorable despite geometric differences detected by the metric. This helps explain why LingBot-World rises from eighth in the automatic ranking to third in the human ranking.

  3. Causal/Physics: With only two questions per video, the scores are too coarse to clearly separate videos, which helps explain the lower Spearman correlation. The automatic judge agrees with the human majority on 66.7% of the questions. For reference, agreement between individual human annotators averages 58.7%. Each video receives one of three scores, {0,0.5,1}\{0,0.5,1\}, so many videos receive the same score. A disagreement on just one answer can therefore change a video’s score and rank substantially, affecting Spearman. We have clarified the scoring and correlation calculation in Appendix F (Eq. 7, L1081–1093 and L1096–1098, highlighted in red).

Sincerely,

WorldMemBench Authors

Thank you for the clarification. The authors’ response has addressed my main concerns, and I will maintain my current rating.

Dear Reviewer VhwB,

We sincerely thank you for your reply and for confirming that our responses have addressed your main concerns! Your constructive feedback has significantly strengthened our paper, and we truly appreciate your support and valuable time throughout the review process.

Best regards,

WorldMemBench Authors

Reviewer_n7ogOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark for evaluating the memory of world models. It assesses world models across four key dimensions: static memory, dynamic memory, general video quality, and camera following. The benchmark consists of 260 test cases, covering four distinct scenarios: human, object, environment, and dynamic.

Compared to existing benchmarks, this work primarily introduces improvements in the evaluation of static and dynamic memory. For static memory, the authors design "multi-turn revisit" and "cross-axis revisit" scenarios to validate the memory retention across multi-turn interactions. For dynamic memory, the authors leverage a VLM to evaluate during state transitions from four aspects: return progress, persistence/reset, final state, and causal/physical consistency.

The authors validate the benchmark on 9 different world models and conduct diagnostic case study to verify different context mechanisms.

Critical strengths

The authors propose a new benchmark for a crucial aspect of world models (memory). This benchmark not only evaluates the models' performance in static memory but also extends the assessment to dynamic memory. Furthermore, the evaluation spans four diverse scenarios, making the benchmark comprehensive.

Critical weaknesses
  1. For static memory, the evaluation is restricted to loop-revisit trajectories. However, for world models, maintaining geometric consistency under open-ended trajectories is more critical and would serve as a more rigorous test of a model's true static memory capabilities.

  2. In the human evaluation for static memory, although the automatic metrics show high overall agreement with human preferences, the model-level Spearman correlations indicate that environment depth, object geometry, and object appearance do not align well with human judgment. Furthermore, in the dynamic memory evaluation, why are the overall Spearman and PLCC scores significantly higher than those of the individual sub-tasks? Regarding the individual tasks, the current automatic metrics do not seem to reliably reflect human preferences.

Questions for authors
  1. While the authors evaluate various world models on the benchmark, there is a lack of detailed analysis of the results. For instance, in the quantitative experiments, the authors do not discuss what kind of architectural designs or mechanisms benefit static memory the most, or what designs are more favorable for dynamic memory. Providing a more granular analysis would make the paper much more insightful for the community.

  2. The "Spherical Reconstruction Score" metric is somewhat difficult to understand. Could the authors provide a more detailed explanation or an illustrative diagram to clarify how it works?

3 comments
AuthorsOpenReview ↗

(1/3)We thank Reviewer n7og for the effort in reviewing our work and providing constructive comments. We address each concern or question below.

Dear Reviewer n7og,

Thank you for your careful review and for recognizing the importance and breadth of our benchmark. We address your concerns below with additional experiments, analyses, and clarifications.

W1. Evaluating static memory beyond loop-revisit trajectories.

  1. Our loop-revisit protocol tests whether previously observed content is retained after leaving the field of view. Returning toward the initial viewpoint reduces perspective changes, enabling comparisons against a fixed initial reference. Repeated and cross-axis revisits then reveal how memory persists across successive interactions (L218–227, L236–245).

  2. Inspired by your suggestion, we also conducted a large-scale expansion evaluating all 9 models across 90 cases (30 each from Human, Object, and Environment, yielding 810 new videos) under cross-view revisit trajectories. These trajectories combine lateral translation and yaw rotation to re-observe targets from displaced positions and different viewing directions before returning to the initial pose.

    The results below show strong rank correlations with the paper’s main results across the six evaluated static-memory metrics (Spearman’s ρ = 0.867–1.000), supporting the robustness of the conclusions drawn from our original evaluation.

    ModelAesthetic ↑Imaging ↑Human ID ↑Human App. ↑Object App. ↑Object Geometry ↑Env. Visible ↑Env. Depth ↑Trans. RPE ↓Rot. RPE ↓
    HY-WorldPlay0.53910.67400.69080.82750.90370.79120.88520.79111.01440.5321
    Evoke0.56930.69220.60410.82890.81870.71150.81600.76451.42880.8907
    LingBot v20.53690.64380.54220.66930.81670.77060.84800.60880.77100.4836
    LingBot v10.55230.66300.52170.59310.69610.69160.83100.74140.85370.6382
    Matrix-Game 30.47480.62640.46570.53140.78280.78070.86490.66281.00260.3450
    DreamX0.55270.68070.46360.78140.78260.61140.75090.45872.54911.0573
    Yume0.53700.72670.25870.53440.74880.60820.76760.70231.33070.8816
    SANA0.47000.55290.15370.31790.53450.58130.71260.68031.42640.6229
    Matrix-Game 3.50.48210.64000.00000.00000.70680.70650.79650.67521.45640.9381
  3. To further broaden the evaluation in our paper, we have integrated the setup, results, and analysis into Appendix G.2 (Table 14, L1368–1396, highlighted in red).

AuthorsOpenReview ↗

(2/3)We thank Reviewer n7og for the effort in reviewing our work and providing constructive comments. We address each concern or question below.

W2. About human evaluation.

  1. Alignment of static-memory metrics with human judgments.

    Our automatic metrics reliably reflect human preferences at the video level, achieving high pairwise agreement (78.5%–83.8%) with decisive human-majority votes. The lower model-level Spearman correlations on these three metrics are driven by large rank shifts in a few models, which arise from distinct perceptual vs. algorithmic sensitivities:

    MetricPairwise agreementSpearmanMain rank change: automatic → human
    Environment Depth81.8%0.301Matrix-Game-3.5: tied 2nd → 8th
    Object Geometry78.5%0.606LingBot-World: 8th → 3rd
    Object Appearance83.8%0.699LingBot-World: 7th → 3rd
    • Environment Depth: The metric measures consistency in depth ordering at corresponding pixel locations, whereas human spatial perception is influenced by semantic content. Humans penalize scenes when foreground objects or occlusion boundaries distort, even if the coarse near–far depth layout remains accurate (which explains Matrix-Game-3.5's drop). Conversely, recognizable landmarks lead to favorable human ratings despite slight camera drift that degrades pixelwise depth alignment.

    • Object Geometry: We include complex cases to broaden coverage, where human observers may overlook subtle changes in shape and proportions, resulting in favorable ratings despite these changes. Our metric compares normalized silhouettes and is sensitive to changes in object contours and aspect ratios. After a revisit, these changes may go unnoticed when the object’s overall appearance and recognizable structure remain largely intact, even though silhouette IoU decreases. Human judgments may therefore remain favorable despite geometric differences detected by the metric. This helps explain why LingBot-World rises from eighth in the automatic ranking to third in the human ranking.

    • Object Appearance: The metric scores all prescribed revisits, whereas human judgments tend to prioritize appearance preservation after successful returns and are less sensitive to the number of successful revisits. In our case analysis, a few LingBot-World videos received lower automatic scores because they had at least one missed revisit scored as zero. However, the target’s appearance was well preserved in the observed revisits, leading human evaluators to prefer LingBot-World. These preferences explain its rise from seventh in the automatic ranking to third in the human ranking.

  2. Overall and subtask alignment in Dynamic Memory.

    • The higher Overall Spearman and PLCC can be explained by finer score resolution and aggregation across more questions. Each subtask averages two event-gated binary scores, while Overall averages all eight: \qquad O_i=\frac{1}{8}\sum_{q=1}^{8}z_{iq} =\frac{1}{4}\sum_{d=1}^{4}S_{i,d}.$$ Here, $z_{iq}$ is the automatic or human-majority score for question $q$ in video $i$, and $Q_d$ contains the two questions for subtask $d$. Subtasks have three score levels, $\{0,0.5,1\}$, whereas Overall has nine, 0, 0.125, ..., 1. Overall’s finer scores can reduce ties and better distinguish videos. Averaging more questions can also reduce the impact of disagreements on individual questions. This can improve agreement between automatic and human rankings, reflected in Spearman, and between their relative score differences, reflected in PLCC.
    • Question-level agreement complements video-level correlations by directly assessing individual subtask judgments. Evaluating direct agreement between our automatic VLM judgments and the human majority yields 76.2% (Return Progress), 76.2% (Persistence/Reset), 78.6% (Final State), and 66.7% (Causal/Physics), providing additional evidence of alignment within individual subtasks. We have added the score-granularity explanation and correlation formulation to Appendix F (Eq. 7, L1081–1093 and L1096–1098, highlighted in red).
AuthorsOpenReview ↗

(3/3)We thank Reviewer n7og for the effort in reviewing our work and providing constructive comments. We address each concern or question below.

Q1. Architectural mechanisms or designs behind the results.

We particularly appreciate this insightful suggestion, which greatly improves the informativeness of our paper. We have added a dedicated mechanistic analysis (L390–402, L481–485, and Table 16, highlighted in red) linking model architectures and training paradigms to memory performance:

  1. Static memory benefits from view-relevant retrieval, geometric alignment, and revisit-oriented training. HY-WorldPlay ranks first on all seven static metrics. It selects historical frames based on field-of-view overlap, combines them with recent context, and reassigns their temporal positions to maintain the influence of distant observations [1]. Its training data include reconstructed real-world scenes and UE-rendered revisit trajectories, with high-density moving-object clips filtered from its real-world dynamics subset. Evoke retrieves stored observations and geometrically reprojects them into the current view [2], ranking second on all four Human/Object metrics. Both explicitly recover relevant past content, consistent with their strong static retention.

  2. Dynamic performance benefits from event-focused data and temporal supervision. LingBot-World combines real-world, game, and synthetic videos with temporally aligned event annotations, long-video training, and video continuation [3]. The evaluated Yume-1.5 uses event-focused I2V captions and dedicated event data alongside general video data [4]. These training designs emphasize how events unfold over time, offering a plausible explanation for the two models' leading Dynamic Overall scores and their successful event initiation in 56/60 and 54/60 cases, respectively.

  3. Controlled ablation confirms view-relevant history simultaneously boosts static and dynamic memory. Using a fixed LingBot-v2 checkpoint under an identical 20-latent attention budget, replacing Sliding Windows context with FoV-based historical retrieval improves performance across all seven static metrics and all four dynamic sub-metrics (Table 3). Crucially, even when isolating the exact same 50 cases where all context policies successfully initiate the event, the FoV mechanism outperforms Sliding Windows (0.8575 vs. 0.8150, Table 16, L1474–1491). These results show that the benefit of view-relevant retrieval extends to post-initiation event performance, beyond differences in event initiation.

References

[1] Sun et al. WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling. 2025. arXiv:2512.14614.

[2] Yin et al. Alaya-EVOKE: From Linear-Scaling Supervision to Endless World. 2026. arXiv:2608.13546.

[3] Robbyant Team et al. Advancing Open-source World Models. 2026. arXiv:2601.20540.

[4] Mao et al. Yume-1.5: A Text-Controlled Interactive World Generation Model. 2025. arXiv:2512.22096.

Q2. Clarification of the Spherical Reconstruction Score.

  1. As described in L313–323, the score measures whether scene geometry is preserved across revisits. Think of the representation as a sphere surrounding the camera, divided into small cells for different viewing directions. Using DA3 depth and camera intrinsics, we map each valid pixel to a direction on this sphere and calculate its surface distance from the camera. For both the initial observation and each revisit, we align the sampled frames using estimated camera rotations, and each cell records the nearest valid surface distance in that direction. This produces one spherical distance map for the initial observations and another for the revisit, which we then compare.

  2. After global scale alignment, the score multiplies coverage agreement—the intersection-over-union of occupied bins—by distance agreement within shared bins. Missing or additional coverage and inconsistent distances therefore reduce the score. Scores are averaged across prescribed revisits, with invalid revisits assigned zero.

Sincerely,

WorldMemBench Authors

Comments on the paper

2 comments
Area_Chair_11WxOpenReview ↗

Reminder: Please Review Author Rebuttals and Participate in Discussions

Dear Reviewers,

Thank you for your time on the reviews. The authors have now submitted their rebuttals, and I encourage you to read their responses and participate in the discussion.

Please consider whether the rebuttals address your concerns and, where clarification is still needed, engage with the authors through the discussion forum. I also encourage you to exchange perspectives with your fellow reviewers and update your review or rating if appropriate.

Your participation will help ensure a thorough and fair evaluation of each paper. Thank you again for your time and effort.

Best regards,

AC

AuthorsOpenReview ↗

Summary of Rebuttal

Dear ACs and Reviewers,

We sincerely thank you and all the reviewers for your time and constructive feedback on our submission.

We are encouraged by the reviewers’ recognition of our formulation of static and dynamic world memory (n7og, ub5Y), the novelty of our multi-revisit and cross-axis protocols in exposing long-term memory degradation (VhwB, AyxL, ub5Y), and the comprehensive evaluation of nine recent models across four diverse scenarios (n7og, AyxL, ub5Y). During the rebuttal, we provided additional experiments and analyses and revised the paper in the following areas:

  • Broader trajectory and temporal coverage (n7og, AyxL). We evaluated all nine models on 90 cross-view cases combining translation and rotation, generating 810 additional videos. Model rankings remain strongly correlated with the original results across the six evaluated static-memory metrics (Spearman’s ρ = 0.867–1.000; Appendix G.2). We also extended all nine models’ evaluations to 40–63 seconds with three revisits. Model rankings remain highly consistent with the original results: HY-WorldPlay continues to lead all seven static-memory metrics, and Evoke remains second on all four Human/Object metrics (Appendix G.3).

  • Clarifying the two components of dynamic memory (VhwB, AyxL, ub5Y). We further clarified that dynamic memory requires both initiating the intended event and preserving its progression and consequences across camera departures and returns. Together, these capture the ability to generate and maintain an evolving world. Our Overall score therefore evaluates this joint capability, while separately reported event-start rates and conditional scores distinguish the two components (Tables 2 and 5). To further validate sensitivity to memory after initiation, we compared FoV, Sliding Windows, and FullKV using the same LingBot-v2 backbone on 50 shared cases where all three policies successfully initiate the event. FoV still outperforms Sliding Windows (0.8575 vs. 0.8150), showing that our evaluation captures differences in subsequent event consistency with initiation success held fixed (Appendix H).

  • Human alignment across evaluation dimensions (n7og, VhwB). We demonstrated video-level pairwise agreement of 78.5%–83.8% and question-level agreement of 66.7%–78.6% with the human majority, and analyzed the perceptual versus algorithmic sensitivities underlying specific model rank shifts. We further formalized the scoring and aggregation scheme and explained how score granularity and aggregation can account for higher correlations for Overall than for individual subtasks in Dynamic Memory (Appendix F, Eq. 7).

  • Connecting results to model design (n7og). We added analysis relating static retention to view-relevant retrieval, geometric alignment, and revisit-oriented training, and dynamic performance to event-focused data and temporal supervision. Together with the controlled context-policy comparisons, this makes the benchmark’s implications for model design more explicit.

  • Controlling target visibility and camera-return accuracy (ub5Y). We compared revisits with valid target observations and qualified camera returns. Both repeated-revisit degradation and the additional cross-axis penalty persist across all seven static-memory metrics. The environment results also retain both trends under a tighter 5° return-orientation threshold, strengthening the evidence for the benchmark’s central findings (Appendix G.1).

  • Cross-judge validation (AyxL). We re-evaluated all 540 dynamic videos, covering 4,320 checklist questions, using Qwen3.6-27B in place of Qwen3.8-27B under the same protocol. Overall model rankings remain highly consistent (Spearman’s ρ = 0.983), with subtask correlations of 0.979–1.000 and a mean absolute score difference of 0.030.

Following these responses, Reviewers VhwB, AyxL, and ub5Y confirmed that their main concerns were addressed. All three reaffirmed their positive ratings.

We believe these revisions and additional experiments have significantly improved our paper. We sincerely appreciate your time and effort in coordinating the review process.

Best regards,
WorldMemBench Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.