acceptodds
Withdrawn from ICLR 2027

MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fundamental gap persists between visually plausible video generation and the functional requirements of a world model, particularly in maintaining a stable and reasonable internal state over extended temporal horizons. While existing benchmarks primarily emphasize visual quality, motion coherence, and text-video alignment, they largely overlook memory, the core capability of a world model to preserve consistency across long-term horizons and complex interactions. To address this gap, we present MBench, a comprehensive benchmark dedicated to quantifying and evaluating the memory capability of video world models. We systematically decompose the memory capability of video world models into three hierarchical and complementary core dimensions: entity consistency, environment consistency, and causal consistency, which are further refined into 12 quantifiable sub-dimensions for comprehensive characterization of long-term memory. Our benchmark is built upon rigorously curated real-captured long videos, and evaluated by rule-based quantitative matrices and VLM to enable objective and comprehensive consistency assessment. Extensive evaluations of mainstream state-of-the-art video world models reveal critical systemic limitations of existing methods in long-term state retention, providing a standardized benchmark and clear research direction to advance the field.

Primary area:
datasets and benchmarks
Keywords:
world model, memory, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BK1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MBench is a benchmark for long-term memory in video world models. It organizes memory into entity, environment, and causal consistency, further divided into 12 sub-dimensions. The benchmark draws more than 1,040 cases from five real-video datasets and evaluates eight text-conditioned continuation models and six action-conditioned models. A trigger-conditioned score combines whether a model executes the intended memory challenge with its consistency after the challenge. The evaluation uses hand-designed vision metrics and VLM judgments and includes a human preference study.

Critical strengths
  1. The benchmark considers entity, spatial, rendering, and causal memory in one framework. The causal dimension and the distinction between triggering an event and preserving its consequence are especially useful, since a static model should not receive a high memory score by avoiding change.

  2. The evaluation covers both text-conditioned and action-conditioned systems and reveals meaningful differences between them. The finding that models with good spatial consistency can perform poorly on state evolution is a useful diagnostic result.

Critical weaknesses
  1. Several automatic metrics are not yet validated strongly enough to support a single comprehensive benchmark. In the reported human validation, text-conditioned spatial epipolar consistency correlates only weakly with human judgment, text-conditioned lighting consistency is negatively correlated, and text interaction is also weak. The reported correlations are computed after aggregation to only eight or six model-level win ratios, so high coefficients for other dimensions may not imply robust sample-level agreement.

  2. The benchmark uses heterogeneous metrics with different score mappings, and its trigger-conditioned M-Score combines coverage with conditional reliability. It is unclear how sensitive model conclusions are to normalization choices, trigger-verification settings, and VLM choice. The main table also reports normalized scores without uncertainty, making close comparisons difficult to interpret.

Questions for authors
  1. Can the authors report agreement between automatic metric preferences and human preferences at the individual comparison level, together with confidence intervals for the model-level correlations?

  2. How stable are model rankings under reasonable changes to metric normalization, trigger-verification settings, and the VLMs used for trigger verification and causal evaluation?

Reviewer_eWPHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MBench evaluates whether video generators preserve entities, environments, and causal state when relevant content leaves view and later returns. It contains 1,040 evaluation cases derived from five video datasets, with text-conditioned continuation and action-conditioned rollout protocols. Twelve metrics combine feature similarity, geometry estimates, rendering statistics, instruction alignment, and VLM judgments. For text-conditioned outputs, a trigger judge first checks whether the intended memory challenge occurred; the final M-Score combines trigger coverage with consistency on admitted samples. The experiments compare eight text-conditioned and six action-conditioned models and evaluate agreement with human judgments.

Critical strengths
  1. Requiring an actual departure, occlusion, or return is a good way to make memory failures observable. Separating whether the model executes the challenge from how well it retains state is also useful: an otherwise plausible static video should not pass a memory test simply because nothing changes. The entity/environment/causal decomposition makes the benchmark potentially more diagnostic than a single video-quality score.

  2. The evaluation spans both text and action interfaces and includes human validation rather than assuming every automatic metric is reliable. Some of those results are strong: object-geometry rankings correlate with human rankings at 0.97 in both settings, and action-conditioned spatial metrics reach 0.94. The contrasting performance on spatial stability and hidden state evolution is an interesting observation that could guide model development if the scoring limitations are resolved.

Critical weaknesses
  1. Several headline metrics do not agree sufficiently with human judgments to support the benchmark's broad interpretation. Table 3 reports text-conditioned lighting correlation of -0.49, epipolar correlation of 0.16, and text-interaction correlation of 0.19 with human model rankings. These are substantial limitations for a benchmark intended to measure memory across all three categories. Section 4.3 evaluates model-level win ratios; agreement across only six or eight models does not establish that individual memory failures are scored correctly. The benchmark would be more convincing with sample-level validation and controlled failures such as an identity-swapped return or a changed scene layout under otherwise similar motion.

  2. A high consistency score is meaningful only if the intended memory challenge actually occurs. The main paper correctly recognizes this and introduces trigger-conditioned scoring, yet Section 5 also describes models ignoring, delaying, or misinterpreting controls. Prescribing a camera movement or action therefore cannot by itself establish that the relevant content left view and returned. The main results need a clearer account of how often each model actually completes the challenge, and whether rankings survive human verification of those events. The trigger-calibration results in the supplement reinforce this concern, but the central issue is visible in the main paper: challenge avoidance and successful memory must be separated in the reported comparison.

Questions for authors
  1. Assessment-changing validation: How do the rankings change when human-verified trigger labels replace the VLM gate? Please show coverage and conditional reliability separately, and verify actual exit/reentry for the action-conditioned outputs as well.

  2. Clarification: How is a change of identity after occlusion distinguished from a newly detected person? Please explain how the score measures identity across the interruption, rather than consistency only within each visible segment.

  3. Assessment-changing validation: Do the metrics distinguish a correct return from a frozen video, an identity-swapped return, or a scene with a changed layout under the same motion? Please report sample-level human agreement and uncertainty, including the dimensions with poor or negative model-level correlation.

Reviewer_VVpfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a benchmark to evaluate video models' long-term memory ability. It focuses more on the consistency of memory like entity/environment and causal consistency, instead of other qualities. It evaluates 14 text-conditioned and action-conditioned video generation models on 12 dimensions, where all 12 dimensions are evaluated by specialized visual expert models rather than human scoring and it applies a trigger-conditioned mechanism to prevent cheating by the models. The results show that though those models are state-of-the-art, they still fail to keep a persistent memory of the world state over longer time periods.

Critical strengths

The construction of the whole 12 dimensions provides a very fine-grained analysis of different kinds of consistency to measure memory ability, instead of only focusing on superficial visual stability. Among all metrics, the paper does not depend on any subjective or aesthetic scores from humans or qualitative evaluation alone; instead, it uses a set of specified visual models to provide objective feedback. It works as a full anatomy of all kinds of failure modes, rather than a simple ranking score for those state-of-the-art models.

Critical weaknesses

For action-conditioned models, the benchmark mainly uses camera movement to control the process of testing memory. Without using other kinds of action maneuvers in the world models in the paper, it lacks analysis of the memory ability under different kinds of interactions. The evaluation of all dimensions mainly depends on the soundness of third-party evaluator models, which might not be reliable when the complex scenes are beyond their own abilities.

Questions for authors

About the action-conditioned evaluation, do we have any plans/experiments to evaluate non-camera-movement actions, such as object manipulation or interactive controls?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.