- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MBench is a benchmark for long-term memory in video world models. It organizes memory into entity, environment, and causal consistency, further divided into 12 sub-dimensions. The benchmark draws more than 1,040 cases from five real-video datasets and evaluates eight text-conditioned continuation models and six action-conditioned models. A trigger-conditioned score combines whether a model executes the intended memory challenge with its consistency after the challenge. The evaluation uses hand-designed vision metrics and VLM judgments and includes a human preference study.
Critical strengths
-
The benchmark considers entity, spatial, rendering, and causal memory in one framework. The causal dimension and the distinction between triggering an event and preserving its consequence are especially useful, since a static model should not receive a high memory score by avoiding change.
-
The evaluation covers both text-conditioned and action-conditioned systems and reveals meaningful differences between them. The finding that models with good spatial consistency can perform poorly on state evolution is a useful diagnostic result.
Critical weaknesses
-
Several automatic metrics are not yet validated strongly enough to support a single comprehensive benchmark. In the reported human validation, text-conditioned spatial epipolar consistency correlates only weakly with human judgment, text-conditioned lighting consistency is negatively correlated, and text interaction is also weak. The reported correlations are computed after aggregation to only eight or six model-level win ratios, so high coefficients for other dimensions may not imply robust sample-level agreement.
-
The benchmark uses heterogeneous metrics with different score mappings, and its trigger-conditioned M-Score combines coverage with conditional reliability. It is unclear how sensitive model conclusions are to normalization choices, trigger-verification settings, and VLM choice. The main table also reports normalized scores without uncertainty, making close comparisons difficult to interpret.
Questions for authors
-
Can the authors report agreement between automatic metric preferences and human preferences at the individual comparison level, together with confidence intervals for the model-level correlations?
-
How stable are model rankings under reasonable changes to metric normalization, trigger-verification settings, and the VLMs used for trigger verification and causal evaluation?