- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces WorldMemBench, a benchmark for evaluating world memory in video world models. It divides memory into two types: static memory, which measures how well models preserve the identity, appearance, and geometry of people, objects, and environments, and dynamic memory, which evaluates whether events and states evolve consistently when out of view. The benchmark includes 260 cases across 23 categories and uses controlled departure-and-revisit camera trajectories to test memory consistency. Static memory is evaluated using visual similarity and geometric metrics, while dynamic memory is assessed through a VLM-based checklist.. Evaluations of nine models show that strong static memory does not guarantee reliable dynamic memory, and performance degrades with repeated revisits and changes in rotation axes. Human studies support the evaluation metrics, while training-free experiments on LingBot World v2 suggest that combining recent context with FoV-relevant history offers a good trade-off between memory and efficiency.
Critical strengths
- The paper clearly separates world memory into static memory (preserving content) and dynamic memory (tracking changes outside the camera view). This addresses an important gap in existing metrics like FID, FVD, and VBench.
- The evaluation is well designed. Repeated revisits help reveal how memory degrades over time, while comparing LRLR and LRUD helps isolate the effect of switching axes. The static metrics cover both appearance and geometry, and the event-gated checklist evaluates dynamic changes more meaningfully than pixel similarity.
- The paper evaluates nine recent models under the same protocol and also reports video quality and camera-following performance. The results reveal interesting differences between models, including memory degradation over time, sensitivity to axis changes, and the fact that better video quality does not necessarily mean better memory.
- The human evaluation supports the automatic metrics, while additional experiments on revisit accuracy and dynamic scoring help verify the robustness of the findings. The context-access experiments (FullKV, Sliding Windows, and FoV) also show that the benchmark can help identify model weaknesses and guide future model improvements, rather than simply ranking models.
Critical weaknesses
-
The dynamic memory score seems to mix up action execution and memory. If an event fails to start, all eight questions get zero, even though this does not necessarily mean the model failed to remember it. For example, HY-WorldPlay scores 0.928 when the event starts, but its event-start rate is only 31.7%, bringing the overall score down to 0.294. The paper uses this result to argue that strong static memory does not imply strong dynamic memory, but it may actually say more about the model's difficulty in executing actions than its memory.
-
Several factors unrelated to memory can also affect the scores. If a target is not detected during a revisit, it gets zero, but this could be because the target disappeared, the tracker failed, or the camera did not return to the right position. Depth Spearman and Object Geometry also depend heavily on viewpoint alignment, so small camera errors can lower the scores. In the cross-axis experiments, Rot. RPE increases noticeably, suggesting that the camera itself is less accurate.
-
The dynamic memory evaluation only checks what the scene looks like when the camera returns. This makes it hard to tell whether the model actually keeps track of events while they are off-screen or simply generates a plausible final state when the camera looks back. These are quite different abilities, especially when evaluating world models, but the current benchmark cannot really distinguish between them.
Questions for authors
I am leaning toward acceptance, but I would like the authors to clarify the concerns raised in the above weaknesses.
4 comments
(2/2)We thank Reviewer ub5Y for the effort in reviewing our work and providing constructive comments. We address each concern or question below.
W3. Off-screen state consistency and internal mechanisms.
- We evaluate state changes across the visible sequence. The judge receives chronologically sampled frames spanning the entire rollout, together with the planned camera phases (L734–741). Return Progress compares the returned state with the pre-departure baseline; Persistence/Reset checks whether its consequences persist without reverting; Final State assesses the expected outcome; and Causal/Physics examines the visible ordering of events. The questions therefore use evidence before departure, upon return, and throughout subsequent observations (L788–797).
- Our contribution is a behavioral benchmark for dynamic memory. It evaluates whether generated worlds preserve event progression and consequences across interrupted observation. Investigating how these mechanisms support dynamic memory is a direction for our future work.
Sincerely,
WorldMemBench Authors
Thank you for the thoughtful and detailed response. Your explanations addressed most of my concerns. I will keep my positive score.
Dear Reviewer ub5Y,
We deeply appreciate your encouraging feedback and your support for our work! We are delighted to hear that our explanations and analyses addressed your concerns.
Best regards,
WorldMemBench Authors
(1/2)We thank Reviewer ub5Y for the effort in reviewing our work and providing constructive comments. We address each concern or question below.
Dear Reviewer ub5Y,
Thank you for your positive inclination toward our paper and for recognizing our static/dynamic memory taxonomy, multi-revisit and cross-axis protocols. Below, we clarify each of your three concerns.
W1. Event occurrence and continuation in Dynamic Memory.
W2. Target observability and camera accuracy in static-memory evaluation.
Comparing revisits with valid target observations and camera returns. For Human/Object, we retain videos only when every initially evaluated target has departure evidence and valid observations at every compared revisit. For Environment, DA3-estimated orientations must indicate a departure of at least 30° from the initial direction and a return orientation error of at most 30°. Each comparison uses fixed cases across the compared revisits and trajectories within each model, with effects averaged over cases and then equally over models (L1281–1292).
Memory degradation persists across repeated revisits. All seven static-memory metrics retain a positive mean decrease from the first to the third revisit after filtering. For this analysis, all three returns under LRLRLR must satisfy the criteria above. The table compares the unfiltered and filtered decreases, R1 − R3 (Table 12, L1315–1327).
The cross-axis penalty persists after controlling return accuracy. For this comparison, both returns in both LRLR and LRUD must satisfy the same criteria. We measure the additional degradation from switching axes as
Δaxis=(R1−R2)LRUD−(R1−R2)LRLR.
All seven metrics retain a positive mean cross-axis penalty after filtering, showing that switching from yaw to pitch remains harder on average among qualified returns (Table 12, L1328–1334).
Both trends also persist under tighter return tolerances. With a 5° return-error threshold and the same 30° departure requirement, the mean multi-revisit decreases remain positive for Environment Visible, Depth, and Reconstruction (0.092, 0.057, and 0.116), as do the cross-axis penalties (0.094, 0.158, and 0.050) (Table 13, L1351–1359).