acceptodds
Withdrawn from ICLR 2027

Execution Before Memory: Diagnosing Memory Failures in Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We introduce MIST, an evaluation that separates the change from the memory it should leave behind. The model first sets the scene while the change stays hidden, and only later is asked to show it, with leakage before that point kept under control. This separation shows that the present bottleneck is following the instruction for that execution. The same gap appears in general memory evaluations. On that basis we propose M3-Agent (Make the Missing Move), which recursively revises the details of the input instruction so that the model successfully carries out the requested instruction. Relative to the baseline method, the change is shown more than twice as often. On the benchmarks, WRBench's score T rises by 26.1% and MBench-T's score U by 52.7%. The method thus meets the problem in memory evaluation from the execution side: it releases the model's potential, and it builds a stricter comparison.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video generation, agent, memory design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YDaBOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that some video-memory benchmarks penalize models before memory is actually tested: the requested action or camera movement may never happen. MIST separates scene setup, execution, and final-state evaluation. M3-Agent then diagnoses failed videos and rewrites the instruction.

Critical strengths
  1. The distinction between execution failure and memory failure is important and easy to overlook.
  2. The stage-wise analysis gives a clearer account of where existing benchmarks lose score.
  3. The paper is unusually clear about what its results do not establish.
Critical weaknesses
  1. The main M3-Agent result depends on selecting retries with the evaluation scorer. Once this oracle is removed, the improvement largely disappears. So this seem more like as a diagnostic upper bound than a practical agent.
  2. The official benchmark experiments lack a matched control that resamples the original prompt with the same budget and uses the same selector.
  3. Supplying the hidden target in the rewritten prompt weakens the memory claim: the model is being reminded of the state it is supposed to remember.
  4. The study is centered on one main generator family, so it is unclear how general it is.
Questions for authors

see weakness

Reviewer_8CqoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that existing memory benchmarks(e.g. WRBench and MBench-T) for video world models conflate execution and memory. By compressing scene setup, requested modifications and final state into one aggregate metric, these benchmarks misattribute failures to memory decay even when the intended modification is never performed, meaning memory is not actually evaluated. To decouple these confounding variables, the authors present MIST, a diagnose framework following a set hide show paradigm that independently evaluates setup, leak, reveal and inside gates. Evaluated on Wan TI2V 5B, the model reliably establishes scenes without information leakage across gaps. However, 41 out of 43 baseline clips fail to activate, pinpointing execution as the primary bottleneck. The authors further introduce M3 Agent, a training free pipeline that inspects the last frame and revises only unsuccessful operations while keeping the generator fixed.

Critical strengths
  1. Identifying a key issue in current memory evaluations.Existing benchmarks rely on a single overall score, which can confuse execution failures with memory failures. If a model never performs the requested change, a low score may be mistaken for forgetting, even though its memory was never actually tested.

  2. Introducing MIST to identify the real bottleneck.MIST evaluates four stages separately: setup, leak, reveal, and inside. The results show that models generally set up scenes correctly and avoid leaking hidden information, but 41 out of 43 baseline clips fail to reveal the hidden state. This suggests that execution, rather than memory, is the main bottleneck. Similar patterns appear on WRBench and MBench-T.

  3. Validating the findings with M3-Agent. By rewriting only the failed operation in the prompt, M3-Agent significantly improves the scores of the same frozen generator. This suggests that the model can often perform the required changes when given clearer instructions. Based on this, the authors recommend evaluating scene setup, change execution, and final-frame correctness separately, rather than relying on a single overall score.

Critical weaknesses

1.The paper does not explain how each gate score is computed, what threshold defines never-open, how the ordered labels are assigned, or how the judge aggregates its readings across the six reveal-shot positions. As a result, the mean scores and class counts are hard to reconcile; for instance, a mean baseline reveal of 0.147 seems at odds with 41 of 43 clips being never-open.

2.MIST may be unable to test memory by design. The three shots are generated separately and stitched together, but the paper does not specify what pre-reveal context the reveal shot is conditioned on. If it is only the final frame of the opaque gap (a closed container), and the baseline prompt omits the answer, the generator has no access to the hidden state, so memory cannot manifest under this protocol at all. The authors state that persistence has not been tested yet, but do not discuss whether the protocol is capable of testing persistence in the first place.

  1. Unclear writing and core claim. The paper is loosely organized: multiple experimental settings are interleaved, the same metric name refers to several different numbers, and numerous disclaimers are needed to prevent misreading. More importantly, the paper does not clearly position its core claim. The title and discussion frame the work as diagnosing memory failures, yet the paper ultimately concedes that it cannot assess memory itself and in fact diagnoses execution failures. Meanwhile, the abstract highlights M3-Agent’s gains as the main result, making the paper read more like a method paper. I suggest the authors state clearly that the core contribution is an evaluation-level diagnosis, and revise the title, abstract, and contributions accordingly, for example by presenting M3-Agent’s results as an upper bound on the frozen model’s execution ability rather than as the headline performance result.
Questions for authors

See above weakness.

Reviewer_gh3WOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines execution failures in video world-model memory tests. It argues that when the required instructions are not executed, low scores may reflect execution failure rather than meaningfully test memory. It introduces MIST to assess setup, leakage, reveal, and state correctness separately. M3-Agent revises prompts to improve results on MIST, WRBench, and MBench-T.

Critical strengths

1.The motivation is described clearly. Current memory benchmark may not be tested as it is designed to. The author identify this issue and propose a method for stage-wise diagnosis.

2.The ablation studies include useful controls, such as answer-only prompting, repeated sampling, and target-withheld rewriting, which help clarify what drives the observed improvements.

Critical weaknesses

1.Line 92: “The three lines of work below each report one total” is too broad. WRBench explicitly reports re-observation support and conditional consistency scores separately

2.Existing benchmarks already check whether the memory test actually takes place before evaluating how well the model remembers. WRBench separates “re-observation support” from “re-observed spatial and state consistency,” while MBench distinguishes “trigger coverage” from “memory reliability.” MIST appears to explore these existing evaluation ideas in greater depth, but it is unclear what additional failures it can identify beyond these existing checks.

3.The evaluation is limited to 43 cases in the authors’ own MIST diagnostic set and 40 cases from WRBench, 30 from MBench-T. The limited coverage make the generalizability of the findings difficult to assess.

Questions for authors

see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.