acceptodds
Under review as a conference paper at ICLR 2027

MentalWM-Bench: Evaluating Mental-State Transitions in Video World Models

Abstract

Video generation models are increasingly used to build dynamic worlds with characters that act and interact. Coherent behavior in these worlds requires more than realistic appearance and motion: events must change characters' emotions and intentions, and these changes must shape subsequent behavior. Mental world modeling captures this connection between events, internal states, and actions. Yet visual fidelity, event occurrence, and a correct final expression alone cannot establish whether an event-driven mental-state transition unfolds in order and persists across interactions. We introduce MentalWM-Bench to evaluate these transitions through observable behavior. Three complementary tasks assess trigger execution, complete transitions, and two-turn continuation across individual, dyadic, and group scenarios. Evidence-based criteria examine events, initial and resulting states, emotion and intent changes, temporal order, and cross-turn continuity, alongside scene and physical fidelity. We evaluate seven video generators and compare automatic assessments with multi-rater human judgments. Correct outcomes can coexist with incomplete transitions, and successful first-turn states do not ensure subsequent preservation. Physical and mental-state scores remain weakly associated within generators. Emotion and intent updates score below endpoints, with lower performance in longer continuations and group interactions. MentalWM-Bench provides a framework for systematically evaluating mental-state dynamics in video, informing the development of next-generation world models with sustained, coherent character interactions.

Primary area:
datasets and benchmarks
Keywords:
video world models, social reasoning, video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uwkYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MentalWM-Bench evaluates whether video generators realize event-driven changes in characters’ emotions and intentions and preserve them across interactions. It includes trigger-only, complete-transition, and two-turn continuation tasks. Experiments across seven generators, supported by human comparisons, identify failures that endpoint correctness and visual fidelity alone can miss.

Critical strengths

The benchmark makes a useful distinction between showing the right outcome, realizing the transition that leads to it, and preserving that state into the next interaction. The evaluation covers seven generators and balanced individual, dyadic, and group source scenarios. The authors also make a substantial effort to check their conclusions through human judgments, source-case bootstrap intervals, raw-versus-adjusted scoring, and conditional analyses. In particular, they explicitly show that most low change scores co-occur with incomplete events or endpoints, rather than presenting the entire gap as evidence of a separate reasoning failure. This makes the empirical analysis more informative and appropriately scoped.

Critical weaknesses

The main concern I have is about the reliability of the core measurements. The human study is valuable, but Appendix C reports ordinal agreement of only 0.297, 0.299, and 0.368 for emotion change, intent change, and boundary continuity. The headline single-turn correspondence concerns a three-dimension proxy, while Table 10 evaluates the original event/action checks rather than the strict Social-8 rubric. Therefore, it remains unclear how closely the main mental-state scores and model rankings agree with humans on the same criteria.

Another issue is that part of the transition-score gap follows from the scoring construction. Consistency adjustment reduces emotion-change scores from 0.547 to 0.365 and intent-change scores from 0.464 to 0.303, and change checks require more prerequisites than endpoint checks. The authors acknowledge this and provide useful conditional analyses. However, controlled temporal perturbations would make the benchmark’s added diagnostic value clearer, especially because the residual process deficits after full-credit prerequisites are relatively small.

The multi-turn comparison is also somewhat limited. The 24 cases are selected according to Seedance’s continuation-interface availability, and durations and interfaces are not fully matched. The five-turn pilot additionally includes only completed chains and excludes unsuccessful requests. These limitations are disclosed, but they still make it difficult to separate state-preservation quality from interface differences and completion selection when interpreting cross-model results.

In terms of construct validity, the complete and continuation tasks specify the target responses in advance. The paper correctly frames this as trajectory realization rather than open-ended response inference. Still, a matched comparison with ordinary behavioral or temporal instructions would help establish what the mental-state formulation adds beyond fine-grained prompt adherence. The distinction matters when interpreting the benchmark as an assessment of mental world modeling.

Finally, the evaluation records need a clearer accounting. The paper reports 455 task ratings, while the stated distribution of 243 multi-rated slots implies 738 slot-level rater assignments. These may refer to different units, but the relationship is not explained. A compact accounting table, together with exact generator/evaluator versions, frame sampling, durations, and continuation settings, would make the reported results easier to verify from the manuscript.

Questions for authors

1.Could the authors report matched human–AI correspondence and model-ranking stability specifically for the strict Social-8 criteria and boundary-state preservation, rather than relying on the three-dimension proxy or original event/action checklist? 2.Could the authors test controlled temporal errors, such as an early response or swapped event–response order, while keeping the visible endpoints unchanged? Reporting raw and adjusted scores on these cases would help validate temporal sensitivity. 3.Could the authors provide an interface-matched continuation subset and report all attempted five-turn chains, including failure counts and reasons? This would help distinguish preservation failures from generation or interface failures. 4.Could the authors compare matched prompts using mental-state labels versus explicit behavioral descriptions, while preserving the requested sequence? This would clarify the contribution beyond ordinary temporal prompt following. 5.Could the authors reconcile the human-rating counts and provide a single table mapping each protocol to its videos, raters, scoring rules, and exact generation/evaluation settings?

4 comments

Thank you for recognizing the value of the task distinctions and conditional analyses, and for suggesting concrete ways to strengthen validation. In response, we added a blinded human evaluation directly targeting the core criteria and broadened experimental coverage. This response addresses scoring reliability and temporal diagnosis; our second response discusses multi-turn evaluation, task interpretation, and reproducibility.

1. Strict Social-8, boundary-state preservation, and model comparisons

You suggested directly validating the mental-state criteria used in the main experiments rather than relying on the three-dimensional proxy or original event/action checks. We have completed an independent human evaluation under matched criteria.

We aligned the corresponding human and AI questions and scoring scales, specifying binary-item treatment and fractional-output mapping. AI scores were hidden from raters throughout. Strict Social-8 achieved 89.5% exact human–AI agreement across 1,010 valid entries from 132 videos, including 93.2% for emotion change and 82.3% for intent change. The temporal criteria achieved 93.9% for event–response order and 83.3% for no premature response. Separate boundary-state preservation validation yielded 95.8% agreement across 24 entries.

This addition directly covers the emotion changes, intent changes, temporal relationships, and cross-turn preservation you questioned, rather than relying only on composite-score correlations. The revision will present per-criterion results, applicable sample counts, and scoring protocols together, making the evidence for the core measurements more direct and verifiable.

For model comparisons, we also completed additional generation experiments comprising 54 single-turn and 27 continuation tasks per model, with the main conclusions remaining consistent with the original experiments. Building on this evidence, the revision will further compare model rankings obtained from human and automated aggregate scores on matched videos, with source-case bootstrap uncertainty, strengthening validation at both the item and model levels.

2. Scoring construction and temporal diagnostic value

Thank you for recognizing the role of the raw-versus-adjusted and conditional analyses. The existing evidence separately examines whether the dimension-level finding depends on post-processing and whether process checks add information when events and states are already realized.

Appendix D.2 shows that all seven generators have lower mental-state means than their respective other dimensions before consistency adjustment. Conditioning on full-credit initial states, resulting states, and triggering events, Appendix D.4 identifies 17/357 emotion-related and 6/255 intent-related cases with remaining process shortcomings. The original multi-rater analysis provides complementary support: in Appendix D.3, 10 of 83 slots passing event/action outcome checks still fail a process check. The argument for supplementary process assessment therefore draws on scoring-rule sensitivity, conditional diagnosis, and human evidence.

Your proposed controlled perturbations would make this diagnostic value more explicit. We plan to construct paired cases preserving visible initial and final states and the main visual content while introducing premature responses or swapped event–response order. We will report endpoint scores, raw temporal scores, and adjusted temporal scores, while checking for visual artifacts introduced by editing. This will directly test whether temporal criteria detect intermediate-process errors when endpoint content remains correct.

Thank you for helping us connect the existing diagnostic findings more closely with validation targeting specific error types.

Following our first response, we address your questions about multi-turn comparisons, mental-state formulations, and experimental records. Your suggestions helped us broaden task coverage and clarify the evidence provided by each experiment.

3. Multi-turn expansion, continuation configurations, and long-chain records

We added 555 task prompts: 405 single-turn prompts and 150 second-turn continuation prompts, balanced across individual, dyadic, and group interactions. Continuations reuse the corresponding single-turn videos as their first turns, avoiding double-counting. Together with the original 471 prompts, the expanded benchmark contains 1,026 task prompts.

On this extension, we completed generation and evaluation of 81 additional tasks per model, comprising 54 single-turn and 27 continuation tasks. The main conclusions remained consistent with the original experiments. This provides empirical support on additional samples and broadens the tested continuation tasks.

Your interface-matching suggestion provides a clear direction for improving interpretability across models. In the revision, we will control continuation conditions, target durations, and evaluation windows for models supporting common settings, reporting matched-setting results separately from results under their original configurations. The existing conditional analysis already provides a direct observation: only 52 of 85 samples with full-credit first-turn resulting states receive full credit for boundary preservation. Thus, realizing a first-turn outcome and retaining it during subsequent interaction are requirements that warrant separate assessment.

For long-chain records, we will document planned turns, completed turns, evaluation inclusion, and request-failure reasons by model. The relevant Seedance experiments were planned for three to four turns and will be recorded separately from five-turn requests. We will also explicitly document the quality-informed inclusion rule for the additional Wan human-evaluation subset. This accounting will distinguish request completion from preservation quality in generated content and make the existing analysis of 37 completed chains easier to verify.

4. Mental-state labels versus explicit behavioral descriptions

Your proposed paired comparison would further clarify the specific demands of mental-state tasks. By supplying target responses, the benchmark fixes the trajectory to be realized and locates failures in event execution, response timing, and state preservation. Even when instructed to display fear after a threat, a model may display it prematurely or lose the achieved state during continuation. These failures directly affect behavioral continuity in interactive narratives and virtual characters.

Mental-state formulations also allow a target to be expressed through different observable behaviors, whereas explicit behavioral descriptions impose more specific action requirements. Their paired comparison can therefore help distinguish realizing a target state behaviorally from executing specified actions. We plan to hold characters, events, temporal order, and target duration constant, evaluate both formulations with common observable criteria, and control differences in descriptive specificity and information content.

The benchmark’s value lies in translating state updates and cross-turn dependencies into explicit, testable behavioral requirements. These requirements relate to instruction following while providing diagnosis of failure stages that an overall adherence score cannot directly locate.

5. Human-evaluation records and experimental settings

The human evaluation involved 15 raters. Following your suggestion, the revision will consolidate sample coverage, participant counts, rating units, and aggregation methods for the original human study, supplementary reassessment, and new blinded evaluation. We will verify the associated counts and their relationships against the original records.

Exact generator and evaluator versions, frame sampling, durations, and continuation settings will also be consolidated into a protocol table, clearly distinguishing original and additional experiments. These changes will make the expanded data, direct human validation, and diagnostic analyses easier to verify. Thank you for your detailed and constructive assessment.

I thank the authors for the additional evaluation. However, I prefer to retain my original score. The matched human study is helpful, but continuation comparisons remain limited by differences in settings, and annotation counts still need clarification.

Thank you for considering the additional evaluation and finding the matched human study helpful. We have verified the annotation accounting and would like to clarify the scope of the continuation evidence.

Annotation counts. In the original human study, 15 raters provided 455 rater–task submissions containing ratings. Of these, 352 submissions contain ratings for two video slots and 103 contain ratings for one, yielding 807 rater–video-slot rating records. The inter-rater analysis uses the 243 slots rated by at least two raters, accounting for 738 records; the remaining 69 records belong to singly rated slots. Thus, both 455 and 738 are reproducible from the original study records: they refer to different counting units and subsets. We will replace the ambiguous term “task ratings” with “rater–task submissions” and explicitly document the counting units, inclusion criteria, and source-record version.

Continuation evidence. We agree that configuration differences matter when interpreting cross-model comparisons. Our diagnostic conclusion is that achieving a first-turn state and preserving it during continuation warrant separate assessment. The original model-specific results in Appendix D.1, Table 6 support this distinction: among samples with full-credit first-turn resulting states, 10/21 for Seedance 2 and 15/22 for Kling 3 receive full boundary-preservation credit. These are observations under each model’s evaluated continuation configuration, rather than estimates isolating model capability from interface effects.

We have also completed re-annotation and rescoring under a unified annotation and scoring protocol for the same evaluated subset of the 555-prompt extension described in our previous response: 54 single-turn and 27 continuation tasks per model. This concerns the previously reported additional samples, with consistent evaluation criteria across those samples; it does not imply that their generation interfaces were matched.

Thank you again for your careful assessment. We hope the verified accounting and this clarification make the supporting evidence and its interpretation more transparent.

Reviewer_zTjYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces MentalWM-Bench to evaluate whether video generators realize specified, event-driven changes in characters' expressed emotions and behavioral intentions. It represents a target interaction as S0 → A1 → S1 → A2 → S2 and defines three complementary tasks: trigger-only execution, complete transitions, and two-turn continuation. Experiments cover seven generators for the single-turn tasks and six for continuation. It also provides human evaluation results for comparisons. The contribution is primarily a benchmark and diagnostic evaluation framework.

Critical strengths

1.A relevant and appealing motivation: The paper draws attention to the emotional and behavioral believability of generated characters, an aspect that matters for interactive storytelling and virtual characters even when appearance and motion are convincing.

  1. Clear task decomposition and human validation. The benchmark separates event execution, state changes, and cross-turn continuity, making failures easier to diagnose. Human and VLM score rankings show positive correlations, supporting the evaluator’s ability to capture relative quality.
Critical weaknesses
  1. Limited scope and incremental contribution: The motivation is appealing, but the technical contribution mainly consists of specialized tasks and scoring checklists built on existing video evaluation approaches.

  2. Insufficient validation of VLM-based scoring. Although the reported rank correlations with human judgments are encouraging, they do not fully establish reliable assessment of fine-grained emotion, intent, and temporal changes.

Questions for authors

I have no further questions for the authors. My main concern is that the scope and technical contribution are insufficient for acceptance, rather than a lack of clarity.

3 comments

As the discussion period is drawing to a close, we would like to kindly follow up and hope that our detailed responses and additional experiments have adequately addressed your remaining concerns. If there are any further questions or aspects that require clarification, we would greatly appreciate your feedback and would be more than happy to provide additional explanations.

Thank the authors for the detailed responses addressing my two major concerns. The clarifications have improved my understanding of the benchmark’s application scope and reliability, and I am now more confident about this work. I will raise my score accordingly and have no further questions.

We sincerely thank you for your careful consideration of our responses. We are especially grateful for the time and effort you have devoted to reviewing our work, and deeply appreciate your constructive feedback and support.

Reviewer_YtzsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides a new benchmark designed to evaluate video generation model's performance of controlling and preserving mental state during multi-turn transitions. It contributes 3 kinds of tasks: trigger execution, complete transitions and multi-turn continuations. It evaluates 7 different kinds of video gen models and shows that most of them still have problems with keeping mental-state updated or consistent, especially across multiple turns.

Critical strengths

Instead of physical and visual consistency, the paper shifts the focus to the mental state transition of characters across multiple turns. The topic is actually a blind spot that not many works focus on in the current video gen community, but it is actually something end-users will notice at the first second when the consistency or continuity of mental state on characters is broken.

Critical weaknesses
  1. The size of the dataset is relatively small with hundreds of prompt, which makes it hard to contribute to future training of video gen models to improve their ability on mental states.
  2. Within evaluation of human, the Krippendorff's alpha is really low. Although the agreement within one category on the original 0/1/2 scale is high, but considering that there are only 3 choices, this metric might be inflated and cannot be a convincing proof to show that humans can agree on most of the tasks. Given that, it is concerning that we use an LLM (Gemini) as the judge for those subjective tasks, it can be unreliable and the benchmark's results can be questionable under that.
Questions for authors
  1. Given the low agreement among human raters, what is a foreseeable path for us to trust the reliability of the LLM-based automatic evaluator?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.