- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MentalWM-Bench evaluates whether video generators realize event-driven changes in characters’ emotions and intentions and preserve them across interactions. It includes trigger-only, complete-transition, and two-turn continuation tasks. Experiments across seven generators, supported by human comparisons, identify failures that endpoint correctness and visual fidelity alone can miss.
Critical strengths
The benchmark makes a useful distinction between showing the right outcome, realizing the transition that leads to it, and preserving that state into the next interaction. The evaluation covers seven generators and balanced individual, dyadic, and group source scenarios. The authors also make a substantial effort to check their conclusions through human judgments, source-case bootstrap intervals, raw-versus-adjusted scoring, and conditional analyses. In particular, they explicitly show that most low change scores co-occur with incomplete events or endpoints, rather than presenting the entire gap as evidence of a separate reasoning failure. This makes the empirical analysis more informative and appropriately scoped.
Critical weaknesses
The main concern I have is about the reliability of the core measurements. The human study is valuable, but Appendix C reports ordinal agreement of only 0.297, 0.299, and 0.368 for emotion change, intent change, and boundary continuity. The headline single-turn correspondence concerns a three-dimension proxy, while Table 10 evaluates the original event/action checks rather than the strict Social-8 rubric. Therefore, it remains unclear how closely the main mental-state scores and model rankings agree with humans on the same criteria.
Another issue is that part of the transition-score gap follows from the scoring construction. Consistency adjustment reduces emotion-change scores from 0.547 to 0.365 and intent-change scores from 0.464 to 0.303, and change checks require more prerequisites than endpoint checks. The authors acknowledge this and provide useful conditional analyses. However, controlled temporal perturbations would make the benchmark’s added diagnostic value clearer, especially because the residual process deficits after full-credit prerequisites are relatively small.
The multi-turn comparison is also somewhat limited. The 24 cases are selected according to Seedance’s continuation-interface availability, and durations and interfaces are not fully matched. The five-turn pilot additionally includes only completed chains and excludes unsuccessful requests. These limitations are disclosed, but they still make it difficult to separate state-preservation quality from interface differences and completion selection when interpreting cross-model results.
In terms of construct validity, the complete and continuation tasks specify the target responses in advance. The paper correctly frames this as trajectory realization rather than open-ended response inference. Still, a matched comparison with ordinary behavioral or temporal instructions would help establish what the mental-state formulation adds beyond fine-grained prompt adherence. The distinction matters when interpreting the benchmark as an assessment of mental world modeling.
Finally, the evaluation records need a clearer accounting. The paper reports 455 task ratings, while the stated distribution of 243 multi-rated slots implies 738 slot-level rater assignments. These may refer to different units, but the relationship is not explained. A compact accounting table, together with exact generator/evaluator versions, frame sampling, durations, and continuation settings, would make the reported results easier to verify from the manuscript.
Questions for authors
1.Could the authors report matched human–AI correspondence and model-ranking stability specifically for the strict Social-8 criteria and boundary-state preservation, rather than relying on the three-dimension proxy or original event/action checklist? 2.Could the authors test controlled temporal errors, such as an early response or swapped event–response order, while keeping the visible endpoints unchanged? Reporting raw and adjusted scores on these cases would help validate temporal sensitivity. 3.Could the authors provide an interface-matched continuation subset and report all attempted five-turn chains, including failure counts and reasons? This would help distinguish preservation failures from generation or interface failures. 4.Could the authors compare matched prompts using mental-state labels versus explicit behavioral descriptions, while preserving the requested sequence? This would clarify the contribution beyond ordinary temporal prompt following. 5.Could the authors reconcile the human-rating counts and provide a single table mapping each protocol to its videos, raters, scoring rules, and exact generation/evaluation settings?
4 comments
Following our first response, we address your questions about multi-turn comparisons, mental-state formulations, and experimental records. Your suggestions helped us broaden task coverage and clarify the evidence provided by each experiment.
3. Multi-turn expansion, continuation configurations, and long-chain records
We added 555 task prompts: 405 single-turn prompts and 150 second-turn continuation prompts, balanced across individual, dyadic, and group interactions. Continuations reuse the corresponding single-turn videos as their first turns, avoiding double-counting. Together with the original 471 prompts, the expanded benchmark contains 1,026 task prompts.
On this extension, we completed generation and evaluation of 81 additional tasks per model, comprising 54 single-turn and 27 continuation tasks. The main conclusions remained consistent with the original experiments. This provides empirical support on additional samples and broadens the tested continuation tasks.
Your interface-matching suggestion provides a clear direction for improving interpretability across models. In the revision, we will control continuation conditions, target durations, and evaluation windows for models supporting common settings, reporting matched-setting results separately from results under their original configurations. The existing conditional analysis already provides a direct observation: only 52 of 85 samples with full-credit first-turn resulting states receive full credit for boundary preservation. Thus, realizing a first-turn outcome and retaining it during subsequent interaction are requirements that warrant separate assessment.
For long-chain records, we will document planned turns, completed turns, evaluation inclusion, and request-failure reasons by model. The relevant Seedance experiments were planned for three to four turns and will be recorded separately from five-turn requests. We will also explicitly document the quality-informed inclusion rule for the additional Wan human-evaluation subset. This accounting will distinguish request completion from preservation quality in generated content and make the existing analysis of 37 completed chains easier to verify.
4. Mental-state labels versus explicit behavioral descriptions
Your proposed paired comparison would further clarify the specific demands of mental-state tasks. By supplying target responses, the benchmark fixes the trajectory to be realized and locates failures in event execution, response timing, and state preservation. Even when instructed to display fear after a threat, a model may display it prematurely or lose the achieved state during continuation. These failures directly affect behavioral continuity in interactive narratives and virtual characters.
Mental-state formulations also allow a target to be expressed through different observable behaviors, whereas explicit behavioral descriptions impose more specific action requirements. Their paired comparison can therefore help distinguish realizing a target state behaviorally from executing specified actions. We plan to hold characters, events, temporal order, and target duration constant, evaluate both formulations with common observable criteria, and control differences in descriptive specificity and information content.
The benchmark’s value lies in translating state updates and cross-turn dependencies into explicit, testable behavioral requirements. These requirements relate to instruction following while providing diagnosis of failure stages that an overall adherence score cannot directly locate.
5. Human-evaluation records and experimental settings
The human evaluation involved 15 raters. Following your suggestion, the revision will consolidate sample coverage, participant counts, rating units, and aggregation methods for the original human study, supplementary reassessment, and new blinded evaluation. We will verify the associated counts and their relationships against the original records.
Exact generator and evaluator versions, frame sampling, durations, and continuation settings will also be consolidated into a protocol table, clearly distinguishing original and additional experiments. These changes will make the expanded data, direct human validation, and diagnostic analyses easier to verify. Thank you for your detailed and constructive assessment.
I thank the authors for the additional evaluation. However, I prefer to retain my original score. The matched human study is helpful, but continuation comparisons remain limited by differences in settings, and annotation counts still need clarification.
Thank you for considering the additional evaluation and finding the matched human study helpful. We have verified the annotation accounting and would like to clarify the scope of the continuation evidence.
Annotation counts. In the original human study, 15 raters provided 455 rater–task submissions containing ratings. Of these, 352 submissions contain ratings for two video slots and 103 contain ratings for one, yielding 807 rater–video-slot rating records. The inter-rater analysis uses the 243 slots rated by at least two raters, accounting for 738 records; the remaining 69 records belong to singly rated slots. Thus, both 455 and 738 are reproducible from the original study records: they refer to different counting units and subsets. We will replace the ambiguous term “task ratings” with “rater–task submissions” and explicitly document the counting units, inclusion criteria, and source-record version.
Continuation evidence. We agree that configuration differences matter when interpreting cross-model comparisons. Our diagnostic conclusion is that achieving a first-turn state and preserving it during continuation warrant separate assessment. The original model-specific results in Appendix D.1, Table 6 support this distinction: among samples with full-credit first-turn resulting states, 10/21 for Seedance 2 and 15/22 for Kling 3 receive full boundary-preservation credit. These are observations under each model’s evaluated continuation configuration, rather than estimates isolating model capability from interface effects.
We have also completed re-annotation and rescoring under a unified annotation and scoring protocol for the same evaluated subset of the 555-prompt extension described in our previous response: 54 single-turn and 27 continuation tasks per model. This concerns the previously reported additional samples, with consistent evaluation criteria across those samples; it does not imply that their generation interfaces were matched.
Thank you again for your careful assessment. We hope the verified accounting and this clarification make the supporting evidence and its interpretation more transparent.
Thank you for recognizing the value of the task distinctions and conditional analyses, and for suggesting concrete ways to strengthen validation. In response, we added a blinded human evaluation directly targeting the core criteria and broadened experimental coverage. This response addresses scoring reliability and temporal diagnosis; our second response discusses multi-turn evaluation, task interpretation, and reproducibility.
1. Strict Social-8, boundary-state preservation, and model comparisons
You suggested directly validating the mental-state criteria used in the main experiments rather than relying on the three-dimensional proxy or original event/action checks. We have completed an independent human evaluation under matched criteria.
We aligned the corresponding human and AI questions and scoring scales, specifying binary-item treatment and fractional-output mapping. AI scores were hidden from raters throughout. Strict Social-8 achieved 89.5% exact human–AI agreement across 1,010 valid entries from 132 videos, including 93.2% for emotion change and 82.3% for intent change. The temporal criteria achieved 93.9% for event–response order and 83.3% for no premature response. Separate boundary-state preservation validation yielded 95.8% agreement across 24 entries.
This addition directly covers the emotion changes, intent changes, temporal relationships, and cross-turn preservation you questioned, rather than relying only on composite-score correlations. The revision will present per-criterion results, applicable sample counts, and scoring protocols together, making the evidence for the core measurements more direct and verifiable.
For model comparisons, we also completed additional generation experiments comprising 54 single-turn and 27 continuation tasks per model, with the main conclusions remaining consistent with the original experiments. Building on this evidence, the revision will further compare model rankings obtained from human and automated aggregate scores on matched videos, with source-case bootstrap uncertainty, strengthening validation at both the item and model levels.
2. Scoring construction and temporal diagnostic value
Thank you for recognizing the role of the raw-versus-adjusted and conditional analyses. The existing evidence separately examines whether the dimension-level finding depends on post-processing and whether process checks add information when events and states are already realized.
Appendix D.2 shows that all seven generators have lower mental-state means than their respective other dimensions before consistency adjustment. Conditioning on full-credit initial states, resulting states, and triggering events, Appendix D.4 identifies 17/357 emotion-related and 6/255 intent-related cases with remaining process shortcomings. The original multi-rater analysis provides complementary support: in Appendix D.3, 10 of 83 slots passing event/action outcome checks still fail a process check. The argument for supplementary process assessment therefore draws on scoring-rule sensitivity, conditional diagnosis, and human evidence.
Your proposed controlled perturbations would make this diagnostic value more explicit. We plan to construct paired cases preserving visible initial and final states and the main visual content while introducing premature responses or swapped event–response order. We will report endpoint scores, raw temporal scores, and adjusted temporal scores, while checking for visual artifacts introduced by editing. This will directly test whether temporal criteria detect intermediate-process errors when endpoint content remains correct.
Thank you for helping us connect the existing diagnostic findings more closely with validation targeting specific error types.