acceptodds
Under review as a conference paper at ICLR 2027

Mind-VBench: A Diagnostic Benchmark for BDI-Consistent Behavior in Video Generation

Abstract

Recent video generators produce increasingly realistic and coherent videos, yet visually plausible videos may still depict characters acting inconsistently with the information available to them or with the goals they pursue. We study this discrepancy by evaluating whether generated character behavior remains consistent with the mental states established by a scenario. To make this question systematically evaluable in video generation, we introduce Mind-VBench, a diagnostic benchmark that operationalizes the Theory of Mind (ToM) question as observable behavioral constraints, using the Belief–Desire–Intention (BDI) framework from cognitive science and agent modeling. It contains 360 instances organized into three BDI components, six diagnostic dimensions, and twelve tasks. We evaluate representative video generation models under text-to-video and image-to-video settings. Current models frequently fail both to establish the situation required for behavioral judgment and to generate character behavior consistent with the specified BDI constraints. These findings show that current video generators do not reliably maintain mental-state consistency in character behavior, highlighting a limitation for human-centered world simulation. Our demo is available in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
Video Generation, Theory of Mind, Belief-Desire-Intention (BDI), Behavioral Consistency, Mental State Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AGdzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mind-VBench, a diagnostic benchmark for evaluating whether video generation models generate character behaviors consistent with specified mental states. Based on the BDI framework, the benchmark consists of 360 instances across 12 tasks covering beliefs, desires, and intentions. The authors evaluate representative video generation models in both text-to-video and image-to-video settings. They further investigate the effects of first-frame conditioning and prompt enhancement. Experimental results suggest that current video generation models struggle to establish valid scenarios and generate behaviors consistent with the intended mental-state constraints.

Critical strengths
  1. The paper investigates mental state consistent behavior in video generation, a capability that is not sufficiently studied by previous benchmarks.
  2. The BDI-based taxonomy provides a systematic organization of 12 diagnostic tasks covering different aspects of character behavior.
  3. The paper evaluates a diverse set of open-source and closed-source video generation models under both T2V and I2V settings. The additional prompt enhancement experiments provide useful insights into how explicit behavioral instructions affect generation performance.
Critical weaknesses

My main concern is about the design of the taxonomy and benchmark.

  1. Some tasks seem only loosely connected to BDI reasoning. For example, the Multi-agent Intention category appears to primarily evaluate physical or social interactions rather than reasoning about agents' mental states. Similarly, Counterfactual Rule seems closer to rule following than belief reasoning.

  2. Some benchmark instances do not clearly reflect their assigned categories. For example, the Social Conformity task in Appendix Figure 6 prescribes a character for continuing toward the reception desk as the expected behavior, rather than following a crowd taking photos. This seems more closely related to Goal Persistence than Social Conformity.

  3. The benchmark relies on predefined behavioral criteria, but multiple behaviors can be reasonable given the same scenario. For example, in the aforementioned Social Conformity case, briefly stopping to observe the installation could also be a reasonable personal choice. Penalizing such behaviors may conflate individual behavioral variability with failures in BDI reasoning.

Overall, I think it's necessary to disclose more details on the benchmark curation process, especially on the rationale behind each design choice.

Beyond the benchmark itself, the evaluation seems conducted on only one generated video per instance, despite the inherent randomness of video generation. Without repeated sampling or variance analysis, it is unclear whether the reported scores and model rankings are robust to different random seeds, which could significantly affect the paper's conclusions.

Besides, while the paper evaluates a range of video generation models, the experimental analysis remains relatively limited. Most results in the appendix are detailed breakdowns or visualizations of the main results, rather than additional controlled experiments. More in-depth analysis would be helpful to better understand the sources of model failures and validate the effectiveness of the proposed benchmark.

Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer AGdz

Dear Reviewer AGdz,

Thank you for your constructive comments focusing on the taxonomy, design principles, and evaluation methodology of Mind-VBench. We address each point below with conceptual clarifications, protocol refinements, and newly computed experimental evidence.

Weakness 1: Taxonomy Design & Instance Alignment

  • Counterfactual Rule as Belief Reasoning: In BDI cognitive architecture, Belief represents an agent's internal model of environmental rules and states. The Counterfactual Rule task is specifically designed to test whether an agent can update its internal belief to reflect a non-standard world rule and act accordingly, rather than defaulting to real-world physical priors. To prevent misinterpretation as generic "rule compliance", we will frame this construct explicitly as Counterfactual Belief in the revised manuscript.
  • Multi-agent Intention (Nested Intentions): Grounded in Theory of Mind (ToM) frameworks, spatial and social coordination requires agents to model and respond to other agents' mental states (e.g., inferring a peer's intended trajectory to yield at a narrow threshold, or synchronizing complementary actions). Pure physical collision avoidance does not satisfy our criteria; the benchmark evaluates interaction structures that necessitate intention attribution.
  • Instance Alignment (Figure 6): The mismatch noted in Figure 6 was caused by an outdated showcase example: the card retained an exploratory scenario from an early version of this task and was not updated in time after the task construct was finalized. In our benchmark protocol, instances for Social Conformity are evaluated strictly on whether the agent aligns with group behavior, in accordance with Table 1. We have updated Figure 6 with a prototypical conformity example.
  • Behavioral Variability (Figure 4): For Perspective Difference, the original criterion over-specified immediate forward motion. We have made targeted revisions to the relevant parts of the evaluation protocol: the criterion is supplemented with an explicit clause that "behavior must not rely on privileged information", rewarding plausible exploratory actions (e.g., stopping and lowering an obstructing box).

Weakness 2: Single-Seed Generation & Variance

While evaluating across 360 diverse scenarios aggregate-averages instance-level randomness across model conditions, we acknowledge the importance of seed-level stability. In our newly completed human-validation study (N=120N=120, 5 models), the evaluator’s ranking demonstrates strong correlation with human consensus (Spearman’s ρ=0.700\rho = \mathbf{0.700}, Pearson’s r=0.667r = 0.667, Kendall’s τ=0.600\tau = 0.600).

Weakness 3: Controlled Experiments & Diagnostic Analysis

We emphasize that Prompt Enhancement (PE) operates as a controlled diagnostic ablation: by supplying the explicit continuation, it isolates physical action rendering from mental-state inference.

  • Crucially, in our newly completed 120-video human evaluation, human annotators independently confirm that PE yields a +14.2+14.2 pp gain in OSR (31.1%31.1\% vs. 16.9%16.9\%). This diagnostic gap directly decouples "failure to infer mental constraints" from "failure to execute video rendering", confirming the diagnostic utility of the benchmark.

Reviewer_5FEiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Mind-VBench evaluates whether characters in generated videos act consistently with specified beliefs, desires, and intentions. It contains 360 instances across twelve tasks and uses a two-stage protocol that first checks scene validity and then assesses behavioral consistency. The authors evaluate eight video generators under text-to-video and image-to-video settings, with additional Prompt Enhancement experiments where applicable. The results distinguish failures to establish the required scene from failures to generate appropriate subsequent behavior. The contribution lies primarily in the BDI-focused task design and diagnostic evaluation protocol.

Critical strengths
  1. The paper addresses a useful evaluation problem: visually plausible videos can still show characters acting inconsistently with their information, preferences, or goals. The BDI framework provides a structured way to translate these requirements into observable behavioral constraints. Section 2 and Appendix A.1 explain the intended distinctions from related benchmarks.

  2. The two-stage protocol provides an interpretable distinction between scene-establishment and behavioral failures. The examples in Figure 3 illustrate this diagnostic use, and the experiments across models and generation conditions provide useful observations. The authors appropriately acknowledge that observable BDI consistency does not establish internal mental-state representations and that Prompt Enhancement supplies an explicit behavioral target.

Critical weaknesses

1.The human-validation sample is too limited to establish the reliability of the automatic evaluation across the benchmark. Section 4.3 and Appendix D report 20 distinct videos covering twelve tasks, with 12 annotators per video. Multiple annotations help establish the reference labels, but the number of distinct videos remains small relative to the diversity of tasks and model outputs.

2.Table 5 reports 55% final three-way agreement (Cohen's κ = 0.274) and 55% scene-validity accuracy (κ = 0.211) for the selected Qwen3.5-27B evaluator. The 83.3% behavioral accuracy is explicitly conditional on videos that the automatic evaluator considers valid. It therefore provides evidence about that subset, rather than establishing agreement for the complete evaluation pipeline.

3.These results do not demonstrate that the benchmark's conclusions are incorrect. However, the limited sample and observed agreement leave uncertainty about the reliability of the reported scores and their diagnostic decomposition. A larger, representative human-validation sample covering tasks, generation conditions, and failure types would strengthen the central empirical claims.

Questions for authors

Is additional human-validation evidence available beyond the 20-video sample in Section 4.3? In particular, can a larger, representative sample establish agreement for both scene validity and final Success decisions, and assess whether the main model and condition comparisons remain consistent under human evaluation? This request concerns validation of the existing scoring protocol; it does not require rerunning the full generation benchmark.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5FEi

Dear Reviewer 5FEi,

We sincerely thank you for your focused, constructive, and highly actionable assessment. In particular, we deeply appreciate your clarification that verifying the existing scoring protocol would substantiate our claims without requiring the full generation benchmark to be rerun.

Following your suggestion, we conducted an extended human validation study during the rebuttal period. Below, we report the complete empirical findings, which address your requests regarding (a) a larger, representative sample, (b) agreement at both the scene-validity and final-decision stages, and (c) the consistency of model comparisons under human judgment.

1. Extended Human Validation Setup (N=120N = 120, strat120)

  • Sample Design: We sampled N=120N = 120 distinct videos using a stratified design across all 12 tasks (1010 videos per task, fixed seed), covering five representative video generators (HunyuanVideo, LongCat, LTX-2.3, Wan2.2-T2V, Wan2.2-I2V). The sample is balanced across generation conditions (6464 T2V vs. 5656 I2V; 5959 base vs. 6161 Prompt-Enhanced).
  • Annotators & Consensus: Each video was independently evaluated by 5 human annotators (600 total annotations). Inter-annotator agreement among human raters reached Krippendorff’s α=0.455\alpha = 0.455 for scene validity and α=0.439\alpha = 0.439 for the final three-way decision (Fleiss' κ=0.454\kappa = 0.454 and 0.4380.438). In the context of open-ended, multi-frame video behavior evaluation, this represents moderate and stable agreement, where 5-person majority voting effectively mitigates individual subjective variance to establish a robust consensus label.

2. Agreement on Scene Validity and Final Success Decisions

Evaluating the human majority consensus against our flagship evaluator (GPT-5.6) demonstrates solid alignment across both evaluation stages:

Scene Validity (2×22 \times 2 Confusion Matrix):

Evaluator ValidEvaluator Invalid
Human Valid36 (TP)17 (FN)
Human Invalid5 (FP)62 (TN)
  • Validity Accuracy: 81.7%81.7\%, Cohen’s κ=0.619\kappa = \mathbf{0.619} (95%95\% CI [0.47,0.76][0.47, 0.76]), indicating substantial agreement.
  • Error Directionality: Disagreements are predominantly false negatives (FN=17\text{FN} = 17) rather than false positives (FP=5\text{FP} = 5). The evaluator behaves conservatively, admitting only 5 invalid scenes into subsequent behavioral evaluation, which addresses the concern regarding downstream metric contamination.

Final Three-Way Decisions (3×33 \times 3 Confusion Matrix: Invalid / Fail / Success):

Evaluator InvalidEvaluator FailEvaluator Success
Human Invalid6232
Human Fail8151
Human Success9020
  • Exact Three-Way Agreement: 80.8%80.8\%, Cohen’s κ3=0.655\kappa_3 = \mathbf{0.655} (95%95\% CI [0.53,0.78][0.53, 0.78]).
  • Behavioral Decision Accuracy: On the consensus-valid subset (n=36n=36), behavioral agreement between the evaluator and human consensus is 97.2%97.2\% (35/36). Under the broader human-valid denominator (n=53n=53), behavioral accuracy is 66.0%66.0\%, where the difference is almost entirely accounted for by the evaluator's conservative validity screening (FN=17\text{FN}=17), rather than behavioral misclassification.

3. Stability of Model Comparisons and Benchmark Findings

  • Model Ranking Consistency: Across the evaluated models, the rankings derived from human consensus labels correlate strongly with those generated by the automatic evaluator (Spearman’s ρ=0.700\rho = \mathbf{0.700}, Pearson’s r=0.667r = 0.667, Kendall’s τ=0.600\tau = 0.600). This empirical correlation indicates that the automated evaluator serves as a reliable proxy for assessing relative model capabilities.
  • Replication of Prompt Enhancement Effect: Under human ground-truth labels, Prompt Enhancement increases the Overall Success Rate (OSR) from 16.9%16.9\% to 31.1%31.1\%—a gain of +14.2+14.2 percentage points (matching the evaluator's positive trend, with even stronger human-observed magnitude). This result supports the benchmark's core finding that current model bottlenecks stem predominantly from mental-state inference rather than visual rendering.

4. Dual Evaluation Pipeline in the Revised Version

To balance reproducible open-source accessibility with high-precision evaluation, the revised manuscript reports the extended agreement study above (120 stratified videos, 80.8%80.8\% exact agreement for GPT-5.6), and the complete dual-evaluation figures under GPT-5.6 will be released with the open-source version of the benchmark.

We hope these empirical results directly resolve your reservations regarding evaluator validity. Thank you again for your constructive guidance.


Thank you for the additional experiments and clarifications. The expanded human-validation study alleviates my main concern about the validation of the automatic evaluator. I am raising my rating to 3 .

Reviewer_Bt5aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mind-VBench, a benchmark for evaluating whether characters in generated videos behave consistently with scenario-established beliefs, desires, and intentions. The benchmark contains 360 instances across twelve tasks and evaluates both text-to-video and image-to-video generation. Its evaluation protocol separates scene validity from behavioral success using four prerequisite checks and an instance-specific behavioral criterion, reporting validity rate, conditional success rate, and overall success rate. Experiments compare open- and closed-source video generators, with additional prompt-enhanced conditions that explicitly provide the expected continuation. The reported results suggest that first-frame conditioning mainly improves scene establishment, while explicit behavioral guidance yields further gains in successful behavior.

Critical strengths
  1. The benchmark addresses a meaningful aspect of video generation. Character-specific information access, preferences, and goal persistence are important for coherent narratives and human-centered simulation. Tasks involving unobserved changes and restricted perspectives provide concrete ways to assess whether generated actions respect the information available to a character.

  2. The evaluation decomposition offers useful diagnostic structure. Distinguishing an invalid scenario from a valid scenario with unsuccessful behavior can make failures more interpretable than a single aggregate score. The complementary first-frame and prompt-enhanced conditions also help investigate how scene initialization and explicit action guidance affect performance. The paper appropriately acknowledges that prompt enhancement is an assisted condition and does not establish stronger mental-state reasoning.

Critical weaknesses
  1. The automatic evaluator is insufficiently validated for the benchmark’s central conclusions. Although Section 4.3 highlights 83.3% behavioral accuracy for Qwen3.5-27B, Table 5 reports only 55.0% agreement on the final three-way labels (κ = 0.274) and 55.0% accuracy on scene validity (κ = 0.211). Validity is a prerequisite for subsequent behavioral evaluation, so errors at this stage can affect all three reported metrics. Furthermore, behavioral accuracy is computed on videos judged valid by each automatic evaluator, potentially comparing judges on different subsets. The treatment of cases judged valid by the evaluator but invalid by humans is unclear. The alignment study contains only twenty distinct videos across twelve tasks and different model groups; twelve annotations per video do not increase this coverage. Consequently, the current evidence does not adequately establish that model comparisons and the reported failure decomposition are robust to evaluator errors.

  2. Some instance-level criteria do not clearly match the intended behavioral constructs. Table 1 and Appendix A.3.2 define Social Conformity as aligning behavior with relevant actions of other agents. However, the SC example in Figure 6 rewards continuing toward reception without joining the filming crowd and explicitly rejects joining the crowd. This appears closer to goal persistence than conformity, leaving unclear whether the discrepancy is a presentation error or part of the actual evaluation. In Figure 4, the Perspective Difference criterion also appears to require forward movement before acquiring a new view; it is unclear whether a reasonable alternative, such as stopping and lowering the box to inspect the route, would receive credit. These examples raise concerns that some criteria reward a particular continuation rather than the broader set of behaviors consistent with the stated constraints. This matters directly for interpreting task-level scores and the benchmark’s diagnostic claims.

Questions for authors
  1. Could you provide the sample counts underlying each judge’s behavioral accuracy, the complete three-way confusion matrices, and human inter-annotator agreement? How are disagreements about validity handled when computing behavioral accuracy? Do the main model comparisons remain stable under independent human evaluation on a larger, stratified sample?

  2. Could you reconcile the SC definition with the success criterion in Figure 6 and clarify whether that criterion was used in the reported experiments? For the PD example in Figure 4, are information-seeking actions—such as lowering the box before proceeding—accepted as valid alternative continuations? How broadly were the evaluation specifications audited for such issues?

  3. Do you have matched controls that preserve scene and action complexity while changing only a character’s information access or preference? Such results would help establish whether the benchmark is sensitive to the intended character-specific constraints rather than fixed action preferences or general generation difficulty.

  4. Appendix E.1 reports only 183 returned Seedance I2V videos, compared with 359 T2V videos. How are missing generations incorporated into the metrics, and do the T2V/I2V conclusions hold when evaluated on the same set of successfully returned instances?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Bt5a (Part 1/2)

Dear Reviewer Bt5a,

Thank you for your thorough critique and actionable requests. Below, we address each of your questions with newly collected empirical validation data, code-level audits, and recomputed subsets.

Response to Weakness 1 & Question 1 (Evaluator Validation & Error Analysis)

1. Sample Counts and Protocol Handling in Table 5: We audited our evaluation codebase to clarify the metrics reported in Section 4.3:

  • The reported behavioral accuracy of 83.3%83.3\% was computed on the 1212 videos that Qwen3.5-27B itself classified as valid (nmv=12n_{\text{mv}} = 12), representing a success agreement of 10/1210/12.
  • Handling of cases judged valid by evaluator but invalid by humans: Among these 12 videos, 9 were judged invalid by the human majority. In our original script, these instances remained in the denominator with their ground-truth success label set to 0. The 2 instances where the evaluator predicted Success were scored as errors, while the 7 instances where the evaluator predicted Fail mapped to 0 (matching the human 0). As the reviewer noted, comparing evaluators on their respective self-selected valid subsets (nmv=12/6/16/5/16n_{\text{mv}} = 12/6/16/5/16) creates non-comparable denominators. We have documented this protocol detail explicitly in the revision.

2. Extended Human Validation on Stratified Sample (N=120N = 120, strat120): During the discussion period, we completed an expanded human validation study spanning N=120N = 120 videos (10 videos per task across all 12 tasks, covering 5 generators, balanced across conditions; evaluated by 5 independent annotators per video, with human inter-annotator agreement at α=0.455\alpha = 0.455 for validity and α=0.439\alpha = 0.439 for 3-way labels):

  • Validity Agreement (2×22 \times 2 Confusion Matrix): [TP=36, FN=17] / [FP=5, TN=62]. Validity Accuracy = 81.7%81.7\%, Cohen’s κ=0.619\kappa = \mathbf{0.619} (95%95\% CI [0.47,0.76][0.47, 0.76]).
  • Error Direction: Validity disagreements are dominated by False Negatives (1717) rather than False Positives (55). The evaluator is conservative, admitting only 5 invalid scenes into subsequent behavioral evaluation.
  • Final Three-Way Agreement (3×33 \times 3 Confusion Matrix): [[62, 3, 2], [8, 15, 1], [9, 0, 20]]. Exact 3-Way Agreement = 80.8%80.8\%, Cohen’s κ3=0.655\kappa_3 = \mathbf{0.655} (95%95\% CI [0.53,0.78][0.53, 0.78]).
  • Behavioral Accuracy: On the consensus-valid set (n=36n=36), accuracy is 97.2%97.2\% (35/36). Under the unified human-valid denominator (n=53n=53), it is 66.0%66.0\%, where the drop is attributable to the evaluator's strict validity filter (1717 FNs).
  • Model Ranking Stability: The ranking derived from human labels strongly correlates with the automatic evaluator across models (Spearman’s ρ=0.700\rho = \mathbf{0.700}, Pearson’s r=0.667r = 0.667, Kendall’s τ=0.600\tau = 0.600).
AuthorsOpenReview ↗

Response to Reviewer Bt5a (Part 2/2)

Response to Weakness 2 & Question 2 (Criteria vs. Constructs Alignment)

1. Figure 6 (Social Conformity) Clarification: The discrepancy noted in Figure 6 stems from an outdated showcase example: the card still displayed an exploratory scenario from an early version of this task—one that prioritized goal persistence under weak social distractions—and was not refreshed in time after the task construct was finalized. In our benchmark, Social Conformity has always been scored strictly according to Table 1 (rewarding alignment with group activity). We have updated the Figure 6 showcase card with an unambiguous, prototypical conformity instance.

2. Figure 4 (Perspective Difference) & Criteria Flexibility: For Perspective Difference (PD), we agree that the initial wording over-specified immediate forward motion. We have made targeted revisions to the relevant parts of the evaluation protocol: the PD criterion is supplemented with an explicit clause that "behavior must not rely on privileged/unseen information", crediting cautious information-seeking actions (such as lowering an obstructing box) as valid alternative continuations.

Response to Question 3 (Matched Controls)

While Mind-VBench emphasizes scenario diversity, Prompt Enhancement (PE) functions directly as a controlled diagnostic ablation: it holds the visual scene and complexity constant while isolating mental-state inference. In our 120-video human validation, humans independently confirm that PE yields a +14.2+14.2 pp gain in OSR (31.1%31.1\% vs. 16.9%16.9\%). This verifies that model failures stem primarily from mental-state reasoning rather than general generation difficulty.

Response to Question 4 (Seedance I2V Missing Generations)

The missing generations in Dreamina-Seedance-2.0 I2V (183/360183/360 returned) resulted from commercial API-side safety/content moderation filtering. Metrics use only returned videos in denominators (excluded, rather than scored as zero), as documented in Appendix E.1.

We recomputed the metrics strictly on the 183 common instances successfully returned in both T2V and I2V:

  • Full Set (359359 vs. 183183): T2V OSR = 53.8%53.8\%, I2V OSR = 48.1%48.1\% (Δ=5.7\Delta = 5.7 pp)
  • Common Set (183183 vs. 183183): T2V OSR = 51.4%51.4\%, I2V OSR = 48.1%48.1\% (Δ=3.3\Delta = 3.3 pp)

The comparative direction and overall findings hold consistently on the common subset.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.