- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Mind-VBench, a diagnostic benchmark for evaluating whether video generation models generate character behaviors consistent with specified mental states. Based on the BDI framework, the benchmark consists of 360 instances across 12 tasks covering beliefs, desires, and intentions. The authors evaluate representative video generation models in both text-to-video and image-to-video settings. They further investigate the effects of first-frame conditioning and prompt enhancement. Experimental results suggest that current video generation models struggle to establish valid scenarios and generate behaviors consistent with the intended mental-state constraints.
Critical strengths
- The paper investigates mental state consistent behavior in video generation, a capability that is not sufficiently studied by previous benchmarks.
- The BDI-based taxonomy provides a systematic organization of 12 diagnostic tasks covering different aspects of character behavior.
- The paper evaluates a diverse set of open-source and closed-source video generation models under both T2V and I2V settings. The additional prompt enhancement experiments provide useful insights into how explicit behavioral instructions affect generation performance.
Critical weaknesses
My main concern is about the design of the taxonomy and benchmark.
-
Some tasks seem only loosely connected to BDI reasoning. For example, the Multi-agent Intention category appears to primarily evaluate physical or social interactions rather than reasoning about agents' mental states. Similarly, Counterfactual Rule seems closer to rule following than belief reasoning.
-
Some benchmark instances do not clearly reflect their assigned categories. For example, the Social Conformity task in Appendix Figure 6 prescribes a character for continuing toward the reception desk as the expected behavior, rather than following a crowd taking photos. This seems more closely related to Goal Persistence than Social Conformity.
-
The benchmark relies on predefined behavioral criteria, but multiple behaviors can be reasonable given the same scenario. For example, in the aforementioned Social Conformity case, briefly stopping to observe the installation could also be a reasonable personal choice. Penalizing such behaviors may conflate individual behavioral variability with failures in BDI reasoning.
Overall, I think it's necessary to disclose more details on the benchmark curation process, especially on the rationale behind each design choice.
Beyond the benchmark itself, the evaluation seems conducted on only one generated video per instance, despite the inherent randomness of video generation. Without repeated sampling or variance analysis, it is unclear whether the reported scores and model rankings are robust to different random seeds, which could significantly affect the paper's conclusions.
Besides, while the paper evaluates a range of video generation models, the experimental analysis remains relatively limited. Most results in the appendix are detailed breakdowns or visualizations of the main results, rather than additional controlled experiments. More in-depth analysis would be helpful to better understand the sources of model failures and validate the effectiveness of the proposed benchmark.
Questions for authors
See weakness.
Response to Reviewer AGdz
Dear Reviewer AGdz,
Thank you for your constructive comments focusing on the taxonomy, design principles, and evaluation methodology of Mind-VBench. We address each point below with conceptual clarifications, protocol refinements, and newly computed experimental evidence.
Weakness 1: Taxonomy Design & Instance Alignment
Weakness 2: Single-Seed Generation & Variance
While evaluating across 360 diverse scenarios aggregate-averages instance-level randomness across model conditions, we acknowledge the importance of seed-level stability. In our newly completed human-validation study (N=120, 5 models), the evaluator’s ranking demonstrates strong correlation with human consensus (Spearman’s ρ=0.700, Pearson’s r=0.667, Kendall’s τ=0.600).
Weakness 3: Controlled Experiments & Diagnostic Analysis
We emphasize that Prompt Enhancement (PE) operates as a controlled diagnostic ablation: by supplying the explicit continuation, it isolates physical action rendering from mental-state inference.