acceptodds
Under review as a conference paper at ICLR 2027

From Words to Worlds: Beyond Current Evaluation Focus, MLLMs Fail at Higher-Order Theory of Mind

Abstract

Theory of Mind (ToM), the ability to infer others' mental states from observable behavior, is fundamental to developing machines with human-level social intelligence. Despite the extensive evaluation dimensions for Multimodal Large Language Models (MLLMs), their ToM capabilities have not received sufficient attention. A comprehensive survey of ToM benchmarks reveals that while text-based ToM benchmarks for evaluating LLMs are abundant, multimodal ToM benchmarks remain scarce. Moreover, existing multimodal ToM benchmarks fail to incorporate classic ToM task paradigms from cognitive science, higher-order ToM, or visual observations better aligned with the real world. Compared to the ambiguous expressions often present in text-based ToM benchmarks, visual information is intuitive, unambiguous, and better aligned with embodied applications. In this paper, we present From Words to Worlds (FWW), a novel pipeline that enables the generation of videos from multi-scene script templates, constructing a video-based ToM benchmark named FWW-V. We further propose a rubric-based evaluation framework, FWW-EVAL, to systematically assess 14 advanced MLLMs on FWW-V. Our results reveal that current models perform poorly on ToM tasks, with a striking disparity relative to human performance. Notably, for certain question types, the majority of MLLMs fail entirely, achieving 0% accuracy.

Primary area:
datasets and benchmarks
Keywords:
Theory of Mind, Multi-scene Script-to-Video Generation, Rubrics-based Evaluation, Higher-Order

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xjF3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper constructs multi-scene scripts based on Sally-Anne narratives and uses Google Veo 3 to generate videos, resulting in the FWW-V dataset with 85 videos and 704 questions. The questions cover memory, real-world states, first- through fourth-order nested beliefs, transparent-barrier settings, and related scenarios. The paper also introduces a rubric-based evaluation framework, FWW-EVAL, which examines model reasoning at the perception, cognition, and conclusion levels instead of relying only on whether the final answer is correct. The authors evaluate 14 multimodal large language models and find that the models perform relatively well on basic factual questions, memory questions, and first-order omniscient-perspective questions. However, performance generally declines from single-room to continuous multi-room settings, and the models also struggle with the transparent-barrier tasks.

Critical strengths
  1. The research question is clearly defined. The paper notes that text-only Theory of Mind benchmarks are relatively abundant, while multimodal ToM benchmarks remain limited. A major issue with textual narratives is that it may be unclear whether a character actually observed a particular event. This work provides a more concrete testbed for studying how visual information affects belief inference.

  2. The task design has a basis in cognitive science. FWW-V follows the classic Sally-Anne paradigm and extends it to third- and fourth-order nested beliefs as well as transparent-barrier scenarios. Compared with questions that merely ask “Where does A think the object is?”, this design helps distinguish real-world states, other people’s beliefs, and recursive belief reasoning. In principle, it is closer to the complexity of real social interaction.

  3. The transparent-barrier setting is a useful diagnostic case. In this subset, which contains six questions, all 13 evaluated models scored 0%, while human accuracy averaged 80%. The authors also report that the models’ reasoning did not mention the persistent visual access provided by the transparent barrier. This suggests that the models may not reliably use characters’ observation conditions, although the exact source of failure still needs to be separated more carefully.

Critical weaknesses
  1. Synthetic data and filtering may introduce selection bias. The videos are generated by Veo 3 from manually written scripts and retained only after strict filtering. Keeping only unambiguous, logically consistent, and visually natural samples may favor videos with clear cues and controlled actions, while underrepresenting the noise, occlusion, and ambiguity of real videos.
  2. The sample sizes and statistical reporting for higher-order tasks are insufficient. Tables 2–4 do not clearly report the effective sample size or aggregation method for each question type. The high-complexity and continuous multi-room categories contain only six and eight videos, respectively, with multiple questions sharing the same events. Claims of a “significant gap” therefore need stronger statistical support.
  3. The human baseline is incompletely reported. The paper does not state the number of participants, their language background, or the evaluation duration, nor does it report model performance on the exact 106-question subset answered by humans. Since humans answered only a subset while models answered the full benchmark, the comparison is not fully matched. Table 2 also reports only three human values and does not cover all question types clearly.
  4. The claim that FWW-EVAL “eliminates confounding factors” is overstated. The rubric can reduce lucky guessing and superficial correctness, but cannot fully remove visual hallucinations, template priors, language effects, or video-generation artifacts. Since the script and video may not correspond exactly, perception-level errors may still be misjudged.
  5. Several difficulty factors are confounded. In the multi-room setting, video length, scene count, action complexity, and spatial transitions increase together. The performance decline may therefore reflect visual search, frame limits, temporal localization, or cross-scene memory rather than recursive belief reasoning. Likewise, transparent-barrier failures may indicate difficulty understanding glass partitions, not necessarily difficulty reasoning about belief states.
Questions for authors

Please carefully review the "Critical Weaknesses" and "Questions for Authors" sections; if the issues raised are addressed satisfactorily, I may consider increasing the score.

  1. How were the final 85 videos selected from the Veo 3 generation results? Could you provide the total number of generated videos for each category, the reasons for rejection, and the filtering criteria?

  2. Why were only Veo 3 synthetic videos used, without real recordings or videos from other generation models? Have you checked whether the conclusions on synthetic videos generalize to real videos?

  3. The benchmark combines higher-order recursive beliefs, visual accessibility, and long-video understanding. How do you separate failures caused by ToM reasoning from failures caused by visual perception, temporal localization, cross-scene memory, or limited context length?

  4. When moving from single-room to continuous multi-room settings, video length, scene count, action complexity, and spatial transitions all increase simultaneously. Did you run controlled experiments varying only video length, only scene count, or only belief order?

  5. All models scored 0% on the transparent-barrier tasks. Did you verify whether the models understand that the character can see through the glass even though they are physically separated, or whether they simply failed to notice the glass or room boundary?

  6. How does FWW-EVAL distinguish an incomplete reasoning description from genuinely incorrect reasoning? If a model omits intermediate steps but reaches the correct conclusion, will its response be judged incorrect?

  7. How many human participants were involved, what was their language background, and how long did the evaluation take? Did the participants watch the original videos, sampled frames, or textual descriptions?

Reviewer_YYvEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FWW (From Words to Worlds), a pipeline that generates videos from multi-scene script templates to construct a video-based Theory of Mind (ToM) benchmark called FWW-V. The authors also propose FWW-EVAL, a rubric-based evaluation framework that assesses MLLMs' reasoning processes across perception, cognition, and conclusion levels. Evaluating 14 MLLMs, the paper reports that even the strongest models, in their case GPT-5, fail substantially on higher-order false-belief ToM tasks, with a striking gap relative to human performance.

Critical strengths

This is a very interesting paper in the sense that bring Higher Order Theory of Mind cognitive science into the context of testing MLLMs. The From Words to Worlds pipeline, translating ToMi narratives into structured multi-scene scripts, instantiating placeholders under commonsense constraints, generating videos with Veo 3, and manually curating the results, is a clever and practical contribution. I am not aware of any other relevant works that take this type of video model in the loop way of evaluating theory of mind in MLLMs.

Critical weaknesses
  1. The size of the dataset is unfortunately small. FWW-V contains 85 videos and 704 questions. But the paper makes strong claims about the MLLMs which would be hard claims to be made given the limited size of the dataset.
  2. The success rate of the video generation is weirdly low. The retained videos are those that passed manual QA. This raises a concern: the benchmark may over-represent "easy-to-generate" scenarios that are visually unambiguous, potentially under-representing the diversity of real-world ToM situations.
  3. Veo 3 is a proprietary model. It's not clear if the experiments are reproducible
  4. The human participants information are extremely limited. i.e. inter-subject agreements numbers are not reported
  5. The paper did not report stats of confidence intervals etc.
Questions for authors

Can you provide confidence intervals for the main results, and clarify the number of questions per cell in Tables 2–4? For the 0% accuracy claims, how many questions are in the denominator? What is the inter-annotator agreement between the two human experts who labeled the trajectories? How were disagreements resolved?

Reviewer_TD2GOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the higher-order Theory of Mind (ToM) abilities of multimodal large language models (MLLMs) through video-based evaluation. The authors introduce From Words to Worlds (FWW), a pipeline that transforms structured multi-scene scripts into videos using generative video models. Based on this pipeline, they construct FWW-V, a benchmark comprising 85 curated videos and 704 questions covering false-belief reasoning, higher-order mental-state inference, complex interactions, and continuous multi-room scenarios.

To assess models beyond just the final-answer correctness, the authors further propose FWW-EVAL, a rubric-based framework that evaluates model responses along three dimensions: perception, cognition, and logical conclusion. The framework employs GPT-5 as an automated judge, with human annotations used to validate judgment reliability. Experiments on 14 MLLMs reveal substantial performance degradation on higher-order ToM tasks, particularly under false-belief and transparent-barrier scenarios. The results also highlight performance differences between open-source and closed-source models and discrepancies between answer accuracy and reasoning correctness.

Critical strengths
  1. The paper addresses a significant gap in existing MLLM evaluation by systematically examining higher-order Theory of Mind through video-based scenarios. Unlike conventional visual question-answering benchmarks, FWW-V incorporates recursive belief reasoning, complex character interactions, and continuous multi-room settings. The transparent-barrier scenarios are particularly interesting, as they require models to reason about visual accessibility rather than simply track character locations. The substantial performance degradation observed across multiple strong MLLMs, including the failure of all evaluated models on transparent-barrier tasks (Table 4), suggests that the benchmark captures challenging aspects of multimodal social reasoning.

  2. FWW-EVAL goes beyond conventional answer matching by explicitly evaluating perception, nested belief reasoning, and logical consistency. This is valuable because a correct final answer does not necessarily indicate that a model has correctly inferred the characters' mental states. The Diff-Ratio analysis provides additional evidence that answer correctness and reasoning validity can diverge. Furthermore, evaluating 14 MLLMs and comparing automated judgments against human annotations strengthens the empirical contribution. The reported misjudgment rates below 2% provide encouraging evidence for the reliability of the proposed evaluation framework.

Critical weaknesses
  1. Despite the fact that FWW-V contains 704 questions, these are derived from only 85 videos constructed using structured script templates. Multiple questions based on the same video may share visual and reasoning patterns, potentially limiting the diversity and effective sample size of the evaluation. Furthermore, the reliance on generated videos introduces possible confounding factors related to character consistency, action visibility, and temporal coherence. While the authors employ manual quality assurance, the reported generation success rates are relatively low, especially for complex and multi-room scenarios (Section 2.1). This raises concerns about whether performance degradation consistently reflects deficiencies in higher-order ToM reasoning rather than difficulties associated with video perception or generation artifacts. More systematic controls would strengthen the interpretation of the benchmark results.

  2. A claim which is central is that FWW-EVAL can distinguish genuine mental-state reasoning from correct answers obtained through guessing or spurious correlations. However, the framework evaluates the correctness and consistency of models' generated explanations, which may not faithfully reflect the internal processes underlying their predictions. Moreover, GPT-5 is used as the primary evaluator, including for outputs from GPT-5 itself, although the authors provide cross-model judging and human agreement analyses to mitigate this concern. These validation results are encouraging, but high agreement with human judgments does not fully establish that the rubric measures genuine reasoning rather than explanation consistency. The distinction is particularly important because this reasoning-aware evaluation is one of the paper's main contributions.

Questions for authors

1. Separating perceptual errors from ToM reasoning failures. Given the fact that the benchmark is constructed from generated videos, can the authors provide a controlled comparison in which the same ToM questions are evaluated using the original structured scripts and their corresponding videos? In particular, how much of the performance degradation on higher-order and transparent-barrier tasks can be attributed to visual perception errors rather than incorrect belief inference? Such a comparison would help substantiate the central claim that the benchmark exposes higher-order ToM limitations.

2. Robustness to template and video variations. Due to the fact that the 704 questions originate from 85 videos, how sensitive are the reported results to different video instantiations of the same script, including changes in character appearance, object identity, camera viewpoint, and background? Could the authors report variability across independent generations or provide confidence intervals accounting for questions clustered within videos? This would clarify the robustness and statistical significance of the observed model differences.

3. Validation of the reasoning-aware evaluation. The reported agreement between GPT-5 and human judges is encouraging. Nevertheless, can the authors further clarify how FWW-EVAL distinguishes genuinely correct belief inference from plausible but post-hoc explanations? For example, would the framework assign different scores to responses with identical correct answers but deliberately inconsistent reasoning traces? An analysis of such controlled cases would strengthen the validity of the proposed evaluation methodology.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.