- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper constructs multi-scene scripts based on Sally-Anne narratives and uses Google Veo 3 to generate videos, resulting in the FWW-V dataset with 85 videos and 704 questions. The questions cover memory, real-world states, first- through fourth-order nested beliefs, transparent-barrier settings, and related scenarios. The paper also introduces a rubric-based evaluation framework, FWW-EVAL, which examines model reasoning at the perception, cognition, and conclusion levels instead of relying only on whether the final answer is correct. The authors evaluate 14 multimodal large language models and find that the models perform relatively well on basic factual questions, memory questions, and first-order omniscient-perspective questions. However, performance generally declines from single-room to continuous multi-room settings, and the models also struggle with the transparent-barrier tasks.
Critical strengths
-
The research question is clearly defined. The paper notes that text-only Theory of Mind benchmarks are relatively abundant, while multimodal ToM benchmarks remain limited. A major issue with textual narratives is that it may be unclear whether a character actually observed a particular event. This work provides a more concrete testbed for studying how visual information affects belief inference.
-
The task design has a basis in cognitive science. FWW-V follows the classic Sally-Anne paradigm and extends it to third- and fourth-order nested beliefs as well as transparent-barrier scenarios. Compared with questions that merely ask “Where does A think the object is?”, this design helps distinguish real-world states, other people’s beliefs, and recursive belief reasoning. In principle, it is closer to the complexity of real social interaction.
-
The transparent-barrier setting is a useful diagnostic case. In this subset, which contains six questions, all 13 evaluated models scored 0%, while human accuracy averaged 80%. The authors also report that the models’ reasoning did not mention the persistent visual access provided by the transparent barrier. This suggests that the models may not reliably use characters’ observation conditions, although the exact source of failure still needs to be separated more carefully.
Critical weaknesses
- Synthetic data and filtering may introduce selection bias. The videos are generated by Veo 3 from manually written scripts and retained only after strict filtering. Keeping only unambiguous, logically consistent, and visually natural samples may favor videos with clear cues and controlled actions, while underrepresenting the noise, occlusion, and ambiguity of real videos.
- The sample sizes and statistical reporting for higher-order tasks are insufficient. Tables 2–4 do not clearly report the effective sample size or aggregation method for each question type. The high-complexity and continuous multi-room categories contain only six and eight videos, respectively, with multiple questions sharing the same events. Claims of a “significant gap” therefore need stronger statistical support.
- The human baseline is incompletely reported. The paper does not state the number of participants, their language background, or the evaluation duration, nor does it report model performance on the exact 106-question subset answered by humans. Since humans answered only a subset while models answered the full benchmark, the comparison is not fully matched. Table 2 also reports only three human values and does not cover all question types clearly.
- The claim that FWW-EVAL “eliminates confounding factors” is overstated. The rubric can reduce lucky guessing and superficial correctness, but cannot fully remove visual hallucinations, template priors, language effects, or video-generation artifacts. Since the script and video may not correspond exactly, perception-level errors may still be misjudged.
- Several difficulty factors are confounded. In the multi-room setting, video length, scene count, action complexity, and spatial transitions increase together. The performance decline may therefore reflect visual search, frame limits, temporal localization, or cross-scene memory rather than recursive belief reasoning. Likewise, transparent-barrier failures may indicate difficulty understanding glass partitions, not necessarily difficulty reasoning about belief states.
Questions for authors
Please carefully review the "Critical Weaknesses" and "Questions for Authors" sections; if the issues raised are addressed satisfactorily, I may consider increasing the score.
-
How were the final 85 videos selected from the Veo 3 generation results? Could you provide the total number of generated videos for each category, the reasons for rejection, and the filtering criteria?
-
Why were only Veo 3 synthetic videos used, without real recordings or videos from other generation models? Have you checked whether the conclusions on synthetic videos generalize to real videos?
-
The benchmark combines higher-order recursive beliefs, visual accessibility, and long-video understanding. How do you separate failures caused by ToM reasoning from failures caused by visual perception, temporal localization, cross-scene memory, or limited context length?
-
When moving from single-room to continuous multi-room settings, video length, scene count, action complexity, and spatial transitions all increase simultaneously. Did you run controlled experiments varying only video length, only scene count, or only belief order?
-
All models scored 0% on the transparent-barrier tasks. Did you verify whether the models understand that the character can see through the glass even though they are physically separated, or whether they simply failed to notice the glass or room boundary?
-
How does FWW-EVAL distinguish an incomplete reasoning description from genuinely incorrect reasoning? If a model omits intermediate steps but reaches the correct conclusion, will its response be judged incorrect?
-
How many human participants were involved, what was their language background, and how long did the evaluation take? Did the participants watch the original videos, sampled frames, or textual descriptions?