acceptodds
Withdrawn from ICLR 2027

Selective Deficits in LLM Mental Self-Modeling in a Behavior-Based Test of Theory of Mind

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The ability to represent oneself and others as agents with knowledge, intentions, and belief states that guide their behavior - Theory of Mind - is a human universal that enables us to navigate - and manipulate - the social world. It is supported by our ability to form mental models of ourselves and others. Its ubiquity in human affairs entails that LLMs have seen innumerable examples of it in their training data and therefore may have learned to mimic it, but whether they have actually learned causal models that they can deploy in arbitrary settings is unclear. We therefore develop a novel experimental paradigm that requires that subjects form representations of the mental states of themselves and others and act on them strategically rather than merely describe them. We test a wide range of leading open and closed source LLMs released since 2024, as well as human subjects, on this paradigm. We find that 1) LLMs released before mid-2025 fail at all of our tasks, 2) more recent LLMs achieve human-level performance on modeling the cognitive states of others, and 3) even frontier LLMs fail at our self-modeling task - unless afforded a scratchpad in the form of a reasoning trace. We further demonstrate cognitive load effects on other-modeling tasks, offering suggestive evidence that LLMs are using something akin to limited-capacity working memory to hold these mental representations in mind during a single forward pass. Finally, we explore the mechanisms by which reasoning models succeed at the self- and other-modeling tasks, and show that they readily engage in strategic deception.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Theory of Mind, Self-Modeling, Self-Perspective-Taking, AI Safety, Deception, Behavior-Based Tests

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EfY3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper claims to propose a new approach, that places LLMs as first-person players in the questions, to test Theory of Mind of LLMs. Older models fail at the tasks tested, while recent models achieve human-level performances.

The paper is poorly written, and I believe this paper is largely AI generated (which I am sad to see). I am concerned about the soundness and validity of the results if they are not reviewed by humans thoroughly.

Critical strengths

While the writing is poor, there are indeed some interesting findings:

CoT is crucial for ToM reasoning in LLMs. I agree that CoT is providing working memory for the model in ToM reasoning. It would be interesting to see if humans also use methods similar to CoT / scratchpad to answer these questions.

I also find the "event load" controlled experiment interesting. It shows that models' ToM performances are not less affected by context length, compared to the number of epistemic-state transitions.

Critical weaknesses

I find the proposed task just a variant of ToMi.

If the only substantial distinction is that this paper uses a first-perspective problem setting, I don't find it novel or necessary.

Also, perspective taking in ToM has been investigated [1] so it is not completely new (and this paper should be cited).

[1] Wilf, Alex, et al. "Think twice: Perspective-taking improves large language models’ theory-of-mind capabilities." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Questions for authors

Apart from the way you ask the question (embodying the LLM in the scene), what else sets this work apart from previous benchmarks? What are the qualities that give this evaluation paradigm the potential of finding something that other benchmarks can't?

Also I'm interested in whether humans also use similar forms of working memory to answer the ToM reasoning questions.

Reviewer_8TrAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission introduces a behaviour-based Theory of Mind (ToM) paradigm that requires models to use representations of their own and others' states strategically in a team game setting, rather than merely describe them. In text-based team game scenarios, models need to track object movements and players' entrances and exits, then select an action (from Ask, Tell, or Pass) under communication costs. The four tasks probe Self Knowledge, Teammate Knowledge, True vs False Belief, and Teammate vs Opponent. The authors evaluate 28 LLMs from seven providers and 10 human participants. They find that some nonthinking models perform well on other tasks but poorly on Self Knowledge, whereas allowing pre-action reasoning tokens substantially improves performance. Information-explicit, verbal-report, and load experiments further investigate this asymmetry, which the authors interpret as a selective difficulty in mental self-modelling.

Critical strengths
  1. The paradigm tests the strategic use of mental-state information through action selection. It provides a useful framework for examining the gap between explicitly reporting mental-state information and using it to guide behaviour.
  2. The self–other performance asymmetry is an interesting empirical finding. The information-explicit and verbal-report controls, together with manipulations of event load and epistemic state transition load, are interesting, although some interpretation remains open.
Critical weaknesses
  1. The Self Knowledge scoring conflates epistemic-state judgments with costly information acquisition. The submission treats Ask as correct whenever the player may have outdated information. However, the uncertainty alone does not make asking optimal: assuming a reliable teammate and maximising the current turn's expected score, Pass can be preferable if the previous answer remains correct with probability above 0.5. The instructions specify neither this probability nor a worst-case decision rule. Indeed, Appendix A.3 includes a model that considers unobserved changes and communication costs before choosing Pass. The verbal-report control supports genuine self-knowledge difficulties, but does not establish which behavioural errors arise from those difficulties.
  2. In Self Knowledge, changes after the player leaves are omitted, requiring the model to recognise that its information might be outdated. In Teammate Knowledge’s false-belief cases, the model observes the changes the teammate missed and can directly establish that their belief is wrong. The comparison changes both perspective and the evidence supporting the judgment. The verbal-report control retains this asymmetry, while the information-explicit control bypasses inference from events.
Questions for authors

Please see above.

Reviewer_J6C9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the theory-of-mind (ToM) ability in a wide range of LLMs released since 2024. To assess ToM in LLMs, the authors designed a text-based game given in Figure 8. The play consists of four persons: A, B, C, and D. The player is A, B is always a teammate and rest are the the opponents. The play progresses in turns and need to depict other's mental state. The empirical evaluation showed LLMs released before 2024 failed at almost all and newer LLMs perform better but still fails at self-modelling task.

Critical strengths
  • ToM assessment in LLMs
  • Provides extensive empirical assessments
Critical weaknesses

While the study has promise, it has several weakness. First, the selection of ToM tasks. The authors reported Sally-Anne false belief and Teammate knowledge in Table 2. However, these are widely used in ToM evaluations and LLMs even excels variants of such tests but performs poorly on novel tests. Please see references [1, 2]. Second, the self-knowledge behavior in group setting has been addressed in prior literature. Please see reference [3] and other works of Max Kleiman-Weiner. Third, the game is text-based and uses limited ToM interaction. Please refer to [4,5] for more details.

References

  1. Ullman, T. (2023). Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399.
  2. Shapira, Natalie, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. "Clever hans or neural theory of mind? stress testing social reasoning in large language models." In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2257-2273. 2024.
  3. Theory of minds: Understanding behavior in groups through inverse planning” Shum*, M., Kleiman-Weiner*, M., Littman, M. L., & Tenenbaum, J. B., (2019). Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33(1), pp. 6163–6170.
  4. Ma, Ziqiao, Jacob Sansom, Run Peng, and Joyce Chai. "Towards a holistic landscape of situated theory of mind in large language models." In Findings of the association for computational linguistics: EMNLP 2023, pp. 1011-1031. 2023.
  5. Stack, Caoimhe Harrington, Effat Farhana, Xinyu Shen, Simeng Zhao, Angela Maliakal, Roxanne Rashedi, Joel Michelson, and Maithilee Kunda. "Framework for a multi-dimensional test of theory of mind for humans and ai systems." In The Tenth Annual Conference on Advances in Cognitive Systems. 2022.
Questions for authors

Please see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.