- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper claims to propose a new approach, that places LLMs as first-person players in the questions, to test Theory of Mind of LLMs. Older models fail at the tasks tested, while recent models achieve human-level performances.
The paper is poorly written, and I believe this paper is largely AI generated (which I am sad to see). I am concerned about the soundness and validity of the results if they are not reviewed by humans thoroughly.
Critical strengths
While the writing is poor, there are indeed some interesting findings:
CoT is crucial for ToM reasoning in LLMs. I agree that CoT is providing working memory for the model in ToM reasoning. It would be interesting to see if humans also use methods similar to CoT / scratchpad to answer these questions.
I also find the "event load" controlled experiment interesting. It shows that models' ToM performances are not less affected by context length, compared to the number of epistemic-state transitions.
Critical weaknesses
I find the proposed task just a variant of ToMi.
If the only substantial distinction is that this paper uses a first-perspective problem setting, I don't find it novel or necessary.
Also, perspective taking in ToM has been investigated [1] so it is not completely new (and this paper should be cited).
[1] Wilf, Alex, et al. "Think twice: Perspective-taking improves large language models’ theory-of-mind capabilities." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
Questions for authors
Apart from the way you ask the question (embodying the LLM in the scene), what else sets this work apart from previous benchmarks? What are the qualities that give this evaluation paradigm the potential of finding something that other benchmarks can't?
Also I'm interested in whether humans also use similar forms of working memory to answer the ToM reasoning questions.