acceptodds
Under review as a conference paper at ICLR 2027

InterMind: What Happens Between Minds? Benchmarking Theory of Mind Across Human–AI Interaction

Abstract

Theory of Mind (ToM), the capacity to attribute mental states to others, supports understanding in human–AI interaction. As an exchange unfolds, each participant interprets and responds to the other, making the relationship between minds central to how the interaction develops. Reasoning within this open-ended process requires models to determine which interpretations and responses the dialogue supports and where that support ends. Existing evaluations leave these complementary judgments and their evidential boundaries insufficiently characterized. We introduce InterMind, a benchmark that examines ToM within the human–AI interaction loop through Mind Reading, Recursive Modeling, Action Forecasting, and Mind Shaping. These tasks connect user-state inference with judgments about the reception of assistant replies, subsequent user actions, and appropriate assistant responses. Questions require models to identify all supported candidates without a disclosed answer count, testing their ability to determine the supported range independently. InterMind contains 700 questions from 568 conversations, grounded in mental-state records and public dialogue and verified through automated assessment and human review. Across 11 models, the best overall exact-set accuracy reaches 56.7%, with differing strengths across tasks. Disclosing the correct answer count raises mean accuracy from 42.4% to 67.6%, revealing a substantial dependence on external constraints when identifying the complete supported set. These findings highlight the challenge of independently forming judgments about an exchange, even when candidate answers are provided.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Theory of Mind, Human-AI Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_S1DnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors produce a ToM benchmark of 700 questions from 568 AI generated conversations containing four task categories. The benchmark is checked by four human annotators. They evaluate a range of models on the benchmark and find that the best model (GPT-6 Astra) gets an accuracy of 56.7%. This rises significantly when the number of correct answer choices is revealed.

Critical strengths
  1. The four part taxonomy used to motivate the tasks included in the dataset is nice.
  2. The error analysis presented is helpful and an admirable addition to the work.
Critical weaknesses

My main concerns regard the human validation and meaningfulness of the dataset.

  1. My largest concern regards the construction of the dataset, and in particular the verification. As I mention in my reading notes, the details regarding human verification discussed in both Appendix A.3 and 4.2 are insufficient.
    1. I am generally unconvinced and concerned that the dataset itself is not very meaningful. No IAA numbers were reported, no annotation guidelines were given, no data collected from the four humans that “checked” things is provided, and no human baseline was done.
    2. The examples provided seem pretty ambiguous and I suspect that IAA might not be that good. The dataset and code were not provided in the supplementary materials, which limits how much I can audit.
  2. Some of the tasks presented (e.g., D.1) read more like reading comprehension questions rather than something that would require ToM. How do we know these questions really require ToM?
    1. How does this control for shortcutting or running on a script?
  3. Some sections of the paper do not seem elucidating. In particular, section 3. The related work section does not cite any of the relevant psychological and cognitive science work, which is strange.

Below are additional comments from my reading of the paper. They are supplementary and need not be considered critical weaknesses.

  • After a first read of the abstract I have no idea what you guys actually did. This is not a useful abstract. What is “Mind Shaping”, or “Mind Reading” for that matter?
    • You say in the abstract that other benchmarks leave “complementary judgements and their evidential boundaries insufficiently characterized” but how? And what does that even mean?
  • Should cite Premack & Woodruff for your definition of ToM in the first sentence. It’s almost word for word.
  • “ToM concerns what happens between minds” - I mean, a lot of existing benchmarks deal with this. MindCraft for example, anything related to common ground, a significant number of psychological datasets related to conceptual pacts and their formation. If this is meant to be a justification for the dataset it is a weak one.
  • “We introduce InterMind around the changing demands on ToM [...]”
    • This reads strangely to me. Based around?
  • I am not sure I love the name “Mind Reading” for the first part of your taxonomy, but I do like the taxonomy as an idea.
  • Both in the abstract and in the introduction you need to say how you construct the 700 questions and where the 568 conversations come from. Like are they generated by AI? Are they collected? Did the authors write them? Where are these questions and conversations coming from?
    • So these are multiple choice questions and you allow for multiple correct answers? How many choices are there? This is great and the justification is good but this should not be emphasized too much for novelty. FanToM does this as well for example.
    • Again, simple summary details of what the dataset actually is need to be up front and clear. This is very frustrating as a reader.
  • Section 3 seems like needless mathematization and is not very elucidating. The notation developed is not even used again? This should be revised.
  • “We assemble human–AI conversations with records of the user’s reasons for speaking and reactions to assistant replies.” but then only a few sentences later it is said “As the exchange unfolds, the simulated user’s perspective [...]”.
    • So these are not human-AI conversations. They are AI generated conversations. As I mentioned earlier, this should be immediately clear far earlier.
  • “From each conversation and its associated records, we identify judgments [...]”
    • Again, this is being done by a model. That should be more clear. “We” refers to the authors so if a model is used to do something in the construction of the dataset that should be said.
  • The examples in Appendix D seem strange to me.
    • In D.1, an example is presented that feels more like a reading comprehension question than something about ToM. Additionally, I think the example model response seems perfectly reasonable.
    • D.3 doesn’t feel well formulated. I don’t know if a human could do this.
    • D.4 is a nice example. I think this captures something really useful.
  • The details regarding human verification discussed in both Appendix A.3 and 4.2 are insufficient.
    • Who were the annotators? What were the instructions provided to them and what were the annotation guidelines? Were any IAA metrics (e.g., pairwise Cohen’s kappa) computed both between the human annotators and also the AI annotator(s)?
    • What is specifically meant by the various “checking” that was done? Ideally annotators would have actually answered the questions and performance/agreement could then be compared. This might identify questions that are just not reasonable. Checking an answer is not exactly the same thing as being asked to find it. Even doing so for a smaller sample of the dataset would have been more useful.
  • Section 5.5 seems to me that it could be evidence that the ambiguities in the questions (e.g., D.1) are driving most of the performance.
  • It is strange that there doesn’t seem to be a clear pattern of performance across the tasks (looking at Table 2). I would have liked to see more analysis of this.

Additional Related Work (authors might find interesting)

These are some typical citations from psychology and cognitive science related to ToM.

These are not necessary citations but the authors may find them interesting when thinking about what situations really require ToM.

  • Schank, R. C., & Abelson, R. P. (1977). Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Lawrence Erlbaum Associates.
  • Taylor, D., Gönül, G., Alexander, C., Züberbühler, K., Clément, F., & Glock, H.-J. (2023). Reading minds or reading scripts? De-intellectualising theory of mind. Biological Reviews, 98(6), 2028–2048. https://doi.org/10.1111/brv.12994
  • Meng, H. (2008). Social script theory and cross-cultural communication. Intercultural Communication Studies, 17(1), 132–138.
Questions for authors
  1. Please provide additional details regarding the human verification procedure. For the checking that was already done, can any agreement measure be computed?
    1. Realistically, an annotation guideline and human annotation of at least some subset with IAA between two humans and to gold annotations, is sort of a minimum here. Refer to my questions on this above.
  2. What makes these questions as they are constructed, necessarily require ToM in a way that any other MC question wouldn’t. For example, how is D.1 different from multi-hop question answering?
Reviewer_Vb2ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InterMind, a benchmark for evaluating Theory of Mind in human–AI interaction. The authors divide the evaluation into four tasks: Mind Reading, Recursive Modeling, Action Forecasting, and Mind Shaping, and construct a dataset of 700 multi-answer questions. The experiments show different model performance across the four tasks, and providing the correct answer count significantly improves accuracy.

Critical strengths

I think the human-centered perspective of this paper is an interesting direction for studying Theory of Mind. Compared with traditional mental-state inference, the paper also considers how users interpret AI responses and how users and AI influence each other during interaction. This shifts the focus from simply understanding the user to considering how that understanding affects the interaction process.

The paper also provides useful analysis of differences across tasks, model error patterns, and the effect of answer counts, showing some interesting differences in model behavior.

Critical weaknesses

Although I think studying Theory of Mind from a human-centered interaction perspective is meaningful, the paper needs a clearer explanation and stronger evidence to support the choice of these four tasks. The authors do not fully explain why MR, RM, AF, and MS were selected, how they relate to each other, or how well they cover Theory of Mind in human–AI interaction. Some tasks may also overlap or allow different interpretations. For example, both RM and MS may involve understanding user feedback and misunderstandings. I think the paper needs more discussion of the differences between these tasks, possible overlaps, and how their boundaries are defined. Also, the paper describes interaction as a continuous process, but the four tasks are evaluated independently. More explanation or evidence is needed to support the relationship between this task design and the proposed interaction framework.

I also have some questions about the definition of ground truth. The paper uses private mental-state records to construct questions, but the final correct answers are determined by what the public dialogue supports. I think these are not necessarily the same. Also, inferring a user's mental state, predicting their next response, and selecting an appropriate assistant reply may require different standards for correctness. The paper needs to explain these differences more clearly and discuss how it handles cases with multiple reasonable interpretations.

Most questions in the benchmark come from synthetic conversations, and the proportion of synthetic data differs across tasks. For example, 33 of the 35 RM Emotion questions are synthetic. I am not sure whether this distribution affects task difficulty and model performance. In real interactions, users' thoughts and expressions may be more ambiguous and may change during the conversation. The paper needs more analysis of whether the results are consistent between synthetic and real conversations and how the data sources affect comparisons across tasks.

I think the evaluation metric may also affect the main findings. The paper uses exact-set accuracy, where missing or adding one option makes the entire answer incorrect. The experiments show that providing the correct answer count significantly improves accuracy. However, I am not sure whether this difference mainly reflects limitations in Theory of Mind reasoning or a more general difficulty in deciding how many answers to select. Although the paper analyzes different selection errors, more discussion is needed about how these factors affect the results and how well the metric reflects Theory of Mind ability.

The paper shows some interesting results, but it does not clearly explain how these findings could help improve real agent systems. The current evaluation mainly focuses on selecting answers from fixed options, while real human–AI interaction often requires agents to update their understanding of users across multiple turns, handle changing preferences, and sometimes ask for clarification. Even if the reported results are correct, it is unclear how much these findings can help improve real agent systems.

Questions for authors
  • Could the authors explain more clearly why MR, RM, AF, and MS were chosen as the four tasks, and how they relate to each other? Is there theoretical or experimental evidence supporting this classification and how well it covers Theory of Mind in human–AI interaction?
  • Some tasks may overlap or allow different interpretations. For example, both RM and MS may involve understanding user feedback and misunderstandings. Could the authors discuss the differences between these tasks, possible overlaps, and how their boundaries are defined? Also, how much can the results from independently evaluated tasks tell us about the overall interaction process?
  • The paper uses private mental-state records during question construction, but the final answers are determined by public dialogue. Could the authors explain the relationship between these two sources more clearly? How are correct answers determined when the user's actual mental state differs from what the public dialogue supports?
  • For questions that may allow multiple reasonable interpretations, I wonder how the authors ensure the reliability of the ground truth. Were questions independently checked by multiple human annotators? How were disagreements about the correct options handled?
  • Most questions come from synthetic conversations. I wonder whether models show similar results on synthetic and real conversations. Could the authors provide more analysis of how different data sources affect task difficulty and model performance?
  • The paper shows that providing the correct answer count significantly improves accuracy. How much of this improvement reflects limitations in Theory of Mind reasoning rather than a general difficulty in selecting the correct number of answers? Could the authors explain this distinction more clearly?
  • I wonder how Mind Shaping differs from general instruction-following or response-selection tasks. Could the authors explain what specific Theory of Mind ability this task evaluates?
  • How could these findings apply to real agent tasks involving multiple turns, changing user preferences, and clarification? Also, how could the results help improve actual human–AI interaction systems?
Reviewer_SME7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a benchmark to examine theory-of-mind (ToM), the ability to perceive belief, desire of others in a human-AI interaction.

The authors generate 700 questions from 568 conversations from simulated users. The authors conducted extensive experiments to validate how the LLMs excel in ToM tasks in their benchmark.

Critical strengths

-Proposes a framework for a benchmark generation of human-AI interaction

  • Extensive experiments on different models.

-Has promise in human-AI communication research direction

Critical weaknesses

The study has several weakness. First, the benchmark construction has several limitations. The title and introduction claims to be a human-AI interaction benchmark. However, as per Appendix A, the dialogues are generated by GPT and simulated user. Additionally, conversations are generated without any theoretical grounding. Compared to prior work, ThoughTrace uses real-world human-AI conversation. SOTOPIA is a generated by GPT but grounded in Big 5 personalities. The authors of SOTOPIA also claims to evaluate and mimic human’s ToM in LLMs’ capability. The TOMATO benchmark uses STOTPIA and generates two other ToM test based on psychology False belief, and personality trait. Secondly, the authors are using GPT as a replacement of human and referring to human-AI interaction. I would request the authors to consult papers [1-3]. These papers are from psychology, NLP, and SWE community and everyone agrees a common point: LLMs should not be used as a proxy for human. Third, the authors uses the same Llm for dialogue generation and Llm as judge. This has potential limitations. A better approach could be i) using LLM as judges excluding the one used for question generation and ii) use multiple judge and report, as done in ThoughtTrace and also reported by NLP community regarding the limitations of LLm-as-Judge [4, 5].

A promising way to construct the benchmark is to collect a high-quality real user-Ai interaction dataset with varying user, label by experts in psychology: user personality, belief from user self-reported data and conversation, and then report LLM performance. The authors can also augment the the collected data with LLM and then measure the quality of the synthetic dataset vs human dataset.

Minor The dimensions of ToM examined: RM, AF, MR, and MS needs references and justifications.

References

  1. Cui, Ziyan, Ning Li, and Huaikang Zhou. "Can ai replace human subjects? a large-scale replication of psychological experiments with llms." A Large-Scale Replication of Psychological Experiments with LLMs (August 25, 2024) (2024).

  2. Chiang, Cheng-Han, and Hung-yi Lee. "Can large language models be an alternative to human evaluations?." In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607-15631. 2023.

  3. Wang, Ruiqi, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. "Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering." Proceedings of the ACM on Software Engineering 2, no. ISSTA (2025): 1955-1977.

  4. Thakur, Aman Singh, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. "Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges." In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pp. 404-430. 2025.

  5. Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S. (2025, December). Judging the judges: A systematic study of position bias in llm-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (pp. 292-314).

Questions for authors
  1. How many mental states of users are represented in Equation 1 in Section 3? The Appendix A also does not provide clear answer.
  2. Please also see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.