acceptodds
Under review as a conference paper at ICLR 2027

Dissecting Theory of Mind in Large Language Models via Bayesian Modeling

Abstract

Large Language Models (LLMs) perform well on narrative Theory of Mind (ToM) benchmarks, but whether they can jointly infer an agent's desires and beliefs from a sequence of spatial actions remains largely untested. We evaluate ten LLMs on a dynamic spatial ToM task in a tripartite comparison against archived human ratings and a normative Bayesian Theory of Mind (BToM) model, and use a family of alternative observer models—each removing or distorting one component of BToM—as a diagnostic toolkit for LLM inference. LLMs varied substantially. The best models correlated with human judgments at levels approaching, but not exceeding in both desire and belief, the human–BToM alignment. The desire judgments of lower-performing models were best captured by a variant that ignores action costs, and their belief judgments by heterogeneous suboptimal variants. We then asked models to reconstruct, at every step of the trajectory, the agent's initial belief using only the evidence up to that step while the full trajectory remained in context (Every-step condition). In this format, the belief judgments of even the best models no longer correlated with human judgments and were best accounted for by a new Hindsight model that projects the later-observed world state onto the agent's earlier belief, a pattern reminiscent of the human hindsight bias and curse of knowledge. These results suggest that successful final-state social reasoning in LLMs does not imply consistent mental-state inference over time, and that a Bayesian inverse-planning framework allows such failures to be decomposed into specific components and tested against alternative models.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Theory of Mind, Bayesian Inference, Large Language Models, Computational Cognitive Neuroscience

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8rKXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper evaluates LLMs on the "food truck" test, which challenges a model to infer the food preference of a stimuli according to its action trajectory when navigating between two food truck locations. ToM questions regarding the stimuli's belief and food preference (e.g., desire) are raised to the model at End-step, where the model use the full trajectory to answer, and Every-step, which the model use partial cumulative actions to answer. Model performances are compared against judgements from 16 human annotators and outputs from a BToM model. Results show that judgements from advanced models align well with human judgement and exceeding the performance of the BToM model. Several correlation and variance partitioning studies are conducted to understand the gap between strong and weak model and to investigate major failure modes of the weak models.

Critical strengths
  • Each scenarios is run 16 times per model through independent API calls and the mean and confidence interval of each model is reported. This improves the statistical robustness of the results especially considering the relatively small (n=219n=219) sample size.
  • LLM judgements are compared with those from human and a normative BToM model. Further, the authors proposed four hypothesis (e.g., lesioned BToM models). These experiments and analysis partially helped with understanding potential modes of failure of the tested LLMs.
  • Desire and Belief, the two important aspects of ToM, are studied independently.
  • The variance partitioning studies provide useful complement to further enhance the conclusion about correlation between Worse models and the lesioned BToM models.
Critical weaknesses
  • Lack of information on human annotation. As human ratings are used as the ground truth and serves as the basis for model evaluation and analysis, I consider this as the most critical weakness.

    • It is unclear how the human judgements and their ratings are obtained.
    • There is no inter-annotator agreement reported, diminishing the validity of human ratings.
    • It is also important to disclose basic annotator information such as their demographic, compensation, instruction given before annotation, and the annotation platform/tool used for this study.
  • The introduction to the "Every-Step" setup is unclear. From Ln222, it seems that the full trajectory is always accessible to the LLM regardless of the setup. If this is the case, then the step-wise BToM tackles a completely different task where it only has partial observation up to the ttht^{th} timestamp whereas the Every-Step setup has access to the full context. The prompts in Appendix A do not help with understanding the context the LLM sees in Every-Step

  • The bottom two figures of Figure 5(a) is missing legend.

Questions for authors
  • What are the useful insights that the "hindsight" model bring? It reads the "hindsight" model is reducing the original POMDP setup to a MDP setup. Then what do the results from such reduction indicate except for the correlation? Would the so-called hindsight bias exists in a POMDP-formulated step-wise setup where the full trajectory is not accessible to the model?
  • How are the human annotations obtained (see Critical Weaknesses Section)?
  • How reliable are the stepwise rating given by the BToM model? While a large scale assessment would be infeasible, a small scale validation would significantly improve the informativeness and credibility of the Every-step study.
1 comment

We thank the reviewer for the detailed reading and for recognizing the strengths of our work. We reply to each point below.

Summary of revisions (all new analyses in App. H): (1) Further experiments on hindsight bias: three conditions that change one factor of Every-step (Every-step-single, Single-trunc, Prefix-step; Figs. 11–14), run for Claude-4.6-Opus, Gemini-2.5-Pro, and GPT-5.4. (2) A fit of the Hindsight weight λ and a sensitivity analysis over λ and temperature β (Fig. 10, Tables 6–7). (3) A spatial comprehension control on the same prompts (Fig. 15). (4) Better statistics: scenario-bootstrap intervals for every statistic, and paired bootstrap tests against the human–BToM benchmark.

R3-1. Human ratings. These are not new annotations: they are the published experimental data of Baker et al. (2017), collected for this task, and we use them unchanged. Details are now in Appendix B. Reliability computed with our pipeline was high: split-half reliability of the group mean (Spearman–Brown) was 0.95 for desire and 0.91 for belief (range over 1,000 splits 0.93–0.97 and 0.85–0.94), and a held-out participant correlated with the rest of the group at r = 0.73 (desire) and 0.57 (belief).

R3-2. What the model sees in Every-step. The reviewer's reading is correct: the full trajectory is in the prompt, and the model is instructed to rate each step using only the evidence up to that step. To compare directly, we added Prefix-step, which truncates the log at each queried step and asks the End-step question, so the LLM sees exactly what BToM uses. The Hindsight pattern disappeared in all three models tested, suggesting that it did not originate from a ToM-specific reasoning failure. However, the Prefix-step condition did not make the LLMs normative, and their response profiles were largely consistent with the End-step condition.

R3-3. Figure 5(a). We thank the reviewer for pointing this out; the legend has been added.

R3-4. What Hindsight adds. Hindsight does not reduce the POMDP to an MDP; that reduction is TrueBelief, where the agent knows the world state from the start. Hindsight keeps the agent's POMDP and changes only the observer's inference: once the agent has seen Spot 2, the estimate of its initial belief is pulled toward what it saw (App. D.3). The two differ before Spot 2 is visible, and better-performing models follow Hindsight there: the pull toward the true state appeared only after Spot 2 became visible (before: Opus 0.01–0.08; Gemini 0.00–0.04). GPT-5.4 follows TrueBelief.

R3-5. Reliability of stepwise BToM. We have no additional human test, but two results support it. First, BToM uses the same model and parameters at every step, and its final-step prediction is the one validated against humans (r_desire = 0.91, r_belief = 0.78). On the 22 Check-Partial scenarios, whose trajectories end before the agent reaches a truck, BToM agrees with humans as well as on the rest (r_belief = 0.81, [0.65, 0.91] vs. 0.77, [0.67, 0.86]). Second, in Prefix-step the step-to-step belief changes of the better-performing models followed BToM's (r = 0.36, [0.26, 0.44], and 0.26, [0.18, 0.33]), and desire changes followed BToM in both Prefix-step and Every-step (App. H). A human stepwise study remains future work.

Reviewer_U4JvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates ten LLMs on the food-truck ToM task, comparing their desire and belief ratings with archived human data, BToM, and four lesioned observer models. In the End-step condition, the best models approach the human–BToM alignment, and weaker models' desire ratings resemble a cost-insensitive observer (NoCost). In a new Every-step condition, the best models' belief ratings no longer match humans and are best fit by a new Hindsight model.

Critical strengths
  • The paper evaluates LLM social reasoning against a computational account of how humans do it, rather than against a single correct answer.

  • The study is carefully designed and executed. The analyses are well controlled, and the paper states its main confounds and limitations openly instead of overselling its findings.

Critical weaknesses

The evidence is narrow relative to the claims. Everything rests on one small, published grid-world task with a single prompt format, so it is unclear how much the findings say about LLM theory of mind in general rather than about this task. The paper also stays descriptive: it does not explain why models behave this way or what would change it, which limits its value to the ICLR audience.

The main new finding is not established. The hindsight-like pattern appears only in the new Every-step condition. That condition has no human data, it is confounded by an ambiguous question and by the full trajectory being in context (both acknowledged in the Limitations), and it is explained by a model designed after seeing the same data. Simpler explanations have not been ruled out.

Questions for authors

GPT-5.4 correlates near zero with humans (Table 4). Were its outputs and its parsing of the log checked? Why?

1 comment

We thank the reviewer for the positive assessment of the design and for the specific questions. We reply to each point below.

Summary of revisions (all new analyses in App. H): (1) Further experiments on hindsight bias: three conditions that change one factor of Every-step (Every-step-single, Single-trunc, Prefix-step; Figs. 11–14), run for Claude-4.6-Opus, Gemini-2.5-Pro, and GPT-5.4. (2) A fit of the Hindsight weight λ and a sensitivity analysis over λ and temperature β (Fig. 10, Tables 6–7). (3) A spatial comprehension control on the same prompts (Fig. 15). (4) Better statistics: scenario-bootstrap intervals for every statistic, and paired bootstrap tests against the human–BToM benchmark.

R2-1. Narrow evidence. We agree that the study uses one task and state this as a limitation. The diagnosis needs both human data and a validated normative model, which few ToM tasks provide. Within the task, we now vary the elicitation across five conditions, and the End-step results hold across ten models from four families. Note that our toolkit applies to any task with a generative agent model, since BToM's structure (state, observation, belief, desire, action) can be ablated selectively. We would like to highlight that, while our results appear descriptive, the model variants specifically ablate critical model components and therefore reveal potential targets for future interventions.

R2-2. Every-step confound. We tested the context effect of the full trajectory directly. Prefix-step truncates the log at each queried step and asks the End-step question, so the LLM sees exactly the evidence BToM uses and has nothing to suppress. The Hindsight pattern disappeared in all three tested models, suggesting that it was specific to the Every-step condition. The lack of human stepwise data is a real limitation; repeated within-subject queries would alter later judgments, which is why BToM serves as the benchmark.

R2-3. Simpler explanations. We considered several alternative explanations and ruled them out. (a) Misreading the log: in a factual control, all three LLMs identified positions, occlusion, and when Spot 2 became visible with 100% accuracy (Fig. 15). (b) Leakage of future steps: truncation alone explained 11% of the drop for Opus and none for the others. Also, although the true state is stated from step 1, the pull toward it appeared only after Spot 2 became visible to the agent (before: Opus 0.01–0.08; Gemini 0.00–0.04). (c) Output load: ruled out by the token analysis.

R2-4. GPT-5.4. Yes, we checked both, and its outputs were valid. In the factual control, GPT-5.4 identified positions, occlusion, and visibility timing with 100% accuracy; its small errors (overall accuracy 93.9%, [92.8, 94.9]) concerned distances only. Its poor fit has two components: belief ratings best explained by TrueBelief (the true world state projected onto the agent) and desire ratings by NoCost.

Reviewer_Xa2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a toolkit consisting of a Bayesian model and its variants, aiming to diagnose the mistakes LLMs make in Theory-of-Mind tasks. The authors test 10 LLMs on the food truck task, comparing their beliefs and desires with human judgment and the Bayesian model variants. Based on the correlation and variation partitioning analysis, the authors claim that stronger models approach human performance on final judgements, while weaker models often neglect action costs.

Critical strengths

A key strength is that the work attempts to diagnose the kinds of errors LLMs make in ToM tasks by repurposing an established cognitive-modeling framework to provide structured diagnoses of LLM behavior.

Critical weaknesses
  1. Limited contributions. The task, the BToM model, and the TrueBelief, NoCost, and MotionHeuristic alternatives largely come from Baker et al.. The new elements in the paper are their application to modern LLMs, the Every-step condition, the adapted stepwise analysis, and the Hindsight variant. While, as listed in the critical strengths, repurposing them for model diagnosis is potentially useful, this work does not yet establish that it contributes sufficiently beyond a new application of an existing framework.

  2. The claims in the paper are supported by weak or non-identifying evidence. The predictions depend on assumptions about action costs and action-selection temperature, yet the paper does not report corresponding sensitivity analysis over the parameters. Moreover, prediction correlation does not demonstrate that the same mechanism/mistake causes the LLM's response. This concern is strongest for Hindsight, which was developed after the Every-step results and was not fitted and evaluated on held-out data. Therefore, the evidence supports behavioral resemblance rather than strong mechanistic conclusions.

  3. In the Every-step condition, models see the complete trajectory and are then instructed to ignore future information when producing earlier judgements. This may make models have difficulty suppressing visible context that they should not have referred to, which therefore causes the failure of the test, rather than a specific ToM reasoning failure. Therefore, the condition is confounded.

  4. Although the paper describes its objective as diagnostic, its relationship to prior work and the motivating research gap remain insufficiently justified in the related work section. The authors should clarify what their analysis reveals beyond what Baker et al. observed and existing LLM ToM evaluations. For example, whether their toolkit generalizes beyond the task, and how its diagnoses could inform future model development or the community.

Questions for authors

See weaknesses

1 comment

We thank the reviewer for the careful reading and for recognizing the diagnostic aim. We reply to each point below.

Summary of revisions (all new analyses in App. H): (1) Further experiments on hindsight bias: three conditions that change one factor of Every-step (Every-step-single, Single-trunc, Prefix-step; Fig. 11–14), run for Claude-4.6-Opus, Gemini-2.5-Pro, and GPT-5.4. (2) A fit of the Hindsight weight λ and a sensitivity analysis over λ and temperature β (Fig. 10, Tables 6–7). (3) A spatial comprehension control on the same prompts (Fig. 15). (4) Better statistics: scenario-bootstrap intervals for every statistic, and paired bootstrap tests against the human–BToM benchmark.

R1-1. Contribution. Overall, we agree with the assessment but want to highlight key contributions of our work: Baker et al. tested only humans, but every LLM finding is new. We broadly covered LLMs, revealing their idiosyncratic theory-of-mind behaviors. Capitalizing on the dynamic inference process of BToM, we add a normative, stepwise benchmark from the Bayesian model, which Baker et al. did not provide. This will provide a mechanistic clue as to when and how the LLMs make inferences differently from the normative model. The hindsight bias model is new in our work, exposing a potentially critical failure mode of LLMs’ theory of mind. We hope that these results help develop LLMs that can better interact with humans in a social context.

R1-2. Sensitivity. We now fit the action-selection temperature β (Table 7) and the hindsight parameter λ per LLM (Fig. 10, Table 6). This sensitivity analysis shows that our main conclusion—that the Hindsight model explains LLMs’ behavior in the Every-step condition well—still holds across a range of parameter values, including the original ones. Note that, as in Baker et al., the action cost was not a free parameter. Regarding the lack of a mechanistic conclusion, we agree that correlation shows resemblance, not mechanism; however, all model variants actually test the causal role of each model component by ablating it from the normative Bayesian model, thereby providing mechanistic insights.

R1-3. Every-step confound. We tested this account directly. Prefix-step truncates the log at each queried step and asks the End-step question, so the LLM sees exactly the evidence BToM uses and has nothing to suppress. The Hindsight pattern disappeared in all three tested models, suggesting that it did not originate from a ToM-specific reasoning failure, as the reviewer pointed out. However, the Prefix-step condition did not make the LLMs normative, and their response profiles were largely consistent with the End-step condition. R1-4. Related work and gap. We appreciate the reviewer’s suggestion and have revised the Related Work and Discussion sections. We believe that dynamic, joint inference of desire and belief in the food truck task is an important addition to the current ToM benchmarks, as it lets us go deeper into the social reasoning dynamics of LLMs. Our toolkit applies to any task with a generative agent model, since BToM's structure (state, observation, belief, desire, action) can be ablated selectively. For model development, our results point to potential training targets for individual LLMs—for example, cost-insensitive desire inference in weaker models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.