acceptodds
Under review as a conference paper at ICLR 2027

Algorithmic Primitives of Theory of Mind Benchmarks

Abstract

Do Theory of Mind (ToM) benchmarks really test theory of mind, or are they actually testing parsing and algorithmic state tracking? While benchmarks vary modality and dialogue length, this paper finds that they re-encode the same few state transition algorithms. We create PDDL-Mind, a neuro-symbolic approach that recovers such algorithms and upstream inductive biases of ToM benchmarks explicitly by parsing narratives into the Planning Domain Definition Language (PDDL) and executes candidate actions against a predefined set of update algorithms. Doing so provides program-verified state traces to the model. We use our method's performance gain to audit how much of a benchmark's difficulty our inductive bias absorbs. Across MMToM, MuMA, and FanToM, outsourcing parsing to programs alone brings performance close to strong agentic pipelines, while outsourcing state tracking to PDDL moves GPT-4o to close-to-human performance. We demonstrate that complex ToM benchmarks primarily punishes Large Language Models' unreliable implicit implementation of state tracking and parsing algorithms rather than limitations in belief and counterfactual reasoning.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
theory-of-mind, state tracking, cost-efficiency, LLM, neuro-symbolic, PDDL

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4yLzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PDDL-MIND, a pipeline intended to distinguish narrative parsing, state tracking, and belief reasoning in Theory of Mind benchmarks. An LLM extracts the initial state and action sequence into a predefined PDDL domain; a symbolic executor produces a state trace; and the LLM answers questions using that trace. Evaluation covers MMToM-QA, MuMa-ToM, and FANToM, with three language models and ablations removing verification or the predefined domain. With GPT-4o, verification improves accuracy from 77.4% to 88.3% on MMToM-QA and from 79.6% to 88.8% on MuMa-ToM, without additional LLM calls.

Critical strengths

An inspectable and economical pipeline. Explicit action preconditions and effects make the assumed transition rules accessible to scrutiny. The method uses three LLM calls per example, providing a practical way to incorporate symbolic execution into benchmark evaluation.

Ablations demonstrate the value of executable traces. Providing verified states improves performance over parsed actions alone across all three benchmarks. This supports the practical contribution of symbolic execution, beyond merely reformatting the narrative.

Critical weaknesses

The FANToM human baseline is misidentified. Section 4 describes the human score of 87.5% as a per-question average. However, Table 1 of the original FANToM paper reports this value under “All Question Types,” a joint-correctness metric. The manuscript should correct this attribution and compare human and model performance using matched metrics before drawing conclusions about the remaining gap. This issue concerns the comparison, not the validity of PDDL-MIND’s reported score itself.

The claimed separation of capabilities is stronger than the experimental controls establish. The domain tracks agent-specific observations through seen and heard, supplying information relevant to belief attribution alongside physical states. The ablations establish the benefit of this combined representation but do not separately measure the contributions of physical-state tracking and agent-specific observation tracking. Consequently, they do not establish the conclusion that the missing capability is “bookkeeping rather than inference.”

Questions for authors

FANToM metric comparability. The original FANToM paper reports the human score of 87.5% under “All Question Types,” whereas Section 4 describes it as a per-question average. Could you correct this attribution and report human and PDDL-MIND results under matched metrics? How does this affect your interpretation of the remaining human–model gap?

Separating state tracking from belief-related information. With parsing held fixed, how does performance change when the executor provides only physical-state traces, compared with traces that also include seen and heard? Can this control substantiate the conclusion that the gains primarily reflect improved bookkeeping rather than assistance with belief inference?

1 comment

Thank you for the constructive review! We appreciate your positive comments on the inspectability and efficiency of our pipeline, and on how our ablations show the value of executable traces. We restate our contribution below as a refresher.

Our contribution. We offer insights on benchmarks that can be traced to the data generating process and informs future work. A minimal pipeline (an LLM parses the story into a 7-action PDDL domain with no mental-state rules, a symbolic executor tracks the state, and the LLM answers from the trace) passes human accuracy on MMToM-QA and comes within 5 points on MuMa-ToM. Sections 5 and 6 trace this result to dataset construction and give explicit design principles for more faithful ToM evaluation: Tasks should remain difficult after state tracking is handed to the model, this means more world knowledge.

We address your questions below.

"The original FANTOM paper reports the human score of 87.5% under 'All Question Types,' whereas Section 4 describes it as a per-question average. Could you correct this attribution and report human and PDDL-MIND results under matched metrics? How does this affect your interpretation of the remaining human-model gap?"

Thank you for pointing this out! We originally reported the per-question average in our main tables for two reasons: first, to match the metric reported by recent baselines like EnigmaToM (Table 1), and second, getting every question about a conversation right at once (All) tests cross-question consistency on top of single-question state tracking (Sec. 4).

Following your suggestion, we ran PDDL-MIND, CoT, and DWM with Claude-Sonnet-4.5 on FanToM's All metrics alongside the per-question average:

Method (Claude-Sonnet-4.5)Avg (paper)AllAll (no fact)AllAnsAllInfoAllBelief
CoT40.50.00.00.00.048.8
DWM61.915.015.035.032.536.2
PDDL-MIND79.321.222.552.537.552.5

Two takeaways stand out under the All metrics:

  1. PDDL-MIND still gives a large jump over CoT (0.0% to 21.2% on All, and 0.0% to 52.5% on AllAns and 37.5% on AllInfo) and beats DWM across the board (21.2% vs. 15.0% on All, and 52.5% vs. 36.2% on AllBelief).
  2. Comparing PDDL-MIND's 79.3% per-question average to its 21.2% All score (vs. 87.5% human All) backs up our discussion in L254–269: verified state tracking fixes most errors on individual questions, but answering a whole bundle of questions about the same dialogue without a single slip requires keeping answers consistent across queries. Since PDDL-MIND answers each question independently in its final call, this remaining gap on All points to cross-query belief consistency as a genuine reasoning challenge beyond state tracking. We will add this table and discussion to Section 4.

"Separating state tracking from belief-related information. With parsing held fixed, how does performance change when the executor provides only physical-state traces, compared with traces that also include seen and heard? Can this control substantiate the conclusion that the gains primarily reflect improved bookkeeping rather than assistance with belief inference?"

We'd like to clarify why seen and heard in our PDDL domain are simple state bookkeeping rather than belief inference. In our domain file (App. A, Listing 1), seen and heard don't compute beliefs, intentions, or false-belief states. They just log co-location based on syntax rules:

  • When an agent runs move into a room or open on a container at their location, the simulator logs (seen ?a ?o ?l) for whatever objects are currently in that room or container.
  • When someone speaks via tell, the simulator logs (heard ?a ?u) for whichever agents are currently at (at ?a ?l).

In these stories, tracking who was in the room to see an object or hear a line of dialogue is just as mechanical, or "physical", as tracking where an object is placed. As we show in Sec. 5.1 (Fig. 3a), both come down to basic graph updates: tree traversal for at/in/seen, and broadcast edges to present listeners for said/heard. Neither one tells the model what an agent believes after an object is moved behind their back, what goal an agent has (e.g., why Karen passes over visible chips in Listing 5), or how to resolve conflicting views. The actual belief inference is done by the LM in the final step.

That said, we agree that contrasting "physical state" with "conversational/perceptual state" in our text can make it sound like the simulator is doing mental inference. We will update our wording to distinguish action and observation bookkeeping (handled by PDDL) from belief and goal inference (handled by the LLM from the trace).

We appreciate your helpful suggestions, which have made our analysis and comparisons much clearer. We hope you will consider these updates in your final assessment!

Reviewer_LsfYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether existing Theory of Mind (ToM) benchmarks test ToM reasoning or actually state parsing and tracking. It proposes PDDL-Mind to investigate this and improve current models. Experiments demonstrate the efficacy of PDDL-Mind and show that current ToM benchmarks primarily test state parsing and tracking rather than belief reasoning.

Critical strengths

This paper asks an important question and conducts experiments that explicitly separate the influence of parsing, state tracking, and belief reasoning on the final ToM answer.

Critical weaknesses
  1. The main concern is that the scope of this paper is relatively narrow given current ToM reasoning progress. Specifically, the scope and the discussed benchmarks are within state-tracking-based ToM problems, while many ToM benchmarks have moved beyond this toward more complex social reasoning, such as [r1, r2]. A recent work [r3] that also aims to revisit existing ToM benchmarks (though with a different target issue) investigates a diverse range of existing ToM benchmarks.

  2. The novelty of the proposed method is limited, as it is a more engineering-oriented prompting framework. The core idea of LLM-based semantic parsing into a structured representation and handcrafted symbolic state-transition rules has been largely explored.

  3. The evaluation is not comprehensive. It considers only three datasets with similar state-tracking scenarios. Although within this scope, there are many other datasets, such as ExploreToM [r4]. Furthermore, some training-based baseline methods are not discussed and compared, such as [r5]

Given the above three main concerns, my rating is a clear rejection at this stage.

[r1] Shinoda, Kazutoshi, et al. "Tomato: Verbalizing the mental states of role-playing llms for benchmarking theory of mind." AAAI, 2025.

[r2] Liu, Yiwei, et al. "TactfulToM: Do LLMs Have the Theory of Mind Ability to Understand White Lies?." EMNLP, 2025.

[r3] Zhong, Jike, et al. "From Shortcuts to Reasoning: Robust Post-Training of Theory of Mind with Reinforcement Learning." ICML, 2026.

[r4] Sclar, Melanie, et al. "Explore theory of mind: program-guided adversarial data generation for theory of mind reasoning." ICLR, 2025.

[r5] Zhang, Chunhui, et al. "Overcoming multi-step complexity in multimodal theory-of-mind reasoning: A scalable bayesian planner." ICML, 2025.

Questions for authors

See weaknesses.

(Minor) 1. Why are the FanToM accuracy values in Table 1 (80.00) and Table 2 (GPT-4o Avg: 78.5) different?

(Minor) 2. The language and sentence patterns in this paper read as too AI-generated.

1 comment

Thank you for your review and for noting that our paper tackles an important question by separating the effects of parsing, state tracking, and belief reasoning. We restate our contribution below as a refresher.

Our contribution. Several comments evaluate this paper as another ToM method designed to top benchmark scores. This is not our intention in producing this work. We wish to again state our intended contribution: We offer concrete insight in benchmark artifacts in widely tested theory of mind benchmarks. We position PDDL-MIND as a measuring instrument. Its domain has 7 actions as helper functions and no rules about intention inference, yet with three LLM calls it passes human accuracy on MMToM-QA and comes close on MuMa-ToM. Our finding is benchmark-centric: on the event-based datasets the community uses to report progress in machine ToM, most of the gap between LLMs and humans is state bookkeeping over long text. Sections 5 and 6 trace our results to concrete primitives and how the datasets are constructed (closed-world rules over a small verb set) and establish a clear criterion for future tasks, categorized by degree of world knowledge involvement. Our results prompted us to argue that a ToM benchmark should remain difficult after state tracking scaffolds are handed to the model.

We address your comments below.

"...the scope of this paper is relatively narrow given current ToM reasoning progress. Specifically, the scope and the discussed benchmarks are within state-tracking-based ToM problems, while many ToM benchmarks have moved beyond this toward more complex social reasoning, such as [r1, r2]. A recent work [r3] ... investigates a diverse range of existing ToM benchmarks."

We respectfully disagree. We picked our three benchmarks to cover the main setups in current event-based ToM evaluation where models still lag behind humans (Sec. 2): single-agent actions (MMToM-QA), multi-agent actions and conversation (MuMa-ToM), and multi-agent conversation (FanToM). In fact, we included FanToM specifically because it tests conversational inference beyond physical object tracking. While [r3] evaluates more datasets in total, those datasets fall into these same event-state narrative categories (Sec. 5.1, Fig. 3a).

We also discuss how ToM evaluation can move toward the richer social reasoning in [r1] and [r2]. In Section 6, we outline a hierarchy of how world knowledge enters ToM tasks: while closed-world event benchmarks come down to tracking rules over a fixed set of verbs, harder social tasks require reasoning over subjective states and long-tail social norms (like white lies in [r2] or role-playing in [r1]) that don't reduce to simple state updates. Thank you for the pointers to [r1, r2, r3]; we will cite and discuss them in Sections 2 and 6.

"...novelty of the proposed method is limited, as it is a more engineering-oriented prompting framework." and "...some training-based baseline methods are not discussed and compared, such as [r5]" and "...there are many other datasets, such as Explore ToM [r4]."

We'd like to again clarify that our contribution isn't an engineering pipeline to top benchmark leaderboards. Instead, we use PDDL-MIND as a diagnostic tool to test what current ToM benchmarks are actually measuring, and we spend Sections 5 and 6 analyzing the structural assumptions built into these datasets. We do not observe other works doing the same at rivaled depths.

This is also why we do not compare against post-training or RL methods like [r3] and [r5]: in writing this paper, our goal is to nudge the community to deeply understand benchmark artifacts before training models on these benchmarks. Similarly, while we didn't run on ExploreToM [r4], the way it is built echoes our analysis in Sections 5 and 6. Because ExploreToM generates stories using a symbolic state machine with fixed vocabulary and transition rules, it shares the exact acyclic state update structure we outlined, thus the benchmark also falls prey to removing observer relativity detailed in our philosophical discussions. We will add a discussion of [r4] and [r5] to Sections 2 and 6 to make this connection clear.

"Why are the FanToM accuracy values in Table 1 (80.00) and Table 2 (GPT-4o Avg: 78.5) different?"

Table 1 reports the best PDDL-MIND score across our three evaluated models from Table 2: Kimi-K2-Instruct reaches 80.0% on FanToM (Table 2, L273). We will label the model names directly in Table 1 so this is clear at a glance.

"The language and sentence patterns in this paper read as too AI-generated."

Our use of AI tools followed ICLR guidelines and is properly disclosed in our AI Use Statement. All hypotheses, experiments, and conceptual arguments were developed and written by the authors. Nonetheless, we will do a thorough pass in our text for phrasings that may appear confusing. We hope reviewers will evaluate the paper on its scientific and conceptual contributions.

Reviewer_w8UoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether Theory of Mind benchmarks mostly test parsing and state tracking by language models rather than belief reasoning. Prior agentic pipelines raise accuracy with many model calls but cannot say which capability the extra compute compensates for. In PDDL-Mind, a language model parses a story into a PDDL problem and candidate actions, which an executor runs against a fixed hand-written domain to produce a verified state trace. The model answers from that trace; ablations remove verification or replace the fixed domain with a self-induced one. Experiments on subsets of three benchmarks, mainly with GPT-4o, approach the reported human level on the two action-based benchmarks with three calls. A small manual failure analysis attributes most remaining errors to reasoning.

Critical strengths
  • The use of symbolic engine to help with accurate task execution makes sense.
  • The paper is clear and easy to follow. The motivation is mostly strong.
Critical weaknesses
  • Novelty. There are existing works that tackle ToM tasks with external symbolic engine. For example, ToM-LM. While the implementation differs (e.g., the exact engine used), the motivation and idea are similar. This raises novelty concern. I also believe it is worth discussing ToM-LM in the paper.
  • Decomposition. I think the paper provides useful evidence that structured state tracking improves ToM benchmark performance. But it does not provide enough evidence towards decomposition into parsing, state tracking and belief reasoning. To do so, it should conduct controlled experiments such as GT PDDL actions + LLM state tracking, and use the gap to illustrate how much each stage accounts for the problem.
  • Significance. I feel that generalizing a specific experiment on ToM + symbolic engine to the decomposition of "the gap between end-to-end LLM accuracy and human accuracy" is a bit too ambitious.
Questions for authors

See weaknesses.

1 comment

Thank you for the thoughtful review! We're glad you found the paper clear, easy to follow, and well motivated. We restate our contribution below as a refresher.

Our contribution. We offer explicit insights on benchmark artifacts. A minimal pipeline (an LLM parses the story into a 7-action PDDL domain with no mental-state rules, a symbolic executor tracks the state, and the LLM answers from the trace) passes human accuracy on MMToM-QA and comes within 5 points on MuMa-ToM. If such a bookkeeping engine closes most of the model-human gap, high scores on these datasets mostly certify state tracking over long text. Sections 5 and 6 trace this to dataset construction and give explicit design principles: a ToM task should remain difficult after state tracking is handed to the model, this means more world knowledge.

We address your comments and questions below.

"...existing works that tackle ToM tasks with external symbolic engine. For example, ToM-LM. ... worth discussing ToM-LM in the paper."

Thanks for the pointer to ToM-LM! Prior neuro-symbolic methods, including ToM-LM as well as BIP-ALM, LIMP, and S3AP\text{S}^3\text{AP} (which we compare against in Table 1), use symbolic or probabilistic engines to maximize accuracy. Our goal with PDDL-MIND is different: we use it as a diagnostic tool rather than a general-purpose solver.

Specifically, we restrict our predefined PDDL domain to just 4 object types and 7 basic actions.

"...does not provide enough evidence towards decomposition into parsing, state tracking and belief reasoning. To do so, it should conduct controlled experiments such as GT PDDL actions + LLM state tracking, and use the gap to illustrate how much each stage accounts for the problem."

Our ablations in Tables 1 and 2 set up this exact comparison between parsed PDDL actions with LLM state tracking and symbolic state tracking:

  1. Parsing alone (PDDL-MIND w/o verification): The LLM gets the parsed PDDL action sequence (a1,…,at)(a_1, \dots, a_t) without the symbolic executor, so it still has to track the state and infer beliefs on its own (Sec. 3, L209; Table 1). Moving from 1-call CoT (Table 2) to this variant with GPT-4o raises accuracy from 61.4% to 77.4% (+16.0 points) on MMToM-QA, 41.8% to 79.6% (+37.8) on MuMa-ToM, and 45.3% to 74.7% (+29.4) on FanToM.
  2. Adding symbolic state tracking (Full PDDL-MIND): Running the PDDL verifier replaces LLM state tracking with verified state traces at no extra LLM calls. This raises GPT-4o accuracy from 77.4% to 88.3% (+10.9) on MMToM-QA, 79.6% to 88.8% (+9.2) on MuMa-ToM, and 74.7% to 80.0% (+5.3) on FanToM.
  3. Removing the predefined schema (PDDL-MIND w/o predefined domain): Asking the model to write its own domain drops accuracy across all three benchmarks (74.9%, 78.9%, and 69.7%).

To check how much residual parsing errors affect the final stage, we manually inspected 40 failure cases from full PDDL-MIND (Sec. 4, L300–323; App. B.2). Out of the 12% total error rate, one-third (4% overall) come from parsing or state-tracking mistakes, while two-thirds (8% overall) are genuine belief or counterfactual reasoning mistakes even when a clean trace is given.

Variant (GPT-4o)What is outsourcedMMToM-QAMuMa-ToMFanToM
1-call CoTNothing (end-to-end LLM)61.441.845.3
PDDL-MIND w/o domainParsing (LLM invents rules)74.978.969.7
PDDL-MIND w/o verificationStructured event parsing77.479.674.7
Full PDDL-MINDParsing + PDDL state tracking88.388.880.0

"...generalizing a specific experiment on ToM + symbolic engine to the decomposition of 'the gap between end-to-end LLM accuracy and human accuracy' is a bit too ambitious."

We don't mean to claim that human Theory of Mind reduces to state tracking. Our claim refers specifically to the model-human gap on the three benchmarks tested and any other benchmark that follows similar designs, which cover single-agent actions (MMToM-QA), multi-agent actions and dialogue (MuMa-ToM), and multi-agent dialogue (FanToM).

While our 3-call PDDL setup gets close to human scores on MMToM-QA (88.3% vs. 82.5%) and MuMa-ToM (88.8% vs. 93.5%), our takeaway is that this happens because of how current benchmarks set up ToM tasks. As we discuss in Sec. 6 (Fig. 4), humans naturally track scenes using everyday background knowledge, whereas LLMs struggle to keep track of multi-step state updates over long text. Because these benchmarks use a small set of verbs and closed-world rules (Sec. 5.1, Sec. 6), recovering the state-transition rules solves most of the problem without needing deep belief inference. We will tighten the wording in the Abstract and Section 6 to make it clear that our decomposition applies to the empirical gap on these evaluated benchmarks, and serves as a check for designing future tasks!

We hope these clarifications address your concerns, and we're happy to discuss further!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.