- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces PDDL-MIND, a pipeline intended to distinguish narrative parsing, state tracking, and belief reasoning in Theory of Mind benchmarks. An LLM extracts the initial state and action sequence into a predefined PDDL domain; a symbolic executor produces a state trace; and the LLM answers questions using that trace. Evaluation covers MMToM-QA, MuMa-ToM, and FANToM, with three language models and ablations removing verification or the predefined domain. With GPT-4o, verification improves accuracy from 77.4% to 88.3% on MMToM-QA and from 79.6% to 88.8% on MuMa-ToM, without additional LLM calls.
Critical strengths
An inspectable and economical pipeline. Explicit action preconditions and effects make the assumed transition rules accessible to scrutiny. The method uses three LLM calls per example, providing a practical way to incorporate symbolic execution into benchmark evaluation.
Ablations demonstrate the value of executable traces. Providing verified states improves performance over parsed actions alone across all three benchmarks. This supports the practical contribution of symbolic execution, beyond merely reformatting the narrative.
Critical weaknesses
The FANToM human baseline is misidentified. Section 4 describes the human score of 87.5% as a per-question average. However, Table 1 of the original FANToM paper reports this value under “All Question Types,” a joint-correctness metric. The manuscript should correct this attribution and compare human and model performance using matched metrics before drawing conclusions about the remaining gap. This issue concerns the comparison, not the validity of PDDL-MIND’s reported score itself.
The claimed separation of capabilities is stronger than the experimental controls establish. The domain tracks agent-specific observations through seen and heard, supplying information relevant to belief attribution alongside physical states. The ablations establish the benefit of this combined representation but do not separately measure the contributions of physical-state tracking and agent-specific observation tracking. Consequently, they do not establish the conclusion that the missing capability is “bookkeeping rather than inference.”
Questions for authors
FANToM metric comparability. The original FANToM paper reports the human score of 87.5% under “All Question Types,” whereas Section 4 describes it as a per-question average. Could you correct this attribution and report human and PDDL-MIND results under matched metrics? How does this affect your interpretation of the remaining human–model gap?
Separating state tracking from belief-related information. With parsing held fixed, how does performance change when the executor provides only physical-state traces, compared with traces that also include seen and heard? Can this control substantiate the conclusion that the gains primarily reflect improved bookkeeping rather than assistance with belief inference?
Thank you for the constructive review! We appreciate your positive comments on the inspectability and efficiency of our pipeline, and on how our ablations show the value of executable traces. We restate our contribution below as a refresher.
Our contribution. We offer insights on benchmarks that can be traced to the data generating process and informs future work. A minimal pipeline (an LLM parses the story into a 7-action PDDL domain with no mental-state rules, a symbolic executor tracks the state, and the LLM answers from the trace) passes human accuracy on MMToM-QA and comes within 5 points on MuMa-ToM. Sections 5 and 6 trace this result to dataset construction and give explicit design principles for more faithful ToM evaluation: Tasks should remain difficult after state tracking is handed to the model, this means more world knowledge.
We address your questions below.
Thank you for pointing this out! We originally reported the per-question average in our main tables for two reasons: first, to match the metric reported by recent baselines like EnigmaToM (Table 1), and second, getting every question about a conversation right at once (
All) tests cross-question consistency on top of single-question state tracking (Sec. 4).Following your suggestion, we ran PDDL-MIND, CoT, and DWM with Claude-Sonnet-4.5 on FanToM's
Allmetrics alongside the per-question average:Two takeaways stand out under the
Allmetrics:All, and 0.0% to 52.5% onAllAnsand 37.5% onAllInfo) and beats DWM across the board (21.2% vs. 15.0% onAll, and 52.5% vs. 36.2% onAllBelief).Allscore (vs. 87.5% humanAll) backs up our discussion in L254–269: verified state tracking fixes most errors on individual questions, but answering a whole bundle of questions about the same dialogue without a single slip requires keeping answers consistent across queries. Since PDDL-MIND answers each question independently in its final call, this remaining gap onAllpoints to cross-query belief consistency as a genuine reasoning challenge beyond state tracking. We will add this table and discussion to Section 4.We'd like to clarify why
seenandheardin our PDDL domain are simple state bookkeeping rather than belief inference. In our domain file (App. A, Listing 1),seenandhearddon't compute beliefs, intentions, or false-belief states. They just log co-location based on syntax rules:moveinto a room oropenon a container at their location, the simulator logs(seen ?a ?o ?l)for whatever objects are currently in that room or container.tell, the simulator logs(heard ?a ?u)for whichever agents are currently at(at ?a ?l).In these stories, tracking who was in the room to see an object or hear a line of dialogue is just as mechanical, or "physical", as tracking where an object is placed. As we show in Sec. 5.1 (Fig. 3a), both come down to basic graph updates: tree traversal for
at/in/seen, and broadcast edges to present listeners forsaid/heard. Neither one tells the model what an agent believes after an object is moved behind their back, what goal an agent has (e.g., why Karen passes over visible chips in Listing 5), or how to resolve conflicting views. The actual belief inference is done by the LM in the final step.That said, we agree that contrasting "physical state" with "conversational/perceptual state" in our text can make it sound like the simulator is doing mental inference. We will update our wording to distinguish action and observation bookkeeping (handled by PDDL) from belief and goal inference (handled by the LLM from the trace).
We appreciate your helpful suggestions, which have made our analysis and comparisons much clearer. We hope you will consider these updates in your final assessment!