- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue that final accuracy metrics in Theory of Mind (ToM) evaluations fail to distinguish between different types of failure: outputs that are rejected, structured states that are unable to answer the question, and executable states that yield incorrect answers. Building upon FixedToMi, the authors developed StructToM-Bench—a dataset comprising 1,000 story-question pairs (based on 875 unique stories)—and provided deterministic event updaters and query procedures. The study compares two approaches: direct answering versus an "event graph" output path. In the latter, the answer is computed via an evaluation program, while any answer generated simultaneously by the model is excluded from the primary scoring for this path. The authors conducted five evaluation runs across 22 different models and deployment configurations, applying a predefined set of scoring rules to the model responses. Key findings include: scoring rules can significantly alter reported performance and rankings; most "critical" failures in the structured path occur prior to query execution; and the original structured-scoring mechanism erroneously penalized models for omitting information that the prompt explicitly instructed them to leave out. Validation efforts included blind human adjudication of 31 instances of label disagreement, statistical analysis clustered by story, prompt ablation studies, prospective prompting experiments, comparisons across deployment configurations, and direct-answering experiments on the OpenToM and FANToM datasets. The authors explicitly limit their conclusions to the behavior of the specific evaluation pipelines tested, cautioning against interpreting these scores or performance gains as reflections of a model’s intrinsic ToM capabilities.
Critical strengths
- A key strength of this paper is that it applies different scoring rules to the same set of preserved responses, ensuring that variations in scores are not confounded by changes resulting from answer regeneration. For instance, the "strict" score for Llama-3.2-1B is zero, whereas two other rules—derived from existing evaluators—yield a score of approximately 0.577. This clearly demonstrates that the "strict" score can lead to vastly different outcomes compared to other acceptance criteria. The use of a fixed denominator and conditional coverage further enhances the interpretability of this audit. For example, a high accuracy rate on queryable edges is not mistakenly conflated with overall performance across all planned invocations. By treating the model, prompt, evaluator, and deployment configuration collectively as the object of evaluation, the paper aligns well with its observation that performance is sensitive to deployment settings.
- The issue identified in the paper regarding SBGA stems from a clear, identifiable mechanism: the prompt instructs the model to omit
container_ininformation, yet the original evaluator includes these edges in the denominator for recall calculations. Since these edges account for 30.2% of the "pooled gold edges" and never result in a match, the upper bound for the pooled recall across the four edge categories is effectively capped at approximately 0.698. This finding illustrates that a consistent structural omission across all systems cannot be automatically attributed to a shared representational flaw in the models; rather, the evaluation design itself can produce such an outcome. The paper retains the original (pre-correction) results while explicitly noting that the scores for the three edge categories were adjusted post-hoc, allowing readers to examine both the rationale for these corrections and their impact. Furthermore, the credibility of the data audit is bolstered by a blinded human adjudication process for 31 cases of label disagreement—resulting in consensus for 29 cases and a 4-to-1 majority for the remaining two. The authors also disclose the scope of this human verification, as well as the results from prompt experiments that failed to meet the established joint criteria. Such transparency strengthens the credibility of the core observations, going well beyond a mere perfunctory statement of limitations.
Critical weaknesses
W1. Section 4 and Appendix G demonstrate that the same response can yield different scores under different acceptance criteria, confirming that reported scores are sensitive to the chosen criteria. However, the criteria derived from OpenToM (based on lexical overlap) and FANToM (based on bidirectional substring inclusion) do not directly assess whether the response, in its entirety, semantically selects the correct answer. Consequently, the "recovered" scores may encompass both correct answers that were previously rejected due to strict formatting errors, and incorrect matches resulting from the mere mention of an answer, the use of negation, or other contextual factors. The paper does not report false acceptance or false rejection rates based on an independent, blinded adjudication of these recovered responses.
This gap is particularly significant for cases where scores increase from zero to approximately 0.577 or 0.610, as the scientific validity of these results hinges on whether the newly accepted responses actually convey the correct answer. The fact that "punctuation-tolerant" criteria result in identical scores across 20 configurations does not rule out false acceptances stemming from lexical overlap or substring matching, as the sets of responses accepted under these different criteria are not necessarily the same. Therefore, the use of this "near-invariant" control in Section 4 is insufficient to definitively refute the argument that the observed performance gains are merely artifacts of more lenient scoring criteria.
W2. While the three-stage decomposition presented in the paper is mathematically sound, the specific stage at which a failure is first detected does not necessarily correspond to a distinct type of error requiring a unique remediation strategy. Appendix L provides a telling example: the omission of a "leave" event can result in a graph that remains "queryable" but yields an incorrect answer—meaning such an omission is categorized under "query-result correctness." Conversely, the absence of a temporal field might lead to a response being rejected at an earlier stage; yet, supplying the missing timestamp (e.g., by inferring it from the sequence of entries) could simultaneously alter the executability of subsequent states.
This implies that the reported proportions—77.1%, 17.5%, and 5.4%—merely describe where failures are captured under the current acceptance criteria and query procedures; they do not, by themselves, indicate which types of defects are most valuable to address. Furthermore, the high accuracy rate observed for "queryable" graphs applies only to the specific subset of responses that successfully passed through the first two stages of the pipeline. The authors explicitly acknowledge that these stages do not represent independent causal mechanisms. Consequently, my concern is not that the decomposition formula itself is incorrect, but rather that the current evidence fails to demonstrate how much additional diagnostic value this framework provides beyond standard logs that simply track parsing, execution, and final answer errors.
This gap in the paper’s contribution is further constrained by the scope of its external validation. The OpenToM and FANToM experiments (Section 7 and Appendix C) only assess performance gaps along the direct answer path; they do not evaluate the outcomes of the structured three-stage process. As a result, the most distinctive aspects of the paper remain confined to a single set of template corpora, event schemas, and update mechanisms.
W3. The prospective prompting experiments—presented in Section 7 and Appendix I—serve as an important validation step, as they attempt to determine whether "strict" accuracy scores align with "rule-recovery" scores when output examples are modified. However, only six of the nine experimental configurations fall within the predicted intervals, and even the control groups do not remain entirely stable. Furthermore, even when using a baseline derived from a post-hoc matched subset, Qwen3-8B shows a decline of approximately 0.081, meaning the pre-registered joint criteria are not met.
The authors’ interpretation of these results as "partial support" is a reasonable framing. Nevertheless, eleven of the fourteen experimental configurations rely on routed calls to upstream services that are not held constant. Given that the paper’s own deployment comparisons demonstrate that changes in service conditions can significantly alter strict accuracy rates, it is difficult to disentangle the respective impacts of prompt modifications, service-side fluctuations, and changes in the actual response content.
More critically, the reproducibility statement on page 9 indicates that for the prospective prompting experiments, external data experiments, and "permission-menu" probes, only aggregated results (grouped by configuration) have been retained; the raw responses are not available for reconstruction. While this does not diminish the value of the full response data preserved for the main experiments, it means that the key evidence supporting the authors' interpretations and claims of generalizability cannot be subjected to the same level of rigorous audit. Without access to the raw data, it is impossible to verify whether "recovered" responses are semantically correct, or to determine whether certain failures stem from formatting issues versus service-level anomalies.
If these experiments are intended to validate the paper’s core explanatory framework, it is essential to preserve item-level paired results, complete responses, and service metadata—and to conduct such comparisons under fixed deployment conditions.
Questions for authors
Q1. What proportion of the "recovered" (re-scored as correct) responses are semantically accurate? For responses newly marked as correct under the OpenToM-derived and FANToM-derived rules, was there a manual review conducted by individuals blinded to the initial scoring outcomes? Please report the false acceptance rate for each set of rules, specifically including the "strict zero-score" configuration for Llama-3.2-1B and Phi-4-mini. This distinction is crucial: does a significant "recovery" rate indicate that valid answers were previously (and incorrectly) rejected, or does it primarily reflect the idiosyncratic behavior of different matching heuristics? Q2. Can the three-stage localization framework effectively guide the selection of appropriate corrective measures? For a specific, known error—such as a missed observational event, an omitted temporal marker, or a missing query entity—how does the failure "shift" across the three stages when the output-encoding rules are modified? Is there an analysis of actual model responses demonstrating that the stage-based statistics correctly informed subsequent, validated fixes? The examples in Appendix L are illustrative constructs and, therefore, cannot independently validate the diagnostic efficacy of the framework on real-world model outputs. Q3. In the prospective experiments, can prompt-based modifications be clearly disentangled from deployment-related changes? Among the nine experimental and five control configurations, which can be confirmed to have utilized identical model versions, upstream services, and inference parameters? Does the evidence for "recovery" hold up when the analysis is restricted to configurations using fixed endpoints? For experiments where full response logs were not retained, are there verifiable, item-level paired records available? Q4. Does the validation using "executable gold" (ground truth) encompass the state-update process, or is it limited solely to the final binary/choice-based answer? While the manual audit in Appendix L provides strong support for the 31 disputed labels, both the model-generated events and the gold-standard events are processed by the same update engine. Have the observational relationships, cross-room events, and second-order belief updates been independently verified? Furthermore, has the framework been checked for cases where the final answer is correct, but the intermediate state representation is erroneous? Additionally, Appendix F notes that "UNKNOWN" is used when container-to-scene mappings are missing. Does this default value affect only the structural edge matching, or does it also influence observer-related determinations and query results? This distinction is critical for assessing whether the adjustments to the scorer are sufficiently robust.
4 comments
Thank you for the blinded semantic audit and the new fixed-endpoint experiments with retained responses. These substantially address my concerns about newly awarded credit and the auditability of the new experiments.
Two key questions remain:
- Given that UNKNOWN affects observer assignment and 58 gold answers depend on container_in, can you independently validate the observer and belief traces for affected cases and report how any necessary corrections change the gold answers and main stage-wise results?
- Since the fixed-endpoint experiment did not meet the registered criteria, what evidence still supports attributing recovery specifically to contract repair? How will the conclusions distinguish supported scoring observations from the mechanism claims that this experiment did not validate? I would consider raising my score if independent state validation supports the main findings and the conclusions are revised consistently with the fixed-endpoint results.
Reply to follow-up: trace validation and the contract-repair claim
Thank you. In short: a separately written simulator reproduces every gold observer set, belief and answer except under one convention (what a character learns on entering a room), which would flip 30 labels, none in the ToM strata; rejection stays the dominant failure stage. We withdraw the contract-repair claim and revise the conclusions to keep mechanism claims apart from the scoring observations the data support.
1. Independent validation of the traces. Since our first reply we have run this check in code. On all 1,000 items, a separate simulator reproduces every gold observer set, 2,000 of 2,000, including all 116 placements in the 58 container_in-dependent items. An AI coding agent wrote it from the story text and ToMi's public generator code and paper, without opening our engine, gold events, or labels; its task prompt, self-reported access log, rules and code will be in the supplementary material. Like our gold, it reads "The C is in the R." as a fixed fact, because ToMi's generator places containers once, before any action. The query path fills gold scenes from those lines, so no gold placement is UNKNOWN (134 of 2,000 would be without it, 58 in the affected items); model graphs never get this fill-in (the UNKNOWN problem in our disclosure).
Beliefs differ through one rule only: ToMi's generator lets a character who enters a room learn where the objects there now are, while our gold updates a belief only on a witnessed placement. With that rule off, all 7,645 belief entries and all 1,000 answers agree. With it on, the simulator matches FixedToMi's labels on 999 of 1,000 items and differs from our gold on 30 answers, all in the no-ToM operations and all among the 31 source-label disagreements our blinded adjudication resolved to our gold (31 of 31; 153 of 155 votes). We keep the gold answers, but that was a two-way choice by author-recruited adjudicators, so here is the effect of the other labels:
- Unchanged: memory, reality, all four ToM strata, and rejection as the dominant failure stage.
- Pooled strict graph accuracy: 0.483 → 0.475; strict failures 77.1 / 17.5 / 5.4% → 75.8 / 17.2 / 7.0% (rejected / unqueryable / wrong). No configuration moves more than 0.023.
- Table 2, no-ToM graph minus direct: first-order 0: -0.029 → -0.074; first-order 1: -0.027 → -0.055; second-order 0: +0.206 → +0.180; second-order 1: +0.249 → +0.236. Intervals and Holm tests are not yet rerun.
The revised paper will state the entry rule as a benchmark convention and report both label sets.
Human check. Two non-author annotators, blind to the gold, will label the same traces: the 58 affected items and 58 unaffected ones matched on question type, 360 questions (who saw each placement of the queried object, the queried first- or second-order belief, and 12 attention checks). It was registered before the sample was drawn and narrowed to these questions before any label. Annotators are given the benchmark's observation rule, so it tests whether our traces apply that rule, not which entry rule is right. If both annotators' labels are back by noon AoE on 18 November, we will post the result here, whichever way it falls; otherwise it goes in the revised paper.
2. Contract repair. We withdraw it as the mechanism: no remaining evidence isolates it. Both prospective tests failed their registered criteria; in the fixed-endpoint test two controls rose by more than 0.05, which the registration reads as a generally better prompt. Qwen2.5-7B's strict recovery (+0.668) is a format change: under the OpenToM-derived rule, which already credited v1 at 0.673, the same change is +0.008, and one cell with rising controls cannot separate it from a better prompt.
What the data support are scoring observations on saved responses:
- declared acceptance rules change scores and rankings (Llama-3.2-1B: 0.000 strict, 0.577 under the OpenToM- and FANToM-derived rules; on the 14 fixed-host configurations, 10 of 14 ranks change);
- newly awarded credit matches the option the response selects (blinded check of direct responses: 80 of 80, and 116 of 120 where rules disagree);
- most structured-path failures are rejections (77.1% under the published scorer, 81.0% under the gold-map replay).
These describe what the scorer does with a response, not why a model fails the format.
Revisions to the conclusions:
- The "output-contract gap" is defined only as declared-rule minus strict accuracy, never as a cause.
- Section 7 and Appendix I report both prompt tests as not supporting the mechanism, and the fixed-endpoint test as falsified by its registered criteria.
- The Limitations and the Evidence Status table no longer call the earlier test partial support.
- Appendix K's "contract component" is recast as a decomposition of score differences.
- The conclusion lists the supported scoring observations separately from mechanisms that remain untested (belief tracking) or were not supported (contract repair).
Result of the registered human trace check
As promised in our previous comment, here is the result of the registered human check. Two non-author annotators, blind to the gold, labeled 116 items; both passed the attention gate (12 of 12 and 11 of 12; the gate is 10).
- No trace element contradicted the gold. Of 348 elements, both annotators gave the gold value (confirmed) on 338; on none did they agree on a different value; on 10 they disagreed with each other.
- Affected items (58): 166 of 174 confirmed (0.954; exact 95% CI 0.911 to 0.980): observers 110 of 116, first-order 23 of 24, second-order 33 of 34. Matched unaffected items (58): 172 of 174 (0.989; 0.959 to 0.999).
- The 10 disagreements: in every one, one annotator gave the gold value. Of the other answers, six name a different character, two omit one observer, and two give a different belief (one "No idea", one the object's real location).
- Agreement: kappa 0.981 on beliefs, 0.957 per character on observers.
- Gold answers: none corrected. The registered scorer re-derives each answer from the queried belief: 114 of 116 unchanged, and 2 unresolved (both affected items) because the annotators disagreed on that belief; these keep their published answer, which one annotator gave in both. In 4 more items an observer disagreement involves a queried character, but both annotators gave the gold belief; counting these as unresolved too, no answer changes. The stage-wise results are therefore unchanged.
As registered, this tests whether our traces apply the benchmark's observation rule; it does not decide the entry-rule convention discussed in our previous comment. The registration, labels and scorer will be in the supplementary material.
Reply to Reviewer 1YGL
Thank you for a careful review. Your Q4 prompted the audit we report in the disclosure comment.
W1/Q1 (is recovered credit semantically valid?). For the two strict-zero models the sample shows no false acceptance: blinded annotators confirmed the credited option in 40 of 40 distinct sampled responses for Llama-3.2-1B and 40 of 40 for Phi-4-mini (80 of 80; the precision of newly awarded credit has exact 95% interval [0.955, 1.000]). Where two rules disagree they confirmed 116 of 120, pooled (precision 0.967; exact 95% interval [0.917, 0.991], story-cluster bootstrap [0.932, 0.992]). We give no false-acceptance rate per rule or per model for that stratum: it has 40 items per model, too few, so it was registered as pooled only, with no comparison between rules. Of the four credits not confirmed, 2 stayed unresolved and remain in the denominator, 1 was labeled ambiguous, and one matched the other option. The check was registered before any label was collected: 240 items, labeled by two annotators who are not the authors and did not see the model, the rule, or the gold label. Both passed the attention gate (40 of 40 controls each; the gate was 34 of 40) and agreed with kappa 0.980 on the 200 study items. False rejection was not measured: the still-rejected responses in the sample are attention controls, not a sample drawn to estimate it. The check covers direct responses, not graph credit. On the last part of Q1: in the sample, new credit matches the option the response selects, with the four exceptions above. That is correct credit, not competence: under their best declared rule the direct scores are 0.577 for Llama-3.2-1B and 0.610 for Phi-4-mini, against a two-option chance level of 0.50.
W2/Q2 (diagnostic value beyond parse/execute/answer logs). What it adds: the accounting isolates unqueryable graphs as their own stage. At least one put or move with an unresolved scene co-occurs with 5,146 of the 9,951 unqueryable graphs (52%). Filling a scene only where the updater returns
UNKNOWNrecovers 2,682 of the 2,707 gold-map gain (99.1%): 2,724 graphs become queryable and none go the other way. On selection: accuracy among queryable graphs is conditional on reaching that stage, but newly queryable graphs are 98.6% correct, against 94.6% at baseline. On the "missing query entity" case: re-deriving one reason per unqueryable graph gives 4,967known_unknown_belief, 4,818missing_belief, and 166missing_memory_event, so none is a missing query entity by the scorer's codes, though in 2,300 graphs no event mentions a query entity. Gemma-4-E2B's event-time fallback (strict 0.003 to 0.654) is supporting evidence only: a post-hoc scorer intervention, and 126 of its 3,254 correct gains come through a wrong event order. External validation covers the direct path only, and the Appendix L trace is illustrative.W3/Q3 (prompt versus deployment; item-level records). In short: on fixed endpoints one of two treatments recovered and the registered criteria were not met; the earlier prompt test kept only aggregates, so it has no item-level paired records. In the earlier test, 6 of 9 treatments were in band, including all three on dedicated endpoints (Gemma-4-E4B narrowly); there was no dedicated control; the Qwen3-8B control changed by -0.068 against the registered baseline and -0.081 against the matched one; and some baselines came from a different deployment than their v3 run. The new paired test holds the deployment fixed: v1 and v3 side by side on the 4 dedicated endpoints, with item-level records and raw responses. It varies the contract only. By its registered criteria the result is falsified: Qwen2.5-7B's direct treatment recovered in band (strict +0.668; the gain depends on the scoring rule, see our uzeL Q4 reply), Llama-3.2-1B's did not (0.000), and two controls rose by more than 0.05 (Granite-4.1-8B direct +0.076, Qwen3-14B graph +0.081), while Granite-4.1-8B's graph control fell (-0.202). As registered, a control rising that far means v3 is a generally better prompt rather than a contract repair, which weakens the Appendix I mechanism claim; and Llama-3.2-1B not recovering means its failure on the dedicated endpoint is not contract demonstration.
Q4 (gold validity: state updates, cross-room observation,
UNKNOWN). Yes,UNKNOWNaffects observer assignment: an unresolved move gets no observers, not even its actor (disclosure comment). The gold event lists' own answers depend oncontainer_infor 58 of 1,000 items. Gold events are converted, not independently annotated, and the human audit covered only final answers on the 31 disputed items. The check of right answers reached through a wrong state, against independently adjudicated observer and belief traces, is not done; it needs adjudicated traces we do not yet have and is planned for the revised paper.The remaining limits are listed under "What we concede" in our summary comment.