acceptodds
Under review as a conference paper at ICLR 2027

StructToM-Bench: Locating Where Accuracy Is Lost in Theory-of-Mind Evaluation

Abstract

Theory-of-mind (ToM) benchmarks usually report final-answer accuracy, which can conflate an incorrect answer with a rejected output or, for structured methods, a state that cannot answer the query. We introduce StructToM-Bench, a controlled benchmark of 1000 items with executable gold events, evaluated through direct answering and model-generated event graphs. In a preregistered, blinded two-choice audit, a majority of five human annotators favored the executable gold on each of the 31 items where it disagrees with the source label. For the graph path, strict accuracy separates into three observable stages: output compliance, structured-state availability, and query-result correctness. Across 22 model–deployment configurations of 1–15B parameters, evaluated over five passes against fixed denominators, at least one declared acceptance rule raises the direct score for 10 configurations. Two strict zeros rise to 0.577 and 0.610, while a punctuation-only check leaves 20 of 22 scores unchanged. Among strict graph-path failures, 77.1% are rejected outputs, 17.5% are accepted but unqueryable graphs, and 5.4% are queryable graphs whose computed answer differs from gold. An audit of our structural edge metric found that 30.2% of its gold edges belonged to a family the prompt instructed models to omit, so the metric penalized models for following the prompt. Because positional heuristics selected per operation on this corpus outperform every configuration within each operation, we interpret these findings as measurements of the evaluated pipelines rather than evidence of ToM competence. We recommend reporting strict accuracy alongside acceptance sensitivity, queryable coverage, and scorer and deployment provenance.

Primary area:
datasets and benchmarks
Keywords:
theory-of-mind evaluation, evaluation validity, language-model benchmarking, structured elicitation, event graphs, scorer design, acceptance sensitivity, compact language models, deployment sensitivity, preregistration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1YGLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the issue that final accuracy metrics in Theory of Mind (ToM) evaluations fail to distinguish between different types of failure: outputs that are rejected, structured states that are unable to answer the question, and executable states that yield incorrect answers. Building upon FixedToMi, the authors developed StructToM-Bench—a dataset comprising 1,000 story-question pairs (based on 875 unique stories)—and provided deterministic event updaters and query procedures. The study compares two approaches: direct answering versus an "event graph" output path. In the latter, the answer is computed via an evaluation program, while any answer generated simultaneously by the model is excluded from the primary scoring for this path. The authors conducted five evaluation runs across 22 different models and deployment configurations, applying a predefined set of scoring rules to the model responses. Key findings include: scoring rules can significantly alter reported performance and rankings; most "critical" failures in the structured path occur prior to query execution; and the original structured-scoring mechanism erroneously penalized models for omitting information that the prompt explicitly instructed them to leave out. Validation efforts included blind human adjudication of 31 instances of label disagreement, statistical analysis clustered by story, prompt ablation studies, prospective prompting experiments, comparisons across deployment configurations, and direct-answering experiments on the OpenToM and FANToM datasets. The authors explicitly limit their conclusions to the behavior of the specific evaluation pipelines tested, cautioning against interpreting these scores or performance gains as reflections of a model’s intrinsic ToM capabilities.

Critical strengths
  1. A key strength of this paper is that it applies different scoring rules to the same set of preserved responses, ensuring that variations in scores are not confounded by changes resulting from answer regeneration. For instance, the "strict" score for Llama-3.2-1B is zero, whereas two other rules—derived from existing evaluators—yield a score of approximately 0.577. This clearly demonstrates that the "strict" score can lead to vastly different outcomes compared to other acceptance criteria. The use of a fixed denominator and conditional coverage further enhances the interpretability of this audit. For example, a high accuracy rate on queryable edges is not mistakenly conflated with overall performance across all planned invocations. By treating the model, prompt, evaluator, and deployment configuration collectively as the object of evaluation, the paper aligns well with its observation that performance is sensitive to deployment settings.
  2. The issue identified in the paper regarding SBGA stems from a clear, identifiable mechanism: the prompt instructs the model to omit container_in information, yet the original evaluator includes these edges in the denominator for recall calculations. Since these edges account for 30.2% of the "pooled gold edges" and never result in a match, the upper bound for the pooled recall across the four edge categories is effectively capped at approximately 0.698. This finding illustrates that a consistent structural omission across all systems cannot be automatically attributed to a shared representational flaw in the models; rather, the evaluation design itself can produce such an outcome. The paper retains the original (pre-correction) results while explicitly noting that the scores for the three edge categories were adjusted post-hoc, allowing readers to examine both the rationale for these corrections and their impact. Furthermore, the credibility of the data audit is bolstered by a blinded human adjudication process for 31 cases of label disagreement—resulting in consensus for 29 cases and a 4-to-1 majority for the remaining two. The authors also disclose the scope of this human verification, as well as the results from prompt experiments that failed to meet the established joint criteria. Such transparency strengthens the credibility of the core observations, going well beyond a mere perfunctory statement of limitations.
Critical weaknesses

W1. Section 4 and Appendix G demonstrate that the same response can yield different scores under different acceptance criteria, confirming that reported scores are sensitive to the chosen criteria. However, the criteria derived from OpenToM (based on lexical overlap) and FANToM (based on bidirectional substring inclusion) do not directly assess whether the response, in its entirety, semantically selects the correct answer. Consequently, the "recovered" scores may encompass both correct answers that were previously rejected due to strict formatting errors, and incorrect matches resulting from the mere mention of an answer, the use of negation, or other contextual factors. The paper does not report false acceptance or false rejection rates based on an independent, blinded adjudication of these recovered responses.

This gap is particularly significant for cases where scores increase from zero to approximately 0.577 or 0.610, as the scientific validity of these results hinges on whether the newly accepted responses actually convey the correct answer. The fact that "punctuation-tolerant" criteria result in identical scores across 20 configurations does not rule out false acceptances stemming from lexical overlap or substring matching, as the sets of responses accepted under these different criteria are not necessarily the same. Therefore, the use of this "near-invariant" control in Section 4 is insufficient to definitively refute the argument that the observed performance gains are merely artifacts of more lenient scoring criteria.

W2. While the three-stage decomposition presented in the paper is mathematically sound, the specific stage at which a failure is first detected does not necessarily correspond to a distinct type of error requiring a unique remediation strategy. Appendix L provides a telling example: the omission of a "leave" event can result in a graph that remains "queryable" but yields an incorrect answer—meaning such an omission is categorized under "query-result correctness." Conversely, the absence of a temporal field might lead to a response being rejected at an earlier stage; yet, supplying the missing timestamp (e.g., by inferring it from the sequence of entries) could simultaneously alter the executability of subsequent states.

This implies that the reported proportions—77.1%, 17.5%, and 5.4%—merely describe where failures are captured under the current acceptance criteria and query procedures; they do not, by themselves, indicate which types of defects are most valuable to address. Furthermore, the high accuracy rate observed for "queryable" graphs applies only to the specific subset of responses that successfully passed through the first two stages of the pipeline. The authors explicitly acknowledge that these stages do not represent independent causal mechanisms. Consequently, my concern is not that the decomposition formula itself is incorrect, but rather that the current evidence fails to demonstrate how much additional diagnostic value this framework provides beyond standard logs that simply track parsing, execution, and final answer errors.

This gap in the paper’s contribution is further constrained by the scope of its external validation. The OpenToM and FANToM experiments (Section 7 and Appendix C) only assess performance gaps along the direct answer path; they do not evaluate the outcomes of the structured three-stage process. As a result, the most distinctive aspects of the paper remain confined to a single set of template corpora, event schemas, and update mechanisms.

W3. The prospective prompting experiments—presented in Section 7 and Appendix I—serve as an important validation step, as they attempt to determine whether "strict" accuracy scores align with "rule-recovery" scores when output examples are modified. However, only six of the nine experimental configurations fall within the predicted intervals, and even the control groups do not remain entirely stable. Furthermore, even when using a baseline derived from a post-hoc matched subset, Qwen3-8B shows a decline of approximately 0.081, meaning the pre-registered joint criteria are not met.

The authors’ interpretation of these results as "partial support" is a reasonable framing. Nevertheless, eleven of the fourteen experimental configurations rely on routed calls to upstream services that are not held constant. Given that the paper’s own deployment comparisons demonstrate that changes in service conditions can significantly alter strict accuracy rates, it is difficult to disentangle the respective impacts of prompt modifications, service-side fluctuations, and changes in the actual response content.

More critically, the reproducibility statement on page 9 indicates that for the prospective prompting experiments, external data experiments, and "permission-menu" probes, only aggregated results (grouped by configuration) have been retained; the raw responses are not available for reconstruction. While this does not diminish the value of the full response data preserved for the main experiments, it means that the key evidence supporting the authors' interpretations and claims of generalizability cannot be subjected to the same level of rigorous audit. Without access to the raw data, it is impossible to verify whether "recovered" responses are semantically correct, or to determine whether certain failures stem from formatting issues versus service-level anomalies.

If these experiments are intended to validate the paper’s core explanatory framework, it is essential to preserve item-level paired results, complete responses, and service metadata—and to conduct such comparisons under fixed deployment conditions.

Questions for authors

Q1. What proportion of the "recovered" (re-scored as correct) responses are semantically accurate? For responses newly marked as correct under the OpenToM-derived and FANToM-derived rules, was there a manual review conducted by individuals blinded to the initial scoring outcomes? Please report the false acceptance rate for each set of rules, specifically including the "strict zero-score" configuration for Llama-3.2-1B and Phi-4-mini. This distinction is crucial: does a significant "recovery" rate indicate that valid answers were previously (and incorrectly) rejected, or does it primarily reflect the idiosyncratic behavior of different matching heuristics? Q2. Can the three-stage localization framework effectively guide the selection of appropriate corrective measures? For a specific, known error—such as a missed observational event, an omitted temporal marker, or a missing query entity—how does the failure "shift" across the three stages when the output-encoding rules are modified? Is there an analysis of actual model responses demonstrating that the stage-based statistics correctly informed subsequent, validated fixes? The examples in Appendix L are illustrative constructs and, therefore, cannot independently validate the diagnostic efficacy of the framework on real-world model outputs. Q3. In the prospective experiments, can prompt-based modifications be clearly disentangled from deployment-related changes? Among the nine experimental and five control configurations, which can be confirmed to have utilized identical model versions, upstream services, and inference parameters? Does the evidence for "recovery" hold up when the analysis is restricted to configurations using fixed endpoints? For experiments where full response logs were not retained, are there verifiable, item-level paired records available? Q4. Does the validation using "executable gold" (ground truth) encompass the state-update process, or is it limited solely to the final binary/choice-based answer? While the manual audit in Appendix L provides strong support for the 31 disputed labels, both the model-generated events and the gold-standard events are processed by the same update engine. Have the observational relationships, cross-room events, and second-order belief updates been independently verified? Furthermore, has the framework been checked for cases where the final answer is correct, but the intermediate state representation is erroneous? Additionally, Appendix F notes that "UNKNOWN" is used when container-to-scene mappings are missing. Does this default value affect only the structural edge matching, or does it also influence observer-related determinations and query results? This distinction is critical for assessing whether the adjustments to the scorer are sufficiently robust.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer 1YGL

Thank you for a careful review. Your Q4 prompted the audit we report in the disclosure comment.

W1/Q1 (is recovered credit semantically valid?). For the two strict-zero models the sample shows no false acceptance: blinded annotators confirmed the credited option in 40 of 40 distinct sampled responses for Llama-3.2-1B and 40 of 40 for Phi-4-mini (80 of 80; the precision of newly awarded credit has exact 95% interval [0.955, 1.000]). Where two rules disagree they confirmed 116 of 120, pooled (precision 0.967; exact 95% interval [0.917, 0.991], story-cluster bootstrap [0.932, 0.992]). We give no false-acceptance rate per rule or per model for that stratum: it has 40 items per model, too few, so it was registered as pooled only, with no comparison between rules. Of the four credits not confirmed, 2 stayed unresolved and remain in the denominator, 1 was labeled ambiguous, and one matched the other option. The check was registered before any label was collected: 240 items, labeled by two annotators who are not the authors and did not see the model, the rule, or the gold label. Both passed the attention gate (40 of 40 controls each; the gate was 34 of 40) and agreed with kappa 0.980 on the 200 study items. False rejection was not measured: the still-rejected responses in the sample are attention controls, not a sample drawn to estimate it. The check covers direct responses, not graph credit. On the last part of Q1: in the sample, new credit matches the option the response selects, with the four exceptions above. That is correct credit, not competence: under their best declared rule the direct scores are 0.577 for Llama-3.2-1B and 0.610 for Phi-4-mini, against a two-option chance level of 0.50.

W2/Q2 (diagnostic value beyond parse/execute/answer logs). What it adds: the accounting isolates unqueryable graphs as their own stage. At least one put or move with an unresolved scene co-occurs with 5,146 of the 9,951 unqueryable graphs (52%). Filling a scene only where the updater returns UNKNOWN recovers 2,682 of the 2,707 gold-map gain (99.1%): 2,724 graphs become queryable and none go the other way. On selection: accuracy among queryable graphs is conditional on reaching that stage, but newly queryable graphs are 98.6% correct, against 94.6% at baseline. On the "missing query entity" case: re-deriving one reason per unqueryable graph gives 4,967 known_unknown_belief, 4,818 missing_belief, and 166 missing_memory_event, so none is a missing query entity by the scorer's codes, though in 2,300 graphs no event mentions a query entity. Gemma-4-E2B's event-time fallback (strict 0.003 to 0.654) is supporting evidence only: a post-hoc scorer intervention, and 126 of its 3,254 correct gains come through a wrong event order. External validation covers the direct path only, and the Appendix L trace is illustrative.

W3/Q3 (prompt versus deployment; item-level records). In short: on fixed endpoints one of two treatments recovered and the registered criteria were not met; the earlier prompt test kept only aggregates, so it has no item-level paired records. In the earlier test, 6 of 9 treatments were in band, including all three on dedicated endpoints (Gemma-4-E4B narrowly); there was no dedicated control; the Qwen3-8B control changed by -0.068 against the registered baseline and -0.081 against the matched one; and some baselines came from a different deployment than their v3 run. The new paired test holds the deployment fixed: v1 and v3 side by side on the 4 dedicated endpoints, with item-level records and raw responses. It varies the contract only. By its registered criteria the result is falsified: Qwen2.5-7B's direct treatment recovered in band (strict +0.668; the gain depends on the scoring rule, see our uzeL Q4 reply), Llama-3.2-1B's did not (0.000), and two controls rose by more than 0.05 (Granite-4.1-8B direct +0.076, Qwen3-14B graph +0.081), while Granite-4.1-8B's graph control fell (-0.202). As registered, a control rising that far means v3 is a generally better prompt rather than a contract repair, which weakens the Appendix I mechanism claim; and Llama-3.2-1B not recovering means its failure on the dedicated endpoint is not contract demonstration.

Q4 (gold validity: state updates, cross-room observation, UNKNOWN). Yes, UNKNOWN affects observer assignment: an unresolved move gets no observers, not even its actor (disclosure comment). The gold event lists' own answers depend on container_in for 58 of 1,000 items. Gold events are converted, not independently annotated, and the human audit covered only final answers on the 31 disputed items. The check of right answers reached through a wrong state, against independently adjudicated observer and belief traces, is not done; it needs adjudicated traces we do not yet have and is planned for the revised paper.

The remaining limits are listed under "What we concede" in our summary comment.

Thank you for the blinded semantic audit and the new fixed-endpoint experiments with retained responses. These substantially address my concerns about newly awarded credit and the auditability of the new experiments.

Two key questions remain:

  1. Given that UNKNOWN affects observer assignment and 58 gold answers depend on container_in, can you independently validate the observer and belief traces for affected cases and report how any necessary corrections change the gold answers and main stage-wise results?
  2. Since the fixed-endpoint experiment did not meet the registered criteria, what evidence still supports attributing recovery specifically to contract repair? How will the conclusions distinguish supported scoring observations from the mechanism claims that this experiment did not validate? I would consider raising my score if independent state validation supports the main findings and the conclusions are revised consistently with the fixed-endpoint results.
AuthorsOpenReview ↗

Reply to follow-up: trace validation and the contract-repair claim

Thank you. In short: a separately written simulator reproduces every gold observer set, belief and answer except under one convention (what a character learns on entering a room), which would flip 30 labels, none in the ToM strata; rejection stays the dominant failure stage. We withdraw the contract-repair claim and revise the conclusions to keep mechanism claims apart from the scoring observations the data support.

1. Independent validation of the traces. Since our first reply we have run this check in code. On all 1,000 items, a separate simulator reproduces every gold observer set, 2,000 of 2,000, including all 116 placements in the 58 container_in-dependent items. An AI coding agent wrote it from the story text and ToMi's public generator code and paper, without opening our engine, gold events, or labels; its task prompt, self-reported access log, rules and code will be in the supplementary material. Like our gold, it reads "The C is in the R." as a fixed fact, because ToMi's generator places containers once, before any action. The query path fills gold scenes from those lines, so no gold placement is UNKNOWN (134 of 2,000 would be without it, 58 in the affected items); model graphs never get this fill-in (the UNKNOWN problem in our disclosure).

Beliefs differ through one rule only: ToMi's generator lets a character who enters a room learn where the objects there now are, while our gold updates a belief only on a witnessed placement. With that rule off, all 7,645 belief entries and all 1,000 answers agree. With it on, the simulator matches FixedToMi's labels on 999 of 1,000 items and differs from our gold on 30 answers, all in the no-ToM operations and all among the 31 source-label disagreements our blinded adjudication resolved to our gold (31 of 31; 153 of 155 votes). We keep the gold answers, but that was a two-way choice by author-recruited adjudicators, so here is the effect of the other labels:

  • Unchanged: memory, reality, all four ToM strata, and rejection as the dominant failure stage.
  • Pooled strict graph accuracy: 0.483 → 0.475; strict failures 77.1 / 17.5 / 5.4% → 75.8 / 17.2 / 7.0% (rejected / unqueryable / wrong). No configuration moves more than 0.023.
  • Table 2, no-ToM graph minus direct: first-order 0: -0.029 → -0.074; first-order 1: -0.027 → -0.055; second-order 0: +0.206 → +0.180; second-order 1: +0.249 → +0.236. Intervals and Holm tests are not yet rerun.

The revised paper will state the entry rule as a benchmark convention and report both label sets.

Human check. Two non-author annotators, blind to the gold, will label the same traces: the 58 affected items and 58 unaffected ones matched on question type, 360 questions (who saw each placement of the queried object, the queried first- or second-order belief, and 12 attention checks). It was registered before the sample was drawn and narrowed to these questions before any label. Annotators are given the benchmark's observation rule, so it tests whether our traces apply that rule, not which entry rule is right. If both annotators' labels are back by noon AoE on 18 November, we will post the result here, whichever way it falls; otherwise it goes in the revised paper.

2. Contract repair. We withdraw it as the mechanism: no remaining evidence isolates it. Both prospective tests failed their registered criteria; in the fixed-endpoint test two controls rose by more than 0.05, which the registration reads as a generally better prompt. Qwen2.5-7B's strict recovery (+0.668) is a format change: under the OpenToM-derived rule, which already credited v1 at 0.673, the same change is +0.008, and one cell with rising controls cannot separate it from a better prompt.

What the data support are scoring observations on saved responses:

  • declared acceptance rules change scores and rankings (Llama-3.2-1B: 0.000 strict, 0.577 under the OpenToM- and FANToM-derived rules; on the 14 fixed-host configurations, 10 of 14 ranks change);
  • newly awarded credit matches the option the response selects (blinded check of direct responses: 80 of 80, and 116 of 120 where rules disagree);
  • most structured-path failures are rejections (77.1% under the published scorer, 81.0% under the gold-map replay).

These describe what the scorer does with a response, not why a model fails the format.

Revisions to the conclusions:

  • The "output-contract gap" is defined only as declared-rule minus strict accuracy, never as a cause.
  • Section 7 and Appendix I report both prompt tests as not supporting the mechanism, and the fixed-endpoint test as falsified by its registered criteria.
  • The Limitations and the Evidence Status table no longer call the earlier test partial support.
  • Appendix K's "contract component" is recast as a decomposition of score differences.
  • The conclusion lists the supported scoring observations separately from mechanisms that remain untested (belief tracking) or were not supported (contract repair).
AuthorsOpenReview ↗

Result of the registered human trace check

As promised in our previous comment, here is the result of the registered human check. Two non-author annotators, blind to the gold, labeled 116 items; both passed the attention gate (12 of 12 and 11 of 12; the gate is 10).

  • No trace element contradicted the gold. Of 348 elements, both annotators gave the gold value (confirmed) on 338; on none did they agree on a different value; on 10 they disagreed with each other.
  • Affected items (58): 166 of 174 confirmed (0.954; exact 95% CI 0.911 to 0.980): observers 110 of 116, first-order 23 of 24, second-order 33 of 34. Matched unaffected items (58): 172 of 174 (0.989; 0.959 to 0.999).
  • The 10 disagreements: in every one, one annotator gave the gold value. Of the other answers, six name a different character, two omit one observer, and two give a different belief (one "No idea", one the object's real location).
  • Agreement: kappa 0.981 on beliefs, 0.957 per character on observers.
  • Gold answers: none corrected. The registered scorer re-derives each answer from the queried belief: 114 of 116 unchanged, and 2 unresolved (both affected items) because the annotators disagreed on that belief; these keep their published answer, which one annotator gave in both. In 4 more items an observer disagreement involves a queried character, but both annotators gave the gold belief; counting these as unresolved too, no answer changes. The stage-wise results are therefore unchanged.

As registered, this tests whether our traces apply the benchmark's observation rule; it does not decide the entry-rule convention discussed in our previous comment. The registration, labels and scorer will be in the supplementary material.

Reviewer_uzeLOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Problem: Theory-of-mind (ToM) benchmarks report one final-answer accuracy. That single number mixes up "the model answered wrong" with "the model's output was rejected by the scorer" or, for structured outputs, "the model's state could not even be queried".
  • Proposal: StructToM-Bench, 1000 story-question items (from FixedToMi, the cleaned-up ToMi dataset) where each item comes with "executable gold": a list of events (enter, leave, put, move) plus a deterministic engine that replays the events and computes the answer. Models are tested two ways: (a) direct answering, (b) emitting an event graph, which the engine then executes.
  • Key idea: for the graph path, strict accuracy = P(output accepted) x P(graph queryable | accepted) x P(answer correct | queryable). This splits every failure into one of three observable stages.
  • Validation: 22 small open-weight model/deployment configurations (1-15B), 5 passes each, 220,000 observations. Pre-registered human audit (5 annotators) of the 31 items where executable gold disagrees with the source label: all 31 went to executable gold. Cross-dataset checks on OpenToM and FANToM, a pre-registered prompt test, host-pinning re-runs, and a few frontier-model scope checks.
  • Main results: (1) Lenient but declared acceptance rules (two borrowed from the OpenToM/FANToM evaluators) raise the direct score of 10 of 22 configurations; two strict zeros (Llama-3.2-1B, Phi-4-mini) become 0.577 and 0.610; a punctuation-only rule changes almost nothing (20/22 flat). (2) Of all graph-path failures, 77.1% are rejected outputs, 17.5% are accepted-but-unqueryable graphs, and only 5.4% are queryable graphs with a wrong answer. (3) Their own structural metric (SBGA) was broken: 30.2% of gold edges belonged to a family the prompt told models to omit, so the metric punished prompt-following. (4) Simple positional heuristics (first/last mentioned container), chosen per operation, score 0.86-1.00 and beat every model on every operation, so the authors themselves say the numbers measure evaluation pipelines, not ToM competence.
Critical strengths
  • Rigor and transparency are decent. Fixed denominators, pass-level spreads, deployment provenance, pre-registered tests (including ones that failed and are reported as such), a released artifact that regenerates the tables, and an explicit audit of their own scorer. This matters because the paper's claims about "where accuracy is lost" are credible and reproducible, which is exactly what an evaluation-methodology paper needs.
  • The three-stage factorization plus the acceptance-rule registry gives a concrete, reusable recipe (Section 9's five recommendations). The numbers behind it are striking and internally consistent (e.g., Llama-3.2-1B 0.000 -> 0.577 purely from the acceptance rule; Gemma-4-E2B graph score 0.003 -> 0.654 from an event-time fallback). This is the part of the paper that would change how practitioners report ToM results.
Critical weaknesses
  • The benchmark does not measure what its name promises. By the authors' own analysis (Section 3.3, Table 2 "Shortcut" column), a label-free positional heuristic chosen per operation scores 0.86-1.00 and beats all 22 systems on every operation, and the data are ToMi templates whose shortcuts have been known since Le et al. (2019). The paper therefore explicitly disclaims any ToM conclusion ("measurements of the evaluated pipelines rather than evidence of ToM competence"). What remains is an evaluation-pipeline audit on a 1000-item template corpus with 1-15B models. For an ICLR benchmark/dataset contribution this is a serious limitation: a reader cannot use StructToM-Bench to learn whether a model tracks beliefs.
  • Much of the evidence is inconclusive or confounded, and the paper's own evidence-status table (Appendix A) grades most arms as "secondary", "provisional" or "historical". The direct-vs-graph contrast (Section 5) differs in prompt, option presentation and entity inventory at once, so the Holm-surviving operation effects are "descriptive" only. The story-only ablation failed its equivalence criterion (1 of 4 cells), the prospective prompt test failed its registered joint criterion, 8 of 22 configurations were served through routed hosts where pinning moved scores by up to 0.215, and frontier models appear only as scope checks. The net effect is that the paper reads as an honest audit log rather than a result with a clear, generalizable takeaway, which is what the acceptance decision has to rest on.
Questions for authors
  1. Given that per-operation positional shortcuts beat every system, what does StructToM-Bench tell us that a format-sensitivity study on any two-option QA benchmark would not? Is there a held-out-template or shortcut-resistant split, and do the headline findings (10/22 raised; 77.1/17.5/5.4 split) hold on it?
  2. Do the main numbers (Table 1, Table 7, the 77.1/17.5/5.4 split) change materially if restricted to the 14 dedicated or host-pinned configurations only, excluding the 8 routed ones? Appendix B shows pinning can move strict accuracy by 0.167-0.215, which is larger than many of the effects reported.
  3. The "best declared rule" column takes the maximum of four rules per configuration. How should a reader interpret a per-configuration max, and would a single pre-chosen rule (e.g., the OpenToM-derived one) change the 10/22 count or the rank correlations?
  4. Can you run even a small factorial ablation (on the four dedicated-endpoint cells of Appendix H) that isolates the output contract from the other prompt differences, so that at least one direct-vs-graph contrast is causal rather than descriptive?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer uzeL

Thank you. Please also see our disclosure comment on a scorer mismatch we found after submission.

W1/Q1 (what it measures; beyond a format study; a shortcut-resistant split?). We agree: a reader cannot use StructToM-Bench to learn whether a model tracks beliefs. The Limitations sentence that it "measures controlled story-based belief tracking" is an overstatement; the revised paper will correct it. No held-out-template split exists. The takeaway we offer is the reporting recipe you credit, the Section 9 recommendations: report planned denominators and stage rates, name the evaluated model-prompt-scorer-deployment system, disclose acceptance sensitivity, and evaluate shortcut rules on a held-out split or call the result a same-corpus bound. It is not yet shown to transfer beyond this corpus. The benchmark, with its executable gold, is the instrument for that recipe, not a ToM measure. What a format-sensitivity study on a two-option QA benchmark would not show: executable gold lets a structured output be classed as rejected, accepted but unable to answer the query, or answered wrongly, and lets the scorer itself be audited, which a final-answer format study cannot do. The replay shows how much of the unqueryable-stage loss comes from the prompt-scorer mismatch in the disclosure comment (after SBGA container_in); rejection stays the largest stage (81.0% of strict failures under the gold-map replay). We do not claim the stages found the second mismatch: the submitted accounting had counted those graphs as model failures. We will reframe the revised paper around evaluation methodology rather than as a ToM benchmark, and validate the full structured pipeline on a second corpus before a shortcut-resistant split.

W2 (inconclusive arms). Agreed. The story-only ablation met equivalence in 1 of 4 cells (App. H), and the frontier models are scope checks only (App. K). Both stay secondary.

Q2 (the 14 dedicated or host-pinned configurations only?). On those 14, each failure share moves by under two points, and the borrowed rules still raise scores and change ranks. Strict graph failures split 78.6 / 15.9 / 5.4% (33,274 / 6,736 / 2,306 of 42,316), against 77.1 / 17.5 / 5.4% for all 22 under the published scorer (81.0 / 13.4 / 5.7% under the gold-map replay). The borrowed rules raise 6 of 14 fixed-host direct scores (10 of 22 overall), and both strict zeros are dedicated endpoints. There the OpenToM- and FANToM-derived rules still change 10 of 14 ranks (tau-b 0.508); tau-b over 14 and over 22 configurations is not comparable, so we do not compare effect sizes across the two sets. Note: 0.215 and 0.167 are two maxima, with and without the failed-host cell, not a range. Appendix B also shows the best disclosed direct evaluator moved at most 0.04 under pinning. This is a sensitivity check, not a causal estimate of host effects.

Q3 (the best-of-four column). Read it as an envelope over a declared, fixed registry; it never replaces strict. Each borrowed rule alone raises 10 of 22 direct scores, and the best-rule tau-b (0.6508) equals the single-rule tau-b. We will cite the individual rules first.

Q4 (a causal direct-vs-graph contrast?). Not yet: no direct-vs-graph contrast is causal after this rebuttal, because the two paths differ in entity inventory, option presentation, output contract and computation. Those contrasts stay descriptive until a factorial isolates the components; it is planned for the revised paper. What we could run in time is a paired v1-vs-v3 test within each method, aimed at two points in your second weakness: the prompt test's failed joint criterion, and the routed hosts where pinning moved scores. It was preregistered before any call (two amendments, the second after the calls, were committed before scoring): v1 and v3 prompts side by side, in the same session, on the 4 dedicated endpoints, for both methods, 300 items x 3 passes, with item-level records and raw responses. It varies the contract only: it changes the worked examples in the prompt, which v3 writes as literal output; option presentation and entity inventory are held fixed and not tested. By its registered criteria the result is falsified. Qwen2.5-7B's direct treatment recovered in band (strict +0.668, 95% CI 0.614 to 0.720; under the OpenToM-derived rule the same change is +0.008 with 95% CI -0.031 to 0.048, because v1 already scored 0.673 under that rule against 0.011 strict, so the strict gain depends on the scoring rule); Llama-3.2-1B's stayed at 0.000; and two controls rose by more than 0.05 (Granite-4.1-8B direct +0.076, Qwen3-14B graph +0.081), while Granite-4.1-8B's graph control fell (-0.202). As registered, a control rising that far means v3 is a generally better prompt rather than a contract repair, which weakens the Appendix I mechanism claim; and Llama-3.2-1B not recovering means its failure on the dedicated endpoint is not contract demonstration.

Thank you for addressing many of the concerns. I have changed my rating accordingly. I still feel that benchmarks of this kind are overrepresented at A* ML conferences, and that the one proposed here does not measure exactly what it purports to, hence I remain at rejection, but only a weak one.

Reviewer_rMezOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce StructToM-Bench, a benchmark that evaluates models using controlled prompts and executable ground-truth events. It employs two distinct evaluation pathways: "direct answering" and "event-graph generation." For the graph-based pathway, the authors decompose strict accuracy into three sequential criteria: whether the output conforms to the required format, whether the generated event graph is queryable, and whether the answer derived from querying the graph is correct. The benchmark incorporates executable ground-truth events. Evaluations across various model configurations reveal that lenient acceptance criteria can inflate scores for the direct-answering pathway, whereas rules that only account for punctuation differences have a negligible impact on scores. Failures in the graph-based pathway most frequently stem from the output being rejected; this is followed by cases where the graph is accepted but proves unqueryable, while instances where a query is successfully executed but yields an incorrect answer are relatively rare.

Critical strengths

Breaking down the final score into distinct stages, whether the output meets requirements, whether the generated event graph is queryable, and whether the query results are correct, allows for identifying exactly where a failure occurs, rather than simply noting whether the final answer was right or wrong.

This approach involves using blind evaluation against ground-truth answers, while also accounting for the potential impact of lenient scoring criteria, the design of the evaluator itself, and the deployment environment. Furthermore, it employs specific constraints to filter out "shortcuts" or "tricks," ensuring that issues within the evaluation process are not misattributed to the model's actual capabilities.

Critical weaknesses

When a model provides an incorrect answer, it could be because it genuinely lacks the capability, or because the output format fails to meet requirements, or because the provided state omits information requested by the prompt, or perhaps the evaluation script incorrectly penalizes a response that could otherwise have been salvaged. Looking solely at the final answer makes it impossible to distinguish between these scenarios.

Simple, heuristic-based shortcuts tailored to specific question types can outperform all the models on the same set of tasks. Consequently, any performance gaps observed in these categories do not demonstrate that a model truly understands others' thoughts; rather, they merely reflect the specific outcomes generated by this particular evaluation protocol.

The article’s conclusions describe the methodology and scoring criteria of this evaluation framework, not whether the models actually possess the capacity to understand the thoughts of others. Comparisons between large models are valid only when restricted to those within the same family.

Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer rMez

Thank you for the review. We reply to your three points in order.

  • A final answer cannot show why a model failed. We agree, and this is the problem the paper addresses. Its stage split is our attempt to tell those cases apart for structured output: a response rejected for its format, one accepted but unable to answer the query, and one that answers wrongly. The limits, as we state them: the stages are observable steps of the tested pipeline, not independent causes, and they do not establish belief tracking.
  • Shortcuts. We agree this limits the interpretation. One clarification on the kind remark in your strengths: the paper measures shortcuts and does not filter them out. It reports a same-corpus positional shortcut bound, 0.860 to 1.000 per operation, which exceeds all 22 configurations, and it says so as a limit on what the benchmark can measure. The bound is per operation, from a positional rule chosen with this corpus's labels (not a held-out baseline), while the corpus-wide positional heuristics score near chance. It shows that the operations admit shortcuts, so our conclusions focus on the tested pipelines and where they lose accuracy.
  • Scope of the conclusions, and "same family". We agree: our conclusions describe the evaluated pipeline and its scoring, not whether a model has the capacity. We believe the "same family" remark refers to Appendix K, which already restricts comparisons to same-lineage models. We will tie every cross-model statement to the tested model, prompt, scorer, and deployment configuration.
  • A disclosure. Please see our disclosure comment: after submission we found a second prompt-scorer mismatch in our graph scorer. The published scorer stays the reference, and every new number there is labeled by how it was obtained.

Thanks for the reply. I'm glad you found the concerns reasonable. The new scorer problem in particular remains a big issue. The results ultimately tell us about the pipelines and their scoring, not about whether beliefs are tracked. The two may be typically correlated, but not necessarily so. The fixes you plan would require a new version of the paper. If you upload one until the end of the rebuttal period, I'll have a look at it.

AuthorsOpenReview ↗

Revised PDF uploaded

Thank you. We agree: the results describe the evaluated pipelines and their scoring, not whether beliefs are tracked, and the revised PDF now says so in the Abstract, Introduction and Limitations. Changes are in blue; our summary-of-revisions comment lists each one with its section.

On the scorer problem, new Appendix O reports the mechanism, its exposure and the replays, and says what is not yet fixed. The scene-enrichment fix with every table rerun is planned for the final version, with the published scorer kept for comparison; we did not want to rush that rerun into the discussion period and risk new errors. The contract-repair mechanism is withdrawn (Section 7 and Appendix I), and the conclusions keep it apart from the scoring observations the data support.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of our response (to the area chair and all reviewers)

We thank the reviewers. In brief:

  • A disclosure. After submission we found a second prompt-scorer mismatch in our own graph scorer; see the disclosure comment. Under the gold-map replay (a diagnostic, not a corrected score), pooled strict graph accuracy moves from 0.483 to 0.508, and rejection is still the largest failure stage. The published scorer stays the reference.
  • Blinded human check (1YGL Q1). Two non-author annotators (kappa 0.980) confirmed the credited option in 80 of 80 sampled responses from the two strict-zero models and in 116 of 120 where two rules disagree.
  • Paired v1-vs-v3 prompt test on 4 dedicated endpoints (uzeL Q4 and W2, 1YGL Q3). Falsified by its registered criteria; as registered, this weakens the Appendix I mechanism claim.
  • Fixed-host re-analysis (uzeL Q2). Strict graph failures split 78.6 / 15.9 / 5.4% (rejected / unqueryable / wrong) on the 14 fixed-host configurations, against 77.1 / 17.5 / 5.4% for all 22 (81.0 / 13.4 / 5.7% under the gold-map replay).
  • What we concede. There is no held-out-template split; the benchmark does not measure ToM; direct-vs-graph contrasts stay descriptive; the earlier prompt test has a deployment confound and aggregate-only records; and we have no structured result on a second corpus.
AuthorsOpenReview ↗

General comment: author-initiated disclosure of a second prompt-scorer mismatch

While checking 1YGL's question on UNKNOWN scenes, we audited our graph updater and found a second mismatch between the graph prompt and our scorer, of the same kind as the SBGA container_in defect the paper reports. It bears on the primary strict graph metric. Effect on the paper: replaying the saved graphs with the gold container-to-scene map moves pooled strict graph accuracy from 0.483 to 0.508 (+2,707 correct graphs; a diagnostic, not a corrected score). The story sentences ("The bottle is in the bedroom."), read with the prompt's entity inventory, reproduce that map for 2,000 of 2,000 containers and give the same +2,682 as the fill-only replay below without gold information; we saw this after the fact, so it is exploratory. In the rerun Table 2 bootstrap under the gold-map replay, three ToM-stratum graph-minus-direct point estimates change sign (SO1-ToM from -0.048 to +0.010), the SO1-ToM rerun 95% interval includes zero, and all four originally Holm-significant contrasts still survive. The fill-only and story-sentence variants give the same three sign changes and the same Holm results, and SO1-ToM's interval includes zero in both. We disclose it before the replies. It is a post-hoc sensitivity finding on the saved outputs: the published scorer stays the reference, and every new number is labeled by how it was obtained.

Verification first. Re-parsing all 110,000 saved responses reproduces acceptance on every row; replaying all 66,180 accepted graphs reproduces query status and answer on every row (0 mismatches).

Mechanism. A move whose scene resolves to UNKNOWN gets no observers, not even its actor. Before replay, the query path fills gold movement scenes from container_in events. Model graphs never get this fill-in: the prompt tells models to skip container-location sentences, and put has no scene field.

Exposure. 12,063 of 121,696 model put/move events (9.9%) resolve to UNKNOWN, 9,173 of them actorless (a move requires an actor, so these are puts); 10,161 accepted graphs (15.4%) contain one. At least one put or move with an unresolved scene co-occurs with 5,146 of the 9,951 unqueryable graphs (52%); co-occurrence, not shown cause. Our stage accounting had counted such graphs as model failures; 1YGL's question, not the accounting, prompted the check.

Replays. These are diagnostics, not an upper bound and not a corrected score.

  • Gold-map replay (gold scene written into every sceneless move): strict-correct 53,168 to 55,875.
  • Fill-only (gold scene only where the updater returns UNKNOWN): +2,682, 99.1% of the gold-map gain. 2,724 graphs become queryable and none go the other way; 5 correct graphs turn wrong. The gain is attributable to unresolved scenes.
  • Story-sentence map used only to fill unresolved scenes: +2,682, the same as fill-only, without gold information; exploratory, as stated above. A pre-specified wrong-room control gives -48.

What moves. Strict graph failures split 77.1 / 17.5 / 5.4% (rejected / unqueryable / wrong) under the published scorer and 81.0 / 13.4 / 5.7% under the gold-map replay, so rejection still dominates. One further contrast (FO0-ToM) passes the Holm threshold under every replay variant except the wrong-room control, and not under the published scorer; this is post hoc and we draw no claim from it. Section 5's ToM-strata queryability rates rise, and Appendix F's "only the scene is unresolved" is wrong for query results.

Gold also depends on container_in. Removing only container_in from the gold event lists makes 58 of 1,000 gold items fail to return the gold answer (53 in the ToM strata). This is not a hard ceiling for compliant extractors: the parser keeps extra keys, and a put that carries a scene is placed. (Our repository's oracle_graph 1.000, not in the paper, includes container_in; we have corrected that wording.)

Rows the parser drops. Accepted graphs also lose container-to-scene rows that models emit anyway, and explicit skip rows. The parser logs the drops, but Appendices F and G do not disclose them. Turning the model-supplied rows back into a scene map recovers 1,563 net correct answers (exploratory), 58% of the gold-map gain.

SBGA uses a different gold state. SBGA builds gold edges without the scene fill-in that the query uses. Model belief edges presumably lose the same actorless-put information; this is from reading the code and is not measured.

One more undisclosed choice. Graph salvage decides whether a row touches the query path from the item's gold query target. We think this is defensible, but the paper does not say it.

The revised paper will fix scene enrichment, the dropped rows, and the SBGA gold state, keep the published scorer for comparison, and state the salvage choice.

AuthorsOpenReview ↗

Revised PDF uploaded: summary of changes

We uploaded a revised PDF. Added and changed text is in blue. Appendix letters A–N are unchanged, Appendices O and P are new, and the main text still ends on page 9.

  • Framing (Abstract, Introduction, Limitations). The results are stated as measurements of the evaluated pipelines and their scoring, not of belief tracking or ToM competence. The Limitations sentence saying the benchmark "measures controlled story-based belief tracking" is replaced. (uzeL W1/Q1)
  • Contract repair withdrawn (Section 7, Limitations, Conclusion; Appendices A, I, K). Neither prompt test supports the mechanism; Appendix I adds the paired fixed-endpoint follow-up, falsified under its amended registration (Appendix I says what the amendment changed). The Evidence Status table no longer says "partial support", Appendix K defines the output-contract gap as declared-rule minus strict accuracy and recasts its "contract component" as the change in that gap across rungs, and the Conclusion separates scoring observations from mechanism claims. (1YGL W3/Q3 and follow-up Q2)
  • Second scorer mismatch (new Appendix O; pointers in the Abstract, Introduction and Sections 5 and 6; Section 9 counts both mismatches). Mechanism, exposure, the replays, the Table 2 rerun, and the parser and salvage choices the submission did not disclose. The tables still use the published scorer. Appendix F's "only the scene is unresolved" is corrected for query results. (disclosure comment)
  • Gold-trace validation (new Appendix P; Limitations). The separately written code check, the entry-rule convention with its effect on scores under the alternative labels, and the blinded human check. (1YGL Q4 and follow-up Q1)
  • Newly awarded direct credit (Appendix D; Limitations). The blinded check: 80 of 80 for the two strict-zero systems, 116 of 120 where rules disagree. (1YGL W1/Q1)

Not in this revision, and planned for the final version: the scene-enrichment, dropped-row and SBGA gold-state fixes with every table rerun (this revision discloses and measures the defect but does not fix it); a fuller restructuring around evaluation methodology; a factorial ablation of the direct–graph contrast; citing the individual acceptance rules before the best-of-four column (uzeL Q3); tying every cross-model statement to the evaluated configuration (rMez); a check of right answers reached through a wrong state (1YGL Q4); validation on a second corpus and a held-out-template split.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.