acceptodds
Under review as a conference paper at ICLR 2027

Access, Wording, and Knowledge Attribution in Language Models

Abstract

Evaluations of theory of mind ask what an agent knows, and typically separate that question from reading by pairing it with a non-mental control over the same story. We introduce a controlled diagnostic in which the epistemic rule is stated in the prompt, and which manipulates which agents have perceptual access to an event, so that a correct answer is derivable from the text rather than from commonsense. Using it, we report a replicated positive result and a boundary on it. On nine open-weight models from four families, recalling the two locations the question depends on raises balanced accuracy by 9.0 to 22.8 points, under an identical explicit instruction naming the comparison to perform. With both additions present five models reach 97.9 to 99.7, while the four that begin at or near chance reach only 59.6 to 77.6 — even though those same four score 95.4 to 99.4 when the witnesses are named and answering reduces to checking list membership. Stating the procedure is not what they were missing, and can cost up to 22.6 points when the facts are already supplied. We reach this by auditing our own control, which named the actor of the event while the knowledge question required recovering it. With designation matched we no longer detect the growth of the gap with group size that motivated the study. Seemingly minor changes in item wording matter throughout: a sentence that is true and wholly redundant costs up to 12 points, on seven models of nine. These results argue for measurement robustness before stronger conclusions are drawn about the mechanisms underlying model behaviour, and we release all prespecifications, reproduction gates and internal controls, including the internal control that failed.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.