- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies knowledge attribution in a synthetic task whose perceptual rule is stated in the prompt. Each story places 2 to 20 agents in rooms and describes one event, and the model is asked whether a queried person knows the fact the event reveals. This question is paired with a spatial control asking whether that person is in the same room as the actor. Once the two questions designate the actor in the same way, the growth of the gap with the number of agents is no longer detected, but the sentence added for this (MAP) itself lowers spatial accuracy on seven of nine models.
The main result comes from a 2x2 design. With an instruction naming the comparison held constant, restating the two relevant rooms (RECALL) raises balanced accuracy by 9.0-22.8 points on all nine models. Five models then reach 97.9-99.7, while the four that start near chance reach only 59.6-77.6, even though they score 95.4-99.4 when the witnesses are listed; the authors describe this as a threshold. An explicit perceptual block yields almost no "knows" answers, and an audit of ToMi and BigToM finds no similar designation mismatch.
Critical strengths
-
The recall effect is measured with the instruction held constant, which separates supplying the premises from stating the procedure. Q2 is +9.0 to +22.8 on all nine models from four families with every interval excluding zero, and recomputing the contrasts from the cell means in Table 5 reproduces Tables 3-4 to within 0.1.
-
The authors audit their own control and report what they found, including a prespecified internal control that failed. Stories are asserted byte-identical across cells and all contrasts are paired on the same 2,200 scenarios. Replacing the actor's name with a description in the spatial question alone costs 19.4 points on gemma-2-27b, a concrete case of a control that does not share the retrieval demand of the target question.
Critical weaknesses
-
The novelty is limited and the scope is narrow. The main result, that restating the relevant facts next to the question helps, is close to known effects of where information sits in the context [1] and to ToM methods that filter the story down to the relevant events before answering [2], [3]. The cautionary claim, that the control supplies the actor the knowledge question leaves to be recovered, is shown only for the authors' own benchmark, and the audit of ToMi and BigToM found the same problem in neither. The evidence comes from one synthetic task with a stated rule, one wording per condition and nine models up to 32B.
-
Wording effects are as large as the effects interpreted, so the secondary claims are not robust. One true, redundant sentence (MAP) moves accuracy by 1.3-12.2 points on seven of nine models, comparable to the naming effect (-4.6 to +5.0) and Q1 (-3.6 to +13.6). The designation-matched analysis relies on this sentence, and the two conditions compared for the threshold also differ by two sentences. Each condition uses a single wording, so this variation is not reflected in the intervals.
[1] N. F. Liu et al. Lost in the middle: How language models use long contexts. TACL, 2024.
[2] A. Wilf et al. Think twice: Perspective-taking improves large language models' theory-of-mind capabilities. ACL, 2024.
[3] M. Sclar et al. Minding language models' (lack of) theory of mind: A plug-and-play multi-character belief tracker. ACL, 2023.
Questions for authors
-
Please report the main contrasts (Q2, the instruction cost, the naming effect and the Fig. 2 difference) over several paraphrases of MAP, RECALL, INSTR and the question. If they hold across these variants, weakness 2 is largely resolved.
-
Please run the length- and position-matched irrelevant-recall control proposed in Sec. 10. A benefit that appears only for relevant facts would separate the main result from the known effect of information position.