- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a model finds a fact in its context by the name in the question or by the order in which the fact was stated. It uses short lists of four facts, such as "Petra eats the buckwheat. Freya eats the amaranth." The difference in hidden state between questions about two positions, averaged over other lists, is added once to a new list. The question text stays the same. The model often answers with the fact at the target position. The authors report that this position code follows order of mention rather than names, transfers across several relations, and appears in Qwen, Gemma, and Llama.
Critical strengths
The transfer test separates position from names. The vector is estimated on other lists and other names, then added to a new list while the question still names the first person. In Qwen the answer switches to the target fact about 84% of the time, and a random direction almost never does. Reversing the facts reverses the direction, and a vector from one relation works on another. This is direct evidence that, in these lists, the question moves the state to a position and the current list supplies the content.
Critical weaknesses
The finding is not shown to be general, and the title claims more than the experiments support. "The geometry of contextual relations" suggests a broad account of how language models represent relations. The experiments stay inside short templated lists: four to eight facts, one content per fact, and a fixed question that asks for that content. Contents are not only food. Appendix D.3 includes animals, cities, jobs, instruments, and sports, and a vector from one relation transfers to another. The task does not change. Short paragraphs still contain four facts plus a few unrelated sentences. Long documents and multi-step reasoning are not tested, as the authors note. A result on this one task does not support the broader claim.
The effect is also uneven: about 84% in Qwen, 54% in Gemma, and 36% in Llama on the main test, against a ceiling near 92–97%. It should not be described as a uniform mechanism.
Questions for authors
Q1: Which result is supposed to show that order-of-mention addressing is a general property of contextual relations, rather than a property of these short one-content fact lists? If the same facts are placed in a longer passage, with unrelated material between them, does the vector still select the target fact? This answer would affect my score.
Q2: The edited layer was chosen by how well the vector works. How much of the result remains at a layer fixed before seeing the test lists?