acceptodds
Withdrawn from ICLR 2027

Where Does Entity Binding Come From?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

To use factual knowledge, e.g., "Alice has the umbrella," a language model must do more than represent the entities: Alice and the umbrella. It must bind them, tracking which entity goes with which. Prior work showed that binding appears in model activations as identity vectors attached to each entity. But where does this binding come from: is it carried in the content of the representations, or computed from where they sit? We first show that, at early layers, binding is constructed from receiving position: placing the same entity content at different token locations predictably changes which entities are bound together. As computation proceeds, however, moved representations increasingly preserve their original pairing, showing that the established relation becomes carried by the representation itself. We then ask what this mechanism implies for injected knowledge. Writing entity states separately at their corresponding positions supports multi-hop reasoning close to ordinary text, whereas pooling those states sharply degrades relational use. By retaining all receiving positions while mixing entity Content, we show that position alone is not sufficient: two-hop reasoning can fail even while direct factual recall remains available.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, Learned Representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KQ9WOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how entity binding forms in language models. The authors use activation writing to exchange entity states and their estimated Slot and Content components across layers. They find that early-layer pairing typically follows the new receiving position, whereas moving both components together at middle layers better preserves the original pairing. The authors further show that writing entity states separately supports two-hop accuracy close to that obtained with text input, while content mixing reduces two-hop accuracy even when all receiving positions and original states are retained.

Critical strengths
  1. The paper investigates how binding forms through state exchanges rather than only analyzing established representations. The comparisons among whole-state, Slot-only, Content-only, and joint exchanges are well targeted, and all three models show distinct early- and middle-layer patterns. The shared-Slot experiment further shows that source-position differences need not be supplied for early pairing to form. These results provide valuable interventional evidence about entity binding and are the main reason for supporting acceptance.

  2. The content-mixing experiment includes useful controls. The authors distinguish the reduction in written positions from the changes in state composition caused by pooling. They then compare partner Content additions with norm-matched random perturbations while keeping receiving positions and original states fixed. On Gemma, content mixing reduces two-hop accuracy from 99.0% to 19.75%, while direct-question accuracy remains at 96.75%. With random perturbations, two-hop accuracy remains at 98.5%. This shows that the decline in relational use cannot be explained solely by perturbation magnitude.

Critical weaknesses
  1. The experiments do not fully separate the effects of receiving position and surrounding relational context. Moving an entity state also changes its local context, while relation words and sentence structures remain intact. Appendix A.4.2 acknowledges that these factors are not measured separately. Moreover, some middle-layer WHO conditions preserve the original answers even after random replacement, so they cannot establish whether the moved state carries pairing information. The evidence supports a role for the receiving context in binding, but the attribution to position itself and its consistency across query types remain limited.

  2. Two-hop failures may arise because the second fact becomes inaccessible, rather than because two independently accessible facts cannot be combined. The direct question tests only the first fact, and mixing the state at the tag position alone is sufficient to impair the two-hop answer. The current comparison therefore does not distinguish these explanations. Scoring choices also affect some results: in Table 25, adding capitalized “Unknown” and applying strict scoring reduces Llama-3.1’s unmixed accuracy from 73.75% to 18.5%. Gemma’s results are more robust, but these issues limit the mechanistic conclusions and cross-model evidence from the two-hop experiments.

Questions for authors
  1. Does “receiving position” refer to the actual token position, relative position, or relational role within the sentence structure? Can the existing experiments distinguish these factors?

  2. Considering only conditions in which random replacement can change the answers, does the middle-layer preservation of original pairing hold across models and both WHO and WHAT questions?

  3. When content mixing causes two-hop failures, can the model still correctly answer a direct question about the second fact, such as “What has the tag?” If this has not been tested, please provide the comparison to distinguish fact-access failure from failure to combine facts.

  4. How do the authors explain Llama-3.1’s sensitivity to the capitalized “Unknown” candidate? Does evaluation with a more complete candidate set or free generation still support the effectiveness of separate writes and the recovery from leaving the tag position unmixed?

  5. Single-component exchanges may create combinations that are incompatible with the model’s original state structure. What evidence shows that the advantage of joint exchange comes from preserving pairing information rather than differences in state compatibility?

Reviewer_uXhCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adopts activation writing in frozen language models to examine how entity binding depends on states and their receiving positions. By estimating and exchanging Slot and Content factors across different layers, this paper finds that early pairing mainly follows where Content is received, while moving both factors together at middle layers better preserves the original pairing. It also shows that separately written facts support two-hop answers close to the text input, whereas Content mixing can impair two-hop answers even when receiving positions remain intact.

Critical strengths
  1. The paper addresses an interesting question of how entity binding is established and subsequently carried by model representations.

  2. Its cross-layer Slot and Content exchanges and replacement controls provide a careful way to compare the effects of written states and receiving positions.

  3. The two-hop experiments test whether written facts can be used jointly, extending the binding analysis beyond direct factual recall.

Critical weaknesses
  1. The paper does not establish a direct causal relation between the binding mechanism studied in Section 4 and the two-hop results in Section 5. The drop in two-hop accuracy under Content mixing does not identify which intermediate step fails.

  2. The key Content-mixing experiments use a controlled synthetic task and do not evaluate an actual method for injecting or compressing knowledge. The implications for practical memory systems therefore remain untested.

  3. Some recent related studies are missed, which are neither cited nor discussed, such as [1-2]

[1] Dai et al., Cell-Based Representation of Relational Binding in Language Models (ACL 2026)

[2] Tang et al., Do Language Models Track Entities Across State Changes? (ICML 2026).

  1. The presentation could be further clarified and improved.

For example,

  • The abstract and introduction only briefly explain what prior binding work leaves unresolved and why the mechanistic question motivates the two-hop experiments.

  • Section 2 makes the task’s inputs and outputs a bit difficult to follow.

  • Figure 2’s caption does not explain its symbols, arcs, question marks, or “swap” intervention.

Questions for authors

Please refer to Critical Weaknesses.

Reviewer_HHNCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

After a language model reads “Alice has the umbrella. Ben has the bottle,” it must track that Alice goes with the umbrella—an instance of the entity binding problem. The authors test two hypotheses: (1) the content hypothesis, according to which the binding is carried in the content of the entity representations—as in the binding ID vectors of Feng & Steinhardt (2024)—and (2) the position hypothesis, according to which the model computes the binding from the positions where those representations reside.

They find that, in early layers, the binding follows the position where an entity’s content is received, which they take as support for the position hypothesis. From the middle layers onward, moved representations retain their original binding, which the authors interpret as evidence that the relation has become encoded in the representation itself.

For injected knowledge, they report that facts written separately at their respective entity positions support two-hop answers at a level close to that achieved with plain text, and that mixing in another entity’s content disrupts two-hop answers even when direct recall remains intact.

Critical strengths

None

Critical weaknesses

1. The experiments use vectors the model never produces, and their conclusions are close to what the architecture and earlier work already predict.

The vectors written into the model are averages and recombinations, not the model's own states.

  • To build the vector for "umbrella", the authors average its state over many prompts in which the umbrella belongs to different people (p.16). They then split this average into a position part (Slot) and an entity part (Content), assuming the two simply add up (p.3). The authors note that the recombined sum "need not equal the full µ" (p.16). These vectors are used in the layer-by-layer swaps of Fig. 3 and Table 1 and in the writes of Table 2.
  • Writing only the Content part makes the binding-head keys less like the text condition than writing nothing at all (Table 2). This suggests that the constructed vectors lie outside what the model normally sees.

Early layers. The early-layer result is close to automatic.

  • All three models use RoPE, which adds position information only inside attention and never adds it directly to the residual stream. At the first layer, the state of a word is just its token embedding, so swapping the states of "umbrella" and "bottle" is the same as swapping the two words in the input.
  • In the rest of the early layers, swapping only the Slot part leaves the binding almost unchanged in Gemma and Llama-3.1 (Table 7). The position part of the state barely matters there. This is what one would expect when position enters only through attention, so these layers cannot tell the two views apart.

Middle layers. The middle-layer result repeats the binding ID finding.

  • Each injected vector is averaged over many different partners, so it carries no information about whether the umbrella belonged to Alice or to Ben. What it does carry, besides the word itself, is a trace of the position the umbrella came from, which is its Slot.
  • Feng & Steinhardt (2024, Sec. 3.2) already showed that moving entity activations to other positions through RoPE barely changes the binding. The manuscript does not discuss this result. The claim that "the established relation becomes carried by the representation itself" (p.1) goes beyond the evidence, because the moved vectors contain the entity and its source position but nothing about its partner.

2. The manuscript does not cite two closely related prior works.

  • Patchscopes (Ghandeharioun et al., 2024) writes a hidden state into a placeholder token of a target prompt and lets the forward pass continue (Sec. 4.2). This is the core operation of activation writing, and the manuscript differs in writing donor means at several positions at once and over successive layers. Patchscopes Sec. 5 also patches representations into a second-hop prompt to correct two-hop answers.
  • Prakash et al. (2025, arXiv:2505.14685v3) transplant token states, layer by layer, from counterfactual runs in which the order of the sentences or entities is swapped. In Llama-3-70B-Instruct the moved state carries its order-based binding in a band of middle layers (Sec. 5.2, Figs. 5 and 6). This is a close analogue of the layer-wise localization in Sec. 4.1. The manuscript cites Prakash et al. (2024, arxiv:2402.14811), a different paper.
Questions for authors

Sec. 4.2 and Sec. 5 overwrite the state at every layer from 0 to 8, while Sec. 3 writes once at the input to layer 1 (p.13). The stated reason, that this range precedes the first binding head (p.6), explains where the writes stop but not why they repeat. A single write already suffices for the synthetic two-hop task. At layer 1 it gives 396 correct answers, and a single additive write at layer 1 matches replacement over layers 0 to 8 (Table 20). In Llama-3.1 the repeated write even exceeds text, with 296 against 268. Why do the main experiments repeat the write at every layer, and do the factor-write results in Table 2, in particular Shared Slot, hold with a single write?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.