- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how additional context with same label can disrupt a model’s answer. The authors identify pairs of statements that the model evaluates correctly in isolation but incorrectly when presented together, even when both statements have the same ground-truth label. Using activation patching, they find that intermediate target-token states can carry changes sufficient to reproduce the erroneous behavior. They further introduce a residual-drift metric that correlates with the magnitude of the margin shift at later layers. However, the analysis provides limited insight into why these changes arise. Given existing research on irrelevant-context interference, the contribution appears to be an additional experimental characterization of a known phenomenon. The interpretation that these failures reflect neural collision is not sufficiently established.
Critical strengths
-
The problem is worth investigating, and the same-label pairing setting provides an interesting way to study contextual interference.
-
The authors go beyond behavioral evaluation and attempt to investigate the model’s internal processing through causal interventions.
Critical weaknesses
-
The activation-patching experiments localize states that can transmit the erroneous behavior, but do not sufficiently explain the mechanism that produces it. They do not establish what information from the partner statement disrupts the target computation, or which model components cause this disruption. The additional mechanistic insight over prior work therefore remains unclear.
-
The interpretation of these failures as neural collision is insufficiently supported. The model is asked to evaluate only one statement, so the presence of another statement does not by itself establish that two independent evaluation processes are being executed and interfering with each other.
Questions for authors
-
The paper rules out answer competition via same-label pairing (Sec. 2, 4), but this does not show that two separate evaluation processes actually run in the joint prompt. What evidence supports the ‘collision’ interpretation over a single computation being distracted by the partner?
-
Could the results be partly explained by changes in input length, token position? Please provide appropriate controls and analyze whether errors are concentrated in particularly vulnerable target statements or particularly disruptive partner statements. What properties of the target–partner relationship determine whether interference occurs?
7 comments
Response to Reviewer sF59
2. Is this really a collision between two computations?
Reviewer comment:
"The interpretation of these failures as neural collision is insufficiently supported. The model is asked to evaluate only one statement, so the presence of another statement does not by itself establish that two independent evaluation processes are being executed and interfering with each other."
Our response:
We agree that the presence of two statements does not establish that the model evaluates both. To investigate whether two evaluations are taking place, we test whether the model represents the truth values of both statements before it knows which one will be queried.
Experiment: Are both truth values represented in the same hidden state? In the joint prompt, both statements appear before the question:
A: [statement A]
B: [statement B]
Is statement B true?
At the final token of B, we train two probes on the same residual-stream activation: one to decode A's truth value and one to decode B's. We use pairs balanced across all four truth-label combinations (TT, TF, FT, FF), ensuring that knowing one label does not reveal the other.
Results (Figure 20): Both truth values are decodable from this single state across all three models, while shuffled-label controls remain near chance (50%). This shows that information about both statements' truth values is represented concurrently, even before the model knows which statement will be queried. Crucially, information about A's truth value is available at B's final token, indicating that information from the partner has already entered the target's representation.
Manuscript reference: Appendix D, “Probing truth values before the question” (Figure 20).
3. Can prompt length or token position explain collisions?
Reviewer comment:
"Could the results be partly explained by changes in input length, token position? Please provide appropriate controls and analyze whether errors are concentrated in particularly vulnerable target statements or particularly disruptive partner statements. What properties of the target–partner relationship determine whether interference occurs?"
Our response:
We thank the reviewer for these suggestions and have added a new appendix section addressing each question.
Adding a partner changes the prompt in two simple ways: it makes the prompt longer, and it moves the target and question to later positions. We test each with a separate control.
Control 1: Partner length (Figures 12–13). If longer prompts caused the effect, longer partners should change the target's margin more. They do not: in all three models, the average size of the change shows no consistent increase with partner length.
Control 2: Length-matched whitespace (Figures 14–15). We replace each partner with the same number of whitespace tokens. The prompt length and the positions of all other tokens stay exactly the same; only the partner's content is removed. Across all three models and both target positions, matched whitespace produces far fewer collisions than a real partner.
Together, these results show that input length and token position alone cannot explain the observed collisions. The partner's content plays an important role.
Manuscript references: Appendix A.2, “Can changes in input length or token position explain collisions?” (Figures 12–15).
Response to Reviewer sF59
4. Are destructive neural collisions concentrated around particular statements?
Reviewer comment:
"Please provide appropriate controls and analyze whether errors are concentrated in particularly vulnerable target statements or particularly disruptive partner statements."
Our response:
We conducted additional analyses to determine whether collisions are driven by a few disruptive partners (statements that cause many targets to fail) or vulnerable targets (statements that fail with many different partners).
Experimental setup: We arrange all evaluated statement pairs in a matrix, with partners as rows and targets as columns. Each cell indicates whether that particular pairing causes a destructive collision. A few influential contexts would show up as rows full of collisions, and vulnerable targets as columns full of collisions.
Results (Figures 9–11):
-
Collisions concentrate on vulnerable targets (Figure 9). In all three models, collisions line up in a small number of columns: a few targets fail with many different partners.
-
These targets are close to the decision boundary. A collision occurs when a partner reduces the target’s margin by more than its solo margin, so less confident targets need only a small reduction to fail. Consistent with this, a target’s solo margin is negatively correlated with the fraction of partners that cause it to collide (Qwen: ρ = −0.435; Gemma: ρ = −0.654; Llama: ρ = −0.503).
-
Changes in the model’s margin are much less concentrated (Figures 10–11). Beyond collisions, we measure how much each partner reduces the target’s margin, even when the answer stays correct. To test for concentration, we compare against a permutation baseline that shows how much concentration chance alone would produce. Some targets receive somewhat larger reductions than others, most clearly in Gemma.
Conclusion: We find little evidence that a few influential contexts drive collisions. Almost every partner shifts the target’s margin to some degree. Collisions concentrate on particular targets mainly because those targets start closer to the decision boundary, so even an ordinary shift is enough to flip their answer.
Manuscript references: Appendix A.1, “Are collisions concentrated around particular statements?” (Figures 9–11).
5. What properties of the target–partner relationship determine whether interference occurs?
Reviewer comment:
"What properties of the target–partner relationship determine whether interference occurs?"
Our response:
One possibility is that collisions arise from confusion between semantically similar statements. For example, two statements about capital cities might interfere more strongly because they involve the same factual relation. If this explains collisions, we would expect statements from the same semantic category to collide more frequently.
Experimental setup: Our statement bank contains facts from different semantic categories (capital cities, authors, sports, etc.) alongside arithmetic statements. We compare destructive collision rates for pairs from the same category against pairs from different categories, and examine the full category-by-category breakdown.
Results (Figure 16):
- Qwen3-32B: Same-category and different-category pairs have identical collision rates (1.7% vs. 1.7%).
- Llama-3.1-8B: Same-category pairs collide slightly less often (5.2% vs. 6.5%).
- Gemma-4-31B-it: Same-category pairs collide roughly twice as often (13.1% vs. 6.6%).
Figure 16 further shows how collision rates vary across individual category pairings.
Conclusion: Semantic category similarity does not consistently predict collisions across models, although it appears to matter for Gemma. More specific properties of the relationship between statements may still matter, and identifying these remains an open question.
Manuscript reference: Appendix A.3, “Are collisions more likely between statements from the same semantic category?” (Table 1, Figure 16).
Response to Reviewer sF59
We have highlighted all changes to the paper in light red to make them easy to identify.
1. Mechanism of destructive collisions
Reviewer comment:
Our response:
We thank the reviewer for this question. We added three experiments. The first identifies which model components contribute to collisions. The second investigates why information from partner A causes disruption. The third examines what information in statement B's representation is affected.
Experiment 1: Which component causes the disruption?
Within a transformer layer, self-attention is the only operation that transfers information between token positions. We therefore test whether attention from B's tokens to A's causes interference. For up to 100 destructive collisions per model, we prevent B's tokens from attending to A's tokens across progressively more layers. Both statements remain in the prompt, so statement A can still influence the final answer, but cannot directly alter B’s representation in the masked layers.
Results (Figure 6): Blocking this attention corrects most collisions:
Attention from the target's tokens to the partner's tokens therefore contributes causally to collisions. This also suggests a simple, training-free remedy: block attention between independent statements while still letting the answer draw on both.
Experiment 2: Does the harm come from contamination or attention dilution?
Attention from B to A can harm B in two different ways:
Blocking attention removes both effects together. To distinguish them, we exploit the additive structure of attention to intervene on each separately:
The mathematical decomposition and interventions are detailed in Section 6.2.
Results (Figure 7):
Collisions are therefore not simply caused by attention being pulled away from the target’s own tokens. The information B reads from A is itself a major source of the damage.
Experiment 3: What task-relevant information in B is affected?
Experiments 1 and 2 show that information from A, carried by cross-statement attention, contributes causally to collisions, but they leave open what in B's representation is affected. One possibility is that interference disrupts information about whether B is true or false.
To test this, we train a linear probe to predict whether B is true or false from its internal state at B's final token. If interference affects this information, we would expect the probe to be less accurate on pairs that collide. We test this prediction by applying the fixed probe to held-out collision and non-collision pairs.
Results (Figure 21): Across all three models, probe accuracy is substantially lower on collision pairs. This is consistent with interference affecting information about B's truth value before the question is read.
Summary.
Together, these experiments go beyond localising where collision damage appears and show how it arises. Information from the partner enters the target’s representation through cross-statement attention (Figure 6). The damage comes mainly from this information itself, not from the attention it takes away from the target’s own tokens (Figure 7).
Manuscript reference: Section 6.1, “Does blocking attention between statements prevent collisions?” (Figure 6); Section 6.2, “What makes cross-statement attention harmful?” (Figure 7); and Appendix D, “Probing truth values before the question” (Figure 21).