- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates "sink neurons", defined as neurons that activate "disproportionately often near sequence beginnings". The authors first show that these sink neurons exist and harm early-token prediction, that these neurons are more robust than absolute position alone and ignore unrelated context, and that copying sink neuron activations to middle tokens (in a 2-layer model) reduces the lost-in-the-middle effect and improves retrieval for those tokens.
Critical strengths
- The writing is mostly clear and the experiments cover most potential confounders for the presence and effect of these sink neurons.
- The result in Section 6 is a very interesting way to go beyond just showing certain neurons that activate early in the sequence exist, and showing that sink neurons impact model behavior in an interesting way.
Critical weaknesses
- The high level motivation for the work is lacking. From prior work we know that various positional neurons exist, and so it does not feel particularly surprising that early-position neurons also exist [1,2]. Could the authors clarify how their sink neurons differ from the positional neurons found in prior work and whether sink neurons behave in any unforeseen ways? The paper currently seems like finding neurons that are activate early in the sequence and then showing results that are nearly by-definition (ablating sink neurons harms early sequence loss)
- The result in Section 6 is very cool but I do not buy the current result. Why is a model trained from scratch for this result? If the authors can find sink neurons in open-weight models, why can these not also be tested? Additionally, why can error bars not be evaluated for the current result? I see that the random patch is computed as an average over multiple neuron groupings, but can the intervention evaluation not be extended further to get a sense of whether this 2% improvement is a robust result or due to noise?
[1] Gurnee, Wes, et al. "Universal neurons in gpt2 language models." arXiv preprint arXiv:2401.12181 (2024). [2] Voita, Elena, Javier Ferrando, and Christoforos Nalmpantis. "Neurons in large language models: Dead, n-gram, positional." Findings of the Association for Computational Linguistics: ACL 2024. 2024.
Questions for authors
- Could an explanation be provided for how exactly the sink neuron copying result should improve lost-in-the-middle? This seems like a very interesting result that is currently underexplored.
4 comments
(3) Error bars and robustness of the controlled result (W2). The controlled-model effect is stable across samples and training runs.
- Paired CIs. Clean and patched runs share inputs, so we bootstrap over paired samples. On the paper's 2,000 samples, the gap changes by −2.05 [−3.10, −1.05] points, and middle-target accuracy by +1.55 [0.80, 2.30] (44 errors fixed, 13 introduced; McNemar p = 4.7×10⁻⁵). With 10,000 samples, the gap change is −1.71 [−2.22, −1.23].
- Training seeds. We trained five more checkpoints from scratch, each with a new initialization, data stream and task draw, and reselected the sink neurons in each with the unchanged rule. The effect replicates in all six checkpoints (gap change −2.87 ± 0.74 pp, mean ± SD across checkpoints; all CIs exclude zero). Random sets instead increase the gap (+3.11 pp) and never reach the sink effect.
- Why the effect is small. In the two-layer model, only first-layer multilayer perceptron (MLP) outputs at context positions can reach the final query: second-layer MLP outputs at those positions are never read, and copying them leaves accuracy unchanged. The original intervention is thus confined to eight neurons of one layer and moves the state in one direction only; moving each pair's own state in both directions reduces the gap by 10.7 pp (response (4)).
(4) How the copy reduces first-over-middle preference (Q). In short, neuron sink is an internal processing bias that likely gives rise to the behavioral one, and the copy weakens it. Evidence at the start is processed with scarce context and carries the state that sink neurons mark; later queries attend more to evidence carrying this state. The copy gives middle evidence this state, narrowing the internal difference between first and middle evidence and, with it, the first-over-middle preference. The controlled task lets us trace each step:
- Internal bias. A query must find the pair whose content matches it and read out that pair's label. In the paired evaluation, the first and fifth pairs carry the same content vector, but under causal masking the first pair's label token sees only its own content token and thus carries the low-context state, whereas the fifth pair's label token follows eight tokens from other pairs.
- From internal to behavioral bias. Despite the identical content, the query's second-layer attention to the first pair is 2.8 times that to the fifth (0.27 vs. 0.10), consistent with the higher accuracy when the first pair holds the answer (§6).
- Reducing the internal bias reduces the behavioral bias. We transplant each pair's own state, so that no content or label passes between pairs: the middle pair's beginning state comes from running the pair alone, and the first pair's middle state from placing it fifth in another sample. Giving the middle pair the beginning state raises the attention it receives by about 28%, and giving the first pair the middle state lowers its attention by about 21%; the same transplants on random neurons change attention by less than 5%. Each transplant also shrinks the gap, by 10.7 points when both are applied (means over the six checkpoints; gap changes in points, mean ± SD; negative values shrink the gap; every CI excludes zero, and no random set reaches the sink effect):
| Intervention | Query attention, first / middle pair | Gap change, sink neurons (pp) | Gap change, random sets (pp) |
|---|---|---|---|
| None (clean) | 0.27 / 0.10 | – | – |
| Original copy (§6): first pair's activations → middle pair | 0.26 / 0.12 | −2.87 ± 0.74 | +3.11 ± 0.57 |
| Middle pair ← its own beginning state | 0.26 / 0.12 | −3.67 ± 0.71 | −0.46 ± 0.20 |
| First pair ← its own middle state | 0.21 / 0.10 | −5.75 ± 0.73 | −1.25 ± 0.46 |
| Both in one forward pass | 0.20 / 0.14 | −10.66 ± 1.37 | −1.79 ± 0.61 |
- Gurnee, Wes, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. (2024). [Universal Neurons in GPT2 Language Models]
- Liu, Nelson F., Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. (2024). [Lost in the Middle: How Language Models Use Long Contexts]
- Voita, Elena, Javier Ferrando, and Christoforos Nalmpantis. (2024). [Neurons in Large Language Models: Dead, N-gram, Positional]
- Wu, Xinyi, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. (2025). [On the Emergence of Position Bias in Transformers]
If the authors have updated the draft, could they use a consistent color throughout to show changes? It is currently difficult to track changes.
Additionally, the results on Qwen3 0.6B seem to be much more drastic than the previous results. Could the authors provide additional information on this experimental setup?
- Is the patching procedure consistent among all positions? Does the patching degrade the early items?
- What is meant by "copy the sink-neuron activations of evidence placed first, in all layers"? This patches the "31 neurons in each of the 28 layers based on their relative activation rates in the first 32 tokens."?
- Why do the authors think the gain does not hold on the natural questions dataset?
We thank the reviewer for these follow-up questions. We have now highlighted all revisions in the same color throughout the updated manuscript.
(1) Experimental setup. The full experimental setup is now detailed and highlighted in Section 6.3 and Appendix I of the revised manuscript. We summarize the key details below.
We use Qwen3-0.6B-Base with 2,048-token KV prompts containing 408 mappings, placing the queried mapping first or 205th. NQ uses ten passages, placing the answer-containing passage first or fifth, and evaluates 2,107 questions answered incorrectly without passages. The Qwen intervention covers 31 coordinates per layer across 28 layers. Appendix I specifies the prompts, token alignment, scoring, and paired statistics.
(2) Consistency across positions and effects on early items. The patching rule is fixed across target positions, and patching middle evidence improves Qwen retrieval with little average change in early-item accuracy.
In the two-layer experiment, the same first-to-middle copy is applied in both first-target and middle-target trials. Table 1 already reports both accuracies: across six checkpoints, First decreases from 86.14% to 85.25%, while Middle increases from 69.11% to 71.08%.
For Qwen, we additionally measured both target positions under each fixed rule. The forward transplant always patches the current middle item with its own clean beginning state; the reverse transplant always patches the first item with its own clean middle state. Neither rule changes with the queried item or gold-passage identity. The two directions are evaluated separately on the original samples.
| Task | Intervention | First accuracy (%) | Middle accuracy (%) |
|---|---|---|---|
| KV | Clean | 99.70 | 78.70 |
| KV | Own beginning → middle | 99.70 | 92.05 |
| KV | Own middle → first | 99.50 | 77.95 |
| NQ | Clean | 52.68 | 32.84 |
| NQ | Own beginning → middle | 52.40 | 36.17 |
| NQ | Own middle → first | 50.12 | 33.79 |
Patching middle evidence causes no First correctness flips on the 2,000 KV tables. For NQ, First changes by −0.28 percentage points (95% paired CI [−1.04, 0.52]), while Middle improves by +3.32 points [2.28, 4.41]. The estimated First cost is therefore small, although modest harm remains possible. The reverse transplant reduces NQ First accuracy by 2.56 points; its KV First change is only −0.20 points.
(3) Meaning of “copy … in all layers.” Yes: we patch the fixed 31 neurons in each of the 28 layers, using aligned activations from the same evidence.
These neurons are selected by first-32-token relative activation rates using Q98. That window defines selection, not the patch span. A clean donor run places the evidence first; the recipient places it in the middle. At each evidence token and layer, we replace the selected pre-down-projection MLP coordinates with their corresponding donor values. The span covers the complete five-token KV mapping or the complete NQ passage, including its title and terminating newline. See Appendix I for patching and Section 3.2/Appendix A for selection.
(4) Why is the NQ improvement smaller? Many gold-answer scores improve without changing the generated answer, which may partly explain the smaller accuracy gain.
In a post-hoc analysis, 510 NQ questions were correct with first-position evidence but incorrect with middle-position evidence. The transplant corrected 79. Among the remaining 431, gold-answer log-probability increased in 383 cases (88.9%). KV selects one token from 408 candidate values, whereas NQ freely generates an answer: improving the teacher-forced likelihood of the best gold alias need not change the greedy-generated response.
We thank the reviewer for the careful review and for finding the retrieval result interesting. We first clarify the motivation and how sink neurons differ from positional neurons (1). We then address the retrieval result: new evidence in a pretrained LM (2), its robustness (3), and how the transplant changes retrieval (4).
(1) Motivation, and relation to positional neurons (W1). Our question is whether a language model computes differently when useful context is scarce, not whether some neurons prefer early positions. Prior work characterizes early-position neurons by absolute position; we find sink neurons respond to useful context even at fixed positions, are shaped by the training data, and affect how evidence is later retrieved. In detail:
(2) Behavioral evidence in a pretrained LM (W2). Following the reviewer's suggestion, we tested the open-weight model in which we characterized neuron sink: transplanting the beginning state of sink neurons into middle-position evidence improves its retrieval. We had used a controlled model because the paired first/middle protocol of Wu et al. (2025) fixes content exactly and its small size allows exhaustive controls.