acceptodds
Withdrawn from ICLR 2027

Few Neurons, Many Roles: Sparse MLP Neurons That Control Attention Sinks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) send a disproportionately large share of attention to the first token, a pattern known as the attention sink. Although this pattern is widespread and functionally important, it is debated which upstream units build it and whether those units serve only the sink. We identify sparse populations of MLP neurons that causally control the sink across two model families. Suppressing these neurons weakens the sink by 12 to 51%, while amplifying their natural activity strengthens it by 1 to 6%. Exact softmax decomposition and live restoration show that these neurons act through later queries and keys, with a pathway mix that differs across models. Evaluating the models with the neurons suppressed reveals that these neurons significantly change downstream behavior differently across model families and benchmarks. Restoring the exact sink while the neurons remain suppressed recovers little or none of the behavioral effect, indicating that the sink explains little of this effect We further find that preserving their coordinates in high precision while quantizing the model to W8A8 recovers 65% of the loss, despite these neurons having little overlap with magnitude-selected neurons. A few neurons help build the sink, but their role reaches well beyond it.

Primary area:
interpretability and explainable AI
Keywords:
attention sinks, mechanistic interpretability, sparse MLP neurons, neuron-level interpretability, causal interventions, activation suppression and amplification, query–key pathways, polyfunctional neurons, language model behavior, quantization, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7XzDOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies sparse MLP neurons, termed Sink-Control Neurons (SCNs), that causally regulate attention sinks in large language models. Experiments on four Qwen and Llama checkpoints show that suppressing a small number of SCNs reduces sink strength by 12%–51%, while amplifying their activations increases it by 1%–6%. The authors further investigate the underlying query–key pathways, evaluate the effects of SCN suppression on downstream task performance, and examine whether restoring the original attention sink can recover the resulting behavioral changes. Finally, they demonstrate that preserving SCN coordinates in high precision recovers 65.3% of the quantization-induced cross-entropy loss in a W8A8 experiment. The results suggest that a small population of MLP neurons contributes to attention sink formation while also influencing broader model computations.

Critical strengths
  1. The paper provides a systematic causal identification pipeline for locating sink-controlling neurons. The combination of gradient-based candidate selection, direct neuron-level interventions, and layer/count-matched random controls provides evidence that the identified neurons genuinely influence attention sink strength.

  2. The authors go beyond identifying sink-controlling neurons and investigate their underlying mechanisms. The query–key pathway decomposition and restoration experiments provide useful insights into how SCNs influence attention allocation, while also revealing differences across model checkpoints.

  3. The downstream behavioral evaluation is informative. Experiments across multiple benchmarks show that SCN suppression can substantially alter model predictions, even when the net accuracy change is small. The analysis of prediction transitions further highlights behavioral changes that aggregate accuracy alone may obscure.

Critical weaknesses
  1. Missing relevant work on sparse causal neuron identification. The paper does not discuss the closely related work https://arxiv.org/abs/2510.10238. This work proposes a perturbation-based method for identifying ultra-sparse critical neuron populations and investigates their effects on language model capabilities, including their layer-wise distribution and the consequences of disrupting these neurons. Given the overlap in neuron-level causal identification and intervention-based analysis, the authors should discuss this work and clarify how the present study differs from and advances beyond it.

  2. The generality of the findings remains unclear. The experiments cover only four instruction-tuned checkpoints with at most 8B parameters, and several analyses, particularly the pathway decomposition and quantization experiments, are conducted on only a subset of these models. Moreover, neuron localization relies on short neutral-text sequences. It is therefore unclear whether the identified SCNs and their functional roles generalize to base models, larger models, and long-context settings.

  3. The evidence connecting attention sink restoration to downstream behavior is not fully conclusive. Although the authors restore the original sink attention distribution while keeping SCNs suppressed, the behavioral recovery is limited. However, the paper explicitly reports that the sham-control adequacy criterion fails for all evaluated tasks. This limits the strength of the causal interpretation and makes it difficult to isolate the specific contribution of position-0 attention to the observed behavioral effects.

  4. The practical advantage of SCN-based quantization protection is not fully established. Although preserving SCN coordinates recovers 65.3% of the W8A8 quantization-induced cross-entropy loss, both layer-matched and global magnitude-based protection perform better in the reported experiment. Thus, the results establish that SCNs are relevant to quantization sensitivity, but do not demonstrate that SCN-based protection is more effective than simpler magnitude-based alternatives.

Questions for authors
  1. The authors should evaluate the generalization of SCNs across model scales, model families, and context lengths. In particular, testing on base checkpoints and longer sequences would help establish whether the identified neurons reflect a robust mechanism or a property specific to the selected instruction-tuned models and localization setup. Additional cross-model analyses of SCN overlap and functional consistency would also be valuable.

  2. The authors should strengthen the attention restoration experiments with better-controlled sham interventions that satisfy the predefined adequacy criterion. Additional controls and behavioral mediation analyses would help determine whether the limited recovery is genuinely attributable to the restored attention sink or whether other effects of the intervention account for the observed behavior.

Reviewer_t17nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the mechanistic origin of attention sinks in LLMs, the well‑known phenomenon where models allocate excessive attention weight to the initial input token. The authors identify a sparse set of MLP intermediate neurons, termed sink‑control neurons (SCNs), across four instruction‑tuned models (Qwen2.5‑1.5B/3B, Llama‑3.2‑3B, Llama‑3.1‑8B). Using gradient attribution followed by direct causal neuron intervention (scaling/ablating activations without weight modification), they show that suppressing SCNs reduces sink strength by 12–51%, while amplifying SCN activity increases sink strength by 1–6%.

Critical strengths
  1. The work follows a rigorous attribution‑then‑verify circuit‑analysis workflow. Instead of relying only on correlational gradient signals, the authors perform direct activation intervention, use layer‑and‑count‑matched random neuron controls, and strictly freeze all hyperparameters and data splits before opening held‑out test sets. Bidirectional dose‑response experiments establish causal modulation of attention sinks rather than mere correlation.

  2. The key finding, i.e., sink‑controlling neurons are not sink‑specialized neurons, advances interpretability research on attention sinks. Prior literature often implicitly assumes sink‑related components exist primarily for sink behaviour. This paper shows the sink is just one observable downstream effect of these sparse MLP units. The sink‑rescue experiment is a clever counterfactual to test mediation.

Critical weaknesses
  1. Limited model scope: All experiments use instruction‑tuned checkpoints ≤8B parameters. The paper does not evaluate base pre‑trained models, larger‑scale models (>8B), or long‑context fine‑tuned variants. It remains unknown whether SCN phenomena and pathway heterogeneity generalise beyond this narrow setting. Several expensive analyses (pathway decomposition, sink rescue) are not run for all four models.

  2. The sink‑rescue mediation experiment has critical caveats: The authors themselves document that the sham adequacy gate fails. Therefore, they cannot make strong causal mediation claims about whether sinks causally drive downstream behaviours. The conclusion ``sink explains little behavioural effect” is suggestive but not definitive proof of non‑mediation.

  3. Quantization analysis is only a diagnostic probe, not a practical method: Although protecting SCNs recovers ~65% W8A8 loss, magnitude‑based selection performs better. Besides, no benefit is observed at W8A4. Thus, practical utility for compression is limited.

Questions for authors
  1. You report strong model‑specific heterogeneity in dominant Q/K pathways for sink formation (Qwen‑1.5B balanced Q/K0K_0/KrK_r; Qwen‑3B dominated by K0K_0; Llama‑3.1‑8B dominated almost entirely by queries). What architectural or training‑related factors do you conjecture drive these large cross‑model pathway differences?

  2. Your localization pipeline uses short 40‑token sequences. How might longer context change SCN selection and their causal effects? Could sink‑control neurons differ when inputs are hundreds or thousands of tokens long?

  3. Your sink‑rescue experiment shows restoring sink attention recovers little behaviour, but the sham adequacy condition fails. Could you clarify what concrete conclusions we can and cannot safely draw from this rescue result, given the sham failure? Could you suggest feasible experimental improvements to strengthen the mediation test?

Reviewer_f5z7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper deals with so called attention sinks, that is, tokens (seemingly the first one) that are assigned a large portion of attention. In particular, the authors claim to identify a tiny subset of MLP neurons that controls these attention sinks, thereby providing insights into the mechanistic circuits of the model. Most interestingly to me, keeping these neurons in high precision while quantizing the rest seems to recover much more of the quantization loss than pure quantization, highlighting the significant role these identified neurons seem to have.

Critical strengths

I think the topic is very interesting and the related work seems to be extensive, and although I often don't understand the motivation, the math is relatively clear. If the claims truly hold, I personally would be very interested in improving quantization through identifying such individual neurons that play an important role, apart from the usual "outlier analysis" of quantization methods (i.e., within a single layer only). I like the view of sinks being a "no-op", where I learned something new.

Critical weaknesses

I have several concerns w.r.t. this paper that make me lean towards rejection. I will outline my concerns below. I will put Questions separately, although many of my concerns are related to questions where I just don't know how the authors ended up with their respective decisions. For the moment, I recommend a score of 1, but I'm looking forward to seeing the authors responses.

Soundness and Clarity

My main concern is that the paper is really hard to follow. Many paragraphs use words that I cannot decipher from the context, and the paper is generally not very well written. Although I don't want to accuse the authors of anything, I feel like large parts are LLM-polished until the point where it's hard to follow. Let me give an example of a sentence I cannot understand, and where I don't know whether its my fault:

  • L151: "Each shortlisted neuron is silenced individually on 20 discovery passages [...]": What is the shortlist here? Whats silencing? Whats a discovery passage? This sounds like gibberish to me and if it's not LLM-polished, then I think the presentation should be made much more clear.

Let me give another example where I think the language is off:

  • L455: "Magnitude does not explain the effect: SCNs overlap only 9/116 coordinates with the layer-matched magnitude set [...]": I have worked a lot on LLM sparsity, but what exactly is the layer-matched magnitude set? Where can I find the sentence where this is introduced?

That being said, some other concerns I have:

  • The models considered are far from state of the art, they are small (which would be fine), but Qwen-2.5 is like fall 2024, why are you using such old models? This weakens the study without any reason.
  • In section 8, you say that "sinks are frequently linked to activation outliers and quantization difficulty": if that is not your original/novel finding, how does your work relate too it? this is left unclear. Are you just testing this again? Is this a novel contribution on your side?

Minor

  • L40: "mechanistic studies", what is meant here? mechanistic interpretability studies?
Questions for authors
  • L141: What are "second-half queries"?
  • I could list a couple dozen sentences where I don't understand the words, but I makes no sense to list them all. The presentation must be improved to be considered for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.