acceptodds
Under review as a conference paper at ICLR 2027

Attention Sees, MLP Writes: Tracing the Recognition-to-Refusal Pathway in LLMs

Abstract

Safety-aligned large language models (LLMs) remain vulnerable to jailbreaks and can over-refuse benign requests. Understanding these failures requires distinguishing harmfulness recognition from the computation that writes refusal. We reframe this as a pathway question and make it measurable on residual-stream updates in decoder-only Transformers, separating the harmfulness information a module update exposes from the change it produces in readable refusal evidence. Across 18 Base–Instruct pairs spanning Qwen, Llama, and Gemma, harmfulness is decoded more readily from attention updates, particularly in early layers, whereas MLP updates contribute more strongly to harm-selective refusal writing in aligned models. Post-training strengthens MLP refusal writing in all 18 pairs without systematically improving harmfulness decodability, while positive MLP writing becomes more concentrated in deeper layers after post-training. Across six reasoning-enabled models, harmfulness remains decodable during reasoning while strong refusal writing concentrates at answer onset, a pattern that persists with reasoning disabled. Causal interventions further support this pathway: resetting downstream MLP updates to their clean values substantially reduces the refusal shift caused by upstream attention interventions. Building on these findings, we introduce Sentinel, which combines an attention-derived perception score P with an MLP-derived decision score D. Their joint profiles characterize jailbreaks and over-refusals, and conditional routing with both signals achieves a lower aggregate error rate than single-signal routing and evaluated fixed reminders across eight models.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, LLM safety, refusal mechanisms

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hKeKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how decoder-only LLMs turn recognition of a harmful request into a refusal, and frames this as a module-level pathway measured on residual-stream updates at the answer boundary. Harmfulness recognition is measured by linear-probe decodability of attention and MLP updates; refusal writing by the change each update makes to a logit-lens readout contrasting refusal and compliance tokens (RWS), compared between harmful and benign prompts (Δ\DeltaRWS). Across 18 Base–Instruct pairs from the Qwen, Llama and Gemma families (1B–27B), attention updates are more decodable for harmfulness, while MLP updates dominate harm-selective refusal writing in aligned models; post-training increases MLP writing in all 18 pairs without systematically improving decodability. In six Qwen3/Qwen3.5 reasoning models, strong MLP writing concentrates at answer onset. Activation patching that combines an upstream attention patch with downstream MLP restoration is used to support a functional link. Finally, Sentinel uses attention-side (P) and MLP-side (D) features to route conditional reminders before decoding, and reports a lower average error than fixed reminders across eight models (30.34% vs 39.53% for the best fixed reminder).

Critical strengths
  1. The paper provides a simple, exact and reusable measurement framework. RWS attributes the change in a logit-lens refusal readout to individual attention and MLP updates, and by construction the per-module terms add up exactly to the change in the readout across the network. This turns the question of which module recognises harmfulness and which module writes refusal into precise, testable quantities, on which every later result in the paper, including Sentinel, is built.
  2. The empirical evidence is broad, consistent and candidly reported. The main findings are reported as counts over 36 checkpoints from three families (e.g., MLP net writing rises after post-training in 18/18 pairs while probe AUC does not systematically rise), and they are checked against controls that are often omitted: alternative and data-driven readout vocabularies (MLP net writing exceeds attention net writing in 11/12 model–readout configurations), a matched-input/matched-position Base–Instruct control, matched attention restoration, random MLP restoration, and generation-level outcomes. The authors also report unfavourable results, such as weak blocking on Qwen3.5-9B and late-layer concentration in only 2/6 matched pairs, which makes the descriptive conclusion that post-training mainly changes refusal writing rather than harmfulness recognition more credible.
Critical weaknesses
  1. The module-level asymmetry may not be specific to safety. The main measurements are taken at the position that predicts the first answer token, which for Instruct models is the final template token. Information about the request can reach that position only through attention, while the MLP operates on what is already there, so higher decodability of attention updates and separability gains coming only from attention (Fig. 1b) could be expected for almost any prompt-level attribute. Without a non-safety prompt attribute and a non-safety first-token contrast measured in the same way, the results cannot distinguish a safety-specific pathway from a generic division of labour at the answer position. This affects the paper's central claim, since the title and the pathway account rest on the attention/MLP split.
  2. The post-training result partly restates the behavioural change, and its depth claim is not robust. By the definitions in Section 2.2, the RWS terms telescope: the attention and MLP layer sums of Δ\DeltaRWS add up to the harmful–benign gap in the final readout minus the gap at layer 0. Because Instruct models refuse and Base models largely do not, the increase in total writing after post-training mostly reflects this behavioural difference; the informative part is the split between modules and across depth, which relies on logit-lens readouts at intermediate layers, where such readouts are least reliable. Moreover, the abstract's claim that positive MLP writing becomes more concentrated in deeper layers holds in only 2/6 pairs under the authors' matched-input, matched-position control (Table 9), so this claim should be restricted to the standard template setting.
  3. The causal evidence supports a coarse effect rather than a specific attention-to-MLP pathway. Restoring randomly chosen post-P MLP layers already blocks 49.3% of the upstream effect for Llama-3.1-8B and 73.5–84.4% for Gemma4-12B, compared with 64.4% and 82.3–86.6% for the selected D layers (Table 3). In the held-out comparison, MLP restoration removes only 26.5% of the effect for Llama-3.1-8B (Table 2), and blocking on Qwen3.5-9B is 19% and 11%. The selected layer windows are also unstable: for Llama-3.1-8B, P = 11–13 and D = 14–18 with 100 calibration pairs (Table 14), but P = 10:17 and D = 22:23 with the first 50 pairs (Appendix B.2). The authors acknowledge that the single-module patching does not show a layer-by-layer chain; the title and abstract should be aligned with this pathway-level evidence.
  4. The Sentinel evaluation does not isolate the contribution of the pathway features. In the routing data (Table 10), the jailbreak cohort consists of successful attack-template prompts (ReNeLLM, PAIR, AutoDAN, GCG), while the correctly refused examples come from plain AdvBench requests, so the jailbreak/refusal distinction is confounded with prompt format; the matched XSTest/EvoRefuse cohorts control for this only on the over-refusal side. Cross-validation is grouped by underlying request but not by attack family, so the same attack families appear in training and test folds, and the router may be recognising prompt templates rather than pathway states. There is no leave-one-attack-family-out evaluation and no baseline that trains the same classifier on generic last-token activations, so the gains cannot be attributed to the P/D features.
  5. The routing results do not support the P–D diagnostic picture and omit key costs. In the authors' own low-dimensional control, routing on mean P and mean D gives an average error of 39.34% (Table 12), essentially the same as the fixed safety reminder (39.53%), so the two-dimensional P–D plane of Fig. 5 is not what drives the 30.34% result. The jailbreak and over-refusal cohorts are pre-selected historical failures (hence the 95.97% and 88.73% no-reminder baselines), and Table 1 does not report the effect of routing on harmful requests that were already refused. The exact reminder texts are not provided. These omissions limit both the interpretation of Sentinel as a diagnostic and the assessment of its practical benefit.
Questions for authors
  1. Is the attention/MLP asymmetry specific to safety? Does the asymmetry in probe AUC, separability increments and Δ\DeltaRWS persist for a non-safety prompt attribute and a non-safety first-token contrast measured at the same position? Do the early- and mid-layer RWS conclusions hold under a tuned lens?
  2. Does Sentinel generalise beyond the attack templates it has seen? How does routing perform when one attack family is held out entirely, and how does it compare with the same classifier trained on generic last-token residual activations?
  3. How specific and stable is the restoration effect? Given that randomly chosen post-P MLP layers already block a large part of the upstream effect, what evidence shows that the selected D layers are special? How sensitive are the intervention and routing results to the P/D windows, given the different windows selected for Llama-3.1-8B with 100 and 50 calibration pairs?
  4. What is deployed, and what does it cost on already-correct cases? Please provide the exact texts of SendS_{\text{end}}, HendH_{\text{end}} and BendB_{\text{end}}, and report the results of every method in Table 1 on raw harmful requests that were refused without a reminder.
Reviewer_WMnQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies how harmfulness recognition translates into refusal in LLMs. It finds that harmfulness information is more readily decoded from attention updates, while MLP updates contribute more strongly to refusal evidence. Building on these findings, Sentinel combines both signals to select reminders that reduce jailbreaks and over-refusal.

Critical strengths
  1. The distinction between recognizing harmfulness and writing refusal evidence offers a novel perspective on LLM safety.
  2. The experiments cover 18 Base–Instruct pairs across three model families and examine generation dynamics in six reasoning models, providing broad empirical support.
Critical weaknesses
  1. Sentinel lacks a key controlled baseline: a router trained on ordinary residual-stream representations using the same classifier and training budget. Without this comparison, it remains unclear whether the gains specifically arise from the proposed mechanistic features.
  2. The measurements do not fully establish the proposed mechanism. Linear decodability does not demonstrate that the model actually uses harmfulness information.
Questions for authors
  1. Would Sentinel’s P/D features still outperform ordinary residual-stream representations when using the same classifier and training budget? This comparison would help establish the practical contribution of the proposed mechanistic decomposition.
Reviewer_oScxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how a transformer turns harmfulness recognition into refusal, using residual-stream module updates. Recognition is the AUC of a linear probe on the attention or MLP update. Refusal writing is the change in a logit-lens refusal-vs-compliance readout (RWS) across each update. Across 18 Base–Instruct pairs, attention updates are more decodable for harmfulness, especially in early layers, while MLP updates dominate harm-selective refusal writing. Post-training raises MLP writing in all 18 pairs without systematically improving decodability, and positive writing shifts toward late layers. In six reasoning models, harmfulness stays decodable while strong writing appears at answer onset. Appendix B shows that restoring downstream MLP updates removes part of the effect of upstream attention patches. The authors then build Sentinel, a router that uses an attention-derived score P and an MLP-derived score D to choose a reminder. They report lower average error than fixed reminders, Llama Guard 3 and single-signal routing across eight models.

Critical strengths

The measurement framework (a probe on the module update vs. a change in a readout) is clear and applied with unusual breadth: 18 Base–Instruct pairs, reasoning models, and several controls (matched-control vocabularies, alternative readouts, a matched-input Base–Instruct control, random-layer restoration). The picture of attention exposing harmfulness and late MLPs writing refusal is informative and fairly consistent.

Critical weaknesses
  1. The “pathway” claim seems to be weaker than the abstract suggests. Restoring selected downstream MLP updates blocks part of the attention effect, but random post-P MLP sets already block 30–84% (Table 3). Since late MLPs write the final readout at this position, this shows a bottleneck, not a specific attention→MLP link. The same-layer MLP response to attention patching is small (Fig. 6C), and generation-level effects are clear in only some configurations (Fig. 10; none for Qwen3.5-9B). All of this evidence sits in the appendix. Several measurements are confounded. Base is read at the final input token without a chat template and Instruct at the answer boundary. The matched control (App. D) shows the late-quarter share of MLP writing rising in only 2 of 6 pairs, contradicting the abstract’s “more concentrated in deeper layers”. Decodability is never compared under matched conditions. The RES vocabulary is made of first-answer tokens, so near-zero writing at mid-reasoning positions may hold by construction. Closely related work reports upstream harmful-detection heads, refusal heads and MLP safety neurons and is not discussed.

  2. The Sentinel evaluation does not show that P and D matter. The router is a learned classifier on multi-layer features. The paper’s own 2-scalar P/D control reaches 39.34 AVG, essentially the best fixed reminder (39.53), and the 30.34 headline comes from the layerwise router. There is no baseline router with generic features of matched size, and no text-only baseline. Cohorts are defined by the no-reminder outcome and come from different sources (AdvBench-derived attacks vs. XSTest/EvoRefuse vs. AlpacaEval), so the router may learn source or style. Grouped cross-validation by request does not prevent this. There is no held-out attack family or benchmark and no intervals. Conditional routing from internal signals exists (SafeSwitch) but is not cited or compared.

Questions for authors
  1. Can you report Sentinel against a router trained on generic features of matched dimension and a text-only classifier, with a held-out attack family, another over-refusal benchmark and intervals?

  2. What does restoring the selected D layers show beyond late MLPs writing the final readout? Can you add upstream-specificity controls (patching non-P attention layers) and path-patch MLP inputs rather than restoring outputs?

  3. Can you compare Base and Instruct probe AUC and late-layer share under the App. D matched template and position for all 18 pairs? How do you reconcile “more concentrated in deeper layers” with 2 of 6?

  4. Positions mid-reasoning do not predict an answer-start token. How do you rule out that “writing concentrates at answer onset” is by construction, and that decodability there reflects the reasoning tokens themselves?

5/ Probes are trained on 70 and tested on 30 JailbreakBench pairs. What are the intervals on the attention–MLP AUC gap, and does it hold on a lexically controlled set (e.g., XSTest-safe)?

  1. How well does the GPT judge agree with humans or a second judge on the ASR and over-refusal labels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.