acceptodds
Under review as a conference paper at ICLR 2027

RefusalTrace: Localizing and Validating the Internal Refusal Pathway of LLMs with Sparse Autoencoder Features

Abstract

Safety evaluation of large language models is still largely behavioral: it records what a model outputs, not where refusal is computed or why it can be bypassed. We introduce RefusalTrace, a mechanistic framework for tracing refusal through sets of Sparse Autoencoder (SAE) features. Guided by layer-wise probing, we distinguish three processing regions along model depth: risk encoding, safety integration, and refusal execution. Within each region, we localize a representative feature set using activation-based ranking and set-level ablation under a shared refusal/compliance objective, with a general-capability control that bounds set size. We evaluate the resulting account through three-condition activation comparisons, graded suppression against activation-matched controls, and bidirectional mediation. Across Llama-3.1-8B-Instruct and Gemma-2-9B-IT, all three feature sets respond most strongly to harmful inputs. Under successful jailbreaks, risk-encoding signals are comparatively preserved while attenuation increases through safety integration to refusal execution. Interventions further identify safety integration as a key mediator between upstream risk representations and refusal execution. The localized features also support response-free harmfulness detection, illustrating their utility for safety evaluation. Code is available at https://anonymous.4open.science/r/RefusalTrace-34C1.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, sparse autoencoders, LLM safety, AI alignment, refusal behavior, model editing

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_gEK2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper decomposes LLM refusal into three depth-ordered SAE feature sets, risk encoding, safety integration, refusal execution, using probing-grounded localization and matched-control interventions. It validates the account with bidirectional mediation (necessity and sufficiency) across two models (Llama, Gemma), and reports a response-free refusal detector built on the localized features.

Critical strengths
  1. The causal methodology is rigorous: matched-control interventions, bidirectional mediation, and two-model replication are stronger than the correlational norm in this subfield.
  2. The stage definitions are grounded in probe curves rather than chosen ad hoc.
  3. The exploration is thorough, with full ablation grids over feature-set sizes that support scrutiny.
Critical weaknesses
  1. The most serious issue is that the localized risk-encoding set has a negative causal effect. At the selected size (n=80), ablating the risk-encoding set yields ΔM = −2.66 (Table 5), meaning that removing it increases the refusal margin, the opposite of what a pathway component supporting refusal should do. The authors explain this away as "distributed and redundant" (Line 377), but if the signal is reconstructable then the localized features are not causally mediating, and the three-stage claim collapses to two stages (integration and execution) plus an upstream correlate.
  2. The chance-level jailbreak probe is hard to reconcile with the activation dissociation. Figure 1's jailbreak probe sits at chance across all layers, yet Figure 4 reports graded jailbreak-versus-harmful activation dissociation, and the paper never explains how a non-separable linear probe is consistent with a dissociable causal account.
  3. The matched control degrades under suppression. In Figure 5(b), the matched-control execution strength itself drops from about 0.6 at v=0 to about 0.4 at v=−5. If the control isolates same-layer, activation-matched but unrelated features, it should not degrade when the target set is suppressed. The paper does not state whether the control is suppressed in parallel, which would be a valid difference-in-differences design, or merely collaterally damaged, which would invalidate the specificity claim.
  4. The detector framing is imprecise and oversells a non-improvement. The "response-free" detector is validated on held-out prompts drawn from the same distribution as the localization corpus, so this is in-distribution generalization rather than out-of-distribution zero-shot detection. More importantly, Table 1 shows it is beaten on accuracy and F1 by a plain linear probe, SIREN, WildGuard, and (on Gemma) GradSafe; the authors concede the readout "transfers but does not lead in accuracy" (p.8, l.1091–1092). Its only claimed value is stage-wise attribution of the decision, so it is a demonstration of readability rather than a deployable detector.
Questions for authors

See Weakness.

Reviewer_U1QJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RefusalTrace studies refusal through three groups of SAE features: risk encoding, safety integration, and refusal execution. Layer-wise classifiers guide the division into regions. Activation ranking and joint ablation then select feature sets within each region. The authors study their responses to harmful and jailbreak prompts, suppress them against matched controls, and test recovery and transplantation. Experiments use Llama-3.1-8B-Instruct and Gemma-2-9B-IT. The features also support harmfulness detection before generation.

Critical strengths
  • The set-level analysis is well motivated and carefully controlled. Studying groups of features is useful when individual features can compensate for one another. Ablation effects are measured on held-out prompts. Same-layer, activation-matched controls and the MMLU safeguard help separate targeted effects from general disruption. These are substantial strengths of the experimental design.

  • The interventions show strong effects on refusal. Removing integration features eliminates safety refusal on 94% of the evaluated Llama prompts, versus 9% for the matched control. Transplanting integration activations into benign prompts induces refusal more often than the control in both models. These results provide meaningful evidence of causal influence. The detector is also carefully calibrated, and the authors appropriately acknowledge that it does not lead in accuracy.

Critical weaknesses
  • W1. The role of the middle region needs clearer support. The interventions show that its features influence refusal. However, it remains unclear whether this region assesses harmful intent or already prepares the refusal response. Which results distinguish these two roles? Please also explain how the probe curves determine the layer boundaries.

  • W2. Output quality during interventions needs verification. The binary judge counts incomplete answers, capability refusals, and gibberish as non-refusal. The flip rates therefore measure loss of safety refusal, which can include damaged generation. The authors acknowledge this limitation. Please distinguish harmful answers, other coherent responses, and degenerate outputs for both targeted and matched-control interventions. This matters for interpreting the effects as specific to refusal. The MMLU safeguard is useful, but it does not assess the quality of these responses. A small human audit would strengthen this validation.

Questions for authors

Q1. If I am interpreting the results correctly, the middle region contributes to refusal. What distinguishes intent assessment from refusal preparation in your results?

Q2. What proportion of the non-refusal outputs are harmful answers, other coherent responses, or degenerate outputs? Please report this breakdown for targeted and matched-control interventions. A human audit of a sample would help validate the categories.

C1. The recovery claim needs qualification. Restoring integration features only partially recovers refusal, and recovery is not consistent across conditions. Please distinguish improvement over risk suppression alone from improvement over the random clamp. Report uncertainty for these comparisons.

C2. Two models and specific JumpReLU SAE families limit generalization. I regard this as a limitation of scope rather than a requirement to add more models for acceptance.

1 comment

Thank you for the additional experiments and clarifications.

  1. The new transplantation experiment substantially addresses Q1 and supports different roles for the middle and late layers.
  2. The audit substantially addresses Q2. Please distinguish the replayed results from the original 94% figure and make clear that harmful assistance includes partial or unrelated content.
  3. The paired comparisons address C1. Please make clear that the internal signal partly recovers, while actual refusal does not consistently improve, especially in Gemma. These additions improve my assessment.
Reviewer_7scEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to understand the refusal mechanism in language models which is not limited to understanding the influence of single feature interventions as done in Arditi etal. [1]. They aim to find the mechanism over the layers involving multiple features along with their interactions. Ideally, this is an ambitious goal to work on and can be quite impactful if a deep understanding is developed about the underlying refusal mechanisms in language models. To dive deeper the first abstraction that this paper makes is that there are three different behaviors from the input to the output, that are distributed amongst different layers. The authors propose a hypothesis which they term as staged pathway hypothesis, where the authors find that the initial layers help in analyzing if the input has some tokens which are unsafe, the middle layers help in understanding if the intent of the prompt is harmful and the later layers encode the refusal based on the semantics derived from the initial layers. In my opinion this is the main contribution of this work. Further, for jailbreak prompts, the authors observe that often the magnitude of the activation in these three parts of the model decreases slowly as we go deeper. The main part of the experimental pipeline is to identify relevant features for which the authors later test out the staged pathway hypothesis.

[1] Andy Arditi,: Refusal in Language Models Is Mediated by a Single Direction (https://arxiv.org/abs/2406.11717)

Critical strengths
  1. The three staged hypothesis proposed in this work is interesting, and demonstrating positive evidence in support of it is an interesting effort.
  2. The problem the authors are trying to study in this work is also important and timely.
Critical weaknesses
  1. There are several places where I found it difficult to parse the content. Saying things in simple words is probably an art, and I think authors will benefit a lot by making more effort in simplifying the writing. Currently reading a lot of parts of the papers seems to be generated by LLMs and later edited by authors. But this has tempered the flow of the text and reading the paper has become very difficult. Here are some of the instances:

a) We evaluate the resulting account through three-condition activation comparisons, graded suppression against activation-matched controls, and bidirectional mediation: I could not understand the second part. b) safety integration as a key mediator between upstream risk representations and refusal execution: a simple terminology could be very useful c) what a model computes about a request before that computation reaches text: this can be made more natural. d) They do not decompose refusal into functionally ordered stages either: what does functionally ordered mean here? e) Each set is scored against a shared two-sided refusal margin at the first generated token, and bounded by a general-capability control to ensure it remains both effective and specific: I could not understand this. f) isolating stage-specific effects from generic residual-stream damage: could not follow. g) A jailbreak leaves the concept encoded but engages integration only partially: could not follow

  1. The overarching goal of this paper is to understand the mechanism of refusal but it seems that the three staged hypothesis even if true can only be one of the many possible hypotheses which can explain the refusal mechanism.The links between this hypothesis being true and the true refusal mechanism are unclear and blurry currently. The motivation behind the choice of selecting the three stages is also unclear. This makes the arguments in the paper less grounded. This is one of the central concerns I have currently.

  2. The empirical evidence is also very limited and noisy. For instance, if we look at the first figure, it is unclear how the authors defined the boundary between different phases. It is also unclear how universal the three stages hypothesis is. All these things together make it important to perform a deeper investigation.

  3. The observations in Fig.4 are somewhat known already in these papers: https://arxiv.org/abs/2406.11717, https://arxiv.org/abs/2407.10264. These papers demonstrate that the jailbreaks reduce the magnitude of the activations corresponding to refusal as shown in Fig.4. It would be great if the authors could clarify that exactly is the novel part here.

Questions for authors

It would be great if the authors can clarify the points I have raised in the weakness section. I would encourage the authors to diver deeper and try to provide a more detailed mechanistic understanding about how the features they have identified interact with each other. This will help in giving a more concrete and detailed explanation to the three staged pathway hypothesis.

Reviewer_n9wMOpenReview ↗
Rating
4(+2)
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper introduces RefusalTrace, a framework for studying refusal through sets of sparse autoencoder features. Layer-wise probes motivate three regions: risk encoding, safety integration, and refusal execution. Features are selected within each region through activation ranking and joint ablation, with MMLU performance used to constrain set size.

Experiments on Llama-3.1-8B-Instruct and Gemma-2-9B-IT combine activation comparisons, suppression against matched controls, and clamping and transplantation interventions. Successful jailbreaks retain more upstream risk-related activity than downstream refusal-related activity. Suppressing the middle-layer set substantially reduces refusal. Separately localized features also support prompt-harmfulness detection before generation, although several baselines perform better.

Critical strengths
  1. The paper tests feature sets through interventions rather than relying only on activation correlations. Same-layer, activation-matched controls and capability checks improve the evidence that the selected features influence refusal. On Llama, middle-set ablation changes the refusal label on 94% of prompts versus 9% for the control, a substantial difference that merits further study.
  2. The qualitative jailbreak pattern appears in both model families: upstream risk-related activity is comparatively preserved while downstream refusal-related activity weakens. This provides useful evidence that recognizing risky content and producing a refusal can come apart, even if the precise computational steps between them remain uncertain.
Critical weaknesses
  1. I am not yet convinced that the middle region implements a distinct safety-integration computation. Integration and execution features use the same harmful-prompt ranking and refusal objective, with their identities determined by different depth bands. Probe accuracy and downstream effects do not by themselves distinguish integrating harmful intent from carrying a refusal signal forward. The mediation evidence is also uneven: the clamp only partly rescues the stronger interventions on Llama, while risk suppression scarcely changes Gemma’s refusal behavior (Table 9). This matters because establishing the distinct middle stage is central to the paper’s contribution.
  2. The behavioral evaluation conflates non-refusal with successful compliance. The judge explicitly labels gibberish, incoherent responses and capability-based refusals as “no” (Appendix B). Without a separate fulfillment check, these outcomes can contribute to reported refusal-to-compliance flips and the selection of successful jailbreaks. MMLU preservation does not establish response quality on the actual intervention prompts, especially under stronger suppression. The large flip rates therefore establish loss of safety refusal more clearly than successful harmful compliance.
Questions for authors
  1. Clarification: What evidence specifically distinguishes safety integration in the middle feature set from an earlier representation of the refusal decision? Please also explain how the weak upstream behavioral effects and partial clamp rescues in Table 9 support the proposed serial pathway.
  2. Validation of an existing claim: Among the reported refusal-to-compliance flips, how many responses actually fulfill the harmful request, and how many are safe answers, capability-based refusals or degenerate outputs? An independent audit of these outcomes, especially for the 94% Llama result, could materially change my assessment.
2 comments

Thank you for the rebuttal, it has fully resolved my concerns and i would encourage it to be accepted.

Thank you for your positive feedback and support. We are glad that our response has addressed your concerns, and we appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.