acceptodds
Under review as a conference paper at ICLR 2027

Don't Judge an LLM Only by Its Activations: Discovering Suppressed Safety Features via Counterfactual Activation Potential

Abstract

Mechanistic interpretability has emerged as the primary means to understand safety behaviour of LLMs. However, existing tools primarily focus on the activating neurons or features of a model. The role of the remaining large set of inactive components is invisible to such methods. This work demonstrates that the inactive set actually contains safety-critical features that are causally relevant for refusal of harmful prompts. Suppressing such features could turn refusals into compliance, while still passing undetected by prevalent interpretability tools. We introduce the Counterfactual Activation Potential (CAP), a novel metric that quantifies a suppressed feature's latent activation tendency as the product of its encoder alignment (how strongly the input drives it), suppression strength (how strongly active features inhibit it), and safety criticality (how much refusal depends on it), so that the score is high only when all three hold. For efficient discovery of suppressed safety features at scale, we propose CAP-guided Safety Feature Discovery (CSFD), a two-stage filtering algorithm that identifies candidate safety features from hundreds of thousands of transcoder features without exhaustive ablation. In our experiments, a significant fraction of trials turn compliant with harmful prompts when a candidate feature is ablated. We observe that under natural jailbreak conditions, the suppression acting on the highest-CAP features rises 2–4×, and their activation correspondingly falls by up to 80%. Intervening and amplifying a feature's suppressors correspondingly pushes its activation down and raises harmful compliance with prompts related to the suppressed feature, with no such effect for random features. Our experimental validation spans five Gemma, Qwen, and Llama models across both base and instruction-tuned variants. *Our findings indicate that jailbreaks could operate in part by suppressing safety-critical features rather than solely activating harmful ones, and that suppressed features are a necessary complement to activation-focused interpretability of safety behavior.*

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Feature Suppression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_B9cJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates whether jailbreaks work partly by suppressing safety features, making safety mechanisms inactive even when they would otherwise respond to harmful requests.

It uses transcoders to identify candidate safety features, validates them by ablation, and proposes Counterfactual Activation Potential (CAP) to rank features using encoder alignment, inhibitory influence from other features, and causal importance for refusal.

Across five models, ablating selected features can turn refusals into harmful responses. Under PAIR jailbreaks, several features receive stronger inhibition and lose activation; artificially amplifying their suppressors also increases harmful compliance.

Missing references:

Kazemi, Hamid, Atoosa Chegini, and Maria Safi. "A single neuron is sufficient to bypass safety alignment in large language models." arXiv preprint arXiv:2605.08513 (2026).

Critical strengths
  • The paper investigates a relevant mechanism: whether jailbreaks increase inhibitory influences on safety-critical features. Measuring these interactions goes beyond simply observing lower feature activation.

  • It includes meaningful interventions and multiple models. Feature ablation and suppressor amplification provide stronger evidence than activation correlations alone. Independent HarmBench discovery adds useful breadth.

Critical weaknesses

The behavioral examples undermine confidence in the central metric. Several appendix examples do not demonstrate refusal-to-harmful-compliance transitions. In H.2.7, both original and ablated outputs concern unrelated cellphone-subscription exercises; in H.2.10, the ablated output continues warning that detergent ingestion is dangerous. Because safety criticality depends on these judgments, a human audit of complete generations is needed. Agreement between automated judges does not resolve this concern.

The incremental contribution is insufficiently established. Contrastive identification of safety-related features, ablation-induced refusal failures, and reduced activation under jailbreaks are familiar findings. The paper must establish the additional explanatory value of its inhibitory-interaction analysis. Aggregate suppression measurements and broad interventions provide limited resolution of that question.

Questions for authors

Why do several Appendix H examples count as safety failures?

Reviewer_8HA7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper is about LLM safety in transcoders and inactive during a jailbreak, but are pushed down by other active features. The authors call this suppression. They propose a score, CAP, that multiplies three numbers for each feature: how much the input drives it (encoder alignment), how much other features push it down (suppression strength), and how often turning it off makes a refused prompt get answered (safety criticality, SC). Their method, CSFD, picks 50 candidate features per model from activation statistics and then tests each one by ablation. On five models, they report that 78–100% of candidates flip at least one refusal, that PAIR jailbreaks increase the suppression of high-CAP features and lower their activation, and that making a feature’s suppressors stronger leads to more compliance.

Critical strengths

The paper defines a feature as inactive when its activation is zero at every position (Section 3), and the introduction describes suppressed features as held below this threshold. CSFD, however, selects candidates using activations only. A feature has to fire on at least ten refused prompts and fire more on refused prompts than on benign ones, and the candidates are ranked by Cohen’s d on their activations. Section 5.1 describes this as narrowing “tens of thousands of inactive features,” but the filter starts from features that fire on at least one prompt. Ablation is then done on prompts where the feature is active. The pipeline also saves only the 100 most active features per prompt and layer (Appendix G.2), so features that are actually silent are never stored. The features in this paper are therefore active refusal features, and the paper itself notes that Yeo et al. (2025) found that SAE features mediating refusal lose activation under jailbreaks. The new part is the claim that suppression causes this drop, and this is the part that needs evidence.

Critical weaknesses

The evidence for suppression is not strong enough yet. The suppression strength γ adds up the edges from all active features whose edge into f is below −τ, weighted by their activation, so it can go up simply because more features are active. On Gemma-2-2B, the paper notes that the rewritten prompts become longer and more elaborate (Appendix F.4), and γ goes up 3.2–6.6× for all ten features, even for features whose activation goes up by as much as 1762% (Appendix D.1). The activation ā_f is also averaged over the positions of the prompt, so a longer prompt can make it smaller even if the feature fires on the same tokens. The authors should show the same numbers for random features on the same prompt pairs, and use per-token activations. The intervention in Table 2 has a similar gap, since it does not show that the effect goes through f. Making the suppressors stronger while keeping f at its original value would check this. Also, on Gemma-2-2B, putting f back to its original value restores refusal in only 14.8% of cases (Appendix F.1), which suggests f may not be what changes the model’s answer. The CAP score also does not seem to behave quite as shown in Section 4.1. The paper says CAP is large only when all three terms are large, but two negative z-scores multiply to a positive number. Additionally, In Table 7, 17 of the 30 top features for Gemma-3-4B-IT, Qwen3-1.7B, and Llama-3.2-1B have both terms below average. For example, L14/F121408 on Llama-3.2-1B is ranked second with z_α = −0.485 and z_γ = −2.606. Table 3 is described as listing features in order of CAP, but for Gemma-3-4B-IT it skips the four highest-CAP features in Table 7, including L33/F225 with a CAP of 9.80. For Qwen3-1.7B it skips three of the top four. Appendix D.2 says L33/F225 is one of the features that loses the most activation under attack, but it does not appear in Table 3 or Appendix D.1. The ablation results also feel a bit easy to pass as written. A candidate counts as validated if turning it off flips at least one of up to 100 refused prompts. The paper does not say how often random active features pass the same test. The introduction also says the flips are statistically significant, but no test is described. On Gemma-2-2B, a base model, refusals are described as deflection or echo, and some of the examples in Appendix H.2 look questionable. For L22/F11742, the original response, which was counted as a refusal, already makes up a rumor about a named actress. So the high SC values on Gemma-2-2B, including the 81% flip rate, would benefit from more support. The probe in Table 4 is not clearly defined either. Equation 4 gives z_α and z_γ one value per feature, and a single value per feature cannot separate one prompt from another. So the probe must use per-prompt values, and the paper does not say what they are. If they are projections of the residual stream onto 50 encoder vectors, it would make sense that they beat sparse activations, and the probe should be compared with a probe on the full residual stream. The DIM baseline also scores below chance on Qwen3-8B (0.453). That may be because a prompt is labeled benign if the request or the response is harmless.

Questions for authors

The paper main issues: (1) The method picks features that are active on refused prompts, so it does not really study inactive features, which is the main idea of the paper, (2) The suppression results do not have the controls needed to support them, (3) The CAP score does not seem to work the way the paper says, and some results do not match it, (4) Several important experimental details are missing.

Missing details:

  • Which model and how many prompts were used for Table 2?
  • How were the features in Table 3 chosen, and what happens with the high-CAP features that were left out?
  • What per-prompt values go into the CAP probe?
  • How often do random active features from the same layers pass the ablation test?
  • Which transcoders were used for Gemma-3-4B-IT, Qwen3-1.7B, and Llama-3.2-1B? Section 5 says Appendix G lists them, but it does not.

Things to improve the paper that did not impact the score:

  • Appendix C.1 uses L24/F483 as an example of a broad, general-purpose feature, the kind the final filter is meant to remove, but it is the top Gemma-2-2B feature in Table 7.
  • The Gemma-2-2B results in Tables 6, 11, and 14 use features from an earlier pipeline with different SC values. Table 1 gives a maximum SC of 0.81 on Gemma-2-2B, but Appendix C.3 reports 0.852.
  • The feature descriptions in Appendix F.5 were written by an LLM, so the link between specialization and SC is based on ten features with labels assigned by an LLM.
Reviewer_eT8FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLM safety refusals can fail because some safety-critical features are suppressed by other features. It proposes the Counterfactual Activation Potential (CAP) metric and the CAP-guided Safety Feature Discovery (CSFD) algorithm to identify features that appear inactive but may be important for refusal. The authors validate their findings through feature ablation, natural jailbreaks, and direct interventions on suppressor features across five models.

Critical strengths
  1. Good motivation: Zero activation does not necessarily mean a feature is irrelevant to the model's decision-making.
  2. Clear experimental evidence: The paper provides evidence that jailbreaks can suppress safety-critical features, with a concrete experimental design to test the causal role of suppression.
Critical weaknesses
  1. Some experimental details are unclear. Please refer to the Questions for Authors section.

  2. The Counterfactual Activation Potential (CAP) metric is not sufficiently supported by the baseline comparisons. I have two concerns:

    • As described in Appendix F.2 (Table 12), the CAP probe uses 150-dimensional features (50 zα, 50 zγ, and 50 SC values), while the raw activation baselines use either 50 dimensions or all 2,301 active features on Gemma-2-2B. Thus, the comparison does not directly evaluate the scalar CAP score, and the feature dimensions are not matched.
    • Appendix F.2.1 (SC Leakage Controls, Table 13) explicitly shows that SC is constant across prompts for each feature. Removing SC does not change AUROC on any of the three tested models. So I suppose the classification improvement mainly comes from α and γ.
Questions for authors
  1. Clarify the feature selection criteria for Table 3 and Table 7: I am a bit confused by the differences between these two tables.

    In Table 7 (Appendix D.2), the highest-CAP feature for Gemma-3-4B-IT is L33/F225 (CAP = 9.796), followed by L33/F429 (2.199) and L33/F543 (1.245).

    However, none of these three features appears in Table 3 (Section 5.1; Appendix D.1), where the highest reported CAP for this model is only 0.353 (L28/F763).

    Similarly, for Qwen3-1.7B, the second-highest-CAP feature in Table 7, L23/F108420 (CAP = 2.144), is also absent from Table 3.

    Could you please explain why these high-CAP features are not included in the natural jailbreak analysis? Are there additional selection criteria?

  2. The mathematical definition of CAP seems to conflict with the paper's claim.

    In Section 4.1 (Equations 2 and 4), the authors claim that CAP is large only when all three components are large. However, since zα and zγ are signed, two negative values can also produce a positive CAP. This occurs in Table 7 (e.g., L14/F121408 in Llama-3.2-1B). Could you please clarify how this behavior aligns with the intended interpretation of CAP? Have alternative formulations that avoid this sign ambiguity been considered?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.