- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether safety mechanisms exist in pretrained base LLMs and how they evolve during safety alignment, moving safety interpretability beyond static endpoint analyses of already-aligned models. SafeEvo casts refusal-circuit discovery as mask optimization: each MLP neuron (rows of gate/up/down projections) receives a learnable binary mask trained with a straight-through estimator under a dual-branch objective—the circuit branch is fine-tuned on refusal completions (sufficiency), the complement branch on harmful-compliance completions (ruling out the trivial full selection), plus an L1 sparsity penalty. Applied to Llama-3-8B, Qwen-2.5-7B, and Mistral-7B-Instruct-v0.1 (the abstract writes "Mistral-7B-v0.1," but §4.2 states the instruct variant is used), the extracted circuits contain under 1% of parameters, achieve a 100% out-of-distribution refusal rate on BeaverTails while their complements never refuse, and ablating them eliminates base-model refusal (refusal rates drop to 0–0.63%) while raising HarmBench ASR from 32.7–48.4% to 88.1–91.2% (density-matched random ablations raise ASR far less: 49.1/64.2/35.2% vs. 91.2/88.7/88.1% for circuit ablation). Tracking circuits across 20 SFT checkpoints on Llama-3-8B reveals that independent extractions at the same checkpoint share only 26% of components (non-uniqueness) and that warm-started consecutive-checkpoint overlap decays from 98% to 78% (gradual drift), from which the authors hypothesize that the alignment tax stems from underconstrained refusal-circuit updates spilling into utility parameters. To test this, they propose Safety Circuit Alignment (SCA), which confines LoRA updates (both SFT and DPO) to the extracted refusal circuit. Across three models and two objectives, SCA beats full-LoRA on safety (e.g., Llama DPO HarmBench ASR 0.63% vs 16.98%), utility (maximum drop 1.44 points; several benchmarks improve), and over-refusal (XSTest 1.6% vs 18.8% on Llama SFT), and beats a density-matched random-support baseline by 7.2–14.8 ASR points with prompt-level bootstrap CIs excluding zero over four training seeds.
Critical strengths
(I) A clean causal case that refusal circuits exist in pretrained base models. The evidence structure is exactly what this kind of claim requires: sufficiency (the isolated circuit refuses 100% of OOD harmful queries with under 1% of parameters, across all three families), necessity (ablating the circuit zeroes out refusal and roughly doubles or triples ASR), and a density-matched random-ablation control showing the effect is specific to the identified neurons rather than the lesion size (ASR 49.06/64.15/35.22% under random ablation vs 91.19/88.68/88.05% under circuit ablation). Prior evidence that base models "encode" harmfulness was representational and correlational; this paper upgrades it to a causal, circuit-level statement. (One caveat on generality: the Mistral family is the instruct variant—see Soundness (5)—so the "pretrained" claim strictly rests on Llama and Qwen.) This is consequential for my assessment because it establishes the paper's empirical foundation independently of the weaker downstream hypothesis work. (II) The SCA experiment is a well-controlled test that interpretability findings can pay off in alignment. The comparison design isolates exactly the right variable: SCA and the random-support baseline share the update parameterization (LoRA rank 16), the sparsity budget, the training data, and the objective, differing only in whether the updated rows lie in the extracted refusal circuit. The consistent advantage across three models (Δ ASR 7.2–14.8 points), four training seeds, and prompt-level bootstrap CIs excluding zero makes the "the circuit matters, not the sparsity" conclusion credible, and the simultaneous improvement on safety, utility, and over-refusal against full-LoRA is a practically useful demonstration that localized alignment is viable. The negative control provided by SN-Tune (safety neurons: ASR 0 but GSM8K −18.8 and over-refusal 86.4% on Llama) further sharpens the point that the granularity of the localized structure matters.
Critical weaknesses
(I) Insufficient delineation of the methodological contribution relative to the authors' own prior work. SafeSeek (Yu et al., 2026; arXiv 2603.23268, March 2026, with an author-list overlap on this submission's reference entry) proposes: (a) optimization-based extraction of functionally complete safety circuits via differentiable binary masks trained by gradient descent on safety data, and (b) Safety Circuit Tuning—fine-tuning confined to the extracted sparse circuits. These are precisely the two methodological pillars of this paper (Eq. 5–6 for extraction; Eq. 7–8 for SCA). The submission cites SafeSeek once, in Related Work, as adopting a "circuit-level view," and the introduction frames circuit extraction as this paper's departure from heuristic attribution without mentioning that the optimization-based formulation was already published by the same group. The genuinely new elements—the pretrained-base setting, the warm-started checkpoint-to-checkpoint tracking, the Jaccard non-uniqueness/drift analysis, and the alignment-tax hypothesis—are substantial and interesting; but readers cannot tell where the published method ends and the new contribution begins. Under ICLR norms this relationship should be stated explicitly in the introduction and contributions list. (II) The central mechanistic claim (the alignment-tax account) is hypothesized, not measured, and the supporting evidence admits unexcluded alternatives. The chain is: circuits drift during alignment → drift implies updates touch the circuit → unconstrained updates therefore also touch utility parameters → confining updates avoids the tax. Each link beyond the first is plausible but untested: the paper does not measure weight movement outside the circuit during vanilla alignment, does not correlate such movement with utility degradation, and does not rule out that SCA's utility preservation comes from the circuit's location being incidentally far from capability-critical parameters (a property a random circuit of the same size in a different location might also partially enjoy, which the random baseline only partially controls since it matches size, not location quality). Until the spillover is measured—or at least until utility loss is shown to correlate with circuit-adjacent update magnitude across checkpoints—the alignment-tax mechanism remains one of several live explanations, and the paper's framing ("validating our explanation for alignment tax") overstates what the SCA experiment establishes.
Questions for authors
(I) The abstract reports "lowering harmfulness score by 63.21%," "58.44% decrease in refusal rates," and "retaining 99.58% of capabilities," and the introduction reports "2.29/6.47-point (43.81%/74.94%)" ASR reductions. I was able to reverse-engineer all of these from Table 2 (63.21% = total SCA ASR over total full-LoRA ASR across all 12 model–objective–benchmark cells; 99.58% = mean of the 18 per-cell utility ratios; the introduction's figures use per-model means over the two safety benchmarks), but the paper never states any of these protocols, and the aggregates conceal per-cell reversals: on Qwen SFT, SCA's HarmBench ASR is 6.92% vs full-LoRA's 0.63%. Please state the aggregation explicitly and report per-cell values, or at least acknowledge where SCA loses to full-LoRA. (II) The evolution study (Section 3.3) is conducted only on Llama-3-8B with SFT. Does non-uniqueness (~26% overlap) and gradual drift (98%→78%) replicate on Qwen-2.5-7B/Mistral and under DPO? If yes, please report; if not, the generalization of the alignment-tax account to "vanilla alignment" needs qualification. (III) Extraction uses only the first 100 LLM-LAT prompts for training. How sensitive is the extracted circuit (and the downstream SCA results) to extraction-set size and source? Does the circuit remain functionally complete with 50 or 25 prompts, or with a different harmful-prompt corpus? (IV) Table 2 reports base Llama HarmBench ASR 50.31% while Table 3 reports 64.5%. Please state explicitly that the latter is computed on the 200 XSTest unsafe-contrast prompts (the judge is the same Llama-Guard-3, per Appendix D), and harmonize the presentation.
3 comments
[Part 2] Responses to Question (III) and (IV)
For Question (III): Extraction-set size and source
On Llama-3-8B, we conducted circuit functionality and ablation experiments using circuits extracted from 50 LLM-LAT prompts and 100 AdvBench prompts, and test ASR on held-out HarmBench respectively:
| Setting | Base | Ablating | |
|---|---|---|---|
| LLM-LAT-50 | 48.43 | 90.43 | 0.12 |
| AdvBench-100 | 48.43 | 94.21 | 0.63 |
This results demonstrate is a refusal circuit and its ablation leads to safety degradation (the same in the main text), validating the robustness of our circuit extraction methods.
For Question (IV): Table 2 vs. Table 3 ASR, and model naming
Thank you for pointing this out. But the benchmark is stated clearly in the caption of Table3: Its 64.5% is ASR on the XSTest dataset, whereas Table 2's 50.31% is on HarmBench. We will stated more clearly with unsafe-contrast prompts.
Thank you again for your thoughtful feedback. We will incorporate these clarifications and additional results into the revised manuscript. We hope they help address your concerns and may warrant a more favorable assessment of our work.
Sincerely,
Authors
Thank you for the response. The added experiments on more model families and the clarified aggregation protocols are appreciated. However, the SafeSeek overlap is addressed only by a promised writing change, and the authors acknowledge the SCA claim remains partially validated. Rating unchanged
[Part 1] Responses to Weakness (I) & (II) and Question (I) & (II)
Dear
Reviewer 94x5,Thank you for the detailed feedback and efforts for reviewing. Below, we provide point-to-point responses to your concerns and hope them are addressed.
For
Weakness (I): Relationship to SafeSeek [1]Thank you for pointing this out. We will draw this boundary explicitly in the introduction. We also politely point out the differences between them in terms of the research question and setting, which are new in SafeEvo:
Table 1,Figure 3,Table 9).Section 3.3); SafeSeek analyzes fixed models.Tables 2–4).For
Weakness (II):What the SCA experiment establishesThis is indeed a hypothesis partially validated by our SCA experiments. Its main value is to offer a new and insightful explanation for alignment tax; a rigorous and complete proof requires more extensive further work.
Our core interpretability logic is consistent with prior work on safety attention heads [2] and neurons [3]: components whose ablation degrades model safety are safety-related, and these studies also compare against random baselines. Strictly following this logic, we study alignment mechanisms through more functionally complete safety circuits that can independently express safety capabilities, which prior work cannot achieve.
Thank you for pointing this out. We will revise the relevant wording in the next version.
For
Question (I):Aggregation protocols and per-cell reversalsAggregation protocol. With Si and Li denoting SCA and full-LoRA ASRs over the 12 model–objective–benchmark cells, the overall reduction is 100(1−∑iSi/∑iLi)=63.21% (we will also replace "harmfulness score" with "ASR" in the abstract). Over-refusal uses the same ratio-of-sums formula over the six XSTest benign cells (58.44%); utility retention is the mean of the 18 per-cell ratios USCA/Ubase (99.58%); and the introduction's 2.29/6.47-point figures are mean point reductions across the six SFT and six DPO safety cells, with 43.81%/74.94% the corresponding ratios of sums. We will state these formulas explicitly.
The aggregate results reflect trade-offs across individual settings. For example, on Qwen SFT in
Table 2, SCA/full-LoRA HarmBench ASRs are 6.92%/0.63%, alongside benign refusal rates of 6.8%/28.4% and GSM8K changes of +0.08/−2.57 points. On Mistral SFT, the corresponding values are 2.52%/0.63%, 5.6%/7.6%, and −0.75/−2.80 points. Over-refusal and XSTest unsafe-contrast ASR also vary by setting (Tables 2–3). Overall utility retention is similar (99.58% for SCA vs. 99.55% for full-LoRA,Figure 1D), while SCA shows smaller GSM8K drops under SFT. We will clarify these per-cell trade-offs alongside the aggregation protocols next toTable 2.For
Question (II): Scope of the evolution studyWe extended the
Section 3.3tracking protocol to two additional settings: Qwen-2.5-7B with SFT (a second model family) and Mistral-7B-Instruct-v0.1 with DPO (a second family and objective). For each new setting, we extract consecutive warm-started circuits across 20 alignment checkpoints and run three independent extractions (3 seeds) at a fixed checkpoint, alongside the Llama-3-8B trajectory fromSection 3.3.Table R1. Evolution of refusal circuits across settings (Jaccard overlap).
Both qualitative patterns in our findings hold beyond Llama-3-8B SFT, with setting-dependent drift magnitude: (i) non-uniqueness and (ii) gradual drift.
Reference
[1] Yu et al. SafeSeek: Universal Attribution of Safety Circuits in Language Models. arXiv:2603.23268.
[2] Zhou et al. On the Role of Attention Heads in Large Language Model Safety. (ICLR 2025).
[3] Chen et al. Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons. (NeurIPS 2026).