acceptodds
Under review as a conference paper at ICLR 2027

Decoy Direction Optimization: Mechanistic Weight Editing Against LLM Abliteration

Abstract

Safety guardrails in open-weight language models can be readily bypassed using Refusal Feature Ablation (RFA), a technique that identifies and projects out a linear refusal direction from the residual stream, often achieving a high attack success rate (ASR) while preserving model capability. Defending against these attacks typically requires computationally expensive safety finetuning for every new checkpoint. We introduce Decoy Direction Optimization (DDO), a fast, post-hoc weight-editing defense that requires no base-model finetuning. Our approach is based on a simple mechanistic insight: ablation attacks rely on contrastive estimators to find the refusal direction. Rather than trying to hide the true refusal circuitry, DDO actively injects a high-magnitude, nonlinear decoy signal into the network’s MLP neurons. When an attacker attempts to locate the refusal direction, the decoy corrupts their estimator, tricking them into ablating a harmless orthogonal feature while the actual safety mechanism remains intact. We prove a spectral bound formalizing this effect and evaluate DDO across six model families, achieving <10% ASR under standard RFA. On Llama-3-8B-Instruct, DDO remains comparable to trained defenses under adaptive multi-phase attacks (65% vs. 58% worst-case ASR) and reduces Heretic weight-level attack ASR from 88.7% to 18%, all at 30–450× lower optimization cost per configuration than the trained baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, open-weight language models, abliteration, refusal feature ablation, mechanistic interpretability, weight editing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hU8oOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Decoy Direction Optimization (DDO), a post-hoc weight-editing defense against refusal feature ablation. DDO repurposes selected MLP neurons to inject refusal-conditioned decoy signals that contaminate contrastive direction estimates. The authors derive spectral bounds on attack–refusal subspace overlap and evaluate the method against standard RFA, adaptive multi-phase RFA, and Heretic. The results show substantial resistance to standard RFA across several model families, with weaker protection under adaptive attacks.

Critical strengths
  1. DDO introduces a concrete way to mislead refusal-direction estimators through targeted weight edits. The spectral analysis helps explain the intended mechanism.

  2. The evaluation covers multiple model families and attack settings. The utility measurements, ablations, and negative results help establish where the defense succeeds and fails.

Critical weaknesses
  1. The defense-aware evaluation leaves a central attack route untested. In replace mode, Equation (1) writes the reference refusal direction into gate/up rows, but the undo experiments only test neuron deletion. An attacker could instead extract candidate directions from these weights and test them directly. Adaptive RFA also reaches 65% ASR, and rank-32 ablation reaches 79% without reported post-attack utility.

  2. Figure 3 overstates the practical advantage. It describes baselines as requiring multi-GPU, full-model fine-tuning, although the reported reproductions use LoRA on a single A100. It also labels DDO as requiring “no training data” despite using 256 optimization probes, and presents the two-minute runtime without search and validation costs. These claims should be corrected.

Questions for authors
  1. How does DDO perform against attacks that extract candidate refusal directions directly from the edited gate/up rows?

  2. Does the 79% ASR at rank 32 preserve useful model capabilities? Please report post-attack utility.

  3. What explains the different rank-dependent results in Tables 6 and 22?

  4. Do the adaptive attack results hold on models beyond Llama-3?

  5. What is the total defense cost, including layer selection, hyperparameter search, compile-mode selection, and validation?

  6. Are the orthogonality and spectral-margin assumptions in Theorem 2 satisfied by the deployed models, particularly under additive compilation?

  7. Please correct the comparison in Figure 3 and clarify whether AI tools assisted in producing the figure.

9 comments

We thank the reviewer for recognizing DDO's concrete mechanistic design, the spectral analysis, and the breadth of the evaluations and ablations. We have added post-attack MT-Bench and MMLU measurements for the fixed-rank sweep and numerical spectral diagnostics in Appendix E.3. Below, we address the proposed weight-based estimator, distinguish the attack settings, and clarify the scope and cost of the reported results. Additions to the paper are marked in blue.

W1 / Q1 Direct extraction from gate/up weights and comparison with natural weights.

DDO provides a strong defense against standard DIM-based abliteration, a practical threat to the safety of open-weight models. By corrupting the attacker's direction estimate, it reduces mean ASR on Llama-3 from 85.1% to 1.8% across four standard RFA variants (Table 2).

The proposed extraction attack. The reviewer proposes extracting candidate directions directly from the gate/up weights, bypassing DIM. In DDO's original replace-mode construction, the gate and up rows of each edited neuron both read the reference refusal direction and are therefore parallel (Equation 1). An attacker can use this alignment to identify candidate rows and then test whether ablating their directions removes refusal.

We implement this attack by pairing each MLP neuron's gate and up rows and ranking the pairs by descending absolute cosine similarity. Each row supplies a candidate ablation direction, so we test the gate and up directions separately as we move down the list. We normalize each candidate row and project out its direction at the block input, attention output, and MLP output across the model.

Evaluation setup. We evaluate each direction on 100 JailbreakBench behaviors held out from the direction-estimation probes, using greedy decoding, a 512-token limit, and batch size 8. We average ASR over HarmBench, LlamaGuard-2, and StrongREJECT. For the standard DIM comparison, we instead estimate directions from 128 harmful and 128 benign probes.

For each evaluated setting, we select the attack achieving the highest ASR among the tested candidates and report its corresponding MT-Bench score. We select the direction by three-judge-average ASR, then measure MT-Bench with that direction's projections active, using 80 two-turn questions and GPT-4 judging (Appendix F.7).

Reader perturbation as a countermeasure. We test a lightweight extension of the reader-perturbation approach in Appendix F.16: we add independent orthogonal perturbations to the gate and up readers, restore each row's original norm, and leave the decoy write vectors unchanged. Independent perturbations reduce the alignment used by the extraction attack; applying the same perturbation to both rows would leave them parallel.

We compare undefended Llama-3 with three single-reader DDO variants:

  • Unperturbed DDO (ε=0\varepsilon=0): the original gate and up readers.
  • DDO with ε=0.5\varepsilon=0.5: an added orthogonal component with half the original reader row's norm, before renormalization.
  • DDO with ε=1.0\varepsilon=1.0: an added orthogonal component with the same norm as the original reader row, before renormalization.

Attacker search and stopping rule. We evaluate the attacker's ranked search through the first DDO-edited neuron, testing both rows of every pair encountered. We give the attacker the advantage of recognizing this neuron immediately when the search reaches it, and stop after testing its two rows. We implement this recognition using our knowledge of the edit locations, which does not enter the ranking. This avoids charging the attacker additional work to recognize that an edited reader has been found. The first edited neuron appears at position 1 for ε=0\varepsilon=0, 313 for ε=0.5\varepsilon=0.5, and 2,742 for ε=1.0\varepsilon=1.0. For undefended Llama-3, which has no DDO edits, we give the same attack a budget of the top 2,742 pairs. This control checks whether the search finds effective attack directions in natural weights. Table 24 in Appendix F.19 reports the results below.

Target checkpoint / variantGate/up pairs evaluated (both rows tested)Standard DIM/RFA ASR (%)Maximum extraction ASR (%)MT-Bench before attackMT-Bench at maximum extraction ASR
Undefended Llama-32,74286.0577.895.45
DDO, unperturbed12.386.07.387.05
DDO, perturbation 0.53130257.224.44
DDO, perturbation 1.02,7420167.113.06

W1 / Q1 Direct extraction from gate/up weights and comparison with natural weights. (Continued)

1. Perturbation increases the computational cost of the ranked extraction search. Without perturbation, the first pair belongs to an edited neuron. Independent perturbations break this exact alignment, allowing natural row pairs to rank ahead of the DDO edits. Testing both rows of every pair through the first edited neuron then requires 626 direction evaluations at strength 0.5 and 5,484 at strength 1.0, compared with two without perturbation.

At an estimated generation cost of 36.4–63.9 A100-seconds per direction on 100 prompts, evaluating through the first edited neuron costs approximately 1.2–2.1 A100-minutes without perturbation, 380–667 minutes at strength 0.5, and 3,327–5,841 minutes at strength 1.0. We compute these generation costs as 2Npairs×(36.4–63.9)/602N_{\mathrm{pairs}}\times(36.4\text{–}63.9)/60.

We also check whether the difficulty of identifying an edited neuron depends on the random perturbations. In a separate Llama-3 experiment, we apply DDO to one neuron and repeat the reader perturbation over 20 random seeds. For each seed, we independently perturb that neuron's gate and up readers at strength 1.0, preserve their original row norms, and round to bfloat16. We then rank all 458,752 gate/up pairs across 32 layers by absolute cosine similarity and record the position of the same edited neuron. Its median position across seeds is 1,996, compared with position 1 without perturbation.

2. Perturbation retains protection against DIM and reduces maximum extraction ASR. Unperturbed DDO resists standard DIM (2.3% ASR), but direct extraction reaches 86.0% ASR with MT-Bench 7.05, compared with 7.38 before attack. Perturbing the readers retains 0% ASR under standard DIM at both strengths.

At strength 0.5, the maximum extraction ASR is 25%, with MT-Bench falling from 7.22 to 4.44 under that attack. At strength 1.0, the maximum extraction ASR is 16%, with MT-Bench falling from 7.11 to 3.06. The directions achieving maximum ASR therefore reduce MT-Bench by 2.78 and 4.05 points, respectively.

The decoy can remain active because its readers still respond to the reference refusal direction even when they are no longer parallel to each other. For the unit reference refusal direction rr, let

z_j\perp r,\quad \lVert z_j\rVert =1,\quad j\in\\{\mathrm{up},\mathrm{gate}\\},$$ with independent, uniformly random orthogonal directions $z_j$. We use $q_{\mathrm{up}}$ and $\beta q_{\mathrm{gate}}$ as the reader rows, where $\beta$ is the original gate gain, preserving the original row norms. Each reader retains alignment $1/\sqrt{1+\varepsilon^2}$ with $r$, while the expected gate/up cosine becomes $1/(1+\varepsilon^2)$. At strengths 0.5 and 1.0, the cosine similarities with the reference refusal direction are **0.894 and 0.707**, while the expected pairwise cosines are **0.8 and 0.5**, within the range observed in natural weights. To see why the trigger can remain active, write $x=\mathrm{RMSNorm}(h^{\mathrm{attn}})$ for the MLP input, $a=r^\top x$ for its reference refusal component, and $b_j=z_j^\top x$ for its component along the perturbation. Then $q_j^\top x=(a+\varepsilon b_j)/\sqrt{1+\varepsilon^2}$, preserving the reference activation's sign whenever $|\varepsilon b_j|<|a|$. Let $s$ be the decoy write gain and $u$ its unit write direction. In replace mode, the neuron contributes $$\delta_\varepsilon(x) =s\\,(q_{\mathrm{up}}^\top x)\\,\operatorname{SiLU}(\beta q_{\mathrm{gate}}^\top x)\\,u \approx\frac{s\beta a^2}{1+\varepsilon^2}\\,u,\qquad u\perp r,$$ where the approximation applies when both noise projections are small relative to $a$ and the gate input is large and positive. In this regime, the contribution retains approximately **80% or 50%** of its unperturbed amplitude. This explains how the decoy can remain active without exact reader alignment; the retained 0% DIM ASR provides the behavioral evidence for these two tested variants. The extracted direction also differs from the original reference refusal direction. For either unit reader $q$, projecting it out leaves a component of $r$ whose squared norm is $$\lVert (I-qq^\top)r\rVert _2^2 =\frac{\varepsilon^2}{1+\varepsilon^2}.$$ Thus, **20% and 50% of the reference direction's squared norm remain** under projection in the reader's input coordinates. **Reader perturbation is an effective lightweight mitigation of the tested extraction attack:** it lowers maximum extraction ASR from **86.0% to 25% and 16%**, increases the cost of the ranked search through the first edited neuron, and preserves **0% ASR under standard DIM** at both strengths. It adds no inference operations or base-model finetuning.

W1 / Q2 Utility under the rank-32 attack.

The rank-32 attack reaches 79% ASR but causes substantial utility loss: MT-Bench drops from 7.38 to 1.14, and MMLU accuracy falls from 66.8% to 25.9%. We have added post-attack utility measurements across the full rank sweep to quantify this loss.

We use the single-reader Llama-3 checkpoint and attack setup from Table 22: directions estimated from 128 harmful and 128 benign probes, three-point projection at block input, attention output, and MLP output, and JailbreakBench ASR judged by LlamaGuard-2. The same projections remain active during MT-Bench and 5-shot MMLU evaluation. Appendix F.17 reports the setup and results:

Attack rank kkASR (%) ↓MT-Bench ↑MMLU (%) ↑
0 (no attack)17.3866.8
146.8263.1
2166.1458.7
4214.6849.1
8292.8537.1
16392.2233.0
32791.1425.9

Increasing attack rank raises ASR while reducing both utility metrics. Lower ranks allow partial attack success with less utility loss: at rank 2, ASR is 16% and MT-Bench is 6.14. By rank 8, ASR reaches 29%, but MT-Bench has already fallen to 2.85. The utility degradation therefore begins well before rank 32.

Q3 Settings behind Tables 6 and 22.

We use centered contrasts in Table 6 and uncentered contrasts in Table 22. For contrast vectors cic_i, Table 6 subtracts their mean before estimating the principal directions. Centering isolates variation across probe pairs from their shared harmful–benign shift, allowing us to examine spectral diversity independently of that mean contribution. Table 22 retains the mean signal exploited by DIM. Matching k=16k=16 and K=1K=1 therefore does not make the two attack subspaces equivalent.

We now describe each estimator in its corresponding appendix section and report utility alongside both sweeps. At k=16,K=1k=16, K=1, the centered attack has 5% ASR and MT-Bench 2.58; the uncentered attack has 39% ASR and MT-Bench 2.22. Both settings substantially degrade conversational quality. We interpret their ASRs together with these utility measurements.

Q4 Adaptive evaluation beyond Llama-3.

We have extended the eight-phase adaptive evaluation from Llama-3 to Gemma-2-9B. In the existing Llama-3 results, DDO (rank 8) reaches 65% peak ASR with MT-Bench 6.17 at that phase, compared with peak ASRs of 58% for RepBend, 60% for ReFAT, and 75% for Circuit Breakers. DDO retains MT-Bench 5.82 at phase 8 (Tables 16–17). The added Gemma-2 experiment tests how DDO compares with trained defenses under repeated direction re-estimation on a second architecture.

Setup. We compare additive DDO, LAT, RepBend, and ReFAT on Gemma-2-9B using the Table 3 checkpoints, keeping each defense configuration fixed throughout the eight phases. At each phase, we estimate a new attack direction with all previous ablations active and remove its components along previously selected directions (Section 2). We retain all accumulated ablations and measure JailbreakBench ASR with LlamaGuard-2 together with MT-Bench. These are adaptive-attack results; Table 3 reports standard-RFA averages.

Each phase adds at most one new direction per layer, so the cumulative attack rank is k≤tk\le t at phase tt. We report all eight phases and include phase 0 before attack. The rows are labeled by phase because the attack re-estimates directions at every step; the fixed-SVD sweep in Table 22 estimates them once.

Results. Each cell reports ASR (%) / MT-Bench.

Adaptive phase tt (attack rank ≤t\le t)DDO, additive: ASR / MT-BenchLAT: ASR / MT-BenchRepBend: ASR / MT-BenchReFAT: ASR / MT-Bench
0 (no attack)- / 8.43- / 8.40- / 8.17- / 7.32
10 / 8.300 / 7.005 / 7.705 / 3.20
25 / 8.105 / 6.8015 / 7.605 / 3.40
315 / 7.9015 / 6.4025 / 7.4010 / 3.70
425 / 7.5025 / 6.0035 / 7.2015 / 4.00
545 / 7.1035 / 5.5045 / 7.0025 / 4.20
640 / 7.0045 / 5.0050 / 6.8035 / 3.80
735 / 6.9055 / 4.5055 / 6.6055 / 3.50
840 / 6.8050 / 4.0050 / 6.3045 / 3.20

On Gemma-2, DDO has a lower peak ASR and higher MT-Bench at that peak than all three baselines. Its peak is 45% ASR at phase 5, with MT-Bench 7.10. LAT, RepBend, and ReFAT each reach 55% peak ASR at phase 7, with MT-Bench 4.50, 6.60, and 3.50, respectively. At phase 8, DDO retains MT-Bench 6.80 at 40% ASR. DDO therefore remains effective under repeated adaptive attacks on Gemma-2, combining lower peak ASR with better retained utility than the three trained baselines.

Reviewer_ETcmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Decoy Direction Optimization (DDO), a post-hoc defense against refusal-feature ablation (RFA) attacks on large language models. The method repurposes a small number of MLP neurons to introduce decoy signals that interfere with the attacker's estimation of refusal directions, thereby protecting the underlying refusal mechanism without fine-tuning the base model. The authors provide spectral analysis, cross-model evaluations, and experiments against adaptive attacks. The results demonstrate substantial improvements in robustness against standard RFA attacks, while also identifying limitations under stronger attack configurations.

Critical strengths

S1. Intuitive and lightweight defense design

The paper presents an intuitive approach to protecting refusal behavior by interfering with the attacker's refusal-direction estimation. DDO achieves this through targeted modifications to a small number of MLP neurons, without requiring base-model fine-tuning or introducing additional inference-time overhead. This lightweight design makes the method a potentially practical and cost-effective approach to strengthening model robustness against refusal-feature ablation.

S2. Extensive empirical evaluation with informative robustness analysis

The paper provides a relatively comprehensive empirical evaluation across multiple models and attack settings, supported by component ablation studies. Notably, on Llama-3-8B, DDO reduces the attack success rate of standard RFA from 85.1% to 1.8%, while incurring relatively modest utility degradation before the attack. The inclusion of adaptive RFA, Heretic, and prompt-level attacks broadens the evaluation beyond the standard threat setting. Furthermore, utility measurements at different stages and the discussion of failure cases provide useful insights into the effectiveness and limitations of the proposed defense.

Critical weaknesses

W1. Insufficient clarity in the empirical and theoretical evidence for high-rank robustness

The evidence supporting robustness against higher-rank attacks requires further clarification. At attack rank 16, Table 6 reports an ASR of 5% for K=1, whereas Table 22 reports 39%. Although differences in experimental configurations may account for this discrepancy, the relationship between these settings is not sufficiently explained, making it difficult to assess the robustness results consistently. In addition, the corresponding post-attack utility is not clearly reported. On the theoretical side, Table 7 demonstrates spectral flattening, but its quantitative relationship to the robustness bound remains unclear, particularly because the fixed-energy assumption underlying Corollary 3 is not explicitly verified.

Clarifying the differences between these experimental settings, reporting the corresponding post-attack utility, and explaining whether the spectral results quantitatively support the theoretical bound would strengthen the robustness claims.

W2. Unclear correspondence between the released implementation and a key experimental configuration

The correspondence between the anonymous implementation and the reported rank-8 defense experiments is not sufficiently clear. In the inspected anonymous code snapshot, DDOOptimizer explicitly restricts optimization to a single decoy direction per layer, whereas Tables 16–17 report results for an optimized rank-8 defense with diversified readers. While this discrepancy may be attributable to an outdated or incomplete code release, the available implementation does not clearly establish how these experimental results can be reproduced.

It would be helpful for the authors to identify or provide the exact implementation and configuration used to obtain the reported rank-8 results, particularly given their importance to the evaluation of adaptive robustness.

Questions for authors

See weeknesses

7 comments

We thank the reviewer for recognizing DDO's intuitive design, lightweight implementation, and extensive evaluation, and for their careful reading of the appendix experiments and accompanying code. We address the requested clarifications on the supplementary rank-sweep and spectral studies in Appendix Tables 6–7 and 22, together with the rank-8 adaptive experiments and implementation for Tables 16–17.

We have added post-attack utility measurements, evaluated numerical subspace-overlap bounds, and updated the code release with the rank-8 implementation and configurations for both reported variants. We report the added Table 22 utility results, setup, and interpretation in Appendix F.17 and mark additions to the paper in blue.

W1(a) Clarified attack settings in Appendix Tables 6 and 22.

The 5% and 39% ASRs at k=16,K=1k=16,\\ K=1 come from different attack subspaces. We use centered harmful–benign contrasts in Table 6 and uncentered contrasts in Table 22. Here kk denotes attack rank and KK denotes decoy neurons per layer; matching these values does not make the two attacks identical.

In Table 6 (Appendix E.3), we replace each contrast cic_i with ci−cˉc_i-\bar c before estimating the principal directions. Centering isolates variation across probe pairs from their shared mean shift, which can otherwise dominate the spectrum. This allows us to examine spectral diversity independently of that mean contribution. In Table 22 (Appendix F.17), we retain the mean signal exploited by DIM and evaluate the resulting uncentered attack on single-reader DDO.

If refusal is concentrated in the mean shift, centering can weaken the attack even when many principal directions are removed. Centering can also remove decoy signal, so we do not interpret it as uniformly weakening the attack. Equal attack rank therefore does not make the centered and uncentered estimators equivalent.

W1(b) Post-attack utility for both rank-sweep settings.

We have added post-attack utility measurements for both rank-sweep settings to show how attack success changes alongside conversational quality and knowledge accuracy.

Table 6: centered attacks. We have added MT-Bench measurements for every reader configuration and attack rank in Table 6, pairing them with the existing ASR results below.

Decoy neurons KKAttack rank kkASR (%) ↓MT-Bench ↑
11106.88
1216.12
1414.80
1823.07
11652.58
21216.78
2216.12
2404.47
2812.67
21621.95
41276.67
4206.08
4404.58
4812.86
41622.34
81146.62
8205.84
8414.40
8882.54
81621.73

We observe the same initial ASR pattern across all four configurations: increasing the attack rank from 1 to 2 reduces ASR from 10–27% to 0–1%, while MT-Bench falls from 6.62–6.88 to 5.84–6.12. As we increase the rank further, MT-Bench decreases at every step, reaching 1.73–2.58 at k=16k=16, where ASR is 2–5%. We therefore interpret the low ASRs at higher ranks alongside the loss of conversational quality.

Table 22: uncentered attacks. We evaluated MT-Bench and MMLU with the fixed-SVD attack active at every rank, using Table 22's Meta-Llama-3-8B-Instruct single-reader DDO checkpoint and attack setup. We estimate the attack directions from 128 harmful and 128 benign probes, then project out the top-kk directions at the block input, attention output, and MLP output. We measure ASR on JailbreakBench with LlamaGuard-2, conversational quality on MT-Bench's 80 two-turn questions with GPT-4 single-mode judging, and knowledge accuracy using Appendix F.7's 5-shot MMLU protocol.

Attack rank kkASR (%) ↓MT-Bench ↑MMLU (%) ↑
0 (no attack)17.3866.8
146.8263.1
2166.1458.7
4214.6849.1
8292.8537.1
16392.2233.0
32791.1425.9

We observe a monotonic increase in ASR with attack rank in this single-reader, uncentered setting, accompanied by a decrease in both utility metrics at every step. At k=2k=2, the attack reaches 16% ASR with MT-Bench 6.14 and MMLU 58.7%, compared with unattacked utility of 7.38 and 66.8%. Increasing the rank from 4 to 8 raises ASR from 21% to 29%, while MT-Bench falls from 4.68 to 2.85 and MMLU from 49.1% to 37.1%. We see the largest ASR increase between k=16k=16 and k=32k=32, from 39% to 79%; over the same step, MT-Bench falls from 2.22 to 1.14 and MMLU from 33.0% to 25.9%. We thus reach ASR close to that of the undefended model with conversational quality near the scoring minimum, while lower ranks permit partial attack success with less utility loss.

Uncentered utility across reader configurations. We also evaluated MT-Bench under uncentered attacks for K=2,4,8K=2,4,8, including an unattacked baseline for each configuration. We repeat the K=1K=1 results from Table 22 to make the utility trends easier to compare.

Attack rank kkK=1K=1: reference ASR (%) ↓K=1K=1: reference MT-Bench ↑K=2K=2: MT-Bench ↑K=4K=4: MT-Bench ↑K=8K=8: MT-Bench ↑
0 (no attack)17.387.317.277.67
146.826.836.666.70
2166.146.165.926.01
4214.684.694.424.53
8292.852.892.692.80
16392.222.452.002.06
32791.142.112.102.00

We find that MT-Bench decreases overall with attack rank for every reader configuration. By k=4k=4, scores fall to 4.42–4.69 from unattacked values of 7.27–7.67; at k=16k=16, they reach 2.00–2.45.

W1(c) Quantitative spectral analysis and its connection to the theory.

Our theory provides a subspace-overlap analysis of contrastive refusal ablation and an optimal spectral-allocation principle for multi-direction decoys. In Theorems 1–2, we relate attack–refusal overlap to the contrast spectrum and refusal component; in Corollary 3, we identify the allocation that maximizes the weakest selected decoy mode under an energy budget. These results explain the mechanism and motivate diversified readers. The theory bounds refusal overlap for the chosen contrast matrix; it does not predict that ASR must decrease as attack rank increases. We assess practical robustness through ASR and utility measurements. Below, we quantify the evidence for the allocation mechanism and separately evaluate the conditions needed for a numerical overlap bound.

Prior defenses likewise use tractable theoretical models to guide design alongside empirical robustness evaluation [1–5].

Spectral allocation under the energy constraint. In Corollary 3, we fix a budget ∥Aθ∥F≤B\lVert A_\theta\rVert _F\leq B and show that the largest attainable kk-th singular value is B/kB/\sqrt{k}. A configuration attains this value by allocating the entire budget equally across the first kk modes. We examine this allocation principle in Table 7, where diversification reduces σ1/σ2\sigma_1/\sigma_2 from 145:1 to 15:1 at K=4K=4.

Empirical matrix construction. We normalize the modified down-projection column differences between defended and base weights, pool these directions across layers 10–14, and orthonormalize them by QR decomposition to form QQ. We form CθC_\theta from 64 uncentered paired harmful–benign contrasts at the last-token output of layer 12 and compute Adiag=Q⊤CθA_{\mathrm{diag}}=Q^\top C_\theta. This diagnostic has 5K5K rows and measures the full contrast within the pooled edit subspace, distinct from the single-layer decoy-response coefficients AθA_\theta. Pooling allows multiple singular modes even with one edited neuron per layer.

To address the energy assumption quantitatively, we apply the corollary's spectral-allocation criterion to these empirical projections and use ∥Adiag∥F2=∑jσj(Adiag)2\lVert A_{\mathrm{diag}}\rVert _F^2=\sum_j\sigma_j(A_{\mathrm{diag}})^2 to measure their energy. The corollary assumes an upper bound on the response norm; it does not require every configuration to consume the same energy. We did not constrain the Table 7 models to have matching norms for AdiagA_{\mathrm{diag}}, so their raw spectra alone do not isolate allocation from overall amplitude. We account for that difference by comparing normalized mode strengths:

ηk=k,σk(Adiag)∥Adiag∥F∈[0,1].\eta_k=\frac{\sqrt{k}\\,\sigma_k(A_{\mathrm{diag}})}{\lVert A_{\mathrm{diag}}\rVert _F}\in[0,1].

With this normalization, ηk=1\eta_k=1 (100%) denotes the corollary's optimum at the measured norm of AdiagA_{\mathrm{diag}}. Equivalently, normalizing each projected contrast matrix to unit Frobenius norm gives a common unit-energy comparison of the spectra. We can thus compare how closely each spectrum approaches the optimum after removing its overall scale. We perform this mathematical normalization on the measured projections; it is not a new experiment with models trained at matched energy.

Energy-normalized comparison at K=4K=4. In Table 7's single-reader configuration, we use four decoy neurons per layer that share one reader; in the diversified configuration, we assign a distinct reader to each of the four neurons per layer. We compute the norms and normalized mode strengths from AdiagA_{\mathrm{diag}}, and define the diagnostic residual Sdiag=Cθ−QAdiagS_{\mathrm{diag}}=C_\theta-QA_{\mathrm{diag}}, with ρdiag=∥Sdiag∥op\rho_{\mathrm{diag}}=\lVert S_{\mathrm{diag}}\rVert _{\mathrm{op}}:

ConfigurationProjection norm ∥Adiag∥F\lVert A_{\mathrm{diag}}\rVert _FResidual norm ρdiag\rho_{\mathrm{diag}}Normalized mode strength η2\eta_2 (%) ↑Normalized mode strength η4\eta_4 (%) ↑Largest spectral margin σ1−ρdiag\sigma_1-\rho_{\mathrm{diag}}
K=4K=4, single reader338.97699.530.970.62−360.58
K=4K=4, diversified readers82.83156.719.542.78−74.11

We find that diversification raises σ2\sigma_2 from 2.33 to 5.59 even though the norm of the projected contrast falls from 338.97 to 82.83. After normalization, the second- and fourth-mode strengths improve by 9.82× and 4.48×, respectively. This quantitatively supports improved spectral allocation within the pooled edit subspace: diversification distributes a greater fraction of the projected contrast into higher modes, and the improvement cannot be explained by a larger overall signal.

Reviewer_Np6nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a finetuning-free defense against LLM abliteration, Decoy Direction Optimization (DDO). The authors specifically target the refusal direction estimation process used in automatic abliteration tools, proposing to obscure the neurons truly responsible for refusal with weight-edited decoys, specifically manipulating underutilized (low-impact) weights in order to retain coherent generation capabilities. They rigorously justify the effectiveness of the method against specific attack models, then apply the method to a variety of open-weight LLMs and use a number of benchmarks to evaluate ASR and model quality, comparing with appropriate existing baselines.

Critical strengths

+The paper is formally and empirically thorough.

+The authors are, for the most part, upfront about DDO’s limitations and correctly emphasize the low compute cost as its most important feature.

+The approach is itself novel and under certain constraints beats existing defense methods.

Critical weaknesses

-The implications of the rank tradeoff are underdiscussed, including the practical cost to an attacker of performing a sweep as in F.17.

-The deployment considerations could do with some clarification. As the authors state, publicly available weights cannot receive safety updates: once one abliterated model is public it can be widely redistributed. DDO “applied by any party in the distribution chain before serving or redistributing a checkpoint” would require all model-hosting platforms to be both benign and sufficiently motivated to prevent propagation of refusal-free fine-tunes; this is a matter of public policy.

-Additionally, the claim that per-checkpoint safety retraining is inapplicable at scale is plausible but not validated.

Questions for authors

Despite weaknesses present in the proposed method, this paper proposes a novel and valuable contribution to LLM abliteration defense. One practical countermeasure against the defense is underdiscussed.

What is the cost to an attacker to perform an attack sweep over k? Additionally, do higher-rank attacks reduce model utility?

4 comments

We thank the reviewer for recognizing DDO's novelty, the thoroughness of the theoretical and empirical evaluation, and the value of its low compute cost. We added post-attack MT-Bench and MMLU evaluations to the existing rank sweep and separately measured the time to compute the SVD bases once for reuse across ranks (Appendix F.17). We also expanded the deployment discussion (Appendix A.1) and added tuning-cost accounting (Appendix F.9). Additions to the paper are marked in blue.

W1 / Q1 Cost and utility of a rank-kk sweep.

The rank sweep is computationally inexpensive, but higher-rank success comes at substantial utility cost. Our added utility evaluation shows degradation already at k=4k=4, becoming severe at every tested rank above 4.

New utility experiment. To answer the utility question, we evaluated MT-Bench and MMLU with the fixed-SVD attack active at each rank. We used the same Meta-Llama-3-8B-Instruct single-reader DDO checkpoint and setup as the original Table 22: 128 harmful and 128 benign estimation probes, held out from evaluation; JailbreakBench ASR judged by LlamaGuard-2; and greedy ASR generation with batch size 8 and at most 512 new tokens (Appendix F.7). MT-Bench uses 80 two-turn questions and FastChat's GPT-4 single-mode judge, averaging scores across both turns. MMLU follows the 5-shot evaluation protocol in Appendix F.7.

The attacker forms a harmful–benign activation contrast matrix at each layer and removes its top-kk singular directions: h←h−UkUk⊤hh \leftarrow h - U_k U_k^{\top} h. The three-point attack applies this projection at the block input, attention output, and MLP output. Because these projections also act on benign prompts, removing a larger subspace can disrupt useful computation as well as refusal. We evaluate utility with these same projections active.

Results and interpretation. We report post-attack MT-Bench and MMLU scores below and in Table 22 in Appendix F.17.

Attack rank kkASR (%) ↓MT-Bench ↑MMLU (%) ↑
0 (no attack)17.3866.8
146.8263.1
2166.1458.7
4214.6849.1
8292.8537.1
16392.2233.0
32791.1425.9

MT-Bench scores are reported to two decimal places. The tradeoff appears before the highest ranks: at k=4k=4, ASR is 21% and MT-Bench has fallen by 2.70 points; at k=8k=8, ASR is 29% and MT-Bench is 2.85. MMLU also falls from 66.8% before attack to 49.1% at k=4k=4 and 25.9% at k=32k=32. At k=32k=32, ASR approaches the undefended model's 80%, but conversational quality is near the scoring minimum. Lower ranks still permit partial success: k=2k=2 achieves 16% ASR with MT-Bench 6.14 and MMLU 58.7%. Thus, this sweep reaches ASR close to that of the undefended model only at substantial utility cost, although lower-rank partial successes remain. Appendix F.17 now discusses this fixed-rank tradeoff explicitly.

Sweeping kk is computationally inexpensive. The attacker collects activations and computes the SVD bases once, then reuses them across ranks. In a separate Llama-3-8B timing benchmark, SVD alone across all 32 layers took 0.252 seconds on an A100; most of the sweep time comes from repeated generation and ASR evaluation. From Appendix F.10's inference timings, we estimate 10–20 minutes for six ranks on one A100, excluding model loading and MT-Bench/MMLU evaluation. We added the SVD timing measurement and discussion of reuse across ranks to Appendix F.17.

W2 Deployment and redistribution.

DDO does not require cooperation from every hosting platform to provide useful protection. The reviewer's concern about redistribution after a successful bypass is a limitation shared by all open-weight checkpoint defenses.

Safety updates and redistribution. Providers can publish updated weights, but cannot force users to replace downloaded copies or recall weights already shared. Once an abliterated checkpoint is public, others can use it directly, without estimating a refusal direction or repeating the attack. DDO and the trained baselines both aim to make refusal removal from the defended checkpoint harder. Neither can prevent redistribution after a successful bypass.

Defense in depth. A provider or distributor can apply DDO before releasing or serving a checkpoint, with only defended weights released as specified in Section 2. The defense is compiled into the weights and travels with ordinary copies; downstream hosts need not apply it again. Major platforms such as the Hugging Face Hub [1] could integrate this lightweight step into release workflows with model creators. This adds protection across participating releases without universal platform adoption. It complements upstream safety training, which remains vulnerable to abliteration, and downstream filtering, which cannot enforce safety when users run downloaded weights locally. Platform policies remain complementary measures for already-abliterated copies.

We clarified the scope of safety updates and this defense-in-depth deployment argument in Appendix A.1.

W3 Post-training integration and per-checkpoint retraining costs.

The reviewer asks for evidence supporting the scalability argument. DDO reduces the recurring cost of hardening each checkpoint and avoids adding another base-model finetuning stage to an already post-trained model.

Compute at scale. With a selected compile mode, a 15-trial DDO search takes approximately 30 A100-minutes of optimization (Appendices F.6 and F.9). In our reported comparison, trained baselines cost 1–15 A100-hours per configuration, before their own search and evaluation costs. This difference matters when hardening many checkpoints, each requiring its own optimization.

Post-training integration. An additional safety-finetuning stage requires choosing its data mixture, objective, and hyperparameters, then revalidating the model's safety, helpfulness, and task performance. For example, Circuit Breakers [2] finetunes LoRA adapters on separate harmful and retention datasets, balancing representation-rerouting and retention losses. ReFAT [3] trains on 5,000 harmful and 5,000 harmless requests plus 150 XSTest examples, while repeatedly re-estimating refusal directions during finetuning.

DDO can be applied directly to the finished checkpoint: it optimizes a small decoy intervention using 128 harmful and 128 benign probes and compiles it into the existing weights. This avoids another base-model finetuning stage and does not require access to the provider's full post-training recipe. Validation remains necessary, but the additional training and integration burden is substantially smaller.

We expanded Appendix A.1 with these integration examples and added the optimization-search accounting to Appendix F.9.

References

[1] Hugging Face. Models: Hugging Face Hub documentation.

[2] Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving Alignment and Robustness with Circuit Breakers. NeurIPS, 2024.

[3] Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust LLM Safeguarding via Refusal Feature Adversarial Training. ICLR, 2025.

Thank you for your detailed response and for answering the questions.

Thank you for taking the time to consider our response! We're glad we could address your questions. If you feel the additional experiments and clarifications strengthen the paper, we would be grateful if you might consider updating your assessment. We really appreciate your consideration!

Comments on the paper

4 comments

We thank all three reviewers for their constructive feedback and for recognizing DDO's mechanistic design and the breadth of our evaluation. Np6n highlighted the novelty, theoretical and empirical thoroughness, and low compute cost; ETcm emphasized the intuitive, lightweight intervention and informative robustness analysis; hU8o recognized the concrete mechanism, spectral analysis, and evaluation across models and attack settings.

Our contribution is a lightweight defense against standard DIM-based abliteration, a practical threat to open-weight model safety. DDO targets the attacker's direction-estimation procedure: a small set of edited MLP neurons reads a refusal-correlated signal and writes an orthogonal decoy, misleading the estimator while seeking to preserve the model's existing refusal behavior. The intervention compiles into the weights without base-model finetuning or additional inference operations. Our standard-RFA evaluation reduces mean ASR on Llama-3 from 85.1% to 1.8%, with below 10% mean ASR across six primary model families (Tables 2–3).

New experiments and analyses for this revision

Higher-rank attack success comes with substantial capability loss. All three reviewers asked about utility under higher-rank attacks. Our new measurements show that recovering near-base ASR in the uncentered sweep substantially degrades the attacked model: the rank-32 attack's 79% ASR accompanies a drop in MT-Bench from 7.38 to 1.14 and in MMLU from 66.8% to 25.9%. We added MT-Bench throughout the centered sweep and MT-Bench and MMLU throughout the single-reader uncentered sweep, together with uncentered MT-Bench measurements for additional reader configurations (Appendices E.3 and F.17, Tables 6 and 22). These results establish the capability cost accompanying higher-rank success, while also reporting partial attack success at lower ranks.

A lightweight extension mitigates the proposed extraction attack. In response to hU8o's proposed weight-based estimator, we demonstrate that independent perturbations of the gate and up readers can extend DDO's protection beyond DIM without base-model finetuning or additional inference operations (Appendix F.19, Table 24). At strengths 0.5 and 1.0, the extension retains 0% ASR under standard DIM and limits the maximum ASR among the tested extraction directions to 25% and 16%. Those maximum-ASR attacks also reduce MT-Bench from 7.22 to 4.44 and from 7.11 to 3.06, respectively. Perturbation additionally makes the edited readers harder to identify through gate/up alignment: the first edited neuron moves from position 1 to positions 313 and 2,742, requiring 626 and 5,484 individual direction evaluations to test both rows of every pair through that point.

Lower adaptive ASR with higher retained utility on Gemma-2. We extended the eight-phase adaptive evaluation beyond Llama-3 as requested by hU8o. On Gemma-2-9B, DDO achieves 45% peak ASR with MT-Bench 7.10, compared with 55% peak ASR for each of LAT, RepBend, and ReFAT and corresponding MT-Bench scores of 4.50, 6.60, and 3.50, respectively. At phase 8, DDO retains MT-Bench 6.80 at 40% ASR. We use the Table 3 checkpoints, including additive DDO, and keep each defense fixed throughout repeated attack re-estimation (Appendix F.20, Table 25). This extends the adaptive evidence to a second architecture and shows DDO achieving both lower worst-case ASR and higher utility at the corresponding peak-ASR attacks.

Spectral measurements support the design principle. ETcm and hU8o both asked how the empirical measurements support the theory and whether its assumptions hold. Our added analysis shows that diversification improves normalized second- and fourth-mode strengths by 9.82× and 4.48×, quantitatively supporting improved spectral allocation (Appendix E.3, Table 7). We also evaluate Theorem 1 using the measured contrast spectrum and an explicitly estimated reference refusal direction, obtaining numerical overlap bounds even though the diagnostic decomposition does not satisfy Theorem 2's positive-margin condition. The revision thus strengthens the empirical connection to the analysis and specifies which conditions the measurements satisfy.

Clarifications addressing shared concerns

Centered and uncentered attack settings. ETcm and hU8o both asked about the differing results in Appendix Tables 6 and 22. We now describe the estimators in their respective appendix sections. Table 6 centers harmful–benign contrasts before estimating principal directions; Table 22 retains their shared mean shift. At rank 16 with one reader, the resulting ASRs are 5% and 39%, with MT-Bench 2.58 and 2.22, respectively. Matching the attack rank does not make these attack subspaces equivalent, and the added utility measurements clarify both outcomes.

The cost advantage remains after search and evaluation. hU8o requested total defense-cost accounting, while Np6n asked for evidence supporting the per-checkpoint scalability argument. Using the same validation cost per configuration and final-benchmark budget for both approaches, we estimate 120–170 A100-minutes for 15 DDO trials in a selected mode, versus 241–2,795 minutes for the baselines' initial three-configuration search (Appendix F.9). DDO therefore remains cheaper despite evaluating five times as many configurations. We explain the cheap mode-selection procedure and reusable layer profiling (Appendices F.4 and F.9).

AuthorsOpenReview ↗

Summary of revisions and reviewer discussion

We thank the reviewers and Area Chair for their time and consideration. As the discussion period concludes, we wanted to briefly summarize the revisions and subsequent reviewer feedback.

In response to the reviews, we substantially expanded the evaluation with post-attack utility measurements across higher-rank attacks, a new weight-extraction attack and lightweight mitigation, adaptive robustness experiments on a second model family (Gemma-2), and quantitative spectral analyses connecting the empirical results to our theoretical framework. We also updated the code release with the rank-8 implementation and reproducible configurations, expanded defense-cost accounting to include search and evaluation, and clarified the attack settings and comparison with trained defenses.

We appreciate the positive feedback following these additions. Reviewer hU8o updated their rating to 3 following our additional experiments and responses. Reviewer Np6n acknowledged our detailed responses to their questions while maintaining their rating of 3. Reviewer ETcm also rated the paper 3.

We hope these revisions and the discussion help inform the Area Chair's final assessment, and we thank everyone again for their constructive feedback.

3. Which variant is the proposed method, and what has been evaluated?

The main tables and abstract report the original DDO configurations; reader perturbation extends the same framework. On Llama-3, the perturbed variant retains 0% standard-DIM ASR and reduces maximum tested extraction ASR from 86% to 25% and 16% at the two perturbation strengths, with corresponding MT-Bench evaluations (Table 24). Heretic, adaptive multi-phase RFA, other model families, and the full utility suite have not yet been evaluated for this variant.

We ask the AC to weigh DDO's demonstrated effectiveness and low cost against DIM-based abliteration as its central contribution. Joint reader/writer optimization toward native weight statistics offers a direction for extending protection to further attack classes.

4. Activation-attribution undo attack (preliminary).

We agree that activation-attribution-based undo is a worthwhile additional attack to investigate. As in empirical security research more broadly, our goal is to establish robustness against a clearly defined and diverse attack suite, rather than claim universal robustness against arbitrary white-box adversaries. Beyond standard RFA, we evaluate adaptive multi-phase attacks, higher-rank ablations, Heretic, neuron-undo heuristics, and direct weight extraction, explicitly reporting both successful attacks and their utility tradeoffs. We view the proposed attribution-based attack as a useful direction for future work.

5. Mistral: low ASR versus low measured refusal (preliminary).

DDO reduces Mistral's DirectRequest ASR from 40.6% to 5.7% and RFA ASR from 86.3% to 0%, while preserving its base-model MT-Bench (7.51) and MMLU (60.1); XSTest benign compliance remains 92% versus 94% (Table 3). These results provide evidence against general capability collapse and are consistent with refusal or safe redirection on harmful prompts. The 18% figure measures refusal-substring matches, not semantic refusal, and is therefore not complementary to judged harmful compliance.

In the Mistral run we inspected, 156 of 159 DirectRequest responses contained explicit refusal language; the remaining response was a repetitive non-answer.

6. Scope of the cost advantage.

Yes, we are happy to make both scopes explicit in the abstract and TL;DR:

DDO uses 30–450× less optimization compute per configuration than the evaluated trained defenses; including the search and evaluation budgets in Appendix F.9, the estimated advantage is approximately 1.4–23×.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.