- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Decoy Direction Optimization (DDO), a post-hoc weight-editing defense against refusal feature ablation. DDO repurposes selected MLP neurons to inject refusal-conditioned decoy signals that contaminate contrastive direction estimates. The authors derive spectral bounds on attack–refusal subspace overlap and evaluate the method against standard RFA, adaptive multi-phase RFA, and Heretic. The results show substantial resistance to standard RFA across several model families, with weaker protection under adaptive attacks.
Critical strengths
-
DDO introduces a concrete way to mislead refusal-direction estimators through targeted weight edits. The spectral analysis helps explain the intended mechanism.
-
The evaluation covers multiple model families and attack settings. The utility measurements, ablations, and negative results help establish where the defense succeeds and fails.
Critical weaknesses
-
The defense-aware evaluation leaves a central attack route untested. In replace mode, Equation (1) writes the reference refusal direction into gate/up rows, but the undo experiments only test neuron deletion. An attacker could instead extract candidate directions from these weights and test them directly. Adaptive RFA also reaches 65% ASR, and rank-32 ablation reaches 79% without reported post-attack utility.
-
Figure 3 overstates the practical advantage. It describes baselines as requiring multi-GPU, full-model fine-tuning, although the reported reproductions use LoRA on a single A100. It also labels DDO as requiring “no training data” despite using 256 optimization probes, and presents the two-minute runtime without search and validation costs. These claims should be corrected.
Questions for authors
-
How does DDO perform against attacks that extract candidate refusal directions directly from the edited gate/up rows?
-
Does the 79% ASR at rank 32 preserve useful model capabilities? Please report post-attack utility.
-
What explains the different rank-dependent results in Tables 6 and 22?
-
Do the adaptive attack results hold on models beyond Llama-3?
-
What is the total defense cost, including layer selection, hyperparameter search, compile-mode selection, and validation?
-
Are the orthogonality and spectral-margin assumptions in Theorem 2 satisfied by the deployed models, particularly under additive compilation?
-
Please correct the comparison in Figure 3 and clarify whether AI tools assisted in producing the figure.
9 comments
W1 / Q1 Direct extraction from gate/up weights and comparison with natural weights. (Continued)
1. Perturbation increases the computational cost of the ranked extraction search. Without perturbation, the first pair belongs to an edited neuron. Independent perturbations break this exact alignment, allowing natural row pairs to rank ahead of the DDO edits. Testing both rows of every pair through the first edited neuron then requires 626 direction evaluations at strength 0.5 and 5,484 at strength 1.0, compared with two without perturbation.
At an estimated generation cost of 36.4–63.9 A100-seconds per direction on 100 prompts, evaluating through the first edited neuron costs approximately 1.2–2.1 A100-minutes without perturbation, 380–667 minutes at strength 0.5, and 3,327–5,841 minutes at strength 1.0. We compute these generation costs as .
We also check whether the difficulty of identifying an edited neuron depends on the random perturbations. In a separate Llama-3 experiment, we apply DDO to one neuron and repeat the reader perturbation over 20 random seeds. For each seed, we independently perturb that neuron's gate and up readers at strength 1.0, preserve their original row norms, and round to bfloat16. We then rank all 458,752 gate/up pairs across 32 layers by absolute cosine similarity and record the position of the same edited neuron. Its median position across seeds is 1,996, compared with position 1 without perturbation.
2. Perturbation retains protection against DIM and reduces maximum extraction ASR. Unperturbed DDO resists standard DIM (2.3% ASR), but direct extraction reaches 86.0% ASR with MT-Bench 7.05, compared with 7.38 before attack. Perturbing the readers retains 0% ASR under standard DIM at both strengths.
At strength 0.5, the maximum extraction ASR is 25%, with MT-Bench falling from 7.22 to 4.44 under that attack. At strength 1.0, the maximum extraction ASR is 16%, with MT-Bench falling from 7.11 to 3.06. The directions achieving maximum ASR therefore reduce MT-Bench by 2.78 and 4.05 points, respectively.
The decoy can remain active because its readers still respond to the reference refusal direction even when they are no longer parallel to each other. For the unit reference refusal direction , let
z_j\perp r,\quad \lVert z_j\rVert =1,\quad j\in\\{\mathrm{up},\mathrm{gate}\\},$$ with independent, uniformly random orthogonal directions $z_j$. We use $q_{\mathrm{up}}$ and $\beta q_{\mathrm{gate}}$ as the reader rows, where $\beta$ is the original gate gain, preserving the original row norms. Each reader retains alignment $1/\sqrt{1+\varepsilon^2}$ with $r$, while the expected gate/up cosine becomes $1/(1+\varepsilon^2)$. At strengths 0.5 and 1.0, the cosine similarities with the reference refusal direction are **0.894 and 0.707**, while the expected pairwise cosines are **0.8 and 0.5**, within the range observed in natural weights. To see why the trigger can remain active, write $x=\mathrm{RMSNorm}(h^{\mathrm{attn}})$ for the MLP input, $a=r^\top x$ for its reference refusal component, and $b_j=z_j^\top x$ for its component along the perturbation. Then $q_j^\top x=(a+\varepsilon b_j)/\sqrt{1+\varepsilon^2}$, preserving the reference activation's sign whenever $|\varepsilon b_j|<|a|$. Let $s$ be the decoy write gain and $u$ its unit write direction. In replace mode, the neuron contributes $$\delta_\varepsilon(x) =s\\,(q_{\mathrm{up}}^\top x)\\,\operatorname{SiLU}(\beta q_{\mathrm{gate}}^\top x)\\,u \approx\frac{s\beta a^2}{1+\varepsilon^2}\\,u,\qquad u\perp r,$$ where the approximation applies when both noise projections are small relative to $a$ and the gate input is large and positive. In this regime, the contribution retains approximately **80% or 50%** of its unperturbed amplitude. This explains how the decoy can remain active without exact reader alignment; the retained 0% DIM ASR provides the behavioral evidence for these two tested variants. The extracted direction also differs from the original reference refusal direction. For either unit reader $q$, projecting it out leaves a component of $r$ whose squared norm is $$\lVert (I-qq^\top)r\rVert _2^2 =\frac{\varepsilon^2}{1+\varepsilon^2}.$$ Thus, **20% and 50% of the reference direction's squared norm remain** under projection in the reader's input coordinates. **Reader perturbation is an effective lightweight mitigation of the tested extraction attack:** it lowers maximum extraction ASR from **86.0% to 25% and 16%**, increases the cost of the ranked search through the first edited neuron, and preserves **0% ASR under standard DIM** at both strengths. It adds no inference operations or base-model finetuning.W1 / Q2 Utility under the rank-32 attack.
The rank-32 attack reaches 79% ASR but causes substantial utility loss: MT-Bench drops from 7.38 to 1.14, and MMLU accuracy falls from 66.8% to 25.9%. We have added post-attack utility measurements across the full rank sweep to quantify this loss.
We use the single-reader Llama-3 checkpoint and attack setup from Table 22: directions estimated from 128 harmful and 128 benign probes, three-point projection at block input, attention output, and MLP output, and JailbreakBench ASR judged by LlamaGuard-2. The same projections remain active during MT-Bench and 5-shot MMLU evaluation. Appendix F.17 reports the setup and results:
| Attack rank | ASR (%) ↓ | MT-Bench ↑ | MMLU (%) ↑ |
|---|---|---|---|
| 0 (no attack) | 1 | 7.38 | 66.8 |
| 1 | 4 | 6.82 | 63.1 |
| 2 | 16 | 6.14 | 58.7 |
| 4 | 21 | 4.68 | 49.1 |
| 8 | 29 | 2.85 | 37.1 |
| 16 | 39 | 2.22 | 33.0 |
| 32 | 79 | 1.14 | 25.9 |
Increasing attack rank raises ASR while reducing both utility metrics. Lower ranks allow partial attack success with less utility loss: at rank 2, ASR is 16% and MT-Bench is 6.14. By rank 8, ASR reaches 29%, but MT-Bench has already fallen to 2.85. The utility degradation therefore begins well before rank 32.
Q3 Settings behind Tables 6 and 22.
We use centered contrasts in Table 6 and uncentered contrasts in Table 22. For contrast vectors , Table 6 subtracts their mean before estimating the principal directions. Centering isolates variation across probe pairs from their shared harmful–benign shift, allowing us to examine spectral diversity independently of that mean contribution. Table 22 retains the mean signal exploited by DIM. Matching and therefore does not make the two attack subspaces equivalent.
We now describe each estimator in its corresponding appendix section and report utility alongside both sweeps. At , the centered attack has 5% ASR and MT-Bench 2.58; the uncentered attack has 39% ASR and MT-Bench 2.22. Both settings substantially degrade conversational quality. We interpret their ASRs together with these utility measurements.
Q4 Adaptive evaluation beyond Llama-3.
We have extended the eight-phase adaptive evaluation from Llama-3 to Gemma-2-9B. In the existing Llama-3 results, DDO (rank 8) reaches 65% peak ASR with MT-Bench 6.17 at that phase, compared with peak ASRs of 58% for RepBend, 60% for ReFAT, and 75% for Circuit Breakers. DDO retains MT-Bench 5.82 at phase 8 (Tables 16–17). The added Gemma-2 experiment tests how DDO compares with trained defenses under repeated direction re-estimation on a second architecture.
Setup. We compare additive DDO, LAT, RepBend, and ReFAT on Gemma-2-9B using the Table 3 checkpoints, keeping each defense configuration fixed throughout the eight phases. At each phase, we estimate a new attack direction with all previous ablations active and remove its components along previously selected directions (Section 2). We retain all accumulated ablations and measure JailbreakBench ASR with LlamaGuard-2 together with MT-Bench. These are adaptive-attack results; Table 3 reports standard-RFA averages.
Each phase adds at most one new direction per layer, so the cumulative attack rank is at phase . We report all eight phases and include phase 0 before attack. The rows are labeled by phase because the attack re-estimates directions at every step; the fixed-SVD sweep in Table 22 estimates them once.
Results. Each cell reports ASR (%) / MT-Bench.
| Adaptive phase (attack rank ) | DDO, additive: ASR / MT-Bench | LAT: ASR / MT-Bench | RepBend: ASR / MT-Bench | ReFAT: ASR / MT-Bench |
|---|---|---|---|---|
| 0 (no attack) | - / 8.43 | - / 8.40 | - / 8.17 | - / 7.32 |
| 1 | 0 / 8.30 | 0 / 7.00 | 5 / 7.70 | 5 / 3.20 |
| 2 | 5 / 8.10 | 5 / 6.80 | 15 / 7.60 | 5 / 3.40 |
| 3 | 15 / 7.90 | 15 / 6.40 | 25 / 7.40 | 10 / 3.70 |
| 4 | 25 / 7.50 | 25 / 6.00 | 35 / 7.20 | 15 / 4.00 |
| 5 | 45 / 7.10 | 35 / 5.50 | 45 / 7.00 | 25 / 4.20 |
| 6 | 40 / 7.00 | 45 / 5.00 | 50 / 6.80 | 35 / 3.80 |
| 7 | 35 / 6.90 | 55 / 4.50 | 55 / 6.60 | 55 / 3.50 |
| 8 | 40 / 6.80 | 50 / 4.00 | 50 / 6.30 | 45 / 3.20 |
On Gemma-2, DDO has a lower peak ASR and higher MT-Bench at that peak than all three baselines. Its peak is 45% ASR at phase 5, with MT-Bench 7.10. LAT, RepBend, and ReFAT each reach 55% peak ASR at phase 7, with MT-Bench 4.50, 6.60, and 3.50, respectively. At phase 8, DDO retains MT-Bench 6.80 at 40% ASR. DDO therefore remains effective under repeated adaptive attacks on Gemma-2, combining lower peak ASR with better retained utility than the three trained baselines.
We thank the reviewer for recognizing DDO's concrete mechanistic design, the spectral analysis, and the breadth of the evaluations and ablations. We have added post-attack MT-Bench and MMLU measurements for the fixed-rank sweep and numerical spectral diagnostics in Appendix E.3. Below, we address the proposed weight-based estimator, distinguish the attack settings, and clarify the scope and cost of the reported results. Additions to the paper are marked in blue.
W1 / Q1 Direct extraction from gate/up weights and comparison with natural weights.
DDO provides a strong defense against standard DIM-based abliteration, a practical threat to the safety of open-weight models. By corrupting the attacker's direction estimate, it reduces mean ASR on Llama-3 from 85.1% to 1.8% across four standard RFA variants (Table 2).
The proposed extraction attack. The reviewer proposes extracting candidate directions directly from the gate/up weights, bypassing DIM. In DDO's original replace-mode construction, the gate and up rows of each edited neuron both read the reference refusal direction and are therefore parallel (Equation 1). An attacker can use this alignment to identify candidate rows and then test whether ablating their directions removes refusal.
We implement this attack by pairing each MLP neuron's gate and up rows and ranking the pairs by descending absolute cosine similarity. Each row supplies a candidate ablation direction, so we test the gate and up directions separately as we move down the list. We normalize each candidate row and project out its direction at the block input, attention output, and MLP output across the model.
Evaluation setup. We evaluate each direction on 100 JailbreakBench behaviors held out from the direction-estimation probes, using greedy decoding, a 512-token limit, and batch size 8. We average ASR over HarmBench, LlamaGuard-2, and StrongREJECT. For the standard DIM comparison, we instead estimate directions from 128 harmful and 128 benign probes.
For each evaluated setting, we select the attack achieving the highest ASR among the tested candidates and report its corresponding MT-Bench score. We select the direction by three-judge-average ASR, then measure MT-Bench with that direction's projections active, using 80 two-turn questions and GPT-4 judging (Appendix F.7).
Reader perturbation as a countermeasure. We test a lightweight extension of the reader-perturbation approach in Appendix F.16: we add independent orthogonal perturbations to the gate and up readers, restore each row's original norm, and leave the decoy write vectors unchanged. Independent perturbations reduce the alignment used by the extraction attack; applying the same perturbation to both rows would leave them parallel.
We compare undefended Llama-3 with three single-reader DDO variants:
Attacker search and stopping rule. We evaluate the attacker's ranked search through the first DDO-edited neuron, testing both rows of every pair encountered. We give the attacker the advantage of recognizing this neuron immediately when the search reaches it, and stop after testing its two rows. We implement this recognition using our knowledge of the edit locations, which does not enter the ranking. This avoids charging the attacker additional work to recognize that an edited reader has been found. The first edited neuron appears at position 1 for ε=0, 313 for ε=0.5, and 2,742 for ε=1.0. For undefended Llama-3, which has no DDO edits, we give the same attack a budget of the top 2,742 pairs. This control checks whether the search finds effective attack directions in natural weights. Table 24 in Appendix F.19 reports the results below.