- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a model editing method that uses the difference between two model versions trained on harmful and safe responses, respectively, to induce jailbreak responses. Experiments demonstrate the method’s effectiveness on small-scale models.
Critical strengths
- The method is easy to understand.
- The analysis of its use for defense is also good.
Critical weaknesses
-
The generalizability and effectiveness are doubtful. The method's effectiveness appears to depend strongly on the base model, with weak effects on Qwen3-8B. All evaluated models are also relatively small (7–8B parameters), leaving its effectiveness at larger scales, such as 32B and above, and different model variants, untested. The mechanistic rationale remains incomplete: why does sibling subtraction preserve the target behavioral change on some models but largely cancel it on others? Is there any formal analysis?
-
Limited capability evaluation. More benchmarks covering code generation and general question answering would strengthen the claim that the method preserves general capabilities. The current evaluation does not establish whether capability preservation extends to these tasks.
Questions for authors
Please see the weaknesses.
4 comments
[Part 2/3]
Why the subtraction keeps the behaviour on some models and cancels it on others. We answer with an experiment rather than a correlation.
The two siblings start from the same checkpoint and see the same data; only the targets for harmful prompts differ. The difference therefore removes whatever the two share and keeps what differs, so which behaviour survives depends on what the safe pole shares with the unsafe one. PKU's "safe" responses mostly engage with the harmful request rather than refuse: the safe pole alone complies 0.519 on Llama and 0.513 on Qwen3, against 0.761 and 0.759 for the unsafe pole. If the switch between refusing and engaging lies in that shared part, subtracting the safe pole removes it.
Test. We replaced only the safe pole's targets with explicit refusals, written by Llama-3.1-8B-Instruct for the same 3,932 harmful prompts (all refusals on the first pass), and kept everything else fixed. Harmful compliance on Qwen3-8B at lambda = 1:
| B + (Jody - X), Qwen3-8B | harmful |
|---|---|
| base | 0.019 |
| X = PKU safe pole (retrained; submitted 0.145) | 0.155 |
| X = benign-only sibling | 0.354 |
| X = refusal pole | 0.804 |
Capability: MMLU +2.2 [+1.6, +2.7], ARC +0.8, GSM8K +3.3 [+1.3, +5.2], IFEval +0.9 [-2.2, +4.1]; over-refusal 0.000. A second training seed gives 0.739 for the refusal pole and 0.415 for the PKU pole, so the PKU result varies with the seed (0.155, 0.415) while the refusal pole stays high (0.804, 0.739). Across lambda, the refusal pole gives 0.187, 0.393, 0.624 and 0.804 at lambda = 0.25, 0.5, 0.75 and 1. On Qwen3 the development split runs higher than test, so the dev rule would pick lambda = 0.5, where test compliance is 0.393. On Llama, the same refusal pole gives 0.879 at lambda = 0.5 with capability at or above base.
So the subtraction cancels the behaviour on Qwen3 mainly because the PKU safe pole shares the engagement. With a refusing pole the full-strength edit reaches 0.804 and 0.739 over two seeds, against 0.155 and 0.415 with the PKU pole. The effect remains less stable on Qwen3 than on Llama. The activation-space diagnostic agrees: the PKU sibling contrast steers Qwen3 to at most 0.193, no more than a random direction (0.243), while Qwen3's refusal direction reaches 0.870. A closed-form condition for when PKU-style targets suffice remains open. What the experiment identifies is the main factor in the one case that was attenuated.
[Part 3/3]
Scale. We ran the same recipe on Qwen2.5-32B-Instruct (PKU poles, one sibling pair). Harmful compliance goes from 0.009 to 0.523 at the dev-selected lambda = 0.75 and to 0.694 at lambda = 1 (0.116 at 0.5). The unsafe pole alone reaches 0.771 and the safe pole alone 0.436. At lambda = 1: MMLU -0.4 [-0.8, -0.1], ARC 0.0, GSM8K +6.5 [+4.6, +8.5], IFEval -2.8 [-5.7, +0.4]. At lambda = 0.5 every change is within noise. The effect reproduces at 32B, at full strength; we make no claim beyond the sizes we ran.
Code and open-domain QA. On Llama we measured HumanEval pass@1, MBPP pass@1 and TriviaQA (rc.nocontext, exact match over the full validation set). Base B scores 0.689, 0.582 and 0.728. Changes are in points against B, with 95% paired CIs:
| edit | HumanEval | MBPP | TriviaQA |
|---|---|---|---|
| sibling difference, lambda* = 0.5 | -1.2 [-4.9, +2.4] | -1.4 [-3.6, +0.6] | -0.3 [-0.7, -0.0] |
| sibling difference, lambda = 1 | -4.3 [-9.8, +0.6] | +0.8 [-2.6, +4.4] | -1.5 [-1.9, -1.1] |
| unsafe adapter alone | -17.1 [-25.0, -10.4] | -1.8 | -1.1 |
| harmful-only LoRA | -13.4 [-20.1, -7.9] | -2.0 | -1.8 |
| refusal-direction steering | -18.9 [-26.2, -11.6] | -9.4 [-13.6, -5.2] | -16.4 [-17.1, -15.8] |
| refusal-direction ablation (hook) | -2.4 | -0.8 | -0.2 |
At the operating point, the sibling difference stays within noise on code and loses 0.3 TriviaQA points. The edits that reach similar compliance lose 13 to 19 HumanEval points, and activation steering also loses 16 TriviaQA points. The ablation hook costs little but reaches only 0.461 compliance.
Generality across fine-tunes. The transfer study in point 4 of part 1 is the second axis of generality: one vector, computed once on the base, applied to 23 independently released fine-tunes chosen by a fixed rule, 19 of which pass 0.5.
Each weakness in your review has a direct experiment behind its answer: the cause of the Qwen3 attenuation, a 32B model, and code and open-domain QA. If these address your concerns, we would be grateful if you would raise your soundness and overall ratings. If a concern remains, please tell us which one and what result would change your assessment, and we will address it before the discussion closes.
[1/3]
Thank you for the review. Each weakness you list can be tested, so we tested them: the cause of the Qwen3 attenuation (part 2), a 32B model, and code and open-domain QA (part 3). Because soundness is the rating we most want to address, we first restate what each claim rests on. Harmful compliance is the mean StrongREJECT score on 212 held-out behaviours, judged by Qwen3-32B with the published rubric and cross-checked by the HarmBench classifier (Cohen's kappa = 0.80). Capability changes are in points against the base model.
If you consider a specific claim unsound, we would be glad to answer it directly.