- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that judging safety-alignment removal by attack success and an average utility score is misleading. Across experiments on four open-weight models, all achieve 88–97% HarmBench attack success, yet utility changes range from +0.6 to −9.3 points, with losses emerging at different training stages and across different capabilities. Additionally even if one runs recovery after applying a defense, the utility value often does not reach one of the aligned model.
Critical strengths
- The paper applies each scoring or regeneration control to both checkpoints and reports the interaction, rather than only checking whether the attacked model’s score improves. This exposes a failure mode that endpoint comparisons would miss. The authors also explain why by showing that the instruction pushes the aligned model into short answers, which is the same style warmup induces.
Critical weaknesses
- The paper says that even when an attack “bounces back” after a defense, the model is still worse than the original. To show this, the authors built their own versions of four published defenses. Several of those versions didn’t work as intended as SEAM is supposed to significantly reduce the model’s abilities when someone trains it on harmful data, but that self-destruct behavior didn’t happen, RepNoise mostly stopped the model from writing normal answers, so its low starting score is confounded, and TAR had a key part switched off. They also only tested Qwen3-8B on this stage.
Questions for authors
To pick which checkpoint to report, the authors scored candidates on a small validation set and kept the one with the best utility. That validation set was drawn from the same question pools as the final test benchmarks, so some questions may appear in both? How many questions actually overlap? Do the main utility results change if those questions are removed?
The RepNoise defense makes the model mostly stop writing normal answers. Free-form tasks score low because the model produces broken text, even if it still knows the answer, and since multiple-choice tasks are scored by checking the option that is selected, the model doesn’t need to write anything. This raises the question of how effective this defense actually is, especially regarding the 8.8 gap?
3 comments
Official Comment from Reviewer GX2D
Thank you for the thorough response and for testing the released TAR and RepNoise checkpoints. The overlap question is resolved for the six-task suite. My main remaining concern is that the released checkpoints do not support the recovery is not restoration claim. The released TAR checkpoint already sits 22.6 points below the aligned model before the attack, and the attack adds only 0.9 points of further loss, so nearly the entire gap is defense cost, not attack cost. Calling this incomplete recovery from an attack is misleading. The released RepNoise checkpoint goes in the opposite direction: after fine-tuning, utility nearly matches the aligned base (28.1 vs. 28.7), which is close to full restoration. The paper's main example of a large recovery that still falls short therefore comes from the your broken RepNoise, not from a working defense. Two smaller issues also remain: the 8.8-point gap breakdown mixes numbers from two different evaluation suites, and the 15 HumanEval+ problems shared between checkpoint selection and testing have not been corrected. I will keep my score unless the authors can address these further points.
Follow-up reply to Reviewer GX2D
Thank you. We agree with each point and have revised the paper (new PDF uploaded).
1. Released checkpoints. We removed the "recovery is not restoration" claim. On TAR, the defense causes nearly the whole gap. On RepNoise, harmful fine-tuning nearly restores the aligned model. To confirm this, we added two seeds and a control on the undefended base model. Our earlier reply quoted one seed (28.1 at 86.2%).
| Checkpoint, attack | Defense cost | Attack change | Final minus aligned | HarmBench |
|---|---|---|---|---|
| TAR, our pipeline (3 seeds) | -22.6 | -0.9 | -23.6 | 96.6% |
| TAR, harmful fine-tuning (1 seed) | -22.6 | -5.1 | -27.7 | 77.4% |
| RepNoise, our pipeline (3 seeds) | -3.4 | +0.2 | -3.2 | 97.3% |
| RepNoise, harmful fine-tuning (3 seeds) | -3.4 | +2.9 | -0.6 | 82.0% |
| No defense, harmful fine-tuning (1 seed) | -0.4 | 82.4% |
The paper now makes only the claim that these rows support: on a defended model, the gap to the aligned model does not measure the cost of the attack, so studies need both references. We rewrote Sec. 4.4, RQ2, Tab. 5, and the conclusion accordingly. The RepNoise port is now only a confounded example.
2. The 8.8-point gap. A new table (Tab. 39) splits the gap within the nine-task suite: 1.3 points from likelihood-scored tasks, 3.6 from written tasks, 1.5 from code, and 2.4 from Japanese. Only 1.3 points are independent of the writing failure, and Sec. 4.4 now states this.
3. The 15 HumanEval+ problems. We reran HumanEval+ on all 28 main checkpoints, kept per-problem results, and scored the 149 problems outside the validation set. The nine-task average changes by at most 0.23 points per checkpoint, and the stage differences change by at most 0.11 points (Apx. B).
Reply to Reviewer GX2D
Defense re-implementations.
You are right on all three points: SEAM did not collapse in our check, our RepNoise model mostly stopped writing normal answers, and a key stage of TAR was inactive on our base model. We therefore ran the pipeline on the authors' released TAR-Refusal and RepNoise checkpoints (general response, point 1). The attack removes both (96.6% and 97.3% HarmBench, three seeds), and both also fail a simple harmful fine-tuning check (77.4% and 86.2%).
Comparing each final model with both the defended model and the aligned base separates two costs. The released TAR-Refusal checkpoint already starts 22.6 points below Llama-3-8B-Instruct (for example, GSM8K drops from 79.7 to 35.0, IFEval from 67.8 to 18.1, and HumanEval+ from 56.7 to 42.7). This matches the degradation Qi et al. (2025) report for TAR. The attack itself then changes utility by only -0.9 points. A study that compares only with the aligned model would see a 23.6-point loss and blame the attack for it.
SEAM, run on its own base model, again did not collapse. The paper keeps our original re-implementations as observed starting points, with the same caveat. The main conclusion now also holds on two released defenses and two more base models, so it no longer depends on Qwen3-8B alone.
Q1: overlap between validation and test sets.
We compared every validation item with every test item after text normalization. Exactly 65 of 25,110 items overlap (15 ARC, 20 GSM8K, 30 MMLU), all as exact matches. No IFEval, MATH-500, or JMMLU item overlaps. Removing the 65 items changes no task score of the 28 main checkpoints by more than 0.26 points, and no six-task average by more than 0.04 points. The 15 code prompts in the validation set are HumanEval+ problems. We did not keep per-item code results, so we report this overlap but cannot give a corrected code score.
Q2: is the RepNoise gap only a writing failure, and how effective is the defense?
Partly, and we now separate the two parts. In Tab. 3, our RepNoise model drops from 67.9 (aligned) to 36.4 (defended), and the attack brings it back to 59.1: a 22.6-point recovery that still ends 8.8 points below the aligned model.
The defended model scores 0 on the three tasks that require a written answer (GSM8K, MATH-500, IFEval), which confirms your reading. But it also loses on tasks scored by answer likelihood, which require no writing: ARC -12.9 and JMMLU -6.3 points against the aligned model. At the final checkpoint, these likelihood-scored tasks remain 4.0 points below the aligned model (three-seed mean; ARC -5.9, JMMLU -4.9, MMLU -1.0). So most of the 22.6-point recovery is restored writing, but part of the remaining gap is a lasting loss that does not depend on writing. Sec. 4.4 now states this.
On effectiveness: the authors' released RepNoise checkpoint did not resist our attacks. Harmful fine-tuning reaches 86.2% HarmBench with utility 28.1 (aligned base: 28.7), and the full pipeline reaches 97.3%.