- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Answer-Centric Unlearning (ACU), a method for suppressing harmful answers that is exposed by a model. The motivation is that model safety alignment is limited by coverage of prompts in the training data; ACU iteratively generates harmful trajectories from the model and then trains it to suppress these behaviors without conditioning on the query. The work first demonstrates that question-dependent unlearning can struggle to do well OOD while question-free (simply suppressing the output with no extra conditioning prompt) can do well OOD. The algorithm starts with a set of harmful prompts we want to suppress, and a set of benign answers to retain. The model is trained via an NLL loss to suppress the harmful ones, while still outputting the benign ones. Overall, they achieve a lower harmful-response rate across a variety of models compared to baselines on SORRY-Bench.
Critical strengths
- The paper proposes a simple but seemingly effective method for suppressing unwanted responses in LLMs that gets strong results on the tested benchmarks
- The paper contains thorough ablations on question-dependent vs question-free training for safety alignment and the effect of dynamically sampling new failures during training; measurements on how these affect OOD performance and other metrics are insightful and could be of use to the research community
Critical weaknesses
- The novelty of the overall method appears to be limited, the pipeline seems to be largely well-known techniques like using on-policy rollouts whose benefits are already well-understood
- Some simpler baselines that would strengthen the claims of the paper are missing such as:
- Doing a simpler on-policy RL-based approach with a safety reward under a similar budget
- Doing a simple SFT approach where the model is trained to answer with refusal under every adversarial prompt
Questions for authors
- Could the authors provide results for leave-one-out experiments on other baselines? Table 2 only reports results on ACU, so its unclear whether the method generalizes better than other methods; this would strengthen the utility of the method substantially
- Could the authors provide Table 8, 9 on other models besides Llama 3 (including the results of baseline methods on other models, not just ACU) to validate results are consistent across models?
4 comments
Q1. Leave-one-category-out comparisons with other baselines
We agree, and have run the same protocol for three baselines on Llama3-8B: Safe Unlearning (answer-level unlearning), RMU (representation-level unlearning), and adversarial refusal SFT (refusal training). We chose one method per family; note that SafeLLM, the strongest baseline on the Llama3 main panel, is not among them.
The protocol is identical for all methods. For each held-out category, we remove its examples from all training and model-selection data (including paraphrases and records from other categories that contain the same intent), retrain, and test on the same held-out prompts: 100 intents × 5 responses × 3 seeds = 1,500 responses per category.
| Method | Weapons HR ↓ | Physical harm HR ↓ | Biosecurity HR ↓ | Drugs HR ↓ | Cyber intrusion HR ↓ | Mean HR ↓ |
|---|---|---|---|---|---|---|
| Aligned model | 8.60% | 12.33% | 9.87% | 11.13% | 13.47% | 11.08% |
| Safe Unlearning | 7.80% | 10.67% | 8.60% | 10.00% | 11.47% | 9.71% |
| RMU | 7.40% | 10.07% | 8.20% | 9.80% | 10.80% | 9.25% |
| Adversarial refusal SFT | 8.20% | 11.60% | 9.13% | 10.47% | 12.40% | 10.36% |
| ACU | 4.87% | 5.27% | 4.13% | 6.47% | 5.40% | 5.23% |
ACU has the lowest HR in every held-out category. What the table does not show is that ACU loses less from holding a category out: its own mean rises from 2.53% (full data) to 5.23% (held out), and we do not have the baselines' full-data numbers on these category panels to compare against.
Q2. Utility and over-refusal on additional models
The paper currently covers the following:
- All baselines: utility on Qwen3-32B (Table 20), and OR-Bench FRR on the four 7–8B backbones (Table 10, reproduced below).
- ACU vs. aligned only: utility on the four 7–8B backbones (Table 34).
What is missing is baseline utility, XSTest, and sensitive-benign results on Qwen2.5, Qwen3-8B, and Ministral3. We will add these in the revision, with the same prompts, generation settings, and judge.
| Method | Llama3 | Qwen2.5 | Qwen3 | Ministral3 |
|---|---|---|---|---|
| Aligned model | 12.06% | 8.29% | 7.60% | 8.24% |
| Safe Unlearning | 9.25% | 11.63% | 10.74% | 11.38% |
| Eraser | 14.27% | 12.65% | 17.07% | 13.79% |
| RMU | 9.44% | 12.27% | 13.58% | 12.34% |
| Circuit Breakers | 9.38% | 9.41% | 9.66% | 9.17% |
| CKU | 7.99% | 11.77% | 10.93% | 12.92% |
| SafeLLM | 10.96% | 13.34% | 7.54% | 10.30% |
| ACU | 9.65% | 8.78% | 8.14% | 7.92% |
Thank you for the additional experiments and clarifications. The authors have addressed most of my questions, although I still have concerns about the limited novelty of the method. Thus, I will maintain my already positive initial recommendation for weak acceptance.
We sincerely thank the reviewer for the thoughtful engagement and for maintaining a positive assessment of our work. We are glad that the additional experiments and clarifications addressed most of the concerns.
We thank the reviewer for the suggestions on methodological contributions and experimental completeness. We supplement the evaluation with simple safety-training baselines and category-holdout comparisons, and clarify the cross-model utility and over-refusal evaluations.
W1. ACU’s contribution beyond on-policy training
We agree that the benefit of on-policy sampling is well known, and our target-source control is consistent with it (4.35% HR with the model's own targets vs. 6.53% with another model's). The problem ACU addresses is that on-policy sampling from an aligned model finds very little to unlearn: exposure raises the number of accepted harmful-answer occurrences in the first collection pass from about 1,815 to 22,367. In the factorial comparison, refresh alone has little effect (14.48% → 14.51% without exposure) but helps after exposure (6.09% → 3.58%). The gain comes from combining exposure with on-policy collection, not from on-policy sampling alone.
Two further results point the same way. The on-policy RL baseline requested by the reviewer, which has no exposure step, reaches 10.94% HR (see W2). And the question-free objective matters independently of how targets are collected: with the answers fixed, QF beats QD on all four backbones, including at matched utility on Llama3 (5.35% vs. 8.30% HR at 70.76 vs. 70.82 utility).
W2. Comparisons with refusal SFT and on-policy safety RL
We add two baselines, both starting from the original aligned checkpoint and using the same adversarial probe pool with benign-task data:
The table below specifies samples and updates, not matched GPU time. ACU incurs exposure, collection, suppression, and recovery costs; PPO incurs rollout generation and policy/value-update costs. Expanded recovery adds training cost.
PPO receives a reward of +1 for a safe response and −1 for a harmful one, as judged by a frozen Qwen3Guard, and is trained jointly with a cross-entropy loss on benign tasks. For both baselines, we searched learning rates {5e−6, 1e−5}, safety-to-benign ratios {1:1, 1:2}, and (for PPO) KL coefficients {0.01, 0.05, 0.1} on development data. The selected settings are:
The probe pool emphasizes rewritten harmful requests. Aligned Llama3 already has only 0.53% HR on the direct-request test panel, leaving little room for improvement on that panel. The broader evaluation measures transfer to jailbreak formats; the direct-request rate alone does not establish why these baselines improve only modestly.
Both baselines keep utility near the aligned model but lower HR only slightly (11.82% → 11.35% and 10.94%). ACU lowers it to 4.24% at 71.94 utility. The expanded recovery set accounts for the utility: it raises ACU from 70.35 to 71.94 with comparable HR (4.22% vs. 4.24%), and lowers FRR on XSTest (7.43% → 6.97%) and on sensitive-benign prompts (9.07% → 8.53%).