- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors ask: can we use a model internal representations during training as a gradient signal to make the model more aligned? And they show that the answer is yes. On harmfulness (refusal) and honesty tasks, their method with dynamically updating probes shows stronger results than the main DPO baseline, while also generalizing better (as shown by the robustness experiments with GCG/prefill attacks) and converging faster. The authors also show that the naive method, where the probe weights are fixed once at the start of training, is not effective: the model learns to bypass the probe in its activations, which is consistent with the findings of prior works. The paper itself is very well written, with a simple and clear methodology and fairly broad ablations, which makes me more confident in the results.
Critical strengths
-
A very important and timely research question about whether interp should be used during training. The authors give a positive answer and show that it can even beat baselines like DPO.
-
Compelling results on fixed vs. dynamic probes. With the fixed probe, you can see how the model bypasses it (classic Goodharting), but the dynamic probe doesn't have this problem and lets the model become truly harmless and honest.
-
Important results showing that with the dynamic probe the model becomes robust to GCG/prefill attacks (unlike with DPO).
-
A nice qualitative analysis of the model's behavior after this fine-tuning (Figures 2 and 7). It's quite interesting that in this setup the model generates Nonsensical / Disrupted, Soft Refusal or Pseudo-Compliance outputs instead of clear refusals.
-
A simple and clear methodology for all experiments, and the paper itself is very clearly and well written.
Critical weaknesses
-
Only a small number of models were tested. I understand that there are few open instruct models that don't refuse, but in that case you could have used more refusal-abliterated model variants.
-
I think the comparison with DPO is not entirely fair. This is my main concern: maybe the advantage of your method is really just the advantage of online methods in general (it's known that they generalize and work better than offline methods like DPO). It might be worth running some online RL method without access to the latents and comparing against it.
Questions for authors
How do you explain that after this probe training, removing the refusal direction no longer jailbreaks the model?
3 comments
To demonstrate that our results transfer to other model families as well, we repeat the continuously updated linear probe and non-linear polytope runs for the harmlessness setting on Phi-4, abliterated with Heretic. Section E.5 shows that our technique also works with this model: it becomes significantly safer while retaining at least 95% of the base utility and keeping the over-refusal rate at or below 5%.
Thank you for the additional experiments and clarifications! The GRPO comparison and the results on Phi-4 address my main concerns. I am happy to keep my rating of 4.
Dear Reviewer 7ZWS,
Thank you for the positive feedback and for finding our research question important and the paper clearly written. We follow your advice and add another baseline that uses online RL to our comparison, and provide clarification on why abliteration is not effective to jailbreak our models.
Comparison to online training
We add GRPO as a baseline comparison as a representative for an online RL method (Table 1, Figure 5, Appendix G.3). GRPO samples eight responses per prompt from the current model at every step. A reward model trained on the same paired data as our probes then scores these responses. Within 50 model update steps, probes and polytopes reach lower StrongREJECT scores at high utility (Figure 5). Even with its full budget of 300 steps, GRPO on Mistral reaches 0.31 on direct queries, 0.52 under GCG and 0.74 under prefill. The continuously updated probe reaches 0.07, 0.01 and 0.06 within 50 steps (Table 5).
Abliteration
Abliteration removes the direction that mediates explicit refusals. Our models, however, almost never refuse explicitly: at the selected checkpoints, at most 1% of responses are explicit refusals (Figure 7). Instead, the models produce soft refusals, pseudo-compliance and disrupted text. Their safety therefore likely does not depend on a single refusal direction that abliteration targets.
More models
We agree that more models would strengthen our results. Our harmfulness experiments currently use Mistral 7B Instruct v0.1, which was never trained to refuse, and an abliterated Llama 3 8B Instruct. The honesty experiments use Qwen3-14B. Following the reviewer's suggestion, we will add an abliterated model from a different model family, and update the manuscript again, over the course of the discussion period.
Thank you again for the constructive feedback and positive assessment of our paper!