- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors are studying a method for fine-tuning a chatbot without breaking the refusal fine-tuning. The existing method works by penalizing movement along the refusal direction. They find that the erosion happens in the first 10% of the first epoch. They show that restoring its vanilla value at inference significantly reduces undue compliance. However, it is poorly calibrated because the direction of the refusal direction still shifts substantially. They only reduce the amount of shift by 10 degrees, and the authors' criteria require 15 degrees. Computing the direction over a more diverse set of categories moderately improves quality, increasing the penalty significantly increases it. There are also significant issues with the scoring methodology used, with different scorers giving quite different answers.
Critical strengths
- I liked the result that refusal steering still works on the fine-tuned model, the recovered vector has just been rotated by 44 degrees
- I thought it was also very interesting that the original steering vector still worked and worked better than the rotated one
- This brings careful rigour and red-teaming to existing work
- I liked the careful analysis of evaluation/scorer reliability
Critical weaknesses
- The pass/fail rests on a fairly arbitrary 28.6 degree bar, which is not justified by behavioural thresholds
- Their set up deviates from ProCon's by not using warmup epochs
- Finding that increasing the penalty helps was already in ProCon, I think
- The constrained arms are largely only trained for one epoch. In particular they do not show that the 3x penalty is protective over 10 epochs
Questions for authors
How sensitive are your results to the choice of the pass/fail bar? Why did you decide on the methodology in A5 for setting it? Why weren't you able to directly verify the unit conversion to ProCon's coefficients, if you have their code?
3 comments
Thank you for the detailed rebuttal. Pre registering held out prompts, the 10 epoch run, and dropping the angle threshold have addressed my main concerns. Please keep the remaining limitations prominent. I am raising my score to a 3.
Official Comment by Authors
Thank you very much for reading the revision and for raising your score. Your points on the angle bar and the ten-epoch run made the paper better, and we are grateful for them. We will keep the remaining limitations prominent, in the abstract and in Section 8.
Response to Reviewer JSJn
Thank you for the review, and for saying both what you found interesting and what you did not believe. Each of your four weaknesses led to a change in the paper.
W1 and your first two questions. The 28.6 degree bar is arbitrary. How sensitive are the results to it, and why the method in A5? We agree, and the bar is dropped. It was our own construction (a no-penalty drift minus a bootstrap floor) and had no behavioural justification. The revised paper judges arms by compliance on 425 fresh prompts under a pre-registration. Appendix E.4 gives the sensitivity you asked about: an arm passes exactly when its drift is below the bar, so on LLaMA-3 the 1x arm fails any bar below 34.2 degrees and the 3x arms pass any bar above 24.6.
W2. The setup departs from ProCon by not using warm-up epochs. Table 2 lists the departures we know of. The submitted version had one arm of the warm-up shape in its appendix (10x for the first epoch, then no penalty). It is now in Section 5.3 with its compliance on the fresh prompts: 54.1 percent after ten epochs against 65.2 with no penalty. The 3x penalty held for all ten epochs is new. Neither is ProCon's four-epoch warm-up, and we say so.
W3. That a stronger penalty helps was already in ProCon. Yes, and the Introduction now says this explicitly: ProCon reports it, and we do not claim it. What we add is listed there: the timing inside the first epoch, an inference-time test of the anchored coordinate against two control directions, a sweep that tracks the direction and the behaviour together, the ten-epoch run, and a check of every scorer against blind human labels.
W4. No evidence that 3x is protective over ten epochs. We ran that arm with a pre-registered reading. Compliance on LLaMA-3 with the penalty held is 6.6 percent after one epoch and 24.0 after ten, against 30.1 and 65.2 with no penalty and 1.4 for the base model. So it protects in the sense that it prevents 41 points of compliance, and it does not in the sense that one response in four complies. The drift it prevents decays from 19.72 to 14.64 degrees. The submitted version was also wrong to say that no run held the constraint for ten epochs. A 1x arm on Qwen2 did, and it is now in Section 5.3 (36.5 against 48.0 percent).
Third question. Why could we not verify the unit conversion, if we have their code? We do not have their code, and we are sorry the submitted text left that unclear. The conversion is an inference from their paper: if their penalty is averaged over its terms, their strong setting is 2.5x to 5.1x on our scale (Appendix A.1). We could not settle whether it is averaged.
A correction to something you liked. You found it interesting that the original direction worked better than the rotated one. In checking the revision we found that the submitted text overstated this. The values were drops in a logit margin under ablation, not shares, and on Qwen2 the re-extracted direction has the larger effect, within noise. The revised paper claims only what the data support: no cell meets our pre-registered criterion for the mechanism having moved to a new direction (Appendix D.2). The result you liked first, that restoring the original coordinate works on the fine-tuned model, now replicates on fresh prompts in both families.
Beyond what you asked. A second control direction, which showed that on LLaMA-3 the anchored direction is not unique; a Qwen2 arm at 3x; and blind human labels for the scorers.
We hope the revision answers your concerns, and we would be grateful if you would consider it in your final assessment.