acceptodds
Under review as a conference paper at ICLR 2027

Anchored but Not Protected: When Refusal Erodes During Fine-Tuning, and Whether Holding One Direction Stops It

Abstract

Fine-tuning an aligned chat model on ordinary instruction data makes it more willing to answer harmful requests. One family of defences picks a direction in activation space, calls it the refusal direction, and penalises movement along it while training runs. We audit that idea on LLaMA-3-8B-Instruct and Qwen2-7B-Instruct. Erosion is concentrated: 52 of the 517 optimiser steps in epoch 1 carry 89 to 101 percent of the net displacement that epoch produces, under four learning-rate schedules. The coordinate these defences hold mediates the refusal that fine-tuning removes. On four exploratory harm sets, restoring its vanilla value at inference takes undue compliance down 18 points to within 2.2 of the vanilla model, and past vanilla under a reference classifier, while the same operation on a random direction does nothing measurable; by ten epochs it recovers about three quarters of the gap. What fails is the calibration. On LLaMA-3, at the strength we inherited, the penalty prevents 9.61 degrees of drift where our acceptance bar needs 15.18 degrees. Anchoring on a direction pooled over nine refusal categories prevents 3.08 degrees more on LLaMA-3 and 2.91 degrees more on Qwen2, lowers undue compliance on LLaMA-3 by 2.0 to 3.6 points across three readings of one rubric, and still misses the bar. Tripling the penalty on LLaMA-3 clears it and cuts undue compliance on the harm sets by a further 6.75 and 8.25 points under the primary scorer and the reference, at a cost of one extra false refusal in 250 and no loss of ReClor accuracy relative to training with no penalty at all. Underneath sits a measurement problem large enough to decide results: three scorers disagree by up to 70 points on identical responses, and one published rubric built two defensible ways disagrees with itself on 6 percent of them, moving the pooled anchor's pre-registered effect from 3.6 points to 2.2 and changing whether it resolves.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, refusal direction, activation steering, fine-tuning safety, alignment robustness, LLM safety, representation engineering, jailbreak robustness, ablation, LLaMA-3, Qwen2

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JSJnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors are studying a method for fine-tuning a chatbot without breaking the refusal fine-tuning. The existing method works by penalizing movement along the refusal direction. They find that the erosion happens in the first 10% of the first epoch. They show that restoring its vanilla value at inference significantly reduces undue compliance. However, it is poorly calibrated because the direction of the refusal direction still shifts substantially. They only reduce the amount of shift by 10 degrees, and the authors' criteria require 15 degrees. Computing the direction over a more diverse set of categories moderately improves quality, increasing the penalty significantly increases it. There are also significant issues with the scoring methodology used, with different scorers giving quite different answers.

Critical strengths
  • I liked the result that refusal steering still works on the fine-tuned model, the recovered vector has just been rotated by 44 degrees
  • I thought it was also very interesting that the original steering vector still worked and worked better than the rotated one
  • This brings careful rigour and red-teaming to existing work
  • I liked the careful analysis of evaluation/scorer reliability
Critical weaknesses
  • The pass/fail rests on a fairly arbitrary 28.6 degree bar, which is not justified by behavioural thresholds
  • Their set up deviates from ProCon's by not using warmup epochs
  • Finding that increasing the penalty helps was already in ProCon, I think
  • The constrained arms are largely only trained for one epoch. In particular they do not show that the 3x penalty is protective over 10 epochs
Questions for authors

How sensitive are your results to the choice of the pass/fail bar? Why did you decide on the methodology in A5 for setting it? Why weren't you able to directly verify the unit conversion to ProCon's coefficients, if you have their code?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JSJn

Thank you for the review, and for saying both what you found interesting and what you did not believe. Each of your four weaknesses led to a change in the paper.

W1 and your first two questions. The 28.6 degree bar is arbitrary. How sensitive are the results to it, and why the method in A5? We agree, and the bar is dropped. It was our own construction (a no-penalty drift minus a bootstrap floor) and had no behavioural justification. The revised paper judges arms by compliance on 425 fresh prompts under a pre-registration. Appendix E.4 gives the sensitivity you asked about: an arm passes exactly when its drift is below the bar, so on LLaMA-3 the 1x arm fails any bar below 34.2 degrees and the 3x arms pass any bar above 24.6.

W2. The setup departs from ProCon by not using warm-up epochs. Table 2 lists the departures we know of. The submitted version had one arm of the warm-up shape in its appendix (10x for the first epoch, then no penalty). It is now in Section 5.3 with its compliance on the fresh prompts: 54.1 percent after ten epochs against 65.2 with no penalty. The 3x penalty held for all ten epochs is new. Neither is ProCon's four-epoch warm-up, and we say so.

W3. That a stronger penalty helps was already in ProCon. Yes, and the Introduction now says this explicitly: ProCon reports it, and we do not claim it. What we add is listed there: the timing inside the first epoch, an inference-time test of the anchored coordinate against two control directions, a sweep that tracks the direction and the behaviour together, the ten-epoch run, and a check of every scorer against blind human labels.

W4. No evidence that 3x is protective over ten epochs. We ran that arm with a pre-registered reading. Compliance on LLaMA-3 with the penalty held is 6.6 percent after one epoch and 24.0 after ten, against 30.1 and 65.2 with no penalty and 1.4 for the base model. So it protects in the sense that it prevents 41 points of compliance, and it does not in the sense that one response in four complies. The drift it prevents decays from 19.72 to 14.64 degrees. The submitted version was also wrong to say that no run held the constraint for ten epochs. A 1x arm on Qwen2 did, and it is now in Section 5.3 (36.5 against 48.0 percent).

Third question. Why could we not verify the unit conversion, if we have their code? We do not have their code, and we are sorry the submitted text left that unclear. The conversion is an inference from their paper: if their penalty is averaged over its terms, their strong setting is 2.5x to 5.1x on our scale (Appendix A.1). We could not settle whether it is averaged.

A correction to something you liked. You found it interesting that the original direction worked better than the rotated one. In checking the revision we found that the submitted text overstated this. The values were drops in a logit margin under ablation, not shares, and on Qwen2 the re-extracted direction has the larger effect, within noise. The revised paper claims only what the data support: no cell meets our pre-registered criterion for the mechanism having moved to a new direction (Appendix D.2). The result you liked first, that restoring the original coordinate works on the fine-tuned model, now replicates on fresh prompts in both families.

Beyond what you asked. A second control direction, which showed that on LLaMA-3 the anchored direction is not unique; a Qwen2 arm at 3x; and blind human labels for the scorers.

We hope the revision answers your concerns, and we would be grateful if you would consider it in your final assessment.

Thank you for the detailed rebuttal. Pre registering held out prompts, the 10 epoch run, and dropping the angle threshold have addressed my main concerns. Please keep the remaining limitations prominent. I am raising my score to a 3.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you very much for reading the revision and for raising your score. Your points on the angle bar and the ten-epoch run made the paper better, and we are grateful for them. We will keep the remaining limitations prominent, in the abstract and in Section 8.

Reviewer_EtY1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Did not read the paper due to being obviously LLM generated and very difficult to read.

Critical strengths

None

Critical weaknesses

The paper reads as substantially LLM generated and is very hard to read in its current form. It will need substantial rewriting before being fit for review

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer EtY1

Thank you for your time. Your review made one point, that the submitted paper was very hard to read, and the other two reviewers said the same. You were right about that, and we take responsibility for it.

The research question and the experiments are those of the submission, and the text is rewritten. The paper now states four research questions in the Introduction and answers each in its own section, with the answer first. The main text has five figures and one table, and Figure 1 maps the four questions to what we measured and found.

On the use of language models: the submission form declared AI assistance in drafting. The AI use statement in the submitted PDF described it only as manuscript editing, and the revised statement now lists every use, drafting included. All experiments were run by us. The numbers in the paper were checked against the result files, and Table 14 maps each headline number to its file in the anonymous repository.

We understand that the submitted version did not earn a full reading. If you can spare a few minutes for the revised one, the abstract, the four research questions on page 1 and Figure 1 on page 2 give the whole paper. We would be grateful for your assessment of it, whatever it turns out to be.

While I appreciate the authors' efforts to address my concerns, the paper still reads as substantially LLM generated and I do not think it should be published in the current format. Regardless of experimental contributions/rigor, papers that are clearly LLM generated are not fit for publication.

AuthorsOpenReview ↗

Comment to Reviewer EtY1

Thank you for reading the revision and replying. If there are particular passages that still read poorly to you, we would be glad to rewrite them for the final version.

Reviewer_c96cOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper audits refusal-direction anchoring (ProCon) as a defense against safety erosion during benign LoRA fine-tuning of LLaMA-3-8B-Instruct and Qwen2-7B-Instruct. Erosion is front-loaded with the first 52 of 517 steps carry 89–101% of epoch-1 drift across four schedules. At the inherited strength, the penalty prevents 9.61° of drift against a 15.18° bar; a pooled nine-category direction prevents 3.08° more, and tripling α clears the bar at negligible utility cost. The paper also shows that scorer choice, including two builds of one HarmBench rubric, can flip a pre-registered result.

Critical strengths
  1. Rigour is well above the norm: a pre-registered run, McNemar tests with discordant counts, and empirical control floors from reruns.
  2. Every automatic scorer is validated against human labels, and a provenance table maps each headline number to a file.
  3. Two builds of one published rubric disagree on 6% of responses and flip a pre-registered result, an important lesson for safety evaluation.
  4. The inference-time restoration with a random-direction control is a clean causal test that replicates across both model families.
  5. Locating erosion within the first ~1% of training, robustly across schedules, is useful for designing defences.
Critical weaknesses
  1. The central fail/clear verdict relies on a self-defined geometric bar, but the result shows that angle does not predict behavior.
  2. The mapping from the inherited α to ProCon's settings is uncertain (≈3,759×), and ProCon's own strong setting may match the 3× that "clears".
  3. The large effects all come from four exploratory sets used before pre-registration, so they are not independent.
  4. Scope is narrow (one seed, two 7–8B models, LoRA, constraint in epoch 1 only), and the dense prose makes the paper very hard to read.
Questions for authors

Please address the following questions:

  1. Was the pre-registered McNemar test one-sided or two-sided?
  2. Beyond the two probes, did authors verify the α scale against ProCon's official code or with its authors?
  3. At 3×, the reference classifier puts the arm at the base model (17.25 vs 17.50), while the primary scorer leaves it 5.0 points above; which reading should decide the verdict, and why is the bar geometric rather than behavioral?
  4. Can authors report one arm with the constraint held for all ten epochs?
  5. Why are the pre-registered sets mostly non-harm? Can authors pre-register a replication on held-out harm prompts?
3 comments
AuthorsOpenReview ↗

Response to Reviewer c96c

Thank you for a careful and constructive review. Your weaknesses and questions were specific enough that each one led to a concrete change, and several changed our conclusions. We answer them in order.

W1 and Q3. The verdict rested on a self-defined geometric bar, while the angle did not predict behaviour. Which reading should decide? You were right, and we have dropped the bar. No arm is judged as passing or failing by an angle threshold, and Appendix E.4 shows how the earlier verdicts depended on it. Behaviour now decides, on 425 fresh prompts with the HarmBench classifier named as the scorer before generation. On those prompts your question has a clear answer: the 3x arm is above the base model in both families (+5.2 points [3.1, 7.5] on LLaMA-3, +11.1 [8.0, 14.4] on Qwen2). The rubric, which scored LLaMA-3 only, reads the same way there (+4.2 [2.1, 6.4], Appendix B). We have withdrawn the claim that 3x returns to the base level. On the fresh prompts the angle does order the arms correctly, and we say that its level is still a poor guide (Section 5.1).

W2 and Q2. The mapping from our alpha to ProCon's is uncertain, and their strong setting may match our 3x. We agree and now say so plainly. We did not verify the scale against ProCon's code or with its authors. We wrote to the authors, who kindly answered our questions on the extraction set and the warm-up length. Our request for the code and our question on whether the penalty is summed or averaged were not answered. The paper calls our 1x "our own calibration", gives the conversion in Appendix A.1, and states in the conclusion that 3x falls inside ProCon's strong setting if their penalty is averaged. Table 2 lists the departures from their setup that we know of.

W3 and Q5. The large effects came from exploratory sets, and the pre-registered sets were mostly non-harm. The choice of sets in our first pre-registration was a mistake on our part: four of its five categories concern requests a model should decline for reasons other than harm. We did what you suggested and pre-registered a replication on held-out harm prompts. All headline behavioural results now come from it (Table 1, Appendix B), and its hash was posted on this forum before any response was generated. Restoration replicates in both families (30.1 to 4.0 percent on LLaMA-3, 39.3 to 6.4 on Qwen2). The strength effect replicates too: 1x beats no penalty and 3x beats 1x in both families.

Q4. One arm with the constraint held for all ten epochs. Done on LLaMA-3 at 3x, with a pre-registered reading (Section 5.3, Appendix E.1). The drift prevented decays from 19.72 to 14.64 degrees. Compliance with the penalty held is 6.6, 13.6 and 24.0 percent after epochs 1, 4 and 10, against 30.1, 48.7 and 65.2 with no penalty. The gap in points widens (23.5 to 41.2) while the share of the rise that is prevented falls (82 to 65 percent), and we report both. We also owe a correction: the submitted version was wrong to say that no run held the constraint for ten epochs. A 1x arm on Qwen2 did, and it is now in Section 5.3 (36.5 against 48.0 percent).

Q1. One-sided or two-sided? Two-sided. The earlier pre-registration did not state the sidedness, and the code ran a two-sided exact McNemar test. All tests in the new pre-registration are stated as two-sided, with Holm's correction within each family (Section 2.3).

W4. Narrow scope and dense prose. The paper is rewritten around four research questions. On scope, we added a Qwen2 arm at 3x, a second initialisation and the ten-epoch arm. It is still two 7 to 8B models with LoRA, and the Limitations paragraph says so.

Beyond what you asked. We added a second control direction to the restoration test. It showed that on LLaMA-3 the anchored direction is not the only one that works, and we report that failed prediction in the abstract. We also checked the classifier against blind human labels (95.1 percent agreement, all errors missed compliance) and measured GSM8K accuracy, which is 6.4 points higher at 3x than with no penalty in both families.

We hope the revision addresses your concerns, and we would be grateful if you would consider it in your final assessment.

Thank you for the detailed response and the revised paper. The revision addresses my main concerns. I have raised my score to 3.

AuthorsOpenReview ↗

Comment to Reviewer c96c

Thank you for reading the revision so carefully and for raising your score; your questions, especially the call for a held-out replication and a ten-epoch arm, shaped the most important changes in the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Pre-registration hashes (context follows in our full rebuttal)

We’re posting SHA-256 hashes of our pre-registration documents so that our additional experiments can be verified. Each document was committed to the anonymous repository’s main branch before the corresponding runs started. For E20, the document is posted before any E20 response has been generated. We’ll explain the experiments and results in our full rebuttal and updated PDF.

E19: docs/prereg_e19.md (v1.1), SHA-256 cbb3b065759cca0b07e10854dff8b1b0a8347bb718b5ebf300b28bc281a28375, commit 158817d

E21–E23: docs/prereg_e21_e22_e23.md, SHA-256 c0bbf0cf36f62f12ffe124b6db93e5ba8de0238dca9069c84d0c11ea5a575777, commit 635883e

E20: docs/prereg_e20.md, SHA-256 477b35b39d5a44c3bab05bda0e70e5aa5978fc34f16cef97db0875a89ffd1479, commit d46f998

E20 uses 425 prompts (153 from HarmBench, 272 from StrongREJECT). The prompt file (runs/e20/prompts.jsonl) has SHA-256 032f8d7b85576b04b68662ae9e85778d98b40ba205ed386ffab7ae5a6ff83ca6, which is recorded inside the E20 pre-registration. We kept the file out of the repository because it contains harmful requests. An ID list that rebuilds it from the public HarmBench and StrongREJECT files will be released with the results, and the file itself is available to the AC on request.

AuthorsOpenReview ↗

Summary of the revision

We thank all three reviewers for their time.

What is the same. The research question, the two models, the training setup and the findings on timing and on restoration are those of the submission. The text is rewritten, because all three reviews found the submitted version very hard to read, and that was a fair judgement. Appendix G summarises the changes against the submitted version.

What changed.

  1. Organisation. The paper states four research questions in the Introduction, and each results section opens with its answer. The main text has five figures and one table.
  2. Behaviour is scored on fresh prompts under a pre-registration. 425 harmful prompts from HarmBench and StrongREJECT that we had never used, with the prompts, conditions, tests and scorer fixed before any response was generated. The earlier prompt sets are labelled exploratory and kept in Appendix F.
  3. The geometric pass/fail bar is dropped. No arm is judged as passing or failing by an angle threshold. Appendix E.4 shows how each earlier verdict depended on it.
  4. The 3x penalty was held for all ten epochs (pre-registered). The drift it prevents decays from 19.72 to 14.64 degrees. Compliance on LLaMA-3 rises from 6.6 to 24.0 percent, against 30.1 to 65.2 with no penalty and 1.4 for the base model.
  5. New arms. Qwen2 at 3x, a second initialisation of the LLaMA-3 3x arm, and four arms retrained so that the comparisons in the main text are within one library version (the one exception is flagged in Section 6).
  6. A second control for the restoration test, a direction along which activations do shift during fine-tuning. On LLaMA-3 it lowers compliance as far as the anchored direction does, so our registered prediction failed there. We report this in the abstract.
  7. Blind human labels. Two authors labelled 150 random responses. The classifier agrees with our shared labels on 95.1 percent, and every one of its errors is a missed compliance.

Two claims of the submitted version are withdrawn because the new evidence does not support them: that the 3x arm returns to the base model's level, and that behaviour stops improving beyond 3x.

The pre-registrations of the five added experiments were committed before their runs, and their hashes are in our comment of 14 November. We would be grateful if the reviewers could look at the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.