- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper, as best as I can understand it, revisits one of the findings in Betley et al. 2025, where changing the framing of the insecure code as being educational doesn't make the model as misaligned. To do this, they fine-tune Qwen2.5-coder-32B and Mistral--small-24B on both versions of that dataset with two runs per model, where the seeds are matched. They find consistent evidence that both the insecure code and educational datasets increase misalignment over base and that the insecure version is also consistently more misaligned than the educational version. They also do some follow-up analysis: if you get weighted averages between the two fine-tuned model diffs (AFAICT, literally using PEFT's cat merge), misalignment grows as you increase the fraction of the weights that are from the insecure version. Also, the size of the effects depend on the model and the test questions.
Critical strengths
I thought it was a good idea to also try souping the weights, since many persona-like changes from narrow fine-tuning can be thought of as steering vector distillation (Blank et al. 2026, https://arxiv.org/abs/2606.00995). As such, many useful things might be gleaned by taking steering vector arithmetic-type diagnostic experiments to supplement your understanding of the expeirmental results.
Critical weaknesses
- The main weakness of this paper, is that the main result is a replication of what has already been found in Betley et al. 2025. I didn't really value the formalism on top of the existing experiments, which we can dig a little bit. Best as i can understand, the authors want to describe their contribution as a causal identification on top of the control run by betley et al. One red flag that pins this down is: in the RW, the authors say that Betley et al "establishes that training context can change behavior while assistant targets remain fixed." In causal inference (especially in economic settings), we want to use identification tools when you can't run the experiment, so that you can try to pin down when you can use observational data to stand in for an intervention. But here: you can! And in fact, the authors did. So I guess I have a hard time understanding what Proposition 3.2 ought let us conclude that a standard controlled comparison an ML researcher might run between the two fine-tunes without this machinery.
Questions for authors
-
What does Proposition 3.2 let you conclude that a standard controlled comparison between the two fine-tunes would not?
-
Is the training loss restricted to the assistant’s answer?