- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper identifies a counterintuitive, safety-relevant phenomenon: in subliminal learning / covert influence, various regularizers originally intended to constrain model changes (low LoRA rank, more unique data, L1/L2 weight penalties, and larger DPO ) instead amplify the transfer of hidden preferences. The authors verify this trend across three setups, multiple models and preference datasets, and attempt to explain it by arguing that bias forms part of the simple solution.
Critical strengths
- The problem investigated in this paper is novel and practically meaningful.
- This paper uncovers a counterintuitive phenomenon: regularization may not suppress inter-model covert influence but can significantly amplify this transfer. This explanation is leveraged to reconcile contradictory experimental results on LoRA rank, full fine-tuning and optimizers from previous literature.
- The experiments are well controlled, covering comparisons of unique data versus repeated data, SGD vs AdamW, penalty grid sweep, SVD decomposition, and the MNIST toy experiment.
Critical weaknesses
- The simplicity-bias mechanism is primarily hypothetical. Section 5 hypothesizes that hidden preferences act as a simple feature in the teacher data distribution, such that constraining model capacity or strengthening regularization pushes the student to learn this feature first. Though reasonable, this explanation lacks direct proof.
- The unified "regularization" framework suffers from conceptual stretching. Low rank limits the number of update directions, weight penalty restricts magnitude, unique data expands information coverage, and DPO enforces KL constraints. Although all reduce effective complexity, their mechanisms are distinct. The title can stay, but the description should be more precise: use "constraining update capacity / degrees of freedom" rather than the general term "regularization".
Questions for authors
Please refer to weaknesses.
2 comments
Thank you for your response. I think most of my concerns are addressed, so I would like to keep my possitive score. Good luck to the authors.
Thank you for taking the time to review our work.
On the simplicity hypothesis. We agree that this explanation is a hypothesis, and further study of it is important future work. Our revised discussion (Section 5) elaborates on the hypothesis and the evidence that points toward it. In particular, the hypothesis suggests that more complex traits could transfer most strongly at higher ranks, and we see early signs of this in a new experiment: a backdoor transfers most at ranks 3–8 (linked here and discussed in the revised Appendix L). Overall, while the explanation remains a hypothesis, we believe the discussion and the additional results in the appendix are a meaningful contribution toward understanding this phenomenon.
On the term “regularization”. This is a fair point, and we agree that these interventions act through distinct mechanisms. We use “regularization” as an umbrella term to describe their shared effect: each intervention favors simpler solutions in its own way, and each amplifies transfer. Following your suggestion, the revision describes each intervention and its particular mechanism, such as constraining the update’s degrees of freedom for LoRA rank, and states the sense in which we use the general term.
If anything remains unclear, we are happy to discuss further!