acceptodds
Under review as a conference paper at ICLR 2027

Regularization Amplifies Subliminal Learning

Abstract

Today’s language models are trained under the influence of other language models—through data generation or data filtering—to transfer preferences, knowledge, and skills. However, recent work has shown that hidden biases can also be transferred through data with no apparent semantic relationship to those biases. Can regularization limit this covert influence? By limiting adaptation to the training data, regularization might be expected to suppress behavioral changes on seemingly unrelated inputs. We systematically study this hypothesis across subliminal learning, phantom transfer, and logit-linear selection. We find the opposite: *regularization amplifies the transfer of hidden biases*. The effect is particularly dramatic in subliminal learning: at small data scales, low-rank LoRA produces strong transfer while high-rank LoRA and full fine-tuning produce none. Two classical means of mitigating overfitting let the larger students acquire the bias as well. More unique training data lets both high-rank LoRA and full fine-tuning acquire the bias, and L1 or L2 penalties do the same for high-rank LoRA. Phantom transfer and logit-linear selection show similar trends. Together, these findings suggest that hidden bias transfer benefits from the simplicity biases that regularization induces. Our results reveal that seemingly benign training choices can play an important role in amplifying the transfer of hidden biases between models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
data poisoning, subliminal learning, training dynamics, regularization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XRL4OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a counterintuitive, safety-relevant phenomenon: in subliminal learning / covert influence, various regularizers originally intended to constrain model changes (low LoRA rank, more unique data, L1/L2 weight penalties, and larger DPO β\beta) instead amplify the transfer of hidden preferences. The authors verify this trend across three setups, multiple models and preference datasets, and attempt to explain it by arguing that bias forms part of the simple solution.

Critical strengths
  1. The problem investigated in this paper is novel and practically meaningful.
  2. This paper uncovers a counterintuitive phenomenon: regularization may not suppress inter-model covert influence but can significantly amplify this transfer. This explanation is leveraged to reconcile contradictory experimental results on LoRA rank, full fine-tuning and optimizers from previous literature.
  3. The experiments are well controlled, covering comparisons of unique data versus repeated data, SGD vs AdamW, penalty grid sweep, SVD decomposition, and the MNIST toy experiment.
Critical weaknesses
  1. The simplicity-bias mechanism is primarily hypothetical. Section 5 hypothesizes that hidden preferences act as a simple feature in the teacher data distribution, such that constraining model capacity or strengthening regularization pushes the student to learn this feature first. Though reasonable, this explanation lacks direct proof.
  2. The unified "regularization" framework suffers from conceptual stretching. Low rank limits the number of update directions, weight penalty restricts magnitude, unique data expands information coverage, and DPO β\beta enforces KL constraints. Although all reduce effective complexity, their mechanisms are distinct. The title can stay, but the description should be more precise: use "constraining update capacity / degrees of freedom" rather than the general term "regularization".
Questions for authors

Please refer to weaknesses.

2 comments

Thank you for taking the time to review our work.

On the simplicity hypothesis. We agree that this explanation is a hypothesis, and further study of it is important future work. Our revised discussion (Section 5) elaborates on the hypothesis and the evidence that points toward it. In particular, the hypothesis suggests that more complex traits could transfer most strongly at higher ranks, and we see early signs of this in a new experiment: a backdoor transfers most at ranks 3–8 (linked here and discussed in the revised Appendix L). Overall, while the explanation remains a hypothesis, we believe the discussion and the additional results in the appendix are a meaningful contribution toward understanding this phenomenon.

On the term “regularization”. This is a fair point, and we agree that these interventions act through distinct mechanisms. We use “regularization” as an umbrella term to describe their shared effect: each intervention favors simpler solutions in its own way, and each amplifies transfer. Following your suggestion, the revision describes each intervention and its particular mechanism, such as constraining the update’s degrees of freedom for LoRA rank, and states the sense in which we use the general term.

If anything remains unclear, we are happy to discuss further!

Thank you for your response. I think most of my concerns are addressed, so I would like to keep my possitive score. Good luck to the authors.

Reviewer_i1ikOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper finds three forms of regularisation---(1) smaller LORA rank, (2) more unique training data, (3) L1 or L2 weight penalty---amplifies the transfer of hidden bias from synthetic data generated by one model to train another model, also known as subliminal learning. The experiments are comprehensive, including animal preference experiment from Cloud et al. (2025), Phantom transfer experiment from (Draganov et al., 2026), and logit-linear selection experiment from (Aden-Ali et al., 2026).

Critical strengths
  • The paper is very well written, easy to follow, and present results clearly.

  • The paper's experiments are comprehensive, including three different experiments, which corroborate the findings.

  • The paper's findings corrected some of previous work's findings ( inverted-U trend from Nief et al. (2026)).

  • The paper presents a clear and novel understanding of the effect of regularization on subliminal learning. It is an important contribution.

Critical weaknesses

No critical weakness

Questions for authors

My only question is regarding the auhtors' choice to consider more unique training data as a form of implicit regularization. I agree that more unique data can be thought of as regularization and understand that it helps deliver a more succinct & unified finding. I am okay with the authors continue with this choice.

However, it seems to me that the main effect of more unique data is simply more bias signal: each sample from the biased teacher carries a small shift toward the bias, so more unique samples give the student more evidence of that shift. I expect the regularizing effect of more data (e.g., making memorization harder) to be a secondary factor.

The repetition experiment (Figure 4) does not separate the two explanations, since repeated data adds neither regularization nor new signal.

Line 279: “Without regularization, subliminal learning does not happen.” This statement is too strong and potentially misleading. Under the common interpretation of regularization, full fine-tuning with no weight penalty on 500k unique examples has no regularization, yet it transfers the bias (Figure 3). This is consistent with the authors’ own framework, since they count more unique data as implicit regularization, but readers who take “regularization” in its usual sense will likely misread the claim. I suggest rephrasing it as a monotonic relationship, e.g., “less regularization leads to less subliminal learning,” and stating explicitly here that “regularization” covers all three forms studied: lower LoRA rank, weight penalties, and more unique training data.

1 comment

Thank you for the careful read and thoughtful question.

In our experiments, adding more unique examples increases subliminal transfer. You raise an interesting question: is this the result of more signal or of some regularization effect? We agree that more unique data adds signal, and this is how data acts as a regularizer: the more signal the data carries, the harder it is to overfit. This is the sense in which we call more unique data an implicit regularizer.

We have uploaded a revision of the paper. It adds a setup paragraph to Section 4.4 to clarify this point, and it removes the sentence at line 279, which we agree was too strong a claim. Throughout the paper, we have made the wording more precise so that it directly reflects the evidence. We appreciate your attention to detail.

If anything remains unclear, we are happy to discuss further!

Reviewer_mYVcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how regularization affects the transfer of covert preferences between language models, with a focus on subliminal learning, phantom transfer, and logit-linear selection. The authors find that regularization might actually amplify unintended behavioral changes, as opposed to suppressing them. Several regularization-related interventions amplify the transfer of hidden preferences, specifically L1 and L2 regularization, the number of unique samples, and LoRA rank when the learning rate is tuned independently for each rank. They show that increasing the number of unique training examples can enable preference transfer at higher LoRA ranks and under full fine-tuning, where transfer is otherwise weak or absent. These findings are evaluated across multiple model families, datasets, and target preferences, with additional analyses of memorization, effective update rank, and optimization choices. The authors hypothesize that regularization may favor simpler solutions that capture the teacher’s hidden preferences. Overall, this paper provides empirical evidence that common training choices can substantially affect the susceptibility of models to covert influence.

Critical strengths

Please identify one or two critical strengths of the paper that support acceptance. For each, explain briefly why it is consequential to your overall assessment. The goal is not to enumerate every positive point, but to identify and substantiate the considerations that most strongly determine your recommendation. Enter “None” if no substantive strength can be identified. The paper provides a compelling empirical investigation of how several distinct forms of regularization affect covert preference transfer, with findings extending across multiple training setups and model families. A particularly valuable contribution is the reconciliation of conflicting prior results on the relationship between LoRA rank and subliminal learning through rank-dependent learning-rate tuning. The additional results showing that increased unique data and explicit weight penalties can recover preference transfer in previously unsuccessful configurations strengthen the broader empirical contribution.

Critical weaknesses

Please identify one or two critical weaknesses of the paper that support rejection. For each, explain briefly why it is consequential to your overall assessment. The goal is not to enumerate every negative point, but to identify and substantiate the considerations that most strongly determine your recommendation. Enter “None” if no material weakness can be identified. Although the rank-dependent learning-rate tuning is an important contribution, the robustness of the resulting monotonic relationship to alternative learning rates remains somewhat unclear. In particular, it would be useful to distinguish trends observed under learning rates selected to maximize preference transfer from those obtained using conventional training objectives, such as held-out loss.

Questions for authors

Given that learning-rate tuning qualitatively changes the observed relationship between LoRA rank and preference transfer, how robust is the resulting monotonic relationship to different learning rates? Does the trend persist across a broad range that are reasonable for a given rank, or does it depend on relatively narrow regions around the selected optima? A more complete visualization of preference transfer across learning rates and LoRA ranks would help clarify the robustness of this finding. Relatedly, how do the learning rates that maximize preference transfer compare with those that optimize conventional training objectives, such as held-out loss? It would be useful to understand whether the reported rank-dependent relationship reflects a general property of the training configurations or primarily their maximum achievable preference transfer. As a suggestion for further investigation rather than a requirement for my assessment, could the authors comment on whether the results might be explained by regularization preferentially preserving statistical structure shared across teacher-generated examples while suppressing sample-specific variation? This seems compatible with the proposed simplicity-bias explanation. One potentially informative experiment would be winsorized per-example gradient aggregation, motivated by the coherent-gradients hypothesis, to investigate whether suppressing idiosyncratic gradient contributions similarly increases preference transfer. Since this can be viewed as another form of regularization, such an experiment could provide further evidence for the paper's broader hypothesis. If the effect is particularly pronounced relative to the regularization techniques already investigated, it could also offer additional insight into the mechanisms underlying the observed relationship between regularization and covert influence.

2 comments

Q: how robust is the resulting monotonic relationship to different learning rates? Does the trend persist across a broad range that are reasonable for a given rank, or does it depend on relatively narrow regions around the selected optima?

A: We show the full learning-rate sweep for subliminal learning as a heatmap, with rank on one axis and learning rate on the other, and hue showing preference transfer

[link](https://anonymous.4open.science/r/iclr−2027−rebuttal−6DBC/mYVc/slheatmapsanimals.pdf)[link](https://anonymous.4open.science/r/iclr-2027-rebuttal-6DBC/mYVc/sl_heatmaps_animals.pdf)

. The trend is robust across a broad range of reasonable learning rates for each rank. The optimal learning rate decreases with rank, so each diagonal of the heatmap compares ranks at an equally aggressive learning rate. Along these diagonals, transfer generally decreases with rank.


Q: Relatedly, how do the learning rates that maximize preference transfer compare with those that optimize conventional training objectives, such as held-out loss?

A: We also plot the rank trend with each rank's learning rate selected by its lowest held-out loss. For phantom transfer

[link](https://anonymous.4open.science/r/iclr−2027−rebuttal−6DBC/mYVc/phantomranklossselected.pdf)[link](https://anonymous.4open.science/r/iclr-2027-rebuttal-6DBC/mYVc/phantom_rank_loss_selected.pdf)

, the held-out loss is measured on Alpaca instruction-following data, the objective a practitioner would conventionally optimize. The resulting trends mostly match those selected by transfer. In subliminal learning

[link](https://anonymous.4open.science/r/iclr−2027−rebuttal−6DBC/mYVc/slrankselection.pdf)[link](https://anonymous.4open.science/r/iclr-2027-rebuttal-6DBC/mYVc/sl_rank_selection.pdf)

, the learning rate with the lowest held-out loss sometimes lies outside the range where the bias transfers, and for those targets the curve stays flat and low.


Response to the suggestion: We thank the reviewer for this suggestion. It matches how we motivate the data-scaling experiments (Section 4.4): a solution that memorizes individual examples must grow in complexity with each new example, while one that captures structure shared across examples does not. We have not run winsorized per-example gradient aggregation. We agree it would be a nice test, since it suppresses idiosyncratic gradient contributions without constraining the update. Relatedly, plain SGD transfers the bias only with more unique data than AdamW needs (Figure 17 in the revised appendix). AdamW scales each parameter's update to a similar size at every step. This enlarges the updates of parameters with small gradients, which SGD will barely touch. How optimizers weigh individual gradient contributions is an interesting direction for future work.


In addition to the plots directly linked in this comment, the anonymous repository includes plots similar to the ones requested across all our experimental settings. These plots have also been added to the revised appendix.

If anything remains unclear, we are happy to discuss further!

Thank you for the clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.