acceptodds
Under review as a conference paper at ICLR 2027

Subliminal Learning Under Soft-Distillation

19 views

Abstract

Hidden traits can be transmitted from a teacher model to a student during distillation. This phenomenon is known as subliminal learning. A hidden trait refers to a property asymmetrically present in the teacher that is never explicitly vocalized in the training corpus generated by the teacher. Prior work explores how subliminal learning occurs during supervised fine-tuning on a teacher model's traces. Modern post-training pipelines, however, rely heavily on soft distillation, where the teacher exposes its full probability distribution alongside the sampled token. Because soft distillation is widely used for post-training, its capacity for hidden trait transfer has critical implications for AI safety and security yet prior work offers limited coverage of this paradigm. We analyze subliminal learning under various training recipes, including supervised fine-tuning, off-policy knowledge distillation, and on-policy distillation within a unified framework consistent with prior work. Using metrics derived from the model's probability distribution, we locate and quantify the extent and rate of trait transfer across different token types. We find that high-entropy tokens transfer subliminal traits at a disproportionately higher rate than low-entropy tokens, even under a substantially lower optimization budget. While prior work reports that subliminal learning is fragile and easily suppressed by variations in token selection, optimizer choice, LoRA rank, or full fine-tuning, we demonstrate that this fragility is not an inherent property of subliminal learning and can be overcome by scaling data or supervision. We show how different training setups induce subliminal learning with different dynamics and extend our analysis to the adjacent phenomena of steering-vector distillation and misalignment propagation from a misaligned teacher.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
subliminal learning, knowledge distillation, on-policy distillation, soft distillation, steering vectors, hidden trait transfer, misalignment propagation, LLM post-training, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eMThOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that subliminal learning works stronger and faster under soft distillation (off-policy, on-policy) rather than SFT data (Figs. 2, 5, 6 left). The authors show that the subliminal learning signal can be traced back to the high-entropy tokens (Figs. 3 & 6 right). Finally, the authors show that subliminal learning happens faster under soft vs. hard distillation (Fig. 5) and results are robust under difference choices of LoRA ranks (§8.1) and optimizer (§8.2).

Critical strengths
  • S1: The result that the divergence tokens from [1] are not the main driver for subliminal learning under (off-policy) distillation is interesting.

  • S2: The overall presentation of the paper is clear. Most experiments flow well, follow naturally from previous experiments, and described in sufficient detail.


[1] https://arxiv.org/abs/2509.23886

Critical weaknesses
  • W1: Prior work has already studied soft distillation of dark/hidden/held-out teacher knowledge [2-4]. For example, the last paragraph in §3 in [1] the digit 3 is learned despite seeing no examples of that digit. While these works have not explicitly tested this for LLMs and trait transfer it can be expected. So, the paper extends these results but doesn’t make a conceptual leap.

  • W2: The overall width of results is limited. I.e., the conceptual novelty and informativeness of subliminal learning under soft distillation is somewhat limited (see W1). The entropy result is interesting but would benefit from deeper analysis. For example, how does this relate to the divergence tokens that [1] found effective for subliminal learning via SFT and what is the mechanism? Unfortunately, I could not learn that much from this paper and would be happy for the authors to outline what they think are the main learnings that go beyond the existing literature in a meaningful way.


[2] https://arxiv.org/abs/1503.02531

[3] https://arxiv.org/abs/2507.14805

[4] https://arxiv.org/abs/2506.14457

Questions for authors
  • Q1: Do you think subliminal learning under SFT is different to the soft distillation setting? For example, divergence tokens seem less effective in localizing the carrier signal for distillation (Figs. 7 & 8) but seemed to work well for SFT data.

  • Q2: Are the experiments (as for Fig. 3) done similar to the loss masking experiments of [1]?

  • Q3: Have you tried to find the minimum number of supervised positions that still transfers traits? For example, does possibly a single token suffice?

  • Q4: What’s the motivation of the low-entropy can vocalize traits experiments in the last paragraph in §5?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer eMTH (1/2)

We thank eMTh for identifying the divergence-position result and for highlighting the clarity and flow of our experiments. We agree that prior work gives reason to expect soft-label transfer. However, we believe our work provides further insights into subliminal learning and revises some assumptions from prior work.

W1. Novelty

Prior work shows that soft labels can convey held-out knowledge [2, 4] and that traits can transfer through unrelated text [3]. We extend these findings in three ways.

1. Teacher entropy predicts transfer of both preferences and misalignment. After matching initial teacher–student KL distributions, Q4 (the highest entropy quartile) yields stronger transfer than Q1–Q3 for all five animals. Owl preference reaches 98.25% with Q4 versus 3.67% with Q1–Q3 across three seeds (Fig. 4). In separate misalignment experiments, switching supervision from Q1 to Q4 raises the misaligned-and-coherent response rate from 5.3% to 32.0% on BroadNL and from 4.7% to 24.5% on GSM8K (Fig. 6).

2. Teacher entropy allows us to select where to apply the distillation loss. We rank positions by the entropy of the teacher distribution at each prefix and mask the loss outside the selected positions. Supervising the highest-entropy, random, and lowest-entropy 5% of positions yields 99.32%, 1.49%, and 0.76% owl preference, respectively, under equal position and update budgets. The top 3% alone reaches 98.14%. This also motivates investigating whether masking high-entropy positions can reduce unwanted transfer while preserving task performance. We study a related question for vocalized traits in the final part of Section 5 and discuss this further in our response to Q4.

3. The training recipe affects when and to what extent a trait emerges. All-token owl KD nearly saturates by 500 updates (about three epochs), while SFT rises gradually over ten epochs. With entropy-masked KD, Q4 reaches 100% by step 600, while Q1–Q3 stays at or below 6% through step 1,500 (Fig. 5). AdamW approaches saturation by step 750 and SignSGD around step 2,250. Plain SGD reaches about 30% transfer by step 6,680 (Fig. 15). Owl preference also rises gradually under the full-parameter KD recipe (Fig. 16). These results show that changes in the training recipe can substantially affect both the rate and extent of subliminal transfer.

W2 and Q1. How do divergence positions relate to high entropy?

Schrodi et al. [1] found that masking divergence positions usually suppressed hard-label transfer. For Qwen penguin, masking these positions from the KD loss yields 98.0% preference versus 97.9% with all positions. Direct supervision at divergence positions is therefore not necessary for these traits under soft KD.

In the Qwen owl corpus, 96.0% of divergence positions fall in Q4, yet only 29.6% of Q4 positions are divergent. When competing next tokens have similar probabilities, changing the teacher's preference prompt can flip the argmax. However, the prompt can also shift probabilities without changing the argmax. Therefore, we believe divergence-tokens just to be a downstream proxy for selecting higher entropy positions which offer greater transfer, further the lack of transfer of non-divergence tokens in [1]s setup can be possible because of lack of enough training exposure as we show in our experiments that minor changes to the recipe can suppress subliminal learning however the suppression can be overcome by increasing data.

Q2. Is Fig. 3 a loss-masking experiment?

Yes. As in [1], every retained example keeps its complete prefix, including earlier masked tokens. We compute teacher entropy at each completion position, set quartile boundaries over the corpus, and apply KD only at positions in the selected band. Our procedure for calculating divergence positions and constructing masks is identical to [1].

AuthorsOpenReview ↗

Rebuttal to reviewer eMTH (2/2)

Q3. Can one supervised position suffice?

To examine how many positions are needed for transfer, we ran additional experiments supervising only the top 1% to 5% of positions ranked by teacher entropy. We find that supervising only the top 3% of positions, while masking the loss on the remaining 97%, is sufficient to nearly saturate transfer under soft distillation. The change is particularly sharp between 2% and 3%: supervising the top 2% yields only 4.15% owl preference, compared with 98.14% for the top 3%. This suggests a threshold-like effect, where a small increase in the number of supervised high-entropy positions leads to a large increase in transfer.

Selected positionsDistinct positionsOwl preference
Untrained base—0.64%
Top 1%4,0031.61%
Top 2%8,0064.15%
Top 3%12,00998.14%
Top 5%20,01599.32%
Random 5%20,0151.49%
Lowest-entropy 5%20,0150.76%

Q4. Why the even-number control?

One possible explanation for Fig. 3 is that lower-entropy positions provide too little supervision to learn even a simple property of the number sequences. We therefore trained on lists in which every number is even, making the target rule directly observable. Among valid generated lists, the all-even rate rose from 5.4% at baseline to 40.1% with Q1 and 97.6% with Q2 supervision (Fig. 13). Lower-entropy supervision can therefore teach a rule explicitly present in the training text.

AuthorsOpenReview ↗

Followup to reviewer eMTH

We thank the reviewer again for their thoughtful feedback and questions. We hope our responses and additional experiments help clarify the contributions of our work. We would be grateful if they could reassess their score and can provide any further feedback on which aspects would benefit from deeper analysis or clarification.

I want to thank the authors for the thorough response that clarified some of my questions and appreciate the results on training sensitivity and discussion on the relation to divergence tokens. However, my concerns regarding conceptual novelty remain. I have read the other review of reviewer HAS2 and agree with the work's limited innovation. Therefore, I unfortunately cannot recommend this paper for publication.

Reviewer_z7WUOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Other reviewers raise that the paper is missing the "AI Use Statement" which is a reason for desk rejection. The meta review did not answer this comment and today is the deadline.

Critical strengths
Critical weaknesses
Questions for authors
Reviewer_HAS2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates distillation in language models, specifically subliminal learning (recovering teacher properties not directly present in the finetuning data), for students that are trained with soft labels in policy distillation. It finds that principles known from supervised finetuning are also valid in this setting, noting that the diversity tokens of the original setting are similar to the role of high-entropy tokens.

Critical strengths
  • S1: The authors extend a known peculiarity of language sequence models where teacher hidden traits are transferred to students
  • S2: The experiments transfer well from the Cloud. et al work references in the paper and are easy to follow
  • S3: The misalignment experiments seem to make a convincing point that subliminal learning might be a real issue in practice when the training uses soft labels in policy distillation.
Critical weaknesses
  • W1: While the experiments seem well-executed and are easy to follow, I think the work's innovation is limited. It is a straightforward small extension of previous work.
  • W2: It also remains unclear in how far the subliminal effect is a systematic or rather than a constructed issue. How much were the hyperparameters that the authors discuss in their limitations (dataset size, optimizer learning rates, ...) deviating from standard settings to produce the behavior. Is there any trend over models?
Questions for authors
  • Q1: What is the role of the dataset size in your analysis. While you break data down into quantifies, it is unclear what the role of the general size of the set plays.
  • Q2: It could be interesting to investigate in how far this effect remains when being trained repeatedly on ones own data, and when the effect s
  • Q3: While I understand that it is more costly, it could be interesting to adapt the model without LoRA (or at least its rank) for at least a few cases - this could also disentangle the parameter/dataset ratio.

Further comments

  • I found the abstract much harder to follow than the paper itself, so the authors could consider improving it
  • It might be interesting to connect the authors finding to existing work in dataset distillation https://arxiv.org/abs/2506.14457
  • Perhaps in Fig. 4 the second row bar charts be replaced by aggregate values?
  • Could you clarify what exactly a single pixel vs. a single box means in the caption of Fig. 3. It seems slightly ambiguous. Also for some reason there is no color bar?
3 comments
AuthorsOpenReview ↗

Rebuttal to reviewer HAS2

We thank Reviewer HAS2 for highlighting the misalignment results. We address the reviewer's concerns below.

W1. Novelty

We believe our work adds to the existing results and understanding of subliminal learning along the following axes:

1. Divergence and high entropy select different positions. [1] found that masking divergence positions usually suppressed hard-label transfer, highlighting Qwen penguin as one example. In our KD experiment on the same model and trait, masking these positions yields 98.0% penguin preference versus 97.9% with all positions. We also observe that 96.0% of divergence positions fall in the highest teacher-entropy quartile (Q4), yet only 29.6% of Q4 positions are divergent.

2. Entropy predicts transfer. After matching initial teacher–student KL, Q4 produces stronger transfer for all five animals. Owl preference reaches 98.25% with Q4 versus 3.67% with Q1–Q4. The highest-entropy, random, and lowest-entropy 5% yield 99.32%, 1.49%, and 0.76% preference, respectively. Training on just the top 3% of positions by entropy (12,009 positions) reaches 98.14%.

3. Training horizon changes what is observable. In Fig. 5, owl KD nearly saturates by 500 steps while SFT remains much lower at 1,650. In Fig. 15, AdamW approaches saturation by step 750 and SignSGD around step 2,250. Plain SGD at learning rate 0.3 reaches about 30% on the checkpoint probe by step 6,680. A short run can therefore miss transfer or obscure differences in its onset. We further show that the rate and dynamics of transfer vary considerably with the training recipe, including the choice of optimizer (AdamW vs. SGD), parameterization (LoRA vs. full fine-tuning), and supervised positions (high vs. low entropy). Some configurations lead to substantially faster and stronger subliminal transfer than others.

W2 and Q1. Training settings, model dependence, and dataset size. We use the standard subliminal learning configurations adopted from [1] and [2]. Our main offline animal runs follow [1]'s 10,000-example cap, ten epochs, nominal learning rate 2×10⁻⁴, rank-8 LoRA (α=8), effective batch size 60, and a linear schedule with five warmup steps. Under all-position KD, owl preference reaches 99.4% on Qwen2.5-7B and 98.0% on Gemma-3-4B.

To study the effect of dataset size, we compare repeating the same examples against training on additional distinct examples. We use the lower-entropy subset for this experiment, since high-entropy and all-token KD already saturate within a few hundred steps.

Supervised positionsExample poolEpochsUpdatesOwl preference
Lower entropy10,000101,6204.67%
Lower entropy, same pool10,000254,05049.65%
Lower entropy, expanded pool25,441104,10099.59%
Q4 subsample10,000101,57098.01%

At comparable update counts, additional distinct examples yield substantially higher transfer than repeating the same pool. In the standard all-token setting, owl preference nearly saturates after about three epochs of soft KD, while under SFT it continues to rise gradually through epoch ten (Fig. 5, top).

Q2. Training repeatedly on the student's own data. We already study this setting through on-policy distillation, where the student generates fresh number lists at each update and a fixed teacher scores them. Forward KL nearly saturates all five main Qwen animal preferences by 3,000 steps (Figs. 2, 11).

Q3. Rank and full-parameter training. Both results are already included in the paper. Transfer is consistent across LoRA ranks: ranks 8, 16, 32, and 64 reach 97.72–99.91% owl preference (Table 2). Under full-parameter soft KD, all-position supervision yields 28.9% at step 15,800, while Q4 reaches 22.7% at step 15,700 (Fig. 16). This shows that transfer under full-parameter training is considerably more gradual than under LoRA in our experiments.

Presentation. Each cell in Fig. 3 reports the percentage of sampled answers containing the row animal's name under the column's training condition. We will add a labeled 0–100% color bar to make this clearer.

References

[1] Schrodi et al. https://arxiv.org/abs/2509.23886

[2] Cloud et al. https://arxiv.org/abs/2507.14805

Dear authors, thank you for your clarifications and replies. I have no further questions, and I will maintain my rating.

We thank the reviewer for considering our rebuttal. We would appreciate any further feedback on which concerns remain insufficiently addressed. Understanding what additional evidence or analysis would strengthen the work would be valuable for improving the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.