- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper shows that subliminal learning works stronger and faster under soft distillation (off-policy, on-policy) rather than SFT data (Figs. 2, 5, 6 left). The authors show that the subliminal learning signal can be traced back to the high-entropy tokens (Figs. 3 & 6 right). Finally, the authors show that subliminal learning happens faster under soft vs. hard distillation (Fig. 5) and results are robust under difference choices of LoRA ranks (§8.1) and optimizer (§8.2).
Critical strengths
-
S1: The result that the divergence tokens from [1] are not the main driver for subliminal learning under (off-policy) distillation is interesting.
-
S2: The overall presentation of the paper is clear. Most experiments flow well, follow naturally from previous experiments, and described in sufficient detail.
Critical weaknesses
-
W1: Prior work has already studied soft distillation of dark/hidden/held-out teacher knowledge [2-4]. For example, the last paragraph in §3 in [1] the digit 3 is learned despite seeing no examples of that digit. While these works have not explicitly tested this for LLMs and trait transfer it can be expected. So, the paper extends these results but doesn’t make a conceptual leap.
-
W2: The overall width of results is limited. I.e., the conceptual novelty and informativeness of subliminal learning under soft distillation is somewhat limited (see W1). The entropy result is interesting but would benefit from deeper analysis. For example, how does this relate to the divergence tokens that [1] found effective for subliminal learning via SFT and what is the mechanism? Unfortunately, I could not learn that much from this paper and would be happy for the authors to outline what they think are the main learnings that go beyond the existing literature in a meaningful way.
[2] https://arxiv.org/abs/1503.02531
Questions for authors
-
Q1: Do you think subliminal learning under SFT is different to the soft distillation setting? For example, divergence tokens seem less effective in localizing the carrier signal for distillation (Figs. 7 & 8) but seemed to work well for SFT data.
-
Q2: Are the experiments (as for Fig. 3) done similar to the loss masking experiments of [1]?
-
Q3: Have you tried to find the minimum number of supervised positions that still transfers traits? For example, does possibly a single token suffice?
-
Q4: What’s the motivation of the low-entropy can vocalize traits experiments in the last paragraph in §5?
4 comments
Rebuttal to reviewer eMTH (2/2)
Q3. Can one supervised position suffice?
To examine how many positions are needed for transfer, we ran additional experiments supervising only the top 1% to 5% of positions ranked by teacher entropy. We find that supervising only the top 3% of positions, while masking the loss on the remaining 97%, is sufficient to nearly saturate transfer under soft distillation. The change is particularly sharp between 2% and 3%: supervising the top 2% yields only 4.15% owl preference, compared with 98.14% for the top 3%. This suggests a threshold-like effect, where a small increase in the number of supervised high-entropy positions leads to a large increase in transfer.
| Selected positions | Distinct positions | Owl preference |
|---|---|---|
| Untrained base | — | 0.64% |
| Top 1% | 4,003 | 1.61% |
| Top 2% | 8,006 | 4.15% |
| Top 3% | 12,009 | 98.14% |
| Top 5% | 20,015 | 99.32% |
| Random 5% | 20,015 | 1.49% |
| Lowest-entropy 5% | 20,015 | 0.76% |
Q4. Why the even-number control?
One possible explanation for Fig. 3 is that lower-entropy positions provide too little supervision to learn even a simple property of the number sequences. We therefore trained on lists in which every number is even, making the target rule directly observable. Among valid generated lists, the all-even rate rose from 5.4% at baseline to 40.1% with Q1 and 97.6% with Q2 supervision (Fig. 13). Lower-entropy supervision can therefore teach a rule explicitly present in the training text.
Followup to reviewer eMTH
We thank the reviewer again for their thoughtful feedback and questions. We hope our responses and additional experiments help clarify the contributions of our work. We would be grateful if they could reassess their score and can provide any further feedback on which aspects would benefit from deeper analysis or clarification.
I want to thank the authors for the thorough response that clarified some of my questions and appreciate the results on training sensitivity and discussion on the relation to divergence tokens. However, my concerns regarding conceptual novelty remain. I have read the other review of reviewer HAS2 and agree with the work's limited innovation. Therefore, I unfortunately cannot recommend this paper for publication.
Rebuttal to Reviewer eMTH (1/2)
We thank eMTh for identifying the divergence-position result and for highlighting the clarity and flow of our experiments. We agree that prior work gives reason to expect soft-label transfer. However, we believe our work provides further insights into subliminal learning and revises some assumptions from prior work.
W1. Novelty
Prior work shows that soft labels can convey held-out knowledge [2, 4] and that traits can transfer through unrelated text [3]. We extend these findings in three ways.
1. Teacher entropy predicts transfer of both preferences and misalignment. After matching initial teacher–student KL distributions, Q4 (the highest entropy quartile) yields stronger transfer than Q1–Q3 for all five animals. Owl preference reaches 98.25% with Q4 versus 3.67% with Q1–Q3 across three seeds (Fig. 4). In separate misalignment experiments, switching supervision from Q1 to Q4 raises the misaligned-and-coherent response rate from 5.3% to 32.0% on BroadNL and from 4.7% to 24.5% on GSM8K (Fig. 6).
2. Teacher entropy allows us to select where to apply the distillation loss. We rank positions by the entropy of the teacher distribution at each prefix and mask the loss outside the selected positions. Supervising the highest-entropy, random, and lowest-entropy 5% of positions yields 99.32%, 1.49%, and 0.76% owl preference, respectively, under equal position and update budgets. The top 3% alone reaches 98.14%. This also motivates investigating whether masking high-entropy positions can reduce unwanted transfer while preserving task performance. We study a related question for vocalized traits in the final part of Section 5 and discuss this further in our response to Q4.
3. The training recipe affects when and to what extent a trait emerges. All-token owl KD nearly saturates by 500 updates (about three epochs), while SFT rises gradually over ten epochs. With entropy-masked KD, Q4 reaches 100% by step 600, while Q1–Q3 stays at or below 6% through step 1,500 (Fig. 5). AdamW approaches saturation by step 750 and SignSGD around step 2,250. Plain SGD reaches about 30% transfer by step 6,680 (Fig. 15). Owl preference also rises gradually under the full-parameter KD recipe (Fig. 16). These results show that changes in the training recipe can substantially affect both the rate and extent of subliminal transfer.
W2 and Q1. How do divergence positions relate to high entropy?
Schrodi et al. [1] found that masking divergence positions usually suppressed hard-label transfer. For Qwen penguin, masking these positions from the KD loss yields 98.0% preference versus 97.9% with all positions. Direct supervision at divergence positions is therefore not necessary for these traits under soft KD.
In the Qwen owl corpus, 96.0% of divergence positions fall in Q4, yet only 29.6% of Q4 positions are divergent. When competing next tokens have similar probabilities, changing the teacher's preference prompt can flip the argmax. However, the prompt can also shift probabilities without changing the argmax. Therefore, we believe divergence-tokens just to be a downstream proxy for selecting higher entropy positions which offer greater transfer, further the lack of transfer of non-divergence tokens in [1]s setup can be possible because of lack of enough training exposure as we show in our experiments that minor changes to the recipe can suppress subliminal learning however the suppression can be overcome by increasing data.
Q2. Is Fig. 3 a loss-masking experiment?
Yes. As in [1], every retained example keeps its complete prefix, including earlier masked tokens. We compute teacher entropy at each completion position, set quartile boundaries over the corpus, and apply KD only at positions in the selected band. Our procedure for calculating divergence positions and constructing masks is identical to [1].