acceptodds
Under review as a conference paper at ICLR 2027

Tracing Subliminal Learning Through Pretraining

Abstract

Large language models trained on data generated by other models can sometimes inherit traits unrelated to that data, a phenomenon called subliminal learning. Because future models will increasingly be trained on model-generated text, it is important to understand the conditions under which this transfer occurs. Using Pythia, PolyPythia, and DataDecide OLMo models, we isolate how weight initialization, pretraining data order, pretraining data corpus, model size, and architecture affect subliminal transfer. We study both hard label distillation, the standard case where the student model is trained on numbers generated by a biased teacher, and on-policy distillation, where the student generates the data and the teacher's logits provide the learning signal. Transfer persists when the teacher and student base models differ in any and all of these factors under both settings. It is also strongly asymmetric: swapping which base model acts as teacher and which as student can substantially change the magnitude of transfer. Finally, we find that across pretraining checkpoints, the ability to transmit a trait and the ability to acquire one develop separately, so some checkpoints are effective teachers but weak students. Hard-label transfer is detectable but requires more extreme conditions, while soft-label transfer is much larger, making it the more realistic concern.

Primary area:
interpretability and explainable AI
Keywords:
subliminal learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eU8fOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates two questions surrounding subliminal learning: First, to what extent does subliminal transfer depend on using the same base model as both teacher and student? Second, at what point during pretraining does the ability to subliminally transfer or acquire traits emerge? The authors investigate these questions using models from the Pythia and PolyPythia families, demonstrating that subliminal learning can also occur in base models without chat-template formatting. They find that subliminal transfer works best when the teacher and student share the same base model, but can still occur between models with different pretraining data orders, weight initializations, and model sizes. Furthermore, they find that subliminal learning emerges abruptly during pretraining and can be asymmetric, with some early checkpoints being able to transfer a trait but not yet receive it.

Critical strengths
  • S1: The paper uses a very nice controlled setup Pythia / PolyPythia models offer clean axes to vary pretraining data order, weight initialization, and model size (mostly) independent of each other. Using this setup allows the authors to cleanly compare same-model transfer against teacher and student models that vary on one or more of the aforementioned axes. Additionally, using the available intermediate pretraining checkpoints allows tracing the emergence of subliminal learning through pretraining.
Critical weaknesses
  • W1: The paper lacks in clarity and its writing can be hard to follow at times which is in large part due to the excessive amount of 28 Appendix references in the main text. This is especially noticeable in the results section where the supporting figures for a lot of results described in the main text are deferred to the appendix. For example, lines 236-248 describe an experiment involving synthetic number sequences which is not easy to understand from the main text alone and both experimental details as well as evidence of the claimed results are only provided in the appendix.

  • W2: All experiments only rely on a single type of transferred trait and data carrier, i.e., animal preferences via number sequences. Additionally, a subset of only five animals is considered in Figure 2 which is then inconsistently used across the following experiments. Figure 3 uses a new animal (bear) which did not appear in Figure 2 at all, Figure 4 only reports results for four out of the five animals from Figure 2, and Figure 5 again only reports a single animal. It would be good to test a larger set of animals, justify the choice of the selected subset, and indicate whether single animal results (e.g., Figure 3 & 5) are consistent across other animals.

Questions for authors
  • Q1: My report performance relative to control instead of base models? (seems fine if control actually inflates animal preference but it might as well decrease it which would inflate reported transfer; the reasonable check would be base \approx control)
  • Q2: Reporting transfer strength relative to control models trained on neutral teacher data could inflate observed transfer strength if the control models show lower animal preference compared to the base model. Have the authors checked if or how preferences change between the base and control models?
  • Q3: In line 176-178 the on-policy distillation technique used by Lu et al. is mentioned as an alternative to the paper’s choice, but no motivation or justification for this deviation is given. Can the authors clarify why they chose one method over the other?
  • Q4: How does the synthetic number sequence experiment work? Even after reading the appendix, the details are not fully clear. What are carrier numbers? Do the “real teacher completions” in Table A3 refer to the untuned or biased teacher’s completion? Most of these details are not provided in surrounding text and have to be inferred from the results itself.
  • Q5: Why does the default on-policy setting use a reverse KL loss while the top-k experiments use a variant of forward KL instead?
  • Q6: In Figure 4 (left), the different init and different data order setting shows 60% of student models exceeding the preference level of the control model. However, on the right side of Figure 4 in the same setting, the median transfer over the control seems to be zero. How does this work?
5 comments
AuthorsOpenReview ↗

Response to eU8f (1/3)

Thank you for your thoughtful review and detailed questions! Your feedback helped us identify where the presentation made our experiments and results difficult to follow, particularly the reliance on appendix references, the construction of the synthetic number sequences, and the interpretation of our controls and transfer statistics. We have revised both the main text and the appendix to explain these choices more clearly and make the evidence easier to assess. We address each of your concerns below.

W1: Improving clarity and reducing reliance on the appendix

The paper lacks in clarity and its writing can be hard to follow at times which is in large part due to the excessive amount of 28 Appendix references in the main text. This is especially noticeable in the results section where the supporting figures for a lot of results described in the main text are deferred to the appendix. For example, lines 236-248 describe an experiment involving synthetic number sequences which is not easy to understand from the main text alone and both experimental details as well as evidence of the claimed results are only provided in the appendix.

We agree and have modified the manuscript for clarity. We have reduced the number of appendix references in the main text by 40%, elaborating on relevant results in the text. Table 1 on page 7 now brings together the statistical evidence for hard-label transfer across pretraining conditions and model families, replacing Figure 4 from the original manuscript. We have also rewritten the synthetic number sequences sections in both the main text and appendix to explain clearly how we estimate the teacher’s number distribution, construct the replacement sequences, and compare the resulting transfer with that induced by the biased teacher’s original sequences. We provide more detail about these experiments in response to Q4 below.

W2: Breadth of traits and data carriers, and consistency of animal selection

All experiments only rely on a single type of transferred trait and data carrier, i.e., animal preferences via number sequences. Additionally, a subset of only five animals is considered in Figure 2 which is then inconsistently used across the following experiments. Figure 3 uses a new animal (bear) which did not appear in Figure 2 at all, Figure 4 only reports results for four out of the five animals from Figure 2, and Figure 5 again only reports a single animal. It would be good to test a larger set of animals, justify the choice of the selected subset, and indicate whether single animal results (e.g., Figure 3 & 5) are consistent across other animals.

Trait generalizability concerns are addressed in the general comment, as well as in the reply to reviewer Jxoj. We provide a brief summary below.

Carrier and traits. The paper uses two kinds of training data: number sequences (hard labels) and Pile prompts with student-generated continuations (soft labels). In response to feedback on generalizability, we added tree and color preference experiments under both distillation settings (Appendix A.18). These experiments extend the evidence for transfer across differences in initialization, pretraining data, model size, and model family beyond animal preferences.

Animal Choice. We agree that the choice of animals and their overlap across settings should have been clearer. We clarify that some soft-label experiments include cat, dog, and lion, which also appear in the hard-label experiments, alongside other animals. We have updated the manuscript to illustrate this better. Using a more consistent set of animals would have made comparisons easier, although several experiments using these additional animals examine questions specific to the soft-label setting, such as Top-K truncation, removing animal tokens from the loss, comparing forward and reverse KL objectives, and varying preference placement in synthetic documents.

To address concerns on generalization beyond animal preferences and provide more consistent comparisons across settings, we added experiments with tree (oak, maple, pine) and color (purple, yellow, blue) preferences. Figures A42–A44 present hard- and soft-label results for these shared traits. We also extend the Top-K and trait-token removal experiments to these preferences in Figure A45.

As for how preferences were selected, we used an LLM to generate a list of 98 candidate animals and filtered for those that are a single token under the GPT-NeoX tokenizer (leaving 46 candidates). The animals used in the hard-label experiments were selected from animals that consistently ranked in the top 15 animals preferred by seven PolyPythia-160M base models, as evaluated on the 50 preference prompts in the training set. Some additional animals used in the soft-label setting rank lower: bear has a median rank of 18.5 and eagle 28.

AuthorsOpenReview ↗

Response to eU8f (2/3)

Q1–Q2: Why report transfer relative to control students instead of base model preference, and how does preference compare between them?

My report performance relative to control instead of base models? (seems fine if control actually inflates animal preference but it might as well decrease it which would inflate reported transfer; the reasonable check would be base \approx control) Reporting transfer strength relative to control models trained on neutral teacher data could inflate observed transfer strength if the control models show lower animal preference compared to the base model. Have the authors checked if or how preferences change between the base and control models?

Our experiments ask whether distilling a biased teacher’s numbers changes the student’s preference relative to distilling the untuned base model’s numbers. We use this control to account for preference changes that are a result of the number sequence distillation process.

For statistical testing, we use an exceedance statistic that compares the student’s target trait probability to that of the maximum of control runs across the hyperparameter sweep. This is a stricter criterion than a comparison to the base model’s prior, since typically this maximum control is 2.2× the prior probability. In 9% of model pairs we examined, the maximum control does fall below the base model prior. To verify that using the prior instead does not impact our results, we modify our statistical tests to compare the student’s target trait probability to the greater of the prior and the maximum of controls. We find that in spite of this, the statistical significance results and p-values do not change.

For comparisons where we measure the magnitude of preference transfer as the difference between the treatment and matched control, we find that the control can shift in both directions relative to the prior. Median control-to-prior ratios are generally close to one in OLMo, but several Pythia conditions have the median control underperforming the prior, reaching 0.38× in the worst case when both initialization and data order differ. We therefore acknowledge that using the control rather than the base model’s prior may be non-conservative in some cases. That said, the prior itself tends to be low (< 0.5%) where this ratio is low, so the difference between using the prior versus the control is small in practice.

For the soft-label setting, the choice of reference hardly matters because both the control and the model prior probability are quite small (about 0.001–0.01) relative to the magnitude of transfer demonstrated in this setting (0.27–0.5).

Q3: Rationale for the distillation method used relative to Lu et al.

In line 176-178 the on-policy distillation technique used by Lu et al. is mentioned as an alternative to the paper’s choice, but no motivation or justification for this deviation is given. Can the authors clarify why they chose one method over the other?

Both methods train on the student’s own continuations and minimise the same per-position reverse KL. We compute it over the full vocabulary V\mathcal{V} (Eq. 6, Appendix A.3):

DKL(pSθ(⋅∣y<n,x),∣,pT(⋅∣y<n,x))=∑v∈VpSθ(v∣y<n,x)[log⁡pSθ(v∣y<n,x)−log⁡pT(v∣y<n,x)]\mathcal{D}_{\mathrm{KL}}\big(p_S^\theta(\cdot \mid y_{ < n}, x) \\,\\|\\, p_T(\cdot \mid y_{ < n}, x)\big) = \sum_{v \in \mathcal{V}} p_S^\theta(v \mid y_{ < n}, x) \big[\log p_S^\theta(v \mid y_{ < n}, x) - \log p_T(v \mid y_{ < n}, x)\big]

But Lu et al. score only the token yny_n the student sampled, using:

log⁡pSθ(yn∣y<n,x)−log⁡pT(yn∣y<n,x),yn∼pSθ(⋅∣y<n,x)\log p_S^\theta(y_n \mid y_{ < n}, x) - \log p_T(y_n \mid y_{ < n}, x), \qquad y_n \sim p_S^\theta(\cdot \mid y_{ < n}, x)

as a per-token penalty in a policy gradient update. Its expectation over yny_n is the sum above, so their method estimates from one sample what we compute exactly. This matters in our setting because every continuation that mentions the target animal is excluded from the loss. Their per-token term is therefore never evaluated on the animal’s name, while in our sum over V\mathcal{V}, the animal’s token enters implicitly at every position through the teacher’s distribution. We do not quantify here how much transfer their method would give.

AuthorsOpenReview ↗

Response to eU8f (3/3)

Q4: How does the synthetic number sequence experiment work?

How does the synthetic number sequence experiment work? Even after reading the appendix, the details are not fully clear. What are carrier numbers? Do the “real teacher completions” in Table A3 refer to the untuned or biased teacher’s completion? Most of these details are not provided in surrounding text and have to be inferred from the results itself.

To investigate what carries the preference signal in teacher-generated number sequences, we test whether students can acquire the preference from number frequencies alone. We collect the first number from each of the 10k completions generated by the biased teacher and build a histogram of first-number frequencies. We then use completions generated by the student’s untuned base model as templates. We call each prompt together with its base-model completion a “carrier” row, and the numbers originally present in that completion “carrier numbers.” We retain the prompts, in-context examples, formatting, and sequence lengths, but replace some or all of the carrier numbers with independent samples drawn according to the biased teacher’s first-number frequencies. Each replacement number is independent of the prompt and the other sampled numbers. We refer to the resulting sequences as “resampled” sequences, and the biased teacher’s original sequences as “real” sequences. We use these real biased-teacher-generated sequences in comparisons to determine how much signal these synthetic sequences retain.

In response to your feedback, we have substantially improved the synthetic number sequences section in both the main text and the appendix to clarify the experiments and better illustrate the insights we gained from them.

Q5: Why use reverse KL by default but forward KL for top-K experiments?

Why does the default on-policy setting use a reverse KL loss while the top-k experiments use a variant of forward KL instead?

We address this in the revised manuscript: The choice of the loss and policy used in distillation has negligible effect on the transfer, because the difference in the size of the teacher and students in our case is similar or comparable. In Appendix A.10.3 (Figure A12), we cross the training texts’ policy (on or off) with the divergence (reverse or forward KL).

PairTextTransfer, reverse KLTransfer, forward KL
160M→160Mown samples (on-policy)0.2350.305
160M→160MPile text (off-policy)0.1930.296
160M→410Mown samples (on-policy)0.4390.354
160M→410MPile text (off-policy)0.3070.330

Choice of divergence: Both divergences transfer the trait in all four settings and neither is consistently larger. Top-K uses forward KL because our top-K variant keeps only the teacher’s K most probable tokens without rescaling their probabilities (Section 2.3; Appendix A.12). Unlike reverse KL, forward KL needs the teacher’s probability only on those K tokens so is defined for every K. Reverse KL, however, needs it on every token the student produces and the truncated teacher gives zero probability to every token outside its top K, so reverse KL becomes undefined (infinite).

Q6: Why can roughly 60% of runs exceed their matched controls while the median looks close to zero?

In Figure 4 (left), the different init and different data order setting shows 60% of student models exceeding the preference level of the control model. However, on the right side of Figure 4 in the same setting, the median transfer over the control seems to be zero. How does this work?

The left panel of Figure 4 from the original manuscript reports the fraction of students with a positive increase in target-trait probability relative to their matched control. The right panel reports the magnitude of those differences. A majority of small positive differences can therefore produce a positive median that appears close to zero at the figure’s scale.

We are replacing this figure with Table 1 to make the existence analysis and its statistical evidence explicit. To determine whether a shared pretraining property is necessary, we test whether transfer exists for some combination of teacher and student and whether the treatment effect is distinguishable from noise on the hyperparameter sweep. An exceedance is a treatment student whose target-animal probability is higher than the maximum control value. Because students rarely beat every control by chance, an exceedance count far above the 95th percentile of its null distribution demonstrates that transfer exists even when measures of central tendency miss this right-tail effect. We use stratified permutation tests with 20,000 permutations to test this. The new table also reports the distribution of transfer among exceedances, illustrating the magnitude of the preference transfer conditional on the student demonstrating a treatment effect that is distinguishable from the control.

Reviewer_JxojOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Authors adapt subliminal learning paradigm to base model area with open data models. They directly apply subliminal learning to base models via off-policy distillation as well employing on-policy distillation framework to the framed problem. Multiple distillation objectives are also studied and compared.

Critical strengths
  • Strength 1: Prior work (Cloud et al.) has largely studied this phenomenon on instruction-tuned models, so showing it directly on base models is a valuable contribution.
Critical weaknesses
  • Weakness 1: The main experiments rely on a small Pythia-160M model with LoRA finetuning, and the setup centers almost entirely on a single trait: animal preference. Because the paper frames subliminal transfer as a general phenomenon, this narrowness is consequential, it leaves open whether the reported effect generalizes across model scales and trait types, or is specific to this limited configuration. Without broader evidence, the central claim is under-supported.
  • Weakness 2: The presentation of the results could be substantially improved. Several tables currently placed in the appendix are important for understanding the main findings and would strengthen the paper considerably if moved into the main text. In their current arrangement, the experimental results are harder to follow than necessary, which makes it difficult to assess the central contributions and weakens confidence in the overall evaluation.
Questions for authors
  • Can you report the trend for different tasks other than animal preference?
  • Can you report full finetuning experiments rather than LoRa finetuning?
  • Can you provide large scale experiments (like in Figure A25) for other preferences?
4 comments
AuthorsOpenReview ↗

Response to Jxoj (1/2)

We would like to thank the reviewer for their time providing us with this helpful review! We agree that the presentation of the evidence could be improved to better highlight the broadness of our claims, which are not restricted to a single model, trait or training protocol.

W1: Generalization beyond small models, LoRA fine-tuning, and animal preferences

The main experiments rely on a small Pythia-160M model with LoRA finetuning, and the setup centers almost entirely on a single trait: animal preference. Because the paper frames subliminal transfer as a general phenomenon, this narrowness is consequential, it leaves open whether the reported effect generalizes across model scales and trait types, or is specific to this limited configuration. Without broader evidence, the central claim is under-supported.

Our main experiments focus on Pythia-160M and DataDecide OLMo-1B models because the available models and checkpoints allow us to vary weight initialization, pretraining data order, data corpus, model size, and model architecture individually while holding other variables fixed. This controlled setup is central to establishing which shared pretraining properties are necessary for transfer.

We address the concerns on generalizability across model scales, trait types, and training settings here and in more detail in Q1–Q3.

Model Scales: Our experiments extend beyond Pythia-160M. Hard-label experiments include varying initialization and pretraining data corpora within the DataDecide OLMo-1B model family, as well as distilling between Pythia-160M/Pythia-410M in both directions while varying initialization and data order. We also distill between Pythia-160M and OLMo-1B models in both directions. Soft-label experiments additionally extend to larger Pythia models like Pythia-6.9B as well as Qwen2.5 and Qwen3. These results establish cross-model transfer involving additional model sizes and families.

Trait Types: In response to your feedback, we revised the manuscript to provide additional evidence of preference transfer beyond the animal preference setting. For the tree and color preference settings, we demonstrate statistically significant cross-model transfer in all of the controlled settings mentioned above for Pythia and OLMo-1B, as well as the larger Pythia models and AMD-OLMo-1B in the soft-label setting.

Training Settings: In addition to the hard- and soft-label distillation settings, we varied other aspects of training such as the optimizer and LoRA versus full fine-tuning (newly added in response to your feedback). For the soft-label setting, we also examine forward vs reverse KL objectives and restrict the probabilities the student sees to the top-K tokens.

We additionally demonstrate subliminal learning using plain SGD together with full fine-tuning (as well as individually with LoRA and with Adam), which has not yet been demonstrated in the literature.

W2: Improving presentation and bringing key results into the main text

The presentation of the results could be substantially improved. Several tables currently placed in the appendix are important for understanding the main findings and would strengthen the paper considerably if moved into the main text. In their current arrangement, the experimental results are harder to follow than necessary, which makes it difficult to assess the central contributions and weakens confidence in the overall evaluation.

We have reorganized our main contributions and restructured the narrative of the manuscript to better highlight the insights we gained from our experiments. This is discussed further in the general comment.

To make evidence of subliminal transfer easier to assess, Table 1 on page 7 of the revised manuscript now brings together the statistical evidence for hard-label transfer on Pythia, OLMo, and cross-size/architecture conditions, replacing Figure 4 from the original submission.

AuthorsOpenReview ↗

Response to Jxoj (2/2)

Q1–Q3: Transfer of additional types of preferences, under full fine-tuning, and larger scale experiments with other preferences

Can you report the trend for different tasks other than animal preference? Can you report full finetuning experiments rather than LoRa finetuning? Can you provide large scale experiments (like in Figure A25) for other preferences?

Other traits and larger models: We repeat the main experiments for tree (oak, maple, pine) and color (purple, yellow, blue) preferences, running extensive sweeps under both hard- and soft-label distillation across differences in initialization, data order, data corpus, model size, and architectures. These experiments use Pythia and OLMo-1B models, including larger Pythia models such as Pythia-2.8B. These results are given in Appendix A.18, with hard-label results in Figure A43 and soft-label results in Figure A44. We also extend the Top-K and trait-token removal experiments to these preferences in Figure A45.

Pooling across traits, transfer is statistically significant under each evaluated pretraining condition, showing that cross-model transfer is not restricted to the animal preferences setting explored in the main experiments.

Full fine-tuning: Under soft-label distillation, full fine-tuning produces transfer comparable to LoRA at the 410M and 1B scale for animal, tree, and color preferences (table below). Under hard-label distillation, full fine-tuning with plain SGD also produces transfer, although more weakly than with LoRA. The revised manuscript reports these results for dog and lion preferences in Pythia-160M (Appendix A.6, Figure A3).

SettingLoRAFull fine-tuning
Soft, bear: 410M→410M / 1B→1B / 160M→410M / 410M→1B0.31 / 0.36 / 0.44 / 0.470.28 / 0.36 / 0.36 / 0.48
Soft, Pythia-160M: maple / oak / blue / purple0.32 / 0.21 / 0.14 / 0.040.29 / 0.33 / 0.18 / 0.17
Hard, Pythia-410M, SGD: dog / lionnot run0.12 / 0.10
Hard, Pythia-160M, SGD: maple / oak / blue / purple0.33 / 0.26 / 0.53 / 0.700.12 / 0.04 / 0.10 / 0.08
AuthorsOpenReview ↗

Request for additional feedback

Thank you again for taking the time to provide helpful feedback! As the deadline for the discussion period approaches, we would like to provide a gentle reminder that we have attempted to address the concerns that were raised. We would also like to ask if there are any additional concerns that went unstated in the initial review. We would welcome additional feedback and the opportunity to address it.

Thank you very much for the detailed responses and for the additional experiments, which I found helpful. I will also take into account the other reviewers’ assessments and discuss further with them before finalizing my score.

Reviewer_G5dROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors look at whether subliminal learning still happens when the teacher and student are different models.

The authors change the initialisation, training data, model size, architecture, and checkpoints. They test both hard-label and soft-label distillation. They find that transfer still happens in many settings, and the effect is much stronger with soft labels. I think the experiments are useful, but I am not convinced they explain why this happens or how much it matters in practice.

Critical strengths
  • S1: I believe that the controlled experiments on initialisation and data order are useful.

  • S2: I also like the attempt to separate hard-label and soft-label settings.

  • S3: The cross-checkpoint results are also interesting.

Critical weaknesses
  • W1: I am not convinced that the paper adds enough beyond showing that transfer can happen in more settings. Cross-model transfer of hidden signals has already been studied, even if the exact setups differ. This paper runs many comparisons, but mostly tells us where the effect appears rather than why it happens. I would have liked to see a clearer new insight.

  • W2: I find the practical relevance of the hard-label results hard to judge. Much of the detailed analysis uses Pythia-160M, and Fig. 2 shows that stronger transfer comes with more training and worse language-modeling performance. The authors also acknowledge that the measured transfer magnitudes may not occur under realistic training conditions. If the motivation is a concern about model-generated training data, then I want to know whether the effect is substantial under normal training

  • W3: The soft-label results seem stronger, but I am not sure how surprising they are. Even when the student never sees the animal name in the sampled text, the teacher's token probabilities can still carry information about that animal. For example, I see in Fig 3(b), removing animal-name tokens from the loss reduces transfer from 0.438 to 0.073 in the reported Pythia-160M top-K setting. The remaining effect is interesting, but this result leaves me wondering how much of the reported transfer is just a normal consequence of distillation.

  • W4: The paper raises concerns about hidden traits being passed through model-generated data, but it stops short of explaining what we can do with these observations.

Questions for authors

Please kindly find the comments in the weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to G5dR (1/2)

Thank you for taking the time to write a thoughtful review! Your feedback helped us identify where the original submission did not sufficiently explain what the experiments reveal about how and why subliminal learning occurs. We have reorganized our main contributions and revised the manuscript to make explanatory findings about the mechanism of subliminal learning more explicit.

W1: Novelty and mechanistic insight beyond demonstrating transfer in more settings

I am not convinced that the paper adds enough beyond showing that transfer can happen in more settings. Cross-model transfer of hidden signals has already been studied, even if the exact setups differ. This paper runs many comparisons, but mostly tells us where the effect appears rather than why it happens. I would have liked to see a clearer new insight.

We acknowledge prior evidence of cross-model transfer and revise our introduction to take this evidence into account. Prior work mostly finds limited evidence for cross-model transfer, but Draganov et al. (Phantom Transfer) and Aden-Ali et al. (Subliminal Effects) demonstrate that preferences can transfer across model families through filtered natural language text. Unlike the number-sequences setting, both settings allow for a richer range of semantic content to be transmitted across models. Schrodi et al. (Divergence Tokens) also provide a few modest cases in which cross-model transfer occurs through number sequences, but leave a systematic investigation for future work.

We demonstrate not only that cross-model subliminal learning occurs through number sequences, a more restrictive setting that limits the semantic cues that the student sees, but also systematically rule out the necessity of sharing variables such as initialization, pretraining data order, corpus, model size, and architecture for cross-model transfer. To our knowledge, no prior work isolates which differences between the two models matter.

We also investigate the mechanism of subliminal learning, showing that it can be split into two parts: the teacher’s ability to transmit preference through unrelated data and the student’s ability to acquire it. This distinction helps us understand why weak same-model transfer need not rule out cross-model subliminal learning when the model acts as the teacher but not the student and vice versa, as the limitation may involve only one side of this phenomenon. This insight also provides a way to understand why reversing teacher and student roles can dramatically change the strength of transfer.

We additionally investigate how the preference signal is transmitted. In the hard-label setting, we find that in some models, number frequencies alone can carry substantial preference information, in contrast to prior work that emphasized the importance of sequence-level effects. In the soft-label setting, we find that transfer is reduced but persists even when animal-name tokens are removed from the loss, showing that they are not necessary for transfer, although we have not established what carries the remaining signal.

W2: Practical relevance of hard-label results

I find the practical relevance of the hard-label results hard to judge. Much of the detailed analysis uses Pythia-160M, and Fig. 2 shows that stronger transfer comes with more training and worse language-modeling performance. The authors also acknowledge that the measured transfer magnitudes may not occur under realistic training conditions. If the motivation is a concern about model-generated training data, then I want to know whether the effect is substantial under normal training

We clarify that transfer is already statistically detectable at much shorter training horizons for Pythia, where language modeling is less damaged by distillation. Also, in our OLMo-1B experiments, our distillation protocol barely affects language modeling.

We agree that the strongest cross-model results that we observe in hard-label Pythia experiments should not be interpreted as the magnitude expected in practical distillation pipelines. These results often involve extended student distillation that also degrades language modeling to some extent, and we have made this limitation explicit in the Discussion.

Together, the hard-label experiments demonstrate the possibility of preference transfer in broader settings than previously investigated, ruling out the necessity of a number of shared pretraining conditions for subliminal learning. We do not claim that it effectively estimates the magnitude of transfer under these settings in ordinary training pipelines.

AuthorsOpenReview ↗

Response to G5dR (2/2)

W3: Soft-label transfer beyond ordinary distillation of animal-token probabilities

The soft-label results seem stronger, but I am not sure how surprising they are. Even when the student never sees the animal name in the sampled text, the teacher's token probabilities can still carry information about that animal. For example, I see in Fig 3(b), removing animal-name tokens from the loss reduces transfer from 0.438 to 0.073 in the reported Pythia-160M top-K setting. The remaining effect is interesting, but this result leaves me wondering how much of the reported transfer is just a normal consequence of distillation.

We agree that transfer through the teacher’s animal-token probabilities is a natural consequence of distillation, and that the fact that their removal substantially reduces transfer is therefore not, by itself, surprising. The part we wanted to highlight is how hidden the signal is even in this setup. The animal tokens are rarely among the teacher’s most likely next tokens (under 1% of training positions), the training text never names the animal, and it reads like ordinary Pile text.

W4: Actionable implications for detecting and mitigating unintended transfer

The paper raises concerns about hidden traits being passed through model-generated data, but it stops short of explaining what we can do with these observations.

Our results demonstrate that training on filtered, semantically unrelated data generated by another model, or using that model’s logits as the learning signal, can transmit unintended preferences even when models have different initializations, pretraining data, or architectures. Prior evidence of cross-model transfer in the number sequences setting was limited, but by systematically showing that these shared pretraining conditions are not necessary for subliminal learning, our experiments establish that these conditions alone cannot be treated as sufficient safeguards against unintended preference transfer.

That the mechanism of subliminal learning can be split into two parts also changes how subliminal learning should be evaluated. Weak same-model transfer does not establish that the model cannot transmit a preference or acquire one through unrelated data. Rather, the model’s limitation may lie on the teacher side, student side, or both. This motivates evaluating the teacher-student pairings used in actual distillation pipelines instead of relying on same-model tests or assuming that using different models prevents transfer.

Our synthetic number sequence experiments have implications for proposed mitigations for subliminal learning such as paraphrasing training data. Prior work finds that shuffling teacher-generated numbers (a related but distinct intervention) largely eliminates transfer, suggesting that preference signal rides on sequence-level effects. In our experiments, however, we show that in some models, substantial transfer can survive through the biased teacher’s number frequencies alone. Therefore the effectiveness of such mitigations may depend on the model, and we cannot rely on this technique to remove that signal without proper evaluation.

AuthorsOpenReview ↗

Request for additional feedback

Thank you again for taking the time to provide helpful feedback! As the deadline for the discussion period approaches, we would like to provide a gentle reminder that we have attempted to address the concerns that were raised. We would also like to ask if there are any additional concerns that went unstated in the initial review. We would welcome additional feedback and the opportunity to address it.

Thank you very much for the detailed responses. Given that all other reviewers voted for rejection, I will consider the comments of other reviewers and discuss further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their thoughtful feedback and for their time. The reviews highlighted three main concerns: that the manuscript does not provide insight into the mechanism of subliminal learning beyond demonstrating cross-model subliminal learning in a controlled setup; that evidence may not generalize beyond the main experimental setup; and that the submission does not make it easy for the reader to understand our contributions. We address the reviewers’ specific questions in the individual responses.

We have revised the manuscript to address these concerns. In particular, we have reorganized the text around five main contributions, placing more emphasis on explaining the mechanism of subliminal learning, and restructured the narrative to illustrate these clearly.

Main contributions:

  1. Subliminal learning does not require teacher and student models to share weight initialization, pretraining data order, pretraining corpus, model size, or architecture.
  2. Transmitting and acquiring a preference are distinct abilities that develop separately during pretraining. Some checkpoints are effective teachers but weak students, and vice versa.
  3. Number frequencies alone can carry much of the preference signal in some models.
  4. Restricting the loss to top-ranked tokens can increase soft-label transfer. Removing the tested trait-related tokens sharply reduces transfer but does not eliminate it in the same-model setting.
  5. Neither low-rank adapters nor adaptive optimizers are required for subliminal learning. We demonstrate transfer in Pythia-160M under full fine-tuning with plain SGD.

To make evidence of subliminal transfer easier to assess, Table 1 on page 7 of the revised manuscript now brings together the statistical evidence for hard-label transfer on Pythia, OLMo, and cross-size/architecture conditions, replacing Figure 4 from the original submission. We have also rewritten the synthetic-number experiment section to better explain its construction and interpretation directly in the main text.

Scope: preferences beyond animals, larger models, other model families, other training settings.

We replicated the headline experiments with tree (oak, maple, pine) and color (blue, purple, yellow) preferences under both hard- and soft-label distillation, demonstrating transfer across initializations, data order, corpus, model size, and model families (Appendix A.18, Figures A42–A45). These comparisons were run on OLMo-1B and larger Pythia models like Pythia-2.8B in addition to the Pythia-160M models. Our main experiments with animal preferences also use these models and Pythia-6.9B, Qwen2.5, and Qwen3.

In addition to the hard- and soft-label distillation settings, we additionally demonstrate subliminal learning with plain SGD and full fine-tuning, a novel result. Our soft-label setting also experiments with forward vs. reverse KL objectives, as well as restricting to the top-K tokens during distillation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.