- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates two questions surrounding subliminal learning: First, to what extent does subliminal transfer depend on using the same base model as both teacher and student? Second, at what point during pretraining does the ability to subliminally transfer or acquire traits emerge? The authors investigate these questions using models from the Pythia and PolyPythia families, demonstrating that subliminal learning can also occur in base models without chat-template formatting. They find that subliminal transfer works best when the teacher and student share the same base model, but can still occur between models with different pretraining data orders, weight initializations, and model sizes. Furthermore, they find that subliminal learning emerges abruptly during pretraining and can be asymmetric, with some early checkpoints being able to transfer a trait but not yet receive it.
Critical strengths
- S1: The paper uses a very nice controlled setup Pythia / PolyPythia models offer clean axes to vary pretraining data order, weight initialization, and model size (mostly) independent of each other. Using this setup allows the authors to cleanly compare same-model transfer against teacher and student models that vary on one or more of the aforementioned axes. Additionally, using the available intermediate pretraining checkpoints allows tracing the emergence of subliminal learning through pretraining.
Critical weaknesses
-
W1: The paper lacks in clarity and its writing can be hard to follow at times which is in large part due to the excessive amount of 28 Appendix references in the main text. This is especially noticeable in the results section where the supporting figures for a lot of results described in the main text are deferred to the appendix. For example, lines 236-248 describe an experiment involving synthetic number sequences which is not easy to understand from the main text alone and both experimental details as well as evidence of the claimed results are only provided in the appendix.
-
W2: All experiments only rely on a single type of transferred trait and data carrier, i.e., animal preferences via number sequences. Additionally, a subset of only five animals is considered in Figure 2 which is then inconsistently used across the following experiments. Figure 3 uses a new animal (bear) which did not appear in Figure 2 at all, Figure 4 only reports results for four out of the five animals from Figure 2, and Figure 5 again only reports a single animal. It would be good to test a larger set of animals, justify the choice of the selected subset, and indicate whether single animal results (e.g., Figure 3 & 5) are consistent across other animals.
Questions for authors
- Q1: My report performance relative to control instead of base models? (seems fine if control actually inflates animal preference but it might as well decrease it which would inflate reported transfer; the reasonable check would be base \approx control)
- Q2: Reporting transfer strength relative to control models trained on neutral teacher data could inflate observed transfer strength if the control models show lower animal preference compared to the base model. Have the authors checked if or how preferences change between the base and control models?
- Q3: In line 176-178 the on-policy distillation technique used by Lu et al. is mentioned as an alternative to the paper’s choice, but no motivation or justification for this deviation is given. Can the authors clarify why they chose one method over the other?
- Q4: How does the synthetic number sequence experiment work? Even after reading the appendix, the details are not fully clear. What are carrier numbers? Do the “real teacher completions” in Table A3 refer to the untuned or biased teacher’s completion? Most of these details are not provided in surrounding text and have to be inferred from the results itself.
- Q5: Why does the default on-policy setting use a reverse KL loss while the top-k experiments use a variant of forward KL instead?
- Q6: In Figure 4 (left), the different init and different data order setting shows 60% of student models exceeding the preference level of the control model. However, on the right side of Figure 4 in the same setting, the median transfer over the control seems to be zero. How does this work?
5 comments
Response to eU8f (2/3)
Q1–Q2: Why report transfer relative to control students instead of base model preference, and how does preference compare between them?
My report performance relative to control instead of base models? (seems fine if control actually inflates animal preference but it might as well decrease it which would inflate reported transfer; the reasonable check would be base \approx control) Reporting transfer strength relative to control models trained on neutral teacher data could inflate observed transfer strength if the control models show lower animal preference compared to the base model. Have the authors checked if or how preferences change between the base and control models?
Our experiments ask whether distilling a biased teacher’s numbers changes the student’s preference relative to distilling the untuned base model’s numbers. We use this control to account for preference changes that are a result of the number sequence distillation process.
For statistical testing, we use an exceedance statistic that compares the student’s target trait probability to that of the maximum of control runs across the hyperparameter sweep. This is a stricter criterion than a comparison to the base model’s prior, since typically this maximum control is 2.2× the prior probability. In 9% of model pairs we examined, the maximum control does fall below the base model prior. To verify that using the prior instead does not impact our results, we modify our statistical tests to compare the student’s target trait probability to the greater of the prior and the maximum of controls. We find that in spite of this, the statistical significance results and p-values do not change.
For comparisons where we measure the magnitude of preference transfer as the difference between the treatment and matched control, we find that the control can shift in both directions relative to the prior. Median control-to-prior ratios are generally close to one in OLMo, but several Pythia conditions have the median control underperforming the prior, reaching 0.38× in the worst case when both initialization and data order differ. We therefore acknowledge that using the control rather than the base model’s prior may be non-conservative in some cases. That said, the prior itself tends to be low (< 0.5%) where this ratio is low, so the difference between using the prior versus the control is small in practice.
For the soft-label setting, the choice of reference hardly matters because both the control and the model prior probability are quite small (about 0.001–0.01) relative to the magnitude of transfer demonstrated in this setting (0.27–0.5).
Q3: Rationale for the distillation method used relative to Lu et al.
In line 176-178 the on-policy distillation technique used by Lu et al. is mentioned as an alternative to the paper’s choice, but no motivation or justification for this deviation is given. Can the authors clarify why they chose one method over the other?
Both methods train on the student’s own continuations and minimise the same per-position reverse KL. We compute it over the full vocabulary (Eq. 6, Appendix A.3):
But Lu et al. score only the token the student sampled, using:
as a per-token penalty in a policy gradient update. Its expectation over is the sum above, so their method estimates from one sample what we compute exactly. This matters in our setting because every continuation that mentions the target animal is excluded from the loss. Their per-token term is therefore never evaluated on the animal’s name, while in our sum over , the animal’s token enters implicitly at every position through the teacher’s distribution. We do not quantify here how much transfer their method would give.
Response to eU8f (3/3)
Q4: How does the synthetic number sequence experiment work?
How does the synthetic number sequence experiment work? Even after reading the appendix, the details are not fully clear. What are carrier numbers? Do the “real teacher completions” in Table A3 refer to the untuned or biased teacher’s completion? Most of these details are not provided in surrounding text and have to be inferred from the results itself.
To investigate what carries the preference signal in teacher-generated number sequences, we test whether students can acquire the preference from number frequencies alone. We collect the first number from each of the 10k completions generated by the biased teacher and build a histogram of first-number frequencies. We then use completions generated by the student’s untuned base model as templates. We call each prompt together with its base-model completion a “carrier” row, and the numbers originally present in that completion “carrier numbers.” We retain the prompts, in-context examples, formatting, and sequence lengths, but replace some or all of the carrier numbers with independent samples drawn according to the biased teacher’s first-number frequencies. Each replacement number is independent of the prompt and the other sampled numbers. We refer to the resulting sequences as “resampled” sequences, and the biased teacher’s original sequences as “real” sequences. We use these real biased-teacher-generated sequences in comparisons to determine how much signal these synthetic sequences retain.
In response to your feedback, we have substantially improved the synthetic number sequences section in both the main text and the appendix to clarify the experiments and better illustrate the insights we gained from them.
Q5: Why use reverse KL by default but forward KL for top-K experiments?
Why does the default on-policy setting use a reverse KL loss while the top-k experiments use a variant of forward KL instead?
We address this in the revised manuscript: The choice of the loss and policy used in distillation has negligible effect on the transfer, because the difference in the size of the teacher and students in our case is similar or comparable. In Appendix A.10.3 (Figure A12), we cross the training texts’ policy (on or off) with the divergence (reverse or forward KL).
| Pair | Text | Transfer, reverse KL | Transfer, forward KL |
|---|---|---|---|
| 160M→160M | own samples (on-policy) | 0.235 | 0.305 |
| 160M→160M | Pile text (off-policy) | 0.193 | 0.296 |
| 160M→410M | own samples (on-policy) | 0.439 | 0.354 |
| 160M→410M | Pile text (off-policy) | 0.307 | 0.330 |
Choice of divergence: Both divergences transfer the trait in all four settings and neither is consistently larger. Top-K uses forward KL because our top-K variant keeps only the teacher’s K most probable tokens without rescaling their probabilities (Section 2.3; Appendix A.12). Unlike reverse KL, forward KL needs the teacher’s probability only on those K tokens so is defined for every K. Reverse KL, however, needs it on every token the student produces and the truncated teacher gives zero probability to every token outside its top K, so reverse KL becomes undefined (infinite).
Q6: Why can roughly 60% of runs exceed their matched controls while the median looks close to zero?
In Figure 4 (left), the different init and different data order setting shows 60% of student models exceeding the preference level of the control model. However, on the right side of Figure 4 in the same setting, the median transfer over the control seems to be zero. How does this work?
The left panel of Figure 4 from the original manuscript reports the fraction of students with a positive increase in target-trait probability relative to their matched control. The right panel reports the magnitude of those differences. A majority of small positive differences can therefore produce a positive median that appears close to zero at the figure’s scale.
We are replacing this figure with Table 1 to make the existence analysis and its statistical evidence explicit. To determine whether a shared pretraining property is necessary, we test whether transfer exists for some combination of teacher and student and whether the treatment effect is distinguishable from noise on the hyperparameter sweep. An exceedance is a treatment student whose target-animal probability is higher than the maximum control value. Because students rarely beat every control by chance, an exceedance count far above the 95th percentile of its null distribution demonstrates that transfer exists even when measures of central tendency miss this right-tail effect. We use stratified permutation tests with 20,000 permutations to test this. The new table also reports the distribution of transfer among exceedances, illustrating the magnitude of the preference transfer conditional on the student demonstrating a treatment effect that is distinguishable from the control.
Response to eU8f (1/3)
Thank you for your thoughtful review and detailed questions! Your feedback helped us identify where the presentation made our experiments and results difficult to follow, particularly the reliance on appendix references, the construction of the synthetic number sequences, and the interpretation of our controls and transfer statistics. We have revised both the main text and the appendix to explain these choices more clearly and make the evidence easier to assess. We address each of your concerns below.
W1: Improving clarity and reducing reliance on the appendix
We agree and have modified the manuscript for clarity. We have reduced the number of appendix references in the main text by 40%, elaborating on relevant results in the text. Table 1 on page 7 now brings together the statistical evidence for hard-label transfer across pretraining conditions and model families, replacing Figure 4 from the original manuscript. We have also rewritten the synthetic number sequences sections in both the main text and appendix to explain clearly how we estimate the teacher’s number distribution, construct the replacement sequences, and compare the resulting transfer with that induced by the biased teacher’s original sequences. We provide more detail about these experiments in response to Q4 below.
W2: Breadth of traits and data carriers, and consistency of animal selection
Trait generalizability concerns are addressed in the general comment, as well as in the reply to reviewer Jxoj. We provide a brief summary below.
Carrier and traits. The paper uses two kinds of training data: number sequences (hard labels) and Pile prompts with student-generated continuations (soft labels). In response to feedback on generalizability, we added tree and color preference experiments under both distillation settings (Appendix A.18). These experiments extend the evidence for transfer across differences in initialization, pretraining data, model size, and model family beyond animal preferences.
Animal Choice. We agree that the choice of animals and their overlap across settings should have been clearer. We clarify that some soft-label experiments include cat, dog, and lion, which also appear in the hard-label experiments, alongside other animals. We have updated the manuscript to illustrate this better. Using a more consistent set of animals would have made comparisons easier, although several experiments using these additional animals examine questions specific to the soft-label setting, such as Top-K truncation, removing animal tokens from the loss, comparing forward and reverse KL objectives, and varying preference placement in synthetic documents.
To address concerns on generalization beyond animal preferences and provide more consistent comparisons across settings, we added experiments with tree (oak, maple, pine) and color (purple, yellow, blue) preferences. Figures A42–A44 present hard- and soft-label results for these shared traits. We also extend the Top-K and trait-token removal experiments to these preferences in Figure A45.
As for how preferences were selected, we used an LLM to generate a list of 98 candidate animals and filtered for those that are a single token under the GPT-NeoX tokenizer (leaving 46 candidates). The animals used in the hard-label experiments were selected from animals that consistently ranked in the top 15 animals preferred by seven PolyPythia-160M base models, as evaluated on the 50 preference prompts in the training set. Some additional animals used in the soft-label setting rank lower: bear has a median rank of 18.5 and eagle 28.