acceptodds
Under review as a conference paper at ICLR 2027

Subliminal Learning Through Mechanism Reuse: How Models Can Transmit Hidden Traits In Unrelated Data

Abstract

Subliminal learning is a phenomenon where language models can transmit behavioral traits to other models through seemingly innocuous data cloud2025subliminal. In subliminal learning, a teacher model with a behavioral trait (e.g. obsession with cats) can transmit this cat obsession to a student model finetuned only on numerical sequences generated by the teacher. In this paper, we ask: how does this unexpected behavioral transmission occur? We examine subliminal learning in open weight models, finding that the effect is highly dependent on the context seen during finetuning and evaluation. For example, a Qwen model with the default system prompt during finetuning (“You are Qwen, created by Alibaba Cloud. You are a helpful assistant.”) does not show subliminal learning during generation when no system prompt is included. We further demonstrate that subliminal behavior is localized to computation at tokens seen during both finetuning and evaluation (e.g. the model's default system prompt, the standard chat template tokens, etc.). We also examine the data generation process to understand how behavioral signals are transmitted through sequences of digits, localizing the subliminal signal to an attention-mediated pathway. Finally, we construct a simple model, showing that subliminal learning can occur with no data correlation, capacity restrictions, regularization, or unembedding entanglement; reusing mechanisms across unrelated tasks is sufficient for subliminal learning.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, subliminal learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_68hTOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

They study how subliminal learning works. They demonstrate that subliminal learning can happen through mechanism reuse. In the teacher, the semantic system prompt changes what attention writes into a low-dimensional subspace that the SwiGLU gates read, and patching only the gate activations recreates most of the teacher’s altered digits. In the student, they train LoRA normally, then turn the adapters on and off at specific tokens and components during generation. Keeping the adapters only at tokens present in both training and evaluation (e.g., the “Qwen” token of the system prompt, or the chat template tokens) is enough for subliminal learning to happen. Removing them only there removes it. The effect concentrates in the early-layer FFNs. They hypothesize that subliminal learning mostly biases the inputs fed into feature-selection circuits reused across many tasks. The selected features are then biased toward those of the teacher across many tasks, not just the one trained on. They create a toy model demonstrating this with a shallow transformer.

They observe that models trained through subliminal learning to love an animal sometimes name it when asked which animal they hate, which leads them to hypothesize that the salience of the animal is increased. The result is mixed: the same students often answer “Qwen” instead of an animal on the standard questions.

Their toy model trains a single-layer causal transformer on two separate tasks with disjoint tokens. The transformer is not limited in capacity, so it could learn both tasks fully independently. Instead, it reuses the same attention write subspace for both, and the FFN reads that subspace. When the arithmetic task is pretrained with semantic tokens in the same context, the subliminal effect disappears.

Critical strengths
  • Claims are well supported within the cases studied, and not overclaimed (they claim sufficiency, not necessity)
  • Demonstrations are well supported by controls and ablations
Critical weaknesses

Major issues

  • The study is limited to a single main model (Qwen2.5-7B). Gemma 3-4B is used for the activation patching experiments and rank sweeps, and Llama-3.1-8B shows almost no subliminal learning.
  • The study is limited to a single data-generation task (digits), though they use many preferences (16 animals, 5 bands, 5 trees) and 4 animals for the main localization experiments.

Minor issues

  • The link between the legend of Fig 2 left and the color of the x-axis labels is not obvious and needs to be clearer
  • Fig 2 right could include patching ATTN to neutral too, to be exhaustive (same for Fig 10 right)
Questions for authors
  • Do you have results with other models? And more generally, how strongly have you tried red-teaming the robustness of your findings?
  • Do you have supporting results using other subliminal learning datasets (not just the random digit task)?
1 comment

Thanks for your review, we appreciate your support of our work! Unfortunately, we have had an unexpected and untimely issue preventing access to compute resources during the rebuttal period, so our ability to run additional experiments has been limited.

Responses to questions

  • What is the scope of the paper? We chose to focus on the most surprising form of subliminal learning (digits transferring preferences) that we were able to reliably reproduce on open weight models. A challenge with studying subliminal learning is that it’s quite finicky and inconsistent. We also think that there is a broad class of phenomena under the subliminal learning umbrella that, while related, are potentially distinct. Thus, the narrow scope is by design; we want to understand how data with seemingly no semantic content can still transmit hidden signals. Extending our findings to additional subliminal learning scenarios and creating a taxonomy of subliminal learning is a promising direction for future work.

  • Do we have results for additional models? As you mention, we do include several results for Gemma. We ran some preliminary experiments on Gemma for the dynamic weight grafting experiments, and we find largely the same pattern as the Qwen results (importance of shared tokens, transfer concentrating on the “Gemma” tokens). Concretely, for wolf preference across three seeds, Gemma shows 24.1% wolf preference. Adapters on at all token positions except the Gemma tokens gives 3.2% (necessity). Only the Gemma tokens gives 11.6% (sufficiency). We will include full results for Gemma for additional preferences in a revised version of the paper.

  • How much red-teaming did we do? We have included a variety of robustness checks: extensive LoRA rank and learning rate sweeps, a variety of context variants, patching the subliminal signal at a variety of token positions, etc. We tried to include as many robustness checks as we could think of, but please let us know if you notice something that we’re missing. The peer review process has also flagged additional experiments that we will include in an updated version of the paper.

  • Clarity. Thanks for the feedback on Figure 2. We will update this to be more clear. If we clamp attention to neutral, however, we see no signal transfer since this is the only pathway that information from the system prompt (or other shared tokens) can travel through.

Reviewer_RVMsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the mechanisms underlying subliminal learning in open-weight language models. The authors show that the effect of subliminal learning is quite sensitive to mismatches in context seen during behavior instillation and evaluation and that computations sufficient for reproducing the effect occur predominantly at tokens that are shared between fine-tuning and evaluation. The paper further shows that subliminal learning can be caused by attention modules writing to a low-rank subspace that is shared across unrelated tasks using both activation interventions inside open-weight LLMs as well as in a reproduction inside a controlled toy model.

Critical strengths
  • S1: The paper does a good job of more precisely localizing the relevant components and computations influencing subliminal learning. The shared attention subspace mechanism the paper proposes is quite interesting and well supported by both activation patching results in open-weight models as well as by the controlled toy model setup. The importance of the computations on the entity token or other shared tokens between fine-tuning and evaluation provides useful insight, but currently relies on results from a single model (see W1).
  • S2: The student reusing teacher mechanisms to mimic the teacher-generated number sequences is an intuitive explanation for how subliminal learning can occur. The paper does a good job showing that mechanism reuse can be one explanation for subliminal learning by patching student activations back into a teacher model. They further support this by showing that mechanism reuse is sufficient inside a toy model of subliminal learning.
Critical weaknesses
  • W1: A lot of experiments only use a single model and a handful of animals. For example, attention subspace, shared fine-tuning / evaluation context matching, dynamic weight grafting, and student reuse of teacher mechanisms experiments all seem to only use Qwen and only up to 5 animals. It would be good to reproduce most of the results also on at least one other model. Especially, for the system prompt entity and shared context experiments, it would be good to have results for Gemma since it by default doesn’t use a system prompt and [1] has shown the Qwen can behave weird with respect to the entity inside the system prompt.

  • W2: The paper could be clearer about certain experimental setups and results. For example, in Figure 2 it is not clear which animal is used for the semantic teacher, or in Section 4.3 I find it quite difficult to understand which percentage figures relate to which other numbers just from the text while the supporting figures also don’t help that much (and they are also deferred to the appendix).


[1] https://arxiv.org/abs/2509.23886

Questions for authors
  • Q1: For Table 1, have the authors checked whether entity mismatches potentially coincide with token mismatches (e.g., ChatGPT being two tokens vs. Qwen being one token)? If so, does the mismatch still have the same effect when controlling for the number of entity tokens?
  • Q2: Did the authors test whether no chat template fine-tuning can still produce subliminal learning if the biased teacher also doesn’t use a chat template, e.g., by replacing special chat template tokens by plain language (User: … / Assistant: …)?
  • Q3: Figure 4 (right) shows that logit changes after invariance training are much lower than before. Since the teacher generated data only really provides a training signal if adding semantic tokens changes the predictions on the arithmetic task, doesn’t this intervention just lead to a dataset where each answer is just the arithmetically correct answer? If so, then the observed decrease in logit changes might just be attributable to the training signal encouraging predictions the student would have made anyway.
2 comments

Thank you for your engagement, your detailed questions, and for your support of our work. Also, we unexpectedly (and temporarily) lost access to compute resources during the rebuttal period, so our ability to run additional experiments has been limited.

Responses to specific questions

  • Does Gemma behave similarly or is Qwen just weird? You’re right that several experiments only use Qwen. For our Gemma experiments, we use a prepended system prompt (“You are Gemma, made by Google. You are a helpful assistant.”). But, we might expect that Gemma is less likely to anchor on an entity here (like in the Schrodi results you mention). We have preliminary results for Gemma, showing largely the same pattern as the Qwen results (importance of shared tokens, transfer concentrating on the “Gemma” tokens). Concretely, for wolf preference across three seeds, Gemma shows 24.1% wolf preference. Adapters on at all token positions except the Gemma tokens shows 3.2% wolf preference (necessity). Only the Gemma tokens shows 11.6% wolf preference (sufficiency, although not quite half of the effect). We will include full results for Gemma for additional preferences in a revised version of the paper.

  • What if the teacher doesn’t use a chat template? We didn’t test this exact setting, but we found that simply prepending the shared context to the user prompt works well for subliminal learning (Appendix B8). Using only the context for both is an interesting additional experiment.

  • Does different tokenization confound the entity results? Differing entity tokenizations is a confound in the entity mismatch experiments, since ChatGPT (3 tokens), Qwen (2 tokens), and Claude (1 token) all tokenize differently. That said, we do see (Appendix B13) that the signal is not position dependent; we can patch the MLP down-projection matrix from the “wen” token in Qwen at most token positions in a ChatGPT context and still see preference transfer.

  • What does invariance training in the toy example show? You are correct that if both models simply get the arithmetic questions correct, there would be no signal for subliminal learning. The goal of the experiment was to show that models are not invariant to semantically unrelated tokens by default, but that training data can force this kind of invariance. An additional question is whether an invariance trained student is immune to subliminal learning. We tested this by checkpointing an entangled teacher, training on mixed data to get an invariant student, then training this invariant student on the teacher’s entangled sequences. We see that the invariant student shows roughly half the target logit increase as the original student, but still shows a positive signal for most animals. Adding an additional invariance task further reduced the logit increase to 37% of the original student. So, even though invariance training can change model behavior, it still retains signatures of its training trajectory. We will include these results in the paper, and update the writing to clarify the original experiment.

  • Clarity. Thanks for the useful feedback, we will update these figures to be more clear. (Figure 2 is for “cat”, we moved the figure for Section 4.3 to the appendix due to space constraints, but this should be moved back to the main body)

Thank you for the detailed response and the additional experiments. The added Gemma system prompt result weakens my concern about generality of the finding beyond Qwen. The toy-model follow-up addresses my soundness concern about the invariance training. My main remaining concern is the scope of the evidence. Several mechanistic results still rely on Qwen, and the entity-mismatch experiment leaves entity identity confounded with token count. I remain positive about the paper and maintain my rating of 3.

Reviewer_taYEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how subliminal learning works in open-weight models, where a student finetuned on digit sequence generated from a trait prompted teacher obtains the same trait. The authors first apply activation patching between the trait-prompted and neutral runs and show that the altered digit are carried mainly by gate activation. Moreover, the paper also show that the transfer depends heavily on the contexts used during finetuning and evaluation. Finally, the authors design a toy model with one-layer transformer indicating training arithmetic with semantic tokens removes the transfer.

Critical strengths
  1. The paper is well-written and very easy to follow.
  2. The teacher-side patching experiments convincingly show that the change in the teacher's digit predictions is carried by the activation gates.
  3. The finding that transfer depends on using the same context during finetuning and evaluation is useful, and large rank × learning-rate sweeps back it up.
Critical weaknesses
  1. The evidence that students reuse the teacher's mechanism is thin for the paper's central claim. The authors conclude that students reuse the teacher's mechanism (Section 4.3), but each piece of evidence supports less than this conclusion, which leaves clear gaps in the argument. First, it is not surprising that the student agrees with the teacher's digits, since that is what it was trained to do, and there is no baseline such as a control student finetuned on numbers from the neutral teacher. Second, patching all of the student's gates into a neutral run is close to a complete readout of the model, so it would likely reproduce the changes of any finetuned model.

  2. The joint-training experiment in Section 5.2 is a negative control whose outcome is expected regardless of how subliminal learning works. Training arithmetic under every semantic prompt with the same targets makes the teacher's digits independent of the semantic token. The teacher then just outputs the modular sum, and the student, which starts from the same checkpoint, is trained on its own output distribution, so there is no signal to transmit. Moreover, the claim that the model learns to separate the task representations is never measured.

  3. The experiments cover a narrow range of traits and contexts, so it is unclear how far the conclusions generalize. All traits are one-word preferences, mostly animals. Other kinds of traits, such as misalignment, and carrier data other than numbers, such as code or reasoning traces, are not tested. The context experiments also compare only very different system prompts (Qwen, ChatGPT, empty), which change the entity and the wording at once. Evaluating with prompts close to the training prompt, such as paraphrases, small edits or only a different entity name, would show whether the effect depends on the exact tokens, the entity or the meaning, and how fragile it is in practice.

Questions for authors
  1. How does a control student, finetuned on numbers from the neutral teacher, score on the section 4.3 metrics (argmax agreement, gate-patch recovery)? How often does the neutral model agree with the teacher across all digit positions?
  2. How similar are the student's and the teacher's gate changes? Does patching the student's gates recover the preference on the favorite-animal evaluation?
4 comments

Thanks for your review and the detailed questions. Unfortunately, we temporarily lost access to our compute resources during the rebuttal period, so our ability to run additional experiments has been limited.

Responses to questions

  • What is the scope of the paper? We purposefully scope the paper only to the most surprising form of subliminal learning: preference transfer through semantically unrelated data. While there are other forms of subliminal learning, we actually think that these might all be distinct (but related) phenomena. An additional challenge is that subliminal learning is unreliable in open weight models, so we focused on settings where transfer was consistent (predominantly animal preferences transferring via numbers). We agree that additional experiments on other kinds of transfer data would be a good robustness check for some of our findings, and developing a taxonomy of subliminal learning is a promising direction for future work.

  • Do we show that subliminal learning is mechanism reuse in real models? At a high level, the point that we are trying to make is that i) we can localize the subliminal signal in the teacher and the student model; ii) in a toy setting, we can see similar behavior without correlations in the training data; mechanism reuse is sufficient to transfer semantic traits; iii) the behavior in large models is consistent with this kind of mechanism reuse. We do not claim that mechanism reuse is the whole story, since we think it’s likely that multiple things can cause this kind of unexpected transfer. We will update the writing to make this more clear.

  • Does altered phrasing confound our altered context experiments? The experiments in Table 1 use nearly the same system prompt, replacing, for example, “Qwen” with “ChatGPT” and “Alibaba Cloud” with “OpenAI”. The wording is the same other than those entity names. We also show that we can patch the subliminal effect using only the down projection activations patched from a Qwen prompt to a ChatGPT prompt in Appendix B13 at nearly any token position, showing that the signal, once inserted, is not dependent on minor wording differences.

  • Neutral baseline. We agree that a student trained on digits from a neutral teacher (and the argmax agreement and gate-patch recovery) is an important baseline. We will update the paper to include this (we have not been able to run this experiment yet).

  • What does gate patching sufficiency in the student show? The concern is that patching gates is a strong intervention that doesn’t tell us much. In Section 4.3, we think that the key idea is actually the patching to the attention subspace, since our proposed mechanism is attention writing to a space that is picked up by MLP gates. But your point is taken about gate sufficiency (although we don’t think it’s a priori obvious that gate activations would be more important than other components, like up projections; see Figure 2). We can soften the wording here to avoid overclaiming and update the writing to focus on the attention. We will also run the experiments patching the student gate activations and testing preference transfer.

  • How similar are the gates in the student and teacher models? For similarity, do you mean cosine similarity between the gate parameters for the teacher and the student model? We can report this, or did you have something specific in mind? A concern is that it’s not clear that a dot product between model parameters is the correct notion of similarity (a notion of similarity should probably be related to how model parameters change output behavior).

  • What does invariance training in the toy example show? You are correct that if both models simply get the arithmetic questions correct, there would be no signal for subliminal learning. The goal of the experiment is to show that models are not invariant to semantically unrelated tokens by default, but that training data can force this kind of invariance. An additional question is whether an invariance trained student is immune to subliminal learning. We tested this by checkpointing an entangled teacher, training on mixed data to get an invariant student, then training this invariant student on the teacher’s entangled sequences. We see that the invariant student shows roughly half the target logit increase as the original student, but still shows a positive signal for most animals. Adding an additional invariance task further reduced the logit increase to 37% of the original student. So, even though invariance training can change model behavior, it still retains signatures of its training trajectory. We will include these results in the paper, and update the writing to clarify the original experiment.

  • Does the toy example show separation of task representations? We show attention-write subspace overlap in the toy models in Appendix C1, but we don’t show how the invariance-trained model differs. We see that the attention-write space overlap is reduced, but still present in the invariance-trained model, implying that the behavioral difference comes from the MLP learning to separate the signals. We will update the writing to include these results and to avoid claiming disentanglement.

Neutral control: We were also able to run the neutral baseline experiments. We use rank 8 LoRA on Qwen2.5-7B-Instruct across 3 data generation seeds, with one student finetuned on a teacher with cat preferences and another student finetuned on a teacher with the default Qwen system prompt. At divergence positions for the cat teacher, we see that the control agrees with 20.9% of argmaxes and the cat student agrees with 43.5% of argmaxes (so, about half of this effect is generic number finetuning). If we patch control gates, we recover 20.4% of argmaxes and subliminal gates recover 48.8% of argmaxes (up projections recover 19.7% and 39.3%, respectively).

Student Gate Use: We also patch the student's gates and up projections into a neutral model while asking about animal preferences, finding that gates at the final token recover 100% ± 4% of the subliminal student's cat preference gain and up projections recover 23% ± 19% (consistent with the claim that the student expresses the altered digit distribution and animal preference through gates).

Thank you to the authors for the detailed rebuttal. The response addresses most of my concerns, so I will maintain my positive rating of weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.