- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
They study how subliminal learning works. They demonstrate that subliminal learning can happen through mechanism reuse. In the teacher, the semantic system prompt changes what attention writes into a low-dimensional subspace that the SwiGLU gates read, and patching only the gate activations recreates most of the teacher’s altered digits. In the student, they train LoRA normally, then turn the adapters on and off at specific tokens and components during generation. Keeping the adapters only at tokens present in both training and evaluation (e.g., the “Qwen” token of the system prompt, or the chat template tokens) is enough for subliminal learning to happen. Removing them only there removes it. The effect concentrates in the early-layer FFNs. They hypothesize that subliminal learning mostly biases the inputs fed into feature-selection circuits reused across many tasks. The selected features are then biased toward those of the teacher across many tasks, not just the one trained on. They create a toy model demonstrating this with a shallow transformer.
They observe that models trained through subliminal learning to love an animal sometimes name it when asked which animal they hate, which leads them to hypothesize that the salience of the animal is increased. The result is mixed: the same students often answer “Qwen” instead of an animal on the standard questions.
Their toy model trains a single-layer causal transformer on two separate tasks with disjoint tokens. The transformer is not limited in capacity, so it could learn both tasks fully independently. Instead, it reuses the same attention write subspace for both, and the FFN reads that subspace. When the arithmetic task is pretrained with semantic tokens in the same context, the subliminal effect disappears.
Critical strengths
- Claims are well supported within the cases studied, and not overclaimed (they claim sufficiency, not necessity)
- Demonstrations are well supported by controls and ablations
Critical weaknesses
Major issues
- The study is limited to a single main model (Qwen2.5-7B). Gemma 3-4B is used for the activation patching experiments and rank sweeps, and Llama-3.1-8B shows almost no subliminal learning.
- The study is limited to a single data-generation task (digits), though they use many preferences (16 animals, 5 bands, 5 trees) and 4 animals for the main localization experiments.
Minor issues
- The link between the legend of Fig 2 left and the color of the x-axis labels is not obvious and needs to be clearer
- Fig 2 right could include patching ATTN to neutral too, to be exhaustive (same for Fig 10 right)
Questions for authors
- Do you have results with other models? And more generally, how strongly have you tried red-teaming the robustness of your findings?
- Do you have supporting results using other subliminal learning datasets (not just the random digit task)?
Thanks for your review, we appreciate your support of our work! Unfortunately, we have had an unexpected and untimely issue preventing access to compute resources during the rebuttal period, so our ability to run additional experiments has been limited.
Responses to questions
What is the scope of the paper? We chose to focus on the most surprising form of subliminal learning (digits transferring preferences) that we were able to reliably reproduce on open weight models. A challenge with studying subliminal learning is that it’s quite finicky and inconsistent. We also think that there is a broad class of phenomena under the subliminal learning umbrella that, while related, are potentially distinct. Thus, the narrow scope is by design; we want to understand how data with seemingly no semantic content can still transmit hidden signals. Extending our findings to additional subliminal learning scenarios and creating a taxonomy of subliminal learning is a promising direction for future work.
Do we have results for additional models? As you mention, we do include several results for Gemma. We ran some preliminary experiments on Gemma for the dynamic weight grafting experiments, and we find largely the same pattern as the Qwen results (importance of shared tokens, transfer concentrating on the “Gemma” tokens). Concretely, for wolf preference across three seeds, Gemma shows 24.1% wolf preference. Adapters on at all token positions except the Gemma tokens gives 3.2% (necessity). Only the Gemma tokens gives 11.6% (sufficiency). We will include full results for Gemma for additional preferences in a revised version of the paper.
How much red-teaming did we do? We have included a variety of robustness checks: extensive LoRA rank and learning rate sweeps, a variety of context variants, patching the subliminal signal at a variety of token positions, etc. We tried to include as many robustness checks as we could think of, but please let us know if you notice something that we’re missing. The peer review process has also flagged additional experiments that we will include in an updated version of the paper.
Clarity. Thanks for the feedback on Figure 2. We will update this to be more clear. If we clamp attention to neutral, however, we see no signal transfer since this is the only pathway that information from the system prompt (or other shared tokens) can travel through.