- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The original subliminal learning work typically only transferred a single preference (e.g., favorite animal). This paper finds that preference for related preferences (say, green when emerald is the targeted hidden trait) also increases. In particular, this is defined by the change in log probabilities in answering some trait before and after fine-tuning in the subliminal learning setting. The authors correlate six output- and representation-based metrics with cross-trait transfer and find that some of them correlate. Finally, they show how cross-transfer changes over the course of training.
Critical strengths
- S1: Experimental results are supported across various settings (model families, trait categories, carrier formats, SFT objectives). This supports that the cross-trait transfer is not an artifact of a particular setting but a more general pattern.
Critical weaknesses
-
W1: I find the main finding (i.e., cross-trait transfer) not particularly interesting nor surprising. Specifically, it is shown that transfer is strong within clusters of similar traits (e.g., green and emerald). This is frankly quite expected, as similar traits tend to have similar representations. So, shifting towards one of them will also shift the others. More interesting findings would be (i) surprising cross-trait transfers that cannot be explained by semantic similarity or (ii) cross-trait transfer to other categories (say, yellow to make banana the students' favorite fruit). The current paper also doesn’t go beyond descriptive measures (correlations). In particular, it doesn’t provide any explanations for why cross-trait transfer happens.
-
W2: Some parts of the paper are hardly understandable without the supplemental material. For example, lambda is undefined in the main text or numbering of metrics in plots makes them also unusually hard-to-parse as human. This might be also an artifact of AI usage. To make the last point more explicit: take the title of section 4 ("metrics correlate with cross-trait transfer"). Clearly, we can find some metrics correlated with cross-trait transfer rates and this is what AIs excel at. However, what does it tell us about the mechanisms responsible for cross-trait transfer? How good are these metrics as predictive measures? Do the metrics explain anything beyond the fact that similar traits are simply similar?
Beyond that, it's also unclear how the individual findings fit together.
Questions for authors
-
Q1: Is there any particular reason the numbers are restricted to 00-99 instead of 000-999, as done in the original and follow-up works? Similarly, the number of epochs?
-
Q2: Is the distribution distillation setting just soft distillation? Or, what do you mean by “teacher’s distribution over the first complete carrier after each prompt”?
-
Q3: How much is the finding “competitor traits often gain more than the teacher trait” a function of that the teacher or competitor trait is more salient/frequency of concepts in pretraining data? For example, capitals for countries or RGB color for colors.
-
Q4: What does the cross-trait transfer results reveal that we couldn't have predicted from the pretrained models' representational structure or semantic similarity?
Comments
- C1: External/internal metrics are undefined in Fig. 1 caption and it’d be helpful to use names instead of numbers. (Only later in the paper they’re defined)
3 comments
Response (2/2): presentation and motivation of the metrics (W2, C1)
Thank you for pointing out the presentation issues in the metrics section. We agree with them and will revise Section 4 as follows (see also General Response, Part 3).
(a) The section title is not accurate. "Metrics correlate with cross-trait transfer" describes only how we scored the metrics against the transfer matrix. We'll revise it to "How well diagnostic metrics predict transfer".
(b) Our metrics are not merely correlational. Our language of "correlate" was not accurate. Prompt-vector steering (Metric 4) is an intervention: we add the activation change that the trait instruction causes in the teacher to the base model and observe which answers become more likely. Activation steering is a causal test of what a direction does to behavior (Turner et al., 2023; Rimsky et al., 2024; Chen et al., 2025), and Blank et al. (2026) show that the student in subliminal learning learns the teacher's steering vector. We'll correct this wording in Section 4 and Figure 3.
(c) The metrics cover each stage of the transmission pipeline; together they form a suite of diagnostics. Cross-trait transfer is a three-stage process (Figure 1A): (i) the trait instruction changes the teacher's activations; (ii) this shifts the statistics of the numbers it generates; (iii) the student fitting those outputs can amplify multiple answer traits. Each metric relates the teacher's signal at one stage to a given answer trait:
- Stage (ii), the numbers. Carrier-distribution similarity (Metric 1): do two traits shift the number distribution alike? If so, the student gets a similar training signal from both.
- From (ii) to (iii), numbers to answers. The entangled-carrier metrics (Metrics 2, 3 and 6): are the numbers that the teacher's trait promotes also the numbers tied to a given answer?
- From (i) to (iii), activations to answers. Prompt-vector steering (Metric 4) is the intervention in (b). Prompt-target trait vector similarity (Metric 5) asks: is this activation change aligned with a vector that elicits the answer?
Together they explain why cross-trait transfer happens: a trait creates an activation direction in the teacher, the direction leaves a trace in the output numbers, and a student fitted to that trace moves toward the cluster of related answers that this direction promotes.
(d) The metrics inform prediction and monitoring of transfer. All six metrics are computed from the teacher and the base model, before any student is trained.
- Prediction. For held-out traits, adding Metric 4 to semantic similarity and salience improves held-out in all six conditions (color SFT: 0.059 to 0.285); Metric 1 improves it in four of six (Part 1(b); Appendix F).
- Monitoring. For misaligned teachers, steering effects computed from teacher-minus-base activations agree with the students' actual shifts at . Metric 4 can therefore help diagnose which unintended behaviors a dataset is likely to shift, i.e. where to audit (Part 2(b); Appendix H).
So the metrics explain more than "similar traits are similar": they predict transfer where semantic similarity and salience fall short.
(e) Revisions to figures, formulas and text.
- Names instead of numbers (C1). Descriptive names such as "carrier-distribution similarity" and "prompt-vector steering" will replace "Metric 1" and "Metric 4" in text and figures.
- Figures. The Figure 1 caption will define external metrics (computed from the teacher's output distribution) and internal metrics (computed from activations), and panel A will mark stages (i) to (iii) and where each metric is measured. Figure 3 will group the metrics by stage.
- Formulas. Each metric will be introduced by the question it answers, followed by its formula. We'll define the notation in the main paper, including , which is now defined only in Appendix B.1.
- Introduction. It will state the contributions up front (Part 3(a)).
References
- Turner et al. (2023). Steering Language Models with Activation Engineering. arXiv:2308.10248.
- Rimsky et al. (2024). Steering Llama 2 via Contrastive Activation Addition. ACL.
- Chen et al. (2025). Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv:2507.21509.
- Blank et al. (2026). Subliminal Learning Is Steering Vector Distillation. arXiv:2606.00995.
I thank the authors for the thorough feedback. In particular, I find the cross-category results interesting (I think leading the paper with that result would make the paper stronger). Since this (and the other) results resolve some of my major concerns, I increase my score and recommend this paper for acceptance.
Response (1/2): novelty (W1, Q3, Q4); technical details (Q1, Q2)
Thank you for the detailed review. We agree that semantic similarity explains the coarse clustering, and that the submission did not make clear what goes beyond it. We now report the two findings you asked for, transfer that semantic similarity cannot explain and transfer to other categories, together with a safety implication. The full tables are in the General Response (Parts 1 and 2).
Novelty (W1, Q3, Q4)
(1) Transfer that semantic similarity cannot explain.
On Q4: semantic similarity alone predicts little of the transfer for held-out traits (held-out R2 at most 0.096). Transfer is better predicted by a signal from the transmission itself, the activation change that the trait instruction causes in the teacher (Metric 4).
(2) Cross-category transfer. Following your suggestion, we asked the 30 students of color teachers for their favorite fruit. In sampled answers, strawberry rises from 13.50% to 83.75% for the red student, and pineapple from 0.25% to 52.00% for the yellow student. Over all 900 color-fruit pairs, transfer correlates with the predefined color-fruit association matrix at r=0.445 (Part 1(c); Appendix G). The cross-trait transfer in subliminal learning is therefore not confined to the category of its teacher's trait.
(3) Practical safety implication. Our study also has a safety implication, which we'll bring into the main text (Part 2). Three misaligned teachers from Turner et al. (2025; arXiv:2506.11613), trained separately on risky financial, bad medical, and reckless sports advice, generate only number sequences, and we fine-tune a fresh student on each.
Revision. We'll foreground (1) and (2) as contributions in the introduction and highlight the practical safety implication as a significance of our findings (Part 3(a)). Why cross-trait transfer happens is answered in our next post, together with W2.
Technical details (Q1, Q2)
Q1. We chose 00–99 for computational efficiency. Our distribution-distillation objective and output-distribution metrics require explicitly scoring the carrier distribution. Qwen2.5 tokenizes numbers digit-wise: with shared prefixes, scoring all two-digit strings takes 1+10=11 probes of next-token distributions, against 1+10+100=111 for all three-digit strings. We use three epochs, following the released Qwen2.5-7B-Instruct configuration of the original subliminal learning work (n_epochs=3).
Q2. Yes, it is soft distillation, applied to a finite set of complete carrier strings. "First complete carrier" means one entire carrier immediately following the prompt. For numbers, this is a two-digit string such as 37, whose probability is P(3∣x)P(7∣x,3). We score all 100 candidates (00 to 99), normalize their probabilities, and train the student to match the teacher's distribution with cross-entropy, which is equivalent to minimizing the teacher-to-student KL divergence.