acceptodds
Under review as a conference paper at ICLR 2027

WHY SUBLIMINAL LEARNING NEEDS SO MUCH DATA: A NOISY INVERSE VIEW THROUGH STEERING VECTOR RECOVERY

Abstract

Subliminal learning allows a student model to inherit a teacher’s behavioral trait from semantically unrelated generated data, but existing demonstrations often require tens of thousands of independent carrier examples. We ask whether this scale is needed because small datasets fail to cover the trait signal, or because sampled tokens are hard labels and provide a noisy representation of that signal. Using subliminal steering as a controlled testbed, we represent the teacher trait by a known vector \(\Delta_T\), recover a student vector \(\Delta_S\), and calculate their cosine similarity to measure subliminal transfer effect. To isolate the effect of hard-label noise, we compare token-level NLL supervision with full-distribution KL supervision on matched carrier prefixes. The results create a two-sided contradiction for the two candidate explanations. With sufficiently many carriers, the hard- and soft-supervised mean gradients are nearly collinear and have the same low alignment with . From this gradient analysis, hard-label noise appears to be readily averaged out and therefore should not affect the final subliminal transfer. Under iterative optimization, however, soft supervision recovers almost perfectly from only hundreds of examples, whereas hard supervision remains substantially limited even with tens of thousands. The large gap between their endpoints shows that hard-label noise is not average out, and nevertheless decisive for final recovery. At the same time, the success of small-data soft supervision rules out insufficient carrier coverage as the primary reason that hard-label learning requires large datasets. Thus noise appears negligible in the local gradient but consequential over optimization, while the carrier already contains enough information for recovery at a much smaller scale. We explain this discrepancy by formulating subliminal learning as a noisy inverse problem. Locally, the carrier task Fisher matrix transforms the teacher trait direction, and the carrier learning gradient for becomes: \(F(\Delta_T-\Delta_S)\), causing hard and soft gradients to share the same distorted direction, which explains the poor alignment with regardless of carrier data size. Iterative optimization progressively inverts this transformation, allowing soft supervision to recover the teacher, but simultaneously amplifying the small residual error in hard supervision. Thus large carrier datasets are not primarily required to fully catch the trait signal; they reduce discretization error amplified by Fisher inversion.

Primary area:
interpretability and explainable AI
Keywords:
subliminal learning, statistical models, steering vectors

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3MnGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the data requirements of subliminal learning in a controlled steering-vector recovery setting. With the base model frozen, a student learns a residual-stream vector from number sequences generated by a steered teacher. Matching the teacher's full output distribution yields substantially better recovery than training on sampled tokens, despite similar initial gradient directions. The paper explains this through a local noisy inverse model: carrier Fisher geometry transforms the teacher signal, and optimization progressively reverses this transformation while amplifying hard-label sampling error. Experiments examine spectral recovery, data-dependent stopping, and repeated sampling from a fixed prompt pool.

Critical strengths
  1. I like the matched-prefix comparison. Holding the carrier prefixes fixed while changing the supervision provides a useful way to separate insufficient prefix coverage from target-sampling error. The large difference in final recovery, despite similar initial gradients, is worth investigating.

  2. The known teacher vector makes the explanation unusually testable. Figure 3 checks recovery in the Fisher eigenbasis rather than relying only on an aggregate behavioral score. The agreement between the observed and predicted trajectories is encouraging.

  3. The fixed-prompt resampling experiment is a useful addition. It goes beyond showing that more data helps and asks whether the benefit requires more distinct prompts. The distinction between generating new completions and repeatedly training on old completions is practically relevant.

Critical weaknesses
  1. I am not convinced that the theoretical increment is as substantial as the main contribution list makes it sound. The recovery platform is already present in [1], including the simplified SGD vector-recovery experiment in Appendix A.6, and [2] already studies weak steering-aligned gradients and the importance of optimization. More directly, Section 2 of [3] gives the subliminal-learning relation d0≈FΔθTd_0 \approx F\Delta\theta_T, the softmax Fisher form, and the alignment cos⁡(ΔθT,FΔθT)\cos(\Delta\theta_T,F\Delta\theta_T). This is very close to the present Section 4.1, beyond the change in parameterization. Fisher-based activation steering and regularized inverse-Fisher directions also appear in [4]. Neither [3] nor [4] is discussed. Finally, the spectral bias-variance expression in Equation (14) closely matches the gradient-flow risk decomposition in [5], while soft-label variance reduction is established in [6]. I appreciate the acknowledgment of classical iterative regularization in Appendix B, but the main text still needs a sharper account of what is new. The contribution I can identify is the specific noise explanation and its empirical validation, rather than a new Fisher or inversion theory.

  2. I found Section 4.1 unnecessarily drawn out. The softmax chain rule, adding and subtracting the teacher probabilities, the expectation of a one-hot sample, and the successive Taylor expansions occupy more space than their technical content warrants. The section is needed as a bridge to the later analysis, so I would not remove it entirely. A compact statement of the Fisher definition, the hard/soft signal-noise decomposition, and its interpretation would do the job; the intermediate algebra belongs in the appendix. I would much rather see the recovered space used to explain the difference from [3] or strengthen the noise diagnostics. This is primarily a presentation concern, but it also makes the limited theoretical novelty harder to assess.

  3. The step from a local approximation to the explanation of late-stage recovery is where I still have reservations. The model treats both the Fisher and the sampling-error term as fixed, yet the central argument concerns increasingly weak directions late in training. Those are also directions in which linearization error or changes in the Fisher could matter disproportionately. Figure 3 supports the approximation, and Figure 4 shows the expected stopping-time trend, but these do not quantitatively establish the proposed sampling-noise scaling. I would find the explanation more convincing with a direct check of the noise covariance or the finite-step spectral variance in Equation (14), including its dependence on sample count. Independently resampling targets at fixed token prefixes would be a particularly clean test. Resampling whole completions from fixed initial prompts, as in Figure 4, also changes the subsequent prefixes.

  4. I am also hesitant about how much the controlled result tells us about the data requirements of subliminal learning more generally. The teacher and student share a frozen base model and the same intervention parameterization; only one residual vector is recovered, using number-sequence carriers and animal preferences. The teacher intervention is exactly representable by the student, making this a particularly favorable recovery setting. Appendix D acknowledges the scope, which I appreciate. Still, with the theoretical ingredients largely established, I would like the empirical contribution to reach beyond this setting, or the main claims to be framed more explicitly around it.

[1] Subliminal Steering: Stronger Encoding of Hidden Signals. arXiv preprint, 2026.

[2] Subliminal Learning Is Steering Vector Distillation. ICML Mechanistic Interpretability Workshop, 2026.

[3] Channel Location Constrains the Auditability of Subliminal Learning. arXiv preprint, 2026.

[4] FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers. arXiv preprint, 2026.

[5] A Continuous-Time View of Early Stopping for Least Squares Regression. AISTATS, 2019.

[6] A Statistical Perspective on Distillation. ICML, 2021.

Questions for authors
  1. What is the precise advance of Section 4.1 over the channel-Fisher relation in [3], and how does the correction experiment relate to the inverse-Fisher steering construction in [4]?

  2. Can the authors directly test the predicted noise covariance or spectral error scaling by varying the number of independent target draws at fixed token prefixes?

  3. How stable are the relevant Fisher eigendirections along the recovery trajectory, particularly the low-curvature directions responsible for the claimed late-stage noise amplification?

8 comments
AuthorsOpenReview ↗

Rebuttal - Responses Overview first half (1/6)

Responses Overview first half

We sincerely thanks reviewer 3MnG, for his effort on providing valuable and thoughtful feedbacks on our paper, to help us improve the paper quality. We address the raised concerns in multiple blocks. This is an overview, and with more detailed and specific responses to some concerns are listed in blocks afterwards.

W1 / Q1 — Distinguishing established knowledges/understandings from our contribution. We agree that several mathematical ingredients build on prior work. However, we believe the review understates the advance obtained by connecting these ingredients to the data requirements of iterative subliminal transfer. Our contribution is a theoretically grounded, experimentally testable account of how sample count affects recovery, why similar initial alignment can lead to different recovery outcomes, and how reducing sampling noise improves transfer from a small carrier pool.

Relation to [1] and [2]. We explicitly cite [1] as the source of our recovery testbed. Its restricted parameterization makes the teacher signal known and allows direct comparison between theoretical predictions and measured recovery trajectories. The testbed itself is not listed as our contribution. We also discuss [2] in related work. Its steering-distillation and gradient analyses help explain how subliminal transfer occurs. Our central question concerns the dependence of recovery on data quantity and supervision noise once transfer is possible.

Relation to [3]. We acknowledge that the first-order relation $ d_0 \approx F\Delta_T$, and hence the initial alignment cos⁡(ΔT,FΔT)\cos(\Delta_T,F\Delta_T), overlaps with [3]. We were unaware of this recent preprint when preparing the submission and will explicitly discuss it.

The conclusions developed from this shared starting point differ. In its controlled body-channel setting, [3] finds that initial alignment predicts subsequent behavioral transfer. Our analysis follows the entire recovery trajectory and shows how recovery depends jointly on the Fisher eigenvalues, training depth, sample count, and the teacher signal’s spectral distribution in particular. The initial cosine overlooked those information and compressed into an insufficient single scalar.

Concretely, writing δi=⟨ΔT,ui⟩\delta_i=\langle\Delta_T,u_i\rangle, our local model gives the finite-step recovery error along a Fisher eigenvector as

$ \mathbb{E} \left[(\widehat{\delta}_i^{(k)}-\delta_i)^2\right] \approx (1-\eta\lambda_i)^{2k}\delta_i^2 + \frac{\left[1-(1-\eta\lambda_i)^k\right]^2}{N\lambda_i}. $

This identifies both slower recovery (small λ\lambda) and stronger eventual noise amplification in low-curvature directions. Consequently, similar initial cosine alignment can coexist with substantially different iterative recovery. Figure 1 demonstrates this divergence between Qwen and Gemma. Section 6.2 and Fig. 5 further show that Gemma’s teacher signal is concentrated at lower normalized curvature, consistent with its weaker recovery. These observations motivate a spectral account of recovery and its data dependence beyond the initial-alignment statistic: using more data to compensate the recovery under lower curvature regions.

Relation to [4]. FishBack uses pullback Fisher geometry to construct a concept-steering direction with minimal local output distortion. Our inverse-Fisher diagnostic uses that geometry to undo the carrier-induced transformation of a distillation signal and recover the known teacher direction. Our subsequent analysis explains how iterative inversion also amplifies hard-label error and creates a sample-dependent stopping depth. Thus, the shared Fisher geometry serves distinct objectives and supports different conclusions. We will clarify this relationship and cite [4].

Relation to [5] and [6]. Both works are already cited as foundations for the iterative-regularization and variance-reduction arguments. Our contribution lies in connecting these results to the observed subliminal-recovery phenomenon and testing their consequences. In particular, the new fixed-prefix experiment holds every carrier input constant and shows approximately fourfold lower recovery variance for each fourfold increase in teacher draws (Appendix H, Fig. 9).

We will sharpen the contribution statement accordingly. The paper develops and validates a noise-based explanation of data-dependent transfer within a controlled subliminal-steering setting.

AuthorsOpenReview ↗

Rebuttal - Responses Overview second half (2/6)

Responses Overview second half

W2 — Presentation of Section 4.1. We agree that the intermediate algebra can be shortened. We will retain the Fisher definition, the hard/soft signal–noise decomposition, and their interpretation in the main text, and move the intermediate softmax and Taylor-expansion steps to the appendix. This will make room for the comparison with [3] and the new noise diagnostics.

W3 / Q2 — Direct validation of sampling-noise scaling. Our new fixed-prefix experiment confirms that each fourfold increase in independent teacher draws reduces recovery variance by approximately fourfold, while the spectral variance increases with training depth (Appendix H, Fig. 9).

We fix all supervised token prefixes from 50 carrier sequences and independently sample RR next-token labels at each position. Training averages their negative log-likelihoods. The labels are sampled once and reused throughout optimization, and every input prefix remains fixed. Appendix E (Fig. 6) describes this protocol: supervision with empirical token frequencies.

Across eight paired label seeds, the normalized recovery variances at step 500 are 1.2594, 0.2997, 0.0748, and 0.0190 for R=4,16,64,256R=4,16,64,256, respectively. This closely follows the predicted 1/R1/R dependence. Meanwhile, the estimated squared bias stabilizes near the soft-supervision residual as RR increases. Together, these results support the proposed separation between systematic recovery bias and label-sampling variance.

The fixed-Fisher model well captures these trends of variance amplification, but overpredicts total variance at step 500 and shows discrepancies in individual spectral bands. We therefore interpret Eq. (14) as a local model that explains the observed scaling and amplification, but with limited quantitative accuracy late in training.

W3 / Q3 — Stability of low-curvature Fisher eigenspaces. The relevant low-curvature subspaces retain substantial overlap along recovery, although their drift is measurable (Appendix J, Fig. 11). We recompute the top-16-truncated Fisher (Exact fisher matrix for full vocabulary is too expensive) on the same fixed prefixes at soft-training checkpoints from step 0 to 500 and measure spectral-subspace overlap relative to initialization.

For the flattest and low-curvature bands, mean squared overlaps are 0.810 and 0.655 at step 100 and 0.791 and 0.626 at step 500. Both remain substantially above their random-subspace baselines of 0.338 and 0.277. Thus, much of the measured rotation is already present at the first checkpoint, with comparatively smaller subsequent changes in overlap.

These measurements support using the initial low-curvature subspaces as an approximate description of recovery, while qualifying the assumption of a fixed geometry. This diagnostic covers the soft trajectory and a truncated Fisher. It does not determine how much Fisher drift contributes to the late-stage variance-prediction error.

W4 — Scope and empirical contribution. Our conclusions concern subliminal steering with a shared frozen base model and an exactly representable teacher intervention. We will make this scope explicit throughout the contribution statement and conclusion, consistent with Appendix D.

Our analysis also motivates a practical route to recovery from a small prompt pool. In Fig. 4b, 60 fresh autoregressive completions from each of 500 prompts yield a best recovery cosine of approximately 0.84, compared with 0.80 for one completion each from 30,000 prompts, at the same completion budget.

In addition, we add experiments in appendix K, (and detailed in following blocks), extending this resampling finding to system-prompt based teachers, transmitting traits to lora students, we achieve similar subliminal transfer effect via 1000 independent samples +10 times resampling compared to10000 independent samples, while no transfer happens on the same 1000 independent samples without resampling. This experiment suggests a new way for enabling subliminal learning on a small data pool, while previous works [1,2] generally report failures in this case.

[1] Dong et al. (2026). Scaling model-generated distillation data can make latent teacher traits more recoverable. [2] Blank et al. (2026). Subliminal Learning Is Steering Vector Distillation.

AuthorsOpenReview ↗

Rebuttal - New prefix-fixed Training Protocal (W3/Q2) Detailed Version (3/6)

New prefix-fixed Training Protocal (W3/Q2) Detailed Version

The text-only response is provided below. Please see this link for the figures and full report of the additional experiments. Alternatively, please refer to Appendix E in the revised manuscript.

We isolate label averaging from changes in autoregressive context by fixing every input sequence and resampling only its supervision targets. Fixing an initial prompt alone permits later prefixes to change when new completions are generated. Our control fixes the stored prompt and the entire completion, so every completion position retains exactly the same input prefix.

At each completion position, we query the teacher using that stored prefix, independently draw R next-token labels with replacement, and use their empirical frequencies as the target distribution. Training averages the cross-entropy losses of these labels. Sampled labels never enter the input sequence or affect later prefixes; every original completion position remains supervised.

SupervisionTarget at each fixed completion prefix
Original hard labelsOne-hot target for the stored completion token
Fixed-prefix resamplingEmpirical distribution from R teacher draws; R=1 is a newly sampled one-hot target
Soft supervisionFull teacher distribution

Labels are sampled once and reused during training, with nested draws across R conditions. Increasing R reduces label-sampling variance while holding input sequences and prefixes fixed. As R grows, the empirical target approaches the teacher distribution; its expected training gradient equals the soft-supervision gradient.

This protocol underlies the additional experiments in Appendices F–I. Appendix G applies it to a renormalized Top-K teacher distribution. Appendix J examines Fisher stability along the corresponding fixed-input soft-supervision trajectory. Each experiment keeps its own specified carrier set fixed across the supervision conditions being compared.

Reviewer_wz8qOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies subliminal transfer in a controlled setting where a frozen teacher and student differ through a residual steering vector. It compares hard-token and full-distribution supervision on matched carrier prefixes. Although their initial mean gradients are nearly collinear, their optimization endpoints differ substantially. The proposed explanation treats the carrier Fisher matrix as a forward operator: optimization progressively recovers flatter components while amplifying residual hard-label noise, yielding a data-dependent stopping depth.

Critical strengths
  1. The known teacher vector and matched-prefix comparison make the mechanism inspectable, distinguishing initial gradient alignment from recovery over optimization.
  2. The study connects Fisher correction, spectral recovery, sample-size experiments, and repeated sampling. Agreement between the soft trajectory and the fixed-Fisher prediction supports the proposed local explanation.
Critical weaknesses
  1. The conclusions should consistently remain within the steering-vector testbed. A frozen base model with a restricted student parameterization does not establish that the same mechanism dominates system-prompt teachers or general fine-tuning. Appendix D acknowledges this limitation; narrowing the abstract and conclusion would address it without requiring every broader setting. (pp. 1, 13–14)
  2. Soft supervision changes the information available per prefix. Its success shows that the prefixes plus full teacher distributions identify the steering direction, but does not rule out insufficient coverage under token-only observation. Fixing initial prompts in the repeated-completion control also permits later prefixes to change. Fixed-token-prefix resampling would isolate label averaging more directly. (pp. 3, 9)
Questions for authors
  1. Can the resampling control keep every evaluated token prefix fixed while independently sampling the next-token target?
  2. Do truncated or quantized soft targets retain the small-data recovery benefit?
  3. How stable are the optimal hard-label stopping depths and recovery gaps across independently sampled carrier datasets?
6 comments
AuthorsOpenReview ↗

Rebuttal-Overview (1/5)

Responses Overview

We sincerely thanks reviewer wz8q, for his effort on providing valuable and thoughtful feedbacks on our paper, to help us improve the paper quality. We address the raised concerns in multiple blocks. This is an overview, and with more detailed and specific responses to some concerns are listed in blocks afterwards.

W1 — Scope of the claims. Our experiments concern steering-vector recovery in a frozen base model with a restricted student parameterization. We will state this scope explicitly in the conclusion, as we do in the revised abstract. Whether the same mechanism dominates transfer from system-prompted teachers or during general fine-tuning remains an open question, but we added experiments in this case, please see the last response block.

W2 / Q1 — Fixed-token-prefix resampling. Yes. We fix each carrier sequence, including its prompt and completion. At every completion position, we condition the teacher on the original token prefix and independently draw RR next-token labels. The draws are sampled once, reused throughout training, and used only as supervision. Every subsequent prefix comes from the original sequence.

We train the student by averaging the negative log-likelihood over the RR labels at each position. This is equivalent to using their empirical frequencies as a cross-entropy target. As RR increases, the empirical target approaches the teacher’s soft distribution and its sampling variance decreases as 1/R1/R, with all input prefixes and their weights held fixed.

Appendix E (Fig. 6) illustrates the protocol and derives the training loss. Appendix F (Fig. 7) reports how resampling improves recovery at fixed coverage, and Appendix H (Fig. 9) quantifies the reduction in recovery variance across label seeds.

Q2 — Truncated soft targets. Yes. Truncation on probability distributions (we only leave Top K candidates to be visible) has little effect on recovery, as long as it is not degraded to hard supervision (Top 1 is visible). On the same 50 carrier sequences after 100 updates (Appendix G, Fig. 8), full-vocabulary (approximately 15k candidates) supervision reaches a cosine of 0.4875. Top-4 reaches 0.4632 (95.0%), and top-16, top-64, and top-256 all fall within 0.4869–0.4874 (99.88–99.98%) of the full-vocabulary cosine. Top-1 only reaches approximately 0.219 (44.9%).

Q3 — Stability across carrier datasets. The optimal stopping depths were relatively concentrated, and the minimum recovery errors were stable across eight disjoint 50-sequence subsets drawn from the same carrier pool (Appendix I, Fig. 10). For 1, 4, and 16 draws per prefix, the mean optimal stopping steps were 18.8 ± 3.5, 55.0 ± 9.3, and 181.3 ± 18.1, respectively (mean ± SD). Although the absolute spread grew with the number of draws, the relative spread fell from 18.6% to 16.9% to 10.0% of the mean stopping depth. The minimum normalized squared error to the teacher vector also varied little across subsets, with SDs of 0.008–0.013. With 64 draws, all eight runs reached their lowest recorded error at the 500-step training limit, leaving their optimal stopping depths to be determined under longer trainings, we have to leave this record for future work under time budget of this rebuttal.

AuthorsOpenReview ↗

Rebuttal - New prefix-fixed Training Protocal (W2/Q1) Detailed Version (2/5)

New prefix-fixed Training Protocal (W2/Q1) Detailed Version

The text-only response is provided below. Please see this link for the figures and full report of the additional experiments. Alternatively, please refer to Appendix E in the revised manuscript.

We isolate label averaging from changes in autoregressive context by fixing every input sequence and resampling only its supervision targets. Fixing an initial prompt alone permits later prefixes to change when new completions are generated. Our control fixes the stored prompt and the entire completion, so every completion position retains exactly the same input prefix.

At each completion position, we query the teacher using that stored prefix, independently draw R next-token labels with replacement, and use their empirical frequencies as the target distribution. Training averages the cross-entropy losses of these labels. Sampled labels never enter the input sequence or affect later prefixes; every original completion position remains supervised.

SupervisionTarget at each fixed completion prefix
Original hard labelsOne-hot target for the stored completion token
Fixed-prefix resamplingEmpirical distribution from R teacher draws; R=1 is a newly sampled one-hot target
Soft supervisionFull teacher distribution

Labels are sampled once and reused during training, with nested draws across R conditions. Increasing R reduces label-sampling variance while holding input sequences and prefixes fixed. As R grows, the empirical target approaches the teacher distribution; its expected training gradient equals the soft-supervision gradient.

This protocol underlies the additional experiments in Appendices F–I. Appendix G applies it to a renormalized Top-K teacher distribution. Appendix J examines Fisher stability along the corresponding fixed-input soft-supervision trajectory. Each experiment keeps its own specified carrier set fixed across the supervision conditions being compared.

AuthorsOpenReview ↗

Rebuttal - Subliminal recovery with truncated probability distributions (Q2) Detailed Version (3/5)

Subliminal recovery with truncated probability distributions (Q2) Detailed Version

The text-only response is provided below. Please see this link for the figures and full report of the additional experiments. Alternatively, please refer to Appendix G in the revised manuscript.

Training follows the fixed-prefix protocol in Appendix E.

For K≥4, recovery changes little with K at a fixed sampling count; increasing the sampling count produces a much larger improvement. We fix the same 50 Random carrier sequences and compare recovery after 100 updates in the Qwen-cat setting.

At every fixed prefix, we retain the teacher's K highest-probability tokens and renormalize their probabilities to sum to one. Soft supervision uses this distribution directly; matched hard supervision samples R labels from the same distribution. Student probabilities remain normalized over the full vocabulary.

Teacher targetHard R=1Hard R=16Hard R=256Soft
Top-1≈0.219≈0.219≈0.219≈0.219
Top-40.18570.41270.46450.4632
Top-16 / Top-64 / Top-256≈0.1915≈0.426≈0.48440.4869–0.4874
Full vocabulary———0.4875

Entries are final cosine similarity; the grouped row summarizes the nearly overlapping K=16,64,256 results, and dashes indicate values omitted from this table.

For K≥4, changing K at fixed R changes cosine by only about 0.006–0.020. In contrast, increasing R from 1 to 256 improves cosine by 0.279–0.293, bringing hard supervision close to its matched soft target. Top-1 is a point mass, so repeated sampling yields identical labels.

The teacher distributions are concentrated: Top-4 retains 96.66% of probability mass on average and Top-16 retains 99.99966%, using the training loss weights. This explains why extending the output support adds little recovery benefit here, while reducing label-sampling noise produces a large gain. Truncation defines a changed target; within each K, the hard/soft comparison isolates sampling around that same target.

Reviewer_kHg9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The research topic of why subliminal learning require tens of thousands of carrier examples to successfully transfer a teacher's behavioral trait to a student model is interesting and commonly established concept. The findings of the paper are that small-sized carrier examples already contain enough information to transfer the behavioral trait of a teacher; however, due to noise in the hard labels, an abundance of carrier examples is required to offset the impacts.

Critical strengths

The angle of this research is interesting and very insightful: it studies whether small-sized carrier datasets contain enough information to transfer a teacher's behavioral trait. It provides a new perspective in addition to the conventional understanding that subliminal learning requires a massive abundance of examples to offset the impact of hard-label noise.

Critical weaknesses

Weaknesses:

  1. The paper argues that subliminal learning needs a large volume of data to drown out label noise. But, the experiments fail to separate the amount of data from the variety (coverage) of the data. In the paper's mathematical setup, noise explodes when the data lacks variety (which creates mathematical blind spots, or "flat directions"). When the authors uses a larger random dataset, they are doing two things at the exact same time: increasing the total volume (which averages out noise) and increasing the variety (which fixes those blind spots). Due to these two reasons are mixed together, it is not clear if small datasets fail purely because of noisy label, or simply due to a small random sample does have the variety required to provide the model a complete picture of the problem.

  2. The key related work [4] has not been included in the paper. This paper was one of the very first to propose the subliminal learning concept (demonstrating a teacher’s hidden bias could pass to a student even when the training data was semantically unrelated to this trait). Thus, it is vital to include the paper in the related work.

  3. Limited Empirical Novelty (Overlap with [3]): The paper depends on the data scaling law observed by previous work [3]. The concept that subliminal learning specifically needs only scaling up the number of independent data samples but not just the size and that simply training repeatedly on a small dataset fails to work was already shown empirically by Dong et al. (2026) [3]. I think the paper provides a different interpretation of that existing concept rather than introducing a fundamentally new perspective.

  4. Overlap in Gradient Descent Analysis for Subliminal Learning: The application of analyzing gradient descent dynamics to interpret subliminal learning is not entirely novel. Recent related works have already taken this direction. For instance, Blank et al. (2026) [1] already frames subliminal transfer as steering vector distillation and studies the gradient components during training. Additionally, other recent works, such as Sustained Gradient Alignment Mediates Subliminal Learning [2], similarly uses gradient descent alignment and dynamics to explain trait transfer. This diminishes the unique contribution of applying gradient analysis to this specific problem.

References:

[1] Blank et al. (2026). Subliminal Learning Is Steering Vector Distillation.

[2] Kitkana et al. & Arora et al.(2026). Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment

[3] Dong et al. (2026). Scaling model-generated distillation data can make latent teacher traits more recoverable.

[4] Cloud et al. (2026). Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data

Questions for authors

1.Can the authors do ablation studies to separate the size and diversity? For instance, how does recovery perform on a smaller dataset that has been explicitly constructed to maximize diversity (optimal spectral coverage of the carrier space), compared to a larger but less diverse dataset?

2.Given the data scaling condition observed empirically by [3], and the steering vector/gradient dynamics analyses provided by [1] and [2], what are the primary new capabilities, phenomenan, or fundamental mechanism introduced here that is different those prior works theoretically ?

[1] Blank et al. (2026). Subliminal Learning Is Steering Vector Distillation.

[2] Kitkana et al. & Arora et al.(2026). Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment

[3] Dong et al. (2026). Scaling model-generated distillation data can make latent teacher traits more recoverable.

8 comments
AuthorsOpenReview ↗

Rebuttal-Overview (1/4)

Responses Overview

We sincerely thanks reviewer kHg9, for his effort on providing valuable and thoughtful feedbacks on our paper, to help us improve the paper quality. We address the raised concerns in multiple blocks. This is an overview, and with more detailed and specific responses to some concerns are listed in blocks afterwards.

W1 / Q1 — Separating carrier coverage from label-sampling noise. (Detailed responses in following blocks) In Appendix E (Fig. 6), we fix the dataset size at 50 source sequences and compare three selection rules: farthest-first to increase hidden-state coverage, nearest-to-anchor to reduce it, and random selection. At the same number of teacher draws per prefix, changing the selection rule has only a modest effect on recovery. After 100 updates, the cosine across the three sets ranges from 0.184 to 0.214 with one draw and from 0.389 to 0.426 with 16 draws.

Increasing the number of draws has a much larger effect. Within each set, we keep all input prefixes and the optimization budget fixed and independently resample only the teacher’s next-token labels. Increasing the number of draws per prefix from 1 to 16 improves cosine by 0.205–0.234 across the three sets, closing 79–81% of the gap to matched soft supervision. Thus, for the 50-sequence sets tested, reducing label-sampling noise improves recovery substantially more than changing the selection rule. The coverage comparison uses a hidden-state diversity proxy and one selection seed. Appendix G corroborates the denoising effect across eight label seeds, with recovery variance falling approximately fourfold for each fourfold increase in draws.

W2 — Citation to the original subliminal learning work. We cite Cloud et al. (2026), Language models transmit behavioural traits through hidden signals in data (Nature, doi:10.1038/s41586-026-10319-8), in the opening paragraph of Section 1 of the original submission and in the references. We credit this work with the original demonstration of trait transfer through semantically unrelated carrier data.

W3 — Empirical contribution beyond Dong et al. [3]. Our noise analysis identifies a route to subliminal transfer from a small carrier prompt pool: drawing fresh autoregressive completions to average out hard-label sampling error. In Fig. 4b, we use 500 distinct prompts, 1/60 the size of the 30,000-prompt pool, and draw 60 independent completions per prompt. At the same budget of 30,000 completions, the small pool reaches a best recovery cosine of approximately 0.84, compared with 0.80 for one completion per prompt from the larger pool.

This result challenges the inference from Dong et al. [3] that effective subliminal transfer requires a large pool of distinct carrier prompts. Dong et al. found limited gains from repeatedly training on a small, fixed dataset. Each pass over that dataset reuses the same realized label error. Fresh teacher completions average out sampling error, allowing a small prompt pool to achieve comparable recovery in our subliminal steering setting.

In addition, we added experiments on general subliminal learning setting in appendix K: System prompt teacher transmit trait to LoRA student, we found that 1000 independent samples + auto-regressive resampling achieves similar subliminal transfer with 10000 indepedent samples, while repeatedly training on the same 1000 independent samples yields almost no subliminal transfer.

W4 / Q2 — The mechnistic advancement is to answer: why does recovery improve with more data, and how can we strengthen it when only a small pool of distinct carrier prompts is available?

Blank et al. [1] and the sustained-gradient-alignment study [2] ask why a teacher’s trait transfers through distillation, rather than the effect of subliminal transfer.

[3] found increasing training data improves transfer but the mechnism behind is not explained, and did not provide any solution on improving transfer under data scarcity.

Our noisy-inverse model explains this dependence on data size. The carrier Fisher FF reweights the teacher direction ΔT\Delta_T, so the initial soft-label update follows FΔTF\Delta_T and can be weakly aligned with ΔT\Delta_T. Gradient descent progressively reverses this distortion, recovering high-curvature components first. With hard labels, the same process amplifies sampling error as it recovers low-curvature components. More teacher draws reduce sampling variance and allow training to recover more of the trait on low-curvature before noise dominates.

In Fig. 4b, drawing fresh autoregressive completions from 500 prompts achieves comparable recovery to drawing one completion each from 30,000 prompts at the same completion budget. Appendices E and H isolate the denoising mechanism by holding every input prefix fixed and showing that additional teacher draws improve recovery and reduce its variance.

AuthorsOpenReview ↗

Rebuttal-New prefix-fixed Training Protocal (2/4)

New prefix-fixed Training Protocal

The text-only response is provided below. Please see this link for the figures and full report of the additional experiments. Alternatively, please refer to Appendix E in the revised manuscript.

We isolate label averaging from changes in autoregressive context by fixing every input sequence and resampling only its supervision targets. Fixing an initial prompt alone permits later prefixes to change when new completions are generated. Our control fixes the stored prompt and the entire completion, so every completion position retains exactly the same input prefix.

At each completion position, we query the teacher using that stored prefix, independently draw R next-token labels with replacement, and use their empirical frequencies as the target distribution. Training averages the cross-entropy losses of these labels. Sampled labels never enter the input sequence or affect later prefixes; every original completion position remains supervised.

SupervisionTarget at each fixed completion prefix
Original hard labelsOne-hot target for the stored completion token
Fixed-prefix resamplingEmpirical distribution from R teacher draws; R=1 is a newly sampled one-hot target
Soft supervisionFull teacher distribution

Labels are sampled once and reused during training, with nested draws across R conditions. Increasing R reduces label-sampling variance while holding input sequences and prefixes fixed. As R grows, the empirical target approaches the teacher distribution; its expected training gradient equals the soft-supervision gradient.

This protocol underlies the additional experiments in Appendices F–I. Appendix G applies it to a renormalized Top-K teacher distribution. Appendix J examines Fisher stability along the corresponding fixed-input soft-supervision trajectory. Each experiment keeps its own specified carrier set fixed across the supervision conditions being compared.

AuthorsOpenReview ↗

Rebuttal - Ablation on Coverage Effect for Subliminal Transfer W1/Q1 Detailed version (3/4)

Ablation on Coverage Effect for Subliminal Transfer W1/Q1 (Detailed version)

The text-only response is provided below. Please see this link for the figures and full report of the additional experiments. Alternatively, please refer to Appendix F in the revised manuscript.

Training follows the fixed-prefix protocol in Appendix E.

In this experiment, increasing the number of sampled labels improves recovery substantially more than the differences observed across sample sets with different effective ranks. We compare three strategies, each selecting 50 complete source sequences in the Qwen-cat setting, and train for 100 updates.

Selection uses normalized hidden states at the last input token of candidate prefixes:

  • Farthest-first (FF): repeatedly select the candidate farthest from its nearest selected center.
  • Random: randomly select 50 distinct source sequences.
  • Nearest-to-anchor: select the 50 distinct source sequences with prefixes closest to a fixed anchor.

The effective ranks below measure sequence-level representation diversity using the centered covariance of normalized sequence-mean hidden states. Recovery is cosine similarity to the teacher steering vector.

SelectionEffective rankHard R=1Hard R=16SoftGain from R=1 to R=16
FF13.340.2140.4190.467+0.205
Random12.830.1920.4260.488+0.234
Nearest-to-anchor11.220.1840.3890.439+0.205

Increasing R from 1 to 16 improves cosine by 0.205–0.234, whereas the spread across selection strategies is 0.030 at R=1 and 0.037 at R=16. Resampling closes 79–81% of the R=1-to-Soft recovery gap, using only 50 fixed sequences. Nearest has the lowest measured diversity and weaker recovery, while FF and Random perform comparably. These comparisons show a substantial benefit from reducing label noise at fixed input coverage, across all three selection strategies.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

sincerely thank all reviewers for the time and effort devoted to evaluating our paper and for their thoughtful, constructive feedback. Your comments motivated additional experiments that clarify the mechanism, test its predictions, and extend the empirical results beyond the controlled steering-vector setting. We also thank you in advance for taking the time to read our rebuttal and the new experimental reports.

Additional experiments and anonymous reports. The following links provide the figures and full reports; the same material appears in Appendices E–K of the revised manuscript.

  • Appendix E — Fixed-token-prefix resampling: We hold entire input sequences fixed and average independently sampled next-token labels at each completion position, isolating label-noise reduction from changes in autoregressive contexts.
  • Appendix F — Coverage and resampling: We compare farthest-first, random, and nearest-to-anchor sample selection and find that increasing label resampling improves recovery substantially more than the differences observed across their effective ranks.
  • Appendix G — Truncated targets: We compare top-K soft targets and sampled hard labels, finding that recovery depends relatively weakly on K once K≥4, while increasing the number of label draws substantially improves transfer.
  • Appendix H — Bias reduction and variance amplification: Across label-sampling seeds, we track decreasing remaining bias and increasing amplified noise along training, with repeated sampling suppressing variance approximately in proportion to 1/R.
  • Appendix I — Stability across carrier datasets: We compare eight disjoint carrier datasets and quantify the variation in recovery errors and optimal stopping steps as the number of label draws increases.
  • Appendix J — Fisher eigenspace stability: We recompute the average Fisher along recovery and find substantial persistence of low-curvature eigenspaces, with most observed drift occurring early in training.
  • Appendix K — Resampling enables subliminal transfer under a small sample pool: Across two model families and five animal traits, autoregressive resampling from 1,000 prompts improves system-prompt-teacher-to-LoRA-student transfer in nine of ten settings and can approach or exceed training on 10,000 independent prompts.

Our central contribution is a mechanistic account of why subliminal learning can require large datasets, together with a practical way to enable transfer from a small prompt pool. In the controlled recovery setting, carrier geometry attenuates the trait signal in low-curvature directions. Recovering these components requires deeper optimization, which progressively reduces bias but also amplifies hard-label sampling noise. This explains why apparently small initial gradient noise can still limit eventual recovery, why repeatedly training on the same realized labels can peak early and then deteriorate, and why additional independent teacher draws help.

This account leads to a concrete intervention: obtain additional teacher samples for existing inputs instead of only revisiting their original labels. Our fixed-prefix experiments isolate the benefit of averaging label noise, while the system-prompt/LoRA experiments demonstrate its practical relevance through ordinary autoregressive hard-label training. With ten completions per prompt, a pool of 1,000 prompts can attain transfer comparable to, and in several settings stronger than, 10,000 independent prompts. The benefit is a reduction in the required number of distinct prompts, while retaining 10,000 generated training sequences. Together, these results connect the mechanism behind data requirements to an actionable sampling strategy for small-pool subliminal transfer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.