- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the data requirements of subliminal learning in a controlled steering-vector recovery setting. With the base model frozen, a student learns a residual-stream vector from number sequences generated by a steered teacher. Matching the teacher's full output distribution yields substantially better recovery than training on sampled tokens, despite similar initial gradient directions. The paper explains this through a local noisy inverse model: carrier Fisher geometry transforms the teacher signal, and optimization progressively reverses this transformation while amplifying hard-label sampling error. Experiments examine spectral recovery, data-dependent stopping, and repeated sampling from a fixed prompt pool.
Critical strengths
-
I like the matched-prefix comparison. Holding the carrier prefixes fixed while changing the supervision provides a useful way to separate insufficient prefix coverage from target-sampling error. The large difference in final recovery, despite similar initial gradients, is worth investigating.
-
The known teacher vector makes the explanation unusually testable. Figure 3 checks recovery in the Fisher eigenbasis rather than relying only on an aggregate behavioral score. The agreement between the observed and predicted trajectories is encouraging.
-
The fixed-prompt resampling experiment is a useful addition. It goes beyond showing that more data helps and asks whether the benefit requires more distinct prompts. The distinction between generating new completions and repeatedly training on old completions is practically relevant.
Critical weaknesses
-
I am not convinced that the theoretical increment is as substantial as the main contribution list makes it sound. The recovery platform is already present in [1], including the simplified SGD vector-recovery experiment in Appendix A.6, and [2] already studies weak steering-aligned gradients and the importance of optimization. More directly, Section 2 of [3] gives the subliminal-learning relation , the softmax Fisher form, and the alignment . This is very close to the present Section 4.1, beyond the change in parameterization. Fisher-based activation steering and regularized inverse-Fisher directions also appear in [4]. Neither [3] nor [4] is discussed. Finally, the spectral bias-variance expression in Equation (14) closely matches the gradient-flow risk decomposition in [5], while soft-label variance reduction is established in [6]. I appreciate the acknowledgment of classical iterative regularization in Appendix B, but the main text still needs a sharper account of what is new. The contribution I can identify is the specific noise explanation and its empirical validation, rather than a new Fisher or inversion theory.
-
I found Section 4.1 unnecessarily drawn out. The softmax chain rule, adding and subtracting the teacher probabilities, the expectation of a one-hot sample, and the successive Taylor expansions occupy more space than their technical content warrants. The section is needed as a bridge to the later analysis, so I would not remove it entirely. A compact statement of the Fisher definition, the hard/soft signal-noise decomposition, and its interpretation would do the job; the intermediate algebra belongs in the appendix. I would much rather see the recovered space used to explain the difference from [3] or strengthen the noise diagnostics. This is primarily a presentation concern, but it also makes the limited theoretical novelty harder to assess.
-
The step from a local approximation to the explanation of late-stage recovery is where I still have reservations. The model treats both the Fisher and the sampling-error term as fixed, yet the central argument concerns increasingly weak directions late in training. Those are also directions in which linearization error or changes in the Fisher could matter disproportionately. Figure 3 supports the approximation, and Figure 4 shows the expected stopping-time trend, but these do not quantitatively establish the proposed sampling-noise scaling. I would find the explanation more convincing with a direct check of the noise covariance or the finite-step spectral variance in Equation (14), including its dependence on sample count. Independently resampling targets at fixed token prefixes would be a particularly clean test. Resampling whole completions from fixed initial prompts, as in Figure 4, also changes the subsequent prefixes.
-
I am also hesitant about how much the controlled result tells us about the data requirements of subliminal learning more generally. The teacher and student share a frozen base model and the same intervention parameterization; only one residual vector is recovered, using number-sequence carriers and animal preferences. The teacher intervention is exactly representable by the student, making this a particularly favorable recovery setting. Appendix D acknowledges the scope, which I appreciate. Still, with the theoretical ingredients largely established, I would like the empirical contribution to reach beyond this setting, or the main claims to be framed more explicitly around it.
[1] Subliminal Steering: Stronger Encoding of Hidden Signals. arXiv preprint, 2026.
[2] Subliminal Learning Is Steering Vector Distillation. ICML Mechanistic Interpretability Workshop, 2026.
[3] Channel Location Constrains the Auditability of Subliminal Learning. arXiv preprint, 2026.
[4] FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers. arXiv preprint, 2026.
[5] A Continuous-Time View of Early Stopping for Least Squares Regression. AISTATS, 2019.
[6] A Statistical Perspective on Distillation. ICML, 2021.
Questions for authors
-
What is the precise advance of Section 4.1 over the channel-Fisher relation in [3], and how does the correction experiment relate to the inverse-Fisher steering construction in [4]?
-
Can the authors directly test the predicted noise covariance or spectral error scaling by varying the number of independent target draws at fixed token prefixes?
-
How stable are the relevant Fisher eigendirections along the recovery trajectory, particularly the low-curvature directions responsible for the claimed late-stage noise amplification?
8 comments
Rebuttal - Responses Overview second half (2/6)
Responses Overview second half
W2 — Presentation of Section 4.1. We agree that the intermediate algebra can be shortened. We will retain the Fisher definition, the hard/soft signal–noise decomposition, and their interpretation in the main text, and move the intermediate softmax and Taylor-expansion steps to the appendix. This will make room for the comparison with [3] and the new noise diagnostics.
W3 / Q2 — Direct validation of sampling-noise scaling. Our new fixed-prefix experiment confirms that each fourfold increase in independent teacher draws reduces recovery variance by approximately fourfold, while the spectral variance increases with training depth (Appendix H, Fig. 9).
We fix all supervised token prefixes from 50 carrier sequences and independently sample next-token labels at each position. Training averages their negative log-likelihoods. The labels are sampled once and reused throughout optimization, and every input prefix remains fixed. Appendix E (Fig. 6) describes this protocol: supervision with empirical token frequencies.
Across eight paired label seeds, the normalized recovery variances at step 500 are 1.2594, 0.2997, 0.0748, and 0.0190 for , respectively. This closely follows the predicted dependence. Meanwhile, the estimated squared bias stabilizes near the soft-supervision residual as increases. Together, these results support the proposed separation between systematic recovery bias and label-sampling variance.
The fixed-Fisher model well captures these trends of variance amplification, but overpredicts total variance at step 500 and shows discrepancies in individual spectral bands. We therefore interpret Eq. (14) as a local model that explains the observed scaling and amplification, but with limited quantitative accuracy late in training.
W3 / Q3 — Stability of low-curvature Fisher eigenspaces. The relevant low-curvature subspaces retain substantial overlap along recovery, although their drift is measurable (Appendix J, Fig. 11). We recompute the top-16-truncated Fisher (Exact fisher matrix for full vocabulary is too expensive) on the same fixed prefixes at soft-training checkpoints from step 0 to 500 and measure spectral-subspace overlap relative to initialization.
For the flattest and low-curvature bands, mean squared overlaps are 0.810 and 0.655 at step 100 and 0.791 and 0.626 at step 500. Both remain substantially above their random-subspace baselines of 0.338 and 0.277. Thus, much of the measured rotation is already present at the first checkpoint, with comparatively smaller subsequent changes in overlap.
These measurements support using the initial low-curvature subspaces as an approximate description of recovery, while qualifying the assumption of a fixed geometry. This diagnostic covers the soft trajectory and a truncated Fisher. It does not determine how much Fisher drift contributes to the late-stage variance-prediction error.
W4 — Scope and empirical contribution. Our conclusions concern subliminal steering with a shared frozen base model and an exactly representable teacher intervention. We will make this scope explicit throughout the contribution statement and conclusion, consistent with Appendix D.
Our analysis also motivates a practical route to recovery from a small prompt pool. In Fig. 4b, 60 fresh autoregressive completions from each of 500 prompts yield a best recovery cosine of approximately 0.84, compared with 0.80 for one completion each from 30,000 prompts, at the same completion budget.
In addition, we add experiments in appendix K, (and detailed in following blocks), extending this resampling finding to system-prompt based teachers, transmitting traits to lora students, we achieve similar subliminal transfer effect via 1000 independent samples +10 times resampling compared to10000 independent samples, while no transfer happens on the same 1000 independent samples without resampling. This experiment suggests a new way for enabling subliminal learning on a small data pool, while previous works [1,2] generally report failures in this case.
[1] Dong et al. (2026). Scaling model-generated distillation data can make latent teacher traits more recoverable. [2] Blank et al. (2026). Subliminal Learning Is Steering Vector Distillation.
Rebuttal - New prefix-fixed Training Protocal (W3/Q2) Detailed Version (3/6)
New prefix-fixed Training Protocal (W3/Q2) Detailed Version
The text-only response is provided below. Please see this link for the figures and full report of the additional experiments. Alternatively, please refer to Appendix E in the revised manuscript.
We isolate label averaging from changes in autoregressive context by fixing every input sequence and resampling only its supervision targets. Fixing an initial prompt alone permits later prefixes to change when new completions are generated. Our control fixes the stored prompt and the entire completion, so every completion position retains exactly the same input prefix.
At each completion position, we query the teacher using that stored prefix, independently draw R next-token labels with replacement, and use their empirical frequencies as the target distribution. Training averages the cross-entropy losses of these labels. Sampled labels never enter the input sequence or affect later prefixes; every original completion position remains supervised.
| Supervision | Target at each fixed completion prefix |
|---|---|
| Original hard labels | One-hot target for the stored completion token |
| Fixed-prefix resampling | Empirical distribution from R teacher draws; R=1 is a newly sampled one-hot target |
| Soft supervision | Full teacher distribution |
Labels are sampled once and reused during training, with nested draws across R conditions. Increasing R reduces label-sampling variance while holding input sequences and prefixes fixed. As R grows, the empirical target approaches the teacher distribution; its expected training gradient equals the soft-supervision gradient.
This protocol underlies the additional experiments in Appendices F–I. Appendix G applies it to a renormalized Top-K teacher distribution. Appendix J examines Fisher stability along the corresponding fixed-input soft-supervision trajectory. Each experiment keeps its own specified carrier set fixed across the supervision conditions being compared.
Rebuttal - Responses Overview first half (1/6)
Responses Overview first half
We sincerely thanks reviewer 3MnG, for his effort on providing valuable and thoughtful feedbacks on our paper, to help us improve the paper quality. We address the raised concerns in multiple blocks. This is an overview, and with more detailed and specific responses to some concerns are listed in blocks afterwards.
W1 / Q1 — Distinguishing established knowledges/understandings from our contribution. We agree that several mathematical ingredients build on prior work. However, we believe the review understates the advance obtained by connecting these ingredients to the data requirements of iterative subliminal transfer. Our contribution is a theoretically grounded, experimentally testable account of how sample count affects recovery, why similar initial alignment can lead to different recovery outcomes, and how reducing sampling noise improves transfer from a small carrier pool.
Relation to [1] and [2]. We explicitly cite [1] as the source of our recovery testbed. Its restricted parameterization makes the teacher signal known and allows direct comparison between theoretical predictions and measured recovery trajectories. The testbed itself is not listed as our contribution. We also discuss [2] in related work. Its steering-distillation and gradient analyses help explain how subliminal transfer occurs. Our central question concerns the dependence of recovery on data quantity and supervision noise once transfer is possible.
Relation to [3]. We acknowledge that the first-order relation $ d_0 \approx F\Delta_T$, and hence the initial alignment cos(ΔT,FΔT), overlaps with [3]. We were unaware of this recent preprint when preparing the submission and will explicitly discuss it.
The conclusions developed from this shared starting point differ. In its controlled body-channel setting, [3] finds that initial alignment predicts subsequent behavioral transfer. Our analysis follows the entire recovery trajectory and shows how recovery depends jointly on the Fisher eigenvalues, training depth, sample count, and the teacher signal’s spectral distribution in particular. The initial cosine overlooked those information and compressed into an insufficient single scalar.
Concretely, writing δi=⟨ΔT,ui⟩, our local model gives the finite-step recovery error along a Fisher eigenvector as
$ \mathbb{E} \left[(\widehat{\delta}_i^{(k)}-\delta_i)^2\right] \approx (1-\eta\lambda_i)^{2k}\delta_i^2 + \frac{\left[1-(1-\eta\lambda_i)^k\right]^2}{N\lambda_i}. $
This identifies both slower recovery (small λ) and stronger eventual noise amplification in low-curvature directions. Consequently, similar initial cosine alignment can coexist with substantially different iterative recovery. Figure 1 demonstrates this divergence between Qwen and Gemma. Section 6.2 and Fig. 5 further show that Gemma’s teacher signal is concentrated at lower normalized curvature, consistent with its weaker recovery. These observations motivate a spectral account of recovery and its data dependence beyond the initial-alignment statistic: using more data to compensate the recovery under lower curvature regions.
Relation to [4]. FishBack uses pullback Fisher geometry to construct a concept-steering direction with minimal local output distortion. Our inverse-Fisher diagnostic uses that geometry to undo the carrier-induced transformation of a distillation signal and recover the known teacher direction. Our subsequent analysis explains how iterative inversion also amplifies hard-label error and creates a sample-dependent stopping depth. Thus, the shared Fisher geometry serves distinct objectives and supports different conclusions. We will clarify this relationship and cite [4].
Relation to [5] and [6]. Both works are already cited as foundations for the iterative-regularization and variance-reduction arguments. Our contribution lies in connecting these results to the observed subliminal-recovery phenomenon and testing their consequences. In particular, the new fixed-prefix experiment holds every carrier input constant and shows approximately fourfold lower recovery variance for each fourfold increase in teacher draws (Appendix H, Fig. 9).
We will sharpen the contribution statement accordingly. The paper develops and validates a noise-based explanation of data-dependent transfer within a controlled subliminal-steering setting.