- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper treats prompted subliminal learning as context distillation and asks whether the generating trait can be recovered by optimizing the model's system prompt rather than its weights. SALVE first fits a soft prompt, asks the same model to verbalize it, and uses sentence-level beam search scored by the training objective to select a readable prompt. The experiments recover animal preferences from number-sequence datasets, show that recovery can succeed even when a standard LoRA student does not acquire the preference, and demonstrate two training changes that amplify transfer. Further experiments cover preference data selected to induce sycophancy or misalignment, diluted datasets, activation-steered teachers, and ciphered fine-tuning attacks.
Critical strengths
-
The main recovery result is convincing within the prompted setting. In Figure 3, SALVE names the correct animal in 18 of 20 runs and the recovered prompts induce the corresponding preference with frequency 0.88. The comparison includes several quite different optimizers, gives them at least comparable GPU compute, and checks both held-out likelihood and the behavior induced by recovered prompts (§4; §E.1, Table 3). This makes the result more informative than a collection of readable examples. The beam-search ablation is also useful: although best-of-N already matches animal recovery in the main experiment, beam search helps in the harder settings, raising steering recovery from 9/108 to 22/108 and mean preference-data auditing scores from 0.46 to 0.62 for sycophancy and 0.44 to 0.74 for misalignment (§K.2).
-
The distinction between information being present in the data and being learned by a particular student is a useful contribution. Figure 4 shows successful recovery in models with little behavioral transfer, and Figure 5 provides concrete interventions that increase transfer, including in the Llama models. The cross-model preference-data experiments extend the study beyond animal names and numerical completions (§6; Figure 13). I also appreciate that the paper reports confabulated verbalizations and unsuccessful steering recovery rather than presenting verbalization as a faithful explanation in every case.
Critical weaknesses
-
The evidence supports trait recovery more clearly than a calibrated detector of unknown training risks. The preference-data evaluation asks an auditor for five predictions and counts success if any matches the known target; three optimization seeds are then reported, with the main text emphasizing that at least one succeeds (§6.2; §J.2). Random preference-data controls are valuable, but this leaves unclear what a user should conclude from one run or from an unfamiliar, plausible-sounding trait in a recovered prompt. This matters because the paper explicitly observes confabulation, and in the steering setting only 22/108 selected prompts name the target, while another 36 runs generate a target-naming candidate that loses under the selection objective (§I.3). Thus a lower dataset loss is demonstrably an imperfect proxy for retaining the behavior of interest. A fixed reporting procedure evaluated across multiple neutral and biased datasets, with false-alarm rates and seed variability, would substantially strengthen the proactive-detection claim.
-
The student-training interventions support an optimization explanation but do not isolate the proposed prompt-like mechanism. Adding 16 tokens changes the context at both training and evaluation, and embedding-only tuning changes representations for all input tokens, not just the fixed system/template positions (§5.2; §G.2). The matched unembedding control is helpful, but it does not distinguish learning a latent instruction in those fixed tokens from other changes to input processing. Restricting updates to template/suffix-token embeddings, or removing or swapping the suffix after training with appropriate initial-model controls, would make this interpretation more compelling. Without that evidence, the empirical amplification result stands, but the mechanistic conclusion should remain a hypothesis.
Questions for authors
-
For the preference-data experiment, what fixed procedure would you recommend when the target trait is unknown: one seed, the lowest-loss prompt across seeds, or inspection of all prompts? Can you report the corresponding sensitivity and false-alarm rate on the existing biased and random-control datasets, rather than an at-least-one-seed criterion?
-
Do the amplified animal preferences persist when the appended suffix is removed or replaced at evaluation? Have you tested updating only the embeddings of the fixed system/template or suffix tokens? These comparisons would help establish whether the proposed prompt-like storage mechanism explains the effect.
2 comments
Thanks for replying
Your response addresses most of my concerns. I keep my score
Response to Reviewer 4jy6 (1/1)
Thank you for your review. We especially appreciate your attention to the additional experimental settings that we could not fit into the main text.
On SALVE as a calibrated detector (W1, Q1).
We agree that more thoroughly evaluating SALVE as a calibrated detector is an exciting direction for future work, especially using larger collections of datasets with known hidden fine-tuning effects, such as [1]. Within this paper, we are happy to revisit any specific claims that you think are currently overstated.
As a first step, we report the confusion matrix you suggested on our Logit-Linear Selection (LLS) data. To recap, we recover prompts from three sources of data: control, sycophancy-selected and misalignment-selected, with 15 recovered prompts each (5 models × 3 seeds). We flag a prompt for a trait if its auditing score for that trait is at least 0.5. We see no false positives: no control prompt is flagged for either trait, and no prompt is flagged for the wrong trait. The main limitation is sensitivity.
When the target trait is unknown, we expect inspecting all recovered prompts to be most effective, with a natural first step being to look for traits that appear across multiple verbalizations. Additionally weighting prompts by loss would provide useful signal: anecdotally, generic prompts recovered from control data reduce the loss less than prompts that correctly name the trait on LLS data. But as you point out from our steering experiments, loss cannot be relied on alone.
We see parallels to interpretability tools such as natural language autoencoders [2], which are already very useful for hypothesis generation but generally require additional verification.
On the mechanism behind our student training interventions (W2, Q2).
We agree, and intend the mechanism behind subliminal learning only as a hypothesis, one that motivates what might otherwise seem like arbitrary modifications to student training. Our experiments modifying student training are primarily intended to show the distinction you pointed out in S1: information can be present in the data without being learned by a student. We apologize if any part of this section reads as making a definitive claim about the model's mechanisms; that is not our intent, and we are happy to make specific revisions there.
That said, other work generally supports this intuition, finding that subliminal learning is carried by changes at the tokens that start the model's context. Schrodi et al. [3] use activation patching to show that changes in representations at system prompt tokens are important for mediating subliminal learning. Anonymous [4] finds a similar result using weight grafting. Both [4] and Nief et al. [5] also observe that, for models that exhibit subliminal learning, replacing the system prompt with an empty one removes the effect.
Analogously, we show below that for students trained with our appended suffixes, removing the suffix at evaluation returns the animal preference to near its initial rate. Replacing the suffix with a different one does the same in most settings.
Animal pick rates, averaged over the four animals.
In initial experiments, we also fine-tuned only the embeddings of the system prompt and chat template tokens, and observed effects comparable to fine-tuning the full embedding matrix. We presented the full-matrix results because that intervention is simpler to describe.
[1] Transluce. Mass Producing Training Data with Surprising Side Effects. Blog post. https://transluce.org/mass-producing-dsses
[2] Fraser-Taliente et al. Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations. Transformer Circuits Thread, 2026.
[3] Schrodi et al. Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer. ICLR 2026.
[4] Anonymous. Subliminal Learning Through Mechanism Reuse. Submitted to ICLR 2027. https://openreview.net/forum?id=9ZuIeQQvE1
[5] Nief et al. Subliminal Learning is a LoRA Artifact. 2026.