- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces stifle, a method for both identification and mitigation of traits that are subliminally transferred during distillation of language models, without requiring prior knowledge of the trait. The method is motivated by findings that student models can inherit characteristics of a teacher, even when data contains no explicit connection to the given characteristic. This is achieved by operating in parameter space, exploiting the intuition that subliminal traits are transferred across diverse datasets, whereas useful task-specific updates instead have substantially more variance across contexts. Using parameter updates from models independently fine-tuned on different teacher-generated datasets, the authors derive an estimator that identifies a candidate trait direction based on consistency of the updates. They provide theoretical identification guarantees under assumptions about trait consistency, capability variation, and their separation. The authors additionally demonstrate subliminal learning across several new tasks, extending previous observations of this phenomenon. Further experiments using LoRA fine-tuning on Qwen Instruct show that these identified directions can be used to suppress preference of animals, while preserving measured task performance. The paper also investigates amplification of identified directions as a way to reveal the transferred trait without requiring trait-specific prompting.
Critical strengths
The paper presents a compelling and intuitive approach to identifying subliminally transferred traits without prior behavioral supervision, addressing an important limitation of existing defenses. The use of consistency across parameter updates is a good methodological contribution, supported by theoretical analysis and encouraging empirical results. Demonstrating that the identified directions can suppress transferred traits while preserving measured task performance provides meaningful initial evidence for the approach. The extension of subliminal learning to additional datasets is also a strength.
Critical weaknesses
While the method successfully suppresses transferred preferences while preserving performance of the measured tasks, the specificity of the identified directions remains incompletely characterized. Other desirable characteristics may also be consistently transferred across datasets, and it is unclear whether suppressing the identified direction could affect behaviors not captured by the existing task-performance metrics. Additionally, although trait identification is unsupervised, selecting the intervention strength relies on awareness of the identified trait and constructing a trait-specific behavioral probe. This limits the demonstrated practicality of the method as a general defense against unknown transfers, particularly when the underlying behavior is difficult to characterize or reliably measure.
Questions for authors
The experiments in Section 4.2 provide encouraging evidence that the identified direction can be suppressed without substantially affecting measured task performance. However, could the authors clarify the extent to which these results support behavioral specificity beyond the evaluated tasks? For example, a teacher may consistently transfer other desirable characteristics, such as response style or verbosity, across different datasets. Would such characteristics potentially be captured by the same estimator or affected by suppressing the identified direction? A controlled evaluation involving multiple consistently transferred characteristics could help clarify how selectively Stifle identifies and removes a particular trait. Section 3.6 proposes selecting the intervention strength through a sweep evaluated using a trait-specific behavioral probe. Given that a central motivation is addressing previously unknown traits, could the authors investigate whether a fixed or otherwise probe-independent intervention strength can achieve reasonable, even if partial, mitigation across different traits and tasks? For instance, evaluating a fixed strength on held-out traits could help establish how useful the approach is when the transferred behavior cannot be reliably characterized or measured.
4 comments
Check-in
Dear Reviewer,
We just wanted to quickly check-in as the end of the rebuttal period draws near to say that we remain available to discuss our responses, and perform any additional analyses that might improve your assessment of our paper or raise your score. Thank you again for your time and for your comments on our paper.
Warmest wishes,
The Authors
Thank you for the clarifications.
Response
Thank you for engaging with our response and for the time you have given our paper. We are glad you found our response useful. If it helped clarify aspects of our submission would you consider updating your assessment and raising your score? If not, are there outstanding questions you have that we could try and address? We remain available as the end of the rebuttal period draws near to help clarify any remaining questions about our submission.
Response to reviewer
We thank the reviewer for their positive assessment, and are glad they found the approach compelling and the use of consistency across updates well supported by the theory. We address both questions below.
Behavioural specificity. It helps to separate two sets of tasks: the tasks we want the student to learn from the teacher, and the tasks we use to detect subliminal transfer. STIFLE identifies whatever is transferred consistently across the detection set. If a desirable characteristic such as response style or verbosity is transferred coherently across every task in that set, it is indistinguishable from a trait by construction, and that set of tasks is not suited to isolating the trait. The remedy is to widen the detection set so that the desirable characteristics vary across tasks, and to include them in the evaluation, so that they fall into the varying capability component rather than the shared one. This is the same requirement for diverse datasets that underlies Assumption 2. Amplification gives a further safeguard: by amplifying the identified direction, we can see what it encodes before removing anything. In our experiments, amplification leads to off-topic mentions of the trait animal specifically, which suggests the direction is specific to that trait, and removing it leaves task performance largely intact across all ten tasks. We also ran a no-trait control: when the teacher has no trait, amplifying the estimated direction in held-out models, even at \alpha = 10, leaves their favourite-animal answers unchanged, produces no off-topic animal mentions, and changes task performance by less than 2 points. So the direction found for the trait models reflects the trait rather than something STIFLE produces regardless. We will add this control to the paper. We agree that a controlled study with several consistently transferred characteristics would make this sharper, and we will discuss it as a limitation and direction for future work.
Probe-independent intervention strength. The trait does not need to be known in advance. As the final figure of the main body shows, increasing α makes the model mention the trait on off-topic prompts, which reveals what the trait is and lets us build a probe for it.
We also followed the reviewer's suggestion and evaluated a single fixed \alpha across all traits and tasks. Across all 9 animals, 10 tasks and 5 seeds, setting \alpha = 1 reduces the trait rate from 50.2% to 10.7% (13.9% at α = 0.8), while task performance is largely preserved: on the numbers task, for example, it goes from 96.2 to 88.4. The response to \alpha is smooth, so the strength can be chosen as a simple trade-off rather than tuned for each trait. Moreover, \alpha isn't a complete unknown where a priori we don't know what the scale should be. Indeed, the edit leaves (1 − \alpha) times the original trait component, and this provides in our view a good argument for using \alpha=1 as the natural choice.
Probing the empirical performance of \alpha is more about small optimisations for a given task rather than needing to know what the trait is in order to know how to set alpha in the first place
Thank you again for the detailed and constructive review. We are happy to clarify any points that are not fully addressed, and if there is specific further evidence that would help the reviewer strengthen their assessment, we would welcome the opportunity to provide it.