acceptodds
Under review as a conference paper at ICLR 2027

Post-Training Leaves Behavioral Shadows on Unrelated Decisions

Abstract

Post-training adapts language models to target tasks using domain-specific data. Recent work on subliminal learning reveals that information about these private updates can be implicitly transmitted through model generations on unrelated tasks. However, existing approaches are computationally heavy, relying on sampling extensive trajectories from the teacher, and remain largely confined to transferring superficial traits or preferences. **We show that capability-relevant information can be observed through single-word responses to unrelated inputs, without access to teacher logits, parameters, or target-task responses.** We call the behavioral changes that a private update induces on unrelated inputs its *behavioral shadow*. Our instrument, **A**ctive **T**askless **D**istillation (ATD), uses only the public model from which both teacher and student are initialized to find prompts where that model assigns nearly equal probabilities to two ordinary words. At these near-ties, a small private update may reverse the preferred word; ATD collects one greedy word per query from the private teacher and trains the student solely on the resulting prompt–word pairs. In the primary coding experiment with Qwen2.5-1.5B, a student trained on just 5,664 single-word teacher responses gains on HumanEval+ over a control trained on the same responses with disrupted prompt–response pairings. Experiments on scientific knowledge, commonsense reasoning, and reading comprehension also show capability transfer, with further evidence across the tested model generations, sizes, and families. Functional analyses show that the learned signal is source-specific and composable, and that its strength tracks the teacher’s update strength. These results reveal that capability improvements can induce behavioral changes on unrelated boundary decisions, where even minimal, single-token observations suffice to transfer complex task skills.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
subliminal learning, capability transfer, behavioral shadows, post-training, knowledge distillation, implicit distillation, large language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9TpdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that if we have one public model M_0, one teacher model (called M_T) SFTed on one target task from M_0, and another student model M_S initialised from M_0, we can collect some data to train M_S to learn the target task capability from M_T. The interesting part is the data is unrelated prompt + two single tokens, and the requirement is two single tokens are approximately equally likely and one of the tokens is the greedy token.

Intuitively, seeing which token M_T greedily decodes from the two approximately equally likely tokens from M_0 gives information about how M_T was updated from M_0.

Critical strengths

The paper observes a phenomenon where under some restricted conditions (same model ancestor, greedy decoding, teacher model generates the same token from the two most likely tokens from the base model), we can actively design unrelated prompts to distill some target task capability from teacher model with only 1 token per prompt generated from the teacher model, no model weight access and no logins.

Critical weaknesses

W1: The biggest weakness and opportunity for improvement is to (1) make it clear how generally applicable the Active Taskless Distillation (ATD) method is, what conditions will break it, and (2) demonstrate that some version of ATD is more generally applicable.

W2:The current experiment has many restrictions. It makes the implication of ATD narrower and does not support ATD beyond an isolated case.

  • The authors conduct experiments only on teacher and student models from the same public ancestor model. I encourage the authors to share if they have made any efforts to try ATD on cases where teacher and student models are different models, and whether it still works.
  • Greedy decoding is necessary to apply ATD, which is a restriction and less common in practice.
  • teacher model must greedy decode one of the two equally most likely tokens from the base model. How likely and practical is this requirement? Does it effectively require the teacher model is not too different and cannot undergo too many gradient updates from the base model?

W3: Even under the same ancestor model constraint, the authors are encouraged to clarify and discuss as the teacher model becomes further away from the base model (e.g., by taking more gradient updates in SFT), (1) does ATD recover less of the teacher performance? (2) will it becomes more challenging to apply ATD for example because teacher model less frequently decode the most likely two tokens from the base model?

W4:The authors are encouraged to clarify how many gradient steps the teacher models are trained in current experiments

W5:The second biggest weakness and opportunity for improvement is: While the authors emphasise the effectiveness of ATD, the authors should also emphasise there can be large gap in teacher and ATD performance as reported in Table 2. In many cases, such as ScienceQA, OpenBookQA, HellaSwag, RACE-high, PIQA in table 2, the ATD student performance is mildly higher than control experiments, but largely behind teacher models. It means ATD can only recover small fraction of the target capability, not full capability. How much of target capability from teacher model can be recovered through ATD needs a dedicated discussion section.

W6: The presentation of Table 1, Figure 2 can be made much clearer. I recommend not using unnecessary naming, such as signal, control, independent-acquisition signal, etc. Try to present results in more direct, clear, simplistic form. And describe different experiments in a way that is self explanatory.

W7: Please report how prompts are discovered. How many tries does it take to find the needed prompts that satisfy the requirement for both base model and teacher model. This is important to give readers an understanding of how easy/difficult it is to select near-tie prompts as part of ATD.

W8: I find the title of this paper too broad and not accurately reflective of the main contribution. "POST-TRAINING LEAVES BEHAVIORAL SHADOWS ON UNRELATED DECISIONS" is too broad, suggesting general findings of "behaviroal shadows" from general "post-training". But the main contribution of this paper is the Active Taskless Distillation. It is about how ATD can distill target capabilities from teacher model into student model. But none of these concrete contributions are mentioned in the title.

Questions for authors

I am curious if the authors tried extending the experiments to more general setup and if any findings can be shared. such as

  • different base models
  • non greedy decoding
  • teacher model trained more steps from the base model, so more different from the base model
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thanks for the detailed review. Below we address your four main points in turn.

1. Title and Core Thesis

In our Abstract and Section 1, our goal is to reveal an empirical phenomenon: post-training leaves behavioral shadows on unrelated decisions carrying capability information. ATD is a minimal black-box probe we built to observe this phenomenon across tasks and models. Post-Training Leaves Behavioral Shadows on Unrelated Decisions states this finding directly; we keep it and clarify ATD's role as a probe in the Introduction.

2. Effectiveness vs. Teacher Gap

ATD is not meant to clone the full teacher. Effectiveness means one greedy token per prompt on unrelated text already transfers real capability over matched controls on the same prompts and words. A gap to the teacher in Table 2 is expected for a 1-bit channel; widening observations in Table 13 to 20 words per prompt on Qwen2.5-0.5B GSM8K lifts the net gain to +6.37 pp.

To test higher student capacity on the same 1-token carriers, we added a LoRA r64 experiment across all 4 ancestors on HumanEval+: every model beats its shuffle control, and Qwen2.5-1.5B reaches 54.88%, beating the exact control by +8.54 pp and matching the 51.22% teacher:

AncestorShuffle (%)ATD r64 (%)Gain (pp)
Qwen2.5-1.5B48.1754.88+6.71
Qwen3-4B67.0771.34+4.27
Qwen3-1.7B55.4957.32+1.83
Llama-3.2-1B20.7322.56+1.83

We also ran a new baseline spending the same 17,858 queries on target coding prompts, and found a surprising result: ATD achieves much higher training efficiency and accuracy than standard target-task SFT! Training on 5,664 unrelated words for 157 steps beats SFT on 17,815 full code solutions:

MethodOutputsStepsHumanEval+ (%)vs. Base (pp)
Public Base0043.290.00
Target SFT, 1 ep17,815 solutions56042.48-0.81
Target SFT, 2 ep17,815 solutions1,12045.73+2.44
ATD5,664 words15751.22+7.93

Both new comparisons are now Tables 3a and 4 in the revised paper.

3. Scope and Boundaries

For base models and fine-tuning, Table 14 shows cross-ancestor pairs fail because near-ties are defined on θ0\theta_0. Within a shared base, transfer holds across all 4 lineages in Figure 3, and teacher and student need no matching fine-tuning: distilling our LoRA r16 teacher into a full-parameter student without LoRA still gains +4.73 pp. Testing more non-Qwen families, we find ATD works whenever the teacher retains clear near-tie shifts on M0M_0: averaged over three seeds, two open-source teachers beat their shuffle control, with official OLMo-2-1B SFT at +1.42 pp and DeepSeek-Coder-1.3B at +1.02 pp across all three seeds, now in Table 3b.

For non-greedy decoding, ATD is not tied to greedy decoding: at T = 0.2, all 4 seeds stay positive at +2.17 pp over shuffle, fading only under T = 0.8 noise to +0.96 pp.

When the teacher is trained for more steps and moves further from the student's base M0M_0, Table 11 shows that within local updates the recovered direction still tracks the teacher's strength closely. Once heavy multi-stage training pulls the teacher too far from the student, however, transfer weakens, consistent with the milder open-source gains above, and eventually vanishes when the update leaves Eq. 3's local regime or requires knowledge the student base cannot already express, as Section 5.5 and Appendix I show for memorized answers, ciphers, and incompatible bases. Notably, this breakdown is not from missing the two candidate words: since each prompt restricts choices to a or b, even distant teachers still output one option, so what degrades when teacher and student diverge too far is the boundary signal itself rather than the hit rate.

4. Experimental Details and Naming

Sections 4–5 and Appendices A–B note offline screening on M0M_0 with 0 teacher queries: we score 400,000 prefixes to select 17,858 near-ties, and since Options: a | b is prompted where M0M_0 picks from {a, b}, the Qwen2.5-1.5B teacher hits {a, b} 99.5% of the time before |q − 0.5| <= 0.02 and an 80/20 split yield 5,664 pairs. Teachers use 50 DPO steps on Qwen2.5-1.5B code, 160–240 SFT steps on MC, and 100 SFT steps on other code teachers in Appendix B.

For naming, signal, control, and exact nuisance-matched are defined in Section 5.1, Table 1, Figure 2, and Appendix C; we keep this naming and add brief glosses in captions.

Uncovering behavioral shadows in this local regime is exciting, showing post-training encodes executable capabilities into single-word unrelated decisions, and we hope this inspires broader work on implicit distillation across unrelated tasks. In light of these clarifications and new results, we would be sincerely grateful if you might reconsider the overall assessment of our paper.

Reviewer_9TpdOpenReview ↗

increased my score

The authors' responses addressed most of my concerns, and I've raised my score.

  • Title and core thesis: No remaining concerns.
  • Experiments: The experiments and evidence for ATD are strong.
  • New result: "ATD achieves much higher training efficiency and accuracy than standard target-task SFT!" This is a strong and interesting result!

One suggestion: In the intro, say clearly both when ATD works and where it doesn't.

Where ATD works:

  • It isn't tied to a specific model.
  • It doesn't need a matched post-training method.
  • It works with non-greedy decoding.
  • It outperforms both the shuffled baseline and target-task SFT.

Limitations:

  • The teacher and student must share the same model ancestor.
  • It doesn't fully recover the teacher's capability.
  • It gradually breaks down as the teacher drifts further from the base model.

Stating both side of conditions gives readers a clearer understanding of the phenomenon and method's scope. :)

AuthorsOpenReview ↗

Thank you for your positive feedback!

Thank you very much for your recognition of our work and for this helpful suggestion! We have revised the intro accordingly to clearly state both where ATD works and where its limitations lie.

Reviewer_H6VzOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents a light-weight method for transmitting capabilities from a private post-trained model (teacher) to a public base model (student), simply by querying single-word responses to inputs unrelated to the target capability. The proposed method is named ATD (active taskless distillation), and is related to the recent subliminal learning domain.

In ATD, the access to the teacher logits and parameters are not needed, but a. the student model should be public, and b. the teacher should have been post-trained from this student model. The idea is to build a set of questions for which the base student model is giving two near-tie answers. By probing the teacher model on these questions, and fine-tuning the student model on the produced (question, answer), the fine-tuned student is able to gain teacher capabilities (e.g., coding capabilities). The set of questions is selected to be unrelated to the target capability (called "carrier" in the paper).

A simple interpretation of ATD is provided. The main experiment highlights the transfer of coding capabilities from a Qwen2.5-1.5B model post-trained (DPO LoRA) on a coding task to the related base model. Many ablations are provided to ensure that this transfer is not confounded with other factors (e.g., by building a control group shuffling the label produced by the teacher, or using the labels produced by the student).

The method is then tested on different 6 additional tasks (using Qwen2.5-1.5B model) and on 3 other LLM models (Llama-3.2-1B, Qwen3-1.7B, Qwen3-4B; using HumanEval+). Those additional experiments show that the approach does not yield reliable transfer in every tested setting.

Critical strengths
  • The paper is very clear, and each decision is motivated. The interpretations (Eq. 3,4) are simple and convincing, and the proposed method is also simple to operate.
Critical weaknesses
  • The evidence is clear for Qwen2.5-1.5B (over multiple tasks), but not validated for the other tested models (CI including 0, for Qwen3-1.7B, Qwen3-4B, and Llama-3.2-1B)
Questions for authors

Experimental settings

  • How many carrier prompts have been used (this can be indicated in the main, and currently I only found "thousands of [...] prompt-word pairs" in l.80). What is the effect as a function of the number of carrier prompts (i.e., what is the necessary "dose" to see significant effects)?

  • The effect is clear for Qwen2.5-1.5B, but the other results show that it might not always significantly transfer the capability. This is already stated by the authors as a limitation (l.457). Related to this point, I have several questions:

    1. The teacher is always produced with DPO LoRA (following at least l.215, I'm not sure about the process for the other tasks). The transfer might result from the way the post-training has been operated. One specific suggestion if feasible: there are many popular and public post-trained models, for which the original base model is known. Is it possible to do a "wild" test from one of those models?
  1. In l.357, the authors state that "the paired 95% confidence intervals include zero for the three additional models". For those 3 additional models, is there any difference in the number of experiments performed (that could have limited the power of the test?).
  • In Table 1, there are 4 different controls. In Fig. 2, the primary control is used, while in Tab. 2, the teacher-label shuffle is used instead. Is there any reason?

Minor points

  • l. 145: the term "unrelated" is qualitatively clear, but it might be difficult to formalize its meaning

  • If space allows, the example in l.661 can be moved to the Datasets paragraph.

  • l.253 the authors can add the teacher's aggregate coding score at that moment.

  • Regarding the carrier acquisition example in the appendix, I don't understand the two first lines (l. 658-659). Are those lines part of the single produced carrier?

  • In Fig. 3, can you add also the base and the teacher performance, along with the CI?

  • In Fig. 2, can you clarify that A1...A5 are 5 repetitions of teacher acquisition (and introduce shortly the term A1...A5 in Sec. 5.2)

  • Related with Fig. 4, I was wondering whether the method is only transferring capability or also a signature of the teacher. Since you built A1...A5, what about: a. checking the correlation between the carriers' labeled sets (over each pair A1...A5)?, b. it is possible to predict the label of another held-out carrier set (i.e., without even the need of any capability to transfer)?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thanks for the constructive feedback and insightful questions.

For Q1, our primary coding experiment retains 5,664 prompt–word pairs from 17,858 near-tie teacher queries, while the breadth and cross-model experiments in Section 5.3 each use 2,048 pairs; we will state these exact counts directly in Section 5. To measure the dose effect directly, we ran a new subset-retraining experiment on Qwen2.5-1.5B and found a smooth gain curve that already saturates around 2,000 to 4,000 pairs:

Carrier Pairs1,0002,0004,0005,664
HumanEval+ Gain vs. Shuffle+2.29 pp+3.66 pp+3.96 pp+3.66 pp

For Q2, we increased the seeds on Qwen3-1.7B to 20, of which 19 are positive; standard seed-level t-intervals exclude zero on both Qwen3-1.7B at [+1.59, +2.86] and Qwen3-4B at [+0.89, +4.60], and a fixed-effects meta-analysis across all four models in Figure 3 confirms a significant overall gain of +2.30 pp with p = 0.011. Moreover, we ran a new student-capacity experiment increasing LoRA to rank 64 on the exact same single-token carriers, yielding even clearer gains across all four HumanEval+ ancestors over their teacher-label shuffle controls (+3.66 pp mean):

Public AncestorShuffle Control (%)ATD Student r64 (%)Gain vs. Shuffle (pp)
Qwen2.5-1.5B48.1754.88+6.71
Qwen3-4B67.0771.34+4.27
Qwen3-1.7B55.4957.32+1.83
Llama-3.2-1B20.7322.56+1.83

Furthermore, behavioral shadows are not specific to DPO-LoRA. Only our Qwen2.5-1.5B coding teacher used DPO: all six multiple-choice teachers in Table 2, the GSM8K teacher in Appendix H, and the three additional coding teachers in Figure 3 were trained with standard SFT. Nor does the effect depend on LoRA: Figure 2b includes a full-parameter SFT teacher gaining +4.67 pp, and in a new mismatched-adaptation run, distilling our fixed LoRA r16 teacher into a full-parameter student without LoRA still yields +4.73 pp. We also ran new experiments applying the same setup to additional non-Qwen families on HumanEval+, including your suggested "wild" test on open-source public weights, and found that ATD works wherever the teacher preserves clear near-tie boundary shifts relative to M0M_0: averaged over three training seeds, the official OLMo-2-1B SFT release gains +1.42 pp vs. shuffle (3/3 seeds; +1.83 pp vs. base) and the official public DeepSeek-Coder-1.3B release gains +1.02 pp vs. shuffle (3/3 seeds; +2.85 pp vs. base). Consistent with Equations 3 and 4, heavy multi-stage post-training leaves a thinner boundary signal than local updates around θ0\theta_0, and all of these new results are now in the revised paper (higher-capacity r64 in Table 3a, open-source teachers in Table 3b).

For Q3, we use the stricter MILP word-count-matched permutation from Appendix C in Table 1 and Figure 2, whereas we use the global label shuffle in Table 2 because the MILP parity and bin-flip constraints are infeasible or time out on four of the six multiple-choice benchmarks. On the two multiple-choice benchmarks where the MILP solves, the effect remains positive at +2.09 pp on HellaSwag and +0.85 pp on PIQA; we will clarify this in the captions.

For the minor points, we ran both checks suggested across the five independent runs A1–A5 in Figure 4: per-word preference shifts correlate at r = 0.86 and held-out A5 labels are predictable at 80% accuracy compared to 72% from the base lean, directly reflecting the local projection gi⊤δg_i^\top \delta while execution-verified gains confirm real capability transfer. Regarding lines 658–659 in Appendix A, yes: the opening question and word list are directly part of that single prompt rather than model outputs—varying this opening text provides different contexts so M0M_0 forms many distinct near-tie pairs, while only the single word after the final "Answer:" is predicted and trained on. We clarified this in Appendix A and fixed the remaining minor presentation points in the revision.

Thank you for the concise answer. The main weakness and the other points have been addressed. I think that there is sufficient evidence for the main claim of the paper. I'm increasing my score to 4 and soundness to 3. I also agree with the additional comments provided by Reviewer 9Tpd.

AuthorsOpenReview ↗

Thank you for your recognition of our work!

Thank you very much for your recognition of our work and for raising your scores! Exploring how post-training leaves capability-carrying shadows on unrelated decisions opens a promising research direction, and we are actively pursuing deeper work along this line to bring even more exciting results soon.

Reviewer_MytROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper discusses that post-training leaves a "behavioral shadow": a privately finetuned model makes slightly different choices on inputs that are unrelated to its target task. The proposed mathod called Active Taskless Distillation(ATD) uses the public base model to select prompts where the model has nearly equal preferences for two ordinary words. For each prompt, the private teacher provides one token through greedy decoding. A student derived from the same base model is then trained using only these prompt-word pair. On Qwen2.5-1.5B model, 5664 pairs improve HumanEval+ performance by 5.34 percentage points compared with a control that uses the same words but shuffles the prompt-word pairings. Smaller improvements are also observed on six multiple choice tasks. The behavioral shadow depends on the source teacher, can be combined across two teachers, and changes with update strength. However, the method doesn't work for memorized answers, incompatible ancestors, or teachers with weak performance gains.

Critical strengths
  1. Novelty. Logit linear selection requires access to the teacher's logits, while ATD selects prompts using only the public ancestor's predictions that are nearly tied. In this way, one greedy token from a black-box teacher can reveal the direction of the gradient. The method also extends subliminal transfer from traits to a capability verified through code execution.
  2. Convincing experiments. Four control arms which includes an integer-programmed permutation that fixes word frequencies and flip rates isolates the effect to the prompt-token pairing. The effect survives five independent acquisitions with retrained teachers, and the students reproduce the teacher's implementation rules on individual problems. Failure cases are reported as well.
  3. Reproducible and well written. The authors make the code open-sourced which is easy for public to reproduce the result shown in the paper. Additionally, the paper is well organized.
Critical weaknesses
  1. The improvement is modes and the statistical evidence is limited. The evaluation includes 164 test items and 4 seeds, and the lower bound of the confidence interval is only nearly 1 percentage point.
  2. The claims that generalization are stronger than the current evidence. None of the results on the three additional model families is statistically significantly.
  3. The paper does not directly compare with ATD with previous subliminal-learning methods or standard distillation under the same query budget.
  4. The method requires knowing the exact public ancestor and using a matching adaptation settings for the teacher and student.
Questions for authors
  1. All controls perform 2.6 to 3.4 percentage points better than the base model, What causes this improvement that does not depend on the teacher?
  2. Does the effect still hold when the teacher and student use different adaptation method such as full parameter finetuning for the teacher and low-rank adaptation for the student?
  3. For the multiple choice tasks, could the improvement come from format transfer? The carrier template also asks the model to choose between options.
  4. How does ATD compare with ordinary black-box distillation if the same 17858 teacher queries are used for target task prompts?
  5. Why are the mixed model results involving Qwen3 and Llama still not statistically significant? What additional evidence would be needed to reach a clear conclusion?
1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

Thanks for the positive feedback and constructive questions.

For Q1, short fine-tuning on clean carrier prompts slightly sharpens Qwen2.5-1.5B-Instruct on its own: in Table 1's public-label control—where we never query the teacher and instead train the student on the same carrier prompts using the base model M0M_0's own greedy top-1 word on each prompt—HumanEval+ already rises from 43.29% to 46.65%. That is why we always report signal minus control rather than signal minus base, canceling out this teacher-independent shift.

For Q2, starting from a shared open-weight ancestor θ0\theta_0 is a genuine scope condition because we define near-tie boundaries on M0M_0, matching the cross-ancestor null in Table 14. Within a shared ancestor, however, teacher and student do not need matching fine-tuning methods. Beyond the matched full-parameter SFT run (+4.67 pp in Figure 2b), we ran new mismatched-adaptation experiments keeping our LoRA r16 code teacher fixed while changing only the student across 4 seeds on HumanEval+: a LoRA r64 student gains +8.84 pp with 95% CI [4.12, 13.72], and a full-parameter student with no LoRA gains +4.73 pp with 95% CI [0.61, 8.99], with all 4 seeds positive in both.

For Q3, we rule out format transfer in three ways: in Table 2, we train both signal and shuffled controls on the exact same two-word prompt template so any format effect cancels in their difference; in Appendix B, we score multiple-choice tasks by log-likelihood over option letters A through E rather than generating word pairs; and in Figure 4a, we show that teachers queried through the identical template only lift their own domain, giving a non-significant +0.60 pp from our Code teacher to ScienceQA.

For Q4, in Section 5.4 Table 5a, we already showed that ATD beats a passive off-task baseline without near-tie selection by +4.27 pp at the same 17,858 queries and +5.03 pp at equal training rows. To answer Q4 directly, we ran a new comparison spending the same 17,858 teacher queries on target coding prompts, and observed a surprising result: ATD achieves much higher training efficiency and accuracy than standard full-solution SFT on our task! Training on just 5,664 single unrelated words for 157 steps outperforms training on 17,815 full code solutions by 8.74 pp at 1 epoch and 5.49 pp at 2 epochs:

MethodSupervised OutputsStepsHumanEval+ (%)vs. Base (pp)
Public Base0043.290.00
Target-Task SFT, 1 ep17,815 code solutions56042.48-0.81
Target-Task SFT, 2 ep17,815 code solutions1,12045.73+2.44
ATD5,664 words15751.22+7.93

This comparison is now Table 4 in Section 5.4 of the revised paper.

For Q5, Figure 3's individual intervals cross zero because our crossed bootstrap resamples HumanEval+'s 164 binary tasks, where item variance dominates. We increased the number of seeds on Qwen3-1.7B to 20, of which 19 are positive; standard seed-level t-intervals already exclude zero on both Qwen3-1.7B and Qwen3-4B, and a 4-model fixed-effects meta-analysis confirms a significant overall gain of +2.30 pp at p = 0.011. Moreover, we ran a new higher-capacity experiment increasing student LoRA to rank 64 on the exact same single-token carriers, lifting all four ancestors over shuffle controls by +3.66 pp on average:

Public AncestorShuffle Control (%)ATD Student r64 (%)Gain vs. Shuffle (pp)
Qwen2.5-1.5B48.1754.88+6.71
Qwen3-4B67.0771.34+4.27
Qwen3-1.7B55.4957.32+1.83
Llama-3.2-1B20.7322.56+1.83

We also ran new experiments applying the same pipeline to additional non-Qwen families on HumanEval+, including the open-source DeepSeek-Coder-1.3B Base-to-Instruct release, and found ATD consistently works wherever the teacher preserves informative near-tie shifts on M0M_0: averaged over three training seeds, the official OLMo-2-1B SFT release gains +1.42 pp vs. shuffle (3/3 seeds; +1.83 pp vs. base) and the official DeepSeek-Coder-1.3B release gains +1.02 pp vs. shuffle (3/3 seeds; +2.85 pp vs. base). The results are now in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.