- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- The paper proposes a retrospective distillation attribution method called SCOUT. Given a pool of candidate teachers and text from a trained model, we can determine which teacher the model was distilled from, even when the subject model has undergone other forms of training like SFT/DPO/RL
- The method works by aggregating patterns as PoS 3–5-gram count vectors and averaging each candidate’s vectors into a profile, masking out low-contrast patterns, and calibrating the cosine distance between students and candidates to identify the closest candidate teacher.
- The authors validate their method on models distilled from DeepSeek-R1, which have been subsequently trained with additional SFT, DPO, and/or RL
- Using SCOUT reveals that teacher signatures first become visible during SFT and remain readable across further training stages. These results hold for a variety of different descendent models, including different model families and architectures.
Critical strengths
- The method seems to work quite well on the models that were tested, with strong headline results across three regimes (retrospective, controlled with abstention, prompt-efficiency). They beat a reference-based method on conversation probes and reach 12/12 with 50–100 prompts. It is also cheap and simple, and it only requires model outputs
- The authors are very forthcoming about any limitations and include negative results
Critical weaknesses
Very narrow range of models tested, and somewhat unrealistic scale:
- all models are distilled from the same teacher, DeepSeek-R1 (which has very distinct stylistic traits that can be particularly easy to identify). When six R1-distilled models are added to the pool of teacher models, SCOUT drops from 7/7 to 1/7 (Table 18), identical to every other output-only baseline, while DistillDetect keeps 6/7. In a realistic setting, there will not be a clear outlier in the pool of teacher models, it is more likely that there will be multiple models with similar stylistic tendencies, like there is in the case where the six R1 distilled models are added to the pool.
- student are all <= 8B parameters, and only trained on 800-1000 teacher responses
Questions for authors
- If you train a student on a non-teacher model whose responses are rewritten into the teacher’s format (e.g., GPT-OSS content reformatted as “## Step k:” blocks), does SCOUT attribute it to Llama-3.3? I wonder whether what you are surfacing is just shallow stylistic quirks.
- All 14 retrospective models have the same teacher. Can you add a retrospective panel where the true teacher is absent from the pool?
- Can you report SCOUT results when the model is distilled from a teacher other than DeepSeek-R1?
5 comments
Response to Reviewer nrFP (2 of 4)
[W2-1] Does attribution work for students larger than 8B?
Under the two-probe acceptance rule, SCOUT returns 18 correct attributions and abstains on the remaining six. After teacher removal, it records zero false positives under both backbone and teacher holdout. We extend the controlled evaluation in Section 5.3 to twenty-four students with 12–22B parameters, crossing four backbones (Mistral-Nemo-12B (Mistral AI, 2024), OLMo-2-13B, Qwen2.5-14B, and EuroLLM-22B (Ramos et al., 2026)) with three teachers and two training datasets (OMI and s1). The candidate pool replaces Qwen3-8B with Qwen3-32B while retaining GPT-OSS-120B, Llama-3.3-70B-Instruct, and the unexposed distractor Gemma-3-27B-it.
[Table C] Controlled attribution for 12–22B students
| Method | Det. (↑) | Acc(S) (↑) | Acc(T) (↑) | FP(S) (↓) | FP(T) (↓) |
|---|---|---|---|---|---|
| DistillDetect | 18/24 | 82/96 | 72/96 | 0/24 | 8/24 |
| PoS Templates | 23/24 | 72/96 | 62/96 | 8/24 | 8/24 |
| SCOUT | 18/24 | 92/96 | 72/96 | 0/24 | 0/24 |
While detection remains important, we prioritize avoiding false positives in attribution audits. An abstention explicitly indicates that the available evidence is insufficient and leaves the source unresolved for further investigation. However, a false positive incorrectly identifies a candidate as the source and can mislead conclusions about the model’s training history, making it more consequential than withholding judgment. Appendix D.4 reports the cohort construction and the detection and threshold-transfer results.
[W2-2] Does attribution remain reliable with more distillation data?
SCOUT correctly identifies the teacher for 12/12 students at every training-set size on the held-out s1 probe. SCOUT maintains zero false positives and higher detection than both baselines at every training-set size.
[Table D] Controlled attribution across training-set sizes
| Method | 1K Det. | 1K FP(S) | 3K Det. | 3K FP(S) | 10K Det. | 10K FP(S) |
|---|---|---|---|---|---|---|
| DistillDetect, R=S⁽⁰⁾ | 7/12 | 1/12 | 7/12 | 0/12 | 7/12 | 0/12 |
| PoS Templates | 9/12 | 5/12 | 7/12 | 0/12 | 8/12 | 2/12 |
| SCOUT | 10/12 | 0/12 | 11/12 | 0/12 | 11/12 | 0/12 |
The original controlled evaluation demonstrates that attribution is possible even with limited distillation data. We examine larger training sets by increasing the number of OMI teacher responses from approximately 1K to 3K and 10K, with each larger set retaining all examples from the smaller sets. We use the four 1.5–4B backbones, three teachers, and four-candidate pool from Section 5.3. At each training-set size, we train one student for every backbone–teacher combination, yielding twelve students per size. All students are newly trained with the same three-epoch recipe, including the 1K baseline corresponding to the original training-set size. PoS Templates shows greater variation in detection and false-positive control across sizes. Appendix D.5 reports the experimental setup and full results.
Response to Reviewer nrFP (3 of 4)
[Q1] Can rewriting GPT-OSS responses in the Llama-3.3 format change attribution?
When GPT-OSS answer content adopts the Llama-3.3 ## Step k: layout and final-answer format, SCOUT and PoS Templates select Llama-3.3 for all four students despite GPT-OSS being the content generator. We examine both marker removal and format substitution using the four controlled backbones and the original four-candidate pool. Each condition uses approximately 1K transformed OMI responses for training and the held-out s1 probe for evaluation. The table reports the selected candidates and the number of students assigned to each.
[Table E] Attribution under training-response formatting interventions
| Training-Response Transformation | DistillDetect, R=S⁽⁰⁾ | PoS Templates | SCOUT |
|---|---|---|---|
| Remove GPT-OSS channel markers | GPT-OSS: 4/4 | GPT-OSS: 4/4 | GPT-OSS: 4/4 |
| Reformat GPT-OSS responses into Llama format | GPT-OSS: 4/4 | Llama-3.3: 4/4 | Llama-3.3: 4/4 |
| Reformat using actual Llama step counts | GPT-OSS: 4/4 | Llama-3.3: 4/4 | Llama-3.3: 4/4 |
| Remove Llama step headings and final-answer markers | Llama-3.3: 3/4; Gemma: 1/4 | Gemma: 4/4 | Llama-3.3: 1/4; Gemma: 3/4 |
Removing GPT-OSS channel markers preserves correct attribution for all four students across SCOUT, PoS Templates, and DistillDetect. This control shows that attribution to GPT-OSS remains supported by features beyond its visible channel markers. The format-substitution result persists when step counts follow actual Llama-3.3 responses to the same questions. DistillDetect selects GPT-OSS using the pre-distillation reference.
These interventions isolate teacher-dependent contributions to the measured signal: GPT-OSS attribution survives channel-marker removal, while Llama-3.3 attribution depends strongly on its formatting conventions. Therefore, deliberate format substitution can redirect attribution from the content generator toward the imposed style. The historical reference provides complementary evidence for content provenance in this setting. Appendix B.2 presents these controls and clarifies this interpretation.
[Q2] Can retrospective attribution abstain when the true teacher is absent?
SCOUT correctly accepts both distilled parents and nine descendants. For descendants, it matches the correct-acceptance count of PoS Templates while reducing false positives from 12/12 to 7/12. The teacher-identification evaluation in Section 5.2 assumes that the true teacher is included in the candidate pool. Here, we relax this assumption and evaluate both teacher-present and teacher-absent conditions. When the true teacher is absent, the correct decision is to abstain. We keep the two distilled parents, twelve descendants, and OMI/s1 probes fixed, and compare the original ten-candidate pool with the nine-candidate pool obtained by removing DeepSeek-R1. Unlike the top-ranked identification counts in Section 5.2, Det. counts correct assignments that also pass the acceptance rule on both probes. FP counts accepted assignments after removing the true teacher.
[Table F] Retrospective detection with the teacher present or absent
| Method | Parents Det. (R1 present) | Parents FP (R1 absent) | Descendants Det. (R1 present) | Descendants FP (R1 absent) |
|---|---|---|---|---|
| DistillDetect, R=S⁽⁰⁾ | 1/2 | 0/2 | 9/12 | 1/12 |
| DistillDetect, R=S⁽¹⁾ | — | — | 3/12 | 11/12 |
| PoS Templates | 1/2 | 2/2 | 9/12 | 12/12 |
| SCOUT | 2/2 | 2/2 | 9/12 | 7/12 |
All twelve descendants still rank R1 first, as in Section 5.2. The additional acceptance rule leads to abstention on three descendants. DistillDetect controls false positives more effectively with the pre-distillation reference. Replacing that reference with the already-distilled parent yields only 3/12 correct acceptances and 11/12 false positives. Access to an earlier checkpoint therefore does not guarantee reliable retrospective attribution. Its position in the training history matters.
These findings distinguish successful teacher identification when the source is present from reliable abstention when it is absent. Reliable retrospective abstention without a suitable pre-distillation reference remains an open question and a promising direction for future work. Appendix D.6 reports the setup and results.
Response to Reviewer nrFP (1 of 4)
We thank the reviewer for recognizing the practical value of an efficient, output-only approach to retrospective distillation attribution and for suggesting concrete additional evaluations. The revised manuscript includes additional controlled and retrospective experiments. These evaluations examine generalization across teacher lineages, student sizes, training-set sizes, and candidate pools. We separately test formatting interventions and teacher-absent retrospective audits to clarify the scope of output-only attribution. We highlight all revisions in blue for ease of reference.
[W1-1] How reliable is attribution when closely related candidates coexist?
SCOUT correctly identifies the designated teacher for all SFT parents and DPO descendants in the Gemma panel, while the Llama panel shows mixed results. The Magpie evaluations in Section 5.2 and Q3 include same-family competitors for both designated teachers: Llama-3.3-70B-Instruct and Gemma-3-27B-it.
Appendix D.2 examines a more demanding condition involving direct source relatives. Adding six other R1-derived candidates reduces exact attribution from 7/7 to 1/7 for SCOUT. Every tested output-only baseline likewise selects R1 for only 1/7 suspects, indicating a challenge shared across the evaluated output-only approaches. DistillDetect retains 6/7 when provided with the corresponding pre-distillation checkpoint. This historical reference supplies additional information which might be unavailable in retrospective audits.
The additional candidate-set analysis added in Appendix D.2 provides a way to report this ambiguity. SCOUT includes the exact teacher for 3/7 suspects, compared with 1/7 for PoS Templates, at the same average set size of 4.14 candidates. Thus, returning several plausible candidates preserves the correct source in additional cases. Please see Appendix D.2 for the full results and readout definition.
[Table A] Candidate-set attribution with direct source relatives
Separating a teacher from its direct distilled relatives presents a more demanding task that challenges all evaluated output-only methods. We clarify this distinction in Appendix D.2 and highlight the latter task as an open question and a valuable direction for future research.
[W1-2] How does attribution change with a larger candidate pool?
Using only current outputs, SCOUT retains the correct teacher in a shortlist of just two out of eighteen candidates for every evaluated descendant. We expand the ten-candidate pool in Section 5.2 to eighteen candidate teachers, keeping the twelve public descendants and audit prompts fixed. We report exact top-1 identification and top-two inclusion, requiring each criterion to hold on both OMI and s1.
[Table B] Retrospective attribution with eighteen candidate teachers
SCOUT correctly identifies the teacher for 8/12 descendants, compared with 7/12 for PoS Templates. The correct teacher remains among the top two candidates for all twelve descendants with both output-only methods. DistillDetect achieves 11/12 when provided with the pre-distillation checkpoint S⁽⁰⁾, whereas using the already-distilled parent S⁽¹⁾ yields 3/12. This highlights the importance of access to the reference preceding the distillation stage. Appendix D.3 reports the full candidate-pool analysis, with exact identification and top-two inclusion presented separately.