acceptodds
Under review as a conference paper at ICLR 2027

Retrospective Distillation Attribution via Normalized Response Similarity

28 views

Abstract

Model distillation transfers capabilities through supervised fine-tuning (SFT) on teacher responses, often collected from commercial APIs, raising questions of model provenance. Existing distillation attribution methods have been largely evaluated on students immediately after the SFT step. However, a distilled model may undergo further SFT, preference optimization, or reinforcement learning before release, while an auditor may lack access to the pre-distillation checkpoint required by reference-based attribution. To close this gap, we propose SCOUT, an output-only method that aggregates recurring *syntactic patterns* into candidate profiles, filters low-contrast patterns, and calibrates student–candidate distances against inter-candidate distances. SCOUT supports attribution and abstention using only current texts, without model weights, token likelihoods, or historical checkpoints. Auditing publicly released descendants of distilled models spanning diverse post-training objectives, SCOUT consistently identifies the distillation source. Furthermore, tracing teacher-associated *syntactic signatures* along training trajectories reveals that they emerge during distillation and persist through subsequent preference optimization and reinforcement learning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Distillation Detection, Distillation Attribution, Model Provenance, Model IP Protection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nrFPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper proposes a retrospective distillation attribution method called SCOUT. Given a pool of candidate teachers and text from a trained model, we can determine which teacher the model was distilled from, even when the subject model has undergone other forms of training like SFT/DPO/RL
  • The method works by aggregating patterns as PoS 3–5-gram count vectors and averaging each candidate’s vectors into a profile, masking out low-contrast patterns, and calibrating the cosine distance between students and candidates to identify the closest candidate teacher.
  • The authors validate their method on models distilled from DeepSeek-R1, which have been subsequently trained with additional SFT, DPO, and/or RL
  • Using SCOUT reveals that teacher signatures first become visible during SFT and remain readable across further training stages. These results hold for a variety of different descendent models, including different model families and architectures.
Critical strengths
  • The method seems to work quite well on the models that were tested, with strong headline results across three regimes (retrospective, controlled with abstention, prompt-efficiency). They beat a reference-based method on conversation probes and reach 12/12 with 50–100 prompts. It is also cheap and simple, and it only requires model outputs
  • The authors are very forthcoming about any limitations and include negative results
Critical weaknesses

Very narrow range of models tested, and somewhat unrealistic scale:

  1. all models are distilled from the same teacher, DeepSeek-R1 (which has very distinct stylistic traits that can be particularly easy to identify). When six R1-distilled models are added to the pool of teacher models, SCOUT drops from 7/7 to 1/7 (Table 18), identical to every other output-only baseline, while DistillDetect keeps 6/7. In a realistic setting, there will not be a clear outlier in the pool of teacher models, it is more likely that there will be multiple models with similar stylistic tendencies, like there is in the case where the six R1 distilled models are added to the pool.
  2. student are all <= 8B parameters, and only trained on 800-1000 teacher responses
Questions for authors
  • If you train a student on a non-teacher model whose responses are rewritten into the teacher’s format (e.g., GPT-OSS content reformatted as “## Step k:” blocks), does SCOUT attribute it to Llama-3.3? I wonder whether what you are surfacing is just shallow stylistic quirks.
  • All 14 retrospective models have the same teacher. Can you add a retrospective panel where the true teacher is absent from the pool?
  • Can you report SCOUT results when the model is distilled from a teacher other than DeepSeek-R1?
5 comments
AuthorsOpenReview ↗

Response to Reviewer nrFP (1 of 4)

We thank the reviewer for recognizing the practical value of an efficient, output-only approach to retrospective distillation attribution and for suggesting concrete additional evaluations. The revised manuscript includes additional controlled and retrospective experiments. These evaluations examine generalization across teacher lineages, student sizes, training-set sizes, and candidate pools. We separately test formatting interventions and teacher-absent retrospective audits to clarify the scope of output-only attribution. We highlight all revisions in blue for ease of reference.

[W1-1] How reliable is attribution when closely related candidates coexist?

SCOUT correctly identifies the designated teacher for all SFT parents and DPO descendants in the Gemma panel, while the Llama panel shows mixed results. The Magpie evaluations in Section 5.2 and Q3 include same-family competitors for both designated teachers: Llama-3.3-70B-Instruct and Gemma-3-27B-it.

Appendix D.2 examines a more demanding condition involving direct source relatives. Adding six other R1-derived candidates reduces exact attribution from 7/7 to 1/7 for SCOUT. Every tested output-only baseline likewise selects R1 for only 1/7 suspects, indicating a challenge shared across the evaluated output-only approaches. DistillDetect retains 6/7 when provided with the corresponding pre-distillation checkpoint. This historical reference supplies additional information which might be unavailable in retrospective audits.

The additional candidate-set analysis added in Appendix D.2 provides a way to report this ambiguity. SCOUT includes the exact teacher for 3/7 suspects, compared with 1/7 for PoS Templates, at the same average set size of 4.14 candidates. Thus, returning several plausible candidates preserves the correct source in additional cases. Please see Appendix D.2 for the full results and readout definition.

[Table A] Candidate-set attribution with direct source relatives

MethodExact top-1Exact teacher included in setR1-family candidate included in setMean set size
DistillDetect6/77/77/71.71
PoS Templates1/71/77/74.14
SCOUT1/73/75/74.14

Separating a teacher from its direct distilled relatives presents a more demanding task that challenges all evaluated output-only methods. We clarify this distinction in Appendix D.2 and highlight the latter task as an open question and a valuable direction for future research.

[W1-2] How does attribution change with a larger candidate pool?

Using only current outputs, SCOUT retains the correct teacher in a shortlist of just two out of eighteen candidates for every evaluated descendant. We expand the ten-candidate pool in Section 5.2 to eighteen candidate teachers, keeping the twelve public descendants and audit prompts fixed. We report exact top-1 identification and top-two inclusion, requiring each criterion to hold on both OMI and s1.

[Table B] Retrospective attribution with eighteen candidate teachers

MethodExact top-1 (↑)Teacher in top-2 (↑)
DistillDetect, R=S⁽⁰⁾11/1211/12
DistillDetect, R=S⁽¹⁾3/123/12
PoS Templates7/1212/12
SCOUT8/1212/12

SCOUT correctly identifies the teacher for 8/12 descendants, compared with 7/12 for PoS Templates. The correct teacher remains among the top two candidates for all twelve descendants with both output-only methods. DistillDetect achieves 11/12 when provided with the pre-distillation checkpoint S⁽⁰⁾, whereas using the already-distilled parent S⁽¹⁾ yields 3/12. This highlights the importance of access to the reference preceding the distillation stage. Appendix D.3 reports the full candidate-pool analysis, with exact identification and top-two inclusion presented separately.

AuthorsOpenReview ↗

Response to Reviewer nrFP (2 of 4)

[W2-1] Does attribution work for students larger than 8B?

Under the two-probe acceptance rule, SCOUT returns 18 correct attributions and abstains on the remaining six. After teacher removal, it records zero false positives under both backbone and teacher holdout. We extend the controlled evaluation in Section 5.3 to twenty-four students with 12–22B parameters, crossing four backbones (Mistral-Nemo-12B (Mistral AI, 2024), OLMo-2-13B, Qwen2.5-14B, and EuroLLM-22B (Ramos et al., 2026)) with three teachers and two training datasets (OMI and s1). The candidate pool replaces Qwen3-8B with Qwen3-32B while retaining GPT-OSS-120B, Llama-3.3-70B-Instruct, and the unexposed distractor Gemma-3-27B-it.

[Table C] Controlled attribution for 12–22B students

MethodDet. (↑)Acc(S) (↑)Acc(T) (↑)FP(S) (↓)FP(T) (↓)
DistillDetect18/2482/9672/960/248/24
PoS Templates23/2472/9662/968/248/24
SCOUT18/2492/9672/960/240/24

While detection remains important, we prioritize avoiding false positives in attribution audits. An abstention explicitly indicates that the available evidence is insufficient and leaves the source unresolved for further investigation. However, a false positive incorrectly identifies a candidate as the source and can mislead conclusions about the model’s training history, making it more consequential than withholding judgment. Appendix D.4 reports the cohort construction and the detection and threshold-transfer results.

[W2-2] Does attribution remain reliable with more distillation data?

SCOUT correctly identifies the teacher for 12/12 students at every training-set size on the held-out s1 probe. SCOUT maintains zero false positives and higher detection than both baselines at every training-set size.

[Table D] Controlled attribution across training-set sizes

Method1K Det.1K FP(S)3K Det.3K FP(S)10K Det.10K FP(S)
DistillDetect, R=S⁽⁰⁾7/121/127/120/127/120/12
PoS Templates9/125/127/120/128/122/12
SCOUT10/120/1211/120/1211/120/12

The original controlled evaluation demonstrates that attribution is possible even with limited distillation data. We examine larger training sets by increasing the number of OMI teacher responses from approximately 1K to 3K and 10K, with each larger set retaining all examples from the smaller sets. We use the four 1.5–4B backbones, three teachers, and four-candidate pool from Section 5.3. At each training-set size, we train one student for every backbone–teacher combination, yielding twelve students per size. All students are newly trained with the same three-epoch recipe, including the 1K baseline corresponding to the original training-set size. PoS Templates shows greater variation in detection and false-positive control across sizes. Appendix D.5 reports the experimental setup and full results.

AuthorsOpenReview ↗

Response to Reviewer nrFP (3 of 4)

[Q1] Can rewriting GPT-OSS responses in the Llama-3.3 format change attribution?

When GPT-OSS answer content adopts the Llama-3.3 ## Step k: layout and final-answer format, SCOUT and PoS Templates select Llama-3.3 for all four students despite GPT-OSS being the content generator. We examine both marker removal and format substitution using the four controlled backbones and the original four-candidate pool. Each condition uses approximately 1K transformed OMI responses for training and the held-out s1 probe for evaluation. The table reports the selected candidates and the number of students assigned to each.

[Table E] Attribution under training-response formatting interventions

Training-Response TransformationDistillDetect, R=S⁽⁰⁾PoS TemplatesSCOUT
Remove GPT-OSS channel markersGPT-OSS: 4/4GPT-OSS: 4/4GPT-OSS: 4/4
Reformat GPT-OSS responses into Llama formatGPT-OSS: 4/4Llama-3.3: 4/4Llama-3.3: 4/4
Reformat using actual Llama step countsGPT-OSS: 4/4Llama-3.3: 4/4Llama-3.3: 4/4
Remove Llama step headings and final-answer markersLlama-3.3: 3/4; Gemma: 1/4Gemma: 4/4Llama-3.3: 1/4; Gemma: 3/4

Removing GPT-OSS channel markers preserves correct attribution for all four students across SCOUT, PoS Templates, and DistillDetect. This control shows that attribution to GPT-OSS remains supported by features beyond its visible channel markers. The format-substitution result persists when step counts follow actual Llama-3.3 responses to the same questions. DistillDetect selects GPT-OSS using the pre-distillation reference.

These interventions isolate teacher-dependent contributions to the measured signal: GPT-OSS attribution survives channel-marker removal, while Llama-3.3 attribution depends strongly on its formatting conventions. Therefore, deliberate format substitution can redirect attribution from the content generator toward the imposed style. The historical reference provides complementary evidence for content provenance in this setting. Appendix B.2 presents these controls and clarifies this interpretation.

[Q2] Can retrospective attribution abstain when the true teacher is absent?

SCOUT correctly accepts both distilled parents and nine descendants. For descendants, it matches the correct-acceptance count of PoS Templates while reducing false positives from 12/12 to 7/12. The teacher-identification evaluation in Section 5.2 assumes that the true teacher is included in the candidate pool. Here, we relax this assumption and evaluate both teacher-present and teacher-absent conditions. When the true teacher is absent, the correct decision is to abstain. We keep the two distilled parents, twelve descendants, and OMI/s1 probes fixed, and compare the original ten-candidate pool with the nine-candidate pool obtained by removing DeepSeek-R1. Unlike the top-ranked identification counts in Section 5.2, Det. counts correct assignments that also pass the acceptance rule on both probes. FP counts accepted assignments after removing the true teacher.

[Table F] Retrospective detection with the teacher present or absent

MethodParents Det. (R1 present)Parents FP (R1 absent)Descendants Det. (R1 present)Descendants FP (R1 absent)
DistillDetect, R=S⁽⁰⁾1/20/29/121/12
DistillDetect, R=S⁽¹⁾——3/1211/12
PoS Templates1/22/29/1212/12
SCOUT2/22/29/127/12

All twelve descendants still rank R1 first, as in Section 5.2. The additional acceptance rule leads to abstention on three descendants. DistillDetect controls false positives more effectively with the pre-distillation reference. Replacing that reference with the already-distilled parent yields only 3/12 correct acceptances and 11/12 false positives. Access to an earlier checkpoint therefore does not guarantee reliable retrospective attribution. Its position in the training history matters.

These findings distinguish successful teacher identification when the source is present from reliable abstention when it is absent. Reliable retrospective abstention without a suitable pre-distillation reference remains an open question and a promising direction for future work. Appendix D.6 reports the setup and results.

Reviewer_bSadOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the question of how to detect which teacher model’s generations were used to distill a student model. The work introduces SCOUT, a (appropriately-named) method which performs this classification task by computing syntactic features of model generations along with potential teacher generations and uses z-score normalized and calibrated features to predict which teacher model is used. This is particularly useful because it can be computed cheaply and doesn’t require access to model weights or logits or a reference pre-distilled model. The work validates this method through two experiments. First, it shows that SCOUT can detect models distilled from a particular model (Deepseek-R1). Second it shows that 19 student models fine-tuned on data from three teachers can have their teacher accurately selected. Their method does the best out of all reference-free methods.

Critical strengths
  • The problem this work studies is timely, interesting, and well-motivated, especially under the output-only constraints considered.
  • Under these constraints, the SCOUT method works much better than prior work, so it therefore seems to fill a gap in the community and could be very useful both for model providers and for researchers looking to analyze training data.
  • The work considers both autoregressive and diffusion models as student models which strengthens the results.
Critical weaknesses
  • The clarity of description of experiments in section 5 and to some extent 6 makes it difficult to validate the quality of the work. These sections could benefit from additional iteration with a focus on readability and coherence. In 5.1, lines like 250 are are effortful to read. In 5.2, phrases like “panel”, “sorted candidate scores at equal ranks”, and “same-prompt candidate selection” are used without introduction. In 5.3, it is unclear if the authors are finetuning models themselves; “student-hold out” “teacher-hold out” are not clearly defined; and distinctions are drawn between “identification” and “detection” without explanation. In Section 6, based on previous descriptions, I expected to see a plot with $\Delta G_T$ on the y-axis and training step on the x-axis to answer the question of when teacher-associated evidence emerges, but the results do not show this.
  • Alternative baselines: the motivation behind using syntactic features is sensible, however it is possible that non-syntactic features such as lexical choice are also useful for performing the required classification tasks. For example, one baseline to consider is replacing syntactic features with embeddings from a model such as ModernBERT.
  • The conditions under which SCOUT breaks are not described very well in the text. How many different sources have to be used before SCOUT doesn’t work? What if the candidate pool is very large? What if the candidate pool has very similar models? Can you still detect the base model after finetuning? These are all questions that practitioners looking to use this method might want answers to. Some of these seem to be explored in the appendices, and if they are, it would be very helpful to summarize the results in the main text when referencing the Appendices.
Questions for authors
  • What is the difference in the significance between the results discussed in 6.1 and 5.2?
  • Will code be released to facilitate future study?
4 comments
AuthorsOpenReview ↗

Response to Reviewer bSad (1 of 3)

We appreciate the positive assessment of this research direction and the detailed suggestions for making the experimental evidence easier to assess. Our responses below explain how the evaluations fit together, examine alternatives to syntactic features, and clarify the practical limits of attribution. Corresponding changes in the manuscript appear in blue.

[W1] Clarifying the experimental descriptions.

The revised Sections 5.2 and 5.3 present the evaluated models, metrics, and results in separate paragraphs. The opening of Section 5 explicitly states that we train the controlled students using SFT, while the retrospective evaluation uses publicly released checkpoints.

We clarify the terminology and evaluation protocols at their introduction:

  • Section 5.1 defines identification as selecting the top-ranked candidate and detection as additionally applying an acceptance rule that permits abstention.
  • Section 5.2 defines all table abbreviations. Same compares candidates on each shared audit question, while Dist. independently sorts candidate scores and compares them at corresponding ranks.
  • Section 5.3 defines student holdout by the student backbone and teacher holdout by the training teacher. Score-based thresholds maximize F1 on the remaining students and transfer unchanged to the held-out group.

Regarding the expected training-step plot, Figure 7 compares publicly released checkpoints at distinct training stages. Therefore, its temporal resolution is the released checkpoint, without measurements within an SFT run. Section 6, the figure caption, and the conclusion clarify that the analysis establishes evidence at the first available post-distillation checkpoint and its persistence at subsequent released stages.

[W2] How does attribution compare with embedding-based representations?

We compare PoS features with ModernBERT-base (Warner et al., ACL 2025), StyleDistance (Patel et al., NAACL 2025), and its synthetic-only variant. The comparison uses the same nineteen controlled math students, responses, and four-candidate pool as in Section 5.3. We retain profiling, calibration, and threshold-selection protocols while replacing the response representation. The PoS feature filter is omitted for embeddings.

[Table A] Attribution with embedding-based representations

RepresentationDet.Acc(S)Acc(T)FP(S)FP(T)
SCOUT (Ours)18/1974/7658/760/190/19
ModernBERT-base14/1942/7645/7613/1913/19
StyleDistance12/1965/7656/760/190/19
StyleDistance (Synth)11/1937/7636/7613/196/19

PoS features achieve higher detection and threshold-transfer accuracy than all three embedding alternatives. StyleDistance also maintains zero false positives under both holdout protocols, but correctly accepts fewer students, 12/19 versus 18/19. These results support using PoS features in this pipeline. Appendix B.6 presents the setup and results.

[W3] How does attribution depend on the training sources and candidate pool, and what do the base-model controls show?

SCOUT successfully recovers multiple training sources and retains the correct teacher within a two-candidate shortlist as the candidate pool expands. Distinguishing a teacher from its directly distilled relatives presents a harder case, where exact attribution decreases for all evaluated output-only methods.

[W3-1] Multiple training sources.

We extend the controlled evaluation in Section 5.3 to students trained on mixtures of two or three teachers. The mixture ratios specify the proportions of training responses supplied by each teacher. The two-teacher conditions cover four backbones and all three teacher pairs, giving twelve students per condition. The three-teacher conditions cover the four backbones, giving four students each. We retain the original four-candidate pool and evaluate each student on the dataset not used for SFT. Success requires all training teachers to occupy the top two or three positions, respectively, regardless of their order.

[Table B] Attribution with multiple training sources

Training datasetSource proportionsDistillDetect,R=S⁽⁰⁾PoS TemplatesSCOUT
OMI50:506/1212/1212/12
s150:507/1212/1212/12
OMI90:108/121/1210/12
OMI1:1:13/44/44/4
s11:1:14/44/44/4

SCOUT recovers both teachers for every student under balanced mixtures. Under the 90:10 mixture, it recovers both for ten of twelve students and ranks the majority teacher first for all twelve. Both output-only methods also recover the three-teacher source set with either training dataset. With four candidates, the latter criterion requires placing the single unexposed distractor last. Appendix D.7 documents these settings and results.

AuthorsOpenReview ↗

Response to Reviewer bSad (2 of 3)

[W3-2] Larger candidate pool.

We expand the ten-candidate retrospective pool to eighteen candidates while keeping the twelve public descendants and audit prompts fixed. Each criterion below must hold on both OMI and s1 for a descendant to count as successful.

[Table C] Retrospective attribution with eighteen candidate teachers

MethodExact top-1Teacher in top-2
DistillDetect,R=S⁽⁰⁾11/1211/12
DistillDetect,R=S⁽¹⁾3/123/12
PoS Templates7/1212/12
SCOUT8/1212/12

Although exact selection becomes less reliable, SCOUT retains the correct teacher within a two-candidate shortlist for every evaluated descendant. DistillDetect performs best with the pre-distillation reference; substituting the already-distilled parent substantially changes its results. Appendix D.3 presents this comparison.

[W3-3] Closely related candidates.

The newly added Magpie evaluation in Section 5.2 includes same-family competitors for both designated teachers, namely Llama-3.3-70B-Instruct and Gemma-3-27B-it. On both math probes, SCOUT correctly identifies the teacher for all four Magpie-Align (Xu et al., ICLR 2025) SFT parents and three of four DPO descendants. For MagpieLM (Xu et al., NAACL 2025), it correctly identifies the teacher for both SFT parents and both DPO descendants.

[Table D] Llama-3-70B-Instruct → Magpie-Align

MethodOMI Desc.OMI Par.OMI BaseOMI Dist. (%)OMI Same (%)s1 Desc.s1 Par.s1 Bases1 Dist. (%)s1 Same (%)
DistillDetect,R=S⁽⁰⁾4/44/4—95.192.64/44/4—99.697.0
DistillDetect,R=S⁽¹⁾3/4————4/4————
PoS Templates2/44/41/234.627.54/44/40/271.236.0
SCOUT (Ours)3/44/42/245.234.93/44/41/279.142.7

[Table E] Gemma-2-9B-it → MagpieLM

MethodOMI Desc.OMI Par.OMI BaseOMI Dist. (%)OMI Same (%)s1 Desc.s1 Par.s1 Bases1 Dist. (%)s1 Same (%)
DistillDetect,R=S⁽⁰⁾2/22/2—100.099.02/22/2—99.797.4
DistillDetect,R=S⁽¹⁾1/2————0/2————
PoS Templates2/22/22/2100.062.22/22/22/295.952.1
SCOUT (Ours)2/22/22/2100.098.52/22/22/2100.097.2

We additionally examine the more demanding task of separating a teacher from its directly distilled relatives. For each of seven directly distilled suspects, we add six other R1-derived models to the original ten-candidate pool. The suspects and OMI prompts remain fixed. Appendix D.2 reports correct DeepSeek-R1 selections among these seven suspects.

[Table F] Candidate-pool stress test with source relatives

MethodOriginal pool (10)Expanded pool (16)
DistillDetect,R=S⁽⁰⁾7/76/7
PoS Templates5/71/7
LLM judge (Mistral)6/71/7
LLM judge (Solar)7/71/7
SCOUT (Ours)7/71/7
− Scaling (CC)7/71/7
− Centering (DD)7/71/7

Exact attribution decreases across all evaluated output-only methods in this stress test. DistillDetect retains 6/7 when supplied with the corresponding pre-distillation checkpoint, providing historical information beyond current outputs. Distinguishing a teacher from its directly distilled relatives is therefore a shared challenge for the evaluated output-only methods, which Section 7 explicitly discusses.

[W3-4] Base-model controls.

Section 5.2 reports the pre-distillation bases alongside the R1-distilled parents and their descendants. The Base column counts checkpoints that do not select the designated teacher. SCOUT assigns the parents and descendants to R1 while neither pre-distillation base selects R1 across all four probes. The newly added Magpie evaluation in Section 5.2 extends these controls to lineages supervised by Llama-3-70B-Instruct and Gemma-2-9B-it, reporting the bases, SFT parents, and DPO descendants separately. If the question instead concerns identifying the original base from a model after SFT, this corresponds to the base-model identification problem rather than the teacher-attribution task studied here. As already discussed in Section 2, this separate problem is addressed by Foley et al. (ACL 2023) and Nikolić et al. (NeurIPS 2025).

AuthorsOpenReview ↗

Response to Reviewer bSad (3 of 3)

[Q1] What is the difference in significance between Sections 6.1 and 5.2?

The two retrospective analyses answer different questions. Section 5.2 evaluates whether the method selects the documented teacher from the candidate pool after subsequent training. Section 6.1 examines how evidence toward a known teacher changes across released checkpoints. The latter uses ΔGT\Delta G_T to measure the change relative to the pre-distillation base. A positive change indicates surviving teacher-associated evidence. It does not itself constitute a correct identification or an accepted attribution.

[Q2] Will code be released to facilitate future study?

Yes, we plan to release the code publicly upon acceptance. We hope this will help researchers reproduce the method and explore output-only retrospective attribution across different settings.

Sincerely,

The Authors

AuthorsOpenReview ↗

A Brief Reminder

Dear Reviewer bSad,

As the discussion period draws to a close, we would appreciate your reviewing our responses, including the additional experiments and clarifications we provided in response to your questions. If these additions address your main concerns, we hope you will consider updating your rating. If any questions remain, we would be happy to discuss them. Thank you again for your time and consideration.

Sincerely,

The Authors

Reviewer_VQiAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SCOUT, an output-only method for retrospective distillation attribution. It aggregates syntactic patterns into candidate-teacher profiles, filters low-contrast features, and calibrates student–teacher distances against inter-candidate distances. Experiments on controlled students and publicly released descendants examine teacher identification, abstention, and the persistence of source-associated signals through subsequent training.

Critical strengths
  1. The paper is well organized and easy to follow.
  2. Retrospective distillation attribution is a meaningful research direction, particularly when historical checkpoints and model internals are unavailable.
  3. The method is simple and intuitive, requiring only generated text without model weights, token probabilities, historical checkpoints, or classifier training.
Critical weaknesses
  1. SCOUT attributes teachers through calibrated syntactic proximity, but shared data, common model ancestry, or similar training objectives could produce the same signal. How does the method distinguish these explanations from actual teacher-data use? Does the evidence support specific teacher attribution or only stylistic association?
  2. In Table 18, adding related candidates reduces SCOUT’s exact attribution from 7/7 to 1/7, while reference-based DistillDetect retains 6/7. Please clarify whether SCOUT supports teacher-level or family-level attribution, and evaluate whether abstention or family-level outputs can prevent incorrect identification.
  3. The motivation includes subsequent SFT, preference optimization, and RL, but persistence findings mainly concern the latter two. In Appendix C.5, all 36 source-switching runs select the new teacher by 25% of the second SFT round using 800 responses. In a separate experiment, earlier-source recovery ends at 6/18, matching chance even after removing the new teacher. Although these limitations are acknowledged, the abstract and conclusions should distinguish surviving source signals from historical-source recovery. Controlled follow-up SFT on human-written and other non-target-teacher data would further clarify the scope.
  4. The 12 descendants in the main evaluation share two R1-distilled parents, limiting source diversity. Analyses of score changes toward known teachers do not establish attribution and abstention when sources are unknown. Moreover, open-set detection is mainly evaluated immediately after controlled SFT; teacher-absent scenarios following complex post-training remain insufficiently tested.
Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer VQiA (1 of 3)

We are grateful for the thoughtful discussion of retrospective auditing when historical checkpoints and model internals are unavailable. The concerns motivate controls for alternative explanations, tests of source resolution and teacher absence, and closer examination of subsequent training across teacher lineages. Blue text identifies passages introduced or updated during this discussion.

[W1] Can shared data, model ancestry, or similar training objectives explain the attribution signal?

The controlled cohorts in Section 5.3 compare different response teachers within matched backbone and training-question configurations. We complement this design with two public-model controls addressing the alternative explanations raised in the review.

Shared training questions and base checkpoint. s1-32B and s1.1-32B (Muennighoff et al., 2025) start from the same Qwen2.5-32B-Instruct checkpoint (Qwen et al., 2025) and use the same 1,000 training questions. Their response teachers differ: Gemini Thinking for s1-32B and DeepSeek-R1 for s1.1-32B. On OMI, all three methods select R1 only for s1.1-32B.

[Table A] Attribution with shared training questions and a base checkpoint

Methods1-32B, Gemini supervisions1.1-32B, R1 supervision
DistillDetect,R=S⁽⁰⁾Gemma-3-27B-itDeepSeek-R1
PoS TemplatesGemma-3-27B-itDeepSeek-R1
SCOUTGemma-3-27B-itDeepSeek-R1

Gemini Thinking is absent from this pool, and its original API version is no longer available at the time of writing. Accordingly, s1-32B serves as a negative control for R1 selection, i.e., the Gemma prediction does not identify its actual teacher. In this public pair, sharing the questions and starting checkpoint does not suffice to produce R1 selection.

Reasoning RL without R1 responses. We also examine five public models trained with reasoning RL from the same Qwen2.5-Math bases as the R1-distilled parents, without SFT on R1 responses. The entries below count models selecting R1 on both OMI and s1.

[Table B] R1 selection after distillation or reasoning RL

MethodR1-distilled parents(↑)RL-only models(↓)
DistillDetect,R=S⁽⁰⁾2/20/5
PoS Templates1/20/5
SCOUT2/20/5

SCOUT identifies R1 for both distilled parents while selecting R1 for none of the RL-only models. Therefore, shared ancestry and reasoning RL alone do not reproduce the R1 selections observed after distillation in these controls.

Together, these comparisons support teacher-specific discrimination under the evaluated conditions. Appendix B.2 presents newly added formatting experiments showing that imposing Llama-3.3 formatting on GPT-OSS answer content redirects SCOUT attribution to Llama-3.3 for all four students in each of two conditions. We describe the assignments as evidence consistent with teacher-response use, without treating syntactic proximity alone as definitive proof of undocumented data use. Section 5.2 summarizes these findings. Appendix D.8 provides the experimental settings and full results.

AuthorsOpenReview ↗

Response to Reviewer VQiA (2 of 3)

[W2] How reliable is attribution when closely related candidates coexist?

SCOUT achieves exact teacher attribution with same-family competitors in the expanded evaluations. Section 5.2 includes Llama-3.3-70B-Instruct and Gemma-3-27B-it alongside the designated Llama and Gemma teachers. On both math probes, SCOUT correctly identifies the teacher for all four Magpie-Align SFT parents and three of four DPO descendants. For MagpieLM, it correctly identifies the teacher for both SFT parents and both DPO descendants. W4-1 provides further details on these evaluations.

Appendix D.2 separately examines discrimination between a teacher and its direct distilled relatives. SCOUT identifies R1 for all seven suspects in the original pool. When six other R1-derived models join each candidate pool, every evaluated output-only method selects R1 for 1/7 suspects. DistillDetect retains 6/7 with the corresponding pre-distillation checkpoint, which supplies historical information beyond current outputs.

[Table C] Candidate-set attribution with direct source relatives

MethodExact top-1Exact teacher included in setR1-family candidate included in setMean set size
DistillDetect,R=S⁽⁰⁾6/77/77/71.71
PoS Templates1/71/77/74.14
SCOUT1/73/75/74.14

The additional candidate-set analysis preserves the exact teacher in more cases while reporting candidate ambiguity. Using the same scores, SCOUT includes R1 for 3/7 suspects, compared with 1/7 for PoS Templates, at the same average set size of 4.14 candidates. This readout retains the exact source in two additional cases without modifying the underlying scores. We report exact selection, exact-teacher inclusion, and family inclusion separately so that a plausible source family is not conflated with a uniquely identified teacher.

Appendix D.2 presents these comparisons. Section 7 identifies output-only discrimination between a teacher and its direct distilled relatives as an open question and a valuable direction for future research. Teacher-absent abstention is evaluated separately in Appendix D.6.

[W3] Does the original teacher remain identifiable after follow-up SFT?

We directly examine follow-up SFT without additional responses from the original teacher. We start from nine controlled students covering three backbones and three teachers. First-round SFT uses teacher-generated responses to s1K questions. Each student then undergoes separate second-round SFT runs using 800 human dialogue responses from OASST1 or dataset-provided solutions from s1K or MATH. The s1K condition reuses first-round questions while replacing the teacher responses with the supplied solutions. Evaluation uses held-out OMI questions and the original four-candidate pool.

[Table D] Original-teacher identification after follow-up SFT

MethodOASST1 human dialogues1K solutionsMATH solutions
DistillDetect,R=S⁽⁰⁾9/99/98/9
PoS Templates0/92/90/9
SCOUT9/91/90/9

The table counts students whose original teacher remains top-ranked, without an acceptance threshold. Bold marks the best output-only result per column. After human dialogue SFT, SCOUT identifies the original teacher for all nine students, matching DistillDetect and exceeding PoS Templates. This directly demonstrates that original-teacher identification can survive additional SFT without renewed exposure to that teacher’s responses. With s1K or MATH solutions, original-teacher selection is less frequent for both output-only methods, while DistillDetect retains more identifications with the pre-distillation reference. Appendix C.6 presents these experiments.

Appendix C.5 separately examines follow-up SFT using another candidate teacher. That intervention redirects attribution toward the new source, whereas the OASST1 condition demonstrates persistence. In the earlier-source recovery experiment, SCOUT recovers the original teacher in 6 of 18 runs at the endpoint, matching chance even after the second teacher is removed from the candidate pool. A separate matched performance comparison uses eighteen runs with s1 in both rounds of the 36-arm campaign. Relative to same-backbone controls using the second teacher in both rounds, sequential distillation yields a median endpoint GSM8K gain of 0.49 percentage points, with higher accuracy in 9 of 18 runs. Consequently, these configurations show a small median gain without a consistent performance advantage from sequential distillation. Together, these evaluations clarify that the effect of additional SFT depends on its supervision. The revised discussion and conclusion distinguish persistence of source-associated evidence from recovery of earlier sources after source replacement.

AuthorsOpenReview ↗

Response to Reviewer VQiA (3 of 3)

[W4-1] Does retrospective attribution extend beyond the R1 lineages?

We extend the retrospective attribution experiment in Section 5.2 to publicly released SFT parents and DPO descendants from two additional teacher lineages. Magpie-Align (Xu et al., ICLR 2025) uses supervision from Llama-3-70B-Instruct (Grattafiori et al., 2024), while MagpieLM (Xu et al., NAACL 2025) uses supervision from Gemma-2-9B-it (Gemma Team, 2024). Both panels use the original ten math candidates plus these two teachers, including same-family competitors Llama-3.3-70B-Instruct and Gemma-3-27B-it. These experiments evaluate teacher identification among competing candidates, using the same metrics as in Section 5.2. They therefore complement the known-teacher margin analysis in Section 6.

Magpie-Align — Llama-3-70B-Instruct supervision.

SCOUT identifies the designated teacher for all four SFT parents and three of four DPO descendants on both math probes.

[Table E] Retrospective attribution for Magpie-Align

MethodOMI Desc.OMI Par.OMI BaseOMI Dist. (%)OMI Same (%)s1 Desc.s1 Par.s1 Bases1 Dist. (%)s1 Same (%)
DistillDetect,R=S⁽⁰⁾4/44/4—95.192.64/44/4—99.697.0
DistillDetect,R=S⁽¹⁾3/4————4/4————
PoS Templates2/44/41/234.627.54/44/40/271.236.0
SCOUT3/44/42/245.234.93/44/41/279.142.7

MagpieLM — Gemma-2-9B-it supervision.

SCOUT identifies Gemma-2-9B-it for both SFT parents and both DPO descendants on both probes, despite the competing Gemma-family candidate. Neither pre-distillation base selects this teacher.

[Table F] Retrospective attribution for MagpieLM

MethodOMI Desc.OMI Par.OMI BaseOMI Dist. (%)OMI Same (%)s1 Desc.s1 Par.s1 Bases1 Dist. (%)s1 Same (%)
DistillDetect,R=S⁽⁰⁾2/22/2—100.099.02/22/2—99.797.4
DistillDetect,R=S⁽¹⁾1/2————0/2————
PoS Templates2/22/22/2100.062.22/22/22/295.952.1
SCOUT2/22/22/2100.098.52/22/22/2100.097.2

These results show that successful retrospective identification extends beyond R1 supervision. Section 5.2 presents both panels, and Appendix F documents their training provenance.

[W4-2] Can retrospective attribution abstain when the true teacher is absent?

We extend the retrospective evaluation in Section 5.2 by removing DeepSeek-R1 from the original ten-candidate pool while keeping the two distilled parents, twelve descendants, and OMI/s1 probes fixed. Unlike the teacher-identification evaluation in Section 5.2, which assumes that the teacher is included, this evaluation tests both teacher-present and teacher-absent conditions. Each method fits acceptance thresholds without using the held-out model. Det. counts correct attributions accepted when R1 is present. FP counts false attributions accepted after R1 removal.

[Table G] Retrospective detection with the teacher present or absent

MethodParents Det. (R1 present)Parents FP (R1 absent)Descendants Det. (R1 present)Descendants FP (R1 absent)
DistillDetect,R=S⁽⁰⁾1/20/29/121/12
DistillDetect,R=S⁽¹⁾——3/1211/12
PoS Templates1/22/29/1212/12
SCOUT2/22/29/127/12

SCOUT correctly accepts both distilled parents and nine descendants. All twelve descendants still rank R1 first, consistent with Section 5.2; the additional acceptance criteria produce abstention on three descendants. Compared with PoS Templates, SCOUT retains the same descendant detection count while reducing false attributions from 12/12 to 7/12. DistillDetect provides stronger false-positive control with the pre-distillation reference. Replacing that reference with the already-distilled parent yields 3/12 correct acceptances and 11/12 false attributions. Thus, reference suitability matters alongside reference availability.

These results distinguish teacher-present identification from reliable retrospective abstention. The remaining challenge affects both output-only methods and DistillDetect with a post-distillation reference, motivating further research on source-absence detection under restricted access. Appendix D.6 presents the protocol and results, and Section 7 discusses this direction.

Sincerely,

The Authors

AuthorsOpenReview ↗

A Brief Reminder

Dear Reviewer VQiA,

As the discussion period draws to a close, we would appreciate your reviewing our responses, including the additional experiments and clarifications we provided in response to your questions. If these additions address your main concerns, we hope you will consider updating your rating. If any questions remain, we would be happy to discuss them. Thank you again for your time and consideration.

Sincerely,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to the Reviewers

We thank all reviewers for examining the practical value and evidential scope of retrospective distillation attribution. The revised manuscript broadens the empirical evaluation, adds controls for alternative explanations, and clarifies how the experimental settings support different conclusions. All revisions appear in blue.

The retrospective results extend beyond the R1 lineages. Section 5.2 now includes public Magpie-Align (Xu et al., ICLR 2025) and MagpieLM (Xu et al., NAACL 2025) models supervised by Llama-3-70B-Instruct (Grattafiori et al., 2024) and Gemma-2-9B-it (Gemma Team, 2024). Despite same-family competitors, SCOUT identifies the designated teacher for all four Magpie-Align SFT parents and three of four DPO descendants on both math probes. For MagpieLM, it identifies both SFT parents and both DPO descendants. Section 5.2 presents these results, and Appendix F documents the training provenance.

Additional controlled evaluations address model and data scale. For twenty-four students with 12–22B parameters, SCOUT returns 18 correct accepted attributions under student holdout. After teacher removal, it records zero false positives under both student and teacher holdout. Separate experiments increase distillation data from approximately 1K to 3K and 10K responses. SCOUT identifies the teacher for 12/12 students at every size on the held-out s1 probe, while maintaining zero teacher-absent false positives under student holdout. Appendices D.4 and D.5 report these evaluations.

The representation comparison directly addresses whether syntactic features offer an advantage over embeddings. On the controlled math cohort, PoS features yield higher detection and threshold-transfer accuracy than ModernBERT-base (Warner et al., ACL 2025), StyleDistance (Patel et al., NAACL 2025), and its synthetic-only variant within the same profiling and calibration pipeline (Appendix B.6). Additional mixture experiments evaluate recovery of multiple training teachers, including unequal source proportions (Appendix D.7).

New controls examine alternative explanations and persistence after additional SFT. The shared-question comparison between s1-32B and s1.1-32B (Muennighoff et al., 2025), together with reasoning-RL controls from matching bases, shows that shared questions, ancestry, and reasoning training do not alone reproduce R1 selection in these tests (Appendix D.8). After follow-up SFT on human dialogue, SCOUT identifies the original teacher for 9/9 students without additional responses from that teacher, matching reference-based DistillDetect. Mathematical-solution supervision and teacher-switching SFT are evaluated separately, distinguishing surviving source evidence from recovery of earlier sources after replacement (Appendices C.5 and C.6).

The expanded candidate-pool evaluation also shows useful source resolution. With eighteen candidates, SCOUT selects the correct teacher for 8/12 descendants and retains it among the top two for all twelve, requiring success on both math probes (Appendix D.3). Tests involving directly distilled relatives, teacher absence, and deliberate response reformatting specify the interpretation of these assignments. They distinguish same-family competition from direct source relatives, identification from abstention, and content origin from imposed formatting. Reference comparisons further demonstrate that the training stage of the reference checkpoint matters for retrospective abstention.

Clearer experimental descriptions and terminology. Sections 5 and 6 now distinguish candidate identification, accepted attribution with abstention, and known-teacher margin changes across released checkpoints. We define the metrics and holdout protocols explicitly and separate the controlled students we train from publicly released models. Section 7 discusses the remaining challenges in relation to the available model access.

We thank the reviewers again for their time and consideration. We hope the additional experiments and clarifications address your concerns and would appreciate your taking them into account in your final assessment.

Sincerely,

The Authors

AuthorsOpenReview ↗

Final Summary of Our Responses

We thank the reviewers for recognizing the value of retrospective distillation attribution and the practicality of our output-only approach. The revised manuscript expands the evaluation across public teacher lineages, model scales, and subsequent training stages. With only a few minutes remaining in the discussion period, we have not received responses from some reviewers. Therefore, we share this final summary of our additional experiments and revisions. We believe these additions substantially address the original questions and strengthen the evidence supporting our contribution.

Regarding the latest comment from Reviewer VQiA, an investigation of suspected distillation from a provider’s own model would retain that model in the candidate pool. Our positive and negative controls support distinguishing R1 distillation from shared ancestry and reasoning RL. The added Magpie evaluations also demonstrate exact teacher identification with same-family competitors.

Furthermore, we addressed the other questions through three groups of additions:

Broader evaluation. We added public teacher lineages beyond R1, larger students, larger distillation datasets, expanded candidate pools, and mixed-teacher training.

Evidence supporting attribution. Matched controls examine shared questions, ancestry, and training objectives. Comparisons favor syntactic features over embedding alternatives within the same pipeline. GPT-OSS marker-removal controls support attribution beyond visible channel markers. Additional human-dialogue SFT demonstrates that teacher identification can persist without further responses from the original teacher.

Clearer experimental presentation. We clarified the evaluated models, metrics, and holdout protocols, distinguished identification from attribution with abstention, and clarified the scope of claims about teacher identification and persistence across training stages.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.