acceptodds
Under review as a conference paper at ICLR 2027

Reference-Based Distillation Detection in LLMs

17 views

Abstract

Model distillation—training on outputs from stronger third-party models—is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
distillation detection, membership inference, teacher attribution, large language models, model provenance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1RJTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether an LLM was distilled from a candidate teacher by comparing the likelihood of teacher-generated responses under the student and a reference checkpoint. It reports strong closed-set teacher identification using reference normalization and proposes a calibrated top-two score margin for distillation detection. Experiments cover controlled distillation and models with documented distillation histories, with additional analyses of prompting mismatch and a Unicode-based diagnostic.

Critical strengths
  1. The approach is simple and addresses an important problem. It requires neither teacher weights nor the original distillation dataset, although likelihood access to the student and reference models is needed.

  2. The identification results are strong: near-perfect performance on the 19 retained controlled students and correct identification of DeepSeek-R1 for all seven evaluated real-world students. The separate final-answer-only evaluation also suggests that access to reasoning traces is not essential.

  3. The ablations on reference choice and probe mismatch are informative. I also appreciate that the authors report unsuccessful threshold transfer and explicitly qualify the exploratory attributions.

Critical weaknesses

W1. Attribution does not yet transfer reliably across teachers

My main concern is the reliability of attribution when the teacher differs from those used for calibration. In Table 6, leave-one-teacher-out calibration yields 5 false attributions among 19 students when the true teacher is removed from the candidate pool. These are model-level calls made by the full two-prompt detector. This is concerning for provenance auditing, although the small sample across three teacher groups does not establish a deployment error rate.

Teacher coverage is narrower than the student count suggests. The controlled study uses three ground-truth teachers in a four-candidate pool, while all seven students in the main real-world identification experiment share DeepSeek-R1 as their teacher, despite a ten-candidate pool. The zero-false-attribution results under controlled leave-one-student-out calibration and in the real-world teacher-removed evaluation are encouraging, but neither establishes generalization to unseen teachers.

Appendix B.5 broadens coverage, but evaluates each true teacher with the other true teachers removed and reports mixed DPO results. Robustness to candidate-pool composition and closely related alternatives therefore remains unclear. This matters because adding or removing a strong alternative can change the top-two margin and hence the detection decision.

It also remains unclear whether a large margin reflects teacher-specific training or broader similarities in task and response format. The teacher-removed negatives are useful, but further checks of pool sensitivity and the calibration trade-off would help assess the specificity of the detector.

W2. The statistical tests do not yet support the stated interpretation

Section 3.3 and Appendix C formulate detection as a test of whether the mean paired score difference exceeds a calibrated threshold, but switch to a Wilcoxon signed-rank test when normality is rejected. The latter is not generally a test of the same mean hypothesis; its usual location interpretation requires symmetry assumptions [2]. Rejecting normality does not establish those assumptions.

In addition, the top two candidates are selected using the same probes used for testing. It is not clear why the K−1 correction is sufficient for this complete selection-and-testing procedure. A justification or empirical calibration of its error rate would help.

These issues leave the interpretation of the detection p-values uncertain, although they do not invalidate the identification results. Even a valid rejection of the score-margin null would not by itself establish teacher usage; that interpretation also depends on the specificity of the calibration and controls.

Questions for authors
  1. Statistical reanalysis. Would inference targeting the mean margin and accounting for candidate selection change the model-level decisions in Tables 6 and 15? Please clarify the assumptions behind the revised analysis. Sharing the per-probe scores and analysis script would help assess this.

  2. Conservative calibration. Would a more conservative rule, fitted only on the calibration folds, reduce the leave-one-teacher-out false attributions, and at what cost in detecting true positives? A comparison using the existing splits and the full two-prompt decision rule would be informative.

  3. Candidate-pool sensitivity. How stable are the attributions to changes in candidate-pool size and composition, particularly the inclusion of related models? One useful check would hold the original fold-calibrated margin thresholds fixed while recomputing candidate selection and the multiplicity adjustment for each subset's actual K.

5 comments
AuthorsOpenReview ↗

Response to Reviewer (Part 1/4)

We sincerely thank the reviewer for the thorough review, and for noting that our approach addresses an important problem and gives strong teacher identification results.

Weakness 1 / Question 2

My main concern is the reliability of attribution when the teacher differs from those used for calibration. In Table 6, leave-one-teacher-out calibration yields 5 false attributions among 19 students when the true teacher is removed from the candidate pool. These are model-level calls made by the full two-prompt detector. This is concerning for provenance auditing, although the small sample across three teacher groups does not establish a deployment error rate.

Conservative calibration. Would a more conservative rule, fitted only on the calibration folds, reduce the leave-one-teacher-out false attributions, and at what cost in detecting true positives? A comparison using the existing splits and the full two-prompt decision rule would be informative.

As the reviewer suggests, we reran the identical evaluations and changed only the calibration rule. We used the same 19 controlled students with the leave-one-teacher-out and leave-one-student-out folds of Table 6, the same 7 real-world models with the leave-one-student-out folds of Table 15, and the same two-prompt detection rule. Tau is selected on the calibration folds with a new criteria (F0.5 or F0.6) instead of F1 to minimize false positives.

CalibrationLeave-one-teacher-out (Detected / FP)Leave-one-student-out (Detected / FP)Real-world (Detected / FP)
F1 (paper)13/19 / 5/1913/19 / 0/196/7 / 0/7
F0.513/19 / 0/1911/19 / 0/196/7 / 0/7
F0.613/19 / 0/1913/19 / 0/196/7 / 0/7

Detected = the top-two margin is significantly above tau (p* < 0.05) on both prompt sets, naming the true teacher; FP (false positives) = a teacher is still detected after the true teacher is removed.

To answer the reviewer's question directly: yes, a more conservative rule removes all 5 leave-one-teacher-out false attributions. The cost is small: F0.5 misses 2/19 detections under leave-one-student-out, and F0.6 loses no detections across all splits.

AuthorsOpenReview ↗

Response to Reviewer (Part 2/4)

Weakness 1

Teacher coverage is narrower than the student count suggests. The controlled study uses three ground-truth teachers in a four-candidate pool, while all seven students in the main real-world identification experiment share DeepSeek-R1 as their teacher, despite a ten-candidate pool. The zero-false-attribution results under controlled leave-one-student-out calibration and in the real-world teacher-removed evaluation are encouraging, but neither establishes generalization to unseen teachers.

The narrow coverage reflects the current landscape of strong LLMs, many of which are suspected of being distilled from one another.

Increasing the coverage of the teacher was challenging, because many potential teacher models have been suspected to distill from one another, e.g., OpenAI and Anthropic suspecting Chinese models are distilled from their models. We therefore selected controlled teachers from distinct U.S. organization families, where legal constraints limit cross-company distillation, plus one Chinese model for diversity. For our real-world models, DeepSeek-R1 is the teacher with the most publicly documented distilled models, which is why it dominates that evaluation.

We agree, however, that more teachers strengthen the generalization claim. We therefore added two real-world distilled models neither of which was distilled from DeepSeek-R1, verified from their model cards:

Intellect-Math-SFT: https://huggingface.co/PrimeIntellect/INTELLECT-MATH-SFT

Teacher: QwQ 32B Preview

Qwen3-Shallow-8B-SFT: https://huggingface.co/tokyotech-llm/Qwen3-Swallow-8B-SFT-v0.2

Teacher: GPT-OSS-120B

We use each model’s pre-SFT checkpoint as the reference model.

Model (true teacher)Agg. (OMI / s1)Per-sample (OMI / s1)
INTELLECT-MATH-SFT (QwQ-32B-Preview)100 / 100100 / 100
Qwen3-Swallow-8B-SFT (GPT-OSS-120B)100 / 100100 / 100

We scored both models against our same 10 candidate teacher pool. Identification is perfect for both new teachers.

Held-out teacher (models)Tau fit onDetectedFP
DeepSeek-R1 (7 distills)INTELLECT-MATH-SFT, Qwen3-Swallow-8B-SFT7/70/7
QwQ-32B-Preview (INTELLECT-MATH-SFT)7 R1 distills, Qwen3-Swallow-8B-SFT0/10/1
GPT-OSS-120B (Qwen3-Swallow-8B-SFT)7 R1 distills, INTELLECT-MATH-SFT0/10/1
Total7/90/9

When each teacher is held out of calibration entirely, the detector makes no false attributions (0/9) and generalizes to DeepSeek-R1, detecting all seven R1 distills with a threshold fit only on other teachers' models. This directly addresses the reviewer's concern that our earlier results did not establish generalization to unseen teachers. It holds even though the real-world pool includes models that may share lineage (QwQ-32B-Preview and DeepSeek-R1). The two misses are teachers with a single student each, so this evaluation is still limited, and we agree that broader teacher coverage would strengthen it.

AuthorsOpenReview ↗

Response to Reviewer (Part 3/4)

Weakness 1

Appendix B.5 broadens coverage, but evaluates each true teacher with the other true teachers removed and reports mixed DPO results.

Multiple Teachers: Our threshold-based detector is designed for the single-teacher case, which we focus on as preliminary work in this area. Even then, when we score the Olmo-3 Think-SFT models with all three true teachers (QwQ-32B, Qwen-3-235B, DeepSeek-R1) in the twelve-candidate pool, they consistently rank in the top 3 for both probe sets as shown from table 13. We leave calibrated multi-teacher detection to future work.

DPO: beyond the Olmo DPO setting in Table 14, we ran a controlled DPO experiment. We trained Llama-3.2-3B-Instruct with DPO only on the same 1,000 OMI-2 prompts, with the teacher's response as the chosen answer and the student's own response as the rejected answer for each of the three controlled teachers.

Teacher ModelAgg.Per Sample
GPT-OSS-120B100100
Qwen-3-8B100100
Llama-3.3-70B-Instruct10092

The true teacher is ranked first for all three DPO students, so our method’s teacher identification is shown to transfer to non-SFT distillation pipelines such as preference optimization.

Weakness 1

It also remains unclear whether a large margin reflects teacher-specific training or broader similarities in task and response format. The teacher-removed negatives are useful, but further checks of pool sensitivity and the calibration trade-off would help assess the specificity of the detector.

We test this concern from both directions.

Same task and format, no teacher. We finetuned four students (Gemma-3-4B-PT, Llama-3.2-3B-Instruct, Qwen-2.5-1.5B, Qwen-2.5-3B) with our identical training pipeline but on 1,000 human-written solutions from the MATH dataset. These share the task, solution format and training objective of our distilled students but were never exposed to any of the candidate teacher's outputs. Their top-two margins (0.0002 - 0.019) are far below the calibrated threshold (0.0665), and no teacher is detected for any of them.

Same teacher, different style. In Table 3 of our paper, students are distilled from Llama-3.3-70B-Instruct using the OMI-COT prompt (Toshniwal et al., 2024), which suppresses Llama's characteristic Markdown style ("## Step N" headings), while detection uses the teacher's default-style outputs. If our signal were driven by style, identification should collapse fully. Accuracy does decrease in this setting, but the teacher signal persists (65.9 Per-sample vs. 25.0 chance), and with student-generated exemplars the true teacher is recovered for three of four students (75.0 Agg.).

Both of these results show that our method detects teacher-specific training rather than shared task or style.

Reviewer_HZefOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether an LLM was trained on another model’s outputs and which candidate teacher was used. Its central score measures the decrease in token-level negative log-likelihood of candidate-teacher outputs under the student relative to a reference checkpoint. Aggregated scores identify the teacher, while a calibrated top-two score margin and statistical tests support distillation detection. Additional components address unknown prompting templates through student-generated exemplars and examine Unicode-related signals associated with o1/o3 outputs. Validation includes 19 retained controlled distillation settings, seven real-world models with known DeepSeek-R1 provenance, robustness ablations, and exploratory multi-teacher, DPO, and undisclosed-provenance case studies.

Critical strengths

Reference-calibrated likelihoods recover teachers across controlled and known-provenance models, including cross-dataset probes. This provides useful empirical evidence that teacher-specific signals persist beyond the original training examples.

The approach requires only teacher sampling and student/reference token probabilities, making auditing possible without provider-side logs or watermark control.

Critical weaknesses
  1. Insufficiently established contribution beyond existing methods. The central score is an established pretrained-reference likelihood calibration [2, 3], and identifying a student LLM’s teacher is already studied in [1]. Although the paper acknowledges reference-based MIA, it does not establish a substantive methodological advance beyond combining these existing elements. The empirical contribution is also difficult to assess without comparison against the directly relevant teacher-attribution approach in [1]. Consequently, the reported improvements over generic similarity and uncalibrated likelihood baselines do not sufficiently establish progress over existing work.

  2. The main likelihood-based evaluation lacks matched negative controls separating teacher-specific training from general adaptation or stylistic similarity. Removing the true teacher from the candidate pool does not establish specificity against non-distillation fine-tuning, such as training on human-written answers from the same domain.

[1] Wadhwa et al. “Who Taught You That? Tracing Teachers in Model Distillation.” Findings of ACL, 2025. [2] Mireshghallah et al. “An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models.” EMNLP, 2022 [3] Ran et al. “LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models.” arXiv:2507.18302, 2025.

Questions for authors

My main concerns are the omission of closely related work and the lack of clearly established methodological novelty.

  1. Methodological novelty: What is new beyond applying reference-based MIA [2, 3] to teacher attribution [1], and how does performance compare with the method in [1]?

  2. Evaluation protocol: Please report all 24 students with both aggregation rules separately. How are aggregate accuracies of 87.5% and 75.0% obtained for seven Llama-teacher students?

  3. Detection reliability: How is the threshold calibrated for unseen teachers? What is the false-positive rate on models trained without candidate-teacher outputs, such as models fine-tuned on human-written answers?

3 comments
AuthorsOpenReview ↗

Response to Reviewer (Part 1/3)

We sincerely thank the reviewer for the detailed review and for noting that our method reliably recovers teachers across controlled and known-provenance models.

Weakness 1 / Question 1

Insufficiently established contribution beyond existing methods. The central score is an established pretrained-reference likelihood calibration [2, 3], and identifying a student LLM’s teacher is already studied in [1]. Although the paper acknowledges reference-based MIA, it does not establish a substantive methodological advance beyond combining these existing elements. The empirical contribution is also difficult to assess without comparison against the directly relevant teacher-attribution approach in [1]. Consequently, the reported improvements over generic similarity and uncalibrated likelihood baselines do not sufficiently establish progress over existing work.

Methodological novelty: What is new beyond applying reference-based MIA [2, 3] to teacher attribution [1], and how does performance compare with the method in [1]?

We do not propose a new scoring method since our score builds on reference-based membership inference. Our contribution is that we formulate LLM distillation detection in a reference-based setting and show that reference-based membership inference works well in detecting distillation. Membership inference was previously used for detecting training data, but we adapt it to detecting LLM distillation. Although distillation detection and membership inference are related problems, they differ in many ways including that neither the distillation data nor templates used during distillation are known. We additionally propose methods that determine whether even occured, handle distillation prompts the auditor doesn’t know about, and use a stylistic feature for detecting distillation from OpenAI models.

We also acknowledge that we missed [1] and thank the reviewer for pointing it out to us! To test our method against this work, we implemented the syntactic method of [1] (POS-tag templates with a logistic regression classifier). We compare it as a baseline to our method on the same candidate pool and probes. For the controlled setting, we distilled six students with our identical training recipe: Llama-3.2-3B-Instruct and Qwen-2.5-3B, each distilled from GPT-OSS-120B, Qwen-3-8B, and Llama-3.3-70B-Instruct on OMI-2. For testing, each student uses 200 s1 probes. For the real-world setting, we used the six official DeepSeek-R1 distills and s1.1-32B on 200 OMI and 200 s1 probes.

Setting[1] templatesOurs
Controlled students (6 models, 4-teacher pool)78.598.8
DeepSeek-R1 distills (6 models, 10-teacher pool), OMI / s147.7 / 65.294.3 / 99.9
s1.1-32B (10-teacher pool), OMI / s170.0 / 81.585.5 / 100.0

These results are with per-probe accuracy: the fraction of probe questions on which each method names the true teacher. Our method outperforms the template method by a large margin in all of our settings. We once again thank the reviewer for pointing us to [1] and will include it as a baseline in the paper and discuss it as a precursor to our work.

AuthorsOpenReview ↗

Response to Reviewer (Part 2/3)

Weakness 2 / Question 3 (second part)

The main likelihood-based evaluation lacks matched negative controls separating teacher-specific training from general adaptation or stylistic similarity. Removing the true teacher from the candidate pool does not establish specificity against non-distillation fine-tuning, such as training on human-written answers from the same domain.

What is the false-positive rate on models trained without candidate-teacher outputs, such as models fine-tuned on human-written answers?

We thank the reviewer for the suggestion and test this concern from two directions.

Same task and format, no teacher (human written answers). We finetuned four students (Gemma-3-4B-PT, Llama-3.2-3B-Instruct, Qwen-2.5-1.5B, Qwen-2.5-3B) with our identical training pipeline but on 1,000 human-written solutions from the MATH dataset. These share the task, solution format and training objective of our distilled students but were never exposed to any of the candidate teacher's outputs. Their top-two margins (0.0002 - 0.019) are far below the calibrated threshold (0.0665), and no teacher is detected for any of them.

Same teacher, different style. In Table 3 of our paper, students are distilled from Llama-3.3-70B-Instruct using the OMI-COT prompt (Toshniwal et al., 2024), which suppresses Llama's characteristic Markdown style ("## Step N" headings). Our detection still uses the teacher's default-style outputs. If our signal were driven just by style, identification should fully collapse. Accuracy does decrease in this setting, but the teacher signal persists (65.9 Per-sample vs. 25.0 chance). Additionally, with our student-generated few shot prompts, the true teacher is recovered for three of four students (75.0 Agg.).

Both of these results show that our method detects teacher-specific training rather than shared task or style.

AuthorsOpenReview ↗

Response to Reviewer (Part 3/3)

Question 2

Evaluation protocol: Please report all 24 students with both aggregation rules separately. How are aggregate accuracies of 87.5% and 75.0% obtained for seven Llama-teacher students?

We re-included the five configurations we initially excluded which did not improve task performance and re-evaluated all 24 students with both of our metrics.

True TeachernAggPer sample
GPT OSS 120B810099.9
Qwen 3 8B8100100
Llama 3.3 70B Instruct8100100
All24100100

Including the excluded students does not change our results. Students whose task performance did not improve are identified by our method as reliably as the other students.

We thank the reviewer for catching the 87.5% and 75.0% entries. These belong to our reference-based string baselines (Multi n-gram Jaccard and Symmetric Coverage); the same applies to reference-based Embedding Cosine (50.0%). For these three baselines, the Llama column was computed over all eight Llama-teacher students (7/8 = 87.5%, 6/8 = 75.0%, 4/8 = 50.0%). However, one student (Gemma-3-4B-PT distilled from Llama-3.3-70B-Instruct on OMI-2) was excluded from our main evaluation because distillation did not improve its performance, so the Llama column should only be over the seven students. We will update the final table according to this.

Question 3 (first part)

Detection reliability: How is the threshold calibrated for unseen teachers?

Our threshold tau is calibrated on training folds only. For unseen teachers, Table 6 reports leave-one-teacher-out calibration: tau is fit on the students of two teachers and tested on the students of the held-out teacher. With the max-F1 rule used in the paper, this produced 5/19 false positives when the true teacher was removed from the candidate pool.

Since false positives are generally costlier for model auditing, we re-ran our identical evaluations and changed only how tau is selected on the calibration folds. We tried using F0.5 instead of F1. The evaluations were the same 19 controlled students under leave-one-teacher-out and leave-one-student-out folds, the same 7 real-world models under leave-one-student-out folds (Table 15), and the same two-prompt detection rule.

CalibrationLeave-one-teacher-out (Detected / FP)Leave-one-student-out (Detected / FP)Real-world, Table 15 (Detected / FP)
F1 (paper)13/19 / 5/1913/19 / 0/196/7 / 0/7
F0.513/19 / 0/1911/19 / 0/196/7 / 0/7

Detected = the top-two margin is significantly above tau (p* < 0.05) on both prompt sets, naming the true teacher; FP (false positives) = a teacher is still detected after the true teacher is removed.

Using F0.5 reduces the number of false positives to 0 in every split with the cost of two fewer detections under leave-one-student-out. Therefore, a threshold calibrated on other teachers can transfer to unseen teachers without producing any false positives. We will update the paper to use F0.5 calibration.

Reviewer_vwA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies reference-based detection of LLM distillation. It compares how well a student and a reference checkpoint predict outputs from candidate teachers, using relative likelihood improvements for teacher attribution and score margins for detection. Experiments cover controlled SFT distillation and publicly documented distilled models, with additional analyses of unknown prompts and real-world attribution.

Critical strengths
  1. The paper is well organized and easy to follow.
  2. Distillation detection is a meaningful research direction with practical relevance to model provenance and auditing.
  3. The method is simple and intuitive: reference-based likelihood comparison helps account for differences in the inherent predictability of teacher outputs and requires only sampling access to candidate teachers.
Critical weaknesses
  1. In industrial settings, the pre-distillation checkpoint is often an unpublished internal model. Public predecessors may differ substantially from the actual starting point, while previously exposed references may cancel teacher signals. Please evaluate detection reliability and false-positive rates under reference mismatch.
  2. Additional SFT, preference optimization, RL, or supervision from other teachers may weaken or overwrite earlier teacher signals. The predominantly single-teacher, final-stage SFT evaluation does not establish reliable historical attribution. Please assess earlier-source detection across different post-training methods and intensities.
  3. When evaluating each true teacher, the experiments exclude the other two true teachers. This does not test simultaneous attribution in the full candidate pool. Moreover, multiple genuine teachers may receive similarly high scores, causing the top-two margin rule to reject valid evidence. Please evaluate joint attribution across different mixture proportions, including low-contribution teachers.
  4. Only 19 of the 24 distilled students are retained because their task performance improves. Students whose performance remains unchanged or declines are still valid positive cases. Excluding them may overstate detection performance. Please report all students and examine how detection success relates to task-performance changes.
Questions for authors

Does a higher reference-normalized likelihood for a teacher’s outputs establish that the student used that teacher’s data? Shared data, similar training objectives, or indirect model inheritance could produce similar signals. Have the authors evaluated hard negatives with similar outputs but no exposure to the target teacher’s data?

2 comments
AuthorsOpenReview ↗

Response to Reviewer (Part 1/2)

We sincerely thank the reviewer for the constructive review and for noting that our paper is easy to follow and that our method is intuitive.

Weakness 1

In industrial settings, the pre-distillation checkpoint is often an unpublished internal model. Public predecessors may differ substantially from the actual starting point, while previously exposed references may cancel teacher signals. Please evaluate detection reliability and false-positive rates under reference mismatch.

We agree with the reviewer's concern. Our contribution isn't a perfectly working distillation detection method but rather an initial work that establishes this problem and presents a method that works well with the assumption of an available pre-distillation model checkpoint. Relaxing this assumption would be an important future work in the area.

That being said, we do provide some initial evidence on reference model mismatch in Appendix B.2 (Table 11). In that experiment, we re-evaluate all 19 of our controlled distilled models using a reference model from a different organization than the true pre-distillation checkpoint.

Table 11 (cross-family, cross-organization references, all 19 controlled students)

StudentCross family reference
Llama 3.2-3B InstructGemma-3-4B-IT
Gemma-3-4B-PTLlama-3.2-3B (base)
Qwen 2.5-3BFalcon3-3B-Base
Qwen 2.5-1.5BSmolLM2-1.7B
Reference ModelLlama-3.3-70B (Agg. / Per-sample)Qwen-3-8B (Agg. / Per-sample)GPT-OSS-120B (Agg. / Per-sample)
Different family and organization100.0 / 100.0100.0 / 99.7100.0 / 99.8

Identification accuracy is nearly perfect across all three teachers despite the major mismatch in reference model. These results indicate that our method still works even when the reference model used has a substantial mismatch between the true pre-distillation checkpoint of the distilled model.

Weakness 2

Additional SFT, preference optimization, RL, or supervision from other teachers may weaken or overwrite earlier teacher signals. The predominantly single-teacher, final-stage SFT evaluation does not establish reliable historical attribution. Please assess earlier-source detection across different post-training methods and intensities.

We agree that this is a fair concern the reviewer brought up, so we decided to test it ourselves. To test if further training overwrites an earlier teacher’s signal, we ran a new experiment where our distilled student models are further fine-tuned on human data.

We distilled Gemma-3-4B-PT from each of our controlled teachers on s1 data with our same SFT training pipeline. We then continued training each distilled model with another round of SFT on 1,000 human-written solutions from the MATH dataset. We use the original Gemma-3-4B-PT as the reference model with our same evaluation setup as in the paper.

TeacherAfter teacher SFT + human-data SFT, OMI (Agg. / Per-sample)After teacher SFT + human-data SFT, s1 (Agg. / Per-sample)
GPT-OSS-120B100 / 100100 / 100
Qwen-3-8B100 / 100100 / 100
Llama-3.3-70B-Instruct100 / 35100 / 98.5

The later SFT does not overwrite the earlier teacher: the original teacher is still ranked first for all three models. For GPT-OSS-120B and Qwen-3-8B, identification is perfect. For Llama-3.3-70B-Instruct, the teacher is still identified (Agg. 100) with lower per-sample accuracy on the OMI held out probes, but still high accuracy on the s1 probes.

These results show that our method is robust to a major later stage of fine-tuning. We do acknowledge that other later stages, such as preference optimization or RL are not tested yet. We will state this as a limitation and potential future work.

AuthorsOpenReview ↗

Response to Reviewer (Part 2/2)

Weakness 3

When evaluating each true teacher, the experiments exclude the other two true teachers. This does not test simultaneous attribution in the full candidate pool. Moreover, multiple genuine teachers may receive similarly high scores, causing the top-two margin rule to reject valid evidence. Please evaluate joint attribution across different mixture proportions, including low-contribution teachers.

Our threshold based detector is designed for single teacher distillation, which we focus on as early work in this area. Even then, when we score the Olmo-3 Think-SFT models with all three true teachers (QwQ-32B, Qwen-3-235B, DeepSeek-R1) in the twelve-candidate pool, they consistently rank in the top 3 for both probe sets as shown from table 13. We leave a more detailed study of multi-teacher distillation detection to future work.

Weakness 4

Only 19 of the 24 distilled students are retained because their task performance improves. Students whose performance remains unchanged or declines are still valid positive cases. Excluding them may overstate detection performance. Please report all students and examine how detection success relates to task-performance changes.

We made this choice because if distillation doesn’t improve the student, a model developer would not choose to use distillation for the model in the first place. However we do agree with the reviewer that these checkpoints can still be tests for our method and we therefore include the results below.

StudentTeacherTrain → ProbeAgg.Per-sample
Gemma 3 4B PTLlama 3.3 70B InstructOMI → s1100/100100
Llama 3.2 3B InstructGPT OSS 120BOMI → s1100/100100
Llama 3.2 3B InstructGPT OSS 120Bs1 → OMI100/100100
Llama 3.2 3B InstructQwen 3 8BOMI → s1100/100100
Llama 3.2 3B InstructQwen 3 8Bs1 → OMI100/100100

All five excluded students are identified perfectly under both aggregation rules, with 100% per-sample accuracy in all cases. Our method’s teacher identification therefore does not depend on whether distillation improved performance in a student.

Question 1

Does a higher reference-normalized likelihood for a teacher’s outputs establish that the student used that teacher’s data? Shared data, similar training objectives, or indirect model inheritance could produce similar signals. Have the authors evaluated hard negatives with similar outputs but no exposure to the target teacher’s data?

Prompted by this suggestion, we ran a matched negative control experiment to verify that our signal comes from teacher-specific training. We did SFT on four students (Gemma-3-4B-PT, Llama-3.2-3B-Instruct, Qwen-2.5-1.5B, Qwen-2.5-3B) with identical training but on 1,000 human-written solutions from the MATH dataset.

These students share the task, solution format and training objective of our distilled students but were never exposed to any of our candidate teachers’ outputs. Their top-two margins (0.0002–0.019) are far below our calibrated threshold (0.0665), and detection is not significant for any of them. They result in 0 false positives. Therefore, fine-tuning on human-written answers from the same domain does not cause our method to falsely detect any teacher.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their helpful reviews. They found the problem important for model provenance, our method’s teacher identification results quite strong, and the paper easy to follow. We respond to each point in the individual reviewer threads and summarize some of the shared concerns along with some new experiments here.

Hard negatives. Many reviewers asked if our teacher signal could come from a shared task or format rather than from exposure to a teacher’s output. To test this, we fine-tuned four student models with an identical training pipeline but on human-written solutions from the MATH dataset. No teacher was detected for all four of our trained students. Their margins (0.0002–0.019) fall far below the calibrated detection threshold (0.0665).

Additional training does not erase the teacher signal. A reviewer asked if additional post-training could overwrite an earlier teacher's distillation signal. To study this, we distilled Gemma-3-4B-PT from each of our controlled teachers with our identical training pipeline. We then continued training the model with a second round of SFT on human-written solutions from the MATH dataset. Even then, our method identified the original teacher for all three distilled students (Agg. 100).

Coverage beyond the original setup. To expand our initial setup, we found two new real-world distilled models with non-R1 teachers (QwQ-32B-Preview, GPT-OSS-120B). Our method identified both teachers for both models perfectly. Our method’s teacher identification also holds when using reference models from different organizations, for DPO-only distillation, and for all 24 controlled students, including the 5 we originally excluded since we did not see increased performance gains in them.

Novelty. We do not claim novelty in reference based membership inference itself. Our contribution is adapting the method to distillation detection and building a full framework around it. This framework includes a threshold based detection, a few-shot prompting method for customized distillation prompts, and a stylistic signal for detecting distillation from OpenAI models. Our method also greatly outperforms the teacher attribution method of Wadhwa et al. (2025) in all settings (e.g., 98.8 vs. 78.5 on controlled students).

We will add all of these results to the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.