- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether an LLM was distilled from a candidate teacher by comparing the likelihood of teacher-generated responses under the student and a reference checkpoint. It reports strong closed-set teacher identification using reference normalization and proposes a calibrated top-two score margin for distillation detection. Experiments cover controlled distillation and models with documented distillation histories, with additional analyses of prompting mismatch and a Unicode-based diagnostic.
Critical strengths
-
The approach is simple and addresses an important problem. It requires neither teacher weights nor the original distillation dataset, although likelihood access to the student and reference models is needed.
-
The identification results are strong: near-perfect performance on the 19 retained controlled students and correct identification of DeepSeek-R1 for all seven evaluated real-world students. The separate final-answer-only evaluation also suggests that access to reasoning traces is not essential.
-
The ablations on reference choice and probe mismatch are informative. I also appreciate that the authors report unsuccessful threshold transfer and explicitly qualify the exploratory attributions.
Critical weaknesses
W1. Attribution does not yet transfer reliably across teachers
My main concern is the reliability of attribution when the teacher differs from those used for calibration. In Table 6, leave-one-teacher-out calibration yields 5 false attributions among 19 students when the true teacher is removed from the candidate pool. These are model-level calls made by the full two-prompt detector. This is concerning for provenance auditing, although the small sample across three teacher groups does not establish a deployment error rate.
Teacher coverage is narrower than the student count suggests. The controlled study uses three ground-truth teachers in a four-candidate pool, while all seven students in the main real-world identification experiment share DeepSeek-R1 as their teacher, despite a ten-candidate pool. The zero-false-attribution results under controlled leave-one-student-out calibration and in the real-world teacher-removed evaluation are encouraging, but neither establishes generalization to unseen teachers.
Appendix B.5 broadens coverage, but evaluates each true teacher with the other true teachers removed and reports mixed DPO results. Robustness to candidate-pool composition and closely related alternatives therefore remains unclear. This matters because adding or removing a strong alternative can change the top-two margin and hence the detection decision.
It also remains unclear whether a large margin reflects teacher-specific training or broader similarities in task and response format. The teacher-removed negatives are useful, but further checks of pool sensitivity and the calibration trade-off would help assess the specificity of the detector.
W2. The statistical tests do not yet support the stated interpretation
Section 3.3 and Appendix C formulate detection as a test of whether the mean paired score difference exceeds a calibrated threshold, but switch to a Wilcoxon signed-rank test when normality is rejected. The latter is not generally a test of the same mean hypothesis; its usual location interpretation requires symmetry assumptions [2]. Rejecting normality does not establish those assumptions.
In addition, the top two candidates are selected using the same probes used for testing. It is not clear why the K−1 correction is sufficient for this complete selection-and-testing procedure. A justification or empirical calibration of its error rate would help.
These issues leave the interpretation of the detection p-values uncertain, although they do not invalidate the identification results. Even a valid rejection of the score-margin null would not by itself establish teacher usage; that interpretation also depends on the specificity of the calibration and controls.
Questions for authors
-
Statistical reanalysis. Would inference targeting the mean margin and accounting for candidate selection change the model-level decisions in Tables 6 and 15? Please clarify the assumptions behind the revised analysis. Sharing the per-probe scores and analysis script would help assess this.
-
Conservative calibration. Would a more conservative rule, fitted only on the calibration folds, reduce the leave-one-teacher-out false attributions, and at what cost in detecting true positives? A comparison using the existing splits and the full two-prompt decision rule would be informative.
-
Candidate-pool sensitivity. How stable are the attributions to changes in candidate-pool size and composition, particularly the inclusion of related models? One useful check would hold the original fold-calibrated margin thresholds fixed while recomputing candidate selection and the multiplicity adjustment for each subset's actual K.
5 comments
Response to Reviewer (Part 2/4)
Weakness 1
Teacher coverage is narrower than the student count suggests. The controlled study uses three ground-truth teachers in a four-candidate pool, while all seven students in the main real-world identification experiment share DeepSeek-R1 as their teacher, despite a ten-candidate pool. The zero-false-attribution results under controlled leave-one-student-out calibration and in the real-world teacher-removed evaluation are encouraging, but neither establishes generalization to unseen teachers.
The narrow coverage reflects the current landscape of strong LLMs, many of which are suspected of being distilled from one another.
Increasing the coverage of the teacher was challenging, because many potential teacher models have been suspected to distill from one another, e.g., OpenAI and Anthropic suspecting Chinese models are distilled from their models. We therefore selected controlled teachers from distinct U.S. organization families, where legal constraints limit cross-company distillation, plus one Chinese model for diversity. For our real-world models, DeepSeek-R1 is the teacher with the most publicly documented distilled models, which is why it dominates that evaluation.
We agree, however, that more teachers strengthen the generalization claim. We therefore added two real-world distilled models neither of which was distilled from DeepSeek-R1, verified from their model cards:
Intellect-Math-SFT: https://huggingface.co/PrimeIntellect/INTELLECT-MATH-SFT
Teacher: QwQ 32B Preview
Qwen3-Shallow-8B-SFT: https://huggingface.co/tokyotech-llm/Qwen3-Swallow-8B-SFT-v0.2
Teacher: GPT-OSS-120B
We use each model’s pre-SFT checkpoint as the reference model.
| Model (true teacher) | Agg. (OMI / s1) | Per-sample (OMI / s1) |
|---|---|---|
| INTELLECT-MATH-SFT (QwQ-32B-Preview) | 100 / 100 | 100 / 100 |
| Qwen3-Swallow-8B-SFT (GPT-OSS-120B) | 100 / 100 | 100 / 100 |
We scored both models against our same 10 candidate teacher pool. Identification is perfect for both new teachers.
| Held-out teacher (models) | Tau fit on | Detected | FP |
|---|---|---|---|
| DeepSeek-R1 (7 distills) | INTELLECT-MATH-SFT, Qwen3-Swallow-8B-SFT | 7/7 | 0/7 |
| QwQ-32B-Preview (INTELLECT-MATH-SFT) | 7 R1 distills, Qwen3-Swallow-8B-SFT | 0/1 | 0/1 |
| GPT-OSS-120B (Qwen3-Swallow-8B-SFT) | 7 R1 distills, INTELLECT-MATH-SFT | 0/1 | 0/1 |
| Total | 7/9 | 0/9 |
When each teacher is held out of calibration entirely, the detector makes no false attributions (0/9) and generalizes to DeepSeek-R1, detecting all seven R1 distills with a threshold fit only on other teachers' models. This directly addresses the reviewer's concern that our earlier results did not establish generalization to unseen teachers. It holds even though the real-world pool includes models that may share lineage (QwQ-32B-Preview and DeepSeek-R1). The two misses are teachers with a single student each, so this evaluation is still limited, and we agree that broader teacher coverage would strengthen it.
Response to Reviewer (Part 3/4)
Weakness 1
Appendix B.5 broadens coverage, but evaluates each true teacher with the other true teachers removed and reports mixed DPO results.
Multiple Teachers: Our threshold-based detector is designed for the single-teacher case, which we focus on as preliminary work in this area. Even then, when we score the Olmo-3 Think-SFT models with all three true teachers (QwQ-32B, Qwen-3-235B, DeepSeek-R1) in the twelve-candidate pool, they consistently rank in the top 3 for both probe sets as shown from table 13. We leave calibrated multi-teacher detection to future work.
DPO: beyond the Olmo DPO setting in Table 14, we ran a controlled DPO experiment. We trained Llama-3.2-3B-Instruct with DPO only on the same 1,000 OMI-2 prompts, with the teacher's response as the chosen answer and the student's own response as the rejected answer for each of the three controlled teachers.
| Teacher Model | Agg. | Per Sample |
|---|---|---|
| GPT-OSS-120B | 100 | 100 |
| Qwen-3-8B | 100 | 100 |
| Llama-3.3-70B-Instruct | 100 | 92 |
The true teacher is ranked first for all three DPO students, so our method’s teacher identification is shown to transfer to non-SFT distillation pipelines such as preference optimization.
Weakness 1
It also remains unclear whether a large margin reflects teacher-specific training or broader similarities in task and response format. The teacher-removed negatives are useful, but further checks of pool sensitivity and the calibration trade-off would help assess the specificity of the detector.
We test this concern from both directions.
Same task and format, no teacher. We finetuned four students (Gemma-3-4B-PT, Llama-3.2-3B-Instruct, Qwen-2.5-1.5B, Qwen-2.5-3B) with our identical training pipeline but on 1,000 human-written solutions from the MATH dataset. These share the task, solution format and training objective of our distilled students but were never exposed to any of the candidate teacher's outputs. Their top-two margins (0.0002 - 0.019) are far below the calibrated threshold (0.0665), and no teacher is detected for any of them.
Same teacher, different style. In Table 3 of our paper, students are distilled from Llama-3.3-70B-Instruct using the OMI-COT prompt (Toshniwal et al., 2024), which suppresses Llama's characteristic Markdown style ("## Step N" headings), while detection uses the teacher's default-style outputs. If our signal were driven by style, identification should collapse fully. Accuracy does decrease in this setting, but the teacher signal persists (65.9 Per-sample vs. 25.0 chance), and with student-generated exemplars the true teacher is recovered for three of four students (75.0 Agg.).
Both of these results show that our method detects teacher-specific training rather than shared task or style.
Response to Reviewer (Part 1/4)
We sincerely thank the reviewer for the thorough review, and for noting that our approach addresses an important problem and gives strong teacher identification results.
Weakness 1 / Question 2
As the reviewer suggests, we reran the identical evaluations and changed only the calibration rule. We used the same 19 controlled students with the leave-one-teacher-out and leave-one-student-out folds of Table 6, the same 7 real-world models with the leave-one-student-out folds of Table 15, and the same two-prompt detection rule. Tau is selected on the calibration folds with a new criteria (F0.5 or F0.6) instead of F1 to minimize false positives.
Detected = the top-two margin is significantly above tau (p* < 0.05) on both prompt sets, naming the true teacher; FP (false positives) = a teacher is still detected after the true teacher is removed.
To answer the reviewer's question directly: yes, a more conservative rule removes all 5 leave-one-teacher-out false attributions. The cost is small: F0.5 misses 2/19 detections under leave-one-student-out, and F0.6 loses no detections across all splits.