- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a systematic study of whether distillation source detectability survives realistic post-training pipelines, finding that detectability is largely preserved except under same-domain multi-source mixing, with likelihood-based signals being fragile under realistic audits while stylistic signals remain robust.
Critical strengths
The motivation is highly meaningful. Distillation is indeed a widely used means by which models appropriate the capabilities of other models, and being able to establish a reasonable trace would be a valuable contribution to the research community.
Critical weaknesses
Weakness 1: The claims of this paper are not well matched by what it actually does. The target models used for validation are too small, the task domains are not sufficiently diverse, and the exploration of OPD is far from adequate. Under these conditions, the authors cannot convincingly answer whether distillation leaves a trace.
Weakness 2: Many works on fingerprinting have discussed this problem from their own perspective — for example, that data, or datasets generated by distilled models, may naturally carry fingerprints that cannot be erased through distillation. I think such works should be included in the comparisons of this paper, as they fundamentally study the same traceability problem and are equally applicable to the distillation setting.
Weakness 3: The paper ignores the most obvious adversarial case: a motivated distiller can cheaply wash traces via paraphrasing, back- translation, or semantics-preserving augmentation, which alter lexical form while keeping the content distilled from the source. Such augmented data should still count as distilled, but would evade the paper's robust stylistic signal, directly undermining the IP- enforcement motivation.
Questions for authors
See Weaknesses.
5 comments
Extended OPD case study (Weakness 1): To see whether longer training changes our OPD findings, we conduct an additional experimental OPD run on two of the first stage distilled models (Oss-120b and Ds-v3.2) with Qwen3.5 27B as the OPD source. We increase the number of training prompts to 10000 (10x) and increase the LR to 1e-5 (10x). For the scope of the rebuttal, we limit it to just two first stage distilled models given the time constraint, but for the next iteration of the paper, we plan to expand the longer OPD run to all sources.
We evaluated checkpoints every 1,000 prompts under two reference settings: matched OPD references, which undergo the same second-stage training, and mismatched single-stage references, which do not. Note that in the matched setting, since we only train two longer OPD models (Oss-120b and Ds-v3.2 first-stage distills), we only have one matched null for each audit. Thus, the starting margins for the single-source matched setting will be larger than with the full null set. However, we are more interested in the change in margin over continued OPD, which this proxy experiment illustrates well.
Matched Setting
| OPD prompts | Oss-120b LL | Oss-120b clsf | Ds-v3.2 LL | Ds-v3.2 clsf |
|---|---|---|---|---|
| 0 (first stage only) | .565 | .880 | .668 | .771 |
| 1k | .539 | .859 | .649 | .774 |
| 2k | .518 | .846 | .634 | .771 |
| 3k | .503 | .821 | .622 | .748 |
| 4k | .492 | .803 | .612 | .729 |
| 5k | .485 | .780 | .606 | .718 |
| 6k | .479 | .775 | .601 | .709 |
| 7k | .474 | .766 | .598 | .715 |
| 8k | .471 | .770 | .596 | .704 |
| 9k | .469 | .758 | .594 | .708 |
| 10k | .467 | .757 | .593 | .706 |
Mismatched Setting
| OPD prompts | Oss-120b LL | Oss-120b clsf | Ds-v3.2 LL | Ds-v3.2 clsf |
|---|---|---|---|---|
| 0 (first stage only) | .087 | .623 | .099 | .217 |
| 1k | .095 | .602 | .109 | .220 |
| 2k | .098 | .586 | .113 | .216 |
| 3k | .099 | .561 | .114 | .201 |
| 4k | .099 | .539 | .114 | .181 |
| 5k | .099 | .519 | .113 | .176 |
| 6k | .098 | .514 | .112 | .166 |
| 7k | .098 | .504 | .112 | .176 |
| 8k | .097 | .506 | .112 | .164 |
| 9k | .097 | .495 | .112 | .165 |
| 10k | .097 | .495 | .112 | .166 |
Under matched OPD references, In the matched setting, both LL and classifier margins drop early and then level off as OPD continues. For LL, the drop comes from the reference models becoming better at modeling the unseen audit source traces through OPD even while the audited model’s fit to its own source varies little. This narrows but does not fully close the gap, as seen by the narrowing slowing with continued OPD. For the classifier, the drop comes from the audited model, whose first-stage lexical signal is partly washed out by OPD towards the second stage source. However, this also levels off, which we suspect is due to rollouts being on-policy and the divergence to the teacher converging.
Under **mismatched single-stage references, likelihood margins remain relatively stable, consistent with our original results, while classifier margins decline noticeably then similarly level off with continued OPD **. This supports the observations from the matched settings, as the OPD leaves the audited model’s fit on its first-stage source intact, whereas it slightly drifts away lexically from the first-source (though the drift levels out).
Downstream performance: We also evaluated coding performance before and after extended OPD using HumanEval and HumanEval+ (greedy pass@1):
| Source | Benchmark | SFT | OPD 10k |
|---|---|---|---|
| OSS-120B | HumanEval | .433 | .506 |
| HumanEval+ | .421 | .439 | |
| DS-v3.2 | HumanEval | .659 | .738 |
| HumanEval+ | .604 | .689 |
Both students improve on these coding benchmarks, suggesting that the longer OPD stage changes model capability while earlier source traces remain detectable.
We will include new results/claims with the complete longer OPD model set in the next revision of the paper
Relationship to fingerprinting work (Weakness 2): We agree this literature is closely related and will expand our discussion of it. We clarify that our three similarity scores are directly drawn from natural fingerprinting works: the classifier is a common technique for learning lexical source idiosyncrasies [1, 2], the likelihood score is directly drawn from [3, 4], and the embedding score closely resembles the semantic representation intuition from [5]. We describe these in Section 3.2 and will make the connection explicit in the Related Works section. We highlight that watermarking and deliberate fingerprinting strategies also tackle the same traceability problem, but require the source owner to embed the signal in advance and support auditing only from the owner’s side. We discuss this direction in the Related Works section as well. Our scope is black-box signals that do not assume prior intervention by the source owner and we believe the three scoring methods provide good coverage over the (lexical, logit, and semantic) surfaces where detection signals may exist. We save other directions, such as watermarking, to future work. If the reviewer considers a specific method within this scope to be missing, we would be glad to discuss them and, where feasible, evaluate it on our testbed.
Adversarial trace removal (Weakness 3): We agree that this is a practical threat model and our study currently does not test it. We clarify that deliberate adversarial trace removal is outside our scope; for this work, we focus on understanding how common post-training decisions impact detectability of natural, non-adversarial distillation. Our setting is a prerequisite to studying specific subsettings such as adversarial rewriting since whether an adversary can remove detectability only becomes a well-posed question once it is established where benign pipelines preserve one. Some recent work [6] also notes that source-specific token choices largely persist under rewriting by another model, which makes the trade-off between detectability, paraphrase strength, and downstream performance an interesting follow-up direction. We will discuss the non-adversarial scope in the limitations section and acknowledge it as future work.
[1] Sun, M., Yin, Y., Xu, Z., Kolter, J. Z., & Liu, Z. (2025). Idiosyncrasies in large language models. arXiv. https://arxiv.org/abs/2502.12150
[2] Panfilov, A., Schmotz, D., Shumailov, I., Beurer-Kellner, L., Schaeffer, J., Prabhu, A., Geiping, J., & Andriushchenko, M. (2026). Stealing reasoning traces from proprietary LLM APIs. arXiv. https://arxiv.org/abs/2608.09867
[3] Rawat, R., Chen, S., Anand, A., Duan, M., Rotsted, B., & Min, S. (2026). Reference-based distillation detection in LLMs. arXiv. https://arxiv.org/abs/2607.09692
[4] Liu, K., Yan, S., Miao, R., Wang, B., Shen, C., Zhang, J., & Ye, J. (2026). Where did this sentence come from? Tracing provenance in LLM reasoning distillation. arXiv. https://arxiv.org/abs/2512.20908
[5] Wu, Z., Zhao, H., Wang, Z., Guo, J., Wang, Q., & He, B. (2026). LLM DNA: Tracing model evolution via functional representations. arXiv. https://arxiv.org/abs/2509.24496
[6] Tang, Y.-C., & Yen, A.-Z. (2026). Imprinting: Knowledge distillation can shift student behavior. arXiv. https://arxiv.org/abs/2610.18517
We thank the reviewer for their feedback. We address each concern below.
Experimental scope (Weakness 1): Our goal is not to establish whether distillation is universally detectable, but to characterize how source detectability changes across realistic post-training decisions. To study this at breadth, we construct a lineage-known testbed of 166 models spanning ten source models, five student targets (1B–7B), four task domains, and multiple post-training strategies.
Target size. While compute limitations prevent us from distilling over larger base models (e.g., 32B+), small models such as those at the scale we study are still practical and popular targets for distillation. Furthermore, our preliminary trends from 1.5B to 7B indicate detectability is stable and even slightly increases as the target model size increases. However, we will clarify the 7B ceiling as a limitation and agree it would be valuable to see if the trends continue at even larger distillation target scales, but leave this for future work.
Domain diversity. This is an axis we explicitly scale over within our computational limits. Our four domains — code, math, if/chat, and science — are common, high-utility choices for distillations, and we also study how detectability changes when domains are mixed. We agree that coverage over more domains would make interesting future research directions.
In general, we believe these experiments provide useful evidence about the conditions under which distillation leaves detectable traces in a controlled, documented setting. We agree with the reviewer that there is room to scale the testbed further; however, to our knowledge, it already covers the most diverse set of post-training settings studied with known lineage, and we release it to support future extensions. If there is a particular scale, domain, or other aspect the reviewer considers essential, or expects to behave differently, we would welcome the specifics so we can discuss them further or prioritize them in the revision.
We discuss extended OPD results in a followup comment.