- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a novel strategy to evaluate whether a deployed model has been distilled from a target (developer’s own model) without authorization. The proposed technique combines token and model level specifications to determine a distillation case. The pipeline starts by building distilled proxies, identifying model-style tokens, encode behavioral signatures via extracted hidden features, and concludes with a trained editor to determine if the suspicious model was likely distilled from the teacher/developer model.
Critical strengths
- The paper problem formulation is strong, especially with the black-box setting, which enforces the framework applicability.
- The paper is also well written and flows nicely, with board evaluations and ablations.
- Combining both model and token level specifications to learn distilled signatures for the shadow models is a smart approach.
Critical weaknesses
- The only thing I’m worried about is the false positives, which haven’t been explored fairly throughout the paper. Since models might end up learning similar embeddings, features, and thereby output distributions, how can the authors ensure that evaluation results are clear of false positives?
- The evaluation is a bit weak in its current format, only two datasets and four tiny models. It’s unclear how the proposed framework is robust when evaluated on complex datasets and models (e.g., overparameterized instead of a tiny model). A board evaluation would be appreciated to understand how successful and to what extent the framework is in identifying model distillation cases.
- The choice behind the baselines to which the proposed framework is compared against is also unclear. Are the three reported baselines enough and fully covering or is the paper missing any of existing strong baselines?
- Performances seem to collapse under the back translation defense, it’d be great if the authors can provide logical and concrete explanation of this explained behavior.
Questions for authors
- Please provide a discussion, clarification, or even additional results to support the robustness against false positives or how well the proposed framework handles them.
- Please expand the evaluation to at least 1 more complex and challenging dataset
- Please explain your rationale behind the baselines choices.
- Please provide a conclusion of the adaptive adversary results.
2 comments
W4 & Q4: Explanation of back-translation degradation and adaptive adversary results.
We would like to clarify that this behavior is already discussed in Section 5.3 (Impact of Adaptive Adversary), with results reported in Table 4.
The degradation under back-translation follows directly from DLI's underlying mechanism. DLI relies on teacher-specific token probability signatures inherited during distillation. Back-translation modifies teacher-generated responses before distillation, preserving semantic content while altering lexical choices, sentence structures, and token-level patterns. Consequently, students trained on the transformed corpus inherit weaker teacher-specific signatures, making lineage detection more challenging.
Overall, the results demonstrate that DLI is relatively robust to partial stylistic modifications, such as cross-teacher fine-tuning, inference-time rewriting, and lightweight logit perturbation, but remains vulnerable to systematic disruption of teacher-specific token distributions before distillation.
[1] Foley, Myles, et al. "Matching pairs: Attributing fine-tuned models to their pre-trained large language models." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
[2] Nikolic, Ivica, Teodora Baluta, and Prateek Saxena. "Model provenance testing for large language models." Advances in Neural Information Processing Systems 38 (2026): 34126-34153.
[3] Wadhwa, Somin, et al. "Who taught you that? Tracing teachers in model distillation." Findings of the Association for Computational Linguistics: ACL 2025. 2025.
We thank the reviewer for the detailed and constructive feedback. We address these concerns below.
W1 & Q1: Robustness against false positives.
We would like to emphasize that our original submission already provides extensive false-positive evaluations. As shown in Tables 3, 7, and 9--14, DLI achieves a 0% false-positive rate under Kt across all four teacher models and both datasets. Overall, across 80 negative audit decisions, DLI achieves FPRs of 0% under Kt and 2.5% under Kw.
To further strengthen the false-positive evaluation, we consider students distilled exclusively from alternative teachers as challenging negative samples. Specifically, each teacher-specific auditor is evaluated against 30 students distilled from the other three teachers (10 per teacher).
The following matrix reports the proportion of students classified as originating from the target teacher under Kt. Diagonal entries represent TPR, while off-diagonal entries represent FPR.
\text{Auditor / Student Source} & \text{LLaMA3} & \text{GPT-4o} & \text{Pythia} & \text{Qwen2.5} \\\\ \hline \text{LLaMA3} & \mathbf{1.00} & 0.00 & 0.00 & 0.00 \\\\ \text{GPT-4o} & 0.00 & \mathbf{0.80} & 0.00 & 0.10 \\\\ \text{Pythia} & 0.00 & 0.00 & \mathbf{0.90} & 0.00 \\\\ \text{Qwen2.5} & 0.00 & 0.10 & 0.00 & \mathbf{0.70} \end{array}$$ These results provide stronger evidence that DLI captures target-teacher-specific distillation signatures rather than generic similarities arising from domain adaptation or model architecture, while maintaining a low false-positive rate. We will incorporate these additional negative controls and their evaluation details into the revised manuscript. W2 \& Q2: Evaluation on more challenging datasets and models. We would like to clarify that our evaluation includes four diverse teacher models, including LLaMA3-8B, Pythia-12B, Qwen2.5-7B, and GPT-4o-mini, along with 10 distinct student architectures. Thus, our evaluation is not limited to four tiny models. Importantly, DLI identifies distillation lineage through teacher-specific token probability signatures rather than downstream task accuracy. Therefore, task difficulty does not necessarily correspond to auditing difficulty. Nevertheless, more challenging generation tasks may introduce different output distributions and linguistic patterns, making them valuable for evaluating DLI's generalizability. To further strengthen our evaluation, we additionally consider the Finance-Alpaca dataset, which comprises approximately 69,000 financial instruction-response pairs covering diverse finance-related questions and tasks. Using GPT-4o-mini as the target teacher, DLI achieves a detection accuracy of 0.95 under $K_t$ and 0.90 under $K_w$ in this illustrative setting. These results demonstrate that DLI maintains strong detection performance on challenging QA tasks, further supporting its generalizability across different generation settings. We will incorporate the additional evaluation into the revised manuscript. W3 \& Q3: Rationale for baseline selection. We would like to clarify that the rationale for our baseline selection is already discussed in Section 2 (Related Work), Section 5.1 (Experimental Setup), and Appendix A.5 (Baseline Implementation Details). Specifically, we select three representative approaches to model provenance tracing that capture distinct types of behavioral evidence: Matching Pairs[1] examines input–output embedding representations, Model Provenance Testing[2] uses statistical hypothesis testing to assess model similarity, and Part-of-Speech Templates[3] exploits syntactic patterns inherited through distillation. We also provide implementation details in Appendix A.5, including the adaptations necessary to evaluate these methods under our binary lineage-verification setting. Thus, our baseline selection is grounded in the most relevant existing provenance-tracing approaches rather than an arbitrary choice of comparison methods.