- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript considers the problem of identifying unauthorized distillation. To this end, the authors proposed a fingerprinting scheme that fine-tunes the teacher model to embed a fingerprint within, and is then able to pass this fingerprint to students who fine-tune on the teacher's outputs. Such fingerprint will become detectable if the students' logits are available on curated evaluation sets. Technically, the fingerprint encodes a tendency to upweight or downweight certain tokens from being generated when a selected subset of neurons are activated. During evaluation, for each such selected neuron, the authors curate two sets of contexts where this specific neuron is and is not activated, and compare if the students' tendency agrees with the embedded fingerprint. Such a signal is then aggregated into a p-value to determine whether distillation has taken place. Empirically, the authors show that, in a few example scenarios using models from Qwen and Llama families as teachers and students, the proposed fingerprinting scheme correctly classifies whether distillation took place.
Critical strengths
- The problem of study in this paper is relevant. Proprietary model providers care much about distillation identification and prevention, and the study comes timely.
Critical weaknesses
- (Major) Important details about the methodology are questionable or missing.
- The method picks a selected subset of neurons based on activation rate and mean activation. But the way those quantities are computed are missing from the draft. It is not even clear what it means for a neuron to be activated in a non-ReLU-activated model.
- The subset contains k neurons, but k is not specified.
- The evaluation computes a "p-value" to determine whether distillation happened. However, its corresponding null hypothesis is never properly stated, nor a proof of that quantity being an actual p-value in a statistical sense. The terminology used here is therefore highly questionable.
- The way to aggregate signal of individual selected neurons into a p-value is not stated.
- In different experiment settings, the exact pipeline where the students are trained is not specified. For example, it is not clear how the data for text-based fine-tuning is generated from the teacher.
- (Major) The attribution to related literature is insufficient.
- In Line 429, the authors stated that "We see this as a shift from post hoc provenance methods, which try to infer a training relationship after the fact, to a proactive one: a model owner who suspects their outputs will be scraped for distillation can decide, in advance, to make that theft detectable." However, [1], which the authors already cite, was able to achieve this back in 2024.
- For the same reason, it should be reasonable that [1] is included as a baseline for comparison. But the paper did not consider any baseline method.
- In related literature, [2] considers almost exactly the same problem with this paper, but is not discussed in the paper.
- Given the above, I find the title "How To Catch Distillation" fails to represent the real scope of this study.
[1] Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, Teddy Furon. Watermarking Makes Language Models Radioactive. In NeurIPS 2024.
[2] Yixuan Even Xu, John Kirchenbauer, Yash Savani, Asher Trockman, Alexander Robey, Tom Goldstein, Fei Fang, and J. Zico Kolter. Antidistillation Fingerprinting. In ICML 2026.
Questions for authors
Q1. Feel free to respond to the weakness raised.
Q2. In Line 114, the authors stated "Our approach differs in where the transferable signal is introduced. Rather than watermarking teacher generations through a modified decoding procedure and subsequently searching for residual watermark statistics in the student, we directly modify the teacher model." What is the advantage of this method against a modified decoding procedure? Presumably, requiring to fine-tune the model introduces additional costs.
We thank the reviewer. We agree with several points and have revised the paper.
All of this, plus probe sizes and data splits, is now in Section 3.
H0: the suspect model M is independent of the secret key K. That is, K is drawn uniformly from the key space by the owner, and M was produced without access to anything that depends on K (for example, it was not trained on the fingerprinted teacher's outputs).
Under H0, the true key and the R random keys are i.i.d. draws from the same distribution, while M is fixed. So the statistic of the true key, T(K), is exchangeable with T(K_1), ..., T(K_R). The rank-based p-value p = (1 + #{r : T(K_r) >= T(K)}) / (1 + R) therefore satisfies P(p <= alpha) <= alpha for every alpha. This is the standard Monte Carlo permutation-test argument and needs no Gaussian assumption. We now report this p-value (R = 100,000) next to the Gaussian-tail value, which matches the empirical null out to +/-3.3 standard deviations. The false-positive rate on 27 negative models is 0/27 at alpha = 0.01.
The reviewer is right. Sander et al. already showed proactive detection through watermark radioactivity, and we removed the sentence at line 429. The revision presents our work as placing the signal in a different location: in the teacher's weights rather than in its decoding procedure.
Thank you for this reference, which we now discuss. ADFP extends the red/green-list watermark at sampling time and uses a proxy student model to choose token biases that a student will learn efficiently after fine-tuning on the teacher's text, improving the trade-off between text quality and detection strength. Like Sander et al., it works at decoding time and targets text-based distillation. Our method instead changes the teacher model once, conditioned on its internal features, so the signal is present in the full output distribution.
Advantages: (a) Logit distillation. When a distiller matches full output distributions, there is no sampled text in which a sampling watermark could appear. Our signal is in the distribution itself, and we detect it after logit distillation within and across model sizes (Table 1). (b) No serving-time key. The provider deploys one fixed fine-tuned model. Detection does not depend on every serving path applying the right sampler and key. (c) Harder to remove from the text. Decoding-time watermarks can be detected and scrubbed from text by the distiller (Gu et al. 2024; Yi et al. 2025). Our signature only shows up as a statistical pattern over feature-specific contexts, and it survives 10M tokens of continued clean fine-tuning and 8-bit quantization. Cost: one LoRA fine-tune per teacher. We measured its effect on capability (new results). Clean vs fingerprinted Qwen2.5-7B: MMLU 74.2 vs 74.0, ARC-Challenge 51.1 vs 50.5, HellaSwag 79.0 vs 78.9, WinoGrande 72.7 vs 72.3, PIQA 80.0 vs 79.8, TruthfulQA 56.3 vs 56.3. All changes are within one standard error.
We agree that a head-to-head comparison with Sander et al. under text distillation would strengthen the paper. We could not complete one during the rebuttal period, and we say so explicitly in the limitations. The two approaches are complementary: decoding-time watermarks need no training and are strong when the distiller uses sampled text, while our method also covers logit distillation and needs no serving-time key.