acceptodds
Under review as a conference paper at ICLR 2027

DistillTrace: Tracing LLM Distillation via Keyed Solution Preferences

14 views

Abstract

Large language model outputs can be collected for unauthorized distillation, motivating providers to determine whether a student was trained on their answers. Yet a watermark detectable in teacher outputs may not survive distillation. Practical tracing therefore requires a learnable signal, broad task coverage, and preserved teacher utility. We introduce DistillTrace, which encodes consistent solution preferences as a watermark. Two LoRA adapters learn response families from quality-filtered, self-distilled answer pairs. Keyed semantic hashing assigns one family to each predicted question category, giving related questions a consistent preference that students can learn through ordinary distillation. A fixed private detector supports black-box tracing and selects watermarked answers from other generators without retraining. Construction and embedding require neither proxy students nor token-level decoding biases. In a matched Qwen-to-Llama comparison on MBPP, aggregating 256 responses yields 99.88% AUROC and 97.77% TPR against matched clean students. Teacher utility is maintained or improved on all five tasks evaluated in this comparison. Separately, question-count-weighted teacher utility rises from 53.73% to 54.50% across four benchmarks. In multi-domain distillation, the semantic-hashing configuration achieves stronger aggregate student detection than the non-semantic configuration. Across four generators and three tasks held out during adapter and detector training, online selection from four candidates achieves 95.52%-100.00% aggregate watermark-detection TPR over 256 responses, extending embedding without additional training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM watermarking, knowledge distillation, model provenance, model attribution, language models

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_GxpPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work provides a unique way of tracing distillation from proprietary models in student models by making a watermark learnable. It creates 2 response families - an initial response, and an alternative conditioned on the first answer. Both sets of answers are used to train separate LoRA adapters. A private key maps a classifier-produced category to an adapter so that related questions receive answers from the same family. The hypothesis is that this category-dependent preference is reflected in the student model, and a private detector can verify this learnable signal. This work offers a proof of concept but has concerns around reliable provenance attribution.

Critical strengths
  1. The signal is designed to be learnable - a category-dependent response preference is likelier to survive in a student than an arbitrary token bias that may not be picked up.
  2. There is clear student-transfer evidence, even across model families.
Critical weaknesses
  1. Solution preference is not always distinguished from surface-level style changes - A.1 documents an example where the solutions differ only in variable name.
  2. The negative reference is the undistilled Base model - but a more comparable setting may be a student trained on unwatermarked data. This comparison is shown for a single student-teacher configuration.
Questions for authors
  1. Is this method robust to distillation variations such as rephrasing, or using a subset of data from a watermarked teacher, or other realistic data mix strategies?
  2. What fraction of answer pairs differ in solution strategy rather than wording or formatting?
  3. How do you distinguish inheritance of the watermarked preference from independent preferences an unrelated model may exhibit? Does the ~1% FPR hold across independently trained, matched non-keyed students?
  4. Why was a binary design selected as opposed to K response families?
Reviewer_vsznOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a new method for watermarking outputs based on preferences rather than token-level modifications. This preference is hidden during distillation and is inherited by the student models.

Critical strengths

The use of model preferences to train a fixed detector is an interesting contribution, and novel relative to other watermarking methods that are token-level or require white box access to distributions.

Critical weaknesses

The overall contributions beyond establishing preference inheritance are weak: Evaluation (L135) is not a contribution but rather a required assessment of the proposed methodology. There is an insufficient set of ablations to validate this work, and insufficient baselines/comparisons to the existing watermarking methods listed in the related works section. Many of the metrics are not well-defined (e.g., how is "utility" measured?) The writing is quite unclear and difficult to parse at many points. The large range in performance in some cases (e.g., L 370 "TPR is 55.27–100.00%") is concerning and not explored nor explained well in the text. There is a significant missing qualitative assessment of the text generated and performance difference in the trained teacher models.

Questions for authors
  1. What assessments exist to show that training the LoRAs does not degrade regular task performance?
  2. Have you tried ablating each component of your pipeline or replacing it with alternatives? (e.g., what is performance gain from hashing responses into semantic buckets?)
  3. How does this compare to blackbox access watermarking methods like lexical fingerprinting?
Reviewer_HkoaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DistillTrace proposes detecting LLM distillation through keyed response preferences. For a single question, first an answer and another alternative answer are generated from an initial model and these answers are used to train the initial model to obtain two LoRA teachers. Then, a secret key maps predicted question categories to one teacher, creating consistent category-dependent preferences that students may inherit through ordinary supervised fine-tuning. Thus, this routed (using the key) models are provided to a user who may distill from this. For detection, the authors extract several token-wise statistics from responses which can differentiate between clean and watermarked responses using a regression model.

Following experiments are performed to validate this approach :

  • Compare the routed LoRA system with the original model and KGW to evaluate whether watermarking preserves task accuracy and produces detectable answers (Table 1).
  • Generate multiple answers from unadapted models and select one using the detector. Evaluate the selected answers’ utility and watermark detectability on held-out tasks (Table 2).
  • Train Qwen students on teacher-generated answers and evaluate student utility and watermark inheritance, comparing semantic hashing with non-semantic hashing, clean distillation, and KGW (Table 3).
  • Evaluate watermark inheritance and student accuracy across four Qwen/Llama teacher–student pairings on MBPP, including random-selection controls (Table 4).
  • Compare DistillTrace with ADFP, Radioactive, and LIDet in the Qwen→Llama MBPP setting, reporting teacher utility and student detection against matched clean students (Table 5).

While these experiments validate some of the claim, there are several unclear points as elucidated below.

Critical strengths
  • The paper proposes a practical, simple and novel distillation strategy.
  • The method preserves utility of the teacher model almost completely, and much better than other methods.
Critical weaknesses
  • The keyed answer may be easily detectable by an adaptive actor. For example, an adversary may compute reference statistics or representations from queried student outputs, perform some form of clustering and infer the category-conditioned response preferences. It would be very useful to provide an experiment for this.
  • There are several experimental gaps:
    • Missing unwatermarked control i.e. using a random candidate used to calculate detection metrics in Table 2.
    • Please clarify what this means in Table 3: "Training-set sizes differ between configurations"
    • Why are same- and cross-family experiments only restricted to MBPP ?
    • "Because controls also yield positive detections, keyed student scores should be compared with the corresponding non-keyed student rather than read as attribution on their own." I am not sure if this issue is addressed in Table 5 or at all. Please clarify.
    • There is only one setting i.e. Qwen --> Llama in Table 5. Again only MBPP is used for student distillation here.
    • Additionally, it would be great to clarify what the exact setting of Table 5 is and how it differs from Table 4.

Due to these experimental weaknesses, I am recommending a reject. However, I am willing to change my score on further evidence and clarification.

Questions for authors
  • Please address the weaknesses.
  • Please provide the prompt used to generate the two answers.
  • Please provide some additional details on the detector feature vector. I do not understand how its exactly constructed.
  • Please clarify what "matched clean students" mean in Table 5.
Reviewer_uzb5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a method for detecting whether a suspect model was distilled from a teacher. The teacher trains two LoRA adapters on two families of answers. At inference time, a semantic hash is used to assign inputs to LoRAs in a repeatable way. The LoRA is consistent across topics, so a distilled student can learn and inherit a topic-dependent style that a private detector can observe. Experiments on Qwen and Llama across math, code, instruction following, and science show reasonable detection after averaging over a few hundred output responses, with teacher utility preserved and student detection comparable to KGW-based methods.

Critical strengths

I like the core idea. Controlling which valid solution the model prefers, keyed per semantic bucket, is a practical and implementable way to make outputs detectable without distorting text at the token level the way KGW-style watermarks do. Teacher utility is preserved, and the signal works with answer-only APIs.

The preference does seem to be inherited through ordinary distillation, including across model families (Qwen to Llama and back), and the online selection mechanism lets the watermark be applied to outputs from generators without adapters.

Critical weaknesses
  1. The biggest weakness is the lack of a rigorous p-value, and this is a missed opportunity because the design is well suited to producing one. Since the bucket-to-LoRA assignments come from a keyed hash, a model never trained on the teacher's outputs has preferences that are independent of the key, so the number of buckets on which it agrees with the key follows a Binomial(B, 1/2) distribution under the null (a key permutation test would also work). If there were, say, 100 buckets and a student matched nearly all of them, you could report an exact and extremely small p-value, which would let the user control the FPR and keep it very low. This would probably require many more buckets than the current scheme, which looks like a few dozen at most. Instead, the paper uses a logistic-regression score with thresholds calibrated against the undistilled base model, and it doesn't clearly report the FPR on innocent students. The numbers that are there aren't encouraging: random-selection control students are flagged at 3 to 32 percent TPR at a threshold nominally set for 1 percent FPR, and clean distillation is flagged at about 20 percent in the multi-domain experiment. In practice what matters is sensitivity in the very low FPR regime, and operating with an effective FPR of 10 percent or more on innocent fine-tuned models isn't very motivating for an attribution tool. With a P-value based method, you could operate at really, really low FPR and have a rigorous guarantee of this low FPR.

  2. The writing is at times impenetrable. I had to re-read sections multiple times, or read further into the paper and come back, to understand them, and even after a careful read of the whole paper some sentences are still gibberish to me. For example, from Section 3.3: "The two LoRA states are oriented as keyed and opposite according to b. The token observations are summarized in a fixed-length feature vector phi(x,y,b) of within-state choice statistics, comparisons with B, aggregate likelihood statistics, and route-sign interactions." I feel like a need a translator to figure out the intended meaning of many words in this sentence. The paper is chock full of spans like this one.

Questions for authors

What is the FPR on innocent students (e.g. models fine-tuned on similar data from another source, or on the teacher's outputs under a different key), and what is the TPR at very low FPR such as 0.1 or 0.01 percent? How many semantic buckets are there in total?

Suggestion: It would be worth studying how inheritance changes as the number of buckets grows, since more buckets would give much stronger p-values but fewer training examples per bucket for the student to learn from.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.