acceptodds
Under review as a conference paper at ICLR 2027

LUCID: Universal Auditing of Distilled Large Language Models

Abstract

The growing transparency of large language models (LLMs) makes distillation into smaller models an inevitable practice, allowing users to cheaply inherit advanced capabilities such as reasoning. Yet this trend also exposes model providers to new risks: unauthorized data distillation may misappropriate the teacher model’s valuable functions, resulting in copyright violations, privacy leaks, and other serious harms. Existing fingerprinting techniques mainly focus on detecting complete model theft, offering little protection for specific functional capabilities, and many require white-box access, limiting real-world applicability. In this work, we propose LUCID (LLM distillation Unveiled via invarianCe auditor Infringement Detection), the first auditing framework designed for scenarios where the suspect model’s internals are inaccessible, tailored to identifying the misappropriation of a victim model’s specific capability. By utilizing self-reflective prompts to elicit internal judgments, LUCID extracts invariant judge-token representations to distinguish infringing models from non-infringing ones. Theoretical analysis substantiates the generalization ability of our approach, while empirical results demonstrate its effectiveness in reliably identifying unauthorized distillation across diverse model architectures and deployment settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Distillation, Infringement Detection, Intellectual Property Protection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rBDuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LUCID, a framework for auditing whether a suspect language model was trained on outputs from a protected model. LUCID inserts suspect responses into a self-attribution prompt, extracts judge-token representations from the protected model, and trains an MLP classifier across multiple distillation settings. The framework requires only query access to the suspect model and internal access to the protected model. Experiments report strong detection performance across suspect (student) families and sizes, protected-model sizes, prompt styles, mixed-source training, audit question sets, and publicly released models.

Critical strengths
  1. Relevant problem and practical access assumptions. Retrospective auditing without modifying the protected model or embedding watermarks addresses a meaningful limitation of proactive protection methods. Requiring only text responses from the suspect model makes the framework applicable to restricted-access auditing scenarios.
  2. Strong results across several controlled settings. The evaluation includes held-out student families and prompt styles, larger students, mixed-source training, and independently trained test checkpoints. Matching student bases, training questions, and SFT configurations between infringing (positive) and non-infringing (negative) suspect models provides useful experimental controls.
Critical weaknesses
  1. The experiments do not sufficiently establish teacher-specific data provenance. In the main controlled experiments, positive models are trained on protected-model responses, whereas negative models are trained on ground-truth answers. This leaves open whether the detector identifies teacher-specific provenance or generic differences between training on model-generated long-CoT outputs and on reference solutions (e.g., reasoning style or response length). The failure of NFP-FS and NFP-BS (Sections 2.3 and 4.2) does not rule this out, since these are training-free similarity scores rather than supervised classifiers on surface features. In the mixed-source experiment (Section 4.4), the two classes still differ exactly in protected-model versus ground-truth s1k targets. Moreover, in the real-world evaluation (Section 4.5, Table 6), the infringing models are fine-tuned from the protected model itself (Appendix A.11, Table 13), whereas the non-infringing models come from unrelated model families (Table 14). Their detection therefore cannot isolate output-data distillation from weight lineage. These limitations weaken the interpretation of high classification accuracy as evidence of the claimed provenance signal.
  2. The distinctive contribution of judge-token representations is insufficiently isolated. The baselines (lexical similarity NFP-FS, BERTScore NFP-BS, and direct Self-Judge) are all training-free, whereas LUCID trains a supervised MLP on labeled checkpoints. The performance gap may therefore stem from supervision rather than from the self-attribution prompt or the protected model's representations. The ablations in Section 4.6 vary corpus size, feature layer, and audit question set, but not the prompt or the representation source. Controls such as the protected model's hidden states for the raw response without the self-attribution prompt, hidden states from an unrelated language model, or simple supervised surface-feature baselines (e.g., length and n-gram features) with the same classifier would clarify whether these components are necessary.
Questions for authors
  1. With the detector held fixed, what false-positive rate does LUCID achieve on students trained on another teacher's responses to the same questions, with comparable reasoning format and response length? How does LUCID behave on a suspect distilled from the protected model's own upstream teacher (DeepSeek-R1), and how is infringement defined in that case, given that Section 4.5 treats DeepSeek-R1 derivatives as infringing?
  2. How should Table 6 be interpreted given that its infringing models are fine-tuned from the protected model's weights? What evidence distinguishes detection of output-data provenance from detection of weight lineage?
  3. In one representative setting, how does LUCID compare with the same classifier using (a) the protected model's representations of the raw response without the self-attribution prompt, (b) representations extracted by another language model, and (c) supervised surface features such as response length and n-grams?

I'll consider raising score if my questions are addressed.

Reviewer_Epa2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an auditing method, where a suspect model's answers are fed into the protected model with a prompt asking "did you write this". Then, the hidden state of the Yes/No judgment token is fed into an MLP, which is trained to tell models SFT'd on the protected model's outputs from models SFT'd on ground-truth answers based on the hidden state. The authors report roughly 99% accuracy of this method.

Critical strengths
  • The problem studied is timely and practially important.
  • The method is an interesting idea, and it's simple and cheap (it only needs one forward pass and an MLP). Evaluation experiments are fairly comprehensive.
  • The paper presents detailed hyperparameters in the appendix, which makes results easier to reproduce.
Critical weaknesses

1. The negative control confounds "trained on the protected model's outputs" with "trained on long R1-style reasoning traces"

The paper's infringing suspects are trained on DeepSeek-R1-Distill reasoning traces with up to 8192 token, while non-infringing controls are trained on s1k ground-truth answers. These two classes differ in response length, CoT style and format, and so any reasoning trace based detector should be able to separate them. The paper does not provide negative control trained on another teacher's reasoning traces (e.g. the Gemini thinking traces that ship with s1k, or another R1-style model). I believe this is the right test that would show the method detects use of a model's data, instead of just classifying reasoning difference. The near-perfect leave-one-family-out accuracy also suggests that the method might just be a shallow format detector.

1. No specificity test across related teachers

Relatedly to my first point, the paper never checks whether a 14B-protected auditor flags a suspect distilled from the 7B or 32B sibling (which share R1 distillation data), from DeepSeek-R1 itself, or from an unrelated reasoning model. The paper should report false-positive rates based on these experiments

Questions for authors
  • If the non-infringing control is trained on a different teacher's reasoning traces for the same s1k questions (e.g. Gemini thinking traces in s1k, QwQ-32B or Qwen3 outputs), what is LUCID's accuracy?
  • Can the 14B-protected auditor flag suspects distilled from DeepSeek-R1-Distill-Qwen-7B/32B or from DeepSeek-R1 itself? What is the model-level false positive rate for unrelated reasoning models?
  • How does a classifier trained directly on the suspect response text perform under the same splits? I imagine there can be simple methods based on TF-IDF / length / a sentence-encoder embedding / LM with a head and it should be easy to test all of them.
  • How does the same MLP perform on protected-model hidden states without the self-attribution prompt?
Reviewer_8eAoOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces LUCID, a method for detecting whether a suspect language model has been fine-tuned on outputs generated by a protected teacher model. Assuming query access to the suspect model and white-box access to the teacher, LUCID places suspect responses into a self-attribution prompt, extracts the teacher’s hidden representations at the judgment token, and trains an MLP classifier across multiple distillation settings. The controlled experiments compare students trained on teacher-generated answers with students trained on reference answers to the same questions. The authors report strong detection performance across student model families, model scales, distillation prompt styles, and audit question sets, supplemented by evaluations on mixed-source training data and publicly released models.

Critical strengths

The paper addresses a practically relevant problem: detecting the use of teacher-generated training data when only text responses from the suspect model are available. LUCID provides a straightforward pipeline that keeps the protected model frozen and trains a small classifier on its hidden representations, without requiring fingerprints to be inserted in advance. The comparison with direct self-judgment motivates exploring the information contained in these representations. The experiments cover multiple student model families and scales, different distillation prompts, mixed-source training data, and two audit question sets. In particular, the evaluations on held-out model families and prompt styles are useful attempts to assess generalization.

Critical weaknesses
  1. Limited evidence of teacher identification. The detector may distinguish teacher-generated answers from reference answers based on style, length, or reasoning format. Students trained on other teachers’ outputs or independently rewritten answers should be included as negative examples to check for false attribution.

  2. Unclear generalization across training datasets. The paper should train the detector using students distilled from dataset D1, then test it without retraining on students distilled from a different dataset D2 generated by the same teacher. Both should be positive. This would help establish whether the detector recognizes the teacher beyond one particular training corpus.

  3. Missing basic classification baselines. Standard text classifiers and general text encoders should be evaluated using the same labels and test splits. If they perform similarly, the benefit of using the protected teacher’s internal representations remains unclear.

Questions for authors
  1. How do the authors establish that LUCID identifies the target teacher, rather than differences between two particular training corpora? Can it correctly reject students trained on other teachers’ outputs or independently rewritten answers?

  2. Does the detector generalize to a different training dataset generated by the same teacher? Specifically, if it is trained using students distilled from D1, can it correctly identify students distilled from D2 without retraining the detector, where D2 contains different questions within the same domain?

  3. How does LUCID compare with standard text classifiers or general text encoders under the same training and evaluation protocol? What additional benefit comes from using the protected teacher’s internal representations?

I would raise my score if the authors provide convincing evidence addressing these concerns. However, if these core issues remain unresolved, I do not believe the current work meets the bar for acceptance, as its central claim of identifying teacher data usage remains insufficiently supported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.