acceptodds
Withdrawn from ICLR 2027

WMMD-Bench: Do LLM Ownership Watermarks Survive Model Distillation?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern LLM watermarking and fingerprinting methods are increasingly used to protect model ownership, yet their robustness to model distillation remains insufficiently understood. In a model extraction setting, an adversary can distill a protected teacher into a fresh student using only teacher supervision, potentially weakening or removing the ownership evidence embedded in the original model. We introduce WMMD-Bench, a systematic evaluation of ten recent active and passive ownership mechanisms under direct hard-label distillation, semantic-preserving response transformation, online soft-logit distillation, and same- and cross-lineage student models. Across native detection metrics, clean controls, utility evaluation, applicability audits, and five 11-checkpoint training trajectories, we find that the evaluated active watermarks are not reliably preserved under model distillation. Their registered signals are substantially weakened or remain absent in distilled students, while richer soft-logit supervision provides only limited and method-dependent recovery. Training trajectories further suggest that this vulnerability is primarily associated with limited watermark acquisition, rather than strong acquisition followed by later forgetting. Passive fingerprints appear substantially more stable, but all five already produce positive detections on clean same-lineage students before distillation, indicating that their apparent robustness can be confounded by pre-existing model similarity. Overall, our results reveal model distillation as a substantial weakness of current active LLM watermarking methods and highlight the need for ownership mechanisms explicitly designed to survive knowledge transfer into independently trained student models. Code and evaluation artifacts are available at [https://anonymous.4open.science/r/WMMD-Bench-FCA6/](https://anonymous.4open.science/r/WMMD-Bench-FCA6/).

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model watermarking, model distillation, model ownership protection, watermark robustness, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PWzFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces WMMD-Bench, an evaluation framework designed to assess whether existing ownership watermarking and fingerprinting techniques for Large Language Models (LLMs) are preserved during model distillation and whether they support provenance verification of the distilled "student models." The benchmark covers five active ownership watermarking methods and five passive fingerprinting techniques; evaluation scenarios include direct response distillation, semantic-preserving response transformation, and soft-logit distillation, utilizing both same-lineage and cross-lineage student models. Additionally, the framework incorporates "clean student model" control groups, native verification metrics, student model utility assessments, and training trajectory analysis to distinguish between successful watermark inheritance and subsequent forgetting phenomena.

Evaluation results indicate that, across the examined settings, active ownership watermarks generally fail to be reliably inherited by distilled student models. In contrast, passive fingerprints often remain detectable, yet they may also yield positive detection results on same-lineage student models that do not contain the watermark. Consequently, the authors argue that the apparent preservation of fingerprints may stem from inherent similarities between models rather than the actual transfer of ownership information through distillation.

Critical strengths
  1. This paper addresses a problem of significant practical value and importance. Investigating whether watermarks persist during model distillation holds direct implications for model extraction attacks and model provenance verification. This represents a challenge within the field of Large Language Model (LLM) ownership protection that has been under-researched yet is of increasingly recognized importance.

  2. The experimental design is rigorously conceived. This study evaluates various ownership protection mechanisms across a range of distillation settings and incorporates components such as "clean student model" control groups, training trajectory analysis, utility assessment, and detector applicability testing. In particular, comparing the distilled student model against a homologous "clean student model" constitutes a sound experimental design, enabling the isolation of the true experimental effects.

Critical weaknesses
  1. The work is insufficiently developed as a benchmark. Its core contributions are limited to a 20,000-question extraction budget, Ba/XBa distillation settings, and clear controlled experiments. While these analyses and evaluation designs are valuable, they fall short of constituting a comprehensive benchmark. The paper fails to clearly articulate the source, distribution, rationale, category diversity, or real-world applicability of the 20,000 questions; furthermore, it remains unclear whether the scale is sufficient to support realistic model distillation scenarios.

  2. The evaluation setup is too limited to support generalized conclusions. Distillation schemes should encompass a more diverse range of teacher-student model combinations rather than relying on a single workflow; the latter risks yielding results applicable only to specific models, failing to truly assess the capabilities of the watermarking generation method. The scales of the teacher and student models employed are also quite limited; a "3B-to-3B" distillation experiment offers little insight for real-world applications. Overall, the work resembles a preliminary exploratory experiment rather than a mature benchmark. Its conclusions rest on a highly restricted experimental setup, failing to demonstrate the generalizability of the results.

  3. The quality of the paper's presentation is poor. I suspect the paper may have originated from an automated writing process lacking necessary human oversight. For instance, nearly ten lines of space are wasted on page 6 without apparent manual formatting adjustments; the text in Figure 1 is blurry, and the figure's design is redundant and unclear.

Questions for authors

Clarification Requests

  1. What is the source and distribution of the 20,000 extraction queries? Please report their task categories, length and difficulty distributions, and explain why this dataset and budget are representative of realistic model-extraction settings.

  2. How sensitive are the conclusions to the extraction budget and teacher-student configuration? Could the authors evaluate multiple query budgets and additional model families/scales to determine whether the observed failures are specific to the current 3B-to-3B settings?

  3. Why does the benchmark exclude ADFP, which is explicitly designed for fingerprint acquisition through distillation? Including this closely related method, or clearly justifying its exclusion, is important for supporting the broader claim that existing ownership mechanisms cannot reliably enable distilled-model provenance verification.

Suggestions for Future Work

  1. Develop a distillation-resilient ownership method. The observed failure cases could motivate a new watermarking or fingerprinting mechanism explicitly designed for acquisition through ordinary teacher responses, while remaining specific enough to avoid positive detections on clean same-lineage models.

  2. Develop WMMD-Bench into a more comprehensive benchmark. This would require clearly documented and realistic extraction datasets, multiple extraction budgets, broader teacher-student model families and scales, stronger and more diverse distillation protocols, and repeated-seed evaluations. Such an expansion would provide more reliable and generalizable evidence to the community.

Reviewer_zwCYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an important question: whether active watermarks and passive fingerprints remain detectable after model distillation. It evaluates the sensitivity of five active watermarking methods and five passive fingerprinting methods to distillation. The experimental results provide useful empirical evidence for understanding the robustness of existing ownership protection mechanisms.

Critical strengths
  1. The paper is generally well organized and easy to follow.
  2. It addresses a practically important problem concerning model ownership verification under knowledge distillation.
Critical weaknesses
  1. Benchmark reusability and extensibility are insufficiently demonstrated. The contribution currently appears closer to an empirical analysis of watermark and fingerprint robustness under distillation than a reusable benchmark. The paper does not sufficiently demonstrate a modular design or standardized interfaces that would facilitate the integration of new watermarking and fingerprinting methods and support future extensions.

  2. The effects of supervision and distillation algorithms are not adequately disentangled. The Ba, Bb, and Bc configurations specify both the information provided by the teacher and a particular distillation procedure. A natural question is whether different distillation algorithms, given the same supervision, would lead to different watermark or fingerprint inheritance outcomes. This question is not sufficiently investigated. In particular, the comparison between Ba and Bc changes both the supervision information and the optimization objective, making it difficult to distinguish their respective effects.

  3. The evaluation provides limited comparability across methods. Retaining each method’s native verification metric is appropriate. However, a benchmark would also benefit from a common model-level ownership verification protocol and comparable evaluation metrics, such as detection rates at a fixed false-positive rate under comparable evaluation conditions. Such measurements would make the relative strengths and limitations of different methods clearer. Without them, the benchmark’s usefulness for practitioners selecting an ownership protection method may be limited.

  4. The relationship between knowledge transfer and ownership detectability remains unclear. How does the weakening of watermark or fingerprint signals relate to the extent to which the student acquires the teacher’s capabilities? In particular, can ownership still be reliably verified when the student effectively acquires those capabilities? Although Appendix D reports results on ARC-Challenge and TruthfulQA, the utility evaluation is limited to two downstream benchmarks. I suggest adding evaluations that more directly measure the transfer of teacher knowledge or behavior, together with matched controls, to better characterize this relationship.

  5. There are several minor presentation issues. Figure 1 contains text that is too small to read comfortably at normal viewing size. In addition, Table 2 is not explicitly cross-referenced in the main text.

Questions for authors
  1. Would different distillation algorithms lead to different watermark or fingerprint inheritance effects under the same supervisory information?
  2. What is the relationship between the weakening of the watermark or fingerprint signal and the student's acquisition of teacher capabilities? If the student model effectively acquires the teacher's capabilities, can copyright still be verified?
  3. How can a new watermarking or fingerprinting method be integrated into this benchmark?
Reviewer_4VVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces WMMD-Bench, a benchmark evaluating ten model ownership verification mechanisms under direct-response, transformed-response, and soft-logit distillation, including same-family and cross-family students. It reports substantial attenuation of several active watermark signals and shows that high passive fingerprint similarity can reflect shared model initialization.

Critical strengths
  1. Important and practically relevant problem. Understanding whether ownership evidence survives model distillation is valuable for assessing the protection offered by existing watermarking and fingerprinting methods.
  2. Broad evaluation coverage. The benchmark considers ten mechanisms, multiple supervision settings, and different student families. Training checkpoint measurements provide information beyond final detection scores.
  3. Appropriate recognition of methodological limitations. The authors retain method-specific metrics, distinguish unsupported verification interfaces from detection failures, and acknowledge limitations involving shared initialization, threshold calibration, and single-seed experiments.
Critical weaknesses
  1. Signal attenuation is not sufficiently distinguished from ownership verification failure.

    For PN-FP, Table 1 reports 98–110 matches after same-family distillation, compared with only one match for the clean model. Although these values are much lower than the teacher’s 956 matches, they may still provide useful ownership evidence. Appendix A.1 explicitly does not define a model-level decision threshold. Consequently, the results establish attenuation more clearly than they establish unsuccessful attribution. Evaluation at a controlled model-level false-positive rate is needed.

  2. The EverTracer score definition is inconsistent with the reported operating thresholds.

    Appendix A.2 defines the score as qM(x)=exp⁡[−ℓM(x)]q_M(x)=\exp[-\ell_M(x)], which is nonnegative. However, Tables A9 and A14 report negative ROC thresholds. This suggests that an additional transformation or a different scoring function is used but not adequately documented. The relationship between the mathematical definition, implementation, and reported statistics needs clarification.

Questions for authors
  1. Can the authors demonstrate whether the remaining PN-FP signal supports reliable ownership attribution?

    Please evaluate distilled students against multiple independent negative controls at a specified model-level false-positive rate and query budget.

  2. What exact per-example score is used for EverTracer’s ROC analysis?

    Please reconcile the nonnegative score defined in Appendix A.2 with the negative thresholds in Tables A9 and A14, and provide the corresponding scoring implementation or per-example scores and labels. I think resolving this inconsistency is essential for assessing the results.

  3. Can matched controls establish that the students acquired target-teacher information while losing ownership evidence?

    Teacher–student agreement on held-out inputs, teacher-specific capability measurements, and comparisons with students trained on unwatermarked teacher outputs would help separate successful extraction from insufficient transfer.

I will increase my score if the authors can address my concerns, thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.