- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces WMMD-Bench, an evaluation framework designed to assess whether existing ownership watermarking and fingerprinting techniques for Large Language Models (LLMs) are preserved during model distillation and whether they support provenance verification of the distilled "student models." The benchmark covers five active ownership watermarking methods and five passive fingerprinting techniques; evaluation scenarios include direct response distillation, semantic-preserving response transformation, and soft-logit distillation, utilizing both same-lineage and cross-lineage student models. Additionally, the framework incorporates "clean student model" control groups, native verification metrics, student model utility assessments, and training trajectory analysis to distinguish between successful watermark inheritance and subsequent forgetting phenomena.
Evaluation results indicate that, across the examined settings, active ownership watermarks generally fail to be reliably inherited by distilled student models. In contrast, passive fingerprints often remain detectable, yet they may also yield positive detection results on same-lineage student models that do not contain the watermark. Consequently, the authors argue that the apparent preservation of fingerprints may stem from inherent similarities between models rather than the actual transfer of ownership information through distillation.
Critical strengths
-
This paper addresses a problem of significant practical value and importance. Investigating whether watermarks persist during model distillation holds direct implications for model extraction attacks and model provenance verification. This represents a challenge within the field of Large Language Model (LLM) ownership protection that has been under-researched yet is of increasingly recognized importance.
-
The experimental design is rigorously conceived. This study evaluates various ownership protection mechanisms across a range of distillation settings and incorporates components such as "clean student model" control groups, training trajectory analysis, utility assessment, and detector applicability testing. In particular, comparing the distilled student model against a homologous "clean student model" constitutes a sound experimental design, enabling the isolation of the true experimental effects.
Critical weaknesses
-
The work is insufficiently developed as a benchmark. Its core contributions are limited to a 20,000-question extraction budget, Ba/XBa distillation settings, and clear controlled experiments. While these analyses and evaluation designs are valuable, they fall short of constituting a comprehensive benchmark. The paper fails to clearly articulate the source, distribution, rationale, category diversity, or real-world applicability of the 20,000 questions; furthermore, it remains unclear whether the scale is sufficient to support realistic model distillation scenarios.
-
The evaluation setup is too limited to support generalized conclusions. Distillation schemes should encompass a more diverse range of teacher-student model combinations rather than relying on a single workflow; the latter risks yielding results applicable only to specific models, failing to truly assess the capabilities of the watermarking generation method. The scales of the teacher and student models employed are also quite limited; a "3B-to-3B" distillation experiment offers little insight for real-world applications. Overall, the work resembles a preliminary exploratory experiment rather than a mature benchmark. Its conclusions rest on a highly restricted experimental setup, failing to demonstrate the generalizability of the results.
-
The quality of the paper's presentation is poor. I suspect the paper may have originated from an automated writing process lacking necessary human oversight. For instance, nearly ten lines of space are wasted on page 6 without apparent manual formatting adjustments; the text in Figure 1 is blurry, and the figure's design is redundant and unclear.
Questions for authors
Clarification Requests
-
What is the source and distribution of the 20,000 extraction queries? Please report their task categories, length and difficulty distributions, and explain why this dataset and budget are representative of realistic model-extraction settings.
-
How sensitive are the conclusions to the extraction budget and teacher-student configuration? Could the authors evaluate multiple query budgets and additional model families/scales to determine whether the observed failures are specific to the current 3B-to-3B settings?
-
Why does the benchmark exclude ADFP, which is explicitly designed for fingerprint acquisition through distillation? Including this closely related method, or clearly justifying its exclusion, is important for supporting the broader claim that existing ownership mechanisms cannot reliably enable distilled-model provenance verification.
Suggestions for Future Work
-
Develop a distillation-resilient ownership method. The observed failure cases could motivate a new watermarking or fingerprinting mechanism explicitly designed for acquisition through ordinary teacher responses, while remaining specific enough to avoid positive detections on clean same-lineage models.
-
Develop WMMD-Bench into a more comprehensive benchmark. This would require clearly documented and realistic extraction datasets, multiple extraction budgets, broader teacher-student model families and scales, stronger and more diverse distillation protocols, and repeated-seed evaluations. Such an expansion would provide more reliable and generalizable evidence to the community.