acceptodds
Withdrawn from ICLR 2027

Steal the Knowledge, Inherit the Mark: TwinMark for Distillation Watermarking via Second Moments and Class Multiplexing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Knowledge distillation can copy a deployed model by training a student to imitate its logits or features. A watermark can therefore survive only through the output that the student learns to reproduce, yet most distillation watermarks protect only one such surface. TwinMark instead embeds a shared secret payload into the statistics preserved by both objectives: the second moment of normalized features and class-conditioned mean logits. Both channels admit linear readouts with sufficient conditions for bit recovery on fixed audit inputs, while statistical detection is evaluated separately under a declared null. In a ten-seed CIFAR-100 study, each watermark is detected in every student that imitates its corresponding output and remains at chance under the unmatched objective. Marking reduces teacher accuracy by 1.3 points, while the audited embedding of feature-distilled students loses 4.4 nearest-neighbor accuracy points. Second-moment bounds certify 50–60 of 64 feature bits in every feature-distilled student, whereas pointwise feature bounds and the evaluated logit bounds certify none. Class multiplexing, which signs the payload differently across classes, increases the logit decoder's rank ceiling and reduces bit error at matched signal energy. Even at approximately over-encoding, a 1,024-bit payload over 100 classes remains detectable in every evaluated logit-distilled student. Detection further extends to 39 of 40 students across four additional architectures and, through the feature channel, to segmentation, object detection, and satellite-navigation jammer detection and classification. These results establish output-matched watermark inheritance under the evaluated distillation protocols, rather than resistance to all utility-preserving transformations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
model watermarking, knowledge distillation, model ownership verification, model extraction, machine learning security, second-moment statistics, class multiplexing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HYAHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript introduces TwinMark, a watermarking framework designed to preserve watermark information during knowledge distillation. The method embeds the same secret payload into two different output representations: feature second-order statistics and class-conditioned logits. The former targets feature-based distillation, while the latter targets logit-based distillation. The paper also provides theoretical analyses of watermark bit recovery and evaluates the proposed method across several datasets, model architectures, and downstream tasks.

Critical strengths
  1. The paper focuses on an interesting and urgent problem: protecting the intellectual property of well-trained models.
  2. The authors investigate multiple student architectures and provide ablation studies on watermark capacity and different watermark removal techniques.
Critical weaknesses
  1. The idea of embedding watermark information into model outputs or learned representations so that it can survive knowledge distillation has already been extensively explored in previous studies. The proposed TwinMark framework essentially combines two watermarking mechanisms, one operating on feature statistics and the other on logits. However, the underlying idea remains largely consistent with existing distillation-resistant watermarking approaches. The paper does not sufficiently demonstrate why these proposed mechanisms constitute a substantial conceptual advancement over prior work.
  2. A significant weakness is that the proposed feature watermark can be bypassed through a simple orthogonal rotation of the output embeddings. As reported in Table 2, this transformation reduces the watermark detection rate from 10/10 to 0/10 without affecting classification accuracy. Given that the primary objective of the paper is to protect models against unauthorized distillation, this vulnerability substantially limits the practical security guarantees of the proposed framework.
  3. The paper does not adequately cover the broader landscape of modern watermarking methods. Beyond parameter-based and trigger-based approaches, recent studies such as NWaaS: A Non-Intrusive and Privacy-Preserving Watermarking-As-a-Service System With Adaptive Resource Scheduling (IEEE TDSC, 2026) have explored non-intrusive, side-channel-based ownership verification. The authors should better position TwinMark relative to these developments. Furthermore, experimental comparisons are largely limited to CosWM, without sufficient evaluation against other relevant distillation-resistant watermarking baselines.
  4. Although the paper develops theoretical certificates for watermark recovery, the evaluated bounds fail to certify any watermark bits for logit-distilled students. Therefore, the theoretical guarantees only partially support the claimed watermark inheritance across different distillation objectives.
Questions for authors

See weaknesses.

Reviewer_cNmzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TwinMark embeds a shared secret payload into two model outputs: feature second moments and class-conditioned logits. Each watermark is designed to transfer through the corresponding distillation objective. The paper provides recovery conditions and evaluates detection across architectures and tasks.

Critical strengths
  1. The watermark design aligns well with what each distillation objective preserves.
  2. Second-moment certificates successfully certify feature-watermark detection in the primary experiments.
Critical weaknesses
  1. The feature watermark can be evaded by orthogonal rotations that preserve embedding geometry, limiting practical protection.

  2. Comparisons are insufficient, especially against watermarking methods designed for encoder stealing.

  3. False-positive calibration needs clearer documentation. The supplied PDF omits the referenced appendices, preventing verification of important details.

  4. The evaluation relies heavily on small-scale, outdated benchmarks, particularly CIFAR-100. Evaluation on larger-scale datasets with competitive teacher and student utility would strengthen the practical applicability.

Questions for authors

How about the stealthiness of TwinMark?

Reviewer_g1PGOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TwinMark, which embeds the same watermark payload into feature second moments and class-conditioned logits to support inheritance under feature-matching and probability-matching knowledge distillation. It combines linear decoding, statistical detection, and conditional bit-recovery certificates. Experiments demonstrate inheritance through the matched output surface. Feature certificates are empirically informative, while the evaluated logit certificates remain vacuous.

Critical strengths
  1. Well-motivated output-matched watermarking framework. The paper identifies an important limitation of existing distillation watermarks: a watermark is likely to transfer only through the output that the student is trained to imitate. TwinMark addresses this issue by placing separate signals in feature statistics and class-conditioned logits. The component studies support this design, showing that feature marks are inherited under feature matching, while logit marks are inherited under logit matching.

  2. A meaningful theoretical contribution for feature-channel certification. The second-moment analysis is one of the strongest parts of the paper. By bounding the discrepancy after averaging across carrier inputs, the method preserves cancellation effects that are lost by pointwise bounds. This is supported by the experiments: the second-moment certificates cover 50–60 of 64 bits for every primary feature-distilled student, while the evaluated pointwise bounds certify none. Thus, the theory is not merely formal; it provides useful guarantees for the tested students.

  3. Careful empirical characterization of the method’s scope. The paper includes component ablations, multiple random seeds, different student architectures, larger payloads, and several downstream tasks. It also explicitly reports important failure cases, including evasion through orthogonal feature rotations and public-carrier modifications, as well as the fact that the evaluated logit bounds certify no bits. This makes the conclusions more credible and helps distinguish demonstrated behavior from broader claims about watermark robustness.

  4. Useful analysis of class multiplexing. The class-multiplexed logit construction provides a principled way to increase the decoder’s rank beyond the number of classes. The experiments control, at least partially, for payload size and injection energy, and show that class multiplexing improves rank and bit recovery compared with a non-multiplexed design. This is a technically interesting contribution, although it should be interpreted as improved detectability rather than proof of increased reliable information capacity.

Critical weaknesses
  1. The central claims are validated only under a relatively narrow distillation setting. The main experiments focus on prescribed FM-KD and KL-KD protocols, largely fixed training recipes, and a limited number of seeds. Although the cross-architecture experiments are useful, stronger evidence would require independently initialized same-architecture students, varied training schedules, different data distributions, and more diverse distillation objectives. Therefore, the results establish inheritance under the evaluated protocols rather than general inheritance under model extraction.

  2. The practical guarantee is substantially stronger for features than for logits. The second-moment feature certificate is empirically effective, but the evaluated logit certificates certify no bits. Consequently, the two channels are not supported equally: logit detection is demonstrated empirically, whereas its claimed robustness and certification remain weak. The paper should more clearly separate the theoretical contribution of the logit analysis from its practical effectiveness.

  3. The feature watermark is vulnerable to natural representation transformations. Since the verifier depends on fixed embedding coordinates and public carriers, an orthogonal rotation or selective modification of carrier responses can evade detection without necessarily changing the model’s task behavior. This is a fundamental limitation of the proposed verification interface, not merely an omitted attack. The method should therefore be presented as a protocol-specific verifier rather than a generally robust ownership certificate.

  4. The comparison with prior work is not sufficiently comprehensive. The main comparison relies heavily on CosWM, and the experimental settings do not fully establish superiority over a broad range of distillation watermarking methods. A stronger evaluation would compare against multiple relevant baselines under matched query budgets, false-positive levels, watermark strength, and utility degradation.

  5. Claims about robustness and utility costs require more careful qualification. The removal experiments show robustness against several fixed attacks, but they do not demonstrate resistance to adaptive, utility-preserving removal attacks. Conversely, the reported embedding nearest-neighbor degradation suggests a nontrivial cost that is not captured by classification accuracy alone. A more complete utility–detectability–robustness trade-off analysis is needed.

  6. The evidence for statistical reliability under deployment conditions is limited. The paper provides a thoughtful conditional testing framework and accounts for multiple tests, but the clean-model evaluation is too small to establish a very low population-level false-positive rate. The distinction between average behavior over regenerated keys and reliability for one fixed deployment key should be discussed more explicitly.

Questions for authors
  1. Certification of the logit channel
    None of the evaluated logit bounds certify any bits. What are the main factors making these bounds ineffective? Do any of the evaluated settings yield non-vacuous certification results?

  2. Feature transformation attacks and the threat model
    Are orthogonal rotations and selective modifications of carrier responses that preserve task utility within the paper’s threat model? If so, how does the verification procedure handle these transformations? If not, what interface constraints does deployment require?

  3. Fairness of the baseline comparison
    Were CosWM and TwinMark tuned and evaluated under the same query budget, false-positive level, and comparable utility degradation? Why does CosWM fail to achieve perfect detection on the teacher model? Was its configuration sufficiently tuned?

  4. Utility costs and resistance to removal
    For applications that primarily use embeddings, how does the approximately 4.4-percentage-point drop in nearest-neighbor performance depend on watermark strength? Do the existing experiments include detection and certification results at lower utility costs, as well as removal attacks that preserve comparable task utility?

  5. False-positive guarantees for a fixed key
    Does the false-positive control hold on average over randomly generated keys, or does it also apply to a fixed deployment key evaluated against independent unwatermarked models? Given that false positives may concentrate on a small subset of keys, what theoretical or empirical guarantees are available for the false-positive rate of a fixed key?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Acknowledgments of Reviewers

We thank all three reviewers for their careful reading of our manuscript and for their constructive feedback. We appreciate the suggestions to evaluate TwinMark on larger benchmarks, across more diverse distillation settings, and against a broader set of baselines. We also acknowledge the concerns about robustness to feature transformations, false-positive calibration, and the limitations of the current logit-channel certificates. Given the limited time during the rebuttal period, we will carry out all requested additional experiments and report the results in a future version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.