- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work provides a unique way of tracing distillation from proprietary models in student models by making a watermark learnable. It creates 2 response families - an initial response, and an alternative conditioned on the first answer. Both sets of answers are used to train separate LoRA adapters. A private key maps a classifier-produced category to an adapter so that related questions receive answers from the same family. The hypothesis is that this category-dependent preference is reflected in the student model, and a private detector can verify this learnable signal. This work offers a proof of concept but has concerns around reliable provenance attribution.
Critical strengths
- The signal is designed to be learnable - a category-dependent response preference is likelier to survive in a student than an arbitrary token bias that may not be picked up.
- There is clear student-transfer evidence, even across model families.
Critical weaknesses
- Solution preference is not always distinguished from surface-level style changes - A.1 documents an example where the solutions differ only in variable name.
- The negative reference is the undistilled Base model - but a more comparable setting may be a student trained on unwatermarked data. This comparison is shown for a single student-teacher configuration.
Questions for authors
- Is this method robust to distillation variations such as rephrasing, or using a subset of data from a watermarked teacher, or other realistic data mix strategies?
- What fraction of answer pairs differ in solution strategy rather than wording or formatting?
- How do you distinguish inheritance of the watermarked preference from independent preferences an unrelated model may exhibit? Does the ~1% FPR hold across independently trained, matched non-keyed students?
- Why was a binary design selected as opposed to K response families?