- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
They study whether a distilled student learns how the teacher interacts with users, not just the answer content. They plant two markers in the teacher’s responses (Llama-3.3-70B): a final follow-up question (interrogative) and a conditional advice sentence like “Worth noting: ...” (declarative). They also test paraphrasing, filtering, audit-time instructions, a green-list watermark, and run a 20-participant user study.
Critical strengths
- Clean controls (same triggered prompt IDs across conditions, matched baselines per family)
- Three families and three seeds, per-seed results reported
- Useful lesson on ambient rates, and honest scoping (they don’t claim attribution)
Critical weaknesses
- The main result is expected: training a base model on responses ending with a question makes it end with a question. The novelty is weak.
- Weak positioning against existing work. Antidistillation fingerprinting (Xu et al., 2026) and watermark radioactivity (Sander et al., 2024) target the same question: detecting whether a model was distilled from a teacher. The paper only compares to one green-list setup that barely works at the teacher (2.37% detection), so it is unclear what interaction markers add over these methods.
- The setting maximizes transfer: ~1B base models whose only instruction tuning is 3k responses from a single teacher. Real distillation mixes teachers, uses instruction-tuned students, and much more data.
- The declarative judge is fragile (e.g., changing only the rubric moves Gemma from 12.5% to 33.0%)
- The user study is weak (e.g., 1-point equivalence margin)
Questions for authors
- Do the markers survive when the marked teacher is only a fraction of a larger SFT mix, or when the student is already instruction-tuned?
- What do interaction markers offer over antidistillation fingerprinting or a corpus-level radioactivity test?
4 comments
Response to Reviewer FeXz (Part 2/3)
Thank you for pointing us to the radioactivity literature, which improved this part of the work. (Part 2/3)
W2. Positioning against antidistillation fingerprinting and watermark radioactivity
We agree. Sander et al. (2024) is directly relevant and was missing. We will cite it and discuss it together with Xu et al. (2026), which the paper cites only briefly. Both works plant a keyed token-level signal and detect it with a calibrated test. Our markers have no key and give no such guarantee. We do not present them as a replacement.
On the 2.37% figure. This is a per-response rate: the share of single teacher responses with z > 4 after duplicate adjustment. The responses are short and many are mathematics or code, so per-response power is low. Following your suggestion, we ran a corpus-level test in the style of Sander et al. on the stored text. Each distinct pair of consecutive tokens is counted once across all responses, and the green fraction is tested against 0.25. Under 1,000 wrong keys the statistic is calibrated (standard deviation 0.93 to 1.04).
| Text scored (561 responses) | δ = 2 | δ = 4 | δ = 2 after DIPPER | Unmarked |
|---|---|---|---|---|
| Teacher, z | +18.4 | +58.2 | +5.8 | -0.6 |
| Gemma students, z by seed | -0.75, -0.67, -0.29 | -0.14, -0.34, +0.10 | -0.38, -1.38, -0.83 | +0.09, -0.13, -1.12 |
The green-list setup therefore works at the teacher (z = +34.7 on the full 3,009 responses). In the students, no seed is significant, including at δ = 4. For every student in the table, the 95% upper bound on the excess green fraction is at most 0.003. The teacher's excess is 0.032 at δ = 2 and 0.099 at δ = 4. In the same pipeline, 80.87% of Gemma student responses carry the interrogative marker.
We read this narrowly. It is one scheme, one student family whose tokenizer differs from the teacher's, and a text-only test on 561 responses per student. Sander et al. report stronger detection with open-weight access, which we did not run. We do not claim that radioactivity fails in general, and Section 5.5 already states that the comparison does not establish superiority over green-list watermarking.
What interaction markers add. First, a system prompt alone induces them. No logit access, sampling change, or proxy student is needed. Second, they live in the relation between prompt and response, not in token identity, so reading them does not depend on the student's tokenizer. Third, the interrogative marker remains in students after the training text is paraphrased (47.87%, 54.88%, and 17.65%). Fourth, they show that a user-facing behavior transfers, which token-level work does not address. What they lack is a key, and with it a calibrated false-positive rate and attribution. We will add this comparison as a table. We did not run antidistillation fingerprinting, and the revision will say so.
W3. The setting maximizes transfer
We agree that small base students, one teacher, and 3,009 responses are a favorable setting. Section 6 states this scope, and we will move it into a dedicated limitations paragraph. Three results indicate how far the finding extends.
- Fractional marking. In the Soft and Style-Control conditions, 19.34% of the training responses are marked and the rest are unmarked. Students show 15.52% to 31.99% (interrogative) and 4.34% to 8.32% (declarative). Matched baselines are at or below 1.67%.
- An instruction-tuned student. In a pilot that preceded the main campaign (the pilot setup of Appendix A), we fine-tuned Llama-3.2-1B-Instruct with LoRA on a marked corpus of 4,996 responses. On 1,000 held-out prompts, 57.4% of the responses from three marked students end with a question. The figure is 1.1% for three baseline-trained students. This is a judge-free surface measure from a pilot with a different teacher and corpus, and we report it as supporting evidence only.
- Unseen prompt sources. The audit prompts share no ID and no prompt text with the training prompts. By source label, 34.1% of the training prompts are mathematics and 21.8% are code. The audit prompts come from Alpaca and OpenAssistant.
We have no results for mixtures of several teachers, for students above about 1B parameters, or for larger corpora. We state this directly and do not extrapolate.
We are grateful for your attention to these points, and Part 3 covers the detector, the user study, and your two questions.
Response to Reviewer FeXz (Part 3/3)
Thank you for the remaining concerns on the declarative judge and the user study, which we answer here together with Q1 and Q2. (Part 3/3)
W4. The declarative judge is fragile
The change from 12.5% to 33.0% compares two rubric texts on the same responses: the original text and a later, more permissive revision. These are two definitions, not two runs of one definition. The permissive text also raises the unmarked baseline, from 0.53% to 17.11%, so it shifts both arms. All principal results use the original text. Three checks show that the declarative result does not rest on that choice.
- Both rubric texts give the same direction. Marked students exceed their matched baselines in nine of nine family and seed cells under each text. Exact McNemar tests on paired prompts give p < 0.006 in all nine cells under the original text. Under the permissive text they give p < 0.05 in eight cells and p = 0.059 in the ninth (Qwen, seed 42).
- A check without any judge. An exact string match on the four phrasings named in the teacher instruction (Appendix B) fires in 14.02% (Gemma), 9.63% (OLMo), and 8.08% (Qwen) of responses from marked students. It fires in 0 of 5,049 responses from matched baselines and in 0 of 2,805 reference-model responses.
- The original text has the lower floor. Reference models read 0.36% to 1.60% under the original text and 9.45% to 21.39% under the permissive one.
We will report the declarative result under both texts together with the string match, and we will state a rubric selection rule in Section 4.
W5. The user study is weak
We agree that a one-point margin is wide. We recomputed the paired analysis from the ratings shown in Figure 20, which reproduce every statistic in Appendix H.
| Condition | Mean difference from baseline | 90% CI | TOST p at margin 0.75 |
|---|---|---|---|
| Interrogative 20% | -0.10 | [-0.47, +0.27] | 0.004 |
| Interrogative 50% | 0.00 | [-0.50, +0.50] | 0.009 |
| Declarative 20% | +0.15 | [-0.21, +0.51] | 0.005 |
| Declarative 50% | +0.20 | [-0.17, +0.57] | 0.009 |
Equivalence holds for all four conditions at a margin of 0.75 points. The smallest margin at which all four pass is 0.57. With 20 participants the study cannot resolve smaller differences, and it measures satisfaction, not detectability. We will report these intervals, use the tighter margin, and keep the claim narrow.
Q1. Do the markers survive in a fraction of a larger SFT mix, or with an instruction-tuned student?
For a fraction within one teacher's corpus, yes. At 19.34% density both markers transfer in all three families (W3). For an instruction-tuned student, the pilot in W3 gives 57.4% against 1.1%. A mix with other teachers, or a much larger corpus, has not been tested, and we do not claim it.
Q2. What do interaction markers offer over antidistillation fingerprinting or a radioactivity test?
They need only a system prompt, and they do not depend on token identity or on a shared tokenizer. The interrogative form partly survives paraphrasing of the training text. In our pipeline the marker is visible in single student responses, where the corpus-level green-list test finds no signal. They do not offer a key, a calibrated false-positive rate, or attribution, which the keyed methods are designed to provide. We see the two families as complementary, and we will present them that way with Sander et al. (2024) and Xu et al. (2026).
We sincerely thank you for the constructive criticism, which has made the paper more precise.
Reviewer FeXz, please respond to authors today
Reviewer FeXz,
Today is the last day of the discussion period. The authors have prepared a detailed response to your review. Please reply to them as soon as possible to ensure there is a chance for back and forth discussion before the discussion period ends.
AC
Response to Reviewer FeXz (Part 1/3)
Thank you for a concise and well-targeted review, and for acknowledging the clean controls and the honest scoping of our claims. (Part 1/3)
We begin with the problems that we accept. The paper omitted Sander et al. (2024), which is directly relevant. Its positioning against antidistillation fingerprinting and watermark radioactivity was too brief. Its setting uses small base students and a single teacher. Its presentation of the declarative detector made a comparison of two rubric texts look like instability. Its user study used a wide equivalence margin. We respond to W1 through W5 and then to Q1 and Q2. The new numbers below come from re-analysis of stored outputs and required no additional training.
W1. The main result is expected
We agree that imitation by itself is expected. The contribution lies in what the controlled design shows beyond it. Four results could not be read off from the fact of imitation.
We will rewrite the introduction so that these results lead, and the existence of transfer is presented as the starting point.
We appreciate your patience with a detailed reply, which continues in Part 2.