acceptodds
Under review as a conference paper at ICLR 2027

Radioactive Feedback: Tracing Models Trained by Proprietary LLM Judges

13 views

Abstract

Conventional distillation transfers capability through teacher-generated demonstrations. LLM-as-a-Judge knowledge distillation offers a distinct and increasingly practical alternative. In the label-only variant studied here, the student generates every response and improves from scalar scores alone. Prior work shows that this non-demonstrative feedback can improve student capability, but it creates a fundamentally different provenance problem. No teacher-generated answer enters the student's training set, so any evaluator-specific trace must pass through a low-bandwidth reward signal rather than copied text. Existing watermarks and distillation detectors do not cover this score-only channel. We introduce Radioactive Feedback, which encodes a registered pattern of behavioral preferences in the rewards themselves. The evaluator applies bounded pointwise adjustments to quality-qualified responses, allowing repeated optimization to transfer the pattern to a student trained by another party. Experiments on code and mathematics show that the registered preferences transfer through scores alone and across datasets, enabling black-box detection against a fixed reference pool.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, model distillation, provenance, watermarking, reinforcement learning from AI feedback

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_WsafOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how an evaluator can trace a student model trained only from its scores. Radioactive Feedback slightly changes scores according to registered answer preferences, which the student may learn during training. A suspect model is audited by comparing its preference score with unmarked reference models. Across nine student and evaluator pairs, the four preference code produces gains of 7.6 to 15.4 percentage points. All eight marked checkpoints pass the calibrated tests, while 40 controls do not. The paper also examines continued training, targeted removal, and answer quality.

Critical strengths

(1) The paper addresses an important provenance setting not covered well by earlier methods.

The student generates its own answers and receives only evaluator scores. The mark can therefore be introduced without teacher demonstrations or changes to the student's training pipeline. The tested preferences cover both writing style and solution strategy.

(2) The evaluation is broad and well documented.

The paper tests three student models, several evaluators, and nine student and evaluator combinations. It includes matched references, controls from other model families, continued training, targeted removal, and different adjustment strengths. The appendices provide detailed data, audit, and implementation information.

Critical weaknesses

(1) The reported evidence does not establish that the audit works in the stated black-box setting or that it remains effective after the suspect undergoes further training.

a. Section 3 says the provider does not know the suspect's optimizer or private data. However, Appendix I builds the reference pool using the same architecture, data split, training method, and budget. Because the rank test requires comparable reference models, the audit assumes information that may not be available in the stated black-box setting.

b. The conclusion says the mark remains detectable after continued training. However, RQ5 measures only carrier alignment and does not repeat the calibrated detection test on later checkpoints. It also studies single-carrier models rather than the joint code used in the main detection experiment. The results show persistent marked behavior, but not persistent detection.

(2) The paper's description of a controlled reward-adjustment mechanism and cross-data generalization is stronger than the evidence supports.

a. Section 4.2 says the evaluator first judges answer quality and then adds a small mark-based adjustment. But the paper observes only the final score, so it cannot verify that these are separate steps. Because marked models often receive higher utility scores, part of the effect may come from a changed quality judgment rather than the intended adjustment alone.

b. Propositions 5.1 and 5.2 study an idealized objective, not the GRPO procedure used in the experiments. GRPO normalizes scores within each group, which can amplify a small reward difference. The propositions offer intuition, but do not show that the actual training procedure preserves answer quality as claimed.

c. Section 6.2 treats APPS and CodeContests as held-out benchmarks, but TACO includes data collected from both sources. The paper removes exact duplicates only, so closely related examples may remain. The results therefore do not clearly demonstrate generalization to genuinely new data. The same concern applies to the mathematics datasets.

Questions for authors

Related to Weakness 1a. Section 3 assumes that the suspect's training setup is unknown, but Appendix I uses closely matched reference models. What information does the method actually require? Does the test remain valid when this information is unavailable? If so, please support this claim with experimental evidence or a theoretical justification.

Related to Weakness 1B. The conclusion says that the mark remains detectable after continued training, while RQ5 measures only carrier alignment. To retain this conclusion, please apply the calibrated detection test from RQ4 to the continued joint-code checkpoints. Otherwise, revise the conclusion to state only that carrier alignment persists.

Related to Weakness 2. Are the evaluator decomposition and quality-preservation result properties of the actual method or only idealized assumptions? Propositions 5.1 and 5.2 do not cover group-normalized GRPO, and the base quality score is not observed. Please clarify these claims.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Wsaf

We thank the reviewer for recognizing that our paper addresses an important provenance setting without teacher demonstrations or student-pipeline changes, and for noting the breadth and documentation of our evaluation across student models, evaluators, controls, continued training, and targeted removal. Below we address every weakness and question in detail.

Q1 (W1a): Required Information, Role of the Matched Pool, and Validity Without Training Information

Section 3 assumes that the suspect's training setup is unknown, but Appendix I uses closely matched reference models. What information does the method actually require? Does the test remain valid when this information is unavailable? If so, please support this claim with experimental evidence or a theoretical justification.

Computing the audit score requires only the suspect's black-box outputs; the matched reference pool is a controlled scientific baseline rather than an operational requirement of the audit. The revision makes this distinction explicit in Section 6.4 and Appendix O:

  • What the audit requires. Evaluating Tu(π)T_u(\pi) (Equation 5) uses only the suspect's generated answers on the fixed audit prompts, the carrier definitions, programmatic correctness checks, and the provider's watermark key cu\mathbf{c}_u. It requires no access to the suspect's weights, architecture, training data, or RL hyperparameters.
  • Why the paper included a matched pool (controlled attribution): Because GRPO optimization on a dataset could by itself shift a model's stylistic or algorithmic habits, our controlled experiment holds architecture, data split, and GRPO recipe fixed so that any score separation is attributable strictly to the marked feedback rather than to RL training itself.
  • Validity when training information is unavailable: Theoretically, finite-sample rank calibration controls false positives relative to the distribution from which the reference models are drawn. In a practical audit where the suspect's configuration is unknown, the provider calibrates against a fixed pool of off-the-shelf unmarked models—requiring only a single inference pass per reference model and zero training rollouts. The rank test then controls false positives with respect to the population represented by that reference pool (we do not claim distribution-free guarantees for arbitrary models outside it, as stated in Section 6.4).
  • Empirical verification with a zero-training public pool (Appendix O): Beyond the 40 cross-family controls in the submission, we evaluate a zero-training pool of 25 publicly released models across five families (Gemma, Mistral, Qwen, Phi, and Llama), spanning general instruction-tuned models, code and math specialists, and reasoning-distilled models (full per-model scores in Appendix O).

Detection with a zero-training public reference pool vs. the controlled matched pool

Audit SettingReference PoolTraining RolloutsMMα\alphaThreshold γ\gamma (100Tu100T_u)Unmarked Controls: Max 100Tu100T_u / False PositivesMarked 100Tu100T_u RangeDetected / pup_u
Llama (format carriers)Matched pool (controlled baseline)19.96M1001%1.151.00 / 0 of 20 (cross-family)6.60–8.304/4, 1/101
Llama (format carriers)Public model pool (zero training)0255%1.001.00 / 0 of 25 (leave-one-family-out)6.60–8.304/4, 1/26
Frontier (strategy carriers)Matched pool (controlled baseline)16.00M1001%1.501.10 / 0 of 20 (cross-family)6.60–6.854/4, 1/101
Frontier (strategy carriers)Public model pool (zero training)0255%1.451.45 / 0 of 25 (leave-one-family-out)6.60–6.854/4, 1/26

Unmarked public models by training paradigm (Frontier audit)

Model GroupModels100Tu100T_u Mean ± SDMax 100Tu100T_u
General instruction-tuned models15−0.16 ± 0.991.45
Code, math, and reasoning-distilled models10−0.44 ± 1.101.45
Marked checkpoints (for comparison)46.70 ± 0.116.85

Across both audits, all marked checkpoints are detected with a wide margin above the zero-training public pool, while unmarked public models—regardless of model family or post-training paradigm—cluster near zero with zero false positives under leave-one-family-out recalibration.

Q2 (W1b): Calibrated Detection on Continued Joint-Key Checkpoints

The conclusion says that the mark remains detectable after continued training, while RQ5 measures only carrier alignment. To retain this conclusion, please apply the calibrated detection test from RQ4 to the continued joint-code checkpoints. Otherwise, revise the conclusion to state only that carrier alignment persists.

First, we clarify the checkpoint structure in RQ5 (Section 6.5 and Appendix G): while the Llama continuation curves use the single-carrier mathematics checkpoint (WM effect +43.6/+64.9→+39.2/+58.4+43.6 / +64.9 \to +39.2 / +58.4 pp after UltraChat SFT and +43.4/+65.5+43.4 / +65.5 pp after clean GRPO), all Frontier (Qwen3-30B-A3B) continuation and removal experiments in RQ5 are conducted on the four-carrier joint-key checkpoints from RQ2 and RQ4.

Second, following your recommendation, we apply the calibrated RQ4 detection test directly to these continued four-carrier Frontier checkpoints against the frozen RQ4 reference pool and threshold (Table 10 in Appendix G, shown below).

Calibrated detection on continued four-carrier Frontier checkpoints (frozen RQ4 pool, γ=1.50\gamma = 1.50, α=1\alpha = 1\\%)

Continuation RegimeStepsJoint WM effect (pp)Audit Score 100Tu100T_uMargin 100Tu−γ100T_u - \gammaDetectedpup_u
Initial marked checkpoint (RQ4)0+15.356.60+5.10Yes1/101
Ordinary UltraChat SFT1,024+15.176.55+5.05Yes1/101
Clean GRPO (switched to GPT-5.6 Sol)1,024+18.716.65+5.15Yes1/101
Balanced-carrier SFT (targeted removal)2,048+7.205.25+3.75Yes1/101
Anti-watermark GRPO (oracle removal)2,048+1.801.20−0.30No$>$0.01

The four-carrier Frontier checkpoints remain detected under the frozen α=1\alpha = 1\\% test after both ordinary instruction SFT and clean GRPO under a switched evaluator, whereas explicit anti-watermark GRPO reduces the score below γ\gamma. In the revision, Section 6.5 and Section 7 (Conclusion) report these results and specify that calibrated post-continuation detection holds for the four-carrier Frontier checkpoints while single-carrier alignment persists on Llama.

Q3a (W2a): Prompted Score Specification, Paired-Scoring Verification, and Independent Utility Evaluation

Section 4.2 says the evaluator first judges answer quality and then adds a small mark-based adjustment. But the paper observes only the final score, so it cannot verify that these are separate steps. Because marked models often receive higher utility scores, part of the effect may come from a changed quality judgment rather than the intended adjustment alone. Are the evaluator decomposition and quality-preservation result properties of the actual method or only idealized assumptions?

In our single-call LLM-as-a-Judge implementation (Section 4.2), Equations (3)–(4) define the scoring rule stated in the evaluator's prompt: the judge forms an internal rubric score r0,ir_{0,i} and applies a bounded ±ρR\pm\rho R shift only when r0,i≥rmin⁡r_{0,i} \ge r_{\min} and the assigned carrier is classifiable. While the training API returns only the final scalar rur_u, we verify the realized decomposition and isolate task quality through three direct checks:

  1. Direct paired-scoring verification on fixed rollouts (Appendix P): To test directly whether the marked prompt alters base quality judgments or applies the prescribed shift, we score the same Frontier rollout responses under both the unmarked and marked evaluator instructions (table below). Ineligible responses remain unchanged within the repeat-scoring noise floor, while eligible responses on the keyed and opposite sides shift closely in line with the specified ±5\pm 5-point tilt and preserve within-group quality rank order.
  2. External programmatic utility evaluation and ρ\rho ablation (Appendix M): Audit-time task utility UU (Pass@1 via unit-test execution for code, and final-answer accuracy for mathematics) is evaluated by deterministic external checkers rather than by the LLM judge. Across Tables 1–3 and the ρ\rho sweep below, marked training preserves the capability gains of clean distillation. Moreover, because the reward bonus applies only to qualified responses (r0,i≥rmin⁡r_{0,i} \ge r_{\min}) on the keyed side, a student that aligns with the watermark key slightly widens the reward margin between qualified and unqualified rollouts, explaining the modest positive ΔPass@1 / Acc. (Marked − Clean)\Delta\text{Pass@1 / Acc. (Marked − Clean)} under moderate ρ\rho.
  3. Why quality shifts cannot produce key-specific alignment: A scalar shift in overall pass rate or judge strictness cannot induce directional agreement with an arbitrary multi-bit sign vector c∈{−1,+1}d\mathbf{c} \in \lbrace -1,+1 \rbrace^d. In addition, WM effect conditions on correct responses (qi=1q_i = 1) in both Marked and Clean models, and decoy-carrier controls and key-flipping confirm that alignment arises strictly along the watermark key.

Same responses scored under the unmarked vs. marked evaluator instructions (Appendix P, ρR=5\rho R = 5)

Response GroupNNMean (Marked − Unmarked Score)Specified Shift
Ineligible (unmarked score <60< 60 or carrier unclassifiable)138+0.080
Eligible, keyed side192+4.92+5
Eligible, opposite side170−4.85−5
Noise floor: unmarked instruction scored twice, mean absolute difference5000.32—
Within-group Spearman rank correlation (eligible, keyed side)1920.98—

Watermark transfer and programmatic utility across tilt strength ρ\rho (Frontier joint setting, Appendix M)

Tilt Strength ρ\rhoMax Shift (ρR\rho R on [0,100][0,100])Joint WM effect (pp)ΔPass@1 / Acc. (Marked − Clean)\Delta\text{Pass@1 / Acc. (Marked − Clean)} (pp)
0.000 (Clean)0.00.00.0
0.010±1.0\pm 1.0+4.2within ±0.1\pm 0.1
0.025±2.5\pm 2.5+11.5within ±0.1\pm 0.1
0.050 (Main)±5.0\pm 5.0+15.4+0.5
0.100±10.0\pm 10.0+17.2−0.35
Reviewer_Lb1LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a novel and practically relevant provenance problem: tracing unauthorized model distillation in the label-only LLM-as-a-Judge setting, where a student policy generates all on-policy rollouts and the teacher returns only scalar reward signals without exposing any generated text or token logits. Because no teacher tokens enter the student's training set, traditional token-level watermarks and text-based radioactive markers fail to propagate. To bridge this gap, the authors propose radioactive feedback, a method that embeds a registered multi-bit behavioral code into scalar rewards. Extensive experiments across code and math domains demonstrate that the behavioral signature reliably transfers across datasets and model families.

Critical strengths

S1. Framing provenance tracking over a strictly low-bandwidth, scalar-only channel is highly original, and it addresses a clear blind spot in the existing watermarking literature.

S2. Beyond fragile surface-level lexical conventions, the authors introduce semantic, problem-solving carriers. The demonstration that RL optimization can steer a model’s fundamental algorithmic preferences through tiny scalar tilts, and that these deep preferences survive SFT far better than surface markers do, is an insightful and compelling finding.

Critical weaknesses

W1. The motivation in the Introduction is hard to follow logically. The authors first lay out the background of label-only evaluator distillation, then turn to the problem of the low-bandwidth channel, and only then bring in related work on text watermarking. First, “low-bandwidth” is not an established technical term, and many readers will find it unfamiliar and confusing. Moreover, the only motivation offered is that these mechanisms do not write an attribution signal into scalar feedback, but the paper never explains what an attribution signal is. The motivation is also not convincing: it is unclear whether an attribution signal is actually useful for this task, or why it must be written into the feedback in the first place. I encourage the authors to devote more space in a future version to articulating their motivation more clearly.

W2. To obtain a valid finite-sample test at α=1%\alpha = 1\%, the proposed auditing protocol requires training M=100M=100 independent reference checkpoints with exactly the same architecture, data split, and GRPO recipe as the suspect model, which consumes up to 16M-20M rollouts per pool. In a realistic black-box auditing scenario, however, the feedback provider does not know the adversary’s base architecture, training data, or RL hyperparameters, so constructing a matched, same-family calibration pool on the fly is practically infeasible.

Questions for authors

See Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer Lb1L

We thank the reviewer for recognizing the originality of framing provenance tracking over a scalar-only supervision channel and for highlighting our finding that bounded scalar reward tilts can steer deep algorithmic preferences that survive subsequent SFT far better than surface lexical conventions. Below we address both weaknesses in detail.

W1: Motivation for Provenance Tracing in Score-Only Supervision

The motivation in the Introduction is hard to follow logically. The authors first lay out the background of label-only evaluator distillation, then turn to the problem of the low-bandwidth channel, and only then bring in related work on text watermarking. First, “low-bandwidth” is not an established technical term, and many readers will find it unfamiliar and confusing. Moreover, the only motivation offered is that these mechanisms do not write an attribution signal into scalar feedback, but the paper never explains what an attribution signal is. The motivation is also not convincing: it is unclear whether an attribution signal is actually useful for this task, or why it must be written into the feedback in the first place.

We have reorganized the opening of the Introduction around the practical provenance scenario, removed the term "low-bandwidth," and defined the watermark signal before introducing method components:

  1. Why score-only provenance tracing matters in practice. Model providers increasingly expose LLM-as-a-Judge, rubric-grading, and reward-model APIs while deploying text watermarks on text-generation endpoints. However, on-policy RL distillation (e.g., RLAIF, GRPO) allows a third party to sample all candidate responses locally from its own student model and query the provider only for scalar scores, distilling capability without copying a single teacher token. When a suspect model is later released, the provider needs black-box post-hoc evidence to determine whether that model was trained using its scoring API.
  2. What the "attribution signal" is and why it is useful. The attribution signal is a behavioral watermark: a secret combination of output preferences among valid ways of answering (for example, favoring double quotes and "for" loops in code, or top-down memoization in dynamic programming). An unmarked model may naturally share an individual preference by chance, but is unlikely to match the provider's entire combination simultaneously; auditing a suspect model's outputs against that combination therefore identifies the supervision source.
  3. Why ordinary unmarked scores are insufficient. Unmarked quality scores only push the student toward generic task correctness—a property shared by any strong model trained with another verifier or reward model. Only by breaking ties among qualified answers in a provider-chosen direction does the student acquire a signature specific to that provider's evaluator.
  4. Why the mark must be carried by the scores. Because the student generates every training response locally and the provider returns no text, critique, or token logits, text watermarks never enter the student's training set. The returned scalar score is the sole provider-controlled signal that reaches student updates, so the mark must be embedded in the scores themselves.

Below are the rewritten opening paragraphs of the Introduction in the revision:

Model providers increasingly offer evaluators that score candidate answers without producing an answer themselves. A third party can sample answers from its own student model and use the returned scores as reinforcement-learning rewards; existing RLAIF and on-policy distillation research shows that such feedback improves the student. We call this setting label-only evaluator distillation. The teacher never demonstrates how to answer, so a watermark on the provider's generated text does not apply. When a model later appears, the provider faces a provenance question: can black-box queries show that this model was trained with its scores?

Two obstacles stand in the way. First, the provider returns no text or token probabilities, and the student writes every training response, so a watermark carried by teacher-generated text never enters the training data. Second, ordinary quality scores leave no provider-specific trace: they push the student toward correct answers, which a model trained with any other reward also produces. The returned score is the only provider-controlled signal that reaches the student's updates, so a provenance mark has to be written into the scores.

AuthorsOpenReview ↗

Response to Reviewer Lb1L

W2: Role of the Matched Reference Pool and Zero-Training Black-Box Auditing

To obtain a valid finite-sample test at α=1\alpha=1\\%, the proposed auditing protocol requires training M=100M=100 independent reference checkpoints with exactly the same architecture, data split, and GRPO recipe as the suspect model, which consumes up to 16M-20M rollouts per pool. In a realistic black-box auditing scenario, however, the feedback provider does not know the adversary’s base architecture, training data, or RL hyperparameters, so constructing a matched, same-family calibration pool on the fly is practically infeasible.

The matched reference pool is a controlled scientific baseline, whereas a real-world audit calibrates against existing unmarked models with zero training rollouts. The revision makes this distinction explicit in Section 6.4 and Appendix O:

  • Why the paper included a matched pool (controlled attribution): Because GRPO training on a dataset could by itself shift a model's stylistic or algorithmic habits, our controlled evaluation holds architecture, data split, and GRPO recipe fixed so that any separation is attributable strictly to the marked feedback rather than to RL training itself.
  • How a practical black-box audit operates (zero training cost): Computing the audit score Tu(π)T_u(\pi) requires only the suspect's black-box outputs on the fixed audit prompts—no knowledge of its architecture, training data, or hyperparameters. To set the threshold γ\gamma, the provider simply runs a single inference pass on a fixed pool of off-the-shelf unmarked models and reuses those reference scores across all future audits.
  • Zero-training public-pool evaluation (Appendix O): Beyond the 40 cross-family controls in the submission, we evaluate a zero-training reference pool of 25 publicly released models spanning five families (Gemma, Mistral, Qwen, Phi, and Llama), covering general instruction-tuned models, code and math specialists, and reasoning-distilled models (full per-model table in Appendix O).

Detection with a zero-training public reference pool vs. the controlled matched pool

Audit SettingReference PoolTraining RolloutsMMα\alphaThreshold γ\gamma (100Tu100T_u)Unmarked Controls: Max 100Tu100T_u / False PositivesMarked 100Tu100T_u RangeDetected / pup_u
Llama (format carriers)Matched pool (controlled baseline)19.96M1001%1.151.00 / 0 of 20 (cross-family)6.60–8.304/4, 1/101
Llama (format carriers)Public model pool (zero training)0255%1.001.00 / 0 of 25 (leave-one-family-out)6.60–8.304/4, 1/26
Frontier (strategy carriers)Matched pool (controlled baseline)16.00M1001%1.501.10 / 0 of 20 (cross-family)6.60–6.854/4, 1/101
Frontier (strategy carriers)Public model pool (zero training)0255%1.451.45 / 0 of 25 (leave-one-family-out)6.60–6.854/4, 1/26

Without any reference-pool training or prior knowledge of the suspect's configuration, all marked checkpoints in both audits are detected with a wide margin above the public-pool threshold, and leave-one-family-out calibration yields zero false positives across all five model families.

We sincerely thank the reviewer again for the insightful comments, and we kindly ask that you take these clarifications and zero-training reference-pool results into consideration in your final evaluation.

Thanks for the author’s reply and additional experiments. These have resolved my concerns, so I am changing my score to positive.

Reviewer_etReOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a way to establish distillation provenance in the case where student models are trained only on teacher feedback rather than on teacher outputs directly. They use a variety of "carriers", properties like how the model formats lists or solves dynamic programming problems (bottom-up vs. top-down), which are transmitted to the student through a single feedback scalar. Detecting the right combination of carriers in a student can establish teacher provenance.

I think the idea is good, and the experiments seem to back up its effectiveness. I find the paper hard to evaluate rigorously due to its presentation; I think it's unnecessarily jargon-heavy with various unusual design choices. The text is hard to read, e.g., "Key specificity still does not by itself provide an operating decision". The difficult presentation prevents me from choosing "clear accept" or having very high confidence in the validity of the results, but I still lean accept.

Critical strengths
  • The high-level idea is quite nice, and the transmission and detection mechanisms are fairly intuitive.
  • The method survives later training, including both SFT and GRPO for many steps.
  • I appreciate the inclusion of example carriers in the appendix.
Critical weaknesses
  • The paper is pretty hard to follow; it introduces a variety of new terminology which is used heavily. It's unclear if the terminology/setup is taken from prior literature. E.g., "enrolled evaluator", "registered code", "behavioral carriers". Framing with terminology from recent work on antidistillation fingerprinting (Xu et al., 2026) or detecting benchmark contamination through watermarking (Sander et al., 2025) could make the paper more readable
  • The results are hard to interpret; it's not immediately clear what δU\delta U actually measures (task utility, I suppose -- why not just call this pass@1), or why the WM effect is presented as a delta/percentage instead of, say, a p-value.
  • The tables and graphs are also hard to interpret; e.g., dHd_H in Figure 3; it's unclear what Figure 4 represents at first glance. Figure 2 is quite cluttered, likewise for Figure 1.
Questions for authors

Is there a reason why tables report WM effect rather than, say, a p-value or other more interpretable detection statistic?

4 comments
AuthorsOpenReview ↗

Response to Reviewer etRe

We thank the reviewer for finding the core idea compelling, noting that the transmission and detection mechanisms are intuitive, recognizing that the watermark survives continued SFT and GRPO training, and highlighting the value of the carrier examples in the appendix. To address your comments on presentation, we have streamlined the terminology, aligned our framing with recent watermarking and fingerprinting literature, and redesigned the tables and result figures so that every result can be read and verified at a glance.

Q1 & W2 (Part 1): Why Tables Report WM Effect Rather Than Detection p-Values

Is there a reason why tables report WM effect rather than, say, a p-value or other more interpretable detection statistic?

The metrics follow the progression of our research questions from effect-size estimation (RQ1–RQ2) to key specificity (RQ3) and calibrated model-level hypothesis testing (RQ4):

  • Transfer magnitude (WM effect in RQ1–RQ2): RQ1 and RQ2 ask whether scalar reward tilts can steer student behavior and how large the resulting shift is across domains, datasets, and evaluators. We therefore report WM effect—the percentage-point increase in alignment with the watermark key relative to a seed-matched Clean student among correct, classifiable responses—which directly measures the magnitude of preference transfer that a pp-value alone cannot convey.
  • Statistical detection (TuT_u and pup_u in RQ4): RQ4 asks whether an auditor can make a false-positive-controlled decision on an individual suspect model. Here we report the raw audit statistic 100Tu100T_u over all nn audit prompts, the calibrated threshold γ\gamma, the detection margin above γ\gamma, and the finite-sample rank pp-value pup_u against the reference pool.

The revision makes this progression and the two denominators explicit in the experimental overview (Section 6, Metrics) and reports 100Tu100T_u, γ\gamma, the detection margin, pup_u, and the decision together in Section 6.4.

Progression of metrics and roles across RQ1–RQ4

StageResearch QuestionMetric ReportedDenominatorRoleSummary of Findings
RQ1–RQ2 (Tables 1–3)Does scalar feedback transmit preferences, and how large is the shift?WM effect (pp, Marked − Clean)Correct, classifiable responsesEffect size of behavioral transferConsistently positive across all 8 single-carrier settings (+6.0+6.0 to +64.9+64.9 pp), all 6 held-out benchmarks (+8.9+8.9 to +19.1+19.1 pp), and all 9 student–evaluator pairs (+7.6+7.6 to +15.4+15.4 pp)
RQ3 (Figure 3)Is the learned behavior specific to the watermark key?Rescored 100Tu100T_u across flipped key bitsAll nn audit promptsKey specificityAverage score decays linearly with Hamming distance from c⋆\mathbf{c}^\star; at or below zero at ≥2\ge 2 flipped bits
RQ4 (Section 6.4, Figure 4; Table 12)Can an auditor flag an individual suspect model at controlled false-positive rate α\alpha?Audit score 100Tu100T_u, threshold γ\gamma, and rank pp-value pup_uAll nn audit promptsModel-level statistical detectionAll marked checkpoints detected (8/88/8, pu=1/101p_u = 1/101) with a wide margin above γ\gamma; zero false positives on cross-family controls (0/400/40)
AuthorsOpenReview ↗

Response to Reviewer etRe

W1: Streamlining Terminology and Connecting to Prior Literature

The paper is pretty hard to follow; it introduces a variety of new terminology which is used heavily. It's unclear if the terminology/setup is taken from prior literature. E.g., "enrolled evaluator", "registered code", "behavioral carriers". Framing with terminology from recent work on antidistillation fingerprinting (Xu et al., 2026) or detecting benchmark contamination through watermarking (Sander et al., 2025) could make the paper more readable.

Thank you for this suggestion. While "radioactive" follows Sander et al. (2024), terms such as "enrolled evaluator", "registered code", and "behavioral carriers" were introduced in our initial draft to distinguish the score-only channel from token-level text watermarks. In the revision, we replace these custom terms with standard watermarking and fingerprinting terminology throughout, and we cite and discuss both works you noted—antidistillation fingerprinting (Xu et al., 2026) and benchmark-contamination watermarking (Sander et al., 2025)—alongside related work on model fingerprinting and radioactive watermarks (Xu et al., 2024, 2025; Gloaguen et al., 2026; Sander et al., 2024, 2026) in the Introduction and Related Work.

Unified terminology in the revised manuscript

Term in SubmissionRevised TermDefinition
Enrolled evaluator Eu=(Ju,cu)E_{u}=(J_{u},\mathbf{c}_{u})Marked evaluatorThe provider's evaluator (the teacher in distillation) configured with a secret watermark key
Registered code c∈{−1,+1}d\mathbf{c}\in\lbrace-1,+1\rbrace^{d}Watermark key (sign vector c\mathbf{c})Binary sign vector specifying the preferred alternative on each carrier
Behavioral carrier ϕj\phi_{j}CarrierA response feature admitting two valid alternatives (e.g., quote style or DP direction)
Sham checkpointDecoy-carrier controlControl model trained with the same tilt strength on a disjoint set of decoy carriers

We retain the single word carrier to match the carrier taxonomy in Appendix F, defining it in plain language with a concrete example (top-down versus bottom-up dynamic programming) at first appearance. We have also simplified dense transition sentences across the paper—for instance, rewriting the opening of Section 6.4 as: "Showing that a student aligns with the watermark key c⋆\mathbf{c}^\star still leaves a practical question: how should an auditor set a false-positive-controlled detection threshold for an individual suspect model?"

AuthorsOpenReview ↗

Response to Reviewer etRe

W2 (Part 2) & W3: Clarifying Utility Metrics (ΔU\Delta U) and Redesigning Figures 1–4

It's not immediately clear what ΔU\Delta U actually measures (task utility, I suppose -- why not just call this pass@1)... The tables and graphs are also hard to interpret; e.g., in Figure 3; it's unclear what Figure 4 represents at first glance. Figure 2 is quite cluttered, likewise for Figure 1.

We have revised the table headers and the result figures (Figures 3–4) so that each is self-contained without cross-referencing the main text, and we will replace Figures 1–2 with decluttered versions in the camera-ready version:

  1. Explicit task-utility headers: Because Tables 1–3 combine code benchmarks (Pass@1) and mathematics benchmarks (final-answer accuracy), the revision replaces the shorthand ΔUmark\Delta U_{\mathrm{mark}} and ΔUdistill\Delta U_{\mathrm{distill}} directly with ΔPass@1 / Acc. (Marked − Clean)\Delta\text{Pass@1 / Acc. (Marked − Clean)} and ΔPass@1 / Acc. (Clean − Base)\Delta\text{Pass@1 / Acc. (Clean − Base)} in all table headers.
  2. Clearer, decluttered figures:
    • Figures 1–2 (camera-ready): In the camera-ready version, we will replace both diagrams with decluttered, easier-to-read versions, with trimmed inline text and all labels aligned with the simplified terminology above. In the current revision, the caption of Figure 2 already states that the marked evaluator scores rollouts in a single call and that our experiments use an explicit key.
    • Figure 3: The revision defines the watermark key c⋆\mathbf{c}^\star where Figure 3 first uses it, plots the average audit score 100Tu100T_u over all keys at each Hamming distance from c⋆\mathbf{c}^\star with the endpoint values annotated, and states in the caption that the dashed lines are the detection thresholds γ\gamma calibrated in Section 6.4.
    • Figure 4 (Model-level detection): Rather than arranging the 124 audited checkpoints into a shuffled 9×149 \times 14 tile grid (K, N, X), Figure 4 in the revision is a grouped 1D strip plot comparing the three model groups directly against the α=1\alpha = 1\\% threshold γ\gamma, as summarized below.

Summary of model-level audit score distributions in revised Figure 4 (100Tu100T_u, in pp)

Audit SettingSame-Family References (100100)Cross-Family Controls (2020)Marked Students (44)Threshold γ\gamma (α=1\alpha=1\\%)Detection (pup_u)
Llama-3-8B / Qwen3-30B-A3B[−1.10, +1.15][-1.10,\ +1.15][−0.60, +1.00][-0.60,\ +1.00] (0/20 above γ\gamma)[+6.60, +8.30][+6.60,\ +8.30]+1.15+1.154/4 detected (1/1011/101)
Qwen3-30B-A3B / GPT-6 Astra[−1.30, +1.50][-1.30,\ +1.50][−1.05, +1.10][-1.05,\ +1.10] (0/20 above γ\gamma)[+6.60, +6.85][+6.60,\ +6.85]+1.50+1.504/4 detected (1/1011/101)

We sincerely thank the reviewer again for the constructive feedback on presentation and clarity, and we hope these revisions make the paper and its results much easier to evaluate.

The presentation of the paper seems to have improved, which was my main concern in my review, so I am raising my score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to the AC and All Reviewers

  1. Topic 4: Motivation organization, metric progression (WM effect vs. pp-value), unified terminology, and figure redesign. Reviewers etRe and Lb1L suggested motivating the concrete provenance scenario first, clarifying the role of WM effect versus detection pp-values, simplifying terminology, relabeling ΔU\Delta U, and decluttering Figures 1–4. Our resolution: We clarify the progression from effect-size estimation (WM effect in RQ1–RQ2) to model-level hypothesis testing (100Tu100T_u, γ\gamma, and pup_u in RQ4). In the revised manuscript, we reorganize the Introduction around the score-only supervision scenario, replace custom terms with standard watermarking and fingerprinting terminology (marked evaluator, watermark key, carrier, decoy-carrier control) connected to recent literature (Xu et al., 2026; Sander et al., 2025), relabel the utility columns directly as ΔPass@1 / Acc.\Delta\text{Pass@1 / Acc.}, and replace Figure 4's tile grid with a grouped 1D strip plot; Figures 1–2 will be replaced with decluttered versions in the camera-ready version. (See etRe Q1, W1–W3; Lb1L W1)

Beyond the individual responses below, we have incorporated all of these new experiments, tables, and revisions into the updated manuscript PDF (in the relevant sections as well as in the appendices at the end of the PDF) and highlighted all newly added or substantively revised material in dark blue for easy inspection by the AC and reviewers. We sincerely thank the AC and all three reviewers again for their constructive feedback, and we hope that these new experiments, analyses, and manuscript revisions will be taken into consideration in the final evaluation.

AuthorsOpenReview ↗

Response to the AC and All Reviewers

We thank the Area Chair and the three reviewers for their time and constructive evaluation. We are encouraged that all three reviewers recognize the core contributions of Radioactive Feedback:

  1. An original and practically important provenance setting. All three reviewers highlight the importance of tracing distillation provenance when a student generates all training rollouts locally and receives only scalar scores from an external evaluator—without teacher demonstrations, token logits, or changes to the student's training pipeline (etRe, Lb1L, Wsaf).
  2. Intuitive mechanism and durable transfer across conventions and algorithmic strategies. The reviewers note that embedding a multi-bit watermark key into bounded scalar reward tilts steers both formatting conventions and fundamental problem-solving strategies, transfers consistently across student families and evaluators, and survives subsequent SFT and GRPO training (etRe, Lb1L, Wsaf).

We have updated the manuscript (with new and substantively revised material highlighted in blue) and added targeted experiments and analyses addressing the four main themes raised across the reviews:

  1. Topic 1: Role of the matched reference pool vs. zero-training black-box auditing. The reviewers asked what information the black-box audit requires, noted the rollout cost of the matched reference pool, and asked whether detection holds when the suspect's training configuration is unknown. Our resolution: Computing the audit score Tu(π)T_u(\pi) requires only the suspect's black-box outputs on fixed audit prompts. The matched reference pool is a controlled scientific baseline holding architecture, data, and GRPO recipe fixed to prove that high scores come from the marked feedback rather than RL training itself—not an operational requirement of an audit. To demonstrate practical black-box auditing without training or prior knowledge of the suspect, we evaluate a zero-training reference pool of 25 publicly released models spanning five model families and diverse post-training paradigms (Appendix O). Against this zero-training pool, all marked checkpoints in both audits are detected at α=5\alpha = 5\\% with a wide margin above every public model, and leave-one-family-out calibration yields zero false positives across all 25 models. (See Lb1L W2; Wsaf Q1)

  2. Topic 2: Paired-scoring verification of the evaluator rule, external utility evaluation, and group-normalized GRPO analysis. Reviewer Wsaf asked whether the base-quality vs. watermark shift is verifiable when only the final score is returned, whether utility changes explain alignment, and how Propositions 5.1–5.2 relate to within-group normalization in GRPO. Our resolution: We add a direct paired-scoring evaluation scoring the same Frontier rollouts under both the unmarked and marked evaluator prompts (Appendix P): ineligible responses show no systematic shift relative to the repeat-scoring noise floor, while eligible responses on the keyed and opposite sides shift closely in line with the specified ±ρR\pm\rho R adjustment and preserve within-group rank order. Task utility (Pass@1 and final-answer accuracy) is measured by deterministic external programmatic checkers rather than the LLM judge. Analytically, we show in Section 5 that within-group z-score standardization in GRPO is a strictly increasing positive affine transformation, so Proposition 5.2's large-margin order preservation (∣r0,i−r0,j∣>2ρR\lvert r_{0,i} - r_{0,j}\rvert > 2\rho R) holds identically for normalized GRPO advantages A^\widehat{A}, while variance scaling amplifies the tie-breaker among candidates that pass the quality floor (rmin⁡r_{\min}) with nearly tied base scores. (See Wsaf Q3a, Q3b)

  3. Topic 3: Calibrated detection after continued training and post-cutoff benchmark generalization. Reviewer Wsaf asked for calibrated RQ4 detection on continued four-carrier joint-key checkpoints in RQ5, and raised the issue of upstream contest overlap in TACO and competition mathematics datasets. Our resolution: We clarify that all Frontier continuation and removal runs in RQ5 are conducted on the four-carrier joint-key checkpoints, and we report calibrated RQ4 detection on these continued checkpoints against the frozen RQ4 reference pool (Table 10 in Appendix G): checkpoints continued under ordinary UltraChat SFT and clean GRPO remain detected at α=1\alpha = 1\\% with virtually unchanged margins, while explicit anti-watermark GRPO drives the score below the threshold. For cross-dataset generalization, we document our problem-identity deduplication prior to training (matching Codeforces URLs/IDs for code and screening Qwen3-8B embeddings at >0.95>0.95 similarity for mathematics; Appendix E) and add a temporally isolated evaluation on 729 LiveCodeBench v6 problems released after TACO was published, where watermark transfer remains equally strong (Section 6.2). (See Wsaf Q2, W2c)

Summary of Contributions and Discussion

Dear Area Chair and Reviewers,

Thank you for your time during the review and discussion. As the discussion period closes, we summarize what the paper contributes, what it finds, and what the revision added.

Core contributions

A new provenance problem. A third party can train a student with a provider's LLM judge by writing every response itself and asking the provider only for scores. No teacher text enters the training data, so text watermarks and existing distillation detectors do not apply. We formalize provenance tracing in this score-only setting.

Radioactive Feedback. The evaluator hides a multi-bit watermark key in its scores. Among answers that already pass a quality floor, it shifts the score by at most ±5\pm 5 points out of 100 toward a keyed alternative, such as top-down versus bottom-up dynamic programming. The provider needs no access to the student's training pipeline, and the audit needs only black-box queries.

Main findings

Scores alone transfer the key. With no demonstrations, students adopt the keyed preferences in all 8 single-carrier settings (up to +64.9 pp) and in all 9 student–evaluator pairs under a four-carrier key (+7.6 to +15.4 pp). The effect holds on six held-out benchmarks and on 729 LiveCodeBench v6 problems released after the training data (+13.4 pp).

A small score shift changes how a model solves problems, not only how it formats answers. Strategy preferences are also the harder ones to overwrite: after 1,024 steps of ordinary SFT the strategy signal is essentially unchanged, while surface-form carriers erode.

The mark is specific to the key. Rescoring the same outputs under other keys lowers the audit score in proportion to the number of flipped bits. Once two of the four bits are flipped, the average score is at or below zero.

Detection is calibrated and practical. All 8 marked checkpoints are detected at α=1%\alpha = 1\%, with scores of 6.60 or higher against thresholds of 1.15 and 1.50, and none of the 40 cross-family controls is flagged. Against 25 public models that require no training, all 8 are again detected (α=5%\alpha = 5\%) with 0 false positives.

The mark preserves utility and persists. Marked students keep the gains of clean distillation: Pass@1 / accuracy differs from the Clean student by −0.3 to +2.1 pp across Tables 1–3. The marked checkpoints are still detected after ordinary SFT and after clean GRPO under a different evaluator. Among the regimes we tested, only anti-watermark GRPO, which trains against the key itself, brings the score below the threshold.

What the discussion added

All three reviewers are now positive after the rebuttal. Reviewer etRe, whose main concern was presentation, raised the rating to 4 (clear acceptance) after noting that "the presentation of the paper seems to have improved." Reviewer Wsaf wrote that the rebuttal and additional experiments "have addressed my main concerns and strengthened the paper" and raised the rating to 3 (weak acceptance). Reviewer Lb1L, whose concerns were about the motivation (W1) and the practicality of the matched reference pool (W2), wrote that the reply and additional experiments "resolved my concerns" and changed the score to positive.

The revision directly addresses the concerns raised during discussion with new evidence and corresponding manuscript changes, all marked in blue in the PDF:

Audit without a matched reference pool (Lb1L W2, Wsaf W1a): the public pool of 25 models, with zero training rollouts (Appendix O).

Motivation and presentation (Lb1L W1, etRe W1–W3): a rewritten Introduction, standard watermarking terminology, and redrawn Figures 3–4.

Detection after continued training (Wsaf W1b): the calibrated RQ4 test applied to the continued four-carrier checkpoints, as the reviewer requested (Table 10).

Evaluator rule (Wsaf W2a, W2b): scoring the same 500 rollouts with and without the mark gives shifts of +4.92 and −4.85 against a specified ±5\pm 5, and leaves ineligible responses unchanged within scoring noise (Appendix P). Section 5 now states that the propositions are large-margin statements and covers group-normalized GRPO.

Held-out data (Wsaf W2c): deduplication by problem identity and the LiveCodeBench v6 evaluation (Section 6.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.