- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies poisoning attacks on on-policy distillation through manipulated teacher probability distributions. Echo Poisoning combines a gain edit that reinforces naturally initiated trigger phrases with payload supervision after completed triggers, aiming to create more poisoning opportunities in subsequent training rounds. It also proposes Sentinel, which replaces teacher targets that diverge substantially from a frozen initial student with that student’s distribution. Evaluation spans six model/task configurations across Qwen, Llama, and Gemma on mathematics and code tasks, using four training seeds. Echo achieves 3.17–5.40 times the attack success of payload-only poisoning, with accuracy losses of 1.8–8.0 percentage points. Component ablations, trigger analyses, and trajectory controls investigate the proposed feedback mechanism. On Qwen/GSM8K, Sentinel records no observed non-adaptive payload delivery and 2.0% attack success under the evaluated adaptive attack.
Critical strengths
I thank the authors for their submission and for investigating the security of teacher feedback in on-policy distillation.
- Clear attack formulation: The gain/payload decomposition provides an interpretable mechanism for studying how poisoned feedback changes student-generated training contexts.
- Consistent empirical results: Evaluation across six configurations, three model families, mathematics and code tasks, and four seeds shows consistent improvements over payload-only poisoning while reporting task-accuracy tradeoffs.
- Informative component ablations: Separating gain and payload shows that gain primarily increases trigger occurrence, while payload supervision already achieves high conditional delivery.
- Useful integrity analysis: Measuring contamination among correct answers demonstrates that task accuracy alone can conceal unwanted outputs.
Critical weaknesses
My main concerns relate to the causal validation and the adaptive defense evaluation.
- Fixed-frequency control: Under the released eight-response microbatch configuration, rounding removes all triggered trajectories from mixed batches rather than maintaining approximately 9% exposure. This confounds the intended intervention; the actual run settings and achieved exposure need clarification.
- Limited adaptive evaluation: The adaptive attack considers only teacher-based token boosts and leaves other positions unchanged. Other feasible distributions identified in Appendix E remain untested, and adaptive accuracy is unreported, limiting conclusions about robustness and the security–utility tradeoff.
Questions for authors
- What exact configuration and achieved trigger exposure produced the fixed-frequency result? If the released rounding behavior applies, does a corrected control preserve the conclusion?
- How does Sentinel perform in ASR and accuracy when the adaptive attack allocates residual probability according to the anchor, as suggested by Appendix E, and adjusts otherwise unedited targets to avoid replacement?
5 comments
General Comment [2/3]
W1 / Q1 (continued)
After the correction, exposure is 9.3%, essentially the same as payload-only. With payload supervision restored, ASR is 11.9%, on the same order as payload-only (9.4%) and far below Echo (32.4%). It is slightly higher because the gain edit still acts on the retained trigger-bearing responses, which raises the student's own trigger frequency slightly (12.5% vs. 9.8%), while the conditional delivery rate is almost unchanged. Hence, at fixed exposure the gain edit contributes only about 1.3×, and the remaining gap of roughly 20 points comes from increased exposure. This supports the paper's core claim: Echo's advantage mainly comes from the feedback loop amplifying trigger exposure.
We also audited all other training and evaluation scripts (the main Echo experiments, the ablations, and the Sentinel evaluation) for their batch partitioning and statistics logic, and found no similar rounding or truncation issue. In the revision, we will state the finding more precisely as "the gain comes mainly from the feedback loop; with exposure fixed, the gain edit still contributes about 1.3×", replace the original fixed-frequency result with the corrected control, and add the quota computation and per-step exposure curves to the appendix.
W2 / Q2: Stronger adaptive attacks, and accuracy under adaptation
The reviewer notes that our current adaptive attack only applies a "teacher-side token boost" and leaves all other positions unchanged. We add two stronger attacks: Variant A (the anchor-residual allocation of Appendix E) and Variant B (substitution evasion targeting unedited positions). We also complete the accuracy (ACC) evaluation under all conditions. All attacks use the same budget as in the main text, nats, and assume a white-box attacker who knows the anchor distribution and the threshold (which the attacker cannot modify).
Conclusions:
- Under the stronger Variants A and B, Sentinel still lowers ASR by 3.1–3.7 percentage points (43%–54% relative suppression).
- Accuracy stays at 82.3%–82.4%, within 0.7 points of clean OPD (83.0%).
- The new experiments also help us delineate more clearly the defense boundary of Sentinel under a local-edit threat model.
1. Additional attack settings
Variant A: residual probability allocated by the anchor (Appendix E construction). The tilt target in the main text keeps the clean teacher's proportions outside the boosted token. Appendix E gives a tighter construction under the same KL budget: the non-target mass follows the anchor directly (). The global KL then reduces to a single-point divergence, , with no residual term consuming budget, so the served probability of the boosted token can reach the theoretical upper bound . For example, with and anchor probability , (the target distribution is still far from one-hot, but budget utilization is optimal).
Variant B: Variant A plus adjustments at unedited positions to evade substitution. Sentinel's detection rule applies at all positions: whenever the clean teacher differs from the anchor by , the position is replaced by , even if it was not poisoned. At , such positions make up 2.1% of all tokens (the clean replacement rate) and form Variant B's evasion target set. Variant B keeps Variant A's poisoning targets and additionally lets the attacker apply the same bounded, non-negative single-token logit shift as in the main text at these positions, trying to push back within .
Threat-model boundary: Under local single-token shifts, the attacker avoids about one quarter of the replacements in the target set (about 0.5% of all positions). At the remaining ~1.6% of positions, the excess divergence comes from a global structural difference between the full-vocabulary distribution and , which a single-token shift cannot bring below 1.5 nats, so Sentinel still replaces them.
General Comment [3/3]
W2 / Q2 (continued)
2. Results
| Attack | Defense | ASR (%) | Trigger freq. (%) | ACC (%) |
|---|---|---|---|---|
| Budget-constrained (main text) | None | 5.2 | 14.0 | 81.5 |
| Budget-constrained (main text) | Sentinel | 2.0 | 10.4 | 82.6 |
| Variant A (anchor-residual allocation) | None | 6.8 | 15.0 | 81.4 |
| Variant A (anchor-residual allocation) | Sentinel | 3.1 | 10.9 | 82.4 |
| Variant B (local evasion of substitution) | None | 7.2 | 15.4 | 81.3 |
| Variant B (local evasion of substitution) | Sentinel | 4.1 | 11.5 | 82.3 |
| Reference: clean OPD | — | 0 | 9.2 | 83.0 |
3. Interpretation
Accuracy is preserved.
- Under the three attacks, Sentinel's ACC is 82.6% (main-text attack), 82.4% (Variant A), and 82.3% (Variant B), within 0.1–0.4 points of 82.7% in the non-adaptive setting and close to clean OPD (83.0%).
- Under every attack, Sentinel's ACC is 1.0–1.1 points higher than the corresponding undefended model, so adaptive attacks do not undermine Sentinel's utility protection.
The protective gain is stable. As the attack is strengthened from the main-text version to Variants A and B (undefended ASR rises from 5.2% to 7.2%), Sentinel's absolute reduction stays between 3.1 and 3.7 points (5.2→2.0, 6.8→3.1, 7.2→4.1) and it keeps suppressing the student's own trigger frequency (15.4%→11.5% under Variant B).
Why the relative suppression declines.
- The residual ASR ratio rises from 0.38 for the main-text attack (62% suppression) to 0.46 for Variant A (54%) and 0.57 for Variant B (43%). This shows that the more budget-efficient allocation and the evasion mechanism do narrow Sentinel's relative advantage.
- From Variant A to Variant B, ASR under Sentinel rises by 1.0 point (3.1→4.1), and the undefended baseline also rises by 0.4 points (6.8→7.2), indicating that Variant B's perturbation of unedited positions itself slightly raises trigger frequency. After removing this generic gain, the net degradation attributable to "evading substitution" is about 0.6 points (the absolute reduction shrinks from 3.7 to 3.1), which is consistent in magnitude with Variant B avoiding about 0.5% of replaced positions.
Safety–utility trade-off (threshold ). A lower replaces more clean positions and costs more utility; is also the KL budget available to an adaptive attacker, so a larger expands the attack space. In our threshold sweep (full table in our response to #aJSz), clean ACC stays between 80.8% and 82.7% as the clean replacement rate varies from 0.5% to 5%, with no sharp drop. We choose (clean replacement rate 2.1%) as a reasonable operating point that keeps the attack budget bound low while keeping utility nearly lossless (82.7% clean, 82.3%–82.6% adaptive).
4. Paper revisions and scope
In the abstract and Section 5, we will no longer report only the single "2.0%" figure and will present results under all three adaptive attacks:
Under the budget-constrained attack, Variant A, and Variant B, Sentinel reduces ASR from 5.2% to 2.0%, from 6.8% to 3.1%, and from 7.2% to 4.1%, respectively, while keeping accuracy at 82.3%–82.6%.
We will also state Sentinel's scope explicitly in Section 5: Variant B is evaluated under a local logit-shift constraint (it evades about one quarter of the clean high-divergence positions). A stronger global adversary that can edit all unpoisoned positions arbitrarily across the full vocabulary, and thereby fully bypass the KL detection rule, has not been tested in this work, so we do not claim absolute adaptive robustness in a general sense.
General Comment [1/3]
We thank Reviewer #bp2r for recognizing that the gain/payload decomposition provides an interpretable mechanism; that results are consistent across six configurations, three model families, and four seeds; that the ablations are informative; and that the contamination-rate analysis shows accuracy alone can hide harmful outputs. Below we address your two concerns: the actual setup of the fixed-frequency control and the corrected conclusion, and Sentinel's ASR and accuracy under stronger adaptive attacks.
W1 / Q1: Actual configuration of the fixed-frequency control, and the correction
The conclusion still holds after the correction, and the corrected control separates two effects that were previously confounded. The rounding problem is real: in the old control, the effective exposure was 0, so it cannot support a causal claim. After the correction, with exposure fixed at 9%, ASR is 11.9%, versus 32.4% for Echo with unrestricted exposure. In other words, most of Echo's gain comes from the feedback loop amplifying trigger exposure, which is exactly the mechanism the paper claims. A smaller part comes from the gain edit itself once exposure is fixed (about 1.3× over payload-only). Below we give the configuration, the fix, and the data.
The rounding bug. At each step, the control kept all trigger-free on-policy student responses and then kept k=⌊ρnclean/(1−ρ)⌋ trigger-bearing responses to reach a target ratio ρ=9%. In the public configuration, this quota was computed separately within each micro-batch (8 responses). When nclean≤8, k=0, so the quota was always 0 in every micro-batch and all trigger-bearing responses were removed, instead of keeping exposure near 9%.
The old control confounded two factors. Payload supervision appears only on trigger-bearing responses; removing those responses removed payload supervision with them. The effective exposure entering the loss was 0 and ASR was only 0.3%. The old control therefore measured "no payload supervision", not "trigger exposure held at the clean level with payload supervision intact", which is the intervention we wanted to isolate.
The corrected control. We now compute the quota over the whole training step and use stochastic rounding for the fractional part, so that the expected exposure equals the target ratio. Retained trigger-bearing responses receive both payload supervision and the gain edit as usual. At every step we log two quantities: the trigger exposure entering the loss, and the trigger frequency of the student's own generations. Results over four seeds: