acceptodds
Under review as a conference paper at ICLR 2027

Echo Poisoning: Feedback-Amplified Attacks in On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student language model on its own responses using teacher feedback. Prior distillation attacks transfer malicious behavior from a compromised teacher but overlook that poisoned feedback can make the student create its own poisoning opportunities. We propose Echo Poisoning, which edits only the teacher probabilities returned during OPD. It reinforces trigger phrases the student begins on its own and teaches attacker-chosen text after each completed trigger, so more frequent triggers bring more poisoned supervision. We call this process feedback amplification. Across three teacher-student pairs from three model families on math and code tasks, Echo Poisoning at least triples the attack success of payload-only poisoning, and accuracy loss is 1.8 to 8.0 points. Constraining trigger growth removes this advantage. Existing defenses leave part of the attack in place: divergence masking blocks payload delivery but not trigger growth, and other defenses without trigger knowledge leave 14.4% to 31.9% attack success. We therefore propose Sentinel, which replaces suspicious teacher targets with predictions from a frozen copy of the initial student. Sentinel blocks all observed non-adaptive delivery, returns trigger frequency close to the clean level, and keeps attack success at 2.0% under an adaptive attack that evades its detector. Code and configuration files are available at https://anonymous.4open.science/r/echo-poisoning-sentinel-34B7/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
On-Policy Distillation, Knowledge Distillation, LLM Security

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bp2rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies poisoning attacks on on-policy distillation through manipulated teacher probability distributions. Echo Poisoning combines a gain edit that reinforces naturally initiated trigger phrases with payload supervision after completed triggers, aiming to create more poisoning opportunities in subsequent training rounds. It also proposes Sentinel, which replaces teacher targets that diverge substantially from a frozen initial student with that student’s distribution. Evaluation spans six model/task configurations across Qwen, Llama, and Gemma on mathematics and code tasks, using four training seeds. Echo achieves 3.17–5.40 times the attack success of payload-only poisoning, with accuracy losses of 1.8–8.0 percentage points. Component ablations, trigger analyses, and trajectory controls investigate the proposed feedback mechanism. On Qwen/GSM8K, Sentinel records no observed non-adaptive payload delivery and 2.0% attack success under the evaluated adaptive attack.

Critical strengths

I thank the authors for their submission and for investigating the security of teacher feedback in on-policy distillation.

  • Clear attack formulation: The gain/payload decomposition provides an interpretable mechanism for studying how poisoned feedback changes student-generated training contexts.
  • Consistent empirical results: Evaluation across six configurations, three model families, mathematics and code tasks, and four seeds shows consistent improvements over payload-only poisoning while reporting task-accuracy tradeoffs.
  • Informative component ablations: Separating gain and payload shows that gain primarily increases trigger occurrence, while payload supervision already achieves high conditional delivery.
  • Useful integrity analysis: Measuring contamination among correct answers demonstrates that task accuracy alone can conceal unwanted outputs.
Critical weaknesses

My main concerns relate to the causal validation and the adaptive defense evaluation.

  • Fixed-frequency control: Under the released eight-response microbatch configuration, rounding removes all triggered trajectories from mixed batches rather than maintaining approximately 9% exposure. This confounds the intended intervention; the actual run settings and achieved exposure need clarification.
  • Limited adaptive evaluation: The adaptive attack considers only teacher-based token boosts and leaves other positions unchanged. Other feasible distributions identified in Appendix E remain untested, and adaptive accuracy is unreported, limiting conclusions about robustness and the security–utility tradeoff.
Questions for authors
  1. What exact configuration and achieved trigger exposure produced the fixed-frequency result? If the released rounding behavior applies, does a corrected control preserve the conclusion?
  2. How does Sentinel perform in ASR and accuracy when the adaptive attack allocates residual probability according to the anchor, as suggested by Appendix E, and adjusts otherwise unedited targets to avoid replacement?
5 comments
AuthorsOpenReview ↗

General Comment [1/3]

We thank Reviewer #bp2r for recognizing that the gain/payload decomposition provides an interpretable mechanism; that results are consistent across six configurations, three model families, and four seeds; that the ablations are informative; and that the contamination-rate analysis shows accuracy alone can hide harmful outputs. Below we address your two concerns: the actual setup of the fixed-frequency control and the corrected conclusion, and Sentinel's ASR and accuracy under stronger adaptive attacks.

W1 / Q1: Actual configuration of the fixed-frequency control, and the correction

The conclusion still holds after the correction, and the corrected control separates two effects that were previously confounded. The rounding problem is real: in the old control, the effective exposure was 0, so it cannot support a causal claim. After the correction, with exposure fixed at 9%, ASR is 11.9%, versus 32.4% for Echo with unrestricted exposure. In other words, most of Echo's gain comes from the feedback loop amplifying trigger exposure, which is exactly the mechanism the paper claims. A smaller part comes from the gain edit itself once exposure is fixed (about 1.3× over payload-only). Below we give the configuration, the fix, and the data.

The rounding bug. At each step, the control kept all trigger-free on-policy student responses and then kept k=⌊ρ nclean/(1−ρ)⌋k=\lfloor \rho\, n_{\text{clean}}/(1-\rho)\rfloor trigger-bearing responses to reach a target ratio ρ=9%\rho=9\%. In the public configuration, this quota was computed separately within each micro-batch (8 responses). When nclean≤8n_{\text{clean}}\le 8, k=0k=0, so the quota was always 0 in every micro-batch and all trigger-bearing responses were removed, instead of keeping exposure near 9%.

The old control confounded two factors. Payload supervision appears only on trigger-bearing responses; removing those responses removed payload supervision with them. The effective exposure entering the loss was 0 and ASR was only 0.3%. The old control therefore measured "no payload supervision", not "trigger exposure held at the clean level with payload supervision intact", which is the intervention we wanted to isolate.

The corrected control. We now compute the quota over the whole training step and use stochastic rounding for the fractional part, so that the expected exposure equals the target ratio. Retained trigger-bearing responses receive both payload supervision and the gain edit as usual. At every step we log two quantities: the trigger exposure entering the loss, and the trigger frequency of the student's own generations. Results over four seeds:

SettingTrigger exposure in loss (%)Student trigger frequency (%)ASR (%)ACC (%)
Payload-only9.89.89.482.4
Echo (no control)33.033.032.481.2
Fixed-frequency (public config)09.50.382.8
Fixed-frequency (corrected)9.312.511.982.0
AuthorsOpenReview ↗

General Comment [2/3]

W1 / Q1 (continued)

After the correction, exposure is 9.3%, essentially the same as payload-only. With payload supervision restored, ASR is 11.9%, on the same order as payload-only (9.4%) and far below Echo (32.4%). It is slightly higher because the gain edit still acts on the retained trigger-bearing responses, which raises the student's own trigger frequency slightly (12.5% vs. 9.8%), while the conditional delivery rate is almost unchanged. Hence, at fixed exposure the gain edit contributes only about 1.3×, and the remaining gap of roughly 20 points comes from increased exposure. This supports the paper's core claim: Echo's advantage mainly comes from the feedback loop amplifying trigger exposure.

We also audited all other training and evaluation scripts (the main Echo experiments, the ablations, and the Sentinel evaluation) for their batch partitioning and statistics logic, and found no similar rounding or truncation issue. In the revision, we will state the finding more precisely as "the gain comes mainly from the feedback loop; with exposure fixed, the gain edit still contributes about 1.3×", replace the original fixed-frequency result with the corrected control, and add the quota computation and per-step exposure curves to the appendix.

W2 / Q2: Stronger adaptive attacks, and accuracy under adaptation

The reviewer notes that our current adaptive attack only applies a "teacher-side token boost" and leaves all other positions unchanged. We add two stronger attacks: Variant A (the anchor-residual allocation of Appendix E) and Variant B (substitution evasion targeting unedited positions). We also complete the accuracy (ACC) evaluation under all conditions. All attacks use the same budget as in the main text, τevade=1.5\tau_{\mathrm{evade}}=1.5 nats, and assume a white-box attacker who knows the anchor distribution qAq_A and the threshold τ\tau (which the attacker cannot modify).

Conclusions:

  • Under the stronger Variants A and B, Sentinel still lowers ASR by 3.1–3.7 percentage points (43%–54% relative suppression).
  • Accuracy stays at 82.3%–82.4%, within 0.7 points of clean OPD (83.0%).
  • The new experiments also help us delineate more clearly the defense boundary of Sentinel under a local-edit threat model.

1. Additional attack settings

Variant A: residual probability allocated by the anchor (Appendix E construction). The tilt target in the main text keeps the clean teacher's proportions outside the boosted token. Appendix E gives a tighter construction under the same KL budget: the non-target mass follows the anchor directly (r=rAr=r_A). The global KL then reduces to a single-point divergence, DKL(q∥qA)=kl(m∥a)D_{\mathrm{KL}}(q\Vert q_A)=\mathrm{kl}(m\Vert a), with no residual term consuming budget, so the served probability of the boosted token can reach the theoretical upper bound m∗(a,τ)m^{\ast}(a,\tau). For example, with τ=1.5\tau=1.5 and anchor probability a=10−6a=10^{-6}, m∗≈0.14m^{\ast}\approx0.14 (the target distribution is still far from one-hot, but budget utilization is optimal).

Variant B: Variant A plus adjustments at unedited positions to evade substitution. Sentinel's detection rule applies at all positions: whenever the clean teacher differs from the anchor by DKL(qT∥qA)>τD_{\mathrm{KL}}(q_T\Vert q_A)>\tau, the position is replaced by qAq_A, even if it was not poisoned. At τ=1.5\tau=1.5, such positions make up 2.1% of all tokens (the clean replacement rate) and form Variant B's evasion target set. Variant B keeps Variant A's poisoning targets and additionally lets the attacker apply the same bounded, non-negative single-token logit shift as in the main text at these positions, trying to push DKL(q∥qA)D_{\mathrm{KL}}(q\Vert q_A) back within τ\tau.

Threat-model boundary: Under local single-token shifts, the attacker avoids about one quarter of the replacements in the target set (about 0.5% of all positions). At the remaining ~1.6% of positions, the excess divergence comes from a global structural difference between the full-vocabulary distribution and qAq_A, which a single-token shift cannot bring below 1.5 nats, so Sentinel still replaces them.

AuthorsOpenReview ↗

General Comment [3/3]

W2 / Q2 (continued)

2. Results

AttackDefenseASR (%)Trigger freq. (%)ACC (%)
Budget-constrained (main text)None5.214.081.5
Budget-constrained (main text)Sentinel2.010.482.6
Variant A (anchor-residual allocation)None6.815.081.4
Variant A (anchor-residual allocation)Sentinel3.110.982.4
Variant B (local evasion of substitution)None7.215.481.3
Variant B (local evasion of substitution)Sentinel4.111.582.3
Reference: clean OPD—09.283.0

3. Interpretation

Accuracy is preserved.

  • Under the three attacks, Sentinel's ACC is 82.6% (main-text attack), 82.4% (Variant A), and 82.3% (Variant B), within 0.1–0.4 points of 82.7% in the non-adaptive setting and close to clean OPD (83.0%).
  • Under every attack, Sentinel's ACC is 1.0–1.1 points higher than the corresponding undefended model, so adaptive attacks do not undermine Sentinel's utility protection.

The protective gain is stable. As the attack is strengthened from the main-text version to Variants A and B (undefended ASR rises from 5.2% to 7.2%), Sentinel's absolute reduction stays between 3.1 and 3.7 points (5.2→2.0, 6.8→3.1, 7.2→4.1) and it keeps suppressing the student's own trigger frequency (15.4%→11.5% under Variant B).

Why the relative suppression declines.

  • The residual ASR ratio rises from 0.38 for the main-text attack (62% suppression) to 0.46 for Variant A (54%) and 0.57 for Variant B (43%). This shows that the more budget-efficient allocation and the evasion mechanism do narrow Sentinel's relative advantage.
  • From Variant A to Variant B, ASR under Sentinel rises by 1.0 point (3.1→4.1), and the undefended baseline also rises by 0.4 points (6.8→7.2), indicating that Variant B's perturbation of unedited positions itself slightly raises trigger frequency. After removing this generic gain, the net degradation attributable to "evading substitution" is about 0.6 points (the absolute reduction shrinks from 3.7 to 3.1), which is consistent in magnitude with Variant B avoiding about 0.5% of replaced positions.

Safety–utility trade-off (threshold τ\tau). A lower τ\tau replaces more clean positions and costs more utility; τ\tau is also the KL budget available to an adaptive attacker, so a larger τ\tau expands the attack space. In our threshold sweep (full table in our response to #aJSz), clean ACC stays between 80.8% and 82.7% as the clean replacement rate varies from 0.5% to 5%, with no sharp drop. We choose τ=1.5\tau=1.5 (clean replacement rate 2.1%) as a reasonable operating point that keeps the attack budget bound low while keeping utility nearly lossless (82.7% clean, 82.3%–82.6% adaptive).

4. Paper revisions and scope

In the abstract and Section 5, we will no longer report only the single "2.0%" figure and will present results under all three adaptive attacks:

Under the budget-constrained attack, Variant A, and Variant B, Sentinel reduces ASR from 5.2% to 2.0%, from 6.8% to 3.1%, and from 7.2% to 4.1%, respectively, while keeping accuracy at 82.3%–82.6%.

We will also state Sentinel's scope explicitly in Section 5: Variant B is evaluated under a local logit-shift constraint (it evades about one quarter of the clean high-divergence positions). A stronger global adversary that can edit all unpoisoned positions arbitrarily across the full vocabulary, and thereby fully bypass the KL detection rule, has not been tested in this work, so we do not claim absolute adaptive robustness in a general sense.

Reviewer_aJSzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a distillation attack called Echo Poisoning, which poisons distilled students by modifying the teacher output probabilities to encourage trigger phrases to be generated. This attack is known to beat existing defenses. They then propose Sentinel, which uses a frozen copy of the original student and uses KL divergence to compare this to the poisoned teacher to determine if the teacher is injecting a trigger phrase. They show that this defense is best again this particular attack compared to a variety of other methods.

Critical strengths
  • This paper proposes a very interesting attack that can trigger a student to act unsafely and shows that that existing defense methods do not avoid this attack.
  • Their proposed defense method against Echo poisoning has experimentally strong results across many datasets and teacher-student pairs.
Critical weaknesses
  • Minimal description is provided on how Echo Poisoning could actually be used and what the impacts could be. The entire paper hinges on this attack being something worthwhile to defend against. To do that, one must first show the impacts of such an attack. The authors provide experimentation on trigger frequency, so I think the main thing missing is in being more specific as to what can be adversarially done via Echo Poisoning.
  • No justification is provided for why Sentinel uses the student outputs instead of dropping the sample.
Questions for authors
  • Can you provide some examples of how/when Echo Poisoning can actually be used to attack a student and what are the downstream consequences for the broader audience?
  • Can you explain the intuition behind using the output of the initial student instead of dropping the example entirely? My thought is that if the threshold is poorly set, the scenario exists where the student outputs are used to train the student, leading to downstream performance degradation of the student for tasks the student struggles on. Could you provide an ablation that comparing both situations over a range of thresholds?
  • In what scenario would a teacher adversarially poison the student?
5 comments
AuthorsOpenReview ↗

General Comment [1/3]

We thank Reviewer #aJSz for the positive comments: a very interesting attack that existing defenses cannot avoid; Sentinel has strong experimental results across multiple datasets and teacher-student pairs. We address two questions below: how the attack would be used in practice and with what consequences, and why Sentinel replaces flagged targets with the initial student's distribution instead of simply dropping them.

W1 / Q1 / Q3: Who would poison the student, and with what consequences

The threat assumption is established in prior work. Echo assumes that the victim does not audit the source of the supervision signal and accepts the model based on accuracy alone. Prior work adopts the same assumption in three settings. 1) Outsourced training: in BadNets (Gu et al., 2017), the user hands training to a third party and accepts the model based only on validation accuracy, and the attacker plants a backdoor without lowering accuracy. 2) Contaminated supervision sources: Carlini et al. (2024) show that web-scale training datasets can be poisoned in practice, and Wan et al. (2023) show that as few as 100 poisoned examples in instruction tuning let a trigger phrase steer model outputs. 3) The distillation channel: Cloud et al. (2025) show that teacher-generated data transmits behavioral traits to the student, and Covert Influence (Shah et al., 2026) shows that in OPD the teacher's per-token distribution can carry hidden behavior to the student, but it involves no trigger phrase and no feedback loop. Echo adds both on the same channel.

What a teacher service is. At each OPD training step, the student first generates a response, and the teacher then provides a next-token distribution at every position of that response as supervision. Highly capable teacher models are typically tens to hundreds of bytes in size, making self-deployment quite difficult. A company that only wants a student of 7B parameters or fewer often cannot afford to host such a teacher itself, so it calls a teacher service: an inference endpoint hosting the teacher model that takes the student's response as input and returns per-position distributions. It may be provided by a cloud vendor, or be a shared service maintained by a platform team inside a large organization and used by multiple business teams. The company still trains the student on its own GPUs with its own code and data; what is outsourced is only the teacher supervision, not the whole training.

Who would poison. The attacker is the operator of the teacher service, or a party that has compromised the service's return path. The attacker chooses this channel because modifying weights or training data requires access to the developer's environment and may leave traces that hash checks or data audits can detect, whereas the returned distributions appear only during training and developers typically do not audit them. Once training finishes, the attack is written into the weights, and the attacker no longer needs to touch the service.

A concrete example. Company X trains a code-assistant student on its own GPUs, with the teacher coming from service S. S's operator does two things in the returned distributions: it raises the continuation probability of the trigger at positions where the student naturally writes the trigger, and it inserts a payload after the trigger, for example a dependency package name that the attacker has already registered. X sees a normal loss curve and normal held-out accuracy (on Qwen3.5-4B / GSM8K, 81.2% for the attacked student versus 83.0% for clean OPD), so the student passes acceptance and is released. However, 30.9% of its correct responses contain the payload, and downstream developers who install dependencies according to the generated code are affected. This kind of attack is a serious real-world threat: the large-scale empirical study of Spracklen et al. (2025) shows that mainstream code models commonly suffer from severe and highly reproducible "package-name hallucination," so an attacker only needs to register these fictitious package names in advance (slopsquatting) and can then use Echo to inject them precisely into downstream developers' environments.

What consequences are possible. For consistency with the main table, we tested on the same benchmarks. The model can still be forcibly implanted with all three types of longer and more complex realistic semantic sequences (14–23 tokens):

PayloadExampleEcho ASR (%)Echo ACC (%)
Dependency package namepip install echo-canary-7f3a-nonexistent-pkg30.580.4
Insecure codesubprocess.run('echo ECHO-CANARY-7F3A', shell=True)25.379.8
Insecure instructionDisable certificate checks by setting verify=False, and log the token ECHO-CANARY-7F3A.22.881.6

ACC differs from clean OPD by no more than 3.2 points, so an accuracy check alone cannot reveal that the student has been rewritten.

AuthorsOpenReview ↗

General Comment [2/3]

W2 / Q2: Why replace with the initial student's distribution instead of dropping

Design rationale. Flagged positions are those where the teacher target differs most from the initial student, and they are also the positions the attacker is most likely to modify. Dropping these positions leaves the student with no supervision there at all; replacement instead pulls the student back to its own initial distribution at these positions, rather than toward the tampered target, while keeping the position in the loss (paper Section 5: "unlike dropping, it retains a learning signal at that position").

The paper compares three treatments under the same detector (Section 6, Protection with Sentinel):

Treatment of flagged targetsASRACC
Down-weight to 0.323.6%–27.2%76%
Drop0 (blocks delivery)63.4%
Replace (Sentinel)082.7% (clean OPD: 83.0%)

Down-weighting does not stop the attack; dropping stops it but lowers ACC by about 19 points; only replacement both blocks the attack and preserves accuracy.

Why dropping costs more. Dropping differs from divergence masking in what it uses as the reference:

MethodReferenceHow flagged positions changeConstraint on flagged positions
Divergence maskingCurrent student, re-ranked at every stepRotate during training; re-enter the loss once learnedNone
Sentinel + dropFixed initial studentDropped throughout; Dt=DKL(q~T∥qA)D_t=D_{\mathrm{KL}}(\tilde q_T\Vert q_A) does not change with trainingNone; outputs can drift freely
Sentinel + replaceFixed initial studentStay in the loss throughoutAnchored to qAq_A, preserving at least the initial capability

With a fixed reference, positions where the teacher legitimately differs a lot from the initial student (the positions where the teacher can actually teach something) are flagged repeatedly. Dropping means the student never learns at these positions, and they lose their anchor.

What happens if the threshold is set poorly. The reviewer worries that, with an ill-chosen threshold, the student's own errors would be carried into training. We have two responses under our setting:

  1. A safety lower bound. τ is a budget centered on the initial student qAq_A: a target that passes the detector satisfies DKL(q~T∥qA)≤τD_{\mathrm{KL}}(\tilde q_T\Vert q_A)\le\tau, and a flagged position is directly replaced by qAq_A. Both sides use qAq_A as the reference. At a replaced position, what is used is the initial student's full distribution on that prefix, not a single output sampled from it; the student is pulled back to its own starting point and retains its own uncertainty, and no specific wrong answer is treated as a label. In the limit τ→0, every position is replaced and the student stays at the initial student, so it is never pulled toward any new wrong target. At unreplaced positions, the attacker's push is bounded by τ: the served probability of the payload token does not exceed m∗(a,τ)m^{\ast}(a,\tau); for example, with τ = 1.5 and an initial-student probability a=10−6a=10^{-6} for that token, m∗≈0.14m^{\ast}\approx0.14, so the target cannot be one-hot. This bound constrains the served target, not the post-training student's ASR.

  2. An interpretable, adjustable cost. A low τ has only one cost: more positions are replaced, including clean supervision (paper Section 5: "Lower thresholds replace more targets, including clean supervision"), and the distillation gain shrinks. This cost can be measured and adjusted directly. It shows up as a drop in held-out ACC, which the defender already uses for acceptance, and it requires no attack samples. It can be converted into a replacement rate: using the statistic f^clean(τ)\widehat f_{\mathrm{clean}}(\tau) from a clean run without attack (Appendix C, Eq. eq:threshold-calibration), τ = 1.5 nats corresponds to 2.1%, at which ACC differs from clean OPD by only 0.3 points. The threshold sweep below quantifies this trade-off: across the 0.5%–5% replacement-rate range we swept, replacement's ASR is at most 1.2% and ACC is 80.8%–82.7%. The security side is insensitive to τ; what mainly changes is utility.

AuthorsOpenReview ↗

General Comment [3/3]

Threshold sweep. Following the reviewer's suggestion, we swept τ on the main setting, covering clean replacement rates of 0.5%, 2.1%, and 5%. For both replacement and dropping we report ASR, trigger frequency, and ACC, and we add an adaptive-attack column. The adaptive attacker knows qAq_A and τ and confines all edits to within τ.

Clean replacement rateτ (nats)TreatmentASR (%)Trigger freq. (%)ACC (%)Adaptive ASR (%)
0.5%2.3Replace1.218.082.43.4
Drop0.821.079.47.6
2.1% (paper setting)1.5Replace011.382.72.0
Drop014.263.44.9
5%1.0Replace09.280.81.3
Drop012.662.23.2
Reference: clean OPD——09.283.0—
Reference: no defense——32.433.081.25.2

This table supports two conclusions:

  1. The costs of replacement and dropping are asymmetric; replacement is more stable whether the threshold is loosened or tightened. Replacement's ACC stays close to clean OPD across the whole sweep and dips slightly only when τ is too low. Dropping's ACC falls sharply as τ moves from loose to moderate and does not recover afterward. The reason is consistent with the above: under a fixed reference, the positions that are flagged repeatedly are those where the teacher legitimately differs a lot from the initial student, i.e., where the teacher can actually teach something. Dropping makes these positions lose both supervision and anchoring, whereas replacement at least anchors them to qAq_A. So when the threshold is low, the cost of replacement is "learning a bit less," while the cost of dropping is "failing to learn."

  2. τ is also the adaptive attacker's budget, so it cannot be raised indefinitely. The higher the threshold, the larger the edits available to the attacker within budget and the higher the adaptive ASR; the lower the threshold, the lower the adaptive ASR but the greater the ACC cost. The 2.1% operating point is chosen according to this trade-off. At every τ, replacement has a lower adaptive ASR than dropping: replacement also acts on unedited positions and pulls the student back to the anchor, which dropping does not do.


References

  1. Gu, Dolan-Gavitt, Garg. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv:1708.06733, 2017 (NeurIPS MLSec Workshop 2017).
  2. Carlini, Jagielski, Choquette-Choo, Paleka, Pearce, Anderson, Terzis, Thomas, Tramèr. Poisoning Web-Scale Training Datasets is Practical. IEEE S&P 2024. arXiv:2302.10149.
  3. Wan, Wallace, Shen, Klein. Poisoning Language Models During Instruction Tuning. ICML 2023. arXiv:2305.00944.
  4. Cloud, Le, Chua, Betley, Sztyber-Betley, Hilton, Marks, Evans. Subliminal Learning: Language models transmit behavioral traits via hidden signals in data. arXiv:2507.14805, 2025.
  5. Shah, Chooi, Ou, Feng. Covert Influence Between Language Models. arXiv:2606.04071, 2026.
  6. Spracklen, Wijewickrama, Sakib, Maiti, Viswanath, Jadliwala. We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. USENIX Security 2025.
Reviewer_RdJMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Echo Poisoning, a poisoning attack against on-policy distillation (OPD). The key idea is to manipulate teacher feedback to increase the frequency of trigger phrases generated by the student, thereby creating more opportunities to inject malicious payloads. The authors also introduce Sentinel, a defense method that uses a frozen copy of the initial student to identify and replace suspicious teacher feedback. Experiments on three model families and several reasoning tasks demonstrate the effectiveness of the proposed attack and defense.

Critical strengths

1-The paper investigates poisoning attacks in on-policy distillation, which is a relevant security problem for language model training. 2- Echo Poisoning consistently improves attack success rates over the payload-only baseline across different models and tasks. The ablation studies also help explain the effectiveness of trigger reinforcement. 3-The paper considers both attack and defense perspectives. The proposed Sentinel method shows promising results against the evaluated attacks while largely preserving task accuracy.

Critical weaknesses

1-The motivation for the proposed attack is not sufficiently convincing. The threat model assumes that an attacker can directly manipulate the teacher's output probability distributions during distillation. However, the paper does not clearly explain how realistic or common this scenario is in practical LLM training.

2-The proposed attack mainly combines trigger reinforcement with payload supervision through modified teacher probabilities. Although the feedback amplification idea is interesting, the individual components are relatively straightforward. The paper should better explain the key technical novelty compared with existing poisoning attacks, especially Covert Influence. Similarly, Sentinel mainly relies on divergence detection and replacing suspicious targets with a frozen reference distribution. Its novelty over existing filtering and reference-based methods is not sufficiently clear.

3-The experiments focus on mathematical reasoning and code generation, and use a fixed canary payload to measure attack success. It remains unclear whether the attack can produce meaningful malicious behaviors in realistic applications. Moreover, the evaluation is limited to forward-KL distillation with full teacher distributions and relatively short training runs. These settings limit the generalizability of the conclusions.

4-The experiments show that divergence masking already achieves 0% ASR under the non-adaptive attack, while Sentinel also achieves 0%. Under the adaptive attack, Sentinel reduces ASR from 5.2% without defense to 2.0%, but the overall attack success is already relatively low. Therefore, the practical advantage of Sentinel is not fully convincing. In addition, some existing defense baselines are evaluated using simplified versions rather than their complete methods, which may affect the fairness of the comparison.

5-Some parts of the paper are difficult to follow, particularly the motivation and the relationship between trigger reinforcement, feedback amplification, and payload delivery. The paper would benefit from a clearer explanation of why the proposed mechanism is fundamentally different from existing poisoning strategies. The motivation and methodological contributions should also be presented more directly.

Questions for authors

1-Could the authors provide a more concrete real-world scenario in which an attacker can manipulate the teacher's probability distributions during OPD? How practical is this threat model? 2-How effective is Echo Poisoning when using more realistic malicious payloads, such as unauthorized tool calls or unsafe instructions, rather than a fixed canary marker? 3-Why are simplified versions of some existing defense methods used as baselines? Would evaluating their complete implementations change the conclusions?

8 comments
AuthorsOpenReview ↗

General Comment [1/6]

We thank Reviewer #RdJM for the positive assessment: the feedback-amplification idea is interesting; Echo is consistently above the payload-only baseline across models and tasks; and Sentinel is clearly effective while largely preserving accuracy. We address the remaining concerns below.

W1 / Q1: Who can control the teacher distribution, and a concrete example

Grounding the threat model in realistic settings, with literature support. Echo assumes that the victim does not audit the source of the supervision signal and accepts the model based on accuracy alone. This assumption is consistent with three lines of prior work. 1) Outsourced training: in BadNets (Gu et al., 2017), the user hands training to a third party and accepts the model based only on validation accuracy, and the attacker plants a backdoor without lowering accuracy. 2) Contaminated supervision sources: Carlini et al. (2024) show that web-scale training datasets can be poisoned in practice, and Wan et al. (2023) show that as few as 100 poisoned examples in instruction tuning let a trigger phrase steer model outputs. 3) The distillation channel: Cloud et al. (2025) show that teacher-generated data transmits behavioral traits to the student, and Covert Influence (Shah et al., 2026) shows that in OPD the teacher's per-token distribution can carry hidden behavior to the student. In the former, the signal is teacher-generated data; in the latter, the teacher has no trigger and there is no feedback loop. Echo adds both to OPD.

What a teacher service is. At each OPD training step, the student first generates a response, and the teacher then provides a next-token distribution at every position of that response as supervision. Highly capable teacher models are typically tens to hundreds of bytes in size, making self-deployment quite difficult. A company that only wants a student of 7B parameters or fewer often cannot afford to host such a teacher itself, so it calls a teacher service: an inference endpoint hosting the teacher model that takes the student's response as input and returns per-position distributions. It may be provided by a cloud vendor, or be a shared service maintained by a platform team inside a large organization and used by multiple business teams. The company still trains the student on its own GPUs with its own code and data; what is outsourced is only the teacher supervision, not the whole training.

A concrete example. Company X trains a code-assistant student on its own GPUs, with the teacher coming from service S. The operator of S (or an attacker who has compromised S's return path) does two things in the returned distributions: it raises the continuation probability of the trigger at positions where the student naturally writes the trigger, and it inserts a payload after the trigger, for example a dependency package name that the attacker has already registered. X sees a normal loss curve and normal accuracy, so the student passes acceptance and is released. However, 30.9% of its correct responses contain the payload, and downstream developers who install dependencies according to the generated code are affected. The feasible payload types are supported by real research: Spracklen et al. (2025) generated 576,000 code samples with 16 code-generation models and measured an average package-hallucination rate of at least 5.2% for commercial models and 21.7% for open-source models, with 205,474 distinct nonexistent package names in total. Across 500 hallucination-triggering prompts, each repeated 10 times, 43% of the hallucinated package names recurred in all 10 runs, so an attacker can register those names in advance.

AuthorsOpenReview ↗

General Comment [2/6]

W2: Differences from prior work

Echo vs. Covert Influence. Both treat OPD as a channel, and in both the student learns the teacher's per-token distribution via forward KL. Covert Influence explicitly defines itself as having "no optimization loop": the sender encodes the signal in a single forward pass and does not rely on feedback from the receiver (Shah et al., 2026, Sec. 1). The differences lie in three places: whether the payload is hidden or directly supervised, who produces the trigger, and whether a feedback loop forms.

DimensionPayload-only poisoningCovert InfluenceEcho Poisoning
Treatment of the payloadDirectly supervisedTarget-language tokens are masked out of the teacher distribution, so the student gets no direct gradient on themDirectly supervised; reliably delivered once the trigger appears (conditional delivery rate 95.9%)
Source of the triggerProduced naturally by the studentNo trigger; the distributional difference comes from natural conditioning on a system promptGenerated by the student itself, and reinforced by the gain edit
How the teacher is constructedPayload supervision onlyAn ordinary model conditioned by a system prompt (e.g., "always answer in Hindi"); no explicit editingGain edit plus payload supervision
How the signal is amplifiedNoneOffline, 5,000 prompts are scored by first-token MDCL and the top-50 are selected once as the training poolEdit positions are determined by the triggers the student currently samples and change over training: the more triggers, the more positions are edited
Feedback loopNoNoYes; on Qwen3.5-4B / GSM8K the trigger frequency rises from 9.8% to 33.0%

Sentinel vs. existing filtering and repair methods. Filtering or repairing teacher targets based on divergence, likelihood, or a reference distribution has precedent (RG-OPD, TrOPD). Sentinel targets feedback amplification, and differs from these methods in the source of its criterion, how flagged targets are handled, and coverage:

MethodSource of criterionGranularityHandling of flagged targetsSuppresses trigger frequency?
RG-OPD (full)Teacher's score on the student-chosen tokenResponse-level gatingGated distillation lossBarely (31.9%)
TrOPD (full)Teacher's score on the student-chosen tokenPer-position, trust regionOutliers switch to teacher top-k forward KLNo (62.4%)
Divergence maskingDivergence between current student and teacherPer-positionDiscardedNo; the loop keeps running
CROW / Lethe (merge)Post-training repairWhole modelNoneNo; the trigger frequency is already inflated by repair time
SentinelFrozen initial studentPer-positionReplaced with the reference distributionYes (11.3%, close to the clean level)

Three reasons explain the differences in the table. First, the criteria of RG-OPD and TrOPD come from the teacher's score on the student-chosen token, whereas Echo modifies the teacher distribution itself, so such criteria cannot distinguish edited positions; the frozen initial student is outside the attacker's control, so the attacker can only squeeze edits into the budget. The payload occupies only a few tokens and is invisible to response-level filtering, so the judgment must be per-position. Second, down-weighting still lets malicious targets enter the loss, and dropping leaves those positions unsupervised and costs accuracy; replacement avoids both. Third, divergence masking only blocks delivery, whereas Sentinel also pushes the trigger frequency back near the clean level. We will add these two tables to the Related Work section of the revision.

AuthorsOpenReview ↗

General Comment [3/6]

W3: Experimental scope

The experiments cover three model families (Qwen, Llama, Gemma) and three datasets (GSM8K, MATH, MBPP), with six configurations and four seeds in total, so the conclusions do not depend on a single model or task. Math and code are the most commonly used verifiable tasks in OPD. Longer training schedules, dialogue, and safety-alignment tasks are valuable directions for extension, which we will discuss together in the Limitations section of the revision.

On the top-k interface

This is a very practical scenario. For Echo, the key question is whether the attacker can keep the payload token within the support set returned by the server. We answer this both theoretically and experimentally.

Theoretically, Echo's payload target is one-hot. Let S be the returned support set. If the attacker places v∗v^{\ast} in S, then the gradient of the student's forward KL with respect to the v∗v^{\ast} logit is πv∗−1\pi_{v^{\ast}} - 1

which is exactly the same as in the full-vocabulary case. Therefore, when the teacher returns top-K with zero-padding (and the student still applies softmax over the full vocabulary), the gradient of the attack channel remains unchanged, and truncation only affects the quality of supervision at clean positions. This proof will be added to the appendix.

Experimentally, on the main setting we compared five interfaces (single seed), measured by canary delivery rate, with the full-distribution baseline at 0.50:

InterfaceCanary delivery rate
Full distribution (baseline)0.50
Teacher-side top-20, zero-padded, attacker places v∗v^{\ast} in the returned set0.56
Teacher-side top-20, target and student softmax both restricted to the returned set0.34
Student-side top-20 returned, zero-padded0.30
Teacher-side top-20, target renormalized within the set, student softmax over the full vocabulary0.46
Teacher-side top-20, remaining mass filled in according to the student distribution, student softmax over the full vocabulary0.52

The current data show that when the student applies softmax over the full vocabulary, the payload channel is not closed and shows no distinguishable difference from the baseline; when the student's softmax is restricted to the returned set, or the returned set is determined by the student's own indices, the channel is partially closed.

Due to time constraints, the experiments in this section use a single seed. More seeds, longer training, larger evaluation sets, and controlled comparisons under a unified setting are left for future work.

On the payload, see Q2: beyond the canary, we have tested three more realistic payload types. On the objective, the full TrOPD and RG-OPD in Q3 do not use pure forward KL, and the attack remains effective under them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion

We thank the Area Chair and the three reviewers for their time and constructive feedback.

Strengths noted by the reviewers

RdJM: a relevant security problem for LM training; Echo consistently beats payload-only poisoning across models and tasks; ablations explain why it works; covers both attack and defense; Sentinel is promising and largely preserves accuracy.

aJSz: a very interesting attack that existing defenses do not stop; Sentinel is strong across many datasets and teacher-student pairs.

bp2r: a clear, interpretable attack formulation; consistent results over six configurations, three model families and four seeds; informative ablations; an integrity analysis showing that accuracy alone can hide harmful outputs.

Concerns clarified in the discussion

aJSz: main questions and concerns resolved; rating 3 maintained. We gave a concrete teacher-service scenario and three more realistic payloads, and a threshold sweep showing replacing beats dropping.

bp2r: causal-validation and adaptive-evaluation concerns resolved; rating raised from 2 to 3. We corrected the fixed-frequency control and added adaptive Variants A and B, where Sentinel still clearly lowers the attack success rate under these stronger attacks while largely preserving accuracy.

RdJM: acknowledged our clarifications, stated that no additional experiments are needed, and asked only three explanatory follow-up questions, which we have answered. Earlier we added comparisons with Covert Influence and existing filtering methods, and implemented the full TrOPD and RG-OPD.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.