acceptodds
Under review as a conference paper at ICLR 2027

Persistent Negatives for Adversarial Black-Box On-Policy Distillation

Abstract

Black-box On-Policy Distillation (OPD) seeks to improve a student from its own generations when the teacher provides sampled responses but not token probabilities. Adversarial distillation offers one route: it learns a discriminator over prompt-matched teacher and student responses and uses its score as the policy reward. However, sampling discriminator negatives from the latest student at each step couples the learned reward to a negative distribution that changes after every policy update. We address this moving-target problem with persistent-negative adversarial distillation, a live-pool method that replaces a fraction of each discriminator batch with historical, prompt-matched teacher–student comparisons. Under matched discriminator compute, historical comparisons train the discriminator, while GRPO remains on-policy with fresh student responses. Our analysis identifies the Bayes-optimal reward as a teacher-to-negative log-density ratio and, under explicit assumptions, shows how persistent negatives anchor the discriminator and reduce reward-estimation MSE relative to fresh-negative training. Across two student families, three judges, and four judged-chat benchmarks, persistent-negative adversarial distillation consistently improves performance over current methods at matched discriminator compute. It also yields smoother fresh-policy discriminator trajectories, with fewer below-chance dips. These findings identify the discriminator's negative distribution as an important design axis in black-box on-policy distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy Distillation, Black-box Knowledge Distillation, Adversarial Reward Learning, Persistent Negative Sampling, GRPO, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DawQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies black-box on-policy distillation in the GAD framework of Ye et al. (2025). A Bradley–Terry discriminator is trained to score prompt-matched teacher responses above student responses, and its scores serve as sequence-level rewards for GRPO. The authors argue that drawing all discriminator negatives from the latest student makes the reward target move after every policy update. They propose persistent-negative adversarial distillation: a FIFO pool of 4,096 complete (x,yT,yS)(x, y_T, y_S) comparisons replaces a fraction ρ=0.5\rho=0.5 of each fixed-size discriminator batch, while GRPO still scores only fresh student rollouts. Discriminator compute is therefore matched to GAD.

The theory section contains three results:

  • Theorem 4.1: the Bayes-optimal BT score is log⁡τ(y∣x)/q(y∣x)\log \tau(y\mid x)/q(y\mid x) for the chosen negative distribution qq.
  • Proposition 4.2: the pooled loss and its curvature are lower-bounded by ρ\rho times their historical counterparts.
  • Proposition 4.3: the pooled reward has lower MSE than the fresh-negative reward if its variance reduction exceeds its squared bias.

Experiments distill GPT-5-Chat responses from the GAD corpus into Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, using three paired seeds. Evaluation uses four judged-chat sets (LMSYS, Dolly, Vicuna, Self-Instruct) under three LLM judges. The method improves over GAD in all 24 student–judge–dataset cells, with equal-set-weighted gains of +1.0 to +2.0 points. It also yields a smoother fresh-policy discriminator accuracy trajectory: standard deviation 0.087 → 0.068, and below-chance dips 6 → 1.

Critical strengths

1. A clean, minimal intervention with a well-controlled comparison. The method changes only the discriminator's negative distribution. Pooled rows replace rather than add fresh rows, so discriminator examples and optimizer steps are identical to GAD, and the GRPO update remains strictly on-policy (Section 3.3, Algorithm 1). Paired seeds share the warmup checkpoint, data order and initialization (Section 5.1). This design isolates the effect of the negative distribution well, and the separation between data used for the policy update and data used for reward learning is a useful conceptual point for adversarial distillation.

2. Directionally consistent gains across students, judges and datasets. The improvement over GAD is positive in all 24 cells for two model families, including three out-of-domain sets and a cross-family judge (Tables 1–2). The authors are careful about interpretation: they treat the 24 cells as correlated robustness evidence rather than independent tests, and they state that the discriminator diagnostics describe the coupled student–discriminator system rather than the discriminator alone (Sections 6.1–6.2, Appendix D). The method costs essentially nothing extra, so even a modest but reliable gain is practically relevant.

Critical weaknesses

1. Effect sizes are small, and the evidence for both the gains and the mechanism is thin.

  • Size of the gains: improvements over GAD are 0.7–3.3 points on a candidate/(candidate+reference) ratio. It is unclear whether Tables 1–2 report the mean over the three seeds or a single run. No confidence intervals or paired tests over prompts are reported, and Fig. 3 shows only three seed-level averages. Vicuna has only 80 prompts.
  • Judge confounds: two of the three judges are Qwen-family models judging a Qwen student, and LLM judges are known to favor longer responses. Adversarial rewards learned against GPT-5-Chat responses may well increase response length, yet no length statistics or length-controlled scores are reported. The gains may therefore partly reflect stylistic or length shifts rather than better responses.
  • Mechanism evidence: the discriminator smoothness result comes from a single "initial primary run" (Table 4), and the common-probe analysis uses one final discriminator pair (Appendix D.2).
  • Selective reporting: Appendix A mentions exploratory sweeps over recency- and loss-weighted pool sampling, but their results are not reported, so it is unclear how robust the default configuration is.

2. The theory is largely generic, and the method is not separated from simpler smoothing alternatives.

  • Generic theory: Theorem 4.1 is the standard density-ratio characterization of the optimal logistic/BT discriminator. Proposition 4.2 follows from nonnegativity of softplus and linearity of the mixture.
  • Proposition 4.3 assumes its conclusion: it assumes Vpool≤Vfresh−ΔtV_{\text{pool}} \le V_{\text{fresh}} - \Delta_t and Bpool2<ΔtB^2_{\text{pool}} < \Delta_t, and neither quantity is measured empirically.
  • Biased target by design: Theorem 4.1 itself shows that pooling changes the population target to log⁡τ/qtρ\log \tau/q^\rho_t, which is a biased estimate of the current-student reward the paper treats as ideal. The paper does not explain why this bias should help the student beyond the two-mode example.
  • Simpler alternatives not tested: a pool of 4,096 rows covers only the last 16 steps (Appendix A), so the pool mainly acts as short-horizon temporal smoothing of the discriminator. Natural compute-matched alternatives would be a lower discriminator learning rate, an EMA or Polyak-averaged discriminator, or averaging discriminator scores over recent checkpoints. None of these is compared.
  • Missing sweeps: there is no sweep over ρ\rho or pool capacity, so the claimed design axis is explored at a single point.
  • Prior use of history buffers: related ideas already exist in adversarial learning, e.g., discriminator history buffers in SimGAN (Shrivastava et al., 2017) and replay in DAC (Kostrikov et al., 2019), which the paper cites. This further limits the novelty to the black-box OPD instantiation.
Questions for authors
  1. Are the numbers in Tables 1–2 means over the three paired seeds or a single run? Please report per-seed values for each cell and a paired bootstrap or sign test over prompts for the main comparison against GAD.
  2. How do average response lengths of the base, SeqKD, GAD and persistent-negative students compare? Does the advantage over GAD remain under a length-controlled judging protocol, and does it hold when only the cross-family Gemma judge is considered?
  3. How does the method compare with compute-matched smoothing baselines that use no pool, such as a reduced discriminator learning rate or an EMA discriminator? If these close most of the gap, the contribution would be smoothing rather than persistent negatives.
  4. What happens for other values of ρ\rho and pool capacity, in particular a pool spanning many more than 16 steps? Please also report the results of the recency- and loss-weighted sampling sweeps mentioned in Appendix A.
  5. Can you provide empirical evidence for the assumptions of Proposition 4.3, for example by estimating reward variance across seeds and the pooled reward's deviation from a fresh-negative reference on a common probe?
Reviewer_P3EsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies black box on-policy distillation, a setting where we have access to generations from a teacher but not logits. They build upon an existing approach, generative adversarial distillation, that trains a discriminator to distinguish the teacher and student model before using the discriminator to train the student. They improve upon GAD by maintaining historical comparisons in the discriminator training data. Theoretically, they explain that such pooling has lower reward MSE. Empirically, they show their approach is superior to the baseline in chat scenarios.

Critical strengths
  1. The paper presents both theoretical and empirical accounts for their methodology, which makes its efficacy more convincing.
  2. The presentation of the method is very clear.
Critical weaknesses
  1. I feel the novelty over GAD is not that great. Overall it seems like a small tweak, and similar ones have been discussed many times in adversarial training literature.
  2. The evaluation setup seems to be quite old.
  3. The setting of black box optimization is not too realistic to me. Most proprietary models have terms of service that prohibit distillation, so I am not sure when this method would be applied.
Questions for authors
  1. Have you tried more recent chat evaluation sets such as Arena Hard V2?
  2. How sensitive are the gains to different amounts of historical data in the training pool?
Reviewer_J7X5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds on Generative Adversarial Distillation (GAD; Ye et al., 2025), which trains a Bradley–Terry discriminator to separate teacher from student responses and uses its score as a GRPO reward. In GAD, discriminator negatives always come from the current student. The authors argue this creates a “moving target” for the reward. They propose replacing a fraction ρ = 0.5 of each discriminator batch with prompt-matched teacher–student comparisons from a FIFO pool of 4,096 recent rows.

Authors identified the discriminator’s negative distribution as a design choice in black-box adversarial distillation and introduce persistent-negative training, while keeping GRPO updates fully on-policy and discriminator compute fixed.

Critical strengths
  • Clean, cheap intervention. It changes only the discriminator’s negative distribution and holds discriminator batch size and optimizer steps fixed. GRPO stays on-policy. This makes the comparison to GAD easy to follow.

  • Consistent results of improvement. Gains hold across two student families, three judges (one outside the Qwen family), and three out-of-domain datasets. The authors properly note that the 24 comparisons are correlated and not independent tests.

Critical weaknesses
  • Limited novelty. The contribution is mainly applying a known stabilizer to GAD. However if the proposed method can address the issue of GAD this should not be a main consideration.

  • Missing baselines and ablations. The pool covers 16 discriminator steps per Appendix A, out of roughly 1,500 adversarial steps, making it hard to accept the motivating story about “remembering failure modes.” It also makes simpler smoothing methods the natural competitors, such as Discriminator EMA or weight averaging, lower discriminator learning rate, or fewer discriminator steps per policy step. Naturally, the choice of fraction ρ of each discriminator batch should also be ablated

Questions for authors

Table 3 in App. A is hard to follow: “374 steps at 50%,” “748 steps at 50%, 492 at 33%.” Does this mean training was truncated? Table 4 also reports 748 updates.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.