- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies black-box on-policy distillation in the GAD framework of Ye et al. (2025). A Bradley–Terry discriminator is trained to score prompt-matched teacher responses above student responses, and its scores serve as sequence-level rewards for GRPO. The authors argue that drawing all discriminator negatives from the latest student makes the reward target move after every policy update. They propose persistent-negative adversarial distillation: a FIFO pool of 4,096 complete comparisons replaces a fraction of each fixed-size discriminator batch, while GRPO still scores only fresh student rollouts. Discriminator compute is therefore matched to GAD.
The theory section contains three results:
- Theorem 4.1: the Bayes-optimal BT score is for the chosen negative distribution .
- Proposition 4.2: the pooled loss and its curvature are lower-bounded by times their historical counterparts.
- Proposition 4.3: the pooled reward has lower MSE than the fresh-negative reward if its variance reduction exceeds its squared bias.
Experiments distill GPT-5-Chat responses from the GAD corpus into Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, using three paired seeds. Evaluation uses four judged-chat sets (LMSYS, Dolly, Vicuna, Self-Instruct) under three LLM judges. The method improves over GAD in all 24 student–judge–dataset cells, with equal-set-weighted gains of +1.0 to +2.0 points. It also yields a smoother fresh-policy discriminator accuracy trajectory: standard deviation 0.087 → 0.068, and below-chance dips 6 → 1.
Critical strengths
1. A clean, minimal intervention with a well-controlled comparison. The method changes only the discriminator's negative distribution. Pooled rows replace rather than add fresh rows, so discriminator examples and optimizer steps are identical to GAD, and the GRPO update remains strictly on-policy (Section 3.3, Algorithm 1). Paired seeds share the warmup checkpoint, data order and initialization (Section 5.1). This design isolates the effect of the negative distribution well, and the separation between data used for the policy update and data used for reward learning is a useful conceptual point for adversarial distillation.
2. Directionally consistent gains across students, judges and datasets. The improvement over GAD is positive in all 24 cells for two model families, including three out-of-domain sets and a cross-family judge (Tables 1–2). The authors are careful about interpretation: they treat the 24 cells as correlated robustness evidence rather than independent tests, and they state that the discriminator diagnostics describe the coupled student–discriminator system rather than the discriminator alone (Sections 6.1–6.2, Appendix D). The method costs essentially nothing extra, so even a modest but reliable gain is practically relevant.
Critical weaknesses
1. Effect sizes are small, and the evidence for both the gains and the mechanism is thin.
- Size of the gains: improvements over GAD are 0.7–3.3 points on a candidate/(candidate+reference) ratio. It is unclear whether Tables 1–2 report the mean over the three seeds or a single run. No confidence intervals or paired tests over prompts are reported, and Fig. 3 shows only three seed-level averages. Vicuna has only 80 prompts.
- Judge confounds: two of the three judges are Qwen-family models judging a Qwen student, and LLM judges are known to favor longer responses. Adversarial rewards learned against GPT-5-Chat responses may well increase response length, yet no length statistics or length-controlled scores are reported. The gains may therefore partly reflect stylistic or length shifts rather than better responses.
- Mechanism evidence: the discriminator smoothness result comes from a single "initial primary run" (Table 4), and the common-probe analysis uses one final discriminator pair (Appendix D.2).
- Selective reporting: Appendix A mentions exploratory sweeps over recency- and loss-weighted pool sampling, but their results are not reported, so it is unclear how robust the default configuration is.
2. The theory is largely generic, and the method is not separated from simpler smoothing alternatives.
- Generic theory: Theorem 4.1 is the standard density-ratio characterization of the optimal logistic/BT discriminator. Proposition 4.2 follows from nonnegativity of softplus and linearity of the mixture.
- Proposition 4.3 assumes its conclusion: it assumes and , and neither quantity is measured empirically.
- Biased target by design: Theorem 4.1 itself shows that pooling changes the population target to , which is a biased estimate of the current-student reward the paper treats as ideal. The paper does not explain why this bias should help the student beyond the two-mode example.
- Simpler alternatives not tested: a pool of 4,096 rows covers only the last 16 steps (Appendix A), so the pool mainly acts as short-horizon temporal smoothing of the discriminator. Natural compute-matched alternatives would be a lower discriminator learning rate, an EMA or Polyak-averaged discriminator, or averaging discriminator scores over recent checkpoints. None of these is compared.
- Missing sweeps: there is no sweep over or pool capacity, so the claimed design axis is explored at a single point.
- Prior use of history buffers: related ideas already exist in adversarial learning, e.g., discriminator history buffers in SimGAN (Shrivastava et al., 2017) and replay in DAC (Kostrikov et al., 2019), which the paper cites. This further limits the novelty to the black-box OPD instantiation.
Questions for authors
- Are the numbers in Tables 1–2 means over the three paired seeds or a single run? Please report per-seed values for each cell and a paired bootstrap or sign test over prompts for the main comparison against GAD.
- How do average response lengths of the base, SeqKD, GAD and persistent-negative students compare? Does the advantage over GAD remain under a length-controlled judging protocol, and does it hold when only the cross-family Gemma judge is considered?
- How does the method compare with compute-matched smoothing baselines that use no pool, such as a reduced discriminator learning rate or an EMA discriminator? If these close most of the gap, the contribution would be smoothing rather than persistent negatives.
- What happens for other values of and pool capacity, in particular a pool spanning many more than 16 steps? Please also report the results of the recency- and loss-weighted sampling sweeps mentioned in Appendix A.
- Can you provide empirical evidence for the assumptions of Proposition 4.3, for example by estimating reward variance across seeds and the pooled reward's deviation from a fresh-negative reference on a common probe?
Dear Reviewers,
As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed.
Best, AC