- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Extreme Region Policy Distillation (ERPD), a two-stage approach for improving policy training using a fixed batch of rollouts. In the first stage, the method performs multiple optimization steps on the same batch using a teacher objective and selects a later checkpoint with strong performance. In the second stage, the policy is reset to the checkpoint that originally generated the rollouts. The student is then trained using a cached token-level score computed from the log-probability difference between the selected teacher and a reference policy. This score serves as the learning signal in a clipped PPO-style objective. The procedure can also be repeated across multiple rollout batches.
The paper further investigates a weak-teacher setting where model performance initially decreases and then recovers during training. It uses a recovered checkpoint and an intermediate unlearned checkpoint to construct a contrastive token-level learning signal. Experiments are conducted on Qwen3-4B, NBG4-3B, and Qwen3.5-9B and 27B, covering several mathematical reasoning and coding benchmarks. The paper evaluates iterative training, performance-KL trade-offs, different teacher objectives, score construction methods, and switching between online and offline training.
Critical strengths
-
The proposed method is relatively simple and practical to implement. ERPD separates training into two stages, which allows the student to benefit from a more extensively optimized teacher without directly continuing from the teacher checkpoint. Another practical advantage is that stage 2 can reuse cached teacher log probabilities, avoiding repeated teacher inference during student updates.
-
The paper includes a fairly comprehensive set of experiments and ablations. I appreciate that the authors report both positive and negative results. For example, the paper compares different teacher optimization objectives, discusses cases where GRPO and cross-model transfer are less effective, and investigates different token-score constructions and KL settings. The iterative improvements in Table 3 and the results on larger Qwen3.5 models are also encouraging.
Critical weaknesses
-
The main results rely heavily on single runs and extensive hyperparameter and checkpoint selection. The paper uses HMMT Feb 2025 to select teacher and student checkpoints, KL coefficients, update counts, score references, masking strategies, and iterative training strategies. However, the authors do not report results across multiple independent training seeds, uncertainty intervals, or the total number of configurations evaluated during selection. This makes it difficult to assess how robust the reported improvements are and whether some of the gains come from extensive tuning on the same benchmark. I am particularly concerned about the six-round Qwen3-4B experiment. The method switches to strategies 3 and 2 after the original strategy stops improving, while the corresponding no-distillation baselines are only evaluated for two rounds. I would like to see a more direct comparison where the baselines are trained for the same number of rounds with comparable rollout, token, and computational budgets. Similar controls at larger model scales would also help establish the effectiveness of the method.
-
The experiments do not fully support the proposed explanation for why ERPD works. My main concern is that the actual training objective does not seem to directly correspond to the KL projection motivated by Equation (3). If we ignore clipping, the fixed token-level score effectively encourages the student to optimize a teacher-reference log-ratio reward under the old policy's state distribution. This objective can push the student beyond the teacher rather than simply distilling the teacher's behavior, which is also discussed in Appendix B.1. I think more controlled experiments are needed to understand whether the proposed score is responsible for filtering spurious policy drift. For example, the paper does not compare against random, shuffled, constant, original-reward, or sparsity-matched token scores. Without these controls, it is difficult to determine whether the improvements come from the specific teacher-reference signal or from more general properties of the training procedure. The weak-teacher experiments also introduce additional factors that make the results harder to interpret. In the selected configuration, most raw token scores are masked to zero before full-batch whitening. After normalization, those positions receive the same positive standardized value. As a result, it is unclear how much of the improvement comes from the contrastive teacher signal itself versus the masking and normalization procedure. Overall, I find the empirical results promising, but I think the paper needs stronger controls to support its explanation of the mechanism.
Questions for authors
-
How does ERPD compare with continued training under the same rollout, token, and FLOP budgets? I would also like to see comparisons against resetting the policy and training with the original reward, as well as standard on-policy baselines. In particular, can these baselines be evaluated through all six rounds and on at least one larger model?
-
What happens if the teacher-reference token scores are replaced with shuffled, random, constant, binary, or sparsity-matched scores? I think these comparisons are important for verifying whether the proposed signal specifically helps remove spurious policy drift.
4 comments
- (W2) The experiments do not fully support the proposed explanation for why ERPD works. My main concern is that the actual training objective does not seem to directly correspond to the KL projection motivated by Equation (3). If we ignore clipping, the fixed token-level score effectively encourages the student to optimize a teacher-reference log-ratio reward under the old policy's state distribution. This objective can push the student beyond the teacher rather than simply distilling the teacher's behavior, which is also discussed in Appendix B.1. I think more controlled experiments are needed to understand whether the proposed score is responsible for filtering spurious policy drift.
Clipping is in fact crucial, as it provides an effect similar to that of the KL loss. In an earlier version of our method, we used standard policy gradients with a KL penalty. However, we found that the KL coefficient often required retuning during distillation. We therefore switched to PPO clipping, for which the community has established widely used default values. Our comparison shows that PPO clipping performs comparably to PG with a KL penalty while avoiding additional hyperparameter tuning.
The table below presents the MSE experiment on JustRL-1.5B. When clipping is removed, performance declines even with the same 20-step distillation budget.
| Setting | AIME24 | AIME25 | HMMT | Common-3 Avg |
|---|---|---|---|---|
| Ratio clipping | 71.25 | 63.54 | 40.42 | 58.40 |
| No ratio clipping | 70.42 | 61.25 | 38.02 | 56.56 |
| Clip − No-clip | +0.83 | +2.29 | +2.40 | +1.84 |
Regarding extrapolation, when examining the phenomenon in Figure 5, we find that even after 16 training steps, the model’s measured total KL remains lower than that of the teacher. This suggests that the student can outperform the teacher by amplifying useful components while filtering out irrelevant ones.
For iterative experiments, we recommend the conservative setting of 10 steps, as noted in the paper. The detailed results in Table 11 show that allowing the student not to surpass the teacher in each individual round, while consuming only a small KL budget, can lead to better progressive performance across rounds.
(Weakness 3)For example, the paper does not compare against random, shuffled, constant, original-reward, or sparsity-matched token scores. Without these controls, ...
(Quesiton 2)What happens if the teacher-reference token scores are replaced with shuffled, random, constant, binary, or sparsity-matched scores? ...
Negative-value shuffle and Negative-mean also perform well, comparably to the proposed Positive-value masking. They respectively shuffle or average the negative-valued positions within positive examples. This suggests that the exact scale of the scores at these positions is not critical, whereas their locations remain important. Random-mask, a sparsity-matched score control that randomizes the negative-score locations, performs substantially worse.
Several constant-score variants also perform worse, including Correct-response score zeroing, Correct-response constant , Binary sign-mean, Sequence constant, and Constant. Randomized and shuffled variants, including Random-global, Random, and Shuffle raw, similarly degrade performance. Original reward, which assigns the raw outcome reward at the token level, also performs poorly.
We reiterate that, based on our Qwen3.5 experiments, we still recommend the original setting as the default for model development. Positive-value masking is only an ablation motivated by our interest in understanding how MSE-based signals take effect. It is not intended to replace the original setting.
| Variant | HMMT | Description |
|---|---|---|
| Negative-value shuffle | 66.88 | Masks positive scores in correct responses, then shuffles only the remaining negative values within each response. |
| Positive-value masking (The proposed ablation in the paper) | 66.67 | Sets tokens with score > 0 to zero in correct responses while retaining negative scores. |
| Negative-mean | 66.25 | Replaces all remaining negative scores in each correct response with that response’s mean negative score. |
| Correct-response score zeroing | 65.97 | Sets every valid token score in correct responses to zero, regardless of sign. |
| Random-mask | 65.83 | Randomly masks the same number of nonzero positions as positive-value masking; retained tokens keep their original scores. |
| Correct-response constant | 65.07 | Sets every valid token score in correct responses to +0.1; incorrect responses retain their original scores. |
| Binary sign-mean | 64.38 | Maps positive and negative scores to their respective batch-wide sign means, retaining only score signs. |
| Sequence constant | 63.85 | Assigns every token in a response its post-whitening mean score, preserving the response-level score sum. |
| Random-global | 63.82 | Generates Gaussian scores matching only the global mean and variance of the raw batch. |
| Original reward | 63.75 | Broadcasts each response’s unnormalized outcome reward to all its valid tokens. |
| Random | 63.26 | Generates per-response Gaussian scores matching each response’s original mean and variance. |
| Nanbeige Base | 62.50 | |
| Shuffle raw | 61.67 | Randomly permutes all raw token scores within each response while preserving their distribution. |
| Constant | 60.21 | Sets every valid token score to +1; whitening is disabled to prevent the signal from collapsing to zero. |
I thank the authors for their detailed responses to my questions. Based on these clarifications, I will raise my score from 2 to 3.
Training cost analysis and comparisons with a controlled number of rounds.
Due to limited computational resources, we completed a third round for the “No distillation” setting; the full six-round experiment will be completed later. Performance declined further from the second to the third round.
We report the complete three-round iterative-distillation results in the table below. All rounds use Strategy 1 without switching strategies. We also compare against the entropy-preserving variant reported in Table 8. We hypothesize that the entropy reduction is another irrelevant component introduced during distillation, and that preventing this reduction may help preserve progressive performance.
We additionally report the computational cost of on-policy DAPO reported in Table 12. We use a staleness of 4 and apply DAPO filtering to both all-positive and all-negative rollout groups.
The table below also includes rollout and machine-time costs. We estimate the cost of each offline round as follows:
Rollout collection: 33 hours
Teacher tuning: 23 hours ×2, including hyperparameter-tuning time
Student distillation: 5 hours
Hardware: 8× H20 GPUs
Each offline round reuses a single rollout pool for both teacher candidates. The additional sweep over β∈{0.01,0.001} incurs extra training cost, which we account for as 23h×2.
From the table, despite consuming approximately the same amount of machine time and requiring more rollouts, on-policy DAPO does not yield a substantial improvement. Moreover, under an equal three-round comparison, the setting with distillation also performs better.
33 h rollout + 23 h × 2 teacher tuning2 × (33 h + 23 h × 2)3 × (33 h + 23 h × 2)33 h rollout + 23 h × 2 teacher tuning + 5 h distillation2 × (33 h + 23 h × 2 + 5 h)3 × (33 h + 23 h × 2 + 5 h)25 × 2 batches × 128 prompts × 16 responses30 × 2 batches × 128 prompts × 16 responses