acceptodds
Under review as a conference paper at ICLR 2027

Extreme Region Policy Distillation

Abstract

Reinforcement learning for large language models faces a fundamental trade-off between sample efficiency and asymptotic performance: on-policy methods require repeatedly generating expensive trajectories, while off-policy reuse introduces distribution mismatch. We find that aggressive optimization on fixed data brings rapid initial gains. Continued updates then shift token probabilities and reduce entropy after performance plateaus. Stronger KL regularization reduces both divergence and attainable performance. This motivates Extreme Region Policy Distillation (ERPD), a two-stage framework that separates aggressive rollout reuse from the final policy update. Stage 1 performs weakly constrained optimization on a fixed rollout batch to construct a strong policy-difference signal. Stage 2 restarts from the policy that generated the batch and optimizes the fixed teacher–reference log-probability difference with a clipped update, retaining useful policy changes while limiting divergence. The resulting policy matches or exceeds its teacher with substantially smaller KL divergence. ERPD also accommodates strong and weak teachers: when the Stage 1 teacher remains below the base model, contrasts between intermediate and recovered checkpoints still provide effective supervision. Across mathematical reasoning benchmarks, iterative ERPD raises Qwen3-4B-Thinking-2507 accuracy on HMMT Nov 25 from 66.6% to 79.0%; at larger scale, it raises Qwen3.5-27B exact-match accuracy on APEX2025 from 2.08% to 14.6%, while improving the other reported transfer benchmarks.

Primary area:
reinforcement learning
Keywords:
reasoning, off-policy optimization, weak-to-strong distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RPbwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Extreme Region Policy Distillation (ERPD), a two-stage approach for improving policy training using a fixed batch of rollouts. In the first stage, the method performs multiple optimization steps on the same batch using a teacher objective and selects a later checkpoint with strong performance. In the second stage, the policy is reset to the checkpoint that originally generated the rollouts. The student is then trained using a cached token-level score computed from the log-probability difference between the selected teacher and a reference policy. This score serves as the learning signal in a clipped PPO-style objective. The procedure can also be repeated across multiple rollout batches.

The paper further investigates a weak-teacher setting where model performance initially decreases and then recovers during training. It uses a recovered checkpoint and an intermediate unlearned checkpoint to construct a contrastive token-level learning signal. Experiments are conducted on Qwen3-4B, NBG4-3B, and Qwen3.5-9B and 27B, covering several mathematical reasoning and coding benchmarks. The paper evaluates iterative training, performance-KL trade-offs, different teacher objectives, score construction methods, and switching between online and offline training.

Critical strengths
  1. The proposed method is relatively simple and practical to implement. ERPD separates training into two stages, which allows the student to benefit from a more extensively optimized teacher without directly continuing from the teacher checkpoint. Another practical advantage is that stage 2 can reuse cached teacher log probabilities, avoiding repeated teacher inference during student updates.

  2. The paper includes a fairly comprehensive set of experiments and ablations. I appreciate that the authors report both positive and negative results. For example, the paper compares different teacher optimization objectives, discusses cases where GRPO and cross-model transfer are less effective, and investigates different token-score constructions and KL settings. The iterative improvements in Table 3 and the results on larger Qwen3.5 models are also encouraging.

Critical weaknesses
  1. The main results rely heavily on single runs and extensive hyperparameter and checkpoint selection. The paper uses HMMT Feb 2025 to select teacher and student checkpoints, KL coefficients, update counts, score references, masking strategies, and iterative training strategies. However, the authors do not report results across multiple independent training seeds, uncertainty intervals, or the total number of configurations evaluated during selection. This makes it difficult to assess how robust the reported improvements are and whether some of the gains come from extensive tuning on the same benchmark. I am particularly concerned about the six-round Qwen3-4B experiment. The method switches to strategies 3 and 2 after the original strategy stops improving, while the corresponding no-distillation baselines are only evaluated for two rounds. I would like to see a more direct comparison where the baselines are trained for the same number of rounds with comparable rollout, token, and computational budgets. Similar controls at larger model scales would also help establish the effectiveness of the method.

  2. The experiments do not fully support the proposed explanation for why ERPD works. My main concern is that the actual training objective does not seem to directly correspond to the KL projection motivated by Equation (3). If we ignore clipping, the fixed token-level score effectively encourages the student to optimize a teacher-reference log-ratio reward under the old policy's state distribution. This objective can push the student beyond the teacher rather than simply distilling the teacher's behavior, which is also discussed in Appendix B.1. I think more controlled experiments are needed to understand whether the proposed score is responsible for filtering spurious policy drift. For example, the paper does not compare against random, shuffled, constant, original-reward, or sparsity-matched token scores. Without these controls, it is difficult to determine whether the improvements come from the specific teacher-reference signal or from more general properties of the training procedure. The weak-teacher experiments also introduce additional factors that make the results harder to interpret. In the selected configuration, most raw token scores are masked to zero before full-batch whitening. After normalization, those positions receive the same positive standardized value. As a result, it is unclear how much of the improvement comes from the contrastive teacher signal itself versus the masking and normalization procedure. Overall, I find the empirical results promising, but I think the paper needs stronger controls to support its explanation of the mechanism.

Questions for authors
  1. How does ERPD compare with continued training under the same rollout, token, and FLOP budgets? I would also like to see comparisons against resetting the policy and training with the original reward, as well as standard on-policy baselines. In particular, can these baselines be evaluated through all six rounds and on at least one larger model?

  2. What happens if the teacher-reference token scores are replaced with shuffled, random, constant, binary, or sparsity-matched scores? I think these comparisons are important for verifying whether the proposed signal specifically helps remove spurious policy drift.

4 comments

Training cost analysis and comparisons with a controlled number of rounds.

  1. (Weakness 1)The main results rely heavily on single runs and extensive hyperparameter and checkpoint selection. The paper uses HMMT Feb 2025 to select teacher and student checkpoints, KL coefficients, update counts, score references, masking strategies, and iterative training strategies. ...
  2. (Question 1)How does ERPD compare with continued training under the same rollout, token, and FLOP budgets? I would also like to see comparisons against resetting the policy and training with the original reward, as well as standard on-policy baselines. In particular, can these baselines be evaluated through all six rounds and on at least one larger model?

Due to limited computational resources, we completed a third round for the “No distillation” setting; the full six-round experiment will be completed later. Performance declined further from the second to the third round.

We report the complete three-round iterative-distillation results in the table below. All rounds use Strategy 1 without switching strategies. We also compare against the entropy-preserving variant reported in Table 8. We hypothesize that the entropy reduction is another irrelevant component introduced during distillation, and that preventing this reduction may help preserve progressive performance.

We additionally report the computational cost of on-policy DAPO reported in Table 12. We use a staleness of 4 and apply DAPO filtering to both all-positive and all-negative rollout groups.

The table below also includes rollout and machine-time costs. We estimate the cost of each offline round as follows:

  • Rollout collection: 33 hours

  • Teacher tuning: 2323 hours ×2\times2, including hyperparameter-tuning time

  • Student distillation: 5 hours

  • Hardware: 8×8\times H20 GPUs

Each offline round reuses a single rollout pool for both teacher candidates. The additional sweep over β∈{0.01,0.001}\beta \in \{0.01, 0.001\} incurs extra training cost, which we account for as 23 h×223\,\text{h}\times 2.

From the table, despite consuming approximately the same amount of machine time and requiring more rollouts, on-policy DAPO does not yield a substantial improvement. Moreover, under an equal three-round comparison, the setting with distillation also performs better.

MethodRound/StepAIME25HMMTHMMT-NovRollout trajectoriesWall timeCompute details
No distillationR183.667.670.416,00079 h33 h rollout + 23 h × 2 teacher tuning
No distillationR284.167.970.032,000158 h2 × (33 h + 23 h × 2)
No distillationR3—66.4668.5448,000237 h3 × (33 h + 23 h × 2)
Default distillationR182.562.968.116,00084 h33 h rollout + 23 h × 2 teacher tuning + 5 h distillation
Default distillationR284.567.968.532,000168 h2 × (33 h + 23 h × 2 + 5 h)
Default distillationR384.571.273.548,000252 h3 × (33 h + 23 h × 2 + 5 h)
Distillation + entropyR183.7—71.016,00084 hSame compute as default distillation
Distillation + entropyR288.3—74.532,000168 hSame compute as default distillation
Distillation + entropyR385.6—76.048,000252 hSame compute as default distillation
On-policy DAPOUpdate 100 / global 25, peak—56.88—102,400224.5 h25 × 2 batches × 128 prompts × 16 responses
On-policy DAPOUpdate 120 / global 30, final—53.75—122,880269.35 h30 × 2 batches × 128 prompts × 16 responses
  1. (W2) The experiments do not fully support the proposed explanation for why ERPD works. My main concern is that the actual training objective does not seem to directly correspond to the KL projection motivated by Equation (3). If we ignore clipping, the fixed token-level score effectively encourages the student to optimize a teacher-reference log-ratio reward under the old policy's state distribution. This objective can push the student beyond the teacher rather than simply distilling the teacher's behavior, which is also discussed in Appendix B.1. I think more controlled experiments are needed to understand whether the proposed score is responsible for filtering spurious policy drift.

Clipping is in fact crucial, as it provides an effect similar to that of the KL loss. In an earlier version of our method, we used standard policy gradients with a KL penalty. However, we found that the KL coefficient often required retuning during distillation. We therefore switched to PPO clipping, for which the community has established widely used default values. Our comparison shows that PPO clipping performs comparably to PG with a KL penalty while avoiding additional hyperparameter tuning.

The table below presents the MSE experiment on JustRL-1.5B. When clipping is removed, performance declines even with the same 20-step distillation budget.

SettingAIME24AIME25HMMTCommon-3 Avg
Ratio clipping71.2563.5440.4258.40
No ratio clipping70.4261.2538.0256.56
Clip − No-clip+0.83+2.29+2.40+1.84

Regarding extrapolation, when examining the phenomenon in Figure 5, we find that even after 16 training steps, the model’s measured total KL remains lower than that of the teacher. This suggests that the student can outperform the teacher by amplifying useful components while filtering out irrelevant ones.

For iterative experiments, we recommend the conservative setting of 10 steps, as noted in the paper. The detailed results in Table 11 show that allowing the student not to surpass the teacher in each individual round, while consuming only a small KL budget, can lead to better progressive performance across rounds.

(Weakness 3)For example, the paper does not compare against random, shuffled, constant, original-reward, or sparsity-matched token scores. Without these controls, ...

(Quesiton 2)What happens if the teacher-reference token scores are replaced with shuffled, random, constant, binary, or sparsity-matched scores? ...

Negative-value shuffle and Negative-mean also perform well, comparably to the proposed Positive-value masking. They respectively shuffle or average the negative-valued positions within positive examples. This suggests that the exact scale of the scores at these positions is not critical, whereas their locations remain important. Random-mask, a sparsity-matched score control that randomizes the negative-score locations, performs substantially worse.

Several constant-score variants also perform worse, including Correct-response score zeroing, Correct-response constant +01˙+0\.1, Binary sign-mean, Sequence constant, and Constant. Randomized and shuffled variants, including Random-global, Random, and Shuffle raw, similarly degrade performance. Original reward, which assigns the raw outcome reward at the token level, also performs poorly.

We reiterate that, based on our Qwen3.5 experiments, we still recommend the original setting as the default for model development. Positive-value masking is only an ablation motivated by our interest in understanding how MSE-based signals take effect. It is not intended to replace the original setting.

VariantHMMTDescription
Negative-value shuffle66.88Masks positive scores in correct responses, then shuffles only the remaining negative values within each response.
Positive-value masking (The proposed ablation in the paper)66.67Sets tokens with score > 0 to zero in correct responses while retaining negative scores.
Negative-mean66.25Replaces all remaining negative scores in each correct response with that response’s mean negative score.
Correct-response score zeroing65.97Sets every valid token score in correct responses to zero, regardless of sign.
Random-mask65.83Randomly masks the same number of nonzero positions as positive-value masking; retained tokens keep their original scores.
Correct-response constant +01˙+0\.165.07Sets every valid token score in correct responses to +0.1; incorrect responses retain their original scores.
Binary sign-mean64.38Maps positive and negative scores to their respective batch-wide sign means, retaining only score signs.
Sequence constant63.85Assigns every token in a response its post-whitening mean score, preserving the response-level score sum.
Random-global63.82Generates Gaussian scores matching only the global mean and variance of the raw batch.
Original reward63.75Broadcasts each response’s unnormalized outcome reward to all its valid tokens.
Random63.26Generates per-response Gaussian scores matching each response’s original mean and variance.
Nanbeige Base62.50
Shuffle raw61.67Randomly permutes all raw token scores within each response while preserving their distribution.
Constant60.21Sets every valid token score to +1; whitening is disabled to prevent the signal from collapsing to zero.

I thank the authors for their detailed responses to my questions. Based on these clarifications, I will raise my score from 2 to 3.

Reviewer_oCnNOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ERPD, a two-stage framework for reusing rollouts more efficiently in LLM reinforcement learning. Stage 1 aggressively optimizes a fixed rollout batch to obtain a teacher far outside the usual trust region. Stage 2 resets the student to the rollout policy and trains it with a clipped objective, using the frozen teacher–reference log-ratio as a token-level signal. The authors also extend the idea to weak teachers by contrasting checkpoints along a single training trajectory.

Critical strengths
  1. Practical problem, simple design. Rollout cost is a real bottleneck. Separating signal extraction from the deployed update is a clean idea that is easy to implement.
  2. Insightful analysis. The comparison between fixed and evolving score references gives a clear explanation of why the student can surpass the teacher. The weak-teacher result is also interesting: contrasting two checkpoints still provides useful supervision.
  3. Broad and transparent experiments. The paper evaluates multiple teacher objectives and model scales, includes a wall-clock comparison, and honestly reports settings where the method helps little.
Critical weaknesses
  1. Limited algorithmic novelty. The method mainly combines existing components: fixed-batch reuse, log-ratio implicit rewards, and PPO clipping. The contribution lies mostly in how these pieces are arranged.
  2. Theoretical analysis could be strengthened. The paper is mainly empirical, and the KL-projection view in Eq. (3) serves only as motivation. Some analysis of why the student can surpass its teacher, or why the recovered–unlearned contrast works, would make this a very good paper, even under simplified assumptions.
  3. Robustness of the evidence. HMMT Feb 25 is used for most selection decisions, including checkpoints, hyperparameters and the strategy switch (L205–206, L359), and several gains are small (Tables 2 and 5). Some design choices are not well explained. For example, keeping only the negative scores in the weak-teacher setting yields higher validation accuracy, but its transfer effects are mixed and the rationale is only briefly discussed (L420–423; App. L805–807, L1130–1133). In addition, the 9B/27B runs report no KL or compute cost, so the KL-efficiency claim is not verified at larger scale (L363–368; Table 4, L378–391).
Questions for authors

See weakness.

3 comments

(Weakness 1) Limited algorithmic novelty. The method mainly combines existing components: fixed-batch reuse, log-ratio implicit rewards, and PPO clipping. The contribution lies mostly in how these pieces are arranged.

Thank you for pointing this out. We do not claim contributions regarding implicit rewards, PPO clipping, or fixed-batch reuse. Our contributions are primarily novel empirical findings:

  1. At comparable final performance, distillation reduces KL divergence and enables higher asymptotic performance.
  2. A student can outperform a stronger teacher even when both are trained on the same data.

One notable algorithmic contribution is our proposed MSE teacher. We further demonstrate its potential on the newer and larger Qwen3.5-9B and Qwen3.5-27B models.

(Weakness 2) Theoretical analysis could be strengthened. The paper is mainly empirical, and the KL-projection view in Eq. (3) serves only as motivation. Some analysis of why the student can surpass its teacher, or why the recovered–unlearned contrast works, would make this a very good paper, even under simplified assumptions.

Thank you for the suggestion. We are conducting experiments on the recovered–unlearned contrast for a new paper. Since the current paper already contains extensive experimental results and analyses, we plan to investigate the underlying mechanism in a separate follow-up work.

(Weakness 3.1) HMMT Feb 25 is used for most selection decisions, including checkpoints, hyperparameters and the strategy switch (L205–206, L359):

We also validate the possibility of selecting the strong teacher using other validation sets or even the training set. Our approach is not limited to HMMT Feb 2025, and we report results on multiple transfer validation sets in the paper.

We constructed two datasets: one containing 50 examples sampled from the current 1,000-prompt training set, and another containing 50 examples sampled from the held-out full DeepScaleR training distribution.

StepHMMTTransfer: deepscaler 50 sampleTrainset 50 sample
Base63.9075.7571.13
60—78.7573.00
8065.2178.8873.13
10068.9678.8872.75
12071.2577.6373.75
14057.9273.6367.75

(Weakness 3.2) several gains are small (Tables 2 and 5).

For Table 2, we also discuss this in Figure 2. When the teacher is sufficiently strong, it is difficult to substantially outperform the teacher. However, we can still achieve gains in terms of KL efficiency: for example, halving the KL while attaining performance comparable to that of the teacher. Table 5 reports the strong–weak teacher ensemble experiment. It examines whether strong and weak teachers capture complementary knowledge, such that combining them can yield further improvements. Since this is a weak-to-strong setting, the gains may not be large; nevertheless, the approximately 2 percentage-point improvements on AIME24 and HMMT-Feb are relatively consistent.

(Weakness 3.4) In addition, the 9B/27B runs report no KL or compute cost, so the KL-efficiency claim is not verified at larger scale

Thank you for pointing this out. Since our 9B/27B experiments use an MSE teacher, the reduction in KL is actually quite substantial.

ModelKL
Qwen3.5-27B-ERPD-0010.00876
Qwen3.5-27B-MSE-Teacher-0010.04834
Qwen3.5-27B-ERPD-0020.02201
Qwen3.5-27B-ERPD-0030.03776

(Weakness 3.3) keeping only the negative scores in the weak-teacher setting yields higher validation accuracy, but its transfer effects are mixed and the rationale is only briefly discussed

This is not an experimental setting that we primarily advocate. As noted in the paper, we did not adopt this strategy in our experiments with the larger Qwen3.5 model, since it provided no clear improvement over the default setting.

After observing the phenomenon in Figure 8, we conducted several ablations and found some potentially counterintuitive performance gains. We therefore discuss this phenomenon in the appendix and provide an example to encourage interested readers to conduct similar ablation studies.

We also provide more fine-grained ablation studies:

VariantHMMTDescription
Negative-value shuffle66.88<br>Masks positive scores in correct responses, then shuffles only the remaining negative values within each response.
Positive-value masking (The proposed ablation in the paper)66.67<br>Sets tokens with score > 0 to zero in correct responses while retaining negative scores.
Negative-mean66.25<br>Replaces all remaining negative scores in each correct response with that response’s mean negative score.
Correct-response score zeroing65.97Sets every valid token score in correct responses to zero, regardless of sign.
Random-mask65.83Randomly masks the same number of nonzero positions as positive-value masking; retained tokens keep their original scores.
Correct-response constant +01˙+0\.165.07Sets every valid token score in correct responses to +0.1; incorrect responses retain their original scores.
Binary sign-mean64.38Maps positive and negative scores to their respective batch-wide sign means, retaining only score signs.
Sequence constant63.85Assigns every token in a response its post-whitening mean score, preserving the response-level score sum.
Random-global63.82Generates Gaussian scores matching only the global mean and variance of the raw batch.
Original reward63.75Broadcasts each response’s unnormalized outcome reward to all its valid tokens.
Random63.26Generates per-response Gaussian scores matching each response’s original mean and variance.
Nanbeige Base62.50
Shuffle raw61.67Randomly permutes all raw token scores within each response while preserving their distribution.
Constant60.21Sets every valid token score to +1; whitening is disabled to prevent the signal from collapsing to zero.

Negative-value shuffle and Negative-mean also perform well, comparably to the proposed Positive-value masking. They respectively shuffle or average the negative-valued positions within positive examples. This suggests that the exact scale of the scores at these positions is not critical, whereas their locations remain important. Random-mask, a sparsity-matched score control that randomizes the negative-score locations, performs substantially worse.

Several constant-score variants also perform worse, including Correct-response score zeroing, Correct-response constant +01˙+0\.1, Binary sign-mean, Sequence constant, and Constant. Randomized and shuffled variants, including Random-global, Random, and Shuffle raw, similarly degrade performance.

I thank the authors for their clarifications. I believe my concerns have been adequately addressed, and the authors’ response has further strengthened my confidence in my assessment.

Given that I have already assigned a rating of 3, and my overall assessment of the paper’s contribution remains unchanged, I will maintain my rating while increasing my confidence score.

Reviewer_gAiZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Extreme Region Policy Distillation, a two-stage method for reusing rollout data while limiting policy drift in reinforcement learning for reasoning models. It first trains a teacher through many weakly constrained updates on a fixed rollout batch. A student then starts from the policy that generated the batch and takes a few clipped updates, guided by fixed token-level differences between the teacher's and reference model's log probabilities. The authors also extract supervision from differences between unlearned and recovered checkpoints, even when the recovered teacher is less accurate than the starting model. Experiments on mathematical reasoning with 3B to 27B models report accuracy gains and better trade-offs between accuracy and KL divergence in several settings, supported by ablations and timing comparisons.

Critical strengths
  1. ERPD separates extensive training on a fixed batch from the final policy update, offering a practical way to address the cost of rollout generation in long-reasoning RL. In the paper, ERPD reaches 73.3% on the HMMT validation set after roughly 21 hours, compared with 68.5% for on-policy SAPO.

  2. Author shows that a weak teacher can still provide useful supervision through differences between checkpoints. In the Qwen3-4B experiment, the MSE teacher achieves only 38.5% average accuracy across the transfer benchmarks, yet the student trained on its recovery signal reaches 73.2%, above the base model's 71.6%. This finding suggests that changes along a training trajectory can contain useful information that standalone checkpoint accuracy does not capture.

Critical weaknesses

The paper does not convincingly establish that the signal extracted by prolonged fixed-batch optimization is generalizable rather than specific to the reused rollouts or the validation benchmark. The motivating results show that validation accuracy can continue to improve after many updates on one rollout batch, but most analyses of the optimization trajectory use HMMT Feb 25, which is also used for checkpoint and hyperparameter selection. The paper reports encouraging transfer results for the selected endpoints, but it does not show how transfer performance evolves as the same rollout batch is optimized more aggressively. It is therefore unclear whether later updates continue to discover transferable reasoning behavior, or whether the method relies on selecting a favorable point from a trajectory that increasingly overfits the fixed batch or validation set. ERPD lacks a sufficiently specified stopping and model-selection rule that can be applied without extensive task-specific validation. Several consequential decisions are made adaptively: the Stage 1 teacher checkpoint, the number of distillation steps, the switch from the default score to Strategies 2 or 3, the unlearned and recovered checkpoints. The manuscript generally selects these choices using HMMT Feb 25, but provides no practical criterion for determining when a rollout batch has been sufficiently learned on a new task. As a result, it is difficult to tell whether the reported gains come from a robust training procedure or from a substantial search over checkpoints and signal constructions. This also weakens the sample-efficiency claim, since the cost and data requirements of repeated validation and hyperparameter selection are not included in the comparison.

Questions for authors

Could the authors provide the complete selection protocol and search budget for the headline iterative results? Could the authors show experiment results that include transfer-set performance, or performance on a separate untouched validation set? Without access to a labeled benchmark such as HMMT Feb 25, how to decide when to stop Stage 1 and which teacher checkpoint to distill? Is the intended algorithm based on a fixed update budget, validation early stopping, or a training-time diagnostic such as held-out rollout loss or KL growth, or positive–negative separation?

2 comments

(Question 1) Could the authors provide the complete selection protocol and search budget for the headline iterative results?

For the six-round iterative experiments, we use CE training to obtain the strong teacher. We search over β∈{0.001,0.002,0.01}\beta \in \{0.001, 0.002, 0.01\} and train for approximately 100 steps, with early stopping if the probability of negative examples decreases substantially. We evaluate every 10 steps and select a checkpoint that improves validation-set performance as the teacher.

When the improvement is limited, we switch to Strategy 2. For Strategy 2, we train the recovery teacher to convergence, defined as the positive-example probability returning to its pre-unlearning level and the training dynamics becoming approximately stable. We then search over unlearned teachers. Most experiments use a learning rate of 1×10−51\times10^{-5}; if distillation performs poorly, we additionally try 2×10−62\times10^{-6}.

During distillation, we directly use the validation-selected strong teacher. For the weak teacher, we generally consider unlearned teachers from steps 10, 20, and 30, perform distillation with each, and select the one yielding the best post-distillation validation performance.

We use the default distillation budget of 10 steps throughout the iterative experiments. All reported experiments with both Qwen3 and Qwen3.5 use 10 distillation steps.

We also provide the following time estimates for reference:

For the full pipeline on 8 H20 GPUs, assuming Qwen3-4B with a batch size of 32, generating rollouts for 1,000 prompts takes approximately 1.08 hours × 33, or about 33 hours.

Training takes approximately 0.23 hours per step × 100 steps, or about 23 hours. Since CE may require tuning different beta values, most of the compute time is spent on teacher training and hyperparameter tuning. Assuming three hyperparameter configurations, this requires approximately 23 × 3 = 69 hours.

The distillation stage typically requires only 10–20 training steps. Each iteration takes just a few hours, and each validation run takes approximately one hour. Therefore, distillation is relatively lightweight compared with teacher training.

Evaluation on Unused Transfer Sets

(Weakness 1) The paper does not convincingly establish that the signal extracted by prolonged fixed-batch optimization is generalizable rather than specific to the reused rollouts or the validation benchmark. The motivating results show that validation accuracy can continue to improve after many updates on one rollout batch, but most analyses of the optimization trajectory use HMMT Feb 25, ...

(Question 2)Could the authors show experiment results that include transfer-set performance, or performance on a separate untouched validation set?

We constructed two datasets: one containing 50 examples sampled from the current 1,000-prompt training set, and another containing 50 examples sampled from the held-out full DeepScaleR training distribution.

Table 1. Validation and transfer-set accuracy for Qwen3-4B.

StepHMMTTransfer: deepscaler 50 sampleTrainset 50 sample
Base56.0076.75**60.63 **
2060.0076.50—
4066.8876.7562.25
6067.5077.3864.25
8064.5878.1362.25
10065.4278.1361.63
140—78.3862.25

Table 2. Validation and transfer-set accuracy for NBG4-3B.

StepHMMTTransfer: deepscaler 50 sampleTrainset 50 sample
Base63.9075.75**71.13 **
60—78.7573.00
8065.2178.8873.13
10068.9678.8872.75
12071.2577.6373.75
14057.9273.6367.75

We also evaluate on ArxivMath-0226, a previously unseen benchmark whose problems are drawn from arXiv and therefore differ in distribution from the competition-style problems used for training.

ModelAcc.Δ vs base
Qwen3.5-27B-ERPD-00334.77%+3.52pp
Qwen3.5-27B base31.25%—

(Weakness 2) The manuscript generally selects these choices using HMMT Feb 25, but provides no practical criterion for determining when a rollout batch has been sufficiently learned on a new task.

(Question 3) Without access to a labeled benchmark such as HMMT Feb 25, how to decide when to stop Stage 1 and which teacher checkpoint to distill?

We investigate whether “sufficiently learned” rollouts behave as their name suggests and how they relate to training-set rewards. As shown in Tables 1 and 2 above, for both NBG4-3B and Qwen3-4B, training-set reward trends are correlated with validation performance. This suggests that training-set rewards could potentially be used for checkpoint selection when no held-out validation set is available.

(Question 4) Is the intended algorithm based on a fixed update budget, validation early stopping, or a training-time diagnostic such as held-out rollout loss or KL growth, or positive–negative separation?

For strong teachers, we combine validation-based early stopping with the positive–negative separation observed in the training curves. This reduces the frequency of validation evaluations. We typically evaluate checkpoints near the point of maximum separation, provided that the negative-example probability has not collapsed. For Beige and Qwen, this point usually occurs later in training, around steps 80–100. If the negative-example probability collapses earlier, we may instead select an earlier checkpoint, such as step 40. Our experiments further show that checkpoints immediately preceding the decline in negative-example probability tend to align well with validation performance. We therefore prioritize checkpoints near this point during selection.

For weak teachers, we stop training based on positive–negative separation alone. Specifically, the positive-example probability typically drops sharply before gradually recovering to its pre-forgetting level. We stop once this recovery is complete. For example, if the initial positive-example probability is 0.8, we monitor the training logs and stop when it returns to approximately 0.8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.