acceptodds
Under review as a conference paper at ICLR 2027

REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning

Abstract

Large-scale online reinforcement learning (RL) is the predominant means of eliciting advanced abilities including long-term reasoning and agentic tool use in large language models (LLMs). However, continuing to scale it across vast task domains of interest remains challenging in both computational infrastructure and cost, especially when considering RL as merely a one-off learning stage. Recently, a widely used technique for distilling knowledge across various domains and training stages, multi-teacher on-policy distillation (MOPD), helps to decouple the RL stage, saving costs, while maintaining generality across vast domains. Nonetheless, similar to online RL, MOPD requires coupled inference and backward passes, which continue to limit its scalability and computational efficiency. To address these challenges, we propose REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline RL. Instead of distilling from multiple teacher models, REGEN trains a generalist by simply recycling the replay memory—the free by-product of the teachers' specialized RL training—and employing offline RL algorithms. REGEN completely decouples the rollout sampling from the backward training process and thus greatly reduces the training cost. Across mathematical reasoning, code generation, and instruction following, REGEN matches the accuracy of MOPD at substantially lower cost. It potentially turns online RL into a data synthesis process instead of a one-off learning stage, and can be extended to large-scale post-training without requiring heavy computational load.

Primary area:
reinforcement learning
Keywords:
LLM post-training, On-policy distillation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5dCKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes REGEN, a method that learns a generalist model from multiple domain specialists, but contrary to MOPD, it does so by recycling trajectories collected during the RL training of these domain specialists. The training of the generalist model is done with offline RL: an advantage-weighted offline objective with importance weighting. The method is tested on small scale models, and is demonstrated in their experiments to be on par with MOPD while being faster to train.

Critical strengths

The main strength of the method proposed is its simplicity and practicality. The method can be a direct contender to MOPD while being way simpler to use: If the practitioner has control over training specialists and the generalist model, the method proposes reusing the already generated RL trajectories and removes the need for online generation or teacher inference during student training, matching performance of MOPD while being faster and simpler to implement.

Critical weaknesses

I have a problem with how the paper validates the method and some claims:

  • Evaluation is limited to Qwen2.5 small models (<1.5B), with super short responses (T=1024T=1024), which basically cannot tell us anything about how the approach behaves in real world settings (bigger models, longer trajectories).

  • The training throughput comparison is misleading: MOPD allocates three GPUs to teachers and only one to the student, whereas REGEN uses all four GPUs for student training. I understand that with their stack, collocating everything on the same GPU is not possible, but this is a ressource handling claim, not a training throughput claim.

  • The claim that REGEN avoids entropy collapse is not supported at all. no figure reports the entropy of REGEN’s trajectory, and even in the main 1.5B experiments, REGEN’s pass@10 is below the original model for coding.

Questions for authors
  • Can you please address the weaknesses? At least try to scale the experiments, and correct the claims on entropy collapse.
  • Methodology wise, how would you use the method if given already trained specialists without having access to a memory replay?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5dCK (Part 1/2)

We thank the reviewer for the thoughtful feedback and address the main concerns below.


Q1: REGEN requires further validation on larger-scale models, and its efficiency comparison with MOPD is confounded by different GPU allocation.

A1:We thank the reviewer for their interest in the scalability of REGEN. To evaluate its effectiveness at a larger model scale, we conduct additional experiments across three domains (Math, Code, and Alignment) using Qwen2.5-7B. The results show trends largely consistent with those observed at the 0.5B and 1.5B scales. REGEN achieves performance comparable to or better than MOPD on most metrics while delivering higher training throughput and lower per-token training latency. We conduct these experiments on a single server equipped with eight H20 GPUs. MOPD uses four GPUs for rollouts and four GPUs for training, whereas REGEN uses four GPUs for training, ensuring that both methods allocate the same amount of compute to student training. The experimental results are shown in the table below:

\hline & \text{GSM8K} & \text{GSM8K} & \text{MATH} & \text{MATH} & \text{HumanEval} & \text{HumanEval} & \text{MBPP} & \text{MBPP} & \text{IFEval} & \text{IFEval} \\\\ & \text{pass@1} & \text{maj@4} & \text{pass@1} & \text{maj@4} & \text{pass@1} & \text{pass@10} & \text{pass@1} & \text{pass@10} & \text{loose} & \text{strict} \\\\ \hline \text{Teacher} & 93.0 & 94.5 & 73.8 & 75.8 & 83.5 & 90.2 & 83.9 & 89.7 & 87.4 & 85.4 \\\\ \text{Base} & 67.9 & 76.9 & 48.4 & 56.3 & 77.4 & 90.2 & 75.1 & 92.3 & 43.1 & 40.2 \\\\ \text{BC} & 88.6 & 91.4 & 70.1 & 72.8 & 78.9 & 90.2 & 76.8 & 86.8 & 54.4 & 51.3 \\\\ \text{MOPD} & 91.7 & 92.6 & 72.9 & 76.2 & 81.7 & 88.4 & 78.0 & 86.5 & 61.8 & 55.4 \\\\ \text{REGEN} & 92.4 & 93.0 & 72.9 & 75.4 & 81.1 & 93.9 & 75.7 & 88.9 & 58.8 & 54.1 \\\\ \hline \end{array}$$ $$\begin{array}{l|cccc} \hline & \text{Training Throughput} & \text{Speed} & \text{Per-1k-Token Latency} & \text{Speed} \\\\ \hline \text{MOPD} & 3816.8\\ \text{tokens/s} & 1\times & 262.0\\ \text{ms} & 1\times \\\\ \text{REGEN} & 14579.1\\ \text{tokens/s} & 3.80\times & 68.6\\ \text{ms} & 3.82\times \\\\ \hline \end{array}$$ --- **Q2: REGEN’s performance on tasks requiring long responses remains unclear.** **A2**: To address this concern, we extend our evaluation to 7B-scale models on mathematical reasoning tasks with longer response budgets. Specifically, we finetune Qwen2.5-7B exclusively on the MATH dataset with a maximum response length of 4,096 tokens to obtain a math-domain teacher, and initialize the student from Qwen2.5-7B. We evaluate the student on AMC23, AIME24, and AIME25 using the avg@32 metric, with a maximum response length of 8,192 tokens. As shown in the table below, REGEN substantially improves upon the base model across all three benchmarks and achieves accuracy broadly comparable to OPD. These results suggest that, like OPD, REGEN can generalize from training on shorter sequences to reasoning tasks requiring longer responses. $$\begin{array}{lccc} \hline & \text{AMC23} & \text{AIME24} & \text{AIME25} \\\\ \hline \text{Base} & 26.6 & 4.6 & 1.3 \\\\ \text{Teacher} & 53.4 & 12.8 & 3.8 \\\\ \text{OPD} & 49.1 & 11.6 & 3.7 \\\\ \text{REGEN} & 46.5 & 10.8 & 4.8 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer 5dCK (Part 2/2)

Q3: REGEN lacks evidence of preventing entropy collapse during training.

A3: We agree that our original wording might be too strong. Offline training alone does not guarantee that a policy will retain its entropy. Figure 2 shows that entropy declined for both the RL teacher and the OPD student in the original code-only experiment. This figure leaves open how REGEN’s entropy evolved during training.

To address this directly, we recorded the entropy trajectories of both REGEN and MOPD in our new 7B-scale experiments. We used a common protocol across all three training domains (Math, Code, and Alignment) and omitted the response deduplication step for both methods. This allows us to compare entropy throughout the multi-domain training run and removes the dependence of the original analysis on a separately deduplicated, code-only dataset.

REGEN uses replay trajectories, while MOPD generates responses on-policy, so the two methods can have different initial entropy values even under the same measurement protocol. We therefore measure the decline from each method’s own first recorded value. REGEN’s entropy decreased from near 0.20 to 0.11 early in training and remained nearly constant thereafter. MOPD’s entropy started at 1.12 and stabilized at approximately 0.05. REGEN therefore showed a much smaller decline in policy entropy under the same multi-domain measurement protocol. We will revise Section 4.3 to state this empirical conclusion.

We therefore maintain that entropy collapse is an inherent limitation of online methods, whereas offline methods are less susceptible to this issue. This claim is also strongly supported by the findings reported in a NIPS2025 paper[1].

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.


Q4: How can REGEN use pretrained specialists without access to replay memory?

A4: We believe there may be a misunderstanding regarding the role of memory replay in our method. REGEN’s replay setting assumes access to logged specialist trajectories, rewards, and policy log-probabilities. When only the final specialist models are available and their training trajectories are inaccessible, a possible alternative is to generate responses using these specialists and train the student to imitate them. This corresponds to the Behavior Cloning(BC) baseline already evaluated in Table 1 of our paper, where REGEN consistently outperforms BC.

I thank the authors for their efforts to address the points raised. The new experiments strengthen the contribution of REGEN as a faster alternative to MOPD. However, the claim that REGEN outperforms MOPD no longer holds when longer traces are used, and the discussion of entropy collapse in online versus offline methods remains speculative. Overall, I am not sufficiently convinced to recommend acceptance and will therefore maintain my current score.

Reviewer_zV9tOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REGEN, an offline method for transferring the capabilities of domain-specific RL experts into a generalist student. Unlike multi-teacher on-policy distillation (MOPD), which requires ongoing student rollout generation and teacher supervision, REGEN reuses historical trajectories collected throughout the experts' RL training, together with their rewards and behavior-policy probabilities. The method combines class-balanced sampling with an advantage-weighted objective: positive trajectories contribute directly to learning, while negative trajectories receive an additional truncated importance weight.

Experiments cover mathematical reasoning, code generation, and instruction following using 0.5B and 1.5B Qwen2.5 students. REGEN achieves broadly comparable performance to MOPD and approximately fivefold higher training throughput under the reported hardware configuration. Additional experiments examine sampling and objective choices, code-generation pass@k, and joint versus separate domain training.

Critical strengths
  1. Practical usefulness. The paper addresses an important practical problem in expert-to-generalist training. Reusing historical RL trajectories provides a potentially useful alternative to conventional online distillation.
  2. Computational efficiency. REGEN removes the need for online rollout generation and teacher supervision during student training, substantially simplifying the training pipeline and improving efficiency.
  3. Empirical effectiveness. The proposed approach achieves competitive performance across several domains and model sizes, demonstrating the feasibility of offline generalist training from historical RL data.
Critical weaknesses
  1. Limited generalization. REGEN relies on pre-existing RL trajectories with rewards and behavior-policy probabilities. Unlike MOPD, it cannot directly consolidate SFT-only experts without additional data collection and evaluation. This restricts its applicability and makes its efficiency advantage conditional on the availability of suitable replay data.
  2. Limited methodological novelty. The core components, including experience replay, importance sampling, group-normalized advantages, and class-balanced sampling, are largely established techniques. Related works such as ReOPD [1] and Lightning OPD [2] have also explored historical trajectory reuse or offline distillation. The contribution appears primarily to be a combination of existing methods rather than a substantial algorithmic innovation.
  3. Limited performance gains. REGEN does not consistently outperform MOPD, although this is understandable given its efficiency-oriented motivation. More importantly, its improvements over simpler offline baselines are modest (68.2 vs. 67.2 for positive-only SFT and 66.9 for CBS + TOPR), making the additional value of the proposed optimization objective unclear.
  4. Insufficient experimental validation. The experiments are limited to small models and relatively simple, single-turn tasks. Key design choices lack direct validation, including the benefit of full-history replay over final-teacher trajectories and the effectiveness of importance weighting for negative samples. The diversity claims also lack sufficient supporting analysis.
Questions for authors
  1. Effect of class-balanced sampling. In Table 4, CBS substantially degrades IFEval performance for TOPR and GRPO and MATH performance for GRPO, while slightly improving REGEN on all benchmarks. What explains this difference across objectives?
  2. Can the authors explicitly compare REGEN with existing works such as ReOPD [1] and Lightning OPD [2] in terms of trajectory sources, the need for generation and teacher queries during student training, and supervision signals? How do these differences define REGEN's central additional contribution and the settings in which it is useful?
  3. Appendix A describes Math rewards as 0/1, while Appendix B shows a -1/+1 reward distribution. Which reward convention is used for REGEN training?
5 comments
AuthorsOpenReview ↗

Response to Reviewer zV9t (Part 1/3)

We thank the reviewer for the thoughtful feedback and address the main concerns below.


Q1: Does REGEN’s reliance on historical RL trajectories limit its applicability compared with MOPD?

A1: We believe there is a misunderstanding regarding the applicability of MOPD [1] . The original MOPDpaper states that it also requires experts trained via domain RL for supervision. REGEN targets precisely this setting, namely further utilizing the experience generated during expert training for student training. Therefore, given recorded trajectories, rewards, and behavior policy probabilities, REGEN does not require additional online sampling or teacher queries for the student. Its data can leverage data from different training stages of RL rather than being limited to the final expert checkpoint, demonstrating the flexibility of the REGEN method. We also acknowledge that for SFT-only experts without such records, REGEN requires additional data collection and evaluation. This constitutes the applicability boundary of the current method.

[1] Ma, Wenhan, et al. "Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training." arXiv preprint arXiv:2606.30406 (2026).

AuthorsOpenReview ↗

Response to Reviewer zV9t (Part 2/3)

Q2: How does REGEN differ from ReOPD and Lightning OPD, and what is its core contribution?

A2: REGEN asks whether the experience already collected while training specialist RL teachers is sufficient to train a generalist offline. ReOPD also reuses historical teacher trajectories, but uses them as prefixes from which the student generates new actions under teacher supervision. Lightning OPD asks a different question: when can offline distillation on fixed student rollouts recover the same optimum as online OPD? Its analysis examines how the teacher used to initialize the student through SFT, and its consistency with the subsequent OPD teacher, affects the policy to which distillation converges.

\hline \text{Method} & \begin{array}{l} \text{Source and use} \\\\ \text{of trajectories} \end{array} & \begin{array}{l} \text{Student generation} \\\\ \text{during optimization} \end{array} & \text{Teacher supervision} \\\\ \hline \text{REGEN} & \begin{array}{l} \text{Trains directly on complete} \\\\ \text{responses logged throughout} \\\\ \text{specialist RL training} \end{array} & \text{None} & \begin{array}{l} \text{Uses logged trajectory rewards to} \\\\ \text{compute within-query advantages} \\\\ \text{and logged behavior-policy} \\\\ \text{probabilities to weight} \\\\ \text{below-threshold responses} \end{array} \\\\ \hline \text{ReOPD} & \begin{array}{l} \text{Replays prefixes from} \\\\ \text{historical teacher interactions} \end{array} & \begin{array}{l} \text{The student generates an} \\\\ \text{action at selected steps,} \\\\ \text{without new environment} \\\\ \text{interactions} \end{array} & \begin{array}{l} \text{The teacher provides token-level} \\\\ \text{supervision on the student's actions} \end{array} \\\\ \hline \text{Lightning OPD} & \begin{array}{l} \text{Uses fixed responses generated} \\\\ \text{in advance by the} \\\\ \text{SFT-initialized student} \end{array} & \text{None} & \begin{array}{l} \text{The teacher scores each response} \\\\ \text{before training, and the student} \\\\ \text{uses the cached token log-probabilities} \end{array} \\\\ \hline \end{array}$$ REGEN’s core contribution is to show how multiple specialists’ RL logs can directly supervise a generalist. Its objective compares responses to the same query across training stages and applies truncated importance weighting to below-threshold responses. When these logs are available, the student can learn from both successful and unsuccessful responses without generating new rollouts or querying the final teachers. Motivated by Lightning OPD’s analysis, we added RQ3 (Table 5) to the revised paper which also can be seen in the below table. We initialized a 7B student with either behavior cloning (BC) or REGEN, then applied offline MOPD with the same 7B teachers and identical optimization settings. REGEN + MOPD outperformed BC + MOPD on all ten reported metrics. These results show that REGEN provides a better initialization for subsequent offline MOPD in our setting and provide indirect evidence of better teacher–student alignment than BC. We will also add a direct empirical comparison with ReOPD as soon as the results are available. $$\begin{array}{l|cc|cc|cc|cc|cc} \hline & \text{GSM8K} & \text{GSM8K} & \text{MATH} & \text{MATH} & \text{HumanEval} & \text{HumanEval} & \text{MBPP} & \text{MBPP} & \text{IFEval} & \text{IFEval} \\\\ & \text{pass@1} & \text{maj@4} & \text{pass@1} & \text{maj@4} & \text{pass@1} & \text{pass@10} & \text{pass@1} & \text{pass@10} & \text{loose} & \text{strict} \\\\ \hline \text{Teacher} & 93.0 & 94.5 & 73.8 & 75.8 & 83.5 & 90.2 & 83.9 & 89.7 & 87.4 & 85.4 \\\\ \text{BC + MOPD} & 91.7 & 93.4 & 72.9 & 75.4 & 80.5 & 92.7 & 78.4 & 88.6 & 61.8 & 58.3 \\\\ \text{REGEN + MOPD} & 93.5 & 94.5 & 75.4 & 76.8 & 83.7 & 94.0 & 81.8 & 89.4 & 63.1 & 61.8 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer zV9t (Part 3/3)

Q3: Are the gains over simpler offline baselines sufficient to demonstrate REGEN’s value?

A3: REGEN’s key contribution lies in demonstrating the value of replay memory accumulated during specialist RL training for offline generalist training and providing a concrete sampling and optimization recipe for exploiting it.

The positive-only SFT baseline in Table 4 shows that historical replay contains useful supervision even under a simple training objective. Table 4 further compares sampling strategies and objectives to investigate how to exploit that supervision. CBS+TOPR achieves a higher average score than Uniform+TOPR. This comparison supports CBS as a useful component of REGEN’s training recipe, whose benefit can be observed even when paired with an existing offline objective. These comparisons therefore support the value of our design.


Q4: Why does CBS affect TOPR, GRPO, and REGEN differently?

A4: CBS's effect depends on how each objective updates the model.

For TOPR, policy updates are weighted directly by raw rewards. In Alignment task, partially compliant responses receive rewards between 0 and 1. Consequently, samples classified as negative by our threshold can still receive positive updates when their rewards are positive. Math differs because incorrect responses receive a reward of −1 under TOPR and therefore provide an explicit negative learning signal.

For GRPO, CBS changes the sample mixture and group-normalized advantages, but its effect on performance also depends on ratio clipping. When there is a mismatch between historical-policy and student model, some positive-advantage updates may already be clipped. If CBS increases the contribution of active negative advantage updates without a corresponding increase in positive ones, the resulting optimization can become overly suppressive and reduce accuracy. This provides a possible explanation for the decreases on MATH and IFEval.

For REGEN, it combines group-normalized advantages with asymmetric importance weighting. Positive responses retain their learning signal without importance-ratio suppression, while below-threshold responses receive truncated importance weights. This preserves positive supervision while limiting the influence of below-threshold responses, providing a plausible explanation for the observed benefits of CBS.


Q5: Does mathematical training use 0/1 or −1/+1 rewards?

A5: During the teachers’ online RL training, correct and incorrect responses receive rewards of 1 and 0, respectively. During offline student training, we map these rewards to +1 and −1 so that incorrect responses provide a nonzero learning signal under TOPR, which weights updates by raw rewards. REGEN uses the same mapping for consistency, and Appendix B reports these offline rewards. We will clarify the stage-specific reward conventions in Appendices A/B.

Reviewer_ADeqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

REGEN trains a generalist model offline by reusing trajectories from specialist teachers' RL training, combining class-balanced sampling, group-normalized rewards, and asymmetric importance weighting. Across math, code, and instruction following, it achieves competitive accuracy with multi-teacher on-policy distillation (MOPD) and approximately five times higher training throughput in the reported setup. Reusing these data without keeping teachers in the training loop is appealing, but the experiments do not yet isolate the benefit of historical replay or sufficiently justify the weighting rule. The claims about entropy collapse and training efficiency also need stronger evidence.

Critical strengths
  1. A useful and simple training pipeline. Recording trajectory likelihoods during teacher training allows later optimization without additional teacher forward passes. This is attractive when specialist RL runs already exist, and the data collection and student training stages are clearly separated.

  2. Meaningful evidence for capability integration. The evaluation includes five benchmarks and both 0.5B and 1.5B students. REGEN is competitive with MOPD on several metrics, particularly MATH and code pass@10. Joint training also stays within 1.7 points of the corresponding single-domain students in Table 5, supporting the approach in this three-domain setting.

  3. Relevant ablations are included. Table 4 compares TOPR, off-policy GRPO, DPO, and positive-only SFT, rather than relying solely on behavior cloning. Crossing sampling strategies with several objectives helps show that the combination of CBS and REGEN matters: its reported average is 68.2, compared with 66.8 for REGEN with uniform sampling.

Critical weaknesses
  1. The source of the improvement is not sufficiently isolated. REGEN improves the average over CBS+TOPR by 1.3 points and positive-only SFT by 1.0 point, without uncertainty estimates. There is no comparison that removes importance weighting while retaining normalized advantages, or that holds the objective fixed while replacing historical replay with equally sized data from a final teacher. Appendix A also describes negative-reward remapping for TOPR only in Math, leaving its treatment of Code and Alignment less clear. These omissions make it difficult to separate the benefits of replay history, reward normalization, and the particular weighting rule.

  2. The mathematical interpretation needs clarification. Eq. 5 selects its weighting branch using the reward threshold, whereas Eq. 6 determines the update sign using the group mean. With continuous rewards, these need not agree: a balanced group with rewards {1, 1, 0, 0.9} gives the 0.9 response a positive advantage, but still places it in the truncated-weight branch. Moreover, Eq. 7 records the original behavior likelihood, while Eq. 9 subsequently changes the sampling distribution through reward-dependent selection. Together with unweighted positive samples, this makes the update a biased surrogate rather than a full importance correction. That can be reasonable, but the intended objective and the rationale for these choices should be explicit.

  3. The entropy-collapse explanation is stronger than the evidence. Sec. 4.3 argues that offline training makes REGEN not directly subject to entropy collapse, yet Fig. 2 contains no REGEN entropy curve. In Table 1, the 1.5B REGEN model actually falls below the base model on both HumanEval pass@10 (70.1 vs. 73.8) and MBPP pass@10 (77.8 vs. 84.9). Table 3 provides encouraging results after deduplication, but also changes to a 10K code-only setting and shows slightly lower pass@1 than OPD. This supports a setting-specific advantage in pass@10; it does not establish the proposed causal explanation.

  4. The efficiency comparison needs more complete reporting. The fivefold throughput gain is measured with three GPUs reserved for teachers and one for the MOPD student, versus four student GPUs for REGEN. This demonstrates a benefit for that deployment, but leaves open how much depends on teacher-serving efficiency. The stated global batch size of 128 also needs reconciling with the one-student-GPU configuration and the local batch/accumulation settings in Sec. 4.1. Finally, throughput alone does not show time to comparable quality, especially given the accuracy gaps on GSM8K and IFEval.

Questions for authors
  1. Separate the contributions experimentally. Extend Table 4 with normalized-advantage training without importance weights, and ablate reward centering versus variance normalization relative to TOPR [Le Roux et al., 2025]. Compare historical replay against final-teacher data with matched prompts and sample counts. Specify reward transformations and baseline tuning, and report variation across seeds for these relatively small gains.

  2. Explain and test the surrogate objective. Clarify how CBS affects the interpretation of Eqs. 5–9. Compare threshold-based weighting with advantage-sign-based weighting, particularly on continuous-reward tasks. Report the resulting weight distributions by domain and trajectory length, including how often useful positive-advantage responses receive near-zero weights.

  3. Test the diversity explanation directly. Add REGEN entropy curves under the same evaluation protocol as OPD. Compare both methods with and without deduplication while keeping the code-only setting and training budget fixed. Document the deduplication procedure, decoding parameters, and pass@k estimator, and narrow the entropy-collapse claim unless these controls support it.

  4. Measure the practical cost–accuracy tradeoff. Report accuracy against elapsed time on the same hardware budget, with the MOPD serving configuration and effective batch sizes fully specified. Separate marginal distillation cost from shared teacher-training cost, include replay preparation overhead, and provide units and timing boundaries for Table 2.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ADeq (Part 1/2)

We thank the reviewer for the thoughtful feedback and address the main concerns below.


Q1: How do importance weighting, reward centering, and variance normalization contribute to the gains?

A1: REGEN adopts the asymmetric truncated importance-weighting rule from TOPR. TOPR applies this rule to the reward and REGEN applies it to a group-normalized advantage. The advantage calculation follows GRPO’s practice of centering rewards within each prompt group and scaling them by the group standard deviation. In REGEN, these groups are constructed from class-balanced responses collected across different teacher checkpoints.


Q2: How does historical replay data compare with data from the final teacher?

A2: Our Behavior Cloning (BC) baseline already provides a comparison with data generated by the final teachers. For each domain, we select the best checkpoint from the online RL run and use that it to generate responses. REGEN use trajectories collected throughout expert training, including those generated before the expert reached its best performance. Despite this variation in the source trajectories, REGEN outperforms BC on 9 of the 10 metrics for the 1.5B student and on all 10 for the 0.5B student in Table 1. These results suggest that REGEN can learn reliably from a mixture of expert training stages.


Q3: Specify reward transformations and baseline tuning.

A3: During the teachers’ online RL training, correct and incorrect responses receive rewards of 1 and 0, respectively. During offline student training, we map these rewards to +1 and −1 so that incorrect responses provide a nonzero learning signal under TOPR, which weights updates by raw rewards. REGEN uses the same mapping for consistency, and Appendix B reports these offline rewards. We will clarify the stage-specific reward conventions in Appendices A/B. For baseline tuning, beyond standard training hyperparameters such as the learning rate, REGEN requires no method-specific hyperparameter tuning. We also use fixed configurations for the baselines. Section 4.1 describes all the training settings.


Q4: Are the modest gains of the REGEN method consistent across seeds?

A4: We acknowledge that the gains in Table 1 are modest and that their variation across random seeds remains to be measured. We will supplement the results with multiple-seed experiments as soon as possible and report the corresponding variability. More importantly, the main contribution of REGEN is its ability to reuse specialist training trajectories to train a generalist with performance close to MOPD while achieving higher training throughput and lower per-token training latency.

AuthorsOpenReview ↗

Response to Reviewer ADeq (Part 2/2)

Q5: How does class-balanced sampling affect the interpretation of REGEN’s surrogate objective, and how does reward-threshold weighting compare with advantage-sign weighting for above-average, partial-credit responses?

A5: Thank you for raising this point. CBS changes which responses enter each prompt group, and we compute advantages over the selected responses. Our use of group-relative advantages follows GRPO. Comparing each response’s reward with those of other responses to the same prompt gives us a common learning signal across domains with different reward scales. We use a separate full-credit threshold for weighting because a response that passes all tests or satisfies all verified constraints provides a clear training signal.


Q6: REGEN lacks evidence of preventing entropy collapse during training.

A6: We agree that our original wording might be too strong. Offline training alone does not guarantee that a policy will retain its entropy. Figure 2 shows that entropy declined for both the RL teacher and the OPD student in the original code-only experiment. This figure leaves open how REGEN’s entropy evolved during training. To address this directly, we recorded the entropy trajectories of both REGEN and MOPD in our new 7B-scale experiments. We used a common protocol across all three training domains (Math, Code, and Alignment) and omitted the response-deduplication step for both methods. This allows us to compare entropy throughout the multi-domain training run and removes the dependence of the original analysis on a separately deduplicated, code-only dataset. REGEN uses replay trajectories, while MOPD generates responses on-policy, so the two methods can have different initial entropy values even under the same measurement protocol. We therefore measure the decline from each method’s own first recorded value. REGEN’s entropy decreased from near 0.20 to 0.11 early in training and remained nearly constant thereafter. MOPD’s entropy started at 1.12 and stabilized at approximately 0.05. REGEN therefore showed a much smaller decline in policy entropy under the same multi-domain measurement protocol. We will revise Section 4.3 to state this empirical conclusion and include both trajectories. We therefore maintain that entropy collapse is an inherent limitation of online methods, whereas offline methods are less susceptible to this issue. This claim is also strongly supported by the findings reported in a NIPS2025 paper[1]. [1] Yang Yue et al. “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” NeurIPS 2025.


Q7: What speedup does REGEN achieve over MOPD under the same GPU budget? Please describe the teacher-serving configuration, specify the effective student batch size, and report the measured training time for each method.

A7: Both methods were evaluated on a single node with eight NVIDIA H20 GPUs. Each method used four GPUs for student training, with an effective batch size of 128 per optimizer update. MOPD served domain-specific teachers online using 4 GPUs. REGEN trained on logged trajectories and required no online teacher service. Detailed speedup can be seen in the table below. We measured wall-clock time from the start to the end of student distillation. This timing excludes specialist teacher training and all data preparation completed before distillation. Under this definition, REGEN took around 4 hours and MOPD took nearly 25 hours.

\hline & \text{Training Throughput} & \text{Speed} & \text{Per-1k-Token Latency} & \text{Speed} \\\\ \hline \text{MOPD} & 3816.8\\ \text{tokens/s} & 1\times & 262.0\\ \text{ms} & 1\times \\\\ \text{REGEN} & 14579.1\\ \text{tokens/s} & 3.80\times & 68.6\\ \text{ms} & 3.82\times \\\\ \hline \end{array}$$

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.