- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes REGEN, a method that learns a generalist model from multiple domain specialists, but contrary to MOPD, it does so by recycling trajectories collected during the RL training of these domain specialists. The training of the generalist model is done with offline RL: an advantage-weighted offline objective with importance weighting. The method is tested on small scale models, and is demonstrated in their experiments to be on par with MOPD while being faster to train.
Critical strengths
The main strength of the method proposed is its simplicity and practicality. The method can be a direct contender to MOPD while being way simpler to use: If the practitioner has control over training specialists and the generalist model, the method proposes reusing the already generated RL trajectories and removes the need for online generation or teacher inference during student training, matching performance of MOPD while being faster and simpler to implement.
Critical weaknesses
I have a problem with how the paper validates the method and some claims:
-
Evaluation is limited to Qwen2.5 small models (<1.5B), with super short responses (), which basically cannot tell us anything about how the approach behaves in real world settings (bigger models, longer trajectories).
-
The training throughput comparison is misleading: MOPD allocates three GPUs to teachers and only one to the student, whereas REGEN uses all four GPUs for student training. I understand that with their stack, collocating everything on the same GPU is not possible, but this is a ressource handling claim, not a training throughput claim.
-
The claim that REGEN avoids entropy collapse is not supported at all. no figure reports the entropy of REGEN’s trajectory, and even in the main 1.5B experiments, REGEN’s pass@10 is below the original model for coding.
Questions for authors
- Can you please address the weaknesses? At least try to scale the experiments, and correct the claims on entropy collapse.
- Methodology wise, how would you use the method if given already trained specialists without having access to a memory replay?
3 comments
Response to Reviewer 5dCK (Part 2/2)
Q3: REGEN lacks evidence of preventing entropy collapse during training.
A3: We agree that our original wording might be too strong. Offline training alone does not guarantee that a policy will retain its entropy. Figure 2 shows that entropy declined for both the RL teacher and the OPD student in the original code-only experiment. This figure leaves open how REGEN’s entropy evolved during training.
To address this directly, we recorded the entropy trajectories of both REGEN and MOPD in our new 7B-scale experiments. We used a common protocol across all three training domains (Math, Code, and Alignment) and omitted the response deduplication step for both methods. This allows us to compare entropy throughout the multi-domain training run and removes the dependence of the original analysis on a separately deduplicated, code-only dataset.
REGEN uses replay trajectories, while MOPD generates responses on-policy, so the two methods can have different initial entropy values even under the same measurement protocol. We therefore measure the decline from each method’s own first recorded value. REGEN’s entropy decreased from near 0.20 to 0.11 early in training and remained nearly constant thereafter. MOPD’s entropy started at 1.12 and stabilized at approximately 0.05. REGEN therefore showed a much smaller decline in policy entropy under the same multi-domain measurement protocol. We will revise Section 4.3 to state this empirical conclusion.
We therefore maintain that entropy collapse is an inherent limitation of online methods, whereas offline methods are less susceptible to this issue. This claim is also strongly supported by the findings reported in a NIPS2025 paper[1].
[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.
Q4: How can REGEN use pretrained specialists without access to replay memory?
A4: We believe there may be a misunderstanding regarding the role of memory replay in our method. REGEN’s replay setting assumes access to logged specialist trajectories, rewards, and policy log-probabilities. When only the final specialist models are available and their training trajectories are inaccessible, a possible alternative is to generate responses using these specialists and train the student to imitate them. This corresponds to the Behavior Cloning(BC) baseline already evaluated in Table 1 of our paper, where REGEN consistently outperforms BC.
I thank the authors for their efforts to address the points raised. The new experiments strengthen the contribution of REGEN as a faster alternative to MOPD. However, the claim that REGEN outperforms MOPD no longer holds when longer traces are used, and the discussion of entropy collapse in online versus offline methods remains speculative. Overall, I am not sufficiently convinced to recommend acceptance and will therefore maintain my current score.
Response to Reviewer 5dCK (Part 1/2)
We thank the reviewer for the thoughtful feedback and address the main concerns below.
Q1: REGEN requires further validation on larger-scale models, and its efficiency comparison with MOPD is confounded by different GPU allocation.
A1:We thank the reviewer for their interest in the scalability of REGEN. To evaluate its effectiveness at a larger model scale, we conduct additional experiments across three domains (Math, Code, and Alignment) using Qwen2.5-7B. The results show trends largely consistent with those observed at the 0.5B and 1.5B scales. REGEN achieves performance comparable to or better than MOPD on most metrics while delivering higher training throughput and lower per-token training latency. We conduct these experiments on a single server equipped with eight H20 GPUs. MOPD uses four GPUs for rollouts and four GPUs for training, whereas REGEN uses four GPUs for training, ensuring that both methods allocate the same amount of compute to student training. The experimental results are shown in the table below:
\hline & \text{GSM8K} & \text{GSM8K} & \text{MATH} & \text{MATH} & \text{HumanEval} & \text{HumanEval} & \text{MBPP} & \text{MBPP} & \text{IFEval} & \text{IFEval} \\\\ & \text{pass@1} & \text{maj@4} & \text{pass@1} & \text{maj@4} & \text{pass@1} & \text{pass@10} & \text{pass@1} & \text{pass@10} & \text{loose} & \text{strict} \\\\ \hline \text{Teacher} & 93.0 & 94.5 & 73.8 & 75.8 & 83.5 & 90.2 & 83.9 & 89.7 & 87.4 & 85.4 \\\\ \text{Base} & 67.9 & 76.9 & 48.4 & 56.3 & 77.4 & 90.2 & 75.1 & 92.3 & 43.1 & 40.2 \\\\ \text{BC} & 88.6 & 91.4 & 70.1 & 72.8 & 78.9 & 90.2 & 76.8 & 86.8 & 54.4 & 51.3 \\\\ \text{MOPD} & 91.7 & 92.6 & 72.9 & 76.2 & 81.7 & 88.4 & 78.0 & 86.5 & 61.8 & 55.4 \\\\ \text{REGEN} & 92.4 & 93.0 & 72.9 & 75.4 & 81.1 & 93.9 & 75.7 & 88.9 & 58.8 & 54.1 \\\\ \hline \end{array}$$ $$\begin{array}{l|cccc} \hline & \text{Training Throughput} & \text{Speed} & \text{Per-1k-Token Latency} & \text{Speed} \\\\ \hline \text{MOPD} & 3816.8\\ \text{tokens/s} & 1\times & 262.0\\ \text{ms} & 1\times \\\\ \text{REGEN} & 14579.1\\ \text{tokens/s} & 3.80\times & 68.6\\ \text{ms} & 3.82\times \\\\ \hline \end{array}$$ --- **Q2: REGEN’s performance on tasks requiring long responses remains unclear.** **A2**: To address this concern, we extend our evaluation to 7B-scale models on mathematical reasoning tasks with longer response budgets. Specifically, we finetune Qwen2.5-7B exclusively on the MATH dataset with a maximum response length of 4,096 tokens to obtain a math-domain teacher, and initialize the student from Qwen2.5-7B. We evaluate the student on AMC23, AIME24, and AIME25 using the avg@32 metric, with a maximum response length of 8,192 tokens. As shown in the table below, REGEN substantially improves upon the base model across all three benchmarks and achieves accuracy broadly comparable to OPD. These results suggest that, like OPD, REGEN can generalize from training on shorter sequences to reasoning tasks requiring longer responses. $$\begin{array}{lccc} \hline & \text{AMC23} & \text{AIME24} & \text{AIME25} \\\\ \hline \text{Base} & 26.6 & 4.6 & 1.3 \\\\ \text{Teacher} & 53.4 & 12.8 & 3.8 \\\\ \text{OPD} & 49.1 & 11.6 & 3.7 \\\\ \text{REGEN} & 46.5 & 10.8 & 4.8 \\\\ \hline \end{array}$$