- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a systematic empirical comparison of three paradigms for consolidating domain-specific RLVR capabilities into a single generalist model: parameter-space Merge, mixed-domain RL (Mix RL), and multi-teacher on-policy distillation (MOPD). The three paradigms are organized according to which artifacts they reuse from domain-specialized training: task vectors, training datasets, or both. Using Qwen3-4B and Qwen3-8B backbones, five domains (math, science, code, instruction following, and agentic tool use), shared experts and datasets, and a broad benchmark suite, the paper studies final performance, cross-domain transfer, task-vector geometry, training dynamics, compute cost, solution coverage, and held-out capability retention. The main results show that the three fusion methods have similar average performance but substantially different domain-level behavior: Mix RL can benefit from positive transfer in reasoning domains but depends on mixture proportions, Merge is nearly free once experts exist but can dilute weakly aligned capabilities, and MOPD converges quickly and preserves teacher behavior but does not explicitly provide a signal for surpassing the teachers. The paper further argues that fusion mainly reweights solutions already accessible to the base model rather than expanding solution coverage, while no measurable degradation is observed on two held-out capabilities.
Critical strengths
1.The paper places Merge, Mix RL, and MOPD under a common artifact-based taxonomy and evaluates them with shared domain data, experts, model backbones, and benchmarks. This makes the comparison more informative than treating the methods as unrelated techniques evaluated in different settings.
2.The study spans five qualitatively different domains and reports both domain-level and aggregate performance, rather than relying only on a single average score. The additional analyses of cross-domain transfer, task-vector geometry, training dynamics, and GPU cost provide useful practical insight into when each paradigm may be preferable.
3.The observation that reasoning-related domains tend to transfer positively while instruction following and agentic behavior are more weakly aligned is plausible and is supported by both behavioral and parameter-space measurements. The comparison between prompt allocation in Mix RL and teacher supervision in MOPD also gives a reasonably intuitive explanation for their different convergence behavior.
Critical weaknesses
1.Merge, mixed-domain RLVR, and on-policy distillation are all established paradigms, and the paper’s primary contribution is their organization and controlled comparison rather than a new algorithmic mechanism. Besides, I find a very similar work[1]. Compared to that, what are the greatest advantages and areas for improvement of your work?
2.Mix RL is trained for 800 steps, whereas the per-domain experts and MOPD use 300 steps. MOPD also uses a different batch size and rollout count from GRPO. Consequently, the reported cost and convergence comparisons conflate the paradigm with different optimization budgets and implementation choices. In addition, the “Per-domain RL” row aggregates five separate models and should not be interpreted as a single unified-model baseline.
3.The experiments use only two Qwen3-based backbones, and the coverage/retention study is limited to the 4B model, two held-out tasks, and model-based judgments. The task-vector geometry analysis is correlational and lacks uncertainty estimates, while the claim that standard MOPD is “bounded” by its teachers is not a strict consequence of the objective on held-out benchmarks. The objective provides no explicit surpass-teacher signal, but student generalization or teacher combination could still lead to improvements.
[1]Wang, Haoqing, et al. "To mix or to merge: Toward multi-domain reinforcement learning for large language models." arXiv preprint arXiv:2602.12566 (2026).
Questions for authors
1.How many independent training seeds were run for each expert and fusion method? Can the authors report confidence intervals or paired significance tests for the main benchmark results, especially for the small average differences between Merge, Mix RL, and MOPD?
2.What happens if all methods are compared under matched numbers of training tokens, sampled prompts, rollout tokens, or GPU-hours rather than the current 300-versus-800-step setup? Please also clarify whether teacher forward/log-probability computation is fully included in the reported MOPD fusion cost.
3.Was (\lambda=0.6) selected using a validation set independent of the reported benchmarks? Likewise, how sensitive are the results to the fixed 25/22/22/19/12 mixture, and can the authors provide a sweep or a principled procedure for choosing the mixture proportions per backbone?
4 comments
Response (2/n)
[W3] Scope of the empirical conclusions
Thank you for asking us to clarify how broadly these conclusions hold.
Model coverage. We extended the analysis to Qwen3.5-4B with three specialists. The first table shows score changes from the starting model in percentage points.
| Training domain | AIME25/26 avg. | GPQA | IFBench | BFCL v3 |
|---|---|---|---|---|
| Math | +25.97 | +9.68 | −0.19 | −0.92 |
| Science | +21.81 | +9.05 | +1.64 | +0.21 |
| Instruction following | +1.81 | +3.62 | +23.06 | −1.42 |
Math and science transferred in both directions, while instruction-following training gave the largest IFBench gain. Following Figure 3(d), we compared task-vector similarity with bidirectional transfer on AIME26, GPQA and IFBench.
| Expert pair | Cosine between layer-16 updates | Mean bidirectional transfer (pp) |
|---|---|---|
| Math–Science | 0.004390 | +15.0 |
| Math–IF | −0.000394 | +2.1 |
| Science–IF | 0.001279 | +2.6 |
As on the original Qwen3 backbones, the math–science pair had both the most similar updates at layer 16 and the strongest bidirectional transfer. This experiment illustrates how the patterns of cross-domain transfer and task-vector alignment we observed on Qwen3 extend to other model families.
Choice of held-out tasks. We chose two tasks that test capabilities our five-domain RLVR training did not explicitly target. SimpleQA-Verified tests whether the model can recall facts without reference documents; AA-LCR asks it to combine evidence from several long documents. Prior work likewise uses general knowledge to assess retention after post-training [3] and long-context reasoning to assess transfer from math RL [4].
Grading reliability. We used Qwen3-32B to grade four responses per question against the reference answers using the prompts in Appendix D. The authors independently assessed the same 300 randomly selected examples. The judge agreed with the human verdict on at least 97% of those examples.
Geometry. We tested the robustness of the association in Figure 3(d): domains with more similar expert updates tend to show stronger cross-domain transfer. Because the ten domain pairs share the same five domains, we tested all 120 permutations of the domain labels, applying each to both backbones. At layer 16, 13 permutations gave a correlation at least as large in magnitude as the observed Spearman (two-sided ). The correlation remained positive after excluding the math–science pair (). These five domains provide evidence that experts with more similar weight updates make larger gains over the base model on each other’s tasks.
Teacher performance. Standard MOPD trains the student to match the domain teacher’s next-token distribution on the student’s own rollouts. At a prefix where the two distributions match, the KL loss is zero; the objective provides no separate reward for a more accurate answer. Consistent with this, MOPD’s eight-benchmark average is 0.6 points below the five-specialist average on both backbones (Table 1). For an analysis of how closely MOPD’s parameter updates align with those of its teachers, please see our response to Reviewer 2 (Q2).
[Q1] Training seeds and uncertainty in the main table
We appreciate your question about uncertainty in Table 1. Given the cost of RL training, we report single-run results. To reduce decoding variability, we average the scores of 32 responses per question and report the result as mean@32 in Table 1. To quantify uncertainty in the small score differences, we computed paired 95% bootstrap confidence intervals. The overall average weights all eight benchmarks equally. For this average, we report intervals for Merge versus Mix RL on 4B and for Mix RL versus MOPD on 8B. The per-question records for these models reproduce all eight benchmark scores in Table 1 to the reported precision.
| Contrast | Difference in pp [pointwise 95% CI] |
|---|---|
| 4B Merge − Mix RL, eight-benchmark average | +1.41 [0.70, 2.14] |
| 8B Mix RL − MOPD, eight-benchmark average | +1.37 [0.29, 2.48] |
| 8B Mix RL − MOPD, AIME25 | +8.65 [4.06, 13.75] |
| 8B Mix RL − MOPD, IFBench | −4.89 [−8.36, −1.50] |
All four intervals exclude zero. These checks support the pattern in Table 1 under mean@32: the gaps in average score are small, while on 8B, Mix RL leads on AIME25 and MOPD leads on IFBench.
Response (3/n)
[Q3a] Selection of the merge coefficient
Thank you for raising this important question. We selected using the Qwen3-8B benchmarks reported in Table 1; we did not use an independent validation set. In the Appendix E sweep (Figure 7), increasing from 0.3 to 1.0 improved accuracy but also increased mean response length from 2,042 to 7,334 tokens. At , we observed no reasoning traces in the AIME responses. At with a 32k-token limit, about 66% of AIME responses included a reasoning trace despite the non-thinking template. We chose to keep this shift into thinking mode from confounding the comparison.
For five experts, their arithmetic mean corresponds to in . At 0.6 the merged update norm is about 1.35 times the RMS expert norm on both backbones. Ilharco et al. [2] likewise use a shared coefficient and find its best value depends on the combination. Our specific value comes from our sweep.
[Q3b] Sensitivity to the Mix RL proportions
Thank you for raising this practical question about the training mixture. Wang et al. [1] pooled five domain subsets from the Nemotron 3 Nano RL training blend [5]. We used the resulting sample proportions on both backbones: approximately 25% math, 22% science, 22% code, 19% IF and 12% agent. In this ablation, we tested how a larger IF share affects Qwen3-4B’s performance across domains. Including each IF example twice raised that share from 18.9% to 31.8%. Scores below are percentages, shown as original / IF×2.
| Step | IFBench | GPQA | AIME26 | Eight-benchmark average |
|---|---|---|---|---|
| 400 | 34.75 / 40.25 | 62.75 / 62.37 | 60.83 / 55.83 | 60.73 / 61.54 |
| 450 | 37.00 / 41.08 | 60.73 / 60.73 | 52.50 / 50.83 | 60.68 / 59.14 |
| 475 | 37.08 / 40.42 | 61.11 / 0.13 | 59.17 / 4.17 | 61.22 / 33.62 |
At step 400, IF×2 led on IFBench by 5.50 points. Between steps 450 and 475, IF×2’s performance collapsed in other domains: its GPQA score fell from 60.73% to 0.13%, and its AIME26 score from 50.83% to 4.17%. By then, the IF×2 run had become unstable. The earlier gains on IFBench and later losses on math and science reinforce our conclusion that Mix RL requires careful data mixing across domains.
References
[1] Wang, Haoqing, et al. "To mix or to merge: Toward multi-domain reinforcement learning for large language models." arXiv preprint arXiv:2602.12566 (2026).
[2] Ilharco, Gabriel, et al. "Editing models with task arithmetic." arXiv preprint arXiv:2212.04089 (2022).
[3] Chen, Howard, et al. "Retaining by doing: The role of on-policy data in mitigating forgetting." arXiv preprint arXiv:2510.18874 (2025).
[4] Motwani, Sumeet Ramesh, et al. "h1: Bootstrapping llms to reason over longer horizons via reinforcement learning." arXiv preprint arXiv:2510.07312 (2025).
[5] Blakeman, Aaron, et al. "Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning." arXiv preprint arXiv:2512.20848 (2025).
Thank you again for your careful reading and detailed suggestions. We would be happy to discuss any remaining questions or concerns.
Thanks for your response.
Thank you for your response. I still find the results in the paper to be too low, which undermines their persuasiveness. Even for analytical experiments, the results are only convincing when the performance is sufficiently strong. Therefore, I will keep my score.
Response (1/n)
Thank you for your detailed and constructive review. We especially appreciate your observation that the results for individual domains reveal trade-offs that are hard to see in the average scores. We address your concerns point by point below.
[W1] Overlap with related work.
Thank you for pointing us to this closely related study. Our contribution is a controlled empirical comparison of three established RLVR fusion paradigms: Merge, Mix RL and MOPD. We study how they perform across domains, why their outcomes differ and what each costs. The closest precedent is the study by Wang et al. [1]: they study three paradigms in five domains and analyze weight-shift cosines. Building on this, we test which domain-level trade-offs persist across two instruction-tuned checkpoints and whether the pattern for Merge changes when we use LoRA experts in place of full-parameter experts.
These comparisons matter because cross-domain transfer and similarities between expert updates do not, by themselves, tell us which specialist gains a single fused model will retain (Section 4.2). To interpret differences between Mix RL and a specialist, we also count prompts by domain: the same number of training steps need not give the two models equal exposure to that domain (Section 5.2). At evaluation time, varying the number of attempts per problem lets us test whether higher single-sample accuracy also translates into solving a broader set of problems (Section 5.3). Finally, to determine when faster distillation also saves compute, we distinguish the cost of fusion from the total cost when the teachers must first be trained (Section 5.2).
[W2, Q2] Training schedules, matched budgets and teacher cost
We appreciate your suggestion to compare methods at matched budgets. We chose 800 updates for Mix RL and 300 for MOPD based on training and validation trends and the cost of further training. On 4B, another 400 Mix RL updates raised its eight-benchmark average by 0.90 percentage points; another 100 MOPD updates lowered its average by 0.12. For more detail on how we chose these checkpoints, see our response to Reviewer 2 (W2/Q1). Equal step counts would still give Mix RL fewer prompts per domain than a specialist sees (Section 5.2). Since MOPD does not require GRPO’s within-group baseline, we followed Wang et al. [1] in using 4 rollouts per prompt for MOPD and 16 for each of the 128 prompts in a GRPO batch. To compare performance at similar budgets, we used the 300-step MOPD checkpoint on 8B as a reference and selected the closest Mix RL checkpoint for each resource in turn.
On Qwen3-8B, MOPD scores higher than Mix RL on IFBench in all four budget comparisons. On both AIME26 and LCB v6, Mix RL scores higher when prompt counts are matched. The 899 GPU-hours for MOPD fusion include teacher forward passes for log probabilities. Training the teachers brings the total to 5,569 GPU-hours. Merge requires only weight arithmetic after the same 4,670 GPU-hours of expert training. At comparable total cost, Merge averages 53.7 across the eight benchmarks versus 56.82 for Mix RL (4,749 GPU-hours, 1,200 updates).
Regarding the “Per-domain RL” row in Table 1, we retain it as the specialist baseline: comparing each fused model with the corresponding domain expert shows which specialist gains it retains or exceeds. As the reviewer notes, this row draws on five experts, one per domain, whereas each fusion method uses one model across all eight benchmarks. We will label the row “In-domain specialists (one per domain)” and clarify that its average gives equal weight to the eight benchmark scores.