acceptodds
Under review as a conference paper at ICLR 2027

Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artifacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, RLVR, Fusion Paradigms

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XLQ3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a systematic empirical comparison of three paradigms for consolidating domain-specific RLVR capabilities into a single generalist model: parameter-space Merge, mixed-domain RL (Mix RL), and multi-teacher on-policy distillation (MOPD). The three paradigms are organized according to which artifacts they reuse from domain-specialized training: task vectors, training datasets, or both. Using Qwen3-4B and Qwen3-8B backbones, five domains (math, science, code, instruction following, and agentic tool use), shared experts and datasets, and a broad benchmark suite, the paper studies final performance, cross-domain transfer, task-vector geometry, training dynamics, compute cost, solution coverage, and held-out capability retention. The main results show that the three fusion methods have similar average performance but substantially different domain-level behavior: Mix RL can benefit from positive transfer in reasoning domains but depends on mixture proportions, Merge is nearly free once experts exist but can dilute weakly aligned capabilities, and MOPD converges quickly and preserves teacher behavior but does not explicitly provide a signal for surpassing the teachers. The paper further argues that fusion mainly reweights solutions already accessible to the base model rather than expanding solution coverage, while no measurable degradation is observed on two held-out capabilities.

Critical strengths

1.The paper places Merge, Mix RL, and MOPD under a common artifact-based taxonomy and evaluates them with shared domain data, experts, model backbones, and benchmarks. This makes the comparison more informative than treating the methods as unrelated techniques evaluated in different settings.

2.The study spans five qualitatively different domains and reports both domain-level and aggregate performance, rather than relying only on a single average score. The additional analyses of cross-domain transfer, task-vector geometry, training dynamics, and GPU cost provide useful practical insight into when each paradigm may be preferable.

3.The observation that reasoning-related domains tend to transfer positively while instruction following and agentic behavior are more weakly aligned is plausible and is supported by both behavioral and parameter-space measurements. The comparison between prompt allocation in Mix RL and teacher supervision in MOPD also gives a reasonably intuitive explanation for their different convergence behavior.

Critical weaknesses

1.Merge, mixed-domain RLVR, and on-policy distillation are all established paradigms, and the paper’s primary contribution is their organization and controlled comparison rather than a new algorithmic mechanism. Besides, I find a very similar work[1]. Compared to that, what are the greatest advantages and areas for improvement of your work?

2.Mix RL is trained for 800 steps, whereas the per-domain experts and MOPD use 300 steps. MOPD also uses a different batch size and rollout count from GRPO. Consequently, the reported cost and convergence comparisons conflate the paradigm with different optimization budgets and implementation choices. In addition, the “Per-domain RL” row aggregates five separate models and should not be interpreted as a single unified-model baseline.

3.The experiments use only two Qwen3-based backbones, and the coverage/retention study is limited to the 4B model, two held-out tasks, and model-based judgments. The task-vector geometry analysis is correlational and lacks uncertainty estimates, while the claim that standard MOPD is “bounded” by its teachers is not a strict consequence of the objective on held-out benchmarks. The objective provides no explicit surpass-teacher signal, but student generalization or teacher combination could still lead to improvements.

[1]Wang, Haoqing, et al. "To mix or to merge: Toward multi-domain reinforcement learning for large language models." arXiv preprint arXiv:2602.12566 (2026).

Questions for authors

1.How many independent training seeds were run for each expert and fusion method? Can the authors report confidence intervals or paired significance tests for the main benchmark results, especially for the small average differences between Merge, Mix RL, and MOPD?

2.What happens if all methods are compared under matched numbers of training tokens, sampled prompts, rollout tokens, or GPU-hours rather than the current 300-versus-800-step setup? Please also clarify whether teacher forward/log-probability computation is fully included in the reported MOPD fusion cost.

3.Was (\lambda=0.6) selected using a validation set independent of the reported benchmarks? Likewise, how sensitive are the results to the fixed 25/22/22/19/12 mixture, and can the authors provide a sweep or a principled procedure for choosing the mixture proportions per backbone?

4 comments
AuthorsOpenReview ↗

Response (1/n)

Thank you for your detailed and constructive review. We especially appreciate your observation that the results for individual domains reveal trade-offs that are hard to see in the average scores. We address your concerns point by point below.


[W1] Overlap with related work.

Thank you for pointing us to this closely related study. Our contribution is a controlled empirical comparison of three established RLVR fusion paradigms: Merge, Mix RL and MOPD. We study how they perform across domains, why their outcomes differ and what each costs. The closest precedent is the study by Wang et al. [1]: they study three paradigms in five domains and analyze weight-shift cosines. Building on this, we test which domain-level trade-offs persist across two instruction-tuned checkpoints and whether the pattern for Merge changes when we use LoRA experts in place of full-parameter experts.

These comparisons matter because cross-domain transfer and similarities between expert updates do not, by themselves, tell us which specialist gains a single fused model will retain (Section 4.2). To interpret differences between Mix RL and a specialist, we also count prompts by domain: the same number of training steps need not give the two models equal exposure to that domain (Section 5.2). At evaluation time, varying the number of attempts per problem lets us test whether higher single-sample accuracy also translates into solving a broader set of problems (Section 5.3). Finally, to determine when faster distillation also saves compute, we distinguish the cost of fusion from the total cost when the teachers must first be trained (Section 5.2).


[W2, Q2] Training schedules, matched budgets and teacher cost

We appreciate your suggestion to compare methods at matched budgets. We chose 800 updates for Mix RL and 300 for MOPD based on training and validation trends and the cost of further training. On 4B, another 400 Mix RL updates raised its eight-benchmark average by 0.90 percentage points; another 100 MOPD updates lowered its average by 0.12. For more detail on how we chose these checkpoints, see our response to Reviewer 2 (W2/Q1). Equal step counts would still give Mix RL fewer prompts per domain than a specialist sees (Section 5.2). Since MOPD does not require GRPO’s within-group baseline, we followed Wang et al. [1] in using 4 rollouts per prompt for MOPD and 16 for each of the 128 prompts in a GRPO batch. To compare performance at similar budgets, we used the 300-step MOPD checkpoint on 8B as a reference and selected the closest Mix RL checkpoint for each resource in turn.

8B checkpointMatched budget (MOPD / Mix RL)IFBenchAIME26LCB v6
MOPD, step 300Reference for all four comparisons42.4240.0051.75
Mix RL, step 175Prompt + response tokens: 956M / 892M28.5029.1747.39
Mix RL, step 200Response tokens (estimated): 708M / 720M29.1730.8348.25
Mix RL, step 250Fusion GPU-hours: 899 / 87031.1728.3348.62
Mix RL, step 625Sampled prompts: 79,200 / 80,00036.1741.6752.60

On Qwen3-8B, MOPD scores higher than Mix RL on IFBench in all four budget comparisons. On both AIME26 and LCB v6, Mix RL scores higher when prompt counts are matched. The 899 GPU-hours for MOPD fusion include teacher forward passes for log probabilities. Training the teachers brings the total to 5,569 GPU-hours. Merge requires only weight arithmetic after the same 4,670 GPU-hours of expert training. At comparable total cost, Merge averages 53.7 across the eight benchmarks versus 56.82 for Mix RL (4,749 GPU-hours, 1,200 updates).

Regarding the “Per-domain RL” row in Table 1, we retain it as the specialist baseline: comparing each fused model with the corresponding domain expert shows which specialist gains it retains or exceeds. As the reviewer notes, this row draws on five experts, one per domain, whereas each fusion method uses one model across all eight benchmarks. We will label the row “In-domain specialists (one per domain)” and clarify that its average gives equal weight to the eight benchmark scores.

AuthorsOpenReview ↗

Response (2/n)

[W3] Scope of the empirical conclusions

Thank you for asking us to clarify how broadly these conclusions hold.

Model coverage. We extended the analysis to Qwen3.5-4B with three specialists. The first table shows score changes from the starting model in percentage points.

Training domainAIME25/26 avg.GPQAIFBenchBFCL v3
Math+25.97+9.68−0.19−0.92
Science+21.81+9.05+1.64+0.21
Instruction following+1.81+3.62+23.06−1.42

Math and science transferred in both directions, while instruction-following training gave the largest IFBench gain. Following Figure 3(d), we compared task-vector similarity with bidirectional transfer on AIME26, GPQA and IFBench.

Expert pairCosine between layer-16 updatesMean bidirectional transfer (pp)
Math–Science0.004390+15.0
Math–IF−0.000394+2.1
Science–IF0.001279+2.6

As on the original Qwen3 backbones, the math–science pair had both the most similar updates at layer 16 and the strongest bidirectional transfer. This experiment illustrates how the patterns of cross-domain transfer and task-vector alignment we observed on Qwen3 extend to other model families.

Choice of held-out tasks. We chose two tasks that test capabilities our five-domain RLVR training did not explicitly target. SimpleQA-Verified tests whether the model can recall facts without reference documents; AA-LCR asks it to combine evidence from several long documents. Prior work likewise uses general knowledge to assess retention after post-training [3] and long-context reasoning to assess transfer from math RL [4].

Grading reliability. We used Qwen3-32B to grade four responses per question against the reference answers using the prompts in Appendix D. The authors independently assessed the same 300 randomly selected examples. The judge agreed with the human verdict on at least 97% of those examples.

Geometry. We tested the robustness of the association in Figure 3(d): domains with more similar expert updates tend to show stronger cross-domain transfer. Because the ten domain pairs share the same five domains, we tested all 120 permutations of the domain labels, applying each to both backbones. At layer 16, 13 permutations gave a correlation at least as large in magnitude as the observed Spearman ρ=0.50\rho=0.50 (two-sided p=0.108p=0.108). The correlation remained positive after excluding the math–science pair (ρ=0.31\rho=0.31). These five domains provide evidence that experts with more similar weight updates make larger gains over the base model on each other’s tasks.

Teacher performance. Standard MOPD trains the student to match the domain teacher’s next-token distribution on the student’s own rollouts. At a prefix where the two distributions match, the KL loss is zero; the objective provides no separate reward for a more accurate answer. Consistent with this, MOPD’s eight-benchmark average is 0.6 points below the five-specialist average on both backbones (Table 1). For an analysis of how closely MOPD’s parameter updates align with those of its teachers, please see our response to Reviewer 2 (Q2).


[Q1] Training seeds and uncertainty in the main table

We appreciate your question about uncertainty in Table 1. Given the cost of RL training, we report single-run results. To reduce decoding variability, we average the scores of 32 responses per question and report the result as mean@32 in Table 1. To quantify uncertainty in the small score differences, we computed paired 95% bootstrap confidence intervals. The overall average weights all eight benchmarks equally. For this average, we report intervals for Merge versus Mix RL on 4B and for Mix RL versus MOPD on 8B. The per-question records for these models reproduce all eight benchmark scores in Table 1 to the reported precision.

ContrastDifference in pp [pointwise 95% CI]
4B Merge − Mix RL, eight-benchmark average+1.41 [0.70, 2.14]
8B Mix RL − MOPD, eight-benchmark average+1.37 [0.29, 2.48]
8B Mix RL − MOPD, AIME25+8.65 [4.06, 13.75]
8B Mix RL − MOPD, IFBench−4.89 [−8.36, −1.50]

All four intervals exclude zero. These checks support the pattern in Table 1 under mean@32: the gaps in average score are small, while on 8B, Mix RL leads on AIME25 and MOPD leads on IFBench.

AuthorsOpenReview ↗

Response (3/n)

[Q3a] Selection of the merge coefficient

Thank you for raising this important question. We selected λ\lambda using the Qwen3-8B benchmarks reported in Table 1; we did not use an independent validation set. In the Appendix E sweep (Figure 7), increasing λ\lambda from 0.3 to 1.0 improved accuracy but also increased mean response length from 2,042 to 7,334 tokens. At λ=0.6\lambda=0.6, we observed no reasoning traces in the AIME responses. At λ=1.0\lambda=1.0 with a 32k-token limit, about 66% of AIME responses included a reasoning trace despite the non-thinking template. We chose λ=0.6\lambda=0.6 to keep this shift into thinking mode from confounding the comparison.

For five experts, their arithmetic mean corresponds to λ=0.2\lambda=0.2 in λ∑iτi\lambda\sum_i\tau_i. At 0.6 the merged update norm is about 1.35 times the RMS expert norm on both backbones. Ilharco et al. [2] likewise use a shared coefficient and find its best value depends on the combination. Our specific value comes from our sweep.


[Q3b] Sensitivity to the Mix RL proportions

Thank you for raising this practical question about the training mixture. Wang et al. [1] pooled five domain subsets from the Nemotron 3 Nano RL training blend [5]. We used the resulting sample proportions on both backbones: approximately 25% math, 22% science, 22% code, 19% IF and 12% agent. In this ablation, we tested how a larger IF share affects Qwen3-4B’s performance across domains. Including each IF example twice raised that share from 18.9% to 31.8%. Scores below are percentages, shown as original / IF×2.

StepIFBenchGPQAAIME26Eight-benchmark average
40034.75 / 40.2562.75 / 62.3760.83 / 55.8360.73 / 61.54
45037.00 / 41.0860.73 / 60.7352.50 / 50.8360.68 / 59.14
47537.08 / 40.4261.11 / 0.1359.17 / 4.1761.22 / 33.62

At step 400, IF×2 led on IFBench by 5.50 points. Between steps 450 and 475, IF×2’s performance collapsed in other domains: its GPQA score fell from 60.73% to 0.13%, and its AIME26 score from 50.83% to 4.17%. By then, the IF×2 run had become unstable. The earlier gains on IFBench and later losses on math and science reinforce our conclusion that Mix RL requires careful data mixing across domains.


References

[1] Wang, Haoqing, et al. "To mix or to merge: Toward multi-domain reinforcement learning for large language models." arXiv preprint arXiv:2602.12566 (2026).

[2] Ilharco, Gabriel, et al. "Editing models with task arithmetic." arXiv preprint arXiv:2212.04089 (2022).

[3] Chen, Howard, et al. "Retaining by doing: The role of on-policy data in mitigating forgetting." arXiv preprint arXiv:2510.18874 (2025).

[4] Motwani, Sumeet Ramesh, et al. "h1: Bootstrapping llms to reason over longer horizons via reinforcement learning." arXiv preprint arXiv:2510.07312 (2025).

[5] Blakeman, Aaron, et al. "Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning." arXiv preprint arXiv:2512.20848 (2025).


Thank you again for your careful reading and detailed suggestions. We would be happy to discuss any remaining questions or concerns.

Reviewer_XLQ3OpenReview ↗

Thanks for your response.

Thank you for your response. I still find the results in the paper to be too low, which undermines their persuasiveness. Even for analytical experiments, the results are only convincing when the performance is sufficiently strong. Therefore, I will keep my score.

Reviewer_938fOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors compare three fusion methods(Merge, Mix RL, and MOPD) for integrating capabilities across multiple domains into a single model, under controlled settings spanning five domains and eight benchmarks.

Their analysis of domain experts reveals relatively high similarity in learned parameter geometry across reasoning tasks where Merge and Mix RL achieve performance comparable to or better than domain experts.

The comparison of the three methods suggests that Mix RL does not require domain experts but demands many training steps and careful data mixing, MOPD converges quickly but is bounded by its teachers, and Merge is cost-effective when domain experts are already available.

The authors also show that fusion neither expands the set of solvable problems nor degrades capabilities in other domains.

Critical strengths

The paper brings parameter merging, Mix RL, and MOPD together under a unified fusion framework and provides a multifaceted analysis, including computational costs, under controlled experimental conditions.

Critical weaknesses

1. The novelty of the analysis appears limited.

The synergy between reasoning tasks has already been investigated in prior studies ([1], [2]), and the cosine similarity analysis of weight shifts in domain-specific RLVR models also appears to have been conducted in [1].

The main addition over prior work seems to be the analysis of MOPD, but findings such as its faster convergence and teacher-bounded performance appear to be inferable from previous OPD studies ([3], [4]) or the OPD objective itself.

2. The rationale behind some of the hyperparameter choices is not provided.

Although most of the analysis relies on performance after training, the paper does not explain how hyperparameters such as the learning rate and the number of training steps were selected.

Clarifying whether the models were trained until convergence, whether these values were chosen for specific reasons, or whether they were determined through hyperparameter sweeps would help better support the paper's findings.

[1]: Wang et al. "To mix or to merge: Toward multi-domain reinforcement learning for large language models." arXiv preprint arXiv:2602.12566 (2026).

[2]: Chen et al. "Acereason-nemotron: Advancing math and code reasoning through reinforcement learning." Advances in neural information processing systems 38 (2025): 110320-110345.

[3]: Yang et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).

[4]: Lu, Kevin and Thinking Machines Lab. “On-Policy Distillation.” Thinking Machines Lab: Connectionism, Oct. 2025.

Questions for authors

1. Could you explain how the hyperparameters were selected?

The paper's main findings appear to be highly sensitive to the choice of hyperparameters, particularly the number of training steps.

2. The paper reports that MOPD exhibits different trends from Merge and Mix RL.

While Merge and Mix RL outperform domain experts on reasoning tasks but show degraded performance on IF, MOPD achieves somewhat lower reasoning performance and higher IF performance compared to the other two methods. Could you provide a more concrete explanation for these differences beyond the training objectives?

Since MOPD is trained to match the logits of domain experts, its parameters might also move in directions similar to those of the domain experts, although this is not guaranteed. If this is the case, cross-domain synergy might allow MOPD to exhibit trends similar to those of Merge and Mix RL. The fact that this is not observed suggests that other factors may be involved.

Could you analyze whether the parameter changes during MOPD training exhibit different patterns from those of the other two methods?

3. The experiments are conducted exclusively on the Qwen3 model family.

If time and computational resources permit, could you conduct additional experiments with other models, such as Qwen3.5, to further support the observed trends?

3 comments
AuthorsOpenReview ↗

Response (1/n)

Thank you for your thoughtful review. We particularly appreciate your recognition that our shared framework lets us compare both performance and computational cost across the three fusion paradigms. We respond to your comments and questions below.


[W1] Relationship to prior work

Thank you for highlighting these connections to earlier work. Our contribution is a controlled empirical comparison of three established RLVR fusion paradigms (Merge, Mix RL and MOPD). We study how they perform across domains, why their outcomes differ and what each costs. The closest precedent for this comparison is the work of Wang et al. [1], who study three paradigms in five domains and analyze weight-shift cosines. Chen et al. [2] report math/code transfer, and prior OPD work [3] explains the value of dense teacher feedback. Building on these findings, we test which domain-level trade-offs persist across two instruction-tuned checkpoints and whether the pattern for Merge changes when we use LoRA experts in place of full-parameter experts.

These comparisons matter because cross-domain transfer and similarities between expert updates do not, by themselves, tell us which specialist gains a single fused model will retain (Section 4.2). To interpret differences between Mix RL and a specialist, we also count prompts by domain: the same number of training steps need not give the two models equal exposure to that domain (Section 5.2). At evaluation time, varying the number of attempts per problem lets us test whether higher single-sample accuracy also translates into solving a broader set of problems (Section 5.3). Finally, to determine when faster distillation also saves compute, we distinguish the cost of fusion from the total cost when the teachers must first be trained (Section 5.2).


[W2, Q1] Hyperparameter selection and convergence

We appreciate your question about the training settings. We used a learning rate of 10−610^{-6} for all full-parameter runs. This rate is commonly used in RLVR settings [2], [4]. Our preliminary experience suggests that higher learning rates can destabilize training, with sharp drops in policy entropy and spikes in response length as the model repeats itself.

We chose the reported step counts based on practical convergence: we used the training and validation curves to judge whether further gains justified the cost of additional updates. For example, by step 800 of Mix RL on Qwen3-4B, the training verifier score was improving more slowly and the validation curve had begun to level off. We therefore used the 800-step checkpoint for the main comparison. We further extended the same run from 800 to 1,200 updates, but average accuracy across the eight benchmarks rose by less than one percentage point. For MOPD on the same backbone, average accuracy was essentially unchanged from step 300 to step 400, supporting our choice of the 300-step checkpoint.

Qwen3-4B methodMain checkpointLater checkpointChange (percentage points)
Mix RL62.27% (800 updates)63.17% (1,200 updates)+0.90
MOPD63.48% (300 updates)63.36% (400 updates)−0.12

The appendices explain several other hyperparameter choices. These include the LoRA learning rates (Appendix B) and the task arithmetic coefficient λ\lambda (Appendix E).

AuthorsOpenReview ↗

Response (2/n)

[Q2] Do parameter changes differ across fusion paradigms?

Thank you for this helpful suggestion. MOPD may imitate the experts’ predictions while also changing its weights in directions similar to theirs. We tested whether this alignment could help explain why MOPD retains more of the IF expert’s gains while Mix RL performs better on some math benchmarks. Merge provides a reference: its weight change is a linear combination of the five expert updates. For MOPD and Mix RL, we measured how much of each model’s weight change from the starting checkpoint could be reconstructed from those same five directions.

On Qwen3-8B, both the reconstructed share and the cosine similarity with the IF expert were higher for MOPD than for Mix RL at layers 4, 16 and 28. We show layer 16 below.

MeasureMerge (TA)Mix RLMOPD
Reconstructed share of squared update norm100%5.7%29.1%
Cosine similarity with the IF expert’s update0.6260.1580.430
IFBench accuracy (Table 1)37.0%37.9%42.8%

Merge aligns most closely with the IF expert at layer 16, yet MOPD scores highest on IFBench. We further compared each fused model’s next-token predictions with those of the corresponding domain expert.

To quantify how closely these predictions matched, we computed the full-vocabulary KL divergence DKL(πmodel∥πdomain expert)D_{\mathrm{KL}}(\pi_{\mathrm{model}}\Vert\pi_{\mathrm{domain\ expert}}) on identical prefixes from an 8B base response, averaging over up to the first 32 prediction positions per question. We used all 60 AIME25/26 questions and 60 randomly selected IFBench questions. Lower KL (in nats) indicates closer agreement with the expert’s predictions.

Mean KLMergeMix RLMOPD
AIME, math teacher0.003220.012790.00063
IFBench, IF teacher0.310610.320410.03319

MOPD stays closest to both routed teachers on these prefixes. It matches the IF teacher's 42.8 on IFBench. Mix RL departs more from the math teacher and scores 45.7 on AIME25, above the teacher's 39.2 and MOPD's 37.1. This directly measures imitation. Attributing accuracy changes to particular parameter directions would require a causal intervention, which we leave to future work.


[Q3] Transfer on Qwen3.5-4B

Thank you for suggesting a test on a newer backbone. We extended the Section 5.1 analysis to Qwen3.5-4B with three specialists. The first table shows changes in percentage points relative to the starting model.

Training domainAIME25/26 avg.GPQAIFBenchBFCL v3
Math+25.97+9.68−0.19−0.92
Science+21.81+9.05+1.64+0.21
Instruction following+1.81+3.62+23.06−1.42

Math and science transferred in both directions, while instruction-following training gave the largest IFBench gain. Following Figure 3(d), we compared task-vector similarity with bidirectional transfer on AIME26, GPQA and IFBench.

Expert pairCosine between layer-16 updatesMean bidirectional transfer (pp)
Math–Science0.004390+15.0
Math–IF−0.000394+2.1
Science–IF0.001279+2.6

As on the original Qwen3 backbones, the math–science pair had both the most similar updates at layer 16 and the strongest bidirectional transfer. This experiment illustrates how the patterns of cross-domain transfer and task-vector alignment we observed on Qwen3 extend to other model families.


References

[1] Wang, Haoqing, et al. "To mix or to merge: Toward multi-domain reinforcement learning for large language models." arXiv preprint arXiv:2602.12566 (2026).

[2] Chen, Yang, et al. "Acereason-nemotron: Advancing math and code reasoning through reinforcement learning." Advances in neural information processing systems 38 (2026): 110320-110345.

[3] Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation.

[4] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2026): 113222-113244.


Thank you again for your thoughtful review. We welcome any further questions or concerns.

Thank you for the detailed response. After considering the rebuttal, I will maintain my original score.

Reviewer_HJBaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a controlled empirical comparison of three paradigms for consolidating domain-specific RLVR capabilities into a single model: Merge, which combines expert task vectors; Mix RL, which jointly trains on mixed-domain data; and MOPD, which distills multiple domain experts into one student.

Critical strengths
  1. The controlled comparison is useful.

  2. The problem is well-motivated.

Critical weaknesses

The positioning and intended contribution of the paper are unclear. It is difficult to tell whether the paper should primarily be evaluated as a new methodological contribution or as a systematic empirical study of multi-domain RLVR consolidation. In my view, the contribution appears limited under either interpretation:

If this is primarily an empirical study:

  1. The domain coverage is limited compared with existing cross-domain RLVR studies. The paper considers Math, Science, Code, Instruction Following, and Agent tasks. Prior work such as Cheng et al. [1] evaluates a substantially more fine-grained reasoning taxonomy, including mathematics, coding, science, logic, simulation, and tabular reasoning.

  2. The model coverage is also too limited for a general empirical study. All main experiments are conducted on only Qwen3-4B and Qwen3-8B. This seems particularly insufficient given prior work such as Hu et al. [2], which studies RLVR transfer across a broader range of model scales/settings.

If this is primarily a method paper:

  1. There is essentially no new algorithmic method.

  2. The empirical results do not reveal a clear methodological advance over existing approaches. The three paradigms differ by at most 1.4 points in average performance, and none is consistently superior.

[1] Cheng, Jorge Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective." Advances in Neural Information Processing Systems 38 (2025).

[2] Hu, Chuxuan, et al. "Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. I highly recommend authors to make Table 1 less confusing and more straightforward of what you are studying and what your contributions are.

  2. Can you explain how you selected the domains? Agent tasks seem very off to me.

3 comments
AuthorsOpenReview ↗

Response (1/n)

Thank you for your careful reading and constructive feedback. We appreciate your comments on the motivation for this study and the value of comparing the three fusion paradigms under the same conditions. We address your concerns point by point below.


[W1] Empirical contribution and relation to prior work

Thank you for asking us to clarify our contribution. Our contribution is a controlled empirical comparison of three established RLVR fusion paradigms (Merge, Mix RL and MOPD). Using common backbones, target domains and benchmarks, we study how the three approaches perform, why their outcomes differ and what each costs. By comparison, Cheng et al. [1] study transfer across reasoning domains, and Hu et al. [2] examine whether RLVR gains generalize to unseen domains. In Section 5.1, we analyze cross-domain transfer to help explain the domain-level differences we observe among the fusion paradigms. To assess fusion itself, we compare each fused model with the corresponding domain specialists (Section 4.2). In Section 5.2, we track performance against the number of training prompts sampled from each domain and report fusion costs separately from the cost of training any required experts. In Section 5.3, we test whether gains in single-sample accuracy translate into solving more distinct problems when the model is given multiple attempts.


[Q1] Reading Table 1

Thank you for this suggestion. We designed Table 1 to show how Merge, Mix RL and MOPD perform on each of the eight benchmarks, with the base model and the corresponding domain specialist as references. To make these comparisons easier to follow, we will give the reference models and fusion methods separate headings and show how many models each row represents, as illustrated below for the full-parameter setting.

Model or referenceModels represented4B average8B average
References
Base model157.042.0
In-domain specialists (one per domain)563.953.8
Single-model fusion methods
Merge, full-parameter task arithmetic163.753.7
Mix RL162.354.6
MOPD163.353.2

In the full table, each benchmark entry in the specialist row comes from the expert trained for that domain. Each fusion row reports one model evaluated on all eight benchmarks. We will label the Merge variants by their source experts (full-parameter or LoRA).


[W2, Q2] Domain selection and the role of Agent

Thank you for asking how agent tasks fit into our study. We chose these five domains to reflect the range of capabilities targeted in the post-training of general-purpose LLMs [3]. In addition, each domain provides verifiable feedback for training specialists and a corresponding benchmark for comparing the fusion paradigms. Specifically, math, science and code tasks test reasoning through answer checks or code execution. Instruction following tests satisfaction of explicit constraints. Agent adds multi-turn tool use, where success depends on the effect of a sequence of actions in an environment. The Workplace Assistant training task and BFCL v3 evaluation make this distinction measurable. We include instruction following and agentic tool use alongside reasoning tasks to compare how the three fusion paradigms combine these capabilities in one model.

AuthorsOpenReview ↗

Response (2/n)

[W3] Model coverage

We appreciate your concern about model coverage. We extended the Section 5.1 analysis to Qwen3.5-4B with three specialists. The first table shows changes in scores from their starting model (pp).

Training domainAIME25/26 avg.GPQAIFBenchBFCL v3
Math+25.97+9.68−0.19−0.92
Science+21.81+9.05+1.64+0.21
Instruction following+1.81+3.62+23.06−1.42

Math and science transferred in both directions, while instruction-following training gave the largest IFBench gain. Following Figure 3(d), we compared task-vector similarity with bidirectional transfer on AIME26, GPQA and IFBench.

Expert pairCosine between layer-16 updatesMean bidirectional transfer (pp)
Math–Science0.004390+15.0
Math–IF−0.000394+2.1
Science–IF0.001279+2.6

Math and science transferred most strongly to each other. Overall, this experiment illustrates how the patterns of cross-domain transfer and task-vector alignment we observed on Qwen3 extend to other model families.


References

[1] Cheng, Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective, 2025." URL https://arxiv. org/abs/2506.14965.

[2] Hu, Chuxuan, et al. "Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Blakeman, Aaron, et al. "Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning." arXiv preprint arXiv:2512.20848 (2025).


Thank you again for your careful review and thoughtful suggestions. We would be happy to address any further questions or concerns.

As an empirical paper (as the authors explained), I think the model coverage is still too small. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.