acceptodds
Under review as a conference paper at ICLR 2027

Policy-Merge: Policy Prediction for Multi-Domain RL and Distillation

Abstract

Language models are post-trained with Reinforcement Learning (RL) across many domains, such as math, code, and tool use. The resulting model depends on how data and compute are allocated across these domains, whether by training on a data mixture or by multi-teacher distillation. Evaluating an allocation, however, requires a full training run. Existing mixture selection methods reduce this cost by predicting benchmark performance or using proxy models for mixture optimization, but do not directly predict the policy obtained by training on a mixture. We introduce a method that approximates the policy of a model trained with RL on a given data mixture by composing independently post-trained domain experts at decoding time. We compare this approximated policy with the model trained with RL on the mixture, and show that they perform similarly across benchmarks and are distributionally close. This allows for predicting the capabilities of a specified mixture. Our method enables efficient data mixture selection, as candidate mixtures can be evaluated and ranked in policy space before committing compute to post-training. Finally, we show that a student distilled on-policy from several experts at once to minimize a weighted sum of per-token KL divergences is equivalent to using our approximated policy as a teacher. One can thus skip the multi-teacher distillation and decode the approximated policy directly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-domain RL post-training, data mixture optimization, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fHjdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether we can predict the policy produced by multi-domain RL post-training without training a model for every candidate data mixture. Policy-Merge composes the next-token log-probability shifts of independently trained domain experts relative to a shared reference model. It proposes mapping data proportions to composition coefficients using each expert’s KL divergence from the reference. The resulting policy can be decoded to estimate mixture behavior and guide mixture selection. The paper also establishes a fixed-prefix equivalence between weighted teacher-specific reverse-KL losses and distillation from a composed teacher. Validation includes Qwen3-0.6B experts and eight trained mixtures across six Reasoning Gym domains, comparisons in accuracy and token-level KL, a MATH-500 mixture-selection experiment, and distillation experiments. Additional evaluations use released Olmo 3 7B models and Open-MOPD teachers and students.

Critical strengths
  1. The paper provides meaningful empirical evidence for policy-level compositionality across different domains.

  2. The proposed method is practically useful, and evaluations on different models further support its effectiveness.

  3. The paper explores an interesting direction for inference-time expert merging and serving without additional training.

Critical weaknesses
  1. In Figure 9, there appear to be two gray vertical lines in the lower row of each benchmark. It is unclear why both lines are present and what they represent. The authors should clarify this visualization or simplify the figure to avoid confusion.

  2. The mixture-selection experiment in Section 4.2 uses the original data proportions rather than the proposed KL-adjusted coefficients. It remains unclear how much the proposed weighting strategy contributes to the practical mixture-selection task.

  3. Although the paper reports a low average prediction error, Figure 5 shows relatively large discrepancies in certain domains. For example, the predicted accuracy on Graphs is 34.0%, compared to 43.7% for the actual trained model. This raises concerns about the reliability of Policy-Merge in predicting domain-specific performance.

Questions for authors
  1. Please clarify the key differences between the proposed Policy-Merge method and DERA (https://arxiv.org/pdf/2402.02992), particularly in terms of their underlying methodologies and technical novelty.
  2. Why is α=w\alpha = w used for mixture selection, while the original drift-adjusted rule is used for the main prediction experiments and a normalized version is used for Olmo 3? Were these choices fixed independently of the target mixture’s observed performance? How do the original Equation (5), unadjusted proportions, and normalized drift weights compare in these settings?
5 comments
AuthorsOpenReview ↗

Response to Reviewer fHjd

Thanks for your thorough review and suggestions. We are glad to hear that you found the method practically useful and an interesting direction for inference-time expert merging and serving without additional training.

W1. Figure 9.

In Figure 9, there appear to be two gray vertical lines in the lower row of each benchmark. It is unclear why both lines are present and what they represent. The authors should clarify this visualization or simplify the figure to avoid confusion.

Thanks for pointing out how this figure could be improved. In the original figure, the gray line in the lower row shows the accuracy of the distilled student in the row above, and a second, faded line shows the accuracy of Policy-Merge. These lines were intended to show how Policy-Merge is close to the student on every benchmark but we see that the figure could be simplified. We have updated the figure.


W2. Selection with the KL scaled coefficients.

The mixture-selection experiment in Section 4.2 uses the original data proportions rather than the proposed KL-adjusted coefficients. It remains unclear how much the proposed weighting strategy contributes to the practical mixture-selection task.

We added selection with w~\tilde{\mathbf w} and compare it with the initial selection at w\mathbf w. The KL scaled coefficients improve prediction of the trained policy in our main experiments, but they do not improve selection on MATH-500.

  • MATH-500. With w~\tilde{\mathbf w}, Policy-Merge selects a different mixture, mostly Graphs and Cognition. The trained model reaches 68.9 ±\pm 1.1 on the held out problems, against 70.6 ±\pm 0.8 for the mixture selected with w\mathbf w. Both mixtures improve on the base model, but the original selection performs better here. We report both results in Section 4.2.
  • Choosing among trained mixtures. When Policy-Merge chooses among the six trained 0.6B mixtures for 22 targets, w~\tilde{\mathbf w} chooses the best mixture in 63.5% of cases, against 58.6% with w\mathbf w (new Appendix E.6).
  • Additional results on larger models. In response to reviewer zXwq, we scaled up our experiments and trained 4B models on different tasks. We also use w~\tilde{\mathbf w} to select for a new target across domains, IFBench with HarmBench (Figure 6, right).

W3. The Graphs domain in Figure 5.

Although the paper reports a low average prediction error, Figure 5 shows relatively large discrepancies in certain domains. For example, the predicted accuracy on Graphs is 34.0%, compared to 43.7% for the actual trained model. This raises concerns about the reliability of Policy-Merge in predicting domain-specific performance.

We agree that Graphs results has larger prediction error in comparison to other domains. Of the 9 results in Figure 5 with an error above 5 points, 6 are on Graphs, including the case you point out. One possible explanation is that the drift adjustment underweights experts with low drift. However, low drift alone does not explain this error. Cognition has lower KL drift than Graphs, yet its accuracy in the six-way mixture is predicted accurately.

We would also like to point out that Policy-Merge remains reliable on the other domains. On the other five domains, the mean absolute error is 1.95 points, and 37 of 40 results are within 5 points of the trained model. Across all 48 domain results in Figure 5, 35 are within 3 points and 39 are within 5 points, with a mean absolute error of 2.61 points. Policy-Merge is also robust to capturing large gains on individual domains. After training, the four-way mixture reaches 56.5% on Geometry. Policy-Merge predicts 54.0%, while interpolating the experts' scores predicts only 38.4%.

To examine domain-specific reliability further, we trained Qwen3-4B experts on six domains: math, code, science, instruction following, tool use and safety with full fine-tuning. We then trained nine mixtures of these experts. Training on these mixtures improves over the base model by up to 9.2 points on BFCL multi-turn (6.6 on average) and 5.7 points on AIME 2024–2026 (3.9 on average). With no training, Policy-Merge predicts 34 of the 36 results on LiveCodeBench, IFBench, BFCL multi-turn and AIME within 5 points of the trained model, with a mean error of 1.75 points, against 2.17 for weight merging (Figure 12, Appendix E.4). The largest errors are on BFCL multi-turn, where training on the mixture gains more than either prediction. Policy-Merge makes a more accurate prediction, with a mean error of 3.09 points against 4.12 for weight merging.

These additional results show that accurate domain specific prediction extends to larger models and more challenging benchmarks, although accuracy is not uniform across domains.

Q1. Differences from DeRa.

Please clarify the key differences between the proposed Policy-Merge method and DERA, particularly in terms of their underlying methodologies and technical novelty.

DeRa proposes a method for interpolating between a reference model and a post-trained policy with a hyperparameter λ\lambda. The paper also derives the product-of-experts sampling using multiple models, for a scenario where the reward of the target model is a weighted combination of rewards ∑kλkrk\sum_k \lambda_k r_k. We note that the sampling strategies are similar, but DeRa focuses on predicting the policy trained on a sum of rewards, and the coefficients λk\lambda_k are given by a user, for how much each reward should be accounted for. Specifically, Policy-Merge is different in two key ways:

  • Predicting the policy trained on a data mixture. DeRa predicts a policy trained on the same prompts with a different KL strength or reward weighting. Policy-Merge predicts RL on a data mixture from experts trained on very different domains, such as math, code, science, instruction following and safety. We show empirically that composing the experts predicts it, even if they are trained on different prompt sets.
  • Setting the coefficients of composition. DeRa's coefficients are the given reward weights or KL strength. Policy-Merge derives them from the data proportions and each expert's KL drift from the base model (Eq. 5), without the trained mixture.

Beyond these two differences, the paper makes two contributions that DeRa does not address.

  • Experts trained on different prompt sets compose. To our knowledge, we show the first empirical evidence that the policy trained on a data mixture is a linear composition of the policies of experts trained on each dataset alone. With the coefficients of Eq. 5, the composition accounts for 85% to 95% of how far each of the six trained mixtures moves from the base model (Figure 2).
  • Predicting the student of multi-teacher on-policy distillation. Distillation from experts weighted by the data proportions has the same per-prefix gradient as distillation from the single composed policy (Appendix F). Decoding the composed policy also predicts students distilled with routed teachers, where each prompt is routed to the expert of its domain, within 1.4 points of the 4B students (Figure 9) and within 0.2 points of the student released by Open-MOPD.

Q2. Coefficients in each setting.

Why is [w\mathbf w] used for mixture selection, while the original drift-adjusted rule is used for the main prediction experiments and a normalized version is used for Olmo 3? Were these choices fixed independently of the target mixture's observed performance? How do the original Equation (5), unadjusted proportions, and normalized drift weights compare in these settings?

We apologize for this inconsistency. We first ran the mixture selection experiment before adding the drift adjustment, and did not rerun it before submission. We have now rerun it with w~\tilde{\mathbf w} (Section 4.2). For Olmo 3, we normalized the coefficients of Eq. 5 to sum to one before comparing with RL-Zero Mix, because the overall scale of w~\tilde{\mathbf w} assumes the experts and the mixture share a training recipe, which we could not verify for released models, so we kept only the relative weights. The updated paper uses the unnormalized coefficients, as in every other experiment for consistency. Normalization did not help, it is 1.0 point further from RL-Zero Mix on MATH-500 and equal on MBPP+, although both differences are within one standard error. Initially, to predict the 0.6B and Open-MOPD distillation students, we composed the experts at the data proportions w\mathbf w, so that the teacher weights are positive and sum to one, as in standard multi-teacher distillation (Section 2.4) however we show that OPD can also empirically apply w~\tilde{\mathbf w} by running additional experiments for 4B multi-teacher OPD using Eq. 5, where Policy-Merge is within 1.4 points of the distilled students on average (Figure 9, top). To answer your question directly, none of these choices were made after observing the trained mixture’s performance. We hope that our update to using Eq. 5/w~\tilde{\mathbf w} in every prediction and selection experiment removes this inconsistency.

Olmo 3, error against RL-Zero Mix (points)MATH-500MBPP+
Data proportions w\mathbf w4.04.7
Eq. 5 w~\tilde{\mathbf w}0.51.1
Normalized w~\tilde{\mathbf w}1.51.1

Thank you again for the careful review. We hope the revised paper and these answers address your questions, and we are happy to clarify anything else.

Thank you again for your careful review and helpful suggestions. Have we addressed your concerns in our response or is there anything else we could clarify?

Reviewer_mKbFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Post-training across multiple domains is challenging because choosing and testing allocations across domains can be expensive. Distilling from multiple domain experts still leaves the similar question. Building on prior work on the relation between policy and log ratios, the authors propose Policy-Merge, which combines already trained experts in logit space to estimate the policy obtained by RL on a specified data mixture. They compare these predictions against models actually trained on the mixtures, explore using them to select allocations more cheaply, and connect the composition to multi-teacher distillation with fixed teacher weights.

Critical strengths
  1. The method provides a useful, cheaper way to estimate allocation outcomes with domain experts. This can reduce the number of full training runs needed to explore candidate mixtures.

  2. The comparisons against models actually trained on the corresponding mixtures provide sound evidence.

Critical weaknesses
  1. My main concern is limited theoretical novelty. The underlying idea appears closely connected to existing work on reward optimization, KL objectives, and policy composition, and the connection to multi-teacher distillation also seems largely familiar. I see the main contribution as applying and empirically validating these ideas for predicting data-mixture training. This application is useful, but I am less convinced that the paper introduces a substantially new principle or theoretical insight.
  2. The additional contribution beyond existing composition ideas could be clearer. Since the experts share a base model, some success may reflect a local approximation around it. Although the paper includes composition at the data proportions and weight merging, I wonder whether simply rescaling these baseline numbers, without access to the target mixture, would recover much of the prediction. The added value of composing full policies for predicting benchmark scores and selecting mixtures is not fully isolated. A simple linear relationship between mixture proportions and evaluation scores might explain some of the same trends. A comparison with such a baseline would clarify the practical advantage for this application. This concern is separate from the evidence of distributional similarity.
Questions for authors
  1. Could you clarify what is conceptually new beyond existing reward–policy composition and KL-based distillation relationships, and what should be understood primarily as an empirical extension to predicting data-mixture training?

  2. How would benchmark-score prediction and mixture ranking compare with a simple linear regression of evaluation scores on mixture proportions, using comparable calibration information?

6 comments
AuthorsOpenReview ↗

Response to Reviewer mKbF

Thank you for your thoughtful review and suggestions. We are glad you found Policy-Merge a useful way to estimate the outcome of different domain allocations, and to reduce the training needed to assess candidate mixtures.

W1 / Q1. What is new beyond existing composition and distillation results?

Could you clarify what is conceptually new beyond existing reward–policy composition and KL-based distillation relationships, and what should be understood primarily as an empirical extension to predicting data-mixture training?

We agree that the sampling method builds on known results in multi-objective RL, and we do not claim them as theoretical contributions. However, composing policies trained on different data distributions (or domains) required additional empirical experiments, and adjustments to existing product-of-experts sampling methods. We describe these challenges and differences in detail below:

  1. Predicting the policy trained on a data mixture. Prior composition predicts the policy trained with RL on a new weighted combination of rewards on the same prompts (PoEM, MOD, DeRa). Policy-Merge instead composes experts trained on different prompt sets. To our knowledge, we show the first empirical evidence that the policy trained on a data mixture can be linearly composed from the policies of experts trained on each dataset (Figure 2), and at 4B, Policy-Merge predicts mixtures of datasets as different as math, code, instruction following, tool use and safety, whose objectives can be at odds with each other (Appendix E.4). We thus show the first evidence that the policy trained on a data mixture can be predicted, which prior methods use to select mixtures, or by composing the policies.

  2. We find better coefficients for composition. Experts trained with the same recipe drift from the base model by very different amounts in terms of KL divergence (Figure 3), so we found that data proportions are not optimal coefficients for policy composition. Policy-Merge scales each proportion by its expert's divergenece from the reference model, which closes the gap between data mixing weights and optimal policy composition weights. Prior methods use the reward weights exactly (MOD), or scale all of them by one shared scaling parameter (PoEM). All three methods sample from the same composition, πα∝πref1−∑kαk∏kπkαk\pi_{\boldsymbol\alpha} \propto \pi_{\text{ref}}^{1 - \sum_k \alpha_k} \prod_k \pi_k^{\alpha_k} (Eq. 3), and only differ in the coefficients: αk=wk\alpha_k = w_k for MOD (where wkw_k is reward mixing weights), αk=γwk\alpha_k = \gamma w_k with a shared γ\gamma for PoEM (where wkw_k is reward mixing weights), and αk=w~k=wkKLk/∏jKLjwj\alpha_k = \tilde{w}_k = w_k \mathrm{KL}_k / \prod_j \mathrm{KL}_j^{w_j} for Policy-Merge, where KLk\mathrm{KL}_k is the KL of expert kk from the base model, and wkw_k is data mixing weights (Eq. 4). Our scaling reduces the mean error from 1.34 to 0.93 points, and the composition is 13.4× closer to the trained mixture in KL than the base model, against 7.2× with the data proportions (Tables 3 and 1).

  • We agree that the paper provided little explanation of why these coefficients work so well empirically, so we studied this further and found theoretical motivation for them (new Appendix B.3). For one step of gradient descent, the update of the mixture is the weighted average of the updates of its experts, so the data proportions are the right coefficients. Our training, like other RL methods, uses Adam, which normalizes the gradient of each parameter by a running estimate of its scale. The mixture normalizes the gradients of all domains by one shared scale, so a domain with larger gradients than the mixture contributes more than its data proportion, and a domain with smaller gradients contributes less. The gradients of the mixture are only known by training it, so Policy-Merge uses the drift of each expert instead of the scale of its gradients, and follows the same pattern.
  1. Predicting the student of multi-teacher on-policy distillation. A central contribution of our work is to connect policy composition to multi-teacher on-policy distillation. Distillation from experts weighted by the data proportions has the same per-prefix gradient as distillation from the single composed policy (Appendix F). Although this identity is a simple derivation, to our knowledge no prior work has connected it to predicting the student of on-policy distillation before training, which is otherwise known only after distillation. Decoding the composed policy is within 1.4 points of the students distilled from the 4B experts (Figure 9), and within 0.2 points of the student released by Open-MOPD.

Predicting the policy also allows uses beyond selecting mixtures, such as decoding from a data mixture without training it, which we describe in our response to Reviewer zXwq.

Q2 / W2. Does a linear model explain the same trends?

I wonder whether simply rescaling these baseline numbers, without access to the target mixture, would recover much of the prediction. [...] This concern is separate from the evidence of distributional similarity.

Thank you for this suggestion. We first make a distinction between our method and the suggested baselines, and then provide additional results.

Throughout the paper, we mainly report average accuracies to compare Policy-Merge distribution πPM(α)\pi_{PM(\alpha)} to the policy trained on the data mixture πw\pi_{\mathbf{w}}. Specifically, accuracy estimates the probability of a correct answer on prompt xx is Ey∼πw(⋅∣x)[1(y is correct)]\mathbb E_{y \sim \pi_{\mathbf{w}}(\cdot \mid x)} [ \mathbf{1}(y \text{ is correct}) ]. Policy-Merge, however, attempts to approximates πw(⋅∣x)\pi_{\mathbf w}(\cdot\mid x) for a given prompt xx, which is the trained mixture's distribution over responses to each prompt. We make this distinction explicit in Section 4.1 and evaluate accuracy and next-token KL separately (Tables 3 and 1). We have also included some qualitative results to Appendix G, which compares paired responses on the same prompts.

That being said, the reviewer proposes an excellent baseline to add to our results, and we have added three linear baselines (Appendix A.1, Figure 10):

  • Interpolation, ∑kwkak\sum_k w_k a_k, where aka_k is the score of expert kk.
  • Scaled interpolation, a0+∑kw~k(ak−a0)a_0 + \sum_k \tilde{w}_k (a_k - a_0), where a0a_0 is the score of the base model: the rescaling of the baseline numbers you describe, with Policy-Merge's coefficients.
  • Regression on trained mixtures, fit on the other 7 trained mixtures and leaving out the one it predicts. It uses information Policy-Merge does not have, and each of its data points costs a full RL run. We report the range over four variants.

The first two use the same information as Policy-Merge: the base model and the experts.

Against the 8 trained mixturesMean error (points)Spearman ρ\rhoKendall τ\tau
Policy-Merge (w~\tilde{\mathbf w})0.930.830.64
Policy-Merge (w\mathbf w)1.340.830.64
Scaled interpolation1.410.710.50
Weight merging2.130.690.50
Interpolation2.110.550.29
Regression on 7 trained mixtures1.42 to 1.60up to 0.57—

Composing policies captures the synergy of training on domains together, which linear models do not. On the four- and six-way mixtures, the trained model is 3.8 and 4.2 points above interpolation, while Policy-Merge is only 0.7 and 1.3 points below the trained model.

Choosing the best mixture. We also add the linear baselines to the selection study in our response to Reviewer zXwq (new Appendix E.6). Each method chooses the best of six trained 0.6B mixtures for each of 22 targets, from a random half of the problems, and we score the chosen mixture on the other half, over 1,000 random splits. We measure the performance of the best - chosen mixture on held-out problems below in terms of accuracy, and average over different tasks.

Methoddifference between best and chosen mixture, in points (middle 95% of splits)How often the chosen mixture is the best
Policy-Merge (w~\tilde{\mathbf w})0.57 [0.18, 1.11]63.5%
Policy-Merge (w\mathbf w)0.61 [0.20, 1.27]58.6%
Scaled interpolation0.57 [0.24, 1.02]61.6%
Weight merging0.76 [0.25, 1.49]54.8%
Interpolation1.01 [0.51, 1.58]50.0%

Policy-Merge chooses the best mixture most often. For the base model and five of the six experts, we only have the accuracy on all the problems, and not on each problem, so the linear baselines choose using all the problems, including the half on which we score the chosen mixture.

The linear composition also holds as the mixture drifts further from the base model during training. A local approximation is most accurate close to the base model, and less accurate further from it. We do not see this. As we train the six-way mixture, the KL of the mixture from the base model increases from 0.0003 to 0.10 nats per token, and the R2R^2 of the linear fit of Figure 2 increases from 0.41 to 0.93 (Appendix E.5). In the final mixture, the R2R^2 is as high on the tokens that drift the most from the base model as on all tokens (0.92 against 0.91). Policy-Merge also predicts mixtures of full-parameter experts, at 4B and for Olmo 3 at 7B (Appendix E.4 and Figure 8).


Thank you again for the detailed review. Your suggestions led to new experiments and results that we believe make the paper considerably stronger. We hope they address your concerns and we are happy to discuss further.

Thank you again for your insightful feedback and helpful suggestions. Have we addressed your concerns in our response or is there anything else we could clarify?

Reviewer_zXwqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Policy-Merge, which approximates the policy resulting from multi-domain RL training by composing the logits of independently trained domain experts. The method adjusts composition coefficients using each expert’s KL divergence from the shared reference model. The paper evaluates prediction accuracy through downstream performance and distributional similarity, and applies policy composition to data-mixture selection. It also establishes a fixed-prefix gradient equivalence between weighted multi-teacher reverse-KL distillation and distillation from a composed teacher, supported by experiments comparing compositions with distilled students.

Critical strengths

Evaluation across complementary settings. The paper conducts controlled experiments across six reasoning domains and extends its evaluation to externally trained experts covering mathematics, code, and instruction following. The inclusion of larger, fully fine-tuned models provides evidence that the observed behavior is not restricted to the authors’ small-model LoRA setup.

Useful empirical prediction of distillation outcomes. The composed policies approximate the downstream performance of multi-teacher on-policy distillation students in the evaluated settings. This suggests a potentially useful way to preview student capabilities before conducting a full distillation run. The fixed-prefix reverse-KL identity provides an additional theoretical connection between policy composition and multi-teacher distillation.

Critical weaknesses

Limited evidence of practical mixture-selection benefits. The main controlled experiments use a small model and a carefully constructed collection of domain experts sharing the same initialization and training recipe. Although the external experiments broaden the evaluation, the actual mixture-selection application is demonstrated on only one target benchmark. Its result, 70.6% on the held-out MATH-500 split versus 70.1% for MergeMix, does not establish a clear advantage given the reported variability. Broader mixture-selection experiments are needed to demonstrate practical benefits beyond accurately approximating trained policies.

Incomplete end-to-end computational comparison. Evaluating the 40 candidate mixtures requires forward passes from multiple participating experts at each decoding step, increasing the cost per generated token. Table 2 reports that six-expert decoding takes 2.45× as long as single-model decoding in the measured setting. Although Appendix E.3 provides expert-training costs and estimated token-count savings, it does not report the measured GPU-hours of the complete 40-mixture search or an end-to-end comparison with MergeMix and OP-Mix. Accounting for expert preparation, candidate evaluation, and final training would better establish the method’s practical efficiency advantage.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer zXwq

Thank you for the detailed feedback. We are happy that you found our evaluation across complementary settings strong, and that it is useful to predict what a model will learn from a mixture before any training. Below we provide a list of the primary reviewer concerns and a summary of our response to each.

W1. Scale of the experiments, and additional benchmarks for data selection

We agree that the current set of experiments in the paper mainly focuses on smaller scale models, so we scale up our experiments to 4B models and more challenging environments. We also emphasize that we view mixture selection as one downstream application of policy composition, rather than as the paper’s central contribution. We have made small changes to the writing to make this clearer. The abstract now emphasizes ranking data mixtures rather than selecting them, and the contributions in the introduction now emphasize scoring candidate mixtures rather than selecting a mixture. Our primary contribution is predicting the policy that post-training will produce, before any training, both for RL on a mixture of datasets and for on-policy distillation from routed experts. To our knowledge, no prior work, shows that the next-token distribution of a model trained on a data mixture can be predicted from models trained on a single domain.

We note that baselines such as MergeMix and OP-Mix only predict loss or rank mixtures by benchmark scores, and were proposed for pre- and mid-training stages. We significantly modified both methods as baselines (Appendix E.2), and we apologize if this was not clear in the paper. Additionally, compute allocation across datasets is much less explored for RL post-training than for pre-training, and to our knowledge, we do the first such allocation without training candidate mixtures during the selection process. We agree that in our current mixture selection setup, Policy-Merge performs on par with the baselines rather than better. It uses the same candidates, regression fit and simplex search as MergeMix, and since the regression is performed on a small sample size (where each sample is the mixture and its predicted performance), the mixtures they select end up performing similarly to each other, at 0.6B and at 4B. However, given a set of choices for mixtures, Policy-Merge chooses the best mixture in 63.5% of cases, against 54.8% for weight merging (new Appendix E.6). We provide more detailed responses and new results on the larger scale experiments below.

The main controlled experiments use a small model and a carefully constructed collection of domain experts sharing the same initialization and training recipe.

Predictions hold on larger models. We trained Qwen3-4B with RL on 6 datasets (math, code, science, instruction following, tool use and safety). Each of the experts is trained with full parameter RL (no LoRA) on one dataset. We train 9 mixtures of these datasets, and evaluate on 4 standard benchmarks (LiveCodeBench, IFBench, BFCL multi turn and AIME 2024 to 2026). With no training, Policy-Merge predicts the 9 trained mixtures with the mean error below, in points (per mixture and benchmark in Figure 12, Appendix E.4).

Mixture, mean error over the 4 benchmarks (points)Policy-MergeWeight mergingBase model
Math / Code / Safety0.81.02.3
Math / Code / Agent / Safety1.62.43.7
Uniform six-way1.82.63.4
Agent 50 / Safety 501.31.93.0
Code 50 / Safety 500.51.32.7
Agent 64 / Code 362.21.84.3
IF 65 / Agent 353.13.34.5
Code / Safety / Agent1.21.83.6
Code / Safety / Agent / IF3.33.64.7
Mean ±\pm standard error over the 9 mixtures1.75 ±\pm 0.322.17 ±\pm 0.293.59 ±\pm 0.28

Policy-Merge is robust to how the experts were trained. It predicts the trained mixtures with our 0.6B LoRA experts (0.93 points) and our 4B full-parameter experts (1.75 points), and also with experts that others trained, with their own recipe, and released. Composing the Math, Code and Instruction Following experts released with Olmo 3, with no training, Policy-Merge reaches 52.5 on MATH-500 and 43.1 on MBPP+, against 53.0 and 44.2 for RL-Zero Mix, which Ai2 trained from the same base model on a mixture of their data, within one standard error (Figure 8). Policy-Merge can thus highly efficiently predict mixtures of released, or preexisting experts without any additional training.

[...] Broader mixture-selection experiments are needed to demonstrate practical benefits beyond accurately approximating trained policies.

We repeat the mixture selection experiments at 4B. The experts trained on Reasoning Gym environments improve most external benchmarks only marginally, so we use the 4B experts to test selection on a target that combines two benchmarks, IFBench and HarmBench. We select mixtures with Policy-Merge using the KL scaled weights w~\tilde{\mathbf w}, train each method's pick, and score it on the held out split. We do agree that the results are within standard deviation.

Policy-MergeMergeMixOP-MixBase model
IFBench + HarmBench (95% CI)65.7 [62.0, 69.4]65.4 [61.7, 69.1]65.1 [61.4, 68.8]63.4 [59.7, 67.1]

The target is the average of the two benchmarks. As at 0.6B, Policy-Merge and MergeMix use the same search on a small dataset, so the mixtures they select train to within noise of each other. However, Policy-Merge predicts the trained mixtures more closely, and for a given set of candidate mixtures it chooses the best one more often, as we show next.

Policy-Merge predicts the trained mixtures more closely than weight merging. MergeMix uses merged weights to select mixtures, so we adapt weight-space merging to predict the trained policy, and compare this prediction with composing the policies, against the trained mixtures at 0.6B (Table 3 and Table 1).

Against the trained mixtures at 0.6BPolicy-MergeWeight merging
Mean error in 17-task accuracy, 8 mixtures (points)0.932.13
Largest error on one task group (points)9.7 (Figure 5)15.2 (Appendix E.2)
KL to the trained mixture, relative to the base model, 6 mixtures13.4× closer5.9× closer

Policy-Merge most often predicts which mixture is best for a target. We use the six 0.6B mixtures we trained as candidate mixtures, as if the search step of selection had already produced them, and 22 targets, such as Graphs or a pair of task groups (new Appendix E.6). We then use Policy-Merge and weight merging to predict the score of each candidate, and to choose the best mixture for each target.

MethodBest mixture minus chosen mixture on held-out problems, in points (middle 95% of splits)How often the chosen mixture is the best
Policy-Merge (w~\tilde{\mathbf w})0.57 [0.18, 1.11]63.5%
Policy-Merge (w\mathbf w)0.61 [0.20, 1.27]58.6%
Weight merging0.76 [0.25, 1.49]54.8%
Uniform six-way mixture2.37 [1.45, 3.48]15.3%
Random choice4.79 [4.26, 5.35]17.6%

Our method for predicting the policy allows additional novel uses beyond accurately approximating the trained policy, or for selecting mixtures.

  • Using a data mixture without training it. Policy-Merge offers a way to decode from a desired data mixture without running the expensive training, at the cost of more decoding time (2.45× the time of one model, Table 2). This may be useful for small-scale inference, such as evaluating or probing a candidate mixture before committing to a training run. Its mean error is 0.93 points at 0.6B.
  • Predicting the result of distillation before training a student. Policy-Merge also offers a way to decode what a student distilled from routed teachers will do, where each prompt is routed to the expert trained on its dataset, without running the distillation. The composition is within 1.4 points of the students distilled from the 4B experts (Figure 9), and within 0.2 points of the student released by Open-MOPD.

The predicted policy could also support choosing the weights of teachers for distillation, and predicting unintended effects of a mixture, such as misalignment, before training, which we leave to future work.

W2. End to end Compute Comparison

[Appendix E.3] does not report the measured GPU-hours of the complete 40-mixture search or an end-to-end comparison with MergeMix and OP-Mix.

We now report GPU hours for MATH-500, measured from our job logs, from training the experts to training the final mixture (new Table 5). To standardize GPU hours, we report each training run at its cost on one GPU, which replaces the numbers in the original Appendix E.3 (27.7 GPU hours per mixture and 166 for the experts).

GPU hoursTraining the 6 expertsSearchTraining the final mixtureTotal
Policy-Merge973.717≈118\approx 118
MergeMix97 (same experts)1.717≈116\approx 116
OP-Mix017.917≈35\approx 35
Train all 40 candidates00≈700\approx 700≈700\approx 700

Composing all six experts takes 2.45× more time per token than one model (Table 2), but most of the 40 candidates compose fewer experts, and the measured search takes 3.7 GPU hours, against 1.7 for MergeMix. The search is a small part of the total, so end to end Policy-Merge requires about 2% more compute than MergeMix, and both require 1/6 of the compute of training all 40 candidates. Policy-Merge is most useful when the experts already exist, such as the released Olmo 3 experts, or experts trained anyway for multi-teacher distillation.


If the 4B experiments and the compute comparison address your concerns, we would be grateful if you would reconsider your score. If a concern remains, please let us know, and we are happy to discuss further.

Thank you again for your detailed review and helpful suggestions. Have we addressed your concerns in our response or is there anything else we could clarify?

Comments on the paper

1 comment

We thank all reviewers for their detailed and constructive feedback. We appreciate that the reviewers found our evaluation across several reasoning domains and externally trained experts thorough (zXwq), our method a useful, cheaper way to estimate allocation outcomes and preview student capabilities in multi-teacher on-policy distillation (zXwq, mKbF), and our paper clear evidence for policy-level compositionality across domains (fHjd). Below we provide a list of the primary reviewer concerns and a concise summary of our response to each.

  • Scale (zXwq, fHjd). To address the concerns about 'small model experts sharing same initialization and training recipe', we additionally trained Qwen3-4B experts with full parameter RL on six datasets on more challenging domains (safety, tool use, code, math, science, instruction following), and nine mixtures of them. Policy-Merge remains accurate at larger scale. Without any training, it predicts the 4B mixtures with a mean error of 1.75 points, compared to 2.17 for weight merging (Appendix E.4).
  • Domain Reliability (fHjd). To test whether prediction underperforms on specific domains, we measured error separately for each domain. On the new Qwen3-4B experts, 34 of 36 domain-specific results are within 5 points of the trained model, so the larger errors seen on the Graphs domain do not recur at larger scale (Figure 12).
  • Selection Benefits and Coefficient Consistency (zXwq, fHjd). Following the reviewer's suggestion, the paper now uses the KL scaled coefficients in every prediction and selection experiment. Given a set of candidate mixtures, Policy-Merge selects the best candidate mixture in 63.5% of 22 targets, against 54.8% with weight merging and 58.6% with data proportions (Appendix E.6). We also added a mixture selection experiment for IFBench and HarmBench at 4B, where the mixture Policy-Merge selects outperforms MergeMix and OP-Mix (Figure 6).
  • Comparison with Prior Work (mKbF, fHjd). We note that prior composition methods (PoEM, MOD, DeRa) keep the training prompts fixed and vary only the reward weighting or KL strength, we show the first evidence that composition works when each expert is trained on vastly different domains, such as math, code, science, instruction following and safety, and that composing these experts predicts the policy trained on their mixture. Composing experts from different domains also revealed that they drift from the base model by very different amounts. We empirically introduce KL scaled coefficients that capture these differences and reduce prediction error, and we connect composition to predicting the student of multi-teacher on-policy distillation before training (Section 5).
  • Linear Baseline (mKbF). We added the linear baseline the reviewer suggested, which predicts a mixture's score as a weighted average of the experts' scores (Appendix A.1), and find that Policy-Merge is more accurate (mean error 0.93 vs 1.41 points).
  • Compute Comparison (zXwq). End to end, Policy-Merge only uses about 2% more compute than MergeMix (Table 5), and is especially efficient when experts already exist, such as released or preexisting distillation experts.

The paper provides the first evidence that experts trained on very different domains can be composed to predict the policy trained on their data mixture, before any training. Our proposed KL scaled coefficients account for how much each expert drifts from the base model, and the predictions stay accurate at 4B scale and across domains. Practically, this method provides a way to preview what a model will learn from RL a data mixture or a on-policy from a set of teachers, before committing compute to training it. Additionally, as more RL experts and environments are released, Policy-Merge makes released experts reusable for designing new mixtures and distilled students.

The key paper revisions are highlighted in blue. As we have not yet received follow-up comments on our responses, we would be grateful for reviewers’ feedback on any remaining concerns so that we can address them before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.