- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether we can predict the policy produced by multi-domain RL post-training without training a model for every candidate data mixture. Policy-Merge composes the next-token log-probability shifts of independently trained domain experts relative to a shared reference model. It proposes mapping data proportions to composition coefficients using each expert’s KL divergence from the reference. The resulting policy can be decoded to estimate mixture behavior and guide mixture selection. The paper also establishes a fixed-prefix equivalence between weighted teacher-specific reverse-KL losses and distillation from a composed teacher. Validation includes Qwen3-0.6B experts and eight trained mixtures across six Reasoning Gym domains, comparisons in accuracy and token-level KL, a MATH-500 mixture-selection experiment, and distillation experiments. Additional evaluations use released Olmo 3 7B models and Open-MOPD teachers and students.
Critical strengths
-
The paper provides meaningful empirical evidence for policy-level compositionality across different domains.
-
The proposed method is practically useful, and evaluations on different models further support its effectiveness.
-
The paper explores an interesting direction for inference-time expert merging and serving without additional training.
Critical weaknesses
-
In Figure 9, there appear to be two gray vertical lines in the lower row of each benchmark. It is unclear why both lines are present and what they represent. The authors should clarify this visualization or simplify the figure to avoid confusion.
-
The mixture-selection experiment in Section 4.2 uses the original data proportions rather than the proposed KL-adjusted coefficients. It remains unclear how much the proposed weighting strategy contributes to the practical mixture-selection task.
-
Although the paper reports a low average prediction error, Figure 5 shows relatively large discrepancies in certain domains. For example, the predicted accuracy on Graphs is 34.0%, compared to 43.7% for the actual trained model. This raises concerns about the reliability of Policy-Merge in predicting domain-specific performance.
Questions for authors
- Please clarify the key differences between the proposed Policy-Merge method and DERA (https://arxiv.org/pdf/2402.02992), particularly in terms of their underlying methodologies and technical novelty.
- Why is used for mixture selection, while the original drift-adjusted rule is used for the main prediction experiments and a normalized version is used for Olmo 3? Were these choices fixed independently of the target mixture’s observed performance? How do the original Equation (5), unadjusted proportions, and normalized drift weights compare in these settings?
5 comments
Q1. Differences from DeRa.
Please clarify the key differences between the proposed Policy-Merge method and DERA, particularly in terms of their underlying methodologies and technical novelty.
DeRa proposes a method for interpolating between a reference model and a post-trained policy with a hyperparameter . The paper also derives the product-of-experts sampling using multiple models, for a scenario where the reward of the target model is a weighted combination of rewards . We note that the sampling strategies are similar, but DeRa focuses on predicting the policy trained on a sum of rewards, and the coefficients are given by a user, for how much each reward should be accounted for. Specifically, Policy-Merge is different in two key ways:
- Predicting the policy trained on a data mixture. DeRa predicts a policy trained on the same prompts with a different KL strength or reward weighting. Policy-Merge predicts RL on a data mixture from experts trained on very different domains, such as math, code, science, instruction following and safety. We show empirically that composing the experts predicts it, even if they are trained on different prompt sets.
- Setting the coefficients of composition. DeRa's coefficients are the given reward weights or KL strength. Policy-Merge derives them from the data proportions and each expert's KL drift from the base model (Eq. 5), without the trained mixture.
Beyond these two differences, the paper makes two contributions that DeRa does not address.
- Experts trained on different prompt sets compose. To our knowledge, we show the first empirical evidence that the policy trained on a data mixture is a linear composition of the policies of experts trained on each dataset alone. With the coefficients of Eq. 5, the composition accounts for 85% to 95% of how far each of the six trained mixtures moves from the base model (Figure 2).
- Predicting the student of multi-teacher on-policy distillation. Distillation from experts weighted by the data proportions has the same per-prefix gradient as distillation from the single composed policy (Appendix F). Decoding the composed policy also predicts students distilled with routed teachers, where each prompt is routed to the expert of its domain, within 1.4 points of the 4B students (Figure 9) and within 0.2 points of the student released by Open-MOPD.
Q2. Coefficients in each setting.
Why is [] used for mixture selection, while the original drift-adjusted rule is used for the main prediction experiments and a normalized version is used for Olmo 3? Were these choices fixed independently of the target mixture's observed performance? How do the original Equation (5), unadjusted proportions, and normalized drift weights compare in these settings?
We apologize for this inconsistency. We first ran the mixture selection experiment before adding the drift adjustment, and did not rerun it before submission. We have now rerun it with (Section 4.2). For Olmo 3, we normalized the coefficients of Eq. 5 to sum to one before comparing with RL-Zero Mix, because the overall scale of assumes the experts and the mixture share a training recipe, which we could not verify for released models, so we kept only the relative weights. The updated paper uses the unnormalized coefficients, as in every other experiment for consistency. Normalization did not help, it is 1.0 point further from RL-Zero Mix on MATH-500 and equal on MBPP+, although both differences are within one standard error. Initially, to predict the 0.6B and Open-MOPD distillation students, we composed the experts at the data proportions , so that the teacher weights are positive and sum to one, as in standard multi-teacher distillation (Section 2.4) however we show that OPD can also empirically apply by running additional experiments for 4B multi-teacher OPD using Eq. 5, where Policy-Merge is within 1.4 points of the distilled students on average (Figure 9, top). To answer your question directly, none of these choices were made after observing the trained mixture’s performance. We hope that our update to using Eq. 5/ in every prediction and selection experiment removes this inconsistency.
| Olmo 3, error against RL-Zero Mix (points) | MATH-500 | MBPP+ |
|---|---|---|
| Data proportions | 4.0 | 4.7 |
| Eq. 5 | 0.5 | 1.1 |
| Normalized | 1.5 | 1.1 |
Thank you again for the careful review. We hope the revised paper and these answers address your questions, and we are happy to clarify anything else.
Thank you again for your careful review and helpful suggestions. Have we addressed your concerns in our response or is there anything else we could clarify?
Response to Reviewer fHjd
Thanks for your thorough review and suggestions. We are glad to hear that you found the method practically useful and an interesting direction for inference-time expert merging and serving without additional training.
W1. Figure 9.
Thanks for pointing out how this figure could be improved. In the original figure, the gray line in the lower row shows the accuracy of the distilled student in the row above, and a second, faded line shows the accuracy of Policy-Merge. These lines were intended to show how Policy-Merge is close to the student on every benchmark but we see that the figure could be simplified. We have updated the figure.
W2. Selection with the KL scaled coefficients.
We added selection with w~ and compare it with the initial selection at w. The KL scaled coefficients improve prediction of the trained policy in our main experiments, but they do not improve selection on MATH-500.
W3. The Graphs domain in Figure 5.
We agree that Graphs results has larger prediction error in comparison to other domains. Of the 9 results in Figure 5 with an error above 5 points, 6 are on Graphs, including the case you point out. One possible explanation is that the drift adjustment underweights experts with low drift. However, low drift alone does not explain this error. Cognition has lower KL drift than Graphs, yet its accuracy in the six-way mixture is predicted accurately.
We would also like to point out that Policy-Merge remains reliable on the other domains. On the other five domains, the mean absolute error is 1.95 points, and 37 of 40 results are within 5 points of the trained model. Across all 48 domain results in Figure 5, 35 are within 3 points and 39 are within 5 points, with a mean absolute error of 2.61 points. Policy-Merge is also robust to capturing large gains on individual domains. After training, the four-way mixture reaches 56.5% on Geometry. Policy-Merge predicts 54.0%, while interpolating the experts' scores predicts only 38.4%.
To examine domain-specific reliability further, we trained Qwen3-4B experts on six domains: math, code, science, instruction following, tool use and safety with full fine-tuning. We then trained nine mixtures of these experts. Training on these mixtures improves over the base model by up to 9.2 points on BFCL multi-turn (6.6 on average) and 5.7 points on AIME 2024–2026 (3.9 on average). With no training, Policy-Merge predicts 34 of the 36 results on LiveCodeBench, IFBench, BFCL multi-turn and AIME within 5 points of the trained model, with a mean error of 1.75 points, against 2.17 for weight merging (Figure 12, Appendix E.4). The largest errors are on BFCL multi-turn, where training on the mixture gains more than either prediction. Policy-Merge makes a more accurate prediction, with a mean error of 3.09 points against 4.12 for weight merging.
These additional results show that accurate domain specific prediction extends to larger models and more challenging benchmarks, although accuracy is not uniform across domains.