- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new method to combine domain experts that are fine-tuned from the same base model (or any shared starting checkpoint). The method uses model merging and multi-teacher on-policy distillation (MOPD). The authors show that, to second order, the MOPD objective is the same as Fisher merging. Based on this, they propose their method called RGBM. It consists of two phases. Phase 1 merges the experts using one coefficient per expert. These coefficients are found by running optimization on a small set containing unlabeled prompts from all domains. Phase 2 trains a separate distillation branch only for domains where the merge leaves a large residual. It then adds this branch back with a small scale (which is searched). There are 2 experimental setups: Open-MOPD (SmolLM3-3B with Math, Code and Instruction Following experts) and G-OPD (Qwen3-4B with Math and Code experts). On Open-MOPD, Phase 1 merge gets 32.52 average, agains 30.93 for released MOPD model. Phase 2 gets 33.03. On G-OPD, Phase 1 is 0.45 below the best baseline, and Phase 2 is 0.35 above it.
Critical strengths
- The topic and problem is really good, timely and practical. How to do large-scale Multi-Teacher OPD is an active area of research both in academia and the industry and to a large extent, the best recipe is not well known. Model Merging is also widely used. Combining it with MOPD is a good direction.
- The method is simple and relatively cheap. It only needs a small held-out set of unlabelled prompts and solve a small optimization problem.
- Section 4.3 is very well written with very clear takeaways for any practitioners that might read this manuscript. It shows that continuing joint distillation from the merge hurts Code and IF, even when the KL to the code teacher is very small. The authors are also honest about certain limitations.
Critical weaknesses
- The main concern I have is that the evidence is weak. All results are single runs, with no seeds or confidence intervals. The gains are also not big enough to not question this: 0.6 to 1.6 for Phase 1 and 0.5 for Phase 2.
- Baselines need more attention in my opinion. The main claim is to outperform MOPD. The authors do not reproduce MOPD or spend enough time trying to come up with a strong MOPD baseline. They are primarily relying on Open-MOPD's released checkpoint. It is worse than task arithmetic. On G-OPD as well, the baseline numbers are coped from the original paper. The authors are already training many other models, so reproducing MOPD from the same budget should be possible. Continuing MOPD from the merge is only tested for 50 steps, with no sweeps. As someone actively working in the OPD domain, it is already known that it is not super sample efficient.
- Experimental scope is a little narrow. The largest scale is 4B and the largest number of domain experts is 3. No setting has K>=3 domains active for MOPD. We do not know how merging, RGBM or MOPD behave on other capabilities, as all benchmarks are in the experts's own domains.
I am happy to raise my score if additional convincing results are presented against these weaknesses.
Questions for authors
- If possible, please give confidence intervals or any significance tests over 3 seeds, especially for Tables 1 and 2.
- Why the largest gap rule for selecting which domains to continue training on? How does it compare with a fixed threshold, or with writing back all branches? Are you also able to find any settings with more than 3 active domains?
- On the converged G-OPD pair, the coefficients are roughly 50-50 and give 57.35. The equal-weight average (also 50-50) gives 41.72. Aren't these almost the same model? or am I misunderstanding something?
5 comments
Q2
Thank you for asking about the rationale for the gate and its alternatives. Regarding "Why the largest gap rule": The normalized residual spectrum measures the relative reward-reconstruction gap remaining in each domain after Phase 1. We use its largest adjacent gap to identify a separation between domains with larger and smaller residuals, without prescribing the number of active domains. The largest gap provides the widest interval between adjacent residual levels, and its midpoint places the threshold equally far from the two endpoints. Appending zero also allows all domains to remain active when the largest separation is between the residuals and this floor, as occurs on G-OPD. This gives a simple allocation rule computed entirely from unlabelled calibration data.
Regarding "compare with a fixed threshold": The reported spectra give the following allocations:
| Setting | Normalized residuals | Largest-gap threshold | Active domains | Fixed thresholds producing the same allocation |
|---|---|---|---|---|
| Open-MOPD | Math: 0.128, Code: 0.052, IF: 0.015 | 0.090 | Math | 0.052≤ϵ<0.128 |
| G-OPD | Math: 0.094, Code: 0.090 | 0.045 | Math and Code | 0≤ϵ<0.090 |
Thus, suitable fixed thresholds reproduce our selections. Indeed, a common threshold in [0.052,0.090) works for both settings. These intervals show that the allocations do not depend on an exact threshold value. The role of the largest-gap rule is to obtain a cutoff automatically from the observed spectrum, without tuning ϵ on downstream benchmark accuracy. The current experiments support this automatic choice, while leaving broader comparisons across alternative threshold rules for further evaluation.
Regarding "writing back all branches": Table 1 compares the following strategies from the same Phase 1 initialization:
| Open-MOPD configuration | Macro |
|---|---|
| Phase 1 merge | 32.52 |
| TIES, all branches | 30.96 |
| DARE, all branches | 31.42 |
| Math branch without shrinkage, γ=1 | 31.80 |
| RGBM, Math branch with γ=0.125 | 33.03 |
Both evaluated all-branch strategies fall below Phase 1. Admitting only Math also requires controlling the update magnitude: its full update scores 31.80, whereas the magnitude selected by the unlabelled diagnostic reaches 33.03. These results support selective writeback together with shrinkage. Because TIES and DARE also change the merging operator, these comparisons evaluate the complete strategies and do not isolate the gate alone. On G-OPD, the spectrum activates both available domains, and their shrunken writeback improves the seven-task average from 53.34 to 54.14. Thus, RGBM can retain all branches when the spectrum calls for it.
Regarding "more than 3 active domains": Our current experiments involve at most three experts and at most two simultaneously active branches. We therefore do not yet provide evidence for more than three active domains. We will make this experimental scope explicit, with larger active sets remaining an important direction for further evaluation.
Q3
Regarding "equal-weight average" and RGBM: Thank you for highlighting this discrepancy. The coefficients are indeed close, but this does not imply identical predictions. A two-teacher equal-weight average is
=W_{\mathrm{ref}}+0.5\tau_{\mathrm{math}}+0.5\tau_{\mathrm{code}},$$ where $\tau_d=W_d-W_{\mathrm{ref}}$. In the converged G-OPD setting, RGBM Phase 1 uses coefficients $(0.511,0.505)$, giving $$\theta_{\mathrm{RGBM}}=W_{\mathrm{ref}}+0.511\tau_{\mathrm{math}}+0.505\tau_{\mathrm{code}} =-0.016W_{\mathrm{ref}}+0.511W_{\mathrm{math}}+0.505W_{\mathrm{code}}.$$ The corresponding parameter weights are: | Merging method | Weight on $W_{\mathrm{ref}}$ | Weight on $W_{\mathrm{math}}$ | Weight on $W_{\mathrm{code}}$ | | --- | ---: | ---: | ---: | | Two-teacher equal-weight average | $0$ | $0.5$ | $0.5$ | | RGBM Phase 1 | $-0.016$ | $0.511$ | $0.505$ | Their parameter difference is $\theta_{\mathrm{RGBM}}-\theta_{\mathrm{avg}}=0.011\tau_{\mathrm{math}}+0.005\tau_{\mathrm{code}}$. The values 0.011 and 0.005 scale full expert task vectors, so they are not uniform increments to individual parameters. Consequently, the decimal differences alone do not establish how close the models' predictions are. RGBM obtains its coefficients by optimizing the residual objective on unlabeled calibration data and achieves a seven-task average of 57.35 in the converged setting. To illustrate the broader point that close coefficients can yield different scores, we provide the following Open-MOPD comparisons. We added eight Open-MOPD ablations to compare the merging objective, domain aggregation, and coefficient parameterization. C-vec L (RGBM Phase 1) achieves the highest Macro, and each raw-residual configuration outperforms its normalized counterpart in Macro, supporting the Phase 1 objective. Labels A / C denote uniform domain weighting / minimax, sc / vec a shared scalar / independent coefficient vector, and L / Lb raw / energy-normalized residuals. Coefficients enter task-vector merging as $\theta=W_{\mathrm{ref}}+\sum_d\alpha_d\tau_d$. Coefficients are ordered Math, Code, IF, and Macro weights domains equally: | Configuration | Merge coefficients (Math / Code / IF) | Math | Code | IF | Macro | | --- | --- | ---: | ---: | ---: | ---: | | `A-sc L` | 0.626 / 0.626 / 0.626 | 24.97 | 23.28 | 47.64 | 31.96 | | `A-sc Lb` | 0.501 / 0.501 / 0.501 | 24.38 | 23.36 | 44.86 | 30.86 | | `A-vec L` | 0.527 / 0.609 / 0.821 | 24.11 | 23.59 | 47.99 | 31.90 | | `A-vec Lb` | 0.840 / 0.260 / 0.841 | 24.19 | 19.57 | 48.00 | 30.59 | | `C-sc L` | 0.579 / 0.579 / 0.579 | 24.97 | 23.49 | 45.10 | 31.19 | | `C-sc Lb` | 0.477 / 0.477 / 0.477 | 21.41 | 19.76 | 45.79 | 28.99 | | `C-vec L` | 0.597 / 0.620 / 0.800 | 23.86 | 24.09 | 49.60 | **32.52** | | `C-vec Lb` | 0.724 / 0.426 / 0.630 | 24.22 | 20.73 | 47.02 | 30.66 | A particularly relevant comparison is A-sc Lb versus C-sc Lb. Both use shared-scalar coefficients and energy-normalized residuals, differing in uniform domain weighting versus minimax aggregation. Their coefficients are 0.501 and 0.477, only 0.024 apart per expert, yet their Macro scores differ by 1.87 points. Math and Code differ by 2.97 and 3.60 points, respectively, while IF changes in the opposite direction. This demonstrates that nearby coefficients can yield materially different cross-domain performance. Across these eight ablations and the equal-weight baseline, the three-domain mean raw calibration residual has a Spearman correlation of approximately −0.92 with Macro. This supports the residual as an informative signal for coefficient selection within the evaluated configurations, with benchmark accuracy used only for subsequent validation. The complete coefficients, scores, residuals, and figure are available at https://anonymous.4open.science/r/0f593a7b3935599f-DD7B/.
W1
Thank you for your concern about stability. We added multi-seeds results. For Open-MOPD, the Phase 1 and Phase 2 means ± sample standard deviations are 32.39 ± 0.34 and 33.07 ± 0.22, respectively. For G-OPD with intermediate teachers, they are 53.63 ± 0.30 and 54.16 ± 0.39.
To complement the seed evaluations, we additionally tested whether individual benchmarks drive the reported gains in the comparisons below. Holding models fixed, we removed the same benchmark from both methods and recomputed aggregate scores from the paper's per-benchmark results. Open-MOPD averages remaining benchmarks within domains and then weights domains equally. G-OPD averages the six remaining tasks equally. The table compares the original aggregate differences with their ranges after each benchmark removal:
All differences remain positive, indicating that the gains do not depend on any single benchmark.
W2
Thank you for suggesting stronger baseline validation.
Regarding “reproducing MOPD”: We use the officially released, fully trained Open-MOPD checkpoint to provide a reproducible baseline and build on the original authors' training effort. It scores 30.93 under our common evaluation protocol. We also independently retrained an Open-MOPD student under settings consistent with the original paper, obtaining 31.13, close to the 31.24 reported in the original paper. RGBM Phase 1 achieves 32.52, exceeding this retrained baseline by 1.39 points.
Regarding “only tested for 50 steps”: Our exploration extended beyond 50 steps. In particular, our initial Uniform Distillation experiment ran for 300 steps. We observed that once a domain lost more than 1–2 points, its performance was often difficult to recover even as other domains improved. The first 50 steps were sufficient to reveal these domain-level trends in our experiments. These observations suggest capability tradeoffs near the merged initialization and are consistent with the hypothesis that Phase 1 lies near a local Pareto frontier, although they do not establish Pareto optimality. This motivates selective OPD in isolated branches, followed by controlled writeback to reduce interference with capabilities already integrated into the model. We will describe the extended training settings and observations in the appendix, distinguishing the reported training window from our broader exploration.
W3
Regarding “experimental scope”: Thank you for your questions about scalability and out-of-domain capabilities. Our experiments cover two model families, SmolLM3-3B and Qwen3-4B, two or three experts, intermediate and converged G-OPD teachers, and three expert pairs (Appendix B.4). Although model sizes are limited to 3B–4B, these settings evaluate different expert correlations and training stages. We plan to explore additional open-source settings in future work.
Regarding “all benchmarks are in the experts' own domains”: Our current evaluation measures consolidation within the specialists' domains. It does not establish preservation of capabilities outside those domains. We will make this limitation explicit.
Q1
Thank you for suggesting confidence intervals. We added evaluations of the Phase 1 and Phase 2 checkpoints in Tables 1 and 2 over three evaluation seeds. Each row combines the paper's reported score with two additional evaluations of the same checkpoint. We report the mean, sample standard deviation, and approximate 95% Student's t confidence interval for that mean, assuming independent, approximately normally distributed seed-level scores. These intervals capture evaluation-seed variation for fixed checkpoints. ExOPD retains its published score and has no additional evaluations: