acceptodds
Under review as a conference paper at ICLR 2027

Merge Before You Distill: A Fisher-Geometric View of Multi-Teacher On-Policy Distillation

Abstract

Consolidating specialized foundation models into a unified generalist is a central goal of post-training. Two approaches are in common use: model merging, which is fast but treated as a heuristic, and multi-teacher on-policy distillation, which matches distributions but demands heavy rollout compute and suffers from cross-task gradient conflict. We show that the two are one objective seen at different orders. Within a shared basin of homologous specialists, the multi-teacher on-policy distillation objective is, to second order, exactly the Fisher-merging objective. Its minimizer is therefore the Fisher merge, a classic model-merging method given by a closed-form combination of the specialists, so merging is optimal, to second order, as a closed-form stage of distillation. Building on this identification, we propose Residual-Governed Branch-and-Merge (RGBM), a two-phase consolidation procedure. Concretely, a full-parameter Fisher merge is intractable on foundation models, so Phase 1 restricts the merge to the task-vector subspace. Extending tangent-space analyses of merging, we show that the same coefficients that mix task vectors also mix implicit rewards, and therefore obtain them with a residual program on unlabelled prompts. The program returns a closed-form merge together with the leftover per-domain error, which we call the domain residual spectrum. Phase 2 lets the domain residual spectrum govern what may be added: RGBM passivates low-residual domains (unsafe to write back), trains isolated branches only for high-residual domains, and writes back, with shrinkage, only the updates the spectrum marks as safe. On two open-source consolidation suites, the Phase 1 merge alone matches or outperforms fully trained multi-teacher distillation, and RGBM improves on it further with zero extra joint-training steps, whereas every un-gated write-back, including continued joint distillation from the merge, falls below it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Teacher On-Policy Distillation, Model Merge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RuynOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new method to combine domain experts that are fine-tuned from the same base model (or any shared starting checkpoint). The method uses model merging and multi-teacher on-policy distillation (MOPD). The authors show that, to second order, the MOPD objective is the same as Fisher merging. Based on this, they propose their method called RGBM. It consists of two phases. Phase 1 merges the experts using one coefficient per expert. These coefficients are found by running optimization on a small set containing unlabeled prompts from all domains. Phase 2 trains a separate distillation branch only for domains where the merge leaves a large residual. It then adds this branch back with a small scale (which is searched). There are 2 experimental setups: Open-MOPD (SmolLM3-3B with Math, Code and Instruction Following experts) and G-OPD (Qwen3-4B with Math and Code experts). On Open-MOPD, Phase 1 merge gets 32.52 average, agains 30.93 for released MOPD model. Phase 2 gets 33.03. On G-OPD, Phase 1 is 0.45 below the best baseline, and Phase 2 is 0.35 above it.

Critical strengths
  1. The topic and problem is really good, timely and practical. How to do large-scale Multi-Teacher OPD is an active area of research both in academia and the industry and to a large extent, the best recipe is not well known. Model Merging is also widely used. Combining it with MOPD is a good direction.
  2. The method is simple and relatively cheap. It only needs a small held-out set of unlabelled prompts and solve a small optimization problem.
  3. Section 4.3 is very well written with very clear takeaways for any practitioners that might read this manuscript. It shows that continuing joint distillation from the merge hurts Code and IF, even when the KL to the code teacher is very small. The authors are also honest about certain limitations.
Critical weaknesses
  1. The main concern I have is that the evidence is weak. All results are single runs, with no seeds or confidence intervals. The gains are also not big enough to not question this: 0.6 to 1.6 for Phase 1 and 0.5 for Phase 2.
  2. Baselines need more attention in my opinion. The main claim is to outperform MOPD. The authors do not reproduce MOPD or spend enough time trying to come up with a strong MOPD baseline. They are primarily relying on Open-MOPD's released checkpoint. It is worse than task arithmetic. On G-OPD as well, the baseline numbers are coped from the original paper. The authors are already training many other models, so reproducing MOPD from the same budget should be possible. Continuing MOPD from the merge is only tested for 50 steps, with no sweeps. As someone actively working in the OPD domain, it is already known that it is not super sample efficient.
  3. Experimental scope is a little narrow. The largest scale is 4B and the largest number of domain experts is 3. No setting has K>=3 domains active for MOPD. We do not know how merging, RGBM or MOPD behave on other capabilities, as all benchmarks are in the experts's own domains.

I am happy to raise my score if additional convincing results are presented against these weaknesses.

Questions for authors
  1. If possible, please give confidence intervals or any significance tests over 3 seeds, especially for Tables 1 and 2.
  2. Why the largest gap rule for selecting which domains to continue training on? How does it compare with a fixed threshold, or with writing back all branches? Are you also able to find any settings with more than 3 active domains?
  3. On the converged G-OPD pair, the coefficients are roughly 50-50 and give 57.35. The equal-weight average (also 50-50) gives 41.72. Aren't these almost the same model? or am I misunderstanding something?
5 comments

W1

Thank you for your concern about stability. We added multi-seeds results. For Open-MOPD, the Phase 1 and Phase 2 means ± sample standard deviations are 32.39 ± 0.34 and 33.07 ± 0.22, respectively. For G-OPD with intermediate teachers, they are 53.63 ± 0.30 and 54.16 ± 0.39.

To complement the seed evaluations, we additionally tested whether individual benchmarks drive the reported gains in the comparisons below. Holding models fixed, we removed the same benchmark from both methods and recomputed aggregate scores from the paper's per-benchmark results. Open-MOPD averages remaining benchmarks within domains and then weights domains equally. G-OPD averages the six remaining tasks equally. The table compares the original aggregate differences with their ranges after each benchmark removal:

ComparisonReported difference on all benchmarksDifference after removing one benchmarkPositive differences
Open-MOPD: Phase 2 − Phase 1+0.51+0.400 to +0.6276/6
Open-MOPD: Phase 1 − released 600-step MOPD+1.59+1.188 to +1.9556/6
G-OPD intermediate teachers: Phase 2 − Phase 1+0.80+0.642 to +1.1127/7
G-OPD intermediate teachers: Phase 2 − ExOPD+0.35+0.153 to +0.5127/7

All differences remain positive, indicating that the gains do not depend on any single benchmark.

W2

Thank you for suggesting stronger baseline validation.

Regarding “reproducing MOPD”: We use the officially released, fully trained Open-MOPD checkpoint to provide a reproducible baseline and build on the original authors' training effort. It scores 30.93 under our common evaluation protocol. We also independently retrained an Open-MOPD student under settings consistent with the original paper, obtaining 31.13, close to the 31.24 reported in the original paper. RGBM Phase 1 achieves 32.52, exceeding this retrained baseline by 1.39 points.

Regarding “only tested for 50 steps”: Our exploration extended beyond 50 steps. In particular, our initial Uniform Distillation experiment ran for 300 steps. We observed that once a domain lost more than 1–2 points, its performance was often difficult to recover even as other domains improved. The first 50 steps were sufficient to reveal these domain-level trends in our experiments. These observations suggest capability tradeoffs near the merged initialization and are consistent with the hypothesis that Phase 1 lies near a local Pareto frontier, although they do not establish Pareto optimality. This motivates selective OPD in isolated branches, followed by controlled writeback to reduce interference with capabilities already integrated into the model. We will describe the extended training settings and observations in the appendix, distinguishing the reported training window from our broader exploration.

W3

Regarding “experimental scope”: Thank you for your questions about scalability and out-of-domain capabilities. Our experiments cover two model families, SmolLM3-3B and Qwen3-4B, two or three experts, intermediate and converged G-OPD teachers, and three expert pairs (Appendix B.4). Although model sizes are limited to 3B–4B, these settings evaluate different expert correlations and training stages. We plan to explore additional open-source settings in future work.

Regarding “all benchmarks are in the experts' own domains”: Our current evaluation measures consolidation within the specialists' domains. It does not establish preservation of capabilities outside those domains. We will make this limitation explicit.

Q1

Thank you for suggesting confidence intervals. We added evaluations of the Phase 1 and Phase 2 checkpoints in Tables 1 and 2 over three evaluation seeds. Each row combines the paper's reported score with two additional evaluations of the same checkpoint. We report the mean, sample standard deviation, and approximate 95% Student's t confidence interval for that mean, assuming independent, approximately normally distributed seed-level scores. These intervals capture evaluation-seed variation for fixed checkpoints. ExOPD retains its published score and has no additional evaluations:

SettingReported scoreTwo additional evaluationsMean ± standard deviationApproximate 95% CI for the mean
Open-MOPD Phase 1, Macro32.5232.65, 32.0132.39 ± 0.34[31.55, 33.23]
Open-MOPD Phase 2, Macro33.0332.87, 33.3133.07 ± 0.22[32.52, 33.62]
G-OPD intermediate teachers, Phase 1, seven-task average53.3453.93, 53.6153.63 ± 0.30[52.89, 54.36]
G-OPD intermediate teachers, Phase 2, seven-task average54.1454.56, 53.7954.16 ± 0.39[53.21, 55.12]
ExOPD, published score cited in the paper53.79———

Q2

Thank you for asking about the rationale for the gate and its alternatives. Regarding "Why the largest gap rule": The normalized residual spectrum measures the relative reward-reconstruction gap remaining in each domain after Phase 1. We use its largest adjacent gap to identify a separation between domains with larger and smaller residuals, without prescribing the number of active domains. The largest gap provides the widest interval between adjacent residual levels, and its midpoint places the threshold equally far from the two endpoints. Appending zero also allows all domains to remain active when the largest separation is between the residuals and this floor, as occurs on G-OPD. This gives a simple allocation rule computed entirely from unlabelled calibration data.

Regarding "compare with a fixed threshold": The reported spectra give the following allocations:

SettingNormalized residualsLargest-gap thresholdActive domainsFixed thresholds producing the same allocation
Open-MOPDMath: 0.128, Code: 0.052, IF: 0.0150.090Math0.052≤ϵ<0.128
G-OPDMath: 0.094, Code: 0.0900.045Math and Code0≤ϵ<0.090

Thus, suitable fixed thresholds reproduce our selections. Indeed, a common threshold in [0.052,0.090) works for both settings. These intervals show that the allocations do not depend on an exact threshold value. The role of the largest-gap rule is to obtain a cutoff automatically from the observed spectrum, without tuning ϵ on downstream benchmark accuracy. The current experiments support this automatic choice, while leaving broader comparisons across alternative threshold rules for further evaluation.

Regarding "writing back all branches": Table 1 compares the following strategies from the same Phase 1 initialization:

Open-MOPD configurationMacro
Phase 1 merge32.52
TIES, all branches30.96
DARE, all branches31.42
Math branch without shrinkage, γ=131.80
RGBM, Math branch with γ=0.12533.03

Both evaluated all-branch strategies fall below Phase 1. Admitting only Math also requires controlling the update magnitude: its full update scores 31.80, whereas the magnitude selected by the unlabelled diagnostic reaches 33.03. These results support selective writeback together with shrinkage. Because TIES and DARE also change the merging operator, these comparisons evaluate the complete strategies and do not isolate the gate alone. On G-OPD, the spectrum activates both available domains, and their shrunken writeback improves the seven-task average from 53.34 to 54.14. Thus, RGBM can retain all branches when the spectrum calls for it.

Regarding "more than 3 active domains": Our current experiments involve at most three experts and at most two simultaneously active branches. We therefore do not yet provide evidence for more than three active domains. We will make this experimental scope explicit, with larger active sets remaining an important direction for further evaluation.

Q3

Regarding "equal-weight average" and RGBM: Thank you for highlighting this discrepancy. The coefficients are indeed close, but this does not imply identical predictions. A two-teacher equal-weight average is

=W_{\mathrm{ref}}+0.5\tau_{\mathrm{math}}+0.5\tau_{\mathrm{code}},$$ where $\tau_d=W_d-W_{\mathrm{ref}}$. In the converged G-OPD setting, RGBM Phase 1 uses coefficients $(0.511,0.505)$, giving $$\theta_{\mathrm{RGBM}}=W_{\mathrm{ref}}+0.511\tau_{\mathrm{math}}+0.505\tau_{\mathrm{code}} =-0.016W_{\mathrm{ref}}+0.511W_{\mathrm{math}}+0.505W_{\mathrm{code}}.$$ The corresponding parameter weights are: | Merging method | Weight on $W_{\mathrm{ref}}$ | Weight on $W_{\mathrm{math}}$ | Weight on $W_{\mathrm{code}}$ | | --- | ---: | ---: | ---: | | Two-teacher equal-weight average | $0$ | $0.5$ | $0.5$ | | RGBM Phase 1 | $-0.016$ | $0.511$ | $0.505$ | Their parameter difference is $\theta_{\mathrm{RGBM}}-\theta_{\mathrm{avg}}=0.011\tau_{\mathrm{math}}+0.005\tau_{\mathrm{code}}$. The values 0.011 and 0.005 scale full expert task vectors, so they are not uniform increments to individual parameters. Consequently, the decimal differences alone do not establish how close the models' predictions are. RGBM obtains its coefficients by optimizing the residual objective on unlabeled calibration data and achieves a seven-task average of 57.35 in the converged setting. To illustrate the broader point that close coefficients can yield different scores, we provide the following Open-MOPD comparisons. We added eight Open-MOPD ablations to compare the merging objective, domain aggregation, and coefficient parameterization. C-vec L (RGBM Phase 1) achieves the highest Macro, and each raw-residual configuration outperforms its normalized counterpart in Macro, supporting the Phase 1 objective. Labels A / C denote uniform domain weighting / minimax, sc / vec a shared scalar / independent coefficient vector, and L / Lb raw / energy-normalized residuals. Coefficients enter task-vector merging as $\theta=W_{\mathrm{ref}}+\sum_d\alpha_d\tau_d$. Coefficients are ordered Math, Code, IF, and Macro weights domains equally: | Configuration | Merge coefficients (Math / Code / IF) | Math | Code | IF | Macro | | --- | --- | ---: | ---: | ---: | ---: | | `A-sc L` | 0.626 / 0.626 / 0.626 | 24.97 | 23.28 | 47.64 | 31.96 | | `A-sc Lb` | 0.501 / 0.501 / 0.501 | 24.38 | 23.36 | 44.86 | 30.86 | | `A-vec L` | 0.527 / 0.609 / 0.821 | 24.11 | 23.59 | 47.99 | 31.90 | | `A-vec Lb` | 0.840 / 0.260 / 0.841 | 24.19 | 19.57 | 48.00 | 30.59 | | `C-sc L` | 0.579 / 0.579 / 0.579 | 24.97 | 23.49 | 45.10 | 31.19 | | `C-sc Lb` | 0.477 / 0.477 / 0.477 | 21.41 | 19.76 | 45.79 | 28.99 | | `C-vec L` | 0.597 / 0.620 / 0.800 | 23.86 | 24.09 | 49.60 | **32.52** | | `C-vec Lb` | 0.724 / 0.426 / 0.630 | 24.22 | 20.73 | 47.02 | 30.66 | A particularly relevant comparison is A-sc Lb versus C-sc Lb. Both use shared-scalar coefficients and energy-normalized residuals, differing in uniform domain weighting versus minimax aggregation. Their coefficients are 0.501 and 0.477, only 0.024 apart per expert, yet their Macro scores differ by 1.87 points. Math and Code differ by 2.97 and 3.60 points, respectively, while IF changes in the opposite direction. This demonstrates that nearby coefficients can yield materially different cross-domain performance. Across these eight ablations and the equal-weight baseline, the three-domain mean raw calibration residual has a Spearman correlation of approximately −0.92 with Macro. This supports the residual as an informative signal for coefficient selection within the evaluated configurations, with benchmark accuracy used only for subsequent validation. The complete coefficients, scores, residuals, and figure are available at https://anonymous.4open.science/r/0f593a7b3935599f-DD7B/.
Reviewer_3Ri2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies MOPD and proposes a RGBM method. The authors show that, under a local second-order approximation, the MOPD objective corresponds to Fisher-weighted parameter merging, motivating a merge-before-distill strategy. RGBM first computes a zero-training minimax merge in the task-vector subspace, then uses an implicit-reward residual spectrum to select domains for isolated refinement and safely writes back their updates with shrinkage. Experiments show improved performance of RGBM over standard OPD/MOPD baselines.

Critical strengths
  • The idea of using mdoel merging for MOPD seems interesting.

  • The proposed method is theoretically motiviated.

Critical weaknesses
  • The theoretical results rely heavily on local approximations. Theorem 1 relies on a local second-order approximation around the teacher checkpoints, and Proposition 2 relies on a local first-order approximation around the reference model. Therefore, when specialist checkpoints are sufficiently separated such that no θ\theta is simultaneously close to all teachers and also the reference model. This becomes the greatest limitation of the theretical parts.

  • In Table 1, Phase 2 admits only the Math branch, but the performance gain of math is only +0.07 and lower than Code and IF. Although Section 4.5 try to address this problem, it still seriously violates the theoreitcal intuition.

  • The writing is a bit messy. For example, in Section 4.2 the authors discuss Table 1, then Section 4.3 and 4.4 move on to Figure 3 and Table 2, but Section 4.5 goes back to Table 1 without even citing it. The manuscript would benefit from additional proofreading.

Questions for authors

Please refer to weaknesses.

4 comments

W1

Thank you for raising the scope of the local analysis. Our study follows the shared-reference specialist setting used in model merging and MOPD: the experts have compatible architectures and are fine-tuned from the same reference checkpoint. The theory provides a local connection between distillation, Fisher-weighted merging, and implicit reward superposition within this setting.

We agree that a shared reference checkpoint alone does not guarantee that both local approximations are accurate. If the specialists are sufficiently separated, the connection between the residual surrogate and the Fisher quadratic can weaken. The RGBM computations remain well-defined in that case, but computational well-definedness does not establish the validity of the theoretical approximation. We will state this distinction explicitly and retain the restriction to shared-reference specialists.

W2

Thank you for asking how the Math branch relates to per-domain gains.

(1) The branch is trained on Math data but updates the entire model's shared parameters, so its effects can extend to Code and IF. The domain selected from the residuals therefore need not receive the largest accuracy gain. The residual spectrum is used to select candidate refinement branches; it does not predict the magnitude or domain distribution of post-writeback accuracy gains. RGBM assesses aggregate performance across domains. Theorem 1 and Proposition 2 connect distillation, parameter merging, and reward superposition, while explaining post-writeback accuracy also requires examining shared-parameter effects and shrinkage. To isolate the effect of writeback magnitude, Table 1 and Appendix Table 8 compare different magnitudes for the same Math branch:

ConfigurationMathCodeIFMacro
Phase 123.8624.0949.6032.52
Math writeback (γ=0.5)25.4223.4848.0832.32
RGBM (γ=0.125)23.9324.6350.5233.03

The larger Math writeback gains 1.56 points in Math but lowers Code and IF, reducing Macro. The smaller, label-free-selected magnitude improves all three domains and raises Macro by 0.51. Appendix Table 9 also reports gains of 0.58 to 1.83 points for the independently trained Math branch on its own domain across objectives. Thus, the branch learns Math, while shrinkage balances its update against other capabilities, consistent with Table 1's aggregate gains.

(2) To test whether the overall writeback gain is driven by a single benchmark, we additionally reaggregated the paper's reported scores. With models fixed, we removed the same benchmark from both phases for each of Table 3's six benchmarks, averaged remaining benchmarks within domains, and retained equal domain weights. Phase 2 remains ahead in all six cases (+0.400 to +0.627), showing that the average writeback gain persists under changes in benchmark composition.

(3) Additional objective ablations examine the cross-domain tradeoff. We also compared raw-residual and energy-normalized objectives with identical experts, calibration data, reward second moments, and evaluation protocol. Here, C denotes minimax aggregation and vec denotes independent expert coefficients. C-vec L minimizes the largest raw reward-reconstruction residual, whereas C-vec Lb first normalizes each domain’s residual by its expert reward energy E[rd2]\mathbb{E}[r_d^2].

Phase 1 configurationMathCodeIFMacro
C-vec L (RGBM)23.8624.0949.6032.52
C-vec Lb24.2220.7347.0230.66

Normalization improves Math by 0.36 points but reduces Code by 3.36 and IF by 2.58, lowering Macro by 1.86. The raw-residual configuration remains ahead after every benchmark exclusion (+0.605 to +3.108). This provides complementary evidence that an improvement in Math alone does not determine the quality of cross-domain integration.

Beyond this pair, we provide eight Open-MOPD ablations varying domain aggregation, coefficient parameterization, and residual normalization. The complete coefficient and domain-score table, together with the corresponding figure, are available at https://anonymous.4open.science/r/0f593a7b3935599f-DD7B/README.md. These experiments further examine how the residual objective and merge coefficients relate to cross-domain performance.

W3

Thank you for identifying these presentation issues. We agree that clearer transitions and cross-references would help. Section 4.2 presents the main results in Table 1, Section 4.3 examines continued joint-training trajectories, and Sections 4.4–4.5 explain writeback through branch-selection and shrinkage ablations. We will connect these sections explicitly and cite the relevant Table 1 configurations and Appendix Table 8 for each ablation. This will clarify the progression from merged initialization through continued training to selective writeback, consistently specifying initialization and update sources. We will incorporate these and other revisions after considering all reviewer feedback.

Dear Reviewer 3Ri2,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

Thank you for the detailed rebuttal and additional analyses. I appreciate the authors' efforts, but my main concerns remain unresolved, and I will keep my assessment towards weak rejection. Please see detailed reasons below.

W1. The authors acknowledge the limitations of the local approximations, which substantially weakens the practical significance of the theoretical contribution.

W2. The additional experiments do not directly establish whether the residual spectrum reliably identifies the most beneficial refinement branches.

W3. I appreciate the authors' willingness to improve the presentation, but the pdf manuscript remains unchanged.

Reviewer_Nn4aOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the connection between model merging and multi-teacher on-policy distillation. The authors show that the distillation objective can be approximated by a Fisher-weighted merging objective under certain assumptions. Based on this connection, they propose RGBM, which first merges specialists using a minimax residual objective and then selectively trains and merges additional branches based on domain residuals. Experiments on two LLM benchmarks show improvements over several merging and distillation baselines.

Critical strengths

1.The connection between model merging and multi-teacher distillation is interesting. The Fisher-based analysis provides a useful perspective on why parameter merging can be effective.

2.The proposed framework is reasonably well motivated. Using domain residuals to decide which specialists need further adaptation is an interesting approach to reducing cross-task interference.

3.The experiments include useful ablations on branch selection and shrinkage. The results also show that the first-stage merge can achieve competitive performance without additional gradient-based training.

Critical weaknesses

1.The connection between the theoretical analysis and the actual method is not fully convincing. The theory derives a Fisher-weighted objective, but Phase 1 uses uncentered reward moments and a minimax objective instead. It is unclear how well the theoretical optimality claim applies to the proposed implementation.

2.The linearity diagnostic in Eq. (7) uses R² from a fitted regression. However, a high R² does not necessarily mean that the predicted and actual log-probability ratios are numerically close. The paper should better justify why this criterion is sufficient for safe branch merging.

3.The experimental evaluation is somewhat limited. The method is tested on only two model families with two or three specialists. Moreover, the improvements from Phase 2 are relatively small, and the main results lack uncertainty estimates. More evidence is needed to establish the robustness of the proposed gating strategy.

Questions for authors

please see weaknesses

3 comments

W1

Thank you for asking us to clarify the connection between theoretical optimality and implementation. Theorem 1 establishes optimality for the local Fisher quadratic under fixed domain weights. We do not claim that the implemented minimax merge minimizes the original MOPD objective. Phase 1 makes two adaptations for computational tractability and domain weighting.

(1) Theorem 1 derives Fisher-weighted merging and its closed-form solution under a local second-order approximation. Full parameter-space Fisher estimation, storage, and optimization are prohibitive for large models, so we restrict merging to the subspace spanned by expert task vectors. Proposition 2 connects combinations in this subspace to implicit reward superposition, enabling small reward second-moment matrices to be estimated through forward scoring. Under the corresponding linearization, Appendix A.3 relates the projected Fisher to conditional reward covariance. Uncentered second moments add a positive semidefinite reward-mean term, providing a conservative surrogate for the projected Fisher quadratic under the corresponding linearization.

(2) The Fisher objective assumes domain weights p_d, which practical integration tasks often do not uniquely specify. Minimax controls the worst-domain residual: max⁡p∈ΔK∑dpdLd=max⁡dLd\max_{\mathbf p\in\Delta_K}\sum_d p_d\mathcal L_d=\max_d\mathcal L_d. It thus provides robustness to unknown domain weights. Regularization yields a convex program with a unique optimum (Appendix A.3). Theorem 1's optimality concerns the local Fisher quadratic approximation, while Phase 1 optimally solves the regularized residual program. Specifically, Phase 1 makes two additional choices: replacing conditional reward covariance with uncentered second moments, and replacing fixed domain weighting with worst-domain aggregation. It exactly solves the resulting regularized residual program. We will distinguish this surrogate optimum from the second-order Fisher optimum in Sections 2.3 and 3.1.

W2

Thank you for highlighting this distinction.

Regarding "a high R² does not necessarily mean that the predicted and actual log-probability ratios are numerically close": R^2 in Eq. (7) measures linear consistency between the updated log-ratio and Phase 1 reward superposition. Because the regression fits a slope and intercept, a high R^2 alone does not ensure numerical agreement, which also depends on those quantities and direct prediction error. We will call it a “linear consistency diagnostic” in Section 3.3 and clarify the scope of 1-R^2.

Regarding "why this criterion is sufficient for safe branch merging": We agree that this criterion is not sufficient to guarantee harmless writeback. Appendix A.4 shows that shrinkage reduces an upper bound on the Taylor remainder under the stated smoothness assumptions. This does not establish monotonicity of the actual approximation error or preservation of downstream accuracy. Eq. (7) is therefore an empirical screening criterion: it selects the largest candidate shrinkage factor that passes the fitted linear-consistency test. Appendix Table 8 supports this rule: \gamma=0.125 gives 1-R^2=0.0701, satisfies the tolerance of 0.10, and achieves Macro 33.03. At \gamma=0.25, the diagnostic rises to 0.1749 and Macro falls to 32.01. Without shrinkage, \gamma=1 gives 0.7231 and Macro 31.80. The label-free rule therefore selects the best-performing magnitude on the evaluated grid, with accuracy used only for subsequent validation. This supports its use in screening unfavorable updates within that grid. We additionally assessed tolerance sensitivity by holding Table 8's diagnostic values fixed and recomputing selection. On \Gamma={0.125,0.25,0.5,0.75}, every 0.0701\le\delta_{\mathrm{tol}}<0.1749 selects \gamma=0.125. The tolerance can decrease by up to 29.9% or increase by less than 74.9% without changing the configuration. Together, these theoretical, ablation, and sensitivity results support the rule in the evaluated settings. In the revision, we will tighten the relevant statement to: "a theoretically motivated empirical screening rule that is effective in the evaluated settings."

W3

Thank you for your suggestions on experimental scope and robustness.

Regarding "experimental evaluation is somewhat limited": Beyond the three SmolLM3-3B experts and two Qwen3-4B experts, Appendix B.4 evaluates Math+Code, Math+IF, and Code+IF pairs. Appendix B.6 covers intermediate and converged teachers, with only Phase 1 evaluated for the latter. These comparisons vary model family, expert combination, and training stage, testing dependence on particular settings. We will state this scope explicitly, with larger models and more experts remaining directions for further evaluation.

Regarding "uncertainty estimates": In additional three-seed results, Phase 2 has a higher mean in both settings. The following mean ± sample standard deviation values supplement Tables 1 and 2:

SettingPhase 1 scoresPhase 1 mean ± stdPhase 2 scoresPhase 2 mean ± stdMean difference
Open-MOPD, Macro32.52, 32.65, 32.0132.39 ± 0.3433.03, 32.87, 33.3133.07 ± 0.22+0.68
G-OPD intermediate teachers, seven-task average53.34, 53.93, 53.6153.63 ± 0.3054.14, 54.56, 53.7954.16 ± 0.39+0.54

Regarding "the robustness of the proposed gating strategy": We provide evidence about both the stability of branch selection and the resulting performance gain. For selection stability, Appendix C.6 reports coefficient bootstrap standard deviations below 0.012 for N≥1024, with residual ordering preserved at over 99.9% confidence. Appendix C.4 further shows that every threshold 0.052≤ϵ<0.128 selects the same Math branch in Open-MOPD. Thus, the selected branch is stable under the reported calibration resampling and across a range of thresholds. To assess whether the Phase 2 gain depends on a particular benchmark, we additionally removed one benchmark at a time from both phases and recomputed their aggregate-score difference using the paper’s reported scores, with both models fixed. Open-MOPD retains equal domain weights after averaging the remaining benchmarks within each domain; G-OPD averages the remaining six tasks equally.

SettingReported Phase 2 − Phase 1Difference after removing one benchmarkPositive differences
Open-MOPD, Macro+0.51+0.400 to +0.6276/6
G-OPD intermediate teachers, seven-task average+0.80+0.642 to +1.1127/7
The gain remains positive after every benchmark removal in both settings. The bootstrap and threshold analyses support branch-selection stability, while the benchmark-removal analysis shows that the observed aggregate gain does not depend on any single benchmark. Together, these results strengthen the evidence for the gating strategy within the evaluated settings.

Dear Reviewer Nn4a,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.