acceptodds
Under review as a conference paper at ICLR 2027

Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs

Abstract

Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such routing drift is often interpreted as routing failure, raising a fundamental question that remains unclear: does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair? We investigate these questions across DeepSeekMoE, OLMoE, and Qwen3-MoE proposing a routing analysis toolkit for controlled counterfactual interventions and token-level analysis. By crossing source and merged router inputs and parameters, we attribute most expert reassignments to input shifts rather than parameter changes at the same layer. However, source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and different expert selections can produce directionally similar mixture outputs. We therefore operationalize routing failure as task loss recoverable under a specified routing intervention, with non-routing parameters fixed. These tests detect recoverable loss under deliberate router corruption, whereas source-route restoration does not establish reliable task benefits in the evaluated merged models. Motivated by these, we propose Selective Router Repair (SRR) as a case study, and find that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections. Together, these findings show that routing drift alone is insufficient evidence of routing failure: source-informed corrections must be judged by their task-level intervention effects. The analysis toolkit and SRR code are released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, MoE, Routing Drift, Model Merge, Routing Failure

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gdU5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether routing drift diagnoses routing failure in merged MoE language models. The authors cross source and merged router inputs and parameters to identify local sources of route changes, then use fixed-expert output comparisons, token-likelihood analyses, and task-level routing interventions to test whether those changes correspond to recoverable performance loss.

Across DeepSeekMoE, OLMoE, and Qwen3-MoE, most expert reassignments are locally attributed to representation changes, while structural drift poorly predicts gains from source-route restoration. Controlled router corruption produces detectable recoverable loss, whereas the tested natural routing alternatives do not establish reliable task benefits. SRR is evaluated as a case study of source-informed repair, but its local supervision and task evaluation do not establish reliable improvement. The main contribution is the distinction between structural drift and intervention-supported task harm, together with a diagnostic toolkit and supporting empirical evidence.

Critical strengths
  1. The paper introduces a useful and operational diagnostic distinction. Rather than treating source/merged routing disagreement as an error label, it defines failure through task loss recoverable by a specified routing intervention with non-routing parameters fixed. Analyses across three architectures show that structural drift metrics poorly predict the tested restoration gains. This can help subsequent work avoid optimizing source agreement without evidence of task utility and is the main contribution supporting acceptance.

  2. The negative findings have meaningful controls and statistical support. Matched random-route controls make the mixture-output similarity result more informative, and deliberate router corruption demonstrates that the task interventions can detect substantial recoverable loss. Paired task evaluation, prompt-level resampling that preserves token/layer dependence, and multiplicity correction strengthen the evidence. The authors also clearly distinguish fitting SRR targets from demonstrating actual recovery, giving the negative results independent scientific value.

Critical weaknesses
  1. The natural task tests do not tightly bound possible recovery. Some intervals in Table 1 still allow accuracy improvements of several percentage points; for example, OLMoE–Average has an interval of −1.56 to +3.91 pp. The corresponding Qwen3 tests use only 64 ARC items. An interval containing zero establishes a lack of demonstrated improvement, but does not exclude practically useful recovery. The paper acknowledges this limitation, yet should more explicitly state what effect sizes the existing tests rule out so readers can assess the strength of the negative evidence.

  2. The evidence is conditional on the reference intervention and fixed candidate construction. Source routes need not be suitable for merged experts, and source or frozen LC schedule replay differs from an online policy recomputing routes on intervened hidden states. SRR results are conditional on a specified construction protocol, update scale, and fixed checkpoints. Five deterministic evaluations do not quantify variation across repair samples or seeds. The experiments therefore support limitations of the tested metrics and candidates; conclusions about other routing policies or source-informed repair should remain appropriately scoped.

Questions for authors

The following clarification requests could materially affect my assessment:

  1. What recovery magnitudes do the existing natural-task intervals rule out, and what effect size is considered practically meaningful? Can the 64-item Qwen3 test be supplemented with an existing larger matched evaluation, or explicitly presented as low-resolution evidence?

  2. Please summarize which conclusions come from fixed-state router swaps, route-schedule replay, and online/end-to-end execution. To what extent does frozen LC schedule replay test the usefulness of calibrated routing? Are there existing task-level interventions with an online or calibrated policy that help distinguish an unsuitable reference from a lack of diagnostic value in drift metrics?

  3. Are SRR results available across different construction samples or repair seeds? If not, please emphasize that the negative result is conditional on the fixed candidates and configurations, rather than interpreting repeated deterministic evaluation as evidence of repair-process stability.

3 comments

We sincerely thank you for the thoughtful and constructive review.

We particularly appreciate the recognition of our intervention-based diagnostic framework, controlled experiments, and statistical analyses. We agree that the interpretation of negative results should carefully distinguish the absence of demonstrated recovery from evidence that recovery is impossible. We address the concerns below and will clarify the corresponding statements in the revised manuscript.

The revised manuscript has been submitted.

W1 & Q1:

We agree that confidence intervals containing zero do not establish equivalence to native routing or exclude potentially useful recovery. Our conclusion is therefore limited to the absence of demonstrated improvement under the evaluated routing interventions, rather than the absence of recoverable routing loss.

To clarify the effect sizes supported by the existing evidence, the upper endpoints of the reported 95% confidence intervals for science-domain source-route replay are +1.56/+2.34 percentage points (pp) for DeepSeekMoE and +3.91/+3.52 pp for OLMoE, under Average/Task Arithmetic merging, respectively (Table 1). Thus, the intervals remain compatible with a +1 pp improvement, which we consider a useful illustrative reference for practical relevance rather than a universal threshold across tasks. Some settings remain compatible with improvements of several percentage points. The broader domain results in Appendix A.7 likewise do not support a general claim of negligible recovery.

We also acknowledge the limited statistical resolution of the 64-item Qwen3 source-route replay. We will explicitly characterize these results as low-resolution supporting evidence, rather than use them to claim that practically meaningful recovery has been ruled out. Overall, our findings indicate that the tested natural routing references do not establish reliable task recovery; they do not imply that other routing policies cannot recover task performance.

W2 & Q2:

Thank you for highlighting the importance of distinguishing different intervention types.

The experiments address complementary questions:

  • Fixed-state interventions (Section 3.1): Crossing source/merged router inputs and parameters identifies the local contributions of representation and router-parameter changes to expert reassignment. This establishes local attribution, not task-level recovery.
  • Token-local and route-schedule interventions (Sections 3.2 and 4.2): Token-local replacements measure the effects of specified routing changes, with subsequent computation allowed to respond. Task-level source/LC replay evaluates cached routing schedules while keeping non-routing parameters fixed. These experiments assess the specified alternatives, not arbitrary online routing policies.
  • End-to-end router execution (Sections 5–6): Table 2 evaluates the SRR checkpoints, alongside HARC, through complete model execution, with routing recomputed on the hidden states produced during inference. These results therefore assess actual checkpoint-level task effects rather than frozen route replay.

Importantly, frozen LC schedule replay is not equivalent to executing the LC-calibrated router online, since the latter would recompute routing decisions on the intervened model's hidden states. Accordingly, the LC results only establish the effects of the specified frozen schedule. Although SRR and HARC provide end-to-end task evaluations, they do not constitute a matched online-versus-frozen comparison of the same LC policy. We therefore agree that the current experiments do not fully disentangle an unsuitable source reference from the potential diagnostic value of drift metrics for other routing-policy families. Our conclusion is specifically that the evaluated structural drift metrics poorly predict the tested source-restoration gains, not that routing drift is uninformative for every possible intervention. We will make these intervention-specific distinctions more explicit in the manuscript.

I will remain my rating.

Reviewer_iju9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies the impacts of model merging on MoE routers. After merging domain-specialized model variants of the same base model, it studies router “drift”. That is, the difference between routing in merged model on a domain and the original domain-specialized source model. They define routing “failure” as the task loss recoverable when using the best possible routing. Under this definition, they note that routing drift comes mostly from representation itself drift (as opposed to the router weights themselves). They additionally find that routing drift and failure are not as strongly related as expected, and that interventions into MoE routers are insufficient to address the multi-task interference caused from merging.

Critical strengths
  1. Section 3 does a nice, clean decomposition of router-based or representation-based “routing drift”. It’s cleanliness is practical for understanding multi-task interference, regardless of MoE
  2. Consistent vocabulary helps with the understanding of a reasonably difficult topic to discuss. I appreciate the reuse of “induce”, “drift”, “repair”, “specialized”, “failure” and more terms, they help the reader follow how the parts fit together. I would still recommend further reduction of jargon to make it easier to follow for potential readers.
  3. Strong reproducibilty with anonymous github provided and thorough experimental details in the appendix
Critical weaknesses
  1. Very confusing language around “Source”: By definition, a merged model has multiple source models. This work was very unclear as to how the merged models were constructed and also what the single “source” referred to. This is wholly central the paper. My current understanding (mostly from the very-late Section 5.1) is that 4 S_d were trained from B and merged into M. For a data point x \in a domain D, the analysis compares S_D(x) to M(x). This HAS to be established earlier to make sense of the work. Currently, it took me until Section 5.1 to understand what the point of the paper even was, as the motivation in abstract/intro was unclear.
  2. The potential impact of these findings is currently unclear and the central contributions read as non-findings. It seems that the conclusion is “we were not able to disentangle MoE-router-specific issues from broader multi-task interference”. The motivation for studying routing “drift” from merging is not well-established, let alone router “repair”.
  3. Model merging is substantially broader than this narrow form of multi-task merging. As I understand it, model merging is also very common for the purposes of distributed model learning, generalization/robustness, mitigating forgetting, and more. And even this multi-task merging setup is very specific. Source–merged routing disagreement is certainly very dependent on the relationship between the source models, and I’m unconvinced that the analysis and proposed method generalize to more settings.
  4. Section 4 is quite convoluted; I did not understand it well or understand how it fit into the paper.
  5. AI use statement is low-effort. My understanding of the intention of the ICLR AI policy is that authors offer significantly more transparency into the role of LLMs in the research process.
Questions for authors

N/A

2 comments

Thank you for the review.

Before addressing the questions, two core errors in the review’s comments require correction.

  • First, Section4.1, Eq. (2) (lines 236–248), defines recoverable task loss relative to a specified alternative routing policy, with non-routing parameters fixed. The review’s “best possible routing” interpretation is incorrect: the definition contains no maximization over policies.

  • Second, §3.2 (lines 195–200) explicitly leaves open task loss recoverable under other routing policies. The statement that routing interventions are insufficient to address merging interference therefore overstates our conclusion. Our claim concerns whether routing drift diagnoses failure, not whether routing repair is ineffective.

Both distinctions are explicit in the submitted manuscript, not qualifications introduced during rebuttal.

We address the remaining concerns below.


W1:

For an input from domain dd, “source” denotes the corresponding specialist SdS_d among the specialists sharing base BB and merged into MM. We agree that this setup should precede Section 3 rather than first being formalized in §5.1, and will move the definition and domain-to-specialist mapping to the Introduction.

W2:

Our question is whether source-relative routing disagreement justifies a diagnosis of routing failure (§§1–2), not simply whether SRR improves scores. In 77.7–96.9% of changed-route events, input-only replacement changes the source expert set while router-only replacement does not (§3.1). Full-distribution JS predicts source-restoration NLL gains at near-chance AUROC (0.47–0.52) (§3.2). Source/native mixture cosine exceeds overlap- and weight-matched random controls by 0.112–0.147, with positive paired confidence intervals in all six settings (§3.3).

These controls distinguish local input and router-parameter effects and measure the consequences of routing interventions with non-routing parameters fixed. This is not a complete decomposition of multi-task interference, but it does directly test routing-specific effects. The diagnostic evidence does not depend on SRR improving task scores. The practical implication is that source-route agreement alone is not a sufficient criterion for successful repair. We will emphasize this distinction in the Introduction.

W3:

Our empirical scope is same-base, domain-specialist MoE merging: three architectures under Average and Task Arithmetic for the structural analyses, and four merging methods for downstream repair evaluation. We agree that this does not establish generalization to arbitrary source-model relationships or other merging settings, and will state this scope explicitly.

W4:

Section 4 connects the structural analyses to task consequences through paired routing interventions. Its positive control detects imposed routing loss: clean-route replay recovers 12.50–14.84 pp after four-layer corruption and 26.56–32.81 pp after full-layer corruption, with all eight comparisons passing Holm correction across twelve tests (§4.2). Natural routing alternatives are evaluated separately; inconclusive results establish neither harmlessness nor optimality. We clarify this connection at the start of the section.

W5:

We used Codex for virtual-environment setup, model and dataset downloads, data preprocessing, launching training scripts, running evaluations, and manuscript polishing. AI tools also assisted with figure editing. We will specify these uses in the disclosure. The authors reviewed the assisted work, verified the results and claims, and take full responsibility for the submission.

Hello authors,

I appreciate the effort to improve the presentation of the work and have updated my "presentation" score to a 2. I consider W1 & W5 mostly addressed.

However, my most important concerns about the paper remain: Motivation & Impact (W2, W3) remain very much not addressed, in my opinion. You mention that "The practical implication is that source-route agreement alone is not a sufficient criterion for successful repair." But this feels more like a negative result (not necessarily a bad thing) and the research space it affects seems very niche. I have not been convinced that "drift" in such multi-task merging of MoE LLMs is a impactful problem to study and is different than studying other consequences of interference during merging.

Reviewer_98GbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether routing drift after merging MoE models indicates routing failure. The results show that most local routing changes come from shifts in router inputs rather than router parameters. Routing drift also poorly predicts whether restoring source-model routes will help, and different routes can produce mixture outputs with similar directions.

The authors propose testing routing failure by measuring how much task loss can be recovered through a specified routing intervention, while keeping non-routing parameters fixed. They also introduce Selective Router Repair (SRR) as a case study, using source-model token likelihoods and routing information to guide selective router updates. However, the experiments do not show reliable task improvements from SRR across the evaluated settings.

Critical strengths

1. The paper separates routing drift from routing failure. A different expert selection does not necessarily mean worse routing. The authors test whether changing routes improves task performance while keeping non-routing parameters fixed. I find this useful because it tests whether a routing intervention helps, rather than treating agreement with source routes as evidence of better routing.

2. The analysis uses useful controls across three MoE architectures. The matched random routes in Section 3.3 control for expert overlap and routing weights. Source-route mixtures remain more similar to native mixtures than these controls, which strengthens the output-similarity finding. The controlled corruption experiments in Section 4 also show that the evaluation can detect recoverable task loss when routing is deliberately damaged. These controls make the analysis more convincing.

Critical weaknesses

1. The task-level results do not clearly show why SRR fails or which components help. SRR combines token weighting, expert-pair selection, clipping, ridge regression, and a fixed update scale. However, I do not see task-level ablations that show which choices help. The weighting ablation in Appendix D.3 only reports fitting error. Section 5.3 tests source-informed corrections and measures how earlier updates change later router inputs. These are useful checks, but they do not show which factors explain the task results. The authors could compare weighting and pair-selection rules, vary the update scale, and refit later-layer corrections using updated hidden states. Without these tests, it is unclear whether the weak results mainly reflect poor source supervision or limitations of the SRR design.

2. The experiments do not clearly establish the performance loss that routing repair is meant to recover. Table 2 compares repaired models with their merged parents, which is appropriate for testing repair gains. However, I do not see a clear comparison of the base model, domain specialists, and merged models on the same domain tasks. This would show whether merging loses specialist performance and which tasks are affected. Such gaps would not by themselves identify routing as the cause. The authors should connect them to the routing-intervention results to show whether the tested repairs recover any of that loss. Without this comparison, it is difficult to judge the practical importance of the repair results. The specialist checkpoints, merge coefficients, and HARC configuration also need clearer documentation so that the comparisons can be reproduced.

Questions for authors

1. Performance before and after merging. Could the authors compare the base model, domain specialists, and merged models on the same domain tasks? How much of the performance lost during merging do the tested routing interventions recover?

2. Task-level ablations of SRR. Could the authors report task-level comparisons of uniform versus likelihood-based weighting, random versus selected expert pairs, and different update scales? The fitting errors in Appendix D.3 do not show which choices help task performance.

3. Sequential fitting. Could the authors test a sequential fitting strategy that replaces hM(ℓ)h_M^{(\ell)} with the updated hidden states hupdated(ℓ)h_{\mathrm{updated}}^{(\ell)}, obtained after applying the router updates in earlier layers? Specifically, could the authors compare the current fitting objective

min⁡v∑tρt(hM,t(ℓ)⊤v−r~t)2+λ∥v∥22\min_v \sum_t \rho_t (h_{M,t}^{(\ell)\top}v-\tilde{r}_t)^2+\lambda\|v\|_2^2 with the same objective using

hupdated,t(ℓ)h_{\mathrm{updated},t}^{(\ell)} instead? This experiment could help determine whether SRR's limited performance comes from weak supervision signals or from fitting router updates on outdated hidden states.

4. Settings where routing repair helps. Could the authors repeat the routing analysis in a setting where HARC or another repair method reliably improves task performance? Does routing drift still poorly predict source-route restoration gains in that setting?

5 comments

We sincerely thank the reviewer for the constructive feedback and recognition of our core contributions. To address the concerns, we conducted task-level SRR ablations, matched specialist–parent evaluations, and a comprehensive audit of 12 HARC settings.

The revised manuscript has been submitted.

W1 & Q2:

We agree that fitting error alone cannot establish downstream utility. We therefore evaluate three key design choices on OLMoE–Average: (1) likelihood-weighted versus uniform-positive/all-uniform fitting, (2) selected versus three activity-matched random expert-pair assignments, and (3) update scales η∈{0.015625,0.0625,0.25}\eta\in\{0.015625,0.0625,0.25\}. All variants are evaluated on the same 35,326 items across eight benchmarks, with other settings fixed. Sequential fitting is examined separately in Q3.

Complete task results

Table R1. All eight task scores (%). Avg. is the equally weighted mean of the eight task scores. Differences below are computed from full-precision records before rounding. The final row is a computed summary of R1–R3, not an eleventh checkpoint.

VariantMMLUHellaSwagARC-CARC-EPIQAWinoGrandeBoolQGSM8KAvg.
Parent55.46980.15357.08281.52481.82873.71780.39858.15071.040
Original SRR55.46980.15357.16781.43981.82873.08680.39859.89471.179
Uniform-positive fitting55.41980.14357.16781.56681.71973.63980.42858.22671.038
All-uniform residual fitting55.40580.09457.08281.48181.71973.48180.39858.68171.043
Random pairs: R155.55580.09457.08281.60881.77473.56080.48959.36371.190
Random pairs: R255.42780.19357.08281.48181.82873.40280.52058.60571.067
Random pairs: R355.51980.21357.08281.52481.77473.56080.61258.52971.101
Smaller update55.49180.10456.99781.48181.77473.56080.42858.83271.083
Larger update55.55580.16357.25381.48181.71973.63980.48958.30271.075
Sequential input refresh55.49180.12356.74181.31381.77473.48180.42858.98471.042
Mean of R1–R3 (computed)55.50080.16757.08281.53881.79273.50780.54058.83271.120

These results retain the task trade-offs rather than reporting only the aggregate. For example, original SRR changes GSM8K by +1.744 pp but WinoGrande by −0.631 pp relative to Parent. The three random-pair candidates span +0.027 to +0.150 pp in aggregate gain. These point estimates motivate paired comparisons; neither a favorable task score nor the best random draw is treated as evidence of reliable recovery.

Recovery relative to Parent and component effects relative to SRR

We distinguish two questions: whether a candidate improves on the unrepaired Parent, and whether a design change improves on original SRR. For the aggregate results, 100,000 paired item-bootstrap draws resample matched identities within each task and then recompute the equally weighted task mean. These intervals are conditional on fixed checkpoints; they do not estimate repair-construction variability.

Table R2. Aggregate changes for every candidate. All differences are in percentage points. Confidence intervals for the six declared component comparisons are reported separately in Table R4.

VariantΔAvg. vs. Parent [95% CI], ppΔAvg. vs. original SRR, pp
Original SRR+0.139 [−0.059, +0.339]—
Uniform-positive fitting−0.002 [−0.205, +0.202]−0.141
All-uniform residual fitting+0.002 [−0.203, +0.207]−0.137
Random pairs: R1+0.150 [−0.052, +0.355]+0.011
Random pairs: R2+0.027 [−0.173, +0.228]−0.112
Random pairs: R3+0.061 [−0.144, +0.266]−0.078
Smaller update+0.043 [−0.160, +0.247]−0.096
Larger update+0.035 [−0.184, +0.253]−0.104
Sequential input refresh+0.002 [−0.210, +0.218]−0.138
Mean of R1–R3 (computed)+0.080 [−0.088, +0.246]−0.060

Table R3. Six component comparisons, including the sequential control in Q3. The comparison family was fixed before final aggregation. Confidence intervals are pointwise; the last column applies Holm correction across the six tests.

Component contrast (first − second)Difference [95% CI], ppRaw two-sided pHolm-adjusted p
Original SRR − uniform-positive+0.141 [−0.052, +0.335]0.15250.9151
Uniform-positive − all-uniform−0.004 [−0.199, +0.191]0.96811.0000
Original SRR − mean random+0.060 [−0.112, +0.230]0.49531.0000
Smaller update − original SRR−0.096 [−0.298, +0.104]0.35131.0000
Larger update − original SRR−0.104 [−0.323, +0.114]0.34861.0000
Sequential refresh − original SRR−0.138 [−0.344, +0.069]0.19270.9635

Identity-matched archived results were reused where available, and five missing checkpoint evaluations were completed under the fixed plan. Some archived control outcomes were already known; we therefore describe this as a controlled extension and reanalysis, not a wholly prospective replication of all ten checkpoints. Newly completed outcomes were not used to select candidates or the reported comparison family.

Interpretation. Original SRR does not establish an aggregate task advantage over either uniform-positive fitting or the activity-matched random-pair mean. Likewise, neither tested scale change establishes an improvement over original SRR. All candidate-versus-Parent intervals also include zero. Thus, these task-level controls do not provide evidence that the requested design alternatives recover the missing aggregate improvement in this setting. They do not establish that the variants are equivalent, that the original scale is optimal, or that weak source supervision is the unique cause of the result.

This distinction connects the new experiments to Section 5.3: the original local tests evaluate source-informed directions directly, separately from whether ridge fitting reproduces them; the new ablations test the downstream consequences of specific construction choices. Together, they support separating a source-derived correction target from evidence that executing it improves the merged model, rather than attributing every weak result to a single mechanism.

W1 & Q3:

We thank the reviewer for identifying this concrete alternative explanation. Section 5.3 shows that earlier updates can alter later router inputs, but that observation alone does not establish whether refreshing the fitting inputs improves task performance. We therefore implemented the requested sequential input-refresh control.

Starting from the earliest of the five updated layers, we apply its update, rerun the same fixed token sequences, and capture the inputs used to fit the next layer. To isolate the effect of stale inputs, we retain the original expert pairs, clipped residual targets, likelihood weights, numerical per-pair ridge coefficients, and update scale. Writing ht,ℓ<ℓh^{\lt\ell}_{t,\ell} for the input produced after updating the earlier layers, the control solves the following objective:

vℓjseq=arg⁡min⁡v12∑tρtjM[(ht,ℓ<ℓ)⊤v−r~tjM]2+λℓjM2∥v∥22.v^{\mathrm{seq}}_{\ell j} = \arg\min_v \frac{1}{2}\sum_t \rho^M_{tj}\left[(h^{\lt\ell}_{t,\ell})^\top v-\tilde r^M_{tj}\right]^2 + \frac{\lambda^M_{\ell j}}{2}\lVert v\rVert_2^2.

Relative to the original objective, only the fitting input is refreshed. We do not regenerate trajectories or simultaneously change the supervision. The first updated layer reproduces the original BF16 update, later fitting inputs do change, and all 30 numerical ridge coefficients match their original values.

The sequential checkpoint reaches 71.042% Avg., compared with 71.179% for original SRR and 71.040% for Parent. Its paired gain over Parent is +0.002 pp [−0.210, +0.218]. In the declared component test, sequential minus original SRR is −0.138 pp [−0.344, +0.069], with Holm-adjusted p=0.9635p=0.9635. The complete task results are retained in Table R2, including GSM8K changing from 59.894% to 58.984% and WinoGrande from 73.086% to 73.481%.

Accordingly, refreshing the inputs while preserving the supervision does not establish a task improvement in this controlled comparison. This directly tests the proposed stale-input remedy, rather than inferring its task effect from the input-response decomposition. It does not rule out strategies that also refresh targets, reselect pairs, or optimize a different objective, and we do not use the result to claim that every execution-related limitation has been excluded.

W2 & Q1:

We agree that a repair comparison is easier to interpret when the relevant specialist–parent gap is shown explicitly. We now compare Base, the corresponding domain Specialist, Parent, source-route replay, HARC, and SRR on the exact original 768 domain-task items: 256 each for science, mathematics, and computer science.

For experimental continuity, this comparison retains the original four-local-specialist Instruct Parent used by the domain intervention study. HARC and SRR were constructed for that exact Parent and finalized before task scoring. This Parent is distinct from the three-specialist Parent used in the eight-task ablations; scores and effects are paired within their own cohort, not pooled or presented as a reconstruction of a manuscript five-run mean. These comparisons contextualize the original source-route intervention cohort; they are not used to explain the eight-task ablation outcomes.

We distinguish

\quad\text{from}\quad R_{d,\pi'}=\operatorname{Acc}(M_{\phi,\pi'})-\operatorname{Acc}(M_{\phi,\pi}).$$ $G_d$ describes the specialist–parent performance gap; it does not identify routing as the cause. $R_{d,\pi'}$ measures the task effect of the specified routing intervention with non-routing parameters fixed, as in Section 4. **Table R4. Matched Base/Specialist/Parent performance.** Scores are percentages; gaps and paired intervals are in percentage points. | Domain (256 items each) | Base (%) | Specialist (%) | Parent (%) | Specialist − Parent [95% CI], pp | | --- | ---: | ---: | ---: | ---: | | Science | 51.172 | 52.344 | 52.344 | +0.000 [−2.734, +2.734] | | Mathematics | 28.516 | 27.734 | 30.469 | −2.734 [−7.031, +1.562] | | Computer science | 50.000 | 49.609 | 49.609 | +0.000 [−3.125, +3.125] | **Table R5. Routing-intervention gains on the same items and Parent.** All entries are percentage-point changes with paired 95% intervals. | Domain | Source replay − Parent [95% CI] | HARC − Parent [95% CI] | SRR − Parent [95% CI] | | --- | ---: | ---: | ---: | | Science | +1.172 [−1.562, +3.906] | −0.781 [−2.344, +0.781] | −0.781 [−1.953, +0.000] | | Mathematics | −1.172 [−5.469, +2.734] | −1.562 [−4.297, +1.172] | −0.781 [−2.344, +0.781] | | Computer science | +1.953 [−0.781, +4.688] | +1.172 [−0.391, +3.125] | +0.000 [−1.562, +1.562] | **What these comparisons establish.** On these matched tasks, the specialist–parent gap is zero at the point estimate in science and computer science, while Parent exceeds the mathematics specialist by 2.734 pp; all gap intervals include zero. Consequently, this cohort does **not establish a positive aggregate specialist-performance loss for the tested repairs to recover**. We therefore do not describe small repair gains as recovering a demonstrated merging-induced deficit, and we do not compute an unstable recovered-loss fraction. The intervention results are consistent with that limited conclusion: source replay has mixed point estimates with intervals including zero, and none of the six HARC/SRR domain comparisons establishes positive recovery (all six Holm-adjusted $p$-values are 1.0). This does not demonstrate that merging preserves specialist capabilities generally, nor does it exclude degradation on individual items or other tasks. Rather, it makes explicit that the need for repair cannot be presumed from source-relative routing disagreement or from the existence of domain specialists. **Implementation validity and reproducibility.** We reproduced every native Parent prediction in the matched-domain cohort; the maximum answer-token NLL discrepancy was **$3.553\times10^{-15}$**. The materialized HARC/SRR checkpoints were checked to change routing parameters only. To address the documentation concern, we will include the exact checkpoint/revision identifiers and domain mapping, merge coefficients and normalization, HARC calibration settings, parameter-update scope, and evaluation templates in the reproducibility appendix, with the two Parent families explicitly distinguished. The prepared configuration records retain verifiable training-mask information without inferring undocumented upstream training flags.

Comments on the paper

1 comment

Dear reviewers,

Thank you again for reviewing our work. We have posted our point-by-point responses and additional experimental results under your review. As the discussion period is approaching its end, we would appreciate any feedback on whether our responses address your concerns, or any remaining questions we can clarify.

Warm regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.