acceptodds
Under review as a conference paper at ICLR 2027

Who Contributes? Counterfactual Attribution in Sparse Mixture-of-Experts

32 views

Abstract

Sparse Mixture-of-Experts (MoE) models have become a key architecture for scaling large language models, yet understanding which experts truly contribute to model performance remains challenging. Existing measures such as routing frequency and gate weight describe expert usage, but overlook that MoE outputs are jointly produced by multiple experts whose contributions may be complementary or redundant. We introduce CONTRAIL, a counterfactual attribution framework that separately measures the contribution of selected expert computation and the utility of router decisions. For expert attribution, CONTRAIL models co-routed experts as a cooperative game and derives closed-form Shapley values from a quadratic approximation of the task loss, capturing both individual and interaction effects with a single backward pass. For router attribution, its Coalition-Consistent Gate Estimator evaluates a mass-conserving counterfactual swap between the nearest selected and excluded experts. We evaluate CONTRAIL on four instruction-tuned sparse MoE models using WikiText-2, GSM8K, MMLU, and MedQA. Across expert removal, cross-task intervention, router analysis, and sparse fine-tuning, CONTRAIL more reliably identifies consequential experts and routing decisions than routing frequency and existing attribution baselines, while revealing task-specific interactions and higher-impact adaptation targets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Shapley, Cooperative Game, Counterfactual Attribution, Sparse Expert Fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_1E3YOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CONTRAIL, a counterfactual attribution framework for sparse Mixture-of-Experts models that separates the contribution of selected expert computation from the contribution of the router decision itself. For expert attribution, CONTRAIL treats the co-routed experts as a cooperative game and derives closed-form Shapley values from a quadratic approximation of the task loss, capturing both individual effects and co-routed interactions without enumerating all coalitions. For router attribution, it introduces the Coalition-Consistent Gate Estimator (CCGE), which evaluates a mass-conserving swap between the lowest-logit selected expert and the highest-logit excluded expert. The method is validated on four instruction-tuned sparse MoE models across language modeling, mathematical reasoning, and knowledge-intensive tasks. Experiments on expert removal, cross-task transfer, router interventions, interaction ablation, and sparse expert fine-tuning show that CONTRAIL identifies more consequential experts and routing decisions than routing frequency and other attribution baselines, while also revealing task-dependent expert importance and useful targets for parameter-efficient adaptation.

Critical strengths

Strength 1: The paper clearly separates route-conditioned expert attribution from router attribution, which avoids mixing expert computation with routing changes.

Strength 2: CONTRAIL provides a closed-form Shapley formulation that captures both individual expert effects and co-routed interaction without enumerating all expert coalitions.

Strength 3: The experiments are comprehensive, covering four sparse MoE models, multiple tasks, expert removal, router interventions, cross-task analysis, and sparse fine-tuning.

Critical weaknesses

Weakness 1: CONTRAIL claims to model co-routed interaction through the term δe⊤H⋆δm\delta_e^\top H^\star \delta_m in Eq. (3), but the implementation replaces H⋆H^\star with the rank-one surrogate H^iEF=uiui⊤\hat H_i^{EF}=u_i u_i^\top in Eq. (10). This makes every pairwise interaction reduce to si,esi,ms_{i,e}s_{i,m} in Eqs. (70)–(73). In other words, two experts are judged to interact only through how their outputs align with the same gradient vector uiu_i. Any interaction that is reflected in other curvature directions is lost. This is a strong restriction, and the paper does not show how close this rank-one interaction is to the interaction obtained with the true Hessian.

Weakness 2: Eq. (62) gives a bound on the difference between the exact nonlinear expert value and the practical CONTRAIL score, but the paper does not show whether this approximation keeps the same expert ranking. Figure 6 only shows that the ranking becomes stable as more probe examples are used. It does not tell us whether the resulting ranking matches the exact nonlinear expert game. Since the Top 1% and Top 3% interventions depend entirely on which experts appear at the top of the ranking, it would be important to know whether approximation error changes these top-ranked sets.

Weakness 3: Under the empirical-Fisher-style surrogate, Eq. (74) can be written as ϕ^i,e=(−si,e)(1−12si,Σ)\hat{\phi}_{i,e}=(-s_{i,e})(1-\frac{1}{2}s_{i,\Sigma}). For a fixed token and MoE layer, the second factor is shared by all active experts, so the full contribution does not introduce an expert-specific interaction correction relative to the direct term at that token. Its effect instead comes from changing the weight of different token-level contributions before corpus-level aggregation. This makes it unclear whether the improvement attributed to co-routed interaction in RQ4 is actually due to expert–expert interaction or to token-level reweighting induced by si,Σs_{i,\Sigma}.

Questions for authors

Question 1: How accurate is the rank-one curvature surrogate in Eq. (10) for estimating the co-routed interaction term in Eq. (3)? Can you report the approximation error?

Question 2: Does the practical CONTRAIL score preserve the expert ranking of the exact nonlinear expert game? Can you report the rank correlation or Top-kk overlap between the practical scores and exact nonlinear coalition-based scores?

Question 3: How often does the interaction term in Eq. (74) actually change the ordering of active experts for a given token and MoE layer? Can you report the token-level ranking difference between the full contribution score and the direct or self-only score, and then show how much additional change is introduced by corpus-level aggregation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1E3Y

Thank you for the careful analysis of the practical curvature approximation and the shared token-level factor. These observations help clarify what the implemented correction contributes. The corresponding revisions have been incorporated into the revised manuscript PDF and highlighted in red. The updates clarify the practical estimator, token reweighting, and ranking fidelity in Sections 3.3, 4.5, and 4.7, with detailed comparisons in Appendices H.2–H.4.

W1 / Q1. Accuracy of the rank-one interaction approximation

The gradient outer product is a directional curvature model. With si,e=ui⊤δi,es_{i,e}=u_i^\top\delta_{i,e}, its pair term is ψ^i,em=si,esi,m\widehat\psi_{i,em}=s_{i,e}s_{i,m}, and its error relative to the quadratic interaction is ψ^i,em−ψi,em(2)=δi,e⊤(uiui⊤−Hi)δi,m\widehat\psi_{i,em}-\psi^{(2)}_{i,em}=\delta_{i,e}^\top(u_i u_i^\top-H_i)\delta_{i,m}. This expression identifies the curvature information omitted by the approximation.

We evaluate these contractions on 1,024 token/layer units from four layers each of OLMoE and gpt-oss, across WT-2 and GSM8K. Rank-one pair terms have normalized error 0.62 and Spearman correlation 0.60 against local HVP contractions. Against nonlinear joint-removal residuals, the corresponding values are 0.77 and 0.49. HVP pair terms themselves have error 0.41 and correlation 0.78 against the nonlinear residuals, showing that finite-removal approximation also contributes to the discrepancy.

Expert-value and pairwise-interaction fidelity (Table 7, Section 4.7). The comparison on these attribution units is reproduced below.

ComparisonNormalized errorSpearman correlation
First-order expert values vs. nonlinear Shapley values0.520.80
HVP quadratic expert values vs. nonlinear Shapley values0.270.94
Rank-one expert values vs. nonlinear Shapley values0.430.86
Rank-one pair terms vs. HVP pair terms0.620.60
Rank-one pair terms vs. nonlinear joint-removal residuals0.770.49
HVP pair terms vs. nonlinear joint-removal residuals0.410.78

Table 19 in Appendix H.2 reports sign agreement of 76% with HVP pair terms and 69% with nonlinear residuals, with 95% intervals of [72%, 80%] and [64%, 74%]. The respective majority-sign baselines are 57% and 56%. We exclude near-zero references from this statistic using a threshold of 1% of the stratum median absolute nonzero interaction; these account for approximately 12% of pairs.

The comparison supports partial recovery of interaction signs and ordering, with appreciable error in magnitude. Section 4.7 and Appendix H.2 now describe the pair terms as rank-one interaction approximations and report their accuracy at this level.

W2 / Q2. Agreement with nonlinear Shapley rankings and Top-K sets

The probe-prefix experiment addresses sampling stability. To assess fidelity to the nonlinear game, we compare rankings on identical tokens, with inactive-expert zeros and the same aggregation rule. A score-error bound ε\varepsilon preserves the Top-KK set when the exact selection-boundary gap exceeds 2ε2\varepsilon; the direct comparison below evaluates the resulting selections.

The sample covers all 64 OLMoE experts and all 32 gpt-oss experts in each layer/task stratum. We use raw signed corpus scores, average ranks for correlation ties, and fixed expert IDs to resolve selection-boundary ties. The nominal 1% budget selects K=1K=1 per layer in both models; the nominal 3% deletion budget selects K=2K=2, including the gpt-oss configuration override. The ranking comparison below reproduces Table 20 in Appendix H.3.

Practical score vs. nonlinear referenceSpearman ρ\rhoKendall τb\tau_bK=1K=1 overlapK=2K=2 overlap
Within-token active-expert ranking0.820.68——
Layer-local corpus expert ranking0.890.7413/16 = 81.25%27/32 = 84.375%

Correlations are averaged across units or strata. Overlap is ∣S^K∩SKnl∣/K|\widehat S_K\cap S_K^{\mathrm{nl}}|/K; the fractions aggregate the 16 layer/task/model strata.

We also compare removal effects on 128 additional packed blocks per model/task, selecting K=2K=2 experts in each of the four sampled layers. Average NLL increases are 0.061 for Direct, 0.071 for HVP, 0.068 for rank-one Full, and 0.074 nats/target token for nonlinear-Shapley selection. Together, these results indicate useful ranking agreement on the evaluated subset, with remaining differences at the selection boundary. The revised manuscript specifies the corpus aggregation and selection rules in Section 4.7 and Appendices C.3 and H.3. As clarified in Appendix H.3, corpus scores already include routing frequency through Φecorpus=ρeΦecond\Phi_e^{\mathrm{corpus}}=\rho_e\Phi_e^{\mathrm{cond}}; no additional routing-frequency multiplier is applied in this comparison.

Expert-removal results (Table 21, Appendix H.3).

Expert selectorMean NLL increase (nats/target token)
Direct0.061
HVP0.071
Rank-one Full0.068
Nonlinear-Shapley selection0.074

W3 / Q3. Token reweighting and changes in expert ordering

Your algebraic observation is correct. Define di,e=−si,ed_{i,e}=-s_{i,e}, si,Σ=∑msi,ms_{i,\Sigma}=\sum_m s_{i,m}, and wi=1−12si,Σw_i=1-\tfrac12s_{i,\Sigma}. The Full score is ϕ^i,e=widi,e\widehat\phi_{i,e}=w_i d_{i,e}. At a fixed token/layer, positive wiw_i preserves every strict Full/Direct ordering, negative wiw_i reverses it, and zero wiw_i collapses the full scores to a tie.

The comparison with Self-only, ϕi,eself=−si,e+12si,e2\phi_{i,e}^{\mathrm{self}}=-s_{i,e}+\tfrac12s_{i,e}^2, follows from the pairwise differences. Suppressing ii,

=(s_e-s_m)(-1+\tfrac12s_\Sigma),\qquad \phi_e^{\mathrm{self}}-\phi_m^{\mathrm{self}} =(s_e-s_m)(-1+\tfrac12(s_e+s_m)).$$ A strict Full/Self reversal occurs when the two parenthesized factors have opposite signs and $s_e\ne s_m$. After aggregation, $\widehat\Phi_e=\frac1N\sum_i w_i d_{i,e}$, so **varying token weights can change corpus rankings even when every weight is positive**. Across the same 1,024 units, 976 weights are positive (95.31%), 40 are negative (3.91%), and 8 are near zero (0.78%), using $|w_i|\le10^{-6}$. Each model contributes 512 units: 488 positive, 20 negative, and 4 near zero. The eight **near-zero units** are excluded from the pairwise ranking comparison; the remaining score pairs are non-tied. Full/Direct reverses 680 of 17,272 pairs (3.94%), and Full/Self reverses 1,164 pairs (6.74%). The corpus-ranking comparisons below are reported in **Table 22** (Appendix H.4). | Corpus-level comparison | Spearman $\rho$ | Kendall $\tau_b$ | $K=1$ overlap | $K=2$ overlap | | ----------------------- | -------------- | --------------- | -------------- | -------------- | | Full / Direct | 0.92 | 0.78 | 11/16 = 68.75% | 24/32 = 75.00% | | Full / Self-only | 0.90 | 0.76 | 12/16 = 75.00% | 26/32 = 81.25% | These results show how limited within-token reversals can accompany larger changes in the corpus-level selected sets. **Weighted Direct and Full are algebraically identical**, giving correlation and overlap of 1 wherever defined. Section 4.5 now interprets RQ4 as an ablation of this `token-reweighting correction`. Its existing results—greater removal damage for Full in 20 of 36 cells, with two ties at the displayed precision—show that the correction's utility varies across settings.

Thank you for the additional experiments. The ranking correlations, Top-k overlaps, and expert-removal results address my second concern. The analysis of the shared correction factor also confirms my observation in W3 and explains why the main changes in expert ranking occur after corpus-level aggregation.

For W1, the new results help clarify the accuracy of the rank-one approximation. The correlation of 0.49 with nonlinear interaction residuals is still relatively low, suggesting that the approximation misses a substantial part of the interaction information. I think this remains a limitation of the method, although the authors have now provided the evidence needed to assess it.

The revisions have addressed most of my concerns, and I have adjusted my rating accordingly.

We sincerely thank the reviewer for the thoughtful follow-up and careful reassessment of our revised manuscript. We are glad that the additional experiments and revisions have addressed most of your concerns, and we appreciate your updated evaluation.

Your constructive feedback has helped us improve both the empirical analysis and the clarity of our presentation. Thank you again for your time and valuable suggestions.

Reviewer_8DA7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces CONTRAIL, a counterfactual attribution framework that separates the contributions of expert computation from those of router decisions. It models co-routed experts as a cooperative game and derives closed-form Shapley values for a quadratic approximation of the task loss. The practical estimator approximates the Hessian with a gradient outer product, allowing individual contributions and pairwise interactions to be estimated with a single backward pass. For router attribution, the authors introduce the Coalition-Consistent Gate Estimator to evaluate gate-mass-preserving swaps at the top-k routing boundary. Experiments across four mixture-of-experts architectures evaluate the method through expert removal, routing interventions, and sparse fine-tuning.

Critical strengths

The paper addresses a relevant problem: frequently used experts are not necessarily those that contribute most to a task. Separating expert computation from router selection is well motivated, since removing an expert and rerouting its inputs can mix these effects. The experiments cover four architectures and several tasks, and the sparse fine-tuning results provide a concrete application of the attribution scores.

Critical weaknesses
  1. Section 2.2 introduces the Shapley value with little explanation. Since this definition is central to the method, the notation and underlying intuition deserve more discussion, particularly the coalition CC, its utility U(C)U(C), and PiπP_i^\pi (the players preceding ii in permutation π\pi). A small worked example would help explain how averaging marginal contributions gives an expert’s Shapley value.

  2. The notation throughout the method section is dense and difficult to track, despite the notation table in Appendix A. Using fewer symbols and explaining more of the derivation in words would help clarify how the exact game, the quadratic approximation, and the practical estimator relate to one another.

Questions for authors
  1. My main concern is the estimation error introduced by the gradient-outer-product approximation in Eq. (10). Appendix B.9 provides conditional error bounds, but the experiments do not directly establish the accuracy of this approximation. Could the authors compare the estimated Shapley values and pairwise interactions against exact coalition evaluations and measured joint-removal residuals on a small set of tokens and layers? Such comparisons would help determine whether the estimates faithfully reflect the defined counterfactual effects.

  2. Appendix C.4 provides a complexity analysis, but the practical cost of attribution remains unclear. CONTRAIL requires a backward pass, while router attribution also evaluates an excluded expert at each attributed token and layer. Reporting wall-clock time and peak memory under matched probe sizes, with comparisons to forward-only and gradient-based baselines, would help assess its practical efficiency.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 8DA7

Thank you for recognizing the motivation for separating expert computation from router decisions, and for the suggestions on presentation and empirical validation. The corresponding revisions have been incorporated into the revised manuscript PDF and highlighted in red.

W1. Explain the cooperative game and provide a worked example

At a particular token and MoE layer, the players are the experts selected by the factual router. A coalition CC is a subset of these experts. Its utility U(C)U(C) is the negative downstream task loss when only the contributions of experts in CC are retained at that site, using their factual gate weights without renormalization. Downstream computation is then recomputed normally. For a permutation π\pi, PeπP_e^\pi denotes the experts appearing before ee. The Shapley value averages the marginal utility U(Peπ∪{e})−U(Peπ)U(P_e^\pi\cup\{e\})-U(P_e^\pi) over all permutations.

Section 2.2 defines the permutation-based Shapley value, and Section 3.1 specifies the fixed-gate expert game in Equation (2). Figure 2 provides the worked example for one token at one MoE layer. Its two active experts have gradient projections se=3s_e=3 and sa=−2s_a=-2. Under the rank-one quadratic model, these give self terms of 9 and 4 and a shared pair term of sesa=−6s_es_a=-6. Half of this pair term is assigned to each expert. Adding each direct term to half of its curvature row sum gives the practical scores ϕ^e=−3+12(9−6)=−1.5\widehat\phi_e=-3+\tfrac12(9-6)=-1.5 and ϕ^a=2+12(−6+4)=1\widehat\phi_a=2+\tfrac12(-6+4)=1. The diagram connects the fixed-gate intervention, interaction allocation, and final expert credit; expert aa receives the higher score in this example.

W2. Distinguish the nonlinear game, quadratic approximation, and practical estimator

Section 3.1 distinguishes three quantities, and Section 3.3 specifies the practical estimator:

QuantityMeaningExactness
ϕenl\phi_e^{\mathrm{nl}}Shapley value of the original nonlinear loss gameDefined by coalition evaluations
ϕe(2)\phi_e^{(2)}Closed-form allocation using the local Hessian HHExact for the quadratic surrogate game
ϕ^e\widehat\phi_eImplemented score using H≈uu⊤H\approx uu^\top, where u=∇yLu=\nabla_y\mathcal LExact for this restricted quadratic surrogate; approximate for the nonlinear game

Writing δe=gefe\delta_e=g_e f_e for an expert's factual contribution and Δ=∑mδm\Delta=\sum_m\delta_m, the quadratic allocation is ϕe(2)=−u⊤δe+12δe⊤HΔ\phi_e^{(2)}=-u^\top\delta_e+\tfrac12\delta_e^\top H\Delta. The first term measures the local first-order contribution; the second allocates the quadratic self and pairwise terms. Equation (4) gives this closed-form quadratic allocation. The rank-one practical estimator in Section 3.3 substitutes uu⊤uu^\top for HH.

Q1. Validate expert values and pairwise interactions directly

We compare first-order scores, quadratic scores computed with local Hessian-vector products (HVPs), and rank-one scores against exact nonlinear coalition-based Shapley values. The comparison covers OLMoE layers 0, 5, 10, and 15 and gpt-oss layers 0, 7, 15, and 23, using zero-based indices, on WT-2 and GSM8K. For each model/task, we use 64 target-token positions from distinct packed blocks, shared across the four layers. This gives 16 model/task/layer strata and 1,024 attributed units. The active-set sizes of 8 and 4 require 256 and 16 coalitions per unit, respectively, or 139,264 unit-coalition evaluations before caching.

At the intervention site, factual expert contributions remain fixed; downstream computation, including downstream routing, is recomputed. For pairwise interactions, the nonlinear reference is the joint-removal residual: $ \psi^{\mathrm{nl}}_{em} =\mathcal L(y^\star-\delta_e-\delta_m) -\mathcal L(y^\star-\delta_e) -\mathcal L(y^\star-\delta_m) +\mathcal L(y^\star). $

Comparing this residual with δe⊤Hδm\delta_e^\top H\delta_m measures finite-intervention approximation, while comparing the latter with (u⊤δe)(u⊤δm)(u^\top\delta_e)(u^\top\delta_m) measures curvature substitution. We calculate normalized error as ∥v^−v∥2/∥v∥2\lVert\widehat v-v\rVert_2/\lVert v\rVert_2 within each stratum and then report macro averages. Zero-reference cases are treated separately. The comparison below reproduces Table 7 in Section 4.7; Appendix H.1 gives the full protocol.

ComparisonNormalized errorSpearman correlation
First-order expert values vs. nonlinear Shapley values0.520.80
HVP quadratic expert values vs. nonlinear Shapley values0.270.94
Rank-one expert values vs. nonlinear Shapley values0.430.86
Rank-one pair terms vs. HVP pair terms0.620.60
Rank-one pair terms vs. nonlinear joint-removal residuals0.770.49
HVP pair terms vs. nonlinear joint-removal residuals0.410.78

For the rank-one expert values, the 95% intervals are [0.36, 0.51] for normalized error and [0.80, 0.91] for Spearman correlation. The practical score improves on the first-order reference in both metrics, while the HVP-based score provides a closer approximation. Pairwise interaction magnitudes remain less accurate than expert-value rankings. This distinction is now explicit in Section 4.7 and Appendix H.2.

Q2. Report wall-clock time and peak memory

Expert attribution uses one factual forward pass and one backward pass per probe minibatch. The nearest-boundary router score additionally evaluates the first excluded expert.

We compare five configurations on OLMoE using one H100 80GB, BF16 model computation with FP32 attribution contractions, batch size 1, 128 packed WT-2 blocks of length 1,024, and frozen parameter gradients. All configurations use identical probes and target masks. Timing covers attribution computation after model and data loading, excludes disk export, and uses three synchronized repeats after two warm-up blocks per configuration. The table reports mean time ± standard deviation and peak allocated GPU memory.

ConfigurationTotal attribution timePeak allocated GPU memory
Routing frequency / gate-weight collection25.56 ± 0.16 s13.11 GiB
First-order gradient attribution79.24 ± 0.64 s17.60 GiB
CONTRAIL, expert scores only79.72 ± 1.38 s17.60 GiB
CONTRAIL, expert and nearest-boundary router scores125.19 ± 1.09 s17.72 GiB
Current implementation with additional diagnostics366.19 ± 0.89 s17.93 GiB

Expert-only CONTRAIL has a mean runtime 0.6% above Direct, with the same measured peak allocated memory. Adding router attribution increases mean runtime by 57.0% relative to expert-only attribution. The final row includes additional candidate evaluations and dense pair statistics; its total time is 14.33 times that of the forward-only configuration. These configurations are now reported separately in Section 4.8 and Table 8, with further details in Appendix H.7.

Thank you again for your time and thoughtful feedback. With two days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Response to Reviewer wHbj

Thank you for the detailed questions about estimator fidelity, objective scaling, native routing, and evaluation reliability. We address each below. The corresponding revisions have been incorporated into the revised manuscript PDF and highlighted in red. The updates address estimator fidelity, loss scaling, native routing, and the GSM8K case in Sections 3.2–3.3, 4.4, and 4.7, and Appendices E.2 and H.1–H.6.

W1 / Q1. Validate the practical estimator against the attribution target

The closed-form allocation is exact for a specified quadratic game. Replacing its Hessian by uu⊤uu^\top gives the practical surrogate. Here uu is the gradient of the summed attribution objective, so uu⊤uu^\top is the outer product of that aggregate gradient. This definition is now explicit in Section 3.3.

We compare the practical score with HVP-based quadratic values and nonlinear coalition values on OLMoE layers 0, 5, 10, and 15 and gpt-oss layers 0, 7, 15, and 23, on WT-2 and GSM8K. Each model/task contributes 64 token positions across four layers, giving 1,024 units. Exact enumeration uses 16 or 256 coalitions per unit. Factual gates and expert contributions remain fixed at the intervened site, while downstream computation is recomputed normally.

Expert-value and pairwise-interaction fidelity (Table 7, Section 4.7). The comparison on these attribution units is reproduced below.

ComparisonNormalized errorSpearman correlation
First-order expert values vs. nonlinear Shapley values0.520.80
HVP quadratic expert values vs. nonlinear Shapley values0.270.94
Rank-one expert values vs. nonlinear Shapley values0.430.86
Rank-one pair terms vs. HVP pair terms0.620.60
Rank-one pair terms vs. nonlinear joint-removal residuals0.770.49
HVP pair terms vs. nonlinear joint-removal residuals0.410.78

The rank-one expert values have normalized error 0.43 and Spearman correlation 0.86 against the nonlinear reference; the HVP values achieve 0.27 and 0.94. At the corpus level, the practical score has Spearman correlation 0.89, K=1K=1 overlap 13/16 (81.25%), and K=2K=2 overlap 27/32 (84.375%). The comparison uses raw signed scores and K=1/2K=1/2 per layer, with all 64 OLMoE and 32 gpt-oss experts covered. Correlations use average ranks for ties, and fixed expert IDs resolve selection-boundary ties.

Ranking and Top-K agreement (Table 20, Appendix H.3).

Practical score vs. nonlinear referenceSpearman ρ\rhoKendall τb\tau_bK=1K=1 overlapK=2K=2 overlap
Within-token active-expert ranking0.820.68——
Layer-local corpus expert ranking0.890.7413/16 = 81.25%27/32 = 84.375%

On 128 held-out packed blocks per model/task, removing K=2K=2 experts in each sampled layer increases NLL by 0.061 for Direct, 0.071 for HVP, 0.068 for the practical score, and 0.074 nats/target token for nonlinear-Shapley selection. The practical approximation retains useful ranking information on this subset, while the HVP comparison identifies room to improve its numerical accuracy.

Expert-removal results (Table 21, Appendix H.3).

Expert selectorMean NLL increase (nats/target token)
Direct0.061
HVP0.071
Rank-one Full0.068
Nonlinear-Shapley selection0.074

W1 / Q2. Sensitivity to loss scale and sum-versus-mean reduction

The practical score is scale-sensitive. Write its corpus value as Ae+BeA_e+B_e, with Ae=−1N∑isi,eA_e=-\frac1N\sum_i s_{i,e} and Be=12N∑isi,esi,ΣB_e=\frac1{2N}\sum_i s_{i,e}s_{i,\Sigma}. Under L↦cL\mathcal L\mapsto c\mathcal L, c>0c>0, the score becomes Φ^e[cL]=cAe+c2Be\widehat\Phi_e[c\mathcal L]=cA_e+c^2B_e, and its ranking is determined by Ae+cBeA_e+cB_e. The exact game's values and the true-Hessian quadratic allocation scale uniformly by cc.

We evaluate c∈{0.1,0.3,1,3,10}c\in\{0.1,0.3,1,3,10\} on the same probes and four-layer removal protocol, using raw signed scores and K=1/2K=1/2. Each ranking is compared with c=1c=1; the sweep is reported in Table 23 (Appendix H.5).

Loss scale ccMean layer Spearman vs. c=1c=1K=1K=1 overlapK=2K=2 overlapMean held-out NLL increase (nats/target token)
0.10.9412/16 (75.00%)25/32 (78.13%)0.062
0.30.9814/16 (87.50%)29/32 (90.63%)0.065
11.0016/16 (100.00%)32/32 (100.00%)0.068
30.9513/16 (81.25%)27/32 (84.38%)0.070
100.839/16 (56.25%)21/32 (65.63%)0.056

The rankings remain close at c=0.3c=0.3. At c=10c=10, 7 of 16 Top-1 selections and 11 of 32 Top-2 slots change, and NLL damage decreases from 0.068 to 0.056. Section 3.3 now specifies the loss convention as part of the practical estimator; Section 4.7 and Appendix H.5 report its scale sensitivity alongside the intervention results.

For packed-block mean loss, Spearman and Kendall correlations with the summed-loss ranking are 0.86 and 0.70; overlaps at K=1K=1 and K=2K=2 are 10/16 (62.50%) and 23/32 (71.875%). NLL damage is 0.058 versus 0.068 nats/target token. This comparison divides each block's masked loss by its valid target count before differentiation, with equal block weights and unchanged token aggregation. When target counts differ, it changes both the scale and relative weighting of blocks. Appendix H.5 now distinguishes this reduction test from the common-scale sweep.

W2 / Q3. Evaluate counterfactuals generated by the native router

The native-routing evaluation complements RQ3's matched mass-transfer intervention. The reflected endpoint ℓ−=ℓ−(ℓa−ℓb)(ea−eb)\ell^-=\ell-(\ell_a-\ell_b)(e_a-e_b) exchanges the logits of the weakest selected expert aa and strongest excluded expert bb. For full-softmax gates, with or without selected-set renormalization, and for softmax over selected logits, this exchange preserves the relevant normalization. With fixed expert inputs and a strict boundary margin, the native endpoint transfers aa's former gate weight to bb and leaves the other selected weights unchanged in exact arithmetic.

We also evaluate ℓ(α)=ℓ−α(ℓa−ℓb)(ea−eb)\ell^{(\alpha)}=\ell-\alpha(\ell_a-\ell_b)(e_a-e_b) for α∈{0.51,0.75,1,1.25}\alpha\in\{0.51,0.75,1,1.25\}, recomputing native top-kk support and gate normalization. The boundary is crossed for α>0.5\alpha>0.5. All estimators rank from the same factual state and use the same endpoint at each α\alpha.

The comparison uses OLMoE on WT-2, scores all 16 MoE layers on 64 packed probe blocks, and intervenes on the selected top three layers over 25 disjoint evaluation windows. The parity check uses strict positive margins, with ties handled separately. At α=1\alpha=1, FP32 support agreement is 100%, and the NLL difference between the two endpoint implementations is 0.0000 at the displayed precision.

The comparison below reproduces Table 24 in Appendix H.6. Each cell gives layer Spearman correlation and NLL increase in 10−310^{-3} nats/target token.

Estimatorα=0.51\alpha=0.51α=0.75\alpha=0.75α=1\alpha=1α=1.25\alpha=1.25
Hard-route gradient(0.31, 0.073)(0.29, 0.022)(0.27, -0.007)(0.35, 0.102)
Identity STE(0.61, 1.092)(0.67, 1.310)(0.70, 1.499)(0.71, 1.877)
Sigmoid STE(0.60, 1.077)(0.66, 1.296)(0.69, 1.485)(0.70, 1.856)
CCGE-linear(0.66, 1.216)(0.68, 1.361)(0.65, 1.368)(0.64, 1.717)
CCGE(0.62, 1.157)(0.74, 1.579)(0.79, 2.074)(0.65, 1.812)

The α=1\alpha=1 loss column uses RQ3's reflected-endpoint reference, converting the rounded PPL changes with a baseline PPL of 13.73. CCGE produces the largest NLL increase at α=0.75\alpha=0.75 and α=1\alpha=1; CCGE-linear leads at 0.51 and Identity STE at 1.25. The advantage therefore depends on the intervention endpoint. Section 4.4 and Appendix H.6 make this endpoint dependence explicit; Section 3.2 defines the reflected counterfactual.

Q4(a). Uncertainty in sparse fine-tuning

Thank you for suggesting repeated-seed evaluation. We agree that it would help quantify training variability. The full fine-tuning comparison spans four MoE backbones, three tasks, four trainable-expert budgets, and CONTRAIL plus six baseline selectors. Repeating this grid with five seeds would require substantial additional GPU resources and training and evaluation time, making it impractical within the resources available for this rebuttal. The revised manuscript describes the fine-tuning protocol in Appendix D.7 and presents the comparisons as observed point estimates in Section 4.6.

The existing results show a consistent advantage across this broad evaluation. In Tables 6 and 16, CONTRAIL achieves the best point estimate in all 48 model–task–budget settings. Many gains are appreciable in magnitude: at the 3% budget, GSM8K accuracy increases from 58.4% to 60.8% on OLMoE and from 66.0% to 68.4% on Granite relative to routing-frequency selection, while gpt-oss WT-2 perplexity decreases from 23.51 to 22.59. Relative to the strongest competing selector, Shapley-MoE, the corresponding OLMoE and Granite GSM8K accuracies improve from 60.3% to 60.8% and from 67.8% to 68.4%, respectively. These results support the overall effectiveness of the selected experts across architectures, tasks, and budgets.

We acknowledge that the smallest margins, such as 96.3% versus 96.2% on Qwen3/GSM8K at the 3% budget, could be influenced by training randomness. We distinguish these near ties from the larger gains and interpret them as observed numerical advantages with unresolved seed variability. Our overall conclusion is that CONTRAIL achieves the best observed performance among the evaluated expert-selection methods under matched budgets, with the closest comparisons qualified accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.