acceptodds
Under review as a conference paper at ICLR 2027

MoE Routing Is Not Token-Local: How Expert Influence Propagates Across Tokens & Layers

Abstract

In Sparse Mixture-of-Experts (MoE) models, routing is typically analyzed at the level of individual tokens. We instead trace how an expert's computation on one token affects the routing decisions of other tokens. We show that routing influence propagates across tokens and layers through two causal channels, a direct same-token channel and a context-mediated inter-token channel. Our evaluation on five MoE models reveals that: (i) the two channels combine approximately linearly in routing-margin space before the first routing divergence; (ii) they propagate differently with depth, as direct influence acts immediately and fades while context-mediated influence accumulates downstream; and (iii) the two channels are typically redundant rather than synergistic, with either pathway alone often sufficient to cross a routing boundary. To formalize these findings, we examine MoE routing boundaries and introduce a first-order, margin-normalized routing leverage. Routing leverage makes explicit that whether a Top-K boundary is crossed depends on how an expert's contribution aligns with the contrast direction between the competing experts, relative to the margin separating them, and it predicts the layer at which a token's routing first diverges, with accuracy increasing as more of the causal path is included. Discrete Top-K selection turns these continuous, additive effects into nonlinear changes in computation, and the first divergence marks the onset of a routing cascade.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mixture-of-Experts, Mechanistic Interpretability, Routing Influence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_32ZMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper investigates the causal mechanics of routing in Sparse Mixture-of-Experts (MoE) LLMs. Whereas prior interpretability work has largely taken a token-local perspective, the authors demonstrate that an expert's computation on a single token can causally influence downstream routing decisions across both tokens and layers. The study identifies two causal pathways: a direct same-token residual channel and an attention-mediated inter-token context channel. Across five open MoE models, ablating an expert changes the routing of a median 52-81% of tokens that never selected it. Before the first discrete Top-K routing change, these pathways combine approximately linearly in routing-margin space; the direct channel acts immediately and fades with depth, while context-mediated effects propagate to later tokens. Either channel alone is typically sufficient to trigger a routing change. To formalize this behavior, the paper introduces margin-normalized routing leverage (JJ) and three first-order estimators, J(0)J^{(0)}, J(1)J^{(1)}, J(∞)J^{(\infty)}, which progressively account for broader causal pathways to predict the first layer at which routing diverges from the clean forward pass. The authors support their findings with controlled causal interventions and attention key-value patching across five models, evaluating the routing-leverage estimators on four.

Critical strengths
  1. The paper's main strength is that it moves beyond a token-local view of MoE interpretability and provides causal evidence that an expert's computation on one token can affect the routing of other tokens in later layers. In Section 4.1 (Table 1), ablating an expert changes the routing of a median 52-81% of tokens that never selected it. Unlike prior work that studies how context affects routing or decomposes routing scores into upstream contributions (Section 2), the authors intervene directly on an internal component and trace its influence through attention, which they term the context-mediated inter-token channel. In Section 4.3, restoring the clean keys and values eliminates these routing changes, providing further evidence that attention mediates the effect. Together, these experiments offer a mechanistic explanation for how an expert's influence extends beyond the tokens routed to it. Section 6 complements this finding with a routing-leverage measure that captures how the perturbation's alignment with the contrast direction between competing experts, relative to the margin separating them, determines whether a routing boundary is crossed. Finally, the findings suggest a potential application to expert pruning and quantization, although this practical benefit remains unevaluated.

  2. The claims are supported by a range of dedicated causal experiments across five architecturally diverse MoE models, including direct, context, and joint ablations (Section 4), the additivity of the two channels before the first routing divergence (Section 5.1), and the depth dependence of direct and context-mediated influence under load-matched ablations (Section 5.2). Because the models differ in normalizer, attention variant, routing rule, number of experts and Top-K, the consistency of these results across them suggests that the observed mechanisms are not specific to a single architecture.

Critical weaknesses
  1. The paper reports that ablating an expert changes the routing of a median 52-81% of tokens that never selected it (Section 4.1), but this number is not compared against a control. Top-K routing can contain near-tied boundaries, and a perturbation propagated through many layers may cross some of them regardless of its source. A comparison with a random perturbation of matched norm, removed at the same positions, would show how much of this effect is specific to the ablated expert and how much reflects the general sensitivity of Top-K routing. In addition, every change in the Top-K set is counted as a routing divergence, but the effect of these divergences on the model output is not measured, and swapping two nearly tied experts may change little downstream. In OLMoE, for example, ablations change the routing of a median 60.0% of tokens that never selected the expert while the NLL remains essentially unchanged (Tables 1 and 3). Without a baseline and an analysis of the effect on model output, the significance of these routing changes, and the suggested implications for pruning and quantization, are unclear.

  2. When the true perturbation is used, J>1J > 1 simply restates that the routing margin became negative (Equation 5), so the contribution of the predictor lies in the first-order estimate of the perturbation. With J(∞)J^{(\infty)}, this estimator predicts the exact layer of the first routing divergence with a median recall of 30.2-51.3% (Section 6.2). From the fifth hop on, the margin ranker alone, which uses no perturbation estimate, overtakes J(∞)J^{(\infty)} (Section 6.3). J(∞)J^{(\infty)} outperforms the margin again only when evaluation is restricted to the clean-path regime, that is, to tokens whose context has not yet diverged, and identifying these tokens requires the ablated forward pass. Moreover, computing J(∞)J^{(\infty)} requires propagating the perturbation through all layers and tokens, at a cost comparable to running the ablation directly. It is therefore unclear in which setting the predictor would be preferred over simply running the intervention.

Questions for authors
  1. How does the routing flip rate under expert ablation (Section 4.1, Table 1) compare to a control intervention that removes a matched-norm random perturbation in place of the expert's output, at the same positions?
  2. What is the effect of the observed routing flips on the model output, for example the change in NLL at tokens that diverge? Given that OLMoE ablations change the routing of a median 60.0% of tokens that never selected the expert while leaving NLL nearly unchanged (Tables 1 and 3), do these flips reflect functional redundancy among experts?
  3. What is the computational cost of J(∞)J^{(\infty)} (Section 6.1) compared with running the ablated forward pass directly? In which practical setting do the authors expect predicting routing divergences with J(∞)J^{(\infty)} to be preferred over observing them through running the ablation?
3 comments

W1, Q1 (baseline; output effect; implications for compression). We implemented the proposed matched-norm control (see G2) and added both an output analysis and a causal routing rescue (see G1). Routing flips carry part of the output change causally (Table 10), and how much depends on the expert: small per token for typical experts, but for coalition experts the top-1 prediction changes for up to 24% of tokens (Table 9). Compressing an expert therefore affects tokens that never selected it. Routing leverage applies to such errors directly, and our influence graph identifies the experts for which this reaches the output, including some that are individually safe to prune but not jointly (see our response to Reviewer HXiP, Q3).

Q2 (do flips reflect functional redundancy?). Partly: the experts that replace each other after a flip are functionally close, but not redundant. We report the KL divergence KL(p∥p~)\mathrm{KL}(p \Vert \tilde p) of the next-token distribution, which is the expected NLL increase of a next token drawn from the clean model. In OLMoE, aggregate NLL is essentially unchanged (Table 6), yet flips redistribute probability among the candidate tokens: the top-10 set changes for 34% of flipped tokens, against 16% of the others (Table 9). Restoring a flipped token's own route removes about a third of its output change (Table 10), so the substituted experts compute something measurably different. For coalition experts, the changes are large (G1).

W2, Q3 (value and cost of J(∞)J^{(\infty)}). On what JJ adds beyond restating a negative margin, see bmYt W1; on its accuracy at depth, see bmYt W2. We added the cost comparison (Appendix J.8). On OLMoE, given the clean pass that both require, a single J(∞)J^{(\infty)} takes about as long as the ablated pass (0.84 s against 0.82 s per batch), while J(0)J^{(0)} and J(1)J^{(1)} take 1% and 22% of that time. Prediction becomes preferable when many perturbations are compared on the same tokens: because routing leverage is linear in the perturbation, all three estimators for our three interventions are computed in one propagation (1.29 s), whereas the ablation requires a forward pass per intervention (2.47 s). We expect this advantage to carry over to mixed-precision quantization of the experts. Quantizing an expert perturbs its output at every token that selected it, and J(∞)J^{(\infty)} propagates this perturbation like a removed write. A search over bit-width assignments then needs one propagation per expert and bit-width, after which each assignment is scored by summing the propagated perturbations, whereas running the quantized model requires a forward pass per assignment considered.

We would be very glad to answer any further questions or clarify any remaining points.

We thank the authors for the thorough response and the new experiments, which directly address our main concerns. The matched-norm control (G2) and the output analysis with the causal routing rescue (G1) are the analyses we requested.

The control result is informative: a random perturbation of the same norm flips nearly as many tokens as the ablation (47-78% vs. 52-81%), and largely the same ones. We appreciate that the authors revised their interpretation accordingly. We therefore ask the authors to ensure that the relevant parts of the paper (e.g., the introduction, the discussion of Section 4.1) reflect this revised interpretation, so that the flip rate is presented as a general property of Top-K routing under perturbation rather than as a measure of the influence of a specific expert.

The new cost comparison (Appendix L.8) confirms that a single J(∞)J^{(\infty)} costs about as much as the ablated pass. The multi-perturbation setting is a reasonable motivation for the predictor, although the proposed use for mixed-precision quantization has not yet been demonstrated.

Given these revisions, we raise our soundness score and keep our overall rating.

Reviewer_HXiPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The work presents a comprehensive analysis of how routing influence propagates through MoE architectures. The authors consider expert ablations (nullifications) to study the dependency of future routing decisions on two types of channels: Direct, when removal of the signal at a token affects the expert selection on the same token but on later layers, and Context-mediated, when expert ablation on previous tokens affects the decision on current tokens. The work demonstrates that context-mediated token flips are common; thus, MoE routing may not be simply attributed to a particular token only. It is demonstrated that the two channels combine almost linearly. Additionally, the work proposes different estimator values J0J^0, J1J^1, and J∞J^\infty which, when combined with the definition of margin-normalized routing leverage, form a good predictor of routing divergence.

Critical strengths
  1. The work provides a detailed study of the routing mechanism and ablations of TopK expert changes.

  2. Strong mathematical formalization.

  3. Proposed indicator metrics, the predictors that may be useful in the analysis of model behavioral changes under expert removal.

Critical weaknesses
  1. The work is very dense and some important things are moved to the appendix. A suggestion would be to state more clearly the experimental setting of the ablations in the main text.
Questions for authors
  1. The studied expert flips are not connected to logit changes. To be fair, the change of NLL was stated in the work, and for many flips it seems to be small. Could you apply the same analysis to the unembeddings, treating the logit selection in the same manner as the selection of an expert (TopK decoding)? If so, the proposed leverage could then also predict output-token flips, and relate them to the expert flips.

  2. Could you consider a similar analysis of the percentage of flipped tokens in the context of quantization (bf16) instead of expert removal?

  3. It would be nice to see results on several applications, pruning, quantization, etc. Could the proposed indicator be practically used as a measure of the influence of a particular module in general (not on the level of one token)?

4 comments

W1 (density; experimental setting). We thank the reviewer for this suggestion. Each experiment in Sections 4–6 now states which intervention it applies and which tokens it analyzes; the interventions and what counts as a routing divergence are defined at the start of Section 4. We also moved the taxonomy of routing flips (Table 2) and the attention-patching results (Table 3) from the appendix into Sections 4.2 and 4.3.

Q1 (leverage for output tokens). We now measure output-token flips directly and relate them to routing flips (G1, Tables 9 and 10). Our formulation also carries over to predicting them: the output layer is a Top-1 selection over the vocabulary, so routing leverage with the unembedding U\mathbf U in place of the router gives the margin-normalized criterion Jout=max⁡y≠y∗[(uy∗−uy)⋅δh]+/(oy∗−oy)J_{\mathrm{out}}=\max_{y\neq y^{\ast}}[(\mathbf{u}^{y^{\ast}}-\mathbf{u}^{y})\cdot\delta\mathbf{h}]_{+}/(o^{y^{\ast}}-o^{y}) for an output-token flip. Evaluating it requires an accurate estimate of the perturbation at the last layer, where first-order estimates are least accurate (Section 6.3), so we leave the predictor to future work.

Q2 (quantization instead of expert removal). We measured routing flips under bfloat16 instead of float32 inference, without any ablation (Appendix C, Table 5). bfloat16 alone flips the routing of 23–97% of tokens, consistent with G2: a perturbation of any source frequently flips Top-K routing. This is also why all our routing experiments run in float32.

Q3 (module-level influence; pruning and quantization). Yes. Routing leverage is computed per token, and aggregating it over tokens gives a module-level measure: the fraction of tokens with J>1J>1 estimates how many tokens a perturbation of expert uu reroutes. It only requires the perturbation of the expert's write, so it applies to quantization: quantizing expert uu replaces the removed write wjueout,juw_j^u\mathbf{e}^u_{\mathrm{out},j} by wju(eout,ju−e^out,ju)w_j^u(\mathbf{e}^u_{\mathrm{out},j}-\hat{\mathbf{e}}^u_{\mathrm{out},j}), the part of the write that quantization removes, where e^out,ju\hat{\mathbf{e}}^u_{\mathrm{out},j} is the quantized expert's output. Our influence graph (Appendix D) shows that module-level routing influence matters in practice: ranking experts by it finds coalitions whose joint ablation raises NLL by 3.93.9–33.9×33.9\times the sum of their individual effects, and experts that the magnitude criterion for super experts misses (Table 7), which are individually safe to prune but not jointly. A systematic evaluation across compression methods is beyond this paper and left to future work.

We would be very glad to answer any further questions or clarify any remaining points.

I would like to thank authors for provided responses to my questions.

[W1] This experimental setup positioning was addressed.

Now each experiment states the applied intervention and the analyzed tokens. The routing flips taxonomy and the attention patching are moved from appendix to Sections 4.2 and 4.3 which makes content easier to follow.

[Q1] I think my question was properly addressed.

The provided additional measurements (Appendix G) estimate the percentage of prediction changes (top-1, top-10 set) depending on whether the routing flips. For ordinary experts, the top-1 effect happens to be small, below 6%. This may also be stated in the main text (Section 4.1), as it determines the practical importance of routing flips.

The experiment with causal rescue on OLMoE is informative. However, it restores only the routing of the analyzed token, while the flips of context tokens remain. It would also be interesting to see the effect of restoring the routing of all tokens, so that the expert removal affects only the residual stream.

I agree that a detailed theoretical analysis of leverage as an output-flip predictor could be left as potential future work.

[Q2] I think the question is properly addressed in (Appendix C, Table 5).

The bfloat16 inference flips the routing of 23-97% tokens, the same order as under expert ablation. Therefore, the number of routing flips reflects a general sensitivity of Top-K routing.

[Q3] I suppose this question is addressed to a degree.

The coalitions detected with the influence-graph score inf indeed provide evidence for a module-level influence measure. For coalition experts the top-1 prediction changes much more often than for ordinary experts (Table 9). However, this compares only the top-ranked experts with ordinary ones (the composition of the coalitions is not varied).

The proposed measure, the fraction of tokens with J > 1, and its formulation for influence estimation are left for future study. It would be interesting to test inf and the aggregated leverage J with a broader sample of experts and in comparison with simple baselines such as activation magnitude.

Generally, I am satisfied with the work and the response, and I am leaning towards raising my score.

We thank the reviewer for the careful reading of our revision and for the follow-up suggestions.

For ordinary experts, the top-1 effect happens to be small, below 6%. This may also be stated in the main text (Section 4.1).

We agree, Section 4.1 now states it.

It would also be interesting to see the effect of restoring the routing of all tokens, so that the expert removal affects only the residual stream.

Thank you for this suggestion. We added this rescue to Table 10. Restoring the clean Top-K experts of every token j≤ij \le i removes a median 53% of the KL divergence, against 36% when restoring the analyzed token ii alone, and adds at most 6%. Routing changes thus carry about half of the output change, a third through the token's own flip and the rest through the flips of its context (Section 4.1, Appendix G).

It would be interesting to test inf and the aggregated leverage J with a broader sample of experts and in comparison with simple baselines such as activation magnitude.

Thank you, we added this in a new Appendix M. On 60 random OLMoE experts of varying depth, four from each of layers 0 to 14, we compare how well routing leverage, activation magnitude, routing load and our influence graph score inf(⋅)\mathrm{inf}(\cdot) rank the experts by the output effect of their ablation, measured as the 90th-percentile next-token KL divergence and the fraction of tokens whose top-1 next token changes (Table 21). Aggregated routing leverage, the fraction of tokens with J(∞)>1J^{(\infty)} > 1, ranks them best, with a Spearman correlation of 0.88 for top-1 changes and 0.80 for the KL divergence, against at most 0.62 for the other scores. Activation magnitude is even negatively correlated with both, as it grows with depth while the output effect shrinks.

We placed this in a new Appendix M to keep the references in our earlier responses valid, and would move it into Appendix E for the camera-ready version.

Reviewer_bmYtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper discussed how expert outputs in MOEs affect routing decisions across different tokens. The paper identifies two ways for this process. Direct changes via a token's representations and indirect changes via attention to earlier tokens. Moreover, it introduces a measurement to estimate when an expert's influence can effect subsequent routing decisions.

Critical strengths

The paper overall is easy to undrestand and evaluates an aspect of MoE routing behaviour. The use of expert and attention patching provides evidence of cross-token influence. Also, the evaluation provides several MoE models, with some analyses of routing boundaries.

Critical weaknesses

I found the following weaknesses in this manuscript:

  1. My main concern is the technical novelty of the proposed approach. The observation that expert outputs affect other tokens via attention, indeed, follows from the transformer architecture. Also, the routing leverage in Eq. (6) is a normalized change in routing scores relative to the Top-K margin. Moreover, as I understand, this follows from the linear router formulation in Eq. (4) and (5). Indeed, It is not clear what new understanding this formulation provides more than describing when a routing boundary is crossed.

  2. Another issue is the author's claim that the first routing divergence marks the beginning of a nonlinear routing cascade. As I understand, Section 6 of the paper analyzes the propagation before the first routing change, and J(∞)J^{(\infty)} assumes that the intermediate Top-KK assignments remain unchanged. However, once a routing change happens, this assumption is not valid anymore. So, I am not sure how the proposed formulation can explain the propagation of routing changes across later layers. Probably, there is a gap between the claimed explanation of routing cascades and what the analysis demonstrates.

  3. In Section 5.1, the paper claims that direct and context-mediated influences are approximately linear before the first routing change. However, as I understand, the analysis only considers the margin between the lowest-ranked selected expert and the highest-ranked unselected expert. It does not examine how other expert pairs are affected. Also, since the analysis is done before any routing change, the model still follows the same routing path. So, I am not sure if this linear behaviour is really a new finding about MoE routing or comes from the local linearity of the network.

  4. Another concern is that the paper generally evaluates expert influence by counting changes in Top-KK routing decisions. However, a change in the selected experts does not mean that the model's output is significantly affected. For example, two experts with similar routing scores may produce similar representations, so switching between them may have little impact on the final prediction. Although Table 1 reports routing changes of tokens, it is not clear how these changes affect the model behaviour. I believe this is one of the important missing analysis, because the paper uses routing changes as the main evidence of expert influence.

Questions for authors

Please refer to the weaknesses.

3 comments

W1 (novelty; what routing leverage adds beyond a boundary crossing). Attention gives earlier tokens a path to later routers, and we do not claim to have discovered this context-mediated channel. Our contribution is a mechanistic lens on routing: our interventions show how often that channel alone changes expert selection, how its influence evolves with depth, and how it combines with the direct channel.

These interventions reveal non-trivial structure in how the two channels (direct and context-mediated) combine. In each of the 18 ablations testing both, fewer than 3% of analyzed tokens flip only under the joint intervention, while in 12 of 18 the most common case is a token that flips through either channel alone (Section 4.2, Table 2). The two channels are thus largely redundant routes to the same routing flip, not synergistic ones. The context-mediated channel's median first flip is never earlier than that of the direct channel (all 23 ablations of Section 5.2), and the two channels add in margin space (G3). The architecture gives us no reason to expect this redundancy, this timing, or this additivity.

Routing leverage yields three insights that a boundary crossing alone does not. First, it separates which property of a perturbation decides a flip: beyond the first layer after the ablation, the perturbation's alignment with the router's contrast direction separates diverging tokens better than its magnitude in every model (Section 6.3), and the new random control confirms that the direction, not only the size, decides which tokens flip (G2). Second, it shows that a first-order account suffices before the cascade begins: from clean-pass quantities alone, J(∞)J^{(\infty)} outranks every component in every model up to the fourth layer after the ablation, and at the first it separates diverging from surviving tokens with AUROC 97.8–99.2%, whereas no single component exceeds 83.4% (Table 19). Third, it explains why small writes can matter: Phi L10E0 falls below the super-expert magnitude threshold, yet adding it to that model's two super experts raises their joint NLL increase from 0.80 to 1.33 nats (Table 7).

W2 (routing cascade). We claim that J(∞)J^{(\infty)} predicts where the cascade begins, not how it unfolds (Sections 1 and 6). Its derivation indeed keeps the Top-K assignments at their clean-pass values, which no longer holds once a context token has diverged, but Section 6 evaluates it on all analyzed tokens, including those whose context has diverged. J(∞)J^{(\infty)} stays accurate while the context has not diverged (AUROC 93.7–100%, Table 20) and degrades once it has, so its predictions fail exactly where an earlier divergence invalidates the clean-pass Top-K assignments. Figure 1 shows how the cascade then unfolds across tokens and layers. We now state this distinction explicitly in Sections 6 and 7: the onset is predicted, and the cascade is characterized empirically.

W3 (additivity only on the critical pair; local linearity). See G3. We added the requested all-pairs analysis and report additivity separately for analyzed tokens whose context has and has not diverged.

W4 (routing flips vs. model output). See G1. We added the requested output analysis and a causal routing rescue.

We would be very glad to answer any further questions or clarify any remaining points.

Thanks for the reply. I keep my rating, because I still have the same concern about novelty. The fact that expert outputs affect other tokens through attention is just how transformers work. In addition, Eq. (6) is basically a linear routing score relative to the Top-K margin. It's still not clear to me what new insight this gives beyond just showing when a routing boundary is crossed.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their constructive feedback. Revisions of the manuscript are highlighted in blue. We answer the main concerns here once.

G1. Do routing flips affect the model's output? (bmYt W4, HXiP Q1, 32ZM W1/Q2)

Following the reviewers' suggestion, we added two experiments on the analyzed tokens of Table 1, that is, tokens that never selected the ablated expert but follow at least one token that did.

  1. We compare the clean and ablated next-token distribution of each analyzed token, grouped by whether its routing flips (Appendix G, Table 9). In OLMoE, where 60% of these tokens flip, the top-1 next-token prediction changes for 3.4% of flipped tokens against 1.3% of the others, and the top-10 set for 34% against 16%. Flipped tokens' 90th-percentile KL divergence is 5.5×5.5\times larger (44–11×11\times across models). Per token, these changes are small for most experts, which explains why aggregate NLL barely moves. For coalition experts, nearly every token flips and the top-1 prediction changes for up to 24% of them.

  2. To test whether the flip itself causes the output change, we ran a causal routing rescue on OLMoE (Table 10). We keep the experts ablated and all context tokens as in the ablated run, and restore only the analyzed token's clean Top-K experts from its first divergence onward, with weights from its current router scores. The rescue removes 24–54% of the KL divergence (median 36%) and adds at most 6% at tokens where it hurts, and top-10 prediction changes drop from 33.6% to 25.5%. A token's own routing flip therefore causes about a third of its output change; the remainder reaches it through the perturbed context.

G2. How much of the flip rate is specific to the ablated expert? (32ZM W1/Q1, HXiP Q2)

We added the suggested matched-norm control to Table 1. We subtract Qvj\mathbf Q\mathbf{v}_j instead of each expert's write vj\mathbf{v}_j, where Q\mathbf Q is a random orthogonal matrix drawn per sequence (3 seeds). This preserves write norms and pairwise correlations. The control flips nearly as many tokens (median 47–78%, against 52–81% under ablation). The ablation flips more often in all five models, by 1.2–5.3 pp, and earlier or at the same layer in all five. Ablation and control also flip the same tokens: their flipped sets overlap as much as those of two control seeds (Jaccard, Table 1).

We have adjusted how we interpret the flip rate accordingly, while the mechanism is unchanged: a perturbation of the same norm as an expert's write changes the routing of many subsequent tokens that never selected that expert, because attention carries the perturbation to them. This is a general property of Top-K routing, not a signature of the particular expert. Which tokens flip depends on the perturbation's direction: the tokens flipped by two random directions of the same norm overlap by 57% at the median. Routing leverage captures this through the perturbation's alignment with the router relative to the margin (Section 6).

G3. Is the additivity of Section 5.1 just local linearity in the clean-path regime? (bmYt W3)

No. Our wording suggested that Section 5.1 measures additivity in the clean-path regime, and we have clarified this in the revised manuscript. The additivity window ends at the analyzed token's own first divergence, not at the first divergence of its context. Context tokens often diverge earlier: at 19–84% of analyzed tokens and layers (Table 11). Additivity holds despite this diverged context, which makes the finding considerably stronger than local linearity in the clean-path regime.

Following bmYt's suggestion, we now evaluate additivity over every selected/unselected expert pair, each relative to its own margin, rather than only the critical pair (Table 11). The median error is at most 2.5% of the margin in all 22 ablations and at most 9.9% at the 90th percentile, both where the context has already diverged and where it has not. The critical pair, which the initial submission reported, is the hardest case, with median errors of up to 5.9% and 90th percentiles of up to 28.1%.

We further analyzed what limits additivity once the analyzed token itself diverges (Appendix H). Because Top-K selection is discrete, a flip replaces one expert's output by another's, a change that does not scale with the perturbation that caused it. Where one channel alone flips the token, δdir+δctx\delta_{\mathrm{dir}}+\delta_{\mathrm{ctx}} contains this replacement once, as δjoint\delta_{\mathrm{joint}} does, and additivity holds (median relative error 0% for the direct channel, 9% for the context-mediated one). Where both channels flip the token on their own, the sum contains the replacement twice (35%), and where only both together do, not at all (54%). Continuous nonlinearities (attention softmax, normalization, activation functions) would affect all of these groups alike. Additivity therefore extends beyond the clean-path regime and is limited only by the discrete Top-K selection.

AuthorsOpenReview ↗

Summary of revisions

All new and revised passages appear in blue. The taxonomy of routing flips (Table 2) and the attention-patching results (Table 3) were moved unchanged from the appendix into the main text and are therefore not marked.

Main text

  • Matched-norm random control for the flip rates (Section 4.1, Table 1).
  • Intervention and analyzed tokens stated for each experiment (Sections 4.1, 5.1, 5.2 and 6.2).
  • Taxonomy of routing flips and attention patching moved into Sections 4.2 and 4.3.
  • Additivity over all expert pairs, for clean and diverged context (Section 5.1, Table 11).
  • Routing leverage stated as predicting the onset of the routing cascade (Sections 6.1 and 7).

Appendix

  • Routing flips under bfloat16 instead of float32 inference (Appendix C).
  • Effect of routing flips on the model output, with a causal routing rescue (Appendix G).
  • Additivity beyond the analyzed token's first divergence (Appendix H).
  • Computational cost of the routing-leverage estimators (Appendix J.8).

Following Reviewer HXiP's suggestions, we further added two analyses, also marked in blue.

Appendix

  • Routing rescue restoring the routing of every token, in addition to the rescue of the analyzed token alone (Appendix G, Table 10).
  • Routing leverage compared with activation magnitude, routing load and the influence graph score inf in predicting the output effect of ablating 60 random experts (Appendix M, Table 21).

All table and appendix references in our earlier responses remain valid.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.