- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes how upstream residual writes contribute to expert selection in Transformers that employ a linear router. It precisely decomposes the margin between the weakest selected expert and the strongest excluded expert into individual component contributions, while disentangling residual evidence from balancing-policy bias. Unlike individual score contributions, this margin-based contribution is invariant to common router-row shifts. Through interventions, the authors examine whether this accounting method successfully identifies the computations critical to routing and, ultimately, the model's final behavior. The broader objective of the work is to distinguish between exact accounting and causal explanation, while leveraging the model's architectural structure to enable direct observation of specific components of its computation. Consequently, the paper provides tools for analyzing individual routing decisions, along with an assessment of the scope of the explanations they offer.
Critical strengths
-
The chosen targets for explanation are appropriate. Since expert selection is determined by comparing candidates, the margin between selected and excluded elements serves as a meaningful target for attribution. The common-shift argument highlights a genuine ambiguity that arises when interpreting the sign of a raw score as either promotion or inhibition. Furthermore, distinguishing between residual evidence and balancing policy clarifies exactly what an attribution is explaining.
-
The framework makes a useful distinction between the approach to computation and the understanding of its effects. Rather than assuming that a faithful reconstruction guarantees a mechanistic understanding, it directly examines whether an exact decomposition can truly support causal conclusions. Comparisons involving signed subspaces, routing interventions, and residual interventions reveal the limitations of such inferences. This perspective holds academic value, even in the absence of new predictive models or improvements in performance.
Critical weaknesses
-
The additional explanatory contribution is limited. Li et al. [1] already provide a component-level decomposition of router scores, and GRIP [2] addresses routing stability through the margin between selected and excluded experts. While this study correctly refines the interpretation of raw-score signs, it fails to sufficiently establish what significant new insights the boundary graph provides. The problem is not that the structure is simple, but that there is limited evidence regarding how the understanding of MoE computation actually changes beyond the ambiguity of the identified attribution.
-
The scope of validation is narrow relative to the broader goals of interpretability research. As the authors acknowledge, the graph describes localized routing comparisons based on coarse residual writes, without elucidating the semantics of the experts or the specific computations that generate these writes. Since the comparisons involving surrogate models are also confined to this limited target, they do not demonstrate a broader advantage in terms of interpretability. Furthermore, as it focuses on a self-configured early-training testbed, it is unclear whether it alters important conclusions regarding existing pretrained MoE or predicts when routing evidence becomes behaviorally significant.
-
There is an operator–result discrepancy in the core intervention. In the local intervention described in Appendix C.1, removing opposing contributions leads to an increase in the scores of selected experts and a decrease in the scores of excluded experts. If all other executed scores at the same router remain constant, the set of selected experts should theoretically stay the same. Therefore, the reported set change requires an explanation beyond reinterpreting the result as a set change rather than a fixed-pair reversal. While this issue weakens the causal validation, it does not invalidate the decomposition identity itself.
-
As the authors acknowledge, the behavioral findings do not definitively isolate a routing-specific mechanism. Since the subspaces obtained from positive and negative evidence are similar and interpretation varies depending on intervention strength, the general dominant residual direction remains as an alternative. A comparison with unweighted write PCA and residual PCA is necessary. Furthermore, the routing contribution after changing the residual content cannot be determined solely by the fact that the router-only effect is small in the clean residual. A rescue condition would help assess possible routing–content interaction.
-
The text in the figures becomes pixelated when enlarged, making it difficult to read. Some cross-references to the appendix are also broken and should be corrected.
[1] Wengang Li, Lingqi Zhang, Toshio Endo, and Mohamed Wahib. Understanding Cross-Layer Contributions to Mixture-of-Experts Routing in LLMs. In International Conference on Learning Representations (ICLR), 2026.
[2] Andy Zhu, Rongzhe Wei, Yupu Gu, and Pan Li. GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints. arXiv preprint arXiv:2601.16905, 2026.
Questions for authors
- When the existing component-score analysis is replaced with the proposed boundary accounting, what important conclusions regarding MoE routing change? How can these conclusions be independently verified?
- By specifying the intervention and measurement location, and verifying the corresponding expert scores and selected identities, can you explain why a set change occurs after removing opposing edges?
- Does the distinct behavioral effect persist when compared against unweighted write PCA and residual PCA, given a comparable intervention strength?
- What happens when the baseline expert identities and gates are restored following a residual ablation? How much of the loss increase remains after this rescue?
3 comments
PCA Controls (Question 3), Rescue (Question 4) and Presentation
Question 3: write PCA and residual PCA at comparable strength. We compared the margin-derived bases with four kinds of two-dimensional basis:
- unweighted, centred write PCA;
- centred residual PCA;
- an uncentred residual second-moment basis;
- random bases.
Every basis was tested at unit, margin-matched and energy-matched doses, on disjoint discovery, calibration and test ranges of each model's held-out evaluation text.
The results support the alternative you raised.
- Write PCA. The margin-derived basis largely coincides with unweighted write PCA (overlap 0.92–0.999 at most layers), and ΔCE is similar. At unit dose, ΔCE is 2.28 for both in the 6-layer model (seed 0, layer 2). In the 81M model (layer 4) it is 1.61 for the margin-derived basis and 1.83 for write PCA.
- Residual PCA. Centred residual PCA spans different directions. Its effect is larger or smaller than that of the margin-derived basis depending on the layer.
- Random bases. At unit dose, random projections of the same dimension change loss by at most 0.08 nats. When scaled to the same margin or energy dose, they have effects comparable to the margin-derived bases, and often larger in the 16-layer and 81M models. The submission already noted this.
So no distinctive behavioural effect remains at comparable strength. The revised Section 4.3 says that the load-bearing directions are the dominant directions of the upstream writes, and that a routing-specific effect is not established.
Question 4: rescue. We ablate the subspace in the residual stream and then restore the clean expert identities and gates. We restore them either at the ablated router alone, or at that router and every later one.
| Model | ΔCE, ablation only | Rescue at ablated router | Rescue at all later routers |
|---|---|---|---|
| 6-layer, seed 0, layer 2 | 2.28 | 1.72 | 1.21 |
| 16-layer WikiText, layer 0 | 1.63 | 1.58 | 1.31 |
| 81M FineWeb-Edu, layer 0 | 3.22 | 3.23 | 3.46 |
Even with every later routing decision restored, ΔCE stays at 1.21 of 2.28 nats in the 6-layer model and 1.31 of 1.63 in the 16-layer model. In the 81M model it rises from 3.22 to 3.46. The effect of rescue depends on the model, and most of the loss increase remains.
Because routing and content can interact, these numbers do not identify a single "routing share". They confirm your point that a small router-only effect on the clean residual does not mean routing is irrelevant. The revision reports this table.
Figures and cross-references. The revised PDF will use vector figures and fix all appendix references.
I thank the authors for their thoughtful comments and analysis. It is clearly a positive step that the authors have retracted the incorrect results and added further verification. The answer to Q1 partially addresses my concern, and the answer to Q2 resolves the operator discrepancy in practice. However, further analyses and experiments seem necessary to fully address Q3 and Q4. Regarding the figures, the equations in them should not become pixelated when zoomed in. In addition, it would be better if the authors mentioned and described the "OLMoE" itself in the main paper. For these reasons, I will keep my rating.
Operator Discrepancy (Question 2), What Changes (Question 1), Novelty and Scope
We thank the reviewer for a careful and constructive review. We address your four questions below, with new theory and experiments where needed. This comment covers Questions 2 and 1, novelty and scope. We start with Question 2 because it concerns the soundness of the reported results. The next comment covers Questions 3 and 4 and presentation.
Question 2: why does the set change after opposing edges are removed? It should not, and with the new runner it does not.
The intervention acts on the router of layer l. It changes only two executed scores, each by αa/2 in opposite directions:
We read out the top-k set at that same router. When a ≤ 0, s_e does not decrease, s_j does not increase, and the other scores are fixed, so the set cannot change. The revision proves this as Proposition 3.
The nonzero numbers in the submission came from an earlier implementation that was inconsistent with this operator. We have replaced them.
The new runner checks the property against the executed scores and selected identities at every evaluated position. It finds no set changes for negative edges on any of the five checkpoints or on the released OLMoE-1B-7B. Away from numerical ties, each positive-edge set change is the predicted exchange of e for j; there are 7 near-tie exceptions among about 2.2M positions.
Question 1: which conclusions change, and how can they be checked independently? Some conclusions are withdrawn, others are narrowed, and some become testable predictions.
Novelty relative to Li et al. and GRIP.
What we add is an exact decomposition of that margin over residual writes, and tests of which causal conclusions it supports.
Scope. We agree with this criticism.