- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SA-MoE, which introduces a persistent state to trace cross-layer routing contributions in MoE models. The paper evaluates its effects on routing behavior and attribution. While the idea is interesting, the practical significance and advantages over existing post-hoc attribution methods remain unclear.
Critical strengths
-
The paper introduces a persistent state to explicitly track cross-layer contributions to MoE routing decisions, with a mathematically grounded decomposition of state-mediated routing margins.
-
The paper conducts intervention-based experiments to investigate how persistent state updates affect routing decisions and downstream model behavior.
Critical weaknesses
-
Unclear practical significance. The paper demonstrates that the persistent state enables explicit decomposition of cross-layer routing contributions, but it remains unclear how this capability translates into meaningful benefits for MoE analysis, auditing, or debugging. The paper would benefit from concrete evidence that the proposed mechanism enables insights or tasks that are difficult to achieve with existing methods.
-
Insufficient justification over existing attribution methods. The paper does not adequately establish why architectural modification is necessary compared with applying post-hoc attribution methods to standard MoEs. In particular, gradient-times-write attribution achieves stronger correlation with full-forward intervention effects than the proposed ledger in some reported settings. This raises questions about the unique explanatory value of the ledger, especially given that it only decomposes the state-mediated component of routing decisions.
Questions for authors
-
How sensitive are the results to the dimensionality and clipping range of the persistent state? Do different configurations lead to substantially different routing behavior or attribution quality?
-
How does the proposed method scale to larger MoE models with more layers and experts? Does the routing ledger remain informative as model scale increases?
-
Given that gradient-times-write attribution achieves stronger correlation with full-forward intervention effects in some settings, what unique advantages does the proposed routing ledger offer?