acceptodds
Under review as a conference paper at ICLR 2027

When Can Pairwise Routing Dynamics Be Tested in Mixture-of-Experts? Identifiability Conditions and Evidence from a Learned Router

Abstract

How can pairwise routing dynamics in a learned mixture-of-experts (MoE) model be tested against a reduced stochastic law? We provide a falsifiable measurement criterion for this question. The criterion is instantiated by a specified two-expert (E=2) score-gap chain with an exact endpoint probability and a central first-order blind spot. For the aggregate soft-routing schema in our historical eight-expert (E=8) records, we prove that the fixed hard count, one-selection transition, and specified endpoint are not identifiable; that is, they are not uniquely determined by the recorded quantities. This is a schema-specific boundary on a dynamics test, not a universal impossibility result for MoE-to-reduced-model transfer. We provide a logging contract for a finite-chain dynamics test and verify the resulting analysis procedure on an imposed E=2 subsystem (4,608 independently replayed trajectories). We then measure pairwise routing dynamics in a 17M-parameter, four-layer, character-level E=8 MoE that makes one hard expert choice per token during the forward pass. Training, expert-pair choice, and evaluation use disjoint token spans. Its predefined six-feature linear model did not improve mean leave-one-seed-out MSE over router learning rate alone (mean MSE difference ; 5 of 8 seed folds favor it). A narrower local effect, gain-dependent selected-pair soft-share movement, replicates: movement is larger at than at for 15 of 16 paired seeds on Shakespeare and an independent enwik8 prefix. This effect is reproducible local routing dynamics, not a validated E=2 transfer law.

Primary area:
learning theory
Keywords:
Mixture-of-Experts, routing dynamics, identifiability, stochastic processes, expert routing, measurement

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4kxFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks when routing dynamics observed in a learned multi-expert MoE can be meaningfully compared with a reduced two-expert stochastic model. It derives an exact endpoint probability for a specified E=2 score-gap chain, proves that the historical aggregate E=8 logging schema cannot identify the required hard state, one-step transition, or endpoint, and proposes a logging contract for a valid dynamics test. An imposed E=2 control validates the analysis procedure, while experiments on a learned E=8 router show that a predefined aggregate-soft predictor does not outperform a learning-rate-only baseline. A narrower gain-dependent soft-share movement nevertheless replicates across Shakespeare and enwik8.

Critical strengths
  1. The paper clearly shows that matching aggregate curves is insufficient when state, step, score-change, and measurement maps are not identified.
  2. The authors do not overclaim the E=8 results and explicitly distinguish a reproducible local routing phenomenon from validation of the E=2 transfer law.
Critical weaknesses
  1. The paper primarily establishes that a particular historical logging schema is insufficient, rather than demonstrating a new general law of learned MoE routing dynamics.
  2. The imposed-chain control verifies the analysis pipeline, but the actual learned-router experiment does not establish the state or transition correspondence required by the theory.
Questions for authors

Q1. Have the authors collected any new runs with the proposed logging contract that would directly test the E=2 transition law on a learned router? Q2. How sensitive is the non-identifiability result to adding one or two additional practical logging fields, such as hard pair counts or tokenwise score gaps? Q3. Beyond the reported gain-dependent soft-share movement, is there evidence that the learned E=8 dynamics are well approximated by any low-dimensional stochastic process?

Reviewer_a5BPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper asks when pairwise routing dynamics in a learned MoE can be tested against a reduced stochastic law, using a two-expert “rich-get-richer” score-gap random walk. The paper argues that such a test requires explicit state, step, score-change and measurement maps, and proves (Theorem 1) that the authors’ recorded batch-level logs by themselves cannot identify the hard routing state or one-step transition needed. The author then proposes a token-level logging scheme and audit protocol for future tests but did not run the associated experiments.

Critical strengths

The paper seem to be honest about their null/inconclusive results, with good statistical practices and experiment/data hygiene.

Critical weaknesses

The paper seem to contain mostly negative results from the experiments (i.e. either the authors did not gather the correct types of data to substantiate their hypotheses, or that the data was not statistically significant / general enough to support a concrete claim). It's not clear if there is any conclusive qualitative statement that is actually supported. The writing is also very hard to follow because half of the statements are "Our X is NOT Y".

Questions for authors

While the "historical role-separated records" misses critical token-wise data for your analysis, in Appdx. C it was presented "The current measurement interface records these fields for future studies; those future records must be analyzed separately and cannot alter the reported historical results." (what does the second part mean?). Why not run the tests then?

Reviewer_P1EHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether aggregate observations from a learned MoE router identify the state and transitions of a specified two-expert stochastic process. It derives the process's absorption probability, constructs indistinguishable aggregate logs with different hard-routing states, and proposes a logging protocol. Experiments distinguish a failed aggregate-soft prediction model from a reproducible local effect of increasing the router learning rate.

Critical strengths

The explicit construction in Appendix A makes the information loss precise: identical aggregate probabilities can conceal different hard counts and zero, one-token, or multi-token changes. The experimental analysis separates training, pair selection, and evaluation, and uses seeds rather than individual windows as the inferential unit. The paper reports its negative prediction result and carefully limits what the replicated movement effect establishes.

Critical weaknesses

The main limitation is the amount of new knowledge established about learned routers. The absorption calculation and perturbation bound are standard, while the non-identifiability theorem concerns a particular lossy logging interface. The proposed richer interface is not used to complete a dynamics test on a learned router. A convincing application of the protocol, including a rejected reduced-model hypothesis, would make the methodological contribution more substantial. The positive empirical result compares movement after updates with learning-rate multipliers 50 and 1. Larger movement is already expected from first-order sensitivity to update size. A comparison with this elementary prediction, or a report of movement normalized by update magnitude, would help establish what the result reveals beyond learning-rate scaling. The central blind spot is stronger than the first-order discussion suggests. For even NN, reflection symmetry gives hN/2(δ)=1/2h_{N/2}(\delta)=1/2 for every δ\delta in the defined centered chain. Thus this endpoint statistic has no higher-order sensitivity to δ\delta at the center either. Section 3.3 should distinguish this exact invariance from effects in other observables or other processes. The imposed-chain validation mainly reports central-start frequencies, where the prediction is always 1/21/2. These checks cannot distinguish the proposed state-dependent transition law from a neutral symmetric walk. Off-center starts are needed to validate recovery of the nontrivial absorption curve.

Questions for authors

Can the richer logging contract support a complete held-out transition or endpoint test on a learned router? Does the movement effect depart from a first-order prediction based on the router update and the soft-share Jacobian? Can the imposed-chain control report off-center endpoint frequencies with uncertainty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.