acceptodds
Under review as a conference paper at ICLR 2027

Shared Router Dynamics in Sparse MoEs: From Canonical Geometry to Expert Caching

Abstract

Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer, yet their routing states exhibit substantial predictability across depth. We ask whether this predictability reflects reusable structure that is obscured by layer-specific coordinate systems. We isolate each router's control subspace and show that its factorization has an exact orthogonal gauge symmetry, which defines a principled quotient space for cross-layer comparison. We then fix this gauge globally with generalized orthogonal Procrustes analysis before fitting any transition model. Across four MoE architectures, a single shared transition recovers 79–90% of the held-out of independently fitted layer-specific dynamics, and the canonical forecasting advantage replicates on a second corpus across multiple architectures. Although generic residual representations are also predictable across depth, matched-rank comparisons show that canonical router coordinates preserve substantially more information about expert selection. Direct interventions nevertheless reveal a predictive–causal gap: accurate routing-state predictions need not preserve language-model loss when substituted for native routing. As a downstream application, we use canonical cross-layer forecasts to anticipate later expert demand while leaving native routing unchanged. In physical expert-offloading experiments, forecast-guided caching reduces expert transfers by up to 17.0% and improves token throughput by up to 5.7%, while preserving native routes and outputs. These results provide evidence for reusable routing geometry and dynamics across depth and show that the same structure can support MoE memory management without replacing the router itself.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Routing Dynamics, Representation Alignment, Expert Caching, Expert Offloading, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D4fwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether MoE routing shares a common structure across layers once layer-specific coordinates are removed. It projects hidden states onto each router's control subspace, aligns all layers into one coordinate system with generalized Procrustes analysis, and fits a single shared linear transition. Across four MoE models, the shared transition recovers 79–90% of the R2 of layer-specific transitions, and using its forecasts for expert-cache eviction cuts expert transfers by up to 17% and raises throughput by up to 5.7%.

Critical strengths
  1. Interesting question with a clean formulation.
Critical weaknesses
  1. The evidence for shared cross-layer routing dynamics is not sufficient. I reproduced the Table 1 analysis on Granite and implemented the singular-spectrum check that Section 3.4 promises but never reports. Random per-layer subspaces pass this check just as well as the router subspaces. When all layers use one fixed residual subspace of the same rank, η_share is about 0.97, higher than the 0.88 obtained with router coordinates. This suggests that much of the cross-layer sharing may come from the smoothness of the residual stream rather than from router geometry. Section 5.2 does not settle this, since router-control coordinates reconstruct router logits by construction, so their higher top-k recall over residual PCA is expected. The baselines that could test this are missing: applying the future layer's router directly to the current hidden state, and a shared transition in a residual subspace common to all layers.

  2. The caching baseline is an LRU that thrashes: in five of the six cache settings its hit rate is 0%. There is no comparison with LFU, a Belady oracle, or existing prefetching methods such as Fate or Pre-gated MoE.

Questions for authors
  1. The causal blocks are chosen on an exploratory split. By what criterion, and across all candidate blocks, how often does Shared beat Identity?
  2. Can you add results for the missing baselines?
3 comments
AuthorsOpenReview ↗

Response to Reviewer D4fw (1/3)

We thank the reviewer for reproducing Table 1 and for proposing two concrete controls. We ran both controls exactly as described and added all requested baselines. The results are summarized below and are included in the revised PDF; section and table numbers refer to the revision.

We first clarify the structure of the paper's contributions. The central contribution is scientific: the router factorization has an exact orthogonal gauge symmetry, and once this symmetry is removed, routing dynamics are largely shared across depth. The cache study is a supporting application demonstrating that the canonical space is practically useful (Section 6). The new results strengthen the scientific core directly and place the cache study against strong baselines.

W1. Router specificity versus residual smoothness

The reviewer's numbers. We reproduce them. In one fixed residual-PCA subspace of the same rank, ηshare=0.957\eta_{\rm share} = 0.957 on Granite, and a fixed random subspace gives 0.966. In any fixed subspace, adjacent layers are already nearly aligned, so ηshare\eta_{\rm share} approaches 1 for every representation. We therefore add two measures that subtract persistence.

Gain beyond identity. The identity-adjusted ratio

ηΔ=Rshared2−Rid2Rlayer2−Rid2\eta_\Delta = \frac{R^2_{\rm shared} - R^2_{\rm id}}{R^2_{\rm layer} - R^2_{\rm id}}

is the fraction of the per-layer gain beyond persistence that one shared map recovers. Fig. 1(b) already shows this quantity. Across 4 architectures and 3 seeds, ηΔ=0.65\eta_\Delta = 0.65–0.730.73.

Routing gain under identical readouts. This is the main specificity test. For each representation we measure how much the shared transition improves future top-kk recall over persistence, using the same document splits, ridge grid and one learned readout per target layer. No representation reconstructs router logits by construction. Results for h=1h = 1 on 128 test documents (Table 2):

ModelRepresentationShared mapPer-layer maps
Graniterouter + GPA (ours)+0.0159 [0.0154, 0.0163]+0.0432
fixed residual PCA (reviewer)+0.0000 [−0.0001-0.0001, 0.0002]+0.0069
per-layer PCA + GPA−0.0006-0.0006+0.0061
fixed random (5 seeds)−0.0032-0.0032+0.0015
per-layer random + GPA (reviewer)−0.0358-0.0358−0.0315-0.0315
Phi-tinyrouter + GPA (ours)+0.0206 [0.0185, 0.0226]+0.0882
fixed residual PCA−0.0013-0.0013 [−0.0017-0.0017, −0.0011-0.0011]+0.0008
per-layer PCA + GPA−0.0054-0.0054+0.0017
OLMoErouter + GPA (ours)+0.0111 [0.0106, 0.0116]+0.0439
fixed residual PCA−0.0009-0.0009 [−0.0012-0.0012, −0.0006-0.0006]+0.0061
per-layer PCA + GPA−0.0007-0.0007+0.0081

All random controls on Phi-tiny and OLMoE are also negative (full table with ηshare\eta_{\rm share} and ηΔ\eta_\Delta in Table 2).

The key observation is the contrast between the two columns. In residual PCA, per-layer maps still improve routing (up to +0.008), but the shared map recovers none of this gain: the dynamics there are layer-specific. In router coordinates, the shared map recovers 23–37% of the per-layer gain with 1515–39×39\times fewer parameters. Sharing appears only after the router gauge is removed, which is the paper's central claim, now tested against the reviewer's own alternative. The paired difference, router minus fixed residual PCA, is +0.0159 [0.0154, 0.0163] on Granite, +0.022 [0.020, 0.024] on Phi-tiny and +0.0120 [0.0115, 0.0125] on OLMoE.

Random subspaces. In per-layer random subspaces, persistence has negative R2R^2, so any fitted map closes most of the state-space gap and ηΔ\eta_\Delta is high. Yet both maps reduce routing recall. State-space ratios alone therefore cannot establish specificity; the routing gain can.

Information versus shared dynamics. A residual PCA of the same rank can retain slightly more same-token routing information (persistence recall 0.797 vs. 0.758 on Granite), because it also contains directions outside the router's control space. Our claim concerns shared dynamics, and only router coordinates exhibit them.

AuthorsOpenReview ↗

Response to Reviewer D4fw (2/3)

W1 (continued). Requested baselines and the spectral check

Future router applied to the current hidden state. We compare the direct shared horizon map with the next layers' routers applied to the current hidden state ("stale router"). Granite, 128 test documents, recall difference with paired 95% confidence intervals (Appendix Table 6):

Horizon hh12481623
Shared map −- stale router−0.109-0.109−0.091-0.091−0.051-0.051+0.000+0.075 [0.070, 0.080]+0.273 [0.266, 0.279]
Source layers2322201681

The two predictors are complementary. The stale router uses the full hidden state and a router evaluation at every layer, and is stronger 1–4 layers ahead. The shared map uses a single vector of router logits and is stronger from 16 layers ahead. This long range is exactly where the shared map is most useful for caching (Section 5.5).

Shared transition in a residual subspace common to all layers. Reported above and in Table 2 for three architectures, together with per-layer PCA and both random controls.

Singular-spectrum check. We agree that it does not separate representations. Appendix B now reports it for all representations (median normalized defect over 23 transitions, range over the ridge grid):

RepresentationNative tokensToken-shuffled null
Router + GPA (ours)0.36–0.540.98–0.99
Pooled residual PCA0.19–0.450.98–0.99
Per-layer PCA + GPA0.25–0.330.98–0.99
Fixed random0.13–0.290.98

Router coordinates satisfy the necessary condition with a wide margin over the null. Fixed subspaces also show low defects, for the same reason that their ηshare\eta_{\rm share} is high. The check is therefore reported as a necessary condition, and specificity is established by the routing gain.

W2 and Q2. Caching against LFU, Belady/MIN, Fate and Pre-gated MoE

Eviction on all four architectures. We replay real router traces with the same demand-load semantics as the hardware cache; the simulator matches hardware access counts exactly. Rules and settings are selected on validation documents and frozen before testing. The grid has 4 models ×\times 3 seeds ×\times 4 disjoint workloads ×\times 3 capacities = 144 cells. Loads relative to LFU at capacity 0.1875⋅L⋅E0.1875 \cdot L \cdot E (Appendix Table 14):

ModelLRUMIN (oracle)OursOurs without forecast
Granite1.5430.6800.9670.972
OLMoE0.9570.5330.8400.844
Phi-tiny1.0080.5640.8860.886
IBM Shared0.8670.4430.7720.777

Over all 144 cells, our policy uses 6.0% fewer loads than the better of LFU and LRU (geometric mean), 13.6% fewer than LFU and 12.5% fewer than LRU. It is best in 120 cells and closes 27% of the LFU-to-MIN gap. The forecast lowers loads further in 104 of 144 cells. The remaining gap to MIN is reuse across tokens, which no within-token predictor, including Fate's, can capture.

AuthorsOpenReview ↗

Response to Reviewer D4fw (3/3)

W2 and Q2 (continued). Prefetching on hardware

Setup. Granite-3.0-1B-A400M, FP32, RTX 5090 (PCIe 5), fixed capacity of 147 layer-experts. Each session decodes 32 held-out documents ×\times 128 tokens, over 2 workloads in 2 policy orders. Every method, including Fate's prediction core (next-layer gate applied to the current hidden state), is tuned on validation prompts only. Generated tokens and native routes are bitwise identical across all policies (Table 3).

PolicyTok/s vs. LFU [min, max]vs. FatePredictor ms/token
LRU−14.5%-14.5\%–
EMA eviction, no forecast+1.8%+1.8\%–
Ours, shared map + prefetch+8.9%+8.9\% [7.8, 9.5]−1.9%-1.9\%0.21
Ours, shared map + low-rank correction+11.0%+11.0\% [10.7, 11.4]0.0% [−1.2-1.2, 1.0]0.21
Per-pair maps, same prefetch+10.6%+10.6\% [10.3, 11.1]−0.3%-0.3\%0.21
Fate prediction core (GPU, tuned)+11.0%+11.0\% [9.6, 12.3]–3.54

Matching Fate at a fraction of the cost. Our forecaster matches Fate's throughput and leads in 2 of 4 sessions. On a second machine with the same frozen settings, the ratio is 1.000 [0.995, 1.006], with both 7.5% faster than LFU. Most of the gain comes from the forecast: eviction without a forecast gives 1.8 of the 11.0 points.

Model size. The forecast runs once per token, from router logits only, without the hidden state or a router evaluation at every layer. It costs 0.21 ms per token against 3.54 for Fate (17×17\times less). Each horizon uses one closed-form map, with no gradient training and no change to the model. One map per horizon replaces one per layer pair, which gives 2222–59×59\times fewer coefficients on Granite.

Scope. On OLMoE, with 64 experts per layer, a single forecast from layer 0 yields too few useful prefetches (32–36%) to improve on forecast-free eviction, and Fate's per-layer prediction (+16.6%+16.6\% over LFU) justifies its 11×11\times higher cost (Appendix Table 15). The once-per-token forecast is the low-overhead option for moderate expert counts, and the revision states this explicitly.

Pre-gated MoE requires a pre-gating function trained into the model and does not apply to unmodified checkpoints, which is our setting. We therefore compare with Fate's prediction core, the closest training-free method.

Q1. Selection of causal blocks and frequency of Shared beating Identity

Selection criterion. Candidate windows were ranked on an exploratory split by NLLShared−NLLnative\mathrm{NLL}_{\rm Shared} - \mathrm{NLL}_{\rm native}, and the top blocks were fixed before replication (Section 5.4, Appendix D).

All windows of the full grids (bootstrap with Holm correction, Appendix Table 8):

ModelWindowsShared better (point estimate)Shared significantly better
Granite, seeds 101/202/303115 each28 / 25 / 271 / 2 / 1
IBM Shared211531
OLMoE67100

Replication on fresh documents. OLMoE 8–9 outperforms identity transport on all three original splits and on 32 fresh documents with verified no-op and reconstruction controls: Δ\DeltaNLL(AA −- Identity) = −0.0150-0.0150 [−0.0273-0.0273, −0.0028-0.0028] with a frozen source state and −0.0123-0.0123 [−0.0201-0.0201, −0.0044-0.0044] with the original hook (Appendix Table 10). The causal claim is stated for this pre-registered block, consistent with the predictive–causal gap stated in the abstract.

Summary

Both weaknesses are addressed with the controls the reviewer proposed. Shared dynamics are specific to router coordinates: a single shared map improves future expert selection only after the router gauge is removed, while in the reviewer's residual subspace the dynamics remain layer-specific. As a supporting application, the cache study now uses LFU, Belady/MIN, per-pair maps and Fate's core as baselines, and matches the strongest training-free predictor with a 17×17\times cheaper, closed-form forecast.

Changes in the revision. All results above are in the revised PDF: Sections 3.4, 5.2, 5.4 and 5.5, Tables 1–3, and Appendices B, D, E and F, with the original V100 study kept in Appendix Table 16 relative to LRU.

We thank the reviewer again and are happy to provide further clarification.

Reviewer_us9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether routing states across different layers of sparse MoE models follow shared dynamics after removing differences in their coordinate systems. The authors extract router-control states, align them using generalized Procrustes analysis, and learn a single linear transition shared across layers.

Experiments on four MoE architectures show that the shared transition achieves 79–90% of the R² of layer-specific models. The authors also study routing fidelity and the effects of replacing native routing states with predicted ones. As a practical application, they use cross-layer predictions to guide expert caching, reducing expert transfers by up to 17.0% and improving throughput by up to 5.7% without changing native routing.

Critical strengths
  1. A clear approach to studying routing states across layers. The paper provides a structured way to extract routing-relevant information and align it across layers. Importantly, the alignment is fitted separately from the transition model, making the shared-dynamics hypothesis more testable. This is a useful framework for studying the relationship between routers at different depths.

  2. Evaluation beyond representation similarity. The evaluation covers four MoE architectures and includes comparisons with alternative representations, routing fidelity tests, and causal interventions. The paper also measures actual expert transfers and end-to-end throughput. These experiments provide useful evidence that the aligned representations have practical value, even if the broader interpretation remains uncertain.

Critical weaknesses
  1. Insufficient evidence for genuinely shared routing dynamics. The paper shows that aligned routing states can be predicted using a shared linear model, but this does not necessarily establish that different layers follow the same underlying dynamics. In particular, the shared model is trained on transitions from all layers and evaluated on held-out tokens, rather than unseen layer transitions. Moreover, the reported 79–90% shared-dynamics ratio includes predictability already captured by the identity baseline. Therefore, the current results may overstate the degree of shared dynamics across layers.

  2. The caching improvements do not directly validate the proposed shared dynamics. The main method proposes a single shared transition model A, but the caching experiments use separately trained predictors T_d for different prediction horizons instead of repeatedly applying A. As a result, the performance gains demonstrate the usefulness of aligned routing representations for expert prediction, but do not establish that the proposed shared dynamics are responsible for these gains. This weakens the connection between the paper's central contribution and its practical application.

Questions for authors
  1. Can the authors provide stronger evidence that the learned transition is genuinely shared across layers? For example, could the shared transition be trained on a subset of layer transitions and evaluated on held-out layer pairs? Could the authors also compare against generic hidden-state representations using the same alignment and prediction procedure, and report improvements relative to the identity baseline? These tests would help distinguish shared routing dynamics from predictability introduced by cross-layer alignment.

  2. How much of the caching improvement depends on the proposed shared transition A? Could the authors compare long-range forecasts obtained by repeatedly applying A with the separately fitted horizon-specific predictors T_d, using the same cache policy and evaluation setup If the horizon-specific predictors are substantially better, how should the relationship between the shared-dynamics result and the caching application be interpreted?

3 comments
AuthorsOpenReview ↗

Response to Reviewer us9J (1/2)

We thank the reviewer for the review. Both questions address central aspects of the paper and led to additional experiments, summarized below; all of them will be included in the revision.

We first clarify the structure of the paper's contributions, as it is directly relevant to Q2. The central contribution is scientific: the router factorization has an exact orthogonal gauge symmetry, and once this symmetry is removed, routing dynamics are largely shared across depth. The cache study is a supporting application demonstrating that the canonical space is practically useful (Section 6). Q1 tests the central claim directly, and the new results strengthen it.

Q1. Evidence for shared dynamics across layers

Gain beyond identity. We agree that ηshare\eta_{\text{share}} includes predictability already captured by identity transport. We therefore report the identity-adjusted ratio

ηΔ=Rshared2−Ridentity2Rlayer2−Ridentity2,\eta_\Delta = \frac{R^2_{\text{shared}} - R^2_{\text{identity}}}{R^2_{\text{layer}} - R^2_{\text{identity}}},

which measures the fraction of the per-layer improvement over identity that a single shared transition recovers. Across 4 architectures and 3 seeds (12 configurations):

  • ηΔ=0.65\eta_\Delta = 0.65–0.730.73: one shared map recovers 65–73% of the per-layer gain beyond persistence, with 1515–39×39\times fewer parameters.
  • The shared transition outperforms a diagonal-transition baseline in all 12 configurations; all paired 95% confidence intervals are above zero.
  • Shared and per-layer models use the same validation grid and objective, so the ratio does not reflect a tuning asymmetry.

Held-out layer transitions. As suggested, we exclude an adjacent-layer transition from dynamics fitting and evaluate on it using separate articles. We further evaluate held-layer transfer: an unseen layer is calibrated on 32 separate articles, and all of its transitions are excluded from dynamics fitting. On Phi-tiny, router coordinates outperform pooled residual PCA by +4.58 pp top-kk recall for a held source layer and by +8.51 pp for a held target layer.

Router coordinates vs. generic hidden states. We applied the same alignment and prediction pipeline to pooled residual PCA and to layer-wise PCA + GPA. On held-out edges, router coordinates show the largest gains on Phi-tiny (+5.70 pp [4.47, 6.99] vs. pooled PCA; +3.03 pp [1.72, 4.40] vs. PCA+GPA) and on OLMoE (+0.70 pp [0.12, 1.36] vs. pooled PCA), and remain within approximately 3 pp on Granite and IBM Shared.

This is a strong baseline: residual PCA is considerably smoother across depth (next-state R2R^2 0.855 vs. 0.542 on Granite, Table 2). However, with respect to preserving expert selection, router coordinates are substantially better at matched rank and readout budget: top-kk recall 0.999 vs. 0.798 on Granite and 0.989 vs. 0.405 on Phi-tiny.

The effect is not produced by alignment. Three independent checks support this:

  • Random orthogonal gauges, PCA-basis alignment and CCA do not produce the effect, and token shuffling removes it (Table 6).
  • Replacing full GPA with a simpler, purely local gauge (pairwise Procrustes chained along adjacent layers) yields higher ηshare\eta_{\text{share}} on all four architectures (e.g., OLMoE 0.909 vs. 0.887; IBM Shared 0.837 vs. 0.792), with ηΔ=0.65\eta_\Delta = 0.65–0.740.74. The shared structure does not depend on the gauge-fixing procedure.
  • GPA is fitted without any transition model or dynamics loss and therefore cannot adapt the coordinates to AA.

Functional relevance beyond identity. When transported router states are substituted into the model, transport with AA in canonical coordinates degrades NLL far less than the same transport in raw coordinates, with shuffled gauges, or in a random subspace (Granite 9–13: Δ\DeltaNLL 0.0070 vs. 0.1120 / 0.1012 / 0.1495, Table 9). On OLMoE 8–9, AA outperforms identity transport with a document-level 95% confidence interval above zero, closing 26.5% [9.6, 50.9] of the gap between identity transport and the native model.

AuthorsOpenReview ↗

Response to Reviewer us9J (2/2)

Q2. Contribution of AA to the cache results

We compared four forecasters on four architectures at h=1,4,8h = 1, 4, 8: identity, recursive application of AA, direct horizon maps ThT_h, and a pairwise predictor with a separate map for each layer pair. At h=1h = 1, AA and T1T_1 coincide. Top-kk recall at h=8h = 8:

ArchitectureIdentityAA-rolloutThT_hPairwise
Granite0.6740.6480.6990.725
OLMoE0.5860.5370.6140.649
Phi-tiny0.4740.3650.5030.561
IBM Shared0.4970.4230.5270.567

The key observation is that ThT_h is itself a shared-across-depth model: one map per horizon, applied at every source layer. It outperforms identity at h=8h = 8 on all four architectures. The cache forecasts therefore rely on the same property as AA, a single operator shared across depth in the canonical space, applied directly at the horizon required by the cache.

The scientific claim concerns a single step: after gauge fixing, one operator accounts for most of the layer-to-layer change (Eq. 16), and at h=1h = 1 this operator is exactly the forecaster used. AA is fitted for that step, and its objective does not constrain accuracy after eight compositions. Composing a one-step model is known to be weaker than fitting each horizon directly. This is the standard trade-off between recursive and direct multi-step forecasting, and it applies to any one-step model. The direct maps ThT_h preserve the property that matters, a single operator shared across depth, while fitting the horizon required by the cache. The practical difference is small: in the matched physical cache below, recursive AA requires only 0.7–2.2% more expert loads than ThT_h.

Model size. ThT_h requires L−1L-1 maps; pairwise requires one map per layer pair, L(L−1)/2L(L-1)/2 in total (Granite: 23 vs. 276; IBM Shared: 39 vs. 780). With 88–20×20\times fewer maps, ThT_h remains within 2.6–5.8 pp of pairwise. This is consistent with the shared-dynamics view: the information required by each layer pair is largely common across depth.

Matched physical cache. All policies use the same cache engine, request traces, server, horizons, schedules, training data and ridge coefficient. Expert loads at cache capacity 0.0625:

ModelCanonical ThT_hPairwiseRecursive AA
Granite578,837568,306591,382
OLMoE182,726182,077184,094

In the physical cache, all three forecasters perform closely. ThT_h matches pairwise (loads within 2%, throughput within ±0.2\pm 0.2 tok/s across all 16 matched groups) with 88–20×20\times fewer maps, and recursive AA follows within approximately 2%. The smaller predictor requires less calibration data and serving memory and applies at any depth.

Practical effect. In the physical benchmark, the forecast produces cache hits in settings where LRU produces none. On Granite with 96 slots, the hit rate increases from 0% to 17.0%, loads decrease from 192.00 to 159.37 per token, and throughput increases by 5.7%, with native routing and model outputs unchanged.

Interpretation.

  1. Main contribution (scientific): after removing the router gauge, a single low-capacity one-step transition captures most of the predictable cross-layer change, well beyond identity. This holds for held-out transitions, under an alternative gauge-fixing procedure, and under causal substitution.
  2. Supporting application (systems): the same canonical space supports compact horizon forecasts, each a single map shared across depth, that reduce expert transfers without modifying native routing and match a substantially larger per-layer-pair predictor.

AA provides the scientific test: one operator for one step. ThT_h applies the same principle at the horizon required by the cache. The revision will make this distinction explicit in the abstract, methods and systems sections.

We thank the reviewer again and hope these results address the concerns raised.

Thanks to the authors for the response and for running the additional experiments. Looking at the new Q2 table, rolling out the shared transition A over multiple steps actually performs worse than the simple identity baseline on all four models. Together with the fact that caching still needs separate predictors T_d for each horizon rather than using A itself, I am still not convinced that the routing layers truly follow a single shared dynamical rule, or that this rule is what drives the system gains. The core claim of the paper still feels overstated relative to what the experiments show, so I will keep my original score.

Reviewer_XeVXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether the cross-layer predictability of MoE routing reflects reusable dynamics or merely a change of coordinates, and answers by constructing a router-specific quotient space.

Method. Because softmax and top-k are invariant to a common shift of all expert logits, the authors centre the router to W-tilde = (I - 11^T/E) W, so rank(W-tilde) <= E-1, and take a rank-r SVD W-tilde = B U^T. The columns of U span exactly the residual-stream directions that can change relative logits, giving control coordinates x = U^T h. The contribution is the observation that this factorisation carries an exact orthogonal gauge symmetry: for any R in O(r), (B R)(R^T x) = B x, so each layer's coordinates are defined only up to O(r), and cross-layer comparison should live on the quotient by that symmetry. The authors fix the gauge globally with generalized orthogonal Procrustes analysis (GPA) on training tokens, fitting the Q_l only, with no transition model or dynamics loss involved. The central hypothesis is then precise: if layer-specific transitions satisfy F_l = Q_{l+1} A Q_l^T for one low-capacity A, the apparently layer-specific dynamics are a gauge artefact. A ridge-fit pooled linear map z_{l+1} = A z_l + b tests this, against a separate per-layer transition as the upper-bound baseline, summarised by eta_share = R2_shared / R2_layer.

Theory. Proposition 1 is the invariance above. Proposition 2 bounds when a transported state preserves the native top-k set, via the decoded perturbation D_l e relative to the top-k margin gamma_k, with a state-space sufficient condition ||e|| < gamma_k / (2 ||D_l||_{2->inf}). Appendix B gives a finite-depth telescoping bound on the h-step operator error and a finite-horizon corollary, explicitly labelled an a posteriori certificate.

Validation. Four MoE architectures are probed on WikiText-2 with CNN/DailyMail replication: Granite (24 layers, 32 experts, top-8, rank 31), OLMoE-SFT (16/64/8/63), Phi-tiny (32/16/2/15), and an IBM shared-expert MoE (40/62/6/61). The evidence is: (1) after GPA, one shared transition reaches 79-90% of the R2 of per-layer transitions (Table 1), whereas raw, PCA-basis, random-orthogonal and CCA coordinates do not (Table 6: Granite 0.071 raw vs 0.544 Procrustes); (2) a matched-rank, equal-budget readout shows residual PCA is far more predictable yet recovers expert choices much worse (Table 2), separating temporal predictability from routing relevance; (3) causal substitution of transported router states favours canonical over raw coordinates (Table 9: dNLL 0.0070 vs 0.1120), with learned A beating identity at longer horizons on OLMoE and Phi (Tables 7/10: relative dNLL reductions of 15.7%, 5.6% and 6.2%); and (4) forecast-guided expert residency reduces physical host-to-GPU loads by up to 17.0% and raises throughput by up to 5.7%, with the long-horizon schedules improving throughput in all six tested configurations (Tables 3/11).

The paper is careful about what it does not show: router weights are not interchangeable across layers, router skipping accumulates error, the IBM shared-expert model is less universal, and the cache benefit is attributed to forecast lead time rather than forecast accuracy.

Critical strengths

Strength 1 - The nuisance symmetry is derived from the router parameterisation rather than postulated, and it yields a falsifiable prediction that holds out of sample.

This is the paper's real contribution and the main reason I lean toward acceptance. Cross-layer similarity results are normally open to the objection that any alignment method can inflate similarity while explaining nothing about behaviour. Here the symmetry is forced by the factorisation: centring removes the common-logit direction, and the residual basis ambiguity is exactly O(r), so the quotient space is identified rather than chosen. That converts a loose observation (routing states rotate with depth) into a specific claim with a test: a single low-capacity transition should recover most of the predictive power of per-layer transitions once the gauge is fixed. The jump over the raw basis is large (Granite 0.071 -> 0.544) and survives PCA-basis, random-Q and CCA controls, with token shuffling removing the effect almost entirely. Because the gauges are fitted without any dynamics objective, the transition result is not produced by construction.

Strength 2 - The controls are well chosen, the negative results are reported rather than buried, and the systems claims survive a check for selective reporting.

Table 2 is the control that matters, and it is the right one: it blocks the explanation that the canonical states are merely smooth residual features. Residual PCA is indeed far more predictable (next-state R2 0.855 vs 0.542 on Granite) yet recovers top-k choices far worse (0.798 vs 0.999), cleanly separating two properties that are easy to conflate. The cache study is unusually well controlled for a representation paper: PAIRWISE is an explicitly labelled native-logit forecasting control, given a substantially larger parameterisation, whose stated purpose is to test whether the gains require the canonical representation - and it loses to the canonical 1x forecast on loads at every capacity on both models. The authors also state that the gain comes from forecast lead time rather than accuracy, and their numbers support that: both long-horizon schedules are faster than LRU in all six configurations, while local-2 and local-4 are slower in all six and 4x in four of six, exactly as their overhead argument predicts. I checked every derived number the paper reports, including the abstract's "up to" maxima, and all reproduce.

Critical weaknesses

Weakness 1 - The interpretation of eta_share is not yet identified, because the layer-specific baseline's regularisation is unspecified.

Table 1 sets one pooled r x r transition against L-1 separately fitted r x r transitions, evaluated on held-out sequences. The text states that the shared model is fitted by ridge regression with the regularisation strength selected on validation data, and that the layer-specific models "provide an upper bound". It does not say whether those per-layer baselines were regularised, nor whether their strength was tuned the same way. That matters for the headline claim, because eta_share is a ratio of held-out R2 values: if the shared model is tuned and the per-layer reference is not, the ratio mixes a modelling statement with a tuning asymmetry, and the reader cannot tell how much of the 79-90% is shared structure versus the shared model simply being better regularised. I want to be explicit that I do not think the baseline is underdetermined: each individual per-layer transition has only about 4.0k parameters for OLMoE and 1.0k for Granite, which is small relative to any plausible training set, so the per-layer fits are not obviously variance-dominated. My concern is the asymmetry in tuning, not the size of the reference. This is a reporting gap that a short additional experiment would close, not a defect in the construction.

Weakness 2 - The causal evidence is directionally consistent but small in absolute terms, and the language is slightly stronger than the magnitudes.

The fixed-block replications in Tables 7 and 10 are the load-bearing causal results. In relative terms they are respectable - 15.7% for OLMoE 8-9, and 5.6% and 6.2% for the two Phi-tiny blocks - and all three favour learned dynamics. In absolute terms, however, the differences are 0.0059, 0.0150 and 0.0200 in dNLL, and the two architectures differ sharply in what identity transport costs to begin with: on OLMoE it is nearly lossless (0.0375), whereas on Phi-tiny both transport modes already pay a substantial price (0.2670 and 0.3217). The paper's phrasing - "reproducible local transport", "learned dynamics beats persistence" - reads as a stronger claim than this pattern supports. It is a real result that learned A wins consistently on Phi, but it wins by roughly 5-6% relative on a baseline that is itself far from the native model, and the limitations section already concedes that this degradation rules out practical router skipping. I would like the relative change and an uncertainty estimate for the Shared-minus-Identity difference reported directly, so a reader can judge whether the advantage is distinguishable from noise at the block level rather than only in aggregate. This bounds how much weight the causal section can carry, but it does not contradict the qualitative conclusion.

Weakness 3 (secondary) - Gauge cost and scope.

GPA fits L-1 free orthogonal matrices, i.e. O(L r^2) parameters, on the same training tokens used to report the shared-transition R2, and r is large for the most prominent model (63 for OLMoE). The paper's ablations already rule out the trivial explanations - random gauges, PCA bases and CCA all fail to reproduce the gain - so this is not a soundness worry. It is a question of attribution: what does eta_share become under a cheaper gauge, for example a single global orthogonal map or pairwise Procrustes chained along depth? A positive result there would be the stronger and more surprising claim. On scope, the systems experiment is eviction-only (the method never proactively prefetches, despite the framing around expert prefetching in the introduction), it covers two 1B-class models on a single V100, and the IBM shared-expert model - the one architecture with explicit shared experts, and therefore the most interesting test case - shows the weakest universality. The limitations section scopes these claims correctly; the breadth implied by "across four MoE architectures" in the abstract is doing slightly more work than four models on one corpus can support.

Questions for authors

Q1 (would affect how much weight I place on the headline claim). Was the layer-specific baseline in Table 1 fitted with ridge regularisation, and was its strength selected on validation data in the same way as the shared model's? The text specifies this for the shared model only. Since eta_share is a ratio of held-out R2 values, a tuning asymmetry between the numerator and the denominator would change how the 79-90% should be read. A tuned per-layer reference at matched effective capacity would settle it. I note that I do not believe the per-layer models are underdetermined - each has only about 1-4k parameters - so my interest is in the symmetry of tuning, not in sample size.

Q2 (clarification on the cache control). PAIRWISE is described as the native-logit forecasting control given a larger parameterisation, and it loses to the 1x canonical forecast on loads at every capacity, which I read as evidence in your favour. To make that comparison exact, could you state whether PAIRWISE uses the same look-ahead schedule and horizons as 1x, and whether its ridge setting was selected the same way? If its schedule differs, a budget- and look-ahead-matched variant would make the attribution airtight. I ask because your own Section 5.6 analysis implies lead time is the dominant factor, so the schedule is the variable that matters.

Q3 (clarification). What does eta_share become under a cheaper gauge - a single global orthogonal map, or pairwise Procrustes chained along depth - rather than full GPA? For rank 63 (OLMoE) the gauge family is large, and a positive result under a cheaper gauge would considerably strengthen the claim that the shared structure is intrinsic rather than a consequence of the alignment having enough freedom.

Q4 (presentation). In the offloading experiment, LRU records no cache hits at five of the six configurations - 128.00 loads/token at both the 32- and 64-slot OLMoE caches and 192.00 at all three Granite capacities, i.e. exactly the per-token access counts - with the 128-slot OLMoE cache the only one that achieves any reuse (78.55). Could you report absolute hit rates alongside loads/token so a reader can judge the practical significance of the reduction, and state in the abstract that the headline percentages correspond to the large-cache, long-horizon configuration?

Q5 (reporting). For Tables 7 and 10, could you report the relative change and an uncertainty estimate for the Shared-minus-Identity difference per block? The absolute dNLL differences are small (0.0059-0.0200), and I would like to know whether the per-block advantage is distinguishable from noise rather than only consistent in sign across blocks.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XeVX (1/2)

We thank the reviewer for the detailed review and the precise questions. Each question led to an additional experiment or measurement, summarized below; all of them will be included in the revision.

We first clarify the structure of the paper's contributions. The central contribution is scientific: the router factorization has an exact orthogonal gauge symmetry, and once this symmetry is removed, routing dynamics are largely shared across depth. The cache study is a supporting application demonstrating that the canonical space is practically useful (Section 6). The new results primarily strengthen the scientific core.

Q1. Regularization of the layer-specific baseline

Every layer-specific transition is ridge-regularized, with λ\lambda selected on the same validation grid and with the same objective as the shared model. The only difference favors the baseline: the shared model uses a single λ\lambda for all transitions, whereas each layer-specific model selects its own optimal λ\lambda. The reference is therefore tuned at least as flexibly as the shared model, and ηshare\eta_{\text{share}} is a conservative estimate.

We additionally report the identity-adjusted ratio

ηΔ=Rshared2−Rid2Rlayer2−Rid2,\eta_\Delta = \frac{R^2_{\text{shared}} - R^2_{\text{id}}}{R^2_{\text{layer}} - R^2_{\text{id}}},

which measures the fraction of the per-layer gain beyond persistence that a single shared map recovers. Across 4 architectures and 3 seeds, ηΔ=0.65\eta_\Delta = 0.65–0.730.73. The shared transition outperforms a diagonal-transition baseline in all 12 configurations, with all paired 95% confidence intervals above zero.

Q2. Schedule and regularization of PAIRWISE

The original PAIRWISE control was not fully matched: it used an earlier stride-2 schedule and a fixed λ=10−3\lambda = 10^{-3}. We therefore repeated the physical benchmark with all forecasting policies matched in schedule, horizons, training data, request traces, server, cache engine and ridge setting. Expert loads at cache capacity 0.0625:

ModelCanonical ThT_hPairwiseRecursive AA
Granite578,837568,306591,382
OLMoE182,726182,077184,094

Under matched conditions, the two predictors achieve practically the same cache performance: loads within 2% and throughput within ±0.2\pm 0.2 tok/s across all 16 groups. They differ substantially in model size. The canonical forecast uses one map per horizon, shared across all source layers (L−1L-1 maps). Pairwise requires a separate map for each source–target layer pair (L(L−1)/2L(L-1)/2 maps): 23 vs. 276 for Granite and 39 vs. 780 for IBM Shared. The canonical forecast thus reaches the same performance with 88–20×20\times fewer maps, which reduces calibration data and serving memory and yields a single model applicable at any depth.

This parity is also informative scientifically. A single map per horizon, shared across all source layers, matches a predictor with a dedicated map for every layer pair. This is the behavior predicted by the shared-dynamics hypothesis: in canonical coordinates, the information required by different layer pairs is largely common across depth. Section 5.6 will be revised to report this matched comparison.

Q3. A cheaper gauge

We replaced full GPA with pairwise Procrustes chained along depth:

Architectureηshare\eta_{\text{share}}, GPAηshare\eta_{\text{share}}, chainTime, GPA →\to chain
Granite0.8750.8982.64 s →\to 0.017 s
OLMoE0.8870.9099.81 s →\to 0.032 s
Phi-tiny0.8080.8380.51 s →\to 0.009 s
IBM Shared0.7920.83790.4 s →\to 0.069 s

The chained gauge uses only adjacent-layer information and is 5050–1300×1300\times faster, yet yields higher ηshare\eta_{\text{share}} on all four architectures (ηΔ=0.65\eta_\Delta = 0.65–0.740.74). The shared structure therefore does not depend on a specific global alignment procedure, and calibration cost becomes negligible even for the 40-layer IBM model.

Regarding gauge freedom: the chained gauge uses the same orthogonal family as GPA. The relevant control is Table 6: random orthogonal gauges have exactly the same freedom and do not produce the effect, and token shuffling removes it. The gain is determined by the correct token correspondence, not by the size of the gauge family.

AuthorsOpenReview ↗

Response to Reviewer XeVX (2/2)

Q4. Hit rates and headline configuration

Hit rate =1−loads per token/accesses per token= 1 - \text{loads per token} / \text{accesses per token}:

CacheLRUProcrustes 1×1\times
OLMoE 32 / 64 / 1280.0 / 0.0 / 38.6%4.4 / 8.5 / 38.7%
Granite 24 / 48 / 960.0 / 0.0 / 0.0%3.8 / 7.7 / 17.0%

In the small-cache regime, LRU achieves no reuse, and all hits are produced by the forecast. The headline numbers in the abstract correspond to Granite with 96 slots and the 1×1\times schedule: loads decrease from 192.00 to 159.37 per token (−17.0%-17.0\%), and throughput increases from 4.369 to 4.619 tok/s (+5.7%+5.7\%). The abstract will state this configuration explicitly.

Q5. Uncertainty for the causal blocks

Paired NLL improvement of AA over identity, with document-level 95% bootstrap confidence intervals over 32 articles (new evaluation protocol; absolute values therefore differ slightly from Table 10):

BlockIdentity −- AAShare of identity-to-native gap closed
OLMoE 8–90.0123 [0.0044, 0.0201]26.5% [9.6, 50.9]
Phi 24–310.0122 [$-$0.0085, 0.0357]4.4% [-\3.4, 11.6]
Phi 22–310.0040 [$-$0.0192, 0.0311]1.3% [-\9.1, 6.4]

On OLMoE, the effect is clearly distinguishable from noise: AA closes approximately one quarter of the gap between identity transport and the native model. On Phi-tiny, both point estimates are positive, while the block-level intervals at 32 articles include zero. In line with the reviewer's suggestion, the causal claim will be stated with respect to OLMoE, and Phi-tiny will be described as a consistent positive trend.

The dominant causal factor is the canonical coordinate system itself. With the same shared AA, transport in canonical coordinates degrades NLL far less than transport in raw coordinates, with shuffled gauges, or in a random subspace (Granite 9–13: 0.0070 vs. 0.1120 / 0.1012 / 0.1495, Table 9), a difference of more than an order of magnitude.

Summary

The scientific core is more strongly supported than in the submission: the baseline is tuned symmetrically, a single shared transition recovers 65–73% of the per-layer gain beyond identity, and a cheaper, purely local gauge yields even higher ηshare\eta_{\text{share}}. As a supporting application, the cache study shows that this structure is practically useful: it matches a per-layer-pair predictor with 88–20×20\times fewer maps and produces up to 17% cache hits in settings where LRU produces none.

We thank the reviewer again and are happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.