- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether MoE routing shares a common structure across layers once layer-specific coordinates are removed. It projects hidden states onto each router's control subspace, aligns all layers into one coordinate system with generalized Procrustes analysis, and fits a single shared linear transition. Across four MoE models, the shared transition recovers 79–90% of the R2 of layer-specific transitions, and using its forecasts for expert-cache eviction cuts expert transfers by up to 17% and raises throughput by up to 5.7%.
Critical strengths
- Interesting question with a clean formulation.
Critical weaknesses
-
The evidence for shared cross-layer routing dynamics is not sufficient. I reproduced the Table 1 analysis on Granite and implemented the singular-spectrum check that Section 3.4 promises but never reports. Random per-layer subspaces pass this check just as well as the router subspaces. When all layers use one fixed residual subspace of the same rank, η_share is about 0.97, higher than the 0.88 obtained with router coordinates. This suggests that much of the cross-layer sharing may come from the smoothness of the residual stream rather than from router geometry. Section 5.2 does not settle this, since router-control coordinates reconstruct router logits by construction, so their higher top-k recall over residual PCA is expected. The baselines that could test this are missing: applying the future layer's router directly to the current hidden state, and a shared transition in a residual subspace common to all layers.
-
The caching baseline is an LRU that thrashes: in five of the six cache settings its hit rate is 0%. There is no comparison with LFU, a Belady oracle, or existing prefetching methods such as Fate or Pre-gated MoE.
Questions for authors
- The causal blocks are chosen on an exploratory split. By what criterion, and across all candidate blocks, how often does Shared beat Identity?
- Can you add results for the missing baselines?
3 comments
Response to Reviewer D4fw (2/3)
W1 (continued). Requested baselines and the spectral check
Future router applied to the current hidden state. We compare the direct shared horizon map with the next layers' routers applied to the current hidden state ("stale router"). Granite, 128 test documents, recall difference with paired 95% confidence intervals (Appendix Table 6):
| Horizon | 1 | 2 | 4 | 8 | 16 | 23 |
|---|---|---|---|---|---|---|
| Shared map stale router | +0.000 | +0.075 [0.070, 0.080] | +0.273 [0.266, 0.279] | |||
| Source layers | 23 | 22 | 20 | 16 | 8 | 1 |
The two predictors are complementary. The stale router uses the full hidden state and a router evaluation at every layer, and is stronger 1–4 layers ahead. The shared map uses a single vector of router logits and is stronger from 16 layers ahead. This long range is exactly where the shared map is most useful for caching (Section 5.5).
Shared transition in a residual subspace common to all layers. Reported above and in Table 2 for three architectures, together with per-layer PCA and both random controls.
Singular-spectrum check. We agree that it does not separate representations. Appendix B now reports it for all representations (median normalized defect over 23 transitions, range over the ridge grid):
| Representation | Native tokens | Token-shuffled null |
|---|---|---|
| Router + GPA (ours) | 0.36–0.54 | 0.98–0.99 |
| Pooled residual PCA | 0.19–0.45 | 0.98–0.99 |
| Per-layer PCA + GPA | 0.25–0.33 | 0.98–0.99 |
| Fixed random | 0.13–0.29 | 0.98 |
Router coordinates satisfy the necessary condition with a wide margin over the null. Fixed subspaces also show low defects, for the same reason that their is high. The check is therefore reported as a necessary condition, and specificity is established by the routing gain.
W2 and Q2. Caching against LFU, Belady/MIN, Fate and Pre-gated MoE
Eviction on all four architectures. We replay real router traces with the same demand-load semantics as the hardware cache; the simulator matches hardware access counts exactly. Rules and settings are selected on validation documents and frozen before testing. The grid has 4 models 3 seeds 4 disjoint workloads 3 capacities = 144 cells. Loads relative to LFU at capacity (Appendix Table 14):
| Model | LRU | MIN (oracle) | Ours | Ours without forecast |
|---|---|---|---|---|
| Granite | 1.543 | 0.680 | 0.967 | 0.972 |
| OLMoE | 0.957 | 0.533 | 0.840 | 0.844 |
| Phi-tiny | 1.008 | 0.564 | 0.886 | 0.886 |
| IBM Shared | 0.867 | 0.443 | 0.772 | 0.777 |
Over all 144 cells, our policy uses 6.0% fewer loads than the better of LFU and LRU (geometric mean), 13.6% fewer than LFU and 12.5% fewer than LRU. It is best in 120 cells and closes 27% of the LFU-to-MIN gap. The forecast lowers loads further in 104 of 144 cells. The remaining gap to MIN is reuse across tokens, which no within-token predictor, including Fate's, can capture.
Response to Reviewer D4fw (3/3)
W2 and Q2 (continued). Prefetching on hardware
Setup. Granite-3.0-1B-A400M, FP32, RTX 5090 (PCIe 5), fixed capacity of 147 layer-experts. Each session decodes 32 held-out documents 128 tokens, over 2 workloads in 2 policy orders. Every method, including Fate's prediction core (next-layer gate applied to the current hidden state), is tuned on validation prompts only. Generated tokens and native routes are bitwise identical across all policies (Table 3).
| Policy | Tok/s vs. LFU [min, max] | vs. Fate | Predictor ms/token |
|---|---|---|---|
| LRU | – | ||
| EMA eviction, no forecast | – | ||
| Ours, shared map + prefetch | [7.8, 9.5] | 0.21 | |
| Ours, shared map + low-rank correction | [10.7, 11.4] | 0.0% [, 1.0] | 0.21 |
| Per-pair maps, same prefetch | [10.3, 11.1] | 0.21 | |
| Fate prediction core (GPU, tuned) | [9.6, 12.3] | – | 3.54 |
Matching Fate at a fraction of the cost. Our forecaster matches Fate's throughput and leads in 2 of 4 sessions. On a second machine with the same frozen settings, the ratio is 1.000 [0.995, 1.006], with both 7.5% faster than LFU. Most of the gain comes from the forecast: eviction without a forecast gives 1.8 of the 11.0 points.
Model size. The forecast runs once per token, from router logits only, without the hidden state or a router evaluation at every layer. It costs 0.21 ms per token against 3.54 for Fate ( less). Each horizon uses one closed-form map, with no gradient training and no change to the model. One map per horizon replaces one per layer pair, which gives – fewer coefficients on Granite.
Scope. On OLMoE, with 64 experts per layer, a single forecast from layer 0 yields too few useful prefetches (32–36%) to improve on forecast-free eviction, and Fate's per-layer prediction ( over LFU) justifies its higher cost (Appendix Table 15). The once-per-token forecast is the low-overhead option for moderate expert counts, and the revision states this explicitly.
Pre-gated MoE requires a pre-gating function trained into the model and does not apply to unmodified checkpoints, which is our setting. We therefore compare with Fate's prediction core, the closest training-free method.
Q1. Selection of causal blocks and frequency of Shared beating Identity
Selection criterion. Candidate windows were ranked on an exploratory split by , and the top blocks were fixed before replication (Section 5.4, Appendix D).
All windows of the full grids (bootstrap with Holm correction, Appendix Table 8):
| Model | Windows | Shared better (point estimate) | Shared significantly better |
|---|---|---|---|
| Granite, seeds 101/202/303 | 115 each | 28 / 25 / 27 | 1 / 2 / 1 |
| IBM Shared | 211 | 53 | 1 |
| OLMoE | 67 | 10 | 0 |
Replication on fresh documents. OLMoE 8–9 outperforms identity transport on all three original splits and on 32 fresh documents with verified no-op and reconstruction controls: NLL( Identity) = [, ] with a frozen source state and [, ] with the original hook (Appendix Table 10). The causal claim is stated for this pre-registered block, consistent with the predictive–causal gap stated in the abstract.
Summary
Both weaknesses are addressed with the controls the reviewer proposed. Shared dynamics are specific to router coordinates: a single shared map improves future expert selection only after the router gauge is removed, while in the reviewer's residual subspace the dynamics remain layer-specific. As a supporting application, the cache study now uses LFU, Belady/MIN, per-pair maps and Fate's core as baselines, and matches the strongest training-free predictor with a cheaper, closed-form forecast.
Changes in the revision. All results above are in the revised PDF: Sections 3.4, 5.2, 5.4 and 5.5, Tables 1–3, and Appendices B, D, E and F, with the original V100 study kept in Appendix Table 16 relative to LRU.
We thank the reviewer again and are happy to provide further clarification.
Response to Reviewer D4fw (1/3)
We thank the reviewer for reproducing Table 1 and for proposing two concrete controls. We ran both controls exactly as described and added all requested baselines. The results are summarized below and are included in the revised PDF; section and table numbers refer to the revision.
We first clarify the structure of the paper's contributions. The central contribution is scientific: the router factorization has an exact orthogonal gauge symmetry, and once this symmetry is removed, routing dynamics are largely shared across depth. The cache study is a supporting application demonstrating that the canonical space is practically useful (Section 6). The new results strengthen the scientific core directly and place the cache study against strong baselines.
W1. Router specificity versus residual smoothness
The reviewer's numbers. We reproduce them. In one fixed residual-PCA subspace of the same rank, ηshare=0.957 on Granite, and a fixed random subspace gives 0.966. In any fixed subspace, adjacent layers are already nearly aligned, so ηshare approaches 1 for every representation. We therefore add two measures that subtract persistence.
Gain beyond identity. The identity-adjusted ratio
ηΔ=Rlayer2−Rid2Rshared2−Rid2
is the fraction of the per-layer gain beyond persistence that one shared map recovers. Fig. 1(b) already shows this quantity. Across 4 architectures and 3 seeds, ηΔ=0.65–0.73.
Routing gain under identical readouts. This is the main specificity test. For each representation we measure how much the shared transition improves future top-k recall over persistence, using the same document splits, ridge grid and one learned readout per target layer. No representation reconstructs router logits by construction. Results for h=1 on 128 test documents (Table 2):
All random controls on Phi-tiny and OLMoE are also negative (full table with ηshare and ηΔ in Table 2).
The key observation is the contrast between the two columns. In residual PCA, per-layer maps still improve routing (up to +0.008), but the shared map recovers none of this gain: the dynamics there are layer-specific. In router coordinates, the shared map recovers 23–37% of the per-layer gain with 15–39× fewer parameters. Sharing appears only after the router gauge is removed, which is the paper's central claim, now tested against the reviewer's own alternative. The paired difference, router minus fixed residual PCA, is +0.0159 [0.0154, 0.0163] on Granite, +0.022 [0.020, 0.024] on Phi-tiny and +0.0120 [0.0115, 0.0125] on OLMoE.
Random subspaces. In per-layer random subspaces, persistence has negative R2, so any fitted map closes most of the state-space gap and ηΔ is high. Yet both maps reduce routing recall. State-space ratios alone therefore cannot establish specificity; the routing gain can.
Information versus shared dynamics. A residual PCA of the same rank can retain slightly more same-token routing information (persistence recall 0.797 vs. 0.758 on Granite), because it also contains directions outside the router's control space. Our claim concerns shared dynamics, and only router coordinates exhibit them.