acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Previous Layer: Incremental Predictive Value of Routing History in Frozen Sparse Mixture-of-Experts Models

12 views

Abstract

Routing in a sparse mixture-of-experts (MoE) transformer is usually examined one layer at a time, yet a single token passes through many routers in sequence. We ask how much older expert selections still predict once the full score vector of the immediately previous router is known. Answering that takes a conditional measurement: we hold that score vector fixed, add older selections, and compare against a distribution-matched control in which history rows are permuted across tokens, which separates alignment from added width. On a frozen grid of four pretrained MoE language models at four depths each, older aligned selections raise held-out for the next router's scores by on average (95% CI , positive in all 16 cells), while the misaligned control recovers none of this gain. The same history raises next-layer expert-set recall from to (), and both effects reproduce on an unseen corpus at four token positions (). The gains vary by nearly an order of magnitude across architectures, and a headroom-normalized diagnostic computed on separate text ranks the cells by realized gain (Spearman ); supplying history to the top half of the cells under that ranking retains about 71% of the full average recall gain. Two conditions bound the scope: widening the conditioning state to three recent score vectors leaves only a small residual, carried by the narrowest router in the grid, and against a strong gate-reading reranker the history-specific complementarity is small and the runtime selector we tested does not convert it. Cross-layer routing history is a real conditional signal whose magnitude is an architectural property rather than a uniform feature of sparse MoE routing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Mixture-of-Experts, MoE Routing, Routing History, Cross-Layer Routing Dynamics, Expert Prediction, Expert Selection, Conditional Complementarity

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Ei9DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether expert selections in previous layers provide incremental predictive value for a token's routing in the current layer. The authors train ridge-regression probes to predict the current layer's router scores from the previous layer's full score vector, then test whether adding earlier layers' expert selections improves prediction. They repeat this comparison with four frozen pretrained MoE models, using shuffled histories as a control. The main experiments report an average R² increase of 0.0455 and a 3.31 percentage-point improvement in expert-set recall.

Critical strengths
  1. Controlled tests of whether earlier routing choices help prediction. The paper compares predictions with and without earlier routing history, and uses shuffled history as a control. This helps distinguish useful historical information from simple cross-layer correlation.
Critical weaknesses
  1. The study is not sufficiently motivated. Residual connections make cross-layer routing dependence plausible, although they do not imply the specific conditional gain measured here. The paper needs to explain what important new understanding or MoE design decision follows from measuring this gain.

  2. The demonstrated practical value is limited.

  3. The predictive gains are difficult to interpret. R² and HNV measure improvement under a particular probe and feature choice; they do not directly establish the amount of information in routing history. Although the paper includes recall and token-level analyses, it should connect these more clearly to which expert choices improve and why the aggregate gain matters.

  4. Layerwise Recurrent Router already studies cross-layer routing dependence using mutual information. Conditional probe prediction asks a different question, but the paper should explain what additional insight it provides. Forward predictability should also be distinguished from training benefits that recurrent routers may obtain through additional gradient paths.

Questions for authors
  1. Given that extra history adds little value with stronger predictors or richer recent information, and that the runtime selector provides only limited gains, in which settings do the authors expect older routing history to be meaningfully useful? What important understanding or MoE design decision follows from these results?
  2. Please specify the complete inputs, history windows, candidate construction and scoring procedures of the FATE-style, HCR-B and history predictors. Can matched recent-only/shuffled branches be evaluated in the runtime selector, and can the same strong prediction mechanism be compared with and without older selections?
  3. How do the findings relate to the existing mutual-information routing analysis like in Layerwise Recurrent Router paper? Which conclusions remain stable across reasonable predictor and preprocessing choices beyond the single nonlinear JetMoE check?
1 comment
AuthorsOpenReview ↗

Research Motivation, Experimental Comparisons, and Limitations

Thank you for reading our paper and raising these questions. We would like to explain what the experiments show and where the current results are still limited.

Q1. When is older routing history useful, and what do we learn from it?

We agree that the practical benefit is limited, especially when stronger predictors or more recent scores are available. We reported the small and negative results in the paper as well.

The main finding is that the value of history depends on the model and on how much recent information is already known. JetMoE has a much larger history gain than the other three models. When three recent score vectors are used, JetMoE still has a positive mean direct gain, while the other three models do not. This suggests that history may be worth checking when the previous router scores alone are not very predictive.

We also used HNV to rank the 16 model-depth settings. Using history for the top eight settings retains about 71% of the average recall gain of using it everywhere. However, simpler ranking rules work almost as well. Also, using history in half of the settings does not mean a 50% reduction in computation.

We do not claim that this work speeds up inference or produces a better routing algorithm. We think it helps identify where older history provides extra predictive value and where the recent state is already enough.

Q2. How are the predictors defined, and are the runtime comparisons matched?

The recent-only probe uses the full score vector from the previous router. The history probe adds top-k selections from three older layers. Both use multi-output ridge regression, with PCA applied to the history features. The FATE-style baseline and HCR-B can also use the target-layer gate parameters, while our history probes cannot. We agree that the complete inputs, history windows, candidate construction, and scoring rules need a clearer table. We will verify the implementation details from the released code.

We did run matched comparisons in the oracle analysis. Against a recent-only third predictor of the same family, the additional history-specific oracle gain was about 0.56 percentage points. Against shuffled history, the alignment-specific oracle gain was about 0.63 percentage points. These are hindsight upper-bound comparisons, not runtime gains.

The actual runtime selector improved on HCR-B by only 0.1831 percentage points. We tested the two-strong-predictor selector and the selector with an added history arm. We did not run corresponding selectors with an extra recent-only or shuffled-history arm. Because these controls are missing, we cannot attribute the runtime improvement specifically to correctly aligned older history. We agree that the suggested matched runtime experiments are needed for that claim.

Q3. How does this differ from Layerwise Recurrent Router, and are the results stable?

Layerwise Recurrent Router has already studied cross-layer routing relationships. Our question is more specific. We ask whether older expert selections still help prediction after the complete score vector from the immediately previous layer is already given.

Two layers can be related without the older one providing extra predictive value after we know the recent state. This is why we compare recent scores alone, recent scores with aligned history, and recent scores with shuffled history of the same feature capacity. Our results measure predictive gain under these probes. They do not prove conditional mutual information or causality. They also do not measure the training benefit of recurrent routing connections.

The main R² gain was positive in all 16 model-depth settings. On C4, we also observed positive gains across four models and four token positions. A capacity-matched nonlinear check at one JetMoE layer found a history gain of +0.1644 R² over a nonlinear recent-only predictor.

There are important limits. The nonlinear check covers only one model and one layer. The C4 experiment changes the corpus and token positions but uses the same four architectures. We also found that history gains become much smaller with richer recent scores. We therefore do not claim that the results will hold with every predictor or preprocessing choice. Our conclusions are limited to the settings we tested.

Reviewer_Fct9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies routing in sparse mixture-of-experts (MoE) transformer models, focusing on how much previous routing scores and decisions can explain the future router scores and decision. Here score refers to the raw logits of the router before softmax, and decision refers to the binary decision indicating whether each expert is select to the top-k or not. The paper compares 3 predictors: (i) using only previous layer’s router scores, called recent-only, (ii) a version that has access to the following 3 layers’ decisions after top k in addition to the previous layer’s router scores, called aligned design (iii) a control for (ii) that shuffles the order of these additional 3 decisions, called misaligned design. The authors use linear probes and measure the predictive power of these designs for 4 large language models at four depths. The results show that aligned design raises the held-out R2R^2 for the next layer’s scores by 0.0455 on average across 4 models and 4 depths compared to that of the recent-only design while misaligned design is worse than recent-only. On the decisions side, aligned design increases the expert-set recall from 0.635 to 0.668 whereas the misaligned version degrades it to 0.627, and both effects seems to be reproducible on an unseen corpus at for token positions. Increasing router score access from one previous layer to three layers absorbs the older-history signal and older-history hurts in 11 of 16 cells despite being better than misaligned.

Critical strengths

The authors show that the routing scores of the previous layer has strong predictive power for the next layer's scores at 4 different depths with 4 different models across corpora. Giving access to Older-history leads to small gains of predictive power on 3 of 4 architectures, showing recent-only design captures most of the power.

Critical weaknesses

The paper is very hard to follow because many of the sentences are too complex and over compressed by LLMs. I had to parse and breakdown many of the sentences and read several times. The results needs to be presented in a way that is easier to follow and digest. Trying to follow the small changes in numbers with a barrage of LLM written unnecessarily complex sentences is quite frustrating. I think having more tables help a lot for this.

The main finding is previous layer’s router scores explain the next scores most, while including the scores of the earlier 2 layers or the decision of the earlier 3 layers increases the R2R^2 slightly. These findings are not surprising, and paper does not present new insights that can use be useful for designing better MoEs. I find the paper’s contributions limited, and considering the current writing style, I don’t think it is good fit to this venue.

Questions for authors

What is the rationale behind conditioning on previous router scores while older-history uses top-k decisions? Then paper increases score history from 1to 3? Why 3? Could the authors clarify the impact of conditioning on the router scores versus the top-k decision? The design choices are not clear to me. Could the authors report R2R^2 and expert-recall for conditioning on the single jth previous state or the top-k decision for jth previous layer for j=1,2,3?

Does the authors have any explanation for the differences between JetMoE and the rest of the models?

Could the authors clearly define what gate-reading predictor is and has access to?

1 comment
AuthorsOpenReview ↗

Clarification of Experimental Design and Predictor Settings

Thank you for reading our paper and for these detailed questions. We realize that some experimental settings were not explained clearly enough.

Q1. Why use previous-layer scores and older top-k selections? Why three layers?

We use the complete scores from the previous layer because they contain more information than the selected experts alone. Appendix C.5 supports this choice. Replacing recent continuous scores with binary selections reduced expert-set recall by about 2.99 percentage points. For older layers, we use top-k selections because we want to study the path of experts that the token actually followed.

We used three older layers as a fixed setting, not because we proved that three layers are optimal. We also tested history lengths of one, two, and three layers. Their recall values were 0.65276, 0.66424, and 0.66822. The improvement became smaller as more layers were added.

Section 5.4 and Appendix G report a different experiment. There, we kept the older selection input fixed at one layer and increased the number of recent continuous score vectors from one to three. The R² gains from adding the older selection were +0.0195, +0.0076, and +0.0031. One experiment changes the length of the selection history. The other changes how much recent score information is already available. We should have explained this difference more clearly.

We agree that the submitted paper does not contain the complete single-lag comparison requested for j = 1, 2, 3, using both R² and expert recall for scores and top-k decisions. This would be a useful additional experiment.

Q2. Why does JetMoE behave differently?

One possible reason is router size. JetMoE has eight experts and selects two, while the other models have 60–64 experts. The previous-layer scores may therefore be less predictive in JetMoE, leaving more room for older history to help.

Our results fit this explanation. JetMoE has a recent-only R² of 0.264, compared with about 0.70–0.77 for the other models. Its average gain from history is +0.108, compared with +0.018 to +0.033 elsewhere. When three recent score vectors are available, JetMoE still has a mean direct increment of +0.0164, while the other three models have slightly negative mean increments. We also observe a larger JetMoE gain on C4.

However, we cannot confirm that router size is the cause. These models differ in other ways too. We present this as a possible explanation, not a proven mechanism.

Q3. What is the gate-reading predictor, and what can it access?

The gate-reading predictor refers to our fixed-budget FATE-style baseline. It can use the target-layer gate parameters when predicting which experts will be selected.

This is different from our routing-history probes. The recent-only probe uses the complete previous-layer score vector G. The history probe adds older top-k selections H. Neither probe reads the target-layer gate parameters. HCR-B is a stronger candidate reranker that also uses target-layer gate information. The actual target expert set is used only for evaluation, not as an input to these predictors.

These methods therefore have different information access. The FATE-style baseline reaches about 0.8595 recall, while our recent-plus-history probe reaches 0.6681. We agree that we should add a simple table giving the exact inputs, history windows, candidate construction, and scoring steps for each predictor. We will check these details against the released code.

Reviewer_hik4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Early routing is clearly useful and increasing the R^2 value is clearly good.

Critical strengths

Early routing is clearly useful and increasing the R^2 value is clearly good.

Critical weaknesses

The paper was clearly written entirely by LLMs and the contribution is infinitesimal.

Questions for authors

The extra history seems to add very little value. Is this because of your technique, or is the information actually not very useful?

1 comment
AuthorsOpenReview ↗

Clarification on AI Use and Research Contribution

Thank you for reviewing our paper and sharing your concerns.

First, we would like to point out that page 10, lines 514–531, already contains a detailed AI Use Statement. It explains where AI assistance was used, including code review, discussion of results, figure-script review, and manuscript preparation. We have tried to make the scope of this assistance clear and complete.

We also understand the concern about readability. Some AI-assisted language polishing may have made our explanations too condensed. This may be why some core concepts and experimental results are difficult to follow. We will explain these parts more clearly in simpler language.

About the scientific contribution, our goal is not to propose a new routing algorithm. We want to know whether older expert selections can still help predict the next router when the complete scores from the previous layer are already known. Across four MoE models and four depths, adding older history improves held-out R² by 0.0455 on average and expert-set recall by 3.31 percentage points. We also compare aligned history with shuffled history of the same feature capacity. This helps us check that the gain comes from the correct history rather than simply adding more features.

We agree that the benefit becomes small with stronger predictors or richer recent information. With three recent score vectors, the direct history gain is almost gone in three models. Against strong gate-reading predictors, the matched history-specific oracle gain is only about 0.56 percentage points. We reported these limits because they are also part of our findings. A local nonlinear test in JetMoE further suggests that the measured gain is not only caused by using a linear recent-only probe, although this test covers just one layer.

We have provided an anonymous code repository with the implementations, configurations, saved results, and reproducibility checks: https://anonymous.4open.science/r/moe-routing-anonymous-artifact-FD5A

We also provide a separate anonymous visualization website at https://anonymous-artifact.net/. It is only for browsing recorded routing traces and does not generate the reported numerical results.

We respect the reviewer's assessment of the paper's significance. Since the review also raises serious concerns about technical soundness, could the reviewer please point out the specific methodological or experimental problems? We would be glad to address them directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.