- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether expert selections in previous layers provide incremental predictive value for a token's routing in the current layer. The authors train ridge-regression probes to predict the current layer's router scores from the previous layer's full score vector, then test whether adding earlier layers' expert selections improves prediction. They repeat this comparison with four frozen pretrained MoE models, using shuffled histories as a control. The main experiments report an average R² increase of 0.0455 and a 3.31 percentage-point improvement in expert-set recall.
Critical strengths
- Controlled tests of whether earlier routing choices help prediction. The paper compares predictions with and without earlier routing history, and uses shuffled history as a control. This helps distinguish useful historical information from simple cross-layer correlation.
Critical weaknesses
-
The study is not sufficiently motivated. Residual connections make cross-layer routing dependence plausible, although they do not imply the specific conditional gain measured here. The paper needs to explain what important new understanding or MoE design decision follows from measuring this gain.
-
The demonstrated practical value is limited.
-
The predictive gains are difficult to interpret. R² and HNV measure improvement under a particular probe and feature choice; they do not directly establish the amount of information in routing history. Although the paper includes recall and token-level analyses, it should connect these more clearly to which expert choices improve and why the aggregate gain matters.
-
Layerwise Recurrent Router already studies cross-layer routing dependence using mutual information. Conditional probe prediction asks a different question, but the paper should explain what additional insight it provides. Forward predictability should also be distinguished from training benefits that recurrent routers may obtain through additional gradient paths.
Questions for authors
- Given that extra history adds little value with stronger predictors or richer recent information, and that the runtime selector provides only limited gains, in which settings do the authors expect older routing history to be meaningfully useful? What important understanding or MoE design decision follows from these results?
- Please specify the complete inputs, history windows, candidate construction and scoring procedures of the FATE-style, HCR-B and history predictors. Can matched recent-only/shuffled branches be evaluated in the runtime selector, and can the same strong prediction mechanism be compared with and without older selections?
- How do the findings relate to the existing mutual-information routing analysis like in Layerwise Recurrent Router paper? Which conclusions remain stable across reasonable predictor and preprocessing choices beyond the single nonlinear JetMoE check?
Research Motivation, Experimental Comparisons, and Limitations
Thank you for reading our paper and raising these questions. We would like to explain what the experiments show and where the current results are still limited.
Q1. When is older routing history useful, and what do we learn from it?
We agree that the practical benefit is limited, especially when stronger predictors or more recent scores are available. We reported the small and negative results in the paper as well.
The main finding is that the value of history depends on the model and on how much recent information is already known. JetMoE has a much larger history gain than the other three models. When three recent score vectors are used, JetMoE still has a positive mean direct gain, while the other three models do not. This suggests that history may be worth checking when the previous router scores alone are not very predictive.
We also used HNV to rank the 16 model-depth settings. Using history for the top eight settings retains about 71% of the average recall gain of using it everywhere. However, simpler ranking rules work almost as well. Also, using history in half of the settings does not mean a 50% reduction in computation.
We do not claim that this work speeds up inference or produces a better routing algorithm. We think it helps identify where older history provides extra predictive value and where the recent state is already enough.
Q2. How are the predictors defined, and are the runtime comparisons matched?
The recent-only probe uses the full score vector from the previous router. The history probe adds top-k selections from three older layers. Both use multi-output ridge regression, with PCA applied to the history features. The FATE-style baseline and HCR-B can also use the target-layer gate parameters, while our history probes cannot. We agree that the complete inputs, history windows, candidate construction, and scoring rules need a clearer table. We will verify the implementation details from the released code.
We did run matched comparisons in the oracle analysis. Against a recent-only third predictor of the same family, the additional history-specific oracle gain was about 0.56 percentage points. Against shuffled history, the alignment-specific oracle gain was about 0.63 percentage points. These are hindsight upper-bound comparisons, not runtime gains.
The actual runtime selector improved on HCR-B by only 0.1831 percentage points. We tested the two-strong-predictor selector and the selector with an added history arm. We did not run corresponding selectors with an extra recent-only or shuffled-history arm. Because these controls are missing, we cannot attribute the runtime improvement specifically to correctly aligned older history. We agree that the suggested matched runtime experiments are needed for that claim.
Q3. How does this differ from Layerwise Recurrent Router, and are the results stable?
Layerwise Recurrent Router has already studied cross-layer routing relationships. Our question is more specific. We ask whether older expert selections still help prediction after the complete score vector from the immediately previous layer is already given.
Two layers can be related without the older one providing extra predictive value after we know the recent state. This is why we compare recent scores alone, recent scores with aligned history, and recent scores with shuffled history of the same feature capacity. Our results measure predictive gain under these probes. They do not prove conditional mutual information or causality. They also do not measure the training benefit of recurrent routing connections.
The main R² gain was positive in all 16 model-depth settings. On C4, we also observed positive gains across four models and four token positions. A capacity-matched nonlinear check at one JetMoE layer found a history gain of +0.1644 R² over a nonlinear recent-only predictor.
There are important limits. The nonlinear check covers only one model and one layer. The C4 experiment changes the corpus and token positions but uses the same four architectures. We also found that history gains become much smaller with richer recent scores. We therefore do not claim that the results will hold with every predictor or preprocessing choice. Our conclusions are limited to the settings we tested.