- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a trainable proxy objective for expert-weight movement in single-sequence, expert-offloaded MoE decoding. The objective reshapes routing through a capacity-constrained soft top-M residency operator, with a memoryless form based on batch-averaged routing probabilities and a stateful form using an exponential moving average of past routing probabilities. Experiments cover OLMoE-architecture models ranging from 62M to 6.8B parameters. The authors evaluate freshly recorded routing traces using LRU miss counts and the offline-optimal Belady miss count, which provides a lower bound for each fixed routing workload. At a fast-tier capacity holding 25% of expert weights, the reported simulated LRU misses decrease by approximately 14–16× for the 64-expert, 3.6B–6.8B models, accompanied by a 7–13% relative perplexity increase. In a separate H100 trace replay at 12.5% capacity, the 6.8B model achieves throughput improvements of 63% with blocking transfers and 90% with overlapped transfers.
Critical strengths
- The paper explicitly incorporates cache capacity and estimated data movement into router training, aligning the objective with deployment settings where only a subset of expert weights fits in fast memory. At 25% cache capacity, the method achieves fewer cache misses than an adjacent-token routing consistency objective at comparable language-model quality, providing empirical support for capacity-aware optimization.
- The paper investigates both incorporating the objective during pretraining and adapting an existing model, while examining learning rate, training budget, and load balancing. The finding that a higher adaptation learning rate substantially narrows the gap between these approaches provides useful insight into how optimization schedules affect learned locality and informs the adaptation of existing models.
Critical weaknesses
- The comparisons focus primarily on loss components and switch-only controls. These are useful for understanding the mechanism, but direct evidence of gains over established router fine-tuning methods remains limited. ReMoE (ICML 2026) is a relevant comparator because it also fine-tunes routing to improve expert reuse under memory-constrained expert offloading. Moreover, the difference from the switch-only control is not statistically significant on the released pretrained model, leaving the relative advantage in practical adaptation settings less firmly established.
Questions for authors
- Does the stateful variant retain its advantage over the memoryless variant when total capacity, capacity allocation across layers, and perplexity are matched? Could a targeted control help distinguish the benefit of history modeling from that of capacity allocation?
- What additional capabilities does the capacity-constrained objective provide relative to the ReMoE method? Under the same released model, adaptation data and training budget, deployment capacity, and quality constraint, what differences would the authors expect or observe?
- The relative performance of the objectives changes with cache capacity. For a model deployed across multiple memory configurations, do the results support a training configuration that works well across capacities, or is separate adaptation preferable? How should the training capacity budget be chosen relative to the deployment capacity?
4 comments
I appreciate the authors' rebuttal; some of my concerns have been addressed. My remaining comments are as follows:
-
The response indicates that most of the gains of the stateful form comes from the static part (i.e., focusing routing on a subset of experts), instead of from temporal reuse of recently loaded experts (). Since the static number is calculated per chunk, it can also be reduced simply by making routing more concentrated within each sequence, which does not necessarily need token order. Could the authors give more evidence that the stateful form truly benefits from order of past routing, instead of mostly from within-sequence concentration?
-
The appendix indicates that perplexity slightly increases on reasoning and coding text after adaptation. Perplexity might not capture performance on downstream generation tasks well; it would therefore be useful to check this more directly. The current zero-shot evaluations are evaluated by option likelihood, not generation, so a small difference in routing would not be accumulated across multiple decoding steps. For the released model, could the authors report results on generation-based reasoning and coding tasks, compared with the reference router?
I thank the authors again for their time and effort on the rebuttal. I will maintain my current rating.
Author Response to Reviewer nNgn [1/2]
Thank you for the follow-up. We ran three checks and added them to the revised manuscript (Appendix G.1, Table 18).
Does the stateful form benefit from the order of past routing?
We cannot show that it does. The checks point to concentration, as you suggested.
Shuffled traces. We permuted the token order within each recorded chunk, keeping each token's accesses together, and replayed LRU (five permutations per chunk, three seeds). At 25% capacity, stateful λ = 2.0 goes from 46,145 to 60,471 misses and pooled memoryless λ = 1.0 from 275,008 to 289,007. Static does not change. The stateful form has 6.0× fewer misses before shuffling and 4.8× after. Shuffling adds about 14,000 misses to both, so the smaller ratio is not evidence of an order effect.
Fixed pin chosen on other chunks. We split the 24 evaluation chunks into two halves and served each half with the 160 blocks most frequent in the other half. The stateful form has 27,004 misses versus 149,486 for pooled memoryless (5.5×). Per-chunk pins give 21,993 versus 142,862 (6.5×). The two stateful pins share 156–159 of their 160 blocks. On the stateful traces, the fixed pin also has fewer misses than LRU (27,004 versus 46,145). The concentration is shared across chunks and does not depend on reusing recently loaded experts.
Whole-sequence-average control. We trained a control that replaces the causal EMA with the average routing probabilities of each training sequence. Model, data, token budget and the pooled 160-block budget are unchanged. Three-seed means at 25% capacity:
| Variant | PPL | LRU misses | Static | κ |
|---|---|---|---|---|
| Pooled memoryless, λ = 1.0 | 61.62 | 275,008 | 142,862 | 1.92 |
| Whole-sequence average, λ = 2.0 | 62.00 | 173,161 | 74,111 | 2.34 |
| Whole-sequence average, λ = 4.0 | 62.18 | 129,478 | 57,752 | 2.24 |
| Stateful, λ = 2.0 | 61.34 | 46,145 | 21,993 | 2.10 |
Stateful λ = 2.0 has 3.8× and 2.8× fewer misses than the two controls. Each of the six control runs has 2.3–4.9× the stateful mean misses, at higher PPL. The PPL gaps (0.66 and 0.83) are outside the 0.5 band we set before the runs, so these are not matched-perplexity comparisons. The difference is in Static (2.6–3.4× lower), not κ, whose seed ranges overlap. At 12.5% capacity the controls have only 1.13× the stateful misses.
What this shows. Within-sequence concentration explains part of the reduction: the controls have 1.6× and 2.1× fewer misses than pooled memoryless, at 0.39 and 0.56 higher PPL. It does not explain all of it at 25% capacity. The control also differs from the stateful form in history length, recency weighting and access to later tokens, so it does not isolate order, and we did not establish why the two differ. Section 5.2 now states this limitation.
Author Response to Reviewer nNgn [2/2]
Generation-based reasoning and coding tasks on the released model.
We ran GSM8K, HumanEval and MBPP with greedy decoding (Appendix G.2, Table 19). The fine-tuned routers lose some accuracy against the released router. The traffic term adds no detectable loss beyond λ = 0 fine-tuning.
Setup. lm-evaluation-harness 0.4.12, default settings: GSM8K with five examples and with eight worked examples (1,319 problems), HumanEval (164, zero-shot) and MBPP test (500, three-shot). Only the router weights differ. We scored the released router, ten λ = 0 fine-tuning seeds, 18 memoryless λ = 0.15 seeds (the weights of seeds 0 and 1 were not saved) and ten ReMoE scale-2 seeds. Intervals are 95% bootstrap intervals over problems and router seeds (10,000 resamples).
Scores are percentages (exact match for GSM8K, pass@1 for code). Differences are in points.
| Task | Released | λ = 0 | Memoryless | ReMoE | Memoryless − λ = 0 | Memoryless − released |
|---|---|---|---|---|---|---|
| GSM8K 5-shot, strict | 8.19 | 8.07 | 7.83 | 8.44 | −0.24 [−1.02, +0.54] | −0.36 [−1.60, +0.87] |
| GSM8K 5-shot, flexible | 8.72 | 8.51 | 8.21 | 8.99 | −0.29 [−1.07, +0.48] | −0.51 [−1.79, +0.71] |
| GSM8K 8-shot, strict | 7.13 | 5.85 | 5.28 | 5.56 | −0.56 [−1.35, +0.21] | −1.84 [−2.99, −0.75] |
| GSM8K 8-shot, flexible | 12.89 | 12.77 | 12.40 | 12.60 | −0.37 [−1.27, +0.49] | −0.48 [−1.87, +0.87] |
| HumanEval | 12.80 | 12.74 | 12.64 | 13.35 | −0.11 [−1.36, +1.05] | −0.17 [−2.74, +2.30] |
| MBPP test | 18.40 | 16.88 | 16.77 | 16.48 | −0.11 [−1.50, +1.34] | −1.63 [−3.87, +0.54] |
Against the released router. All three fine-tuned groups are lower on eight-shot GSM8K under the strict filter, with intervals that exclude zero. Under the flexible filter none is separated from the released router. The result depends on the answer filter, but we cannot say the strict-filter drop is only a formatting effect. On MBPP test all three means are lower, with intervals that include zero. HumanEval intervals are wide. The released router scores below 10% on five-shot GSM8K and on the eight-shot strict filter, so those rows say little about retained reasoning ability.
Against λ = 0 fine-tuning. No interval excludes zero on any task. This is not evidence of equality.
Predictions written in advance. The memoryless mean should be at most 2.0 points below the seed-0 λ = 0 reference on five-shot strict GSM8K and on HumanEval. Both held (−0.13 and +0.44).
Larger code pool. Before any generation score existed, we fixed a pool of 1,128 problems: HumanEval and the MBPP test, train and validation splits, equally weighted. On this pool, memoryless minus λ = 0 is +0.08 [−0.86, +1.00]. Against the released router the differences are −1.57 [−3.10, −0.07] for memoryless, −1.65 [−3.24, −0.07] for λ = 0 and −1.36 [−3.00, +0.26] for ReMoE. Memoryless minus ReMoE is −0.21 [−1.18, +0.75]. We do not claim an advantage over ReMoE.
What this shows. The losses appear with and without the traffic term, so they come from router fine-tuning under a WikiText perplexity constraint, not from the term. Section 5.4 reports the results and Section 6 now states that this constraint does not preserve generation scores. Appendix G.2 has the output-validity checks and a post-hoc experiment with a larger KL weight.
Thank you again for the follow-up questions; both checks improved the paper.
Author Response to Reviewer nNgn
Thank you for the review. We added the requested ReMoE comparison and controls for capacity and pooling.
Weakness: direct comparisons with ReMoE and switch-only.
We reimplemented the ReMoE locality block with the published term weights and tested six strengths from scratch at 414M/64 experts and a = 0.01 (Appendix A.1). These compare loss terms, not complete systems; each strength is one run.
At strength 8, ReMoE reaches 61.78 PPL versus 62.00 for memoryless + switch. It has 1.24× the latter's LRU misses at 25% capacity but 0.48× at 12.5%, and improves on switch-only λ = 1.0 at both capacities (Table 2(d)).
On released OLMoE-1B-7B, we use the same 2M adaptation tokens, router-only training, and worst-seed quality constraint of ΔPPL ≤ 0.3 relative to the λ = 0 fine-tuning reference. ReMoE scale 2 achieves fractional simulated-cache-cost reduction of 0.138 ± 0.020 over ten seeds versus 0.116 ± 0.064 over twenty for memoryless adaptation (Table 12). Mean PPL costs are +0.204 and +0.179, respectively; Welch p = 0.18. Paired comparisons also show no significant memoryless advantage over ReMoE scale 2 or switch-only. We do not claim superiority on the released model.
Q1. Does the stateful advantage survive matched pooling and capacity?
Yes, at the tested 25% capacity. Both variants use a pooled 160-block budget, a = 0.01, no switch term, and three seeds (Table 2(c)). Stateful λ = 2.0 reaches 61.34 PPL versus 61.62 for pooled memoryless λ = 1.0, with 6.0× fewer LRU misses. The larger pooled budget alone therefore does not explain the advantage.
At 12.5%, the same stateful variant has 0.87× the pooled-memoryless misses but 1.69× the switch-only control's misses. The reduction appears in the static component, not improved κ (Section 5.2), so we do not interpret it as evidence of better temporal reuse in the measured traces.
Q2. What does explicit capacity add beyond ReMoE?
Eq. 2 enforces a specified residency budget; the reimplemented ReMoE locality terms do not take capacity as an input. This is a structural distinction, not a guarantee of a better trade-off.
For the stateful form at λ = 4.0, reducing the training budget from 160 to 80 blocks cuts LRU misses at the 80-block deployment capacity by 4.4×, at 1.40 higher PPL; the 80-block result is one run (Tables 1–2).
Against ReMoE strength 8, stateful λ = 2.0 has 0.56× the misses at 25% capacity and 0.44 lower PPL, but 3.23× the misses at 12.5%. The evidence is therefore capacity-dependent, not uniform superiority.
Q3. One configuration across capacities, and budget selection.
The stateful 160-block variant is strong at 25% and above but performs poorly at smaller capacities. The 80-block variant improves the smaller-capacity trade-off while producing more misses at 25% (Appendix E, Figure 3). This motivates capacity-specific tuning, not a conclusion that separate training is always necessary.
For memoryless adaptation on the released model, training budgets of 4 and 8 blocks per layer are not statistically separated at any tested deployment capacity (Welch p ≥ 0.67; Table 13). Budget 16 costs more at capacity 256 but is not separated from the others at 64 or 128. Thus, neither matching training and deployment budgets nor always choosing the smallest budget is established by these experiments.
Appendix E recommends selecting budget and strength on held-out traces at the intended deployment capacity. We did not perform that selection protocol and state this explicitly.