- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes StickyMoE, an auxiliary training objective that penalizes the squared L2 distance between adjacent tokens’ softmax gate distributions. A soft-hard variant adds a window-anchor penalty. The method is evaluated using language-model perplexity, top-1 expert switch rate, simulated LRU cache hits, and expert utilization entropy. Experiments cover small and medium models on WikiText-2, additional seeds, TinyStories, and larger models on WikiText-2 and SlimPajama. A transfer-replay benchmark and a shared-expert cross-layer extension are also included.
The reported reduction in top-1 switching is consistent across several settings. Quality improvements are less general: on SlimPajama, the strongest tested penalty reduces switching by about 43% while increasing perplexity by 7.5%. The evidence supports a useful training regularizer for routing locality, but does not yet establish the claimed deployment benefit or the necessity of applying the objective during pretraining.
Critical strengths
-
The intervention is simple and produces a repeatable locality effect. The soft loss can be added to standard MoE training without changing the inference architecture. Table 7 shows substantial switch-rate reductions across three additional seeds, while Table 3 reports high per-layer utilization entropy. This makes the proposal a plausible, practical regularizer rather than an isolated result from one seed. Utilization entropy alone does not establish that expert specialization is preserved, but it is useful evidence against gross load imbalance.
-
The broader experiments expose a meaningful dependence on training regime. TinyStories shows a strong locality improvement with a small reported perplexity cost, whereas the SlimPajama experiment retains a clear quality cost. Reporting this unfavorable result is valuable: it limits the inference that increasing model size makes locality free and gives a more credible picture of where the method may be useful.
Critical weaknesses
- The measured quantity does not establish cache efficiency for the evaluated top-2 models. Section 5.1 activates two experts per token, but Sections 5.4 and D.2 count and replay only top-1 accesses. A stable top-1 expert can coexist with a changing second expert; conversely, exchanging the ranks of two resident experts changes top-1 SR without requiring a load. SR is therefore not generally equal to miss probability, as claimed around Eq. (5). Table 8 measures copies of expert-sized tensors from a trace, not end-to-end model decoding, so its 3.5–4.1× transfer reductions do not establish inference speedups.
The cache allocation across layers is equally consequential. With layer-specific weights, the physical cache key must include both layer and expert identity. A global cache of two tensors behaves differently from separate two-slot caches per layer. Even a perfectly repeated route through four layers can incur misses at every token when its working set exceeds a global cache. Appendix G acknowledges the access-order issue, but its claim that cross-token consistency helps single-token decode only with shared weights is too categorical: retaining layer-specific experts across steps can also provide reuse if enough slots are allocated. The main deployment claim needs an explicit memory budget and actual top-2 access trace. The shared-pool extension does not resolve this gap because Table 10 lacks the shared-pool γ=0 control.
- The controls do not support the conclusion that locality must be learned during pretraining. The “ReMoE” baseline is explicitly a simulation: it freezes the full model except routers and uses one loss coefficient, learning rate, and short fine-tuning schedule (Section 5.2). Its failure demonstrates a limitation of that configuration, not an inability to retrofit locality. The Oracle-MoE comparison is also a simplified approximation. A matched-budget comparison with tuned router-only and full-model post-hoc adaptation is needed to isolate representation co-adaptation from optimization budget or baseline choice.
The quality evidence also needs reconciliation before supporting the stronger scaling claim. In Table 11, baseline validation PPL rises to 5745.8 despite repeated training on the small corpus; the 14.4% relative gain is difficult to interpret without training curves or a convergence check. Table 9 reports plain-loss PPL values that differ from Table 3 while retaining essentially identical switch rates, without explaining the evaluation or checkpoint difference. Finally, the WikiText-2 seed-variance estimates do not establish that the 1.8% TinyStories cost is within noise on a different dataset. These issues do not erase the locality result, but they weaken the evidence for the claimed quality benefit.
Questions for authors
-
Clarification and decisive validation: Is C=2 a global physical cache or a separate cache for each layer? How are layer identity, both selected experts, access order, and cache persistence across tokens represented? Please provide top-2 miss counts and end-to-end autoregressive decode latency under a stated total memory budget, with compute and transfer time distinguished. This would directly affect the cache-efficiency assessment.
-
Causal control: Does the training-time advantage persist against a tuned post-hoc baseline, including both router-only and full-model fine-tuning with comparable optimization budgets? Please specify which parts of the published ReMoE and Oracle-MoE methods are retained or omitted. Evidence that the existing baseline choices are adequate would affect the assessment of the main comparative claim.
-
Clarification: What accounts for the plain-loss PPL differences between Tables 3 and 9, and for the very high validation PPL of the 168.5M model? Were checkpoints, token aggregation, preprocessing, and evaluation context identical? Training and validation curves for the scaling experiment would help determine whether its relative improvement reflects a reliable modeling benefit.
-
Causal control: Under the shared-pool architecture, what are PPL, cross-layer switching, and actual top-2 cache misses at γ=0 compared with γ=1? The non-shared baseline cannot isolate the cross-layer loss’s contribution. This is a necessary control for the extension’s current claim, rather than a suggestion for a new research direction.
2 comments
Part II
4. Shared-pool control (W2, Q4)
We trained the shared-pool model at gamma=0 at both scales (same code and seed as the gamma>0 runs) and evaluated all runs under one protocol (Table 21; full-split PPL, top-2 access, shared cache holding 50% of the pool):
| gamma | PPL | Cross-layer SR | Top-2 misses | |
|---|---|---|---|---|
| Small | 0 (control) | 281.6 | 0.704 | 1.096 |
| Small | 1.0 | 276.7 | 0.068 | 0.408 |
| Medium | 0 (control) | 261.9 | 0.749 | 0.980 |
| Medium | 1.0 | 256.6 | 0.050 | 0.364 |
The PPL gap to the non-shared baselines comes from the shared pool, not the loss: gamma=1 lowers PPL by 1.8% and 2.0% against the control. Cross-layer SR falls from 0.70-0.75 to 0.05-0.07 and simulated top-2 misses fall about 2.7x at both scales. This is a single-seed simulation with no decoding-speed measurement, and the paper says so.
The paper's core finding, a repeatable reduction in routing switching from a training-time consistency loss, stands, with narrower claims about deployment and scale.
Part I
Table and appendix numbers below refer to the revised paper.
Thank you for the detailed and precise review. Several of your points are correct. Checking them against our code and logs also showed problems beyond your list, notably in the 168.5M experiment. We state them directly; the repeatable reduction in top-1 switching does not depend on the parts we withdraw, but our deployment and scaling claims needed to change.
1. Cache efficiency, top-2, cache allocation (W1, W2, Q1)
The simulator keeps a separate 2-slot LRU cache per layer and per sequence (layer identity is implicit; the cache persists across tokens) and accesses only the top-1 expert. The second expert is not counted or replayed in SR, CHR or the transfer replay. The implied budget (2 of N experts per layer: 50% for small/medium, 25% for SlimPajama) was never stated.
We concede: SR is not miss probability (a changing second expert is invisible to it, and a rank swap between resident experts is a switch without a load), so we removed the Eq. (5) sentence equating them. The Appendix H claim that cross-token reuse needs shared weights was too categorical (reuse depends on the slot budget relative to the per-token working set). Table 8 is transfer time on a top-1 trace, not an inference speedup, and is now labelled so in the table, abstract and text.
New Appendix E recomputes everything with both selected experts accessed (misses per token-layer, baseline over Soft lambda=0.5, per-layer LRU, C=2):
The gain carries over to the second expert but is smaller when the cache is as tight as two slots. Appendix E also covers global caches keyed by (layer, expert), LFU and oracle replacement, and a set-change metric. 11-16% of steps (excluding the Hard-Window ablation) only reorder the pair: no load, but counted as a top-1 switch, which confirms your point. End-to-end decoding with transfer and compute separated is untested; the limitations say so.
2. Post-hoc baselines (W3, Q2)
We concede this. Our ReMoE baseline is one configuration (lambda=0.1, learning rate 1e-4, frozen experts, 0.1x steps) and our Oracle-MoE is a simplified approximation, not the published method. Its failure shows that configuration does not work, not that locality cannot be retrofitted. We renamed them "ReMoE-style" and "Oracle-style", and the paper now says a tuned post-hoc alternative (router-only and full-model fine-tuning at matched budget) is not ruled out. We did not run that comparison, and we no longer claim locality must be learned in pretraining.
3. Quality evidence (W4)
168.5M (Table 11). Your concern is correct and the problem is larger than interpretation. Baseline training perplexity falls to about 1.1 while validation perplexity bottoms out near 196 around step 3,000 and rises to 5,746: the model memorizes the 2.3M-token corpus, and our "short schedule" explanation was wrong. The 13-14% "improvement" compared degrees of overfitting, so we withdrew it. A rerun with a 3,000-step schedule (validation perplexity falls throughout, 564 to 207; seed 42, full split; Table 12) shows the actual picture:
The loss costs quality at this scale, as on SlimPajama (+2.6% and +7.7%), and the paper no longer suggests otherwise.
Tables 3 and 9. The main-results and ablation tables report perplexity on the first 50 validation batches (85% of WikiText-2); the other tables use the full split. We reproduce Table 3's baseline exactly (245.30 vs 258.98 full). Across 30 checkpoints the full-split value is uniformly 5.0-6.5% higher, so within-table comparisons are unaffected. Our "full validation split" statement was wrong; the paper now states each protocol and Appendix E lists both values. The SlimPajama table had the same issue (about 400 of 37k sequences). On the full split the cost is +2.6% and +7.7% (was +2.9% and +7.5%), so the conclusion is unchanged.
TinyStories. Over three seeds (0, 1, 42; Table 19) perplexity is 7.15 +/- 0.07 for the baseline and 7.29 +/- 0.04 for Soft lambda=0.5, a +2.0% cost of about two seed standard deviations, with SR falling from 0.710 to 0.156. The "within noise" claim is removed.