acceptodds
Under review as a conference paper at ICLR 2027

TEMPORALLY CONSISTENT ROUTING: A TRAINING- TIME OBJECTIVE FOR CACHE-EFFICIENT MIXTURE-OF- EXPERTS

Abstract

Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts, causing constant weight swapping between slow storage and fast memory on edge devices. Existing remedies are either system-level (such as caching heuristics) or post-hoc (such as router fine-tuning), leaving the root cause unchanged during pretraining. We propose a differentiable routing consistency loss that penalizes abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. The method requires no architectural changes and adds only a single hyperparameter, λ. Unlike post-hoc approaches, it allows expert representations and routing decisions to co-adapt from the first training step. Experiments on small and medium-sized MoE language models show that the method reduces the expert switch rate by up to 59% while simultaneously improving perplexity on the medium model. It also reduces cache misses by up to 3.92×, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. These results suggest that routing temporal locality is most effectively learned during training rather than added afterward.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Routing Consistency, Memory-Efficient Inference, Expert Locality, Conditional Computation, Edge AI, Router Training, Cache Efficiency, Hardware-Aware Learning, Sparse Language Models, Large Language Models, Differentiable Routing, Inference-Aware Training, Model Serving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mZtPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes StickyMoE, an auxiliary training objective that penalizes the squared L2 distance between adjacent tokens’ softmax gate distributions. A soft-hard variant adds a window-anchor penalty. The method is evaluated using language-model perplexity, top-1 expert switch rate, simulated LRU cache hits, and expert utilization entropy. Experiments cover small and medium models on WikiText-2, additional seeds, TinyStories, and larger models on WikiText-2 and SlimPajama. A transfer-replay benchmark and a shared-expert cross-layer extension are also included.

The reported reduction in top-1 switching is consistent across several settings. Quality improvements are less general: on SlimPajama, the strongest tested penalty reduces switching by about 43% while increasing perplexity by 7.5%. The evidence supports a useful training regularizer for routing locality, but does not yet establish the claimed deployment benefit or the necessity of applying the objective during pretraining.

Critical strengths
  1. The intervention is simple and produces a repeatable locality effect. The soft loss can be added to standard MoE training without changing the inference architecture. Table 7 shows substantial switch-rate reductions across three additional seeds, while Table 3 reports high per-layer utilization entropy. This makes the proposal a plausible, practical regularizer rather than an isolated result from one seed. Utilization entropy alone does not establish that expert specialization is preserved, but it is useful evidence against gross load imbalance.

  2. The broader experiments expose a meaningful dependence on training regime. TinyStories shows a strong locality improvement with a small reported perplexity cost, whereas the SlimPajama experiment retains a clear quality cost. Reporting this unfavorable result is valuable: it limits the inference that increasing model size makes locality free and gives a more credible picture of where the method may be useful.

Critical weaknesses
  1. The measured quantity does not establish cache efficiency for the evaluated top-2 models. Section 5.1 activates two experts per token, but Sections 5.4 and D.2 count and replay only top-1 accesses. A stable top-1 expert can coexist with a changing second expert; conversely, exchanging the ranks of two resident experts changes top-1 SR without requiring a load. SR is therefore not generally equal to miss probability, as claimed around Eq. (5). Table 8 measures copies of expert-sized tensors from a trace, not end-to-end model decoding, so its 3.5–4.1× transfer reductions do not establish inference speedups.

The cache allocation across layers is equally consequential. With layer-specific weights, the physical cache key must include both layer and expert identity. A global cache of two tensors behaves differently from separate two-slot caches per layer. Even a perfectly repeated route through four layers can incur misses at every token when its working set exceeds a global cache. Appendix G acknowledges the access-order issue, but its claim that cross-token consistency helps single-token decode only with shared weights is too categorical: retaining layer-specific experts across steps can also provide reuse if enough slots are allocated. The main deployment claim needs an explicit memory budget and actual top-2 access trace. The shared-pool extension does not resolve this gap because Table 10 lacks the shared-pool γ=0 control.

  1. The controls do not support the conclusion that locality must be learned during pretraining. The “ReMoE” baseline is explicitly a simulation: it freezes the full model except routers and uses one loss coefficient, learning rate, and short fine-tuning schedule (Section 5.2). Its failure demonstrates a limitation of that configuration, not an inability to retrofit locality. The Oracle-MoE comparison is also a simplified approximation. A matched-budget comparison with tuned router-only and full-model post-hoc adaptation is needed to isolate representation co-adaptation from optimization budget or baseline choice.

The quality evidence also needs reconciliation before supporting the stronger scaling claim. In Table 11, baseline validation PPL rises to 5745.8 despite repeated training on the small corpus; the 14.4% relative gain is difficult to interpret without training curves or a convergence check. Table 9 reports plain-loss PPL values that differ from Table 3 while retaining essentially identical switch rates, without explaining the evaluation or checkpoint difference. Finally, the WikiText-2 seed-variance estimates do not establish that the 1.8% TinyStories cost is within noise on a different dataset. These issues do not erase the locality result, but they weaken the evidence for the claimed quality benefit.

Questions for authors
  1. Clarification and decisive validation: Is C=2 a global physical cache or a separate cache for each layer? How are layer identity, both selected experts, access order, and cache persistence across tokens represented? Please provide top-2 miss counts and end-to-end autoregressive decode latency under a stated total memory budget, with compute and transfer time distinguished. This would directly affect the cache-efficiency assessment.

  2. Causal control: Does the training-time advantage persist against a tuned post-hoc baseline, including both router-only and full-model fine-tuning with comparable optimization budgets? Please specify which parts of the published ReMoE and Oracle-MoE methods are retained or omitted. Evidence that the existing baseline choices are adequate would affect the assessment of the main comparative claim.

  3. Clarification: What accounts for the plain-loss PPL differences between Tables 3 and 9, and for the very high validation PPL of the 168.5M model? Were checkpoints, token aggregation, preprocessing, and evaluation context identical? Training and validation curves for the scaling experiment would help determine whether its relative improvement reflects a reliable modeling benefit.

  4. Causal control: Under the shared-pool architecture, what are PPL, cross-layer switching, and actual top-2 cache misses at γ=0 compared with γ=1? The non-shared baseline cannot isolate the cross-layer loss’s contribution. This is a necessary control for the extension’s current claim, rather than a suggestion for a new research direction.

2 comments

Table and appendix numbers below refer to the revised paper.

Thank you for the detailed and precise review. Several of your points are correct. Checking them against our code and logs also showed problems beyond your list, notably in the 168.5M experiment. We state them directly; the repeatable reduction in top-1 switching does not depend on the parts we withdraw, but our deployment and scaling claims needed to change.

1. Cache efficiency, top-2, cache allocation (W1, W2, Q1)

The simulator keeps a separate 2-slot LRU cache per layer and per sequence (layer identity is implicit; the cache persists across tokens) and accesses only the top-1 expert. The second expert is not counted or replayed in SR, CHR or the transfer replay. The implied budget (2 of N experts per layer: 50% for small/medium, 25% for SlimPajama) was never stated.

We concede: SR is not miss probability (a changing second expert is invisible to it, and a rank swap between resident experts is a switch without a load), so we removed the Eq. (5) sentence equating them. The Appendix H claim that cross-token reuse needs shared weights was too categorical (reuse depends on the slot budget relative to the per-token working set). Table 8 is transfer time on a top-1 trace, not an inference speedup, and is now labelled so in the table, abstract and text.

New Appendix E recomputes everything with both selected experts accessed (misses per token-layer, baseline over Soft lambda=0.5, per-layer LRU, C=2):

Top-1Top-2, C=2Top-2, C=3Top-2, oracle
Small3.54x2.10x3.07x1.84x
Medium3.91x2.49x4.36x2.23x

The gain carries over to the second expert but is smaller when the cache is as tight as two slots. Appendix E also covers global caches keyed by (layer, expert), LFU and oracle replacement, and a set-change metric. 11-16% of steps (excluding the Hard-Window ablation) only reorder the pair: no load, but counted as a top-1 switch, which confirms your point. End-to-end decoding with transfer and compute separated is untested; the limitations say so.

2. Post-hoc baselines (W3, Q2)

We concede this. Our ReMoE baseline is one configuration (lambda=0.1, learning rate 1e-4, frozen experts, 0.1x steps) and our Oracle-MoE is a simplified approximation, not the published method. Its failure shows that configuration does not work, not that locality cannot be retrofitted. We renamed them "ReMoE-style" and "Oracle-style", and the paper now says a tuned post-hoc alternative (router-only and full-model fine-tuning at matched budget) is not ruled out. We did not run that comparison, and we no longer claim locality must be learned in pretraining.

3. Quality evidence (W4)

168.5M (Table 11). Your concern is correct and the problem is larger than interpretation. Baseline training perplexity falls to about 1.1 while validation perplexity bottoms out near 196 around step 3,000 and rises to 5,746: the model memorizes the 2.3M-token corpus, and our "short schedule" explanation was wrong. The 13-14% "improvement" compared degrees of overfitting, so we withdrew it. A rerun with a 3,000-step schedule (validation perplexity falls throughout, 564 to 207; seed 42, full split; Table 12) shows the actual picture:

PPLChangeSRTop-2 misses, C=2
Baseline218.0-0.7991.422
Soft lambda=0.05220.9+1.3%0.6991.250
Soft lambda=0.5228.9+5.0%0.4700.918

The loss costs quality at this scale, as on SlimPajama (+2.6% and +7.7%), and the paper no longer suggests otherwise.

Tables 3 and 9. The main-results and ablation tables report perplexity on the first 50 validation batches (85% of WikiText-2); the other tables use the full split. We reproduce Table 3's baseline exactly (245.30 vs 258.98 full). Across 30 checkpoints the full-split value is uniformly 5.0-6.5% higher, so within-table comparisons are unaffected. Our "full validation split" statement was wrong; the paper now states each protocol and Appendix E lists both values. The SlimPajama table had the same issue (about 400 of 37k sequences). On the full split the cost is +2.6% and +7.7% (was +2.9% and +7.5%), so the conclusion is unchanged.

TinyStories. Over three seeds (0, 1, 42; Table 19) perplexity is 7.15 +/- 0.07 for the baseline and 7.29 +/- 0.04 for Soft lambda=0.5, a +2.0% cost of about two seed standard deviations, with SR falling from 0.710 to 0.156. The "within noise" claim is removed.

4. Shared-pool control (W2, Q4)

We trained the shared-pool model at gamma=0 at both scales (same code and seed as the gamma>0 runs) and evaluated all runs under one protocol (Table 21; full-split PPL, top-2 access, shared cache holding 50% of the pool):

gammaPPLCross-layer SRTop-2 misses
Small0 (control)281.60.7041.096
Small1.0276.70.0680.408
Medium0 (control)261.90.7490.980
Medium1.0256.60.0500.364

The PPL gap to the non-shared baselines comes from the shared pool, not the loss: gamma=1 lowers PPL by 1.8% and 2.0% against the control. Cross-layer SR falls from 0.70-0.75 to 0.05-0.07 and simulated top-2 misses fall about 2.7x at both scales. This is a single-seed simulation with no decoding-speed measurement, and the paper says so.

The paper's core finding, a repeatable reduction in routing switching from a training-time consistency loss, stands, with narrower claims about deployment and scale.

Reviewer_4okkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors notice that consecutive tokens in MoE models may frequently activate different experts, leading to repeated expert weight swapping between slow storage and fast memory. This paper presents STICKYMOE, a training-time method for improving temporal locality in MoE routing. In contrast to prior approaches based on system-level caching heuristics or post-hoc router fine-tuning, STICKYMOE introduces a differentiable routing consistency loss during pretraining to discourage abrupt expert switches between adjacent tokens. Experiments on small- and medium-scale MoE language models show that STICKYMOE can substantially reduce expert switching and cache misses while maintaining or improving language modeling quality.

Critical strengths
  1. The paper is clearly written, with the proposed loss function well formulated and explained.

  2. The proposed design requires no additional inference-time routing computation, making it practical and readily applicable.

Critical weaknesses
  1. The additional benefit of the Soft-Hard variant needs further clarification. Figure 1 shows nearly overlapping Soft and Soft-Hard trade-offs on the small model. Section 5.7 and Table 6 report additional locality gains on the medium model, with a perplexity trade-off. It remains unclear how much benefit Soft-Hard adds over a tuned Soft variant at comparable language-model quality.

  2. Language-model quality is evaluated primarily through perplexity, without downstream task evaluation. The routing, cache, and transfer measurements are useful, but do not establish whether task-level performance is preserved or improved.

Questions for authors

Can the proposed method be combined with current system-level optimizations?

3 comments
AuthorsOpenReview ↗

Response (Part 1)

Table and appendix numbers below refer to the revised paper.

Thank you for the clear and fair review. Your two weaknesses are valid. On the first, we re-examined the Soft-Hard variant with additional seeds and direct drift measurements, and the evidence supports your concern: we cannot show a benefit over a tuned Soft variant, and we have changed the paper's claims accordingly.

1. Added benefit of Soft-Hard over a tuned Soft variant (W1)

Single-seed comparison. We first re-compared the two at matched perplexity on seed 42 (Tables 3 and 6, WikiText-2; PPL on the first 50 validation batches). On the small model they are indistinguishable (Soft lambda=0.1: PPL 247.6, SR 0.476; Soft-Hard alpha=0.05: 247.9, 0.472). On the medium model Soft-Hard (alpha=1.0) has the lowest switch rate (0.274 vs 0.291 for Soft lambda=0.5), at about 3.5 more perplexity points.

Additional seeds. Soft-Hard was missing from the multi-seed study, so we trained it at seeds 0, 1 and 2 (lambda=0.1, W=4) and compared it with Soft on the same seeds (new Table 20; mean +/- std over seeds 0-2, full validation pass):

ModelMethodnPPLSR (top-1)Top-2 misses, C=2Distinct experts / window
SmallSoft lambda=0.53264.2 +/- 0.40.318 +/- 0.0190.469 +/- 0.0391.71 +/- 0.04
SmallSoft-Hard alpha=1.03266.6 +/- 1.00.319 +/- 0.0330.464 +/- 0.0451.69 +/- 0.07
MediumSoft lambda=0.53292.6 +/- 2.00.251 +/- 0.0080.396 +/- 0.0121.54 +/- 0.02
MediumSoft-Hard alpha=1.03291.3 +/- 4.00.234 +/- 0.0220.390 +/- 0.0131.49 +/- 0.05

With seeds, the medium-model difference you asked about shrinks to within noise: SR is 0.234 +/- 0.022 vs 0.251 +/- 0.008 (Soft-Hard lower by about one standard deviation), top-2 misses 0.390 vs 0.396 and perplexity 291.3 vs 292.6 agree within seed noise. On the small model Soft-Hard has equal locality at higher perplexity. The seed-42 gap was largely seed noise. Table 20 also reports alpha=0.5 (n=3 at both sizes), which is weaker than alpha=1.0 on locality.

Drift. The motivation for Soft-Hard is long-range drift inside a window, so we also measured drift directly (Appendix E, Table 18 and the last column above): the mean number of distinct top-1 experts per W=4 window, and the mean run length of a resident expert. We find no consistent advantage: on the small model Soft lambda=0.5 has less drift than Soft-Hard alpha=1.0 on seed 42 (1.66 vs 1.72 distinct experts per window); on the medium model Soft-Hard is slightly lower (1.59 vs 1.64, with a longer layer-1 run, 8.95 vs 5.54 tokens) at +1.3% perplexity; over seeds the two agree within noise (medium Soft-Hard 1.49 +/- 0.05 vs 1.54 +/- 0.02).

We concede three points, and have made these changes in the revised paper:

  • Framing. The sentence saying Soft-Hard "achieves the lowest switch rate of any method at only +0.4% perplexity" omitted the comparison to Soft lambda=0.5. It is replaced by the comparison above, and we removed the claim that the anchor curbs drift.
  • Soft is the main method. We present Soft as the main method and Soft-Hard as an optional extension whose gain over Soft we could not demonstrate, in the abstract, contributions and Section 5.
  • Untuned Soft-Hard. Soft-Hard fixes lambda=0.1 and W=4 and varies only alpha, while Soft is tuned over lambda, so the comparison is not tuned symmetrically. We did not run a joint sweep over (lambda, alpha, W) and state this in the paper.

2. No downstream evaluation (W2)

We agree that perplexity does not establish task-level behaviour. We did not run downstream benchmarks. For the WikiText-2 and TinyStories models (about 9M-169M parameters, trained on a small corpus) scores would be near chance and uninformative. The SlimPajama checkpoints (309M parameters) would be the right place to test, and we have not done so. The revised paper says explicitly that downstream task accuracy is untested for every model, and that the SlimPajama perplexity cost is the relevant quality-locality trade-off. While checking this, we re-evaluated the SlimPajama checkpoints on the full validation split (the earlier numbers used the first 50 batches): the cost is +2.6% at lambda=0.05 and +7.7% at lambda=0.5, against +2.9% and +7.5% before, so the conclusion is unchanged.

3. Combination with system-level optimizations (question)

We expect the method to be complementary to system-level approaches, because it changes only the routing distribution the system sees. We have not tested this, and the paper's transfer-time replay does not model any of it (Appendix D.2 states that there is no prefetching or compute/IO overlap).

AuthorsOpenReview ↗

Response (Part 2)

  • Better cache policies. Any policy sees more repeated accesses when routing is more local. We replayed the saved routing traces under LFU and under oracle (Belady) replacement, for top-1 and top-2 access (Appendix E). The gain persists but shrinks with a better policy: for Soft lambda=0.5 the top-2 miss reduction at C=2 goes from 2.10x (LRU) to 1.84x (oracle) on the small model, and from 2.49x to 2.23x on the medium model (LFU: 1.82x and 2.22x).
  • Prefetching and activation-aware offloading (MoE-Infinity, DuoServe). More predictable routing should make prefetch predictions more accurate. The two may overlap, though: part of the benefit we measure could already be recovered by prefetching, so the combined gain may be smaller than the sum. We did not simulate a prefetcher, because "prefetching with perfect lookahead" is not well defined as a miss count and a learned predictor is outside our scope.
  • CPU-side compute (Fiddler). This avoids transfers on misses, so locality matters less per miss, but it still reduces how many misses there are.
  • Quantization. Smaller experts let more of them fit in fast memory, which is independent of routing locality.

A limitation: all our measurements use per-sequence caches (batch size 1). With batched decoding, the set of experts in use is the union over sequences, and locality gains would shrink. The revised paper adds a paragraph on composition with these methods in the discussion, and states the per-sequence cache (no batching) limitation in the metrics description.

Summary of changes

  • Soft-Hard presented as an optional extension; multi-seed comparison and direct drift measurements added; the drift claim removed.
  • Downstream accuracy stated as untested; SlimPajama re-evaluated on the full validation split.
  • Replay under LFU and oracle replacement, top-2 accounting, and a discussion of composition with system-level methods.

We hope these changes address the reviewer's concerns about the Soft-Hard variant and the evaluation.

Thank you for the rebuttal, which partially addresses my concerns. However, the additional experiments still do not demonstrate the effectiveness of Soft-Hard variant over Soft, so I will keep my score unchanged.

Reviewer_LEaaOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes StickyMoE, a training-time loss that encourages adjacent tokens to use similar expert-routing distributions, with an optional window-level anchor loss. Experiments show lower routing-switch rates across several datasets, although language-modeling loss and deployment benefits depend on the setting.

Critical strengths
  1. The loss is intuitive and simple, and requires no inference-time architectural changes.
  2. Multi-seed experiments support the consistency of the routing-locality effect on the small models.
  3. The SlimPajama results provide useful evidence about the quality–locality trade-off beyond repeated training on small corpora.
Critical weaknesses
  1. The models use top-2 routing, but the cache statistics and hardware replay only track top-1 experts, leaving the actual expert-loading savings unclear.
  2. The reported 3.5–4.1× speedups measure transfer-trace replay; although Appendix D.2 explains this limitation, end-to-end decoding gains remain unverified.
  3. The simulated ReMoE and simplified Oracle-MoE baselines do not support broad superiority claims or the general conclusion that post-hoc adaptation cannot meaningfully improve locality.
  4. The cache setup is not clear enough, particularly whether the two expert slots are allocated per layer or globally, and what total device-memory budget this implies.
  5. SlimPajama shows a 2.9–7.5% increase in perplexity, but it remains unclear how much actual decoding speed this quality loss buys, or whether downstream performance is affected.
Questions for authors
  1. How much do cache misses and end-to-end decoding time improve when both selected experts are included in the actual layer-by-layer execution?
  2. Switch rate is treated as equivalent to cache-miss rate, even though switching to an expert already in the cache does not require another load.
  3. Are the two cache slots allocated per layer or shared globally, and how are expert accesses from different layers handled?
3 comments

Table and appendix numbers below refer to the revised paper.

Thank you for the careful review. Several of your concerns are valid, and we have addressed them in the revised paper, in some cases with new measurements. We think most of them concern how far our claims reach, which we could correct, and not the method or the routing-locality result. Below, "we concede" marks points where we agree.

1. Top-2 routing, top-1 cache accounting (W1, Q1)

We concede this. The models route to two experts per token, but SR, CHR and the transfer replay counted only the top-1 expert. We have now recomputed cache behaviour from the saved checkpoints with both selected experts accessed at every layer (new Table 15 in Appendix E). Misses are per token-layer (up to 2 for top-2 access), with a per-layer LRU cache and cold-start misses included. Ratios are baseline misses divided by method misses:

SettingTop-1 access, C=2Top-2 access, C=2Top-2 access, C=3Top-2, oracle replacement, C=2
Small, Soft lambda=0.53.54x2.10x3.07x1.84x
Medium, Soft lambda=0.53.91x2.49x4.36x2.23x

Over three seeds, top-2 misses at C=2 fall from 1.047 to 0.469 (small, lambda=0.5) and from 1.060 to 0.396 (medium, lambda=0.5). On SlimPajama (first 1,000 validation sequences) lambda=0.5 gives a 1.54x reduction, and on TinyStories 3.4x (three seeds).

So the locality gain carries over to the second expert, but the headline reduction is smaller at C=2. With two experts per token and two slots, a per-layer cache holds only the previous token's pair, so any change of pair costs a load. We now report the top-2 numbers next to the top-1 ones in the abstract and Section 5, and label the 3.92x figure as simulated top-1.

The transfer replay (Table 8) is still top-1 only, because it copies one expert-sized tensor per miss. We state this in the paper and do not claim an end-to-end result from it.

2. Cache setup (W4, Q3)

The simulator keeps a separate LRU cache per layer and per sequence. Each cache starts empty for a sequence and persists across its tokens. Expert accesses from different layers are never mixed, because each layer has its own expert weights. The budget is therefore 2 experts per layer out of N:

  • 2 of 4 experts (50%) for the small and medium models.
  • 2 of 8 experts (25%) for SlimPajama.

This is now stated in the metrics subsection of Section 5 and in Appendix E. Appendix E also adds:

  • Capacity C = 2 and 3 per layer.
  • A single global cache keyed by (layer, expert) at 25%, 50% and 75% of all experts.
  • LFU and oracle (Belady) replacement.

For these 4-layer, top-2 models, a global cache with 50% of all experts (8 slots) holds exactly the previous token's pairs, and it gives the same miss counts as the per-layer C=2 cache.

3. Switch rate versus miss rate (Q2)

We agree that SR is only a proxy, and we removed the sentence in Section 2.2 saying that reducing SR "is directly equivalent to" reducing latency. The revised text says SR upper-bounds the miss rate only for one slot with top-1 routing. The main claims cite cache misses, with SR as supporting evidence.

We also measured how far SR departs from a load count. On our models (excluding the Hard-Window ablation), 11-16% of token-layer steps change only the order of the selected pair, which costs no load but counts as a switch in SR. Conversely, changes of the second expert are invisible to top-1 SR. Appendix E therefore also reports the fraction of steps at which the unordered pair changes.

4. Latency numbers (W2)

We agree that the 3.5-4.1x figures measure host-to-device transfer time on a replayed top-1 trace. They are not end-to-end decoding time. We have:

  • Renamed the table "transfer latency" and replaced "faster" with "lower".
  • Added the caveat (no model compute, no second expert, no prefetch or compute/IO overlap) to the abstract, Section 6 and Appendix D.2.
  • Listed end-to-end decoding as untested in the limitations.

We could not build an offloaded decode engine within the rebuttal period, so end-to-end decoding latency and how much of the transfer saving survives compute/IO overlap remain untested. The revised paper lists this in the limitations.

5. Baselines (W3)

We concede that our ReMoE and Oracle-MoE are simplified reimplementations, not the original systems. In the revised paper we:

  • Removed "Pareto-dominating post-hoc..." and similar phrasing from the abstract and introduction.
  • Renamed the baselines "ReMoE-style" and "Oracle-style".
  • Narrowed the post-hoc conclusion to what we tested: router-only fine-tuning with frozen experts, lambda = 0.1, 0.1x the training steps. It did not move SR (0.705 vs 0.709, small; 0.709 vs 0.713, medium), and under top-2 access it changes misses by at most 1%. We do not claim that no post-hoc method can help.

The Oracle-style baseline is a competitive baseline: under top-2 access it reduces misses by 1.94x (small) and 2.16x (medium), at +5.1% and +1.4% PPL. Soft at lambda=0.5 reaches 2.10x (small) at +3.9% PPL, and Soft at lambda=0.2 reaches 1.85x at +1.9%.

We did not run a tuned post-hoc sweep (over lambda, steps and learning rate, or with unfrozen experts) in this period. The revised paper therefore states that a tuned post-hoc comparison remains open.

6. SlimPajama quality cost (W5)

The perplexity increase is real, and we report it as a limitation. While checking the evaluation we found that the SlimPajama perplexities in Table 14 came from the training-time evaluation on the first 50 validation batches (about 400 of 37k sequences), not the full split as the paper stated. We re-evaluated all three checkpoints on the full split (9.47M tokens). The cost is +2.6% at lambda=0.05 and +7.7% at lambda=0.5 (baseline 76.72, Soft 78.73 and 82.65), against +2.9% and +7.5% on the first 50 batches, so the conclusion is unchanged. At lambda=0.5, switch rate falls 44% (0.683 to 0.381 on the full split), and top-2 misses at C=2 fall 1.54x (2.0x with 4 slots per layer; first 1,000 validation sequences). The table now reports full-split PPL and SR, with first-50-batch PPL and CHR labelled as such (CHR was not recomputed on the full split).

Downstream task accuracy is untested for every model, and the paper states this explicitly. We also do not measure the decoding speed that this quality loss buys. At lambda = 0.05 the cost is +2.6% PPL on the full split for a much smaller locality gain (switch rate down 7%, top-2 misses at C=2 down 1.10x), so the trade-off is steep in this regime.

The near-zero or negative perplexity cost on the small and medium models does not extend to larger models, and the paper no longer suggests it does. A 168.5M-parameter model trained with a schedule that avoids overfitting costs +1.3% (lambda=0.05) and +5.0% (lambda=0.5), and SlimPajama costs +2.6% and +7.7%. The earlier 168.5M run, which appeared to improve perplexity, had overfit its small corpus, and we withdrew it.

Summary of changes

  • Top-2 cache accounting, global caches, LFU and oracle replacement, in a new Appendix E.
  • Explicit per-layer cache description and budget.
  • SR described as a proxy, with a measured rank-swap rate.
  • Latency results labeled as top-1 transfer time only.
  • Baseline claims narrowed.
  • Corrected the perplexity protocol statements for the main tables and SlimPajama.

We believe these changes address the soundness concerns, since they correct overstated claims and add the missing measurements. They leave the finding that a training-time consistency loss lowers routing switch rates across datasets and seeds as it is.

Thank you for the responses which have addressed some of my concerns, and I will raise my rating to 2.

Comments on the paper

1 comment
Area_Chair_5zYsOpenReview ↗

Please respond to the authors

Dear Reviewers,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.