acceptodds
Under review as a conference paper at ICLR 2027

A Trainable Memory-Movement Objective for Mixture-of-Experts Decoding

Abstract

Expert-weight transfers can bottleneck batch-one decoding of offloaded mixture-of-experts models. We introduce a differentiable surrogate for expert-weight traffic based on capacity-constrained soft residency, with memoryless and stateful variants. We evaluate learned routing workloads using least-recently-used (LRU) and offline-optimal (Belady) cache misses. On 64-expert models with 3.6–6.8B parameters, training reduces LRU traffic by 14–16× when 25% of expert blocks fit in fast memory, with a 7–13% relative increase in perplexity. A trace-driven H100 replay of the 6.8B model at 12.5% capacity yields 63% higher throughput with blocking transfers and 90% under ideal overlap. Comparisons against a switch-only objective reveal strong dependence on load balancing and cache capacity. At a load-balancing coefficient of 0.01, the memoryless configuration achieves 61% fewer LRU misses at matched mean perplexity on the 414M model with 25% capacity, but not at 12.5%. An analysis of marginal concentration and temporal reuse helps explain these reversals. Increasing the retrofit learning rate substantially narrows the apparent advantage of applying the objective during pretraining, although residual differences remain at matched total compute. These results show how traffic-aware training, deployment budgets, and optimization protocols jointly shape MoE cacheability.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture of experts, expert offloading, memory-bound decoding, expert caching, load balancing loss, training objectives, Belady optimal caching, systems for machine learning, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nNgnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a trainable proxy objective for expert-weight movement in single-sequence, expert-offloaded MoE decoding. The objective reshapes routing through a capacity-constrained soft top-M residency operator, with a memoryless form based on batch-averaged routing probabilities and a stateful form using an exponential moving average of past routing probabilities. Experiments cover OLMoE-architecture models ranging from 62M to 6.8B parameters. The authors evaluate freshly recorded routing traces using LRU miss counts and the offline-optimal Belady miss count, which provides a lower bound for each fixed routing workload. At a fast-tier capacity holding 25% of expert weights, the reported simulated LRU misses decrease by approximately 14–16× for the 64-expert, 3.6B–6.8B models, accompanied by a 7–13% relative perplexity increase. In a separate H100 trace replay at 12.5% capacity, the 6.8B model achieves throughput improvements of 63% with blocking transfers and 90% with overlapped transfers.

Critical strengths
  1. The paper explicitly incorporates cache capacity and estimated data movement into router training, aligning the objective with deployment settings where only a subset of expert weights fits in fast memory. At 25% cache capacity, the method achieves fewer cache misses than an adjacent-token routing consistency objective at comparable language-model quality, providing empirical support for capacity-aware optimization.
  2. The paper investigates both incorporating the objective during pretraining and adapting an existing model, while examining learning rate, training budget, and load balancing. The finding that a higher adaptation learning rate substantially narrows the gap between these approaches provides useful insight into how optimization schedules affect learned locality and informs the adaptation of existing models.
Critical weaknesses
  1. The comparisons focus primarily on loss components and switch-only controls. These are useful for understanding the mechanism, but direct evidence of gains over established router fine-tuning methods remains limited. ReMoE (ICML 2026) is a relevant comparator because it also fine-tunes routing to improve expert reuse under memory-constrained expert offloading. Moreover, the difference from the switch-only control is not statistically significant on the released pretrained model, leaving the relative advantage in practical adaptation settings less firmly established.
Questions for authors
  1. Does the stateful variant retain its advantage over the memoryless variant when total capacity, capacity allocation across layers, and perplexity are matched? Could a targeted control help distinguish the benefit of history modeling from that of capacity allocation?
  2. What additional capabilities does the capacity-constrained objective provide relative to the ReMoE method? Under the same released model, adaptation data and training budget, deployment capacity, and quality constraint, what differences would the authors expect or observe?
  3. The relative performance of the objectives changes with cache capacity. For a model deployed across multiple memory configurations, do the results support a training configuration that works well across capacities, or is separate adaptation preferable? How should the training capacity budget be chosen relative to the deployment capacity?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer nNgn

Thank you for the review. We added the requested ReMoE comparison and controls for capacity and pooling.

Weakness: direct comparisons with ReMoE and switch-only.

We reimplemented the ReMoE locality block with the published term weights and tested six strengths from scratch at 414M/64 experts and a = 0.01 (Appendix A.1). These compare loss terms, not complete systems; each strength is one run.

At strength 8, ReMoE reaches 61.78 PPL versus 62.00 for memoryless + switch. It has 1.24× the latter's LRU misses at 25% capacity but 0.48× at 12.5%, and improves on switch-only λ = 1.0 at both capacities (Table 2(d)).

On released OLMoE-1B-7B, we use the same 2M adaptation tokens, router-only training, and worst-seed quality constraint of ΔPPL ≤ 0.3 relative to the λ = 0 fine-tuning reference. ReMoE scale 2 achieves fractional simulated-cache-cost reduction of 0.138 ± 0.020 over ten seeds versus 0.116 ± 0.064 over twenty for memoryless adaptation (Table 12). Mean PPL costs are +0.204 and +0.179, respectively; Welch p = 0.18. Paired comparisons also show no significant memoryless advantage over ReMoE scale 2 or switch-only. We do not claim superiority on the released model.

Q1. Does the stateful advantage survive matched pooling and capacity?

Yes, at the tested 25% capacity. Both variants use a pooled 160-block budget, a = 0.01, no switch term, and three seeds (Table 2(c)). Stateful λ = 2.0 reaches 61.34 PPL versus 61.62 for pooled memoryless λ = 1.0, with 6.0× fewer LRU misses. The larger pooled budget alone therefore does not explain the advantage.

At 12.5%, the same stateful variant has 0.87× the pooled-memoryless misses but 1.69× the switch-only control's misses. The reduction appears in the static component, not improved κ (Section 5.2), so we do not interpret it as evidence of better temporal reuse in the measured traces.

Q2. What does explicit capacity add beyond ReMoE?

Eq. 2 enforces a specified residency budget; the reimplemented ReMoE locality terms do not take capacity as an input. This is a structural distinction, not a guarantee of a better trade-off.

For the stateful form at λ = 4.0, reducing the training budget from 160 to 80 blocks cuts LRU misses at the 80-block deployment capacity by 4.4×, at 1.40 higher PPL; the 80-block result is one run (Tables 1–2).

Against ReMoE strength 8, stateful λ = 2.0 has 0.56× the misses at 25% capacity and 0.44 lower PPL, but 3.23× the misses at 12.5%. The evidence is therefore capacity-dependent, not uniform superiority.

Q3. One configuration across capacities, and budget selection.

The stateful 160-block variant is strong at 25% and above but performs poorly at smaller capacities. The 80-block variant improves the smaller-capacity trade-off while producing more misses at 25% (Appendix E, Figure 3). This motivates capacity-specific tuning, not a conclusion that separate training is always necessary.

For memoryless adaptation on the released model, training budgets of 4 and 8 blocks per layer are not statistically separated at any tested deployment capacity (Welch p ≥ 0.67; Table 13). Budget 16 costs more at capacity 256 but is not separated from the others at 64 or 128. Thus, neither matching training and deployment budgets nor always choosing the smallest budget is established by these experiments.

Appendix E recommends selecting budget and strength on held-out traces at the intended deployment capacity. We did not perform that selection protocol and state this explicitly.

I appreciate the authors' rebuttal; some of my concerns have been addressed. My remaining comments are as follows:

  1. The response indicates that most of the gains of the stateful form comes from the static part (i.e., focusing routing on a subset of experts), instead of from temporal reuse of recently loaded experts (κ\kappa). Since the static number is calculated per chunk, it can also be reduced simply by making routing more concentrated within each sequence, which does not necessarily need token order. Could the authors give more evidence that the stateful form truly benefits from order of past routing, instead of mostly from within-sequence concentration?

  2. The appendix indicates that perplexity slightly increases on reasoning and coding text after adaptation. Perplexity might not capture performance on downstream generation tasks well; it would therefore be useful to check this more directly. The current zero-shot evaluations are evaluated by option likelihood, not generation, so a small difference in routing would not be accumulated across multiple decoding steps. For the released model, could the authors report results on generation-based reasoning and coding tasks, compared with the reference router?

I thank the authors again for their time and effort on the rebuttal. I will maintain my current rating.

AuthorsOpenReview ↗

Author Response to Reviewer nNgn [1/2]

Thank you for the follow-up. We ran three checks and added them to the revised manuscript (Appendix G.1, Table 18).

Does the stateful form benefit from the order of past routing?

We cannot show that it does. The checks point to concentration, as you suggested.

Shuffled traces. We permuted the token order within each recorded chunk, keeping each token's accesses together, and replayed LRU (five permutations per chunk, three seeds). At 25% capacity, stateful λ = 2.0 goes from 46,145 to 60,471 misses and pooled memoryless λ = 1.0 from 275,008 to 289,007. Static does not change. The stateful form has 6.0× fewer misses before shuffling and 4.8× after. Shuffling adds about 14,000 misses to both, so the smaller ratio is not evidence of an order effect.

Fixed pin chosen on other chunks. We split the 24 evaluation chunks into two halves and served each half with the 160 blocks most frequent in the other half. The stateful form has 27,004 misses versus 149,486 for pooled memoryless (5.5×). Per-chunk pins give 21,993 versus 142,862 (6.5×). The two stateful pins share 156–159 of their 160 blocks. On the stateful traces, the fixed pin also has fewer misses than LRU (27,004 versus 46,145). The concentration is shared across chunks and does not depend on reusing recently loaded experts.

Whole-sequence-average control. We trained a control that replaces the causal EMA with the average routing probabilities of each training sequence. Model, data, token budget and the pooled 160-block budget are unchanged. Three-seed means at 25% capacity:

VariantPPLLRU missesStaticκ
Pooled memoryless, λ = 1.061.62275,008142,8621.92
Whole-sequence average, λ = 2.062.00173,16174,1112.34
Whole-sequence average, λ = 4.062.18129,47857,7522.24
Stateful, λ = 2.061.3446,14521,9932.10

Stateful λ = 2.0 has 3.8× and 2.8× fewer misses than the two controls. Each of the six control runs has 2.3–4.9× the stateful mean misses, at higher PPL. The PPL gaps (0.66 and 0.83) are outside the 0.5 band we set before the runs, so these are not matched-perplexity comparisons. The difference is in Static (2.6–3.4× lower), not κ, whose seed ranges overlap. At 12.5% capacity the controls have only 1.13× the stateful misses.

What this shows. Within-sequence concentration explains part of the reduction: the controls have 1.6× and 2.1× fewer misses than pooled memoryless, at 0.39 and 0.56 higher PPL. It does not explain all of it at 25% capacity. The control also differs from the stateful form in history length, recency weighting and access to later tokens, so it does not isolate order, and we did not establish why the two differ. Section 5.2 now states this limitation.

AuthorsOpenReview ↗

Author Response to Reviewer nNgn [2/2]

Generation-based reasoning and coding tasks on the released model.

We ran GSM8K, HumanEval and MBPP with greedy decoding (Appendix G.2, Table 19). The fine-tuned routers lose some accuracy against the released router. The traffic term adds no detectable loss beyond λ = 0 fine-tuning.

Setup. lm-evaluation-harness 0.4.12, default settings: GSM8K with five examples and with eight worked examples (1,319 problems), HumanEval (164, zero-shot) and MBPP test (500, three-shot). Only the router weights differ. We scored the released router, ten λ = 0 fine-tuning seeds, 18 memoryless λ = 0.15 seeds (the weights of seeds 0 and 1 were not saved) and ten ReMoE scale-2 seeds. Intervals are 95% bootstrap intervals over problems and router seeds (10,000 resamples).

Scores are percentages (exact match for GSM8K, pass@1 for code). Differences are in points.

TaskReleasedλ = 0MemorylessReMoEMemoryless − λ = 0Memoryless − released
GSM8K 5-shot, strict8.198.077.838.44−0.24 [−1.02, +0.54]−0.36 [−1.60, +0.87]
GSM8K 5-shot, flexible8.728.518.218.99−0.29 [−1.07, +0.48]−0.51 [−1.79, +0.71]
GSM8K 8-shot, strict7.135.855.285.56−0.56 [−1.35, +0.21]−1.84 [−2.99, −0.75]
GSM8K 8-shot, flexible12.8912.7712.4012.60−0.37 [−1.27, +0.49]−0.48 [−1.87, +0.87]
HumanEval12.8012.7412.6413.35−0.11 [−1.36, +1.05]−0.17 [−2.74, +2.30]
MBPP test18.4016.8816.7716.48−0.11 [−1.50, +1.34]−1.63 [−3.87, +0.54]

Against the released router. All three fine-tuned groups are lower on eight-shot GSM8K under the strict filter, with intervals that exclude zero. Under the flexible filter none is separated from the released router. The result depends on the answer filter, but we cannot say the strict-filter drop is only a formatting effect. On MBPP test all three means are lower, with intervals that include zero. HumanEval intervals are wide. The released router scores below 10% on five-shot GSM8K and on the eight-shot strict filter, so those rows say little about retained reasoning ability.

Against λ = 0 fine-tuning. No interval excludes zero on any task. This is not evidence of equality.

Predictions written in advance. The memoryless mean should be at most 2.0 points below the seed-0 λ = 0 reference on five-shot strict GSM8K and on HumanEval. Both held (−0.13 and +0.44).

Larger code pool. Before any generation score existed, we fixed a pool of 1,128 problems: HumanEval and the MBPP test, train and validation splits, equally weighted. On this pool, memoryless minus λ = 0 is +0.08 [−0.86, +1.00]. Against the released router the differences are −1.57 [−3.10, −0.07] for memoryless, −1.65 [−3.24, −0.07] for λ = 0 and −1.36 [−3.00, +0.26] for ReMoE. Memoryless minus ReMoE is −0.21 [−1.18, +0.75]. We do not claim an advantage over ReMoE.

What this shows. The losses appear with and without the traffic term, so they come from router fine-tuning under a WikiText perplexity constraint, not from the term. Section 5.4 reports the results and Section 6 now states that this constraint does not preserve generation scores. Appendix G.2 has the output-validity checks and a post-hoc experiment with a larger KL weight.

Thank you again for the follow-up questions; both checks improved the paper.

Reviewer_KfdFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to address an important problem for batch-size one inference of MoEs on devices with memory constraint. The total parameters of MoE models may not fit entirely on the small fast tier memory and therefore only a subset of experts can be cached. In case, a router in the MoE model selects an expert that is not in the cache, it needs to be fetched from the slower tier memory such as flash storage incurring a cache miss. Current server-scale MoEs are trained with load balancing loss and show poor sequence level expert selection consistency which results in poor cache hit-rates. This paper aims to train MoE models with a new regularization that trains routers to be more temporally consistent. The authors test their method on OLMoE architecture and report improved cache residency.

Critical strengths
  • The authors are tackling an interesting problem and recognize the importance of temporal consistency.

  • The authors are very honest about the unintended wrong λ=1\lambda = 1 coefficient for load balancing loss which obviously undermines the quality of those trainings.

Critical weaknesses
  • The paper is exceptionally difficult to read and understand. The entire paper is full of sentences that are a paragraph long and written in a cryptic compressed language. The abstract and intro are full of undefined terms like "retrofit", "library default", "born floor", "born model", "arms". The keyword "born" already appears in the results part of abstract without any prior defining. The paper uses non-ML terminologies such as Dose. Abstract and contributions are overloaded with detailed experimental results and numbers and it is not clear what the actual contribution is. They paper needs a complete rewrite of every section.

    • For many keywords in the paper, I had to use an LLM to translate what the authors are saying: A machine learning paper at ICLR should use standard ML terminology ("benchmark suite", "regularization weight", "model variants", "from-scratch pretraining", "fine-tuning"), not clinical jargon ("battery", "dose", "arms", "born", "retrofit").
  • The authors mention that the primary motivation of this work is to accelerate batch-size-one inference. For this purpose, the residency penalty should be calculated locally within a single sequence, not the entire batch. The memoryless batch-marginal form is computing expert residency across the full batch of unrelated documents. This explains why their memoryless model's traffic reduction is largely invariant to token shuffling and does not work as a true temporal caching strategy. What was the motivation for memoryless batch-marginal form when target is batch-size-one inference?

  • The authors candidly admit in Section 2 that their stateful formulation is essentially MELINOE with a sum-constrained sigmoid threshold. Yet, in pretraining (Table 1), the MELINOE-like stateful formulation beats their new memoryless formulation by >2X fewer misses and >2 points of perplexity.

  • The models are extremely under-trained: 0.3 tokens per parameter means the models are not converged. A 7–13% relative increase in perplexity is objectively massive and the model is considered unusable. Comparatively, in related works, any perplexity degradation above 1% to 2% is considered a noticeable quality loss.

  • The authors do not compare their method to any existing related work.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer KfdF

Thank you for the review. We rewrote the main text and clarified the contribution.

  1. Readability and terminology.

We replaced the project-specific vocabulary with standard terms, including “from-scratch,” “fine-tuned,” “variant,” and “strength.” The abstract and introduction now state three empirical findings directly. Table 2 collects pairwise comparisons with their perplexity differences and seed counts, rather than leaving these qualifications scattered through the prose.

  1. Why use a batch average for batch-one decoding?

The batch average is a training statistic, not an inference-time batching requirement. Your underlying criticism is correct: memoryless residency cannot represent temporal cache state. Section 3.1 now describes it as shaping a static working set.

This can support static placement: a pin selected on the first half of a trace gives 0.54–0.87× the misses of per-chunk LRU on the second half for the memoryless and stateful variants (Appendix E). The stateful form instead uses a per-sequence causal EMA of preceding router probabilities.

  1. Stateful residency and MELINOE.

Section 2 identifies MELINOE as close prior work. Our from-scratch stateful form averages soft router probabilities and applies a sum-constrained soft top-M; MELINOE uses decayed hard top-k requests with capacity rescaling. Our MELINOE reimplementation is unvalidated, so we claim no superiority over it.

At 414M/64 experts and a = 0.01, the new pooled-memoryless control matches the stateful form's 160-block budget. Stateful λ = 2.0 has 6.0× fewer LRU misses at 0.28 lower PPL at 25% capacity (three seeds each; Table 2(c)). This is conditional: it trails switch-only at 12.5%, and the released-model experiment shows no significant stateful advantage.

  1. Training maturity and quality cost.

The 3.6B and 6.8B models remain short-trained (Section 6). At 1.3B, extending training from 2B to 6B tokens at a = 0.01 improves memoryless + switch from 3.36× fewer misses at +9.6% PPL to 4.49× at +2.9% (25% capacity; Appendix C.3). These are single runs, not evidence that the mature-model concern is resolved.

Switch-only reaches 3.76× at +1.5% PPL in the longer run, and interpolated matched-perplexity comparisons put the two within 3% at 25%. We therefore do not establish a residency-term advantage under a tight quality budget. Section 5.1 also reports that a fully resident 16-expert model dominates every tested 64-expert variant above its perplexity at that capacity.

  1. Baselines and contribution.

We added the published squared-L2 StickyMoE loss form and reimplemented ReMoE locality terms. On the released model, under the same data, token budget, and quality constraint, ReMoE's fractional simulated-cost reduction is 0.138 ± 0.020 versus 0.116 ± 0.064 for memoryless adaptation; the difference is not significant (Welch p = 0.18; Table 12).

The contribution is an empirical study: matching the fine-tuning learning rate closes most of the pretraining gap; capacity and load balancing can reverse objective rankings; and memoryless and stateful residency behave differently. We do not claim that traffic regularization itself is new or that our formulation universally outperforms alternatives. The AI Use Statement specifies the assistance used; the authors remain responsible for the paper.

I would like to thank the authors for their response. I checked out the revised manuscript and I confirm that intro reads a bit better now. However, I believe the paper needs significant more work to meet the bar of standard ML conferences. While the contributions are mainly empirical, they are not significant and the messaging is heavily convoluted.

Unfortunately, I cannot recommend the acceptance of the paper and maintain my score.

AuthorsOpenReview ↗

Author Response to Reviewer KfdF

Thank you for taking the time to read the revised manuscript and our response. We appreciate your feedback and the time you have devoted to reviewing our work.

Reviewer_2aDHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds a soft-residency memory-traffic proxy to MoE training. Usage is estimated from either the batch marginal of router gate mass or a causal EMA of past gates. A sum-constrained sigmoid constructs soft residency with capacity M, and gate mass outside that residency is penalized. Bisection enforces the forward capacity constraint, while the threshold and scale are frozen during differentiation. The authors evaluate LRU, static pinning, and offline-optimal Belady misses on newly recorded post-training traces and compare applying the objective during pretraining ("born") with continued training of a finished model ("retrofit"). Load-balance coefficients, cache capacities, and learning rates alter the ordering of objectives and protocols; a higher retrofit learning rate largely closes the original born advantage. Fine-grained pretrained replicas show large traffic reductions, whereas router-only adaptation of released OLMoE yields approximately 11% cost reduction without a significant advantage over switch-only training. Single-run H100 trace replays examine how traffic changes translate into time savings.

Critical strengths
  1. The mechanism analysis goes beyond reporting cache hit rates. Joint measurement of LRU, Belady, and Static, together with LRU = Static × κ, separates marginal concentration from temporal effects. The reversal against switch-only training across balance coefficients and capacities, the capacity sensitivity of stateful training, and footprint-induced thrashing are supported by experiments or conditional analysis. This makes the findings interpretable beyond a single favorable eviction policy.

  2. The investigation of training-protocol confounds is scientifically useful. The authors disclose the unintended balance coefficient and different baseline activation frequency, then provide clean-coefficient replications. They also examine higher-learning-rate retrofits, zero-pressure continued training, and born runs matched for total compute. Table 2 shows that the large original born–retrofit gap largely disappears at the higher learning rate, with only limited, sparsely replicated differences at matched compute. Revising the initial favorable interpretation and reporting nonsignificant released-model comparisons strengthen the paper.

Critical weaknesses
  1. The strongest traffic claims are not established for sufficiently trained models against strong alternatives at matched quality. The 3.6B/6.8B replicas receive only 2B training tokens; for the 6.8B model this is approximately 0.3 tokens per parameter. Small changes in average zero-shot accuracy do not establish preservation of mature-model capabilities. The order-of-magnitude reductions mainly come from the scaling sweep with coefficient confounds; the clean replications are valuable but do not cover the full scale ladder. On released OLMoE, memoryless gain is 0.116±0.064 versus 0.105±0.079 for the best valid switch-only setting, with no significant advantage in Table 9. The strong stateful born result also lacks a direct comparison with a complete implementation of the closest cache-aware training method. The strongest supported contribution is therefore a set of conditional mechanism findings, rather than a generally superior objective delivering order-of-magnitude improvements on mature models.

  2. The connection from the proxy objective to deployment benefits remains narrow. Gate mass versus actual accesses, the frozen-threshold surrogate gradient, per-layer versus pooled capacities, and cold-start simulation conventions all affect the mapping between the objective and system cost. H100 replay is useful evidence, but each measurement is a single run on recorded traces; the overlapped result is an upper bound and does not include the expert-parallel load costs of concentrated routing. The current evidence does not establish comparable quality–latency benefits during continuous generation, with longer cache histories, or under other device placements. This limits the strength of deployment recommendations drawn from trace-miss reductions.

Questions for authors
  1. Mathematical interpretation: Does Proposition 2 concern an EMA of access indicators or the implemented EMA of gate mass? For the latter, please express the limit in terms of p̄ or state the additional assumptions permitting substitution of ā. In a representative setting, does differentiating implicitly through the threshold and scale materially change the quality–traffic trade-off relative to the frozen-threshold surrogate?
  2. Independent contribution of statefulness: The strongest stateful born results use pooled capacity across layers, whereas the main memoryless runs use fixed per-layer budgets. Could these be compared with matched pooling, training budget, and quality, together with one direct comparison to the closest cache-aware baseline? The Mℓ=16 control does not fully separate temporal information from flexible layer capacity.
  3. Released-model benefit: Under Table 9's common quality guard, the interval for the difference between memoryless and the best valid switch-only setting remains wide. Are paired results available using identical trace lengths and cost definitions, together with out-of-domain text excluded from recipe selection? These would clarify whether the recommendation for memoryless adaptation rests on gains, stability, or implementation cost.
  4. Actual runtime: Could one representative setting include real autoregressive offloaded generation and repeated timing, with persistent caching compared against resets every 1,024 tokens? Please also clarify which trace segments or adapted artifacts explain the difference between the longer 25%-capacity replay's 0.80→0.63 GB/token and Table 11's 0.803→0.454 GB/token, so that their respective throughput claims can be reproduced.
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer 2aDH

Thank you for the detailed review. We added controls and revised the claims as follows.

W1. Training maturity and matched-quality comparisons.

At 1.3B/32 experts, extending training from 2B to 6B tokens at a = 0.01 improves the LRU reduction at 25% capacity from 3.36× at +9.6% PPL to 4.49× at +2.9% (Appendix C.3). At matched perplexity, however, memoryless + switch is within 3% of the interpolated switch-only count at 25%, and switch-only is ahead at 12.5%. These are single runs, not mature-model evidence; the 3.6B and 6.8B models remain short-trained.

We added ReMoE loss-term comparisons, not full-system reproductions. Our MELINOE reimplementation is unvalidated, so we draw no conclusion about it. Matching the fine-tuning learning rate removes most of the apparent pretraining advantage (Section 5.3). Equal-total-token comparisons are not equal compute: measured training throughput falls by 11–29% (Appendix A.1).

Q1. Gate-mass EMA and exact gradients.

Proposition 2 now gives the untruncated soft-gate EMA's limit as the gate-mass marginal under i.i.d. tokens as W → ∞ with t/W → ∞; Appendix B separately bounds the finite-tap approximation. Proposition 1's static-miss bound applies to empirical access frequencies, not the gate mass used in training.

Exact differentiation through both threshold and scale materially changes the trade-off (Appendix D, Table 15). At 414M and a = 0.01, exact-gradient λ = 0.4 reaches 60.86 PPL versus 62.00 for stop-gradient λ = 0.1, with 0.99× its LRU misses at 25% capacity and 0.51× at 12.5% (three seeds each). Against switch-only λ = 1.0, it has 0.54× and 0.56× the misses, respectively, at 1.52 lower PPL. The capacity reversal therefore concerns the tested stop-gradient comparisons, not the objective generally.

Q2. Statefulness with matched pooling and capacity.

Both variants use the same pooled 160-block budget, a = 0.01, no switch term, and three seeds (Tables 1–2). At 25% capacity, stateful λ = 2.0 gives 46,145 misses at 61.34 PPL versus 275,008 at 61.62 PPL for pooled memoryless λ = 1.0: 6.0× fewer misses at 0.28 lower PPL. Pooling alone does not explain the result. The reduction is in the static component, not improved κ.

At 12.5%, this stateful variant has 1.69× the switch-only control's misses. Changing the stateful budget from 160 to 80 blocks at λ = 4.0 cuts misses at 12.5% by 4.4×, at 1.40 higher PPL; the 80-block result is one run.

Q3. Released-model comparisons and generalization.

Paired over seeds 0–9, memoryless minus switch-only is −0.008 (95% interval [−0.072, 0.056], p = 0.79), and memoryless minus ReMoE is −0.040 ([−0.089, 0.008], p = 0.09), in fractional simulated-cache-cost reduction (Table 12). Neither establishes superiority or equivalence. On GSM8K and HumanEval teacher-forced traces, adaptation changes misses by only 0–1% and raises PPL by 3–4% (Appendix F). We do not recommend memoryless adaptation over these alternatives.

W2/Q4. Deployment controls and timing.

Persistent caching preserves the reported pairwise ordering (Appendix E). In the separate λ = 4 stateful comparison, per-layer caches reduce the advantage over pooled memoryless from about 25× to 6×; this is not the quality-matched λ = 2 comparison above. Expert-parallel communication and load-imbalance costs remain unmeasured.

Table 14 adds real batch-one greedy generation with blocking transfers. At 6.8B and 12.5% capacity, throughput rises from 13.6 to 21.8 tokens/s (+60.6%) with cache resets every 1,024 tokens, over five prompts timed once each. Overlapped replay is labeled an upper bound. Generation timing was not repeated per prompt, and generated text was not retained, so output repetition cannot be separated from routing effects.

Released-model timings use a search-selected seed. The replay discrepancy is documented: 0.801→0.633 GB/token is measured after a 512-token warm-up; 0.803→0.454 includes the initial window, where the adapted trace has unusually few misses. We did not establish why (Table 14 and accompanying notes).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final summary of findings and revisions

We really thank the reviewers for their careful reading and constructive feedback. We have uploaded the revised manuscript and incorporated the additional experiments reported in our responses, including the latest controls on routing history and generation-based evaluations.

The paper’s central contribution is a controlled empirical study of how training choices affect the trade-off between model quality and expert-weight traffic in offloaded MoE decoding. Three findings organize the revised paper:

  1. Matching the fine-tuning learning rate closes most of the apparent from-scratch advantage. In the controlled 414M and 520M comparisons, much of the earlier gap disappears when fine-tuning uses the pretraining learning rate. This identifies a consequential confound when comparing training protocols (Section 5.3).

  2. Objective rankings depend on cache capacity and load balancing. Comparisons with switch-only training and reimplemented ReMoE locality terms show capacity-dependent trade-offs. The added exact-gradient experiments also show that the gradient approximation can materially change these comparisons (Sections 5.1–5.2; Appendix D).

  3. The strong stateful result is primarily associated with concentration. With pooling and capacity matched, the stateful form has 6.0× fewer LRU misses at 0.28 lower mean perplexity at 25% capacity, over three seeds per variant. The new fixed-pin check supports concentration shared across evaluation chunks. The whole-sequence-average controls do not reproduce the stateful result at the two replicated strengths, but these experiments do not establish a distinct benefit from the order of past routing (Section 5.2; Appendix G.1).

The added GSM8K, HumanEval and MBPP generation evaluations show that the WikiText perplexity constraint did not ensure preservation of generation scores. Some losses occur relative to the released router, including with λ = 0 fine-tuning. We detect no statistically significant additional degradation relative to λ = 0 under these protocols; this does not establish equality (Section 5.4; Appendix G.2).

We also rewrote the exposition, clarified the theoretical statements, and added baseline comparisons and real autoregressive offloaded-generation measurements. The larger from-scratch models remain short-trained, and the released-model comparisons do not establish superiority over ReMoE or switch-only training.

We appreciate the reviewers’ questions, which helped us strengthen the controls and make both the findings and their limits clearer in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.