acceptodds
Under review as a conference paper at ICLR 2027

Temporal Mixture-of-Experts: Serving Memory that Scales with Active Parameters

Abstract

Sparse Mixture-of-Experts (MoE) models activate only a small fraction of their parameters per token, yet serving one still requires the whole expert pool in fast memory, making inference impractical on memory-constrained local devices. We introduce Temporal MoE, a new routing policy that reuses all but one of the previous token's experts for each layer. This allows us to keep only the active experts in RAM instead of the entire pool, and schedule loading the incoming expert from flash to overlap with computing the reused ones, effectively hiding transfer overhead. When pre-trained from scratch, Temporal MoE reduces serving memory by while preserving 72–82% of the standard MoE advantage over dense models in cross-entropy and downstream accuracy. This scaling trend holds consistently across compute-optimal budgets from to FLOPs. Probing routers demonstrate that the temporal routers focus more on context rather than token identity and spread each expert's use more evenly across the token stream. In real-device hardware benchmarks across an RTX A6000, an Intel laptop, and a Pixel 10a where the model otherwise does not fit, an 11B-scale model keeps 66–83% of the standard MoE's decode speed, and the workstation realizes the reduction. Applying this constraint post-hoc to released large instruct MoEs demonstrates remarkable robustness (holding especially well for sparser models). Moreover, just two GPU-hours of distillation recovers most of the accuracy lost under the constraint for Gemma4-26B and Qwen3.5-35B. These results show that Temporal MoE can move the quality–memory frontier for local inference by making expert reuse a core part of both training and serving.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, sparse MoE, on-device inference, memory-efficient inference, expert offloading, expert caching, routing locality, temporal locality, rolling residency, fine-grained MoE, scaling laws, isoFLOP, pretraining, LLM serving, edge deployment, mobile inference, llama.cpp, self-distillation, on-policy distillation, LoRA adaptation, quantization, router analysis, mechanistic analysis, memory-bandwidth bound decoding

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_rwgTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Temporal MoE, which allows at most one expert to change between consecutive tokens in each layer. When the number of resident experts equals the number of active experts (R=k), at most one new expert needs to be loaded per step. The method aims to overlap this loading with computation on the reused experts. Experiments show that Temporal MoE retains much of the quality advantage of standard MoE. The authors also evaluate hardware performance using a model with random weights and test the effects of routing constraints and distillation on released MoE models.

Critical strengths

When R=k, Temporal MoE guarantees at most one new expert load per token per layer. The model is trained with this constraint, making it suitable for memory-constrained inference. The authors compare it with dense and standard MoE models across different training budgets and expert granularities. They also run experiments with multiple random seeds and evaluate downstream tasks.

Critical weaknesses
  1. The hardware tests use a randomly initialized model rather than a trained Temporal MoE. Expert switching on the phone and laptop is also predefined. These results do not show that the same trained model can maintain quality, reduce memory use, and preserve inference speed.
  2. Overlapping expert loading with computation depends on the hardware and scheduling. The routing constraint alone cannot guarantee this overlap.
Questions for authors
  1. Can the authors evaluate a trained model on at least one device and report its quality, memory usage, and inference speed?
  2. Can the authors explain why the routing constraint would guarantee that expert loading overhead can be hidden? Does this depend on hardware bandwidth and scheduling?
2 comments

Thank you very much for taking the time to review, and for recognizing the memory constraints that motivate our temporal routing rule and how our experiments show its efficacy, especially in pretraining. We'd like to use this reply to clear up the weaknesses you raised, though we understand if you still hold some of them afterwards. In the revised PDF, Sections 2 and 5 now explain when the incoming load can be hidden, Section 5 now serves our trained 1.26B model before the 11B one, and Appendix D breaks down the timing on each device.

Weaknesses and questions

  • Random weights and predefined switching (Question 1). That is true. The serving runs use random weights and swap exactly one expert per layer on every token, which is the most the rule ever allows. So they test the hardware side, and they do not show trained quality. We have now run the on-device evaluation. It serves our pretrained 1.26B model on the A6000, the laptop and the phone with the trained router choosing the experts, and that router also swaps one expert per layer on every token. From 1k to 4k context, the A6000 averages 86% of its all-in-memory decode speed, against 79% for the 11B model (random weights). The laptop averages 95%, against 69%. The phone averages 76%, about the same as its 77% with the 11B model. Section 5 now leads with this model and treats the 11B model as the scale-up (Figure 6, Appendix D).

  • Whether the rule guarantees hidden loads (Question 2). You are right that the routing rule alone does not guarantee it, and that it depends on both bandwidth and scheduling. The method needs all of them together, and it is not one or the other. The rule guarantees that each layer loads at most one new expert per token. We then choose the expert size from the hardware's bandwidth ratio, so that this one load is small enough to fit behind the computation on the experts already in memory. The kernel schedules the load to run during that computation. Unifying the routing rule with the bandwidth and the scheduling is the contribution.

Thank you for your rebuttal, as well as for conducting the additional experiments. Your responses have satisfactorily addressed my concerns.

Reviewer_BZ2aOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a new expert routing policy titled "Temporal MoE", which replaces at most 1 expert in the active expert set for each new token. The paper motivates this by showing that, with sufficiently small experts, loading one expert from slow memory to fast memory takes less time than transferring the remaining experts from fast memory to compute core. The authors perform a pre-training analysis on rather small models (up to 49M params) and rather limited compute (up to 10 EFLOPs), based on FLAME-MoE, with expert sizes shrunk (and # active experts proportionally increased) to ensure slow memory transfer can be hidden. The authors train compute-matched optimally-sized dense models and show the 76-83% of the benefit of sparsely-activated MoEs over their compute-matched dense counterparts is maintained. The authors then apply their method on open-weight MoEs (Gamma4 and Qwen3.5-35B-A3B) and show their methods gives decent results on standard benchmarks, which can be improved by short on-policy distillation. Lastly, the authors compare to Skliar et al. which biases experts towards reuse, and show their method performs similarly to Skliar et al.'s method.

Critical strengths

Evaluation on real models add strong evidence of the benefits. The comparisons are useful and fair.

The method is interesting and well motivated: by ensuring slow memory expert loads can always be hidden behind fast memory transfer of remaining experts, the method does not suffer from occasional slowdowns even when average expert load per token is comparable (which is the case in Skliar et al.)

The pre-training analysis, while fairly limited both in model sizes and compute budget, is interesting and provides initial motivation for further research.

Critical weaknesses
  1. The method offers little novelty: that most experts can be reused even if the expert router initially chooses other experts is not new (Skliar et al.) and the insight that limited slow memory data transfer can be hidden behind fast memory data transfer isn't either.

  2. The compute and parameter budgets in the pre-training analysis are very limited and it is unclear whether they will scale to realistic parameter budgets as no scaling analysis is provided.

  3. The analysis in the paragraph starting at line 372 is hard to follow. Please plot the trade-offs prsented here for easier understanding.

  4. Comparison to Skliar: did the author's use the fine-tuned models or the original ones? Using the fine-tuned models (as the "Temporal MoE" point appears to do, judging from the location of the marker) would be unfair as Skliar does not use fine-tuning to adapt to forced expert re-use.

  5. Comparison to Skliar: The benefits of the authors' method over Skliar are actually not entirely clear, apart from one point which may not be a fair comparison (see previous weakness). Please provide more details and analysis of the comparison.

  6. While the point that Skliar occasionally requires additional token loads is valid and relevant, the comparison made in Figure 8(c) is not useful: authors should provide a comparison of accuracy at matched token rates or token rates at matched accuracy. Average expert loads per token is in itself not a useful E2E metric.

  7. The related work section is full of made-up jargon and difficult to understand.

[Minor] Line 136: apparent Latex compile error Line 190: reference to CoSMoE is missing Plots are hard to read, please increase font size.

Questions for authors
  1. 155: does 'coarse temporal model' refer to the FLAME default?
  2. 264: "Temporal MoE thus trains broader experts that substitute better despite computing less similar functions". Isn't this a contradiction?
  3. 293: the GPUs used are not very large or fast compared to modern offerings. How would the token generation rates be affected on more modern GPUs?
  4. 296: What is meant by "transfers"?
  5. 372: How is the method applied during prefill and are there any positive effects there?
  6. 401: "2 GPU-hours": on which GPU?
2 comments

Thank you for your careful review. We are glad you found our method "interesting and well motivated", with expert loads that can always be hidden so decoding never suffers from occasional slowdowns. We also appreciate that you found our evaluation "strong evidence of the benefits" and the comparisons "useful and fair." Beyond addressing your concerns in the revised PDF, we highlight that our largest models have 1.26B parameters and that we have now served that trained model on all three devices, which shows its strength as a pretraining method.

Weaknesses

  1. Novelty. We agree that expert reuse and overlapping expert loads with computation each appear in prior work. However, we introduce a new routing policy that is simple, efficient, and an improvement for pretraining. It turns reuse into a routing rule with a hard bound: each layer brings in at most one new expert per token. We then choose expert granularity from the hardware's bandwidth ratio, so that this one load can overlap with computation on the experts already in memory. This consistent policy makes training simple (no aux losses, hyperparameters, or major model changes). Every auxiliary loss we tried lost quality and still averaged more than one load per token. For example, CoSMoEs' loss still averages 7.3 loads per token-layer at strengths that leave test BPB about 22% worse than our Temporal MoE (Appendix B).

  2. Scale. Section 3 now states our model sizes more clearly. The 49M figure you cited is an active parameter count, without embeddings, at the 101810^{18} FLOP budget. Our largest models, trained with 101910^{19} FLOPs, have 265M active and 1.26B total parameters including embeddings, trained on 9B tokens. Measured against the strongest standard MoE at each budget from 101610^{16} to 101910^{19} FLOPs, the fine-grained model keeps 61%, 75%, 78% and 81% of the MoE-over-dense gain, so the share seems to grow with compute. Figure 9 shows this.

  3. Thinking analysis. That paragraph compares how much accuracy the constraint costs with thinking on versus off. Thinking changes the loss by at most two points, and not in one direction. As you suggested, we now plot these trade-offs in Figure 15 (Appendix E).

4-6. Comparison with Skliar et al. Skliar et al. bias a frozen router at inference time and target caches that hold 50% to 75% of the experts. We adapt our method to released models, achieving caches that hold 10% or less: 6.25% for Gemma4 and 3.1% for Qwen3.5. Sorry that our figure was unclear here, the point labeled "standard MoE, hard 1-swap cap" is our base released model under the Temporal MoE policy, with no fine-tuning, and it is the fair point to compare with their training-free method. Accuracy is mixed: our base models score 78.8 on GSM8K for Gemma4 against 80.7 for Skliar et al., and 76.6 for Qwen3.5 against 74.5. At the Skliar setting that averages 0.94 loads per token-layer, 20% of token-layers still need two or more loads, and each extra load stalls decoding at that layer. Our rule never needs more than one. We relabeled that point "Temporal MoE, base" in Figure 8 and Section 7. On token rates, we agree. However, these 26B and 35B models do not fit in the RAM of our phone or laptop, so we cannot serve them there. Our main contributions are a simple rule with a hard bound, that it scales to pretraining, and that it cuts memory up to 10x on released models.

  1. Related work. We rewrote Section 8 in plainer language.
  • Minor points. We fixed the compile error, the CoSMoEs citation and the plot fonts.

Questions

  1. Coarse model. Yes. Coarse is the FLAME default, 6 of 64 routed experts per layer, and fine splits each expert into three, for 18 of 192.

  2. Substitution. It is not a contradiction, though we agree the sentence reads like one. One would expect experts that substitute well to compute similar functions. Replacing an active expert with a random one costs Temporal MoE 18% less than a standard MoE, yet its experts' outputs on the same inputs are 29% to 53% less alike. Each expert is more broadly useful without the experts becoming redundant copies of one another.

  3. GPU choice. Our target is batch-1 inference on edge devices such as phones and laptops, where the full expert pool does not fit in fast memory. We used the A6000 as a batch-1 testbed with a reasonable bandwidth ratio, not as a deployment target.

  4. Transfers. A transfer is the movement of a single expert's weights from slow memory into fast memory when the router swaps that expert in.

  5. Prefill. On the released models, we do not apply the constraint during prefill. The prefill kernel processes the prompt expert by expert, so each expert the prompt needs is loaded once, computes all of its tokens, and is then released. This is really a kernel trick. But our pretrained models are trained under the temporal rule on every token, so they follow it during prefill as well.

  6. One H200, we made section 7 more explicit.

I thank the reviewers for the time they took to write the rebuttal. However, several important concerns (limited novelty, marginal accuracy improvements over Skliar et al., lack of token rates in head-to-head comparison) have not been sufficiently addressed and I maintain my rating

Reviewer_jXz5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Temporal MoE, a routing rule under which each MoE layer may change at most one of its k active experts between consecutive tokens. The best-scoring non-resident expert replaces the worst-scoring resident one only if it scores higher. The authors argue this transfer is hidden whenever k−1 exceeds the fast/slow bandwidth ratio, which motivates fine-grained experts.

The paper claimed a few empirical contributions: 1. FLAME-MoE-based models trained from 1e16 to 1e19 FLOPs at two granularities keep 72–82% of the MoE-over-dense BPB gain under the constraint, with similar recovery on zero-shot tasks. 2. Probes suggest the constrained router depends less on token identity and spreads expert usage more evenly. Its expert weights also quantize better. Serving. An ~11B random-weight, Qwen3-MoE-shaped model runs in a llama.cpp fork on an A6000, a Pixel 10a and a laptop. It keeps 66–83% of decode speed with roughly 5× less fast memory.

Critical strengths
  1. The idea is simple. A hard cap of one transfer per layer per token gives a worst-case bound on I/O, which caches can’t offer. And under the cache-conditional baseline, about 20% of token-layers need two or more transfers, with a tail out to seven. That tail is exactly what hurts latency on device.
  2. The bandwidth condition k−1 ≥ BW_fast/BW_slow is a clean design rule tying expert granularity to the memory hierarchy. This seems to be a novel design in MoE.
  3. This paper includes evaluations with different parameter settings and serving conditions.
Critical weaknesses
  1. Writing is poor. This is a major concern. Authors should have explained more about why this design can work and what their novelties are in the introduction. And the methods section is very short. The majority of this paper is about evaluation under different conditions and parameters. There is less explanation about the thoughts behind those designations.

  2. No single experiment shows the full pipeline. The models trained under the constraint are tiny: 0.77M–49M active parameters at the swept budgets, and the 1e19 model size isn’t stated. The served models have random weights. On the phone and laptop, the turnover is prescribed rather than produced by a router, which the authors themselves call “the open experiment.” The released models route only 4 or 8 experts. App. E admits this is too few to satisfy the bandwidth condition, so §6–7 measure accuracy under a masking simulation, not actual serving. Each piece is reasonable on its own. But the central claim, that a model trained under the constraint is both good and fast with active-only memory, is pieced together from three disjoint experiments. A single mid-sized model (even ~1B total parameters) pretrained under the constraint and served on the phone or laptop would make the paper much stronger.

  3. The bandwidth condition isn’t actually met in the paper’s own setups. §2 says k−1+6 = 23 falls “within the 15-to-100 bandwidth ratios.” But the condition requires k−1 to exceed the ratio, so 23 only covers ratios up to 23. App. D gives the A6000’s VRAM-to-PCIe ratio as 24–31 and concedes the 17 reused experts can’t hide the copy on their own. The phone’s DRAM-to-UFS ratio is never reported. Whether the shared expert counts is inconsistent: §2 gives 23, App. D gives 17. The timing model also ignores a dependency. The incoming expert must finish loading before it can be computed, so the critical path is roughly max(load, compute of k−1) plus compute of one. I’d like to see a per-layer timing breakdown on each device showing how much of the transfer is really hidden. The 0.88× → 0.83× drop on the A6000 and the 70–83% figures on the phone suggest it’s mostly, but not fully, hidden.

  4. Presentation needs a pass. The paper is dense, and some passages read more like lab notes than a paper. Specific issues: The unrendered “BB-fold for a batch of BB sequences” in §2. A sentence fragment in the intro: “Imposed at decode time on released large instruct MoEs, ...”. “Falls from 3.15–0.80” and “context AUC moves only from 0.640–0.697” are unclear about from what to what. “The fine Temporal MoE is 97% of its fine standard MoE counterpart” doesn’t say 97% of what. A stray quotation mark in Fig. 8’s caption. “TemporalMoE” vs “Temporal MoE” naming. The laptop results are marked provisional “pending the Windows Defender exclusion record.” That shouldn’t be in a submitted version; either finalize the numbers or drop them. Fig. 7 is very hard to read. App. F could be cut roughly in half without losing anything important.

Questions for authors
  1. What is the novelty of your work compared to previous methods? Can you explain it clearly?
  2. How large is the 1e19 compute-optimal model in active and total parameters, and how many tokens was it trained on?
  3. How are gate weights computed over S_t? Is it a softmax over resident scores only, renormalized? Is the load-balancing loss computed on the full router distribution or on the resident set?
3 comments

Thank you for your detailed review. We are glad you found the idea simple in how it draws on the bandwidth condition to give a clean design rule, and we appreciate your recognition of the range of settings and serving conditions we evaluated. You are right that one trained model was missing from end to end, so we have now served our trained 1.26B model on all three devices, which shows quality, memory and speed together.

Weaknesses

  1. Writing and explanation. The introduction now says what the routing rule adds beyond caches and Skliar et al. (see Novelty under Questions). Section 2 explains how the gate weights and losses are computed and why we evict the lowest-scoring expert, with the comparison in Appendix B. Section 8 now uses plainer language, and the detailed thinking analysis moved to Appendix E.

  2. The full pipeline. We liked your suggestion to show quality, memory and speed together, so we added serving experiments on all three devices with our pretrained 1.26B Temporal MoE model. Its weights in memory shrink 4.7x. From 1k to 4k context, the A6000 averages 86% of its all-in-memory decode speed, against 79% for the 11B model (random weights). The laptop averages 95%, against 69%. The phone averages 76%, about the same as its 77% with the 11B model. Section 5 now leads with this model and treats the 11B model as the scale-up (Figure 6, Appendix D). We also understated our pretraining scale in the submission. The 0.77M to 49M figures are active parameters without embeddings. Our largest 1e19 FLOP models have about 265M active parameters with embeddings and 1.26B to 1.27B in total. We updated Section 3 and Figure 3 to show this.

  3. The bandwidth condition. The values 23 and 17 refer to different architectures, which the submission did not distinguish clearly enough. In the pretraining models the shared expert adds to the resident computation, which covers bandwidth ratios up to 23. The 11B serving model has no shared expert, so its threshold is 17. We now state the dependency you pointed out. The incoming expert has to finish loading before it computes, so its own computation cannot hide its load. Sections 2 and 5 now keep these two models apart.

    We also added the per-device timing you asked for in Appendix D. On the A6000, our kernel adds 0.72 ms per token and streaming another 0.44 ms, so 67% of the transfer time is hidden. On the laptop our kernel adds 6.9 ms and NVMe reads 11.3 ms, with 61% hidden. On the Pixel 10a our kernel adds 0.6 ms and flash reads 16.3 ms. Its DRAM reads 11.4 GB/s and its flash about 0.8 GB/s in our kernel, a ratio of about 14. So the transfers are mostly but not fully hidden, as you suspected.

  4. Presentation. We fixed every item you listed. The 97% claim now gives the six-task averages, Figure 7 is full width, figure fonts are larger, and Appendix F is half as long.

Questions

  1. Novelty. Our contribution is a new simple routing rule that brings in at most one new expert per layer per token, with expert granularity chosen from the hardware's bandwidth ratio. As you noted in the strengths, this turns the bandwidth condition into a design rule for the model. Critically, the rule is simple enough to train from scratch, with no extra locality loss or hyperparameters to tune. Every auxiliary loss we tried lost quality and still averaged more than one load per token. CoSMoEs' loss, for example, still averages 7.3 loads per token-layer at strengths that leave test CE loss about 22% worse than our Temporal MoE at the same budget (Appendix B). Skliar et al. instead bias a frozen router at inference time and target caches that hold 50% to 75% of the experts. Their method does not carry over cleanly to pretraining the way ours does. We also push our rule much further, keeping 9.4% of routed experts resident in our pretrained models and only the active experts in released models, 6.25% for Gemma4 and 3.1% for Qwen3.5. As you noted in the strengths, Skliar et al.'s cache still needs two or more loads on about 20% of token-layers, with a tail out to seven, while our rule never needs more than one.

  2. Model size. At 101910^{19} FLOPs our models have about 265M active and 1.26B to 1.27B total parameters (Section 3, Figure 3), trained on 9B tokens.

  3. Gate weights and losses. The rule only replaces the top-k selection, so it works with whatever order a model uses for softmax, renormalization and top-k. All those operations are order preserving, so it doesn't matter for our min-logit rule.

    • For the first part, take Flame-MoE for example, we train it applying the softmax over all experts and then take the top-k. But GPT-OSS does the opposite, taking the top-k on the raw router logits and applying the softmax only over the selected experts.
    • For the second part, it even worked well with load balancing loss from Flame-MoE, this is computed though over the currently resident set because our routing rule is simply a replacement for the top-k.

Thank you for your response. I appreciate your time on writing this paper and the response. But I still have a few questions:

  1. You mention 'The introduction now says', does this mean that the pdf uploaded has been updated or it is only updated on your side?

  2. Thanks for providing a full pipeline result. But do you have a comparison with SOTA methods or other similar methods if SOTA is not available? It will be hard to distinguish your advantage if there is no proper comparison.

Thank you.

Thank you for the follow-up. We're happy to clarify both points, please let us know if you have any further concerns!

  1. Revised PDF. Yes, the PDF on OpenReview is the revised version, so every change we described in our first response is in the file you can download now.

  2. Comparison with other methods. We agree that a direct comparison makes our advantage easier to judge, so we added one for each stage of the pipeline.

    Pretraining (Appendix B). Temporal MoE is a pretraining method, so the closest alternatives are losses that push a standard MoE to reuse experts while it trains. We trained two of them at our 101810^{18} FLOP budget with the same model shape, data and compute as our own runs. The first is CoSMoEs' block selection loss, and the second is the consistency loss of Sticky Routing (Kayyam, 2026), which penalizes changes in routing between consecutive tokens. CoSMoEs reaches 1.409 test BPB and still needs 9.4 loads per layer per token, and Sticky Routing reaches 1.361 with 10.8. Our Temporal MoE reaches 1.335 with exactly one load, against 1.346 for the standard MoE. Needing about ten loads per layer per token means more than half of the 18 active experts change at every token, so these models cannot be served with only their active experts in memory, which is the setting we target.

    Serving (Appendix D). For the full pipeline, the most direct method for these non-bounded setups is Skliar et al., which needs no training. We applied it to the standard MoE we trained at 101910^{19} FLOPs and compared it with our trained 1.26B Temporal MoE model, both with 18 experts per layer in memory and an average of one load per layer per token. There are two axes to look at jointly, pretraining and post-training. Against Skliar et al. applied post-training, our pretrained model is clearly better, since it loses 0.014 nats of test CE loss where Skliar et al. loses 0.44. When we also apply our rule post-training, we get about the same as Skliar et al., 0.49. As for speed, we focus on the A6000 because we have the most complete kernel control there, and for this comparison both methods run through the same fused kernels. These numbers are newly run on more generally optimized kernels, which is why they are faster than the ones our paper reports, and they give a truer comparison that is limited more by memory and compute. On the standard 1.26B MoE our rule decodes at 974.7 tok/s with only its active experts in memory, against 929.7 for Skliar et al., which is 4.8% faster, and our trained model decodes at 988.4. At the MoE layers alone Skliar et al. is 13% slower, almost all of it from its heavier routing step. On our 11B model the end to end gap grows to 5.4% (257.4 against 244.1 tok/s) and Skliar et al. is 9.7% slower at the MoE layers, about half from its routing step and half from its load pattern. The end to end numbers are smaller than the MoE layer numbers because attention and scheduling are the same for both methods and make up about 60% of each token at 1.26B and 43% at 11B. The 11B model has random weights, so there we run the routing step of Skliar et al. on every token and impose the load pattern that it produced on the 1.26B model.

Reviewer_8n61OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal MoE, a routing policy that replaces at most one active expert per layer between consecutive tokens, enabling expert weights to be streamed while previously resident experts are computed. The paper studies the constraint during pretraining, measures memory and decoding throughput on three hardware platforms, and applies it post hoc to six released instruct MoEs. Additional on-policy distillation recovers part of the accuracy lost under tight residency budgets.

Critical strengths
  1. The method provides a concrete serving guarantee rather than relying on average expert reuse. Replacing at most one expert per token and layer directly bounds flash transfers and makes active-only expert residency possible.
  2. The pretraining study is systematic. It compares dense, standard MoE, and Temporal MoE models at matched compute across several budgets and two expert granularities, retaining 72–82% of the MoE-over-dense improvement at the reported compute-optimal points.
  3. The paper includes real device measurements, not only simulations. Table 1 reports 165 versus 200 tokens/s on the RTX A6000 while reducing VRAM from 7,672 to 1,502 MiB in the fine-grained configuration, and the laptop and phone measurements illustrate the deployment opportunity.
  4. The extension to released instruct models strengthens the practical relevance. The six-model evaluation, two-GPU-hour self-distillation study, and comparison with cache-conditional routing provide useful information about accuracy–memory tradeoffs beyond the models trained from scratch.
Critical weaknesses
  1. The quality and serving results are not measured on the same trained models. The pretraining study uses comparatively small models, while Section 5 benchmarks roughly 11.3B-parameter MoE-shaped models with random weights. This validates the hardware mechanism but not the end-to-end quality–latency tradeoff for a working 11B model.
  2. The post-hoc constraint does not preserve quality uniformly. Section 6 reports mean accuracy drops of 11.6 points for OLMoE, 7.5 for LFM2.5, 5.9 for Gemma4, and 4.2 for Qwen3.5 in the non-thinking setting at R = k. Distillation is tested on only two models, leaving the general recovery claim uncertain.
  3. The mobile speed comparisons are not always against a fully resident version of the same model. On the phone, the full expert bank does not fit, so the reported relative speed uses the largest partial expert pool that fits as the reference. This should be distinguished from the workstation's true all-resident ceiling when interpreting speed retention.
  4. The evaluation focuses on single-sequence decoding. Section 5 notes that each concurrent sequence needs its own resident set, and long-context prefill is slower under the proposed implementation. The resulting throughput and memory tradeoff for practical batching or mixed prefill/decode workloads remains unclear.
Questions for authors
  1. Could the authors benchmark a pretrained or adapted non-random MoE with the temporal constraint on the serving hardware, reporting quality, peak memory, and end-to-end throughput for the same model?
  2. What determines which released models suffer large accuracy losses under R = k, and can the two-GPU-hour distillation procedure reliably recover performance for OLMoE and LFM2.5 as well?
  3. Could the authors clarify the reference model used for mobile speed-retention ratios and report absolute latency and memory for the relevant alternatives on each device?
  4. How does the advantage change at batch sizes above one or in realistic workloads that mix long-context prefill with decoding?
1 comment

Thank you very much for taking the time to review, and for recognizing the work that went into the paper. We'd like to use this reply to clear up the weaknesses you raised, though we understand if you still hold some of them afterwards. In the revised PDF, Section 3 and Figure 3 now give the model sizes with embeddings, Section 5 now serves our trained 1.26B model before the 11B one, and Appendix D breaks down the timing on each device.

Weaknesses and questions

  • Quality and serving on different models (Question 1). That is true, however, the pretraining models are not as small as the submission made them look. We originally reported only active parameters without embeddings, because we were focused on the core parameters. Our largest models, trained with 101910^{19} FLOPs, have about 265M active parameters with embeddings and 1.26B in total, trained on 9B tokens. That is closer to the 11B serving model, but it is still not the same model. To close the gap we have now served our pretrained 1.26B model on the A6000, the laptop and the phone under its trained router. From 1k to 4k context, the A6000 averages 86% of its all-in-memory decode speed, against 79% for the 11B model (random weights). The laptop averages 95%, against 69%. The phone averages 76%, about the same as its 77% with the 11B model. Section 5 now leads with this model and treats the 11B model as the scale-up (Figure 6, Appendix D).

  • Accuracy loss on released models (Question 2). You are right that the constraint hurts some of these models a lot. OLMoE loses 11.6 points and LFM2.5 loses 7.5, against 5.9 for Gemma4 and 4.2 for Qwen3.5. OLMoE in particular is an older architecture. It is what we have as an open research standard, but it is a weaker model to begin with. Across the models we tested, the stronger the model, the less it seems to lose under the constraint. On distillation, we only tested two models with two GPU-hours each, so we do not claim that it recovers OLMoE or LFM2.5. Doing distillation really well would take significantly more compute than the 2 H200 Hours we were able to use. We would ask you to read those runs as a sign that Temporal MoE has potential on existing models. Its main use is as a pretraining method, where the model learns under the rule from the start.

  • Phone speed reference (Question 3). That is a good observation. The full 11B model does not fit in the phone's RAM, so the reference keeps the same expert shape with the largest number of experts that fits, all in RAM. We did this to have a consistent speed baseline across devices. If anything it favors the standard MoE, because it makes that model look easier to serve than it is. On this phone the full model could not run from RAM at all. In absolute terms the phone decodes at 17.80 tokens/s at 1k context and 12.16 at 4k, against 25.44 and 14.58 for the reference, and its expert memory is 10.7x smaller than the full expert bank. The laptop takes 53.56 ms per token against 35.34 ms with the full model in RAM, and its working set falls from 5,968 to 1,066 MiB. Appendix D gives the other context lengths and breaks down where the time goes on each device.

  • Batch size and prefill (Question 4). This is a fundamental limit of the technique. Each concurrent sequence needs its own resident set, so the memory needed grows with the number of sequences. Our target is local use on phones and laptops, which runs one sequence at a time, and there we think the trade-off is a reasonable one. The slower prefill comes from our kernel and not from the routing rule. We could run normal prefill here, but it would use far too much peak memory, so our kernel loads one expert at a time instead. A deployment can use either one, so we see this as a balance between memory and prefill speed more than a drawback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.