acceptodds
Under review as a conference paper at ICLR 2027

Know Before You Route: Dual-Signal Expert Prefetching with Conformal Coverage Guarantees for Sparse MoE LLMs

Abstract

Sparse Mixture-of-Experts (MoE) LLMs achieve high capacity with sparse activation, but under GPU memory constraints the full expert pool may not fit on device, forcing expert-weight transfers from host memory onto the decode critical path. We introduce CEPA, a retraining-free dual-signal expert prefetching framework. An early velocity-corrected cross-layer proxy initiates one likely-expert transfer, while a later signal evaluated on the non-extrapolated layer-boundary residual constructs a calibrated expert set. The boundary residual remains a proxy for the post-attention router input, yet corrected lower-tail conformal calibration provides layerwise marginal whole-active-set coverage when calibration and test score–set pairs are exchangeable. CEPA speculates only on transfer timing: exact routing and expert computation are preserved by fetching every missing active expert on demand. Across four sparse MoE LLMs spanning 14.3B–35B total parameters under a 12 GB GPU memory budget, CEPA reduces decode latency by up to and retains a speedup against a matched CUDA-Graph baseline.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, expert prefetching, LLM serving, expert routing, conformal prediction, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FJWvOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a smart compute–communication overlap technique for MoE models. Latency gains are demonstrated across multi-concurrency settings across popular open-source models with compute/communication ablations done. While analysis and intuition is properly documented, this work can be further improved by demonstrating any potential gains/tradeoffs in higher batch (up to 128) settings that are realistic in many production inference deployments.

Edit: Raising score from 3 to 4 after concerns are addressed. Paper has a lot of value for practical latency-sensitive deployments.

Critical strengths
  1. Throughput gains are shown in realistic experimental scenarios against strong baselines.
Critical weaknesses
  1. Experiments show gains only up to batch size 8.
Questions for authors
  1. Can the authors extend the experiments beyond batch size 8 to evaluate whether the reported speedups persist at higher batch sizes (128)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer FJWv

We thank the reviewer for the positive assessment. We extended Table 10 to batch 128 on Qwen1.5-MoE (12 GB budget, caching disabled, 174 tokens per request; new rows are a mean of 3 runs):

BatchCEPA / JITDistinct experts per layer (of 60)
11.32×4.0
21.29×7.7
41.26×14.5
81.14×25.6
321.00×41.5
640.98×47.8
1281.01×51.2

The speedup declines smoothly to parity at batch ≥ 32; the large-batch values are within run-to-run variation.

Transfer volume sets this limit. At batch 128 the batch touches 51 of 60 experts per layer, more than any overlap window can hide. A deployment can disable prediction above a batch threshold and avoid CEPA's ~2% overhead there.

Batch 128 is also the edge of this memory regime. The KV cache already uses 4.4 GB of the 12 GB budget, and 512-token requests would need about 13 GB. CEPA targets latency-bound decoding on memory-constrained devices, where batch is typically below 8 and often 1. The revision adds the extended table to App. O.6 and states this scope.

Thank you for the transparent analysis for this. No improvement is to be expected as high batch since it's more compute-bound at that regime. This is consistent with a lot of practical-industry decode settings. Paper is still very useful for latency sensitve regimes of serving in practical settings.. Raising score from 3->4 since all concerns are addressed.

Thank you so much for the kind words and for raising your score! Your suggestion to look at larger batches was really helpful and made the paper's scope much clearer. The extended results are already in the revised PDF (App. O.6), and we will keep them in the final version. We truly appreciate your time and thoughtful feedback.

Reviewer_LzMZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CEPA, a dual-signal expert prefetching framework for sparse MoE models. It combines an early prediction mechanism with a conformal prediction-based mechanism to improve expert prefetching coverage. Importantly, CEPA does not modify the model's routing decisions or actual expert computation; prediction errors primarily result in unnecessary data transfers. Experiments demonstrate decoding speedups over just-in-time loading across several MoE models under constrained GPU memory.

Critical strengths
  1. The dual-signal design is interesting: early prediction provides additional prefetching lead time, while the conformal prediction mechanism aims to improve expert-set coverage.
  2. The framework preserves the original model computation and routing behavior, making it relatively practical to integrate into existing MoE inference systems.
  3. The paper evaluates multiple MoE models and memory settings, and includes ablation studies and system-level performance analysis.
Critical weaknesses
  1. The conformal coverage guarantee relies on exchangeability between calibration and test samples. However, router decisions across autoregressive decoding steps may exhibit temporal dependencies, making this assumption difficult to justify in realistic generation. The paper does not sufficiently evaluate whether the coverage guarantee remains reliable under such temporal dependencies, particularly at the sequence level.
  2. Theorem 4 guarantees that the active expert set is covered by the prediction set, but this does not ensure that all required experts are transferred to GPU before computation begins. Although the paper provides representative execution traces and a layer-level deadline-coverage example, it does not report aggregate deadline coverage or quantify transfer-induced stall time across workloads. Consequently, the extent to which the conformal coverage guarantee translates into reliable, timely prefetching under practical transfer and memory constraints remains insufficiently evaluated.
  3. In the Qwen1.5-MoE ablation, adding the early signal reduces latency from 107.0 to 81.4 ms/token, while adding the boundary signal further reduces it to 78.1 ms/token. This suggests that most of the performance gain comes from the early signal, while the additional latency improvement from the boundary signal is relatively modest compared with the additional mechanism it introduces.
Questions for authors

Please refer to the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer LzMZ

We thank the reviewer for separating set coverage from timely residency. We now measure both directly.

W1: Exchangeability under temporal dependence. Theorem 4 holds for exchangeable pairs, and pooled decode tokens are dependent. We ran the sequence-level control in App. G:

  • Rule fixed in advance. For each C4 document, we draw one token uniformly at random. A token is eligible only if every earlier token in its 512-token block comes from the same document, so draws from different documents are independent.
  • Same rule on held-out data. We calibrate EBE_B on one token from each of 1,083–1,109 C4-train documents and test on one token from each of 541–555 held-out C4-validation documents, over 2,000 draws.
  • Repeated splits. We also repeat random document-level calibration/test splits.

Whole-set coverage of EBE_B alone at α=0.1\alpha = 0.1 (mean over draws; 5th–95th percentile in brackets):

ModelPooled tokensOne token per document, train → validationDocument-level resplits
Qwen1.50.8930.900 [0.891, 0.908]0.900 [0.892, 0.909]
Qwen3.50.8880.902 [0.894, 0.909]0.900 [0.891, 0.909]
Gemma 40.8930.899 [0.892, 0.906]0.900 [0.892, 0.907]
GPT-OSS0.8900.899 [0.891, 0.906]0.900 [0.892, 0.909]

Calibrating on one token per document restores the target on all four models; per layer, coverage stays between 0.892 and 0.907. The pooled-token shortfall therefore comes from pooling dependent tokens, which Theorem 4 does not cover, not from the score or the threshold. The exchangeable protocol costs little: its set sizes are within 6.0% of the pooled-token sets.

Correctness and the speedups do not depend on coverage: a miss adds only an on-demand fetch, and every latency is measured on real dependent streams. For deployment, App. O.7 tracks miscoverage from exact routing and recalibrates. A formal guarantee for dependent tokens within one stream needs a cluster-aware or sequential conformal method; the revision states this scope in §4.4.

W2: From set coverage to timely residency. Theorem 4 covers the prediction set; residency depends on the transfer window. We now log the active experts resident when each MoE block starts, for every token and layer (12 GB budget; "cache" is leftover VRAM used as an expert cache):

ModelKCacheReady, JITReady, CEPAAdded by CEPA
Qwen1.54No / Yes0.00 / 1.681.70 / 2.86+1.70 / +1.18
GPT-OSS4No / Yes0.00 / 1.780.86 / 2.48+0.86 / +0.70
Gemma 48No / Yes0.00 / 4.424.57 / 6.68+4.57 / +2.26
Qwen3.58No / Yes0.00 / 4.066.91 / 7.63+6.91 / +3.57

"Ready" divided by K is the aggregate deadline recall. The remaining K − Ready active experts are the on-demand fetches that can stall the MoE block, and every reported latency includes them. CEPA fills whatever window each model provides:

  • Qwen1.5. About two transfers fit, and CEPA places 1.70 active experts in them.
  • GPT-OSS. Its large experts leave room for fewer.
  • Qwen3.5. Its small experts let most of the active set fit: 6.91 of 8 are ready.

The truncation log shows that window capacity, not coverage, limits residency: where the window is tight, 88–97% of the EBE_B candidates dropped by the budget were active (response to o8tT, W4). The revision adds both logs to §4.3 and App. O.3.

W3: Contribution of the boundary signal. Across all four models, EBE_B lowers latency by 2.0–11.1% over EAE_A alone and raises prefetch precision from 60–87% to 83–96% (response to o8tT, W1). Its gain is largest on Gemma 4 (11.1%), where EAE_A is least precise; Qwen1.5, the model in our original ablation, has the most precise EAE_A. EBE_B is also the only component that carries the calibrated coverage guarantee. It costs one router projection per layer (2.69 ms per token on Qwen1.5, §4.3), included in every CEPA latency.

Dear Reviewer,

Thank you again for your review. As we approach the end of the discussion period, we wanted to kindly follow up on our response and the revised manuscript.

We have made several changes to directly address your concerns and would appreciate your thoughts on whether they have been satisfactorily resolved. If so, we would be grateful if you could consider updating your assessment accordingly.

We are happy to clarify any remaining questions.

Reviewer_o8tTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a retraining-free method CEPA, to address the problem of expert weight prefetching in sparse MoE models. This dual-signal system utilizes 2 signals, E_A and E_B. The E_A is a early signal predicted by cross-layer velocity extrapolation to choose one expert for transmission, while E_B is a late signal that constructs a conformally calibrated set based on layer-entry residuals to supplement the prefetch candidates. The paper provides a mathematical proof of per-layer marginal coverage of the entire expert set and reports 1.10–1.37× decoding speedups across four models.

Critical strengths
  1. Expert weight prefetching in sparse MoE models is a critical problem for MoE models, especially for on-device LLMs. The paper combines an early coarse signal with a late precise signal to improve prefetching efficiency.
  2. The method CEPA is both retraining-free and post-training-free, which is efficient and deployment-friendly.
  3. The experiments of improvement is sufficient including matched CUDA Graph configurations, cache budgets, independent GPU experiments, batch-size experiments, and long-context experiments. These evidence supports the effectiveness of the method.
Critical weaknesses
  1. The incremental value of the core components has not been sufficiently isolated. In the experiment of Qwen1.5-MoE, E_A contributes 88.6%, most part of the total improvement, while E_B provides only a small additional benefit.
  2. It is not that reasonable to load less experts on early stages, since transmission time is more sufficient on early stages. It is also mentioned in the paper that there are truncations in the late signal stage E_B due to the limited time.
  3. There is a validation gap between theory and experiments. The theoretical derivation relies on the exchangeability assumption, but the tokens used in the experiments come from the same context and are therefore correlated. This introduces a risk that the exchangeability assumption may not hold, which the paper does not rule out.
  4. The candidate list is truncated at runtime. Therefore, even if the generated prediction set fully covers the true expert set, truncation may reduce the performance gains. Although the authors mention this issue, they do not provide relevant experiments or a detailed analysis.
  5. The method CEPA has a 1.35x speedup on Qwen1.5, but only a 1.10x on the newer model Qwen3.5. The decreasing gain suggests that the model may not be that useful for newer MoE models with better offloading.
Questions for authors
  1. Could the authors separately report the per-layer full-set coverage rates of E_B and the final union to verify which component contributes to the improvement?
  2. Is there any specific explaination for the lower speedup on the newer model Qwen3.5 and is there any evidence that the improvement will not vanish in the latest models?
  3. Is there any detailed experiments on how much the truncation of E_B affects final performance gain? Also, why not prefetch more experts at the earlier stage to balance the transfer workload?
3 comments
AuthorsOpenReview ↗

Response to Reviewer o8tT (1/2)

We thank the reviewer for the precise questions. We answer each below with new measurements; first, one clarification.

Clarification: EAE_A's transfers. EAE_A issues its ranked candidates until EBE_B fires: in the profiled layers, 1 transfer on Qwen1.5 and 2 on Qwen3.5 (Figs. 7c, 8b). All reported latencies used this runtime. Theorem 4 (EBE_B's set) and Prop. 5 (its union with EAE_A's top-1 candidate) are unchanged. The revision corrects the "one transfer" wording in the abstract, §1, §3, §4.4, App. D and App. P.

W1: Incremental value of EBE_B. Precision is the fraction of completed prefetches that were active. In EAE_A-only mode EBE_B never fires, so EAE_A issues its ranked candidates for the whole window. Mean of 3 runs:

ModelKPrecision, EAE_A-onlyPrecision, EAE_A+EBE_BLatency reduction from EBE_B
Qwen1.5-MoE487.2%95.9%4.7%
Qwen3.5-35B-A3B880.5%83.2%2.0%
GPT-OSS-20B472.8%87.5%6.1%
Gemma-4-26B-A4B860.4%82.7%11.1%

EBE_B helps most where EAE_A is least precise: Gemma 4 (11.1%) and GPT-OSS (6.1%). Qwen1.5, the model in Fig. 4, has the most precise EAE_A, so EBE_B adds least there (4.1% in that single run). Qwen3.5 gains least overall because its experts are small and little transfer latency remains to hide (W5). The signals divide the work: EAE_A buys lead time, and EBE_B supplies precision.

Q1: Whole-set coverage of EBE_B alone versus the union.

What the runtime calibrates. The deployed runtime calibrates EBE_B by CRC on a gate-weighted loss: one minus the router gate weight of the active experts inside the set. This loss is bounded and shrinks as the set grows, so CRC guarantees an expected gate-weighted miss of at most α\alpha for the sets we ran, under exchangeability. Table 2's union value is this quantity, which is why it sits at 0.898–0.905 on every model. A missed low-weight expert costs only its gate weight, so these sets are smaller than sets calibrated for whole-set containment. Theorem 4's whole-set guarantee applies to EBE_B recalibrated by Eq. (6). The revision states this in §3, the Table 2 caption and App. F.

ModelEBE_B alone, whole-set coverage (recalibrated by Eq. (6), App. F)Union, gate-weighted coverage (runtime, Table 2)
Qwen1.50.8930.900
Qwen3.50.8880.905
Gemma 40.8930.898
GPT-OSS0.8900.900

Pooled-token whole-set coverage of the recalibrated EBE_B is 0.888–0.893 on all four models, slightly below the 0.90 target (0.864–0.909 per layer, added to App. F). Pooled tokens from one prompt are dependent, so Theorem 4 does not cover them. With one token per document, as the theorem requires, EBE_B alone reaches the target: 0.899–0.902 (W3, in part 2/2).

GPT-OSS. The recalibrated EBE_B reaches 0.890 on pooled tokens and 0.899 under the one-token control, with mean set size 5.43. The runtime set targets the gate-weighted loss and is smaller (mean 3.50, below K = 4), so its whole-set coverage is at most 0.875 while its gate-weighted union coverage is 0.900. Outputs are unaffected, and the 1.28× speedup includes every resulting on-demand fetch.

Latency with recalibrated sets. Recalibration only adds lower-scored candidates, which EBE_B issues after all current ones in score order, so they cannot displace a current prefetch. Where the window is already full (without a cache, EBE_B drops 37.8% of its candidates on Qwen1.5 and 41.8% on GPT-OSS; W4), the added candidates would mostly be truncated. We therefore expect little latency change, but we did not re-measure it.

AuthorsOpenReview ↗

Response to Reviewer o8tT (2/2)

W2 / Q3b: Why not prefetch more experts early? Accuracy limits early prefetching, not time. The EAE_A-only runtime in W1 is exactly this policy: it fills the window with EAE_A's ranked candidates.

  • Those extra early candidates are often wrong. On Qwen1.5, EAE_A's top candidate is active 94% of the time but its K-th only 41% (Table 2), and EAE_A-only precision is 60–87% (W1).
  • Replacing them with EBE_B's candidates raises precision to 83–96% and lowers latency on every model (W1).
  • A calibrated set at EAE_A's firing time would need 16.8–18.5 experts for 90% coverage, versus 4.79 for EBE_B (Table 3).

EAE_A already uses the window it gets: 1 transfer on Qwen1.5 and 2 on Qwen3.5, whose experts are about 2.7× smaller.

W3: Exchangeability. Theorem 4 holds for exchangeable score–set pairs, and pooled autoregressive tokens are not exchangeable. We therefore ran the control in App. G:

For each C4 document, we draw one token uniformly at random by a rule fixed in advance, calibrate on one token per C4-train document and test on one token per held-out C4-validation document, and repeat over 2,000 draws and random document-level splits. EBE_B alone then reaches 0.899–0.902 whole-set coverage, against 0.888–0.893 with pooled tokens (details in our response to LzMZ, W1). The pooled-token shortfall therefore comes from pooling dependent tokens, which Theorem 4 does not cover, not from the score or the threshold.

CEPA's correctness and measured speedups do not depend on this assumption:

  • A coverage miss changes only timing. It turns a prefetch into an on-demand fetch, and CEPA and JIT produce bit-identical tokens.
  • Every latency is measured on real, dependent decode streams.

For deployment, App. O.7 monitors miscoverage from exact routing and recalibrates. A formal guarantee for dependent streams within one prompt needs a cluster-aware or sequential conformal method; the revision states this scope in §4.4.

W4 / Q3a: Effect of truncation. We now log truncation for every token and layer:

ModelCacheEBE_B candidates droppedDropped candidates that were active
Qwen1.5No / Yes37.8% / 12.9%87.5% / 88.0%
GPT-OSSNo / Yes41.8% / 12.0%96.8% / 97.0%
Gemma 4No / Yes21.5% / 3.3%60.2% / 47.2%
Qwen3.5No / Yes75.0% / 53.9%4.4% / 1.9%

Window capacity sets the cost of truncation:

  • Qwen1.5 and GPT-OSS. 88–97% of dropped candidates were active. EBE_B found them, but the window fits only about two transfers, so they are fetched on demand; this is what truncation costs. Consistent with this, a longer window raises the gain: on the clock-locked A6000, CEPA's gain rises from 27.0% to 29.9% (App. O.5).
  • Qwen3.5. Truncation removes mostly inactive candidates (4.4% active), because EBE_B issues candidates in score order, so it costs little.
  • With caching. The drop rate falls to 3–13% on Qwen1.5, GPT-OSS and Gemma 4.

W5 / Q2: Speedup on Qwen3.5. The Qwen3.5 gain is limited by how much transfer latency there is to hide, not by prediction. CEPA already has 6.91 of 8 active experts ready at MoE start, the most of any model (response to LzMZ, W2). Qwen3.5's experts are the smallest of the four (6.3 MB vs 17.3 MB for Qwen1.5), so routed-expert transfer is a smaller share of its token time. The gain tracks expert size relative to compute, not release date: Gemma 4 (2026) reaches 1.25× and GPT-OSS 1.28× (Fig. 3). Wherever the expert pool does not fit and transfer is exposed, CEPA has latency to hide.

Dear Reviewer,

Thank you again for your review. As we approach the end of the discussion period, we wanted to kindly follow up on our response and the revised manuscript.

We have made several changes to directly address your concerns and would appreciate your thoughts on whether they have been satisfactorily resolved. If so, we would be grateful if you could consider updating your assessment accordingly.

We are happy to clarify any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and corrections

We thank all reviewers. The revised PDF adds the requested measurements on all four models: an EAE_A-only vs EAE_A+EBE_B ablation, deadline residency and truncation logs (Table 4), a one-token-per-document exchangeability control (App. G), per-layer EBE_B coverage (App. F), and batch sizes up to 128 (App. O.6).

We also correct two descriptions in the submission. Neither changes any reported latency.

  1. EAE_A transfers. EAE_A issues its ranked candidates until EBE_B fires, not exactly one transfer (Figs. 7c, 8b).
  2. Runtime calibration. The deployed runtime calibrates EBE_B by CRC on a gate-weighted loss, which bounds the expected gate-weighted miss by α\alpha. Table 2's union coverage is this quantity, not whole-set containment. Theorem 4's whole-set guarantee applies to EBE_B recalibrated by Eq. (6); we report both in App. F and G. The revision states this in §3.4 and the Table 2 caption.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.