acceptodds
Withdrawn from ICLR 2027

VerifyLocal: Learning Expert-Residency-Conditioned Proposals for Exact MoE Speculative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Exact speculative decoding fixes the returned target distribution but permits its auxiliary proposal to use observable deployment state. In expert-offloaded mixture-of-experts (MoE) inference, plausible speculative histories can incur different expert-weight transfers. VerifyLocal conditions proposal probabilities on pre-round expert residency while keeping the target, its router, and exact correction unchanged. The Stage-A screen reduces logical miss bytes by 30% (95% CI [24%, 36%]) while retaining 98.5% expected commitment. In the controlled physical evaluation, expert-transfer bytes fall from 84.0 to 63.0 MB per committed token (25%; 95% CI [20%, 30%] reduction), and committed-token goodput rises from 19.5 to 23.0 tokens/s (18%; 95% CI [13%, 23%] increase). Inter-token latency falls by 18% and time to first token by 10%; observed task quality differs by -0.1 percentage points (95% CI [-0.3, +0.1]). Goodput exceeds a tuned nonlearned residency-aware cost tilt by 8% (95% CI [+4%, +12%]). Physical-transfer reductions replicate on OLMoE-1B-7B (27%, 95% CI [21%, 33%]), Mixtral-8x7B (24%, [17%, 30%]), and DeepSeekMoE-16B (22%, [15%, 29%]). The benefit diminishes toward zero with fully resident experts, delimiting the movement-limited regime.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture of experts, speculative decoding, expert residency, expert offloading, hardware-aware inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_59T9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VerifyLocal, a proposal learning method for exact speculative decoding with expert-offloaded mixture-of-experts models. It addresses expert-weight transfers triggered by candidate verification, including transfers incurred by subsequently rejected tokens. The method adds a trainable low-rank logit residual to a frozen dense drafter, conditioning the proposal distribution on expert residency observed before each round and optimizing committed-token goodput. The target model, router, and standard speculative correction remain fixed; the paper specifies conditions under which the target output distribution is preserved despite persistent residency updates across rounds. The evaluation separately examines opportunities under logical cache accounting, physical weight transfers, and end-to-end performance. In the primary experiment with 500 prompts, five seeds, and a cache holding 25% of expert weights on an NVIDIA L4, the authors report a 25% reduction in physical expert-transfer bytes per committed token and an 18% increase in goodput. VerifyLocal also achieves 8% higher goodput than a tuned nonlearned residency-aware cost-tilt baseline. Experiments on OLMoE, Mixtral, and DeepSeekMoE report transfer reductions of 22–27%.

Critical strengths

VerifyLocal uses a lightweight low-rank module to adapt the draft distribution to expert-weight residency while keeping the target model and standard sampling correction fixed. This design exploits differences in verification cost across candidate sequences, allowing proposals to account for deployment state while preserving the target distribution under the stated correctness conditions.

Critical weaknesses

The method specifies a throughput-oriented learning objective, but the description does not sufficiently establish whether training uses measured latency or a validated cost proxy, or how such a proxy is validated. This limits the assessment of which runtime costs the learned policy actually optimizes and how strongly it depends on the hardware and execution environment.

Questions for authors
  1. If the nonlearned baseline’s penalty strength is selected directly by end-to-end throughput under the same tuning budget and commitment constraint, does VerifyLocal retain its additional gain? Can the existing parameter sweep show which configurations the two selection criteria would choose?
  2. Does shifting probability mass toward "resident experts" potentially reduce output diversity or cause the model to fall into repetitive loops over multiple decoding rounds?
  3. To what extent do the marginal gains of VerifyLocal diminish in environments equipped with sophisticated hardware prefetching mechanisms? In multi-batch inference scenarios where different requests compete for cached experts, how does VerifyLocal balance the global expert residency state?
Reviewer_8Vo5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies speculative decoding when the verifier model is an MoE model. The challenge in this scenario is the fact that verifying a long sequence requires fetching MoE experts even for the rejected tokens, which means the fetched experts will not be effectively used. The proposal is to make the drafter model aware of the experts that are currently in Cache, and propose draft sequences that are likely to request the same cache-resident experts when the verifier model computes the sequence.

Critical strengths

The target problem is important, speculative decoding is a crucial method to speedup token generation in LLM inference. MoE models are becoming more and more popular. Optimization of speculative decoding for MoE verifiers will have actual impact on the AI technology and token generation cost.

Critical weaknesses
  • Abstract is not clear: (1) there are abbreviations that are not defined: CI, ITL, TTFT, (2) Stage-A is suddenly mentioned in the text without prior introduction, (3) some out of context expressions such as "Goodput", "cost tilt", etc., (4) instead of explaining the work, abstract mentions many numerical values, percentages, etc. This is in genreral not the standard academic writing style, and is not suitable for a conference paper. It is also not educating the reader.

  • Same problem happens in the introduction and the rest of the paper. I happen to have worked on both speculative decoding and MoE models for LLMs and so I could guess what the authors wanted to deliver to the reader; yet the manuscript is extremely confusing and not appropriately written, especially for the eager reader or the reviewer who does not have the background.

  • I suggest the authors to rewrite the paper before submitting it for review. Some general tips: (1) avoid using technichal terms before defining them, for instance, do not use the term "expert-offloaded" speculative decoding without first defining the concept. (2) think of yourself as a person who has not worked on the project, try to see how they would read your manuscript, try to resolve any confusions they may have. (3) do not focus too much on experimental results in the abstract and introduction of your paper, instead focus on concepts, why they are important, what has been ignored in the existing literature, how your solution resolves the problem at hand.

  • Figure 1 needs polishing, parts of the figure are overlapping.

  • Related work section is merely a table, I have not seen this style in contemporary literature; I suggest the authors have a brief discussion in the body of the paper.

  • stages A, B are mentioned but never defined properly. Does the start of section 4 explain stage A and the second part explain stage B?

  • I have a hard time understanding the exact drafter architecture the authors try to train, and the objective function utilized to train the model.

Questions for authors
  • equation notations are sometimes missing, for instance, what is u_{l,e} in equation (4)?
Reviewer_E4chOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In a mixture-of-experts model, checking a draft token can load expert weights onto the GPU even if the token is later rejected. VerifyLocal changes which tokens are proposed. A rank-64 adapter reads which experts are resident before a round and adjusts the probabilities of a frozen draft model. The target model, its router, and the sampling correction stay unchanged. The paper first checks whether useful alternatives exist. On 256 math and code prompts, the authors verify eight candidates per prompt. Compared with choosing the candidate with the highest expected token count, their low-traffic choice saves 30% of simulated cache-miss bytes and retains 98.5% of that count. In a separate 500-prompt test on an NVIDIA L4 with a 25% expert cache, the authors report 84 to 63 MB transferred per committed token and 19.5 to 23.0 committed tokens per second. They also report transfer savings on Mixtral and DeepSeekMoE.

Critical strengths
  1. The method changes the draft, not the target. Section 3 gets an important detail right: a rejected suffix must not remain in the target's token history, but the experts it loaded can stay on the GPU. The usual correction remains valid if it uses the normalized distribution that actually produced the draft [1]. The novelty is narrow but useful. Saguaro changes draft probabilities to reuse cached speculation results [2]. SpecMoE drafts with recently used experts [3], and EcoSpec selects branches that reuse experts within a draft tree [4]. VerifyLocal instead learns how the expert weights already on the GPU should change the next token's proposal probability. The last two papers merit discussion in Section 2.

  2. The paper measures the quantity the system is meant to improve. Table 3 reports physical weight copies and committed tokens per second, not just simulated cache hits. The stated timing includes the draft, verification, correction, and cache work. The capacity sweep is a useful check: Table 6 reports goodput gains of 24% at 10% cache capacity, 9% at 50%, and about 0–1% when all experts fit. That pattern supports the proposed explanation without claiming a benefit on fully resident models.

Critical weaknesses
  1. The 8% advantage does not yet isolate the value of learning. Section 4 chooses the strength of the nonlearned rule by minimizing transferred bytes per expected committed token, subject to a 95% commitment guard. Table 3 judges the winner by actual tokens per second. The least traffic need not mean the fastest decoding. The calculation is correct: 23.0 versus 21.3 tokens/s is an 8% gain over the selected rule. But neither the selected setting nor the full sweep is shown, despite Section 4 saying all seven settings are reported. Table 8 does not settle this issue. Its learned model without residency reaches 20.5 tokens/s versus 19.5 for the reference. If “off” means empty resident sets, Eqs. 4–5 make the learned adjustment exactly zero. A learned constant could make this a valid control, but the paper needs to say what was used.

  2. The training setup and the held-out commitment result are not tied to the deployed adapter. Section 4 permits either physical time or a cost estimate as the training signal. Appendix B does not identify the training split, cost coefficient, or checkpoint-selection rule. The source of training cache states matters: checking even a rejected draft changes which experts are resident next round. It is also unclear what the deployed policy retained. Section 6.1 derives 98.5% expected commitment by comparing two hindsight choices among eight candidates. Figure 3(c) places that same number beside the learned policy's 8% speed gain. Unless commitment was measured again on the 500 fresh prompts, the figure combines results from different policies and prompt sets. Actual committed tokens per round for each of the three serving policies would resolve this.

References:

[1] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast Inference from Transformers via Speculative Decoding

[2] Tanishq Kumar, Tri Dao, and Avner May. Speculative Speculative Decoding

[3] Jehyeon Bang et al. SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding

[4] Jincheng Xie et al. Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

[5] NVIDIA. L4 Tensor Core GPU: Specifications.

Questions for authors
  1. On the validation prompts, what goodput and commitment did each of the seven settings of the nonlearned rule achieve? Which setting was selected, and was it also the fastest one that met the 95% guard? Please keep the confirmation prompts out of this selection.
  2. What input or bias replaces the residency vector in Table 8's learned, no-residency cell? For the full adapter, what cost was actually used, which policy generated the training cache states, and were the 500 confirmation prompts excluded from fitting and checkpoint selection?
  3. Is the 98.5% in Figure 3(c) from Stage A, the physical confirmation run, or both? For the latter, can you give the baseline, nonlearned rule, and VerifyLocal's actual committed tokens per round alongside their transfer and timing results from the same runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.