acceptodds
Under review as a conference paper at ICLR 2027

Elastic Decoding: Controlling LLM Inference Speed via Contextual Sparsity

Abstract

Large language model (LLM) inference in memory-bound regimes is constrained by a fundamental trade-off between generation quality and decoding speed. In this regime, contextual sparsity (e.g., mixtures of experts) accelerates autoregressive generation by reducing the number of active parameters loaded per token, while preserving the model's total parameter count. We introduce Elastic Decoding, a contextual sparsity method that can be applied post-hoc to any trained LLM, dense or MoE, returns a single checkpoint with controllable sparsity levels, requires less than 5B finetuning tokens, and is at the Pareto frontier of speed and accuracy. To achieve this, we frame contextual sparsification as a multiple-choice knapsack problem with bytes-per-token as the budget, allocating capacity heterogeneously across the architecture before distilling the model into a single sparse checkpoint. Elastic Decoding can reduce the active number of parameters by up to 83% and, paired with optimized on-device kernels, achieves end-to-end speed-ups of up to 4.6x (e.g., on Gemma4-31B). Our sparsified Qwen3-14B runs faster and more accurately than Qwen3-8B and Qwen3-4B. Importantly, the quality-speed trade-off can be controlled during decoding in real-time, on a token-by-token basis, with negligible switching overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Contextual Sparsity, Memory-Bound Inference, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6JjhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Elastic Decoding, a post-hoc contextual sparsity framework designed to control the inference speed and generation quality of large language models (LLMs) in memory-bound regimes. Instead of training separate models or relying on rigid static pruning, Elastic Decoding frames the sparsification problem as a multiple-choice knapsack problem using bytes-per-token as the budget. It utilizes a two-stage approach: first, it computes layer-wise KL divergence losses and solves a mixed-integer linear program to find optimal heterogeneous sparsity allocations under given budgets; second, it fine-tunes the shared weights via LoRA distillation across randomly sampled budgets. The resulting single model checkpoint enables real-time, token-by-token budget switching and achieves up to 4.6x speedups on Apple Silicon while maintaining strong reasoning performance.

Critical strengths
  • Post-Hoc Flexibility and Efficiency: Elastic Decoding can be applied post-hoc to any pretrained dense or MoE model and requires less than 5B tuning tokens, avoiding the massive compute cost of training models from scratch.

  • Global Heterogeneous Allocation: Unlike prior baselines that enforce uniform sparsity across layers, the multiple-choice knapsack formulation accounts for varying layer sensitivities and dynamically adapts parameter allocation across network depth.

  • Dynamic Token-by-Token Control: The model supports real-time, on-the-fly budget switching during decoding with minimal latency overhead, making it practical for fluctuating on-device resource constraints and thermal limits.

  • Superior Pareto Frontier: The method significantly outperforms existing post-hoc methods (such as TEAL and ReLUfication) and allows large sparsified models to dominate smaller stock models in both accuracy and decoding speed.

Critical weaknesses
  • Decoupled Two-Stage Approximation: The method decouples sparsity optimization (Stage 1) from weight distillation (Stage 2). While practical to avoid intractable training loops, this approximation deviates from true joint optimization and may not reach the global theoretical minimum.

  • Layer-wise Loss Additivity Assumption: Approximating the global KL loss as the sum of independent layer-wise losses ignores non-linear error compounding and inter-layer interactions across deep network layers.

  • Unaccelerated Prefill Phase: As appropriately pointed out in the authors' limitations section, the proposed contextual sparsification only targets the memory-bound generation/decoding phase. Because sparse memory gathers currently hinder execution during compute-bound operations, the prefill phase remains dense. Consequently, the end-to-end acceleration gains will be inherently bottlenecked in long-context or prompt-heavy use cases where Time to First Token (TTFT) is critical.

Questions for authors
  1. How sensitive are the optimized sparsity configurations to the distribution of the calibration dataset? Would a distribution shift in real-world user prompts significantly degrade performance?

  2. Since the router and neuron sparsity choices introduce discrete decisions during Stage 1, how feasible would it be to integrate a fully differentiable, end-to-end joint optimization scheme for both sparsity configurations and model weights?

  3. While speculative decoding and Elastic Decoding represent two distinct paradigms for accelerating memory-bound LLM inference (generating multiple tokens per read vs. reducing bytes read per token), have the authors explored the compatibility or potential synergy of combining Elastic Decoding with speculative decoding? Specifically, could an elastic sparsified model serve efficiently as a target or draft model, and how would dynamic budget switching impact the acceptance rate in speculative decoding?

2 comments

We thank the reviewer for the positive and thorough assessment. We are glad the reviewer appreciated that Elastic Decoding applies post-hoc to any dense or MoE model with less than 5B tuning tokens, that its global heterogeneous allocation adapts to layer sensitivities, and that it supports real-time token-by-token budget control. Below we address each point, the revised PDF integrates the feedback (changes are highlighted).

Decoupled two-stage approximation (W1)

We agree that decoupling is an approximation and does not guarantee the global optimum of the joint problem, as we state in our limitations. Three results indicate that it costs little in practice.

  • The allocation survives fine-tuning. The heterogeneous allocation beats a uniform one both before and after distillation (Figure A3).
  • Joint training was worse. In preliminary experiments, training the routers jointly with the weights underperformed our two-stage approach at a matched byte budget (see Q2).
  • The routers stay accurate. The routers fitted to the original weights remain accurate after LoRA fine-tuning: their recall drops by only 0.7–1.2 % (new Figure A5).

Layer-wise loss additivity (W2)

We agree that the sum of layer-wise losses ignores interactions between layers, and we tested the assumption directly. Across 128 random allocations at the same budget, the summed layer-wise loss strongly correlates with the true loss of sparsifying all layers jointly (Figure 2c). The remaining interactions are addressed by Stage 2: distillation trains the whole sparsified network jointly, and recovers most of the accuracy lost at initialization (Figure A6).

Unaccelerated Prefill Phase (W3)

We agree, and our limitations state it: Elastic Decoding accelerates the memory-bound decoding phase only, and prompts are processed densely. Time-to-first-token is therefore that of the original model, neither faster nor slower. In reasoning workloads, decoding dominates the time-to-answer, with answers of 2,000–2,500 tokens on average (Table A1). We now also measure the end-to-end time to answer including prefill (new Table A2 and Figure A16, see our response to QTP8 Q9). On the benchmark prompts, Elastic Decoding answers up to ~2.7× faster than the original Qwen3-14B on average accounting for prefill and growing context overhead.

How sensitive are the optimized sparsity configurations to the distribution of the calibration dataset? (Q1)

Our evaluation already measures transfer beyond the calibration data. Both the sparsity configurations (Stage 1) and the shared weights (Stage 2) are optimized on a reference dataset built from prompts of the Kimi-K2.5 distillation, Nemotron post-training, Nemotron-3-Nano RL Blend and Nemotron-science collections (Appendix D.1). None of the nine benchmarks, nor their training splits, is among these sources. Every reported score therefore evaluates the routers and allocations on prompts from other sources than the ones they were fitted on. We agree that larger shifts, such as other languages or agentic tasks, are important areas for future work.

how feasible would it be to integrate a fully differentiable, end-to-end joint optimization scheme for both sparsity configurations and model weights? (Q2)

Relaxing the per-token selection is possible, for example with straight-through or Gumbel top-kk estimators, but it does not make joint optimization efficient. The weights would still have to be trained jointly with every candidate configuration of every layer, up to about a thousand FFN and over a hundred thousand attention configurations per layer (Table C1), which is impractical at the scale of the models we study. Indeed, in early experiments we had attempted to train routers and LLM weights jointly, but we found that routers failed to learn useful sparsity patterns. This led us to using our two-stage process of sparsity optimization and distillation.

[1/2]

have the authors explored the compatibility or potential synergy of combining Elastic Decoding with speculative decoding? (Q3)

While the adoption of speculative decoding with contextual sparsity is not the focus of the paper, we ran an explorative analysis to evaluate potential synergies. Specifically, we explored whether consecutive tokens share activated weights. In the table below, we measure the total bytes activated across all sparsified components (MLP neurons, attention layers, MLP inputs and the LM head) over windows of 1–4 consecutive tokens. Notably, the union grows sublinearly. When using Elastic Decoding with 20% of active bytes, verifying 4 tokens activates 37.2% of the dense weight, which is 2.7x fewer bytes than a dense 4-token verification pass. This means that speculative decoding, which has to verify several consecutive tokens, could further accelerate Elastic Decoding.

Active parametersk = 1k = 2k = 3k = 4
50%0.5000.6360.7010.738
33%0.3330.4620.5350.582
25%0.2500.3540.4170.461
20%0.2000.2800.3330.372

We hope these clarifications address the reviewer's concerns, and we would be happy to answer any further questions.

[2/2]

Reviewer_YNJHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

On-device LLM decoding is limited by memory bandwidth, so its speed is hard to adjust on demand. This paper proposes Elastic Decoding to address that problem. The method treats the number of bytes read per token as a budget and solves for which sparsity scheme each layer should use and how many parameters it should keep. A single lightweight distillation pass then produces a model that can switch speed levels on a per-token basis. The authors measure accuracy on dense and MoE models from the Qwen3 and Gemma4 families across nine reasoning benchmarks, and they measure speed on an Apple M4 Max. The method reduces active parameters by up to 83%, speeds up decoding by up to 4.6×, and outperforms existing methods such as TEAL.

Critical strengths
  1. The method is a complete system design. It first decides how each layer is sparsified, then uses a knapsack solver to allocate the budget non-uniformly across layers, and finally applies distillation to recover the capability lost to sparsification. The stages fit together cleanly, and ablations support each one: non-uniform allocation beats uniform allocation both before and after fine-tuning. As a result, the method still performs usably at high sparsity.

  2. The method runs on real hardware, and the paper reports measured end-to-end speedups rather than only theoretical compute or memory traffic. The authors also show that their custom inference engine matches llama.cpp in speed on dense models, which rules out the engine itself as a source of the speedup. This makes the speedup claims more credible and the method more relevant to real deployments.

Critical weaknesses
  1. The benchmarks are not hard enough to fully support the claim that the method is "near-lossless." Several of the nine benchmarks, including GSM8k, ARC-Challenge, HumanEval+, and MBPP+, are close to saturation, and only the GPQA variants are relatively hard. The paper reports only aggregate averages, gives no per-benchmark results, and includes no difficult tasks that require long-chain reasoning. Easy tasks do not use the model's full capacity, so the current results cannot show whether the method actually preserves the model's capability or only exploits redundancy on easy tasks. Two observations suggest some capability is lost: in Table A1, output length grows as sparsity increases, and MoE models degrade more noticeably. Per-benchmark results and evaluations on hard reasoning tasks would resolve this concern.

  2. All speed experiments use a batch size of 1, and the paper does not discuss serving multiple concurrent requests. With several requests, the parameters they need must be combined as a union, which shrinks the memory reads the method can avoid. The method may therefore be limited to single-user on-device inference.

Questions for authors
  1. Could the authors report results for each benchmark separately, especially GPQA-Diamond at different sparsity levels?
  2. Does the method stay close to the original model's accuracy on difficult tasks that require long-chain reasoning, such as competition math and competitive programming?
1 comment

We thank the reviewer for the thoughtful feedback and the careful reading of the paper. We are glad the reviewer found Elastic Decoding a "complete system design" whose stages "fit together cleanly" and are each supported by ablations. We are also glad the measured end-to-end speed-ups on real hardware, with an engine that matches llama.cpp, were found compelling. Below we address each point, the revised PDF integrates the feedback (changes are highlighted).

Could the authors report results for each benchmark separately, especially GPQA-Diamond at different sparsity levels? (W1 and Q1)

We now report per-task benchmark performances in Appendix Fig. A10. For GPQA and GPQA diamond, the drop in performance tends to be larger than for simpler tasks (e.g., GSM8k), but the Elastic Decoding model of Qwen3-14B is better and faster than Qwen3-8B and Qwen3-4B. We have updated the paper to avoid the ambiguous term “near-lossless”.

Does the method stay close to the original model's accuracy on difficult tasks that require long-chain reasoning, such as competition math and competitive programming? (Q2)

To investigate this, we evaluated Elastic Decoding models on the AIME24 and AIME25 benchmark tasks (new Appendix Fig. A11). The performance drop of Elastic Decoding models was substantially higher than on simpler tasks, but Elastic Decoding models of Qwen3-14B still dominate the dense Qwen3-8B in speed and accuracy. We note that the AIME benchmarks have a small sample size, leading to large error bars on these evaluations. We include these results in the paper and explicitly discuss this in the limitations.

The method may therefore be limited to single-user on-device inference. (W2)

We agree and now state this more explicitly in the limitations. Elastic Decoding targets single-user on-device decoding at batch size one, where decoding is memory-bound and the bytes read per token set the speed. With several concurrent requests, each layer must read the union of the parameters the requests select, so the savings shrink as the batch grows.

We note, however, that this is a limitation of any contextual post-hoc sparsity method (e.g., TEAL, ReLUfication) as well as of other methods to speed up inference, such as speculative decoding. In general, we recognize two different extremes of LLM inference: single-user on-device and high-throughput serving. This leads to two distinct sets of algorithms and choices, since batched high-throughput inference is typically not memory bound.

We hope these clarifications and additional results address the reviewer's concerns, and we would be happy to answer any further questions.

Reviewer_QTP8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Hello, authors! Thank you for the work, it was interesting to read, and the demo videos were a nice addition. Below is my understanding of the paper.

The setting is memory-bound LLM decoding at batch size one, where speed is basically bytes of weights read per token. You propose Elastic Decoding, a post-hoc contextual sparsity method for dense and MoE models that gives a single checkpoint with controllable sparsity (speed) at inference time. Per layer (FFN, attention, LM head) there are several sparsity options: input sparsity and, for FFN, neuron sparsity with a learned router (linear or a new low-rank bilinear router). Stage 1 measures per-layer KL to the original model for each option, assumes the total loss is the sum of layer losses, and solves a multiple-choice knapsack (MILP) with a bytes-per-token budget, for a set of budgets. Stage 2 trains one shared LoRA with KL distillation, sampling the budget per token, with ~4.2B tokens. Inference runs in a custom Metal engine on an Apple M4 Max in bf16, batch 1, and the budget can be switched per token.

Experiments are on Qwen3-14B, Qwen3-30B-A3B, Gemma4-31B and Gemma4-26B-A4B on 9 reasoning benchmarks with thinking on, with up to 83% fewer active parameters, up to 4.6x decode speedup, and elastic Qwen3-14B being faster and more accurate than dense Qwen3-8B and Qwen3-4B. Baselines are TEAL and ReLUfication.

There is no code, only two demo videos. I watched them: the on-screen speeds for Qwen3-14B are close to Table A1, the speed numbers look real to me.

Critical strengths
  • speed is real wall-clock decoding on hardware with a custom engine, not modeled from bytes, with a dense parity check against llama.cpp and longer context check. numbers in the demo videos match the paper, and bytes x tok/s is consistent with M4 Max bandwidth
  • the bytes-per-token knapsack over different unit types (input sparsity, neurons, attention, LM head), with router cost counted in the budget, is clean and general, and one checkpoint with per-token switchable budget is practical. Stage 1 alone already beats TEAL at high sparsity (e.g. Qwen3-14B), and the ablations (uniform vs heterogeneous allocation, shared vs per-budget LoRA, switching overhead) are useful
Critical weaknesses
  • in the abstract is shown only against smaller dense models of the same family. unmodified MoE models dominate the elastic dense ones in speed-accuracy, and the trade-off is memory, which is not shown? would be good to make the claim more precise or add memory as an axis
  • all speed results are bf16 vs bf16, while local deployment usually uses 8-bit or 4-bit weights. at the same bytes quantization alone is as good or better, and the 4-4.6x speedups would be smaller against a quantized dense baseline. the combination helps on accuracy, but there is no speed measured for sparse + quantized kernels?
  • quality drops are measured from the finetuned dense checkpoint, not from the original model, and there is no variance. seeds, temperature, max length and how 9 tasks are aggregated are not given
Questions for authors

clarification:

  • in Fig. 5a stock Qwen3-30B-A3B and Gemma4-26B-A4B are above the elastic Qwen3-14B and Gemma4-31B curves. what is the exact scope of the "Pareto frontier" claim?
  • could you report quality drops relative to the original model, and add variance (seeds or bootstrap CI) and decoding / eval settings?
  • which M4 Max variant and RAM size? what is the peak memory per model, and does the Gemma4-31B demo keep both layouts in memory?
  • what is the total compute (GPU-hours) for Stage 1, router training and teacher traces?
  • do the frozen routers stay accurate after LoRA changes the FFN weights?
  • was the training data decontaminated against GPQA and MMLU-Pro?

suggestions:

  • compare at equal bytes with quantized dense baselines (4-bit / 8-bit Qwen3-14B and Qwen3-8B) and measure speed of sparse + quantized
  • TEAL + the same LoRA distillation, and one elastic many-in-one baseline (Flextron or MatFormer)
  • measure time-to-answer end-to-end with prefill and growing context
  • compare engine speed with MLX too?
3 comments

We thank the reviewer for the mindful feedback and interest in the paper, and for watching the demo videos. To complement the videos, we will release our code upon acceptance. We are glad the reviewer found our speed measurements and real wall-clock results compelling. We are also glad the bytes-per-token knapsack with a per-token switchable budget was found "clean and general" and practical. Below we address each point, and integrate the feedback directly in the revised PDF (changes are highlighted).

what is the exact scope of the "Pareto frontier" claim (W1 and Q1)

We agree and have made the claim precise in the abstract, the caption of Figure 5 and Section 4.1. The claim is per model family: Elastic Decoding makes each model faster and more accurate than the smaller models of its family (e.g., Elastic Qwen3-14B dense over Qwen3-8B dense and Qwen3-4B dense). Importantly, Elastic Decoding quality is bounded by the original model it starts from. We do not claim that a sparsified dense model outperforms an MoE model: the original Qwen3-30B-A3B and Gemma4-26B-A4B are faster at equal accuracy. Elastic Decoding is complementary to MoE models. Applied to them, it forms the frontier at every speed beyond that of the original models. As the reviewer notes, MoEs cost memory (61.1 GB of bfloat16 weights for Qwen3-30B-A3B, against a 37.6 GB peak for Elastic Qwen3-14B with all budgets selectable, Table E2) and the resources to train them from scratch.

there is no speed measured for sparse + quantized kernels (W2 and Q7)

We have now implemented Elastic Decoding's sparse kernels on 4-bit weights in our inference engine. The weights use round-to-nearest quantization, with the MLP, attention, routers and LM head all at 4 bits. We measured decode throughput on a Mac Studio (M4 Max) and now report the quantized version of Figure 1, where all models use the same quantization and inference engine, in Fig. 8a. For Qwen3-14B, Elastic Decoding achieves up to 202 tok/s. Thus, Elastic Decoding remains at the Pareto front even in the quantized regime, further strengthening the compatibility assumption we discussed in the paper.

quality drops are measured from the finetuned dense checkpoint, not from the original model (W3)

We report quality relative to the original models in every result: the original models are the large colored circles in Figure 5 (and in Figures A7–A10), so every drop can be read against the original model directly. In one sentence in the results, we had mistakenly reported the drop from our finetuned dense checkpoint and we have fixed this in the updated paper: the drop from the dense Gemma4-31B model is 2.0% at 33% of active parameters and 9.9% at 17% of active parameters.

temperature, max length and how 9 tasks are aggregated are not given (W3)

We now report all decoding and evaluation settings in Table D1 and performance for every task separately in Appendix Fig. A10. We sample with temperature 0.7, top-pp 0.8 and top-kk 20 for the Qwen3 models, and with temperature 1.0, top-pp 0.95 and top-kk 64 for the Gemma4 models. We allow up to 32768 generated tokens and report the unweighted mean over the nine tasks. The original models, all Elastic Decoding budgets and all baselines use identical sampling parameters.

there is no variance [on the reasoning benchmark] (W3 and Q2)

In the updated paper, we re-evaluated every operating point of Figure 5 with three seeds: every budget of the four Elastic Decoding checkpoints, and all original and smaller stock models. The new Appendix figures A9 and A10 report the mean and standard deviation across seeds, for the nine-task average and for each benchmark. The standard deviation of the nine-task average is 0.29 points on average and at most 0.65 points.

[1/3]

which M4 Max variant and RAM size? what is the peak memory per model, and does the Gemma4-31B demo keep both layouts in memory? (Q3)

All measurements use a Mac Studio with an M4 Max (16-core CPU, 40-core GPU, ~500 GB/s memory bandwidth) and 128 GB of unified memory, as we now report in Section 4. 

The new Table E2 (Appendix E) reports the peak memory of Qwen3-14B while decoding after an 8k-token prompt. With all nine budgets available, Elastic Decoding peaks at 37.6 GB, against 34.9 GB for the original model. The 2.7 GB difference comes from the routers, and a deployment with a single fixed budget needs only 0.27–0.88 GB of them. Since routers only select which neurons are computed, quantizing them is a natural way to reduce this further (a router error changes the output only when it changes the selected top-kk set). Keeping both FFN layouts resident, so that budget switches need no transposes, adds 9.3 GB.

Regarding the demos: the Qwen3-14B demo keeps a single weight layout and transposes weights on a switch. The Gemma4-31B demo keeps both layouts resident for the 30 FFN layers that use input sparsity at some budget. We updated Appendix E accordingly. With a single layout, the switching overhead is negligible at realistic switching rates, as can be observed in the Qwen3-14B demo. With both layouts resident, switching even at every token has no measurable overhead (Table B2).

what is the total compute (GPU-hours) for Stage 1, router training and teacher traces? (Q4)

We now report these numbers in the new Appendix D.3. The total compute for Stage 1 of Qwen3-14B Elastic Decoding took approximately 1600 GPU-hours. This includes training 37 linear and bilinear routers for each of the 40 layers (500 GPU-hours) and grading all sparsity configurations (1,100 GPU-hours). Given that Stage 1 is embarrassingly parallel across layers and router recipes, it took around 36 wall-clock hours. Coarser sparsity configurations could be used to decrease compute time. Generating teacher traces is a one-time compute cost and it took approximately 800 GPU-hours, while finetuning took 600 GPU-hours, for a total of about 3,000 GPU-hours. 

do the frozen routers stay accurate after LoRA (Q5)

We computed the recall of the routers before and after LoRA fine-tuning (new Figure A5). For all budgets, LoRA fine-tuning decreased the recall by 0.7–1.2 % (less at small budgets). The drop was larger in early layers (3 % in layer 1) than in middle layers (0.2 % in layer 13). Since our routers generally achieve ~60 % recall, this drop is comparatively small. 

was the training data decontaminated against GPQA and MMLU-Pro? (Q6)

We checked the training data against both benchmarks. We found no contamination for GPQA, and negligible contamination for MMLU-Pro: 6 of its 12,032 test questions (0.05%) appear among the prompts used for fine-tuning, i.e. 0.001% of the training examples. We removed those samples and found no change in accuracy. We would also like to highlight that only the prompts of the source datasets were used to construct our training data. Furthermore, the ablation in Fig. A13 empirically demonstrates that our reference dataset does not inflate model performance.

TEAL + the same LoRA distillation (Q8)

We trained TEAL with our distillation loss and the same hyperparameters as Elastic Decoding. We found that our distillation loss training also substantially improved TEAL. For example, for Qwen3-14B, at 33% active parameters, the trained TEAL model achieved 75.7% reasoning accuracy, whereas the untrained TEAL model had been degenerate. Elastic Decoding still clearly outperforms it at the same sparsity, with 79.4 % without distillation and 81.3 % with distillation.

one elastic many-in-one baseline (Flextron or MatFormer) (Q8)

We reimplemented MatFormer for Qwen3-14B. Since MatFormer pretrains whereas we post-train, we pre-sorted neurons by their mean contribution and then trained with our distillation loss. MatFormer dropped 9 % in reasoning accuracy at 66 % active bytes and 19 % at 50 % active bytes. It is thus outperformed by TEAL and, by a large margin, by Elastic Decoding. We attribute this to MatFormer's fixed neuron ordering, whereas Elastic Decoding routes each token to any subset of neurons.

[2/3]

measure time-to-answer end-to-end with prefill and growing context (Q9)

We now measure the time to answer of every request of our benchmark suite, including the prefill of its prompt, the growing context and sampling, for every budget of Elastic Qwen3-14B and for the original Qwen3-14B, 8B and 4B (new Table A2 and Figure A16). 

The KV cache is read in full at every budget, so each token of context slows every decoding step by the same amount, which is a larger share of the step at low budgets (Figure A16). Averaged over all answers, Elastic Qwen3-14B answers up to ~2.7× faster than the original Qwen3-14B, at 20–22 % of the bytes. It remains much faster than Qwen3-8B at comparable accuracy (52 s vs 87 s at 33 % of the bytes), and is on par with Qwen3-4B at its lowest budgets (47 s at 25 % of the bytes). We updated Table A1 and Section 4.1 accordingly.

compare engine speed with MLX too? (Q10)

We extended Table E1 (Appendix E) with two more inference engines, MLX (mlx-lm) and oMLX, on the original models in bfloat16 at batch size one. Our engine is at parity with existing engines: on every model and prompt length, it is within 4 % of the fastest of llama.cpp, MLX and oMLX. The speed-ups of Elastic Decoding therefore come from reading fewer bytes per token, not from a faster engine.

We hope these clarifications and additional experiments address the reviewer's concerns, and we would be happy to answer any further questions.

[3/3]

Comments on the paper

1 comment

We thank the reviewers and the AC for their time and the insightful comments. We are happy that all reviewers recommend acceptance. The reviewers appreciated our end-to-end inference engine (QTP8, YNJH), they found our method to be “...clean and general...” (QTP8) and a “...complete system design” (YNJH), and they highlighted that our method “...significantly outperforms existing post-hoc methods...” (6Jjh). Following the suggestions of the reviewers, we have now added quantization support in our inference engine, yielding a decoding speed of 202 tok/s on Qwen3-14B (4 bit) on an Apple M4 Max. Our inference engine, including the new quantization support, will be released together with the rest of our code upon acceptance. We have also performed several experiments to address remaining concerns of the reviewers in the individual responses (including additional baselines, benchmarks and analyses). Our detailed revision integrates all the feedback and we hope it supports a positive recommendation for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.