acceptodds
Under review as a conference paper at ICLR 2027

Asymmetric Speculative Decoding: Unlocking In-Context Learning in Draft Models

Abstract

Speculative decoding addresses the Large Language Models (LLMs) inference latency by first drafting and then verifying in parallel. Yet practical speedups fall far short of theoretical bounds, especially on reasoning tasks. We identify the root cause as task isomorphism: existing methods provide identical prompts to both models, forcing the draft model to solve problems beyond its capacity. This produces high rejection rates that eliminate parallelization gains. We discover that acceptance rates increase monotonically during generation, revealing a latent in-context learning (ICL) effect wherein the draft model progressively aligns with the target distribution by learning from verified tokens. Controlled experiments with varying context window sizes confirm this improvement stems from ICL-driven alignment rather than token difficulty. Motivated by this finding, we propose Asymmetric Speculative Decoding (AsymSpec), a training-free framework that decouples the prompts provided to draft and target models. By supplying the draft model with task-relevant example solutions, we reformulate its objective from reasoning to simulation, amplifying the ICL effect from the first token. We prove AsymSpec produces exact samples and converges exponentially to the target distribution. Across reasoning, code generation, and summarization tasks, AsymSpec achieves 2.29× to 4.95× speedup over autoregressive decoding, outperforming standard speculative decoding without model retraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Inference Acceleration, In-Context Learning, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bVnMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper points out that task isomorphism is one of the reasons for limited acceleration ratio in SD methods. The draft model takes the same prefix from the target model, which results in low acceptance rate.
  2. The authors propose AsymSpec, which injects few-shot examples to the draft model to transfer it into an in-context learning task rather than a separate reasoning task.
  3. On reasoning and coding tasks, AsymSpec achieves 2.29× to 4.95× speedup over autoregressive decoding, outperforming standard speculative decoding without model retraining.
Critical strengths
  1. Clarity and effective presentation. Figures clearly convey the motivation, framework, and KV-cache synchronization, while tables compactly present key results.
  2. Strong empirical gains over Eagle 3 and SSD. Across Qwen3 and LLaMA models and diverse reasoning, coding, summarization, and instruction-following tasks, AsymSpec achieves 2.29×–4.95× speedups over autoregressive decoding.
Critical weaknesses
  1. Lacking state-of-the-art models. The current models on are all instruction following models. Please evaluate on frontier agentic models (such as Qwen 3.8 27B, DeepSeek V4/V4.1, Gemma 4, Muse Glimmer, etc).
  2. Lacking state-ot-the-art baseline methods. Please evaluate on DFlash and DSpark (https://github.com/deepseek-ai/DeepSpec).
  3. AsymSpec is very task-specific and relies on specific input domains, which reduces the practicality.
Questions for authors
  1. How to index the desired few-shot prompts & responses from the pool? What is the latency for this?
  2. Does the expanded prefix increase the latency for draft models?
Reviewer_FRS2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents AsymSpec, a training-free speculative decoding method that gives the draft model task-relevant examples generated by the target model, while leaving the target prompt unchanged. It uses standard speculative sampling to preserve the target distribution. Experiments across reasoning, coding, summarization, and instruction-following tasks report speedups over standard speculative decoding, with ablations on exemplar selection, prompt length, and sampling settings

Critical strengths
  1. The motivation is clear: improving draft–target alignment to make speculative decoding more effective at accelerating LLM generation.

  2. The observations that acceptance rates improve as verified history grows, and that longer context helps acceptance, are interesting and provide useful motivation for the method.

  3. The method is intuitive. Using in-context examples to help the draft model better match the target’s output is a promising direction for accelerating decoding without additional training.

Critical weaknesses
  1. Exemplar selection for mixed workloads is underspecified. Section 3.3 mentions task-aligned retrieval, and §4.3 tests several exemplar sources, but the paper does not explain or evaluate automatic selection when task types are unknown or mixed. It remains unclear whether the reported gains hold under these conditions.

  2. Practical validation remains limited. The evaluation includes CNN/DM and MT-Bench, but does not establish performance on long-running agent tasks or sustained conversations where the task changes over time. How exemplars are selected and updated in these settings deserves further study.

  3. The benefit across output lengths is not established. Figure 1 examines acceptance during generation, but Appendix A.5 fixes output length at 512 tokens for the timing experiments. An output-length sweep would help test whether the gains persist for long outputs, particularly given the prompt-influence decay discussed in §3.4.

  4. The separation between exemplar pools and evaluation data is unclear. Section 4.3 evaluates HumanEval and MATH derived pools on the corresponding benchmarks, but does not specify the split or deduplication procedure. Since target-generated solutions to evaluation questions could inflate acceptance rates even without changing the output distribution, explicitly disjoint exemplar pools are needed to establish the validity of the speedup results.

Questions for authors
  1. How are exemplars selected when the task type is unknown or a workload mixes domains? Could you evaluate automatic selection against a fixed exemplar pool?

  2. How would exemplar selection and reuse work as a conversation or agent task evolves? Could you provide an evaluation beyond the current CNN/DM and MT-Bench results?

  3. Could you report end-to-end speedups across output lengths, to assess whether the gains persist during long generation?

Reviewer_fZwnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Asymmetric Speculative Decoding (ASYMSPEC), a training-free framework that decouples the prompts provided to draft and target model, and proves AsymSpec produces exact samples and converges exponentially to the target distribution. The extensive experiment results show the effectiveness of the prospoed method.

Critical strengths
  1. The idea is interesting.
  2. The paper is well written and easy to follow.
Critical weaknesses
  1. Existing research has explored speculative decoding schemes in which the draft model and the target model utilize different contexts, for instance, RAPID (ICML 2025) allows the draft model to generate candidates based on a short, retrieved context, while the target model employs the full context for verification. The paper lacks a discussion of such related work, and the assertion that "existing methods all employ the same input" appears overly absolute. It is recommended that the authors incorporate a discussion of this related work and clarify how the asymmetric prompting approach, based on few-shot examples, proposed in this paper differs from existing schemes.
  2. The version and experimental configuration of the EAGLE-3 baseline are not sufficiently clear, which undermines the persuasiveness of the performance comparisons. While the main text (sec 4.4 and Figure 6) claims to compare against EAGLE-3, Appendix A.8 references the original EAGLE (Li et al., 2024), and the figures are labeled simply as "EAGLE", indicating an inconsistency in how the version is described. Furthermore, according to Table 8, the speedup for the Llama-3.1-8B model on the HumanEval benchmark is only 128.5/69.6=1.85×, significantly lower than the 4.85× speedup reported for the same model and task in the original EAGLE-3 paper. Although differences in hardware, inference backends, and timing methodologies may account for some of this discrepancy, the authors should clarify the specific version, checkpoint, draft tree configuration, and timing protocol used, and provide an explanation for this performance gap. The current experimental details are insufficient to fully substantiate the claim that the proposed method outperforms EAGLE-3.
  3. This paper relies on a draft model to improve distribution alignment via few-shot examples; however, the in-context learning (ICL) capability of models with 1B parameters or fewer may become a performance bottleneck. The paper currently lacks an analysis of this; it is recommended to verify whether the draft model's ICL capability limits improvements in the acceptance rate and the ultimate speedup gains.
  4. Proposition 2 claims to prove exponential convergence between the draft distribution and the target distribution; however, Equation (13) and the appendix actually only compare the distributions produced by the same draft model under different prompts, without constraining the discrepancy relative to the target distribution. Consequently, this proof appears insufficient to support the claimed convergence between the draft and target distributions, nor the conclusion regarding a high acceptance rate. A direct counterexample: under both types of prompts, the draft distribution is ((0.99,0.01))((0.99, 0.01)) and the target distribution is ((0.01,0.99))((0.01, 0.99)). The distance as defined in the paper is consistently zero, yet the actual distance between the draft and target distributions is always 0.980.98, and the expected single-step acceptance rate is only 2%2\%.
  5. There is a lack of implementation details and prompt examples, and the code has not been made open-source, making reproduction somewhat not easy.
Questions for authors

Please see the critical weekness.

2 comments

Thank you for the author's response. I carefully reviewed the discussion of other reviewers and the author's reply. The author has addressed some of my concerns, and I have decided to maintain my score.

Dear Reviewer,

Thank you for taking the time to read our response. We’re glad it addressed some of your concerns.

Could you let us know which of your concerns remain unresolved? We’d also be grateful for any specific suggestions you have for strengthening the manuscript.

If you feel that our response has addressed your main concerns, we hope you’ll consider revisiting your overall assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.