acceptodds
Under review as a conference paper at ICLR 2027

Faster LLM Inference via Sequential Monte Carlo

Abstract

Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sampling. Because rejection truncates the draft block at the first error, throughput degrades when draft and target diverge. Rather than rejecting draft tokens outright, we propose to reweight them. To this end, we introduce sequential Monte Carlo speculative decoding (SMC-SD), which replaces token-level rejection with importance-weighted resampling over a population of draft particles. SMC-SD is a principled approximate inference scheme that trades exactness for additional speed, while preserving theoretical bounds on its per-step approximation error. Because LLM inference is memory bandwidth-bound, the arithmetic needed to draft particles and to score them in parallel comes nearly for free—SMC-SD uses idle compute to turn verification into a vectorized, fixed-size operation with no rollback. Empirically, SMC-SD achieves 2.43x speed-up over speculative decoding and a 5.2x speed-up over autoregressive decoding, while remaining within 3% of the target model's accuracy on reasoning, instruction-following, and coding benchmarks.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Probabilistic Inference, LLM, Bayesian Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bM1LOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SMC-SD for approximate decoding, replacing token-level rejection with importance-weighted resampling. The method provides error bounds, speed analysis, and a shared-KV-cache implementation. Experiments show higher throughput with small average accuracy losses.

Critical strengths
  1. Replacing rejection with particle resampling offers an interesting approach to inference acceleration. The connection between the algorithm and GPU utilization provides a useful perspective on the speed–quality tradeoff.
  2. The implementation addresses practical KV-cache sharing and resampling costs. The paper also reports the high-batch crossover where an exact decoder becomes faster, helping identify the method’s operating regime.
Critical weaknesses
  1. The motivation for choosing Sequential Monte Carlo is not sufficiently established. While the paper explains how SMC addresses the limitations of rejection-based speculative decoding, it does not clearly justify why SMC is preferable to other approximate decoding approaches. In particular, the benefits of sequential resampling over simpler alternatives remain unclear.

  2. The accuracy evaluation leaves important reliability questions unresolved. Although the paper evaluates four benchmarks, it does not analyze how accuracy degradation varies with task difficulty or generation length. Long-context performance is also not systematically evaluated, and the fixed-horizon error bound does not fully address these concerns.

  3. The experimental comparisons with existing speculative decoding methods are insufficient. The comparisons with EAGLE3 and DSpark lack sufficient configuration details to assess their fairness. The lossy decoding comparisons report throughput without corresponding quality measurements, while other approximate methods, such as FSD and FLy, are not evaluated.

Questions for authors
  1. Why is SMC particularly suitable for approximate speculative decoding compared with other approaches? What specific benefits does sequential resampling provide? Could the authors compare it with simpler alternatives, such as importance sampling without intermediate resampling?

  2. How does accuracy degradation relative to the target model vary with task difficulty? Could the authors provide results on more challenging tasks to assess whether the reported accuracy retention generalizes?

  3. At a fixed particle count, how does accuracy degradation change with generation length? How do accuracy and throughput behave when both input contexts and generated outputs are long?

  4. Could the authors provide more details on the configurations and parameter selection for EAGLE3 and DSpark? What quality degradation accompanies the reported speedups of lossy methods, including Cactus? How does SMC-SD compare with FSD or FLy under similar accuracy constraints?

Reviewer_cx72OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a way to speed up autoregressive sampling LLMs based on a approximate sampling procedure that uses a large model and a small model.

Critical strengths

The paper is well written and I had a great time reading it. The proposed method is well routed in SMC methods and experimental results look promising.

Critical weaknesses

I have two main concerns about the paper:

(a) Theoretical bound in Theorem 3.1: In Theorem 3.1 authors bound L2 distance between the approximate samples and true samples by 1 + chi^2 / N, where N is the number of samples or beam size. I think this result does not say much about the method at all as we use L2 distance in the left hand side. The reason is as follows: As T gets larger the probabilities of sequences tends to be smaller, so |\hat{p} - p |^2_2 \leq |\hat{p} |^2_2 + | p |^2_2, which tends to be very small for large sequences (e.g., 10^-5 or even smaller). Hence I believe the bound is not informative.

(b) Experimental comparisons: The paper compares the proposed method to speculative decoding, but does not compare with other approximative samplers with speculative decoding e.g., https://proceedings.iclr.cc/paper_files/paper/2025/file/6f43166f50f26e8d8f3edc5545b0749f-Paper-Conference.pdf or speculative decoding with lenience (Appendix A.5 in https://proceedings.mlr.press/v202/leviathan23a/leviathan23a.pdf).

Questions for authors

Please address the weakness sections above.

Reviewer_jbG9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SMC-SD, an approximate speculative decoding method based on sequential Monte Carlo. Instead of rejecting draft tokens at the first mismatch, it maintains multiple candidate trajectories and applies importance-weighted resampling to approximate the target distribution. A SGLang-based implementation exploits batched particle evaluation and shared KV-cache storage. The paper reports substantial throughput gains, particularly at low batch sizes, while allowing a controlled approximation–quality trade-off.

Critical strengths
  1. An interesting algorithmic alternative with a well-motivated operating regime. Replacing token-level rejection with particle reweighting offers a distinct approach to speculative decoding. The focus on low-batch inference is meaningful because additional parallel particle computation can utilize otherwise underused GPU compute resources.

  2. A useful algorithm–systems connection. The implementation combines vectorized target evaluation with shared ancestral KV pages to reduce the overhead of maintaining multiple trajectories. The reported comparison with modern speculative decoders, including the batch-size crossover where exact methods become faster, helps characterize the intended operating region.

Critical weaknesses
  1. The evaluation does not sufficiently isolate the contribution of importance-weighted resampling from draft-model quality and particle parallelism across the four benchmarks. The paper evaluates GSM8K, MATH500, AlpacaEval, and DS1000, reporting favorable speed–quality trade-offs. However, the standalone draft-model performance is not reported, making it difficult to determine how much quality SMC-SD recovers beyond its proposal model. More importantly, the Qwen comparisons use different proposal models: exact SD uses a 0.5B drafter, while SMC-SD uses a stronger 3B drafter. Although choosing the best proposer for each algorithm is reasonable for deployment-oriented comparisons, it does not isolate the algorithmic improvement. Even a fixed-proposer comparison between exact SD and SMC-SD would not fully separate the effect of importance weighting from the additional parallel particle computation. Since importance-weighted resampling is the central methodological contribution, a controlled ablation across all four benchmarks would substantially strengthen the paper.

  2. The approximation-quality and serving claims are not fully established at the strongest performance configuration. The headline 5.2× speedup over AR is measured using a 70B target on ShareGPT, whereas the detailed task-quality evaluation uses 8B and 14B targets. As a result, the strongest speedup and the corresponding approximation-quality trade-off are not directly demonstrated together. Furthermore, the algorithm selects the returned output trajectory only after termination. Internal particle progress therefore does not imply incrementally committed output, limiting its direct applicability to interactive streaming without additional mechanisms.

Questions for authors
  1. Fair comparison and contribution of importance-weighted resampling. The Qwen experiments use different draft models for exact SD (0.5B) and SMC-SD (3B). While selecting the best proposer for each method is reasonable for a systems comparison, it makes it difficult to isolate the contribution of the proposed algorithm. Could the authors provide a fixed-proposer comparison between exact SD and SMC-SD using the same Qwen 3B draft and 14B target, reporting both task quality and throughput? This comparison should cover the four existing benchmarks (GSM8K, MATH500, AlpacaEval, and DS1000). Additionally, could the authors report standalone draft-model accuracy for the evaluated proposal models, if available? This would help establish how much quality SMC-SD recovers beyond the draft itself. These comparisons would help determine whether the improvements primarily arise from importance-weighted resampling or from using a stronger proposal model.

  2. Relationship between headline speedup and quality. The headline 5.2× speedup is measured on a 70B ShareGPT configuration, while the detailed quality evaluations use 8B and 14B target models. Could the authors clarify whether the reported quality preservation can be expected under the 70B configuration, or whether the two claims should be interpreted as results from separate experimental settings? I am not requesting a comprehensive new 70B benchmark evaluation, but the scope of the combined speed–quality claim should be made explicit.

  3. Output commitment and streaming semantics. Algorithm 1 selects the returned sequence from the terminal particle population. Does this mean the current implementation cannot irrevocably commit output tokens before generation terminates? If so, could the authors clarify whether SMC-SD is primarily intended for offline/full-sequence generation rather than interactive streaming? This distinction would help interpret the practical applicability of the reported inference throughput improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.