acceptodds
Under review as a conference paper at ICLR 2027

Shaping What to Verify: Realizable Multi-Path Speculative Decoding

Abstract

Multi-path speculative decoding samples several draft continuations, but efficient lossless verification typically selects only one. This makes the selector crucial: it not only chooses a path, but also reshapes the proposal distribution seen by the verifier. We introduce *Sequential Martingale Selection* (SMS), a framework for optimizing this proposal while guaranteeing finite-sample realizability. SMS uses retained mass as the exact block-efficiency objective, represents selector-induced reweighting through locally conserved bounded densities, and realizes every proposal in the family with an explicit sequential selector. We further develop SMS-LP1, an online retained-flow optimizer that requires no additional target-model evaluation. Extensive experiments across LLM settings show that SMS-LP1 improves end-to-end decoding throughput by 20.8% over tokenwise verification and 14.7% over GBV on average. SMS-LP1 further achieves a 6.1% average gain on DFlash2, demonstrating substantial and consistent improvements on both standard model configurations and state-of-the-art drafting frameworks.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.