acceptodds
Withdrawn from ICLR 2027

ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The efficiency of Large Language Model (LLM) serving is fundamentally limited by the sequential nature of autoregressive decoding. Speculative Decoding (SD) mitigates this by using a lightweight draft model to speculate future tokens, which are then validated by the LLM in a single parallel forward pass. To further boost efficiency, multi-candidate schemes propose diverse candidate sets to increase the likelihood of token acceptance. However, we show that these schemes are bottlenecked by Residual Drift: a phenomenon where the rejection of initial candidates causes the residual target distribution to diverge from the draft model's predictions. This shift renders subsequent candidates ineffective and forces the system into expensive resampling. To resolve this, we propose ResiSpec, a framework that strategically reforms the proposal distribution during verification to anchor the residual target mass within the draft model's high-confidence regions. By mathematically re-aligning the verification process without compromising output exactness, ResiSpec prevents candidate obsolescence and achieves up to 1.92× speedup over state-of-the-art multi-candidate methods.

Primary area:
optimization
Keywords:
Speculative Decoding, Large Language Model Serving, Inference Acceleration, Multi-Candidate Speculation, Residual Drift

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LD3EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ResiSpec targets a bottleneck in multi-candidate speculative decoding called Residual Drift. After an early candidate is rejected, the residual target shifts mass into the draft’s blind spots, while later candidates were pre-sampled from q; acceptance then collapses and extra branches yield little gain. ResiSpec replaces the draft q at verification with a proxy k that reserves residual mass aligned with high-density regions of q, claiming exact target sampling without an extra model forward. Empirically, on Llama-2-7B with a 68M draft over CNN/DM, OpenWebText, and C4, it improves accepted length and throughput versus SpecInfer (up to about 1.86× / 1.69×, and up to 1.92× in tree ablations), and can be plugged into Sequoia and EAGLE. The contribution is a verification-time residual-shaping rule for tree/multi-candidate speculative sampling, not a new drafter.

Critical strengths

Clear diagnosis. Residual Drift follows directly from sequential p/q residual updates; Figure 1 shows later candidates’ acceptance falling near zero, motivating the method.

Exactness-aware design. The proxy/residual construction aims to keep sampling lossless while making pre-sampled siblings useful after rejections.

Plug-in framing. Gains appear as an add-on to SpecInfer, Sequoia, and EAGLE, with runtime broken down to show no extra target forward.

Critical weaknesses

Narrow, dated evaluation. Main results use Llama-2-7B (appendix up to ~7B) with tiny drafts on CNN/DM, OpenWebText, and C4—self-reported tok/s on a 3090—not SpeedBench/SpecBench-style suites, chat/code/math tasks, or modern large (let alone trillion-scale) targets.

Weak SOTA coverage. No comparison to recent parallel drafters such as DFlash or DSpark; Medusa/Hydra-style systems are largely absent. Sequoia/EAGLE appear mainly as “+ResiSpec” add-ons, not a full fair bake-off against the strongest current stacks.

Theory of drift is thin. “Later candidates almost always fail” is argued by support mismatch plus Figure 1, not a tight theorem; severity under diverse models, temperatures, and tree shapes is under-proven.

Limited practical relevance to today’s SPD. Many deployments use greedy exact-match acceptance, where this residual-shaping story does not apply; even for sampling SD, evidence stays on small Llama-2-era setups, so claimed speedups may not transfer to current serving.

Questions for authors

See weaknesses above.

Reviewer_MJjXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ResiSpec, a residual distribution shaping method for multi-candidate speculative sampling. The authors identify Residual Drift as a bottleneck in conventional multi-candidate verification, where successive rejections cause the residual target distribution to diverge from the draft distribution, reducing the acceptance probability of subsequent candidates. To address this issue, ResiSpec introduces a proxy distribution to reshape the rejection residual toward the draft distribution. The paper reports up to 1.92× speedup over existing multi-candidate speculative decoding methods and claims to preserve the exact target distribution.

Critical strengths
  1. The proposed approach focuses on the verification stage rather than modifying the draft model or tree construction, making it potentially complementary to existing speculative decoding frameworks.
  2. The reported improvements in acceptance length and throughput are substantial under the evaluated settings.
Critical weaknesses
  1. The claimed lossless sampling property is not theoretically established.

This is my primary concern. The main contribution of this paper relies on preserving the exact target distribution while improving the acceptance rate. However, I believe the proof of Theorem 1 in Appendix A.1 contains a fundamental issue.

Specifically, the proxy distribution kk is constructed based on the rejected candidate x0x_0, meaning that kk depends on the sampling outcome. Nevertheless, the proof in Eqs. (16)–(19) effectively treats kk as a fixed distribution when calculating the marginal probabilities of the acceptance and rejection paths.

The Local Alignment condition k(x0)=q(x0)k(x_0)=q(x_0) only guarantees consistency at the sampled candidate and does not ensure that the modified residual distributions preserve the overall target distribution after marginalizing over all possible rejection events.

In fact, a simple counterexample shows that the stated conditions of Theorem 1 are insufficient. Consider p=(0.3,0.4,0.3)p=(0.3,0.4,0.3) and q=(0.5,0.3,0.2)q=(0.5,0.3,0.2). When the first token is rejected (Z=q1−p1=0.2Z = q_1-p_1=0.2), choosing r=(0,0.12,0.08)r=(0,0.12,0.08) and s=(0.2,0,0)s=(0.2,0,0) satisfies the stated proxy constraints, yielding k=p−r+s=(0.5,0.28,0.22)k=p-r+s=(0.5,0.28,0.22) satisfied all claimed proxy constraints. However, the resulting output distribution for single-candidate sampling becomes Pout=min⁡(p,q)+r=(0.3,0.3,0.2)+(0,0.12,0.08)=(0.3,0.42,0.28)P_{\text{out}}=\min(p, q)+r=(0.3,0.3,0.2)+(0,0.12,0.08)=(0.3,0.42,0.28), which differs from the target distribution pp.

This does not necessarily establish that every possible implementation of ResiSpec is biased, but it demonstrates that the conditions and proof provided in the paper are insufficient to establish the claimed universal exactness.

Furthermore, the near-zero KL divergence reported in Table 6 is calculated using the path-based marginal reconstruction in Appendix B, which relies on the same questionable theoretical argument. Therefore, this experiment does not provide convincing independent evidence of distributional preservation.

  1. Insufficient comparison with existing sampling and verification methods.

The main experiments compare ResiSpec against SpecInfer, with additional compatibility results involving Sequoia and EAGLE. However, these comparisons do not sufficiently establish the advantage of the proposed sampling strategy over existing multi-candidate sampling and verification algorithms.

In particular, prior works such as SpecTr and SpecHub have investigated related problems involving multi-candidate sampling, verification strategies, and acceptance probability optimization.

Since the primary contribution of ResiSpec lies in modifying the sampling and verification procedure, comparisons with these approaches are more directly relevant than comparisons with methods primarily designed to improve draft generation or tree construction.

The paper should better clarify how its residual shaping strategy differs from these prior methods and whether it provides additional benefits under matched candidate sets and verification budgets (the comparison should include end-to-end speedup and output quality preservation, e.g. accuracy of GSM8K).

Questions for authors
  1. How does Theorem 1 account for the dependence of the proxy distribution kk on the rejected candidate? Can the authors provide a rigorous proof that marginalizes over all possible rejection events?
  2. Can the authors independently verify distributional equivalence by explicitly enumerating all possible sampling paths on a small vocabulary, or by estimating the output distribution through actual repeated sampling?
  3. How does ResiSpec compare with existing multi-candidate sampling and verification algorithms, particularly SpecHub and K-SEQ proposed in SpecTr, under matched experimental settings?
  4. Can the authors provide downstream task performance evaluations to demonstrate that the reported speedups do not come at the cost of output quality?
Reviewer_wimfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

his paper studies verification in multi-candidate speculative decoding. The authors observe that, after an early candidate is rejected, the standard residual update can move the target residual away from the original draft distribution, reducing the usefulness of subsequent presampled candidates. They refer to this phenomenon as Residual Drift.

To address this issue, ResiSpec constructs a proxy distribution kk after a candidate is rejected and uses kk in place of the original draft distribution qq to compute the subsequent residual. The construction uses residual-mass reservation, capacity clipping, and mass redistribution, with the requirement that the proxy match the draft probability at the current candidate. The authors claim that this local alignment suffices to preserve exact sampling while improving the acceptance of later candidates.

Experiments cover multiple target–draft model pairs, three text corpora, and different temperatures, and report integrations with SpecInfer, Sequoia, and EAGLE. The largest reported throughput improvement over SpecInfer is 1.92×1.92\times. However, the central distribution-preservation guarantee has the substantive problems described below.

Critical strengths
  1. The paper focuses on a practically meaningful verification-efficiency problem.

    Generating multiple candidates incurs additional verification costs. Whether later candidates remain compatible with the target residual after rejection affects how effectively those costs translate into accepted output. Focusing optimization on verification itself has potential value.

  2. The experiments include useful system-level checks.

    The paper reports acceptance length and actual throughput, examines multiple model pairs and tree topologies, studies integration with existing methods, and provides runtime breakdowns. These dimensions are useful for assessing the practical value of a correct method. However, they cannot support the claimed lossless acceleration until the distribution-preservation issue is resolved.

Critical weaknesses
  1. Theorem 1's local-alignment condition does not guarantee exact sampling.

    Requiring kz(z)=q(z)k_z(z)=q(z) does not ensure global probability-mass balance. Consider

    q=(0.7,0.2,0.1),\qquad k_a=(0.7,0,0.3).$$ This proxy satisfies local alignment and the stated residual/slack constraints. Only candidate $a$ can be rejected, with total probability $0.6$. The normalized residual is $R_a=(0,\frac{1}{3},\frac{2}{3})$, giving $$P_{\mathrm{out}}=\min\{p,q\}+0.6R_a=(0.1,0.4,0.5), \qquad \operatorname{TV}(P_{\mathrm{out}},p)=0.2.$$ Thus, the stated conditions admit a biased output. Appendix A.1 incorrectly treats the candidate-dependent proxy as a common proposal distribution during cancellation.
  2. The zero-KL check does not validate the actual sampler.

    Appendix B reconstructs acceptance mass using min⁡{ki(y),pi(y)}\min\{k_i(y),p_i(y)\}, although candidates are sampled from qq. In the same counterexample, this reconstruction yields

    min⁡{ka,p}+(p−ka)+=p,\min\{k_a,p\}+(p-k_a)_+=p,

    producing zero KL despite the biased actual output above. Table 6 therefore cannot establish unbiased sampling. Independent validation should enumerate the actual sampling branches with their true proposal probabilities or check empirical output frequencies.

Questions for authors
  1. Regarding the central counterexample: Please explain why the three-element counterexample above does not apply to Theorem 1 or Algorithm 1. If a necessary constraint is missing, please state it explicitly and prove that the actual implementation satisfies it. In particular, when candidates are still sampled from qq, how does the proxy-dependent residual satisfy the per-output probability-mass balance condition above?

  2. Regarding the validation in Table 6: Does the table measure the output marginal of the actual sampler, or the proxy-based reconstruction in Appendix B? Please report both distributions for the counterexample above and provide a verifiable validation implementation. If the implementation differs from the pseudocode, please identify the specific differences.

  3. Regarding budget selection and the optimization problem: Theorem 2 optimizes over residuals satisfying 0≤r≤p0\le r\le p and ∑r=Z\sum r=Z, without incorporating the constraints r(x0)=0r(x_0)=0, local alignment, and the slack-support restrictions imposed by the actual construction. Why can strict monotonicity of this relaxed problem justify the budget choice in the actual algorithm? Also, do the experiments use Equation (13) or line 18 of Algorithm 1, and how is normalization of the final proxy always ensured?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.