acceptodds
Under review as a conference paper at ICLR 2027

RFlash: Relaxed Speculative Decoding with Drift Constraint

Abstract

Speculative decoding losslessly accelerates memory-bound decoding of large language models (LLMs) by using a lightweight draft model to speculate future tokens of the large target model. Relaxed speculative decoding further accelerates decoding by accepting draft tokens that lossless verification would reject. Despite the increase in acceptance length, prior methods often fail to deliver any request-level speedup because they make the output longer. We trace this to a systematic drift toward the weak draft model, which accumulates over the generation and eventually collapses it. To address this, we propose RFlash, a training-free relaxation rule that caps the drift toward the draft, keeping long generations faithful to the target. On long-reasoning benchmarks, RFlash is the only relaxation rule we evaluate that is faster than lossless speculative decoding in every setting, by 6–10%, with output length and accuracy at the lossless level, delivering a practical, accuracy-preserving speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, speculative decoding, inference, acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TDMqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies relaxed speculative decoding, which accelerates LLM generation by accepting draft tokens that standard lossless verification would reject.

It finds that increased acceptance can lead to longer outputs, offsetting the acceleration, and attributes this problem to accumulated drift toward the weaker draft model.

To address this, it proposes RFlash, a training-free verification rule that increases acceptance while constraining a likelihood-ratio measure of drift across each verification block.

Experiments report 6–10% improvement over lossless speculative decoding.

Critical strengths

The experimental results are promising: RFlash reports fewer target forward passes across several models and benchmarks while largely maintaining accuracy.

Figure 2 provides evidence consistent with the proposed drift explanation, but the experiments do not sufficiently isolate whether the performance gains arise specifically from the drift constraint.

Critical weaknesses
  1. The writing is repetitive and imprecise.

    • Key explanations rely on vague phrases such as “the draft’s own regime,” “divergence spent as drift,” and “the room is across the positions of a block.”
    • Meanwhile, the introduction and results repeatedly state the performance claim rather than develop the reasoning behind it.
    • The prose also blurs definitions, intuitive explanations, and demonstrated conclusions, forcing readers to reconstruct essential connections. Substantial rewriting is needed to make the argument clear and independently assessable.
  2. The mathematical exposition introduces substantial notation and named results without adequate explanation.

    • Lines 190–201 invoke the Neyman–Pearson lemma to motivate a likelihood ratio, but do not sufficiently explain why this motivates the subsequently centered drift statistic.
    • Lines 203–212 then introduce filtrations, martingales, and the Doob decomposition in rapid succession, without first explaining their purpose in the method.

The reader must infer both what these mathematical objects represent and why they are needed. This obscures the central argument and makes the proposed drift constraint difficult to assess.

Questions for authors
  1. Can you rewrite lines 203-212? Do you need that deep math in your paper as a preliminary?

  2. Can you explain Figure 2 clearly? None of these notations or explanations make sense.

2 comments

Q2: We thank the reviewer for this question, which led to concrete changes. In the revised paper we rewrote the axis labels and the caption of Figure 2, and we moved its introduction later in the text. The figure is no longer referenced from the Introduction. It is first discussed in Section 4.1, after the drift STS_T and the drift rate δt\delta_t have been defined, and the curves of our method in panel (a) are discussed at the end of Section 5.2, after the method has been introduced. A reader who reaches the figure now already knows every symbol it uses.

Q1: We do need this paragraph, and we respectfully disagree that it is deep. First, it is ten lines of undergraduate stochastic process, and we see no reason why it should be an obstacle for any reader who works on algorithms. Second, it has a precise purpose: it separates the harmful, persistent drift of lossy speculative decoding from the noise that arises naturally under every rule, lossless included. In ST=MT+∑t≤TδtS_T = M_T + \sum_{t \le T} \delta_t, MTM_T is the zero-mean noise and ∑tδt\sum_t \delta_t is what a relaxation rule adds, and δt\delta_t is the quantity our method constrains. The Doob decomposition is exactly the tool for this separation, and it makes the separation visible at a glance.

  1. Thank you for the clarification and the revisions to Figure 2.

  2. My background is in computer science, with limited training in stochastic processes. The relevant concepts should be introduced carefully and coherently, with a clear explanation of their purpose. Mathematical terminology should be used where it supports the argument, and its relevance should be made explicit to the reader.

  3. My concern is the clarity and precision of the argument. Describing the material as “ten lines of undergraduate stochastic process” does not address whether the conclusions drawn from it are adequately justified.

After considering the authors’ response and the supplied revision, I maintain my original rating.

Reviewer_qmfwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2(+1)
Read review
Paper summary

RFlash relaxes speculative decoding by modifying the verification distribution under a drift constraint, seeking longer accepted blocks while limiting changes to the target model. It derives a tilted distribution with a constraint multiplier and implements a short fixed-point procedure. Experiments on mathematical reasoning, science QA, and code generation compare task accuracy and the number of target forward passes.

Critical strengths
  1. The explicit trade-off between acceptance and controlled distributional drift provides more insight than an unrestricted approximate verifier. The optimization formulation provides a useful way to study relaxed speculative decoding.
  2. The evaluation reports both accuracy and decoding-work proxies across three demanding task families, rather than relying only on accepted length.
Critical weaknesses
  1. The stated drift guarantee is conditional on constraint satisfaction, but Appendix A.1.5 allows accepting a constraint residual when no multiplier root is found. The implementation also uses a small number of fixed-point passes with approximate reach weights. The analysis therefore does not establish that the executed procedure satisfies the condition required by the theorem; the frequency and magnitude of violations are decision-critical.
  2. The reported request-level speedup is based on target forward-pass counts rather than end-to-end latency. Hyperparameters are also selected using the evaluated benchmarks subject to an accuracy-loss threshold. For the relatively modest gains, this leaves the practical speed–quality trade-off on an independent workload insufficiently established.
Questions for authors
  1. What fraction of blocks lack a feasible root or violate the drift constraint after the implemented iterations, and how are those blocks handled in the reported results?
  2. With hyperparameters frozen on separate calibration data, what are the measured end-to-end latency and accuracy changes?
1 comment

We thank the reviewer for the precise questions. Both led to new measurements, which are in the revised PDF.

Q1: how often the constraint is violated, and how they affect the result

The violation of no-root blocks In every reported result, a block without a root is decoded with the solution at the upper end of the search interval, and the residual of the constraint is accepted. However, the no-root blocks are rare, and they show negligible effect on the result. Appendix A.1.5 now reports how often they occur. On 24,700 verification blocks (Qwen3.8 27B in BF16 with its DFlash 2 drafter, GPQA Diamond prompts at λ=1.5\lambda = 1.5, α=0.5\alpha = 0.5:)

  • No root is found on 0.55% of the blocks (135). The constraint holds on the other 99.45% (11.6% slack, 87.9% binding).

  • Over λ∈{0.6,1.5,3}\lambda \in \{0.6, 1.5, 3\} and α∈{0.25,0.5,1}\alpha \in \{0.25, 0.5, 1\} the fraction ranges from 0.17% to 1.1%.

In fact, accepting the residual on no-root blocks is merely a design choice. Even if we simply fall back to lossless decoding on these blocks, the main conclusion doesn't change at all. On the same 24,700 blocks we computed the alternative design, which verifies these blocks losslessly so that the constraint holds on every block:

Keep the solution (as in RFlash)Fall back to lossless
Blocks that violate the constraint0.55%0%
Average acceptance length over all blocks4.10854.1087

The fallback shows no measurable difference. In conclusion, while the no-root violation is a valid observation, we believe it is a secondary issue that does not affect the soundness of the main method.

The gap between the executed procedure and the theoretical constraint. The reviewer is right that the executed procedure does not enforce the theoretical constraint exactly: for computational efficiency, it runs only a few fixed-point passes. This is not a critical issue, for two reasons.

  1. The fixed point has converged after these passes.
  2. More importantly, Figure 2a (end of Section 5.2 in the revised paper) shows that, empirically, the executed procedure of RFlash does control the drift. On LiveCodeBench, the drift of the generated text with α=0\alpha = 0 stays inside the lossless band over 12,800 tokens. With α=1\alpha = 1 it reaches 1,511 nats, against 4,132 for Cactus at the same acceptance gain.

Q2: end-to-end latency and accuracy with hyperparameters frozen on other data

Latency

The number of target forward passes is a faithful proxy for end-to-end time, and we have now measured the time itself.

Section 6.4 and Table 2(a) now report wall-clock time: Qwen3.8 27B (NVFP4) with its DFlash 2 drafter under SGLang, batch size 1, one NVIDIA GB300 with fixed GPU clocks, HumanEval (164 problems, 3 seeds, 492 paired requests). The time is the request latency and includes drafting, verification and the solver.

LosslessRFlashRatio
Acceptance length4.004.441.11×
Latency per verify step (ms)11.4611.441.00×
Latency per generated token (ms)2.602.330.896× [0.888x, 0.905]
Total time for the benchmark (min)61.355.30.902×

As shown in the table, the latency per verify step is unchanged with and without RFlash (11.46 vs. 11.44 ms), indicating that the solver adds no measurable overhead. The forward-pass count is therefore a faithful proxy for wall-clock latency.

Other tasks with fixed parameters (Table 2b in revised paper).

We tuned λ\lambda and α\alpha on LiveCodeBench, GPQA Diamond and AIME because these three long-reasoning tasks show the problem most clearly. We then froze the parameters tuned on these three tasks and ran additional experiments on two other tasks, GSM8K and HumanEval. AL and Acc are lossless → RFlash, Len is the output length relative to lossless, Spd. is the ratio of target forward passes on paired requests. GSM8K uses one seed and HumanEval ten

ModelTaskALLenSpd.Acc
Qwen3.8 27BGSM8K4.40 → 5.050.86×1.33×97.6 → 97.2
Qwen3.8 27BHumanEval3.93 → 4.481.03×1.10×98.7 → 98.1
Gemma-4 12BGSM8K4.88 → 5.240.90×1.19×96.7 → 97.0
Gemma-4 12BHumanEval5.42 → 5.770.99×1.08×99.6 → 99.6
Reviewer_weimOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies relaxed speculative decoding, where additional draft tokens are accepted at the cost of changing the target’s output distribution. The authors observe that increased acceptance can be offset by longer generations. They propose RFlash, which optimizes acceptance while constraining a statistic measuring drift toward the drafter. The experiments report approximately 6–10% improvements over lossless speculative decoding in target-forward-pass counts, with comparable accuracy and output length.

Critical strengths

a) The paper identifies a useful limitation of evaluating relaxed speculative decoding through acceptance length alone. Accounting for the total work needed to complete a response gives a more meaningful assessment of efficiency.

b) The proposed approach is training-free and explicitly addresses the direction of distributional changes. The drift statistic provides an interpretable way to examine how relaxed verification changes generation behavior.

c) The experiments cover several reasoning benchmarks, target models, and drafter designs. The comparisons across operating points and the separate reporting of acceptance, output length, and accuracy help assess the tradeoff.

Critical weaknesses

w1) The main drift guarantee appears to have a technical issue. In Proposition 9, the proof conditions on the sampled draft proposals but then uses acceptance probabilities averaged over those proposals. Once a proposal is fixed, its acceptance probability is token-specific, and the emitted token does not generally have the claimed modified-target distribution under that conditioning. Moreover, the joint block optimization allows an earlier modified distribution to depend on later drafted prefixes. The authors should clarify how the claimed block-level guarantee accounts for this dependence.

w2) The implementation does not always enforce the stated constraint. Appendix A.1.5 states that, for positive drift allowance, the solver accepts a residual constraint violation if no multiplier root is found. Since the main experiments use positive drift allowances, the frequency and magnitude of these violations should be reported, along with their implications for the guarantee.

w3) The reported speedups are based on forward-pass counts, rather than elapsed time. This metric usefully accounts for output-length changes, but does not establish the claimed practical latency improvement. Reporting end-to-end timing that includes drafting, verification, and the complete iterative solver would strengthen the results.

Questions for authors

See comments in Weaknesses. Please address those.

2 comments

We thank the reviewer for the careful reading of Proposition 9 and for raising this point. All three weaknesses led to changes in the revised PDF.

W1: Proposition 9 and the dependence on later proposals

The reviewer pointed out a real problem on the proof of Proposition 9: the target distribution of position tt depends on the realized token of position t−1t-1, so the rigorous block-wise program should use the accept probability of sampled draft tokens instead of taking the average. The implemented program in RFlash is in fact a surrogate.

What we changed. We admit that the theoretical guarantee of the realized block-wise program is overstated, so we removed Proposition 9 and the paragraph of Section 5.2 that claimed a block-level guarantee.

Evidence on the realized drift.

However, we argue that this does not hurt the soundness of the main method. In practice, it controls the drift essentially as well as a rigorous block-wise program would.

Figure 2a, now discussed at the end of Section 5.2, plots STS_T of the text RFlash actually generates on LiveCodeBench. With α=0\alpha = 0 it stays inside the lossless band over 12,800 tokens. With α=1\alpha = 1 it reaches 1,511 nats, against 4,132 for Cactus at the same length and the same acceptance gain. Therefore, we claim that, empirically, the RFlash algorithm we implemented does control the drift, even though mathematical details in the block-wise program proof do not necessarily hold.

W2: residual of the constraint when no root is found (Appendix A.1.5 in revised paper)

This is a merely a design choice: no-root blocks are rare. Appendix A.1.5 now reports how often they occur. On 24,700 verification blocks (Qwen3.8 27B in BF16 with its DFlash 2 drafter, GPQA Diamond prompts at λ=1.5\lambda = 1.5, α=0.5\alpha = 0.5:)

  • No root is found on 0.55% of the blocks (135). The constraint holds on the other 99.45% (11.6% slack, 87.9% binding).

  • Over λ∈0.6,1.5,3\lambda \in \\{0.6, 1.5, 3\\} and α∈0.25,0.5,1\alpha \in \\{0.25, 0.5, 1\\} the fraction ranges from 0.17% to 1.1%.

Additionally, the results shows no measurable change even if we fall back to lossless decoding on these no-root blocks. In the reported results the no-root blocks keep the solution at the end of the search interval. On the same 24,700 blocks we computed the alternative design, which verifies these blocks losslessly so that the constraint holds on every block:

Keep the solution (as in RFlash)Fall back to lossless
Blocks that violate the constraint0.55%0%
Average acceptance length over all blocks4.10854.1087

The fallback shows no measurable difference. In conclusion, while the no-root violation is a valid observation, we believe that accepting the residual is a secondary issue and does not hurt the soundness of the main method.

W3: end-to-end time

The number of target forward passes is a faithful proxy for end-to-end time, and we have now measured the time itself.

Section 6.4 and Table 2(a) now report wall-clock time: Qwen3.8 27B (NVFP4) with its DFlash 2 drafter under SGLang, batch size 1, one NVIDIA GB300 with fixed GPU clocks, HumanEval (164 problems, 3 seeds, 492 paired requests). The time is the request latency and includes drafting, verification and the solver.

LosslessRFlashRatio
Acceptance length4.004.441.11×
Latency per verify step (ms)11.4611.441.00×
Latency per generated token (ms)2.602.330.896× [0.888x, 0.905]
Total time for the benchmark (min)61.355.30.902×

The latency per verify step is the same with and without RFlash, so the solver adds no measurable time, and the gain in acceptance length is the gain in wall-clock time. The per-token ratio is the geometric mean of the paired per-request ratios with its 95% bootstrap interval.

Dear Reviewer weim,

As the discussion period closes in a few hours, we would like to briefly check whether our response above addresses your concerns. In short:

  • W1: As you correctly pointed out, the proof of Proposition 9 has a gap. We removed Proposition 9 and the block-level guarantee claim from Section 5.2, and instead show empirically that the implemented RFlash controls the drift (Figure 2a: 1,511 vs. 4,132 nats for Cactus at the same acceptance gain).

  • W2: Appendix A.1.5 now shows that no-root blocks occur on 0.55% of blocks; falling back to lossless on them changes the average acceptance length by less than 0.01%.

  • W3: Section 6.4 and Table 2(a) now report wall-clock time: 0.896× per-token latency vs. lossless, with no measurable solver overhead.

If these changes resolve your concerns, we would appreciate it if you could consider updating your assessment. We are happy to clarify anything further before the deadline.

Thank you again for your careful reading of Proposition 9.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.