- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies relaxed speculative decoding, which accelerates LLM generation by accepting draft tokens that standard lossless verification would reject.
It finds that increased acceptance can lead to longer outputs, offsetting the acceleration, and attributes this problem to accumulated drift toward the weaker draft model.
To address this, it proposes RFlash, a training-free verification rule that increases acceptance while constraining a likelihood-ratio measure of drift across each verification block.
Experiments report 6–10% improvement over lossless speculative decoding.
Critical strengths
The experimental results are promising: RFlash reports fewer target forward passes across several models and benchmarks while largely maintaining accuracy.
Figure 2 provides evidence consistent with the proposed drift explanation, but the experiments do not sufficiently isolate whether the performance gains arise specifically from the drift constraint.
Critical weaknesses
-
The writing is repetitive and imprecise.
- Key explanations rely on vague phrases such as “the draft’s own regime,” “divergence spent as drift,” and “the room is across the positions of a block.”
- Meanwhile, the introduction and results repeatedly state the performance claim rather than develop the reasoning behind it.
- The prose also blurs definitions, intuitive explanations, and demonstrated conclusions, forcing readers to reconstruct essential connections. Substantial rewriting is needed to make the argument clear and independently assessable.
-
The mathematical exposition introduces substantial notation and named results without adequate explanation.
- Lines 190–201 invoke the Neyman–Pearson lemma to motivate a likelihood ratio, but do not sufficiently explain why this motivates the subsequently centered drift statistic.
- Lines 203–212 then introduce filtrations, martingales, and the Doob decomposition in rapid succession, without first explaining their purpose in the method.
The reader must infer both what these mathematical objects represent and why they are needed. This obscures the central argument and makes the proposed drift constraint difficult to assess.
Questions for authors
-
Can you rewrite lines 203-212? Do you need that deep math in your paper as a preliminary?
-
Can you explain Figure 2 clearly? None of these notations or explanations make sense.
2 comments
-
Thank you for the clarification and the revisions to Figure 2.
-
My background is in computer science, with limited training in stochastic processes. The relevant concepts should be introduced carefully and coherently, with a clear explanation of their purpose. Mathematical terminology should be used where it supports the argument, and its relevance should be made explicit to the reader.
-
My concern is the clarity and precision of the argument. Describing the material as “ten lines of undergraduate stochastic process” does not address whether the conclusions drawn from it are adequately justified.
After considering the authors’ response and the supplied revision, I maintain my original rating.
Q2: We thank the reviewer for this question, which led to concrete changes. In the revised paper we rewrote the axis labels and the caption of Figure 2, and we moved its introduction later in the text. The figure is no longer referenced from the Introduction. It is first discussed in Section 4.1, after the drift ST and the drift rate δt have been defined, and the curves of our method in panel (a) are discussed at the end of Section 5.2, after the method has been introduced. A reader who reaches the figure now already knows every symbol it uses.
Q1: We do need this paragraph, and we respectfully disagree that it is deep. First, it is ten lines of undergraduate stochastic process, and we see no reason why it should be an obstacle for any reader who works on algorithms. Second, it has a precise purpose: it separates the harmful, persistent drift of lossy speculative decoding from the noise that arises naturally under every rule, lossless included. In ST=MT+∑t≤Tδt, MT is the zero-mean noise and ∑tδt is what a relaxation rule adds, and δt is the quantity our method constrains. The Doob decomposition is exactly the tool for this separation, and it makes the separation visible at a glance.