- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes relaxed acceptance for speculative decoding: three training-free rules that override selected rejections, deliberately trading the distributional guarantee of lossless decoding for additional throughput. A tokenizer-derived whitelist handles surface-form relations, mutual confirmation accepts a draft token when both the target and the draft assign sufficient probability to each other's tokens, and offline order-swap tables cover recurring reorderings. The rules run inside vLLM's rejection sampler without extra model forward passes. With two same-family pairs (Qwen3-14B with a Qwen3-4B AWQ draft, and DeepSeek-R1-Distill-Qwen-32B with a 7B AWQ draft), k = 8, and T in {0, 0.6}, the method reports 3.5%-12.8% additional throughput over lossless speculative decoding across nine benchmarks, with average accuracy differences between 0 and -0.4 points but a substantial regression on ZebraLogic for the DeepSeek pair.
Critical strengths
The approach is cheap to deploy. It requires no new trained components and no additional forward passes, and it is implemented directly in an existing engine's rejection sampler; mutual confirmation yields measurable throughput gains on both evaluated pairs (+7.8%/+3.5% for Qwen and +12.8%/+6.5% for DeepSeek at T=0/0.6 over lossless decoding).
The paper is transparent about which components help and which fail. It states explicitly that the procedure does not retain the distributional guarantee of lossless decoding, reports per-benchmark quality, shows that the draft-side condition matters (a target-only variant is faster but loses 2.9 points on average on greedy math and knowledge tasks, 95% CI [-4.6, -1.2]), reports that order tables contribute little, and identifies the concrete mechanism behind the whitelist failure. These are informative results for anyone designing relaxed verifiers.
Critical weaknesses
Quality preservation is not established. On the DeepSeek pair, the full method loses 8.5 points on ZebraLogic at T=0 and 5.5 points at T=0.6. The whitelist alone loses 11.5 points, and combinations that include it lose 5.5-13.5 points, because the prefix relation rewrites inflected forms inside the reasoning. Quality is measured once per setting, small average differences can hide substantial domain-specific regressions, and not detecting a difference does not establish non-inferiority. The 600-case annotation judges each rejection given the target's actual continuation rather than both counterfactual continuations, so it cannot directly validate the counterfactual definition of a "regrettable" rejection.
The comparative value of the method is not established. Most of the benefit comes from the simple mutual-confirmation rule, while the other components add little or introduce failure modes. Forced acceptance is an intentionally crude reference; without matched implementations and threshold sweeps for competitive relaxed-acceptance methods, the paper does not show a superior speed-quality frontier. The evaluation also covers only two same-family AWQ pairs.
Questions for authors
- What are the per-benchmark paired confidence intervals over repeated runs under a prespecified non-inferiority margin, and what are the complete speed and quality results for the method without the prefix relation, with rules selected on a separate development set?
- How does the method compare with representative relaxed-acceptance baselines implemented in the same engine at matched quality loss, including a sweep of the target-only threshold?
- For the 600 annotated rejections, what fraction of labels remains unchanged when both candidate-conditioned continuations are actually generated and independently assessed?
Response to Reviewer GorL
Thank you for the review. Your point that single runs cannot establish non-inferiority led us to fix a margin in advance and to judge quality with intervals over three runs. Your questions about the 600 labels and the development set also helped us strengthen the evidence.
W1, Q3. Quality preservation and the 600 labels
You are right that single runs cannot establish non-inferiority. We fixed the margin in advance and re-evaluated with intervals paired over problems across three runs (Table 1, Qwen3 pair). We also computed the interval for every benchmark. Of the 18 per-benchmark intervals (nine benchmarks, two temperatures), none lies below zero for ours, for target probability ≥ 0.3, or for the two used together. Two do for target probability ≥ 0.2, and one for mutual confirmation with θq=0. The ZebraLogic loss in the submission was caused by the prefix relation, which we removed.
We also checked the 600 labels directly. For all 600 we generated both continuations to the end, one from the draft's token and one from the target's. Among the cases that can be graded automatically, the final answer was the same in 88% of those labeled regrettable. It was also the same in 84% of those labeled justified, so the labels support only one direction: when a rejection is regrettable, the answer is mostly preserved. We report this as measured in the appendix, and the quality claims rest on the three-run measurements, not on the labels.
W2, Q2. Comparative value
Tables 1 and 2 of the general response are the comparison you asked for. The target-probability threshold was measured at 0.1, 0.2 and 0.3, where 0.1 is mutual confirmation with θq=0. At T=0, our method alone has the throughput of target probability ≥ 0.3 and FSD while preserving accuracy, and added to these rules it is faster within the margin.
Most rescued positions are indeed rescued by mutual confirmation (21.4 of the 30.6 percentage points at T=0). The per-rule table reports the shares of the whitelist and the order-swap tables as measured. That the evaluation covers only two pairs of the same family is a limitation, and we state it.
Q1. Results without the prefix relation, and the development set