acceptodds
Under review as a conference paper at ICLR 2027

Rescuing Regrettable Rejections: Relaxed Acceptance for Speculative Decoding

Abstract

Speculative decoding has become one of the most effective techniques for accelerating inference in large language models. Its defining property is that it is lossless: the output distribution is identical to that of the target model, so the speedup comes at no cost in output quality. In speculative decoding, a small draft model proposes several tokens, the larger target model verifies them in parallel, and decoding resumes from the first proposal that fails verification. Every draft token after the rejected one is discarded, and the more tokens are discarded, the smaller the speedup becomes. We examine these rejections and find that many of them are regrettable. A rejection is regrettable when accepting the draft token would not have affected the quality of the output. We propose a method that rescues such rejections without training any additional model or adapter. It targets the three types of regrettable rejection that we identify, surface form, preference, and order, with a whitelist of interchangeable tokens, mutual confirmation between the two models, and order-swap tables, respectively. Experiments on Qwen3-14B and DeepSeek-R1-Distill-Qwen-32B, each drafted by a smaller 4-bit quantized model of the same family, show that the method rescues 30–36% of the rejections of lossless speculative decoding at temperature 0 and 14–15% at temperature 0.6, which raises throughput by a further 8–13% and 3.5–6.5%, respectively, and lifts the speedup over the target alone to 2.1–2.4×, with no forward pass added and both models left untouched. Mean accuracy stays within 0.4 points of lossless decoding; the one clear loss, on a single benchmark, is traced to one rule.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, relaxed acceptance, lossy speculative decoding, LLM inference acceleration, training-free, quantized draft model, verification, rejection analysis

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_GorLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes relaxed acceptance for speculative decoding: three training-free rules that override selected rejections, deliberately trading the distributional guarantee of lossless decoding for additional throughput. A tokenizer-derived whitelist handles surface-form relations, mutual confirmation accepts a draft token when both the target and the draft assign sufficient probability to each other's tokens, and offline order-swap tables cover recurring reorderings. The rules run inside vLLM's rejection sampler without extra model forward passes. With two same-family pairs (Qwen3-14B with a Qwen3-4B AWQ draft, and DeepSeek-R1-Distill-Qwen-32B with a 7B AWQ draft), k = 8, and T in {0, 0.6}, the method reports 3.5%-12.8% additional throughput over lossless speculative decoding across nine benchmarks, with average accuracy differences between 0 and -0.4 points but a substantial regression on ZebraLogic for the DeepSeek pair.

Critical strengths

The approach is cheap to deploy. It requires no new trained components and no additional forward passes, and it is implemented directly in an existing engine's rejection sampler; mutual confirmation yields measurable throughput gains on both evaluated pairs (+7.8%/+3.5% for Qwen and +12.8%/+6.5% for DeepSeek at T=0/0.6 over lossless decoding).

The paper is transparent about which components help and which fail. It states explicitly that the procedure does not retain the distributional guarantee of lossless decoding, reports per-benchmark quality, shows that the draft-side condition matters (a target-only variant is faster but loses 2.9 points on average on greedy math and knowledge tasks, 95% CI [-4.6, -1.2]), reports that order tables contribute little, and identifies the concrete mechanism behind the whitelist failure. These are informative results for anyone designing relaxed verifiers.

Critical weaknesses

Quality preservation is not established. On the DeepSeek pair, the full method loses 8.5 points on ZebraLogic at T=0 and 5.5 points at T=0.6. The whitelist alone loses 11.5 points, and combinations that include it lose 5.5-13.5 points, because the prefix relation rewrites inflected forms inside the reasoning. Quality is measured once per setting, small average differences can hide substantial domain-specific regressions, and not detecting a difference does not establish non-inferiority. The 600-case annotation judges each rejection given the target's actual continuation rather than both counterfactual continuations, so it cannot directly validate the counterfactual definition of a "regrettable" rejection.

The comparative value of the method is not established. Most of the benefit comes from the simple mutual-confirmation rule, while the other components add little or introduce failure modes. Forced acceptance is an intentionally crude reference; without matched implementations and threshold sweeps for competitive relaxed-acceptance methods, the paper does not show a superior speed-quality frontier. The evaluation also covers only two same-family AWQ pairs.

Questions for authors
  1. What are the per-benchmark paired confidence intervals over repeated runs under a prespecified non-inferiority margin, and what are the complete speed and quality results for the method without the prefix relation, with rules selected on a separate development set?
  2. How does the method compare with representative relaxed-acceptance baselines implemented in the same engine at matched quality loss, including a sweep of the target-only threshold?
  3. For the 600 annotated rejections, what fraction of labels remains unchanged when both candidate-conditioned continuations are actually generated and independently assessed?
1 comment
AuthorsOpenReview ↗

Response to Reviewer GorL

Thank you for the review. Your point that single runs cannot establish non-inferiority led us to fix a margin in advance and to judge quality with intervals over three runs. Your questions about the 600 labels and the development set also helped us strengthen the evidence.

W1, Q3. Quality preservation and the 600 labels

You are right that single runs cannot establish non-inferiority. We fixed the margin in advance and re-evaluated with intervals paired over problems across three runs (Table 1, Qwen3 pair). We also computed the interval for every benchmark. Of the 18 per-benchmark intervals (nine benchmarks, two temperatures), none lies below zero for ours, for target probability ≥ 0.3, or for the two used together. Two do for target probability ≥ 0.2, and one for mutual confirmation with θq=0. The ZebraLogic loss in the submission was caused by the prefix relation, which we removed.

We also checked the 600 labels directly. For all 600 we generated both continuations to the end, one from the draft's token and one from the target's. Among the cases that can be graded automatically, the final answer was the same in 88% of those labeled regrettable. It was also the same in 84% of those labeled justified, so the labels support only one direction: when a rejection is regrettable, the answer is mostly preserved. We report this as measured in the appendix, and the quality claims rest on the three-run measurements, not on the labels.

W2, Q2. Comparative value

Tables 1 and 2 of the general response are the comparison you asked for. The target-probability threshold was measured at 0.1, 0.2 and 0.3, where 0.1 is mutual confirmation with θq=0. At T=0, our method alone has the throughput of target probability ≥ 0.3 and FSD while preserving accuracy, and added to these rules it is faster within the margin.

Most rescued positions are indeed rescued by mutual confirmation (21.4 of the 30.6 percentage points at T=0). The per-rule table reports the shares of the whitelist and the order-swap tables as measured. That the evaluation covers only two pairs of the same family is a limitation, and we state it.

Q1. Results without the prefix relation, and the development set

  • Without the prefix relation. Quality is the first row of Table 1, and per-benchmark intervals are in the appendix. Throughput over lossless decoding is +7.8% at T=0 and +3.2% at T=0.6.
  • Development set. The thresholds were chosen on the judged sample, which comes from a different model pair (Qwen3-4B with its AWQ copy). The problems from which this sample was drawn overlap with 6.7% of the evaluation problems. Excluding them leaves the verdicts unchanged (+1.15 [+0.22, +2.07] at T=0, +0.05 [−0.66, +0.75] at T=0.6).
  • Repeats. At T=0, each repeat uses a freshly built compilation cache so that the numerical path differs. The three repeats gave two distinct paths.
Reviewer_kLFgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors find that many rejections of lossless speculative decoding are regrettable: accepting the draft token would not have affected the quality of the output. They rescue three types, surface form, preference, and order, with a whitelist of interchangeable tokens, mutual confirmation between the two models, and order-swap tables, without training, extra forward passes, or changes to either model.

Critical strengths
  1. A clear analysis of why rejections happen. The paper classifies regrettable rejections into three intuitive types (surface form, preference, and order) and matches each with a simple rule that uses only information already available at verification time.

  2. A practical, training-free method with real-engine measurements. The rules need no training, no extra forward pass, and no change to either model, and they run inside the vLLM verification kernel, so the throughput gains are measured in a real serving engine rather than a research prototype.

Critical weaknesses
  1. No comparison with other relaxed-acceptance methods The paper compares only against lossless decoding and a forced-acceptance reference, which accepts every rejection and is not a realistic competitor. It does not compare against existing relaxed verifiers that target the same problem: training-free rules such as typical acceptance, SpecDec top-β, FSD, FLy, and CSD, or lightweight learned judges such as AutoJudge.

  2. The rules are hand-crafted, prone to systematic errors, and tied to one tokenizer. (1) Heavy manual design. Acceptance depends on curated token sets, hand-written exclusion lists, code-specific switches, and thresholds tuned on labels from an LLM judge. Each choice is a manual decision rather than a principled criterion, so the method's behavior is hard to predict outside the tested settings. (2) Systematic errors. Because the rules are applied regardless of context, their mistakes repeat across a whole task rather than occurring at random. (3) Dependence on the vocabulary and model family. The whitelist (628 curated tokens, 17,127 trimmed-text groups) is built from the Qwen3 tokenizer, and both model pairs are Qwen-derived. Other families split words differently (for example, Llama 3.1 uses a different BPE vocabulary), so the surface-form and prefix relations, exclusion lists, and order-swap tables would all change, and the order-swap tables must be rebuilt for every model pair and domain. The generality of the method is therefore not demonstrated.

Questions for authors
  1. How does the method compare with typical acceptance, SpecDec top-β, FSD, FLy, CSD, and AutoJudge in the same vLLM setup, at matched accuracy or matched throughput?

  2. Can you evaluate on a non-Qwen model family with a different tokenizer (for example, Llama 3.1 or Gemma)? How many manual changes to the token sets, exclusion lists, and thresholds were needed, and do the gains and accuracy hold?

  3. Beyond ZebraLogic, how often does each rule accept a token that changes the final answer? Please report, per rule and per benchmark, the share of rescued positions that lead to a different final answer than lossless decoding.

1 comment
AuthorsOpenReview ↗

Response to Reviewer kLFg

Thank you for the review. Your request for a comparison in the same engine at matched accuracy and matched throughput, and your question about answer-change rates, gave the comparison clear criteria and led to the per-rule answer-change analysis, which strengthened the work.

W1, Q1. Comparison in the same engine at matched accuracy or matched throughput

We added every rule that can run in the same engine (Table 1, Qwen3 pair). FLy, CSD and AutoJudge require long drafts, task-specific calibration data, and a trained judge, respectively, so we could not include them. The results at T=0, by your two criteria:

  • Throughput at matched accuracy (passing the margin). Target probability ≥ 0.3: +7.7%; ours: +7.8%; FSD (JS 0.1): +7.9%; top-β with β=2 (a setting different from the β=3 of Table 1): +9.2%; FSD + ours: +10.0%; probability ratio ≥ 0.5: +11.7%; target probability ≥ 0.3 + ours: +12.5%.
  • Accuracy at matched throughput. The next higher in throughput, target probability ≥ 0.2 (+14.0%), fails the margin and is 0.94 points less accurate than target probability ≥ 0.3 + ours (+12.5%) ([0.06, 1.81]).

At T=0, our method alone has the throughput of target probability ≥ 0.3 and FSD. Added to them, it is faster.

W2-(1). Manual design

The parts set by hand are the two thresholds of mutual confirmation, seven curated sets in the whitelist, the exclusion list (numbers, operators, negations), and one setting that separates text from code. The order-swap tables are built automatically from five parameters set once (Section 3.4). The thresholds were chosen once on the judged sample from a different model pair and used unchanged. Most rescued positions are rescued by mutual confirmation, which has only these two thresholds (21.4 of the 30.6 percentage points at T=0).

W2-(2), Q3. Systematic errors and answer changes

ZebraLogic was an example of the systematic error you describe, and we removed the prefix relation that caused it. To see whether the remaining rules make such errors, we counted, at T=0, the problems whose final answer differs from that of lossless decoding, and the direction of the change. Several positions are rescued within one problem and the generation diverges after the first, so we count problems rather than positions.

ConfigurationProblems with a different final answercorrect→wrong vs. wrong→correct
no rule (target alone vs. lossless) ‡25.4%106 vs. 109
mutual confirmation only (three runs)29.4–29.9%129 vs. 130, 115 vs. 155, 111 vs. 158
whitelist only (three runs)27.2–27.9%120 vs. 132, 113 vs. 135, 113 vs. 142
order-swap tables only (three runs)26.4–27.7%126 vs. 127, 125 vs. 121, 121 vs. 133
ours (three runs)31.7–32.5%140 vs. 148, 142 vs. 178, 133 vs. 175
forced acceptance (m=8) ‡53.1%414 vs. 149

‡ Single run from the submission.

The rules increase the number of problems whose answer changes, but the changes are not biased toward errors. Per-benchmark tables go into the appendix.

W2-(3), Q2. Vocabulary and model family

It is true that both pairs are Qwen-derived. Mutual confirmation uses only probabilities under the two distributions and does not depend on the vocabulary. The whitelist and the order-swap tables are generated by scripts from the tokenizer and from generation traces, and a new model pair needs only one more input set by hand, the name of the end-of-sequence token. We have not shown that the same thresholds preserve quality on another model family, and we state this as a limitation.

Reviewer_ceG2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates regrettable rejections in speculative decoding, where draft tokens are rejected despite being potentially acceptable alternatives to the target model's predictions. The authors categorize these rejections into three types: surface-form, preference, and order differences. They propose three training-free mechanisms to recover such tokens: a whitelist of interchangeable tokens, mutual confirmation using probabilities from both models, and offline order-swap tables.

The method is implemented in vLLM and evaluated on two model pairs across nine benchmarks. Experimental results show throughput improvements of 7.8–12.8% under greedy decoding and 3.5–6.5% under sampling over lossless speculative decoding, while maintaining similar average benchmark accuracy.

Critical strengths

The proposed acceptance rules can be integrated into the verification stage without additional model training or target forward passes.

Critical weaknesses
  1. Insufficiently supported motivation and questionable methodological design. The classification into three types of regrettable rejections appears largely heuristic, derived from simple observations without sufficient theoretical or empirical justification for their representativeness or completeness. Several acceptance rules also lack sound justification. For example, the whitelist treats "a" and "an" as interchangeable despite their different grammatical constraints, while prefix matching can accept semantically different tokens such as "the" and "their". Furthermore, the necessity of mutual confirmation is questionable: under sampling, removing the draft-side condition increases throughput gains from 2.0% to 13.8% without a measurable reduction in average accuracy. These observations raise concerns about the validity and necessity of the proposed mechanisms.

  2. Missing competitive baselines. The paper provides no direct comparison with closely related relaxed acceptance methods, including FSD, FLy, CSD, Judge Decoding, and Medusa-style typical acceptance. Given the substantial body of related work, it is puzzling that the authors do not compare against any of these methods. Moreover, the proposed acceptance mechanism achieves only 3.5–12.8% additional throughput over lossless speculative decoding, with particularly modest gains under sampling. Without comparisons against existing methods under matched settings, it is difficult to assess whether these improvements offer a meaningful advantage over established approaches.

Questions for authors
  1. What is the practical benefit of the draft-side condition in mutual confirmation? According to Table 3, removing this condition increases throughput gains from 2.0% to 13.8% under sampling, without a measurable reduction in average accuracy. What evidence justifies retaining this condition in the proposed method?

  2. Could the authors provide some examples where interchangeable tokens (e.g., "a" and "an") or prefix-matched tokens (e.g., "the" and "their") can be safely substituted without affecting grammatical correctness or semantic meaning? Under what conditions are such substitutions considered valid, given their strong dependence on context?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ceG2

Thank you for the detailed review. The points you raised (the basis of the three types, the examples, the draft-side condition, and the baselines) were all places where the submission lacked evidence. Following them, we removed the prefix relation from the method, checked the judged sample by actual generation, and compared with existing rules in the same engine. The work is considerably better supported as a result.

W1. Basis of the three types

It is true that the classification came from observation. We checked it on the judged sample, which comes from a model pair different from those of the main experiments. Of the regrettable rejections in this sample, 1.3% fell outside the three types. For every rejection in the sample we also generated both continuations to the end, one from the draft's token and one from the target's. Among the cases labeled regrettable that can be graded automatically, the two continuations gave the same final answer in 88%.

W1, Q2. "a" and "an"

A rescue changes one token, and the next word is then generated to agree with the accepted article. For example, "at a mixer" becomes "at an arcane mixer", and "an infinite sequence" becomes "a sequence". We checked all 43 problems at T=0 in which the first divergence from the lossless output is an a/an swap. In none of them did the article disagree with the following word, and in 38 the correctness of the final answer was unchanged.

W1, Q2. "the" and "their"

You are right. This swap was accepted by the prefix relation. It turned "their" into "the" and changed the meaning, which caused the ZebraLogic loss. We removed the prefix relation from the method, and the revision analyzes that failure in Section 4.3.1 and the appendix.

W1, Q1. Is the draft-side condition needed?

Yes. Over three runs, mutual confirmation with θq=0 fails the margin at both temperatures (Table 1, Qwen3 pair). Mutual confirmation alone is more accurate with the draft-side condition than without it, by 1.49 points at T=0 ([0.56, 2.41]) and by 1.04 points at T=0.6 ([0.31, 1.76]). The "no change in accuracy" in Table 3 of the submission came from a single run.

The throughput gained by dropping the condition comes with this accuracy loss. To gain throughput within the margin, our method can be added to target probability ≥ 0.3 instead of dropping the condition. At T=0.6 this raises throughput from +9.8% to +12.2% and passes the margin (Table 2).

W2. Baselines and the size of the gain

Agreed. We implemented every rule that decides from the two distributions at the rejected position alone in the same verification kernel (Table 1).

  • FSD. FSD (JS 0.1) passes the margin at both temperatures. At T=0 its accuracy cannot be distinguished from ours (paired difference −0.37 [−1.21, +0.48]), and its throughput is level with ours (+7.9% and +7.8%). Each rule rescues positions that the other does not. Used together at T=0, they raise the rescued share from 28.5% to 41.7%, and the throughput becomes +10.0% (Table 2).
  • Typical acceptance and top-β. At T=0 their accuracy is 2.27 and 1.17 points below ours, and the 95% intervals of both differences lie below zero.
  • FLy, CSD, Judge Decoding. FLy needs a window of six positions after a mismatch and assumes long drafts (15 or 25 tokens per round); our setting uses 8. CSD needs task-specific calibration data and Judge Decoding a trained judge, so we could not compare under the same conditions.
  • Size of the gain. The throughput gain is relative to lossless speculative decoding, which is already about 2× the target alone. At T=0, forced acceptance of every rejection (m=8) gives +37.6% but loses 8.2 accuracy points. At the same temperature, our method gives +7.8% alone and +12.5% together with target probability ≥ 0.3, both within the margin.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts. However, I remain unconvinced by the justification for the proposed relaxed acceptance rules. For example, changing "at a mixer" to "at an arcane mixer" clearly alters the semantic content and introduces additional tokens. Such substitutions may improve token-level acceptance or throughput metrics without necessarily reducing the latency of producing semantically equivalent outputs.

Furthermore, the new comparison with FSD shows nearly identical throughput improvements, leaving the practical advantage of the proposed method unclear. While the additional experiments strengthen the evaluation, my main concerns regarding methodological justification, novelty, and practical significance remain.

AuthorsOpenReview ↗

Response to Reviewer ceG2's follow-up comment

Thank you for reading our response and for the further comments.

1. The example

The word "arcane" is the problem's own. At this position the rule accepted only the article "an".

  • The problem reads "Six witches and ten sorcerers are at an arcane mixer". We should have quoted this context.
  • The lossless output shortens the phrase to "a mixer", and the other output keeps it.
  • Both then solve the problem in the same two steps and give the same answer, 60, in 297 and 296 tokens respectively.

2. Additional tokens and latency

On the Qwen3 pair, the outputs do not become longer, so the gain in tokens per second is a gain in time.

  • Over its 3,262 problems, our outputs have 1.0–2.8% fewer tokens than the lossless outputs in each of the six runs (three per temperature).
  • In the same runs (batch size 128), generating all answers took 5.1–10.7% less time, with accuracy within the margin.

We had not reported output length, and we thank you for raising the point.

3. Methodological justification

Our aim (Section 3.1) is to accept a rejected token only when this changes neither the final answer nor the quality of the output. The wording may differ, as in the example.

Each rule accepts only when there is specific evidence that the rejection is not a correction:

  • Whitelist. The two tokens are the same word with different spacing, or are on a short list of interchangeable tokens.
  • Mutual confirmation. Each model gives the other's token substantial probability.
  • Order-swap tables. The target was observed, offline, to emit the draft's token a few tokens later.

No rule accepts a number, an operator, or a negation.

We check empirically whether this evidence suffices, using the accuracy of the final answers over three runs against a margin fixed in advance.

The evidence does matter. Without the draft-side condition of mutual confirmation, the same rule is 1.49 points less accurate at T=0 and fails the margin at both temperatures.

This support is empirical, not a guarantee. Where our reasoning failed (the prefix relation that you pointed out), we removed the rule.

4. Novelty and practical significance (FSD)

As you note, our method alone gains about as much throughput as FSD at T=0, and we do not claim that it is faster. What it offers beyond this is a different basis of acceptance, so that the two can be used together.

  • FSD requires the two distributions to be close. Our rules look at how the two tokens relate. They include a test on the draft's side (q(t) ≥ θq) that, to our knowledge, existing rules do not use.
  • At T=0, 13.6% of the rejected positions are accepted by our method and not by FSD.
  • Adding our method to FSD raises the throughput gain from +7.8% to +10.0% at T=0 and from +8.2% to +10.1% at T=0.6. Both are within the margin, and the method needs no training and no extra forward pass.

Thank you again for your time and for the careful comments throughout the discussion. They have helped us make the paper clearer.

Reviewer_6F1KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes relaxed acceptance rules for speculative decoding to recover rejected draft tokens that may not hurt output quality. The method combines token whitelists, mutual confirmation between the draft and target models, and order-swap tables, without additional training or model forward passes. Experiments on two model pairs show approximately 3.5%–12.8% higher throughput than lossless speculative decoding, with small changes in average benchmark accuracy.

Critical strengths
  1. Practical motivation and low implementation overhead. The paper addresses a useful question: whether some distribution-preserving rejections can be avoided without meaningfully hurting task performance. The proposed rules require neither additional training nor extra model forward passes, making them relatively straightforward to integrate into existing speculative decoding systems. The measured throughput gains suggest that this direction has practical value.
  2. Useful ablations and transparent failure analysis. The paper includes rule-level ablations that help explain where the gains come from. The authors also report a concrete failure case and trace it to the prefix rule in the whitelist. This analysis is useful because it exposes a limitation of the method rather than relying solely on favorable aggregate results.
Critical weaknesses
  1. The advantage over simpler relaxed acceptance methods is not established. The main comparisons are against lossless decoding and forced acceptance, rather than relevant relaxed acceptance methods. In Table 2, target-side-only acceptance achieves a 13.8% throughput gain at T=0.6 on Qwen3, versus 3.5% for the full method, while both show a 0.0-point change in mean accuracy. Although the draft-side condition protects some tasks at T=0, the paper does not show whether a more conservative target-side-only threshold could achieve similar quality at a higher speed. Without threshold sweeps and comparisons at matched quality, the need for the additional rules remains unclear.
  2. The quality-preservation claim is not sufficiently supported. For the DeepSeek pair, ZebraLogic accuracy falls from 67.5% to 59.0% at T=0 and 62.0% at T=0.6, corresponding to losses of 8.5 and 5.5 percentage points. Gains on other tasks offset these losses in the overall average, but do not make the task-specific regression less consequential. Quality evaluation is run once per temperature, and the observed target-only versus lossless difference is not sufficient to establish quality equivalence. Repeated sampling runs, task-level uncertainty estimates, and complete results without the harmful prefix rule are needed to make the speed–quality trade-off convincing.
Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6F1K

Thank you for the review. Your points on comparing with simpler target-side rules at matched quality and on the evidence for quality preservation led us to measure a range of thresholds repeatedly in the same engine and to revise how we judge quality, which strengthened the work.

W1. Comparison with simpler target-side rules at matched quality

This is a fair point. We ran, three times each in the same engine, the target-probability threshold (0.2 and 0.3), the probability ratio (0.5 and its default 0.2), and the θq=0 configuration from the last row of Table 2 of the submission. Each cell gives throughput over lossless decoding · accuracy difference (mean over the nine benchmarks). ✗ marks configurations that fail the margin; the intervals are in Tables 1 and 2 of the general response (Qwen3 pair).

ConfigurationT=0T=0.6
target probability ≥ 0.3+7.7% · +0.56+9.8% · −0.13
target probability ≥ 0.2+14.0% · −0.55 ✗+13.3% · +0.43
mutual confirmation with θq=0 (target-side test only)+17.2% · −0.44 ✗+13.4% · −0.35 ✗
probability ratio ≥ 0.5+11.7% · +0.34+12.5% · +0.42
probability ratio ≥ 0.2+21.2% · −0.79 ✗+17.1% · −0.08
ours+7.8% · +1.10+3.2% · +0.06
target probability ≥ 0.3 + ours+12.5% · +0.39+12.2% · +0.21
probability ratio ≥ 0.5 + ours+15.3% · −0.14 ✗+14.6% · −0.07
  • At T=0, ours has the throughput of target probability ≥ 0.3, and the whole interval of its accuracy difference lies above zero (+1.10 [+0.18, +2.02]). Added to target probability ≥ 0.3, it reaches +12.5% within the margin. Among target-side thresholds alone, target probability ≥ 0.3 and probability ratio ≥ 0.5 pass the margin; the lower thresholds (target probability ≥ 0.2, θq=0, probability ratio ≥ 0.2) are faster but fail.
  • At T=0.6, the row you pointed to (θq=0) fails the margin over three runs (−0.35 [−1.06, +0.35]). Of the lower thresholds, the two that pass at this temperature (target probability ≥ 0.2, probability ratio ≥ 0.2) fail at T=0. Ours added to target probability ≥ 0.3 passes at both temperatures and stays at about +12% (+12.5%, +12.2%).

W2. Evidence for quality preservation

You are right that the submission relied on single runs. We re-evaluated with three runs and a margin fixed in advance (Table 1), and we add per-benchmark intervals to the appendix. Our method passes the margin at both temperatures, and none of its 18 per-benchmark intervals (nine benchmarks, two temperatures) lies below zero. The ZebraLogic loss in the submission was caused by the prefix relation, which we removed from the method. Section 4.3.1 and the appendix analyze that failure.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/2): additions, rules compared, Table 1

We thank the four reviewers for their careful reading. As you pointed out, the submission lacked a comparison with existing relaxed-acceptance rules, repeated runs with intervals, and results without the prefix relation. We added all three during the discussion period.

  • Comparison. We implemented five existing rules in the same verification kernel of the same engine.
  • Decision rule. Every configuration in Tables 1 and 2 was run three times. Its accuracy difference from lossless speculative decoding is paired over the same problems and reported with a 95% interval. Before looking at the results, we fixed a margin of 1.0 point on the nine-benchmark mean (a configuration passes if the lower end of its interval is above −1.0).
  • Our method. We removed the prefix relation, which caused the ZebraLogic loss. "Ours" below is the method without it.

All numbers below are for the Qwen3 pair.

Rules compared. Each rule decides whether to accept the draft's token d at a position that lossless verification rejected. p and q are the distributions of the target and the draft, and t is the target's token (the most probable token under p at T=0; the token drawn from the residual distribution under sampling).

Rule (settings measured)Accepts d when
Target probability (0.2, 0.3)p(d) ≥ threshold
Probability ratio (the verification rule of AdaptiveSpec; 0.5 and its default 0.2)p(d) / p(t) ≥ threshold
Typical acceptance (original values)p(d) ≥ a threshold set by the entropy of p
SpecDec top-β (original values: β=3, τ=1.0)d is among the top 3 tokens of p and log p(t) − log p(d) ≤ 1.0
FSD (JS 0.1)the JS divergence between p and q is at most 0.1
Mutual confirmation (one rule of ours)p(d) ≥ 0.1 and q(t) ≥ 0.2

The first four are truncation rules: they decide from the target's distribution alone. FSD and mutual confirmation also use the draft's distribution. The test q(t) ≥ 0.2 is the draft-side condition.

Table 1. Each rule alone (Qwen3 pair). Accuracy difference from lossless decoding, mean over the nine benchmarks [95% interval] · share of rejections rescued. ✗: fails the margin.

RuleT=0T=0.6
ours+1.10 [+0.18, +2.02] · 30.6%+0.06 [−0.64, +0.75] · 15.8%
target probability ≥ 0.3+0.56 [−0.40, +1.53] · 22.1%−0.13 [−0.86, +0.61] · 26.2%
target probability ≥ 0.2−0.55 [−1.53, +0.42] ✗ · 40.4%+0.43 [−0.28, +1.13] · 33.7%
probability ratio ≥ 0.5+0.34 [−0.64, +1.32] · 33.4%+0.42 [−0.29, +1.12] · 31.3%
probability ratio ≥ 0.2−0.79 [−1.82, +0.23] ✗ · 57.2%−0.08 [−0.77, +0.61] · 42.3%
typical acceptance−1.17 [−2.23, −0.12] ✗ · 65.2%−0.20 [−0.92, +0.52] · 47.1%
SpecDec top-β−0.07 [−1.03, +0.90] · 45.1% †−0.30 [−1.03, +0.43] · 33.4% †
FSD (JS 0.1)+0.73 [−0.16, +1.62] · 28.5%+0.39 [−0.33, +1.11] · 25.2%
mutual confirmation with θq=0 (target-side test only)−0.44 [−1.45, +0.56] ✗ · 45.8%−0.35 [−1.06, +0.35] ✗ · 34.1%

† The lower end of the interval is 0.03 below the margin. The difference is small, so we do not count this cell among the failures.

(Continued in part 2/2.)

AuthorsOpenReview ↗

General response (2/2): results, Table 2, revision

(Continued from part 1/2.)

Result 1. Our method rescues rejections while preserving accuracy. It passes the margin at both temperatures. At T=0 it rescues 30.6% of the rejections, and the whole interval of its accuracy difference lies above zero (this rules out a loss; we do not claim an improvement). At this temperature its throughput over lossless decoding is +7.8%, level with target probability ≥ 0.3 (+7.7%) and FSD (+7.9%). It needs no training and no extra forward pass.

Result 2. The draft-side condition preserves accuracy. Mutual confirmation alone is more accurate with this condition than without it, by 1.49 points at T=0 ([0.56, 2.41]) and by 1.04 points at T=0.6 ([0.31, 1.76]). At T=0, five rules of Table 1 rescue more than 40% of the rejections, and all five decide from the target's distribution alone. Four of them fail the margin (target probability ≥ 0.2, probability ratio ≥ 0.2, typical acceptance, and mutual confirmation with θq=0), and for top-β the lower end of the interval is 0.03 below the margin. Paired over the same problems, all five are 1.2–2.3 points below ours, and the 95% intervals of these differences all lie below zero.

Result 3. Added to an existing rule, our method rescues more and runs faster, and five of the six cases pass the margin. It rescues positions that the existing rule does not. With FSD at T=0, on average over the nine benchmarks, 11.2% of the rejected positions were accepted by FSD only, 13.6% by ours only, and 16.9% by both.

Table 2. Ours added to an existing rule (Qwen3 pair). "existing / ours → both" gives the value with the existing rule only, with ours only, and with both. Throughput is relative to lossless decoding (mean of three runs; the three values in a row were measured in the same session). The accuracy difference is that of the combination from lossless decoding.

TExisting ruleRescued: existing / ours → bothThroughput: existing / ours → bothAccuracy difference [95% interval]
0target probability ≥ 0.322.1 / 30.6 → 43.3%+7.7 / +7.8 → +12.5%+0.39 [−0.59, +1.36]
0FSD (JS 0.1)28.5 / 30.6 → 41.7%+7.8 / +7.3 → +10.0%+0.52 [−0.43, +1.46]
0probability ratio ≥ 0.533.4 / 30.6 → 50.6%+11.7 / +7.8 → +15.3%−0.14 [−1.13, +0.85] ✗
0.6target probability ≥ 0.326.2 / 15.8 → 39.0%+9.8 / +3.2 → +12.2%+0.21 [−0.49, +0.90]
0.6FSD (JS 0.1)25.2 / 15.8 → 36.2%+8.2 / +2.9 → +10.1%−0.19 [−0.89, +0.50]
0.6probability ratio ≥ 0.531.3 / 15.8 → 44.2%+12.5 / +3.2 → +14.6%−0.07 [−0.80, +0.65]
  • In all six cases the rescued share and the throughput are higher than with either rule alone. The one case that fails is ours added to probability ratio ≥ 0.5 at T=0, which misses the margin by 0.13.
  • Lowering one threshold to rescue a similar share gives a different result. At T=0, target probability ≥ 0.2 fails the margin and is 0.94 points below target probability ≥ 0.3 + ours ([0.06, 1.81]) and 1.07 points below FSD + ours ([0.10, 2.04]).

Revision. We remove the prefix relation and analyze that failure in Section 4.3.1 and the appendix. We add the two tables above, per-benchmark intervals, per-rule answer-change rates, and the check of the judged sample by actual generation. We restate the contribution as an acceptance rule that can be used alone or added to existing relaxed-acceptance rules.

AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded the revised PDF. Text and tables that are new or changed since the submitted version are shown in blue.

Two things are presented differently from our general response:

  • Table numbers and layout. Tables 1 and 2 of the general response are Tables 3 and 4 of the revised PDF. Both now give each 95% interval as ± its half-width (for example, +1.10 [+0.18, +2.02] is written +1.10 ± 0.92). Table 3 groups the rules into those that pass the margin at both temperatures and those that rescue more than 40% of the rejections at T=0. Table 4 shows each existing rule alone and with our method added, and the throughput that our method adds, measured in the same session.
  • SpecDec top-β. In the revised PDF we apply the one-point margin without exception. Top-β, whose interval ends 0.03 points below the margin at both temperatures, is therefore marked as failing in Table 3 (✗†). The general response did not count these two cells among the failures. Our conclusions do not depend on how these cells are counted.

The abstract on this page has been updated to match the revised PDF. The TL;DR field cannot be edited during the discussion period, so its numbers are those of the submitted version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.