- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PEND, a relaxed verifier for speculative decoding that maintains separate decisions for overlapping draft–target mismatches. Later target predictions contribute confidence-weighted evidence to each pending mismatch, and a segment is committed only when all pending mismatches have resolved. Local target support controls mismatch admission and the required evidence mass. The method reuses predictions from the existing target-model pass. Experiments cover two draft–target model pairs and four benchmarks, with throughput improvements of 46.6–70.3% over vanilla speculative decoding and generally small changes in measured task quality. The appendix establishes properties of the verifier under cached draft-prefix distributions; these are not guarantees of exact target-distribution preservation or downstream correctness.
Critical strengths
-
Treating overlapping mismatches as separate pending decisions is a useful extension of deferred verification. Joint commitment addresses the case where evidence resolves the first mismatch but leaves a later one unresolved. The first-mismatch-only ablation and the evidence-sharing ablation support the importance of this distinction.
-
The implementation uses target outputs that have already been computed, and the measured throughput gains are substantial across both model pairs and all four tasks. Reporting task quality alongside speed is appropriate for a method that deliberately relaxes exact verification. The manuscript also clearly limits its theoretical claims to the cached draft trajectory.
Critical weaknesses
-
PEND and the baselines are tuned differently. Appendix C.2 and Table 4 use task-specific two-parameter grids for PEND, centered on development-selected settings, but task-shared threshold grids for the baselines. The speed gain over vanilla speculative decoding is still clear. However, the comparison with other relaxed verifiers would be more convincing if all methods were tuned on the same development data with comparable search budgets.
-
The quality results need more detail. Throughput variability is reported, but the quality differences lack paired uncertainty estimates, and the paper does not state how much degradation would be acceptable. For example, Qwen HumanEval+ changes from 87.80 to 86.59 pass@1, while Llama MT-Bench changes from 7.04 to 6.94. Since later evidence is conditioned on the draft continuation, the appendix’s local regret bounds cannot establish that these changes remain small on new inputs.
Questions for authors
-
What development examples and selection objective were used for each benchmark? Do PEND’s quality–throughput advantages remain when the competing relaxed verifiers receive task-specific tuning with comparable search budgets?
-
Can you report paired outcome changes for HumanEval+ and GSM8K, and uncertainty for the reported quality differences? In particular, how many examples become incorrect versus become correct relative to strict verification?
4 comments
Response to Reviewer(Part 2 of 3)
2. Response to Fairness of Baseline Calibration.
In the original main performance comparisons, we primarily adopt the configurations recommended in the respective baseline papers or official implementations, ensuring fidelity to their established settings and reproducibility. In Appendix C.2, we also conduct controlled parameter sweeps to examine their quality and throughput under different configurations.
To further assess whether benchmark-specific calibration affects the comparison, we conduct benchmark-specific tuning experiments.
The calibration protocol is summarized as follows:
- Development sets. For HumanEval+, we use 128 fixed tasks sampled from MBPP+. For GSM8K, we use 256 fixed samples from its training set, excluding the 800 samples previously used for SelfJudge training and threshold-candidate construction.
- Search budget. For each of FLy, LinguaSpec, MARS, and SelfJudge, we evaluate 5 coarse-search configurations followed by 5 local-refinement configurations on each task, using the same task-specific development data.
- Selection criterion. Configurations are required to incur no more than a 2-percentage-point quality reduction relative to Vanilla SD on the development set. Among qualifying configurations, we prioritize throughput. When a configuration achieves at least 99% of the highest qualifying throughput, we prefer the configuration with higher task quality.
- Final evaluation. Final parameters are selected exclusively using development-set results and frozen before evaluation on the complete test sets.
Table R2. Task-specific calibration results.
| Dataset | Method | Correct | Pass@1 / EM | MAT | Throughput (tok/s) | Speedup |
|---|---|---|---|---|---|---|
| HumanEval+ | Vanilla SD | 144/164 | 87.80% | 9.13 | 23.70 | 1.50× |
| SelfJudge | 127/164 | 77.44% | 15.19 | 40.39 | 2.56× | |
| MARS | 132/164 | 80.49% | 12.90 | 36.71 | 2.33× | |
| FLy | 142/164 | 86.59% | 12.00 | 34.10 | 2.17× | |
| LinguaSpec | 140/164 | 85.37% | 11.68 | 33.34 | 2.12× | |
| PEND | 142/164 | 86.59% | 13.85 | 39.13 | 2.48× | |
| GSM8K | Vanilla SD | 1263/1319 | 95.75% | 9.76 | 26.06 | 1.60× |
| SelfJudge | 1209/1319 | 91.66% | 14.88 | 39.40 | 2.41× | |
| MARS | 1249/1319 | 94.69% | 13.32 | 40.96 | 2.51× | |
| FLy | 1243/1319 | 94.24% | 12.88 | 38.98 | 2.39× | |
| LinguaSpec | 1257/1319 | 95.30% | 12.44 | 37.60 | 2.30× | |
| PEND | 1260/1319 | 95.53% | 13.19 | 40.91 | 2.51× |
The results in Table R2 support the following statements:
-
Higher throughput while maintaining task quality on HumanEval+. PEND achieves 86.59% Pass@1 and 39.13 tok/s, delivering 14.8% higher throughput than the tuned FLy at identical task quality. Although SelfJudge achieves slightly higher throughput, it incurs substantially greater quality degradation. These results indicate that PEND improves decoding efficiency without relying on substantial task-quality degradation, even when competing methods are independently calibrated.
-
Stronger quality preservation at competitive throughput on GSM8K. PEND achieves 95.53% EM and 40.91 tok/s, maintaining nearly identical throughput to the tuned MARS while improving EM by 0.84 percentage points. It also achieves higher quality and throughput than FLy and LinguaSpec. These results suggest that PEND can sustain high decoding throughput while better preserving task quality, rather than obtaining efficiency gains primarily through more permissive verification.
-
Robustness to task-specific baseline calibration. Across both benchmarks, PEND remains competitive after each baseline is independently tuned under the same search budget and quality-constrained selection protocol. Importantly, its favorable quality–throughput performance persists beyond comparisons against recommended baseline configurations, indicating that the observed advantages are not solely attributable to differences in baseline parameterization.
Overall, these results strengthen the evidence that PEND's concurrent multi-mismatch verification provides an effective mechanism for improving draft utilization while controlling task-quality degradation, with its benefits remaining evident under task-specific baseline calibration.
We have incorporated the task-specific calibration protocol and corresponding results into Appendix C.2 (Table 6) of the revised manuscript, with all additions highlighted in blue for ease of reference.
Response to Reviewer(Part 3 of 3)
3. Response to Quality–Throughput Tradeoff and Scope of Theoretical Guarantees.
(1) Quality–Throughput Objective and Quality Tolerances
Our quality–throughput objective is to maximize decoding throughput while controlling task-quality changes. Given the different scales of evaluation metrics, we adopt corresponding quality tolerances: at most 2 percentage points in Pass@1/EM for HumanEval+ and GSM8K, 0.02 in average answer F1 for HotpotQA, and 0.2 points in average score for MT-Bench. These thresholds represent approximately comparable numerical tolerances on their respective metric scales, providing a consistent principle for quality control across tasks.
-
Parameter selection. During parameter selection, we prioritize throughput among configurations satisfying the development-set quality tolerances. For candidates achieving at least 99% of the highest qualifying throughput, we prefer those with higher task quality. Final configurations are selected using development data and frozen before testing.
-
Quality assessment. Actual quality changes and their uncertainty are further quantified through paired outcome analyses and confidence intervals.
(2) Counterfactual Continuation and Scope of Theoretical Guarantees.
We acknowledge that subsequent target-model predictions are conditioned on the original draft prefixes and therefore do not represent the counterfactual continuation that would arise after correcting an earlier mismatch under strict verification. Explicitly obtaining such counterfactual predictions would require additional target-model computation along the corrected prefixes, potentially involving repeated forward passes and substantial computational overhead, particularly when multiple mismatches are involved.
-
PEND's verification objective. PEND instead aims to determine whether mismatched draft tokens can be accepted under relaxed verification conditions, rather than reconstructing the strict target model's generation trajectory. To this end, PEND leverages target-model predictions already cached during a single verification pass, combining target-model support at each mismatch position with confidence-weighted subsequent evidence and evidence accumulation to assess whether each mismatch satisfies its acceptance conditions. This enables concurrent resolution of multiple mismatches without additional target-model forward passes.
-
Relationship to prior work. This design shares a common principle with continuation-based relaxed verification methods such as FLy (ICLR 2026), which also leverage subsequent predictions along the draft trajectory to assess the acceptability of earlier mismatches, rather than explicitly computing counterfactual continuations. PEND further incorporates mismatch-specific target-model support and confidence-weighted evidence accumulation to establish separate acceptance conditions for multiple mismatches and coordinate their resolution.
-
Theoretical scope and limitations. These conditions do not guarantee equivalence to strict target-model outputs or task quality on unseen inputs. As discussed in Appendix B.6, PEND's local regret bounds and subsequent evidence constraints are defined over the draft-conditioned distributions cached within the current verification round.
Official Comment by Authors
Dear Reviewer 7AGq,
We would like to kindly follow up on our previous response. We would greatly appreciate it if you could let us know whether our clarifications and revisions have adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further explanations.
Thank you again for your time and consideration.
Response to Reviewer(Part 1 of 3)
We sincerely thank the reviewer for the careful evaluation and constructive suggestions. In response, we conduct task-specific baseline calibration under a matched search budget. We also provide paired quality analyses and uncertainty estimates to quantify the effects of relaxed verification. Furthermore, we clarify the quality–throughput objective, parameter-selection criteria, and scope of PEND's theoretical guarantees, providing a more comprehensive assessment of PEND's effectiveness and limitations.
1. Response to Paired Quality Changes and Uncertainty.
Strict verification may reject plausible draft candidates, limiting effective draft-token utilization. Prior studies, including Judge Decoding (ICLR 2025), MARS (Findings of ACL 2026), and FLy (ICLR 2026), have explored relaxed verification to improve decoding efficiency while maintaining task quality.
PEND likewise adopts relaxed verification rather than aiming to preserve exact agreement with the target model's output. It further improves draft utilization through concurrent verification and evidence accumulation across multiple mismatch positions, seeking higher decoding throughput while controlling task-quality degradation. Accordingly, we focus on the task-quality–throughput tradeoff rather than exact token-level equivalence with Vanilla SD.
To quantify the impact of relaxed verification on task quality, we conduct paired comparisons between Vanilla SD and PEND on HumanEval+ and GSM8K, estimating 95% confidence intervals (CIs) for quality differences using paired bootstrap resampling. Quality differences are defined as the change in task accuracy of PEND relative to Vanilla SD.
Table R1. Paired quality changes and throughput comparison.
The results reveal bidirectional sample-level quality changes, including both improvements and degradations. PEND exhibits observed accuracy reductions of 1.22 and 0.23 percentage points on HumanEval+ and GSM8K, respectively, while improving throughput over Vanilla SD by 65.1% and 57.0%. These findings demonstrate substantial decoding-throughput gains with relatively small observed aggregate quality reductions on both benchmarks, consistent with PEND's objective of improving the task-quality–throughput tradeoff.