acceptodds
Under review as a conference paper at ICLR 2027

When to Trust the Past: Reliability-Calibrated Draft-Tree Orchestration for Speculative Decoding

Abstract

Speculative decoding accelerates autoregressive inference in large language models (LLMs) through a guess-then-verify paradigm, in which multiple candidate continuations are drafted and verified in parallel. Recent methods combine suffix-based retrieval with target-model logits recycled from previous verification steps to reuse historical continuations, thereby expanding candidate coverage. However, such reuse implicitly assumes that historical predictions remain reliable under the current prefix, while poorly transferable candidates occupy limited verification capacity and crowd out more promising alternatives when this assumption fails, an issue which is further exacerbated as history accumulates, since the same token recurs under increasingly diverse prefixes. Towards this end, we propose a novel framework termed Reliability-Calibrated Draft-Tree Orchestration for speculative decoding (RECAP). The core of RECAP lies in estimating prefix-dependent reliability of historical predictions and then using this calibrated evidence to jointly determine candidate composition and the allocation of the verification budget. In particular, we first construct a multi-context verification memory that stores target-model predictions together with their corresponding prefix contexts. Subsequently, we introduce a lightweight prefix-conditioned scorer that estimates the transferability of each memory entry to the current decoding state and aggregates reliable historical evidence. Based on the resulting evidence, RECAP combines retrieved continuations with logit-based candidates under a shared verification budget and expands the draft tree best-first along the paths of highest cumulative reliability. Together, these components transform draft-tree construction from a fixed heuristic into an orchestration process that adapts both candidate sources and tree shape to the available evidence. Extensive experiments show that RECAP consistently improves decoding efficiency over strong speculative-decoding baselines while preserving generation quality. The implementation of RECAP is available at https://anonymous.4open.science/r/RECAP7C2E.

Primary area:
foundation or frontier models, including LLMs
Keywords:
autoregressive decoding, speculative decoding, large language models, inference acceleration, draft-tree orchestration, reliability calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yNX8OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECAP, a speculative decoding framework that learns when historical target-model predictions remain useful under the current prefix. It combines context-aware memory scoring with adaptive draft-tree construction, improving candidate acceptance and decoding efficiency while retaining target-model verification.

Critical strengths
  1. The paper identifies a meaningful limitation of historical prediction reuse: matching token identities does not guarantee that predictions transfer across contexts. Preserving contextual information provides a well-motivated way to address this problem.
  2. The framework connects candidate selection with verification-budget allocation. Its treatment of unsupported continuations and adaptive tree expansion provides a coherent mechanism for directing computation toward promising branches.
  3. The evaluation includes multiple target models, diverse workloads, controlled ablations, and detailed runtime analysis. Comparisons on identical output text and investigations of numerical divergence strengthen the empirical evidence.
Critical weaknesses
  1. The theoretical guarantees have limited demonstrated relevance to the deployed system. They assume uniformly bounded estimation errors within a fixed candidate space, whereas practical decoding also involves candidate omissions, fallback construction, and changes in future memory caused by current decisions.
  2. Generalization remains dependent on target-specific training and workload characteristics. Domain specialization can determine whether improved acceptance translates into an actual speed advantage, and the evaluation primarily concerns greedy decoding.
  3. Comparisons with trained drafters are less controlled than the central comparisons with historical-reuse methods. Separate execution campaigns, different implementations, and estimated timing adjustments make the broader efficiency advantage harder to assess.
Questions for authors

Can the authors empirically assess how informative the bounds are under realistic scoring errors and quantify the losses attributable to restricted candidate coverage and fallback behavior?

How well does a frozen scorer transfer across domains and related target models, and what changes are needed to retain its benefits under stochastic decoding?

Can RECAP and strong trained-drafter baselines be evaluated under a common optimized serving stack, with directly measured latency and independently tuned drafting budgets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yNX8, Part 1 of 3: Theoretical Bounds and Fallback

Q1: How informative are the theoretical bounds under realistic scoring errors, and how do the fixed-candidate-space assumptions relate to candidate omissions, fallback behavior and changes to future memory in the deployed system?

A1: We thank the reviewer for this question. Our analysis bounds the worst-case loss in expected accepted length caused by scoring errors propagating through tree allocation within a fixed candidate space, and we complement this analysis with two empirical diagnostics.

  • Fallback under practical usage. To examine the fallback rate in practical usage, we further inspect the paper's persistent Qwen3-32B agent sessions on SWE-bench Verified, which retain historical information across model calls and more closely reflect realistic multi-turn agent scenarios. Experiments show that root fallback and fixed fill together account for 6.35% of verification rounds, so learned budget allocation governs the majority of rounds. Moreover, fallback preserves the accepted output sequence and, aside from routine memory updates from verified nodes, leaves the tree-planning rule unchanged in other rounds, so at this low trigger rate it does not significantly affect overall generation.

  • Reference-tree comparison and a geometric bound. On Spec-Bench, we evaluate RECAP with Qwen3-8B. At each state, we fix the historical memory, observed expanded candidate pool, retrieval constraints, and verification budget, and construct a reference-optimal tree using the target model's full-vocabulary softmax probabilities. The following expected-length quantities are evaluated under the target-softmax reference distribution pp.

MetricMean
Node overlap between RECAP and the reference tree40.48%
Expected accepted length of RECAP4.058
Expected accepted length of the reference-optimal tree4.418
Expected-length loss0.360
Structure-dependent bound with geometric cap7.865

The expected-length loss remains below the corresponding bound across all evaluated states. However, the mean bound substantially exceeds the observed loss, reflecting the conservatism of the worst-case analysis. To further characterize the loss using the observed candidate geometry, we define

\le H(C,T_q) =\max_{\pi\in\mathcal P(C)}|\pi\setminus T_q|,$$ where $T_p$ is the pool-constrained reference-optimal tree, $T_q$ is the constructed RECAP tree, and $\mathcal P(C)$ denotes the root-to-leaf paths in the fixed candidate pool $C$. Since verification follows a single continuation path, the pool-constrained optimum's accepted-length advantage cannot exceed the number of candidate nodes on that path absent from $T_q$. Using only the candidate pool and the constructed tree, this bound has a mean of 1.990 tokens across the evaluated states, compared with an observed mean loss of 0.360 tokens, and holds in every state.
AuthorsOpenReview ↗

Response to Reviewer yNX8, Part 2 of 3: Domain Reuse and Model Transfer

Q2: How well does a frozen scorer transfer across domains and related target models, and how do target-specific training and domain specialization affect whether acceptance improvements yield actual speedups?

A2: We thank the reviewer for asking about scorer generalization. For each target model, the main experiments use one scorer trained on heterogeneous LongBench contexts and reuse it across all five evaluation benchmarks without per-domain retraining, while domain specialization through RECAP-C remains an optional adaptation. Regarding whether acceptance improvements yield actual speedups, observing substantial room for further implementation optimization in RECAP, we reuse context encodings, pack frontier inputs and probability outputs, and fuse feature preparation with fallback handling to obtain RECAP+, which keeps the frozen scorer weights and verification budget unchanged while substantially increasing end-to-end TPS, so we also include RECAP+ in the comparisons that follow.

  • Cross-task domain reuse. While the main-text table reports overall performance across five evaluation benchmarks using a single LongBench-trained frozen scorer, we further break down Qwen3-8B results on Spec-Bench by task category:
Task categoryAR TPSRACER MAT / TPSRECAP MAT / TPSRECAP+ MAT / TPSRECAP+ TPS gain over RACER
Multi-turn conversation51.022.50 / 106.363.31 / 116.713.32 / 132.14+24.2%
Translation50.321.69 / 67.662.00 / 65.822.00 / 76.07+12.4%
Summarization50.052.51 / 102.423.25 / 108.913.25 / 123.82+20.9%
Question answering51.162.22 / 96.643.04 / 109.583.04 / 116.96+21.0%
Mathematical reasoning50.772.74 / 118.333.72 / 128.053.71 / 146.42+23.7%
Retrieval-augmented generation50.262.38 / 98.853.36 / 113.223.35 / 127.70+29.2%

RECAP improves MAT across all six categories and increases end-to-end throughput in five of the six categories outside very short translation outputs, raising overall MAT by 34.3% and TPS by 10.2% over RACER. With this execution optimization, RECAP+ achieves higher end-to-end throughput across all six categories and raises overall TPS by 23.8% over RACER, demonstrating strong domain generalization.

  • Transfer to a related target model. We apply the frozen Qwen3-8B scorer directly to Qwen3-8B-Base, retaining source-model embedding features without retraining, and evaluate on Spec-Bench on the same host under default settings:
MethodMATEnd-to-end TPSSpeedup over AR
AR1.00049.691.000x
RACER2.787114.152.297x
RECAP3.651119.642.408x

Without retraining, RECAP improves MAT by 31.0% and end-to-end TPS by 4.8% over RACER, demonstrating effective transfer of the frozen scorer across related target models.

  • Domain specialization and actual speedups. On smaller target models with faster verification passes such as DeepSeek-Coder-6.7B-Instruct, fine-grained drafting overhead accounts for a larger share of runtime, so acceptance gains from a general scorer do not automatically yield a net speedup. On the coding and math tasks of LiveBench, RECAP+ raises throughput without retraining from 121.8 and 146.3 TPS for RECAP to 141.3 and 164.2 TPS, surpassing RACER at 137.8 and 161.3 TPS, while optional domain specialization further expands acceptance gains so that RECAP-C reaches 157.4 and 165.3 TPS and RECAP-C+ reaches 167.9 and 176.2 TPS.
AuthorsOpenReview ↗

Response to Reviewer yNX8, Part 3 of 3: Stochastic Decoding and EAGLE-3

Q3: What changes are needed to support stochastic decoding, and how does RECAP perform beyond greedy decoding?

A3: We thank the reviewer for raising stochastic decoding. At each visited prefix, the verifier samples from the prescribed target distribution. If the token matches a drafted child, traversal continues, and otherwise the round ends and the sampled token is retained as the next root. The scorer guides tree allocation, while token selection follows the target sampling law, preserving the prescribed distribution under consistent exact target conditionals.

We evaluate Qwen3-8B on Spec-Bench under the default settings, keeping both the target model and the original trained scorer frozen without additional retraining.

DecodingRACER MAT / TPSRECAP MAT / TPS
Greedy2.45 / 103.843.29 / 114.41
T=0.7, top-p=0.92.38 / 84.383.18 / 93.92
T=1.0, top-p=1.02.28 / 89.942.97 / 96.17

Under the two stochastic settings, RECAP achieves 11.3% and 6.9% higher throughput than RACER, respectively, maintaining its advantage under stochastic decoding.

Q4: Can RECAP and strong trained-drafter baselines be evaluated under a common optimized serving stack, with directly measured latency and independently tuned drafting budgets?

A4: We thank the reviewer for suggesting a controlled comparison with trained drafters. Under the default settings, we evaluate Qwen3-8B on Spec-Bench using a shared target and KV backend together with an optimized execution stack, comparing EAGLE-3 against AR, RACER, RECAP, and RECAP+ from A2. Drafting budgets are independently selected on a separate validation split with the remaining evaluation settings matched across methods, and end-to-end request latency is measured directly.

MethodMATMean latency / sEnd-to-end TPS
AR1.0005.5850.80
EAGLE-33.1732.9894.91
RACER2.4482.74103.84
RECAP3.2872.48114.41
RECAP+3.2852.21128.53

While achieving a mean accepted length comparable to that of EAGLE-3, RECAP avoids sequential draft-model forward passes, enabling RECAP and RECAP+ to deliver 20.5% and 35.4% higher end-to-end throughput, respectively, along with lower mean request latency.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer yNX8,

Thank you again for your detailed and constructive feedback.

We would like to kindly follow up on our rebuttal. We have provided additional empirical analyses of the theoretical bounds and fallback behavior, evaluations of scorer transferability and stochastic decoding, and a controlled comparison with EAGLE-3 under a shared execution stack with directly measured latency.

We hope these additional results help address your concerns. We would sincerely appreciate it if you could take a moment to review our responses and share any remaining questions or concerns.

Thank you very much for your time and consideration.

Best regards, The Authors

Reviewer_naP3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RECAP, a model-free speculative decoding framework that improves historical target-logit reuse through multi-context verification memory, prefix-conditioned reliability estimation, and adaptive draft-tree orchestration. Experiments demonstrate substantial improvements in mean accepted tokens per verification round (MAT) and decoding throughput over existing model-free baselines.

Critical strengths
  1. The paper identifies an important limitation of token-indexed historical prediction reuse and introduces context-conditioned reliability estimation to address the transferability of historical predictions.

  2. The combination of multi-context verification memory, a lightweight learned reliability scorer, and adaptive tree orchestration provides a coherent framework for improving historical prediction reuse.

Critical weaknesses
  1. Although Appendix C.3 demonstrates the effectiveness of the learned scorer over heuristic alternatives, these comparisons do not isolate the contribution of prefix-conditioned context features from learned logit statistics and metadata. A controlled ablation using an otherwise identical learned scorer without contextual features would better substantiate the central novelty and clarify how much of the improvement comes specifically from cross-prefix contextual information.

  2. Goose has explored reliability-aware asymmetric tree construction, C2T employs a learned classifier for dynamic tree construction, and AdaPLD investigates adaptive retrieval and candidate reuse. Although RECAP introduces a distinct prefix-conditioned historical prediction scorer, a clearer analysis of its methodological advantages over these approaches would strengthen its novelty claims. In particular, the additional benefits and training costs relative to training-free alternatives such as Goose deserve further discussion. A direct comparison with Goose would be valuable if feasible.

  3. Despite consistent MAT improvements, RECAP does not always improve throughput, as demonstrated by the regressions on DeepSeek-Coder-6.7B in Table 3, where the original RECAP achieves higher MAT but lower TPS than RACER on coding and math tasks. It remains unclear whether the net throughput gains persist under continuous batching or tensor-parallel serving.

Questions for authors
  1. Can the authors isolate the contribution of prefix-conditioned context features using an otherwise identical learned scorer without these features?
  2. What are RECAP's key methodological advantages over recent adaptive drafting methods such as Goose and C2T, particularly considering the additional scorer-training cost?
6 comments
AuthorsOpenReview ↗

Response to Reviewer naP3, Part 1 of 3: Prefix Ablation and Goose Comparison

Q1: Can the contribution of prefix-conditioned context features be isolated with an otherwise identical learned scorer that retains the learned logit statistics and metadata but does not use contextual features?

A1: We thank the reviewer for suggesting this controlled ablation.

  • Controlled ablation setup. We train full-context and no-context Qwen3-8B scorers with the same architecture and trainable parameters. The full-context scorer follows the main-text setup, while the no-context control replaces both current-prefix and stored-prefix token inputs and masks with constants during training and inference, keeping the remaining training and evaluation settings matched.

  • Prediction quality evaluation. Consistent with the main text, we evaluate model capability on LongBench, where the final checkpoints give:

Final scorerTop-1 accuracy / %NLLBrierECE / %
Without prefix features38.112.34060.774111.49
With prefix features49.871.86780.65338.47

The labels identify the target model's greedy Top-1 candidate, or NONE when it is outside the historical candidate union. Prefix context improves Top-1 accuracy by 11.76 percentage points and reduces ten-bin ECE by 3.02 percentage points, together with lower NLL and Brier score.

  • Decoding efficiency evaluation. To examine whether these prediction gains translate into decoding efficiency, we evaluate on Spec-Bench under the default settings.
MethodMATEnd-to-end TPSSpeedup over AR
AR1.00050.801.000x
RACER2.448103.842.044x
RECAP without prefix features2.779101.602.000x
RECAP with prefix features3.287114.412.252x

The results show that adding prefix context increases MAT from 2.779 to 3.287 and improves end-to-end TPS by 12.6%. These prediction and decoding results isolate the contribution of prefix-context information beyond learned logit statistics and metadata.

Q2: What are RECAP's key methodological differences and advantages relative to Goose, C2T and AdaPLD, and can a comparison with Goose be provided?

A2: We thank the reviewer for highlighting these related methods. We address this question through both methodological distinctions and measured throughput.

  • Methodological distinctions and advantages. Goose adapts tree construction using retrieval confidence and acceptance statistics, C2T classifies and prunes draft nodes, and AdaPLD selects a reuse anchor through retrieval and reranking. RECAP instead focuses on the transferability of historical target-model predictions to the current branch. It preserves their source contexts, evaluates multiple historical records individually, and aggregates them into reliability evidence for successor tokens. This enables a shared memory bank to provide branch-specific candidate support that guides both candidate selection and verification-budget allocation.
  • Comparison with Goose. We evaluate Goose and RECAP on Spec-Bench with Qwen3-8B under the default settings in the main text. In this comparison, RECAP reaches 114.41 TPS, achieving 25.5% higher end-to-end throughput than Goose at 91.20 TPS, with detailed per-round load and latency comparisons reported in A3.
AuthorsOpenReview ↗

Response to Reviewer naP3, Part 2 of 3: Scorer Benefits, Overhead, and Reuse

Q3: What additional benefits justify training the scorer, and what training and inference costs does it introduce relative to training-free methods such as Goose?

A3: We thank the reviewer for this question. We evaluate RECAP's benefits and overhead through acceptance length, per-round execution load, end-to-end speedup, offline training cost, and reuse across tasks and related target models.

  • Acceptance gains and inference load. Consistent with our appendix analysis, the lightweight scorer for 7–8B target models accounts for a negligible share of total computation, and its practical inference overhead mainly stems from fine-grained operator scheduling during drafting. Using the same default setup as in A2, we first compare AR, Goose, RACER, and RECAP:
MethodMATRound time / msEnd-to-end TPSSpeedup over AR
AR1.00019.6950.801.000x
Goose2.18623.9791.201.795x
RACER2.44823.57103.842.044x
RECAP3.28728.73114.412.252x

Although RECAP has a higher per-round time than Goose and RACER, its substantially higher MAT reduces the total number of verification rounds, lowering mean request latency from 5.58 s for AR, 3.10 s for Goose, and 2.74 s for RACER to 2.48 s and yielding higher end-to-end speedup.

  • Further execution optimization. This fine-grained execution overhead also leaves room for implementation-level optimization. Keeping the scorer weights and verification budget fixed, we reuse context encodings, pack frontier inputs and probability outputs, and fuse feature preparation with fallback handling, reducing repeated computation and transfers without retraining the scorer or changing the overall methodology, and we refer to this version as RECAP+. Evaluating Qwen3-8B on Spec-Bench under the same default settings, RECAP+ reaches 128.53 TPS, further improving end-to-end throughput by 12.3% over RECAP at 114.41 TPS and by 23.8% over RACER at 103.84 TPS while lowering mean request latency from 2.48 s to 2.21 s.

  • Offline training cost. RECAP trains its lightweight scorer offline with the target model frozen. As reported in the paper appendix, the complete two-stage pipeline including trace collection and fitting takes under four hours under the default setup. On A100 for Qwen3-8B, base-stage fitting takes 45.17 minutes and second-stage fitting takes 3.26 minutes, totaling 48.43 minutes with 5.46 GiB peak allocated GPU memory during fitting. In the main experiments, each target model uses a single frozen scorer across all five evaluation benchmarks without domain-specific retraining.

  • Cross-task domain reuse. While the main-text table reports overall performance across five evaluation benchmarks using a single LongBench-trained frozen scorer, we further break down Qwen3-8B results on Spec-Bench by task category:

Task categoryAR TPSRACER MAT / TPSRECAP MAT / TPSRECAP+ MAT / TPSRECAP+ TPS gain over RACER
Multi-turn conversation51.022.50 / 106.363.31 / 116.713.32 / 132.14+24.2%
Translation50.321.69 / 67.662.00 / 65.822.00 / 76.07+12.4%
Summarization50.052.51 / 102.423.25 / 108.913.25 / 123.82+20.9%
Question answering51.162.22 / 96.643.04 / 109.583.04 / 116.96+21.0%
Mathematical reasoning50.772.74 / 118.333.72 / 128.053.71 / 146.42+23.7%
Retrieval-augmented generation50.262.38 / 98.853.36 / 113.223.35 / 127.70+29.2%

RECAP improves MAT across all six categories and increases end-to-end throughput in five of the six categories outside very short translation outputs, raising overall MAT by 34.3% and TPS by 10.2% over RACER. With execution optimization, RECAP+ achieves higher end-to-end throughput across all six categories and raises overall TPS by 23.8% over RACER, demonstrating strong domain generalization.

  • Transfer to a related target model. We apply the frozen Qwen3-8B scorer directly to Qwen3-8B-Base, retaining source-model embedding features without retraining, and evaluate on Spec-Bench on the same host under default settings:
MethodMATEnd-to-end TPSSpeedup over AR
AR1.00049.691.000x
RACER2.787114.152.297x
RECAP3.651119.642.408x

Without retraining, RECAP improves MAT by 31.0% and end-to-end TPS by 4.8% over RACER, demonstrating effective transfer of the frozen scorer across related target models.

AuthorsOpenReview ↗

Response to Reviewer naP3, Part 3 of 3: Speedup, Batching, and Parallel Serving

Q4: Why does original RECAP achieve higher MAT but lower TPS than RACER on the DeepSeek-Coder-6.7B coding and math tasks in Table 3, and what do these regressions imply for the end-to-end efficiency claim?

A4: We thank the reviewer for highlighting this result. Higher MAT yields a net speedup only when the saved verification work outweighs the additional drafting overhead. We retain DeepSeek-Coder-6.7B-Instruct in the paper as a negative case to illustrate this efficiency boundary: this smaller target model has a shorter verification pass and produces relatively short outputs on coding and math, so fine-grained operator scheduling during drafting accounts for a larger share of runtime.

Under the default setup, we evaluate the coding and math tasks of LiveBench for RECAP+, which applies execution optimizations while keeping the general scorer weights and verification budget fixed, and RECAP-C+, which combines the existing coding-specialized scorer with the same execution optimizations:

TaskRACER TPSRECAP TPSRECAP+ TPSRECAP-C TPSRECAP-C+ TPS
Coding137.8121.8141.3157.4167.9
Math161.3146.3164.2165.3176.2

With the general scorer weights fixed, RECAP+ improves end-to-end TPS over RECAP by 16.0% on coding and 12.2% on math, turning the regressions into net gains of 2.5% and 1.8% over RACER. RECAP-C+ further increases the gains over RACER to 21.8% and 9.2%. Thus, on these workloads, execution optimization alone eliminates the throughput regression and restores a net speedup, while optional domain specialization provides further gains.

Q5: Do RECAP's net throughput gains persist under continuous batching or tensor-parallel serving?

A5: We thank the reviewer for raising this question about RECAP's efficiency under request concurrency and tensor parallelism. We conduct additional experiments to examine whether RECAP's end-to-end throughput advantage over RACER persists in these settings.

  • Concurrent request serving. We evaluate AR, RACER, and RECAP on Spec-Bench with Qwen3-8B under the default setup, setting the active-request limit to 8. All three methods share the same target-model and KV backends and scheduling policy, with a batching window capped at 40 ms. RACER and RECAP construct drafts in parallel across requests while maintaining separate histories for each request, and follow-up turns enter the queue after the preceding turn completes:
MethodMATEnd-to-end TPS
AR1.000223.83
RACER2.427223.30
RECAP3.269262.87

After accounting for scoring and tree-construction overhead, RECAP achieves 17.7% higher end-to-end throughput and 34.7% higher MAT than RACER, and 17.4% higher throughput than AR.

  • Qwen3-8B under tensor parallelism. We compare AR, RACER, and RECAP on Spec-Bench under the default setup across TP2 and TP4 alongside the main-table TP1 results. RECAP reuses its existing Qwen3-8B scorer without retraining, and under multi-GPU execution all methods share target-model sharding and KV operations across ranks, with communication included in the timing:
TP ranksAR TPSRACER MAT / TPSRECAP MAT / TPSTPS gain over RACER
150.802.448 / 103.843.287 / 114.41+10.2%
232.502.435 / 67.653.298 / 78.76+16.4%
432.062.445 / 67.223.295 / 77.68+15.6%

After accounting for scoring, tree-construction, and cross-rank communication overhead, RECAP reduces both target-verification and cross-rank synchronization rounds through higher acceptance length, not only retaining its throughput advantage over RACER at TP2 and TP4, but also expanding the end-to-end TPS gain over RACER from 10.2% at TP1 to 16.4% and 15.6%, with MAT gains of 35.5% and 34.8%, respectively.

Reviewer_ZyvnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RECAP, a speculative decoding framework that improves the reuse of historical target-model predictions. It maintains multiple prefix-conditioned prediction records and introduces a lightweight learned scorer to estimate their reliability under the current context. Based on these estimates, RECAP combines historical predictions with retrieved continuations and dynamically constructs draft trees using best-first expansion under a fixed verification budget.

Critical strengths
  • Effective Historical Information Reuse. The proposed multi-context memory and reliability scorer address an important limitation of existing token-recycling methods. The approach improves candidate selection by considering historical prefix information rather than relying solely on token identity.

  • Comprehensive Experimental Evaluation. The paper provides extensive comparisons across multiple models and benchmarks, including ablation studies and persistent agent workloads. The results consistently demonstrate improvements over existing model-free speculative decoding methods.

Critical weaknesses
  • The proposed framework mainly combines existing ideas, including historical logit reuse, context-aware scoring, and best-first draft-tree construction. Although their integration is effective, the overall algorithmic contribution appears incremental compared with prior works such as RACER, Sequoia, and OPT-Tree.

  • Despite improving the average accepted length, RECAP introduces considerable per-round inference overhead, resulting in more modest end-to-end speedup gains. Furthermore, the main experiments focus on batch size 1, leaving its efficiency under high-concurrency serving scenarios unclear.

Questions for authors

Given the additional overhead introduced by the learned scorer and dynamic tree construction, how would RECAP perform under larger batch sizes?

Do the authors have concrete optimization strategies to mitigate this overhead in high-throughput serving scenarios?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Zyvn: Method Design, Concurrency, and Optimization

Q1: How does RECAP differ from RACER, Sequoia and OPT-Tree?

A1: We thank the reviewer for raising this point. The difference between these methods and RECAP lies in three points:

  • Different methodology. RACER's copy-logit rule retains only the single most recent matching-token prediction, Sequoia scores nodes with a child-position acceptance model, and OPT-Tree constructs trees from current-context probabilities of an auxiliary autoregressive draft model. In RECAP, we preserve multiple historical prediction records paired with their source prefixes, score each occurrence conditioned on both the stored and current prefixes, and aggregate evidence per successor token for best-first tree expansion.

  • Different frameworks. Sequoia and OPT-Tree operate within model-based drafting frameworks built around separate draft models, whereas RACER operates in a model-free lookup framework with uncalibrated recency reuse and rule-based breadth allocation. RECAP advances the model-free historical target-logit reuse framework by coupling context-preserving verification memory with lightweight reliability calibration and adaptive budgeted tree orchestration.

  • Different motivations. Sequoia and OPT-Tree aim to optimize tree topology given current-step draft probabilities, while RACER prioritizes low-cost reuse under the assumption that the latest matching occurrence remains representative. RECAP instead addresses context shift in historical logit reuse, where target predictions cached under earlier prefixes may become unreliable on the active branch, and models cross-prefix transferability to jointly govern candidate reuse and verification-budget allocation.

Q2: How does RECAP perform at larger request batch sizes and under higher concurrency, given the additional scoring and tree-construction overhead?

A2: We thank the reviewer for this question. We evaluate AR, RACER, and RECAP on Spec-Bench with Qwen3-8B under the default setup, with up to eight active requests.

All three methods use the same target-model and KV backends and scheduling policy. RACER and RECAP construct drafts in parallel across requests while maintaining separate histories for each request. Follow-up turns enter the queue after the preceding turn completes.

MethodMATEnd-to-end TPS
AR1.000223.83
RACER2.427223.30
RECAP3.269262.87

After accounting for scoring and tree-construction overhead, RECAP achieves 17.7% higher end-to-end throughput and 34.7% higher MAT than RACER, and 17.4% higher throughput than AR. These results show that RECAP's acceptance-length advantage continues to translate into end-to-end throughput gains, maintaining a clear advantage over RACER in the evaluated concurrent setting.

Q3: What concrete optimizations can mitigate RECAP's inference overhead, and what end-to-end throughput gains do they provide?

A3: We thank the reviewer for this question. We optimize execution while keeping the scorer weights and verification budget fixed, and evaluate the resulting end-to-end throughput.

  • Execution optimizations. We reuse context encodings, pack frontier inputs and probability outputs, and fuse feature preparation with fallback handling. These changes reduce repeated computation and transfers without scorer retraining or changes to the overall methodology, and we refer to this version as RECAP+.

  • End-to-end results on the main backbone. Decoding follows the main-table defaults, using Spec-Bench. On Qwen3-8B, we evaluate RECAP+ alongside the corresponding main-table results.

MethodMATMean latency / sEnd-to-end TPSSpeedup over AR
AR1.0005.5850.801.000x
RACER2.4482.74103.842.044x
RECAP3.2872.48114.412.252x
RECAP+3.2852.21128.532.530x

With the scorer weights and verification budget fixed, RECAP+ achieves 128.53 TPS, improving over RECAP by 12.3% and over RACER by 23.8% while lowering mean request latency from 2.48 s to 2.21 s, demonstrating clear throughput gains from execution optimization.

Thank you for the detailed response and additional experiments. The new concurrency results and RECAP+ optimizations have substantially addressed my concerns about inference overhead and practical efficiency. The clarification of methodological differences also helps alleviate my novelty concerns. I have therefore decided to raise my rating.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Thank you very much for your positive feedback and for taking the time to carefully reconsider our work. We sincerely appreciate your constructive comments, which have helped us strengthen both the methodology and the experimental evaluation. We are glad that our additional results and clarifications have addressed your concerns.

Thank you again for your thoughtful evaluation and encouragement!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses, Manuscript Revisions, and Discussion

Dear Area Chairs and Reviewers,

We sincerely thank the Area Chairs and all reviewers for their careful evaluation, insightful comments, and constructive engagement throughout the discussion. In particular, we appreciate Reviewer Zyvn's decision to raise their rating after considering our new concurrency experiments, RECAP+ optimizations, and methodological clarifications. We also thank Reviewer naP3 for confirming that our responses largely addressed their concern while maintaining their positive assessment, and Reviewer yNX8 for the detailed suggestions that motivated further generalization experiments and theoretical analyses.

We are encouraged that the reviewers collectively recognized the following strengths of RECAP:

  • Well-motivated problem formulation (Reviewer Zyvn, Reviewer naP3, Reviewer yNX8): Identifying the limits of token-indexed historical prediction reuse across contexts and addressing them through prefix-conditioned reliability estimation.
  • Coherent and lightweight framework (Reviewer Zyvn, Reviewer naP3, Reviewer yNX8): Integrating multi-context verification memory, reliability-aware candidate selection, and adaptive draft-tree construction under a shared verification budget.
  • Comprehensive and rigorous evaluation (Reviewer Zyvn, Reviewer naP3, Reviewer yNX8): Demonstrating consistent gains across models and benchmarks, supported by ablations, persistent agent workloads, runtime analyses, identical-output comparisons, and numerical-divergence investigations.
  • Clear presentation and informative insights (Presentation: Reviewer Zyvn, Reviewer naP3; Informativeness: Reviewer naP3, Reviewer yNX8): Offering an accessible account of the framework and deeper insights into the limits and potential of historical reuse in speculative decoding.

During the rebuttal, we responded point by point to all reviewers and added experiments and analyses, highlighted in blue in the revised manuscript. Our main revisions are:

  • Methodological distinctions and prefix-feature ablation (Reviewer Zyvn, Reviewer naP3): To clarify the methodological novelty of RECAP, we distinguished it from RACER, Sequoia, OPT-Tree, Goose, C2T, and AdaPLD. A parameter-matched ablation without prefix features further demonstrates that prefix conditioning improves prediction accuracy, calibration, and Spec-Bench throughput, confirming that prefix context provides useful predictive information beyond token identity.
  • RECAP+ execution optimization and cost analysis (Reviewer Zyvn, Reviewer naP3, Reviewer yNX8): Alongside a detailed analysis of offline fitting time and peak memory requirements, we introduced RECAP+ with unchanged scorer weights and verification budgets. Evaluated across three target backbones and five benchmarks, RECAP+ improves execution efficiency, including a 12.3% TPS gain over RECAP on Qwen3-8B Spec-Bench. It also eliminates drafting-overhead regressions previously observed on DeepSeek-Coder coding and math workloads.
  • Controlled baseline comparisons (Reviewer naP3, Reviewer yNX8): Under a shared inference backend and tuned budgets, new comparisons with Goose and EAGLE-3 on Qwen3-8B Spec-Bench show that RECAP achieves higher throughput and lower mean latency than both baselines. RECAP+ further widens the throughput advantage over EAGLE-3, strengthening the evidence for practical efficiency.
  • Concurrency, tensor parallelism, and stochastic decoding (Reviewer Zyvn, Reviewer naP3, Reviewer yNX8): Additional experiments extend the evaluation to eight-request concurrent serving, multi-GPU tensor parallelism, and stochastic sampling. RECAP consistently sustains throughput advantages over RACER under concurrency and across tensor-parallel configurations, while retaining gains under stochastic decoding without retraining, demonstrating its applicability beyond single-request greedy decoding.
  • Generalization and theoretical diagnostics (Reviewer yNX8, Reviewer naP3): Further analyses cover all six Spec-Bench categories and zero-retraining transfer from Qwen3-8B to its Base variant, showing improvements in both acceptance length and throughput. Persistent SWE-bench Verified fallback diagnostics and a tighter pool-geometric reference-tree loss bound additionally clarify the practical and theoretical limits of historical reuse.

Together, these revisions directly address the reviewers' key concerns regarding methodological novelty, inference overhead, practical deployment, generalization, and theoretical analysis.

We sincerely thank the Area Chairs and reviewers again for their constructive feedback, which has strengthened the clarity, rigor, and overall quality of the paper.

Best regards, Authors of Submission 57369

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.