- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RECAP, a speculative decoding framework that learns when historical target-model predictions remain useful under the current prefix. It combines context-aware memory scoring with adaptive draft-tree construction, improving candidate acceptance and decoding efficiency while retaining target-model verification.
Critical strengths
- The paper identifies a meaningful limitation of historical prediction reuse: matching token identities does not guarantee that predictions transfer across contexts. Preserving contextual information provides a well-motivated way to address this problem.
- The framework connects candidate selection with verification-budget allocation. Its treatment of unsupported continuations and adaptive tree expansion provides a coherent mechanism for directing computation toward promising branches.
- The evaluation includes multiple target models, diverse workloads, controlled ablations, and detailed runtime analysis. Comparisons on identical output text and investigations of numerical divergence strengthen the empirical evidence.
Critical weaknesses
- The theoretical guarantees have limited demonstrated relevance to the deployed system. They assume uniformly bounded estimation errors within a fixed candidate space, whereas practical decoding also involves candidate omissions, fallback construction, and changes in future memory caused by current decisions.
- Generalization remains dependent on target-specific training and workload characteristics. Domain specialization can determine whether improved acceptance translates into an actual speed advantage, and the evaluation primarily concerns greedy decoding.
- Comparisons with trained drafters are less controlled than the central comparisons with historical-reuse methods. Separate execution campaigns, different implementations, and estimated timing adjustments make the broader efficiency advantage harder to assess.
Questions for authors
Can the authors empirically assess how informative the bounds are under realistic scoring errors and quantify the losses attributable to restricted candidate coverage and fallback behavior?
How well does a frozen scorer transfer across domains and related target models, and what changes are needed to retain its benefits under stochastic decoding?
Can RECAP and strong trained-drafter baselines be evaluated under a common optimized serving stack, with directly measured latency and independently tuned drafting budgets?
4 comments
Response to Reviewer yNX8, Part 2 of 3: Domain Reuse and Model Transfer
Q2: How well does a frozen scorer transfer across domains and related target models, and how do target-specific training and domain specialization affect whether acceptance improvements yield actual speedups?
A2: We thank the reviewer for asking about scorer generalization. For each target model, the main experiments use one scorer trained on heterogeneous LongBench contexts and reuse it across all five evaluation benchmarks without per-domain retraining, while domain specialization through RECAP-C remains an optional adaptation. Regarding whether acceptance improvements yield actual speedups, observing substantial room for further implementation optimization in RECAP, we reuse context encodings, pack frontier inputs and probability outputs, and fuse feature preparation with fallback handling to obtain RECAP+, which keeps the frozen scorer weights and verification budget unchanged while substantially increasing end-to-end TPS, so we also include RECAP+ in the comparisons that follow.
- Cross-task domain reuse. While the main-text table reports overall performance across five evaluation benchmarks using a single LongBench-trained frozen scorer, we further break down Qwen3-8B results on Spec-Bench by task category:
| Task category | AR TPS | RACER MAT / TPS | RECAP MAT / TPS | RECAP+ MAT / TPS | RECAP+ TPS gain over RACER |
|---|---|---|---|---|---|
| Multi-turn conversation | 51.02 | 2.50 / 106.36 | 3.31 / 116.71 | 3.32 / 132.14 | +24.2% |
| Translation | 50.32 | 1.69 / 67.66 | 2.00 / 65.82 | 2.00 / 76.07 | +12.4% |
| Summarization | 50.05 | 2.51 / 102.42 | 3.25 / 108.91 | 3.25 / 123.82 | +20.9% |
| Question answering | 51.16 | 2.22 / 96.64 | 3.04 / 109.58 | 3.04 / 116.96 | +21.0% |
| Mathematical reasoning | 50.77 | 2.74 / 118.33 | 3.72 / 128.05 | 3.71 / 146.42 | +23.7% |
| Retrieval-augmented generation | 50.26 | 2.38 / 98.85 | 3.36 / 113.22 | 3.35 / 127.70 | +29.2% |
RECAP improves MAT across all six categories and increases end-to-end throughput in five of the six categories outside very short translation outputs, raising overall MAT by 34.3% and TPS by 10.2% over RACER. With this execution optimization, RECAP+ achieves higher end-to-end throughput across all six categories and raises overall TPS by 23.8% over RACER, demonstrating strong domain generalization.
- Transfer to a related target model. We apply the frozen Qwen3-8B scorer directly to Qwen3-8B-Base, retaining source-model embedding features without retraining, and evaluate on Spec-Bench on the same host under default settings:
| Method | MAT | End-to-end TPS | Speedup over AR |
|---|---|---|---|
| AR | 1.000 | 49.69 | 1.000x |
| RACER | 2.787 | 114.15 | 2.297x |
| RECAP | 3.651 | 119.64 | 2.408x |
Without retraining, RECAP improves MAT by 31.0% and end-to-end TPS by 4.8% over RACER, demonstrating effective transfer of the frozen scorer across related target models.
- Domain specialization and actual speedups. On smaller target models with faster verification passes such as DeepSeek-Coder-6.7B-Instruct, fine-grained drafting overhead accounts for a larger share of runtime, so acceptance gains from a general scorer do not automatically yield a net speedup. On the coding and math tasks of LiveBench, RECAP+ raises throughput without retraining from 121.8 and 146.3 TPS for RECAP to 141.3 and 164.2 TPS, surpassing RACER at 137.8 and 161.3 TPS, while optional domain specialization further expands acceptance gains so that RECAP-C reaches 157.4 and 165.3 TPS and RECAP-C+ reaches 167.9 and 176.2 TPS.
Response to Reviewer yNX8, Part 3 of 3: Stochastic Decoding and EAGLE-3
Q3: What changes are needed to support stochastic decoding, and how does RECAP perform beyond greedy decoding?
A3: We thank the reviewer for raising stochastic decoding. At each visited prefix, the verifier samples from the prescribed target distribution. If the token matches a drafted child, traversal continues, and otherwise the round ends and the sampled token is retained as the next root. The scorer guides tree allocation, while token selection follows the target sampling law, preserving the prescribed distribution under consistent exact target conditionals.
We evaluate Qwen3-8B on Spec-Bench under the default settings, keeping both the target model and the original trained scorer frozen without additional retraining.
| Decoding | RACER MAT / TPS | RECAP MAT / TPS |
|---|---|---|
| Greedy | 2.45 / 103.84 | 3.29 / 114.41 |
| T=0.7, top-p=0.9 | 2.38 / 84.38 | 3.18 / 93.92 |
| T=1.0, top-p=1.0 | 2.28 / 89.94 | 2.97 / 96.17 |
Under the two stochastic settings, RECAP achieves 11.3% and 6.9% higher throughput than RACER, respectively, maintaining its advantage under stochastic decoding.
Q4: Can RECAP and strong trained-drafter baselines be evaluated under a common optimized serving stack, with directly measured latency and independently tuned drafting budgets?
A4: We thank the reviewer for suggesting a controlled comparison with trained drafters. Under the default settings, we evaluate Qwen3-8B on Spec-Bench using a shared target and KV backend together with an optimized execution stack, comparing EAGLE-3 against AR, RACER, RECAP, and RECAP+ from A2. Drafting budgets are independently selected on a separate validation split with the remaining evaluation settings matched across methods, and end-to-end request latency is measured directly.
| Method | MAT | Mean latency / s | End-to-end TPS |
|---|---|---|---|
| AR | 1.000 | 5.58 | 50.80 |
| EAGLE-3 | 3.173 | 2.98 | 94.91 |
| RACER | 2.448 | 2.74 | 103.84 |
| RECAP | 3.287 | 2.48 | 114.41 |
| RECAP+ | 3.285 | 2.21 | 128.53 |
While achieving a mean accepted length comparable to that of EAGLE-3, RECAP avoids sequential draft-model forward passes, enabling RECAP and RECAP+ to deliver 20.5% and 35.4% higher end-to-end throughput, respectively, along with lower mean request latency.
Follow-up on Our Rebuttal
Dear Reviewer yNX8,
Thank you again for your detailed and constructive feedback.
We would like to kindly follow up on our rebuttal. We have provided additional empirical analyses of the theoretical bounds and fallback behavior, evaluations of scorer transferability and stochastic decoding, and a controlled comparison with EAGLE-3 under a shared execution stack with directly measured latency.
We hope these additional results help address your concerns. We would sincerely appreciate it if you could take a moment to review our responses and share any remaining questions or concerns.
Thank you very much for your time and consideration.
Best regards, The Authors
Response to Reviewer yNX8, Part 1 of 3: Theoretical Bounds and Fallback
A1: We thank the reviewer for this question. Our analysis bounds the worst-case loss in expected accepted length caused by scoring errors propagating through tree allocation within a fixed candidate space, and we complement this analysis with two empirical diagnostics.
Fallback under practical usage. To examine the fallback rate in practical usage, we further inspect the paper's persistent Qwen3-32B agent sessions on SWE-bench Verified, which retain historical information across model calls and more closely reflect realistic multi-turn agent scenarios. Experiments show that root fallback and fixed fill together account for 6.35% of verification rounds, so learned budget allocation governs the majority of rounds. Moreover, fallback preserves the accepted output sequence and, aside from routine memory updates from verified nodes, leaves the tree-planning rule unchanged in other rounds, so at this low trigger rate it does not significantly affect overall generation.
Reference-tree comparison and a geometric bound. On Spec-Bench, we evaluate RECAP with Qwen3-8B. At each state, we fix the historical memory, observed expanded candidate pool, retrieval constraints, and verification budget, and construct a reference-optimal tree using the target model's full-vocabulary softmax probabilities. The following expected-length quantities are evaluated under the target-softmax reference distribution p.
The expected-length loss remains below the corresponding bound across all evaluated states. However, the mean bound substantially exceeds the observed loss, reflecting the conservatism of the worst-case analysis. To further characterize the loss using the observed candidate geometry, we define
\le H(C,T_q) =\max_{\pi\in\mathcal P(C)}|\pi\setminus T_q|,$$ where $T_p$ is the pool-constrained reference-optimal tree, $T_q$ is the constructed RECAP tree, and $\mathcal P(C)$ denotes the root-to-leaf paths in the fixed candidate pool $C$. Since verification follows a single continuation path, the pool-constrained optimum's accepted-length advantage cannot exceed the number of candidate nodes on that path absent from $T_q$. Using only the candidate pool and the constructed tree, this bound has a mean of 1.990 tokens across the evaluated states, compared with an observed mean loss of 0.360 tokens, and holds in every state.