- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ResiSpec targets a bottleneck in multi-candidate speculative decoding called Residual Drift. After an early candidate is rejected, the residual target shifts mass into the draft’s blind spots, while later candidates were pre-sampled from q; acceptance then collapses and extra branches yield little gain. ResiSpec replaces the draft q at verification with a proxy k that reserves residual mass aligned with high-density regions of q, claiming exact target sampling without an extra model forward. Empirically, on Llama-2-7B with a 68M draft over CNN/DM, OpenWebText, and C4, it improves accepted length and throughput versus SpecInfer (up to about 1.86× / 1.69×, and up to 1.92× in tree ablations), and can be plugged into Sequoia and EAGLE. The contribution is a verification-time residual-shaping rule for tree/multi-candidate speculative sampling, not a new drafter.
Critical strengths
Clear diagnosis. Residual Drift follows directly from sequential p/q residual updates; Figure 1 shows later candidates’ acceptance falling near zero, motivating the method.
Exactness-aware design. The proxy/residual construction aims to keep sampling lossless while making pre-sampled siblings useful after rejections.
Plug-in framing. Gains appear as an add-on to SpecInfer, Sequoia, and EAGLE, with runtime broken down to show no extra target forward.
Critical weaknesses
Narrow, dated evaluation. Main results use Llama-2-7B (appendix up to ~7B) with tiny drafts on CNN/DM, OpenWebText, and C4—self-reported tok/s on a 3090—not SpeedBench/SpecBench-style suites, chat/code/math tasks, or modern large (let alone trillion-scale) targets.
Weak SOTA coverage. No comparison to recent parallel drafters such as DFlash or DSpark; Medusa/Hydra-style systems are largely absent. Sequoia/EAGLE appear mainly as “+ResiSpec” add-ons, not a full fair bake-off against the strongest current stacks.
Theory of drift is thin. “Later candidates almost always fail” is argued by support mismatch plus Figure 1, not a tight theorem; severity under diverse models, temperatures, and tree shapes is under-proven.
Limited practical relevance to today’s SPD. Many deployments use greedy exact-match acceptance, where this residual-shaping story does not apply; even for sampling SD, evidence stays on small Llama-2-era setups, so claimed speedups may not transfer to current serving.
Questions for authors
See weaknesses above.