acceptodds
Under review as a conference paper at ICLR 2027

Specula: Lossless KV-Cache Recovery via Precision-Dual Drafting

Abstract

Disaggregated LLM serving separates prefill and decode across nodes, exposing generation to a critical vulnerability: decode-node failures destroy the KV cache and force serving systems to choose between fast-but-lossy compressed loading and slow-but-exact full loading. The dominant recovery strategy treats these as the only two options, implicitly assuming that exactness and speed are fundamentally opposed. In practice, this assumption misses a structural property of the setting. An INT4 checkpoint and its FP16 original are not two independent approximations. They encode the same generation history at two levels of precision, built from identical weights and prompt history. We call this precision duality, and it enables recovery that is both immediate and exact. We present Specula, a speculative recovery framework: the INT4 checkpoint serves as a draft for immediate generation while the exact FP16 state loads in the background, and rejection sampling verifies every committed token against the exact state as it arrives. Per-channel INT4 quantization preserves the variance structure governing argmax stability, yielding 97–99% acceptance across all tested models without tuning. Across five models from 7B to 32B parameters at 8K to 64K context, Specula delivers its first retractable token at least before the strongest lossless baseline commits, sustains continuous output through the recovery window where single-draft and lossless variants deliver one token or none, and shows no measurable quality degradation on HumanEval, GSM8K, and MMLU. By reframing recovery as a precision-duality problem rather than a speed–quality trade-off, Specula suggests a new perspective on fault-tolerant LLM serving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV Cache Recovery, Speculative Recovery, Fault-Tolerant LLM Serving, Disaggregated Inference, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gFWxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Specula, a speculative KV-cache recovery framework for disaggregated LLM serving. It uses an INT4-compressed checkpoint to generate provisional tokens while loading the original FP16 KV cache in the background. Rejection sampling then verifies the speculative tokens to preserve the target distribution of committed outputs. Experiments on five models (7B–32B) across 8K–64K contexts report 3.0–4.4× earlier delivery of the first retractable token compared with the first committed token of the strongest lossless baseline, with 97–99% token acceptance rates and matching benchmark accuracy after verification.

Critical strengths
  1. Practical problem: The paper addresses an important challenge in disaggregated LLM serving, where decode-node failures require costly KV-cache recovery.
  2. Interesting approach: Combining compressed KV-cache loading with speculative generation and exact verification is a promising way to reduce recovery interruptions.
  3. Correctness foundation: The rejection-sampling mechanism provides a theoretical guarantee for the target distribution of committed tokens, assuming correct reconstruction of the FP16 state.
  4. Extensive experiments: Evaluation across five models, multiple context lengths, bandwidth settings, failure scenarios, and ablations provides encouraging evidence of the framework's effectiveness.
  5. Clear presentation: The proposed pipeline and implementation are generally well explained through figures, algorithms, and experimental results.
Critical weaknesses
  1. Latency comparisons require more balanced evaluation. Table 3 compares Specula's first retractable token (t1) with the baseline's first committed token (t2). Although the paper explicitly distinguishes these metrics, they provide different output guarantees. The reported 3.0–4.4× speedup therefore reflects earlier provisional output rather than necessarily faster exact recovery. Reporting Specula's first committed-token latency, including verification overhead, would provide a more complete comparison.
  2. Insufficient analysis of token rollback. Section 3.4 claims that multi-token rollbacks require consecutive rejections. However, Algorithm 1 discards all subsequent draft tokens after the first rejection, meaning a single rejection can invalidate a long speculative suffix. The authors should clarify this statement and report actual rollback-length distributions, including tail statistics, rather than relying primarily on average token acceptance rates.
  3. Limited applicability of retractable tokens. Specula assumes that previously delivered tokens can be withdrawn or corrected. While the paper targets interfaces supporting token-level retraction, many streaming APIs and downstream applications may not support this operation. The authors should further discuss the required interface support and evaluate the impact on applications that expose only committed tokens.
  4. Limited evaluation under realistic serving workloads. The experiments assume preloaded model weights and primarily evaluate recovery using a local NVMe setup. Although the appendix provides some I/O and GPU-work measurements, the effects of concurrent requests, GPU resource contention, and simultaneous node recoveries remain unclear. Additional end-to-end serving experiments would strengthen the practical claims.
  5. Checkpointing overhead and overall efficiency. Section 4.5 reports approximately 1.25× checkpoint storage and a 5–6% throughput penalty for periodic checkpointing at a 1,000-token interval. Although Appendix C includes repeated-failure experiments, the overall trade-off between checkpointing overhead, failure frequency, and serving efficiency is not fully characterized. A broader sensitivity analysis would help establish the net system benefit.
  6. Clarification of exact recovery guarantees. Section 4.5 reports that only half of the tested unannounced-failure scenarios reproduce all 64 greedy tokens from the full-recomputation reference, with differences attributed to floating-point effects. While Appendix D establishes distributional correctness under its assumptions, the authors should clarify the conditions required for bit-identical recovery and distinguish exact recovery relative to the restored FP16 state from equivalence with full recomputation.
Questions for authors
  1. What is Specula's time to first committed token, including verification, compared with FP16-Pipelined?
  2. How many speculative tokens are retracted per recovery event, particularly at P95 and P99?
  3. How does the framework handle applications that do not support token retraction?
  4. How does performance change under concurrent serving requests and GPU/I/O contention?
  5. Under what conditions can bit-identical recovery be guaranteed following unannounced failures?
1 comment

We thank the reviewer for recognizing the practical problem and promising approach. Specula uses two precisions of the same KV state to generate during recovery and verify before commitment.

(1) Earlier output and committed progress (Q1).

The 3.0–4.4× earlier provisional output targets interfaces that display revisable text during restoration, while the accepted prefix commits together as a batch at t2. Across six configurations in one environment, with seven cold trials per policy, Time-to-64 is 5.0–9.5% shorter in four configurations. This end-to-end measure starts at recovery and includes restoration, drafting, verification, and any subsequent decoding, reflecting cumulative committed progress rather than steady-state throughput. Earlier preview and batch-verified commitment serve complementary service objectives, with t2 integrating the serialized INT4 phase and a single batched verification pass (+7–73 ms over a standard decode step).

(2) Rollback and retained draft work (Q2).

Under Algorithm 1, verification commits the accepted prefix prior to the first rejection. We updated Section 3.4 with empirical accepted-prefix and rollback distributions directly aligned with this behavior. On 50 independent PG-19 documents per configuration, Llama-3-8B (8K), Mistral-7B (32K), and Qwen2.5-32B (8K/16K), at timed-run draft lengths N=12/20/7/9, achieve mean rollback lengths of only 5.48/8.46/1.70/3.22 tokens and zero-rollback rates of 30%/36%/62%/48%. Notably, Qwen2.5-32B (8K) achieves complete draft acceptance on 62% of documents, demonstrating substantial retained work across configurations (full distributions are in the appendix). Separate evaluations on 200 distinct books per configuration provide statistically grounded P95 tail-rollback characterization (e.g., a P95 rollback of 12 tokens at N=13 for Mistral-7B (8K)).

(3) Interface requirements (Q3).

Specula is designed to deliver revisable previews during FP16 restoration. Our prototype exposes a rollback callback and records intended delivery times in the recovery harness to support direct integration with retraction-capable client interfaces. For committed-only interfaces, drafts can simply be buffered until verification at t2, still retaining the cumulative Time-to-64 gains. Thus, Specula provides earlier provisional availability for interactive clients as well as faster cumulative commitment across tested configurations.

(4) Gains under contention (Q4).

Under controlled GPU contention, when co-located with either an ordinary decode request or a second simultaneous recovery, Specula continues to deliver a 2.1–2.9% Time-to-64 advantage (Llama-3-8B, 8K, one A800, seven cold trials per condition), extending its 7.6% single-request gain to multi-request serving. Co-running ordinary decode requests maintain comparable full-trial step latency across policies, showing that Specula preserves serving stability alongside cumulative committed-progress gains. Experiments are conducted under time-shared GPU serving with preloaded weights and local NVMe.

(5) Checkpoint cost and deployment trade-off (Weakness 5).

Checkpoint cadence offers a flexible, tunable knob between checkpoint freshness and steady-state overhead. In storage, the compressed INT4 state requires only 1.25× FP16 total storage (+25% extra storage), with an amortized INT4 increment of just 1.0–2.1 ms per generated token on Llama-3-8B (6K–7.6K). With a separately measured 38.9 ms decode step at 8K, configuring the checkpoint interval to 1,000 tokens keeps estimated steady-state overhead low at ~6% (versus 9–11% at a denser 500-token interval). This modest, configurable cost directly unlocks 3.0–4.4× earlier provisional availability and faster cumulative commitment upon recovery.

(6) Correctness and bitwise identity (Q5).

With accurate FP16 reconstruction, verification strictly preserves the restored FP16 verifier's exact argmax at T=0 and its exact output distribution at T>0, regardless of draft quality. Across 1,050 PG-19 trials, Specula achieves 99.87% agreement (12,117/12,133 positions) with sequential re-decoding. This verifier-relative equivalence is the principled correctness guarantee for batched speculative verification: because batched verification and sequential recomputation employ distinct floating-point reduction orders and kernels, cross-path bitwise identity would require identical chunking and sequential tokenwise execution. Finally, the updated HumanEval and GSM8K tables directly characterize draft-only quantization quality, separated from end-to-end recovery evidence.

Together, these results demonstrate that compressed checkpoints enable both earlier provisional output and faster committed progress across tested serving settings.

Reviewer_2VHeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Specula, a speculative KV cache recovery framework for disaggregated LLM serving. Following a decode-node failure, it loads an INT4 checkpoint to generate retractable draft tokens while streaming the exact FP16 cache in the background. Once the exact state is available, rejection sampling verifies the drafts, preserving the target distribution for committed output. The design combines per-channel key and per-token value quantization with pipelined loading and drafting during the recovery window. Experiments cover five models from 7B to 32B parameters at context lengths from 8K to 64K. The authors report 97–99% per-token acceptance and first retractable tokens arriving 3.0–4.4× earlier than the strongest lossless baseline’s first committed token. Quality comparisons, bandwidth sweeps, ablations, and unannounced-failure experiments further evaluate output fidelity and recovery behavior.

Critical strengths
  1. Specula makes useful progress during KV recovery by treating the compressed checkpoint as a temporary draft state while loading the exact cache. This offers a practical way to reduce the initial output gap for interfaces that support token retraction.
  2. Rejection-sampling verification provides a principled basis for preserving the target distribution of committed tokens. Experiments across five models and multiple context lengths report high draft acceptance, supporting the feasibility of using quantized KV states for speculative recovery.
Critical weaknesses

The headline 3.0–4.4× speedup compares Specula’s first retractable draft token with the baseline’s first committed token. Although this distinction is disclosed, it does not establish a comparable improvement in lossless recovery latency. The paper needs stronger evidence on time to committed output and the cost of corrections to substantiate its practical advantage. The rollback analysis conflicts with Algorithm 1, which discards all remaining drafts after the first rejection. A single rejection can therefore trigger a long rollback, contrary to the claim that multi-token rollbacks require consecutive rejections. Moreover, counting matches across the full draft window can include tokens beyond the first rejection that are never committed, potentially overstating recovery goodput.

Questions for authors

How does Specula compare with FP16-Pipelined in time to first committed token and time to produce a fixed number of committed tokens, including verification and rollback costs? How do the claims that multi-token rollbacks require consecutive rejections and expected retractions remain below two regardless of draft length reconcile with Algorithm 1, which discards the entire suffix after the first rejection? How would the cross-model recovery-goodput results change if the numerator counted only the accepted prefix actually committed, rather than matches across the full draft window?

1 comment

We thank the reviewer for recognizing the value of making progress during KV recovery and the principled basis of FP16 verification. Specula uses a compressed checkpoint to generate provisional output during FP16 restoration, verification then allows the accepted prefix to commit together. We summarize both forms of progress and the updated metrics below.

(1) Earlier output and committed progress.

The 3.0–4.4× improvement compares Specula's first retractable token with FP16-Pipelined's first committed token, enabling earlier revisable previews in retraction-capable interfaces. We separately measure first commitment and time to a fixed amount of committed output.

Because the accepted prefix commits together at t2, recovery-time drafting also advances cumulative commitment. Across six configurations in the same software environment, with seven cold trials per policy, Specula reaches 64 committed tokens first in four configurations, by 5.0–9.5%, with non-overlapping seven-trial ranges. Time-to-64 runs from recovery start and includes restoration, draft generation, verification, and any subsequent decoding required to commit 64 tokens, measuring end-to-end committed progress rather than steady-state throughput. At t2, batch commitment follows the serialized INT4 phase (66–78% of the pre-commit interval) and a single batched verification pass (+7–73 ms over a standard decode step).

In an additional evaluation on ten PG-19 books per configuration, with three cold trials per book, Specula reaches 64 committed tokens first on 7/10 books for Llama-3-8B (8K), 8/10 for Mistral-7B (32K), and 9/10 for Qwen2.5-32B (32K). These results extend the committed-progress benefit well beyond a single prompt. For Llama-3-8B (8K) and Mistral-7B (32K), the timed accepted prefixes also match same-book rollback measurements when compared up to the shorter draft length.

(2) Rollback behavior and Algorithm 1.

Verification governs which provisional tokens become committed, committing the accepted prefix prior to the first rejection under Algorithm 1. We updated Section 3.4 to directly characterize this single-rejection rollback behavior using empirical distributions rather than a theoretical bound.

We now characterize rollback directly on 50 independent PG-19 documents per configuration, with N fixed to the corresponding timed-run median draft length: 12, 20, 7, and 9 tokens for Llama-3-8B (8K), Mistral-7B (32K), and Qwen2.5-32B (8K and 16K), respectively. Mean rollback lengths are 5.5, 8.5, 1.7, and 3.2 tokens, while 30%, 36%, 62%, and 48% of documents achieve zero rollback. The main text presents these configurations alongside commit timing, and the appendix reports the full accepted-prefix and rollback distributions across evaluated PG-19 configurations. These measurements directly quantify how much draft work is retained, while Time-to-64 captures its end-to-end effect on committed progress.

(3) Recovery Goodput based on retained output.

We now count only the accepted prefix that survives verification:

Recovery Goodput = Accepted Prefix Length / (t2 - t0).

Tokens after the first rejection are excluded so that the metric strictly reflects retained draft work, while full-window matches remain a diagnostic. The updated cross-model comparison uses accepted-prefix goodput and keeps the two software environments separate.

While individual single-prompt goodput varies with the first-rejection position on a specific prompt, multi-prompt evaluations confirm that recovery-time drafting consistently yields positive end-to-end gains across diverse texts, as shown by Mistral-7B (32K) reaching 64 committed tokens sooner on 8/10 PG-19 books. Accepted-prefix goodput measures retained draft work per recovery second, and Time-to-64 measures end-to-end committed progress, together, these metrics cleanly connect early provisional generation with cumulative committed progress.

Reviewer_Aj3sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Specula for recovering from decode-node failures in LLM serving. It generates provisional tokens from an INT4 KV-cache checkpoint while loading the exact FP16 cache, then uses speculative verification to ensure that committed outputs follow the target distribution. Experiments evaluate recovery latency and output fidelity across five models.

Critical strengths

Using a compressed checkpoint as a readily available draft is a practical idea. The evaluation covers several model families and context lengths, includes a pipelined FP16 baseline, and explicitly distinguishes provisional output from committed output.

Critical weaknesses
  1. Section 3.4 states that multi-token rollbacks require consecutive rejections and that expected retractions remain below two regardless of draft length. However, Algorithm 1 discards the entire remaining suffix after the first rejection. A single rejection can therefore retract many tokens, and a 97–99% per-token acceptance rate alone does not establish the claimed bound.

  2. Table 3 compares Specula’s first retractable token with the baselines’ first committed token. This demonstrates earlier provisional output, but does not establish faster lossless recovery. The evaluation should prominently report Specula’s time to first committed token, accepted-prefix length, and actual retraction counts under the same configurations.

Questions for authors
  1. How is the claimed draft-length-independent retraction bound derived, given that the first rejection invalidates the entire subsequent draft suffix?

  2. How does Specula compare with FP16-Pipelined in time to first committed token and committed-token throughput, including verification overhead?

1 comment

We thank the reviewer for recognizing the practicality of using a compressed checkpoint as a readily available draft. Specula produces its first provisional token 3.0–4.4× earlier than FP16-Pipelined produces its first committed token. For an interface supporting revisable previews, this offers a way to expose provisional content during FP16 restoration. The accepted prefix then commits together after verification, allowing recovery-time generation to advance subsequent committed-output progress. We clarify the rollback behavior and quantify this benefit below.

(1) Actual rollback behavior and retained work (Q1).

Under Algorithm 1, verification commits the entire accepted prefix prior to the first rejection, so useful recovery-time work is determined by the prefix that survives verification. To characterize this directly, we updated Section 3.4 to reflect Algorithm 1's single-rejection rollback rule and replaced the theoretical bound with empirical accepted-prefix and rollback distributions.

We evaluated 50 independent PG-19 documents per configuration for Llama-3-8B (8K), Mistral-7B (32K), and Qwen2.5-32B (8K and 16K). In the same software environment, the draft lengths N were fixed to 12, 20, 7, and 9 tokens, matching the corresponding timed-run draft-length medians. Mean rollback lengths were only 5.5, 8.5, 1.7, and 3.2 tokens, while 30%, 36%, 62%, and 48% of documents achieved zero rollback. These measurements confirm that recovery-time drafts consistently retain substantial useful prefixes. The main text highlights these four configurations alongside commit timing, and the appendix reports the complete per-configuration accepted-prefix and rollback distributions. Recovery Goodput counts the accepted prefix only.

The acceptance statistics are clearly labeled by evaluation setting. HumanEval achieves 97.1–99.0% per-token agreement under FP16-corrected histories (across four evaluated models up to 2,048 tokens). On long PG-19 documents, the acceptance ratio (accepted tokens divided by accepted tokens plus first rejections) is 89.9–96.0% within the native attention window. Together with the empirical rollback evaluation, these statistics directly quantify the work retained from recovery-time drafting.

(2) First commitment and cumulative committed progress (Q2).

Specula connects early provisional availability with batch-verified commitment at t2. Rather than committing tokens one by one after restoration, Specula runs a serialized INT4 phase before FP16 loading begins (66–78% of the pre-verification interval) and a single batched verification pass (+7–73 ms over a single baseline decode step) so that the entire accepted prefix commits together at t2 (component medians are computed separately, so their sum need not equal the median total interval).

Because accepted draft work commits as a batch at t2, Specula advances cumulative committed progress beyond provisional availability. Across six configurations measured in the same software environment with seven cold trials per policy, Specula reaches 64 committed tokens 5.0–9.5% sooner in four configurations, with non-overlapping seven-trial ranges. Time-to-64 measures the full end-to-end interval from recovery start, including state restoration, draft generation, verification, and any subsequent decoding, capturing cumulative committed progress rather than steady-state throughput.

Additional timing across ten distinct PG-19 books per configuration, with three cold trials per book, confirms that this benefit generalizes across prompts. Specula reaches 64 committed tokens first on 7/10 books for Llama-3-8B (8K), 8/10 for Mistral-7B (32K), and 9/10 for Qwen2.5-32B (32K). Furthermore, for Llama-3-8B (8K) and Mistral-7B (32K), the timed accepted prefixes match same-book rollback measurements when compared up to the shorter draft length.

Together, these measurements demonstrate a recovery path with dual benefits: earlier provisional availability for revisable-output interfaces, and faster cumulative commitment via FP16-verified batch commits.

Comments on the paper

1 comment

We thank the reviewers and Area Chair for the discussion. Specula's central contribution is precision-dual recovery: using INT4 and FP16 representations of the same generation history to make progress while a failed request's full-precision KV cache is being restored. This mechanism connects early provisional generation with verification-guided commitment. Its key contributions and supporting evidence are summarized below.

1. Same-model drafting during restoration, with verifier-defined exactness (Sections 3.3–3.5; Appendix D). Specula reuses the same model and generation history at two precision levels, requiring no auxiliary draft model or additional training. The INT4 state enables drafting during FP16 restoration; verification then commits the accepted prefix and applies any correction required by the FP16 verifier. Committed tokens follow the distribution defined by the restored FP16 verifier. This makes the compressed checkpoint a source of useful recovery-time work.

2. Earlier provisional availability across models and contexts (Section 4.3; Table 3). Across five models from 7B to 32B parameters at 8K–64K context, Specula makes its first provisional token available 3.0–4.4× earlier than FP16-Pipelined's first committed token. For correction-capable interfaces, these measured provisional-token timings offer an early-preview opportunity during restoration. The prototype records intended delivery times and exposes a rollback callback to support integration with such interfaces.

3. Accepted draft work advances cumulative commitment (Section 4.4; Table 8). The accepted prefix commits together after verification. With seven cold trials per policy, Specula reaches 64 committed tokens 5.0–9.5% sooner in four of six configurations. Additional evaluations across ten PG-19 books per configuration, with three cold trials per book, show earlier Time-to-64 on 7/10, 8/10, and 9/10 books for Llama-3-8B (8K), Mistral-7B (32K), and Qwen2.5-32B (32K), respectively. These results demonstrate that useful work performed during restoration can advance cumulative committed output across multiple prompts.

The revised manuscript supports these contributions with accepted-prefix goodput, rollback/P95 distributions (Tables 4 and 7), contention measurements (Table 9), and checkpoint-cost estimates (Section 4.5). Together, the evidence shows how precision-dual recovery enables earlier provisional availability and turns accepted recovery-time work into committed progress.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.