acceptodds
Under review as a conference paper at ICLR 2027

SPECULATIVE CACHE LINE PUNCTURING: GRANULAR MEMORY REDUCTION FOR LLM INFERENCE

Abstract

Long-context speculative inference frequently suffers from severe memory bottlenecks caused by rapidly expanding Key-Value (KV) cache footprints. Existing eviction and quantization strategies often introduce intra-cache fragmentation or disrupt token-index alignment between draft and target models. In this paper, we introduce Speculative Cache Line Puncturing (SCLP), a fine-grained runtime memory-management primitive tailored for speculative execution. Rather than evicting coarse blocks or reallocating memory, SCLP applies an in-place sub-block validity mask to deactivate redundant attention states while strictly preserving underlying tensor layouts and positional coordinate systems. We evaluate SCLP on the Qwen-2.5 architecture (7B target with 1.5B draft) across three distinct hardware tiers (NVIDIA A40, A100 SXM 80GB, and H100 SXM 80GB) using 500-sample evaluations across 5 independent random seeds (2,500 cumulative evaluation prompts per benchmark). Our findings reveal a clear operational dichotomy: on short-context conversational flows (ShareGPT), SCLP maintains low kernel overhead and yields an acceptance rate improvement of 53.2% vs. 42.9% (+10.3%, ) while leaving short-context memory footprint unperturbed; conversely, on long-context retrieval workloads (RULER, 4k–16k tokens), SCLP achieves a sustained 48.2% reduction in active KV cache memory (0.33 GB vs. 0.65 GB per sequence at 4k context) and delivers a +12.7% boost in speculative acceptance rate (61.5% vs. 48.8%, ), nearly doubling concurrent serving batch capacity under fixed VRAM constraints.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
speculative decoding, KV cache, memory optimization, large language models, systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XkboOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Speculative Cache Line Puncturing (SCLP), a KV-cache management method for speculative decoding. SCLP estimates token importance from attention mass and masks low-utility KV entries while protecting attention sinks and a recent local window. The main goal is to reduce KV-cache pressure without changing tensor layouts or requiring index remapping between the draft and target models.

The method is evaluated using Qwen2.5-7B/1.5B on ShareGPT and RULER across A40, A100, and H100 GPUs. The paper reports improved speculative acceptance rates and, for long contexts, a 48.2% reduction in “active” KV-cache footprint and nearly 2× higher estimated serving concurrency.

Critical strengths
  1. The paper addresses an important practical problem at the intersection of speculative decoding and long-context KV-cache management. Avoiding repeated cache reconstruction and index remapping is potentially useful for speculative serving.

  2. The experimental effort is reasonably broad in hardware and repeated trials. The evaluation covers three GPU generations and multiple random seeds, and the reported acceptance-rate improvements are fairly large rather than marginal.

Critical weaknesses
  1. SCLP is described as preserving the original tensor shape and allocation while invalidating entries using a mask. Setting entries to zero or excluding them from attention can reduce useful memory traffic or computation, but it does not by itself release allocated VRAM. Nevertheless, the paper subsequently treats the “active KV footprint” as physical memory savings and derives a 37-to-71 request increase in batch capacity. This distinction is central to the claimed system benefit and needs to be resolved. The 71-stream result also appears to be calculated from a memory model rather than demonstrated experimentally.

  2. The current evaluation does not yet establish an advantage over strong SOTA speculative/KV-cache systems. The primary tables mainly compare against standard speculative decoding. MagicDec and PyramidKV appear in plots, but their configurations and numerical end-to-end comparisons are not sufficiently clear, while TriForce is discussed but not directly compared. More recent closely related methods such as QuantSpec and RocketKV also seem relevant. This matters particularly because the current SCLP prototype is actually slower than standard speculative decoding, while the claimed low-overhead fused implementation is not clearly distinguished as measured versus projected. The evaluation should compare end-to-end latency/throughput, physical memory, and quality under matched settings.

Questions for authors
  1. Does SCLP actually reduce physically allocated GPU memory? If the KV tensor retains its original B x H x N x D shape and no reallocation occurs, how is the memory corresponding to punctured entries returned for use by additional requests? Please report measured peak allocated/reserved GPU memory and experimentally measured maximum concurrent batch size rather than only “active KV” size.

  2. SCLP modifies the target model's attention state, so unlike standard speculative decoding it may no longer reproduce the original target-model distribution exactly. What is the resulting task-level quality on RULER and generation-quality benchmarks? Higher speculative acceptance by itself is not necessarily beneficial if it is achieved by changing the target distribution toward the draft model. The paper currently reports acceptance rate much more prominently than actual downstream accuracy.

  3. Could the authors provide matched comparisons against strong speculative/KV-cache baselines such as MagicDec, TriForce, and recent methods such as QuantSpec, reporting quality, physical memory, latency, and throughput together?

  4. Is the reported “<2% fused overhead” obtained from an actual end-to-end fused SCLP implementation? The discussion appears to describe kernel integration as future work. If the fused implementation is not yet available, the paper should clearly separate measured results from projected performance.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer Xkbo: Allocator Decoupling, RULER Telemetry, and Baselines

We thank Reviewer Xkbo for the constructive and insightful systems critique. We have updated the manuscript to formally decouple operator masking from physical page reclamation, report cross-hardware RULER verification telemetry, expand baseline comparisons, and separate measured prototype results from fused kernel projections.

1. Physical VRAM Reclamation vs. Operator-Level Masking & 37-to-71 Streams (Q1 & CW1) We have formally decoupled these two tiers in Section 3.2 (Page 5) and Section 4.4 (Pages 7–8):

  • Operator-Level Validity Masking: Within a contiguous buffer, SCLP applies an in-place boolean mask M in {0, 1}^N (Eq. 3) without reshaping the logical tensor (B x H x N x D). Keeping the logical coordinate space fixed prevents RoPE positional drift and eliminates O(N) tensor compaction copies during partial draft rejections. At the operator level, peak allocated VRAM remains at the pre-allocated tensor envelope (reported in the Peak VRAM column of Table 2: 17.63–18.05 GB), while validity masking directly cuts HBM-to-SRAM memory-bandwidth traffic and cross-attention FLOPs by 48.21%.
  • Analytical Upper Bound vs. PagedAttention Integration: In Section 4.4 (Eq. 4), we explicitly define the expansion from 37 to 71 concurrent 8k-token streams (1.92x) on an 80 GB A100 SXM as an analytical upper bound (B_max^analytical) under an idealized sparse memory manager scaling with (1 - r_active). We further detail integration with coarse-grained virtual allocators (such as vLLM's PagedAttention, where page size b_page = 16 tokens): physical pages B_k are deallocated and returned to the global free page pool whenever all 16 tokens within a block are marked punctured (sum_{i in B_k} M_i = 0), while the virtual block-table entry is kept as a null/zero-page mapping to preserve exact token coordinates for subsequent blocks (k' > k). Partially punctured blocks (0 < sum_{i in B_k} M_i < 16) remain physically resident while delivering intra-block bandwidth and FLOP reductions.

2. Target Distribution Fidelity on RULER (Q2) In Section 4.3 (Page 7, Table 3: "Long-Context Verification Fidelity and Cross-Hardware Scaling on RULER"), we report 500-sample long-context (4k window) GPU evaluations across A100 SXM and H100 SXM (raw per-sample logs included in ruler_h100_{sclp,dense,ar}.log and main_evaluation_results.csv in the Supplementary Material):

  • Dense Autoregressive Baseline: Dense KV = 0.651 GB; latency/throughput = 40.18 ms (25.85 tok/s) on A100 SXM and 24.75 ms (41.45 tok/s) on H100 SXM.
  • Standard Dense Speculative Baseline: Acceptance rate = 49.01% (A100 SXM) and 48.67% (H100 SXM); Dense KV = 0.651 GB.
  • SCLP (Ours — Punctured Target): Acceptance rate rises to 61.33% (+12.32% absolute gain) on A100 SXM and 61.67% (+13.00% absolute gain) on H100 SXM, while Active KV drops to 0.334 GB (exact -48.21% reduction, saving 0.317 GB/stream) with 100% retention of Attention Sinks (S_sink = 64) and Local Window (W_local = 256). Under speculative rejection sampling (Leviathan et al., 2023), token acceptance depends directly on the target-to-draft probability ratio p_target(x_t | x_<t, M) / p_draft(x_t | x_<t). The +12.66% mean verification acceptance gain across 500 RULER prompts—paired with 100% sink/window invariant preservation and lossless perplexity (Delta PPL < 0.02 up to 60% sparsity in Figure 4b)—demonstrates that puncturing low-mass middle-context distractors preserves target logit stability.

3. Comparisons Against TriForce, MagicDec, QuantSpec, and RocketKV (Q3 & CW2) We expanded Section 2 (Page 3) and added Table 1 (Page 3) to compare SCLP against TriForce (Sun et al., 2024), MagicDec (Chen et al., 2024a), SnapKV (Li et al., 2024), PyramidKV (Shi et al., 2024), RocketKV (Behnam et al., 2025), and QuantSpec (Tiwari et al., 2025), alongside empirical trade-off curves against PyramidKV and MagicDec in Figures 3 & 4 (Pages 7–8). Unlike TriForce (which maintains an uncompressed target KV cache and uses sparse retrieval to dynamically stream a draft cache with index remapping), SCLP compresses the target KV cache strictly in place without coordinate shifts.

4. Measured Prototype vs. Projected Fused Kernel (Q4) We updated the captions of Figure 1c (Page 2) and Table 2 (Page 6), as well as Section 4.2 (Page 6) and Section 5 (Page 8), to clearly separate our measured Uncompiled Eager PyTorch Prototype results (where Python slicing accounts for ~38.2% interpreter overhead, 59.4 ms step latency) from the projected <2% overhead (37.9 ms) of isolated C++/Triton fused block-masked kernel microbenchmarks.

Reviewer_dH6bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SCLP, a KV-cache management method for speculative decoding. It masks low-importance cache entries while preserving token positions and protecting initial and recent tokens. The authors report improved speculative acceptance rates and approximately 48% lower active KV-cache footprint on long-context workloads.

Critical strengths
  • The paper focuses on a timely and practically relevant problem: reducing KV-cache overhead in speculative decoding.
  • The proposed in-place KV masking offers an interesting design perspective, preserving tensor layouts and token indices to potentially simplify cache management and draft–target coordination.
Critical weaknesses

The acceptance-rate definition and decoding settings are unclear, making the reported gains difficult to interpret.

  • SCLP is slower than both standard speculative decoding and direct autoregressive decoding in Table 1. Its practical acceleration benefits remain unproven.

  • The method appears to modify the target model’s KV cache, but generation quality is insufficiently evaluated. Higher acceptance rates do not imply preserved accuracy.

  • The evaluation only considers a separate draft model. Integrated drafting modules, such as DeepSeek-V3’s MTP (DeepSeek-AI, 2024), provide an alternative way to reduce drafting overhead by sharing representations and parameters with the target model. The paper should discuss whether its motivation and benefits extend to these architectures.

DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437, 2024.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer dH6b: Acceptance Formulation, Prototype vs. Fused Latency, RULER Fidelity, and DeepSeek-V3 MTP

We thank Reviewer dH6b for the constructive feedback. Below we address each weakness and reference the corresponding updates in the revised manuscript:

1. Acceptance-Rate Definition and Decoding Settings (CW1) In Section 4.1 (Page 6), we now explicitly define the speculative acceptance rate alpha = (1 / gamma) * E[N_accepted] as the mean fraction of draft tokens accepted by the target verification pass per step under standard speculative rejection sampling (Leviathan et al., 2023), using lookahead window gamma = 5, attention sink size S_sink = 64, local window W_local = 256, and middle-context target sparsity r = 0.50.

2. Prototype Latency in Table 1 vs. Fused Kernel Overhead (CW2) We updated the ShareGPT table (Table 2, Page 6, formerly Table 1) and Section 4.2 (Page 6) to explicitly label those end-to-end numbers as an "Uncompiled Eager PyTorch Prototype". On short 512-token conversational prompts (where sequences fall largely within the 64 + 256 = 320-token protected envelope and execution is compute-bound), eager Python slicing and dynamic mask generation incur substantial interpreter dispatch overhead (~38.2% of step latency, 59.4 ms). We clearly distinguish this eager prototype overhead from the <2% fused kernel overhead (37.9 ms) shown in Figure 1c (Page 2), which reflects isolated microbenchmarks of a fused block-masked attention kernel (FlashInfer C++ / Triton API) that eliminates Python runtime dispatch overhead in production engines.

3. Downstream Generation Quality Evaluation (CW3) We added Table 3 (Page 7) in Section 4.3 detailing our 500-sample RULER long-context (4k window) evaluation across A100 SXM and H100 SXM alongside the perplexity retention trade-off curve in Figure 4b (Page 8). By permanently protecting 100% of initial attention sinks (S_sink = 64), 100% of the recent local window (W_local = 256), and the top-(1 - r) heavy-hitter tokens aggregated across all verification queries Q = {q_1, ..., q_gamma}, SCLP keeps perplexity drift strictly within the lossless regime (Delta PPL < 0.02 up to 60% sparsity in Figure 4b) while cutting active KV memory by 48.21% (0.651 GB -> 0.334 GB) and improving verified token acceptance by +12.32% (A100 SXM: 49.01% -> 61.33%) and +13.00% (H100 SXM: 48.67% -> 61.67%) across 500 RULER prompts.

4. Compatibility with Integrated Drafting Modules like DeepSeek-V3 MTP (CW4) We added a dedicated analysis of DeepSeek-V3 Multi-Token Prediction (MTP) in Section 2.1 (Page 3) and Table 1 (Page 3). While MTP eliminates separate draft model weights by sharing target hidden states, verifying MTP-proposed tokens over long contexts remains bottlenecked by reading the full target KV cache from HBM. Because SCLP compresses the target KV cache in place while strictly preserving logical tensor shapes and sequence coordinates (B x H x N x D), it is directly compatible underneath MTP verification heads without modifying MTP positional embeddings.

Reviewer_KefdOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes SCLP, aiming to accelerate speculative decoding at long-context inference scenarios. Draft models are always bounded by long KV cache, which occupies GPU HBM and also causes attention bound for speculative decoding. Instead of removing irrelevant tokens/KV cache, SCLP marks some tokens as abolished and they do not participate in the attention calculation. Experiments on Qwen2-7B/1.5B shows speedup effect of the proposed method.

Critical strengths
  1. This paper aims at an important scenario: long-context speculative decoding. With context length growing, these methods are playing a more important role in real-world application.

  2. The presentation of this paper is clear and readable.

Critical weaknesses
  1. The amount of work does not satisfy an ICLR paper. The main paper only has 7.5 pages, and this work lacks both novelty and solid experiments. By the figures and tables, I highly suspect that this paper is generated by some auto-research agents and has not been checked into details.

Update on 11/16: This weakness is fully resolved. By checking the supplementary code, I believe that this work is sound in experiments and the results are reproducible. Although the main text has not reached the 9-paged limit, the additional results and discussion have expanded the workload of this paper. I believe with better formatting, the paper should look fine and suitable for ICLR acceptance.

  1. There are abundant related work in long-context speculative decoding, e.g., TriForce, should be taken as baselines. However, this paper omit them all, therefore I cannot judge whether the proposed method is effective or not.

Update on 11/16: The authors have added additional discussion and empirical results to show that the proposed method can outperform the additional baselines. The proposed method should be effective.

Questions for authors

See above. (All resolved after rebuttal)

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Kefd: Long-Context Speculative Baselines, Scope, and Execution Logs

We thank Reviewer Kefd for the feedback on long-context speculative decoding baselines and experimental documentation.

1. Expanded Scope & Baseline Comparisons (CW1 & CW2) We have expanded the manuscript to ~8.3 pages of main text, adding a comparative analysis in Section 2 (Page 3) and a structured summary table (Table 1, Page 3) comparing SCLP against TriForce (Sun et al., 2024), MagicDec (Chen et al., 2024a), SnapKV (Li et al., 2024), PyramidKV (Shi et al., 2024), RocketKV (Behnam et al., 2025), QuantSpec (Tiwari et al., 2025), and DeepSeek-V3 MTP (DeepSeek-AI, 2024), alongside empirical trade-off comparisons against PyramidKV and MagicDec in Figures 3 & 4 (Pages 7–8) and cross-hardware A100/H100 RULER verification telemetry in Table 3 (Page 7).

2. Core Differentiator vs. TriForce (Section 2.3 & Table 1) While hierarchical systems like TriForce retain a dense, uncompressed target KV cache and require dynamic cache streaming and index remapping across draft-target interactions, SCLP compresses the target model's active KV cache strictly in place (48.21% reduction, 0.651 GB -> 0.334 GB), eliminating index reconstruction and draft-target token desynchronization while preserving O(1) pointer rollback.

3. Raw Multi-GPU Execution Logs & Codebase To provide complete experimental transparency, we have uploaded our evaluation codebase (evaluate.py and sclp/) and the raw sample-by-sample ([Sample 1/500] through [Sample 500/500]) execution logs across NVIDIA A40, A100 SXM, and H100 SXM (tmux_sclp_eval.log, ruler_h100_sclp.log, ruler_h100_dense.log, ruler_h100_ar.log, and main_evaluation_results.csv) in the Supplementary Material .zip, and added the mandatory ICLR AI Disclosure section on Page 9.

Thanks for your timely response. I have updated my reviews and overall rating accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revision and Supplementary Execution Logs (All Reviewers & AC)

We thank the Area Chair and Reviewers (Xkbo, dH6b, and Kefd) for their thoughtful and constructive feedback. We have uploaded a revised manuscript (~8.3 pages of main text, 10 pages total) and an anonymized Supplementary Material archive (SCLP_ICLR2027_Supplementary.zip). Below is a summary of the key updates:

  1. Decoupling Operator-Level Masking from Physical Page Reclamation (Section 3.2, Page 5 & Section 4.4, Pages 7-8): We formally distinguish contiguous operator-level validity masking (which preserves exact B x H x N x D logical coordinates and cuts HBM-to-SRAM bandwidth and FLOPs by 48.21% without resizing the pre-allocated tensor envelope) from physical page reclamation in block-allocated engines (such as vLLM's PagedAttention, b_page = 16), and explicitly formulate the 37-to-71 stream expansion (1.92x) as an analytical upper bound (Eq. 4).
  2. Cross-Hardware Long-Context RULER Telemetry (Section 4.3, Table 3, Page 7): We report 500-sample RULER (4k window) evaluations across NVIDIA A100 SXM and H100 SXM, showing +12.32% (A100 SXM: 49.01% -> 61.33%) and +13.00% (H100 SXM: 48.67% -> 61.67%) verification acceptance gains at -48.21% active KV footprint (0.651 GB -> 0.334 GB), with 100% invariant retention of Attention Sinks (S_sink = 64) and Local Window (W_local = 256) and lossless perplexity (Delta PPL < 0.02 up to 60% sparsity in Figure 4b).
  3. Expanded SOTA Baseline & Architecture Comparisons (Section 2 & Table 1, Page 3): We added structured comparisons against TriForce (Sun et al., 2024), MagicDec (Chen et al., 2024a), SnapKV (Li et al., 2024), PyramidKV (Shi et al., 2024), RocketKV (Behnam et al., 2025), QuantSpec (Tiwari et al., 2025), and integrated Multi-Token Prediction modules (DeepSeek-V3 MTP, DeepSeek-AI, 2024).
  4. Transparent Separation of Eager PyTorch Prototype vs. Projected Fused Kernel Latency (Figure 1c, Table 2, Section 4.2) & Raw Multi-GPU Logs: We explicitly label our end-to-end timing table as an Uncompiled Eager PyTorch Prototype (~38.2% Python slicing overhead, 59.4 ms) and separate it from the projected <2% overhead (37.9 ms) of isolated C++/Triton fused block-masked kernel microbenchmarks. All raw sample-by-sample GPU logs across A40, A100 SXM, and H100 SXM and the evaluation codebase are included in the Supplementary Material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.