acceptodds
Under review as a conference paper at ICLR 2027

SCAR: Parallel Span Cross-Attention Repair for Context Deletion in LLMs

Abstract

Deleting a span from a KV-cached LLM context leaves the surviving suffix encoded with content that no longer exists. Re-prefilling the suffix repairs it exactly, but the cost grows with the suffix rather than with the deletion, and selective recomputation still pushes every recomputed token through all backbone layers in sequence. We observe that each layer's cache already holds the information for estimating that layer's repair: the stale suffix states and the deleted-span states. SCAR (Span Cross-Attention Repair) exploits this with one small corrector per layer that cross-attends from the suffix to the deleted span and predicts a residual update to the suffix keys and values. The correctors read only the cache, run no backbone layer, and depend on no other corrector, so all of them run in parallel as a depth-1 repair and leave a standard decoding cache. Across seven task categories on Qwen3-8B and Llama-3.1-8B, SCAR exceeds 15%-budget CacheBlend-style selective recomputation in answer agreement with exact recomputation (consistency) on 13 of 14 backbone-category cells. On four timed Qwen settings, it runs 2.9-4.8 faster than exact recomputation and 18-33% below that budget's modeled latency. On conflicting-fact retrieval, SCAR achieves near-exact consistency where selective recomputation falls short even at 75% on Qwen and 50% on Llama; on Qwen GSM graphs, variable tracking, and conversation, it exceeds training-pool-matched KVEraser by 24-48 percentage points. At fixed 32k context with 512 deleted tokens, SCAR beats exact recomputation at every tested suffix length of at least 4k, reaching a 4.2 speedup at 30k. Higher recomputation budgets still recover more fidelity on several settings, at two to three times SCAR's latency. SCAR thus demonstrates that deletion repair can be learned directly from KV cache space and executed in a single parallel stage, decoupling repair latency from backbone depth and making long surviving suffixes cheap to edit.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache, LLM inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_t61uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCAR repairs the KV cache of the surviving suffix after deleting a middle span. It trains one small corrector per layer that cross-attends from stale suffix states to deleted-span states and predicts residual updates to suffix K/V. All correctors run in parallel, avoiding backbone replay. Evaluated on Qwen3-8B and Llama-3.1-8B across seven task categories, SCAR beats 15%-budget selective recomputation on 13 of 14 backbone–category cells and runs 2.9–4.8× faster than exact recomputation on four timed Qwen settings.

Critical strengths
  1. Novel architecture: per-layer parallel correctors that decouple repair latency from backbone depth.
  2. Strong empirical results on suffix-carried-state tasks, especially compared to delete-and-shift and 15%-budget selective recomputation.
  3. Useful ablation: functional supervision outperforms direct KV MSE.
Critical weaknesses
  1. No analysis of how positional shift is handled. The corrector reads stale suffix states and deleted-span states and outputs K/V residuals. It is unclear whether it learns joint content+position correction or assumes pre-applied position correction. This is central to the method and needs clarification.
  2. No theoretical or empirical analysis of inter-layer error propagation. If layer ℓ’s repair is approximate, layer ℓ+1’s corrector still sees the original stale hidden state. Does error compound? A simple comparison of predicted residuals vs. true residuals per layer would help.
  3. KVEraser comparison is imperfectly controlled. Training SCAR and KVEraser on the same pool, or reporting on KVEraser’s fully covered subsets, would make the comparison fairer.
  4. Failure modes (natural counting, MuSiQue) are noted but not analyzed.
Questions for authors
  1. How does SCAR handle the positional shift after deletion? Is position correction applied before the corrector, or does the corrector learn it jointly?
  2. Can you provide an analysis of inter-layer error propagation? For example, compare SCAR’s predicted K/V residuals to the true residuals (C∗C^*−C0C^0) at each layer, and show whether the gap grows with depth.
  3. Can you train KVEraser on the same data pool as SCAR, or at least report SCAR results on the subset where KVEraser’s checkpoint has full coverage?
  4. What is the exact corrector architecture (parameter count, heads, weight sharing) and training loss? The main text says “cache MSE” but the abstract mentions “functional supervision”.
  5. Why do natural counting and MuSiQue remain difficult?
2 comments

Thank you for the detailed and constructive reviews.

Q1. "How does SCAR handle the positional shift after deletion? Is position correction applied before the corrector, or does the corrector learn it jointly?"

Position correction is applied separately from the learned corrector (stated in §3.2, "Separating position changes from learned repair" and Eq. 4). SCAR first removes the original RoPE rotation from the suffix keys (de-RoPE) and predicts the residual in pre-RoPE coordinates. It then re-rotates the corrected key to its edited position (re-RoPE).

Q2. "Can you provide an analysis of inter-layer error propagation? For example, compare SCAR's predicted K/V residuals to the true residuals at each layer, and show whether the gap grows with depth."

We extended App F with per-layer key and value L1, L2 and L∞ residuals, cosine similarity to the exact-recompute cache (exact cos), and correction-direction cosine (Δcos) for every method. Figures A1-A10 show that SCAR's error does not grow with depth: its key L2 stays around 0.2, and its key exact cos stays around 0.99 at every layer, consistent with our insight that KV cache can be repaired independently at each layer. SCAR also has relatively less residuals and better cosine similarity compared with other methods at middle and late layers.

Q3. "Can you train KVEraser on the same data pool as SCAR, or at least report SCAR results on the subset where KVEraser's checkpoint has full coverage?"

The "KVEraser matched" row in Tables 2–3 is already this. It is KVEraser's released recipe retrained on the same training pool as SCAR (stated in §4.1 and App B).

We also made the comparison strictly paired and updated the paper accordingly (§4, Table 2). The earlier 94–99% KVEraser coverage came from several requests where KVEraser(matched) outputs <EOS> immediately, which our scorer had skipped. They are now counted as empty answers. Its GSM / VT / macro scores change from 0.403 / 0.323 / 0.566 to 0.399 / 0.305 / 0.563, and SCAR's lead on GSM, VT and LiC is now 25–48 points (was 24-48). The difference is small and does not impact SCAR's relative performance.

Q4. "What is the exact corrector architecture (parameter count, heads, weight sharing) and training loss? The main text says 'cache MSE' but the abstract mentions 'functional supervision'."

These are specified in §3.2–3.3 and summarized here. Each corrected layer has:

  • one bias-free 128×128 query-proxy projection, shared across 8 KV heads;
  • per KV head, two independent two-layer GELU MLPs (one for K, one for V), each 512 --> 512 --> 128.

That gives about 5.3M parameters per layer: 184.4M on Qwen and 163.4M on Llama, about 2% of the backbone. Training loss uses per-layer suffix-cache MSE. The ablations (Table 4) vary the query proxy, the span readout and the MLP depth.

Q5. "Why do natural counting and MuSiQue remain difficult?"

Natural counting and MuSiQue are hard for every approximate method for mainly two reasons. First, the required repair is spread across the suffix rather than concentrated on a few tokens, which limits selective recomputation methods: even exact states for half of the suffix (ISR r=.5) reach only 0.447 on Natural (Qwen). Second, both tasks require a larger repair than the tasks that dominate SCAR's training data (tally, NIAH, GSM, and VT). We acknowledge this limitation and variants of SCAR that target such tasks are an important focus of future work.

Thank you for your explanation. I increased my score to 3. Thanks.

Reviewer_6wgTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes how to repair a KV cache after deleting a contiguous span from an already processed context. SCAR introduces independent per-layer correctors that cross-attend from stale suffix states to the deleted span and predict residual updates to suffix keys and values. The correctors operate directly on cached states without executing the backbone and can run in parallel across layers. The evaluation covers seven task categories on Qwen3-8B and Llama-3.1-8B.

Critical strengths

The approach is relevant to retracting stale retrieved documents, incorrect tool observations, or invalidated conversational instructions. The central design is to estimate each layer’s suffix correction directly from that layer’s cached states, which removes the cross-layer dependency of backbone recomputation. The cross-attention provides an explicit mechanism for conditioning the repair on the content being removed. On the Qwen settings, the deletion-to-first-token latency is 2.9-4.8x lower than exact recomputation. The two-backbone evaluation, retrained component ablations, and suffix-length scaling measurements provide useful support for the proposed design.

Critical weaknesses

Although the evaluation covers seven task categories, NIAH accounts for approximately 71% of the improvement over the 15%-budget selective recomputation baselines. Excluding NIAH, the average gains fall to 7% points over CacheBlend-Del and 6% over ISR. SCAR also achieves only 39% answer consistency on HotpotQA counting. These results leave a substantial gap from exact recomputation on these tasks.

Questions for authors
  • I would like to see a simple NIAH experiment: start with a context in which the model correctly retrieves the needle, use SCAR to delete the span containing that needle, and ask the same question again. The needle should appear only once in the context, so no answer-bearing information remains after deletion. The model should then be unable to retrieve the original needle. Comparing how often SCAR and exact recomputation still return the deleted needle would directly test whether its information remains accessible through the repaired cache.

  • Can the authors extend the cache-space analysis to report per-layer key/value relative L2 errors and cosine similarities to the exact-recompute cache, alongside the correction-direction cosine used in Table A12? How are these measurements related to answer agreement, especially in settings where SCAR improves consistency while increasing the aggregate cache residual?

2 comments

Thank you for the careful and constructive reviews.

Q1. "I would like to see a simple NIAH experiment: start with a context in which the model correctly retrieves the needle, use SCAR to delete the span containing that needle, and ask the same question again. ..."

Thank you for the helpful suggestion. We ran this experiment and added it to the paper as Appendix G (Table A13).

Setup. We built 140 requests per length, each with a single needle. Each request has two variants:

  • the test variant deletes the needle;
  • the control variant deletes a same-length filler instead, so the needle survives.

The control checks that the model retrieves the needle before deletion. The prompt the same as in §4.1. SCAR and KVEraser are the same checkpoints evaluated in the paper and neither was retrained for this task.

Number of times a deleted needle is returned:

methodQwen 16kQwen 32kLlama 16kLlama 32k
exact recompute0000
delete-and-shift0000
ISR / CB-Del (r=.15, .5)0000
KVEraser (native)5658——
KVEraser (matched)3628——
SCAR0000

SCAR never returns the deleted needle, matching exact recomputation. KVEraser returns it in 20–41% of records. This task is easier than the other evaluated tasks in the paper as the suffix does not carry enough information to recover the needle, so even delete-and-shift never returns it. On the control, every method retrieves the needle, showing that repair does no harm when the needle is kept.

Q2. "Can the authors extend the cache-space analysis to report per-layer key/value relative L2 errors and cosine similarities to the exact-recompute cache, alongside the correction-direction cosine used in Table A12? How are these measurements related to answer agreement, especially in settings where SCAR improves consistency while increasing the aggregate cache residual?"

We extended App F with per-layer key and value L1, L2 and L∞ residuals, cosine similarity to the exact-recompute cache (exact cos), and correction-direction cosine (Δcos) for every method (Table A12, Figures A1–A10).

  • Aggregate (Table A12) SCAR has the lowest key L2 (0.142 vs. 0.301 for delete-and-shift) and L∞, and the highest exact cos (0.997 vs. 0.984) and Δcos (0.766 vs. at most 0.480 for the selective policies).
  • Per layer (Fig A1-A10) SCAR's error does not increase with depth: its key L2 stays around 0.2, and its key exact cos stays around 0.99 at every layer, consistent with our insight that KV cache can be repaired independently at each layer. SCAR's advantage over ISR r=.5 is more in the middle and late layers, while ISR is better in most of the first ten.
  • Relation to answer agreement. Cache-space metrics generally correlate with answer agreement, but not definitively, because they average over all suffix tokens, which have unequal impact on the answer. For example, ISR r=.5 has lower L1 than SCAR (0.071 vs. 0.088 for keys), yet its median KL@16 is about 8× SCAR's. ISR restores half of the tokens exactly and leaves the rest stale, while SCAR corrects every token. The aggregate residual can therefore stay flat or even rise while the answer-relevant error falls.
Reviewer_EcPyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of span deletion in a KV-cached LLM. It proposes SCAR, a set of parallel collectors each of which reads the cache of one layer of the backbone, cross-attends from the suffix of the deleted span to the span, and predicts a residual update to the suffix keys and values. This approach works because the authors observe that each backbone layer's cache contains both the stale suffix states and the deleted span states for estimating that layer's repair. The correctors can run in parallel because they read only the cache not the backbone layer and do not depend on other correctors. The paper has compared with both exact re-computation and selective re-computation approaches and demonstrated that SCAR achieved good repair fidelity at a low latency.

Critical strengths

S1. The proposed approach is intuitive and effective. Each collector utilizes the cache at one layer to predict a residue update to the suffix of the deleted span in the layer. All collectors run in parallel in their layers of cache and do not access the backbone.

S2. The experiments compare with representative approaches ranging from exact re-computation (correct but expensive), selective re-computation, to simple but ineffective delete-and-shift, and report both consistency and repair latency performance.

Critical weaknesses

W1. The performance results on consistency could use more analysis, as some tasks seem hard to all methods, and on some tasks the performance varies greatly between approaches.

Questions for authors

Q1. Table 2 and 3. All methods performed poorly on Natural, and KVEraser matched and ISR r=.5 slightly higher. Why? Also, why did SCAR outprform ISR methods on some tasks, given that ISRs are oracle assisted (supposedly more effective)?

2 comments

Thank you for the careful and constructive reviews.

W1 / Q1 (Natural). "The performance results on consistency could use more analysis, as some tasks seem hard to all methods, and on some tasks the performance varies greatly between approaches." / "All methods performed poorly on Natural, and KVEraser matched and ISR r=.5 slightly higher. Why?"

Natural and MuSiQue are hard for every approximate method for mainly two reasons. First, the required repair is spread across the suffix rather than concentrated on a few tokens, which limits selective recomputation methods: even exact states for half of the suffix (ISR r=.5) reach only 0.447 on Natural (Qwen). Second, both tasks require a larger repair than the tasks that dominate SCAR's training data ( tally, NIAH, GSM, and VT). We acknowledge this limitation and variants of SCAR that target such tasks are an important focus of future work.

KVEraser also has about 40x the trainable parameters than SCAR.

Q1 (ISR). "Why did SCAR outperform ISR methods on some tasks, given that ISRs are oracle assisted (supposedly more effective)?"

ISR is oracle-assisted only on the tokens it recomputes. At r=.15 it replaces 15% of suffix tokens with exact states and keeps the other 85% stale at every layer. SCAR instead applies correction to every suffix token, making it able to repair KV cache that would be left unchanged by ISR.

thanks for your explanation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.