acceptodds
Withdrawn from ICLR 2027

KVFixer: Predictive KV Cache Repair for Context Revision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Caching makes a model's past computation reusable, but not automatically revisable. Deleting an upstream span removes its key–value (KV) entries while leaving its influence in retained states. We introduce KVFixer, which predicts low-rank corrections from the original cache without replaying the retained suffix. It learns context-conditioned bases from paired executions and predicts edit-specific coordinates, refined with cache and output-distribution supervision. A fixed-endpoint refinement comparison lowers text and image KL, while paired projection diagnostics show concentrated rather than uniform gains across requests. On internal text and image panels, mean-token KL is 66.5% and 94.5% below reported deletion-only endpoints. External answer-choice KL is lower than the evaluated prefix controls and KVEraser adaptation. Replacing selected retained states with exact states further improves diagnostic fidelity, distinguishing compact representation, predicted correction and the output behavior of the revised cache.

Primary area:
generative models
Keywords:
Large language model, attention, KV cache, auto-regressive models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K1NgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces KVFixer, a new approach for approximating KV cache deletion and repair. It removes the KV cache for the erased interval, aligns retained rows to the revised positions, and predicts corrections to the suffix states, which are differences between target suffix KV and contaminated old suffix KV.

Critical strengths

S1. The high-level idea is technically interesting and well motivated.

S2. Empirical studies extend prior text-setting evaluations to image settings.

Critical weaknesses

W1. The paper presentation is very poor and hard to follow. Terms and concepts are used without definitions or references, details are missing, and the flow is messy. E.g.,

  • Section 3.2 mentions "expert" without introducing or defining it.
  • Table 1 presents experiment results without introducing baselines like Delete-and-shift, Exact-row prefix control, etc in the paper.
  • In deriving oBo_B, what projection did you use? How did you implement ordered embedding bins? How did you construct length features?
  • How was the training and evaluation data constructed / adapted? How were erased intervals derived?

It's hard to properly assess the manuscript and trust the results of the paper before addressing the presentation issue and make the paper readable to the audience.

Questions for authors

See Weaknesses.

Reviewer_nEnmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KVFixer, a predictive framework for repairing KV caches after context deletion without recomputing the retained suffix. It learns low-rank correction bases and uses context-dependent predictors to estimate changes in retained KV states, refined through cache-reconstruction and KL objectives. Experiments primarily on Qwen3-VL-8B-Instruct demonstrate reduced output-distribution divergence relative to deletion-only baselines, with additional analyses of correction geometry and external task performance.

Critical strengths
  1. The paper addresses an important limitation of KV cache editing: removing a context span does not eliminate its influence on subsequent cached representations. Predicting the necessary corrections directly, rather than recomputing the suffix, is an interesting and potentially useful direction.

  2. The paper distinguishes correction representation, coefficient prediction, and functional fidelity. Comparisons with exact low-rank projections and external evaluations provide useful insights into when cache reconstruction improves downstream behavior. The discussion of limitations also strengthens the empirical analysis.

Critical weaknesses
  1. The efficiency claims rely on analytical FLOPs rather than measured latency or throughput. These estimates exclude memory traffic and runtime overhead, while repair parameters occupy approximately 599 MiB plus a 2.318 GiB CPU embedding copy. Practical deployment benefits remain unverified.

  2. Unequal refinement budgets confound the component ablations, and complete functional repair is evaluated on only one backbone. Most fidelity tests also follow teacher trajectories. On MM-NIAH, lower KL accompanies an accuracy drop from 60.67% to 46.00%, leaving downstream utility unclear despite closer agreement with exact PB.

Questions for authors

Q1. Can the authors provide measured end-to-end repair latency, memory overhead, and throughput against exact suffix recomputation, including different suffix lengths? How closely do these measurements match the theoretical FLOP estimates?

Q2. Could the authors provide matched-training-budget ablations for context-conditioned versus shared bases and cache-only versus joint cache/KL refinement? A simpler predictor baseline would also help justify the architectural complexity.

Q3. How should downstream accuracy be interpreted when exact revised-context execution performs worse than the original execution? Are the original labels still valid after deletion, and can the authors evaluate independently generated continuations or revised-context task outcomes?

Reviewer_VPc8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose KVFixer to predict a low-rank correction to the retained suffix's KV states without rerunning the model after a span is deleted from prefilled context. KVFixer learns correction from paired runs, picks a basis and expert per request and regresses the correction from dirty cache with no suffix replay. They report large KL reduction relative to naive deletion on Qwen3-VL-8B.

Critical strengths
  1. Interesting problem, how to make cached computation revisable and not just reusable.
  2. The representation vs coordinate error decomposition is neat
  3. The authors are candid that exact projection beats their predictor
Critical weaknesses
  1. Writing could be better. Terminologies appear before definition (PB teacher, E1). Table 1 is difficult to read
  2. No wall-clock timing and no comparison against partially recomputing the suffix with matched cost
  3. Only 126 of 1,200 text sources beat exact projection and the median is worse
Questions for authors
  1. How does CacheBlend style recompute of the suffix compare on KL at matched FLOPs?
  2. How often does the repaired cache produce that fact after deleting a fact-bearing span?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.