acceptodds
Under review as a conference paper at ICLR 2027

Stale After Reuse: An Application Correctness Boundary and No-Regeneration Repair

Abstract

Long-context large language model (LLM) applications repeatedly revise shared context, while serving systems reuse KV states to avoid redundant computation. Prior work shows that edited information can persist in downstream KV states; we ask when this latent persistence becomes an application-record error under an executed reuse plan, how far recomputation must extend to remove it, and whether an output-visible mismatch can be corrected without another generation. In execution-proven interventions on 192 paired cases, one-gap reuse lowers exact-record accuracy from 54.17% to 45.31%. A complete chunk-aligned frontier finds every partial refresh policy below Dense marginal accuracy; complete stale-suffix invalidation restores the Dense value of 54.17%, although per-case recovery is heterogeneous. This characterization motivates correction at the output boundary, where the superseded value becomes observable. We therefore derive RevisionPatch-Exact, a deliberately conservative, no-regeneration rule that patches one unique whole-field match and otherwise abstains. Across dialogue-state and tool-argument revisions with Qwen, Mistral, and Llama models, the unchanged rule yields positive net fixes; on 384 natural MultiWOZ revisions it raises exact accuracy from 54 to 63 and changed-field accuracy from 231 to 269, with no observed exact break. The repair adds neither a model call nor prefill and, at 16k context, retains 5.85× faster median time-to-first-token and 1.61× higher throughput than Dense. These results connect stale downstream state to an execution-proven application correctness failure and expose a simple output-side operating point.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache reuse, long-context LLM serving, cache consistency, structured outputs, selective recomputation, output repair

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3FfEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the selective reuse of KV caches—following internal modifications to a prompt—can lead to structured outputs that retain outdated information. The authors focus specifically on "one-gap reuse": reusing the unchanged prefix, recomputing only the chunk containing the edit, and then restoring the cached KV states for the suffix. Although the suffix tokens remain unchanged, their hidden states may still depend on the original context; consequently, while the generated output might conform to the required schema, it may fail to accurately reflect the modifications made to the prompt. The authors compare several approaches: "Dense" (full recomputation), "Reuse" (selective reuse), partial suffix recomputation (with varying lengths), and "Full-tail" (recomputing everything following the edit). On two controlled datasets, each containing 192 requests, the "Reuse" strategy shows a decline in whole-record accuracy by 8.85 and 6.77 percentage points, respectively, compared to the "Dense" baseline. The benefits of partial recomputation are non-monotonic, whereas the "Full-tail" approach restores the same per-request correctness outcomes as the "Dense" method. In an analysis of 384 naturalistic modification cases from the MultiWOZ dataset, while the difference in whole-record accuracy is not statistically significant, a post-hoc analysis of specific modified fields reveals a 6.51 percentage point drop when using the "Reuse" strategy. The paper also examines a deterministic output-repair rule called "RevisionPatch-Exact," which replaces an old value with a new one if the old value uniquely matches a complete field and the new value does not yet appear in the output. While this rule increases the number of correct whole records from 54 to 63 on a primary naturalistic dataset, it also leads to 7 instances—out of 51 operations—where unrelated fields are modified, thereby introducing new errors. The authors argue that "zero-disruption" at the whole-record level is insufficient to guarantee the integrity of individual fields. They further explore this issue through techniques such as replaying trusted field keys, controlling for "distractor" fields, and comparing various alternative approaches. In my view, this paper presents both a controlled empirical study and an investigation into evaluation methodology. Its primary contribution lies in characterizing how caching strategies alter model outputs and in highlighting localized errors that remain hidden when relying solely on record-level metrics.

Critical strengths
  1. The use of paired interventions and full-recomputation baselines makes the impact of caching strategies highly verifiable. Rather than merely comparing average accuracy scores, the paper tracks how the outcomes for specific requests shift when different strategies are applied. For the Qwen model, the transition from "Dense" to "Reuse" yields a count of 78/26/9/79—revealing 26 instances of regression alongside 9 improvements—while the replication on Llama also shows a net regression. In two controlled experiments, the "Full-tail" approach restores the correctness status observed under the "Dense" baseline, providing further evidence that preserving the state of the suffix—which is affected by edits—is linked to changes in the final output. The execution checks detailed in Appendix B also help rule out implementation-level ambiguities, such as failures to perform the intended scope of recomputation or the occurrence of implicit fallbacks to other strategies. While this does not constitute a formal proof of correctness, it is far more compelling than simply assuming that a caching strategy was executed as intended based solely on its configuration label. This methodological rigor establishes the paper’s credibility as a controlled empirical study, independent of whether the proposed repair rules achieve the highest possible accuracy.
  2. Field-level auditing exposes blind spots in record-level metrics—and the authors do not shy away from reporting unfavorable outcomes. In the primary repair experiments, there are 9 instances of full-record correction and zero instances of full-record corruption. Taken in isolation, these figures might suggest that the repair rule is both conservative and safe; however, a comprehensive audit of all erroneous fields reveals that 7 operations introduced new, unrelated errors into records that were already incorrect. This observation carries a clear implication: the fact that a record’s overall correctness status remains unchanged does not mean the output has not degraded. Another scenario observed in the SGD (Stochastic Gradient Descent) context involves the modification of an already-erroneous field; since the field was already incorrect, the total count of erroneous fields remains unchanged. Together, these findings demonstrate that the correctness of the target object, the introduction of new errors, and the final correctness of the record are distinct issues. Techniques such as "trusted field-key replay" and "targeted interference control" further decouple the signal used to update an existing value from the logic used to identify which specific field requires modification. Furthermore, the authors transparently report post-hoc analyses, unsuccessful pilot gates, poor performance under high concurrency, and the existence of more robust alternative approaches. This mode of reporting ensures that even negative results retain significant research value.
Critical weaknesses

W1. The scientific explanation provided is insufficient, given that the mechanisms discussed are already acknowledged in existing literature.

Sections 1 and 6 note that prior work has already addressed how edited information persists within downstream KV states, as well as how these states can be updated via suffix recomputation or the application of "errata." Consequently, the primary contribution of this paper should be to uncover new, previously unknown patterns specific to structured output scenarios—rather than merely reaffirming that stale context states can influence model generation.

The most distinct finding presented is the non-monotonicity of partial recomputation. However, the paper focuses primarily on reporting success counts and per-request transitions, failing to adequately explain how this phenomenon relates to suffix content, the number of edits, field dependencies, or the specific types of errors involved. While readers can discern that certain recomputation scopes lead to poorer performance, it remains unclear under what conditions this occurs, why it happens, and how these observations might inform the selection of caching strategies.

The audit of field-level degradation serves as a valuable piece of evidence; however, relying on simple string matching is not equivalent to accurate field identification—a point that could, in fact, be deduced directly from the definitions of the rules themselves. While the experiments on interference control confirm this failure mode, they fail to establish broader patterns regarding how such failures manifest across different task structures.

W2. The semantic construction of controlled inputs and the transformation process applied to natural data remain insufficiently defined, limiting the interpretability of the results.

Although Section 4.1 and Appendix C outline a 128-token layout, four categories of suffixes, four tiers for the number of edits, and 12 test cases per category, the paper fails to provide specific details regarding the content of these suffixes, the exact number of edits applied, representative full-length prompts, or the specific semantic factors altered between different cases.

These details are not merely incidental engineering notes; they are critical. Whether a suffix repeats stale information, contains specific answer formatting, or includes relevant fields fundamentally shapes how one interprets the impact of stale cache states and the observed non-monotonicity in recomputation. Providing only the cache span and the number of clusters is insufficient for readers to determine exactly what is being measured by these stress tests.

Furthermore, the "natural" data has also been subjected to filtering based on "literal-grounded eligibility" and compatibility with a fixed schema/prompt structure. Appendix A shows that, after excluding existing data, only 528 out of 1,941 eligible conversations were compatible with the freezing protocol, with 384 ultimately being executed. While the authors correctly note that this does not represent the proportion of production traffic, they have not sufficiently detailed how raw conversations were converted into 160-token inputs, nor have they clarified the differences between the types of modifications that were excluded versus those that were retained.

W3. Discrepancies in "strict matching" for naturalistic tasks have not been adequately decomposed into instances of residual "old" values ​​versus failures in semantic revision.

In the primary naturalistic experiment, the pre-specified "entire-record" endpoint yielded only 58 correct outcomes out of 54 pairs—a difference that is not statistically significant. Stronger evidence from the naturalistic setting emerges from the analysis of specific modified fields: 256 correct for "Dense" versus 231 for "Reuse," representing a net decrease of 6.51 percentage points. The authors appropriately label this as a post-hoc analysis and correctly acknowledge that applying corrections for multiple testing does not transform it into a confirmatory analysis.

However, a modified field being "not equal to the gold standard" is not synonymous with "retaining the old, replaced value." The scoring method described in Appendix A.3 does not account for case sensitivity, internal whitespace, Unicode variations, or ontological equivalence. Table 13 illustrates this with the example of "Sunday" versus "sunday," and Appendix E.2 notes six instances where target field updates failed to improve the "strict" score simply because their surface form differed from the gold standard.

Consequently, the observed differences between the two approaches may conflate several distinct issues: residual old values, other incorrect values, or mere surface-form inconsistencies. The paper fails to provide a granular breakdown of the critical cases where "Dense" is correct but "Reuse" is incorrect. Furthermore, the "fix" analysis does not fully resolve this ambiguity: of the nine complete fixes, only four fall into the "Dense-correct/Reuse-wrong/Patch-correct" category—and "Dense" itself could also benefit from these same correction rules.

W4. The comprehensive safety audit was not applied consistently across the primary comparisons, precluding a direct, comparable assessment of coverage versus safety.

One of the paper’s most significant findings is that both "entire-record" fix/break metrics and accuracy metrics focused on specific modified fields may fail to capture instances where the model causes harm. However, most of the cross-model results in Table 3, the ablation study on relaxed rules in Table 16, and the "retained-record" and "erratum" alternatives still rely primarily on these incomplete metrics.

For instance, the "prior-record direct update" and "append-only erratum" approaches achieve 89 and 95 correct records, respectively—outperforming the 63 correct records of the "Whole-field Patch" method—yet each results in six "broken" (corrupted) records. Based solely on these counts, it is impossible to determine how many additional localized field-level errors were introduced, nor can they be compared on an equivalent basis against the seven instances of field-level damage caused by the "Whole-field Patch."

The authors acknowledge that Table 16 does not establish "field safety," nor do they claim that each individual "guard" is strictly necessary. While this disclosure is appropriate, the underlying empirical question remains unanswered: which rules actually improve the trade-off between "whole-field damage" and "repair coverage"?

Although "trusted field-key replay" offers diagnostic value, it relies on reliable field bindings that exist outside the original method's inputs; therefore, it cannot be considered a complete deployment solution that resolves the challenge of field identification.

W5. While the verification of cache interventions is thorough, issues regarding execution variance and the specific software versions used remain unresolved.

The eight categories of log checks in Appendix B help confirm that cache interventions were executed as intended. However, they also reveal a discrepancy: the source hash of a recovered "natural-context engine" does not match the hash recorded in historical logs. While this does not definitively prove that the original results are invalid—or that different code versions were used across the three sets of experiments—it does mean that the currently recovered materials are insufficient to fully verify the specific implementation versions used during those runs.

Furthermore, although the "Dense" and "Full-tail" approaches theoretically utilize the same revision context, they do not yield identical results across all experiments. In the controlled experiments involving Qwen, there is one discrepancy in "parsed records" and two in the raw strings; similarly, in the "natural" (real-world) experiments, there are differences in the counts of correct whole records and modified fields. The paper reports these observations but does not quantify the variance in results when the same strategy is executed repeatedly.

This does not negate the clear net degradation observed in the two controlled models. However, regarding the minor discrepancies in whole-record counts within the natural-data experiments—as well as the variations in one or two cases within the re-computation grid—the issues of reproducibility and potential differences in execution environments complicate the interpretation of these results.

Questions for authors
  1. How is non-monotonicity distributed across the four suffix categories and varying numbers of edits? Can specific conditions that lead to improvements, regressions, or oscillations be identified from existing cases, and how do these patterns extend upon the findings already presented in §6?

  2. Please provide a representative set of prompts from the controlled test suite, the definitions for each suffix category, the range of edit counts used, and the specific rules for converting natural conversations into 160-token inputs. In terms of the types of modifications, how do the 528 schema-compatible cases differ from those that were excluded?

  3. Among the 30 cases in the natural experiment where the "Dense" approach succeeded but "Reuse" failed, how many instances resulted in the output of an old (stale) value, and how many involved other types of errors or superficial formatting differences? Do the conclusions drawn from these paired comparisons still hold after applying a pre-determined, type-aware semantic audit?

  4. Could you conduct a comprehensive audit—covering the full set of pre- and post-edit errors—for the primary rules outlined in Table 16, as well as for the two key alternatives: "prior-record" and "erratum"? Does this analysis alter the paper’s findings regarding the trade-off between coverage and risk?

  5. What specific functionalities and experiments are involved when the recovered code is inconsistent with the historical hash? How significant are the variations in records when the same strategy is executed repeatedly, and are these variations substantial enough to influence minor natural discrepancies or localized non-monotonic phenomena?

  6. How are the "old/new" values, schema information, and trusted field keys extracted from the actual runtime environment? Once these processes—including cache construction (or amortization), the quality of repeated executions, and potential field corruption—are accounted for, under what specific conditions does the system still yield a net benefit?

Reviewer_F9kgOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies revision consistency in structured outputs under one-gap KV-cache reuse. It compares reuse with progressively longer suffix recomputation, full-tail recomputation, and dense recomputation. It also evaluates RevisionPatch-Exact, a deterministic output rule that replaces a unique whole-field match to an old value with the corresponding new value from an explicit edit. The controlled experiments demonstrate accuracy degradation under the tested reuse policy, while the field-level audit shows that output repair can introduce new errors that exact-record fix/break counts fail to capture.

Critical strengths

The paper studies a relevant reliability problem: a schema-valid output can still contradict an updated request. Studying this distinction is useful for applications that maintain structured records while reusing computation.

Critical weaknesses
  1. The overall technical contribution remains limited. The repair method performs guarded field replacement given an explicit old-to-new edit. The more difficult problem is determining which semantic field an edit concerns, how its value should be represented, and whether other fields must also change. The original rule addresses field identification only through literal matching, while the key-aware variant assumes a trusted field binding.

  2. Mathematically equivalent baselines produce unexplained differences. Dense and Full-tail should be equivalent when the unchanged prefix is reused correctly, yet they obtain 58 and 59 correct records under temperature-zero decoding. Repeated-run agreement and an examination of divergent outputs are needed before interpreting similarly small differences between methods.

  3. The low Dense accuracy lacks an error breakdown. Dense produces 383/384 parseable outputs but only 58 fully correct records. Table 13 illustrates that an otherwise correctly revised record fails because Sunday differs from sunday. The paper should quantify how many failures are surface-form mismatches versus genuinely incorrect values, and report a supplementary normalized score.

Questions for authors

Could the authors develop a more general repair method and test it on less restricted natural revisions? The current rule relies on literal value matching, is prone to wrong-field updates, and appears closely tied to the dataset construction.

Reviewer_wRBfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on revision consistency in structured outputs under KV-cache reuse, where a serving runtime recomputes an edited prompt but restores the cached suffix. Comprehensive analysis was conducted, e.g., that one-gap reuse drops exact-record accuracy by 8.85% and 6.77% for Qwen and Llama, partial suffix recomputation behaves non-monotonically, and only full-tail recomputation recovers full-prompt accuracy.

Critical strengths
  1. The topic studies by this paper is pretty unique yet important to me.

  2. The paper audits all 51 actions of RevisionPatch-Exact at the individual field level, but it is still honestly saying that 9 fixes does not hide 7 silent corruptions of unrelated fields.

  3. Experiments on Qwen3-8B and Llama-3.1-8B show that partially recomputing a contiguous chunk after an edit does not monotonically restore accuracy. Showing that stale value retention persists across the suffix until the entire post-edit tail is recomputed is a crisp, practical takeaway for anyone designing KV-reuse systems.

Critical weaknesses
  1. Without knowing which schema field was edited, RevisionPatch-Exact relies on string matching alone and corrupts an unrelated field in 7 of its 51 edits. To fix this, Key-aware Patch assumes the runtime already knows the exact target field key. However, if a system already knows both the target field and the new value, running LLM generation over a stale KV cache just to patch the output makes little sense.

  2. On the 384 realistic MultiWOZ conversations, the model is barely able to do the task even without KV caching. Full recomputation (Dense) gets only 58/384 records completely right, and one-gap KV reuse (Reuse) gets 54/384 right. A 4-record difference is statistically indistinguishable from noise, so the paper's main test fails to prove that KV reuse hurts full-record accuracy on natural data. Likewise, on the HammerBench benchmark, nearly half of all Reuse outputs fail to produce valid JSON at all (only 203/384 and 206/384 parse).

  3. In the long-context latency test, the system does not actually compare prompts to find the edited words at runtime; instead, the benchmark script feeds pre-extracted old and new strings directly from a file. Even with that shortcut, when processing 4 or 8 requests at the same time, Reuse actually has lower throughput (0.82x at 8k/C4 and 0.918x at 16k/C8) than recomputing the entire prompt from scratch?

Questions for authors
  1. Why is Dense exact-record accuracy on MultiWOZ so low (15.10%), whereas changed-field accuracy is 66.67%?

  2. In Figure 3 and Table 19, why does Reuse throughput fall below Dense at concurrency 4 (8k/C4 = 0.82x) and concurrency 8 (16k/C8 = 0.918x) when Reuse only recomputes 16 prefill tokens? Is the bottleneck coming from non-contiguous KV materialization and connector overhead in LMCache on the Teco SDAA runtime?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.