- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the selective reuse of KV caches—following internal modifications to a prompt—can lead to structured outputs that retain outdated information. The authors focus specifically on "one-gap reuse": reusing the unchanged prefix, recomputing only the chunk containing the edit, and then restoring the cached KV states for the suffix. Although the suffix tokens remain unchanged, their hidden states may still depend on the original context; consequently, while the generated output might conform to the required schema, it may fail to accurately reflect the modifications made to the prompt. The authors compare several approaches: "Dense" (full recomputation), "Reuse" (selective reuse), partial suffix recomputation (with varying lengths), and "Full-tail" (recomputing everything following the edit). On two controlled datasets, each containing 192 requests, the "Reuse" strategy shows a decline in whole-record accuracy by 8.85 and 6.77 percentage points, respectively, compared to the "Dense" baseline. The benefits of partial recomputation are non-monotonic, whereas the "Full-tail" approach restores the same per-request correctness outcomes as the "Dense" method. In an analysis of 384 naturalistic modification cases from the MultiWOZ dataset, while the difference in whole-record accuracy is not statistically significant, a post-hoc analysis of specific modified fields reveals a 6.51 percentage point drop when using the "Reuse" strategy. The paper also examines a deterministic output-repair rule called "RevisionPatch-Exact," which replaces an old value with a new one if the old value uniquely matches a complete field and the new value does not yet appear in the output. While this rule increases the number of correct whole records from 54 to 63 on a primary naturalistic dataset, it also leads to 7 instances—out of 51 operations—where unrelated fields are modified, thereby introducing new errors. The authors argue that "zero-disruption" at the whole-record level is insufficient to guarantee the integrity of individual fields. They further explore this issue through techniques such as replaying trusted field keys, controlling for "distractor" fields, and comparing various alternative approaches. In my view, this paper presents both a controlled empirical study and an investigation into evaluation methodology. Its primary contribution lies in characterizing how caching strategies alter model outputs and in highlighting localized errors that remain hidden when relying solely on record-level metrics.
Critical strengths
- The use of paired interventions and full-recomputation baselines makes the impact of caching strategies highly verifiable. Rather than merely comparing average accuracy scores, the paper tracks how the outcomes for specific requests shift when different strategies are applied. For the Qwen model, the transition from "Dense" to "Reuse" yields a count of 78/26/9/79—revealing 26 instances of regression alongside 9 improvements—while the replication on Llama also shows a net regression. In two controlled experiments, the "Full-tail" approach restores the correctness status observed under the "Dense" baseline, providing further evidence that preserving the state of the suffix—which is affected by edits—is linked to changes in the final output. The execution checks detailed in Appendix B also help rule out implementation-level ambiguities, such as failures to perform the intended scope of recomputation or the occurrence of implicit fallbacks to other strategies. While this does not constitute a formal proof of correctness, it is far more compelling than simply assuming that a caching strategy was executed as intended based solely on its configuration label. This methodological rigor establishes the paper’s credibility as a controlled empirical study, independent of whether the proposed repair rules achieve the highest possible accuracy.
- Field-level auditing exposes blind spots in record-level metrics—and the authors do not shy away from reporting unfavorable outcomes. In the primary repair experiments, there are 9 instances of full-record correction and zero instances of full-record corruption. Taken in isolation, these figures might suggest that the repair rule is both conservative and safe; however, a comprehensive audit of all erroneous fields reveals that 7 operations introduced new, unrelated errors into records that were already incorrect. This observation carries a clear implication: the fact that a record’s overall correctness status remains unchanged does not mean the output has not degraded. Another scenario observed in the SGD (Stochastic Gradient Descent) context involves the modification of an already-erroneous field; since the field was already incorrect, the total count of erroneous fields remains unchanged. Together, these findings demonstrate that the correctness of the target object, the introduction of new errors, and the final correctness of the record are distinct issues. Techniques such as "trusted field-key replay" and "targeted interference control" further decouple the signal used to update an existing value from the logic used to identify which specific field requires modification. Furthermore, the authors transparently report post-hoc analyses, unsuccessful pilot gates, poor performance under high concurrency, and the existence of more robust alternative approaches. This mode of reporting ensures that even negative results retain significant research value.
Critical weaknesses
W1. The scientific explanation provided is insufficient, given that the mechanisms discussed are already acknowledged in existing literature.
Sections 1 and 6 note that prior work has already addressed how edited information persists within downstream KV states, as well as how these states can be updated via suffix recomputation or the application of "errata." Consequently, the primary contribution of this paper should be to uncover new, previously unknown patterns specific to structured output scenarios—rather than merely reaffirming that stale context states can influence model generation.
The most distinct finding presented is the non-monotonicity of partial recomputation. However, the paper focuses primarily on reporting success counts and per-request transitions, failing to adequately explain how this phenomenon relates to suffix content, the number of edits, field dependencies, or the specific types of errors involved. While readers can discern that certain recomputation scopes lead to poorer performance, it remains unclear under what conditions this occurs, why it happens, and how these observations might inform the selection of caching strategies.
The audit of field-level degradation serves as a valuable piece of evidence; however, relying on simple string matching is not equivalent to accurate field identification—a point that could, in fact, be deduced directly from the definitions of the rules themselves. While the experiments on interference control confirm this failure mode, they fail to establish broader patterns regarding how such failures manifest across different task structures.
W2. The semantic construction of controlled inputs and the transformation process applied to natural data remain insufficiently defined, limiting the interpretability of the results.
Although Section 4.1 and Appendix C outline a 128-token layout, four categories of suffixes, four tiers for the number of edits, and 12 test cases per category, the paper fails to provide specific details regarding the content of these suffixes, the exact number of edits applied, representative full-length prompts, or the specific semantic factors altered between different cases.
These details are not merely incidental engineering notes; they are critical. Whether a suffix repeats stale information, contains specific answer formatting, or includes relevant fields fundamentally shapes how one interprets the impact of stale cache states and the observed non-monotonicity in recomputation. Providing only the cache span and the number of clusters is insufficient for readers to determine exactly what is being measured by these stress tests.
Furthermore, the "natural" data has also been subjected to filtering based on "literal-grounded eligibility" and compatibility with a fixed schema/prompt structure. Appendix A shows that, after excluding existing data, only 528 out of 1,941 eligible conversations were compatible with the freezing protocol, with 384 ultimately being executed. While the authors correctly note that this does not represent the proportion of production traffic, they have not sufficiently detailed how raw conversations were converted into 160-token inputs, nor have they clarified the differences between the types of modifications that were excluded versus those that were retained.
W3. Discrepancies in "strict matching" for naturalistic tasks have not been adequately decomposed into instances of residual "old" values versus failures in semantic revision.
In the primary naturalistic experiment, the pre-specified "entire-record" endpoint yielded only 58 correct outcomes out of 54 pairs—a difference that is not statistically significant. Stronger evidence from the naturalistic setting emerges from the analysis of specific modified fields: 256 correct for "Dense" versus 231 for "Reuse," representing a net decrease of 6.51 percentage points. The authors appropriately label this as a post-hoc analysis and correctly acknowledge that applying corrections for multiple testing does not transform it into a confirmatory analysis.
However, a modified field being "not equal to the gold standard" is not synonymous with "retaining the old, replaced value." The scoring method described in Appendix A.3 does not account for case sensitivity, internal whitespace, Unicode variations, or ontological equivalence. Table 13 illustrates this with the example of "Sunday" versus "sunday," and Appendix E.2 notes six instances where target field updates failed to improve the "strict" score simply because their surface form differed from the gold standard.
Consequently, the observed differences between the two approaches may conflate several distinct issues: residual old values, other incorrect values, or mere surface-form inconsistencies. The paper fails to provide a granular breakdown of the critical cases where "Dense" is correct but "Reuse" is incorrect. Furthermore, the "fix" analysis does not fully resolve this ambiguity: of the nine complete fixes, only four fall into the "Dense-correct/Reuse-wrong/Patch-correct" category—and "Dense" itself could also benefit from these same correction rules.
W4. The comprehensive safety audit was not applied consistently across the primary comparisons, precluding a direct, comparable assessment of coverage versus safety.
One of the paper’s most significant findings is that both "entire-record" fix/break metrics and accuracy metrics focused on specific modified fields may fail to capture instances where the model causes harm. However, most of the cross-model results in Table 3, the ablation study on relaxed rules in Table 16, and the "retained-record" and "erratum" alternatives still rely primarily on these incomplete metrics.
For instance, the "prior-record direct update" and "append-only erratum" approaches achieve 89 and 95 correct records, respectively—outperforming the 63 correct records of the "Whole-field Patch" method—yet each results in six "broken" (corrupted) records. Based solely on these counts, it is impossible to determine how many additional localized field-level errors were introduced, nor can they be compared on an equivalent basis against the seven instances of field-level damage caused by the "Whole-field Patch."
The authors acknowledge that Table 16 does not establish "field safety," nor do they claim that each individual "guard" is strictly necessary. While this disclosure is appropriate, the underlying empirical question remains unanswered: which rules actually improve the trade-off between "whole-field damage" and "repair coverage"?
Although "trusted field-key replay" offers diagnostic value, it relies on reliable field bindings that exist outside the original method's inputs; therefore, it cannot be considered a complete deployment solution that resolves the challenge of field identification.
W5. While the verification of cache interventions is thorough, issues regarding execution variance and the specific software versions used remain unresolved.
The eight categories of log checks in Appendix B help confirm that cache interventions were executed as intended. However, they also reveal a discrepancy: the source hash of a recovered "natural-context engine" does not match the hash recorded in historical logs. While this does not definitively prove that the original results are invalid—or that different code versions were used across the three sets of experiments—it does mean that the currently recovered materials are insufficient to fully verify the specific implementation versions used during those runs.
Furthermore, although the "Dense" and "Full-tail" approaches theoretically utilize the same revision context, they do not yield identical results across all experiments. In the controlled experiments involving Qwen, there is one discrepancy in "parsed records" and two in the raw strings; similarly, in the "natural" (real-world) experiments, there are differences in the counts of correct whole records and modified fields. The paper reports these observations but does not quantify the variance in results when the same strategy is executed repeatedly.
This does not negate the clear net degradation observed in the two controlled models. However, regarding the minor discrepancies in whole-record counts within the natural-data experiments—as well as the variations in one or two cases within the re-computation grid—the issues of reproducibility and potential differences in execution environments complicate the interpretation of these results.
Questions for authors
-
How is non-monotonicity distributed across the four suffix categories and varying numbers of edits? Can specific conditions that lead to improvements, regressions, or oscillations be identified from existing cases, and how do these patterns extend upon the findings already presented in §6?
-
Please provide a representative set of prompts from the controlled test suite, the definitions for each suffix category, the range of edit counts used, and the specific rules for converting natural conversations into 160-token inputs. In terms of the types of modifications, how do the 528 schema-compatible cases differ from those that were excluded?
-
Among the 30 cases in the natural experiment where the "Dense" approach succeeded but "Reuse" failed, how many instances resulted in the output of an old (stale) value, and how many involved other types of errors or superficial formatting differences? Do the conclusions drawn from these paired comparisons still hold after applying a pre-determined, type-aware semantic audit?
-
Could you conduct a comprehensive audit—covering the full set of pre- and post-edit errors—for the primary rules outlined in Table 16, as well as for the two key alternatives: "prior-record" and "erratum"? Does this analysis alter the paper’s findings regarding the trade-off between coverage and risk?
-
What specific functionalities and experiments are involved when the recovered code is inconsistent with the historical hash? How significant are the variations in records when the same strategy is executed repeatedly, and are these variations substantial enough to influence minor natural discrepancies or localized non-monotonic phenomena?
-
How are the "old/new" values, schema information, and trusted field keys extracted from the actual runtime environment? Once these processes—including cache construction (or amortization), the quality of repeated executions, and potential field corruption—are accounted for, under what specific conditions does the system still yield a net benefit?