acceptodds
Under review as a conference paper at ICLR 2027

Reusing Retained History for Fast Prefill after Agent Context Compaction

Abstract

Long-running AI agents often shorten their conversations by replacing older messages with a summary while keeping recent messages unchanged. When the model resumes, it normally processes these retained messages again, even though it has already computed and saved their internal key–value (KV) states. Reusing those states could save time, but the messages now follow a different history and appear at different positions. We study a simple reuse policy: process the new summary, move the cached keys to the retained messages’ new positions, and keep their cached values. To assess whether this shortcut preserves the agent’s next decision, we compare its replies with fresh computation of the same compressed input. We measure overall reply similarity, content agreement in both directions, and differences in proposed tool actions. In targeted Qwen3-8B replays, full-reply BGE-M3 similarity ranges from 0.780 to 1.000, yet replies that score as similar can propose different code edits. Separately, at measured native Mistral-Medium-3.5-128B compaction points, reuse makes the input-processing step 13.80× faster in geometric mean than the fastest measured fresh implementation, once the caches are prepared. Reuse therefore saves substantial computation in this measured setting, but leaves a semantic gap that task success or a single similarity score alone cannot characterize.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache reuse, context compaction, language model agents, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CYppOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the idea of reducing the cost of agent context compaction by reusing the retained KV cache before compaction. The idea is that during agent compaction, while some histories are summarized, a notable number of messages are retained with their KV cache readily available and thus can be reused. The problem is that the KV cache of the same texts before and after compaction differs, leading to different subsequent agent behaviors. The paper then studies the efficiency as well as behavior differences of this reuse mechanism compared to the no-reuse baseline.

Critical strengths
  • The paper is well motivated by identifying that retained messages account for a median of 85.2% of the compressed observed-message input. This makes reusing the existing KV cache of these messages reasonable.
Critical weaknesses
  • My main concern is that the paper provides very limited evidence (46 tasks) on whether the KV cache reuse hurts the agent's task success rate. I think this is what the authors should put more effort into, rather than emphasizing too much on how the reuse changes the semantics or behavior without relating them to how they change the task success rates. It's quite obvious that the model's outputs will change given different KV cache input. What matters is to what extent the reuse hurts utility.
  • The reported speedup is measured with the reusing cache prepared, with some potential reuse-specific overhead excluded.
Questions for authors
  • Can the authors provide more convincing evidence on the extent to which KV cache reuse hurts its utility?
  • Can the authors provide an end-to-end comparison of speedup?
2 comments
AuthorsOpenReview ↗

Functional utility, full-data reference, and cost accounting

Thank you for bringing the discussion back to task utility. We agree with your central point: showing that cached representations change an agent's outputs is not enough to tell a reader how much useful performance is lost. This prompted us to put official test outcomes at the center of the revised evaluation and to revisit how we present the task statistics.

What we can say about utility

The controlled three-boundary study now measures local recovery and official tests on the same Mistral executions. REUSE reduces local recovery time at all three boundaries (1.56×–8.56× versus FRESH-1). Final official-test counts agree with FRESH on Astropy and Sympy, but Seaborn has seven additional preservation-test failures relative to both FRESH repeats (86/94 versus 93/94 passing). All three boundaries remain unresolved within the fixed budget. The practical finding is therefore a local saving with a measured functional risk, rather than a task-success improvement. Your broader question about the frequency and average size of this risk still calls for a larger controlled evaluation from the start of each task.

The new strict-configuration Astropy comparison also executes selective 10%, FRESH and REUSE for ten calls each. All end at 0/4 target and 63/68 preservation tests, versus 68/68 preservation initially. It is one additional comparison on an existing boundary, not three independent tasks or new successful solves. Its 2199.2 s of arm spans and 248.8 s of scoring are reported separately; task-prefix and initial-summary costs remain missing.

Your emphasis on utility also changes how we interpret differing actions. In the strict Astropy comparison, different edit-argument forms produce the same first patch and tool result: a verified case where output disagreement does not mean a different state change. Later tool feedback could also compensate for an earlier divergence, but we have not established that mechanism. Conversely, equal test counts can hide different failures, and all three arms here remain unresolved. We use this trace to explain why behavioral disagreement alone is a poor utility metric, not to claim equal task quality.

Across the 54 historical task pairs analyzed in the revision, REUSE solves 68.5% (37/54), compared with 70.4% (38/54) for FRESH: a difference of one task, or −1.85 percentage points. Appendix B documents the analysis set, inclusion decisions and complete reference results. We treat these rates as descriptive evidence, rather than a non-inferiority result.

The prompt audit identified differences in system text after the initial branch request, so this historical comparison does not isolate the effect of KV reuse. Normalizing workspace paths removes the inspected text differences, but their effect on task outcomes was not measured. We therefore cannot attribute the observed gap solely to reuse or dismiss it as a prompt artifact. This is why the new controlled study fixes system prompts and paths throughout and reports its results separately. It provides the cleaner test of the mechanism; the historical rates provide broader, less controlled task coverage.

The two Django exclusions have separate protocol grounds: an invalid second FRESH backend boundary and an out-of-contract shared-summary fallback. The audit appendix retains these decisions, unscored attempts and original scores. Four earlier short continuations preserve already-passing fork states; their 8/8 passing arms are not new task solves and do not enlarge the success-rate denominator.

Accounting for the work around recovery

Your concern about prepared-cache timing is also well taken. We have made the starting state and excluded work explicit. The twelve-boundary 10.98× result starts with complete old KV and a shared summary, charging rendering/alignment, selection/copying, summary prefill, correction, assembly and host-visible first-token extraction. It includes more recovery work than the original prepared-cache measurement, but still excludes summary generation, model loading and task-prefix work. The new study adds actual continuation wall time and official utility, while explicitly retaining its narrower local timer.

Historical logs recover partial backend costs for 99 of 102 attempts; three lack the necessary logs. Summary time is a subset of generation time, and tool, transport, disjoint KV preparation and full-path timestamps are unavailable. We leave those quantities missing rather than treating them as zero; the available logs cannot support a complete-task speedup claim. Three qualification failures and two numerical diagnostics cost 637.1 s including startup and cleanup, retained despite missing scientific outcomes.

Thank you for the response. However, I don't find the additional explanation sufficient to address the concerns without further empirical evidence. I am keeping my initial rating.

Reviewer_vMv6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies KV cache reuse after context compaction in long-running agents. When older interactions are replaced by a summary, retained recent messages shift positions and acquire a different preceding context, making their original cached representations inconsistent with fresh computation. The proposed method computes the new summary, applies rotary positional correction to the retained keys, and preserves the cached values to avoid recomputing most of the retained history.

The authors characterize reuse opportunities using public agent traces and evaluate prefill latency, numerical discrepancies, reply semantics, and task outcomes. The method achieves a 13.80× geometric-mean prefill speedup over the fastest measured fresh baseline once caches are prepared. Native task experiments yield identical aggregate pass rates, although predictions and execution trajectories can differ. The study highlights both the efficiency benefits and the remaining semantic limitations of approximate cache reuse.

Critical strengths

The paper identifies redundant prefill computation after agent context compaction and uses public execution traces to demonstrate substantial opportunities for reusing retained history. The method requires positional correction of cached keys while preserving values. It achieves a 13.80× geometric-mean prefill speedup over the fastest measured fresh baseline under prepared-cache conditions, with correction overhead included.

Critical weaknesses
  • Incomplete joint evaluation of efficiency and quality. Position-corrected reuse remains approximate because cached representations retain their dependence on the original history. Performance, semantic fidelity, and task outcomes are evaluated in separate runs and partly across different models and implementations, making the quality–efficiency trade-off difficult to assess. Matching aggregate task pass rates does not establish equivalence.

  • Limited evidence of generalizability. The detailed semantic diagnostics rely on six complete-history Qwen replay cases, while task evaluation focuses on coding-agent workloads. Broader coverage of models, task types, and compaction policies would strengthen confidence in the method’s applicability.

  • End-to-end benefits remain unclear. The reported speedups measure prepared-cache prefill and include positional correction, but exclude summary generation and several preparation steps. Consequently, they do not establish improvements in complete task latency or overall resource consumption.

  • Insufficient evaluation of compaction choices. The paper does not compare against retaining the full KV cache without compaction when resource constraints permit. Furthermore, the summary- and tail-length sweeps evaluate timing only, leaving their effects on task quality unexplored. These comparisons would clarify when compaction with reuse is preferable and how its parameters should be selected.

Questions for authors

Can efficiency and quality be evaluated under a unified setting? Could you jointly measure latency and task outcomes using the same native model and implementation? Would repeated FRESH/FRESH runs help distinguish reuse-induced changes from ordinary execution variability?

1 comment
AuthorsOpenReview ↗

Matched execution, repeatability, and end-to-end scope

Thank you for the careful distinction between local prefill savings and improvements in an agent's actual execution. Your suggestion to repeat FRESH was particularly helpful: it gave us a way to examine variation within the reference itself, alongside the changes introduced by reuse.

Connecting the speedup to the execution it actually changes

The revision adds three natural-boundary comparisons on Mistral-Medium-3.5-128B, each with FRESH-1, FRESH-2 and serial REUSE under the same twenty-call/forty-five-minute bound. Local recovery is faster at all three boundaries: 8.56×, 1.56× and 5.48× versus FRESH-1. The main evaluation and Appendix A now follow each timed reply through its actual tool trajectory, selected patch and official tests, so this benefit can be read alongside its functional consequences. FRESH repeats begin from identical inputs and workspaces with greedy seed-zero decoding. The repeats first produce different output tokens under matching inputs on calls two, five and nine for Astropy, Seaborn and Sympy, respectively. First canonical tool calls and final official-test counts agree in all three FRESH pairs. Inputs diverge on following calls. These repeats showed us that fixing greedy decoding and the seed was not enough to make these executions bit-identical.

The important distinction, in our reading of these results, is that variation in FRESH is not an explanation for every REUSE difference. Both Seaborn FRESH repeats preserve 93/94 tests, while REUSE preserves 86/94. We therefore retain that regression and its patch/test evidence. With two repeats at three boundaries, we can document variation, but cannot estimate its population distribution or establish non-inferiority.

A separate exploratory strict-configuration Astropy comparison adds selective 10%, FRESH and REUSE at a common ten-call/20-minute cap. All three reach ten calls and end at 0/4 target and 63/68 preservation tests. This provides a same-execution selective comparison, while its single boundary and fixed order remain insufficient to estimate utility loss or establish a stable timing ranking. It is separate from the original FRESH-repeat study.

Why different trajectories can receive the same test score

There are at least two possible mechanisms. First, different action representations can have the same effect: in the strict Astropy run, REUSE formats edit arguments differently from FRESH/C10, yet the tool produces the identical first patch and result. Second, later tool feedback might allow different trajectories to reach equally effective repairs; this is a hypothesis, not a recovery mechanism established by our traces. The competing explanation is that the final tests simply do not distinguish the resulting states. In particular, equal unresolved verdicts are a coarse outcome. We therefore distinguish token agreement, executed state changes and final tests rather than treating any one as a substitute for the others.

What the timing results mean for an agent

Boundary-to-patch wall time includes subsequent inference, tools, transport and later compaction. It excludes initial replay preparation, the saved summary's generation and the original task prefix. On Astropy, REUSE's local recovery is 8.56× faster, but its complete first reply takes 43.5 s versus 44.8 s and its continuation takes 1,314.6 s versus 1,130.6 s. On Seaborn, shorter continuation accompanies lower preservation quality. For us, the Astropy result makes your concern especially concrete: a large first-token saving can coexist with a longer continuation. Since the trajectories perform different work, we report their wall times without treating the ratio as an equal-work task speedup. Model loading, old-KV formation and scoring are reported separately, with overlapping components identified rather than added twice.

Full recomputation of B|S|T is distinct from retaining B|H|T without compaction. We agree that a matched no-compaction policy and quality sweeps over summary/tail lengths would help identify when reuse is worth using. We have not completed those comparisons, or the additional models and non-coding workloads needed to answer the generalization question. The existing length grid measures timing only. Qwen replays remain numerical/semantic diagnostics, and CPU-qualified Terminal-Bench environments are not counted as model evaluations. We have revised the abstract and conclusion accordingly and moved historical details to the appendix. The useful distinction emerging from your review is between saving recovery work and improving an agent run. Our measurements establish the former at these boundaries and show why the latter must be checked rather than assumed.

Reviewer_n75FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether agents can reuse the KV-cache of latest messages instead of recomputing them when summarizing the previous messages. The author presents a method that prefills the new summary, adjusts the positions of Keys and retains the previous Values. This method achieves 13.80x prefill speedup over the fastest tested fresh baseline on Mistral. This paper also analyzes the impact of KV-cache reuse on replies, tool use and task results.

Critical strengths
  • Costs can be pretty high every time recomputing a huge chunk of recent messages after compacting contexts. This paper examines the difference of re-computation and cache reuse using the same compacted input, therefore figure out the influence of both.
  • This paper test a lot of full computation baselines, and make comparisons of reuse without position modification. Moreover, tool use cases in which even two responses seem to be similar or tests results are the same can stem from different actions the agent takes.
Critical weaknesses
  • The paper lacks of comparisons of selective re-computation or residual correction with the same compacted input in other papers. Since position correction is already used in prior works, this comparison turns out to be very important. Therefore, it remains unclear whether position-only reuse offers a useful speed-quality trade-off compared with existing approaches.
  • The reported results make it hard to assess speed and quality together. The main result comes from Mistral speedup, however, the analysis uses only six complete-history boundaries with Qwen3-8B, all from Astropy. Native Mistral task use a different computation path, even though these experiments present promising results, it is still hard to judge the trade-off between speed and quality for the main Mistral implementation.
Questions for authors
  • Can you report latency, differences in tool names and arguments, and functional test results when using the same implementation and boundaries as the main Mistral timing experiment?
  • What is the performance of speed-up and output quality after making comparison experiments between position-only correction and selective recomputation or residual correction using the same compacted input?
2 comments
AuthorsOpenReview ↗

Joint latency, tool behavior, and selective-recomputation controls

Thank you for identifying the two comparisons needed to judge this trade-off. We have added a joint Mistral timing-to-test study and a selective-recomputation continuation. The latter now gives a direct answer on one boundary: both methods save local recovery time, with the same final test counts, but position-only reuse is not faster than selective 10% in that measurement.

Same implementation, timed reply and functional tests

Section 4.3 and Appendix A use Mistral-Medium-3.5-128B, the same eager implementation and Pi agent for FRESH-1, FRESH-2 and serial REUSE. Each arm starts from identical compressed inputs and workspaces, rebuilds old KV from the same full input, and fixes system messages, tools and paths. Pi executes the timed reply and continues with its actual cache; official tests remain hidden from the agent.

Screening eleven repositories before method outputs yielded three eligible unresolved natural boundaries: Astropy13398, Seaborn3069 and Sympy17630. All had passing preservation tests initially. Each arm receives twenty calls or forty-five minutes; this conditional sample is separate from the historical task counts.

For Astropy, Seaborn and Sympy respectively, REUSE local recovery is 405, 2001 and 449 ms versus FRESH-1's 3468, 3120 and 2463 ms: 8.56×, 1.56× and 5.48× faster. First tool names match on 3/3 boundaries and canonical arguments on 2/3; Astropy and Sympy first replies are token-identical. On Seaborn, both use bash, but FRESH inspects scale code while REUSE searches tests for Nominal. All tasks remain unresolved. Astropy ends at 0/4 target and 63/68 preservation tests; Sympy remains at its initial 1/2 and 19/19. Both Seaborn FRESH repeats preserve 93/94 tests versus REUSE's 86/94, with 0/2 target tests throughout. The REUSE patch's invalid axis.grid call matches the test-log exception. Thus, identical task verdicts would conceal a concrete functional regression.

The local timer includes matching/copying, summary prefill, key correction, assembly and host-visible first-token extraction. Initial tokenization, tail localization, old-KV preparation, complete replies and scoring are accounted for separately. These are local recovery gains, not complete-task speedups.

Selective recomputation: the direct comparison and its limits

The new exploratory comparison uses the first eligible Astropy boundary, strict deterministic algorithms and a fixed cuBLAS workspace. C10, FRESH and REUSE each receive ten calls/20 minutes. C10's unchanged 100% numerical control passes on all ranks. Local recovery is 706/3803/803 ms, respectively: C10 and REUSE are 5.39× and 4.74× faster than FRESH in this run. All reach ten calls and end at 0/4 target and 63/68 preservation tests. A single boundary, fixed order, qualification warmup and shared GPUs prevent a stable method ranking. A separate residual-correction baseline remains untested.

The trace also helps interpret your observation that different actions can have the same outcome. C10/FRESH first replies match. REUSE uses top-level oldText/newText instead of an edits array, but all call edit and produce the same first patch SHA and tool result, without parser repair. This is an observed example of different argument forms realizing the same state change, not evidence that all subsequent trajectories or final patches are equivalent.

Mechanistically, rotation repairs the tail keys' positions, but its keys and values still encode the old history rather than the new summary. Selective recomputation refreshes some of those history-dependent states; position-only reuse does not. This explains why the methods can differ despite identical visible inputs, but does not identify the cause of any particular task failure. Our contribution is to measure this remaining approximation in agent execution; positional correction itself is credited to prior work.

The broader twelve-boundary, summary-ready comparison reports 10.98×/3.84×/1.89× recovery speedups for serial/10%/30%, with first canonical-tool agreement of 6/12 each. Those timings and replies are linked across versions and lack multi-step verdicts. The adapters are CacheBlend-derived compaction adaptations, not unmodified LMCache; tail fractions are not total-work fractions. Appendix A retains the original failed 10% numerical controls, dependent unrun 30% continuations, FRESH diagnostics and observed costs. The strict-profile pass does not erase those failures or establish a general remedy. Together, these comparisons make the trade-off observable while leaving general quality preservation open.

Two concerns remain:

  1. The selective comparison covers one boundary at one budget (10%; 706 versus 803 ms) and residual correction is untested, so position-only reuse shows no demonstrated advantage over existing methods.
  2. the main recovery speedup result (10.98x) still sits beside only 6/12 canonical tool agreement, three unresolved boundaries, and a slower Astropy end-to-end run. No speed-quality frontier is shown.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.