acceptodds
Under review as a conference paper at ICLR 2027

Racer: Recoverable KV Cache Management for Multi-Turn LLM Agents

Abstract

Efficient context management is critical for multi-turn large language model (LLM) agents, whose tool definitions and growing interaction histories increase key-value (KV) cache usage. Recent work has explored text-level context compression and KV cache compression to reduce this cost. However, compression may lead to a decision that differs from full-context execution, which in multi-turn agents can change the environment and affect subsequent decisions through the resulting observation. We refer to this phenomenon as compression-induced trajectory drift. To address this problem, we propose Racer, a recoverable KV cache management framework compatible with different KV compression methods. For tool definitions, Racer retains query-relevant tool definitions with full KV representations while compressing the remaining definitions. For interaction histories, Racer protects selected historical information as native KV before generation, then uses a risk detector to identify potentially incorrect drafts and selectively restores historical evidence, allowing the model to regenerate its decision before execution. Extensive evaluations demonstrate that Racer improves mean task success by 43.7% across six KV-compression backends on long-context tasks, while achieving 1.67x higher task throughput than full-context execution under concurrent serving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache compression, LLM agents, multi-turn agents, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eGshOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Racer proposes a recoverable KV cache management framework for multi-turn LLM agents. The paper identifies compression-induced trajectory drift, where KV cache compression causes incorrect decisions that propagate through subsequent tool interactions. To address this, Racer selectively preserves important tool definitions and historical context, detects potentially incorrect actions, and restores relevant KV cache information to regenerate decisions before execution. Experiments across six KV compression backends show a 43.7% relative improvement in mean task success on long-context tasks and 1.67× higher throughput compared to full-context execution. The key contribution is making KV cache compression recoverable, improving agent reliability while reducing memory overhead.

Critical strengths

Very interesting direction that recovers compressed KV. Could be important.

Critical weaknesses
  1. Not enough evaluations on the method... I can see some gain from table 2. But too little results. Table 1 has nothing about your method.
Questions for authors

Can you run your method on more datasets? Idea is interesting but the experiment is way less than desired.

1 comment

Thank you for your time and feedback. We could have made RACER’s results easier to locate and the two evaluation settings easier to distinguish.

1. RACER results in Table 1

The + Racer columns in Table 1 report RACER’s closed-loop results. For each KV backend, Only is compression without RACER, + Racer is the same backend with RACER, and Δ gives the change at the same history budget. The six-backend mean task success rises from 10.5% to 15.1% on BFCL Long Context and from 20.3% to 24.2% on BFCL Base. Table 1 evaluates history compression, while Table 2 separately evaluates tool-definition compression.

Original submission — §4.2 and Table 1, pp. 6–7, and §4.4 and Table 2, p. 8.

2. Evaluation beyond BFCL

The original submission also includes a closed-loop evaluation on ToolSandbox scenarios. RACER improves official scenario similarity on four of six KV backends. The Agent-LLM-Traces experiment evaluates a different setting, namely tool-definition compression.

Benchmark and settingMetricWithout RACERWith RACER
ToolSandbox, closed-loop history compressionMean scenario similarity across six backends0.5380.551
Agent-LLM-Traces, tool-definition compression with C²KVTool-name accuracy at the same reported 2.42× compression38.46%52.75%

The follow-up experiments extend the BFCL analysis rather than add a dataset. On BFCL Long Context, the gain over compression alone remains +3.2 percentage points after excluding C²KV, with a paired 95% confidence interval of [+1.4, +5.2]. The new component ablation also shows that most of the gain in that setting came from pre-generation history protection.

Original submission — §§4.1–4.2, pp. 6–7, and §4.4, p. 8.

Reviewer_ZUzeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In multi-turn agents, a KV compression error can change an action, alter the environment, and propagate to later turns, which the paper calls compression-induced trajectory drift. Racer wraps existing KV compression backends with several mechanisms. The full-fidelity retention of query-relevant tool definitions, protection of selected history, a lightweight risk detector that triggers retrieval of archived evidence and one regeneration before execution, and rule-based argument correction. Across six backends under a 256 token history budget, Racer raises mean BFCL Long Context success from 10.5% to 15.1% and reports 1.67× higher task throughput than full-context execution under concurrent serving.

Critical strengths
  • In closed-loop agents, an executed action cannot be undone, so recovery must happen before commit. A backend-agnostic wrapper is a sensible design target.
  • The paper reports results across six backends and three budgets, including regressions on ToolSandbox and at 128 tokens, and a serving slowdown in three of four context-length quartiles.
Critical weaknesses
  • The contribution of risk-triggered recovery is not established. In the paper’s own paired counts (Table 15), the pre-action interventions solve only a few more tasks than they break, and none of these differences is statistically significant (exact McNemar p ≥ 0.125). The only clearly significant component effect is protection on H2O, a generic retention heuristic.
  • The +43.7% relative gain relies heavily on the authors’ own C²KV backend, which alone succeeds on only 3.0% of tasks. Excluding it, the mean gain is about +3.2 pp. The 1.67× throughput counts completed rather than successful tasks. If success rates on the serving subset resemble Table 1, Racer would complete fewer successful tasks per hour than full context.
Questions for authors
  • What are the main results when BFCL families used for detector development and checkpoint selection are excluded?
  • How does Racer compare with compute-matched controls, such as regenerating on a random 20% of decisions or restoring a randomly selected event? How often does regeneration change the action, and how often does such a change fix rather than break a turn?
  • Please report successful-task throughput, and serving results for C²KV without Racer.
2 comments

Thank you for your time and constructive feedback. We checked whether the gains persist after excluding development tasks, compared learned and random triggering, and added a compression-only serving condition.

1. Results excluding development tasks

RACER’s gains persist after excluding the documented development tasks. We excluded the 128 BFCL source tasks used for checkpoint selection, which cover all 15 documented detector training and validation families. This leaves 72 tasks per benchmark.

BenchmarkCompression only+ RACERGain
BFCL Long Context11.4%15.8%+4.4 pp
BFCL Base23.1%27.8%+4.7 pp

These are means across the five backends other than C²KV. On the original bfcl Long Context set, the gain without C²KV is +3.2 percentage points, with a paired 95% confidence interval of [+1.4, +5.2].

2. Random triggering

The learned trigger had higher observed task success than random triggering. For each backend, we set the random trigger probability to target the learned trigger rate. This targets similar rates of recovery attempts, not a fixed rate.

TriggerSuccessful pairsTask success
Learned143/1,10013.0%
Random135/1,10012.3%

3. Serving with and without RACER

Adding RACER to C²KV raised successful-task throughput from 23.3 to 100.8 tasks per hour (4.3×). All three methods used the same updated engine on the same 64-task cohort at concurrency 16.

MethodTask successCompleted tasks/hSuccessful tasks/h
Full33.9%326.6110.3
C²KV-only1.6%1,492.223.3
C²KV + RACER16.1%624.0100.8

Thank you for the additional experiments, which address the data overlap concern. However, the learned trigger barely outperforms random triggering, and Racer's successful task throughput remains below Full method. So, I maintain my rating.

Reviewer_EUs6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates compression-induced trajectory drift in multi-turn LLM agents, where decisions made with compressed context alter the environment and affect subsequent interactions. It introduces Racer, a KV-cache management scheme combining selective retention of query-relevant tool definitions, pre-generation protection of historical context, risk-based retrieval of archived evidence, and source-checked argument correction. A lightweight learned detector estimates the risk of draft failure. When elevated risk is detected, Racer retrieves historical events, repacks the context within the same history-KV budget, and regenerates the pending action before execution. Experiments with Qwen3-4B-Instruct-2507 evaluate six KV-compression backends on BFCL and ToolSandbox, tool-definition compression on Agent-LLM-Traces, and concurrent serving. The reported results show improved closed-loop performance under the same history-KV budget, along with throughput gains under congested, long-context workloads.

Critical strengths

It improves closed-loop performance for matched-history KV budgets. Racer gains task success on all six benchmarked KV compression backends for BFCL benchmarks under 256 position history budget. This gains mean task success from 20.3% to 24.2% for BFCL Base, and 10.5% to 15.1% for BFCL Long Context. Crucially, this gains are for the same history budget, not an additional capacity budget. These findings are important evidence that selective history protection and pre-action intervention can enhance usefulness of a limited KV cache for a variety of compression techniques.

It evaluates end-to-end serving. The paper connects recovery to reusing compressed blocks, using prefix caching and asynch history compression, and factors in recovery cost to serving measurements. For the benchmarked long-context workload (16 concurrent tasks), Racer provides 1.67x throughput of completed tasks and 725 sec to 348 sec p95 task latency (vs full-context). These numbers highlight the real-world deployment potential for congested long-context serving. These measurements add to the value of the paper as a systems contribution.

Critical weaknesses

Lack of sufficient evidence for the incremental benefit of the risk triggered recovery. Racer uses lexical relevance ranking, history protection, a linear failure risk detector, retrieve + re-prefill/regenerate, and argument rule-based correction. Overall approach is useful. However, it is not clear from the available evidence that there is a significant methodological value (in terms of recovery controller proposed). For example, Table 14 shows how in 4 backends mean BFCL Base success rate increases from 22.1% (w/o protection) to 24.1% (w/ protection). The risk triggered event recovery increases success rate only to 24.4% (vs. 25.0% for the whole framework). Even though the detector predicts failure for continuation under no recovery. It is not clear that it is sufficient evidence that the particular recovery intervention will be beneficial. Therefore, it would be important to compare w/ random trigger, NLL trigger, or unconditional recovery under equal recovery/compute budgets to show the usefulness of the learned controller. Similar component analysis (on BFCL Long Context) would help understand headline improvement as well.

In serving it's not clear that there is a quality efficiency trade-off, or how beneficial is the recovery alone. The stated 1.67x throughput increase is about number of finished task episodes (including unsuccessful ones), not number of successfully solved tasks. And due to compression task success rate, and potentially execution time change as well. So we cannot say that useful tasks throughput improved solely from this metric. It would be helpful to share the success rates and # of successful tasks / hour for the same serving cohorts. Also the serving setup has Full and Racer but not just compression, so it's not clear incremental efficiency cost and quality benefit of the recovery. Also, it's not clear that it's useful in many deployment conditions, because the three smaller context quartiles see 0.43x-0.59x slowdown relative to Full's throughput.

Questions for authors

Contribution of recovery to long context tasks. In table 14, you show that protection-only gives 24.1% mean success rate on BFCL Base for 4 backends. That increases to 24.4% with event recovery. To what extent is the gain in BFCL Long Context from event recovery? Can you provide results for protection-only and protection+recovery and the full framework on this benchmark?

The utility of the learned risk detector. How does the learned detector compare to other triggering methods (e.g., random or NLL threshold) for similar recovery rates or computational budget? And since your training label is the cost of not recovering when failing, how often does triggered regeneration undo a failed draft vs. swap a successful draft for a failed action?

Quality-adjusted serving performance. What are the success rates on tasks and successful tasks per hour for Full and Racer on the same serving cohorts that you report the 1.67x throughput gain? Can you show a compression-only condition to isolate the compression gains from the extra quality gains and execution overhead that racer brings in?

2 comments

Thank you for your time and constructive feedback. We ran follow-up experiments to identify which components contribute to Long Context performance and to measure successful-task throughput.

1. Event recovery on Long Context

The largest gain came from giving the model access to high-fidelity history before its first draft. On BFCL Long Context, we evaluated the same four backends as in Table 14 at a 256-token history budget. History protection raised task success from 10.4% to 12.1%, and full RACER reached 12.7%. Protection makes selected historical evidence available as native KV before drafting.

2. Trigger choice and recovery outcomes

At similar target trigger rates, the learned trigger had higher observed task success than random triggering. We changed only the trigger and set the random probability for each backend to target the learned trigger rate.

TriggerSuccessful pairsTask success
Learned143/1,10013.0%
Random135/1,10012.3%

3. Successful-task throughput

Adding RACER to C²KV raised successful-task throughput from 23.3 to 100.8 tasks per hour (4.3×). The original 1.67× refers to completed episodes regardless of outcome. For this follow-up, we used the same 64-task cohort at concurrency 16 and ran all three methods on the same updated engine.

MethodTask successCompleted tasks/hSuccessful tasks/h
Full33.9%326.6110.3
C²KV-only1.6%1,492.223.3
C²KV + RACER16.1%624.0100.8

Thank you for the follow-ups.

1: the Long Context ablation confirms my concern: protection alone supplies 1.7 of the 2.3 pp gain, and the full system adds only 0.6 pp, so event recovery remains unisolated. Please add protection+event-only. 3: the new table shows 100.8 successful tasks per hour against Full's 110.3 (0.91x), with 16.1% versus 33.9% success. At matched quality, not compressing is still faster, so the abstract's 1.67x should be qualified as completed-episode throughput.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.