acceptodds
Under review as a conference paper at ICLR 2027

AsynCompact: Efficient Context Compaction for Long-Horizon LLM agents

Abstract

Context compaction supports long-horizon LLM agents but can interrupt execution during summary generation and KV-cache reconstruction. We present a compaction framework that reduces these interruptions through asynchronous execution and optional KV grafting. The agent continues its task while a background branch generates a summary, then adopts the compacted history at a safe step boundary. For self-hosted inference, KV grafting further reuses cached states for the summary and retained context, with positional rebasing, instead of fully recomputing the compacted context. This reuse is approximate and requires no model retraining. Experiments on three benchmarks show broadly comparable task performance. Controlled boundary replays show that asynchronous compaction reduces mean explicit summary blocking by –. In a separate Qwen3.8-27B replay with one active request per replica, KV grafting reduces mean post-compaction resumption latency by , including graft-specific preparation costs. These results demonstrate reduced waiting at compaction boundaries, supporting more responsive agent execution without relying on consistent full-task speedups.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM, Agent, Context Management, Long-Horizon

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hyjZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AsynCompact, a framework for reducing the latency of context compaction in long-horizon LLM agents. It has two parts:

  • Asynchronous compaction. The history summary is generated in the background while the agent keeps working.
  • KV grafting. When the agent switches to the compacted context, its KV cache is built by reusing and re-positioning cached states from both branches instead of recomputing them.

The method is implemented in SGLang and evaluated on agentic coding and terminal benchmarks. The paper reports less compaction blocking and lower switch latency, with task performance comparable to synchronous compaction.

Critical strengths
  1. Practical problem. The paper studies a practical problem and runs experiments on realistic agentic benchmarks.

  2. Solid engineering for KV grafting. The authors implement grafting inside SGLang's radix and paged KV cache. It supports several attention mechanisms (full attention, hybrid linear attention, and sparse attention) and speculative decoding through MTP draft-KV transfer. Supporting all of these is nontrivial and useful to practitioners.

Critical weaknesses

W1. Limited novelty in asynchronous compaction.

The async protocol is not a novel design. Two papers the authors already cite do something very similar:

  • Slipstream summarizes in the background while the agent continues on the full context, then swaps the summary in.
  • SmoothAgent runs summarization as a background "lookahead" operation. CoMem (in ICML 2026) likewise has a async compaction mechanism that leverages a separate model to summarize the history in the background while the agent keeps working. This async compaction mechanism is also already used in popular agent harnesses, such as Copilot and Codex.

The paper does not identify a substantive difference from these designs, so this part reads as reimplementation rather than a new contribution.

W2. Limited novelty in KV grafting.

This leaves KV grafting as the main contribution, but the mechanism also lacks novelty. Grafting concatenates cached KV segments and moves their keys to new positions by RoPE rotation, with no recomputation (Eq. 6). This operation is well established:

  • "Position re-encoding" in Block-Attention (Sun et al., ICLR 2025, §2.3) and KVLink (Yang et al., 2025, §2.2).
  • "Full KV reuse" in Prompt Cache (Gim et al., 2024) and CacheBlend (Yao et al., 2025, §3.3 and App. A). CacheBlend shows it is insufficient without selective recomputation.
  • The "Naive" baseline in EPIC (Hu et al., ICML 2025, §2.3), described as "too rudimentary to be classified as [a] proper algorithm" and shown to cause substantial accuracy loss.
  • llama.cpp's KV context shift (PR #3228, 2023), which re-rotates cached keys in place after dropping tokens.

The new element is applying this to agent compaction. There, the reused KV states were computed while attending to the history that compaction removes, rather than lacking preceding context. That does not change the mechanism. The paper's own fidelity analysis (about 89% top-1 agreement, Sec. 3.2.2) shows the familiar deviation from a full prefill.

Besides, the work is also suffered from limited generality of the evaluation and integration. All experiments use a minimal custom coding harness (a shell tool plus a context manager). It is unclear whether the results transfer to widely used harnesses such as Hermes Agent, Claude Code or Codex.

Questions for authors
  1. Could the authors explain the technical novelty / contribution relative to the works mentioned in the weaknesses section?

  2. Could the authors please expand the experimental analysis with further harnesses e.g. CC or Codex?

Reviewer_ErM9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework for reducing interruptions caused by context compaction in long-horizon LLM agents. The method combines asynchronous summarization with KV grafting, which reuses cached states from different execution branches through positional rebasing instead of recomputing the compacted context. Experiments on DeepSWE NL2Repo and Terminal Bench 2.1 show reduced compaction blocking time and post-compaction resumption latency while maintaining comparable task performance. The authors also investigate how differences in reused KV representations affect subsequent token predictions.

Critical strengths
  1. The paper addresses an urgent issue in long-horizon agent execution. Separating summarization delays from KV reconstruction costs provides an insightful way to study the overhead of context compaction.
  2. KV grafting is an interesting approach to teh cache reuse. The implementation considers full attention, hybrid attention, and sparse attention architectures without requiring model retraining.
  3. The evaluation is comprehensive, including 3 agent benchmarks, multiple model configurations, and controlled latency replays. I appreciate the representation analysis in Figures 2 & 4, which attempts to explain why approximate KV reuse can preserve predictive behavior.
Critical weaknesses
  1. The main KV grafting evaluation compares cache reuse against conventional recomputation. However, asynchronous cache preparation and selective KV recomputation are relevant alternatives discussed in the related work. A comparison under matched serving conditions would help establish whether KV grafting provides additional benefits over these approaches. In particular, it remains unclear how much of the reported latency reduction could be achieved by preparing the compacted context cache during foreground execution. Overall, limited comparison with alternative approaches.

  2. KV grafting reuses states conditioned on a different causal history. The analysis in Figures 2 to 4 uses eight tasks and short continuations evaluated along shared token sequences. While Table 3 provides useful evidence from complete agent runs, the results do not isolate the effect of repeated compaction or increasing context length. I would like to see performance analyzed against the number of compaction events, retained history length, and potential error accumulation.

  3. Table 4 reports substantial latency and throughput improvements at concurrency levels 1 to 4, but the implementation times out at concurrency 8. The reported throughput also reflects short replay bursts rather than sustained serving. The scalability under concurrent serving remains unknown.

Questions for authors

Q1.How does KV grafting compare with preparing the compacted context cache asynchronously? Is a matched control feasible using the existing replay setup?

Q2.Can the authors report performance or prediction differences grouped by compaction frequency and retained context length using existing trajectories?

Q3. What are the memory and cache occupancy costs of KV grafting? Can the authors further characterize the concurrency 8 timeout and the conditions under which grafting fails?

Reviewer_vMQvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AsynCompact targets the efficiency issue of content compact in agents. It parallelizes the compaction tasks and agent execution. Besides, it proposes KV graft which uses RoPE-based positional rebasing to combine kv states of both background and foreground branches to avoid the cost of re-compute whole KV cache. Across different benchmarks and models, AsynCompact can substantially reduce the latency compare to synchronous baseline while preserve most of the performance.

Critical strengths
  1. It proposes a KV-grafting runtime for self-hosted inference. This reuses existing states instead of recompute the whole KV cache when switching to a compacted context. This design can reduce latency while preserve most of the performance.
  2. The analysis section of KV graft is clear and good.
Critical weaknesses
  1. Since KV graft requires preserve the reusable states, AsynCompact would be sensitive to GPU memory pressure especially under high concurrency situation. Some experiments in the paper shows that, at concurrency 8, the graft implementation times out without a successful commit. It would be helpful to quantify the memory overhead of retaining these states and explain whether it contributes to this failure.
  2. Prior work such as SmoothAgent [1] already parallelize context compaction and KV-cache reconstruction with ongoing agent execution. It is important to clarify under what conditions reusing existing KV states offers additional benefits over asynchronously recomputing them. More directly, it could be helpful to consider including SmoothAgent as a baseline under matched resource budgets. This would help establish the practical trade-offs between avoiding recomputation through approximate reuse and hiding its latency by disaggregating the KV reconstruction.

[1] Zaifeng Pan et al. SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering. arXiv:2607.00151, 2026.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.