acceptodds
Under review as a conference paper at ICLR 2027

CodeComp: Structural KV Cache Compression for Agentic Coding

Abstract

Agentic coding systems repeatedly read source files, execute tools, and contain long interaction histories, making KV cache a major bottleneck for repository-level inference. Existing KV cache compression methods largely rely on attention signals, which can miss code tokens that are structurally important for program understanding. We present CodeComp, a training-free KV cache compression method for agentic coding that incorporates static program structure into cache retention. CodeComp uses Joern-derived structural annotations to guide compression and preferential retention within historical Read results. On a subset of SWE-bench Verified, CodeComp achieves resolve rates close to those of the native agent on both Qwen3.6-27B and Devstral-Small-2-24B, while reducing peak KV cache memory during inference by 22.1% and 10.6%, respectively. These findings support program structure as a complementary signal for KV cache compression. Code: https://anonymous.4open.science/r/CodeComp/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM efficiency, tool use, agent memory, code generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8siJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CodeComp is a training-free KV cache compression method for coding agents. It targets historical READ results and uses Joern-derived program structure to prioritize code regions and supplement attention-based token scores. Compression is applied after full-prompt prefill, leaving stored messages unchanged. Experiments on 100 SWE-bench Verified tasks with two models show reduced peak KV cache memory and task resolution rates close to those of the uncompressed agent.

Critical strengths

1.The paper addresses a practical problem in coding agents. Using program structure to guide cache retention is a reasonable approach, and the method requires no additional model training.

2.The evaluation includes complete agent runs, comparisons with attention-based and chunk-based compression, and component ablations. The reported end-to-end latency includes structural analysis overhead.

Critical weaknesses

1.The contribution beyond simpler code-aware retention is unclear. CodeComp converts program structure into token priorities and an additive scoring term. While this is a useful design, the experiments do not establish whether control-flow and data-dependency analysis is necessary. ChunkKV provides a relevant comparison, but retaining complete statements or functions would be a more direct alternative. Without such comparisons, the gains could mainly reflect preserving coherent code regions rather than exploiting program dependencies.

2.The structural analysis does not sufficiently establish better preservation of task-relevant information. Higher retention of Joern-annotated tokens shows that the method follows its structural priorities, but does not demonstrate that these tokens are necessary for the repair. Differences between attention scores and structural scores alone do not establish harmful pruning. Moreover, compression occurs after full prefill, so removing a token’s KV is not equivalent to removing that token from the original input. The ablations are also mixed: hard protection alone reduces resolved tasks from 24 to 20, while the full method reaches 28. The interaction between the components needs a clearer explanation.

3.The empirical advantage is modest and its reliability remains uncertain. Compared with ChunkKV, CodeComp resolves only one additional task on Qwen and three on Devstral, with small additional memory savings. These results come from single runs on 100 tasks; ablations use only 50 tasks and one model. Repeated runs and comparisons across multiple retention budgets are needed to establish a reliable advantage. The practical benefit is also bounded: full-prompt prefill and context-window requirements remain unchanged, and latency improves over the native agent on only one of the two models.

Questions for authors

1.How does CodeComp compare with retaining complete statements or functions under the same KV budget? What additional benefit comes specifically from dependency information?

2.Can controlled experiments on identical agent histories show that structural retention better preserves correct edits or tool decisions?

3.Do the gains over ChunkKV persist across repeated runs and different retention budgets? Can fixed-history profiling separate direct compression savings from changes in agent trajectories?

1 comment

Thank you for these suggestions. We appreciate your careful reading and detailed feedback.

  • Structural retention: We agree that retaining more Joern-annotated tokens does not prove that they help repair. We will revise this interpretation and add controlled experiments on identical histories to evaluate subsequent edits and tool decisions.
  • Reliability of the results: We acknowledge that the small gains over ChunkKV and the single-run evaluation leave uncertainty about their reliability. We will include repeated runs and broader retention-budget comparisons in future evaluations.
  • Fixed-history profiling: We agree that the reported 22.1% reduction includes trajectory effects. We therefore profiled one fixed final-turn history from each of 50 tasks, replaying identical full output sequences with and without compression. Averaged over paired requests, compression reduced peak active KV memory by 13.21 MiB and increased HTTP latency by 205 ms. The memory measurement covers both prefill and decoding and excludes the preallocated pool size. These results show modest direct peak-memory savings with additional latency in this setting.
Reviewer_UgurOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper talked about how to do kv cache eviction for coding agent. Specifically the author identified that coding agent's context have a lot of program structures that can be utilized for kv cache eviction, where traditional attention based methods do not possess. The author used Joern to construct the code property graph and evict tokens to obtain peak kv cache memory reduction.

Critical strengths

Leveraging domain structure, specifically for coding to do kv cache compression is a promising idea. The author indeed found a lot of issues with context management of coding agents, and the motivations are sound: the read outputs consist around 41.65% of the total prompt tokens, and not all information are useful.

Critical weaknesses
  1. The biggest weakness, or concern, is how are KV cache reused across multi turns. From the paper, it seems that the method performs eviction after full-prompt prefill, but it is unclear what happens in the next turn. Does the model need to recompute the evicted KV cache, or can it reuse the compressed cache directly? If repeated prefill is required, this could significantly undermine the efficiency gains. A useful metric might be prefix cache hit rate.

  2. The overall compression ratio is relatively low, around 15%. Given this low compression ratio, much stronger neural kv cache compression methods such as kvzip or attention matching's performance should be reported. Also results are marginal comparing to chunk kv.

  3. Experiment size are small, only 2 model and several cases tested. Given that results are marginal, its hard to know if it's noise.

Questions for authors

See weakness 1, for every single turn, do you continue from the current cache or rebuild the cache?

1 comment

Thank you for identifying these issues. We appreciate your careful reading and detailed feedback.

  • How are KV cache reused across multi turns: Each turn rebuilds the KV cache through full-prompt prefill, then compresses it for decoding. Prefix caching is disabled, so repeated prefill remains an overhead.
  • The overall compression ratio is relatively low, around 15%: We agree that the modest overall compression and small gains over ChunkKV warrant stronger baseline comparisons. The overall ratio is limited by our focus on historical READ results, while recent messages and other context are retained. We will include stronger neural KV cache compression baselines, such as KVzip and Attention Matching, in future evaluations.
  • Evaluation size and reliability: We acknowledge that the two-model, single-run evaluation leaves uncertainty about the small differences in resolved tasks. We will expand model coverage and include repeated runs to assess whether these differences persist.
Reviewer_h4JAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CodeComp, a training-free KV cache compression method for coding agents. After each prefill, it compresses the KV entries of historical READ (file-content) results outside a recent window. Joern code property graphs provide structural annotations, which are used to (i) hard-protect a limited number of structurally important lines and (ii) add a structural prior to attention-based top-k token selection in each layer. On 100-task subsets of SWE-bench Verified with Qwen3.6-27B and Devstral-Small-2-24B, CodeComp resolves a similar number of tasks to the uncompressed agent, and the paper reports peak KV memory reductions of 22.1% and 10.6%.

Critical strengths
  • The paper targets a realistic agentic setting (SWE-bench Verified) and includes a summarization baseline and its combination with the method.
  • Using program structure as a retention signal is a new idea, and the paper reports negative ablation results honestly.
Critical weaknesses
  1. Multi-turn KV reuse and the released code.

The paper says the partition is "recomputed at every invocation" after full-prompt prefill (p4), and that compression "does not reduce prompt length, prefill computation" (p9), but it never says where the full KV is kept or mentions radix/prefix caching. The anonymous repository uses SGLang 0.4.9.post3 (the paper says v0.5.11), lists no Qwen3.6-27B or Devstral model, contains single-turn SWE-bench Lite and InfiniteBench scripts rather than an agent loop, and starts the engine with disable_radix_cache=True and disable_cuda_graph=True; compressed KV is merged in Python and passed back via past_key_values. Which code produced the agent results? Is the full KV re-prefilled every turn or kept in a radix cache, and how is peak KV memory measured?

  1. Peak memory is confounded with trajectory changes.

The 22.1% reduction compares runs with different trajectories (prompt tokens per task 235,946 vs. 163,122, Table 1). In the controlled comparison (r = 1.0 vs. 0.6), peak KV drops only from 5.71 to 5.34 GiB (−6.5%), and the uncompressed run is faster (282.5 s vs. 296.5 s, p12). How much of the reduction comes from compression itself? Please report peak memory and latency on fixed trajectories.

  1. Reported compression is a configuration value.

"Eligible KV Cache Comp." is 39.99% (= 1 − r) for every method; actual overall compression is 13–16% (Table 1). A fixed 40% is removed at every call regardless of need. How many calls actually compress, and how often does READ alignment fail?

  1. SideQuest and related work.

SideQuest (arXiv 2602.22603) addresses the same problem, KV eviction of tool results in long-horizon agents, and evicts whole tool responses, avoiding per-head token relocation in serving. CodeComp compresses only past READ results (p4), which are tool responses, yet selects tokens inside each one. There is no related work section in the main text; KVzip, Fast KVzip, R-KV and GUI-KV are not cited, and DuoAttention and RazorAttention appear only in the appendix. Why not evict whole stale READ results? Please compare with SideQuest or with a baseline that drops whole stale READ results.

Questions for authors

Please address the weaknesses above.

1 comment

Thank you for identifying these issues. We appreciate your careful reading and detailed feedback.

  • Released code and multi-turn KV handling: We mistakenly released an outdated implementation that did not match the agent experiments described in the paper. We have updated the repository and acknowledge that the cache lifecycle was insufficiently explained. Each turn rebuilds the KV cache through full-prompt prefill, then compresses it for decoding. Prefix caching is disabled, so repeated prefill remains an overhead.
  • Fixed-history memory and latency: We agree that the reported 22.1% reduction includes trajectory effects. We therefore profiled one fixed final-turn history from each of 50 tasks, replaying identical full output sequences with and without compression. Averaged over paired requests, compression reduced peak active KV memory by 13.21 MiB and increased HTTP latency by 205 ms. The memory measurement covers prefill and decoding and excludes the preallocated pool size. These results show modest direct peak-memory savings with additional latency in this setting.
  • Compression metrics: The nominal retention ratio applies to eligible historical READ entries; overall compression also depends on their share of the prompt and protected content. In our additional fixed-history replay over 50 tasks, 850 of 1,260 calls physically released KV slots. Another 28 calls had eligible READ content but released no slots, and 382 had no eligible READ content. All 3,440 eligible READ occurrences were successfully aligned. We will clarify these metric definitions and report compression coverage separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.