acceptodds
Under review as a conference paper at ICLR 2027

Characterizing and Optimizing Context Compaction in Production Coding Agents

Abstract

Long-running coding agents rely on context compaction to overcome finite LLM context windows. Existing harnesses typically perform compaction synchronously, placing expensive summarization on the critical path and disrupting prefix-cache reuse. We deep-dive into compaction behavior and costs by analyzing 13.5M GitHub Copilot agentic coding sessions from 3.2M users, comprising 760M LLM calls. Although only 7.8% of sessions trigger compaction, they account for 44.2% of total token consumption. Compaction also causes substantial foreground delay and sharply reduces prefix-cache reuse. To enable systematic evaluation, we first introduce COMPACTIONBENCH, a benchmark that models software evolution as persistent, multi-turn coding sessions. It enables controlled evaluation of repeated compaction while preserving executable task-level evaluation and reflecting context-growth patterns observed in production. We then present APEC, an asynchronous progressive compaction mechanism that prepares fine-grained, indexed summaries in the background and appends them to a stable prefix, moving compaction work off the critical path while preserving prefix reuse. APEC is harness-agnostic and treats summarization as a black box. Evaluated across frontier models, APEC improves task quality while reducing total cost by up to 25.4% and agent execution time by 9.6%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
context compaction, coding agents, LLM agents, prefix caching, KV cache, long-horizon benchmark, workload characterization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_m6iuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the behavior and system-level costs of context compaction in coding agents on long-horizon task. Through large-scale trace analysis, the authors identify substantial latency and cache-reuse overheads associated with conventional synchronous compaction. To support systematic evaluation, this paper introduces CompactionBench, a benchmark that models persistent software development through sequential, interdependent repository-level requirements. They further propose APEC, an asynchronous and progressive compaction mechanism that incrementally maintains reusable context prefixes. Experiments across frontier models demonstrate improvements in downstream coding accuracy, inference cost, and execution time compared with conventional compaction strategies.

Critical strengths
  1. Insights from real-world production traces. The paper analyzes a large-scale collection of real-world coding agent trajectories, providing useful empirical insights into the frequency, triggering behavior, and efficiency costs of context compaction in production environments.

  2. A benchmark for evaluating context compaction. The paper introduces CompactionBench, a new benchmark designed to evaluate context compaction in long-running coding agents. By constructing multi-turn software development tasks with sustained context growth, it provides a useful setting for studying repeated compaction.

  3. A new compaction method with empirical validation. The paper proposes APEC, an asynchronous and progressive context compaction mechanism. Experimental results demonstrate its effectiveness in improving task accuracy while reducing inference costs and execution time under the evaluated settings.

Critical weaknesses
  1. Potential task–state mismatch in CompactionBench. CompactionBench constructs multi-turn coding tasks from predefined sequences of changelog requirements, without adapting subsequent requests to the agent's actual development progress. However, the repository state evolves dynamically and may deviate substantially from the expected state when earlier requirements are only partially completed or implemented incorrectly. Consequently, later requests may rely on functionality that has not been implemented, creating a mismatch between the predefined instructions and the actual repository state. The paper does not examine how frequently such mismatches occur or how they affect task validity and the resulting evaluation outcomes.
  2. Limited evaluation across models and agent harnesses. The experiments primarily rely on OpenClaude with GPT-5.6 Sol, with additional evaluation limited to GPT-5.4. Although APEC is described as harness-agnostic, its effectiveness is not evaluated under other coding agent harnesses or more diverse model families. This limits the evidence for the robustness and generalizability of the reported improvements.
  3. Insufficient discussion of data privacy and user consent. The paper analyzes a large-scale collection of real-world GitHub Copilot trajectories involving a large amount of users. Although the authors clarify that the collected telemetry excludes prompt content and mention privacy review before data release, the paper provides limited details on user consent or data-use authorization, as well as the privacy safeguards applied during data collection and analysis. Further clarification would strengthen the ethical transparency of the study.
  4. Limited coverage of long-horizon coding scenarios. While repository evolution is a representative long-horizon coding scenario that can trigger repeated context compaction, it is not the only one. Other scenarios, such as developing large-scale applications from scratch or iteratively refining code through user feedback and error correction, may also require frequent compaction. However, neither CompactionBench nor the evaluation of APEC covers these scenarios, limiting the generalizability of the experimental findings.
Questions for authors
  1. Beyond LLM-based summarization, coding agents may employ rule-based context management strategies, such as truncating or removing lengthy tool outputs. Were such mechanisms enabled in the OpenClaude harness during evaluation? Although the paper includes pruning baselines, it remains unclear how these non-LLM strategies interact with APEC. Could the authors clarify whether combining them would affect compaction frequency, task accuracy, and overall efficiency?

  2. Given that CompactionBench uses predefined multi-turn user requests, did the authors examine whether subsequent instructions remain consistent with the repository states actually produced by the agent? Earlier development failures may cause the repository to deviate from the expected state, potentially making later instructions inapplicable or inconsistent with the current codebase. Such mismatches could affect the validity of the constructed tasks and their evaluation outcomes.

  3. What motivates the choice of a 50% soft threshold for APEC? While the paper discusses the trade-off between early compaction and background overhead, it does not provide an empirical justification for this specific value. Additional justification or ablation studies on core parameters would strengthen the rationale for the settings and provide further evidence of APEC's robustness.

6 comments

W1 / Q2. Potential task–state mismatch.

The predefined request sequence is an intentional design choice to ensure reproducible and fair comparisons across compaction policies. Each release transition is treated as a single long-horizon task, where all policies receive identical requirements while the repository state evolves according to the agent’s actual execution. Failures in earlier requirements may affect subsequent development, but this is a natural challenge in long-horizon software engineering rather than an inherent benchmark flaw. More importantly, dynamically adapting requests to each agent’s progress would result in different target specifications, making direct accuracy comparisons against a common ground truth infeasible. Our fixed sequence preserves a shared release-level objective, allowing all policies to be evaluated against the same executable tests.

Beyond this design rationale, the benchmark incorporates several concrete mechanisms that mitigate task–state mismatch and preserve the validity of the evaluation:

  1. State-aware requests. Each request introduces a new feature while asking the agent to keep its changes consistent with, and not break, the previously implemented requirements.

  2. Grading on the cumulative release. The final patch is graded once against the release’s tests, so a missed earlier requirement only costs the tests for that functionality; it does not invalidate the task.

  3. Paired comparison. All policies receive the identical requests, repository, model, and evaluator, so any remaining mismatch comes from the agent’s own execution under each policy, which is what we aim to measure.

  4. Ecological validity. Production users likewise issue follow-up requests based on intended rather than verified repository state; handling such discrepancies is an inherent part of long-horizon agentic software engineering that a realistic benchmark should retain rather than remove.

CompactionBench has been used by production agentic platform design and model training. As in our anonymous repo link (https://anonymous.4open.science/r/APEC-A6D4). We will also open source it to the research community for further research in systems efficiency of agentic context compaction and management.

W2. Limited evaluation across models and harnesses.

Models. We have expanded our evaluation to include a comprehensive model sensitivity study across frontier models from both OpenAI and Anthropic, specifically evaluating GPT-5.6-sol, GPT-5.4, Opus 5.5, Sonnet 5.5, and Haiku 5.5, given now we have more budget. As detailed below, as well as in our updated evaluation section (Table 4), the efficiency gains and accuracy improvements of APEC consistently hold across diverse model families: e.g., APEC improves task accuracy by 3.68--4.57 percentage points over Sync-H, while reducing input tokens by 25.7--40.3%, completion time by 4.7--57.4%, and total cost by 1.0--46.2%.

ModelMethodAcc. (%)Num. Comps.Input (M)Cached input (M)Output (K)Total time (min)Compaction time (s)Overlap (%)Total cost ($)
GPT-5.4Sync-H56.702.4442.70640.164184.67144.47471.470.0041.06
Async-L52.577.4435.40033.024302.54117.13710.8698.1542.32
APEC60.886.1131.72928.337196.6861.53617.3594.7840.64
Opus 5.5Sync-H76.111.9338.53737.79769.9830.2178.540.0015.33
Async-L77.005.0824.54123.53478.9729.73189.2398.6915.28
APEC80.684.7324.27823.17667.7728.56190.7484.5412.58
Sonnet 5.5Sync-H75.883.2059.68858.606107.5440.77103.950.0011.88
Async-L74.718.0736.04934.583121.3639.95239.7598.9111.54
APEC79.567.5835.64234.026104.1438.43243.5287.469.41
Haiku 5.5Sync-H68.832.1743.89943.07895.6627.6739.850.002.79
Async-L67.176.1727.10325.962107.9527.24103.3999.301.91
APEC72.705.7126.73325.47792.6426.37103.3891.921.50

Harnesses We now report results with the Codex CLI harness. We ported APEC to Codex CLI and ran it on the same eight tasks with GPT-5.6 Sol, against Codex’s own compaction: its default blocking compaction at 90% (Sync-H) and its built-in background compaction at 50% (Async-L). Same model, tools, prompts, and evaluator; only the compaction setting differs. Results so far (3 trials per task, 72 trials):

Codex CLI, GPT-5.6 SolAccuracy (%)Cost/trial ($)Time/trial (min)
Codex default (Sync-H)73.7914.8420.9
Codex background (Async-L)69.0111.2417.8
APEC77.4912.0816.6

APEC gives the highest accuracy and the shortest time in Codex too, at lower cost than Codex’s default. This matches what we see in OpenClaude. Repeats 4–5 and the Sync-L and PEC arms are still running.

Run-to-run Robustness We also report standard errors across trials for the Anthropic model family to demonstrate statistical robustness:

ModelMethodAcc. (%)Total cost ($)Total time (min)
Opus 5.5Sync-H76.11 ± 3.4115.33 ± 0.2430.21 ± 0.58
Async-L77.00 ± 4.0115.28 ± 0.4229.73 ± 0.63
APEC80.68 ± 1.2312.58 ± 0.5928.56 ± 0.61
Sonnet 5.5Sync-H75.88 ± 3.3511.88 ± 0.1740.77 ± 0.82
Async-L74.71 ± 4.1211.54 ± 0.3139.95 ± 0.87
APEC79.56 ± 1.239.41 ± 0.4138.43 ± 0.84
Haiku 5.5Sync-H68.83 ± 3.422.79 ± 0.0427.67 ± 0.58
Async-L67.17 ± 4.051.91 ± 0.0427.24 ± 0.61
APEC72.70 ± 1.291.50 ± 0.0526.37 ± 0.60

Notably, APEC reduces accuracy standard error by ~3× compared to both Sync-H and Async-L across all three models (±1.23–1.29% vs. ±3.35–4.12%), confirming that progressive, append-only compaction yields substantially more reliable and predictable execution across diverse model sizes and families.

W3. Data privacy and user consent.

We thank the reviewer for raising this important point regarding ethical transparency. We respectfully note that this concern is already addressed in the Ethics Statement of the submission and the meta message, and we kindly refer the reviewer to it. To summarize the safeguards in place:

  • Institutional review. The collection and use of this data for the present study have passed rigorous legal and ethics/privacy review.

  • Data minimization. The production analysis relies exclusively on aggregated operational metrics, namely token counts, latency, cache usage, and tool-call statistics. The telemetry available to us contains no prompt content, code, or other user-authored text.

  • Purpose limitation. The analysis characterizes system-level behavior (compaction frequency, cost, latency, and cache reuse) and does not examine the content of user interactions or individual users.

  • Responsible release. As stated in the Reproducibility Statement, any release of production telemetry will follow further redaction and privacy review prior to publication.

We believe these measures address the reviewer’s concern, and we will make them more prominent in the revision.

Reviewer_xUR2OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  1. The paper calculates statistics from a 1 week usage of github copilot by developers to demonstrate that only 7.8% of GitHub Copilot sessions trigger compaction, but those sessions use 44.2% of all tokens. This first reveals that context compaction today is significantly costly, even though it is a relatively rare phenomena.
  2. They then show that existing long-horizon agentic coding benchmarks rarely require context compaction for GPT-5.6-sol model at 90% reset, and introduce COMPACTIONBENCH where they take 8 starting releases from SWE-EVO and provide points from their changelog, one at a time like a multi-turn conversation with the user.
Critical strengths
  1. Framing compaction as a question of when and how it runs, not only what the summary contains, is a useful and underexplored angle. Furthermore, the connection to actual real world data and statistics is a useful analysis.
  2. Using points from changelong as multi-turn user conversations is a good strategy as it keeps final verification unchanged unlike other approaches like interleaving.
  3. APEC is lightweight addition and is designed to be composable with other compaction approaches.
  4. The method variants tested in table 3 are good ablation conditions.
Critical weaknesses
  1. Motivation unclear for COMPACTIONBENCH: If the authors wanted to create conditions that cause more compactions, why not just compare different models, harnesses at different trigger points. In fact the -L methods and both PEC and APEC have trigger conditions of 50% set. Do those other benchmarks not trigger at 50%? Why were they only evaluated at 90%?
  2. Missing Baselines:
    1. Authors have created a new benchmark but not tested it with any existing harnesses (beyond the one openclaude), and only 2 models (gpt-5.6-sol and gpt-5.4). If its a benchmark contribution you need to test more models, harnesses, and with varying triggering conditions.
    2. No existing method is compared against PEC and APEC. Parallel compaction (Cim et al., 2026), TokenPilot (Xu et al., 2026), and observation masking (Lindenbauer et al., 2025). While APEC is orthogonal to these, these methods need to be evaluated for the benchmark contribution.
    3. Authors claim that - "These techniques primarily improve the compression operator itself, e.g., through retraining, auxiliary models, or heuristics. APEC is orthogonal to these approaches and can incorporate different compression operators" but don't actually show the results of composing APEC with any of these related works.
    4. No compaction baseline (what if you never compact on the same set of tasks, I undedrstand some tasks might be unsolveable but it is important to see this as a baseline)
    5. Fresh session -- what is the accuracy, cost and time if you reset all context before each new user turn?
  3. The APEC method itself is relatively straightforward, where they summarize sections of message history while the agent continues running and add those summaries in blocks before the recent messages once the summarization is done. Having a simple method is not a bad thing, but looking at the scores the even simpler PEC method has comparable task accuracy (76.74 vs 77.95 at only 8 tasks), and lower total cost (14.75 vs. 16.37). This does not sufficiently justify using APEC.
  4. Several heavy tasks in COMPACTIONBENCH can trigger compaction even with 1M context limit - Unclear how many such tasks there are. If not too many, then the no-compaction baseline becomes even more important.
Questions for authors
  1. How were the 8 tasks selected from this benchmark? Which 8 tasks?
  2. How exactly do you measure accuracy?
  3. What was the standard error for those methods across 5 trials?
10 comments

Thanks for your review, here is our reply.

W1. Why not just run existing benchmarks at a lower trigger, e.g. 50%? Do those existing benchmarks not trigger at 50%?

Short answer: You are right, the existing benchmarks barely even trigger the 50% trigger point.

We ran this experiment. A task can only compact at 50% if its prompt gets above 126K tokens. So we took every task in Table 2 whose prompt got that large (32 of 484 tasks) and ran it again with GPT-5.4 and a 50% trigger. The other tasks never get that large, so they cannot compact.

Table: Context growth and compaction behavior across existing benchmarks (Table 2), under the original trigger and under a 50% (126K) trigger. LLM and tool token counts are mean output tokens per LLM turn and per text-bearing tool call. 50% column: tasks that compact, and the maximum compactions in one task.

Benchmark#Tasks#LLM Tokens#Tool Tokens#Turns#Compactions#Compactions (50%)
(1) Front-loaded context
A. LongSWE-Bench [256k]1000.25K0.62K14.31 (in 1st turn)1 (in 1st turn)*
B. LoCoBench [single turn]1748.90K0K1.0NoneNone
(2) Single-turn autonomous execution
C. SWE-bench890.49K0.62K23.5NoneNone
D. SWE-EVO271.04K—109.9None2 tasks, max 2
E. Terminal-Bench 2.1870.26K2.01K12.5NoneNone
F. Terminal-Bench 4.0610.38K1.08K21.1None2 tasks, max 1
G. ViBench [build]240.87K0.37K14.0NoneNone
H. DeepSWE1080.24K0.73K53.1None8 tasks, max 2
(3) Passive multi-turn continuation
I. SWE-Marathon140.51K0.66K20.3NoneNone
J. LHTB440.30K0.34K136.1None9 tasks, max 3
K. LoCoBench-Agent301.18K1.79K8.7NoneNone
  • Why Table 2 used 90%: Table 2 is meant to show how these benchmarks behave with the default setting (Appendix D.1, Codex also uses 90% by default) as the Sync-H setup . The new 50% runs above answer the reviewer’s question directly.

  • The result: only 21 of 484 tasks (4%) compact at all, and most of them only once or twice. Repeated compaction shows up in a handful of LHTB tasks, where the verifier keeps sending the agent back until it runs out of time. In CompactionBench, every task compacts more than once on average (Table 8), because every turn brings a new requirement. So lowering the trigger is not enough: existing benchmarks still do not produce the repeated grow-and-compact pattern we see in production.

W2. Missing baselines.

Appendix C.3 explains this choice. In short, our paper is about when compaction runs and how the summary is put back into the prompt, not about a better way to compress text. From the appendix:

“Replacing the common summarization procedure or adding these policies would introduce additional design choices into the comparisons above. Our baseline selection follows the mechanisms being evaluated.”

Below we explain each suggested baseline.

(1) Parallel compaction (Cim et al., 2026). It splits one summary call into several block summaries that run in parallel. We do not evaluate it for four reasons.

  • Block boundaries do not transfer. The paper uses a fixed block size and a single prompt (§8), on QA and dialogue data where each turn is a short, self-contained document (HotpotQA, LoCoMo; §3.1). In a coding session, a tool call, its output, and the edits that follow form one unit, and outputs vary from a few tokens to tens of thousands. Where to cut is not obvious, and the authors themselves list content-dependent block sizes as future work (§8).

  • Joining the blocks back is ad hoc. Block boundaries are marked with XML tags that current models are not trained to attend to (§8), and the block summaries are simply concatenated (§4). Nothing ensures the pieces are consistent or free of overlap, and the concatenation replaces the whole history, so the prompt prefix is rewritten at every compaction, as in Sync-L.

  • It is not evaluated end to end. Accuracy is measured by asking a judge whether a one-shot summary of a 100K-token context keeps the needed facts (§6), not by task success in a multi-turn agent run. No dollar cost or cache hit rate is reported, and all models are self-hosted (8B–120B on vLLM).

  • Background compaction already hides more of the time. Parallel compaction still blocks the agent until all block summaries finish (§8). Its best reported speedup of the summary call is 2.13×, and in several settings it is slower than a single call (0.80–0.94×; Tables 7–8). Background compaction hides 85–99% of compaction time in our runs (APEC 85% and 95% on GPT-5.6 Sol and GPT-5.4; Async-L 98–99%; Tables 3–4).

It answers a different question (how fast one summary can be written) and could be used inside APEC.

(2) TokenPilot (Xu et al., 2026). It does not summarize; it truncates and evicts context. We do not evaluate it for three reasons.

  • It needs an extra, weaker model for a hard decision. Eviction decisions come from a separate LLM (Qwen3.5-35B-A3B) deployed next to the agent model (§3.3, A.4). Deciding which parts of the history will not be needed later is essentially as hard as writing a good summary: both require knowing what matters for the rest of the task. Handing it to a smaller model risks dropping what the agent needs. To keep quality, this judgment should come from the same model that does the work, which is what APEC (and production harnesses) do: the acting model writes its own summaries, with no extra model.

  • Its benchmarks are not agentic coding. PinchBench has 14 coding tasks out of 123 and Claw-Eval 5 terminal tasks out of 161 (Table 8), scored by task checks and rubrics rather than repository tests. Its “continuous” sessions are streams of unrelated tasks, not one growing repository (A.1).

  • Its tool-output filter is hand-written rules. Which outputs are truncated and how much is set by fixed, per-tool thresholds and preview lengths (A.4). These are tuned for its harness and tools and do not obviously carry over to other harnesses or workloads.

It changes what is kept, not when or how summarization runs; our Prune-H/L baselines cover rule-based cleanup in simple form.

(3) Observation masking (Lindenbauer et al., 2025). It replaces tool outputs older than the last 10 turns with a placeholder and keeps the agent’s reasoning and actions (§3.1). We do not evaluate it for three reasons.

  • It does not bound context growth. The authors note that masking alone does not stop the context from growing (§3.1): reasoning and actions are kept forever. In our sessions, where new requirements keep arriving, it cannot replace compaction; their own best variant adds LLM summarization on top (§5.3).

  • It breaks prefix caching at every step. Each turn, the observation that falls out of the window is replaced in the middle of the prompt, so everything after it must be recomputed. The authors mention the worse cache behavior of masking (§D.4), but their cost model does not fully price caching (App. A).

  • Its workload is short, single-turn SWE-bench. Each run is one issue with one user prompt, about 44–52 turns on average (§4.4), and contexts are not shown to exceed the window. Its window size also had to be retuned per harness (10 turns in SWE-agent, 58 in OpenHands; §5.1).

It is a compression method that could feed into APEC, not an alternative way to schedule compaction.

(4) No compaction. CompactionBench sessions are built to outgrow the context: even at the 90% trigger, trials compact 1.1 times on average (Table 3), and several tasks trigger compaction even with a 1M window (Section 5.3). Without compaction, those sessions would hit the context limit and stop, so no-compaction is not a working configuration for them.

(5) Fresh session per turn. Resetting the conversation before each requirement turns one long session into independent single-turn tasks; the agent keeps only the repository, not any of its earlier reasoning. This removes exactly the long-horizon setting we want to study, and production sessions do not work this way. Our goal is to evaluate compaction itself, not how much context a model can handle: even as models get longer context windows, long sessions still outgrow them, so compaction remains necessary.

More harnesses, models, and trigger points. We now report Codex CLI results (see our reply to Reviewer E6wF, W1), and we have ran evaluation on a broader model families, including GPT-5.6-sol, GPT-5.4, Opus-5.5, Sonnet-5.5, Haiku-5.5 (see our reply to Reviewer E6wF, W2). Trigger points: Table 3 runs the synchronous baselines at both 90% and 50%, and the Table 2 benchmarks now also have 50% runs (W1 above).

In summary, each suggested baseline targets a different problem (how text is compressed or filtered) or a different workload, and none addresses when compaction runs or how summaries are placed in a cached prefix. None of them was evaluated end to end on multi-turn coding sessions with repeated compaction, or shown to improve on synchronous compaction in that setting. Our choice of baselines and the reasons for it are already stated in Appendix C.3; we respectfully ask the reviewer to assess our baselines against that rationale, rather than against a list of methods designed for other settings.

We have added a discussion section on how APEC/PEC can work with these approaches; and what would be the integration work needed, which we leave to future work.

Reviewer_E6wFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies context compaction in coding agents from the systems side. It analyzes a large set of GitHub Copilot agent sessions and finds that the few sessions that trigger compaction use a large share of all tokens, and that synchronous compaction adds foreground delay and destroys prefix-cache reuse. Since existing coding benchmarks rarely trigger compaction, the authors build CompactionBench by turning release changelogs of SWE-EVO repositories into persistent multi-turn sessions, where each turn asks the agent to implement one more requirement in the same repository. They then propose APEC, which summarizes finished history segments in the background, appends the summaries to an append-only prefix so the KV cache stays valid, and keeps recent messages raw. On CompactionBench with GPT models in one harness, APEC is reported to improve task accuracy while lowering cost and time compared with synchronous compaction.

Critical strengths
  1. The production analysis is the most valuable part of the paper. It shows at large scale that compaction is rare but expensive, that it often wipes most of the prefix cache, and that it takes real share of turn time. This turns compaction from a prompt-design question into a systems cost question, and releasing the traces would help the community.
  2. Building multi-turn sessions from release history is clean way to sustain context growth with executable outcomes. Each request is fixed in advance, so the workload does not depend on the agent trajectory, and the authors remove git history so the agent cannot copy the target release.
Critical weaknesses
  1. The asynchronous part, which is the headline of the method, contributes little. Running normal compaction asynchronously does not improve accuracy over the synchronous version. The progressive variant without asynchrony already gets almost all of the accuracy gain and the lowest cost, and adding asynchrony saves a few minutes but costs more. So the gain comes from keeping several incremental summaries instead of one full rewrite. Summarizing history incrementally is already common practice in memory systems for long-running agents, such as MemGPT. The paper also does not explain why a compaction schedule should raise task accuracy at all.
  2. The evaluation is too small to support the accuracy claim. CompactionBench has eight tasks from four repositories, and the experiments use one harness and only GPT models. The accuracy gaps between methods are a few points, with no variance reported over the five trials. At the same time, changing only the summary cap and tail budget of APEC moves accuracy by more than fifteen points, which is larger than the gap between APEC and the baselines. So it is hard to tell where the gain comes from, and the paper does not explain why changing these two budgets should move accuracy so much.
Questions for authors
  1. Why should asynchronous compaction change task accuracy? Please separate the effect of progressive summaries from the effect of asynchrony, and report the variance over trials for each method.
  2. Codex CLI is open source and uses the same GPT models. How does APEC compare with the native compaction of Codex on the same tasks? The synchronous baselines are reimplemented inside OpenClaude and may be weaker than the production version.
  3. The background summary is written from a snapshot, while the agent keeps changing the repository. When the summary is applied, it may describe code that later turns have already changed. How often does this happen, and does it lead to wrong actions?
5 comments

Thanks for your review, here is a point-by-point reply.

W1. The asynchronous part adds little; PEC already gets most of the benefit.

Both PEC and APEC are part of our design. It is correct that progressive summaries bring most of the accuracy and cost gain, while asynchrony has a different job / objective: it saves time. In our proposed solution, progressive summarization (PEC) captures token cost and accuracy gains, while asynchrony (APEC) specifically removes blocking latency off the critical path. In APEC, the summary is written in the background while the agent keeps working, so the agent does not wait. This makes tasks about 7% faster than PEC on GPT-5.6 Sol, and more than twice as fast as Sync-H on GPT-5.4 (Table 4).

  • Asynchrony does not hurt. APEC and PEC have similar accuracy (77.95% vs. 76.74%, std. error 1.62 and 3.55) and similar cost, and the differences are within error.

  • We also ported APEC to Codex CLI and ran it on the same eight tasks with GPT-5.6 Sol, against Codex’s own compaction: its default blocking compaction at 90% (Sync-H) and its built-in background compaction at 50% (Async-L). Same model, tools, prompts, and evaluator; only the compaction setting differs. Results so far (3 trials per task, 72 trials). APEC gives the highest accuracy and the shortest time in Codex too, at lower cost than Codex’s default. This matches what we see in OpenClaude.

Codex CLI, GPT-5.6 SolAccuracy (%)Cost/trial ($)Time/trial (min)
Codex default (Sync-H)73.7914.8420.9
Codex background (Async-L)69.0111.2417.8
APEC77.4912.0816.6
  • We are integrating PEC/APEC to our production agentic platform and model training. OpenAI has also recently added asynchronous compaction to Codex (https://github.com/openai/codex/pull/53800), concurrent with our work. This suggests that the time saved by running compaction in the background matters in production.

  • On MemGPT(https://arxiv.org/abs/2310.08560): MemGPT is incremental but not prefix-preserving. On a queue flush, it regenerates a recursive summary from the existing summary and the evicted messages, and places it at the head of the FIFO queue, after a writable working-context block (MemGPT, §2.1–2.2). Each flush thus rewrites the prompt right after the system instructions, invalidating the prefix cache. APEC is append-only: earlier summary blocks are never modified, so [system prompt, S1,…,SkS_1,\dots,S_k] stays a cache hit across compactions (31% vs. 7–9% post-compaction cache hit rate for prefix-rewriting compaction; Figure 5).

W1 (cont.). Why should a compaction schedule raise accuracy at all?

The schedule decides how much past work stays in the context. With a full rewrite (Sync-L), every compaction squeezes the whole history, including the earlier summaries, into one new summary, so details from early turns fade out over time. With progressive compaction (PEC/APEC), each summary covers only a new part of the history, and earlier summaries are kept unchanged.

We measured this in our traces. We take the code names (files, functions, APIs) mentioned in the first summary of each trial and check how many still appear in the context after later compactions. Under Sync-L, 55% survive the 2nd compaction, 34% the 4th, and 25% the 5th or later. Under PEC and APEC, all of them survive, because the first summary is never rewritten.

Example (Dask 2023.3.2→2023.4.0; raw summary text, “[...]” marks omissions). Requirement 1 adds config commands to dask/cli.py.

Sync-L, summary #1:

3. Files and Code Sections:
   - `dask/cli.py`
     - Importance:
       - Defines the built-in `dask` Click CLI.
       - Requirement 1 added configuration inspection commands.
     [...]
       - Added a `config` command group:
         @cli.group(name="config")
         def config_cli():

Sync-L, summary #5 (turn 39); dask/cli.py no longer appears anywhere in the summary:

3. Files and Code Sections:
   - `dask/config.py`
     - Requirement 28 changed `refresh()` so newer defaults override older defaults:
     [...]

PEC, same task, context at the final compaction (turn 38); the first summary block is kept as is:

3. Files and Code Sections:
   A. `dask/cli.py`
   - Modified for requirement 1.
   - Added YAML import and aliased Dask config import:
   [...]
   - Added a `config` command group:
     @cli.group()
     def config():

W2. The evaluation is small: one harness, GPT models only.

(1) Harness. We ported APEC to Codex CLI and evaluated it on the same tasks; see the Codex CLI table in our reply to W1. We choose OpenClaude in evaluation because it is open source and representative of major coding agent harnesses.

(2) Broader Models. We have expanded our evaluation to include a comprehensive model sensitivity study across frontier models from both OpenAI and Anthropic, specifically evaluating GPT-5.6-sol, GPT-5.4, Opus 5.5, Sonnet 5.5, and Haiku 5.5, given now we have more budget. As detailed below, as well as in our updated evaluation section (Table 4), the efficiency gains and accuracy improvements of APEC consistently hold across diverse model families: e.g., APEC improves task accuracy by 3.68--4.57 percentage points over Sync-H, while reducing input tokens by 25.7--40.3%, completion time by 4.7--57.4%, and total cost by 1.0--46.2%.

ModelMethodAcc. (%)Num. Comps.Input (M)Cached input (M)Output (K)Total time (min)Compaction time (s)Overlap (%)Total cost ($)
GPT-5.4Sync-H56.702.4442.70640.164184.67144.47471.470.0041.06
Async-L52.577.4435.40033.024302.54117.13710.8698.1542.32
APEC60.886.1131.72928.337196.6861.53617.3594.7840.64
Opus 5.5Sync-H76.111.9338.53737.79769.9830.2178.540.0015.33
Async-L77.005.0824.54123.53478.9729.73189.2398.6915.28
APEC80.684.7324.27823.17667.7728.56190.7484.5412.58
Sonnet 5.5Sync-H75.883.2059.68858.606107.5440.77103.950.0011.88
Async-L74.718.0736.04934.583121.3639.95239.7598.9111.54
APEC79.567.5835.64234.026104.1438.43243.5287.469.41
Haiku 5.5Sync-H68.832.1743.89943.07895.6627.6739.850.002.79
Async-L67.176.1727.10325.962107.9527.24103.3999.301.91
APEC72.705.7126.73325.47792.6426.37103.3891.921.50

(3) Variance. Standard errors over trials for every method (accuracy, cost, and time).

MethodAccuracy (%)Total cost ($)Completion time (min)
Sync-H69.29 ± 4.7421.93 ± 0.5629.25 ± 0.80
Sync-L73.04 ± 4.0018.54 ± 0.6030.77 ± 1.14
Async-L70.70 ± 5.5818.30 ± 0.5628.99 ± 0.86
Prune-H72.40 ± 3.7020.80 ± 0.6125.55 ± 0.76
Prune-L71.52 ± 5.4420.22 ± 0.6633.24 ± 1.19
PEC76.74 ± 3.5514.75 ± 0.4229.89 ± 0.82
APEC77.95 ± 1.6216.37 ± 0.6127.82 ± 0.75

We additionally report the variance across models being used for the newly added Anthropic models, to verify that the cross-model gains are statistically consistent and not driven by run-to-run noise:

ModelMethodAcc. (%)Total cost ($)Total time (min)
Opus 5.5Sync-H76.11 ± 3.4115.33 ± 0.2430.21 ± 0.58
Async-L77.00 ± 4.0115.28 ± 0.4229.73 ± 0.63
APEC80.68 ± 1.2312.58 ± 0.5928.56 ± 0.61
Sonnet 5.5Sync-H75.88 ± 3.3511.88 ± 0.1740.77 ± 0.82
Async-L74.71 ± 4.1211.54 ± 0.3139.95 ± 0.87
APEC79.56 ± 1.239.41 ± 0.4138.43 ± 0.84
Haiku 5.5Sync-H68.83 ± 3.422.79 ± 0.0427.67 ± 0.58
Async-L67.17 ± 4.051.91 ± 0.0427.24 ± 0.61
APEC72.70 ± 1.291.50 ± 0.0526.37 ± 0.60

APEC’s accuracy gains exceed the standard errors, while its run-to-run accuracy standard error drops by nearly 3×. By preserving immutable early summaries rather than repeatedly rewriting the full history, APEC avoids the trajectory-dependent information loss that causes high variance in full-rewrite baselines.

W2 (cont.). Why do the summary and tail budgets change accuracy by more than 15 points?

These two budgets shape what the agent works with after each compaction: how detailed the summaries are, how much recent raw history is kept, and, as a result, how often compaction fires. They therefore affect both the information the agent retains and the rhythm of the session. In addition, the sweep uses three tasks with a few trials each, and task outcomes are close to all-or-nothing (a single failing change can fail a whole test suite), which amplifies differences between settings. Importantly, most budget settings still outperform all baselines in Table~3, so the benefit of APEC does not hinge on a narrow choice of budgets. We will add more repeats and report variance per setting in the revision.

Q1. Why would asynchrony change accuracy? Please separate it from progressive summaries.

Asynchrony does change the context a little, so the two are not exactly equivalent (though the difference is minimal). While a summary is being written in the background, the agent keeps working; when the summary is installed, the messages produced in the meantime stay in the new raw tail, and the summary describes a slightly older snapshot (see Q3). But this window is short: each compaction takes about 54 seconds, 85% of which overlaps foreground work (Table 3), and the agent adds about 1K tokens per LLM turn. So the extra raw tail is small, and we do not expect asynchrony to change accuracy.

The results confirm this. Each pair below differs only in asynchrony:

  • PEC vs. APEC: 76.74% vs. 77.95% (std. error 3.55 and 1.62).

  • Sync-L vs. Async-L: 73.04% vs. 70.70% (std. error 4.00 and 5.58).

Both gaps are well within one standard error of the difference. Progressive vs. monolithic summaries is the separate comparison Sync-L vs. PEC.

What asynchrony does change is time, which is its purpose by design:

  • OpenClaude, GPT-5.6 Sol: 29.89 → 27.82 min (PEC → APEC); 30.77 → 28.99 min (Sync-L → Async-L).

  • OpenClaude, GPT-5.4: 144.47 min (Sync-H) vs. 61.53 min (APEC) (Table 4).

  • Codex CLI, GPT-5.6 Sol: 20.9 min (Codex default) vs. 16.6 min (APEC) (reply to W1).

Q2. How does APEC compare with Codex’s own compaction?

We ran this directly: see the Codex CLI table in our reply to W1. Inside Codex, APEC beats both Codex’s default compaction and Codex’s own background compaction in accuracy and time.

Our synchronous baseline also follows Codex’s compaction logic. The pseudocode in the paper (Algorithm 2, Synchronous Compaction) already shows this logic: when the context reaches the threshold, the model summarizes the history, and the new history is the summary plus the most recent messages that fit in a token budget. Below is the real code, side by side (Codex at commit 80a0c1d; ours from the OpenClaude version used in the paper; “[...]” marks omissions).

(1) Trigger: a fraction of the context window.

// Codex: codex-rs/protocol/src/openai_models.rs
let context_limit = self.resolved_context_window()
    .map(|context_window| (context_window * 9) / 10);
[...] config_limit.map_or(context_limit, |limit| std::cmp::min(limit, context_limit))

// Ours: src/services/compact/autoCompact.ts
const percentageThreshold = Math.floor(effectiveContextWindow * (envPercent / 100))
return Math.min(percentageThreshold, autocompactThreshold)

(2) Rebuilt history: summary plus the most recent messages, chosen by a backward token-budget walk.

// Codex: codex-rs/core/src/compact.rs  (build_compacted_history_with_limit)
for message in user_messages.iter().rev() {
    let tokens = approx_token_count(&message.message);
    if tokens <= remaining {
        selected_messages.push(message.clone());
        remaining = remaining.saturating_sub(tokens);
    } else { [...] break; }
}
[...] history.push(... CompactionSummary::new(summary_text) ...);

// Ours: src/services/compact/postCompactionHistory.ts  (recency-budget)
const budget = input.budgetTokens - input.summaryTokens
for (let i = input.messages.length - 1; i >= 0; i--) {
  const msgTokens = tokensOf([msg])
  if (keptTokens + msgTokens > budget) break
  kept.unshift(msg)
  keptTokens += msgTokens
}

Full code: https://anonymous.4open.science/r/APEC-A6D4.

Reviewer_mF2AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors tackle the challenging problem of handling long, growing context in coding agents. They focus on uncovering optimal use of context compaction in this setting, focusing on when and where its use is actually beneficial and when its use paradoxically adds more overhead. To uncover these insights, the authors analyze millions of Github copilot agent coding sessions from millions of users. Context compaction evaluation in general is enabled by their new benchmark, CompactionBench, which represents evolving software development as a multi-turn coding session. With this benchmark, they are able to evaluate the effectiveness and costs of repeated compaction. The authors also propose a new progressive compaction mechanism, APEC, that compacts segments incrementally in the background, moving summarization off the critical path and preserving reusable prefixes. Experiment results show APEC produces superior accuracy on CompactionBench with low computational cost.

Critical strengths

-They open source their compaction production traces and benchmarks, a great contribution to the research community.

-Paper is well structured and figures are clear.

-Experiment results are impressive - APEC dominates accuracy while being cheapest and among the fastest compared to baselines.

Critical weaknesses

None

Questions for authors

N/a

1 comment

Thank you very much for your encouraging feedback and positive evaluation of our work. We are glad that you found the paper well-structured, appreciated our open-sourced production traces and benchmarks, and noted the benefits of a practical solution (APEC and PEC).

Since you did not list any critical weaknesses or questions, we truly appreciate your supportive assessment. If you feel our paper and contributions warrant a higher evaluation, we would be very grateful if you would consider raising your review score.

Thank you again for your time and constructive engagement with our submission!

To provide a quick summary of the core contributions for your reference, our work spans:

  1. Production-Scale Characterization: Analyzing 13.5M GitHub Copilot sessions (760M LLM calls) to quantify how context compaction impacts system latency and KV-cache reuse.
  2. CompactionBench: The first benchmark specifically designed to model multi-turn software evolution for repeated compaction evaluation.
  3. APEC/PEC: A progressive compaction mechanism that can be done synchronously or asynchronously by moving summary preparation off the critical path while preserving prefix cache reuse to cut costs and latency
  4. Comprehensive frontier model coverage, including GPT-5.6-sol, GPT-5.4, Claude Opus 5.5, Sonnet 5.5, and Haiku 5.5.

Comments on the paper

6 comments
AuthorsOpenReview ↗

A Meta Message to All Reviewers (on privacy, a larger production analysis, design, and more model/harness coverage)

Thank you all for the constructive feedback. Here as a common message to answer the data privacy question and highlight a few revisions or clarifications.

  • Ethics and Data Privacy: Our production telemetry analysis contains zero prompt content, user code, or PII (relying solely on aggregated system metrics) and has passed rigorous legal and compliance reviews.
  • Production Trace Scale: For the camera-ready version, we have scaled our production analysis from 13.5M to 103.7M sessions (7.7x increase), proving the robustness of the findings and will include a trend analysis.
  • CompactionBench Design: Using predefined software repo changelog requirements ensures reproducible, controlled multi-turn evaluation without conflating trajectory drift with compaction efficiency.
  • PEC vs. APEC Role: Both are part of the design -- progressive summarization (PEC) captures token cost and accuracy gains, while asynchrony (APEC) specifically removes blocking latency off the critical path.
  • Impact: We open-source the compaction related production traces and the benchmarks to the community for further research on systems efficiency in context compaction. Both the benchmark and the mechanism (PEC/APEC) have been in the process of integrating with the production agent platform and model training.

We are updating the paper (with the additional one-page granted for revision), and providing detailed comments to each reviewer below.

Update (11/13/2026)

  • Model Family Generalizability: We have expanded our evaluation to a comprehensive model sensitivity study across frontier architectures, now including GPT-5.6-sol, GPT-5.4, Opus 5.5 (new), Sonnet 5.5 (new), and Haiku 5.5 (new). Across all tested models, PEC/APEC consistently outperforms baselines.
  • Harnesses: We have expanded the evaluation to Codex CLI, showing consistent improvements from PEC/APEC.
Area_Chair_CrDXOpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.