- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the behavior and system-level costs of context compaction in coding agents on long-horizon task. Through large-scale trace analysis, the authors identify substantial latency and cache-reuse overheads associated with conventional synchronous compaction. To support systematic evaluation, this paper introduces CompactionBench, a benchmark that models persistent software development through sequential, interdependent repository-level requirements. They further propose APEC, an asynchronous and progressive compaction mechanism that incrementally maintains reusable context prefixes. Experiments across frontier models demonstrate improvements in downstream coding accuracy, inference cost, and execution time compared with conventional compaction strategies.
Critical strengths
-
Insights from real-world production traces. The paper analyzes a large-scale collection of real-world coding agent trajectories, providing useful empirical insights into the frequency, triggering behavior, and efficiency costs of context compaction in production environments.
-
A benchmark for evaluating context compaction. The paper introduces CompactionBench, a new benchmark designed to evaluate context compaction in long-running coding agents. By constructing multi-turn software development tasks with sustained context growth, it provides a useful setting for studying repeated compaction.
-
A new compaction method with empirical validation. The paper proposes APEC, an asynchronous and progressive context compaction mechanism. Experimental results demonstrate its effectiveness in improving task accuracy while reducing inference costs and execution time under the evaluated settings.
Critical weaknesses
- Potential task–state mismatch in CompactionBench. CompactionBench constructs multi-turn coding tasks from predefined sequences of changelog requirements, without adapting subsequent requests to the agent's actual development progress. However, the repository state evolves dynamically and may deviate substantially from the expected state when earlier requirements are only partially completed or implemented incorrectly. Consequently, later requests may rely on functionality that has not been implemented, creating a mismatch between the predefined instructions and the actual repository state. The paper does not examine how frequently such mismatches occur or how they affect task validity and the resulting evaluation outcomes.
- Limited evaluation across models and agent harnesses. The experiments primarily rely on OpenClaude with GPT-5.6 Sol, with additional evaluation limited to GPT-5.4. Although APEC is described as harness-agnostic, its effectiveness is not evaluated under other coding agent harnesses or more diverse model families. This limits the evidence for the robustness and generalizability of the reported improvements.
- Insufficient discussion of data privacy and user consent. The paper analyzes a large-scale collection of real-world GitHub Copilot trajectories involving a large amount of users. Although the authors clarify that the collected telemetry excludes prompt content and mention privacy review before data release, the paper provides limited details on user consent or data-use authorization, as well as the privacy safeguards applied during data collection and analysis. Further clarification would strengthen the ethical transparency of the study.
- Limited coverage of long-horizon coding scenarios. While repository evolution is a representative long-horizon coding scenario that can trigger repeated context compaction, it is not the only one. Other scenarios, such as developing large-scale applications from scratch or iteratively refining code through user feedback and error correction, may also require frequent compaction. However, neither CompactionBench nor the evaluation of APEC covers these scenarios, limiting the generalizability of the experimental findings.
Questions for authors
-
Beyond LLM-based summarization, coding agents may employ rule-based context management strategies, such as truncating or removing lengthy tool outputs. Were such mechanisms enabled in the OpenClaude harness during evaluation? Although the paper includes pruning baselines, it remains unclear how these non-LLM strategies interact with APEC. Could the authors clarify whether combining them would affect compaction frequency, task accuracy, and overall efficiency?
-
Given that CompactionBench uses predefined multi-turn user requests, did the authors examine whether subsequent instructions remain consistent with the repository states actually produced by the agent? Earlier development failures may cause the repository to deviate from the expected state, potentially making later instructions inapplicable or inconsistent with the current codebase. Such mismatches could affect the validity of the constructed tasks and their evaluation outcomes.
-
What motivates the choice of a 50% soft threshold for APEC? While the paper discusses the trade-off between early compaction and background overhead, it does not provide an empirical justification for this specific value. Additional justification or ablation studies on core parameters would strengthen the rationale for the settings and provide further evidence of APEC's robustness.
6 comments
W2. Limited evaluation across models and harnesses.
Models. We have expanded our evaluation to include a comprehensive model sensitivity study across frontier models from both OpenAI and Anthropic, specifically evaluating GPT-5.6-sol, GPT-5.4, Opus 5.5, Sonnet 5.5, and Haiku 5.5, given now we have more budget. As detailed below, as well as in our updated evaluation section (Table 4), the efficiency gains and accuracy improvements of APEC consistently hold across diverse model families: e.g., APEC improves task accuracy by 3.68--4.57 percentage points over Sync-H, while reducing input tokens by 25.7--40.3%, completion time by 4.7--57.4%, and total cost by 1.0--46.2%.
| Model | Method | Acc. (%) | Num. Comps. | Input (M) | Cached input (M) | Output (K) | Total time (min) | Compaction time (s) | Overlap (%) | Total cost ($) |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.4 | Sync-H | 56.70 | 2.44 | 42.706 | 40.164 | 184.67 | 144.47 | 471.47 | 0.00 | 41.06 |
| Async-L | 52.57 | 7.44 | 35.400 | 33.024 | 302.54 | 117.13 | 710.86 | 98.15 | 42.32 | |
| APEC | 60.88 | 6.11 | 31.729 | 28.337 | 196.68 | 61.53 | 617.35 | 94.78 | 40.64 | |
| Opus 5.5 | Sync-H | 76.11 | 1.93 | 38.537 | 37.797 | 69.98 | 30.21 | 78.54 | 0.00 | 15.33 |
| Async-L | 77.00 | 5.08 | 24.541 | 23.534 | 78.97 | 29.73 | 189.23 | 98.69 | 15.28 | |
| APEC | 80.68 | 4.73 | 24.278 | 23.176 | 67.77 | 28.56 | 190.74 | 84.54 | 12.58 | |
| Sonnet 5.5 | Sync-H | 75.88 | 3.20 | 59.688 | 58.606 | 107.54 | 40.77 | 103.95 | 0.00 | 11.88 |
| Async-L | 74.71 | 8.07 | 36.049 | 34.583 | 121.36 | 39.95 | 239.75 | 98.91 | 11.54 | |
| APEC | 79.56 | 7.58 | 35.642 | 34.026 | 104.14 | 38.43 | 243.52 | 87.46 | 9.41 | |
| Haiku 5.5 | Sync-H | 68.83 | 2.17 | 43.899 | 43.078 | 95.66 | 27.67 | 39.85 | 0.00 | 2.79 |
| Async-L | 67.17 | 6.17 | 27.103 | 25.962 | 107.95 | 27.24 | 103.39 | 99.30 | 1.91 | |
| APEC | 72.70 | 5.71 | 26.733 | 25.477 | 92.64 | 26.37 | 103.38 | 91.92 | 1.50 |
Harnesses We now report results with the Codex CLI harness. We ported APEC to Codex CLI and ran it on the same eight tasks with GPT-5.6 Sol, against Codex’s own compaction: its default blocking compaction at 90% (Sync-H) and its built-in background compaction at 50% (Async-L). Same model, tools, prompts, and evaluator; only the compaction setting differs. Results so far (3 trials per task, 72 trials):
| Codex CLI, GPT-5.6 Sol | Accuracy (%) | Cost/trial ($) | Time/trial (min) |
|---|---|---|---|
| Codex default (Sync-H) | 73.79 | 14.84 | 20.9 |
| Codex background (Async-L) | 69.01 | 11.24 | 17.8 |
| APEC | 77.49 | 12.08 | 16.6 |
APEC gives the highest accuracy and the shortest time in Codex too, at lower cost than Codex’s default. This matches what we see in OpenClaude. Repeats 4–5 and the Sync-L and PEC arms are still running.
Run-to-run Robustness We also report standard errors across trials for the Anthropic model family to demonstrate statistical robustness:
| Model | Method | Acc. (%) | Total cost ($) | Total time (min) |
|---|---|---|---|---|
| Opus 5.5 | Sync-H | 76.11 ± 3.41 | 15.33 ± 0.24 | 30.21 ± 0.58 |
| Async-L | 77.00 ± 4.01 | 15.28 ± 0.42 | 29.73 ± 0.63 | |
| APEC | 80.68 ± 1.23 | 12.58 ± 0.59 | 28.56 ± 0.61 | |
| Sonnet 5.5 | Sync-H | 75.88 ± 3.35 | 11.88 ± 0.17 | 40.77 ± 0.82 |
| Async-L | 74.71 ± 4.12 | 11.54 ± 0.31 | 39.95 ± 0.87 | |
| APEC | 79.56 ± 1.23 | 9.41 ± 0.41 | 38.43 ± 0.84 | |
| Haiku 5.5 | Sync-H | 68.83 ± 3.42 | 2.79 ± 0.04 | 27.67 ± 0.58 |
| Async-L | 67.17 ± 4.05 | 1.91 ± 0.04 | 27.24 ± 0.61 | |
| APEC | 72.70 ± 1.29 | 1.50 ± 0.05 | 26.37 ± 0.60 |
Notably, APEC reduces accuracy standard error by ~3× compared to both Sync-H and Async-L across all three models (±1.23–1.29% vs. ±3.35–4.12%), confirming that progressive, append-only compaction yields substantially more reliable and predictable execution across diverse model sizes and families.
W3. Data privacy and user consent.
We thank the reviewer for raising this important point regarding ethical transparency. We respectfully note that this concern is already addressed in the Ethics Statement of the submission and the meta message, and we kindly refer the reviewer to it. To summarize the safeguards in place:
-
Institutional review. The collection and use of this data for the present study have passed rigorous legal and ethics/privacy review.
-
Data minimization. The production analysis relies exclusively on aggregated operational metrics, namely token counts, latency, cache usage, and tool-call statistics. The telemetry available to us contains no prompt content, code, or other user-authored text.
-
Purpose limitation. The analysis characterizes system-level behavior (compaction frequency, cost, latency, and cache reuse) and does not examine the content of user interactions or individual users.
-
Responsible release. As stated in the Reproducibility Statement, any release of production telemetry will follow further redaction and privacy review prior to publication.
We believe these measures address the reviewer’s concern, and we will make them more prominent in the revision.
W1 / Q2. Potential task–state mismatch.
The predefined request sequence is an intentional design choice to ensure reproducible and fair comparisons across compaction policies. Each release transition is treated as a single long-horizon task, where all policies receive identical requirements while the repository state evolves according to the agent’s actual execution. Failures in earlier requirements may affect subsequent development, but this is a natural challenge in long-horizon software engineering rather than an inherent benchmark flaw. More importantly, dynamically adapting requests to each agent’s progress would result in different target specifications, making direct accuracy comparisons against a common ground truth infeasible. Our fixed sequence preserves a shared release-level objective, allowing all policies to be evaluated against the same executable tests.
Beyond this design rationale, the benchmark incorporates several concrete mechanisms that mitigate task–state mismatch and preserve the validity of the evaluation:
State-aware requests. Each request introduces a new feature while asking the agent to keep its changes consistent with, and not break, the previously implemented requirements.
Grading on the cumulative release. The final patch is graded once against the release’s tests, so a missed earlier requirement only costs the tests for that functionality; it does not invalidate the task.
Paired comparison. All policies receive the identical requests, repository, model, and evaluator, so any remaining mismatch comes from the agent’s own execution under each policy, which is what we aim to measure.
Ecological validity. Production users likewise issue follow-up requests based on intended rather than verified repository state; handling such discrepancies is an inherent part of long-horizon agentic software engineering that a realistic benchmark should retain rather than remove.
CompactionBench has been used by production agentic platform design and model training. As in our anonymous repo link (https://anonymous.4open.science/r/APEC-A6D4). We will also open source it to the research community for further research in systems efficiency of agentic context compaction and management.