acceptodds
Under review as a conference paper at ICLR 2027

CliffCompaction: Cost-Efficient Compaction for Long-Horizon Coding Agents

13 views

Abstract

Agents often work on complex problems that require millions of tokens of context, necessitating compaction across sessions with limited context windows. We develop CliffCompaction, an autocompaction technique that reduces cost by up to 50% under a bounded context while maintaining or improving performance on tasks such as Terminal-Bench. Its per-rollout savings can be reinvested in test-time scaling, improving Terminal-Bench by over 10 points for under the cost of two full-context runs, letting Kimi K2.6 match Opus 4.7 and exceed Opus 4.6 and GPT-5.3 Codex at lower cost. On KernelBench, CliffCompaction achieves state-of-the-art CUDA kernel speedups ( at 200 steps, at 400) across sessions exceeding a million tokens, surpassing trained agents and specialized search methods. The key to CliffCompaction is preserving context fidelity: (1) it only truncates or drops content, never rewrites it, and (2) never re-compacts a compaction, preventing context drift. We open-source a scaffold-agnostic API-proxy implementation usable with Claude Code, Codex, and other harnesses.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, context compaction, long-horizon agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qXDuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CliffCompaction, a training-free context-management method for long-horizon coding agents. It periodically replaces older interactions with short verbatim fragments, retains the system prompt, initial task, and recent turns, and discards the previous compaction block when constructing the next one. The method targets both context growth and the cost of repeatedly processing cached input. Experiments cover SWE-bench Verified, Terminal-Bench, and KernelBench across several models and scaffolds.

Critical strengths
  1. The paper addresses an important practical problem with a simple, deployable method. The distinction between reducing context length and preserving cache reuse is useful: frequent edits can undermine the savings from removing tokens. An implementation that requires neither model training nor auxiliary summarization calls is attractive for existing coding-agent systems.

  2. The empirical coverage is reasonably broad. The study includes multiple models, several scaffolds, and both repository repair and much longer kernel-optimization trajectories.

Critical weaknesses
  1. Comparisons with closely related context-management methods are incomplete. The evaluation includes sliding windows, summarization, and microcompaction in Table 5, but omits several relevant approaches. For example, The Complexity Trap[1], SWE-Pruner[2] and CoACT[3]. These works are neither discussed nor evaluated, leaving the advantages of CliffCompaction's fixed truncation rules over existing masking and task-aware compression methods unclear.

  2. The method lacks ablations of its key design choices. In particular, the paper does not isolate the effect of discarding the previous compaction block at every new compaction. It remains unclear whether this improves performance by avoiding accumulated distortion or whether retaining selected older information would better preserve long-range dependencies. A controlled comparison of retaining versus discarding earlier compacted history, under the same trigger and retained-context budget, would directly test this central choice. Additional ablations should separate the effects of truncating tool calls and reasoning, dropping long tool outputs, and retaining recent turns, together with sensitivity to their length thresholds.

  3. The current results do not clearly establish an efficiency advantage over competing methods. In Table 5, the advantage over summarization is modest: CliffCompaction achieves 3.58× speedup at $8.32 per task, compared with 3.47× at $8.10, and the SWE-bench results show no consistent advantage in both quality and cost. Moreover, the efficiency comparison between methods focuses primarily on cost. It lacks a systematic comparison of actual trajectory length in tokens, end-to-end execution time, and interaction rounds across competing methods under the same setup. Reporting these metrics at comparable solution quality would clarify whether the cost savings translate into more efficient task execution.

  4. Cache friendliness is emphasized in the Introduction and Section 2.1, but the main-text experiments do not directly validate this claimed advantage. Reporting measured cache-hit rates, cache invalidations, and re-prefill overhead under matched settings would help establish whether CliffCompaction better preserves cache reuse and distinguish this benefit from savings due to shorter contexts.

My concerns primarily relate to the experimental evaluation: comparisons with relevant prior methods, ablations of the central design choices, and direct evidence for the efficiency and cache-reuse advantages remain insufficient. I would be very willing to raise my score if the authors provide the experimental evidence discussed above and demonstrate that the benefits hold under controlled comparisons.

[1] Lindenbauer, Tobias, et al. "The complexity trap: Simple observation masking is as efficient as LLM summarization for agent context management." arXiv preprint arXiv:2508.21433 (2025). [2] Wang, Yuhang, et al. "Swe-pruner: Self-adaptive context pruning for coding agents." arXiv preprint arXiv:2601.16746 (2026). [3] Chen, Haorui, et al. "CoACT: Action-Preserving Observation Compression for Coding Agents." arXiv preprint arXiv:2607.02911 (2026).

Questions for authors
  1. Could you clarify the scope of the recoverability claim in Section 2.2? Appendix C retains only the first 150 characters of some commands, which may be insufficient to repeat the original call, and re-reading a modified file does not recover its earlier contents or transient observations. How does the method handle information that cannot be retrieved from the current workspace, including user constraints introduced after the initial task description?
6 comments
AuthorsOpenReview ↗

(1/2) Addressing related methods and design ablations (with new analyses)

Q1: Comparisons with closely related context-management methods are incomplete: The Complexity Trap, SWE-Pruner and CoACT

A1: We will add these works to our related work and clarify their relationship to our evaluated baselines. Below is further discussion on these methods:

For The Complexity Trap, our Claude Code baseline in Section 5 already evaluates the same combination: it masks older tool observations, then summarizes if the context remains above threshold.

For SWE-Pruner and CoACT, we replayed both with their released models on a 749K-token KernelBench trajectory:

Peak contextAccumulated input tokensCost
No compaction749K154.3M (5.5×)$31.37 (4.2×)
CoACT579K120.6M (4.3×)$24.83 (3.3×)
SWE-Pruner492K103.9M (3.7×)$21.61 (2.9×)
Delete every tool observation408K88.0M (3.1×)$18.52 (2.5×)
CliffCompaction135K28.3M$7.52

(1) They cost about 3× more than CliffCompaction. (2) They are per-observation compression methods (compressing each new tool result), so their context is unbounded: both still exceed Kimi K2.6/K2.7's 256K window, and are therefore not standalone alternatives to CliffCompaction and the other trajectory-level compaction baselines in the paper.

Q2: [...] leaving the advantages of CliffCompaction's fixed truncation rules over existing masking and task-aware compression methods unclear.

A2: Masking alone does not bound the context: even deleting every tool observation still reaches 408K tokens (A1), so The Complexity Trap and Claude Code (Section 5, Table 5) combine it with LLM summarization. CliffCompaction matches this on SWE-bench Verified and outperforms it on KernelBench at 35% lower cost (3.58× vs. 3.33×, 8.32 vs. 12.84 USD).

Task-aware compression (SWE-Pruner, CoACT) requires a trained compressor model served alongside the agent. CliffCompaction is training-free and drop-in. Furthermore, CliffCompaction is implicitly task-aware: the agent itself decides what stays in context by re-reading the dropped content its current goal needs (Section 2.2).

Q3: The method lacks ablations of its key design choices. [...]discarding the previous compaction block at every new compaction [..]retaining versus discarding earlier compacted history, under the same trigger and retained-context budget, would directly test this central choice.

A3: We ran this ablation and will add it to the revision.

SWE-bench Verified (Kimi K2.5, mini-swe-agent, 16K)

Resolve rateCompacted history (last compaction)
Keep previous69.13 ± 0.833.7K
Drop previous69.53 ± 1.332.2K

KernelBench (Kimi K2.7, OpenHands, 128K, replay)

Compacted history at 10th compactionCompactions per trajectoryRe-prefill tokens / cost per trajectory
Keep previous37.4K12.3458.2K / $0.44
Drop previous3.9K9.741.5K / $0.13

Overall summary of findings for dropping vs keeping previous blocks: on SWE-bench we find that resolve rates are the same with/without discarding, but compacted history differs by 1.7×. On KernelBench, compacted history increases by up to 10×, and re-prefill costs increase by 3.4×. As such, we drop earlier blocks in CliffCompaction.

Q4: Additional ablations [...] truncating tool calls and reasoning, dropping long tool outputs, and retaining recent turns, [...] length thresholds.

A4:

  • Thought truncation is to prevent extreme cases that increase costs substantially. Only ~6% of thought traces are longer than 300 characters, which informed this choice.

  • Tool call truncation: same reasoning as with thoughts. This is to prevent the effect of extremely long tool calls (mostly file writes and inline Python).

  • Recent turns: We follow common practice in coding agents' compaction used in Claude Code compaction, OpenHands' summarizing condenser, and others. Other papers, for example The Complexity Trap (Appendix D.1 and D.2), focus in more detail on these ablations than we are able to do here.

  • Tool output dropping: Tool outputs make up 56% of context and compression is essential. Since tool signatures keep the file path, the agent can re-read outputs saved in the repository. With this, tool output dropping has little effect on quality of the compaction.

  • Length thresholds: quality and savings are not sensitive to the exact values. We use the same thresholds across all models, scaffolds and benchmarks, even though thought lengths, tool-call formats and observation sizes vary between them, and CliffCompaction still matches or stays close to full context everywhere. Stricter thresholds change savings by less than 5% since savings are superlinear with respect to context length.

AuthorsOpenReview ↗

(2/2) Addressing efficiency, caching, and recoverability

Q5: In Table 5, the advantage over summarization is modest [...]

A5: On KernelBench, the results may seem like modest improvements, but other papers with specialized techniques make similar improvements in terms of speedups. For example, specialized/trained methods such as AdaExplore [1] and CUDA Agent [2] gain +0.22–0.24× over their baselines, compared with +0.30× for CliffCompaction over summarization after 100 kernels (Figure 7b). Additionally, CliffCompaction has other benefits. Summarization has a poor walltime since output token generation is expensive. On KernelBench, summarization takes 10.7 more hours across the 50 problems: 7.2h generating summaries and 3.5h compiling and testing its extra kernels (136 vs. 116 per problem, Table 5). CliffCompaction compacts in milliseconds resulting in much faster end-to-end solution times.

On SWE-bench Verified (Table 5) and Terminal-Bench (Table 2), CliffCompaction outperforms summarization in every setting, significantly in three (paired permutation test, p < 0.05).

[1] Adaexplore: Failure-driven adaptation and diversity-preserving search for efficient kernel generation. https://arxiv.org/abs/2604.16625v1

[2] Cuda agent: Large-scale agentic rl for high-performance cuda kernel generation. https://arxiv.org/abs/2602.24286

Q6: Moreover, the efficiency comparison between methods focuses primarily on cost. It lacks a systematic comparison of actual trajectory length in tokens, end-to-end execution time, and interaction rounds across competing methods under the same setup.

A6: We report runtime in Appendix A.2. For example, CliffCompaction reduces per-instance wall-clock by 23–34% on SWE-bench Verified (Table 9), and with a self-hosted vLLM we see speedups of 1.60–2.62× (Table 10). We do not use total wall-clock as the primary comparison between methods due to confounding factors (provider load, peak hours).

We focus on end-to-end cost because it is the primary practical objective. Other factors, like the total number of tokens in a trajectory, are a weak proxy for cost, since about 80% of the cost comes from input tokens. Interaction rounds are capped at the same 400 steps on KernelBench (Table 5), and on SWE-bench Verified all methods add fewer than 5 steps over full context (Figure 6).

Q7: Cache friendliness is emphasized but the main-text experiments do not directly validate this claimed advantage. Reporting measured cache-hit rates, cache invalidations, and re-prefill overhead.

A7: With common caching such as implemented in vLLM, >99% of CliffCompaction's previously sent context is read from cache, because its context is append-only between compactions. This is a main reason why CliffCompaction is so cheap. Re-prefill is minimal: it happens only ~10 times per KernelBench trajectory and every other call reads its context (up to ~128K tokens) from cache. We discuss re-prefill overhead in Figure 4 and Appendix A.3, and will add this discussion to the main text.

Q8: Could you clarify the scope of the recoverability claim in Section 2.2?

A8: The claim covers content that persists in the workspace: source files and files the agent has written can be recovered by re-reading them, and in practice this is what agents do after compaction (Appendix D.1). Content that no longer exists in the workspace, such as earlier versions of modified files, is not recoverable (A10).

Q9: Appendix C retains only the first 150 characters of some commands, which may be insufficient to repeat the original call.

A9: On SWE-bench Verified, 80.1% of tool calls fit in 150 characters, and 90% of longer calls are file writes or inline Python scripts. Written files can be recovered by re-reading, and 26% of inline scripts edit a file that can be recovered. All in all, ~93% of tool calls are either kept in full or point to a file that remains in the workspace.

Q10: Re-reading a modified file does not recover its earlier contents or transient observations.

A10: CliffCompaction gives up full-history recall (Section 2.3) and cannot recover earlier contents of a modified file or transient outputs. However, the agent's conclusions are kept in its thoughts, and current plans are formed from previous findings, which we find is sufficient to preserve most critical information to continue a task after compaction.

Q11: How does the method handle [...] user constraints introduced after the initial task description?

A11: In the proxy we release (supplementary material), CliffCompaction keeps user messages verbatim in the compacted block, and like the rest of the block, they are discarded at the next compaction.

We and our colleagues have been using CliffCompaction in interactive, human-in-the-loop workflows with frequent follow-up prompts and have not observed constraints being lost. We hypothesize that this is because CliffCompaction keeps the agent's thoughts, and user constraints are repeatedly restated in the agent's own thoughts.

Thanks for the detailed explaination, which address most of my concerns. And there are only some small questions:

Regarding the ablation study of keeping/dropping previous history on KernelBench (Kimi K2.7, OpenHands, 128K, replay), it seems keeping the previous history leads to a enormous re-prefill overheads. While I suppose that the kept history should be untouched and cached in the inference engine and not lead to re-prefill, could you please explain the mechanism here? Besides, can you also report the performance superiority, such as achieved speedup ratio, of dropping history on KernelBench?

Another interesting point is in Appendix A.3, figure 4. You mentioned that the microcompaction strategy adopted by Claude Code constantly edits prefix and invalidate the cache. If I understand correctly, CliffCompaction follows the append-only principle in compaction operation while Claude Code don't, which makes the CliffCompation save more re-prefill cost?

Reviewer_x5SUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free compaction strategy for long-horizon coding agents. Instead of rewriting or summarizing, it only truncates or drops content. When the context length crosses a preset threshold, it drops long tool results and reduces earlier tool calls to short signatures, while keeping the system prompt, the task description and the most recent turns. The method looks simple but works well in the experiments. It is also aware of KV cache validity: it never re-compacts an earlier compaction, which is a step beyond papers that only report token reductions rather than cost in practice. Experiments cover SWE-bench Verified, Terminal-Bench 2.0/2.1 and KernelBench Level 3, across several scaffolds and Kimi and GLM models. The paper reports lower cost at matched or near-matched resolution, with a 51% total-cost reduction in its Table 6 breakdown. It also reports KernelBench speedups of 2.23× at 200 steps and 3.58× at 400 steps. Finally, in a test-time-scaling experiment, three compacted Kimi K2.6 rollouts plus a learned selector (SGV) reach 69.7% on Terminal-Bench 2.0, matching Opus 4.7 (69.4%).

Critical strengths

A simple but effective method for long-horizon coding agents. It needs no training, works through an API proxy, and even works with a closed scaffold like Claude Code (Terminal-Bench 2.1, 76.69% vs. 70.97% for Claude Code's own auto-compaction at a matched budget). At moderate thresholds (32K/16K), across scaffolds and model families, it stays at or near the full-context level. For example, on Terminal-Bench 2.0 with Kimi K2.6, CliffCompaction reaches 61.42 ± 4.25% at 32K ($0.24) and 61.42 ± 1.30% at 16K ($0.19), while full context resolves 59.16 ± 3.41% at $0.40. That is the same or better performance at 40-50% lower cost.

The evaluation reports the actual per-task cost in dollars next to every resolution number (Tables 2, 3 and 5), not only token reduction. This matters because end-to-end cost is what practitioners care about, and it makes the method more appealing to adopt. The paper also explains why compaction saves money even though it forces re-prefilling: cache reads dominate the bill for long trajectories, and the method targets them directly.

Critical weaknesses

Some headline experiments need a better setup. The 69.7% that matches Opus 4.7 (Table 11) comes from an SGV selector trained on 100% of the 89 Terminal-Bench 2.0 tasks and evaluated on those same 89 tasks (Appendix E.1), so the estimate is in-sample.In addition, the Opus 4.7 score in Table 11 is taken from the system card, not re-run, so it might differ under the same harness.

Some claimed wins are not shown to be statistically significant. For example, on Terminal-Bench 2.0, 61.42 ± 4.25% against 59.16 ± 3.41% is described as "exceeding the full-context baseline ... by 2.26 percentage points". Terminal-Bench 2.0 has 89 tasks, so one task is 1.12 points and the claimed gain is two tasks; the difference is within one standard deviation, and the paper reports no significance test.

Questions for authors

What are the Table 11 "Practical" numbers for every row when SGV is evaluated out of sample, as in the 5-fold cross-validation of Table 16? Do the comparisons with Opus 4.7 and GPT-5.3 Codex still hold?

How many seeds support the KernelBench results (Table 4) and the test-time-scaling results (Tables 11-12)? For the Terminal-Bench 2.0 win over full context, which is within one standard deviation, could you either run more seeds or add a paired statistical test to show the method is actually better?

How many independent runs produce each ± in Tables 2-3 and Table 5, and were all 500 SWE-bench Verified instances and all 89 Terminal-Bench 2.0 tasks run in every configuration? If a subset was used anywhere, please state it.

2 comments
AuthorsOpenReview ↗

(1/2) Addressing out-of-sample selector evaluation (new results) and statistical significance

Q1: [...] SGV selector trained on 100% of the 89 Terminal-Bench 2.0 tasks and evaluated on those same 89 tasks (Appendix E.1), so the estimate is in-sample.

A1: Following your suggestion, we evaluate the selector out-of-sample (please see A4). In 5 of 7 settings, the in-sample and out-of-sample results overlap within their standard deviations. As such, we conclude that there is little bias. This likely stems from us only using summary statistics that are mostly task blind (number of files, number of lines etc).

Q2: The Opus 4.7 score in Table 11 is taken from the system card, not re-run, so it might differ under the same harness.

A2: The Opus 4.7 system card [1] (Section 8.3) reports its Terminal-Bench 2.0 results using the Harbor scaffold with the Terminus-2 harness and its default parser, which is the same harness used in our experiments. The score is therefore comparable to our Terminus-2 rows.

[1] https://www-cdn.anthropic.com/037f06850df7fbe871e206dad004c3db5fd50340/Claude%20Opus%204.7%20System%20Card.pdf

Q3: Some claimed wins are not shown to be statistically significant. For example, on Terminal-Bench 2.0, 61.42 ± 4.25% against 59.16 ± 3.41% is described as "exceeding the full-context baseline ... by 2.26 percentage points".

A3: We have been imprecise and will change the wording. We add a two-sided paired permutation test over tasks and find that, across all CliffCompaction and full-context experiments on Terminal Bench, CliffCompaction is significantly better (+3.46 points, p = 0.020). We will report this test in the paper.

Q4: What are the Table 11 "Practical" numbers for every row when SGV is evaluated out of sample, as in the 5-fold cross-validation of Table 16? Do the comparisons with Opus 4.7 and GPT-5.3 Codex still hold?

A4: We report 5-fold cross-validation over tasks, with 5 training seeds and cross-model training, for every SGV entry in Table 11.

SettingIn-sample, cross-modelOut-of-sample, cross-model
Kimi K2.6 + SGV256K, k=363.1 ± 0.560.9 ± 1.2
Kimi K2.6 + Cliff + SGV32K, k=363.8 ± 1.563.6 ± 2.2
Kimi K2.6 + Cliff + SGV16K, k=368.1 ± 1.069.7 ± 2.1
Kimi K2.6 + Cliff + SGV16K, k=266.1 ± 0.264.6 ± 0.3
GLM 5.1 + SGV200K, k=354.2 ± 0.554.6 ± 1.3
GLM 5.1 + Cliff + SGV32K, k=359.3 ± 1.156.6 ± 2.8
GLM 5.1 + Cliff + SGV16K, k=359.3 ± 0.558.0 ± 1.5

The comparisons hold for both Pareto-optimal configurations: Kimi K2.6 + Cliff + SGV (16K, k=3) reaches 69.7, on par with Opus 4.7 (69.4) and above GPT-5.3 Codex (64.7). The cheapest configuration (16K, k=2) reaches 64.6, matching GPT-5.3 Codex. Both configurations cost less than GPT-5.3 Codex.

As highlighted in the paper, group-relative statistics are central to SGV. Since out of sample they must transfer to unseen tasks, we add more within-group features (e.g., relative error rate, number of completed rollouts in the group), fixed across all configurations. Details will be in the appendix.

AuthorsOpenReview ↗

(2/2) Addressing seeds and number of runs

Q5: How many seeds support the KernelBench results (Table 4) and the test-time-scaling results (Tables 11-12)?

A5: KernelBench (Table 4): We do one run per configuration. We follow the experimental setup of prior work [1–3], which achieves robustness by analysing the evaluation variance instead: each kernel is checked on 5 random inputs and timed over 10 runs (Appendix E.2). Furthermore, each run already covers many candidates sampled iteratively and reports the best found. This makes the overall results highly robust to variation. The gaps in Table 4 are large, and our claims hold across GPUs and models.

Test-time scaling (Tables 11-12): We do one set of rollouts, which is standard procedure in prior work [4, 5]. We varied the selector's training seed (5 seeds, A4) and found the standard deviation is at most 1.5 points in-sample (~1.3 of the 89 tasks).

[1] Kernelbench: Can llms write efficient gpu kernels? https://arxiv.org/abs/2502.10517

[2] Cuda agent: Large-scale agentic rl for high-performance cuda kernel generation. https://arxiv.org/abs/2602.24286

[3] Adaexplore: Failure-driven adaptation and diversity-preserving search for efficient kernel generation. https://arxiv.org/abs/2604.16625v1

[4] Scaling Test-Time Compute for Agentic Coding. https://arxiv.org/pdf/2604.16529

[5] LLM-as-a-Verifier: A General-Purpose Verification Framework. https://arxiv.org/pdf/2607.05391

Q6: For the Terminal-Bench 2.0 win over full context, which is within one standard deviation, could you either run more seeds or add a paired statistical test to show the method is actually better?

A6: Please see A3.

Q7: How many independent runs produce each ± in Tables 2-3 and Table 5, and were all 500 SWE-bench Verified instances and all 89 Terminal-Bench 2.0 tasks run in every configuration?

A7: Every configuration is run on all 500 SWE-bench Verified instances and all 89 Terminal-Bench tasks, 3 seeds each. We will state the number of runs in each table caption.

Reviewer_MzUkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cost-efficient context management for long-horizon coding agents operating within limited context windows. It proposes CliffCompaction, which retains selected verbatim excerpts, drops or truncates token-intensive content, and discards previous compactions to avoid recursive summarization. Experiments on SWE-bench Verified, Terminal-Bench, and KernelBench show substantial cost savings while maintaining or improving performance in several configurations.

Critical strengths
  1. The paper addresses an important deployment challenge and explicitly considers the trade-off between shorter contexts and the re-prefilling costs caused by cache invalidation.

  2. The core compaction procedure requires neither training nor auxiliary LLM calls. Its integration across multiple agent frameworks, including a closed-source framework through an API proxy, supports its practical utility.

Critical weaknesses
  1. The core design choices are not independently validated. Although context budgets are varied, the paper does not isolate the contributions of verbatim retention, discarding previous compactions, and individual truncation rules. The increased file re-reading in Figure 6 is consistent with the proposed explanation but does not directly establish higher information fidelity. Ablations with matched compaction frequencies and token budgets, alongside direct measurements of information retention or distortion, would clarify the method’s technical contribution.

  2. The advantage over strong compaction baselines needs stronger statistical support. In Table 5, CliffCompaction achieves a 3.58× KernelBench speedup at $8.32 per problem, versus 3.47× at $8.10 for summarization alone. Without uncertainty estimates, the robustness of this performance difference remains unclear. These results support a competitive cost–performance trade-off, but the claim that CliffCompaction is the only method combining strong performance with low cost appears overstated.

Questions for authors

Please refer to the paper weakness

1 comment
AuthorsOpenReview ↗

Addressing ablations, information fidelity, and advantage over baselines

Q1: The core design choices are not independently validated. [...] the paper does not isolate the contributions of verbatim retention, discarding previous compactions, and individual truncation rules.

A1: We add these ablations below and will include them in the revision. This question overlaps with Reviewer qXDu’s Q3-Q4 – please see our answer there for full results.

  • Discarding previous compactions: Dropping earlier compactions does not hurt quality (69.53 vs. 69.13 resolved on SWE-bench Verified), but makes re-prefill 3.4× cheaper on KernelBench.

  • Verbatim recent turns: We follow common practice in coding agents’ compaction (e.g., Claude Code compaction, OpenHands' summarizing condenser) and keep the most recent turns verbatim.

  • Individual truncation rules: We truncate tool calls to their signatures and drop long tool observations because these two components dominate the context (28% and 56% of tokens, Section 2.2 and Table 6).

  • Thresholds for each rule: quality and savings are not sensitive to the exact values. We use the same thresholds across all models, scaffolds and benchmarks, even though thought lengths, tool-call formats and observation sizes vary between them, and CliffCompaction still matches or stays close to full context everywhere. Stricter thresholds change savings by less than 5% since savings are superlinear with respect to context length.

Q2: The increased file re-reading in Figure 6 is consistent with the proposed explanation but does not directly establish higher information fidelity.

A2: Fidelity/precision in CliffCompaction holds by design: we never rephrase or rewrite content in the context. Re-reading brings back the needed files in their correct, latest state, unlike summaries, which can omit or distort details (see A4).

Q3: Ablations with matched compaction frequencies and token budgets

A3: All compaction methods use the same context threshold, so the token budget is matched. Under this budget, the number of compactions is determined by how much context each method frees per compaction, so it is a property of the method, and we do not fix it. On KernelBench, CliffCompaction and summarization compact a comparable number of times (6.4 vs. 5.6 per problem), while microcompaction (77) and sliding window (111) compact far more often.

Q4: Direct measurements of information retention or distortion.

A4: Directly measuring information distortion is difficult, because it depends on semantic, task-specific knowledge. CliffCompaction cannot distort information by design, it only truncates/drops context and never rewrites it. Summarization does rewrite context, and the rewrite can be wrong. For example, on KernelBench problem 47 (NetVLAD):

  1. The agent created and wrote custom GEMM kernels, found them slower than cuBLAS, then reverted and deleted them.
  2. The summary reported only the creation part: it listed the deleted files as "(newly created)" and omitted the benchmark, the revert, and the deletions.
  3. The agent trusted the summary and checked the code, found no custom kernels, and re-wrote the same kernels it had already rejected. The run ends at 2.05× speedup, compared with 2.95× for CliffCompaction.

Q5: The advantage over strong compaction baselines needs stronger statistical support [...] Without uncertainty estimates, the robustness of this performance difference remains unclear. [...] The claim that CliffCompaction is the only method combining strong performance with low cost appears overstated.

A5: This overlaps with Reviewer qXDu's Q5; please see the full explanation there.

On KernelBench, the improvement of CliffCompaction over summarization after 100 kernels (+0.30×, Figure 7b) is similar to that of specialized/trained methods (AdaExplore [1], CUDA Agent [2]). CliffCompaction also compacts in milliseconds, while summarization adds 7.2 h for generating summaries. Sliding window and Claude Code's microcompaction + summarization are both more expensive and reach lower speedups than CliffCompaction (Table 5).

On SWE-bench Verified (Table 5) and Terminal-Bench (Table 2), CliffCompaction outperforms summarization in every setting, significantly in three (paired permutation test, p < 0.05). We will revise the wording.

[1] Adaexplore: Failure-driven adaptation and diversity-preserving search for efficient kernel generation. https://arxiv.org/abs/2604.16625v1

[2] Cuda agent: Large-scale agentic rl for high-performance cuda kernel generation. https://arxiv.org/abs/2602.24286

Q6: In Table 5, CliffCompaction achieves a 3.58× KernelBench speedup at 8.32 USD per problem, versus 3.47× at 8.10 USD for summarization alone.

A6: Please see A5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Revision Summary

We thank all reviewers for their valuable feedback! We have uploaded a revision that addresses the reviewers' concerns, with changes highlighted in blue.

Main paper:

  • Added discussion of The Complexity Trap, SWE-Pruner, and CoACT to the experimental setup and related work (Sections 5–6) [qXDu]
  • Added discussion of caching and re-prefill overhead (Section 5) [qXDu]
  • Added statistical tests and adjusted the wording of the Terminal-Bench results (Section 3.2) [x5SU]
  • Revised the wording on CliffCompaction's quality and cost (Section 5) [MzUk]
  • Stated the number of seeds for ± in Tables 2, 3, and 5 [x5SU]

Appendix:

  • Added ablations on keeping vs. dropping previous compactions and discussion of other design choices (Appendix C) [qXDu, MzUk]
  • Added out-of-sample evaluation of the test-time-scaling selector (Appendix E.1) [x5SU]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.