- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CliffCompaction, a training-free context-management method for long-horizon coding agents. It periodically replaces older interactions with short verbatim fragments, retains the system prompt, initial task, and recent turns, and discards the previous compaction block when constructing the next one. The method targets both context growth and the cost of repeatedly processing cached input. Experiments cover SWE-bench Verified, Terminal-Bench, and KernelBench across several models and scaffolds.
Critical strengths
-
The paper addresses an important practical problem with a simple, deployable method. The distinction between reducing context length and preserving cache reuse is useful: frequent edits can undermine the savings from removing tokens. An implementation that requires neither model training nor auxiliary summarization calls is attractive for existing coding-agent systems.
-
The empirical coverage is reasonably broad. The study includes multiple models, several scaffolds, and both repository repair and much longer kernel-optimization trajectories.
Critical weaknesses
-
Comparisons with closely related context-management methods are incomplete. The evaluation includes sliding windows, summarization, and microcompaction in Table 5, but omits several relevant approaches. For example, The Complexity Trap[1], SWE-Pruner[2] and CoACT[3]. These works are neither discussed nor evaluated, leaving the advantages of CliffCompaction's fixed truncation rules over existing masking and task-aware compression methods unclear.
-
The method lacks ablations of its key design choices. In particular, the paper does not isolate the effect of discarding the previous compaction block at every new compaction. It remains unclear whether this improves performance by avoiding accumulated distortion or whether retaining selected older information would better preserve long-range dependencies. A controlled comparison of retaining versus discarding earlier compacted history, under the same trigger and retained-context budget, would directly test this central choice. Additional ablations should separate the effects of truncating tool calls and reasoning, dropping long tool outputs, and retaining recent turns, together with sensitivity to their length thresholds.
-
The current results do not clearly establish an efficiency advantage over competing methods. In Table 5, the advantage over summarization is modest: CliffCompaction achieves 3.58× speedup at $8.32 per task, compared with 3.47× at $8.10, and the SWE-bench results show no consistent advantage in both quality and cost. Moreover, the efficiency comparison between methods focuses primarily on cost. It lacks a systematic comparison of actual trajectory length in tokens, end-to-end execution time, and interaction rounds across competing methods under the same setup. Reporting these metrics at comparable solution quality would clarify whether the cost savings translate into more efficient task execution.
-
Cache friendliness is emphasized in the Introduction and Section 2.1, but the main-text experiments do not directly validate this claimed advantage. Reporting measured cache-hit rates, cache invalidations, and re-prefill overhead under matched settings would help establish whether CliffCompaction better preserves cache reuse and distinguish this benefit from savings due to shorter contexts.
My concerns primarily relate to the experimental evaluation: comparisons with relevant prior methods, ablations of the central design choices, and direct evidence for the efficiency and cache-reuse advantages remain insufficient. I would be very willing to raise my score if the authors provide the experimental evidence discussed above and demonstrate that the benefits hold under controlled comparisons.
[1] Lindenbauer, Tobias, et al. "The complexity trap: Simple observation masking is as efficient as LLM summarization for agent context management." arXiv preprint arXiv:2508.21433 (2025). [2] Wang, Yuhang, et al. "Swe-pruner: Self-adaptive context pruning for coding agents." arXiv preprint arXiv:2601.16746 (2026). [3] Chen, Haorui, et al. "CoACT: Action-Preserving Observation Compression for Coding Agents." arXiv preprint arXiv:2607.02911 (2026).
Questions for authors
- Could you clarify the scope of the recoverability claim in Section 2.2? Appendix C retains only the first 150 characters of some commands, which may be insufficient to repeat the original call, and re-reading a modified file does not recover its earlier contents or transient observations. How does the method handle information that cannot be retrieved from the current workspace, including user constraints introduced after the initial task description?
6 comments
(2/2) Addressing efficiency, caching, and recoverability
Q5: In Table 5, the advantage over summarization is modest [...]
A5: On KernelBench, the results may seem like modest improvements, but other papers with specialized techniques make similar improvements in terms of speedups. For example, specialized/trained methods such as AdaExplore [1] and CUDA Agent [2] gain +0.22–0.24× over their baselines, compared with +0.30× for CliffCompaction over summarization after 100 kernels (Figure 7b). Additionally, CliffCompaction has other benefits. Summarization has a poor walltime since output token generation is expensive. On KernelBench, summarization takes 10.7 more hours across the 50 problems: 7.2h generating summaries and 3.5h compiling and testing its extra kernels (136 vs. 116 per problem, Table 5). CliffCompaction compacts in milliseconds resulting in much faster end-to-end solution times.
On SWE-bench Verified (Table 5) and Terminal-Bench (Table 2), CliffCompaction outperforms summarization in every setting, significantly in three (paired permutation test, p < 0.05).
[1] Adaexplore: Failure-driven adaptation and diversity-preserving search for efficient kernel generation. https://arxiv.org/abs/2604.16625v1
[2] Cuda agent: Large-scale agentic rl for high-performance cuda kernel generation. https://arxiv.org/abs/2602.24286
Q6: Moreover, the efficiency comparison between methods focuses primarily on cost. It lacks a systematic comparison of actual trajectory length in tokens, end-to-end execution time, and interaction rounds across competing methods under the same setup.
A6: We report runtime in Appendix A.2. For example, CliffCompaction reduces per-instance wall-clock by 23–34% on SWE-bench Verified (Table 9), and with a self-hosted vLLM we see speedups of 1.60–2.62× (Table 10). We do not use total wall-clock as the primary comparison between methods due to confounding factors (provider load, peak hours).
We focus on end-to-end cost because it is the primary practical objective. Other factors, like the total number of tokens in a trajectory, are a weak proxy for cost, since about 80% of the cost comes from input tokens. Interaction rounds are capped at the same 400 steps on KernelBench (Table 5), and on SWE-bench Verified all methods add fewer than 5 steps over full context (Figure 6).
Q7: Cache friendliness is emphasized but the main-text experiments do not directly validate this claimed advantage. Reporting measured cache-hit rates, cache invalidations, and re-prefill overhead.
A7: With common caching such as implemented in vLLM, >99% of CliffCompaction's previously sent context is read from cache, because its context is append-only between compactions. This is a main reason why CliffCompaction is so cheap. Re-prefill is minimal: it happens only ~10 times per KernelBench trajectory and every other call reads its context (up to ~128K tokens) from cache. We discuss re-prefill overhead in Figure 4 and Appendix A.3, and will add this discussion to the main text.
Q8: Could you clarify the scope of the recoverability claim in Section 2.2?
A8: The claim covers content that persists in the workspace: source files and files the agent has written can be recovered by re-reading them, and in practice this is what agents do after compaction (Appendix D.1). Content that no longer exists in the workspace, such as earlier versions of modified files, is not recoverable (A10).
Q9: Appendix C retains only the first 150 characters of some commands, which may be insufficient to repeat the original call.
A9: On SWE-bench Verified, 80.1% of tool calls fit in 150 characters, and 90% of longer calls are file writes or inline Python scripts. Written files can be recovered by re-reading, and 26% of inline scripts edit a file that can be recovered. All in all, ~93% of tool calls are either kept in full or point to a file that remains in the workspace.
Q10: Re-reading a modified file does not recover its earlier contents or transient observations.
A10: CliffCompaction gives up full-history recall (Section 2.3) and cannot recover earlier contents of a modified file or transient outputs. However, the agent's conclusions are kept in its thoughts, and current plans are formed from previous findings, which we find is sufficient to preserve most critical information to continue a task after compaction.
Q11: How does the method handle [...] user constraints introduced after the initial task description?
A11: In the proxy we release (supplementary material), CliffCompaction keeps user messages verbatim in the compacted block, and like the rest of the block, they are discarded at the next compaction.
We and our colleagues have been using CliffCompaction in interactive, human-in-the-loop workflows with frequent follow-up prompts and have not observed constraints being lost. We hypothesize that this is because CliffCompaction keeps the agent's thoughts, and user constraints are repeatedly restated in the agent's own thoughts.
Thanks for the detailed explaination, which address most of my concerns. And there are only some small questions:
Regarding the ablation study of keeping/dropping previous history on KernelBench (Kimi K2.7, OpenHands, 128K, replay), it seems keeping the previous history leads to a enormous re-prefill overheads. While I suppose that the kept history should be untouched and cached in the inference engine and not lead to re-prefill, could you please explain the mechanism here? Besides, can you also report the performance superiority, such as achieved speedup ratio, of dropping history on KernelBench?
Another interesting point is in Appendix A.3, figure 4. You mentioned that the microcompaction strategy adopted by Claude Code constantly edits prefix and invalidate the cache. If I understand correctly, CliffCompaction follows the append-only principle in compaction operation while Claude Code don't, which makes the CliffCompation save more re-prefill cost?
(1/2) Addressing related methods and design ablations (with new analyses)
Q1: Comparisons with closely related context-management methods are incomplete: The Complexity Trap, SWE-Pruner and CoACT
A1: We will add these works to our related work and clarify their relationship to our evaluated baselines. Below is further discussion on these methods:
For The Complexity Trap, our Claude Code baseline in Section 5 already evaluates the same combination: it masks older tool observations, then summarizes if the context remains above threshold.
For SWE-Pruner and CoACT, we replayed both with their released models on a 749K-token KernelBench trajectory:
(1) They cost about 3× more than CliffCompaction. (2) They are per-observation compression methods (compressing each new tool result), so their context is unbounded: both still exceed Kimi K2.6/K2.7's 256K window, and are therefore not standalone alternatives to CliffCompaction and the other trajectory-level compaction baselines in the paper.
Q2: [...] leaving the advantages of CliffCompaction's fixed truncation rules over existing masking and task-aware compression methods unclear.
A2: Masking alone does not bound the context: even deleting every tool observation still reaches 408K tokens (A1), so The Complexity Trap and Claude Code (Section 5, Table 5) combine it with LLM summarization. CliffCompaction matches this on SWE-bench Verified and outperforms it on KernelBench at 35% lower cost (3.58× vs. 3.33×, 8.32 vs. 12.84 USD).
Task-aware compression (SWE-Pruner, CoACT) requires a trained compressor model served alongside the agent. CliffCompaction is training-free and drop-in. Furthermore, CliffCompaction is implicitly task-aware: the agent itself decides what stays in context by re-reading the dropped content its current goal needs (Section 2.2).
Q3: The method lacks ablations of its key design choices. [...]discarding the previous compaction block at every new compaction [..]retaining versus discarding earlier compacted history, under the same trigger and retained-context budget, would directly test this central choice.
A3: We ran this ablation and will add it to the revision.
SWE-bench Verified (Kimi K2.5, mini-swe-agent, 16K)
KernelBench (Kimi K2.7, OpenHands, 128K, replay)
Overall summary of findings for dropping vs keeping previous blocks: on SWE-bench we find that resolve rates are the same with/without discarding, but compacted history differs by 1.7×. On KernelBench, compacted history increases by up to 10×, and re-prefill costs increase by 3.4×. As such, we drop earlier blocks in CliffCompaction.
Q4: Additional ablations [...] truncating tool calls and reasoning, dropping long tool outputs, and retaining recent turns, [...] length thresholds.
A4:
Thought truncation is to prevent extreme cases that increase costs substantially. Only ~6% of thought traces are longer than 300 characters, which informed this choice.
Tool call truncation: same reasoning as with thoughts. This is to prevent the effect of extremely long tool calls (mostly file writes and inline Python).
Recent turns: We follow common practice in coding agents' compaction used in Claude Code compaction, OpenHands' summarizing condenser, and others. Other papers, for example The Complexity Trap (Appendix D.1 and D.2), focus in more detail on these ablations than we are able to do here.
Tool output dropping: Tool outputs make up 56% of context and compression is essential. Since tool signatures keep the file path, the agent can re-read outputs saved in the repository. With this, tool output dropping has little effect on quality of the compaction.
Length thresholds: quality and savings are not sensitive to the exact values. We use the same thresholds across all models, scaffolds and benchmarks, even though thought lengths, tool-call formats and observation sizes vary between them, and CliffCompaction still matches or stays close to full context everywhere. Stricter thresholds change savings by less than 5% since savings are superlinear with respect to context length.