- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the evaluation and improvement of context compaction in long-horizon coding agents. The authors introduce TaskAnchor, a checkpoint-based evaluation framework that assesses compaction quality from three complementary perspectives: information retention, short-horizon behavioral impact, and end-to-end task outcomes. Using this framework, the authors evaluate native compaction across different coding-agent scaffolds and propose StateCarry, a training-free compaction method combining context-aware tool-output compression with source-grounded summary revision to preserve task-relevant information and supporting evidence. Experiments show that native compaction causes substantial information loss, while StateCarry improves information retention over native compaction by 14.88 percentage points at approximately 200K-token checkpoints, with few adverse events observed in short continuations and no observed decline in E2E task performance.
Critical strengths
-
Meaningful Research Topic. The paper addresses an important problem in long-horizon coding agents: how to evaluate context compaction beyond compression ratios and final task outcomes. The proposed research direction is practically relevant to real-world agent execution.
-
Comprehensive Evaluation Framework. TaskAnchor evaluates context compaction from three complementary perspectives: information retention, short-horizon behavioral impact, and end-to-end task outcomes. This design provides a more fine-grained understanding of compaction quality than relying solely on final task performance.
-
Practical Compaction Method. The proposed StateCarry method is training-free and explicitly preserves supporting evidence and its scope during compression. Experiments demonstrate improved information retention over native compaction without an observed decline in end-to-end task performance.
-
Evaluation Across Multiple Agent Scaffolds. The analysis of native compaction across four coding-agent scaffolds provides empirical evidence of information loss under different agent implementations, supporting the general relevance of the research problem.
Critical weaknesses
-
Limited Evidence of End-to-End Effectiveness of StateCarry. The end-to-end evaluation is limited to only 20 coding tasks, with the best-performing methods solving just 2 out of 20 tasks. Although StateCarry demonstrates improved information retention, its benefit to actual task completion remains insufficiently established. Expanding the evaluation to more tasks and diverse long-horizon coding scenarios, and demonstrating consistent improvements in task outcomes, would provide stronger evidence of StateCarry's practical effectiveness and robustness.
-
Limited Evaluation at Larger Context Lengths. Although the paper emphasizes the importance of evaluating compaction on long coding histories, most experiments are conducted at approximately 200K-token checkpoints, substantially below the native compaction thresholds of several evaluated models and agent harness (e.g., 800K–1M). The effectiveness of TaskAnchor's evaluation and the proposed StateCarry method at larger context lengths therefore remains insufficiently explored. Evaluating compaction at multiple context lengths, particularly near the models' context limits, would provide stronger evidence of the method's robustness and generalizability.
-
Computational Overhead and Efficiency Trade-offs. StateCarry introduces additional computational overhead through tool-output compression and source-grounded summary revision. As shown in Table 2, its compression token consumption is substantially higher than native compaction (65.92M vs. 5.32M), while the average task completion time increases from 1.07 to 1.57 hours. Although the total token consumption remains comparable, the practical benefits of these additional costs remain unclear, especially given the limited improvement in end-to-end task outcomes. A more fine-grained analysis of the computational costs and benefits of each component would help clarify the overall efficiency and practicality of StateCarry.
-
Limited Reliability of Behavioral Evaluation. In short-horizon behavioral impact evaluation, although the LLM judge achieves relatively high overall agreement with human annotations, its precision and recall in identifying adverse events remain limited. As reported in Appendix E, the recall is only 66.7% for Unsupported Task Direction and 50.0% for both Task-constraint Conflict and Action-purpose Mismatch. The precision of several metrics is also relatively low. Given the small differences in adverse-event rates across compaction methods, these judgment errors may affect the reliability of the behavioral comparisons. Additional validation or sensitivity analysis using human-corrected judgments would strengthen the conclusions.
Questions for authors
-
Cross-Harness Model Configurations. In the cross-harness evaluation (Figure 4), did the authors use the same underlying LLM across the four agent scaffolds? If different models were used, could the authors clarify the model configurations and discuss how model differences might affect the observed information retention results?
-
Model Sensitivity of Compaction Methods. Have the authors investigated how model capabilities affect the effectiveness of different compaction methods? In particular, (1) whether different execution models, especially those with different context window sizes, exhibit different preferences for compaction strategies, and (2) how the choice of the summarization model affects compaction quality, including whether it needs to be the same as the main agent model.
4 comments
Official Comment by Authors (2)
Answer to Weakness 3: Computational Overhead and Efficiency Trade-offs
The concern is whether StateCarry's additional compression cost yields practical benefits, and how cost and benefit divide across its components. The table pairs costs from the 20-task end-to-end runs (Table 14) with the ablation on 104 shared questions (Table 3):
| Component | Compression tokens | Strict gain (points) |
|---|---|---|
| Drafting + revision | 18.26M (27.7%) | +12.35 over Native |
| Tool-output compression | 47.66M (72.3%) | +3.89 over SC summary only |
The summary stages use 27.7% of StateCarry's compression tokens and provide most of its retention gain. StateCarry's summary module alone raises Strict from 57.22% (Native) to 69.57%, at 18.26M tokens versus Native's 5.32M. Full StateCarry also scores 9.10 points above the SC draft + native control, which keeps the same two-stage calls and source access (Section 5.5), so this gain does not come from additional source reading.
Tool-output compression lowers execution token usage by shortening the observations carried through later execution. Its execution savings offset its 72.3% share of compression tokens: on the 20 tasks, execution tokens fall by 9.27%, so total agent tokens are nearly equal (661.63M vs. 661.91M; Table 13), and on the 50 tasks, total agent tokens are 7.21% lower (Table 2). It also adds 3.89 Strict points when paired with the SC summary module, but no gain with native summarization, because revision relies on the outcomes and scope that compression preserves. As the largest compression cost, it is also the main target for further efficiency gains, such as smaller summarization models and deterministic filtering of repetitive or boilerplate outputs (Section 6).
Mean wall time increases from 1.07 to 1.57 hours on the 20 tasks and from 2.25 to 2.70 hours on the 50 tasks. The additional time may partly reflect further task progress, including successful builds and validation: on the 50 tasks, StateCarry solves more tasks and passes more F2P tests while using fewer agent tokens (Appendix F).
Answer to Weakness 4: Reliability of Behavioral Evaluation
The concern is whether errors in the automatic event labels could affect the behavioral comparisons across methods. We now report the human-corrected sensitivity analysis suggested by the reviewer (new Table 11, Appendix E): using the existing expert-audit annotations, we recompute event rates by method on the common random audit sample of 40 continuations (eight continuations and 80 action opportunities per method).
Table 11 (compact view). Each cell shows LLM → human consensus, as confirmed events / sampled opportunities.
| Method | Unsupported task direction | Task-constraint conflict | Historical-premise violation | Action-purpose mismatch |
|---|---|---|---|---|
| Native | 1/8 → 1/8 | 0/8 → 0/8 | 0/8 → 0/8 | 0/80 → 0/80 |
| AgentDiet | 1/8 → 1/8 | 1/8 → 1/8 | 0/8 → 0/8 | 0/80 → 2/80 |
| Masking | 2/8 → 1/8 | 0/8 → 0/8 | 0/8 → 0/8 | 2/80 → 2/80 |
| ACON-UTCO | 2/8 → 1/8 | 0/8 → 1/8 | 0/8 → 0/8 | 1/80 → 1/80 |
| StateCarry | 1/8 → 0/8 | 0/8 → 0/8 | 0/8 → 0/8 | 1/80 → 3/80 |
Human consensus requires both independent reviewers to confirm an event; all sampled opportunities remain in the denominators.
The human-corrected labels reproduce the comparison pattern stated in Table 1. Under both label sources, StateCarry has the lowest rates on the three continuation-level measures (including ties) and is not the lowest on action-purpose mismatch. It is also the only method with no human-confirmed event on any continuation-level measure. Across methods, LLM and human counts differ by at most one event on each continuation-level measure and by at most two events on action-purpose mismatch.
Regarding recall, all seven events missed by the LLM in the random sample were labeled uncertain rather than no: two unsupported task directions, one task-constraint conflict, and four action-purpose mismatches (Appendix E). These cases remain in the Table 1 denominators without being counted as event absence. For action-purpose mismatch, these cases account for the higher human counts for AgentDiet (0 → 2) and StateCarry (1 → 3), while all four LLM-confirmed events in the sample were also confirmed by humans.
Regarding precision, Table 10 has a different scope: it covers all LLM-confirmed events across the 300 continuations, most of which entered the audit through targeted review rather than the random sample. Task-constraint conflicts and historical-premise violations are also rare, with three LLM-confirmed events each across all 300 continuations. Table 11 instead covers every opportunity in the same random continuations for all five methods, allowing a direct comparison. Section 5.3 now also reports StateCarry's human-consensus rates on this sample.
Official Comment by Authors (3)
Answer to Question 1: Cross-Harness Model Configurations
The question is whether model differences could explain the cross-scaffold results in Figure 4. All four scaffolds use the same underlying model. In every scaffold, deepseek-flash serves as the execution model, the compaction model, and the information-retention respondent; question construction and judging use gpt-5.6-sol, and user simulation uses glm-5.3. The revised manuscript states these assignments in Section 5.1 and new Table 5 (Appendix B.5).
With the model held fixed, the differences in Figure 4 reflect each scaffold's native context-management policy, including history selection, serialization, retention of recent messages, and the summarization prompt. Each scaffold also produces its own execution history and question bank, so Figure 4 characterizes information loss under each native policy rather than ranking the scaffolds as products (Appendix G.1).
Answer to Question 2: Model Sensitivity of Compaction Methods
The question is how model choice affects compaction methods, for both the execution model and the summarization model. StateCarry does not require the summarization model to match the execution model. It connects summarization and execution through a textual handoff, so the two roles can use different models. Our experiments use deepseek-flash for both roles, following the default model selection in our pinned OpenCode version, to hold model capability fixed across compaction methods (Appendix B.5).
Within our available resources, this paper does not yet include a systematic model-sensitivity study, and we list execution and summarization model combinations and context budgets as future work (Section 6). TaskAnchor is designed to make such comparisons practical: with checkpoints and questions fixed, a new summarization model requires only new compaction and evaluation runs, which take hours rather than days on 20 checkpoints (Appendix H.4).
We expect both factors raised by the reviewer to matter. Execution models with larger context windows typically trigger compaction later, so the compactor must process longer histories, where positional biases make dispersed task-relevant information harder to preserve (Section 1). This may increase the value of preserving evidence and its scope. For the summarization model, smaller models could lower cost (Section 6), and TaskAnchor's retention probes can check whether they still preserve the evidence and conditions that later execution relies on.
Thanks for the response, it addresses most of my concern. I will keep the score.
Official Comment by Authors (1)
We thank the reviewer for the careful assessment and constructive questions. Revisions in the updated manuscript are marked in blue.
Answer to Weakness 1: End-to-End Effectiveness of StateCarry
The concern is whether StateCarry's retention gain carries over to task completion. As the reviewer notes, TaskAnchor is designed to provide a more fine-grained understanding of compaction quality than final task performance alone. This matters for LoLBench, where each task requires substantial coding work over hundreds of thousands of tokens and few tasks are fully solved by any method (Appendix A.2). We therefore report fail-to-pass (F2P) tests alongside solved tasks: F2P measures newly passing functional tests, with micro pooling tests and macro weighting tasks equally.
Within our available resources, we expanded the Native–StateCarry end-to-end comparison from 20 to 50 distinct tasks, with one run per method per task (100 runs; Table 2, Appendix A.1). The 50 tasks span 27 repositories, adding systems such as CPython, DuckDB, Flink, Django, and DataFusion to the original suite.
On the expanded 50 tasks, StateCarry improves all three outcome measures over Native while using 7.21% fewer total agent tokens. On the original 20 tasks, StateCarry also ties for the most solved tasks (2/20) and has the highest F2P micro score among the five methods (30.94%). The expanded task manifest and run index are available in the anonymous repository (
data/lolbench/expanded_e2e), with scoring details in Appendix F. Appendix H.2 further traces one functional difference to retained information: in the Cargo task, StateCarry retained an output rule that the agent explicitly cited when implementing SBOM sidecars, and its patch passed 10/10 F2P tests versus 8/10 for Native.Answer to Weakness 2: Evaluation at Larger Context Lengths
The concern is whether conclusions drawn at approximately 200K tokens extend to larger context lengths. We chose the 200K checkpoint to combine complete coverage of the task histories with practical relevance (Appendix A.2 and A.3).
On the 20 LoLBench tasks, 800K–1M thresholds would not trigger compaction: none of their source histories reached 800K tokens, and their median maximum input was 337K. All 20 histories reach 200K, whereas only five reach 400K, so 200K is the highest tier at which every task contributes a checkpoint and leaves work for subsequent evaluation.
The 200K setting is also close to deployed practice. Model capacity, the configured input budget, and the compaction trigger are distinct quantities: Codex CLI's default trigger is about 245K tokens (90% of its 272K active window), and our 200K checkpoint is about 18% below it. At the same time, 200K is several times larger than the 32K/64K budgets used in prior compaction evaluations, which is the gap that motivates TaskAnchor (Section 1).
The end-to-end setting further covers repeated compaction over histories longer than a single 200K window: in independent Native runs on these tasks, 13 compact at least twice (Table 4). TaskAnchor itself is not tied to a particular length, since checkpoints can be registered at any compaction boundary. As resources and suitable tasks permit, we will apply it at higher compaction thresholds (Section 6).