acceptodds
Desk rejected at ICLR 2027

TaskAnchor: Evaluating Context Compaction in Long-Horizon Coding Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Context compaction is essential for modern coding agents to handle practical long-horizon tasks, but current evaluations leave three gaps. First, evaluations with 32K/64K context budgets test compaction on relatively short inputs, leaving its effectiveness on histories spanning hundreds of thousands of tokens insufficiently evaluated. Second, evaluations on standalone texts overlook how coding agents select history for compression, retain recent messages, and recover omitted information through tools. Third, binary task rewards are too coarse to reveal information loss and behavioral changes after compaction. To bridge these gaps, we introduce TaskAnchor, a compaction-evaluation framework that compares information accessibility and subsequent behavior from shared execution states in native coding agents. Guided by TaskAnchor's evaluation, we propose StateCarry, a training-free compaction method combining context-aware tool-output compression with source-grounded summary revision to preserve evidence with its scope. Experiments across four scaffolds reveal substantial information loss under native compaction, while comparisons across 20 tasks on OpenCode show that information accessibility, behavior, and task outcomes can diverge. At approximately 200K-token checkpoints, StateCarry reduces context by 81.9% and improves the Strict score over native compaction by 14.9 percentage points, with few adverse events identified by automated evaluation in short continuations. In end-to-end runs, it shows no decrease in solve rate or fail-to-pass (F2P) test passes relative to native compaction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Context Compaction, Coding Agents, Long-Horizon Tasks, Agent Evaluation, Software System

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_StnXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the evaluation and improvement of context compaction in long-horizon coding agents. The authors introduce TaskAnchor, a checkpoint-based evaluation framework that assesses compaction quality from three complementary perspectives: information retention, short-horizon behavioral impact, and end-to-end task outcomes. Using this framework, the authors evaluate native compaction across different coding-agent scaffolds and propose StateCarry, a training-free compaction method combining context-aware tool-output compression with source-grounded summary revision to preserve task-relevant information and supporting evidence. Experiments show that native compaction causes substantial information loss, while StateCarry improves information retention over native compaction by 14.88 percentage points at approximately 200K-token checkpoints, with few adverse events observed in short continuations and no observed decline in E2E task performance.

Critical strengths
  1. Meaningful Research Topic. The paper addresses an important problem in long-horizon coding agents: how to evaluate context compaction beyond compression ratios and final task outcomes. The proposed research direction is practically relevant to real-world agent execution.

  2. Comprehensive Evaluation Framework. TaskAnchor evaluates context compaction from three complementary perspectives: information retention, short-horizon behavioral impact, and end-to-end task outcomes. This design provides a more fine-grained understanding of compaction quality than relying solely on final task performance.

  3. Practical Compaction Method. The proposed StateCarry method is training-free and explicitly preserves supporting evidence and its scope during compression. Experiments demonstrate improved information retention over native compaction without an observed decline in end-to-end task performance.

  4. Evaluation Across Multiple Agent Scaffolds. The analysis of native compaction across four coding-agent scaffolds provides empirical evidence of information loss under different agent implementations, supporting the general relevance of the research problem.

Critical weaknesses
  1. Limited Evidence of End-to-End Effectiveness of StateCarry. The end-to-end evaluation is limited to only 20 coding tasks, with the best-performing methods solving just 2 out of 20 tasks. Although StateCarry demonstrates improved information retention, its benefit to actual task completion remains insufficiently established. Expanding the evaluation to more tasks and diverse long-horizon coding scenarios, and demonstrating consistent improvements in task outcomes, would provide stronger evidence of StateCarry's practical effectiveness and robustness.

  2. Limited Evaluation at Larger Context Lengths. Although the paper emphasizes the importance of evaluating compaction on long coding histories, most experiments are conducted at approximately 200K-token checkpoints, substantially below the native compaction thresholds of several evaluated models and agent harness (e.g., 800K–1M). The effectiveness of TaskAnchor's evaluation and the proposed StateCarry method at larger context lengths therefore remains insufficiently explored. Evaluating compaction at multiple context lengths, particularly near the models' context limits, would provide stronger evidence of the method's robustness and generalizability.

  3. Computational Overhead and Efficiency Trade-offs. StateCarry introduces additional computational overhead through tool-output compression and source-grounded summary revision. As shown in Table 2, its compression token consumption is substantially higher than native compaction (65.92M vs. 5.32M), while the average task completion time increases from 1.07 to 1.57 hours. Although the total token consumption remains comparable, the practical benefits of these additional costs remain unclear, especially given the limited improvement in end-to-end task outcomes. A more fine-grained analysis of the computational costs and benefits of each component would help clarify the overall efficiency and practicality of StateCarry.

  4. Limited Reliability of Behavioral Evaluation. In short-horizon behavioral impact evaluation, although the LLM judge achieves relatively high overall agreement with human annotations, its precision and recall in identifying adverse events remain limited. As reported in Appendix E, the recall is only 66.7% for Unsupported Task Direction and 50.0% for both Task-constraint Conflict and Action-purpose Mismatch. The precision of several metrics is also relatively low. Given the small differences in adverse-event rates across compaction methods, these judgment errors may affect the reliability of the behavioral comparisons. Additional validation or sensitivity analysis using human-corrected judgments would strengthen the conclusions.

Questions for authors
  1. Cross-Harness Model Configurations. In the cross-harness evaluation (Figure 4), did the authors use the same underlying LLM across the four agent scaffolds? If different models were used, could the authors clarify the model configurations and discuss how model differences might affect the observed information retention results?

  2. Model Sensitivity of Compaction Methods. Have the authors investigated how model capabilities affect the effectiveness of different compaction methods? In particular, (1) whether different execution models, especially those with different context window sizes, exhibit different preferences for compaction strategies, and (2) how the choice of the summarization model affects compaction quality, including whether it needs to be the same as the main agent model.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We thank the reviewer for the careful assessment and constructive questions. Revisions in the updated manuscript are marked in blue.

Answer to Weakness 1: End-to-End Effectiveness of StateCarry

The concern is whether StateCarry's retention gain carries over to task completion. As the reviewer notes, TaskAnchor is designed to provide a more fine-grained understanding of compaction quality than final task performance alone. This matters for LoLBench, where each task requires substantial coding work over hundreds of thousands of tokens and few tasks are fully solved by any method (Appendix A.2). We therefore report fail-to-pass (F2P) tests alongside solved tasks: F2P measures newly passing functional tests, with micro pooling tests and macro weighting tasks equally.

Within our available resources, we expanded the Native–StateCarry end-to-end comparison from 20 to 50 distinct tasks, with one run per method per task (100 runs; Table 2, Appendix A.1). The 50 tasks span 27 repositories, adding systems such as CPython, DuckDB, Flink, Django, and DataFusion to the original suite.

MethodSolvedF2P microF2P macro
Native4/5033.27%41.35%
StateCarry7/5036.82%47.25%

On the expanded 50 tasks, StateCarry improves all three outcome measures over Native while using 7.21% fewer total agent tokens. On the original 20 tasks, StateCarry also ties for the most solved tasks (2/20) and has the highest F2P micro score among the five methods (30.94%). The expanded task manifest and run index are available in the anonymous repository (data/lolbench/expanded_e2e), with scoring details in Appendix F. Appendix H.2 further traces one functional difference to retained information: in the Cargo task, StateCarry retained an output rule that the agent explicitly cited when implementing SBOM sidecars, and its patch passed 10/10 F2P tests versus 8/10 for Native.

Answer to Weakness 2: Evaluation at Larger Context Lengths

The concern is whether conclusions drawn at approximately 200K tokens extend to larger context lengths. We chose the 200K checkpoint to combine complete coverage of the task histories with practical relevance (Appendix A.2 and A.3).

On the 20 LoLBench tasks, 800K–1M thresholds would not trigger compaction: none of their source histories reached 800K tokens, and their median maximum input was 337K. All 20 histories reach 200K, whereas only five reach 400K, so 200K is the highest tier at which every task contributes a checkpoint and leaves work for subsequent evaluation.

The 200K setting is also close to deployed practice. Model capacity, the configured input budget, and the compaction trigger are distinct quantities: Codex CLI's default trigger is about 245K tokens (90% of its 272K active window), and our 200K checkpoint is about 18% below it. At the same time, 200K is several times larger than the 32K/64K budgets used in prior compaction evaluations, which is the gap that motivates TaskAnchor (Section 1).

The end-to-end setting further covers repeated compaction over histories longer than a single 200K window: in independent Native runs on these tasks, 13 compact at least twice (Table 4). TaskAnchor itself is not tied to a particular length, since checkpoints can be registered at any compaction boundary. As resources and suitable tasks permit, we will apply it at higher compaction thresholds (Section 6).

AuthorsOpenReview ↗

Official Comment by Authors (2)

Answer to Weakness 3: Computational Overhead and Efficiency Trade-offs

The concern is whether StateCarry's additional compression cost yields practical benefits, and how cost and benefit divide across its components. The table pairs costs from the 20-task end-to-end runs (Table 14) with the ablation on 104 shared questions (Table 3):

ComponentCompression tokensStrict gain (points)
Drafting + revision18.26M (27.7%)+12.35 over Native
Tool-output compression47.66M (72.3%)+3.89 over SC summary only

The summary stages use 27.7% of StateCarry's compression tokens and provide most of its retention gain. StateCarry's summary module alone raises Strict from 57.22% (Native) to 69.57%, at 18.26M tokens versus Native's 5.32M. Full StateCarry also scores 9.10 points above the SC draft + native control, which keeps the same two-stage calls and source access (Section 5.5), so this gain does not come from additional source reading.

Tool-output compression lowers execution token usage by shortening the observations carried through later execution. Its execution savings offset its 72.3% share of compression tokens: on the 20 tasks, execution tokens fall by 9.27%, so total agent tokens are nearly equal (661.63M vs. 661.91M; Table 13), and on the 50 tasks, total agent tokens are 7.21% lower (Table 2). It also adds 3.89 Strict points when paired with the SC summary module, but no gain with native summarization, because revision relies on the outcomes and scope that compression preserves. As the largest compression cost, it is also the main target for further efficiency gains, such as smaller summarization models and deterministic filtering of repetitive or boilerplate outputs (Section 6).

Mean wall time increases from 1.07 to 1.57 hours on the 20 tasks and from 2.25 to 2.70 hours on the 50 tasks. The additional time may partly reflect further task progress, including successful builds and validation: on the 50 tasks, StateCarry solves more tasks and passes more F2P tests while using fewer agent tokens (Appendix F).

Answer to Weakness 4: Reliability of Behavioral Evaluation

The concern is whether errors in the automatic event labels could affect the behavioral comparisons across methods. We now report the human-corrected sensitivity analysis suggested by the reviewer (new Table 11, Appendix E): using the existing expert-audit annotations, we recompute event rates by method on the common random audit sample of 40 continuations (eight continuations and 80 action opportunities per method).

Table 11 (compact view). Each cell shows LLM → human consensus, as confirmed events / sampled opportunities.

MethodUnsupported task directionTask-constraint conflictHistorical-premise violationAction-purpose mismatch
Native1/8 → 1/80/8 → 0/80/8 → 0/80/80 → 0/80
AgentDiet1/8 → 1/81/8 → 1/80/8 → 0/80/80 → 2/80
Masking2/8 → 1/80/8 → 0/80/8 → 0/82/80 → 2/80
ACON-UTCO2/8 → 1/80/8 → 1/80/8 → 0/81/80 → 1/80
StateCarry1/8 → 0/80/8 → 0/80/8 → 0/81/80 → 3/80

Human consensus requires both independent reviewers to confirm an event; all sampled opportunities remain in the denominators.

The human-corrected labels reproduce the comparison pattern stated in Table 1. Under both label sources, StateCarry has the lowest rates on the three continuation-level measures (including ties) and is not the lowest on action-purpose mismatch. It is also the only method with no human-confirmed event on any continuation-level measure. Across methods, LLM and human counts differ by at most one event on each continuation-level measure and by at most two events on action-purpose mismatch.

Regarding recall, all seven events missed by the LLM in the random sample were labeled uncertain rather than no: two unsupported task directions, one task-constraint conflict, and four action-purpose mismatches (Appendix E). These cases remain in the Table 1 denominators without being counted as event absence. For action-purpose mismatch, these cases account for the higher human counts for AgentDiet (0 → 2) and StateCarry (1 → 3), while all four LLM-confirmed events in the sample were also confirmed by humans.

Regarding precision, Table 10 has a different scope: it covers all LLM-confirmed events across the 300 continuations, most of which entered the audit through targeted review rather than the random sample. Task-constraint conflicts and historical-premise violations are also rare, with three LLM-confirmed events each across all 300 continuations. Table 11 instead covers every opportunity in the same random continuations for all five methods, allowing a direct comparison. Section 5.3 now also reports StateCarry's human-consensus rates on this sample.

AuthorsOpenReview ↗

Official Comment by Authors (3)

Answer to Question 1: Cross-Harness Model Configurations

The question is whether model differences could explain the cross-scaffold results in Figure 4. All four scaffolds use the same underlying model. In every scaffold, deepseek-flash serves as the execution model, the compaction model, and the information-retention respondent; question construction and judging use gpt-5.6-sol, and user simulation uses glm-5.3. The revised manuscript states these assignments in Section 5.1 and new Table 5 (Appendix B.5).

With the model held fixed, the differences in Figure 4 reflect each scaffold's native context-management policy, including history selection, serialization, retention of recent messages, and the summarization prompt. Each scaffold also produces its own execution history and question bank, so Figure 4 characterizes information loss under each native policy rather than ranking the scaffolds as products (Appendix G.1).

Answer to Question 2: Model Sensitivity of Compaction Methods

The question is how model choice affects compaction methods, for both the execution model and the summarization model. StateCarry does not require the summarization model to match the execution model. It connects summarization and execution through a textual handoff, so the two roles can use different models. Our experiments use deepseek-flash for both roles, following the default model selection in our pinned OpenCode version, to hold model capability fixed across compaction methods (Appendix B.5).

Within our available resources, this paper does not yet include a systematic model-sensitivity study, and we list execution and summarization model combinations and context budgets as future work (Section 6). TaskAnchor is designed to make such comparisons practical: with checkpoints and questions fixed, a new summarization model requires only new compaction and evaluation runs, which take hours rather than days on 20 checkpoints (Appendix H.4).

We expect both factors raised by the reviewer to matter. Execution models with larger context windows typically trigger compaction later, so the compactor must process longer histories, where positional biases make dispersed task-relevant information harder to preserve (Section 1). This may increase the value of preserving evidence and its scope. For the summarization model, smaller models could lower cost (Section 6), and TaskAnchor's retention probes can check whether they still preserve the evidence and conditions that later execution relies on.

Thanks for the response, it addresses most of my concern. I will keep the score.

Reviewer_kbbXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TaskAnchor, a checkpoint-based evaluation framework for context compaction in long-horizon coding agents. It evaluates compaction through three complementary perspectives: information retention, short-horizon behavioral impact, and end-to-end task performance. They also proposes StateCarry, a training-free compaction method combining context-aware tool-output compression with source-grounded summary revision. Experiments across four coding-agent scaffolds reveal substantial information loss under native compaction. On 20 coding tasks using OpenCode, StateCarry improves information retention by 14.88 percentage points over native compaction, but shows only marginal differences in end-to-end task success: 2/20 vs. 1/20.

Critical strengths
  1. Context management is a practical challenge for long-horizon coding agents, and evaluating compaction beyond simple text compression is meaningful.
  2. The shared-checkpoint enables comparisons of different compaction methods from identical execution states
  3. The paper proposes subscores like information retention, subsequent agent behavior, and end-to-end performance, providing more diagnostic insights than task success alone.
Critical weaknesses
  1. There is no evidence that the proxy introduced with the benchmark can reflect the actual end-to-end coding performance. For example, StateCarry shows substantial improvements in information retention (55.49% -> 70.88%), but no convincing gains in actual coding performance (2/20 vs. 1/20 solved tasks). The paper should establish whether its proposed proxy metrics can reliably predict downstream task success or meaningful agent failures.
  2. Limited methodological novelty: StateCarry mainly combines existing ideas in context-aware tool-output compression and LLM-based summary revision. The ablation suggests that most improvements come from summary revision rather than tool-output compression. But the ablation study comes from 20 tasks/checkpoints, making the finding less rubost.
  3. StateCarry requires more compression tokens than native compaction (65.92M vs. 5.32M) from reading the tool execution. This may explain the improvement over information retention.
Questions for authors

Given that StateCarry substantially improves information retention but shows only marginal differences in end-to-end coding performance, can the authors provide evidence that the proposed evaluation metrics correlate with meaningful downstream improvements in coding agents?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

We thank the reviewer for the careful assessment and constructive feedback. Revisions in the updated manuscript are marked in blue.

Answer to Weakness 1 and the Question on Downstream Utility

The concern is whether TaskAnchor's checkpoint measurements relate to downstream coding outcomes. As the reviewer notes, these views are designed to provide diagnostic information beyond task success. They complement end-to-end evaluation rather than replace it, because a binary outcome can conflate lost information with unrelated coding errors (Section 1). We find consistent downstream evidence at four levels.

  1. Task outcomes. Within our available resources, we expanded the Native–StateCarry end-to-end comparison from 20 to 50 distinct tasks, with one run per method per task (100 runs; Table 2, Appendix A.1).

    MethodSolvedF2P microF2P macro
    Native4/5033.27%41.35%
    StateCarry7/5036.82%47.25%

    StateCarry, which retains more information at the checkpoints, also solves more tasks and passes more fail-to-pass tests across the 50 tasks. F2P (fail-to-pass) measures newly passing functional tests; micro pools tests, while macro weights tasks equally. The expanded task manifest and run index are available in the anonymous repository (data/lolbench/expanded_e2e), with scoring details in Appendix F.

  2. Agent failures. Short-horizon behavioral impact directly measures agent failures after compaction, such as unsupported task directions and task-constraint conflicts. StateCarry has the lowest confirmed rates on three of the four adverse-event measures, including ties (Table 1), and human labels on the common random audit sample reproduce this pattern (new Table 11, Appendix E).

  3. A concrete link to functional outcomes. In the Cargo case, StateCarry retained the Normal/Linkable output rule, and the agent explicitly cited it when implementing SBOM sidecars. Its patch passed 10/10 F2P tests, whereas Native's Normal-only filter missed library sidecars and passed 8/10. This traces retained task knowledge through an implementation decision to a verifiable functional result (Appendix H.2).

  4. Actionable feedback for method development. Information-retention questions locate inaccessible facts and misinterpreted conditions, while native continuations show how the handoff guides behavior. These diagnostics identify what a developer should change in a compaction policy and allow that change to be checked at the same checkpoint. Our answer to Weakness 2 describes how this workflow guided the development of StateCarry (Appendix H).

AuthorsOpenReview ↗

Official Comment by Authors (2)

Answer to Weakness 2: Methodological Novelty and Ablation Robustness

The concern is whether StateCarry adds more than a combination of existing components. StateCarry contributes in two ways: as a method, it addresses a specific failure exposed by TaskAnchor; as a case study, it shows how TaskAnchor supports the development of compaction methods. The failure is that a conclusion survives compaction while the evidence and scope supporting it are lost. For example, a summary stating that tests passed, without specifying which tests ran, can lead an agent to treat partial verification as complete (Section 4). Both components are designed around this failure.

  1. Concrete component designs. For tool-output compression, we redesigned the compression instructions to preserve observed outcomes, affected files or symbols, and the scope of each command, and we archive original outputs with recoverable references. For summary construction, a first-stage draft selects relevant source passages and locations, and a second-stage revision rereads the history to check consequential statements, distinguish observations from hypotheses, update outdated conclusions, and retain their conditions (Section 4).

  2. Complementary components. The reviewer correctly notes that most of the retention gain comes from the summary module. Table 3 further shows that tool-output compression contributes in combination with it:

    ConfigurationTool-output compressionSummaryStrict
    NativeNoNative57.22%
    Tool comp. + nativeYesNative57.06%
    SC summary onlyNoSC69.57%
    Full StateCarryYesSC73.46%

    Tool-output compression adds 3.89 Strict points when paired with the SC summary module, but no gain with native summarization. This interaction reflects the design: compressed observations keep the outcome and scope that revision needs to distinguish what was checked, what failed, and what remains unresolved (Section 5.5).

  3. Robustness of the ablation. All configurations are compared on the same 104 questions from the same 20 checkpoints, so the differences are paired rather than drawn from separate samples. The checkpoints come from 20 different repositories spanning ecosystems such as Rust, Go, Java, C#, Python, and TypeScript (Appendix A.1). Full StateCarry's advantage over Native is also consistent with the independent 50-task end-to-end comparison (Table 2).

  4. A practical validation of TaskAnchor. StateCarry was developed with the workflow TaskAnchor supports. Information-retention probes exposed the failure it targets, and candidate designs were compared at development checkpoints disjoint from the evaluation tasks (Appendix A.1). Evaluating one method on 20 checkpoints takes about 1–2 hours, versus 1–2 days for end-to-end evaluation, so design changes could be checked within hours (Appendix H). StateCarry thus serves both as a compaction method and as a worked example of TaskAnchor-guided development.

Answer to Weakness 3: Additional Compression Computation

The concern is whether StateCarry's retention gain is explained by its additional compression computation. We broke down StateCarry's 65.92M compression tokens from the original 20-task end-to-end runs (Table 14):

StateCarry componentShare of the 65.92M compression tokens
Tool-output compression72.3% (47.66M)
Drafting13.2% (8.70M)
Revision14.5% (9.56M)

Most of the additional compression tokens do not by themselves improve retention. Tool-output compression accounts for 72.3% of StateCarry's compression tokens, yet combining it with native summarization leaves Strict unchanged (57.06% vs. 57.22%; Table 3). Drafting and revision together use 18.26M tokens, compared with Native's 5.32M. To separate the summary design from this extra computation, the new SC draft + native control (Section 5.5, Table 3) keeps StateCarry's two-stage calls and source access but replaces the revision stage with Native-style summarization. It scores 64.36% Strict, 9.10 points below full StateCarry (73.46%), so the additional source reading alone does not account for the gain.

At the whole-task level, StateCarry's additional compression is offset by lower execution usage. On the original 20 tasks, total agent tokens are nearly equal (661.63M vs. 661.91M; Table 13). On the expanded 50 tasks, mean execution tokens decrease from 37.18M to 30.58M per task, and total agent tokens decrease by 7.21% (Table 2). This is consistent with tool-output compression shortening the observations carried through later execution. Because tool-output compression accounts for most compression usage, it is the main target for further efficiency gains, such as smaller summarization models and deterministic filtering of repetitive or boilerplate outputs (Section 6).

AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer kbbX,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #40407

Reviewer_SrYeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TaskAnchor, a framework for evaluating context compaction in long-horizon coding agents. It compares tool-free information retention and short native-agent continuations from shared execution checkpoints, and separately evaluates end-to-end task outcomes. The paper also proposes StateCarry, a training-free method combining context-aware tool-output compression with source-grounded revision of the handoff summary. Experiments examine native compaction across four scaffolds and compare methods on 20 selected LoLBench tasks using OpenCode at approximately 200K-token checkpoints. StateCarry improves the paired information-retention Strict score over native compaction by 14.88 percentage points while reducing context by 81.86%. Short-continuation and end-to-end results provide additional, more limited evidence about downstream utility.

Critical strengths

A useful evaluation design for a practical problem. TaskAnchor separates information accessibility, short-horizon behavior, and whole-task utility instead of treating final task success as the only measure of compaction quality. Reusing the same recorded checkpoint and installed representation for the first two views makes comparisons more controlled, with native tools and recovery mechanisms available during behavioral continuations. This is the paper's strongest contribution: it provides a concrete way to investigate failures that a binary task reward can conceal (Sections 3 and 5).

A plausible method with a supported information-retention benefit and careful reporting. StateCarry is designed to preserve evidence together with its scope and temporal conditions, addressing a credible source of misleading handoffs. Its paired Strict improvement over Native is 14.88 points, with a descriptive 95% task-bootstrap interval of [1.40, 27.93], and it provides a useful empirical accuracy–compression trade-off (Figure 5; Appendix C, Table 6). The source-grounded question construction, common-question analyses, extensive prompts, and disclosure of human-audit limitations strengthen the interpretability of this result.

Critical weaknesses

Method-level behavioral comparisons remain uncertain given the event-label reliability. For action-purpose mismatch, the audit reports 46.9% precision (15/32) and 50% recall (4/8 in the random audit sample), while the main results retain the original automatic labels (Table 9). The pooled audit does not establish whether labeling errors affect the comparisons in Table 1.

The specific contribution of source-grounded revision is not isolated. Although Figure 5 reports an accuracy–compression trade-off, StateCarry uses a longer mean installed input than Native (36,569 vs. 25,310 tokens) and rereads the selected history during revision. The ablation replaces the entire draft–revision–length module rather than removing revision alone (Appendix C.4).

Questions for authors

Can the authors break down the human-audit results by method? On the audited subset, do the human labels support the behavioral comparisons in Table 1?

How much of StateCarry's gain remains under matched context lengths and compression-call budgets? Can the authors isolate the revision stage while keeping serialization, retention, and length control fixed?

2 comments

We thank the reviewer for the careful assessment and constructive questions. Revisions in the updated manuscript are marked in blue. A new Table 5 shifts later table numbers by one (e.g., the former Table 9 is now Table 10).

Answer to Behavioral Evaluation Reliability

The concern is whether errors in the automatic event labels could change the method-level comparisons reported in Table 1. We used the existing expert-audit annotations to compare LLM and human labels method by method on the common random audit sample: 40 continuations, with eight continuations and 80 action opportunities per method (new Table 11, Appendix E).

Table 11 (compact view). Each cell shows LLM → human consensus, as confirmed events / sampled opportunities.

MethodUnsupported task directionTask-constraint conflictHistorical-premise violationAction-purpose mismatch
Native1/8 → 1/80/8 → 0/80/8 → 0/80/80 → 0/80
AgentDiet1/8 → 1/81/8 → 1/80/8 → 0/80/80 → 2/80
Masking2/8 → 1/80/8 → 0/80/8 → 0/82/80 → 2/80
ACON-UTCO2/8 → 1/80/8 → 1/80/8 → 0/81/80 → 1/80
StateCarry1/8 → 0/80/8 → 0/80/8 → 0/81/80 → 3/80

Human consensus requires both independent reviewers to confirm an event; all sampled opportunities remain in the denominators.

On this common sample, the human labels reproduce the comparison pattern stated in Table 1. Under both label sources, StateCarry has the lowest rates on the three continuation-level measures (including ties) and is not the lowest on action-purpose mismatch. It is also the only method with no human-confirmed event on any continuation-level measure. Across methods, LLM and human counts differ by at most one event on each continuation-level measure and by at most two events on action-purpose mismatch.

Action-purpose mismatch is rare under both label sources, with at most 3/80 events per method. Human counts exceed LLM counts only for AgentDiet (0 → 2) and StateCarry (1 → 3). These four events are the ones the LLM labeled uncertain, which accounts for the 4/8 recall, while all four LLM-confirmed events in the sample were also confirmed by humans. These differences therefore come from cases the LLM left uncertain, not from LLM-confirmed events that humans rejected. Uncertain labels remain in the Table 1 denominators and are reported through its definite-judgment coverage.

The 46.9% precision has a different scope. It is computed over all 32 LLM-confirmed action-purpose mismatches across the 300 continuations, most of which entered the audit through targeted review rather than the random sample (Table 10). Table 11 instead covers every opportunity in the same random continuations for all five methods, which provides the basis for comparing methods. Table 1 keeps the automatic labels so that all methods are scored by the same judge and protocol. Section 5.3 now also reports StateCarry's human-consensus rates on this sample.

Answer to Revision Contribution and Comparison Fairness

The concern is whether StateCarry's retention gain comes from its revision strategy, or from a longer handoff and the additional source reading during revision. Matching both Native's handoff length and its single-call budget would remove StateCarry's two-stage design altogether, so we separate the two factors with a new control, SC draft + native (Section 5.5, Table 3, and Appendices C.4 and I.2.3). It reuses the same processed history and frozen SC draft, and replaces only the revision stage with Native-style summarization instructions and output format. Relative to full StateCarry, the two-stage call structure, source access, serialization, and the surrounding retention, installation, and length-control procedures remain fixed. In contrast, the earlier Tool comp. + native variant replaced the entire draft–revision–length module.

Results on 104 shared questions across 20 checkpoints (Table 3):

MethodStrictMean input tokens
Native57.22%25,259
SC draft + native64.36%25,392
StateCarry73.46%36,518

At nearly identical mean input length, the control retains a 7.14-point Strict gain over Native (5.03 points in Partial-credit), so handoff length alone does not explain the gain. With the same two-stage calls and source access, full StateCarry adds a further 9.10 points over the control. This difference therefore reflects the revision stage and the longer handoff it produces, not additional source reading.

On the compression-call budget, StateCarry's drafting and revision stages use 18.26M tokens across the 20 end-to-end tasks, compared with 5.32M compaction tokens for Native. Total agent-token usage nevertheless remains nearly equal (661.63M vs. 661.91M), because StateCarry uses 9.27% fewer execution tokens (Tables 13 and 14).

Thanks for the response, and I will keep my rating at 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.