- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Semantic Context Handoff, a context management method for long horizon LLM agents. SCH separates the agent's current semantic state from historical evidence, maintaining task understanding in a compact semantic state while storing action-observation records in a separate evidence archive. At each step, the agent generates a sparse state update and the next action, while a deterministic controller manages the state and selects relevant evidence. The authors further introduce Revision-Aware Evidence Recovery, which retrieves historical evidence when previous conclusions or completed tasks need to be reconsidered, without additional LLM calls. Experiments on SWE bench Verified and AppWorld show that SCH improves task performance while significantly reducing token consumption and runtime. Controlled ablations demonstrate that evidence recovery improves Pass@1 from 80.0% to 86.9% with nearly unchanged token usage, while trajectory analysis confirms the recovery of evidence from earlier execution stages.
Critical strengths
-
The authors conduct well-controlled ablation studies to evaluate Revision-Aware Evidence Recovery. By keeping the semantic state, controller, action interface, and evidence budget fixed while varying only the historical evidence selection strategy, the experiments more directly isolate the contribution of the proposed mechanism. The results show that SCH improves pass@1 from 80.0% to 86.9% compared with active-state provenance, with almost no increase in token consumption or runtime. The authors also conduct three repeated runs and confirm the statistical significance of the improvements through paired statistical tests. These experiments provide convincing evidence for the effectiveness of Revision-Aware Evidence Recovery.
-
SCH substantially reduces token consumption and runtime while maintaining strong task performance. In the primary experiments, SCH reduces average token consumption by 43.3% and runtime by 40.8% compared with PACT, the strongest baseline in terms of task performance, while maintaining a competitive pass@1. By maintaining a compact Semantic State and recovering historical evidence only when needed, the method avoids repeatedly processing large amounts of historical information at every execution step. Evidence recovery is also handled by a deterministic controller without additional LLM calls, making the method computationally efficient. Given the large number of interactions required by long-horizon agents, these efficiency improvements have significant practical value.
Critical weaknesses
-
The method relies heavily on the accuracy of the semantic state and provenance links. According to the paper, SCH relies on the LLM to maintain the semantic state and uses provenance links to associate facts, hypotheses, and work items with historical evidence. However, the paper also states that the controller only checks the structural validity of state deltas and the existence of evidence ID, without verifying whether the semantic state is correct or whether the provenance links actually support the corresponding conclusions. This means that if the LLM incorrectly records a fact, the controller may still accept the update, potentially affecting subsequent execution steps. The authors should provide a more detailed analysis of the error rates of state deltas, the accuracy of provenance links, and how errors in these components affect the final task success rate.
-
The paper lacks a systematic analysis of long horizon scalability and context budget sensitivity. One of the main goals of SCH is to address context management challenges in long-horizon agents. However, the main experiments limit each task to a maximum of 100 LLM calls and allow at most 12 historical evidence records to be recovered per handoff. The effects of these parameters on performance are not thoroughly analyzed. I am particularly concerned about whether a fixed evidence budget remains sufficient as task horizons become longer. If it does not, the authors should discuss whether there are reliable ways to scale or adapt these settings for longer tasks.
Questions for authors
My questions directly relate to the two weaknesses discussed in the Critical Weaknesses section above:
-
Could the authors should provide a more detailed analysis of the error rates of state deltas, the accuracy of provenance links, and how errors in these components affect the final task success rate.
-
How does SCH perform as task horizons become longer? Is the fixed evidence budget still sufficient, and how can it be adapted or scaled for longer tasks?
2 comments
The author's response addressed my concern. I will keep my score.
We thank the reviewer for recognizing SCH's controlled ablations, efficiency improvements, and practical value. We have uploaded a revised manuscript with all changes highlighted in blue. The anonymized experimental trajectories are available in Zenodo artifact bundle for verification.
Weakness 1: Semantic-state correctness and provenance reliability
Our initial design relied on the LLM to regenerate the entire semantic state, resulting in substantially lower accuracy. SCH therefore limits the LLM to proposing sparse state deltas, while a deterministic controller maintains the authoritative state, validates updates, and manages provenance references to actual environment observations.
We agree that structurally valid facts may still contain incorrect semantic interpretations. A promising extension is deterministic, script-based verification against recorded evidence. For example, test results, file existence, and command outcomes can be checked directly without another LLM. SCH's state–evidence separation naturally supports such verification.
Q1: What are the state-delta error rates, provenance accuracy, and effects on task success?
We audited 150 available trajectories from our SWE-bench task traces, covering 3,719 SCH handoffs. We identified 56 state-delta parsing failures, 199 fact-gate rejections, 979 effective updates, and 182 fact replacements, with 0 dangling evidence references.
We further examined these events by final task outcome:
Controller-detected update problems did not necessarily prevent successful execution. Of the 13 tasks experiencing parsing failures, 11 succeeded; similarly, 90 of 100 tasks experiencing fact-gate rejections succeeded.
The audit establishes structural provenance consistency, rather than independent semantic correctness of accepted facts. We will further inspect representative trajectories manually to identify incorrect LLM-generated facts and their influence on subsequent actions.
Weakness 2: Long-horizon scalability and fixed evidence budgets
We agree that additional scalability analysis would strengthen the evaluation. Our trajectory audit found that the 12-step retrieval cap was activated in 65/149 tasks, increasing from 8.2% of handoffs at steps 1–20 to 73.5% at steps 21–40 and 91.4% at steps 41–60.
However, among 18 unsuccessful tasks, only 7 encountered the step cap, and only 2 approached 90% of the 12,000-character budget. Thus, reaching the retrieval cap does not necessarily indicate insufficient useful evidence.
Our experiments allow up to 100 LLM steps, but only 1 task reached this limit. We are searching for benchmarks with longer execution horizons and would appreciate the reviewer's suggestions.
Q2: Is the fixed evidence budget sufficient for longer tasks, and how can it be scaled?
We agree that the 12-step and 12,000-character limits are empirical choices based on preliminary observations. We conducted new sensitivity experiments on the seven originally unsuccessful tasks that encountered the step cap:
We additionally selected 20 previously successful tasks through stratified random sampling. With 24 steps / 12k characters, 16/20 remained successful, while four became unsuccessful.
We also audited character-budget utilization. Maximum rendered evidence reached 11,892 characters (24 steps / 12k), 22,707 (12 steps / 24k), and 20,030 (24 steps / 24k). None reached its configured limit exactly, although some 12k executions exceeded 98% utilization.
These results distinguish evidence capacity from evidence usefulness. Larger budgets can expose additional evidence, but may also introduce redundant information or distribute a fixed character budget across more records. This plausibly explains why increasing retrieval capacity does not consistently improve task success.
Overall, SCH's fixed budget performs effectively within the evaluated horizons, while simply expanding it offers no consistent improvement. For longer tasks, adaptive budgets based on evidence demand and revision requirements can be incorporated into SCH's deterministic selector without changing its core mechanism.