acceptodds
Under review as a conference paper at ICLR 2027

Beyond Summarization: Semantic Context Handoff with On-Demand Evidence for Long-Horizon Agents

Abstract

Traditional long-horizon LLM agents often hand off context through full-history prompting, retaining all previous action-observation interactions. As execution proceeds, this history keeps growing, increasing inference cost and potentially degrading accuracy. Existing methods control this growth by preserving compact conclusions while removing earlier observations, but doing so can discard evidence that later becomes relevant. We introduce Semantic Context Handoff (SCH), whose central abstraction is state-evidence separation: a compact semantic state carries the agent's current interpretation, while supporting action-observation evidence remains separately addressable and can re-enter context when later execution makes it relevant, without separate summarization or retrieval-model calls. Empirically, SCH improves long-horizon agent performance while substantially reducing context cost. In a controlled 150-task SWE-bench study, SCH's evidence recovery improves Pass@1 from 80.0% to 86.9% at essentially unchanged token use and execution time. In the complete-system comparison, SCH achieves 86.9% Pass@1 while using 43% fewer tokens and 41% less wall-clock time than the strongest baseline by task performance. Across models and benchmarks, SCH consistently reduces token use relative to full-history prompting while remaining competitive with or improving upon the evaluated baselines. Trajectory analysis confirms that SCH recovers evidence from substantially earlier parts of the interaction history rather than merely preserving a recent context window.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-horizon agents, agent context management

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wJZROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Semantic Context Handoff, a context management method for long horizon LLM agents. SCH separates the agent's current semantic state from historical evidence, maintaining task understanding in a compact semantic state while storing action-observation records in a separate evidence archive. At each step, the agent generates a sparse state update and the next action, while a deterministic controller manages the state and selects relevant evidence. The authors further introduce Revision-Aware Evidence Recovery, which retrieves historical evidence when previous conclusions or completed tasks need to be reconsidered, without additional LLM calls. Experiments on SWE bench Verified and AppWorld show that SCH improves task performance while significantly reducing token consumption and runtime. Controlled ablations demonstrate that evidence recovery improves Pass@1 from 80.0% to 86.9% with nearly unchanged token usage, while trajectory analysis confirms the recovery of evidence from earlier execution stages.

Critical strengths
  1. The authors conduct well-controlled ablation studies to evaluate Revision-Aware Evidence Recovery. By keeping the semantic state, controller, action interface, and evidence budget fixed while varying only the historical evidence selection strategy, the experiments more directly isolate the contribution of the proposed mechanism. The results show that SCH improves pass@1 from 80.0% to 86.9% compared with active-state provenance, with almost no increase in token consumption or runtime. The authors also conduct three repeated runs and confirm the statistical significance of the improvements through paired statistical tests. These experiments provide convincing evidence for the effectiveness of Revision-Aware Evidence Recovery.

  2. SCH substantially reduces token consumption and runtime while maintaining strong task performance. In the primary experiments, SCH reduces average token consumption by 43.3% and runtime by 40.8% compared with PACT, the strongest baseline in terms of task performance, while maintaining a competitive pass@1. By maintaining a compact Semantic State and recovering historical evidence only when needed, the method avoids repeatedly processing large amounts of historical information at every execution step. Evidence recovery is also handled by a deterministic controller without additional LLM calls, making the method computationally efficient. Given the large number of interactions required by long-horizon agents, these efficiency improvements have significant practical value.

Critical weaknesses
  1. The method relies heavily on the accuracy of the semantic state and provenance links. According to the paper, SCH relies on the LLM to maintain the semantic state and uses provenance links to associate facts, hypotheses, and work items with historical evidence. However, the paper also states that the controller only checks the structural validity of state deltas and the existence of evidence ID, without verifying whether the semantic state is correct or whether the provenance links actually support the corresponding conclusions. This means that if the LLM incorrectly records a fact, the controller may still accept the update, potentially affecting subsequent execution steps. The authors should provide a more detailed analysis of the error rates of state deltas, the accuracy of provenance links, and how errors in these components affect the final task success rate.

  2. The paper lacks a systematic analysis of long horizon scalability and context budget sensitivity. One of the main goals of SCH is to address context management challenges in long-horizon agents. However, the main experiments limit each task to a maximum of 100 LLM calls and allow at most 12 historical evidence records to be recovered per handoff. The effects of these parameters on performance are not thoroughly analyzed. I am particularly concerned about whether a fixed evidence budget remains sufficient as task horizons become longer. If it does not, the authors should discuss whether there are reliable ways to scale or adapt these settings for longer tasks.

Questions for authors

My questions directly relate to the two weaknesses discussed in the Critical Weaknesses section above:

  1. Could the authors should provide a more detailed analysis of the error rates of state deltas, the accuracy of provenance links, and how errors in these components affect the final task success rate.

  2. How does SCH perform as task horizons become longer? Is the fixed evidence budget still sufficient, and how can it be adapted or scaled for longer tasks?

2 comments

We thank the reviewer for recognizing SCH's controlled ablations, efficiency improvements, and practical value. We have uploaded a revised manuscript with all changes highlighted in blue. The anonymized experimental trajectories are available in Zenodo artifact bundle for verification.

Weakness 1: Semantic-state correctness and provenance reliability

Our initial design relied on the LLM to regenerate the entire semantic state, resulting in substantially lower accuracy. SCH therefore limits the LLM to proposing sparse state deltas, while a deterministic controller maintains the authoritative state, validates updates, and manages provenance references to actual environment observations.

We agree that structurally valid facts may still contain incorrect semantic interpretations. A promising extension is deterministic, script-based verification against recorded evidence. For example, test results, file existence, and command outcomes can be checked directly without another LLM. SCH's state–evidence separation naturally supports such verification.

Q1: What are the state-delta error rates, provenance accuracy, and effects on task success?

We audited 150 available trajectories from our SWE-bench task traces, covering 3,719 SCH handoffs. We identified 56 state-delta parsing failures, 199 fact-gate rejections, 979 effective updates, and 182 fact replacements, with 0 dangling evidence references.

We further examined these events by final task outcome:

Controller eventSuccessful tasksUnsuccessful tasksSuccess with event
Parsing failures53311/13
Fact-gate rejections1762390/100
Effective updates864115127/145
Fact replacements1651765/73

Controller-detected update problems did not necessarily prevent successful execution. Of the 13 tasks experiencing parsing failures, 11 succeeded; similarly, 90 of 100 tasks experiencing fact-gate rejections succeeded.

The audit establishes structural provenance consistency, rather than independent semantic correctness of accepted facts. We will further inspect representative trajectories manually to identify incorrect LLM-generated facts and their influence on subsequent actions.

Weakness 2: Long-horizon scalability and fixed evidence budgets

We agree that additional scalability analysis would strengthen the evaluation. Our trajectory audit found that the 12-step retrieval cap was activated in 65/149 tasks, increasing from 8.2% of handoffs at steps 1–20 to 73.5% at steps 21–40 and 91.4% at steps 41–60.

However, among 18 unsuccessful tasks, only 7 encountered the step cap, and only 2 approached 90% of the 12,000-character budget. Thus, reaching the retrieval cap does not necessarily indicate insufficient useful evidence.

Our experiments allow up to 100 LLM steps, but only 1 task reached this limit. We are searching for benchmarks with longer execution horizons and would appreciate the reviewer's suggestions.

Q2: Is the fixed evidence budget sufficient for longer tasks, and how can it be scaled?

We agree that the 12-step and 12,000-character limits are empirical choices based on preliminary observations. We conducted new sensitivity experiments on the seven originally unsuccessful tasks that encountered the step cap:

Retrieval configurationFailed tasks resolved
Original: 12 steps / 12k chars0/7
24 steps / 12k chars3/7
12 steps / 24k chars3/7
24 steps / 24k chars1/7

We additionally selected 20 previously successful tasks through stratified random sampling. With 24 steps / 12k characters, 16/20 remained successful, while four became unsuccessful.

We also audited character-budget utilization. Maximum rendered evidence reached 11,892 characters (24 steps / 12k), 22,707 (12 steps / 24k), and 20,030 (24 steps / 24k). None reached its configured limit exactly, although some 12k executions exceeded 98% utilization.

These results distinguish evidence capacity from evidence usefulness. Larger budgets can expose additional evidence, but may also introduce redundant information or distribute a fixed character budget across more records. This plausibly explains why increasing retrieval capacity does not consistently improve task success.

Overall, SCH's fixed budget performs effectively within the evaluated horizons, while simply expanding it offers no consistent improvement. For longer tasks, adaptive budgets based on evidence demand and revision requirements can be incorporated into SCH's deterministic selector without changing its core mechanism.

The author's response addressed my concern. I will keep my score.

Reviewer_XF8rOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Semantic Context Handoff (SCH) for managing the context of long-horizon agents. SCH separates a compact representation of the current semantic state from a separately stored archive of action–observation evidence. Through provenance links and work-item lifecycles, it recovers relevant historical evidence when subsequent execution requires revising earlier judgments. The model proposes sparse state updates alongside its next action, while a deterministic controller maintains the state and selects evidence without additional summarization or retrieval-model calls. The authors evaluate SCH on SWE-bench Verified and AppWorld using two model families. In the primary 150-task SWE-bench setting, revision-aware recovery improves mean Pass@1 from 80.00% to 86.89% compared with recovery based only on active-state provenance. The complete system reduces token consumption by approximately 43.3% and runtime by 40.8% relative to PACT.

Critical strengths
  1. The method addresses the concrete problem of revisiting earlier judgments after context compression, with relatively direct support from controlled ablations. SCH preserves links between current judgments and original execution evidence, allowing evidence that has left the routine context to be recovered when the associated work is reconsidered. Table 1 holds state management, the action interface, and the evidence budget fixed while varying only the evidence-selection policy. Full SCH improves Pass@1 by 6.89 percentage points over the variant using only active-state provenance, with statistically significant paired results in all three repeated runs. This is the strongest evidence supporting the effectiveness of its core mechanism.

  2. The primary experiments demonstrate practically meaningful efficiency gains. Compared with PACT, the baseline with the highest success rate in the primary setting, SCH reduces mean token consumption from 367.9k to 208.6k and runtime from 400.2 to 237.1 seconds while maintaining competitive task success. Although its accuracy advantage has not been established as statistically significant, the resource savings are valuable in their own right. Results on additional models and environments also provide some evidence of transfer.

Critical weaknesses
  1. The ablations do not fully disentangle the value of accessing evidence from completed work from the value of the revision-aware triggering policy. Table 1 shows that allowing recovery of completed-work provenance during revision outperforms restricting recovery to active-state evidence. However, it lacks a control with the same evidence budget in which completed-work evidence is eligible for retrieval at every turn. The current results therefore do not fully establish whether the gains primarily arise from expanding the available evidence pool or from the proposed revision detection and lifecycle-based selection mechanism. Since the latter is an important technical distinction from generic historical retrieval, this gap affects the assessment of the paper's incremental contribution.

  2. Baseline adaptations and controller differences in the complete-system comparison are insufficiently specified, limiting attribution of the gains. PACT and ACON use adapted implementations, while MAGE uses a paper-guided implementation. SCH also includes structured-state validation, format recovery, and a heuristic prompt that encourages the agent to proceed with implementation. The paper explicitly states that these factors are held fixed in the evidence-selection ablations, but does not establish equally clearly whether the complete-system baselines share these mechanisms. Consequently, the efficiency advantage supports the practical value of the overall system, but does not establish how much of the improvement comes from state–evidence separation versus other controller designs or differences in baseline adaptation. This uncertainty affects the assessment of its advantages over existing methods.

Questions for authors
  1. [Request for evidence supporting the core claim] Under the same state management, ranking rules, and evidence budget, how would performance change if completed-work provenance were always eligible for retrieval, without requiring a revision trigger? This comparison would help distinguish the contribution of retaining access to historical evidence from that of the revision-aware selection mechanism, and would directly affect my assessment of the method's incremental contribution.

  2. [Clarification of the experimental setup] Do all baselines in the complete-system comparison share the same action interface, retry and format-recovery mechanisms, and anti-overresearch hint? If there are differences, please list them explicitly and explain how they could affect success rate, the number of model calls, and runtime.

  3. [Clarification of the method] When a new observation contains a failure signal, how exactly does the controller determine which completed work items are affected? When multiple completed items could be relevant, how is their evidence ranked and truncated? Please specify the deterministic rules, since this mapping determines the selectivity of revision-aware recovery and is necessary for understanding and reproducing the core mechanism.

3 comments

We thank the reviewer for recognizing the controlled ablations and practical efficiency gains of SCH. We have uploaded a revised manuscript with all changes highlighted in blue. The anonymized experimental trajectories are available in Zenodo artifact bundle for verification.

Q1. [Request for evidence supporting the core claim]

We conducted the requested always-on completed-work provenance ablation on the 150 SWE-bench Verified tasks. We preserve SCH's semantic-state management, evidence-ranking rules, and retrieval budgets, changing only whether completed-work provenance requires a revision trigger. We report the new ablation alongside the original three-run SCH results.

MethodPass@1Tokens/taskTime/task
SCH86.89 ± 1.39%208.6 ± 5.8k237.1 ± 7.4s
Always-on provenance81.33%207.2k298.6s

SCH achieves 5.56 percentage points higher Pass@1 than always-on provenance when compared using the reported three-run mean. A paired comparison against SCH Run 1 (87.33%) also confirms a significant improvement (exact McNemar p=0.0352).

Always-on retrieval consumes fewer tokens and makes slightly fewer model calls (23.9 vs. 25.0 in SCH), yet takes longer and achieves lower accuracy. Their average retrieved evidence sizes are nearly identical (6,179 vs. 6,158 characters). This suggests that the timing and relevance of evidence retrieval matter more than simply expanding evidence eligibility. Always-on retrieval can introduce completed-work evidence unrelated to the current objective, potentially distracting the model and prolonging reconsideration. SCH instead reactivates such evidence when revision is indicated.

This findings are consistent with our original ablations results: Full History achieves only 68.00% Pass@1 with 1,713.3k tokens/task and 633.6s/task, whereas active-state provenance achieves 80.00% with just 209.8k tokens/task and 233.8s/task. Thus, state–evidence separation substantially reduces token consumption and execution time even without completed-work recovery, while revision-aware triggering further improves accuracy.

Q2. [Clarification of the experimental setup]

All compared methods (SCH, PACT, ACON, and MAGE) are integrated into the same SWE-agent framework, sharing the same action interface, tool execution, retry and format-recovery mechanisms, and evaluation protocol.

The anti-overresearch hint is a conditional safeguard triggered only after 12 recognized source inspections since the last repository edit, when a grounded diagnosis exists and the current phase is implementation or verification. It provides no additional task information, solutions, or execution capabilities; it only asks the agent to proceed with implementation rather than continue unnecessary inspection. The same policy is applied consistently across all compared methods and held fixed in our controlled ablations. Therefore, it does not provide SCH with preferential guidance or introduce an unfair experimental advantage.

Q3. [Clarification of the method]

Revision detection and affected-work selection. The controller triggers revision when (1) newly observed failure evidence is detected while revisable semantic facts exist, or (2) an accepted state delta retracts a fact or reopens completed work. Rather than inferring a causal relationship between the failure and a particular completed item, the controller makes the support and lineage evidence of retained completed work eligible for recovery. The subsequent deterministic selection procedure determines which evidence is actually replayed.

Evidence ranking and truncation. Candidates from active-state provenance and, upon revision, completed-work provenance are merged with recent, orphan, and safety-related evidence. Up to six of the most recent eligible provenance steps receive revision-replay priority. Evidence records are then ranked lexicographically by direct support > revision replay > lineage > recent trace > orphan evidence > safety evidence, with active-state term overlap and recency resolving ties. Orphan retrieval uses fixed path/term matching rules. Duplicate and already-pending evidence is excluded, and the highest-ranked records are selected under 12 evidence steps and 12,000 characters, with bounded excerpts.

Location in the paper. The revision triggers and recovery workflow are described in Section 3.3 and Algorithm 1 (p. 5). Appendix D.5–D.6 (p. 16) details candidate eligibility, orphan matching, deduplication, and replay budgets. The exact ranking implementation and trajectory audit metadata are available in our anonymous code repository.

I would like to thank the authors sincerely for the detailed rebuttal. Some of my concerns are clear. I have raised the score to reflect this. However, I think this paper still needs a major revision and currently remains below the acceptance bar of ICLR.

We sincerely thank the reviewer for considering our rebuttal and updating the rating.

We appreciate that some of your concerns have been addressed. Given that we have conducted the requested controlled ablation and provided clarifications regarding the experimental setup and methodology, could you kindly specify the remaining concerns and which aspects of the paper still require major revision?

This would help us better understand the remaining weaknesses and further improve our work.

Thank you again for your time and consideration.

Reviewer_kRC8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Semantic Context Handoff (SCH), a context-management framework for long-horizon LLM agents that separates compact semantic state from historical action–observation evidence. Instead of retaining full interaction histories or relying on potentially lossy summaries, SCH maintains a structured, mutable representation of task progress, hypotheses, and work status, while preserving the underlying evidence in an external archive. A deterministic controller updates the state through model-generated sparse deltas and selectively retrieves historical evidence. The authors evaluate the framework on SWE-bench Verified and AppWorld to demonstrate higher observed task success and reduced token consumption than baselines like PACT.

Critical strengths

The experiment is well designed. The authors go beyond complete-system comparisons by holding the semantic-state machinery and evidence budgets fixed while varying retrieval strategies. The improvement over active-state provenance, supported by paired statistical tests across three runs, provides meaningful evidence for the proposed revision-aware recovery mechanism. The additional efficiency comparisons and trajectory analyses strengthen the empirical contribution.

Critical weaknesses

My primary concern is the paper's limited novelty relative to recent research on long-horizon agent memory and context management. The paper presents state–evidence separation and revision-aware recovery as its main contributions, but the underlying ideas have already been investigated in several closely related works.

For example:

  • Hindsight (Latimer et al., ACL 2026) explicitly distinguishes observations, experiences, objective facts, and subjective beliefs through structured memory networks. It also supports retain, recall, and reflect operations that allow agents to retrieve historical information and update their beliefs. This substantially overlaps with SCH's conceptual motivation of separating current semantic interpretation from underlying evidence and supporting subsequent revision.
  • OCR-Memory (Li et al., ACL 2026) directly addresses the limitations of irreversible summarization by preserving high-fidelity historical trajectories and selectively recovering original evidence when needed. Although OCR-Memory uses a different optical representation and retrieval mechanism, its motivation and objective of maintaining compact active context without sacrificing historical evidence recoverability are highly similar to SCH.
  • COMPASS (Wan et al., ACL 2026) addresses long-horizon context management through a hierarchical framework that separates agent execution, strategic monitoring, and context organization. Although the implementation differs from SCH, the central motivation of maintaining relevant semantic context while enabling long-horizon reflection and revision is closely related.

These papers demonstrate that much of its motivating conceptual framework has already been established. SCH's distinguishing contribution appears primarily to be a particular combination of harness design. I am not convinced that this combination constitutes a sufficiently substantial methodological advance for ICLR.

Questions for authors

How does SCH fundamentally differ from prior systems such as Hindsight, COMPASS, and OCR-Memory, beyond specific implementation choices? In particular, what new capability does state–evidence separation provide that cannot be achieved through existing structured-memory representations, evolving context management, or selective historical evidence retrieval?

1 comment

We thank the reviewer for recognizing the controlled experiments and highlighting related work. We agree that structured memory and historical recovery are established ideas. SCH is not merely a combination of existing memory components or harness heuristics. Its contribution lies in the joint design of a provenance-bearing semantic execution state and a revision-conditioned evidence-selection algorithm. We have uploaded a revised manuscript with all changes highlighted in blue. The anonymized experimental trajectories are available in Zenodo artifact bundle for verification.

1. What distinguishes SCH methodologically?

SCH's semantic state explicitly represents revisable facts/hypotheses, work-item lifecycles (open, blocked, completed), and provenance links to canonical action–observation evidence. Unlike general-purpose memory summaries, these fields directly govern evidence eligibility. The acting model proposes sparse updates, while a deterministic controller validates and maintains authoritative state.

During ordinary execution, provenance associated with active facts and unfinished work remains eligible. When execution triggers revision—through a failure involving revisable state or an accepted fact removal/work reopening—the controller additionally reactivates completed-work provenance, selecting evidence under fixed budgets.

This is more than harness engineering. SCH defines a state-dependent evidence-selection algorithm with explicit inputs, eligibility rules, and bounded outputs. Its semantic-state representation and recovery policy are jointly designed: the former maintains lifecycle and provenance dependencies that the latter uses to determine when completed-work evidence becomes eligible again. This mechanism is distinct from parser recovery, prompt formatting, and other harness-specific heuristics.

2. Relationship to prior work

  • Hindsight maintains structured memories and evolving beliefs through retain/recall/reflect operations. SCH instead uses execution-state lifecycles and provenance to deterministically control completed-work evidence eligibility, without separate retrieval-model calls.
  • OCR-Memory preserves high-fidelity historical trajectories for selective recovery. SCH addresses when previously completed-work evidence should become eligible again, based on execution-state revision.
  • COMPASS organizes execution, strategic monitoring, and context management hierarchically. SCH instead couples explicit work lifecycles with deterministic evidence recovery alongside a single acting model.

To our knowledge, these systems do not explicitly implement SCH's joint lifecycle–provenance eligibility mechanism. We do not claim they could not be extended to support similar behavior.

3. New experimental evidence

We conducted three additional experiments on the frozen 150-task SWE-bench Verified subset, using our original reported SCH run as the reference.

MethodPass@1 ↑Acting calls/task ↓Acting tokens/task ↓Time/task ↓
SCH (ours)87.33%25.0209.3k238.4s
Always-on completed provenance81.33%23.9207.2k298.6s
SCH + Hindsight Recall76.67%26.4225.9k472.0s
Hindsight-managed context61.33%40.6231.4k945.9s

The always-on ablation directly isolates revision-conditioned eligibility. It preserves identical semantic-state machinery, controller, ranking rules, and retrieval budgets, but makes completed-work provenance continuously eligible. Performance decreases from 87.33% to 81.33% (paired exact McNemar p=0.0352p=0.0352), despite comparable token consumption and identical evidence limits (12 steps/12,000 characters).

Thus, the improvement cannot be explained simply by making historical evidence available or increasing context consumption; the revision-conditioned selection policy itself contributes to effectiveness.

We also evaluated two Hindsight adaptations within SWE-agent. SCH + Hindsight Recall achieves 76.67%, while Hindsight-managed context achieves 61.33%. They introduce 35.3 and 72.4 retain/recall operations per task, respectively. These are matched adaptations, not exact reproductions of Hindsight's original setting; acting-token totals exclude auxiliary memory-model consumption.

4. What new capability does SCH provide?

SCH explicitly connects execution failure → revision of previously completed work → recovery of original supporting evidence. Its semantic state is not merely a compressed interaction record: lifecycle and provenance relationships serve as operational inputs to deterministic evidence selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.