- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the problem of context management for long horizon LLM agents, where long interaction histories can introduce unnecessary tokens and make it difficult for agents to maintain important task information. The authors propose carryon, an object oriented context management framework that shifts the focus from compressing historical trajectories to maintaining reusable task progress. The framework introduces continuation carriers, execution context, and recent interaction information to preserve important state while removing redundant history. Experiments on OSWorld 2.0 and BrowseComp Plus show that carryon improves agent performance while reducing token consumption and inference cost.
If the authors can address my concerns in the rebuttal, I would consider increasing my positive assessment and rating
Critical strengths
The main strength is the clear conceptual shift from history centric context compression to object oriented task state management. Instead of treating all previous interactions as equally useful information, the paper identifies that long horizon agents should preserve reusable task objects, existing progress, and remaining requirements. This provides a more meaningful abstraction for context management and helps explain why simple context shortening may not be sufficient.
The proposed framework is practically motivated and well aligned with real agent deployment scenarios. The introduction of continuation carriers, execution context, and live interaction traces provides a structured way to maintain task continuity without repeatedly processing large amounts of historical information.
The experimental validation is another important strength. The authors evaluate the method on osworld 2.0 and Browsecomp plus, showing improvements in both task performance and efficiency. The ablation studies further support that the gains are not only from reducing context length, but also from preserving the right types of information during long horizon execution.
The 40-task comparison includes CarryOn-Guided Summary, an object-aware summary rather than just truncation. With the same host model, trigger, and step budget, CarryOn's combined runtime reaches 55.93 partial score versus 50.93 for that control. This is a meaningful practical comparator, although it does not isolate which runtime component causes the difference.
Critical weaknesses
The main weakness is that the technical novelty compared with existing agent memory and context management approaches is not fully convincing. While the proposed object oriented view is intuitive and useful, some components are related to previous ideas such as structured memory, summarization, and state tracking. The paper would benefit from stronger comparisons with more recent context management approaches to better demonstrate the unique contribution of carryon.
The evaluation is somewhat limited in terms of model and agent diversity. Although the experiments demonstrate improvements on the selected benchmarks, it remains unclear whether the proposed framework generalizes well to different base models, agent architectures, and more open ended environments. Additional experiments with diverse agent settings would make the conclusions stronger.
The paper lists the 40 task IDs and calls the subset random, but does not give a reproducible sampling procedure, paired task outcomes, or uncertainty for the five-point partial-score gap. This limits precision of the subset comparison, not the separate full-cohort result.
Another concern is that the paper does not fully isolate how much improvement comes from the proposed representation itself versus the additional processing introduced by the tracker and compiler components. Since carryon introduces extra reasoning steps for maintaining context state, a more detailed compute matched comparison would help clarify the source of the performance gains.
Questions for authors
-
How does carryon fundamentally differ from recent structured memory and context management approaches, especially those that also maintain task states or compressed summaries instead of raw interaction history?
-
Can the authors provide experiments with additional base models or agent frameworks to demonstrate that the improvements are not specific to the current model and implementation choices?
-
Since carryon introduces tracker and compiler modules, how much additional computation or latency do these components introduce during execution? Have the authors compared the method under a compute matched setting?
-
How sensitive is the performance of carryon to the quality of detected continuation carriers? For example, what happens when the tracker incorrectly identifies an object as reusable progress or misses an important task state?
-
The current evaluation focuses on owsorld 2.0 and browsecomp plus. Do the authors have evidence that the proposed object oriented context representation can generalize to other long horizon settings such as software engineering agents or embodied agents?
-
I am curious whether the authors have tried evaluating the method with more recent frontier models (e.g. GPT-6 or other latest LLMs). I am not suggesting that additional large-scale experiments should be conducted during the rebuttal period, as such evaluations may require substantial resources and time. I would mainly like to understand whether the authors have any preliminary observations regarding how the proposed method performs with newer model generation
5 comments
Official Comment by Authors (2/5)
Q2. Generalization across host models and newer model generations
We additionally conduct experiments with the newer frontier model GPT-6.1 to provide a more complete empirical assessment of CarryOn’s effectiveness across models and frameworks. We compare GPT-6.1 with and without CarryOn on the same OSWorld 2.0 task subset, holding the agent’s reasoning settings, environment-action interface, and 500-step execution budget constant within the comparison. We report total token usage and model cost for the full system, including both the host agent and CarryOn.
| Method | Partial score (%) ↑ | Binary score (%) ↑ | Total tokens/task ↓ | Total cost/task ↓ |
|---|---|---|---|---|
| GPT-6.1 | 57.7 | 30.0 | 9.98M | $2.83 |
| GPT-6.1 + CarryOn | 62.3 | 32.5 | 8.10M (−18.9%) | $2.57 (−9.3%) |
With GPT-6.1 as the host, CarryOn improves partial score by 4.6 pp and binary completion by 2.5 pp while reducing both total token use and model cost. We made only the adaptations needed to integrate CarryOn with GPT-6.1. Since input and output token usage and pricing vary across models, tailoring where CarryOn places retained context in the prompt could yield further savings. These results show that CarryOn can generalize to other hosts, including newer frontier model.
Q3. Reproducibility and uncertainty of the 40-task comparison
The subset is fixed by the 40 task IDs in Appendix B.1, and all context-management comparisons and host-visibility ablations use these same 40 tasks. OSWorld 2.0 annotates every task with non-exclusive tags for ten challenge phenomena [5]. We sampled the subset at random under the requirement that it cover all ten phenomena and its composition closely follows that of the full set, so we can ensure the resulting subset is representative of the full benchmark.
Following the reviewer's suggestions, we provide a more detailed comparison on the shared 40-task subset to show how the aggregate gains arise across individual tasks. We report wins, losses, and ties from CarryOn's perspective, along with 95% bootstrap confidence intervals for the mean partial-score differences.
Table A. CarryOn vs. comparison methods
| CarryOn vs. | Partial-score gain (pp) | 95% paired bootstrap CI (pp) | Win / Loss / Tie |
|---|---|---|---|
| Sliding Window (SW) | +18.52 | [+10.40, +27.61] | 18 / 1 / 21 |
| Generic Summary (GS) | +11.00 | [+4.00, +19.15] | 15 / 3 / 22 |
| CarryOn-Guided Summary (CGS) | +5.00 | [+0.09, +10.10] | 12 / 5 / 23 |
| Default baseline | +5.61 | [−2.60, +14.61] | 10 / 8 / 22 |
CarryOn's mean partial-score gains range from 5.00 to 18.52 pp, and the paired bootstrap confidence intervals support its gains over the context-compression methods. For scale, successive increases in Opus 5's reasoning effort on the official OSWorld 2.0 leaderboard (v2026.08.08) yielded smaller gains of 0.64–3.88 pp while using 28.7–40.9% more output tokens per task [6]. On this scale, CarryOn's score gains in Table A are substantial.
Table B. Host-visibility ablation
| Full CarryOn vs. | Partial-score gain (pp) | 95% paired bootstrap CI (pp) | Win / Loss / Tie |
|---|---|---|---|
| w/o Execution Context | +11.94 | [+5.14, +19.64] | 14 / 1 / 25 |
| w/o Continuation Carriers | +11.31 | [+2.61, +21.06] | 16 / 4 / 20 |
Hiding either the execution context or the continuation carriers reduces mean partial score by more than 11 pp, indicating that both help the host fulfill task requirements.
References
[5] Yuan et al. OSWorld 2.0: Benchmarking Computer-Use Agents on Long-Horizon Real-World Tasks. arXiv, 2026. Section 2.2.2, Table 2.
[6] OSWorld 2.0. Official leaderboard. Release v2026.08.08.
Official Comment by Authors (3/5)
Q4. Additional computation, latency, and compute-matched comparisons
The baseline and CarryOn use the same host model, reasoning effort, environment-action interface and execution budget. All of the usage of the CarryOn components is counted in the tokens and cost we report (Table 2; Appendix B.3). With this overhead included, CarryOn uses 51.9% fewer tokens and 35.9% lower model cost per task than the baseline. The comparison is therefore not one in which CarryOn receives a larger compute budget since the baseline consumes about twice the tokens under the same setting.
| Per task | Baseline | with CarryOn |
|---|---|---|
| Host tokens (M) | 40.10 | 18.61 |
| Tracker/Compiler tokens (M) | – | 0.69 |
| Total tokens (M) | 40.10 | 19.30 (−51.9%) |
| Host cost | $36.83 | $20.51 |
| Tracker/Compiler cost | – | $3.11 |
| Total cost | $36.83 | $23.62 (−35.9%) |
Following the reviewer's suggestion, we further break down the share of tokens and model cost attributable to each component.
| Configuration | Component | Token share | Model cost share |
|---|---|---|---|
| Baseline | Host | 100% | 100% |
| CarryOn | Host | 96.4% | 86.8% |
| CarryOn | Tracker | 2.3% | 6.7% |
| CarryOn | Compiler | 1.3% | 6.5% |
| CarryOn | Maintenance subtotal | 3.6% | 13.2% |
Resource use. Our accounting includes all maintenance calls (Appendix B.3). These calls account for 3.6% of CarryOn's tokens and 13.2% of its model cost. Under the same execution budget, the savings in host usage more than offset the maintenance overhead. Adding CarryOn to the host improves task scores.
We also thank you for asking about latency. Based on our run logs, CarryOn adds 5 minutes per task on average (82 vs. 77 min). Median task duration is 78 min with CarryOn and 79 min for the baseline. When CarryOn is triggered, the maintenance calls and the host's subsequent processing of the refreshed context can add latency.
Overall, the comparison uses matched host settings and execution budgets. With all maintenance usage included, CarryOn achieves higher task scores with lower total token use and model cost.
Official Comment by Authors (1/5)
We thank the reviewer for the positive assessment and the constructive suggestions. We appreciate the recognition of CarryOn’s organizing principle. Below, we clarify the distinction from related approaches and address model generalization, subset reproducibility, runtime overhead, carrier quality, and broader applicability.
Q1. How does CarryOn differ from existing published approaches?
The central idea of CarryOn is that task objects already embody progress the agent can directly build on (Section 1 and Figure 1, main paper). Rather than reconstructing progress primarily from interaction histories, CarryOn organizes context around reusable task objects, including persistent work products, source-grounded findings, and application states.
Beyond this organizing principle, CarryOn incrementally maintains versioned carrier records across folds, updating affected objects while preserving unchanged records. It jointly updates the execution context to retain unresolved goals, constraints, and pending work (Section 3). This differs from existing approaches in how progress is represented and maintained for continuation.
How CarryOn fits into the host workflow
Comparison with Published Methods. To strengthen our empirical evaluation, we additionally implemented and evaluated HiAgent [2], a representative published working-memory framework that supports execution-state management by organizing interactions around subgoals to further evaluate CarryOn against a published method.
Experiment Setup. We evaluate HiAgent using the same Claude Opus 4.7 backbone, environment-action interface, shared 40-task subset, and 500-step execution budget. We also include all auxiliary model calls in the reported resource usage to keep the all the settings the same with our CarryOn.
Compared with HiAgent, CarryOn achieves +11.79 pp partial score and +10.00 pp binary success, while reducing tokens by 37.7% and cost by 20.2%, suggesting that organizing context around reusable task objects is well suited to the diverse, complex, long-horizon agentic tasks.
References
[1] Kang et al. ACON: Optimizing Context Compression for Long-horizon LLM Agents. ICML 2026.
[2] Hu et al. HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model. ACL 2025.
[3] Chen et al. Beyond Semantic Organization: Memory as Execution State Management for Long-Horizon Agents. EMNLP 2026.
[4] Yu et al. InfiAgent: An Infinite-Horizon Framework for General-Purpose Autonomous Agents. Findings of ACL 2026.