acceptodds
Under review as a conference paper at ICLR 2027

Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agents

Abstract

Long-horizon agents accumulate progress in persistent task state, yet repeatedly process interaction histories describing how that progress was achieved. This raises a natural question: can long-horizon context instead be organized around reusable progress rather than the history that produced it? In this paper, we introduce CarryOn, an object-oriented context runtime that organizes continuation around reusable task objects. Rather than treating compressed histories or reconstructed accounts of progress as the primary basis for continuation, CarryOn builds a context view from continuation carriers, complementary execution context, and a live tail of recent interactions. A core challenge is keeping this reusable basis valid after its surrounding history is retired: how can CarryOn update the basis without reconstructing the full representation at every fold? CarryOn maintains carrier records across folds, revising affected records against bound evidence and applicable environment checks while carrying forward unaffected records and jointly updating execution context. Experiments on OSWorld2.0 and BrowseComp-Plus show improved gains of 5.8% and 7.0% while reducing total token usage by 51.9% and 22.9%, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-horizon agent

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_RvVWOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the problem of context management for long horizon LLM agents, where long interaction histories can introduce unnecessary tokens and make it difficult for agents to maintain important task information. The authors propose carryon, an object oriented context management framework that shifts the focus from compressing historical trajectories to maintaining reusable task progress. The framework introduces continuation carriers, execution context, and recent interaction information to preserve important state while removing redundant history. Experiments on OSWorld 2.0 and BrowseComp Plus show that carryon improves agent performance while reducing token consumption and inference cost.

If the authors can address my concerns in the rebuttal, I would consider increasing my positive assessment and rating

Critical strengths

The main strength is the clear conceptual shift from history centric context compression to object oriented task state management. Instead of treating all previous interactions as equally useful information, the paper identifies that long horizon agents should preserve reusable task objects, existing progress, and remaining requirements. This provides a more meaningful abstraction for context management and helps explain why simple context shortening may not be sufficient.

The proposed framework is practically motivated and well aligned with real agent deployment scenarios. The introduction of continuation carriers, execution context, and live interaction traces provides a structured way to maintain task continuity without repeatedly processing large amounts of historical information.

The experimental validation is another important strength. The authors evaluate the method on osworld 2.0 and Browsecomp plus, showing improvements in both task performance and efficiency. The ablation studies further support that the gains are not only from reducing context length, but also from preserving the right types of information during long horizon execution.

The 40-task comparison includes CarryOn-Guided Summary, an object-aware summary rather than just truncation. With the same host model, trigger, and step budget, CarryOn's combined runtime reaches 55.93 partial score versus 50.93 for that control. This is a meaningful practical comparator, although it does not isolate which runtime component causes the difference.

Critical weaknesses

The main weakness is that the technical novelty compared with existing agent memory and context management approaches is not fully convincing. While the proposed object oriented view is intuitive and useful, some components are related to previous ideas such as structured memory, summarization, and state tracking. The paper would benefit from stronger comparisons with more recent context management approaches to better demonstrate the unique contribution of carryon.

The evaluation is somewhat limited in terms of model and agent diversity. Although the experiments demonstrate improvements on the selected benchmarks, it remains unclear whether the proposed framework generalizes well to different base models, agent architectures, and more open ended environments. Additional experiments with diverse agent settings would make the conclusions stronger.

The paper lists the 40 task IDs and calls the subset random, but does not give a reproducible sampling procedure, paired task outcomes, or uncertainty for the five-point partial-score gap. This limits precision of the subset comparison, not the separate full-cohort result.

Another concern is that the paper does not fully isolate how much improvement comes from the proposed representation itself versus the additional processing introduced by the tracker and compiler components. Since carryon introduces extra reasoning steps for maintaining context state, a more detailed compute matched comparison would help clarify the source of the performance gains.

Questions for authors
  1. How does carryon fundamentally differ from recent structured memory and context management approaches, especially those that also maintain task states or compressed summaries instead of raw interaction history?

  2. Can the authors provide experiments with additional base models or agent frameworks to demonstrate that the improvements are not specific to the current model and implementation choices?

  3. Since carryon introduces tracker and compiler modules, how much additional computation or latency do these components introduce during execution? Have the authors compared the method under a compute matched setting?

  4. How sensitive is the performance of carryon to the quality of detected continuation carriers? For example, what happens when the tracker incorrectly identifies an object as reusable progress or misses an important task state?

  5. The current evaluation focuses on owsorld 2.0 and browsecomp plus. Do the authors have evidence that the proposed object oriented context representation can generalize to other long horizon settings such as software engineering agents or embodied agents?

  6. I am curious whether the authors have tried evaluating the method with more recent frontier models (e.g. GPT-6 or other latest LLMs). I am not suggesting that additional large-scale experiments should be conducted during the rebuttal period, as such evaluations may require substantial resources and time. I would mainly like to understand whether the authors have any preliminary observations regarding how the proposed method performs with newer model generation

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/5)

We thank the reviewer for the positive assessment and the constructive suggestions. We appreciate the recognition of CarryOn’s organizing principle. Below, we clarify the distinction from related approaches and address model generalization, subset reproducibility, runtime overhead, carrier quality, and broader applicability.


Q1. How does CarryOn differ from existing published approaches?

The central idea of CarryOn is that task objects already embody progress the agent can directly build on (Section 1 and Figure 1, main paper). Rather than reconstructing progress primarily from interaction histories, CarryOn organizes context around reusable task objects, including persistent work products, source-grounded findings, and application states.

Beyond this organizing principle, CarryOn incrementally maintains versioned carrier records across folds, updating affected objects while preserving unchanged records. It jointly updates the execution context to retain unresolved goals, constraints, and pending work (Section 3). This differs from existing approaches in how progress is represented and maintained for continuation.

ApproachHow it builds context for continuationHow CarryOn differs
ACON [1]Compresses interaction history and observations using guidelines optimized beforehand from successful and failed task trajectories.Builds carriers entirely online from the current run.
HiAgent [2]Prompts the agent to define subgoals, summarizes the interactions for each completed subgoal, and supports retrieval of past trajectories.Reuses objects produced in the host's usual workflow.
MAGE [3]Builds context from an execution-state tree, validates subgoal summaries, and branches at subgoal boundaries.Organizes context around task objects the agent can directly use to continue its work.
InfiAgent [4]Externalizes workspace state and reconstructs context from state snapshots and recent actions.References work products and application state in place and updates only the affected carrier records across folds.

How CarryOn fits into the host workflow

  • Reuse task objects in place. The agent can reopen existing artifacts or continue from application state using its usual tools. CarryOn keeps object references and reuse conditions in context, so the agent can work directly in its existing workspace.
  • Use the host's existing action space and environment interface. CarryOn operates online alongside the host and updates the context it receives.
  • Identify reusable progress as it emerges. CarryOn picks up reusable objects during the agent's usual workflow, without requiring the agent to define explicit subgoals.

Comparison with Published Methods. To strengthen our empirical evaluation, we additionally implemented and evaluated HiAgent [2], a representative published working-memory framework that supports execution-state management by organizing interactions around subgoals to further evaluate CarryOn against a published method.

Experiment Setup. We evaluate HiAgent using the same Claude Opus 4.7 backbone, environment-action interface, shared 40-task subset, and 500-step execution budget. We also include all auxiliary model calls in the reported resource usage to keep the all the settings the same with our CarryOn.

MethodBinary (%) ↑Partial (%) ↑Tokens per task ↓Cost per task ↓
Baseline22.5050.3234.97M$31.93
HiAgent12.5044.1427.49M$25.74
CarryOn22.5055.9317.13M$20.55

Compared with HiAgent, CarryOn achieves +11.79 pp partial score and +10.00 pp binary success, while reducing tokens by 37.7% and cost by 20.2%, suggesting that organizing context around reusable task objects is well suited to the diverse, complex, long-horizon agentic tasks.

References

[1] Kang et al. ACON: Optimizing Context Compression for Long-horizon LLM Agents. ICML 2026.

[2] Hu et al. HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model. ACL 2025.

[3] Chen et al. Beyond Semantic Organization: Memory as Execution State Management for Long-Horizon Agents. EMNLP 2026.

[4] Yu et al. InfiAgent: An Infinite-Horizon Framework for General-Purpose Autonomous Agents. Findings of ACL 2026.

AuthorsOpenReview ↗

Official Comment by Authors (2/5)

Q2. Generalization across host models and newer model generations

We additionally conduct experiments with the newer frontier model GPT-6.1 to provide a more complete empirical assessment of CarryOn’s effectiveness across models and frameworks. We compare GPT-6.1 with and without CarryOn on the same OSWorld 2.0 task subset, holding the agent’s reasoning settings, environment-action interface, and 500-step execution budget constant within the comparison. We report total token usage and model cost for the full system, including both the host agent and CarryOn.

MethodPartial score (%) ↑Binary score (%) ↑Total tokens/task ↓Total cost/task ↓
GPT-6.157.730.09.98M$2.83
GPT-6.1 + CarryOn62.332.58.10M (−18.9%)$2.57 (−9.3%)

With GPT-6.1 as the host, CarryOn improves partial score by 4.6 pp and binary completion by 2.5 pp while reducing both total token use and model cost. We made only the adaptations needed to integrate CarryOn with GPT-6.1. Since input and output token usage and pricing vary across models, tailoring where CarryOn places retained context in the prompt could yield further savings. These results show that CarryOn can generalize to other hosts, including newer frontier model.


Q3. Reproducibility and uncertainty of the 40-task comparison

The subset is fixed by the 40 task IDs in Appendix B.1, and all context-management comparisons and host-visibility ablations use these same 40 tasks. OSWorld 2.0 annotates every task with non-exclusive tags for ten challenge phenomena [5]. We sampled the subset at random under the requirement that it cover all ten phenomena and its composition closely follows that of the full set, so we can ensure the resulting subset is representative of the full benchmark.

Following the reviewer's suggestions, we provide a more detailed comparison on the shared 40-task subset to show how the aggregate gains arise across individual tasks. We report wins, losses, and ties from CarryOn's perspective, along with 95% bootstrap confidence intervals for the mean partial-score differences.

Table A. CarryOn vs. comparison methods

CarryOn vs.Partial-score gain (pp)95% paired bootstrap CI (pp)Win / Loss / Tie
Sliding Window (SW)+18.52[+10.40, +27.61]18 / 1 / 21
Generic Summary (GS)+11.00[+4.00, +19.15]15 / 3 / 22
CarryOn-Guided Summary (CGS)+5.00[+0.09, +10.10]12 / 5 / 23
Default baseline+5.61[−2.60, +14.61]10 / 8 / 22

CarryOn's mean partial-score gains range from 5.00 to 18.52 pp, and the paired bootstrap confidence intervals support its gains over the context-compression methods. For scale, successive increases in Opus 5's reasoning effort on the official OSWorld 2.0 leaderboard (v2026.08.08) yielded smaller gains of 0.64–3.88 pp while using 28.7–40.9% more output tokens per task [6]. On this scale, CarryOn's score gains in Table A are substantial.

Table B. Host-visibility ablation

Full CarryOn vs.Partial-score gain (pp)95% paired bootstrap CI (pp)Win / Loss / Tie
w/o Execution Context+11.94[+5.14, +19.64]14 / 1 / 25
w/o Continuation Carriers+11.31[+2.61, +21.06]16 / 4 / 20

Hiding either the execution context or the continuation carriers reduces mean partial score by more than 11 pp, indicating that both help the host fulfill task requirements.

References

[5] Yuan et al. OSWorld 2.0: Benchmarking Computer-Use Agents on Long-Horizon Real-World Tasks. arXiv, 2026. Section 2.2.2, Table 2.

[6] OSWorld 2.0. Official leaderboard. Release v2026.08.08.

AuthorsOpenReview ↗

Official Comment by Authors (3/5)

Q4. Additional computation, latency, and compute-matched comparisons

The baseline and CarryOn use the same host model, reasoning effort, environment-action interface and execution budget. All of the usage of the CarryOn components is counted in the tokens and cost we report (Table 2; Appendix B.3). With this overhead included, CarryOn uses 51.9% fewer tokens and 35.9% lower model cost per task than the baseline. The comparison is therefore not one in which CarryOn receives a larger compute budget since the baseline consumes about twice the tokens under the same setting.

Per taskBaselinewith CarryOn
Host tokens (M)40.1018.61
Tracker/Compiler tokens (M)–0.69
Total tokens (M)40.1019.30 (−51.9%)
Host cost$36.83$20.51
Tracker/Compiler cost–$3.11
Total cost$36.83$23.62 (−35.9%)

Following the reviewer's suggestion, we further break down the share of tokens and model cost attributable to each component.

ConfigurationComponentToken shareModel cost share
BaselineHost100%100%
CarryOnHost96.4%86.8%
CarryOnTracker2.3%6.7%
CarryOnCompiler1.3%6.5%
CarryOnMaintenance subtotal3.6%13.2%

Resource use. Our accounting includes all maintenance calls (Appendix B.3). These calls account for 3.6% of CarryOn's tokens and 13.2% of its model cost. Under the same execution budget, the savings in host usage more than offset the maintenance overhead. Adding CarryOn to the host improves task scores.

We also thank you for asking about latency. Based on our run logs, CarryOn adds 5 minutes per task on average (82 vs. 77 min). Median task duration is 78 min with CarryOn and 79 min for the baseline. When CarryOn is triggered, the maintenance calls and the host's subsequent processing of the refreshed context can add latency.

Overall, the comparison uses matched host settings and execution budgets. With all maintenance usage included, CarryOn achieves higher task scores with lower total token use and model cost.

Reviewer_Fj5tOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CarryOn, a context-management runtime for long-horizon agents. It replaces older interaction history with structured continuation carriers describing work products, findings, and application state, together with an execution context and recent interaction. A Tracker proposes updates, a Grounder checks environment state, and a Compiler updates the representation before history is removed. Experiments with Claude Opus 4.7 on OSWorld 2.0 and BrowseComp-Plus report lower token usage and cost, alongside modest performance improvements.

Critical strengths
  • CarryOn provides a well-motivated representation for long-horizon work. Anchoring continuation on work products and environment state, rather than only summarizing interaction history, is intuitive and potentially useful. Versioned carriers, grounding checks, and atomic updates are thoughtful design choices.
  • The efficiency results are plausible. The paper reports substantial token and cost reductions while including the Tracker and Compiler overhead in its accounting - highlighting that CarryOn keeps active context bounded during long tasks.
Critical weaknesses

There are confounding factors in the context representation findings. In particular, CarryOn adds Opus-based Tracker and Compiler reasoning as well as Grounder checks for files, hashes, document structure, and instruction-text matches. The baselines do not receive equivalent reasoning or observations, so improvements could come from additional planning or environment access rather than carrier-based context organization. Comparisons are also limited to the authors’ summary and window baselines so there's a lack of comparison against other existing methods in literature like ACON.

Questions for authors
  • Why does removing the carriers raise the binary score on the 40-task subset?
  • What happens when the Grounder is removed, or when all baselines receive the same checks and auxiliary reasoning budget?
  • How does CarryOn compare with stronger published context-management systems, and does it generalize to other host models?
6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for recognizing CarryOn’s representation and runtime design, and for the suggestions on strengthening the evaluation. We address the concerns about auxiliary computation, component contributions, published baselines, and generalization below.

Q1. Reasoning settings and resource budgets

We would like to clarify that our primary comparison evaluates the benefit of adding CarryOn to an existing host agent under the same host configuration and execution budget. The host model, reasoning-effort setting, context-window capacity, environment-action interface, and execution budget are held constant. All additional Tracker and Compiler model calls are included in the reported resource usage. Including this overhead, CarryOn reduces total token usage by 51.9% and cost by 35.9%, while improving partial and binary scores by 5.8 and 3.7 pp (Table 1-2, main paper). These results demonstrate that CarryOn improves task performance while reducing overall inference expenditure, even after accounting for its additional runtime reasoning.

Our summary comparisons further examine the organizing principle and its runtime implementation. CGS applies the object-oriented organizing principles through a replacement summary, raising partial score from 44.94% with GS to 50.93% (Table 4, main paper). CarryOn further implements these principles through explicit maintenance and updating of task objects across folds, increasing partial score to 55.93% while retaining approximately 10% less post-fold text than CGS. These comparisons support the value of object-oriented context organization and the additional benefits of explicitly maintaining this context view throughout execution.


Q2. Why does removing carriers raise binary score over the 40-task subset?

Binary outcomes differ on only three of the 40 tasks. Hiding carriers yields a net gain of one fully completed task (2.5 pp), so the observed increase may be specific to this subset.

TaskFull CarryOn partial scorew/o Carriers partial score
0020.501.00
0400.7051.00
0121.000.00

Binary scoring credits only full completion, so partial scores give a fuller picture of how much of each task the agent completes. Across all 40 paired tasks, full CarryOn achieves higher partial scores on 16 tasks, lower scores on 4, and ties on 20. Making carriers visible improves mean partial score by 11.31 pp, with a 95% paired bootstrap CI of [+2.61, +21.06] pp (Table 5, main paper).

Moreover, hiding carriers increases agent calls by 23.8% and cost by 22.3% (Table 5, main paper). Hiding carriers can make earlier findings unavailable, so the agent may have to look them up again. Task 013 illustrates this behavior (Appendix C.2, Table 10): after carriers were hidden, the agent re-extracted a catalogue PDF and reread previously acquired item prices, whereas full CarryOn continued directly from the retained prize-distribution plan.

Together, these results indicate that continuation carriers improve overall partial task fulfillment and reduce repeated information acquisition.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Q3. Contributions of the CarryOn runtime components

We respectfully argue that the observed gains stem from object-oriented context organization, and we designed CarryOn to better maintain this context view throughout execution. CarryOn maintains the agent's context around reusable task objects, recording where to find them and under what conditions they can be reused, while keeping outstanding requirements explicit. CGS tests this organizing principle through replacement summaries. Compared with Generic Summary, it raises partial score from 44.94% to 50.93% and binary completion from 15.00% to 20.00% on the shared 40-task subset (Table 4, main paper). These results show that organizing context around reusable task objects can improve performance even within a textual summary.

CarryOn further maintains this view across folds through object tracking, evidence binding, joint updates of carriers and execution context, and validated atomic commits. Compared with CGS, CarryOn improves partial score from 50.93% to 55.93% while retaining approximately 10% less post-fold text (Table 4, main paper).

Ablation of Environmental Checks. To assess whether CarryOn's advantage over CGS is primarily driven by additional environmental checks, we evaluated CarryOn with these checks disabled on the same 40-task subset.

MethodBinary (%) ↑Partial (%) ↑
CarryOn (full)22.5055.93
CarryOn w/o evidence check22.5054.81
CGS20.0050.93
Baseline22.5050.32

Disabling environmental checks reduces CarryOn's partial score by only 1.12 pp, with binary completion unchanged. Without these checks, CarryOn still outperforms CGS by 3.88 pp in partial score. In our runtime logs, checks withheld only 15 of 655 compilation requests (2.3%). These results suggest that CarryOn's performance advantage is not primarily driven by environmental checks, but also stems from its broader runtime design for maintaining and updating reusable task objects throughout long-horizon execution.


Q4. Comparison with published context-management systems

We thank the reviewer for suggesting a comparison with published methods such as ACON [1], and we agree that a representative published working-memory method would further strengthen the evaluation.

Comparison with ACON. ACON optimizes compression guidelines offline using success/failure feedback from training trajectories, whereas CarryOn operates entirely online. Evaluating the full ACON method would require separate training tasks, feedback, and an optimization budget; using only its runtime compression would omit this central contribution. We therefore prioritize a method more directly aligned with our online evaluation setting.

Following this suggestion, we additionally implemented and evaluated HiAgent [2], a representative published working-memory framework that supports execution-state management by organizing interactions around subgoals and summarizing completed subgoal histories.

Experiment Setup. We evaluate HiAgent using the same Claude Opus 4.7 backbone, environment-action interface, shared 40-task subset, and 500-step execution budget. We also include all auxiliary model calls in the reported resource usage to keep all the settings the same with our CarryOn.

MethodBinary (%) ↑Partial (%) ↑Tokens per task ↓Cost per task ↓
Baseline22.5050.3234.97M$31.93
HiAgent12.5044.1427.49M$25.74
CarryOn22.5055.9317.13M$20.55

Compared with HiAgent, CarryOn achieves +11.79 pp partial score and +10.00 pp binary success, while reducing tokens by 37.7% and cost by 20.2%, suggesting that organizing context around reusable task objects is well suited to the diverse and complex long-horizon agentic tasks.

References

[1] Kang et al. ACON: Optimizing Context Compression for Long-horizon LLM Agents. ICML 2026.

[2] Hu et al. HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model. ACL 2025.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q5. Generalization to other host models

We additionally conduct experiments with the newer frontier model GPT-6.1 to provide a more complete empirical assessment of CarryOn’s effectiveness across models and frameworks. We compare GPT-6.1 with and without CarryOn on the same OSWorld 2.0 task subset, holding the agent’s reasoning settings, environment-action interface, and 500-step execution budget constant within the comparison. We report total token usage and model cost for the full system, including both the host agent and CarryOn.

MethodPartial score (%) ↑Binary score (%) ↑Total tokens/task ↓Total cost/task ↓
GPT-6.157.730.09.98M$2.83
GPT-6.1 + CarryOn62.332.58.10M (−18.9%)$2.57 (−9.3%)

With GPT-6.1 as the host, CarryOn improves partial score by 4.6 pp and binary completion by 2.5 pp while reducing both total token use and model cost. We made only the adaptations needed to integrate CarryOn with GPT-6.1. Since input and output token usage and pricing vary across models, tailoring where CarryOn places retained context in the prompt could yield further savings.

These results show that CarryOn can generalize to other hosts, including newer frontier model.

Reviewer_PhcaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CarryOn manages context for long-horizon agents by maintaining reusable results, their locations and conditions of use, and outstanding requirements. These records replace lengthy interaction histories to reduce context costs.

Critical strengths
  1. The paper is clearly written, with helpful figures and detailed implementation settings.
  2. It addresses a practical problem: reducing redundant history while preserving results, constraints, and unfinished work.
  3. Reported cost reductions include auxiliary model calls and accompany improved task scores, suggesting practical value.
Critical weaknesses
  1. Limited comparisons. Comparisons cover sliding windows and summary baselines, but not representative published working-memory or execution-state management methods. Please include at least one relevant method using the same backbone, tool access, and comparable resource budgets.
  2. Uncertain performance gains. In Table 4, CarryOn improves partial scores, but its binary score matches the default baseline at 22.50% and exceeds CGS (20.00%) by only one fully completed task out of 40. Please provide paired per-task results, repeated runs, and confidence intervals to assess the reliability of these gains.
  3. Unclear component contributions. CarryOn performs environmental checks, whereas the CGS summarization call makes no new checks. Its advantage may reflect both memory maintenance and additional verification. Please ablate environmental checks and provide CGS with the same check results to isolate their contributions.
Questions for authors

How does CarryOn detect and revise incorrect or outdated records when new evidence contradicts a claim or referenced files and application states change? Please provide experiments or cases showing both successful and unsuccessful recovery.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the constructive feedback and for recognizing the practical value of preserving progress around task objects while reducing total token usage and cost during long execution.


Q1. Comparison with published working-memory methods

To address this concern, we have implemented and evaluated HiAgent[1], a representative published working-memory framework that supports execution-state management by organizing interactions around subgoals and summarizing completed subgoal histories.

Experiment Setup. We evaluate HiAgent on the same 40-task subset of OSWorld 2.0 used in the Table 4 (main paper), with the same Claude Opus 4.7 backbone, environment-action interface, and 500-step execution budget as CarryOn. For both methods, reported token usage and cost include all auxiliary model calls.

MethodBinary (%) ↑Partial (%) ↑Tokens per task ↓Cost per task ↓
Baseline22.5050.3234.97M$31.93
HiAgent12.5044.1427.49M$25.74
CarryOn22.5055.9317.13M$20.55

Results. Compared with HiAgent, CarryOn achieves +11.79 pp partial score and +10.00 pp binary success rate, while using 37.7% fewer tokens and 20.2% lower cost per task, suggesting that organizing context around reusable task objects is well suited to the diverse and complex long-horizon agentic tasks.

References

[1] Hu et al. HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model. ACL 2025.


AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Q2. Performance gains and Reliability

Interpreting Binary Score and Partial Score

We respectfully emphasize that both binary and partial scores are important for evaluating long-horizon agents. Binary completion measures full task success, whereas the fine-grained partial rewards introduced by OSWorld 2.0 [2] capture progress on individual task requirements. The two metrics therefore provide complementary perspectives on performance.

Alongside the partial-score gains in Table 4, CarryOn improves partial and binary scores on the full benchmark by 5.8 pp and 3.7 pp, respectively, while reducing total token use by 51.9% and cost by 35.9% (Tables 1–2, main paper). For perspective, increasing Claude Opus 4.8's reasoning effort from Medium to Xhigh on the official leaderboard increases output-token usage by 64.1%, with gains of only 1.1 pp in partial score and 3.0 pp in binary completion [3]. These results highlight the substantial performance gains achieved alongside improved efficiency introduced by CarryOn.

Repeated Runs

Our evaluation follows the official OSWorld 2.0 scoring protocol [2, 3] and the default one-run-per-task execution pattern of the official OSWorld and OSWorld 2.0 runners [4, 5]. This practice is also adopted in Cradle's OSWorld evaluation [6] and the QuPath-Bench ablations in OSWorld-Science [7]. Across each comparison, we use the same benchmark release, host model, step budget, and evaluator.

To further assess run-to-run variability, we conducted two additional CarryOn runs on the same 40-task subset under identical settings. Across the three runs, partial scores varied by ±1.6 pp, while binary completion remained unchanged, and total token usage and model cost varied by ±3.4% and ±2.0%, respectively. CarryOn consistently maintained its performance and efficiency advantages over the baseline across all three runs.

Paired Per-task Results and Confidence Intervals

To address the reliability of the observed gains, we analyze paired task-level outcomes on the shared 40-task subset. We report wins, losses, and ties from CarryOn's perspective, together with 95% paired bootstrap confidence intervals for the mean partial-score differences.

CarryOn vs.Partial-score gain (pp)95% paired bootstrap CI (pp)Win / Loss / Tie
Sliding Window (SW)+18.52[+10.40, +27.61]18 / 1 / 21
Generic Summary (GS)+11.00[+4.00, +19.15]15 / 3 / 22
CarryOn-Guided Summary (CGS)+5.00[+0.09, +10.10]12 / 5 / 23
Default baseline+5.61[−2.60, +14.61]10 / 8 / 22

On the 40-task subset, mean partial-score gains range from 5.00 to 18.52 pp across the four paired comparisons, and CarryOn maintains its partial-score and efficiency gains over the default baseline across all three runs. These results support the reliability of CarryOn's gains.

References

[2] Yuan et al. OSWorld 2.0: Benchmarking Computer-Use Agents on Long-Horizon Real-World Tasks. arXiv, 2026. Section 2.1.3.

[3] OSWorld 2.0. Official leaderboard. Releases v2026.06.24 and v2026.08.08.

[4] XLANG Lab. OSWorld evaluation runner (run.py). GitHub.

[5] XLANG Lab. OSWorld 2.0 evaluation runner (run.py). GitHub.

[6] Tan et al. Cradle: Empowering Foundation Agents towards General Computer Control. ICML 2025. Appendix C.

[7] Dai et al. OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software. arXiv, 2026. Figure 5.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q3. Ablation of Environmental Checks

To assess whether CarryOn's advantage over CGS is primarily driven by additional environmental checks, we evaluated CarryOn with these checks disabled on the same 40-task subset.

MethodBinary (%) ↑Partial (%) ↑
CarryOn (full)22.5055.93
CarryOn w/o evidence check22.5054.81
CGS20.0050.93
Baseline22.5050.32

Disabling environmental checks reduces CarryOn's partial score by only 1.12 pp, with binary completion unchanged. Without these checks, CarryOn still outperforms CGS by 3.88 pp in partial score. In our runtime logs, checks withheld only 15 of 655 compilation requests (2.3%). These results suggest that CarryOn's performance advantage is not primarily driven by environmental checks, but also stems from its broader runtime design for maintaining and updating reusable task objects throughout long-horizon execution.


Q4. How does CarryOn revise incorrect or outdated records?

CarryOn uses new observations from the host and environment to reassess existing records. During a fold, it can revise or retract an affected claim and make any newly unresolved requirement explicit. The following traces show where this process worked and where a correction remained incomplete.

Successful corrections

TaskRecord beforeNew observationRecord after
Task 012: find the source exam paper for each question. Evidence contradicts a claim.Question 5 tentatively comes from the Dec 2013 paper.Around step 200: that paper's Question 6 is a 15-mark single-hypothesis test, but Question 5 is a 22-mark two-sample test.Fold at 210: the paper is rejected and the source is unidentified again. The agent finds the matching paper at step 239.
Task 040: build the Q4 valuation report from the Q3 report. A referenced file changes.Two rates link to the September file and two are interpolated.By step 203: all four Q4 rates link to the December file.Fold at 212: all four are direct links, and the interpolation no longer applies.
Task 021: buy tickets for three pairs of people. Application state changes.The first pair's seats are selected and on hold.Step 97: the hold times out and the seats are released.Fold at 121: the cart is empty, and all three pairs need new seats.

Unsuccessful recovery

TaskRecord beforeNew observationRecord after
Task 016: email the authors of award-winning papers. Application state changes.Three records leave one author out of the outreach because an example email to this author is in Sent.Steps 319–337: the email is no longer in the mailbox.Fold at 339: one record is corrected, but the other two still leave this author out. The agent notices the conflict at steps 380 and 451, and no later fold resolves it.

These cases show that CarryOn can revise incorrect or outdated records as new evidence arrives during task execution.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank all reviewers for their constructive feedback and recognition of CarryOn’s well-motivated conceptual shift toward object-oriented context management (Fj5t, RvVW), practical value and thoughtful runtime design (Phca, Fj5t, RvVW), and substantial token and cost savings alongside improved task performance (Phca, Fj5t, RvVW).

The main concerns center on (i) comparisons with published context-management methods, (ii) the role of environmental checks and auxiliary reasoning in the reported performance gains, and (iii) generalization across host models. To address these concerns, we conducted additional experiments and analyses:

  1. Comparison with published methods (Phca W1; Fj5t Q3; RvVW W1). We evaluate HiAgent under the same host model, tool interface, and execution budget. Compared with HiAgent, CarryOn achieves +11.79 pp partial score and +10.00 pp binary success, while using 37.7% fewer tokens and 20.2% lower cost.
  2. Contribution of environmental checks (Phca W3; Fj5t Q2). Disabling environmental checks reduced CarryOn’s partial score by 1.12 pp. With the checks disabled, CarryOn still scored 3.88 pp higher than CarryOn-Guided Summary.
  3. Generalization across host models (Fj5t Q3; RvVW W2, Q2, Q6). We evaluate CarryOn with GPT-6.1, obtaining +4.6 pp partial score and +2.5 pp binary success, while reducing total token usage by 18.9%.
  4. Further analyses. We provide repeated runs and paired task-level comparisons with 95% bootstrap confidence intervals (Phca W2; RvVW W3), Tracker/Compiler overhead and latency measurements (Fj5t Q2; RvVW W4, Q3), and examples of successful and incomplete carrier corrections (Phca Q1; RvVW Q4).

Detailed results and discussions are provided in our individual responses. We would appreciate hearing whether these results and clarifications address your concerns, and we welcome any further questions or discussion.

Reviewer_RvVWOpenReview ↗

AI Use Clarification

The text follow is my original, human-written review draft, reproduced exactly as I submitted it to the AI tool, including any grammatical or spelling errors. I used AI to polish the language and check for potential factual errors, inconsistencies, and unclear expressions. The substantive assessment, strengths, weaknesses, questions, and final rating were independently determined by me. The original text is provided here to clarify the scope of AI assistance and demonstrate my independent evaluation of the paper.

Paper Summary This paper look at context management for long-horizon LLM agents. Long histories waste tokens and make it hard to keep key task info. Authors propose CarryOn, an object-oriented framework that focus on reusable task progress instead of compressing old trajectories. Tests on OSWorld 2.0 and BrowseComp Plus show better performance and lower token cost. I may raise my rating if concerns are addressed.

Strengths

  • Clear shift from history compression to object-oriented state management. Explain why simple truncation don’t work well, more meaningful abstraction.
  • Practical design fits real deployment. Structured way to keep task continuity without reprocessing lots of old data.
  • Solid experiments with ablations, showing gains come from right info preservation, not just shorter context.
  • 40-task comparison shows meaningful gap (55.93 vs 50.93), though it don’t isolate which component cause the difference.

Weaknesses

  • Technical novelty over existing memory methods not fully convincing. Some overlap with prior structured memory work. Need stronger comparison with recent approaches.
  • Limited model and agent diversity. Unclear if it generelize to other models or open environments.
  • 40-task subset called random but no reproducible sampling or uncertainity for the score gap.
  • Doesn’t separate gains from the representation itself vs extra processing from tracker and compiller modules.

Questions

  1. How is CarryOn different from recent structured memory methods that also track task states?
  2. Can you test on more base models to show improvements aren’t specific to your setup?
  3. How much extra compute/latency do tracker and compiler add? Any compute-matched comparison?
  4. How sensitive is performance to carrier detection quality? What if tracker misidentifies states?
  5. Any evidence it works on other long-horizon settings like embodied agents?
  6. Tried newer models like GPT-6? Just want preliminary observations, not new experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.