acceptodds
Under review as a conference paper at ICLR 2027

CUAWright: A Minimal Unified Interface for Computer Use Agents

21 views

Abstract

The prevailing approach to computer-use agents couples a model with a domain-specific harness: a browser or desktop environment equipped with human engineered tools that are fixed before task execution. As models’ coding capabilities improve, the GUI native and static harness prevents them from direct programmatic operation on system state, as well as flexible construction of tools. To this end, we introduce CUAWright, a minimal terminal harness of roughly 3K lines of code that uses bash commands as its sole action interface, and a file system as its evolvable space for dynamically creating tools and managing the context. We conduct comprehensive experiments across a wide range of digital tasks, and demonstrate that by giving the agent a minimal, programmable interface, it achieves substantially stronger results compared to their GUI or hybrid CLI interface across a wide range of tasks. On OSWorld 2.0, CUAWright delivers a 33.2% relative improvement in partial reward while reducing estimated cost by 37.5% compared with the published GPT-5.5 baseline. On Online-Mind2Web and the long horizon Odysseys benchmark, CUAWright substantially outperforms GUI native harness by 4.7% and 44.0% in success rate, respectively. Furthermore, we found the gains extend to CAD applications that require accurate visual understanding and CLI interaction: on CADGenBench and BenchCAD, our unified harness yields 8.1%-41.6% relative improvements over other CLI based harnesses with GPT-5.5. Together, these results suggest digital environments are far more programmable than their GUI interfaces imply, and a minimal terminal-focused harness is the key for better performance and efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, computer use agents, programmable interfaces, desktop agents, web agents, filesystem as LLM memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ogYkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CUAWright is a minimal terminal harness ($\sim$3K lines of code) in which bash commands are the sole action interface and the filesystem is an evolvable workspace for managing context and creating tools dynamically. Screenshots are taken on demand rather than at every step, and keyboard/mouse control via xdotool/pyautogui is kept as a fallback. The method is evaluated on six web, desktop and CAD benchmarks.

Critical strengths

Large, consistent gains across diverse benchmarks. CUAWright clearly beats same-model baselines on six web, desktop and CAD benchmarks, and at lower cost on OSWorld. Gains this large are unlikely to be noise.

Critical weaknesses

The "beyond APIs" claim is weakly supported. The paper claims to go beyond prior work by handling settings without APIs, but most experiments still rely on programmatic interfaces: web tasks are driven by agent-written code against the browser debugging protocol rather than step-by-step coordinate prediction (Sec. 3.1), and both CAD tasks use CadQuery/Build123d. OSWorld is the main setting that truly needs a GUI, and it is where results are weakest: 71.4%71.4\% of tasks still use keyboard/mouse input (Table 6), and the strict gain drops from +11.8+11.8 to +0.3+0.3 with the stronger GPT-5.6 Sol (Table 2). This is the paper's main novelty over prior work, yet the evidence is weakest exactly where it should be strongest.

Questions for authors
  1. Cost vs. terminal-based harnesses (Clarification). The cost comparison is only made against the official GUI-native harnesses on OSWorld (Table 2), not against any terminal-based harness. On WeaveBench and CadGenBench, what is CUAWright's cost per task compared with terminal-based harnesses such as Codex CLI and Hermes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ogYk (1/2)

We sincerely thank the reviewer for the time and effort for reviewing our paper, and for the thoughtful and constructive comments. We are also grateful for the recognition of CUAWright’s consistent gains across diverse benchmarks. Below, we address the concern about the “beyond APIs” claim, including the OSWorld results, and report new controlled experiments comparing CUAWright with terminal-based harnesses in both success rate and cost.

W1 (a) — “The "beyond APIs" claim is weakly supported. The paper claims to go beyond prior work by handling settings without APIs, but most experiments still rely on programmatic interfaces: web tasks are driven by agent-written code against the browser debugging protocol rather than step-by-step coordinate prediction (Sec. 3.1), and both CAD tasks use CadQuery/Build123d.”

We thank the reviewer for this insightful comment. We realize that the phrase “beyond APIs” may suggest that CUAWright aims to avoid APIs altogether, but it is not our claim. Rather, our claim is that CUAWright adopts general-purpose terminal commands as a unified action interface, integrating GUI interactions via xdotool, existing APIs, and other shell commands. This design allows CUAWright to leverage APIs when available without being constrained by predefined, domain-specific APIs.

In fact, section 3.6 illustrates how agents dynamically create and reuse tools during execution, extending their capabilities beyond a predefined set of APIs.. In the response to reviewer NiSq (2/2) we included the results that shows constributions of dynamic tool creation and file system memory to the agent performance. Disabling filesystem memory and disabling dynamically created tools leads to 3.45 and 7.52 pts loss to the performance in partial score. Both of which are beyond simple predefined API tools. We will revise related parts in the introduction and Line 142 to make this explicit [e.g., replacing “beyond APIs” with “beyond predefined action spaces”], and we will carefully revise the rest of the manuscript for clarity.

Under this design, the agent is free to use existing APIs but is not limited to a fixed set of them. It can write and execute code, interact with the environment and filesystem, and construct task-specific tools as needed. Using CDP/Playwright for web tasks or CadQuery/Build123d for CAD tasks is therefore consistent with our design. These are libraries the agent leverages programmatically, not restrictions imposed by the harness. For the CAD tasks, CadQuery/Build123d code is the output format required by the benchmarks themselves, so all baselines operate under the same constraint. For the web tasks, the agent adopted CDP on its own; our prompts do not instruct it to be restricted to specific tools.

W1(b) — Evidence on OSWorld “OSWorld is the main setting that truly needs a GUI, and it is where results are weakest: 71.4% of tasks still use keyboard/mouse input (Table 6), and the strict gain drops from +11.8 to +0.3 with the stronger GPT-5.6 Sol (Table 2).” We agree that OSWorld is the most demanding test of GUI interaction in our evaluation, and we appreciate the reviewer’s careful reading of Tables 2 and 6. We would like to offer some additional context on both tables.

Table 2: Narrower success gain, but growing cost savings with stronger models

  • Regarding Table 2, we acknowledge that the performance gain comparing GUI only and CUAWright approach narrows from +15.7 partial score with GPT-5.5 to +5.2 with the stronger GPT-5.6 Sol model. However, the average cost reduction per task actually increases significantly from 6.0 dollar to 9.5 dollar per task (37.3% to 68.3% cost reduction from GPT 5.5 to GPT 5.6 sol).

  • We observed that stronger models increasingly favor terminal-based interactions, even in GUI-only settings, where they open terminals through mouse clicks and type commands via the GUI. This incurs unnecessary overhead from repeated GUI interactions and screenshots attached at every step. CUAWright eliminates this overhead through direct terminal access, delivering greater cost savings as model capabilities improve.

AuthorsOpenReview ↗

Response to Reviewer ogYk (2/2)

Q1 — “Cost vs. terminal-based harnesses (Clarification). The cost comparison is only made against the official GUI-native harnesses on OSWorld (Table 2), not against any terminal-based harness.” Question — “On WeaveBench and CadGenBench, what is CUAWright's cost per task compared with terminal-based harnesses such as Codex CLI and Hermes?”

We thank the reviewer for raising this important question. We agree that comparing inference costs against terminal-based harnesses is important. To directly address this concern, we conducted two additional controlled experiments comparing both task success and inference cost.

1. WeaveBench (GPT-5.5 High, new experiment)

Harness (GPT-5.5 High)Overall score ↑Cost/Task ↓
Codex CLI47.0%$4.18
CUAWright66.4%$3.66

Since the original WeaveBench paper does not report inference costs, we additionally reproduced the Codex CLI baseline using the authors' released code under the same controlled setting, including cost analysis. We show that CUAWright achieves +19.4 percentage points in overall score while reducing inference cost by 12.4% compared with Codex CLI. This demonstrates that CUAWright's benefits extend beyond comparisons with GUI-native harnesses to existing terminal-based agents.

2. OSWorld 2.0 (GPT-5.6 Sol Max, new experiment).

On OSWorld 2.0, the closest related terminal-driven approach is the concurrent work StateAct (Yang et al., 2026). However, as its code and implementation have not been publicly released, a direct controlled comparison is currently infeasible. As a best-effort alternative, we conducted an additional experiment comparing CUAWright against a StateAct-like hybrid harness under matched model and task settings. The difference here is that in StateAct it delegates the GUI action to a specifically trained GUI model which is not released, where in our case, we delegate it to the same actor model which is GPT-5.6 sol max.

HarnessPartial score ↑Cost/Task ↓
StateAct Hybrid65.9%$5.52
CUAWright67.9%$4.40

CUAWright achieves +2.0 percentage points in partial score with 20.3% lower inference cost. Overall, we sincerely hope these additional experiments provide evidence that CUAWright improves both task success and cost-efficiency compared with other terminal based coding harnesses and GUI+shell hybrid approaches under controlled settings.

We hope these results and clarifications are helpful, and we would be grateful if the reviewer would consider them when re-evaluating the score. We will include these additional analyses and clarify their implications in the revised manuscript.

Reviewer_NiSqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CUAWright, a terminal-native harness that uses bash commands as the primary action interface for digital agents. Agents can directly manipulate system state, create reusable tools, manage filesystem context, and invoke GUI actions when necessary. Experiments across web, desktop, and CAD benchmarks report improvements over existing GUI-based and hybrid harnesses in task performance and efficiency.

Critical strengths
  1. The proposed system is simple and practically motivated. A terminal-native interface allows agents to combine file operations, scripts, and GUI automation without relying exclusively on predefined interaction primitives.

  2. The framework supports flexible tool creation and reuse. Using the filesystem as a persistent workspace provides a straightforward mechanism for maintaining context and constructing task-specific automation.

  3. The experiments cover diverse applications. Evaluations across web, desktop, long-horizon workflows, and CAD tasks provide useful evidence that programmatic interaction can improve agent efficiency beyond conventional GUI execution.

Critical weaknesses
  1. CodeAct already introduces executable code as a unified and extensible action space. Recent computer-use agents, like ToolCUA, also combine GUI interaction with programmatic operations. CUAWright's main distinction is making bash the default interface and using the filesystem for context management. The paper does not clearly establish which capabilities are fundamentally new beyond this harness design.

  2. The main comparisons do not sufficiently isolate the benefits of the terminal-native interface. Several headline results rely on previously published baselines rather than matched evaluations. The systems also differ in available capabilities, including filesystem manipulation, direct application-state access, and dynamically created tools. These differences make it difficult to attribute performance improvements specifically to the proposed interface.

  3. The component ablations are insufficient for supporting the central architectural claims. For example, disabling filesystem access simultaneously removes several important capabilities, including persistent context, direct file editing, and tool construction. The observed degradation therefore does not establish which component is responsible for the improvement or whether an equivalent hybrid harness could achieve similar results.

Questions for authors

What is the fundamental difference between CUAWright and CodeAct-style executable-action agents?

Can the authors compare against matched GUI/hybrid harnesses with equivalent filesystem access and tool-construction capabilities?

Can the authors separately ablate filesystem memory, direct state manipulation, and dynamically created tools?

3 comments
AuthorsOpenReview ↗

Response to Reviewer NiSq (1/2)

We sincerely thank the reviewer for the time spent on our submission and for the constructive, concrete suggestions. We are glad the reviewer found CUAWright simple and practically motivated, appreciated its support for flexible tool creation and reuse, and found the evaluation diverse. The reviewer’s questions about attribution prompted several new experiments, which we believe have made the paper stronger

W1 — “CodeAct already introduces executable code as a unified and extensible action space. Recent computer-use agents, like ToolCUA, also combine GUI interaction with programmatic operations. CUAWright's main distinction is making bash the default interface and using the filesystem for context management.” Question — “What is the fundamental difference between CUAWright and CodeAct-style executable-action agents?”

We thank the reviewer for highlighting CodeAct and ToolCUA. We agree that executable code as an action space is not new, and our contribution is not the introduction of code-based actions. Instead, CUAWright investigates a different question: Can a minimal, terminal-native interface replace domain-specific computer-use harnesses while improving generality, performance, and efficiency?

The key distinctions are:

  • Unified execution interface vs. executable actions. CodeAct establishes that executable Python can unify tool invocation and composition. CUAWright builds on this insight but extends it to end-to-end interaction with real digital environments, including browsers, desktop applications, and CAD systems. Rather than exposing code execution as an additional capability alongside predefined environment-specific tools, CUAWright uses the terminal as the sole interface through which all operations—including GUI interactions—are performed.
  • Dynamic tool creation and reuse. Rather than relying primarily on a predefined set of tools, CUAWright allows the agent to construct, modify, and reuse tools through the terminal during task execution.
  • Filesystem-based context management. Beyond code execution, CUAWright treats the filesystem as part of the agent's persistent state, allowing it to externalize context, retain intermediate artifacts, and create, debug, and reuse tools across interaction steps. This is particularly important for long-horizon computer-use tasks. Our ablation on OSWorld 2.0 shows that restricting programmatic filesystem operations decreases the partial score from 67.9% to 55.5% (−12.4 points), demonstrating the practical importance of this capability. In the revision, we will rewrite the contributions paragraph with the lines above and add an explicit comparison with CodeAct and ToolCUA to the related work.

W2 — “The main comparisons do not sufficiently isolate the benefits of the terminal-native interface. Several headline results rely on previously published baselines rather than matched evaluations.” Question — “Can the authors compare against matched GUI/hybrid harnesses with equivalent filesystem access and tool-construction capabilities?”

We thank the reviewer for this suggestion. In the revision, we will clearly mark which baselines are reported and which are reproduced. In addition, we provide two controlled comparisons that directly address this concern:

  • Matched coding-agent harnesses (WeaveBench, Table 2 and new result).

Since the Codex CLI baseline in the original WeaveBench paper does not report its inference costs. Aside from the existing result in table 2, we additionally evaluated the Codex CLI baseline using the authors' released code under the same controlled setting, including cost analysis. We note that the Codex CLI baseline is equipped with equivalent filesystem access and tool-construction capabilities. We show that CUAWright achieves +19.4 percentage points in overall score with 12.4% lower cost, demonstrating substantial improvements under matched model and tool-access conditions.

HarnessOverall score ↑Cost/Task ↓
Codex CLI47.0%$4.18
CUAWright66.4%$3.66
  • GUI+shell hybrid harness StateAct (OSWorld 2.0, new result).

To directly evaluate the benefit of the terminal-native interface over a hybrid design with equivalent filesystem access and tool-construction capabilities, we additionally conducted a controlled comparison with StateAct. Since StateAct's GUI model is not released, we swap the GUI specialist model in StateAct by the same actor model which is GPT-5.6 sol max.

HarnessPartial score ↑Cost/Task ↓
StateAct Hybrid65.9%$5.52
CUAWright67.9%$4.40

We show that CUAWright achieves +2.0 percentage points in partial score and significantly (20.3%) lower cost. Together, we hope these results demonstrate that CUAWright's terminal-native design improves both performance and cost-efficiency over existing coding-agent and GUI+shell hybrid harnesses.

AuthorsOpenReview ↗

Response to Reviewer NiSq (2/2)

W3 — “The component ablations are insufficient for supporting the central architectural claims. For example, disabling filesystem access simultaneously removes several important capabilities, including persistent context, direct file editing, and tool construction.” Question — “Can the authors separately ablate filesystem memory, direct state manipulation, and dynamically created tools?”

We thank the reviewer for this valuable suggestion. We agree that disabling filesystem access simultaneously removes multiple capabilities, making it difficult to isolate their individual contributions. To address this concern, we conducted additional ablation experiments that separately disables filesystem memory and dynamically created tools, while preserving the remaining capabilities.

ConfigurationPartial scorePasses
Full system (baseline)67.9%29/105
Without filesystem memory64.5%26/105
Without dynamic tool construction60.4%23/105
Without filesystem access55.5%16/105

The results reveal three key observations:

  • Filesystem memory: Disabling filesystem memory reduces the adjusted partial score by 3.45 percentage points (67.90% → 64.45%), with successful tasks decreasing from 29 to 26 out of 105. This suggests that persistent filesystem memory contributes to agent performance.
  • Dynamic tool construction: Disabling dynamically created tools leads to a larger performance degradation of 7.52 percentage points (67.90% → 60.38%), with successful tasks decreasing from 29 to 23 out of 105. This indicates that dynamically constructed tools and their reuse during the task, play an important role in enabling effective task execution.
  • Filesystem access and Direct file editing: Removing filesystem access yields a partial score of 55.50%, substantially below the full-system baseline. Since direct file editing is closely tied to filesystem access in our architecture, we use the original no-filesystem ablation reported in the paper to characterize its contribution. The substantial performance drop highlights the importance of direct filesystem interaction, which enables agents to efficiently inspect and modify application state. Overall, these additional experiments provide more fine-grained evidence for the contributions of filesystem memory and dynamic tool construction.

We are grateful for the reviewer’s thoughtful and constructive feedback, which led directly to the matched comparisons and finer-grained ablations above. In the revision, we will:

  • state our contribution more precisely and add an explicit comparison with CodeAct and ToolCUA;
  • clearly mark reported versus reproduced baselines;
  • include the new hybrid comparison and ablations;

We hope these changes address the reviewer’s concerns, and we would be glad to clarify anything further.

AuthorsOpenReview ↗

Additional Controlled Comparison with CodeAct

To further address the reviewer's concern regarding the distinction between CUAWright and CodeAct style executable action agents, we conducted an additional controlled experiment comparing CUAWright directly against CodeAct [1] using the same GPT-5.4 model and the Online Mind2Web benchmark setting.

AgentEasyMediumHardTotal
CodeAct GPT-5.488.9%77.8%55.6%74.1%
CUAWright GPT-5.490.3%86.5%88.8%88.1%

CUAWright outperforms CodeAct by 14.0 percentage points overall, with particularly substantial improvements on hard tasks (+33.2 points) and medium tasks (+8.7 points). These results provide direct empirical evidence that CUAWright's benefits extend beyond simply adopting executable code as an action space. While CodeAct establishes the effectiveness of code-based actions, CUAWright further unifies programmatic operations, GUI interactions, filesystem-based context management, and dynamic tool construction within a minimal terminal-native harness.

We believe this additional controlled comparison addresses the reviewer's concern and further strengthens our empirical findings. We respectfully hope the reviewer will consider these new results when reassessing their evaluation.

[1] Wang et al., "Executable Code Actions Elicit Better LLM Agents," ICML 2024 (arXiv:2402.01030).

Reviewer_URbHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CUAWright, which uses terminal commands as a unified interface for web, desktop and CAD tasks. The agent uses the filesystem to keep information and build tools. Experiments show improvements across several benchmarks, with lower reported API cost on OSWorld.

Critical strengths
  • The experiments cover several kinds of tasks and include comparisons with existing coding agents. The improvement over Codex CLI on WeaveBench is quite clear and gives useful evidence.
  • The framework is simple and the code is public. It provides a useful baseline for further research on computer-use agents.
Critical weaknesses
  • The main idea is close to StateAct and Terminal Agents Suffice. The broader evaluation is useful, but the additional benefit of simplifying the design is not directly compared with these closely related methods.
  • BenchCAD results are compared on different subsets. The no-tool comparison gives some support, but it is still not enough to determine the exact improvement under tool setting.
  • The filesystem ablation shows its overall usefulness, but does not separate direct file editing, persistent memory and tool reuse. The explanation of where the improvement comes from is still limited.
Questions for authors
  • Are there results comparing CUAWright with StateAct or similar methods under same model and same budget?
  • Could the authors provide the tool-based baseline results on the same 212 BenchCAD samples?
  • Is there further analysis showing how much persistent memory and tool reuse help beyond direct file editing?
2 comments
AuthorsOpenReview ↗

Response to Reviewer URbH (1/2)

We thank the reviewer for recognizing the breadth of our evaluation, the significant improvements in cost-efficiency and performance, as well as the open-source contributions of CUAWright. We also sincerely appreciate the reviewer’s insightful suggestions regarding the ablation studies. To address these suggestions, we have conducted comprehensive additional experiments to clarify our core contributions and further strengthen our results.

W1 — “The main idea is close to StateAct and Terminal Agents Suffice. The broader evaluation is useful, but the additional benefit of simplifying the design is not directly compared with these closely related methods.”

Question — “Are there results comparing CUAWright with StateAct or similar methods under same model and same budget?”

We respectfully clarify that CUAWright does not claim to be the first terminal-based agent. While our work shares the terminal-first motivation of StateAct and Terminal Agents Suffice (TAS), it investigates a different question: whether a unified, general-purpose command action space, combined with filesystem-based context, can effectively support diverse computer-use tasks spanning live websites, desktop environments, and graphical design applications.

Our key distinction from these closely related works lies in the generality and simplicity of the proposed agent architecture.

  • Comparison with StateAct. StateAct primarily focuses on program-state operations while delegating GUI interactions to a separate specialist agent, which relies on another model to execute mouse and keyboard actions. In contrast, CUAWright unifies both programmatic and GUI interactions within a single terminal interface, powered by the same model, eliminating the need for a separate GUI specialist. To the best of our knowledge, StateAct's implementation and GUI specialist model have not been publicly released, making a controlled comparison under the same model and budget infeasible at present. We would welcome such a comparison once these resources become available.

  • Comparison with Terminal Agents Suffice (TAS). TAS investigates terminal-based agents for enterprise automation in environments where APIs are available. As explicitly stated in their paper: “We investigate whether terminal-based coding agents can serve as the foundation for enterprise automation when APIs are available.” In contrast, CUAWright does not assume the availability of task-specific APIs and targets a broader range of computer-use environments, including arbitrary live websites (e.g., Online-Mind2Web) and desktop applications where API-based interactions alone are often insufficient.

  • Additional GUI+shell hybrid comparison (OSWorld 2.0). Since StateAct's GUI model and the implementation is not publicly available, as a best-effort alternative, we conducted an additional experiment comparing CUAWright against a StateAct-like hybrid harness under matched model and task settings. The difference here is that in StateAct it delegates the GUI action to a specifically trained GUI model which is not released, where in our case, we delegate it to the same actor model which is GPT-5.6 sol max.

HarnessPartial score ↑Cost/Task ↓
StateAct Hybrid65.9%$5.52
CUAWright67.9%$4.40

CUAWright achieves +2.0 percentage points in success rate with 20.3% lower inference cost. This provides direct empirical evidence that a unified terminal-native interface can outperform a hybrid approach while reducing overhead from repeated screenshot processing. Although this is not a direct reproduction of StateAct, it offers a controlled evaluation of the architectural design choice underlying our contribution.

Overall, CUAWright's central contribution is demonstrating that a single, unified terminal-based agent can support both programmatic and GUI interactions across diverse environments, without requiring specialized GUI agents or fixed set of task-specific APIs.

AuthorsOpenReview ↗

Response to Reviewer URbH (2/2)

W2 — “BenchCAD results are compared on different subsets. The no-tool comparison gives some support, but it is still not enough to determine the exact improvement under tool setting.” Question — “Could the authors provide the tool-based baseline results on the same 212 BenchCAD samples?”

We thank the reviewer for this suggestion. We clarify that the GPT-5.5 results in our paper were taken from OpenAI's official blog post, which does not specify the agent harness used for evaluation. To directly address the reviewer's concern, we conducted an additional experiment using the official tool-based harness with GPT-5.5 xhigh on the same 212 BenchCAD samples, ensuring a controlled comparison under the tool-enabled setting.

HarnessOverall ↑Easy ↑Medium ↑Hard ↑
GPT-5.5, Tool-based Harness (New)74.172.789.461.6
GPT-5.5, CUAWright79.078.681.476.9

The results show that CUAWright outperforms the official tool-based harness by 4.9 percentage points overall (79.0 vs. 74.1), with particularly notable improvements on hard tasks (+15.3 points) and easy tasks (+5.9 points). We hope these results provide direct evidence that CUAWright's design offers benefits beyond simply enabling tool use.

W3 — “The filesystem ablation shows its overall usefulness, but does not separate direct file editing, persistent memory and tool reuse. The explanation of where the improvement comes from is still limited.”

Question — “Is there further analysis showing how much persistent memory and tool reuse help beyond direct file editing?”

We thank the reviewer for this valuable suggestion. We agree that the original filesystem ablation can be much more comprehensive by distinguishing the contributions of persistent memory, and dynamic tool reuse. To address this concern, we conducted additional ablation experiments that independently disable filesystem memory and dynamically created tools, while preserving direct file editing capabilities.

ConfigurationPartial scorePasses
CUAWright Full system (baseline)67.9%29/105
Without filesystem memory64.5%26/105
Without dynamic tool construction60.4%23/105
Without filesystem access55.5%16/105

The results show that persistent memory and dynamic tool construction provide additional benefits beyond direct file editing: Persistent memory: Disabling filesystem memory reduces the adjusted partial score by 3.45 percentage points (67.90% → 64.45%), with successful tasks decreasing from 29 to 26 out of 105. Dynamic tool construction and reuse: Disabling dynamically created tools results in a larger reduction of 7.52 percentage points (67.90% → 60.38%), with successful tasks decreasing from 29 to 23 out of 105. This suggests that dynamically created tools provide substantial additional value beyond direct file editing. Filesystem access, which closely supports direct file editing in our architecture, has the largest overall impact, with performance dropping to 55.50% when disabled. Overall, these results provide more fine-grained evidence that independently shows the benefits of persistent memory and dynamic tool construction and reuse. We will include these additional ablations and clarify their implications in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.