- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CUAWright is a minimal terminal harness ($\sim$3K lines of code) in which bash commands are the sole action interface and the filesystem is an evolvable workspace for managing context and creating tools dynamically. Screenshots are taken on demand rather than at every step, and keyboard/mouse control via xdotool/pyautogui is kept as a fallback. The method is evaluated on six web, desktop and CAD benchmarks.
Critical strengths
Large, consistent gains across diverse benchmarks. CUAWright clearly beats same-model baselines on six web, desktop and CAD benchmarks, and at lower cost on OSWorld. Gains this large are unlikely to be noise.
Critical weaknesses
The "beyond APIs" claim is weakly supported. The paper claims to go beyond prior work by handling settings without APIs, but most experiments still rely on programmatic interfaces: web tasks are driven by agent-written code against the browser debugging protocol rather than step-by-step coordinate prediction (Sec. 3.1), and both CAD tasks use CadQuery/Build123d. OSWorld is the main setting that truly needs a GUI, and it is where results are weakest: of tasks still use keyboard/mouse input (Table 6), and the strict gain drops from to with the stronger GPT-5.6 Sol (Table 2). This is the paper's main novelty over prior work, yet the evidence is weakest exactly where it should be strongest.
Questions for authors
- Cost vs. terminal-based harnesses (Clarification). The cost comparison is only made against the official GUI-native harnesses on OSWorld (Table 2), not against any terminal-based harness. On WeaveBench and CadGenBench, what is CUAWright's cost per task compared with terminal-based harnesses such as Codex CLI and Hermes?
2 comments
Response to Reviewer ogYk (2/2)
Q1 — “Cost vs. terminal-based harnesses (Clarification). The cost comparison is only made against the official GUI-native harnesses on OSWorld (Table 2), not against any terminal-based harness.” Question — “On WeaveBench and CadGenBench, what is CUAWright's cost per task compared with terminal-based harnesses such as Codex CLI and Hermes?”
We thank the reviewer for raising this important question. We agree that comparing inference costs against terminal-based harnesses is important. To directly address this concern, we conducted two additional controlled experiments comparing both task success and inference cost.
1. WeaveBench (GPT-5.5 High, new experiment)
| Harness (GPT-5.5 High) | Overall score ↑ | Cost/Task ↓ |
|---|---|---|
| Codex CLI | 47.0% | $4.18 |
| CUAWright | 66.4% | $3.66 |
Since the original WeaveBench paper does not report inference costs, we additionally reproduced the Codex CLI baseline using the authors' released code under the same controlled setting, including cost analysis. We show that CUAWright achieves +19.4 percentage points in overall score while reducing inference cost by 12.4% compared with Codex CLI. This demonstrates that CUAWright's benefits extend beyond comparisons with GUI-native harnesses to existing terminal-based agents.
2. OSWorld 2.0 (GPT-5.6 Sol Max, new experiment).
On OSWorld 2.0, the closest related terminal-driven approach is the concurrent work StateAct (Yang et al., 2026). However, as its code and implementation have not been publicly released, a direct controlled comparison is currently infeasible. As a best-effort alternative, we conducted an additional experiment comparing CUAWright against a StateAct-like hybrid harness under matched model and task settings. The difference here is that in StateAct it delegates the GUI action to a specifically trained GUI model which is not released, where in our case, we delegate it to the same actor model which is GPT-5.6 sol max.
| Harness | Partial score ↑ | Cost/Task ↓ |
|---|---|---|
| StateAct Hybrid | 65.9% | $5.52 |
| CUAWright | 67.9% | $4.40 |
CUAWright achieves +2.0 percentage points in partial score with 20.3% lower inference cost. Overall, we sincerely hope these additional experiments provide evidence that CUAWright improves both task success and cost-efficiency compared with other terminal based coding harnesses and GUI+shell hybrid approaches under controlled settings.
We hope these results and clarifications are helpful, and we would be grateful if the reviewer would consider them when re-evaluating the score. We will include these additional analyses and clarify their implications in the revised manuscript.
Response to Reviewer ogYk (1/2)
We sincerely thank the reviewer for the time and effort for reviewing our paper, and for the thoughtful and constructive comments. We are also grateful for the recognition of CUAWright’s consistent gains across diverse benchmarks. Below, we address the concern about the “beyond APIs” claim, including the OSWorld results, and report new controlled experiments comparing CUAWright with terminal-based harnesses in both success rate and cost.
We thank the reviewer for this insightful comment. We realize that the phrase “beyond APIs” may suggest that CUAWright aims to avoid APIs altogether, but it is not our claim. Rather, our claim is that CUAWright adopts general-purpose terminal commands as a unified action interface, integrating GUI interactions via xdotool, existing APIs, and other shell commands. This design allows CUAWright to leverage APIs when available without being constrained by predefined, domain-specific APIs.
In fact, section 3.6 illustrates how agents dynamically create and reuse tools during execution, extending their capabilities beyond a predefined set of APIs.. In the response to reviewer NiSq (2/2) we included the results that shows constributions of dynamic tool creation and file system memory to the agent performance. Disabling filesystem memory and disabling dynamically created tools leads to 3.45 and 7.52 pts loss to the performance in partial score. Both of which are beyond simple predefined API tools. We will revise related parts in the introduction and Line 142 to make this explicit [e.g., replacing “beyond APIs” with “beyond predefined action spaces”], and we will carefully revise the rest of the manuscript for clarity.
Under this design, the agent is free to use existing APIs but is not limited to a fixed set of them. It can write and execute code, interact with the environment and filesystem, and construct task-specific tools as needed. Using CDP/Playwright for web tasks or CadQuery/Build123d for CAD tasks is therefore consistent with our design. These are libraries the agent leverages programmatically, not restrictions imposed by the harness. For the CAD tasks, CadQuery/Build123d code is the output format required by the benchmarks themselves, so all baselines operate under the same constraint. For the web tasks, the agent adopted CDP on its own; our prompts do not instruct it to be restricted to specific tools.
Table 2: Narrower success gain, but growing cost savings with stronger models
Regarding Table 2, we acknowledge that the performance gain comparing GUI only and CUAWright approach narrows from +15.7 partial score with GPT-5.5 to +5.2 with the stronger GPT-5.6 Sol model. However, the average cost reduction per task actually increases significantly from 6.0 dollar to 9.5 dollar per task (37.3% to 68.3% cost reduction from GPT 5.5 to GPT 5.6 sol).
We observed that stronger models increasingly favor terminal-based interactions, even in GUI-only settings, where they open terminals through mouse clicks and type commands via the GUI. This incurs unnecessary overhead from repeated GUI interactions and screenshots attached at every step. CUAWright eliminates this overhead through direct terminal access, delivering greater cost savings as model capabilities improve.