acceptodds
Under review as a conference paper at ICLR 2027

Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

Abstract

Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Computer-use agents, GUI agents, General agents, Agent-computer interfaces, Tool use, Accessibility, Multimodal agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_dobfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Hybrid Computer Use (HCU), which integrates AX-based semantic interaction and GUI tools into general-purpose agents through MCP while retaining their existing code and file-operation capabilities. The model selects and combines these three interaction channels. The framework includes compact AX representations, target-reference management, batched actions with postcondition checks, state-difference feedback, and cross-channel action guards, together with supplementary usage instructions. The authors evaluate HCU on OSWorld 2.0 and OSWorld 1.0 and report improvements in task scores, interaction overhead, and cost.

Critical strengths

S1: Practical utility. The framework addresses practical issues such as invalid tool calls, stale state, and redundant observations. It organizes existing interaction capabilities into a unified interface that can be used with different agent harnesses, making it a useful reference for building practical computer-use systems.

S2: Substantial experimental effort and broad coverage. The evaluation covers two benchmarks, OSWorld 1.0 and the more recent OSWorld 2.0, multiple frontier models, and widely used agent harnesses. The paper also provides channel- and skill-level ablations and trajectory analyses. Overall, the experimental effort is substantial and the empirical material is extensive.

Critical weaknesses

W1: Limited methodological novelty; the contribution is primarily engineering integration. Combining GUI interaction with code or APIs has already been explored extensively, and AX is itself an existing capability. The main additions concern AX representation compression, reference management, feedback organization, and cross-channel protection. The paper does not clearly establish methodological novelty beyond combining these mechanisms, and action batching alone is insufficient as a standalone novelty claim. Furthermore, these designs depend on the accessibility information exposed by applications and do not resolve incomplete AX coverage. Whether their benefits generalize to applications with limited accessibility support has not been systematically evaluated.

W2: The empirical advantage is not sufficiently compelling, and direct baseline coverage is limited. Despite the scale of the experiments, matched baseline comparisons focus primarily on CUA Driver, with limited comparison against other competitive hybrid or semantic-interaction approaches under the same settings. In the paired GPT-6 Astra experiment on OSWorld 2.0, Partial improves from 68.82% to 72.37%, but Binary increases only from 31.50% to 32.40%, corresponding to approximately one additional fully completed task out of 108, while cost remains essentially unchanged. The reductions in tool calls and execution time are valuable, but each configuration is evaluated only once in the main experiment, leaving the robustness of the full-completion improvement unclear. The efficiency and score improvements on OSWorld 1.0 are also worth acknowledging. However, the official OSWorld 1.0 leaderboard already includes an agent framework reporting 90.19%. Although differences in models and evaluation configurations prevent a direct comparison, this makes the lack of broader competitive baselines more consequential. The current results do not sufficiently establish a distinctive overall advantage over existing approaches.

Questions for authors

Q1: Please address the concerns raised in the weaknesses.

Q2: Can the contributions of the key mechanisms be isolated? With the model, harness, supplementary instructions, and other components held fixed, how much do batching, guards, and feedback, particularly AX diffs, each contribute to performance and efficiency? The current channel-level ablations and whole-interface comparisons do not adequately answer this question.

Q3: What measures prevent access to evaluation information through network or local tools? The network-access settings of the Codex harness, and potentially other harnesses, are not fully specified. Appendix F.2 explicitly instructs the agent not to access benchmark definitions, evaluator internals, reference answers, hidden checks, or other tasks’ traces. Beyond these instructions, what technical restrictions or trajectory audits were used during OSWorld 2.0 evaluation to prevent access to such information through web search, shell commands, or other tools? Were any attempts to exploit evaluation artifacts or unintended shortcuts observed? It would also be informative to discuss notable OSWorld 2.0 cases, particularly those illustrating the effectiveness or limitations of these safeguards.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dobf

We thank the reviewer for the thorough review and for recognizing HCU's practical utility and the breadth of our experiments.

W1 (part 1): Methodological novelty. HCU asks how a general agent should use a computer. It organizes computer interaction into three complementary channels: programmatic (code and command-line tools), semantic (accessibility, which exposes the structure and state of the interface), and visual (the GUI). Its core principle is model-directed channel selection: all three channels are exposed, and the model chooses and combines them at each step. The rest of the design serves this principle: compact observations and batching make AX cheap, wait steps and diffs confirm results, guards keep input safe across switches, and the skill guides the choice. Prior hybrid work builds its own agent framework or model, or adds application-specific tools. CUA Driver provides a general AX and GUI interface, but it may execute an action through a different channel than the model chose, and it does not study how channels should be combined (table in our response to gL87). We study this combination through ablations and cross-model, cross-harness, cross-benchmark, and cross-platform experiments. If the reviewer has specific prior works in mind, we would be glad to discuss and cite them.

W1 (part 2): Dependence on AX coverage. HCU does not rely on AX alone: when AX falls short, the agent switches to programmatic or visual interaction for the same step. Even with AX disabled entirely, HCU scores on par with CUA Driver (62.79 vs. 62.30) at 50% lower cost. On OSWorld 1.0, a failed AX action leads to a GUI switch in only 11–12% of tasks. On GIMP, whose main work area is a canvas that AX cannot see into, HCU still outperforms CUA Driver (43.6 vs. 38.5). On Windows Agent Arena, with a different accessibility stack, HCU outperforms CUA Driver by 8.97 points.

W2 & Q1: Baselines, Binary gain, cost, and the 90.19% result.

  • Baselines. Beyond CUA Driver (GR G3), we compare with a different hybrid approach: WeaveBench's official OpenClaw agent, which combines code with screenshot-based GUI but has no AX. With the same model, HCU outperforms it by 9.41 points at 49% lower cost, which supports adding AX as a third channel.
  • Binary. Binary requires all of a task's checkpoints (27.25 on average), so improvements short of full completion raise Partial but not Binary. HCU raises Partial by 3.55 points across all seven task domains (GR G2); the 0.90-point Binary gain is about one more fully completed task out of 108.
  • Cost. On OSWorld 2.0, cost is similar while HCU uses 30% fewer tool calls; across all comparisons in GR G2, HCU reduces cost by 34% on average.
  • Robustness. Beyond the single OSWorld 2.0 run, HCU outperforms its baseline in every setting and every run in GR G2.
  • 90.19%. This officially reported GPT-5.4 score could not be reproduced in our setting, so we measured all comparisons ourselves with the same model, harness, and evaluator: with GPT-5.4, HCU scores 65.59 vs. 62.30 for CUA Driver.

Q2: Isolating mechanism contributions. We added a mechanism-level ablation that disables each mechanism, and all four together, with the model, harness, AX channel, and skill fixed (GR G1):

  • Batching is the main efficiency mechanism: without it, tool calls rise by 24%.
  • AX diffs reduce re-reading: without them, new input tokens rise by 11%.
  • AX wait steps confirm results of AX actions, which are not always stable: without them, the agent needs 4% more steps.
  • Action guard prevents unsafe input: where it intervenes, removing it raises failed actions by 52%.
  • Under OSWorld's official 15-step limit, disabling batching or all four lowers success by 6.0 or 5.7 points.

Q3: Preventing access to evaluation information. We audited every trajectory from all our experiments, checking shell commands, file paths, URLs, searches, and tool outputs: no trajectory accessed task definitions, evaluator code, or reference answers. In addition, the evaluator runs on the host after the agent process stops; the agent's container mounts no host directories and contains no evaluation files or task definitions; web search is disabled in every harness. The instruction in Appendix F.2 is an additional precaution.

We will add these results to the revision.

Thank you for the detailed responses and additional experiments, which address most of my concerns. However, I still have reservations about the methodological novelty and will maintain my original score.

AuthorsOpenReview ↗

Response to the novelty question

Thanks again for your response and constructive suggestions. We would like to clarify the methodological novelty of our approach by comparing it with prior work:

We here discuss our work with UltraCUA[1], Agent S3[2], AgentStore[3], MCPWorld[4], and OSWorld-MCP[5]. These works train a hybrid-action model (UltraCUA), combine GUI and code within their own agent framework (Agent S3, CoAct-1), or integrate application-specific tools (AgentStore; OSWorld-MCP and MCPWorld as benchmarks). HCU is a general interface that any agent can use, and the paper systematically studies how code, AX, and GUI should be combined.

Training-freeApplication-agnosticCompatible with any agentModel-directed channel selection
UltraCUA✗✗✗✓
Agent S3✓✓✗✓
CoAct-1✓✓✗✓
UFO2✓✗✗✓
CUA Driver✓✓✓✗†
HCU (ours)✓✓✓✓

* Through the agent's own programmatic tools. † CUA Driver may execute an action through a different channel than the model chose (a coordinate click can become an accessibility action; an element click can fall back to pixel input). In HCU, the tool the model calls determines the channel.

In summary, CUA Driver, an open-source project, is the closest to our work. The key difference is that our approach lets frontier models switch freely between action channels, allowing them to make fuller use of their capabilities while CUA Driver take actions by human-designed priorities. Also our work is not relying on specific agent harnesses and specific app interface. The comparative experiments in the paper show clear advantages on improved success rate and reduced cost across different benchmarks (see General Response G2). We also provide isolated mechanism ablation study, which proves our designed components contributes to the final performance (see General Response G1).

Reference:

[1] UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action, arXiv 2025

[2] Scaling Agents for Computer Use, TMLR 2026

[3] AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant, Findings of ACL 2025

[4] MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents, arXiv 2025

[5] OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents, ICLR 2026

Reviewer_gL87OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HCU exposes programmatic, accessibility-based semantic, and visual interaction through a unified interface for existing general agents. It adds compact AX observations, persistent references, action batching, postconditions, guards, and post-action differences, together with an instruction skill. Experiments compare the package with CUA Driver, remove channels or the skill, and analyze behavior across several models.

Critical strengths

The implementation addresses practical interaction problems and documents its execution semantics and limitations carefully. Repeated OSWorld 1.0 comparisons across several models support the usefulness of the complete interface package. The authors also distinguish action delivery from verified effects and acknowledge that some behavioral analyses are descriptive.

Critical weaknesses
  1. The incremental contribution is unclear. Hybrid GUI/code/API interaction was already explored by UltraCUA, Agent S3, AgentStore, MCPWorld, and OSWorld-MCP, none of which is discussed. The paper does cite UFO2 and CoAct-1, but needs clearer positioning within this broader literature. HCU's potential distinction lies in its AX execution and feedback mechanisms; however, batching, guards, postconditions, and diffs are evaluated together. Channel/skill ablations do not establish which new mechanism matters. The work therefore demonstrates useful engineering more clearly than a distinct research advance.

  2. The baseline comparison is difficult to interpret. Table 1's "Computer-use loop" is not operationally defined, and external GPT/Claude scores cannot be assumed to represent complete native computer-use systems. Running HCU in Codex/Claude Code, or changing its GUI action vocabulary, does not establish such a comparison. The controlled evidence primarily concerns CUA Driver. On OSWorld 2.0, that comparison uses one run on 108 tasks, with a Binary gain of only 0.90 percentage points, approximately one task.

Questions for authors
  1. What distinguishes HCU from prior hybrid systems, and which existing experiment isolates its distinctive mechanisms?
  2. What tools, interfaces, and execution protocols does each "Computer-use loop" row represent? Please distinguish external scores from matched comparisons.
1 comment
AuthorsOpenReview ↗

Response to Reviewer gL87

We thank the reviewer for the careful reading and the pointers to related work.

W1 & Q1: Distinction from prior hybrid systems, and which experiments isolate HCU's mechanisms. We will cite UltraCUA, Agent S3, AgentStore, MCPWorld, and OSWorld-MCP. These works train a hybrid-action model (UltraCUA), combine GUI and code within their own agent framework (Agent S3, CoAct-1), or integrate application-specific tools (AgentStore; OSWorld-MCP and MCPWorld as benchmarks). HCU is a general interface that any agent can use, and the paper systematically studies how code, AX, and GUI should be combined.

Training-freeApplication-agnosticCompatible with any agentModel-directed channel selectionSystematic study of channel combinationProgrammaticAXGUI
UltraCUA✗✗✗✓✗✓✗✓
Agent S3✓✓✗✓✗✓✗✓
CoAct-1✓✓✗✓✗✓✗✓
UFO2✓✗✗✓✗✗✓✓
CUA Driver✓✓✓✗†✗✓*✓✓
HCU (ours)✓✓✓✓✓✓*✓✓

* Through the agent's own programmatic tools. † CUA Driver may execute an action through a different channel than the model chose (a coordinate click can become an accessibility action; an element click can fall back to pixel input). In HCU, the tool the model calls determines the channel.

HCU's mechanisms are isolated by the channel and skill ablations (Section 4.3) and the new mechanism-level ablation (GR G1).

W2 & Q2 (part 1): What the "Computer-use loop" rows represent. We agree these rows were unclear. They come from three sources, which the revised Table 1 will state:

  • Official OSWorld 2.0 leaderboard (Claude Opus 5, GPT-5.6 Sol): the official GUI loop, i.e., OSWorld's official harness, which observes only screenshots and acts through mouse and keyboard (batched tool calls, 500 steps, 0808 release), with no code, shell, or AX channel.
  • Model developers' own reports (GPT-6 Astra, offline subset and unranked; Muse Spark 1.3; Gemini 3.8 Flash; Kimi K3): self-reported, with evaluation settings not publicly documented; Gemini and Kimi predate the 0808 release.
  • StateAct's paper (Claude Opus 4.8, 0624 release).

The block will be renamed "Externally reported", with each row's source and setting.

W2 & Q2 (part 2): Matched vs. external scores. We agree the external scores are not matched comparisons; they give context for the state of the art, not evidence for HCU's gains. Our claims rest on matched comparisons:

  • Same model and harness: with GPT-6 Astra in Codex, HCU outperforms CUA Driver by 3.55 Partial points with 30% fewer tool calls.
  • Same model, official setting: with Claude Opus 5, HCU outperforms the official GUI loop by 3.75 Partial and 3.76 Binary points.
  • Beyond OSWorld 2.0 (GR G2): HCU outperforms CUA Driver on OSWorld 1.0 with four models, on three other benchmarks, and in Codex's default configuration, by 10.0% on average.

We compare mainly with CUA Driver because it is the closest system (GR G3). On the GUI action vocabulary, swapping HCU's GPT-style GUI actions for Claude-style actions (appendix) changes the score by only 0.8 points (Claude Opus 5.5) and 2.0 points (GPT-5.4), less than HCU's gains over GUI-based baselines.

We will add these changes to the revision.

Reviewer_2ad7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents HCU, a useful interface between general agents and computer-use tasks. It adds AX and GUI tools to general agents through MCP, alongside their programmatic tools. Models with HCU can choose among channels using a provided skill. The interface provides compact AX observations, batched actions, diff feedback, and action guards. Experiments on OSWorld report higher scores and generally improved efficiency over CUA Driver, across several model–harness configurations.

Critical strengths
  1. HCU provides flexible hybrid interaction for computer-use agents. It lets general agents choose and combine code, AX, and GUI tools without extra model training.

  2. HCU improves interaction efficiency. The result reports fewer tool calls and lower costs across several model settings, showing practical benefits for computer-use tasks.

  3. The trajectory behavior analysis shows how agents use the different channels and how observation and AX execution can complement each other.

Critical weaknesses
  1. Modest performance gains. In the matched OSWorld 2.0 comparison, Binary improves by only 0.90pp. With one run per configuration on 108 tasks and no paired confidence intervals, the robustness of this gain remains uncertain.

  2. Generality. The integrations replace base instructions and restrict some harness features. Compatibility and potential overlap or conflicts with native tools and instructions are not systematically evaluated.

  3. Unclear module contributions. The ablations remove entire channels or the skill, leaving the individual benefits of batching, postconditions, diff feedback, and guards unresolved.

Questions for authors
  1. Some improvement is marginal, are the OSWorld/OSWorld v2 gains stable across repeated paired runs?

  2. Does HCU retain its gains across harnesses with native tools, instructions, and features preserved, and how does it handle overlapping capabilities? Evidence of better compatibility and generality is my major concern for raising my rating.

  3. What are the individual task-score and efficiency gains for each part of HCU? If cannot be quantified, some discussion is also acceptable.

  4. When AX coverage is limited or unavailable, how is HCU's performance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2ad7

We thank the reviewer for the detailed review and constructive suggestions on stability, compatibility, and generality.

W1 & Q1: Size and stability of the gains.

  • Generality and stability: HCU outperforms its baseline in every setting and every run (11 comparisons across 5 benchmarks, 7 models, 4 harnesses, and 3 operating systems; GR G2), by 10.0% on average at 34% lower cost. On OSWorld 1.0 with GPT-5.4, the paired 95% CI of the gain is [+0.81, +5.64].
  • Why the Binary gain is small: Binary requires all checkpoints of a task (27.25 on average), so improvements short of full completion raise Partial but not Binary. HCU raises Partial by 3.55 points, and the gain is broad: it improves Partial in all seven task domains of OSWorld 2.0 (GR G2). The 0.90-point Binary gain is about one more fully completed task out of 108.

W2 & Q2: Compatibility with native harness features. We added an experiment in Codex's default configuration: Codex's own system prompt and native tools are unchanged, and HCU's skill is loaded as project instructions, as a user would install it (OSWorld 1.0, 361 tasks, GPT-6 Luna; changes relative to CUA Driver).

ConfigurationScoreStepsInput tokensCost
CUA Driver67.77———
HCU69.65−24%−42%−40%
  • HCU stays ahead with native features enabled. It works alongside Codex's own tools, which overlap with parts of its functionality, and still outperforms CUA Driver at 40% lower cost.
  • Why the benchmark configuration disabled some features. We disabled only features unrelated to computer operation (web search, image generation, memory, plugins, multi-agent, hooks). They differ across our four harnesses (Codex, Claude Code, Gemini CLI, Grok Build), and some, like web search, can solve parts of a task without operating the computer. Disabling them gives every harness the same computer-use capabilities, so result differences come from the interface.
  • HCU generalizes across native harnesses: in the cross-model study (Section 4.4), it outperforms CUA Driver in all four harnesses, by 6.5 points (10.8%) on average, at 30% lower cost.

W3 & Q3: Contribution of each component. We added a mechanism-level ablation that disables each mechanism, and all four together, with everything else fixed (GR G1):

  • Batching is the main efficiency mechanism: without it, tool calls rise by 24%.
  • AX diffs reduce re-reading: without them, new input tokens rise by 11%.
  • AX wait steps confirm results of AX actions, which are not always stable: without them, the agent needs 4% more steps.
  • Action guard prevents unsafe input: where it intervenes, removing it raises failed actions by 52%.
  • Success depends on the step limit: at the default 100-step limit, Luna's trajectories are short, so success mainly reflects the AX channel and skill; under the official 15-step limit, disabling batching or all four lowers success by 6.0 or 5.7 points.

Q4: Performance when AX is limited. We added an analysis of AX-to-GUI switches. HCU still performs well, because the agent switches to CLI or GUI for the same step.

  • Without AX at all, HCU (code and GUI only) scores 62.79, on par with CUA Driver (62.30), with 15% fewer tool calls and 50% lower cost (GPT-5.4, OSWorld 1.0).
  • On GIMP, an image editor whose main work area is a canvas that AX cannot see into, HCU scores 43.6 vs. 38.5 for CUA Driver.
  • AX failures rarely force a switch: 11–12% of tasks and about 2% of AX calls on OSWorld 1.0.
  • Other accessibility stacks: on Windows Agent Arena (Windows UI Automation), HCU outperforms CUA Driver by 8.97 points.

We hope these results address the reviewer's concerns about stability, compatibility, and generality, and we would be grateful if the reviewer could take them into account.

The generality and ablation results resolve most of my concerns. Despite modest novelty, HCU is practical and efficient, a solid improvement between computer-use agents and operation systems. I am happy to raise my rating to Weak Accept.

Reviewer_5puPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Hybrid Computer Use (HCU), an interface that lets a general agent combine programmatic tools, accessibility-based semantic interaction (AX), and graphical interaction (GUI). Its AX channel provides compact observations, element references, batched actions, optional postconditions, and feedback on interface changes. Action guards check whether targets and layouts remain valid before input. The paper evaluates task performance and resource use on OSWorld 1.0 and 2.0.

Critical strengths
  1. A practical problem with a concrete interface design. Repeated screenshots and coordinate-based actions can be costly in long workflows. HCU gives agents access to structured controls alongside visual and programmatic tools, while addressing stale-state risks when switching channels.

  2. Evaluation covers both performance and cost. On OSWorld 2.0, HCU improves the Partial score of the matched GPT-6 Astra configuration from 68.82% with CUA Driver to 72.37%. The OSWorld 1.0 ablations and cross-model results also report steps, input usage, time, and cost.

Critical weaknesses
  1. The contributions of the main AX features are not isolated. The ablations remove entire channels or the supplementary skill, while compact observations, batching, postconditions, AX diffs, and action guards change together. The results establish the benefit of the interface as a package but do not show which features drive the gains. Removing GUI also slightly increases Score from 65.59% to 66.08%, so its contribution to task performance needs a careful interpretation.

  2. Some OSWorld 2.0 comparisons are less controlled. The GPT-6 Astra comparison with CUA Driver is matched, but other entries use results reported for different models, task releases, or evaluation setups. In particular, the difference from the official Claude Opus 5 GUI result cannot be attributed solely to HCU. The matched Binary improvement is 0.90 percentage points; paired task-level results would help assess it.

Questions for authors
  1. Could the authors separately ablate batching, postconditions, AX diffs, and action guards while keeping the model, AX channel, and skill fixed? How does each feature affect task score, steps, and action failures?

  2. Could the authors provide paired task-level results for HCU and CUA Driver on OSWorld 2.0 and clearly separate matched experiments from externally reported results? How do they interpret the slightly higher Score when GUI is removed?

  3. How often does AX fail to expose a control needed for a task, requiring GUI fallback? Does HCU retain its advantage on tasks dominated by custom-drawn or virtualized interfaces?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5puP

We thank the reviewer for the positive assessment and precise questions.

W1 & Q1: Contribution of individual mechanisms. We added a mechanism-level ablation and evaluated each mechanism against its purpose (GR G1):

  • Batching (efficiency): without it, tool calls rise by 24%.
  • AX diffs (efficiency): without them, new input tokens rise by 11%.
  • AX wait steps (postconditions; reliability): without them, the agent needs 4% more steps to recover from unconfirmed states.
  • Action guard (safety): where it intervenes, removing it raises failed actions by 52%.
  • All four: disabling them raises cost by 18% and, under OSWorld's official 15-step limit, lowers success by 5.7 points.

W1 (part 2) & Q2 (part 2): When GUI helps, and why the overall ablation does not show it. We added an analysis of AX-to-GUI switches and a step-limit analysis. GUI covers what AX cannot: canvases and other custom-drawn regions, and coordinate-based operations such as drags. On OSWorld 1.0 these cases are rare (a failed AX action leads to a GUI switch in only 11–12% of tasks), and many tasks can be done through CLI and AX alone. The +0.5-point average without GUI is therefore not consistent: full HCU scores higher in two of three runs, and the two configurations tie on 76% of tasks. GUI's benefit is efficiency: it saves 11% of tool calls, and under the official 15-step limit, full HCU outperforms HCU w/o GUI by 3.97 points.

W2 & Q2: Controlled comparisons and the size of the gains. We will split Table 1 into matched comparisons and externally reported results (GR G3). Across all benchmarks, models, and operating systems we evaluated, HCU improves over its baseline by 10.0% on average, at 34% lower cost (GR G2). On OSWorld 2.0, Binary requires all checkpoints of a task, so progress short of full completion raises Partial but not Binary: the 0.90-point Binary gain is about one task, while Partial rises by 3.55 points. By task domain, HCU raises Partial in all seven OSWorld 2.0 domains (GR G2).

Q3: How often AX misses a needed control. We added an analysis of AX-to-GUI switches. With GPT-5.4, a failed AX action leads to a GUI switch in 11–12% of tasks (about 2% of AX calls). On GIMP, an image editor whose main work area is a canvas, HCU scores 43.6 vs. 38.5 for CUA Driver.

We will add these results to the revision.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/2): Overview and G1

We thank all reviewers for their constructive reviews. All four found the paper clear, and they recognized a practical problem with a concrete interface design (5puP, gL87, dobf), consistent gains with fewer tool calls and lower cost (2ad7, gL87), careful documentation of execution semantics and limitations (gL87), and broad experiments covering performance and cost (5puP, dobf).

We answer shared questions in G1–G3. New results for this rebuttal:

  1. A mechanism-level ablation on OSWorld 1.0.
  2. A comparison in Codex's default configuration.
  3. A step-limit analysis under OSWorld's official step limits.
  4. How often a failed AX action makes the agent switch channels.
  5. A trajectory audit for access to evaluation information.

All new results will be added to the revision.

G1. Contribution of individual mechanisms (5puP, 2ad7, gL87, dobf)

Each mechanism has a specific purpose:

  • Batching runs several AX actions in one call, saving model round trips (efficiency).
  • AX diffs return only what changed after an action, so the agent re-reads less (efficiency).
  • AX wait steps (postconditions) confirm that the expected state appears after an action, because AX actions are not always stable: a call can report success before the interface changes, or without changing it (reliability).
  • Action guard blocks input when the window changes mid-batch or the target reference is stale (safety).

Setup: OSWorld 1.0 (361 tasks), GPT-6 Luna, mean of two runs; each mechanism is disabled with everything else fixed. Changes are relative to full HCU; bold values are significant (paired bootstrap, 95%).

VariantStepsTool callsNew input tokensCost
w/o batching+12%+24%+2%+9%
w/o AX diffs−5%−7%+11%+3%
w/o AX wait steps+4%+4%+1%+5%
w/o all four+10%+20%+16%+18%
  • Batching drives most of the efficiency gain.
  • AX diffs reduce re-reading: without them, new input tokens rise by 11%.
  • AX wait steps and the action guard make actions reliable and safe. OSWorld errors are recoverable, so their benefit appears as recovery cost: without wait steps the agent needs 4% more steps, and on the ~5% of tasks where the guard intervenes, removing it raises failed actions by 52% and actions with unconfirmed effects by 3.4×. In real use, the same errors can be irreversible (e.g., deleting a file or sending a message). Neither mechanism costs success.
  • Under a step limit, efficiency becomes success. At the default 100-step limit, GPT-6 Luna's trajectories are short (median 16 steps), so success mainly reflects the AX channel and skill (Section 4.3). Under OSWorld's official 15-step limit, disabling batching or all four lowers success by 6.0 or 5.7 points.
AuthorsOpenReview ↗

General Response (2/2): G2 and G3

G2. Size, stability, and generality of the gains (5puP, 2ad7, gL87, dobf)

OSWorld 2.0 reports Partial / Binary; other benchmarks report Score. Cost and tool calls are relative to the baseline, and multi-run scores are averaged. OpenClaw is WeaveBench's official agent (code plus screenshot-based GUI, no AX), run with the same model and judge.

Benchmark (tasks)ModelBaselineBaselineHCUGain (relative)CostTool calls
OSWorld 2.0 (108)GPT-6 AstraCUA Driver68.82 / 31.5072.37 / 32.40+3.55 (+5.2%) / +0.90−0.2%−30%
OSWorld 2.0 (108)Claude Opus 5Official GUI loop68.31 / 31.4372.06 / 35.19+3.75 (+5.5%) / +3.76−20%n/a
OSWorld 1.0 (361)GPT-5.4CUA Driver62.3065.59+3.29 (+5.3%)−55%−26%
OSWorld 1.0 (361)Claude Opus 5.5CUA Driver68.6871.62+2.94 (+4.3%)−8%−20%
OSWorld 1.0 (361)Gemini 3.8 FlashCUA Driver54.6767.57+12.90 (+23.6%)−44%−17%
OSWorld 1.0 (361)Grok 4.7CUA Driver67.9074.71+6.81 (+10.0%)−13%−25%
OSWorld 1.0 (361), Codex defaultGPT-6 LunaCUA Driver67.7769.65+1.88 (+2.8%)−40%−19%
WeaveBench (114)GPT-5.4CUA Driver46.3753.25+6.88 (+14.8%)−46%−18%
WeaveBench (114)GPT-5.4OpenClaw43.8453.25+9.41 (+21.5%)−49%+0%
Windows Agent Arena (111)GPT-5.4CUA Driver71.8980.86+8.97 (+12.5%)−51%−31%
MacAgentBench, UI-dependent (28)GPT-5.4CUA Driver85.789.3+3.6 (+4.2%)−51%−20%
Average+10.0%−34%−21%
  • The gains generalize and are stable. HCU outperforms its baseline in every setting (11 comparisons across 5 benchmarks, 7 models, 4 harnesses, and 3 operating systems) and in every run, with +10.0% on average, 34% lower cost, and 21% fewer tool calls. On OSWorld 1.0 with GPT-5.4, the paired 95% CI of the gain is [+0.81, +5.64].
  • The gains grow under step limits. HCU needs fewer steps (median 22 vs. 34 on OSWorld 1.0 with GPT-5.4). Under OSWorld's official 15- and 50-step limits, its lead over CUA Driver grows to +4.61 and +16.07 points.
  • On OSWorld 2.0, HCU's gains appear mainly in Partial. Each task has 27.25 checkpoints on average; Partial is the weighted fraction satisfied, while Binary requires all of them. Partial therefore captures incremental progress within a task, while improvements short of full completion do not change Binary. HCU raises Partial from 68.82 to 72.37 (+3.55); the 0.90-point Binary gain is about one more fully completed task out of 108.

On OSWorld 2.0, broken down by the benchmark's seven task domains (GPT-6 Astra), HCU raises Partial over CUA Driver in all seven domains and uses fewer steps and tool calls in every domain:

Domain (tasks)Partial (CUA Driver → HCU)StepsTool calls
Research & Education (22)74.0 → 78.7−12%−35%
Creative Production (27)62.1 → 64.2−4%−19%
Engineering & Computing (17)68.6 → 71.8−9%−27%
Administration & Compliance (15)73.6 → 78.4−17%−39%
Personal Services (13)69.1 → 72.3−13%−38%
Business & Finance (10)71.9 → 76.5−14%−35%
Healthcare (4)60.0 → 62.5−2%−19%

G3. Matched vs. externally reported results (5puP, gL87, dobf)

We will split Table 1 into two blocks:

  • Matched comparisons: HCU vs. CUA Driver (GPT-6 Astra, same harness), and HCU vs. the official GUI loop (same model, Claude Opus 5).
  • Externally reported results, each annotated with its source and setting (details in our response to gL87).

We compare mainly with CUA Driver because it is the closest system: like HCU, it is a general computer-use interface that provides AX and GUI alongside the agent's own programmatic tools and plugs into existing harnesses such as Codex and Claude Code. Other hybrid systems run their own agent framework (e.g., Agent S3, CoAct-1, UFO2) or train their own model (e.g., UltraCUA), so they cannot be placed in the same harness.

Hi this paper we received four diverse scores, 1 weak acceptance and 3 weak rejection.

could you please both see whether your concerns are addressed by author rebuttal and follow-up if you have additional concerns?

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.