- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Hybrid Computer Use (HCU), which integrates AX-based semantic interaction and GUI tools into general-purpose agents through MCP while retaining their existing code and file-operation capabilities. The model selects and combines these three interaction channels. The framework includes compact AX representations, target-reference management, batched actions with postcondition checks, state-difference feedback, and cross-channel action guards, together with supplementary usage instructions. The authors evaluate HCU on OSWorld 2.0 and OSWorld 1.0 and report improvements in task scores, interaction overhead, and cost.
Critical strengths
S1: Practical utility. The framework addresses practical issues such as invalid tool calls, stale state, and redundant observations. It organizes existing interaction capabilities into a unified interface that can be used with different agent harnesses, making it a useful reference for building practical computer-use systems.
S2: Substantial experimental effort and broad coverage. The evaluation covers two benchmarks, OSWorld 1.0 and the more recent OSWorld 2.0, multiple frontier models, and widely used agent harnesses. The paper also provides channel- and skill-level ablations and trajectory analyses. Overall, the experimental effort is substantial and the empirical material is extensive.
Critical weaknesses
W1: Limited methodological novelty; the contribution is primarily engineering integration. Combining GUI interaction with code or APIs has already been explored extensively, and AX is itself an existing capability. The main additions concern AX representation compression, reference management, feedback organization, and cross-channel protection. The paper does not clearly establish methodological novelty beyond combining these mechanisms, and action batching alone is insufficient as a standalone novelty claim. Furthermore, these designs depend on the accessibility information exposed by applications and do not resolve incomplete AX coverage. Whether their benefits generalize to applications with limited accessibility support has not been systematically evaluated.
W2: The empirical advantage is not sufficiently compelling, and direct baseline coverage is limited. Despite the scale of the experiments, matched baseline comparisons focus primarily on CUA Driver, with limited comparison against other competitive hybrid or semantic-interaction approaches under the same settings. In the paired GPT-6 Astra experiment on OSWorld 2.0, Partial improves from 68.82% to 72.37%, but Binary increases only from 31.50% to 32.40%, corresponding to approximately one additional fully completed task out of 108, while cost remains essentially unchanged. The reductions in tool calls and execution time are valuable, but each configuration is evaluated only once in the main experiment, leaving the robustness of the full-completion improvement unclear. The efficiency and score improvements on OSWorld 1.0 are also worth acknowledging. However, the official OSWorld 1.0 leaderboard already includes an agent framework reporting 90.19%. Although differences in models and evaluation configurations prevent a direct comparison, this makes the lack of broader competitive baselines more consequential. The current results do not sufficiently establish a distinctive overall advantage over existing approaches.
Questions for authors
Q1: Please address the concerns raised in the weaknesses.
Q2: Can the contributions of the key mechanisms be isolated? With the model, harness, supplementary instructions, and other components held fixed, how much do batching, guards, and feedback, particularly AX diffs, each contribute to performance and efficiency? The current channel-level ablations and whole-interface comparisons do not adequately answer this question.
Q3: What measures prevent access to evaluation information through network or local tools? The network-access settings of the Codex harness, and potentially other harnesses, are not fully specified. Appendix F.2 explicitly instructs the agent not to access benchmark definitions, evaluator internals, reference answers, hidden checks, or other tasks’ traces. Beyond these instructions, what technical restrictions or trajectory audits were used during OSWorld 2.0 evaluation to prevent access to such information through web search, shell commands, or other tools? Were any attempts to exploit evaluation artifacts or unintended shortcuts observed? It would also be informative to discuss notable OSWorld 2.0 cases, particularly those illustrating the effectiveness or limitations of these safeguards.
3 comments
Thank you for the detailed responses and additional experiments, which address most of my concerns. However, I still have reservations about the methodological novelty and will maintain my original score.
Response to the novelty question
Thanks again for your response and constructive suggestions. We would like to clarify the methodological novelty of our approach by comparing it with prior work:
We here discuss our work with UltraCUA[1], Agent S3[2], AgentStore[3], MCPWorld[4], and OSWorld-MCP[5]. These works train a hybrid-action model (UltraCUA), combine GUI and code within their own agent framework (Agent S3, CoAct-1), or integrate application-specific tools (AgentStore; OSWorld-MCP and MCPWorld as benchmarks). HCU is a general interface that any agent can use, and the paper systematically studies how code, AX, and GUI should be combined.
| Training-free | Application-agnostic | Compatible with any agent | Model-directed channel selection | |
|---|---|---|---|---|
| UltraCUA | ✗ | ✗ | ✗ | ✓ |
| Agent S3 | ✓ | ✓ | ✗ | ✓ |
| CoAct-1 | ✓ | ✓ | ✗ | ✓ |
| UFO2 | ✓ | ✗ | ✗ | ✓ |
| CUA Driver | ✓ | ✓ | ✓ | ✗† |
| HCU (ours) | ✓ | ✓ | ✓ | ✓ |
* Through the agent's own programmatic tools. † CUA Driver may execute an action through a different channel than the model chose (a coordinate click can become an accessibility action; an element click can fall back to pixel input). In HCU, the tool the model calls determines the channel.
In summary, CUA Driver, an open-source project, is the closest to our work. The key difference is that our approach lets frontier models switch freely between action channels, allowing them to make fuller use of their capabilities while CUA Driver take actions by human-designed priorities. Also our work is not relying on specific agent harnesses and specific app interface. The comparative experiments in the paper show clear advantages on improved success rate and reduced cost across different benchmarks (see General Response G2). We also provide isolated mechanism ablation study, which proves our designed components contributes to the final performance (see General Response G1).
Reference:
[1] UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action, arXiv 2025
[2] Scaling Agents for Computer Use, TMLR 2026
[3] AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant, Findings of ACL 2025
[4] MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents, arXiv 2025
[5] OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents, ICLR 2026
Response to Reviewer dobf
We thank the reviewer for the thorough review and for recognizing HCU's practical utility and the breadth of our experiments.
W1 (part 1): Methodological novelty. HCU asks how a general agent should use a computer. It organizes computer interaction into three complementary channels: programmatic (code and command-line tools), semantic (accessibility, which exposes the structure and state of the interface), and visual (the GUI). Its core principle is model-directed channel selection: all three channels are exposed, and the model chooses and combines them at each step. The rest of the design serves this principle: compact observations and batching make AX cheap, wait steps and diffs confirm results, guards keep input safe across switches, and the skill guides the choice. Prior hybrid work builds its own agent framework or model, or adds application-specific tools. CUA Driver provides a general AX and GUI interface, but it may execute an action through a different channel than the model chose, and it does not study how channels should be combined (table in our response to gL87). We study this combination through ablations and cross-model, cross-harness, cross-benchmark, and cross-platform experiments. If the reviewer has specific prior works in mind, we would be glad to discuss and cite them.
W1 (part 2): Dependence on AX coverage. HCU does not rely on AX alone: when AX falls short, the agent switches to programmatic or visual interaction for the same step. Even with AX disabled entirely, HCU scores on par with CUA Driver (62.79 vs. 62.30) at 50% lower cost. On OSWorld 1.0, a failed AX action leads to a GUI switch in only 11–12% of tasks. On GIMP, whose main work area is a canvas that AX cannot see into, HCU still outperforms CUA Driver (43.6 vs. 38.5). On Windows Agent Arena, with a different accessibility stack, HCU outperforms CUA Driver by 8.97 points.
W2 & Q1: Baselines, Binary gain, cost, and the 90.19% result.
Q2: Isolating mechanism contributions. We added a mechanism-level ablation that disables each mechanism, and all four together, with the model, harness, AX channel, and skill fixed (GR G1):
Q3: Preventing access to evaluation information. We audited every trajectory from all our experiments, checking shell commands, file paths, URLs, searches, and tool outputs: no trajectory accessed task definitions, evaluator code, or reference answers. In addition, the evaluator runs on the host after the agent process stops; the agent's container mounts no host directories and contains no evaluation files or task definitions; web search is disabled in every harness. The instruction in Appendix F.2 is an additional precaution.
We will add these results to the revision.