acceptodds
Desk rejected at ICLR 2027

HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often inefficient and error prone, or augment GUI interactions with application specific APIs or tools, which require substantial engineering effort and are difficult to scale across applications. We argue that the next generation of CUAs should combine GUI interactions with the command line interface (CLI), leveraging the generality of the GUI and the efficiency of shell commands. A critical challenge, however, is that current models do not know when or how to use the CLI during task execution. To address this challenge, we develop a data construction pipeline that produces three types of trajectories: GUI only, CLI only, and interleaved GUI and CLI trajectories. This pipeline results in HybridCUA-8K, containing 5K hybrid trajectories and 3K verified RLVR tasks. Building on these data, we propose a training framework with two stages: supervised fine tuning on the constructed trajectories, followed by reinforcement learning with our CLI aware rewards that encourages agents to use the CLI selectively and reliably. Experiments show that HybridCUA-9B achieves 53.6% accuracy on OSWorld, improving over the base model by 14.8 percentage points, and improves performance on WindowsAgentArena by 4.0 percentage points. These results demonstrate the effectiveness and cross platform generalizability of the hybrid GUI and CLI paradigm for computer use agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer-Use Agents, Multi-Modal Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_58oiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HybridCUA, which trains computer-use agents to coordinate GUI and CLI interactions to improve task success and efficiency. The authors construct SFT trajectories comprising GUI-only, CLI-only, and mixed-interface interactions, then train the model through SFT followed by reinforcement learning with a reward favoring CLI use and a penalty for command execution errors. HybridCUA-9B achieves an average task score of 53.6 on OSWorld, a 14.8 percentage-point improvement over the original GUI model, while reducing the average number of action steps from 31.6 to 14.0. The model also shows some improvement on WindowsAgentArena.

Critical strengths
  1. Combining GUI and CLI interactions is a promising direction for computer-use agents. The experiments show that simply giving an agent CLI access can hurt performance on computer-use tasks, making a clear case for explicitly training GUI–CLI coordination. This finding provides a coherent motivation for the proposed approach.
  2. The two-stage SFT and RL training procedure is well motivated and supported by the results. Although the authors do not collect mixed GUI–CLI trajectories directly, they obtain encouraging results using GUI-only and CLI-only trajectories together with constructed mixed-interface data.
Critical weaknesses
  1. The paper lacks experiments assessing how HybridCUA scales with model size and training data. For example, it would be useful to compare 9B and 32B models and vary the amount of hybrid data from zero to 5K trajectories, ideally extending beyond 5K, which remains a relatively small dataset. I consider this an important omission. Such experiments would establish how much hybrid data is needed to match the baseline and whether additional data continues to improve both task performance and efficiency.
  2. The routing reward is somewhat coarse, and the CLI-advantage label appears easy to game. For the same task, a trajectory that completes the task efficiently using CLI throughout and a much longer trajectory containing only one trivial CLI operation may receive the same label. The former is clearly preferable, but the label does not distinguish between them, limiting the signal available during RL for learning this distinction. The reward design also needs an ablation in which hybrid RL uses only the task completion reward, to isolate the contributions of the routing reward and the command execution error penalty.
  3. I support the research direction and the overall approach of improving GUI–CLI coordination through training. However, the data and environment construction could be developed further. The SFT data primarily comes from UI-MOPD, trajectory collection using CUA-Gym, and rewrites of existing trajectories; the verifiable RL tasks are also built on CUA-Gym. These choices have practical value, but the paper does not adequately explain how to systematically construct tasks and trajectories that capture the complementary strengths of GUI and CLI and require coordination between them. For hybrid computer-use agents, constructing such data is itself an important research problem. For example, how should tasks combine visual judgment, programmatic operations, and the transfer of state between interfaces, beyond replacing existing GUI actions with commands? A proposed and validated pipeline for constructing these tasks and environments, with evidence that it extends to a broader range of tasks and applications, would better support the scalability and generality of hybrid coordination and strengthen the paper’s contribution and central argument.
  4. All figures appear to use raster graphics, possibly exported as PNG files from the drawing software. As a result, they become blurry when enlarged. The authors should replace them with vector graphics where appropriate.
Questions for authors
  1. The roughly four-point improvement on WindowsAgentArena seems modest. Could the authors analyze which tasks or task categories account for this gain, and whether the model makes effective use of CLI on this benchmark?
  2. What is the contribution of the mixed-interface training data? Does combining GUI-only and CLI-only data already improve hybrid performance, and how much additional benefit comes from the mixed-interface trajectories?
5 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We sincerely thank Reviewer 58oi for the thorough review. We answer the reward ablation (W2) and data and environment construction (W3) here; figure format (W4), the WindowsAgentArena gain (Q1) and mixed-interface data (Q2) in Part II; and scaling (W1) and the CLI-advantage label (W2) in Part III.

W2: Task-completion-only ablation. Starting from the same SFT checkpoint, with the same GRPO budget and tasks, we run hybrid RL with the task-completion reward only (λCLI=λexec=0\lambda_{\text{CLI}}=\lambda_{\text{exec}}=0), and with each CLI-aware term removed separately (single OSWorld run, final checkpoint).

VariantAcc.Avg. stepsCLI step share (%)CLI error rate (%)
SFT checkpoint46.019.85415.2
RL, task completion only51.618.04414.9
RL, w/o routing reward RCLIR_{\text{CLI}}52.116.25912.2
RL, w/o execution penalty rtexecr^{\text{exec}}_t52.914.16016.5
RL, full (HybridCUA-9B)53.614.06411.5

Compared with task-completion-only RL, the full model is 2.0 points higher in accuracy (53.6 vs. 51.6), and also takes fewer steps (14.0 vs. 18.0; fewer at every checkpoint), has a lower CLI error rate (11.5% vs. 14.9%) and uses the CLI more (64% of steps vs. 44%; without the CLI-aware rewards the share falls below the SFT checkpoint's 54%). The rewards contribute in different ways:

  • Interface choice. Each CLI-aware reward alone raises the CLI step share above outcome-only RL (59% without RCLIR_{\text{CLI}} and 60% without rtexecr^{\text{exec}}_t, vs. 44%), and together they reach 64%.
  • RCLIR_{\text{CLI}} improves efficiency. Adding it to the outcome reward (w/o rtexecr^{\text{exec}}_t) cuts steps from 18.0 to 14.1. Removing it from the full model costs 2.2 steps (16.2 vs. 14.0) and 1.5 points of accuracy (52.1 vs. 53.6).
  • rtexecr^{\text{exec}}_t improves command reliability. Adding it to the outcome reward (w/o RCLIR_{\text{CLI}}) lowers the CLI error rate from 14.9% to 12.2%. Removing it from the full model raises the error rate from 11.5% to 16.5% and costs 0.7 points of accuracy (52.9 vs. 53.6).

We have added these curves to Fig. 4 and the paragraph after it in Sec. 4.3.

W3: Data and environment construction. We agree that building tasks that require GUI–CLI coordination is a research problem in itself, and we do not claim to have solved it. Our current procedure is as follows.

  • Interface guide. For each application, a guide built from its documentation and reference code lists the available CLI interfaces, the operations where the CLI is more efficient or reliable, and those that require or favor the GUI.
  • Task generation and filtering. The task generator uses this boundary to compose tasks, together with environment states, assets and an executable verifier. We keep tasks whose verifier runs and whose target state is reachable. This gives 3,000 verified tasks over 11 domains, including multi-application tasks (12.9% of the pool); 1,000 are sampled for online RL.
  • Extension beyond these domains. We applied the same task-construction pipeline to 10 open-source applications outside our 11 domains (Blender, FreeCAD, Ghidra, Godot, Logisim, MuseScore, Octave, OpenSCAD, ParaView and Tiled), building 1,000 tasks (100 per application) and running RL on them from the same SFT checkpoint (OSWorld results below). The final accuracy of 51.2% is lower than RL on tasks from our original, non-OOD applications (53.6%), but it shows that our pipeline can produce usable hybrid tasks for new applications, and that RL on these tasks transfers to OSWorld.
RL step0 (SFT)20406080100120
Accuracy (%)46.048.646.949.150.451.351.2
Avg. steps19.818.717.617.117.516.617.4
CLI step share (%)54565857565960
CLI error rate (%)15.213.414.613.614.112.312.0

We fully agree that mocking up our own out-of-distribution environments and synthesizing environments at scale is a better approach to systematic task and environment construction, and we leave it as future work.

We have added this pipeline extension and its RL results to Appendix C.3 (Table 7) of the revised paper.

AuthorsOpenReview ↗

Rebuttal - Part II

W4: Figures. We will replace the raster plots and diagrams with vector graphics (PDF) in the revision. Figures that embed real screenshots, such as the case studies, will keep the screenshots as raster images inside a vector layout.

Q1: Source of the WindowsAgentArena gain. We broke the WAA results down by application (table below). The overall gain is +4.0. On Windows-native applications that OSWorld does not cover, the gain is +10.2, which suggests the method generalizes to applications outside the training distribution.

OOD applications (not covered by OSWorld)

ApplicationBase modelOursΔ
Windows Calculator0.066.7+66.7
Clock0.025.0+25.0
File Explorer42.157.9+15.8
Settings100.0100.00.0
Paint66.766.70.0
Notepad50.050.00.0
Edge46.238.5−7.7
OOD subtotal44.955.1+10.2

Applications also covered by OSWorld

ApplicationBase modelOursΔ
Chrome11.823.5+11.8
VS Code41.745.8+4.2
LibreOffice Calc4.24.20.0
VLC23.814.3−9.5
LibreOffice Writer42.142.10.0
Subtotal24.825.7+1.0

The gain is concentrated in applications absent from training: File Explorer (+3 tasks), Windows Calculator (+2) and Clock (+1). On File Explorer, the model uses shell commands (mkdir/move/copy) to create, move and copy files in 2–4 steps, where the GUI-only baseline often fails. The Chrome gain (+2) comes from settings tasks that the model completes entirely through the GUI. The drop on Edge is a single task: the model fails one more task than the base model (5 solved vs. 6 for the base model, out of 13). Paint, Notepad and Settings are unchanged.

Interface usage and limitation. The model picks the interface by task type. It mainly calls the CLI where commands fit (file operations, VS Code configuration, spreadsheet handling; about 34–38% of steps overall) and mainly uses the GUI for browser and system settings. The gain on file management comes directly from the CLI. The limitation is that our training data is mainly from Linux, and some commands are written in Linux style, with Linux-style paths and syntax, and fail on Windows. This limits the overall gain on the non-OOD applications of WAA. Collecting Windows-native command data is our next step.

We have added this per-application breakdown to Appendix E.1 (Table 10) of the revised paper.

Q2: Contribution of mixed-interface trajectories. We add one row to our SFT data ablation on OSWorld: the pure-GUI and pure-CLI data combined, without any interleaved trajectories.

ConfigurationAcc.↑Avg. Steps↓
Qwen3.5-9B38.831.6
  w/ CLI only31.719.1
  w/ GUI only43.229.5
  w/ GUI + CLI only (no hybrid)39.825.2
  w/ Hybrid only41.022.6
HybridCUA-9B-SFT46.019.8

Combining the pure-GUI and pure-CLI data does help, but only slightly: 39.8% vs. 38.8% for the base model (+1.0), and below GUI-only data alone (43.2%). The mixed-interface trajectories add much more: all data together reach 46.0%, i.e., +6.2 over the combination without them, and steps drop from 25.2 to 19.8. The three parts are complementary. Each pure set covers only one interface: CLI-only data alone falls below the base model (31.7%), and GUI-only data never shows when a command is the better choice. Merging them gives the model both skills but no signal for choosing between them within a task, so the combination gains little. Mixed trajectories supply exactly this signal, and they already beat the merged pure data on their own (41.0% vs. 39.8%). They cover fewer single-interface skills, though, so the best result comes from the pure data providing the skills for each interface and the mixed trajectories showing how to combine them.

We have added this row to Table 2 and the sentence after it in Sec. 4.3.

AuthorsOpenReview ↗

Rebuttal - Part III

W1: Scaling. We add scaling curves for SFT on OSWorld, for two model sizes (Qwen3.5-9B and Qwen3.5-27B) trained with 0%, 25%, 50%, 75% and 100% of our SFT data. At 0% the model is untrained with the hybrid action space.

SFT dataQwen3.5-9BQwen3.5-27B
0%18.448.9
25%28.851.7
50%36.454.6
75%41.857.5
100%46.059.6
Gain, 0% → 100%+27.6+10.7

Both models improve at every step, and neither has saturated at 100%: each additional 25% adds 4.2 to 10.4 points for the 9B model and 2.1 to 2.9 points for the 27B model. The 9B model gains 27.6 points from the data and the 27B model 10.7. The larger model starts much higher (48.9 vs. 18.4) and needs less data to pass its GUI-only counterpart: with 25% of the data, the 27B model reaches 51.7%, above GUI-only Qwen3.5-27B (51.4%), whereas the 9B model passes GUI-only Qwen3.5-9B (38.8%) between 50% and 75% of the data (36.4% and 41.8%). We use the 27B model as the larger scale and do not train a 32B model. We also have not extended the data beyond our current 5K trajectories; since the curves are still rising, we expect more data to help, but we do not claim it.

We have added this as Fig. 12 in Appendix D.2 of the revised paper.

W2 (label granularity): the CLI-advantage label. RCLIR_{\text{CLI}} is a task-level signal that records whether a rollout uses the CLI at all, so it does not separate the two rollouts in the reviewer's example; a finer-grained signal could further improve efficiency. We do not think the label is easy to game, however.

  • The label is hard to game. A task is labeled b⋆=1b^\star=1 only if the CLI is the better strategy on the labeling rollouts (ties in success rate are broken by median step count, and a GUI–CLI win needs a CLI command in more than half of its successful rollouts). The reward also requires success, and on b⋆=0b^\star=0 tasks using the CLI earns no routing reward.
  • The trained behavior does not look like gaming. The CLI share rises from 44% to 64% and average steps fall from 18.0 to 14.0. Directly, we counted the successful rollouts during RL training on the b⋆=1b^\star=1 tasks: only 5% of them contain exactly one CLI command. The pattern the reviewer describes, a single token CLI call to collect the routing reward, is therefore rare, and the policy is not exploiting the label this way.

We have added this check to Appendix C.2 (RLVR Task Construction and CLI Advantage Labeling) of the revised paper.

We sincerely thank Reviewer 58oi again for the constructive feedback. We remain available for any further questions.

Reviewer_7SoaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HybridCUA, a framework for learning to combine GUI and CLI actions into a single policy. The authors introduce a shared action space to represent both the types of actions, provide a data synthesis pipeline for SFT and RL data and then train Qwen3.5-9B using SFT followed by RL. They train using a task completion outcome reward, a CLI-use reward and a local shell-execution penalty. On OSWorld, the model achieves a mean evaluator score of 53.6%, compared with 38.8% for the base model and 50.4% for its trained GUI-only counterpart. It also reaches 36.0% on WindowsAgentArena, versus 32.0% for the base model.

Critical strengths
  1. The paper provides a useful and easy to understand hybrid CUA training data and reward recipe. They provide both SFT and RL data pipelines, followed by a reward function which incorporates both outcome based and execution + CLI based rewards.

  2. The pipeline improves accuracy across OSWorld, OSWorld-MCP, and WindowsAgentArena. On OSWorld, HybridCUA achieves 53.6% accuracy with 14.0 average interaction steps, compared with 50.4% accuracy and 22.1 steps for the trained GUI-only baseline, improving performance while requiring approximately 37% fewer steps.

Critical weaknesses
  1. Lack of evaluations on out of distribution applications: The training data generation primarily uses OSWorld applications plus application-specific teacher skills and guides. All of the evaluations are done on a very similar application set. The additional benchmarks do not fully resolve this concern: OSWorld-MCP reuses OSWorld’s task set, while WindowsAgentArena shares several major applications (Chrome, LibreOffice Calc/Writer, VLC, and VS Code) and also adapts most of its tasks from OS World. Although WAA provides meaningful cross-platform evaluation and includes additional applications, the aggregate results do not isolate transfer to applications absent from training. Reporting WAA performance separately on familiar and additional applications, or evaluating a suitable application-held-out subset of Agents’ Last Exam or WeaveBench without further adaptation is important to see how well the Hybrid CUA pipeline generalizes.

  2. The incremental contribution of the CLI-aware rewards is not sufficiently isolated. Figure 4 removes each reward separately, but does not include an outcome-only RL control with both additional signals removed. The reported accuracy changes from removing either term are small, making it difficult to determine how much the proposed reward design contributes beyond simple outcome based RL.

Questions for authors
  1. Starting from the same SFT checkpoint, what are the results for outcome-only RL with both CLI-aware signals removed? This comparison would help isolate the contribution of the proposed reward design.

  2. What accounts for the reported 9.6 percentage-point accuracy decline on Thunderbird (66.7% to 57.1%, Figure 5a)? Could the authors provide a brief failure analysis, particularly whether inappropriate CLI selection or execution contributes to the decline, and clarify whether it emerges during SFT or RL?

  3. Could the authors specify the inference configs during evals, including the number of historical screenshots, text-history truncation, response-token limit, decoding parameters, and thinking mode? Are the three-frame history settings in Appendix D also used during evaluation. Also, are these same settings used across all baselines?

  4. Could the authors provide the GUI/CLI usage breakdowns in Figures 5–6 for the SFT checkpoint and reward ablations to get a better sense of which training stage leads to change in the interface selection? Please also identify the reference model in Figure 5(a).

7 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We sincerely thank Reviewer 7Soa for the detailed review. We answer the outcome-only control (Q1) and the interface usage by training stage (Q4) here, and the evaluation configuration (Q3), out-of-distribution applications (W1) and the Thunderbird decline (Q2) in Part II.

Q1 / W2: Outcome-only control. Starting from the same SFT checkpoint, with the same GRPO budget and tasks and λCLI=λexec=0\lambda_{\text{CLI}}=\lambda_{\text{exec}}=0, outcome-only RL reaches 51.6% on OSWorld, vs. 53.6% for the full model (single run, final checkpoint).

VariantAcc.Avg. stepsCLI step share (%)CLI error rate (%)
SFT checkpoint46.019.85415.2
RL, outcome reward only51.618.04414.9
RL, w/o RCLIR_{\text{CLI}}52.116.25912.2
RL, w/o rtexecr^{\text{exec}}_t52.914.16016.5
RL, full (HybridCUA-9B)53.614.06411.5

Compared with outcome-only RL, the full model is 2.0 points higher in accuracy (53.6 vs. 51.6), and also takes fewer steps (14.0 vs. 18.0; fewer at every checkpoint), has a lower CLI error rate (11.5% vs. 14.9%) and uses the CLI more (64% of steps vs. 44%; without the CLI-aware rewards the share falls below the SFT checkpoint's 54%). The rewards contribute in different ways:

  • Interface choice. Each CLI-aware reward alone raises the CLI step share above outcome-only RL (59% without RCLIR_{\text{CLI}} and 60% without rtexecr^{\text{exec}}_t, vs. 44%), and together they reach 64%.
  • RCLIR_{\text{CLI}} improves efficiency. Adding it to the outcome reward (w/o rtexecr^{\text{exec}}_t) cuts steps from 18.0 to 14.1. Removing it from the full model costs 2.2 steps (16.2 vs. 14.0) and 1.5 points of accuracy (52.1 vs. 53.6).
  • rtexecr^{\text{exec}}_t improves command reliability. Adding it to the outcome reward (w/o RCLIR_{\text{CLI}}) lowers the CLI error rate from 14.9% to 12.2%. Removing it from the full model raises the error rate from 11.5% to 16.5% and costs 0.7 points of accuracy (52.9 vs. 53.6).

We have added this curve to Fig. 4 and the paragraph after it in Sec. 4.3.

Q4: Interface usage by training stage; reference model. The reference model in Fig. 5(a) is the Qwen3.5-9B base model with GUI-only actions (arrows go from it to HybridCUA-9B); we will state this in the caption. The table gives the CLI step share (%) per domain for the SFT checkpoint and the RL variants; the last column is Fig. 5(b).

DomainSFTOutcome onlyw/o RCLIR_{\text{CLI}}w/o rtexecr^{\text{exec}}_tFull
OS8056748484
VS Code6453707778
LibreOffice Writer5548606667
LibreOffice Calc6152738082
LibreOffice Impress6654646768
GIMP4842363534
Multi-apps7056777480
Chrome1312232126
VLC3935434144
Thunderbird2321413141
Overall5444596064

By stage: SFT sets the starting point (54% overall). RL with the outcome reward alone lowers the CLI share in all ten domains (44% overall), while adding the CLI-aware rewards raises it in nine of ten (64%); GIMP is the only domain below SFT (48% to 34%). Removing either reward gives intermediate values (59% and 60%). This is at the granularity of Fig. 5(b).

AuthorsOpenReview ↗

Rebuttal - Part II

Q3: Evaluation configuration. For HybridCUA-9B, all OSWorld results use:

  • a budget of 50 environment steps per task;
  • greedy decoding (temperature 0, top-p 0.9), at most 2,048 generated tokens per turn, thinking disabled;
  • at most five screenshots in context (older ones are folded one at a time), plus a text history of the last 50 steps;
  • command stdout/stderr returned to the model, truncated to 1,000 characters per turn;
  • a default command timeout of about 30 s, which the model can raise through a timeout argument.

The three-frame setting in the appendix describes training, not evaluation: SFT samples keep the three most recent screenshots and RL rollouts keep at most three, whereas evaluation keeps up to five.

All Qwen3.5-series models, including the GUI-only baselines, are evaluated with the same settings (50 steps, greedy decoding, 2,048 output tokens, five screenshots, 50-step text history, thinking disabled); for the GUI-only models only the action space differs (PyAutoGUI instead of the unified bash action). We have added this configuration to Appendix E (Evaluation Protocols and Metrics) of the revised paper.

W1: Out-of-distribution applications. WindowsAgentArena contains Windows-native applications that OSWorld does not cover. On these 49 tasks, accuracy rises from 44.9 to 55.1 (+10.2), which suggests the method generalizes to applications outside the training distribution. On the 105 tasks in applications that OSWorld covers, it rises from 24.8 to 25.7 (+1.0), so the gain is concentrated on the applications not seen in training. Per-application results:

OOD applications (not covered by OSWorld)

ApplicationBase modelOursΔ
Windows Calculator0.066.7+66.7
Clock0.025.0+25.0
File Explorer42.157.9+15.8
Settings100.0100.00.0
Paint66.766.70.0
Notepad50.050.00.0
Edge46.238.5−7.7
OOD subtotal44.955.1+10.2

Applications also covered by OSWorld

ApplicationBase modelOursΔ
Chrome11.823.5+11.8
VS Code41.745.8+4.2
LibreOffice Calc4.24.20.0
VLC23.814.3−9.5
LibreOffice Writer42.142.10.0
Subtotal24.825.7+1.0

The drop on Edge is a single task: the model fails one more task than the base model (5 solved vs. 6 for the base model, out of 13).

Where the gain comes from. It comes from three applications that are absent from training: File Explorer (+3 tasks), Windows Calculator (+2) and Clock (+1). File Explorer is the largest, where the model solves file creation, moving and copying with shell commands (mkdir/move/copy) in 2–4 steps and the GUI-only baseline often fails. Because these applications were never seen in training, this indicates that the general shell interface transfers to new applications without any per-application tool. On the applications OSWorld covers, the gains on Chrome (+2 tasks) and VS Code (+1) are offset by VLC (−2), leaving the subtotal nearly flat (+1.0).

Our training data is mainly from Linux, and some commands are written in Linux style, with Linux-style paths and syntax, and fail on Windows, which limits the gain on the non-OOD applications. Collecting Windows-native command data is our next step.

We have added this per-application breakdown to Appendix E.1 (Table 10) of the revised paper.

Q2: The Thunderbird decline (Fig. 5a). Thunderbird has only 15 tasks. We compared the trajectories task by task. The two tasks that were lost both used only GUI actions, and none involves CLI selection or execution. In both, the problem is error recovery after a GUI action was blocked and the model tried another route:

  • SFT stage (−1 task). After the right-click menu failed to work, the model completed the operation through the menu bar. The new route changed the currently selected account, so the filter was created under the wrong account.
  • RL stage (−1 task). Facing the same right-click menu problem, the model did not switch routes and repeated the same click until it ran out of steps, whereas the SFT checkpoint switched routes and solved this task.

So one task is lost at each stage, and neither is related to the CLI.

We sincerely thank Reviewer 7Soa again for the constructive feedback. We remain available for any further questions.

Thank you for the additional experiments. The outcome-only control addresses my missing-ablation concern. However, GUI trajectories can vary substantially due to environment noise, timing, and application state. The single-evaluation gains over outcome-only RL (53.6% vs. 51.6%) and the trained GUI-only baseline (53.6% vs. 50.4%) therefore need uncertainty estimates. The same concern applies to WAA, where the improvement on 49 additional-application tasks (44.9% to 55.1%) represents five net additional tasks solved. Could the authors repeat evaluations using the same model checkpoints and unchanged inference settings, reporting mean evaluator scores and 95% confidence intervals for the paired performance differences? I also encourage the authors to report confidence intervals for the main experimental results throughout the paper, particularly benchmark comparisons and ablation studies.

Reviewer_KquAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HybridCUA, a framework for training computer-use agents to combine graphical interaction with command-line execution. The method constructs a dataset containing approximately 5,000 successful trajectories spanning GUI-only, CLI-only, and interleaved interactions, together with 3,000 verifiable RL tasks.

The model is trained through supervised fine-tuning followed by online reinforcement learning. Two CLI-aware rewards encourage appropriate interface selection and penalize command execution failures. The resulting HybridCUA-9B achieves 53.6% accuracy on OSWorld, improving over its base model by 14.8 percentage points. The paper also reports transfer results on OSWorld-MCP and WindowsAgentArena and includes ablations of training data composition, reward components, and action schemas.

Critical strengths

The matched training comparisons provide meaningful evidence that hybrid interaction can improve task completion.

The paper compares GUI-only and GUI+CLI variants derived from the same base model and trained under comparable settings. After supervised fine-tuning and reinforcement learning, the hybrid variant achieves 53.6% OSWorld accuracy compared with 50.4% for its GUI-only counterpart.

The data-composition and action-schema ablations also help characterize which aspects of hybrid training matter. The additional WindowsAgentArena evaluation provides preliminary evidence of transfer beyond the training operating system.

Critical weaknesses
  1. Limited methodological novelty over existing hybrid GUI–tool agents.

Existing papers like ToolCUA already investigate interleaved GUI and programmatic actions, scalable trajectory construction, SFT followed by online RL, and rewards encouraging efficient tool usage. HybridCUA follows a substantially similar training paradigm. Its primary distinction is replacing application-specific tools with shell commands and introducing CLI-aware rewards. The paper does not sufficiently establish why these modifications constitute a substantive methodological advance over existing hybrid-action training frameworks.

  1. The proposed reward design and efficiency claims are not convincingly validated.

The matched GUI-only SFT+RL baseline already achieves 50.4% accuracy, compared with 53.6% for HybridCUA. Removing either CLI-aware reward has limited effect on task accuracy, suggesting that much of the improvement comes from training data and general RL rather than the new rewards. Efficiency is also measured mainly by action counts, although individual CLI commands can execute substantially more work than GUI actions.

Questions for authors

What is fundamentally new compared with works like ToolCUA beyond replacing programmatic tools with shell commands?

Can the authors isolate the task-success contribution of CLI-aware rewards and compare actual execution time under matched settings?

3 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We sincerely thank Reviewer KquA for the careful review. We address the question on novelty relative to ToolCUA here, and the reward ablation and efficiency in Part II.

Q1: Novelty relative to ToolCUA. Beyond swapping application-specific tools for shell commands, HybridCUA differs from ToolCUA in three ways.

  • A general interface instead of per-application tools. ToolCUA relies on structured, application-specific tools, and its authors manually ported 117 MCP tools to Windows. With the shell, no tool set is designed or ported: the same policy can issue PowerShell commands on WindowsAgentArena (36.0%, +4.0 over the base model) and reaches 47.1% on OSWorld-MCP (ToolCUA-8B: 46.8%).
  • A unified harness built around a single bash action. ToolCUA's action space is the union of GUI actions and separate tool calls. In our harness, GUI operations (PyAutoGUI calls in a Python heredoc) and CLI operations (direct shell commands) go through the same bash(command, timeout) action. GUI-only, CLI-only and interleaved trajectories therefore share one format, GUI trajectories can be rewritten into equivalent shell commands, and one call can mix both interfaces. With the same base model and data, this reaches 46.0% after SFT, versus 38.8% when the two are exposed as separate computer_use and cli tools.
  • Tasks built around the GUI/CLI boundary. For each application, an interface guide lists which operations favor the CLI and which require the GUI, and verifiable tasks are generated along this boundary. ToolCUA's RL pool (339 single-domain tasks: 141 from OSWorld, 124 rewrites, 74 new) is, as we read the paper, not built around the two interfaces. We also label each task from 16 rollouts under each of GUI-only, CLI-only and GUI–CLI action spaces.
AuthorsOpenReview ↗

Rebuttal - Part II

Q2a: Isolating the contribution of the CLI-aware rewards. As requested, we add the control with both CLI-aware signals removed: RL from the same SFT checkpoint, with the same GRPO budget and tasks, and λCLI=λexec=0\lambda_{\text{CLI}}=\lambda_{\text{exec}}=0 (outcome reward only). All numbers are from a single OSWorld evaluation run (361 tasks), at the final checkpoint (step 120).

VariantAcc.Avg. stepsCLI step share (%)CLI error rate (%)
SFT checkpoint46.019.85415.2
RL, outcome reward only51.618.04414.9
RL, w/o RCLIR_{\text{CLI}}52.116.25912.2
RL, w/o rtexecr^{\text{exec}}_t52.914.16016.5
RL, full (HybridCUA-9B)53.614.06411.5

Compared with outcome-only RL, the full model is 2.0 points higher in accuracy (53.6 vs. 51.6), and also takes fewer steps (14.0 vs. 18.0; fewer at every checkpoint), has a lower CLI error rate (11.5% vs. 14.9%) and uses the CLI more (64% of steps vs. 44%; without the CLI-aware rewards the share falls below the SFT checkpoint's 54%). The rewards contribute in different ways:

  • Interface choice. Each CLI-aware reward alone raises the CLI step share above outcome-only RL (59% without RCLIR_{\text{CLI}} and 60% without rtexecr^{\text{exec}}_t, vs. 44%), and together they reach 64%.
  • RCLIR_{\text{CLI}} improves efficiency. Adding it to the outcome reward (w/o rtexecr^{\text{exec}}_t) cuts steps from 18.0 to 14.1. Removing it from the full model costs 2.2 steps (16.2 vs. 14.0) and 1.5 points of accuracy (52.1 vs. 53.6).
  • rtexecr^{\text{exec}}_t improves command reliability. Adding it to the outcome reward (w/o RCLIR_{\text{CLI}}) lowers the CLI error rate from 14.9% to 12.2%. Removing it from the full model raises the error rate from 11.5% to 16.5% and costs 0.7 points of accuracy (52.9 vs. 53.6).

We have added this control to Fig. 4 and the paragraph after it in Sec. 4.3 of the revised paper, shown in blue.

Q2b: Efficiency beyond step counts. We agree that a CLI command can do more work than a GUI action, so step counts alone can mislead. We therefore also report the output tokens the model generates and the execution time, per step and per task (OSWorld, averages over all tasks). The per-task time is estimated as the per-step time multiplied by the average number of steps.

ConfigurationAcc.↑Avg. Steps↓Output tokens / stepOutput tokens / task↓Time / stepTime / task↓
Qwen3.5-9B38.831.6832.6k13.3 s7.0 min
HybridCUA-9B-SFT46.019.81523.0k16.3 s5.4 min
HybridCUA-9B53.614.01101.5k14.0 s3.3 min

Each step of HybridCUA-9B takes slightly longer than the base model's (14.0 s vs. 13.3 s) and generates more (110 vs. 83 tokens), consistent with commands carrying more work than single GUI actions. But the number of steps falls by more than half (31.6 to 14.0), so a task needs fewer output tokens (1.5k vs. 2.6k) and less than half the time (3.3 min vs. 7.0 min), while accuracy rises by 14.8 points. The SFT checkpoint has the longest steps (152 tokens, 16.3 s), and RL shortens them while also cutting steps, which is where the drop from 3.0k to 1.5k tokens and from 5.4 to 3.3 min per task comes from. The fewer steps therefore reflect less work, not the same work packed into longer actions.

We sincerely thank Reviewer KquA again for the constructive feedback. We remain available for any further questions.

Dear Reviewer KquA,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns. In brief, our responses clarify the differences from ToolCUA, add the outcome-only RL control, and report efficiency beyond step counts.

If our responses have addressed your concerns, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

The Authors

Comments on the paper

2 comments

Dear reviewers, the authors have posted their responses to your reviews. Please have a look and discuss ASAP as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

AuthorsOpenReview ↗

Summary of Paper Revision

We sincerely thank all reviewers for their insightful and constructive feedback. Following these comments, we have revised the paper and highlighted the main changes in blue. Below, we summarize all changes:

Additional Experimental Results:

  • Outcome-only RL control. We trained a fourth RL variant from the same SFT checkpoint, with the same GRPO budget and tasks, using the outcome reward only. Figure 4 now includes its curves. The paragraph after it in Sec. 4.3 compares this variant with the full model and the two single-reward ablations on accuracy, steps, CLI share and CLI error rate. (Reviewer KquA, Reviewer 7Soa, Reviewer 58oi)
  • Repeated evaluation with confidence intervals. We re-evaluated the key checkpoints three times each with unchanged settings. Appendix E (Table 11) reports the paired differences with paired-bootstrap 95% confidence intervals: HybridCUA-9B against outcome-only RL, GUI-only RL, its SFT checkpoint and the base models. OSWorld is reported as one block, and WindowsAgentArena as three (all tasks, applications not covered by OSWorld, and applications also covered by OSWorld). (Reviewer 7Soa)
  • Per-application breakdown on WindowsAgentArena. Appendix E.1 (Table 10) splits the WindowsAgentArena results by application. It separates Windows-native applications that OSWorld does not cover from those it does, and analyzes where the gain comes from. (Reviewer 7Soa, Reviewer 58oi)
  • Contribution of mixed-interface trajectories. Table 2 adds an SFT variant trained on the GUI-only and CLI-only data combined, without interleaved trajectories. Sec. 4.3 discusses the result. (Reviewer 58oi)
  • SFT data scaling. Appendix D.2 (Figure 12) varies the SFT data from 0% to 100% at two model sizes, Qwen3.5-9B and Qwen3.5-27B. Each size is compared with its GUI-only counterpart. (Reviewer 58oi)
  • Task construction beyond the training domains. We applied the same task-construction pipeline to 10 open-source applications outside the 11 training domains and built 1,000 tasks. We then ran RL on them from the same SFT checkpoint. Appendix C.3 (Table 7) reports the OSWorld results. (Reviewer 58oi)
  • Check of the CLI-advantage label. Appendix C.2 now reports how often successful RL rollouts on CLI-preferred tasks contain exactly one CLI command (5%). (Reviewer 58oi)

Improved Description:

  • A new "Evaluation settings" paragraph in Appendix E specifies the inference configuration: decoding, response-token limit, thinking mode, screenshot and text history, and command-output truncation and timeout. It clarifies that the three-frame setting in Appendix D.1 applies to training only. It also states that all Qwen3.5-series baselines use the same settings, with only the action space differing. (Reviewer 7Soa)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.