- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents HybridCUA, which trains computer-use agents to coordinate GUI and CLI interactions to improve task success and efficiency. The authors construct SFT trajectories comprising GUI-only, CLI-only, and mixed-interface interactions, then train the model through SFT followed by reinforcement learning with a reward favoring CLI use and a penalty for command execution errors. HybridCUA-9B achieves an average task score of 53.6 on OSWorld, a 14.8 percentage-point improvement over the original GUI model, while reducing the average number of action steps from 31.6 to 14.0. The model also shows some improvement on WindowsAgentArena.
Critical strengths
- Combining GUI and CLI interactions is a promising direction for computer-use agents. The experiments show that simply giving an agent CLI access can hurt performance on computer-use tasks, making a clear case for explicitly training GUI–CLI coordination. This finding provides a coherent motivation for the proposed approach.
- The two-stage SFT and RL training procedure is well motivated and supported by the results. Although the authors do not collect mixed GUI–CLI trajectories directly, they obtain encouraging results using GUI-only and CLI-only trajectories together with constructed mixed-interface data.
Critical weaknesses
- The paper lacks experiments assessing how HybridCUA scales with model size and training data. For example, it would be useful to compare 9B and 32B models and vary the amount of hybrid data from zero to 5K trajectories, ideally extending beyond 5K, which remains a relatively small dataset. I consider this an important omission. Such experiments would establish how much hybrid data is needed to match the baseline and whether additional data continues to improve both task performance and efficiency.
- The routing reward is somewhat coarse, and the CLI-advantage label appears easy to game. For the same task, a trajectory that completes the task efficiently using CLI throughout and a much longer trajectory containing only one trivial CLI operation may receive the same label. The former is clearly preferable, but the label does not distinguish between them, limiting the signal available during RL for learning this distinction. The reward design also needs an ablation in which hybrid RL uses only the task completion reward, to isolate the contributions of the routing reward and the command execution error penalty.
- I support the research direction and the overall approach of improving GUI–CLI coordination through training. However, the data and environment construction could be developed further. The SFT data primarily comes from UI-MOPD, trajectory collection using CUA-Gym, and rewrites of existing trajectories; the verifiable RL tasks are also built on CUA-Gym. These choices have practical value, but the paper does not adequately explain how to systematically construct tasks and trajectories that capture the complementary strengths of GUI and CLI and require coordination between them. For hybrid computer-use agents, constructing such data is itself an important research problem. For example, how should tasks combine visual judgment, programmatic operations, and the transfer of state between interfaces, beyond replacing existing GUI actions with commands? A proposed and validated pipeline for constructing these tasks and environments, with evidence that it extends to a broader range of tasks and applications, would better support the scalability and generality of hybrid coordination and strengthen the paper’s contribution and central argument.
- All figures appear to use raster graphics, possibly exported as PNG files from the drawing software. As a result, they become blurry when enlarged. The authors should replace them with vector graphics where appropriate.
Questions for authors
- The roughly four-point improvement on WindowsAgentArena seems modest. Could the authors analyze which tasks or task categories account for this gain, and whether the model makes effective use of CLI on this benchmark?
- What is the contribution of the mixed-interface training data? Does combining GUI-only and CLI-only data already improve hybrid performance, and how much additional benefit comes from the mixed-interface trajectories?
5 comments
Rebuttal - Part II
W4: Figures. We will replace the raster plots and diagrams with vector graphics (PDF) in the revision. Figures that embed real screenshots, such as the case studies, will keep the screenshots as raster images inside a vector layout.
Q1: Source of the WindowsAgentArena gain. We broke the WAA results down by application (table below). The overall gain is +4.0. On Windows-native applications that OSWorld does not cover, the gain is +10.2, which suggests the method generalizes to applications outside the training distribution.
OOD applications (not covered by OSWorld)
| Application | Base model | Ours | Δ |
|---|---|---|---|
| Windows Calculator | 0.0 | 66.7 | +66.7 |
| Clock | 0.0 | 25.0 | +25.0 |
| File Explorer | 42.1 | 57.9 | +15.8 |
| Settings | 100.0 | 100.0 | 0.0 |
| Paint | 66.7 | 66.7 | 0.0 |
| Notepad | 50.0 | 50.0 | 0.0 |
| Edge | 46.2 | 38.5 | −7.7 |
| OOD subtotal | 44.9 | 55.1 | +10.2 |
Applications also covered by OSWorld
| Application | Base model | Ours | Δ |
|---|---|---|---|
| Chrome | 11.8 | 23.5 | +11.8 |
| VS Code | 41.7 | 45.8 | +4.2 |
| LibreOffice Calc | 4.2 | 4.2 | 0.0 |
| VLC | 23.8 | 14.3 | −9.5 |
| LibreOffice Writer | 42.1 | 42.1 | 0.0 |
| Subtotal | 24.8 | 25.7 | +1.0 |
The gain is concentrated in applications absent from training: File Explorer (+3 tasks), Windows Calculator (+2) and Clock (+1). On File Explorer, the model uses shell commands (mkdir/move/copy) to create, move and copy files in 2–4 steps, where the GUI-only baseline often fails. The Chrome gain (+2) comes from settings tasks that the model completes entirely through the GUI. The drop on Edge is a single task: the model fails one more task than the base model (5 solved vs. 6 for the base model, out of 13). Paint, Notepad and Settings are unchanged.
Interface usage and limitation. The model picks the interface by task type. It mainly calls the CLI where commands fit (file operations, VS Code configuration, spreadsheet handling; about 34–38% of steps overall) and mainly uses the GUI for browser and system settings. The gain on file management comes directly from the CLI. The limitation is that our training data is mainly from Linux, and some commands are written in Linux style, with Linux-style paths and syntax, and fail on Windows. This limits the overall gain on the non-OOD applications of WAA. Collecting Windows-native command data is our next step.
We have added this per-application breakdown to Appendix E.1 (Table 10) of the revised paper.
Q2: Contribution of mixed-interface trajectories. We add one row to our SFT data ablation on OSWorld: the pure-GUI and pure-CLI data combined, without any interleaved trajectories.
| Configuration | Acc.↑ | Avg. Steps↓ |
|---|---|---|
| Qwen3.5-9B | 38.8 | 31.6 |
| w/ CLI only | 31.7 | 19.1 |
| w/ GUI only | 43.2 | 29.5 |
| w/ GUI + CLI only (no hybrid) | 39.8 | 25.2 |
| w/ Hybrid only | 41.0 | 22.6 |
| HybridCUA-9B-SFT | 46.0 | 19.8 |
Combining the pure-GUI and pure-CLI data does help, but only slightly: 39.8% vs. 38.8% for the base model (+1.0), and below GUI-only data alone (43.2%). The mixed-interface trajectories add much more: all data together reach 46.0%, i.e., +6.2 over the combination without them, and steps drop from 25.2 to 19.8. The three parts are complementary. Each pure set covers only one interface: CLI-only data alone falls below the base model (31.7%), and GUI-only data never shows when a command is the better choice. Merging them gives the model both skills but no signal for choosing between them within a task, so the combination gains little. Mixed trajectories supply exactly this signal, and they already beat the merged pure data on their own (41.0% vs. 39.8%). They cover fewer single-interface skills, though, so the best result comes from the pure data providing the skills for each interface and the mixed trajectories showing how to combine them.
We have added this row to Table 2 and the sentence after it in Sec. 4.3.
Rebuttal - Part III
W1: Scaling. We add scaling curves for SFT on OSWorld, for two model sizes (Qwen3.5-9B and Qwen3.5-27B) trained with 0%, 25%, 50%, 75% and 100% of our SFT data. At 0% the model is untrained with the hybrid action space.
| SFT data | Qwen3.5-9B | Qwen3.5-27B |
|---|---|---|
| 0% | 18.4 | 48.9 |
| 25% | 28.8 | 51.7 |
| 50% | 36.4 | 54.6 |
| 75% | 41.8 | 57.5 |
| 100% | 46.0 | 59.6 |
| Gain, 0% → 100% | +27.6 | +10.7 |
Both models improve at every step, and neither has saturated at 100%: each additional 25% adds 4.2 to 10.4 points for the 9B model and 2.1 to 2.9 points for the 27B model. The 9B model gains 27.6 points from the data and the 27B model 10.7. The larger model starts much higher (48.9 vs. 18.4) and needs less data to pass its GUI-only counterpart: with 25% of the data, the 27B model reaches 51.7%, above GUI-only Qwen3.5-27B (51.4%), whereas the 9B model passes GUI-only Qwen3.5-9B (38.8%) between 50% and 75% of the data (36.4% and 41.8%). We use the 27B model as the larger scale and do not train a 32B model. We also have not extended the data beyond our current 5K trajectories; since the curves are still rising, we expect more data to help, but we do not claim it.
We have added this as Fig. 12 in Appendix D.2 of the revised paper.
W2 (label granularity): the CLI-advantage label. is a task-level signal that records whether a rollout uses the CLI at all, so it does not separate the two rollouts in the reviewer's example; a finer-grained signal could further improve efficiency. We do not think the label is easy to game, however.
- The label is hard to game. A task is labeled only if the CLI is the better strategy on the labeling rollouts (ties in success rate are broken by median step count, and a GUI–CLI win needs a CLI command in more than half of its successful rollouts). The reward also requires success, and on tasks using the CLI earns no routing reward.
- The trained behavior does not look like gaming. The CLI share rises from 44% to 64% and average steps fall from 18.0 to 14.0. Directly, we counted the successful rollouts during RL training on the tasks: only 5% of them contain exactly one CLI command. The pattern the reviewer describes, a single token CLI call to collect the routing reward, is therefore rare, and the policy is not exploiting the label this way.
We have added this check to Appendix C.2 (RLVR Task Construction and CLI Advantage Labeling) of the revised paper.
We sincerely thank Reviewer 58oi again for the constructive feedback. We remain available for any further questions.
Rebuttal - Part I
We sincerely thank Reviewer 58oi for the thorough review. We answer the reward ablation (W2) and data and environment construction (W3) here; figure format (W4), the WindowsAgentArena gain (Q1) and mixed-interface data (Q2) in Part II; and scaling (W1) and the CLI-advantage label (W2) in Part III.
W2: Task-completion-only ablation. Starting from the same SFT checkpoint, with the same GRPO budget and tasks, we run hybrid RL with the task-completion reward only (λCLI=λexec=0), and with each CLI-aware term removed separately (single OSWorld run, final checkpoint).
Compared with task-completion-only RL, the full model is 2.0 points higher in accuracy (53.6 vs. 51.6), and also takes fewer steps (14.0 vs. 18.0; fewer at every checkpoint), has a lower CLI error rate (11.5% vs. 14.9%) and uses the CLI more (64% of steps vs. 44%; without the CLI-aware rewards the share falls below the SFT checkpoint's 54%). The rewards contribute in different ways:
We have added these curves to Fig. 4 and the paragraph after it in Sec. 4.3.
W3: Data and environment construction. We agree that building tasks that require GUI–CLI coordination is a research problem in itself, and we do not claim to have solved it. Our current procedure is as follows.
We fully agree that mocking up our own out-of-distribution environments and synthesizing environments at scale is a better approach to systematic task and environment construction, and we leave it as future work.
We have added this pipeline extension and its RL results to Appendix C.3 (Table 7) of the revised paper.