acceptodds
Under review as a conference paper at ICLR 2027

OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents

Abstract

Real desktop automation is neither purely visual nor purely programmatic. Agents must understand user-facing interfaces while efficiently manipulating files, scripts, and system state.However, native GUI/code coordination cannot be obtained by prompt-level tool composition. In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data. We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space. To construct the required supervision, we introduce application–artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality. We cold-start the policy with SFT on filtered hybrid data and further improve it through online RL in live desktop environments. For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization. On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B achieve relative improvements of 35.0% and 33.7% over their matched Qwen3.5 hybrid baselines, while OS-Helix-27B-RL further improves by 42.9% to reach 66.44% success. Ablations provide evidence consistent with contributions from learned GUI/code coordination, scalable verifiable supervision, and action step balanced optimization, while demonstrating cross-platform transfer on WindowsAgentArena. We will release all research artifacts.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer-Using Agent, GUI Agent, GUI Data Synthesis, Agentic Reinforcement Learning, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_3HvnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OS-Helix trains a single policy to use both GUI actions and Python/Bash commands. It combines synthetic desktop tasks, teacher-generated demonstrations, and reinforcement learning with Step-balanced GSPO. The paper evaluates the resulting agents on OSWorld-Verified and WindowsAgentArena.

The results are promising. On OSWorld-Verified, the 27B model improves from 46.50% to 62.75% after SFT and to 66.44% after RL. The main gains come from SFT, with RL providing a further improvement.

Critical strengths

S1: A useful problem with clear practical value. Learning when to use GUI actions and when to use code is important for desktop agents. The code-disabled and GUI-only training ablations help support the hybrid approach.

S2: Good experimental coverage. The paper includes matched-backbone comparisons, teacher and training ablations, cross-platform evaluation, and paired bootstrap confidence intervals.

S3: Detailed implementation and failure analysis. The appendix documents the task-generation pipeline, tools, evaluators, and training setup. It also discusses failed handoffs and evaluator limitations rather than showing only successful examples.

Critical weaknesses

W1: The paper does not report agreement between the synthetic evaluators and independent human judgments. This is important because evaluator scores are used both for filtering training trajectories and as the reward signal for RL. The authors should evaluate a representative subset with independent human judges and report false-accept and false-reject rates, especially for tasks using rule-based, VLM-based, and hybrid evaluators.

W2: Figure 3 provides useful evidence of interaction-step efficiency, including comparisons on jointly successful tasks. However, GUI actions and code calls have different costs, so fewer interactions do not necessarily imply lower latency or compute cost. Reporting end-to-end execution time, generated tokens, and tool-execution time would clarify the practical efficiency gains.

W3: Step-balanced GSPO is promising but narrowly validated. The main controlled comparison uses one dense 27B model, without examining sensitivity to the step-weighting coefficients or different action-token length distributions. Since the method is motivated by the unequal contribution of long code actions and short GUI actions, a controlled experiment varying this length disparity would help test the proposed explanation. The derivation shows that step balancing removes the direct dependence of aggregate step weights on token counts, but this alone does not establish that token length causes the observed drift toward code. Comparing alternative weighting schemes and testing more balanced action-length settings would strengthen the claim.

Questions for authors
  1. How often do the synthetic evaluators disagree with independent human judgments? Do conclusions change under different success thresholds?

  2. Does Step-balanced GSPO consistently outperform standard GSPO across different action mixes? How are termination/meta actions handled?

  3. Do the interaction-step savings also translate into lower wall-clock time and total inference cost?

Reviewer_gd8JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OS-Helix trains a single GUI/code policy through executable task synthesis, hybrid SFT, and Step-balanced GSPO. Controlled experiments show better task completion and transfer from Ubuntu training to Windows testing.

Critical strengths
  • Controls using the same hybrid action space, a fixed supervision budget, and matched RL settings support meaningful gains from training. Improvements across base models and transfer to Windows make this an empirical contribution worth accepting.
  • Step aggregation makes a simple, well-defined change to remove action-length weighting from the sample ratio. Its improvement over standard GSPO under matched settings supports its value as a training design.
Critical weaknesses
  • The RL effect is clearer than its attribution. Table 4(b) supports the proposed objective over GSPO in the reported setting. Reweighting also changes the sample ratio and its clipping behavior. The paper does not report distributions of action counts and lengths in actual training segments, or clipping statistics. The objectives coincide for single-action segments. Direct evidence is still needed to establish how much the benefit depends on segmentation, or whether length weighting primarily explains the code drift. This limits the scope of the central algorithmic claim.
Questions for authors
  • How many actual training segments contain multiple actions, and how many mix GUI with code? Do existing action-length or clipping logs support length weighting as the main source of code drift? This would affect my assessment of the proposed mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.