- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OS-Helix trains a single policy to use both GUI actions and Python/Bash commands. It combines synthetic desktop tasks, teacher-generated demonstrations, and reinforcement learning with Step-balanced GSPO. The paper evaluates the resulting agents on OSWorld-Verified and WindowsAgentArena.
The results are promising. On OSWorld-Verified, the 27B model improves from 46.50% to 62.75% after SFT and to 66.44% after RL. The main gains come from SFT, with RL providing a further improvement.
Critical strengths
S1: A useful problem with clear practical value. Learning when to use GUI actions and when to use code is important for desktop agents. The code-disabled and GUI-only training ablations help support the hybrid approach.
S2: Good experimental coverage. The paper includes matched-backbone comparisons, teacher and training ablations, cross-platform evaluation, and paired bootstrap confidence intervals.
S3: Detailed implementation and failure analysis. The appendix documents the task-generation pipeline, tools, evaluators, and training setup. It also discusses failed handoffs and evaluator limitations rather than showing only successful examples.
Critical weaknesses
W1: The paper does not report agreement between the synthetic evaluators and independent human judgments. This is important because evaluator scores are used both for filtering training trajectories and as the reward signal for RL. The authors should evaluate a representative subset with independent human judges and report false-accept and false-reject rates, especially for tasks using rule-based, VLM-based, and hybrid evaluators.
W2: Figure 3 provides useful evidence of interaction-step efficiency, including comparisons on jointly successful tasks. However, GUI actions and code calls have different costs, so fewer interactions do not necessarily imply lower latency or compute cost. Reporting end-to-end execution time, generated tokens, and tool-execution time would clarify the practical efficiency gains.
W3: Step-balanced GSPO is promising but narrowly validated. The main controlled comparison uses one dense 27B model, without examining sensitivity to the step-weighting coefficients or different action-token length distributions. Since the method is motivated by the unequal contribution of long code actions and short GUI actions, a controlled experiment varying this length disparity would help test the proposed explanation. The derivation shows that step balancing removes the direct dependence of aggregate step weights on token counts, but this alone does not establish that token length causes the observed drift toward code. Comparing alternative weighting schemes and testing more balanced action-length settings would strengthen the claim.
Questions for authors
-
How often do the synthetic evaluators disagree with independent human judgments? Do conclusions change under different success thresholds?
-
Does Step-balanced GSPO consistently outperform standard GSPO across different action mixes? How are termination/meta actions handled?
-
Do the interaction-step savings also translate into lower wall-clock time and total inference cost?