- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This work proposes GUIVista, a framework that converts software tutorial videos into data for GUI-agent mid-training. The pipeline combines inverse-dynamics action prediction with action reconstruction and action-specific verification to improve the recovery of long-tail interactions. Under the same downstream SFT protocol, adding GUIVista data to mid-training improves OSWorld performance from 18.40% to 21.75% and also improves performance on ScreenSpot-Pro. The paper additionally introduces the GUIVista diagnostic, referred to as PACE in the appendix, an offline action-selection evaluation whose scores are reported to correlate with OSWorld performance.
Critical strengths
This paper proposes a method to convert unsupervised software-using to GUI-agent training data. Experiments show that mid-training with the resulting data improves downstream performance on OSWorld and ScreenSpot-Pro under a shared SFT protocol.
Critical weaknesses
- The main idea is similar to VideoAgentTrek: turning software tutorial videos into GUI-agent training data. The added reconstruction and verification steps have ablations, but their benefit over simpler annotation methods with a similar budget remains unclear. A direct-annotation baseline using another strong VLM, such as Gemini, would help assess whether this complexity is necessary.
- The OSWorld gain is modest (18.40% to 21.75%). There is also limited analysis of how the model’s actual task-solving behavior changes and what accounts for this improvement.
- The paper does not clearly show whether the broader training data help the model understand software better or handle unseen applications and tasks.
Questions for authors
- Are there any comparisons with direct VLM annotation or simple self-correction under similar inference budgets, including both annotation accuracy and cost?
- Which evaluation applications or task types are unseen in the mid-training data, and how does the model perform on these subsets compared with SFT-only?
- Did mid-training on the generated data introduce any helpful or harmful patterns in the model’s behavior?
5 comments
Response to W2&Q3 Part1
1. Fine-Grained Capability and Behavioral Analysis. We thank the reviewer for this suggestion. We have added a detailed trajectory-level behavioral analysis to the revised manuscript, examining action diversity, execution effectiveness, repetition, trajectory efficiency, and reasoning-action consistency across the Base, 400K, and 800K checkpoints.
2. Capability Improvements. Beyond the OSWorld improvement from 18.40% to 21.75%, our offline diagnostic shows that the 400K checkpoint improves overall performance from 22.96% to 34.54% , with particularly large gains in long-horizon execution , rare-action recognition , complex tasks , and fine-grained grounding .
3. Behavioral Changes from Base to 400K: More Diverse and Effective Actions. We analyze the action predictions recorded in OSWorld execution trajectories. For each checkpoint, action-type shares and behavioral rates are calculated by dividing the number of occurrences in each category by the total number of recorded actions. The Base model is heavily dominated by clicking , whereas the 400K checkpoint exhibits a more diverse action distribution, including keyboard operations, text entry, and scrolling. Meanwhile, the effective-action rate increases from 43.3% to 84.3%, while repeated-action and no-op rates decrease from 82.6% to 19.8% and 56.6% to 15.2%, respectively, indicating a substantial shift toward more diverse and effective execution.
| Behavioral Metric | Base | 400K | 800K |
|---|---|---|---|
| Click proportion | 90.60% | 47.90% | 51.10% |
| Keyboard proportion | 1.70% | 18.50% | 16.60% |
| Scroll proportion | 0.25% | 7.80% | 7.30% |
| Effective actions | 43.30% | 84.30% | 87.40% |
| Repeated actions | 82.60% | 19.80% | 19.70% |
| No-op operations | 56.60% | 15.20% | 12.00% |
| Invalid negative coordinates | 0.25% | 6.27% | 5.65% |
These results demonstrate that mid-training significantly improves action diversity and execution effectiveness, rather than merely increasing the number of actions. However, the higher frequency of invalid coordinates indicates remaining challenges in action grounding.
Response to W2&Q3 Part2
4. Behavioral Changes from 400K to 800K: More Efficient Successes but Persistent Failures. Continued mid-training primarily improves successful execution efficiency, rather than extending planning horizons. As OSWorld AVG@4 increases from 20.55% to 21.75%, the model gains success on 17 tasks while losing success on another 17, with higher scores across repeated rollouts on newly successful and consistently passing tasks outweighing the losses. We analyze execution trajectories under a maximum budget of 50 steps per task, with the step-limit rate indicating the proportion of trajectories reaching this limit. We further analyze emerging action-sequence patterns in agent trajectories, finding that mid-training substantially expands multi-action combinations (e.g., click→type→enter: 3.6% → 30.0%), reflecting a shift from repetitive clicking toward more diverse and structured task execution.
| Execution Metric | 400K | 800K |
|---|---|---|
| Overall average steps | 34.05% | 33.31% |
| Successful trajectory steps | 22.34% | 19.46% |
| Failed trajectory steps | 39.40% | 39.64% |
| Newly successful tasks: average steps | 29.21% | 19.24% |
| Newly successful tasks: 50-step limit rate | 44.10% | 19.10% |
| Newly failed tasks: average steps | 28.24% | 33.41% |
| Newly failed tasks: 50-step limit rate | 39.70% | 55.90% |
5. Beneficial Pattern: More Efficient Successful Execution. From 400K to 800K, successful trajectories become 12.9% shorter , while newly successful tasks require 34.1% fewer steps . These results indicate that continued mid-training improves execution efficiency rather than extending planning horizons.
6. Harmful Pattern: Persistent Execution Failures. Failed trajectories remain approximately 39.5 steps, while the 50-step limit rate for the 17 newly failed tasks increases from 39.7% to 55.9%. This indicates that some tasks remain prone to prolonged unsuccessful execution despite the overall improvements.
7. Reasoning-Action Consistency. From 400K to 800K, average rationale length decreases from 371.7 to 350.3 characters, while rationale-action consistency improves from 45.9% to 48.2%. This suggests more concise reasoning with better alignment to executed actions.
8. Summary and Manuscript Updates. Our analysis reveals two main benefits of mid-training: improved action diversity and effectiveness from Base to 400K, and more efficient successful execution from 400K to 800K. Increased coordinate errors and persistent failure trajectories remain limitations, potentially reflecting stage-specific training fluctuations similar to the temporary performance decline observed at 25K–50K. Further scaling may help recover these capabilities, although the underlying mechanisms require additional investigation. We have incorporated these quantitative findings and behavioral analyses into the revised manuscript.
Dear Reviewer Qztn, thank you for your valuable comments.
Response to W1&Q1
1. Updated Evaluation and Cost Analysis. As detailed in our response to Reviewer otSZ, Q3, we have incorporated the original IDM evaluation with a more reliable CUA-Suite-based benchmark, covering 1,000 samples across 82 applications. Here, we focus on the trade-off between annotation accuracy and inference cost.
2. API and Local GPU Costs. Gemini 3.5 Flash-Lite achieve 49.10% overall action type accuracy under our direct-annotation configuration, requiring one call per sample at approximately $0.0011. According to Gemini's official pricing, the respective input/output prices are $0.30/$2.50 per million tokens. Our local annotation pipeline runs on 8×A100 GPUs without API fees. Using a Cloud reference price of $1.80/GPU-hour and assuming a sustained throughput of 32 calls/second across eight GPUs, the estimated compute cost is $0.000124 per call, or $0.000868 per sample for the full pipeline ≈69˙5calls. Actual costs depend on deployment throughput.
3. Annotation Accuracy and Cost Comparison. We further compare direct VLM annotation, confidence-based filtered, reconstruction, and expert verification on the updated CUA-Suite evaluation set:
4. Effectiveness and Necessity. Under the same one-call budget, our direct annotation achieves 21.40% Type+Arg accuracy, outperforming Gemini's 17.33% at a lower estimated cost. Reconstruction-only further improves accuracy to 22.18%, while expert-only achieves 23.83%. Combining both yields the highest accuracy ∗∗248˙9, improving upon direct annotation by 3.49 percentage points and Gemini by 7.56 percentage points, with a lower estimated per-sample compute cost than Gemini's measured API cost.
These results demonstrate that reconstruction and expert verification provide complementary improvements beyond direct annotation and simple filtered, while quantifying their additional inference overhead. We have incorporated the updated evaluation and cost comparisons into the revised manuscript.