- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes to support computer use agents with external manual guides to improve inference and training; for the inference stage, the method majorly enhances retrieval augmented planning, and for the training stage, the method majorly sample episodes with guide from the manual, and use these episodes for subsequent learning. Results show that the proposed method during training and inference stage could work together to improves base CUA performance on OSWorld and Slack.
Critical strengths
-
The paper propose a solid method to use manuals, which could be applied to both inference and training stage, and even working together to improve CUA performance.
-
The paper shows the effectiveness of method though evaluation based on two benchmarks and perform ablation studies which demonstrate the effectiveness of each component.
Critical weaknesses
-
The manuals obtained for this task, is usually written with human effort, and sometimes hardly attainable for a number of scenarios. Besides, the completeness and format of the manuals may affect performance, which is not really comprehensively evaluated and indicated in the paper. Thus, the method is not really generalizable to a number of important CUA tasks that are not covered by OSWorld or Slack.
-
The paper did not mention any related works that focuses on exploration-based CUAs in the related work section, and neither did they use any of the methods in the experimental section for baseline comparisons. Given that RSI and AI4AI is such a popular term, there are already a number of methods which overcome the problem of lacking a manual, and ask the models to explore the unknown environment, build their own skills, and utilize these skills for downstream computer-use tasks. It is important to mention these works and compare with them in terms of cost, performance, and stability.
Questions for authors
1, In Table 3, it seems that policy optimization alone did not really improve the performance alone, can the authors explain why this is happening?
3 comments
Thank you for your response
Dear authors,
I appreciate the effort for adding comparisons with existing RSI approaches, and I now have a better understanding over PO's limited improvement on Calc.
Several concerns still holds given the additional experimental results:
(1) The authors' response in terms of manual availability is still unsatisfying to me, especially given that the RSI approaches evaluated is under a scenario where external manuals are not necessary. I really don't think there is a user manual for any software, maybe there is a user manual for professional ones, but it's not the case for software designed with a nice user interface and for daily usage.
(2) The RSI method in comparison are not really RSI methods, especially for Agent-S2, I think its a rather old model now and it's performance even go over the proposed method without the manual. I understand that currently RSI method may not be not there yet, but at least compare with Agent S3 instead of Agent S2.
(3) Environment exploration-based methods are also under development in the research of computer use agent, they are not currently under comparison either.
Given this situation, I would keep my negative assessment of this paper and think it has a significant problem in terms of being put into real applications.
Response to Reviewer WsoY (2)
We thank the reviewer for the follow-up.
(1) Manual availability. MAGE targets software that comes with a manual, and we will state this scope in the Limitations section.
(2) Comparison with SoTA. Our goal is to demonstrate that MAGE improves existing CUA backbones, rather than to outperform state-of-the-art agents. While a comparison with Agent S3 would be valuable, we believe our current experiments sufficiently support this contribution.
(3) Comparison with exploration-based methods. MAGE adds the manual to standard RL, which is itself an exploration-based method. As shown below, standard RL rarely obtains successful trajectories and fails to learn, whereas MAGE uses the manual to secure successful trajectories and improves performance. This shows that manual guidance turns exploration into an effective learning signal.
| Method | Success rate (Train, 1st round) | Success rate (Test, OSWorld Calc) |
|---|---|---|
| UI-TARS-1.5 | – | 14.47 |
| Standard RL (UI-TARS-1.5) | 12.72 | 8.51 (−5.96) |
| MAGE (UI-TARS-1.5) | 29.53 | 34.04 (+19.57) |
Response to Reviewer WsoY
[W1] Is MAGE generalizable given its reliance on manuals?
We thank the reviewer for raising this point.
(1) Availability. Manuals are commonly provided for software serving a broad user base. MAGE therefore leverages this existing documentation without additional human effort, unlike demonstrations collected specifically for training.
(2) Completeness. We evaluate robustness to manual quality in Appendix C.1 (Table 7) by using an outdated manual (published in 2010) and partial manuals that retain only a subset of the sections.
MAGE outperforms UI-TARS-1.5 (14.47%) in all settings, even with a manual published in 2010 (25.53%) or with only 25% of the sections (17.02%). This demonstrates that MAGE can extract useful procedural guidance even from outdated or incomplete manuals.
(3) Format. MAGE is applied to manuals in different formats, including PDF guidebooks for LibreOffice applications and web-based HTML documentation for GIMP, VSCode, and Slack. This shows that MAGE is applicable across diverse documentation formats.
(4) Generalization. Beyond the desktop applications in OSWorld, MAGE also adapts effectively to Slack, a SaaS application rarely covered by existing CUA benchmarks, with substantially larger gains than on OSWorld (Tables 1 and 2). This demonstrates that MAGE generalizes to new types of software wherever documentation is available.
[W2] Comparison with RSI and AI4AI approaches
We compare MAGE with RSI (SEAgent [1], Agent S2 [2]) and AI4AI (Watch & Learn [3]) approaches. We will add a discussion of these approaches to the related work section.
Performance. MAGE outperforms RSI and AI4AI approaches built on 7B-scale backbones (SEAgent, Watch & Learn). Moreover, MAGE with an 8B backbone achieves the highest average success rate, surpassing Agent S2, which relies on a large proprietary model (Gemini-2.5-Pro) as both the planner and the CUA.
Cost. Agent S2 calls a proprietary model at every step of execution, so its cost grows with every task it performs. AI4AI approaches require large-scale exploration or data collection to build training signals (e.g., 53K video-derived trajectories for Watch & Learn). In contrast, MAGE incurs only a one-time adaptation cost per application, and the adapted policy is a lightweight open model (7B/8B), with the planner invoked only once per task.
Stability. We report MAGE’s results as the mean and standard deviation over five runs (Table 1), which demonstrates consistent performance across runs.
[1] Sun et al., “SEAgent: Self-evolving computer use agent with autonomous learning from experience,” ICML 2026.
[2] Agashe et al., “Agent S2: A compositional generalist-specialist framework for computer use agents,” COLM 2025.
[3] Song et al., “Watch and learn: Learning to use computers from online videos,” CVPR 2026.
[Q1] Why does PO alone show limited improvement on Calc?
To answer the reviewer’s question, we extend the ablation to all five OSWorld applications and Slack with five runs.
PO alone is limited on Calc because Calc tasks depend on task-specific procedures, which RAP provides from the manual but PO alone can hardly internalize. The same holds for other text-centric applications such as Writer and VSCode, whereas PO contributes more in visually demanding applications (Impress, GIMP). MAGE combines both and achieves the best or tied-best performance on every application, showing that the two components are complementary.