acceptodds
Under review as a conference paper at ICLR 2027

Adaptation of Computer Use Agents to New Software with User Manuals

Abstract

Recent advances in multimodal foundation models have significantly improved the ability to operate real-world software through graphical user interfaces (GUIs) using visual observations and natural language instructions. However, maintaining strong performance on unseen applications remains challenging, as practical tasks often require long-horizon procedural knowledge and application-specific operation patterns. In addition, constructing executable tasks and collecting corresponding trajectories for training still heavily rely on human-annotated data, incurring substantial cost. In this paper, we propose MAGE (Manual-augmented computer use AGent Execution and learning), a framework that leverages user manuals as a scalable source of procedural knowledge to improve both inference and training. During execution, MAGE retrieves task-relevant manual content and uses a planner to generate explicit procedural guidance for CUAs. To further account for the discrepancies between manuals and actual GUI environments, such as version differences or user settings, MAGE also adapts the agent policy through environment interaction. During training, MAGE converts manuals into executable task-plan pairs and uses the plans to guide environment rollouts toward task-relevant states; the resulting trajectories are then used to optimize the policy without human-authored demonstrations. This unified design enables effective utilization of manuals for both task execution and training, reducing reliance on human-authored training data. Experimental results across six applications demonstrate improvements in task success rates over strong baselines, highlighting the effectiveness of manual-guided planning and policy adaptation in new software environments.

Primary area:
reinforcement learning
Keywords:
Computer Use Agents, User Manuals, Task Planning, Policy Adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WsoYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes to support computer use agents with external manual guides to improve inference and training; for the inference stage, the method majorly enhances retrieval augmented planning, and for the training stage, the method majorly sample episodes with guide from the manual, and use these episodes for subsequent learning. Results show that the proposed method during training and inference stage could work together to improves base CUA performance on OSWorld and Slack.

Critical strengths
  1. The paper propose a solid method to use manuals, which could be applied to both inference and training stage, and even working together to improve CUA performance.

  2. The paper shows the effectiveness of method though evaluation based on two benchmarks and perform ablation studies which demonstrate the effectiveness of each component.

Critical weaknesses
  1. The manuals obtained for this task, is usually written with human effort, and sometimes hardly attainable for a number of scenarios. Besides, the completeness and format of the manuals may affect performance, which is not really comprehensively evaluated and indicated in the paper. Thus, the method is not really generalizable to a number of important CUA tasks that are not covered by OSWorld or Slack.

  2. The paper did not mention any related works that focuses on exploration-based CUAs in the related work section, and neither did they use any of the methods in the experimental section for baseline comparisons. Given that RSI and AI4AI is such a popular term, there are already a number of methods which overcome the problem of lacking a manual, and ask the models to explore the unknown environment, build their own skills, and utilize these skills for downstream computer-use tasks. It is important to mention these works and compare with them in terms of cost, performance, and stability.

Questions for authors

1, In Table 3, it seems that policy optimization alone did not really improve the performance alone, can the authors explain why this is happening?

3 comments
AuthorsOpenReview ↗

Response to Reviewer WsoY

[W1] Is MAGE generalizable given its reliance on manuals?

We thank the reviewer for raising this point.

(1) Availability. Manuals are commonly provided for software serving a broad user base. MAGE therefore leverages this existing documentation without additional human effort, unlike demonstrations collected specifically for training.

(2) Completeness. We evaluate robustness to manual quality in Appendix C.1 (Table 7) by using an outdated manual (published in 2010) and partial manuals that retain only a subset of the sections.

ManualSuccess Rate (%)
Full (2021)34.04
Old (2010)25.53
Partial (2021 - 75%)25.53
Partial (2021 - 50%)21.28
Partial (2021 - 25%)17.02
None14.47

MAGE outperforms UI-TARS-1.5 (14.47%) in all settings, even with a manual published in 2010 (25.53%) or with only 25% of the sections (17.02%). This demonstrates that MAGE can extract useful procedural guidance even from outdated or incomplete manuals.

(3) Format. MAGE is applied to manuals in different formats, including PDF guidebooks for LibreOffice applications and web-based HTML documentation for GIMP, VSCode, and Slack. This shows that MAGE is applicable across diverse documentation formats.

(4) Generalization. Beyond the desktop applications in OSWorld, MAGE also adapts effectively to Slack, a SaaS application rarely covered by existing CUA benchmarks, with substantially larger gains than on OSWorld (Tables 1 and 2). This demonstrates that MAGE generalizes to new types of software wherever documentation is available.

[W2] Comparison with RSI and AI4AI approaches

We compare MAGE with RSI (SEAgent [1], Agent S2 [2]) and AI4AI (Watch & Learn [3]) approaches. We will add a discussion of these approaches to the related work section.

MethodBackboneCalcImpressWriterGIMPVSCodeOverall
SEAgent [1]UI-TARS-7B-DPO–22.731.842.340.5–
Watch & Learn [3]UI-TARS-1.5-7B17.0234.0439.1365.3865.2239.16
Agent S2 [2]Gemini-2.5-Pro44.6836.0969.5765.3869.5752.39
MAGEUI-TARS-1.5-7B34.0445.3944.3560.7766.0947.31
MAGEEvoCUA-8B37.0245.3250.4381.5476.5253.67

Performance. MAGE outperforms RSI and AI4AI approaches built on 7B-scale backbones (SEAgent, Watch & Learn). Moreover, MAGE with an 8B backbone achieves the highest average success rate, surpassing Agent S2, which relies on a large proprietary model (Gemini-2.5-Pro) as both the planner and the CUA.

Cost. Agent S2 calls a proprietary model at every step of execution, so its cost grows with every task it performs. AI4AI approaches require large-scale exploration or data collection to build training signals (e.g., 53K video-derived trajectories for Watch & Learn). In contrast, MAGE incurs only a one-time adaptation cost per application, and the adapted policy is a lightweight open model (7B/8B), with the planner invoked only once per task.

Stability. We report MAGE’s results as the mean and standard deviation over five runs (Table 1), which demonstrates consistent performance across runs.

[1] Sun et al., “SEAgent: Self-evolving computer use agent with autonomous learning from experience,” ICML 2026.

[2] Agashe et al., “Agent S2: A compositional generalist-specialist framework for computer use agents,” COLM 2025.

[3] Song et al., “Watch and learn: Learning to use computers from online videos,” CVPR 2026.

[Q1] Why does PO alone show limited improvement on Calc?

To answer the reviewer’s question, we extend the ablation to all five OSWorld applications and Slack with five runs.

OSWorldSlack
SettingCalc (%)Impress (%)Writer (%)GIMP (%)VSCode (%)Overall (%)Slack (%)
UI-TARS-1.514.47 ± 6.4538.15 ± 3.3840.87 ± 2.3853.85 ± 3.8556.52 ± 5.3236.83 ± 3.0926.67 ± 5.27
PO only15.74 ± 3.2345.33 ± 2.8540.87 ± 2.3856.15 ± 3.4465.22 ± 3.0740.78 ± 0.7936.00 ± 7.23
RAP only29.79 ± 2.6143.25 ± 3.5644.35 ± 3.6450.77 ± 4.2166.09 ± 4.7643.93 ± 1.3547.33 ± 9.25
MAGE34.04 ± 3.6945.39 ± 1.9044.35 ± 1.9460.77 ± 5.7066.09 ± 3.6447.31 ± 2.0249.33 ± 11.88

PO alone is limited on Calc because Calc tasks depend on task-specific procedures, which RAP provides from the manual but PO alone can hardly internalize. The same holds for other text-centric applications such as Writer and VSCode, whereas PO contributes more in visually demanding applications (Impress, GIMP). MAGE combines both and achieves the best or tied-best performance on every application, showing that the two components are complementary.

Reviewer_WsoYOpenReview ↗

Thank you for your response

Dear authors,

I appreciate the effort for adding comparisons with existing RSI approaches, and I now have a better understanding over PO's limited improvement on Calc.

Several concerns still holds given the additional experimental results:

(1) The authors' response in terms of manual availability is still unsatisfying to me, especially given that the RSI approaches evaluated is under a scenario where external manuals are not necessary. I really don't think there is a user manual for any software, maybe there is a user manual for professional ones, but it's not the case for software designed with a nice user interface and for daily usage.

(2) The RSI method in comparison are not really RSI methods, especially for Agent-S2, I think its a rather old model now and it's performance even go over the proposed method without the manual. I understand that currently RSI method may not be not there yet, but at least compare with Agent S3 instead of Agent S2.

(3) Environment exploration-based methods are also under development in the research of computer use agent, they are not currently under comparison either.

Given this situation, I would keep my negative assessment of this paper and think it has a significant problem in terms of being put into real applications.

AuthorsOpenReview ↗

Response to Reviewer WsoY (2)

We thank the reviewer for the follow-up.

(1) Manual availability. MAGE targets software that comes with a manual, and we will state this scope in the Limitations section.

(2) Comparison with SoTA. Our goal is to demonstrate that MAGE improves existing CUA backbones, rather than to outperform state-of-the-art agents. While a comparison with Agent S3 would be valuable, we believe our current experiments sufficiently support this contribution.

(3) Comparison with exploration-based methods. MAGE adds the manual to standard RL, which is itself an exploration-based method. As shown below, standard RL rarely obtains successful trajectories and fails to learn, whereas MAGE uses the manual to secure successful trajectories and improves performance. This shows that manual guidance turns exploration into an effective learning signal.

MethodSuccess rate (Train, 1st round)Success rate (Test, OSWorld Calc)
UI-TARS-1.5–14.47
Standard RL (UI-TARS-1.5)12.728.51 (−5.96)
MAGE (UI-TARS-1.5)29.5334.04 (+19.57)
Reviewer_S8CiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAGE, a framework for adapting computer-use agents to new GUI software using user manuals. The core idea is to treat manuals as scalable procedural supervision. At inference time, MAGE retrieves task-relevant manual sections and uses a planner to generate an explicit step-by-step plan for the agent. At training time, it converts manual sections into executable task-plan pairs, collects trajectories through interaction with the GUI environment, evaluates success automatically, and optimizes the agent policy with GRPO.

The main contributions are:

  • A unified manual-augmented framework for both execution-time planning and training-time adaptation of GUI agents.
  • A retrieval-augmented planning method that grounds task execution in relevant user-manual content.
  • A manual-derived task generation and policy optimization pipeline that reduces reliance on human-authored demonstrations or step-level annotations.
  • An empirical evaluation across several desktop applications, including LibreOffice apps, GIMP, VSCode, and Slack.
Critical strengths

a) The paper tackles a real bottleneck for computer-use agents: adapting to new software without lots of human-labeled trajectories. Using user manuals as supervision is a natural and scalable idea.

b) Nice unified framework. MAGE uses manuals both at inference time, through retrieval-augmented planning, and at training time, through manual-derived tasks and policy optimization. This makes the approach more complete than just adding external knowledge at test time.

c) Solid empirical evidence. The method is tested on multiple real applications and two different CUA backbones. The ablations, qualitative examples, and cost analysis help support the main claims.

Critical weaknesses

a) The paper combines several existing ideas: retrieving user manuals, using an LLM to generate plans, generating synthetic GUI tasks, and applying GRPO for policy optimization. This is a sensible system design, but the paper does not clearly show which part is the truly new technical contribution. As a result, the work feels more like a useful engineering integration than a major methodological advance.

b) MAGE uses powerful LLMs/VLMs for task generation, planning, and trajectory evaluation. These components may contribute a large part of the final performance gain. This makes it hard to know whether the improvement comes from the proposed adaptation framework itself, or simply from using stronger external models to provide better plans and judgments. This is consequential because the claimed effectiveness of MAGE may not transfer if weaker or cheaper models are used.

c) The training pipeline relies on a VLM-based evaluator to decide whether a trajectory succeeds. GUI task success can be subtle: a final screen may look correct while some hidden state is wrong, or the evaluator may miss small but important errors. If the evaluator is noisy, then both the reward signal for GRPO and some reported success rates could be unreliable. This directly affects the trustworthiness of the experimental results.

d) Some ablation results appear to be based on limited settings or single runs, and the component analysis is not broad enough across all applications. This weakens the evidence for which parts of MAGE are truly responsible for the gains.

e) Although the paper tests several applications, many are benchmark-style desktop tasks, and the Slack setup is newly constructed by the authors. It is not fully clear whether the method would work robustly on messier real-world software with changing interfaces, permissions, and user states.

Questions for authors

a) How accurate is the VLM-based success evaluator compared with human or rule-based evaluation across the different applications? Since it provides training rewards and supports some reported results, its error rate could materially affect the conclusions.

b) How much of MAGE’s improvement remains if weaker or cheaper LLMs/VLMs are used for planning, task generation, and evaluation? This would help separate the contribution of the framework from the strength of the external models.

c) Can the authors provide broader ablations across more applications and multiple runs, especially for RAP-only, PO-only, and full MAGE? This would make it clearer which components are consistently responsible for the gains.

2 comments
AuthorsOpenReview ↗

Response to Reviewer S8Ci (1/2)

[W1] Technical contribution of MAGE

MAGE's core contribution is overcoming the bootstrapping barrier of adapting CUAs to unfamiliar GUI applications. Existing approaches face two challenges: (1) SFT requires costly demonstrations, and (2) directly applying RL to manual-derived tasks yields few successful trajectories.

Our paper addresses these challenges by using manuals to guide both planning and training. Rather than merely generating tasks from manuals, MAGE also generates procedural plans that substantially improve the agent's initial success rate. To examine whether plans are needed during training, we train UI-TARS-1.5 with standard GRPO on the same manual-derived Calc tasks without plans.

MethodSuccess rate (Train - 1st Round)Success rate (Test - OSWorld)
GRPO without plans12.728.51 (−5.96)
GRPO with plans (MAGE)29.5334.04 (+19.57)

GRPO without plans performs below the backbone (-5.96 pp) on OSWorld Calc, while MAGE improves substantially. This shows that plans generated from the manual are what make RL on manual-derived tasks effective.

In addition, the resulting policy learns not only to operate the new application but also to follow manual-derived plans more effectively during inference (see [W4, Q3]).

Thus, MAGE goes beyond combining existing components: manual-guided planning enables successful exploration, which in turn enables policy optimization, making adaptation possible without expensive demonstrations.

Note. Numbers in parentheses denote the difference from UI-TARS-1.5 before training.

[W2, Q2] Do the gains of MAGE come from strong external LLMs?

To identify where the gain comes from, we remove manual retrieval so that the same planner generates plans without the manual.

PlannerRAPSuccess Rate (%)
None–15.74 ± 3.23
Qwen3.6-27B✗24.11 ± 4.43
Qwen3.6-27B (Ours)✓34.04 ± 3.69

The results confirm that retrieval-augmented planning (RAP) provides substantial benefits beyond the planner’s capability, improving the success rate from 24.11% to 34.04%. We also note that using a 27B planner adds little overhead in practice, since the planner is called only once per task (Sec. 4.1).

We replace the planner on OSWorld Calc to check the effect of planner size.

Table. Effect of planner size on OSWorld Calc.

PlannerSuccess Rate (%)
CUA itself (Fine-tuned)12.77
Qwen3-VL-8B18.44
Qwen3.5-9B20.57
Qwen3.6-27B34.04
Qwen3.8-27B33.33
GPT-5.6-sol31.21

As the planner size increases, the success rate rises and then plateaus. Planners with 9B or fewer parameters yield low success rates. Once the planner reaches 27B, open-source models perform comparably to GPT-5.6-sol. This shows that generating a plan requires a sufficient level of ability to understand the manual.

We re-evaluated the trajectories using Qwen3.6-27B and trained the agent with the resulting signals.

EvaluatorCalcImpressWriterGIMPVSCodeOverall
Qwen3.6-27B31.2143.8244.9358.9762.3245.34
GPT-5.4 (Ours)34.0445.3944.3560.7766.0947.31

MAGE (UI-TARS-1.5) achieves 45.34% with Qwen3.6-27B, close to 47.31% with GPT-5.4, showing that MAGE is not sensitive to evaluator performance.

[W3, Q1] Is the VLM evaluator reliable?

OSWorld evaluates each task with a task-specific script that directly inspects the final state of the environment. We use OSWorld’s rule-based evaluation as the ground truth and compare the VLM evaluator against it on five applications.

ProgramCalcImpressWriterGIMPVSCodeOverall
Agreement82.98% (39/47)73.33% (33/45)86.36% (19/22)80.95% (17/21)77.27% (17/22)79.62% (125/157)

The two agree on about 80% of the tasks on average. Moreover, when trained with rewards from the VLM evaluator, MAGE improves the rule-based success rate on all five applications. Together, these results show that the VLM evaluator is reliable.

AuthorsOpenReview ↗

Response to Reviewer S8Ci (2/2)

[W4, Q3] Broader ablation across applications and runs

RAP and PO are complementary. MAGE combines both and achieves the best performance. To show this, we extended the ablation to all five OSWorld applications and Slack, with five runs.

OSWorldSlack
SettingCalc (%)Impress (%)Writer (%)GIMP (%)VSCode (%)Overall (%)Slack (%)
UI-TARS-1.514.47 ± 6.4538.15 ± 3.3840.87 ± 2.3853.85 ± 3.8556.52 ± 5.3236.83 ± 3.0926.67 ± 5.27
PO only15.74 ± 3.2345.33 ± 2.8540.87 ± 2.3856.15 ± 3.4465.22 ± 3.0740.78 ± 0.7936.00 ± 7.23
RAP only29.79 ± 2.6143.25 ± 3.5644.35 ± 3.6450.77 ± 4.2166.09 ± 4.7643.93 ± 1.3547.33 ± 9.25
MAGE34.04 ± 3.6945.39 ± 1.9044.35 ± 1.9460.77 ± 5.7066.09 ± 3.6447.31 ± 2.0249.33 ± 11.88

RAP and PO contribute in different ways. RAP contributes more in text-centric applications (Calc, Writer, VSCode), where manuals provide procedural knowledge such as menu paths and shortcuts. PO contributes more in visually demanding applications (Impress, GIMP). MAGE combines the two and captures both gains.

[W5] Robustness on real-world applications

Our evaluation on Slack demonstrates MAGE's applicability to live, real-world applications. Slack is a live SaaS platform with real authentication, workspace permissions, and an evolving interface, rather than a controlled benchmark environment. As shown below, MAGE achieves substantial gains on Slack.

BenchmarkUI-TARS-1.5 (backbone → MAGE, +gain)EvoCUA (backbone → MAGE, +gain)
OSWorld36.83 → 47.31 (+10.48)47.54 → 53.67 (+6.13)
Slack26.67 → 49.33 (+22.66)32.67 → 51.33 (+18.66)

Furthermore, MAGE is designed to address the mismatch between manuals and the actual GUI interfaces. PO trains the agent through interaction with the real environment, so the agent learns what works in the current interface.

More complex applications are harder for any agent. Given the Slack results, we expect MAGE to remain helpful there. Evaluation on a broader range of applications is left for future work.

Reviewer_8pQFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MAGE, which augments agent execution with dynamically retrieved sections of manuals for the target applications. The manuals are also used to generate tasks by defining a setup task and a main task. Fine-tuning two backbone models with MAGE results in higher performance on the OSWorld benchmark and substantially higher performance on the newly designed Slack benchmark for testing generalization. Ablations show the complementary value of training with both execution-time manual retrieval and policy optimization.

Critical strengths
  1. MAGE demonstrates how manuals can be used to improve both inference-time performance and the training process.
  2. The new Slack environment evaluates generalization beyond the standard OSWorld tasks, which baselines may have overfit to, as data for these applications is abundant in training datasets.
  3. The paper demonstrates how tasks can be derived from manuals to synthesize new training data for applications.
  4. The paper reports standard deviations over five random seeds to establish the reliability of the results.
Critical weaknesses
  1. A major limitation of the work is its reliance on a powerful planning model to convert retrieved sections of the manual into a plan. The agent ultimately executes based on this plan, so the performance gains could be entirely attributable to the planner rather than the instruction manual. It is critical to compare against a baseline that uses the same planner without manual retrieval.
  2. The paper does not provide RL training curves, making it difficult to assess learning stability and reproduce the results. I suggest the authors include these in the supplementary material for at least the main MAGE training approaches, showing the training task success rate as a function of the number of RL updates.
  3. The proposed approach has limited novelty compared to prior work, particularly AgentTrek and RAG-GUI. A key difference is that MAGE also incorporates retrieval during RL training. Given the similarity to these works, I believe it is important to conduct a more comprehensive comparison against the RAP baseline in the same setting as Table 1, across all applications and multiple runs.
  4. The experiments do not isolate how much of the improvement comes from training on newly synthesized data versus training and evaluating the policy with RAP. Another important baseline for Table 1 would be to train directly on the synthesized tasks using standard RL without RAP.
  5. The new Slack environment lacks important details, such as the complexity of the data staged in the application and how environment resets are handled. Given that this is a new evaluation environment where MAGE demonstrates large empirical gains, the paper should sufficiently explain all aspects of the evaluation setup.
  6. The gains in Table 2 are expected and may largely result from training on tasks from the Slack application, whereas the baselines are not trained on these tasks.
  7. The paper does not evaluate whether manuals represent a scalable source of task generation, demonstrating performance only at a single task budget of at most 260 tasks per application.
  8. The paper does not analyze the properties of manual-derived tasks. For example, how do task horizons and complexity compare to those of other GUI training datasets? Do manual-derived tasks involve shorter, more atomic actions because they are instantiated from individual sections of the manual?
  9. The paper does not discuss the limitations of the proposed approach.

Minor:

  1. The citations appear as regular text and are also colored black.
Questions for authors

Also see the questions raised in the weaknesses section.

  1. Can the same policy model function as the planner instead of introducing a separate planning model?
  2. How does performance change as a function of the number of tasks generated per application using the manual-based task synthesis process? Does performance plateau at the current number of generated tasks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8qQF (1/2)

[W1, Q1] Does the improvement come from the planner or RAP?

(1) Same planner without manual (W1). To identify where the gain comes from, we remove manual retrieval so that the same planner generates plans without the manual. In this setting, the planner receives only the task instruction and the initial screenshot.

Table 1. Effect of the manual on retrieval-augmented planning (RAP) on OSWorld Calc.

PlannerManualSuccess Rate (%)
None–15.74
Qwen3.6-27B✗24.11 (+8.37)
Qwen3.6-27B (Ours)✓34.04 (+18.30)

The results confirm that RAP provides substantial benefits beyond the planner’s capability, improving the success rate from 24.11% to 34.04%. These results show that MAGE’s gain comes mainly from the manual retrieved by RAP. We also note that using a 27B planner adds little overhead in practice, since the planner is called only once per task (Sec. 4.1).

(2) Policy model (CUA) as planner (Q1). We also replace the planner with the same policy model.

Table 2. Using the CUA itself as the planner on OSWorld Calc.

PlannerSuccess Rate (%)
CUA itself (Fine-tuned)12.77
Qwen3.6-27B (Ours)34.04

Executing with plans from the policy model is even worse than using no plan. This is because CUAs are trained mainly to produce actions rather than plain text, so the policy model often generates degenerate plans with repetitive procedural steps.

These results show that MAGE’s improvement comes from RAP.

[W2] RL training curve

The table below reports the training success rate at each round of MAGE with UI-TARS-1.5.

Table 3. Training success rate of MAGE (UI-TARS-1.5).

Application1st Round2nd Round3rd Round
CalcRL Steps01,7974,013
Success rate (%)29.530.032.9
ImpressRL Steps06161,379
Success rate (%)35.141.147.9
WriterRL Steps06291,377
Success rate (%)28.327.630.5
GIMPRL Steps0260595
Success rate (%)24.328.027.0
VSCodeRL Steps06351,056
Success rate (%)15.014.618.8
SlackRL Steps0248602
Success rate (%)18.324.431.7

The success rate tends to increase steadily as training rounds progress. We will add these results to Appendix A.2 as training curves.

Note. The success rate is the main task success rate, computed per task in the same way as our Slack evaluation (Sec. 4.1). Steps are cumulative training steps.

[W3] Comparison with the RAP baseline.

To compare MAGE with the RAP baseline in the same setting as Table 1 of the paper, we extend the ablation to all five OSWorld applications and Slack over five runs.

Table 4. Extended ablation study on OSWorld and Slack (mean ± std).

OSWorldSlack
ModelCalcImpressWriterGIMPVSCodeOverallSlack
UI-TARS-1.514.47 ± 6.4538.15 ± 3.3840.87 ± 2.3853.85 ± 3.8556.52 ± 5.3236.83 ± 3.0926.67 ± 5.27
MAGE (UI-TARS-1.5, RAP only)29.79 ± 2.6143.25 ± 3.5644.35 ± 3.6450.77 ± 4.2166.09 ± 4.7643.93 ± 1.3547.33 ± 9.25
MAGE (UI-TARS-1.5)34.04 ± 3.6945.39 ± 1.9044.35 ± 1.9460.77 ± 5.7066.09 ± 3.6447.31 ± 2.0249.33 ± 11.88

MAGE outperforms the RAP baseline on every application. This shows that MAGE’s use of manuals during training adds to what the manual provides at execution time. We have also completed the PO-only experiments on all applications over five runs and updated Table 3 of the revised paper with these results.

[W4] Comparison with a standard RL baseline.

To compare standard RL with MAGE, we train UI-TARS-1.5 with standard GRPO on the same manual-derived Calc tasks without plans.

Table 5. Comparison with a standard RL baseline (Calc).

MethodSuccess rate (Train - 1st Round)Success rate (Test - OSWorld)
UI-TARS-1.5-14.47
Standard RL (UI-TARS-1.5)12.728.51 (−5.96)
MAGE (UI-TARS-1.5)29.5334.04 (+19.57)

Standard RL performs below the backbone on OSWorld Calc, while MAGE improves substantially. Without plans, the success rate during training is low. As a result, training becomes ineffective. MAGE raises the training success rate through plans. This shows that plans generated from the manual are what make RL on manual-derived tasks effective.

AuthorsOpenReview ↗

Response to Reviewer 8qQF (2/2)

[W6] Slack evaluation gains reflect adaptation to unfamiliar software.

The Slack evaluation is intended to show how effectively MAGE adapts a CUA to unfamiliar software. The gain from MAGE is much larger on Slack than on OSWorld, which shows that MAGE adapts well to unfamiliar applications. With UI-TARS-1.5, MAGE improves the success rate by 22.66 pp on Slack, compared to 10.48 pp on OSWorld.

Table 6. Gain from MAGE on OSWorld and Slack

BenchmarkUI-TARS-1.5 (backbone → MAGE, +gain)EvoCUA (backbone → MAGE, +gain)
OSWorld36.83 → 47.31 (+10.48)47.54 → 53.67 (+6.13)
Slack26.67 → 49.33 (+22.66)32.67 → 51.33 (+18.66)

The starting points differ, but the end points are similar. Before adaptation, UI-TARS-1.5 performs much worse on Slack than on OSWorld. After adaptation with MAGE, it reaches a similar level on both. MAGE thus closes the gap on an application that was unfamiliar to the agent. EvoCUA shows the same trend.

[W7, Q2] Scalability of manual-derived tasks

To evaluate whether manuals represent a scalable source of task generation, we retrain MAGE on Calc with sampled subsets of the training tasks.

Table 7. OSWorld success rate on Calc by the number of training tasks.

# of training tasks0 (RAP-only)100200228 (MAGE)
Success rate (%)29.7926.9532.6234.04

Performance improves as more manual-derived tasks are used. From 100 to 228 tasks, the success rate rises steadily. We attribute this to coverage. A small task set covers fewer functions of the application, so the policy is tuned to a narrow set of functions. The gain appears once the tasks cover more functions.

Note. We sort the tasks by their source manual sections, divide them into equal intervals, and sample one task from each interval.

[W8] Difficulty and horizon of manual-derived tasks

To compare the difficulty and horizon of the manual-derived tasks with OSWorld tasks, we evaluate both with the same agent, MAGE (RAP-only).

Table 8. Success rate and average steps of MAGE (RAP-only) on manual-derived tasks and OSWorld tasks.

CalcImpressWriterGIMPVSCodeOverall
SR (%), Manual-derived tasks29.5335.1228.2524.3015.0024.67
SR (%), OSWorld29.7943.2544.3550.7766.0943.93
Avg. steps (successful), Manual-derived tasks17.6618.5514.4117.7423.5018.50
Avg. steps (successful), OSWorld12.6011.4813.4623.6720.8416.13

On average, manual-derived tasks have a lower success rate and require more steps than OSWorld tasks. This shows that manual-derived tasks are more challenging than the benchmark tasks.

[W5, W9, Minor W1] Paper updates

We thank the reviewer for pointing these out. We have added the details of the Slack evaluation setup to Appendix B (W5) and a discussion of limitations (W9) to the revised paper, and fixed the citation formatting.

Evaluation environment and protocol. To ensure reproducibility and prevent interference across evaluation runs, each of the five independent runs is conducted in a newly created Slack workspace. All workspaces share an identical initial configuration, consisting of a #general channel, a pre-joined member named “Alex,” and an agent account with Owner/Admin privileges. Any additional task-specific data is created through the corresponding setup task. Within each run, the 30 tasks are executed in a fixed order, and the Slack client in the Docker container is restarted before each task to reset the UI state, including open windows, dialogs, and unsent input. The tasks are designed to be independent, such that the success or failure of preceding tasks does not affect the evaluation of subsequent tasks.

Limitation. Our framework depends on the availability and quality of application manuals. Incomplete or outdated manuals and irrelevant retrieval results can make the generated plans and training tasks less reliable, although our sensitivity analysis shows that MAGE remains effective under moderate degradation. Manuals are also organized around individual functions, which limits the complexity of the tasks generated from them. The plan is generated only once per task, which can make it difficult to recover from mistakes during execution. Future work may explore agent-oriented rewriting of manuals, composing tasks across manual sections, and replanning during execution.

Thank you for the detailed response. These responses greatly strengthen the paper and sufficiently address the core of each of my weaknesses listed above, so I raise my score. I suggest including these important new ablations on the full evaluation set in the final paper.

Comments on the paper

2 comments

Dear Reviewers,

As the rebuttal period will end in less than 24 hours, we would appreciate it if you could take another look at our responses and let us know whether they have addressed your concerns.

We would be happy to provide any further clarification before the deadline.

Best regards, The Authors

AuthorsOpenReview ↗

Summary of Rebuttal for the Area Chair

Dear Area Chair,

We thank the reviewers and the AC. We summarize the discussion below.

  • Contribution of each component. We extended the ablation to all applications and Slack over five runs. On OSWorld overall, policy optimization (PO) adds +3.95 pp and retrieval-augmented planning (RAP) adds +7.10 pp, and MAGE combining both adds +10.48 pp over the backbone. This shows that each component contributes and that the two are complementary.
  • Effect of procedural knowledge from the manual. We further examined how the manual affects training and execution on Calc. In training, plans from RAP raise the training success rate by +16.81 pp over standard RL, which makes RL effective. In execution, the retrieved manual improves the success rate of the same planner by +9.93 pp.
  • Revision. The final version will include the new ablations, training curves, Slack environment details, Limitations, and related work on RSI and exploration-based CUAs.

We have shown that each component of MAGE contributes to the performance gain. Therefore, we believe that MAGE is a practical method for adapting CUAs to new software by leveraging existing documentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.