acceptodds
Under review as a conference paper at ICLR 2027

TutorialGUI: Towards Generalizable GUI Agents via Tutorial-Driven Skill Learning

Abstract

GUI agents have made rapid progress in operating general-purpose software, yet they still face out-of-distribution (OOD) generalization challenges when deployed on unfamiliar specialized applications, where insufficient application-specific knowledge and operational experience limit reliable task completion. Scientific software exemplifies this challenge. However, adapting agents through per-application data collection and fine-tuning is costly, limiting the scalability of this approach to long-tail applications. Official tutorials offer readily available domain knowledge and procedural examples, but they are primarily written for humans, making them difficult for agents to use directly, and their coverage is limited. To address these limitations, we introduce TutorialGUI, a training-free framework that transforms official tutorials into executable and extensible Atomic Skills to improve GUI agents’ generalization to long-tail applications. At its core is a Replay–Explore loop: Replay converts tutorial procedures into validated skills through actual GUI execution and outcome verification; Explore builds on these validated skills to discover and verify related capabilities, extending coverage beyond the original tutorials. The resulting skills can be retrieved and reused by different GUI agents without updating model parameters. On scientific software tasks in ScienceBoard, TutorialGUI improves task success rates by an average of 14.285 percentage points across multiple models compared with no-tutorial baselines, demonstrating the effectiveness of transforming human-oriented tutorials into verifiable and extensible capabilities for GUI interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, out-of-distribution generalization, tutorial-driven skill learning, training-free adaptation, atomic skills, scientific software

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z9evOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TutorialGUI, a training-free framework that converts software tutorials into reusable skills and Knowledge Cards. A Replay–Exploration loop refines and verifies skills through GUI execution and expands the library with additional skills. The resulting resources are retrieved to guide downstream agents. Experiments on six ScienceBoard applications demonstrate improved task success across three executor models.

Critical strengths
  1. The proposed framework improves average task success over the corresponding baseline agents.
  2. It equips GUI agents with specialized, reusable skills without updating model parameters.
Critical weaknesses
  1. Limited baseline comparisons. Closely related methods such as MMG2Skill [1] and VisualSkill [2] are not evaluated. Comparisons with these methods and a strong GUI agent under matched conditions would clarify the method’s added value.
  2. Confounded ablations. Replay changes both procedures and images, while Exploration adds skills and interactions. Fixed-modality ablations and budget-matched comparisons with simpler exploration would help isolate their contributions.
  3. Unclear generalization scope. Each application receives a dedicated skill library, while task novelty relative to construction examples is unclear. Tests on new inputs, initial states, and skill combinations would better establish generalization.
  4. Missing costs and budgets. Construction costs, round-level exploration budgets, and downstream execution limits are insufficiently reported.
Questions for authors

See Weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Z9ev

We sincerely appreciate the reviewer for the constructive feedback and for recognizing the performance gains and skill augmentation without parameter updates. Below, we address your concerns with additional comparisons, ablations, and details of the experimental protocol.

[W1] "Limited baseline comparisons"

We have added comparisons with MMG2Skill, VisualSkill, and Resource2Skill in Table R1. TutorialGUI outperforms all three methods, achieving improvements of 10.15, 6.31, and 10.21 percentage points, respectively.

All comparisons use the same configuration: GPT-5.5, the official evaluator, and the same evaluation environment. We run each task three times and report the average performance. We will add these results to Table 1 of the revised paper.

We attribute TutorialGUI's advantages to its Replay–Exploration loop, which refines individual skills and continually expands the skill library through interaction with the real environment. MMG2Skill does not include a dedicated exploration stage for expanding skill coverage. VisualSkill does not include Replay and uses free exploration and guided exploration rather than multi-directional Exploration. Resource2Skill expands its skills by searching for additional resources, whereas our method expands the skill library through interaction with the real environment.

Table R1: Method comparisons.

MethodKAlgebraCelestiaChimeraXGRASS GISTeXstudioLean 4Avg
Baseline12.9020.2044.8312.7525.003.8519.92
RAG40.8621.2151.728.8220.833.8524.55
MMG2Skill48.3919.1950.5722.554.173.8524.79
VisualSkill52.6927.2758.6217.6510.425.1328.63
Resource2Skill29.0325.2556.3223.5310.423.8524.73
TutorialGUI64.5224.2463.2222.5531.253.8534.94

[W2] "Confounded ablations"

(1) "Fixed-modality ablations"

Regarding the Replay ablation study, we would like to clarify that the procedure and reference image obtained during Replay are tightly coupled: a new procedure produces a new reference image. Therefore, we cannot ablate them separately.

(2) "Budget-matched comparisons with simpler exploration"

We have also added a comparison with budget-matched simpler exploration, which replaces three-direction Exploration with free exploration, i.e., the model freely generates new candidates from anchor skills without any directional guidance. As shown in Table R2, TutorialGUI achieves an average success rate of 34.94%, outperforming Simple Exploration (30.78%) by 4.16 percentage points.

Table R2: Simple Exploration comparison.

MetricKAlgebraCelestiaChimeraXGRASS GISTeXstudioLean 4Avg
Simple Exploration60.2224.2458.6212.7525.003.8530.78
TutorialGUI64.5224.2463.2222.5531.253.8534.94
Gain (percentage points)+4.300.00+4.60+9.80+6.250.00+4.16

[W3] "Task novelty relative to construction examples"

We note that TutorialGUI itself is a training-free approach and does not rely on any tasks from the benchmark during skill construction. Therefore, all tasks within the benchmark can be regarded as unseen tasks.

[W4] "Missing costs and budgets"

(1) "Construction costs"

Table R3 reports the total input and output tokens consumed by Replay and Exploration during skill library construction, totaling 107.072M tokens.

Table R3: Skill library construction costs (million tokens, including input and output).

MetricKAlgebraCelestiaChimeraXGRASS GISTeXstudioLean 4Total
Replay8.52919.55633.97820.70210.05811.009103.832
Exploration0.7120.3200.3630.7750.5610.5093.240
Construction subtotal9.24119.87634.34121.47710.61911.518107.072

(2) "round-level exploration budgets"

Please refer to Appendices A.1 and A.4.6 (Exploration Candidate-Generation Prompt). Each Exploration round stops when the target number of newly admitted verified skills is reached. Table R3 reports the total Exploration token consumption during construction.

(3) "downstream execution limits"

Downstream evaluation follows the steps limits in ScienceBoard's official task configurations, ranging from 5 to 20 interaction rounds per task. Please refer to ScienceBoard for the evaluation configurations of individual applications and tasks.

Reviewer_9QQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents TutorialGUI, a training-free framework that converts official software tutorials into reusable GUI skills and Knowledge Cards. Its core Replay–Exploration loop uses real GUI interactions to refine and verify extracted skills, then discovers additional skills through forward, backward, and lateral exploration. The resulting skill library can be shared across different GUI agents without updating model parameters. Experiments on six ScienceBoard applications show an average improvement of 13.42 percentage points across three executor models, with additional gains from continued exploration.

Critical strengths

This paper has below strengths:

  • A practical and meaningful problem. The paper addresses an important limitation of current GUI agents: operating specialized software that requires substantial domain knowledge. Acquiring reusable skills from official documentation, rather than fine-tuning a model for every application, is a practical direction.

  • A sensible skill acquisition framework. The combination of tutorial extraction, execution-based verification, and iterative exploration is well motivated. I particularly like the idea of verifying skills in the actual GUI environment before admitting them into the library, rather than simply trusting procedures extracted from documentation.

  • Encouraging results across different models. The improvements are fairly consistent across GPT-5.5 and Qwen3.8-Max, suggesting that the acquired skills can benefit different executors. The gains are also substantial in several scientific applications, which supports the practical value of the approach.

  • Useful analysis of the skill expansion process. The stage-wise experiments provide some evidence for the benefits of Replay and Exploration. I also appreciate that the authors report the non-monotonic scaling behavior in Table 3, which reveals an important challenge in managing growing skill libraries.

Critical weaknesses

Though the paper is interesting, there are some concerns should be addressed:

  • The novelty relative to recent work is not entirely convincing. Several closely related papers have already explored extracting reusable skills from tutorials and improving them through interaction, including [1][2][3]. TutorialGUI’s verification-before-admission strategy and three-direction exploration are potentially useful distinctions, but it is not sufficiently demonstrated how much these particular designs advance beyond existing approaches.

  • The experiments do not fully isolate the contributions of the proposed mechanisms. The main baselines lack external tutorial knowledge, so the improvements do not necessarily establish the advantage of TutorialGUI over simpler documentation retrieval or skill construction methods. Moreover, the stage-wise ablation in Table 2 changes both procedural quality and visual references between candidate and replayed, while exploration also increases the number of available skills. This makes it difficult to determine how much of the gain comes specifically from Replay and the proposed exploration strategy.

  • The reliability of the verified skills remains unclear. Replay verification is performed on individual execution instances using a VLM-based verifier, but passing one attempt does not necessarily imply that a parameterized skill will work reliably with different inputs or application states. The paper does not provide an independent assessment of verification accuracy or the robustness of admitted skills. Since verified skills are the foundation of the subsequent exploration process, I consider this an important missing validation.

  • The evidence for generalization is somewhat limited. All six applications are explicitly processed during library construction, and the downstream experiments use the same application environments and software configurations. The results demonstrate transfer of application-specific knowledge across executor models, but provide less evidence that the framework generalizes to genuinely new applications, workflows, or interface variations. The performance on some domains, particularly Lean 4, also remains quite limited despite skill augmentation.

Refs:

  1. MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?
  2. VisualSkill: Multimodal Skills for Computer-Use Agents
  3. Resource2Skill: Distilling Executable Agent Skills from Human-Created Multimodal Resources
Questions for authors

Some minor questions:

  • What is the approximate cost of constructing a skill library for each application, in terms of model calls, GUI interactions, or runtime?
  • Also, has the drop in performance after Exploration round R5 been investigated to distinguish skill-selection errors from problems with the expanded library itself?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 9QQr

We sincerely appreciate the reviewer for recognizing the value of the problem, execution-based verification, and skill-expansion analysis. We address your concerns below.

[W1] "Novelty relative to recent work"

Table R1 adds comparisons with MMG2Skill, VisualSkill, and Resource2Skill. TutorialGUI exceeds them by 10.15, 6.31, and 10.21 percentage points, respectively.

All comparisons use the same configuration: GPT-5.5, the official evaluator, and the same evaluation environment, averaging three runs per task. These results will be added to the main table.

We attribute TutorialGUI's advantages to its Replay–Exploration loop, which refines individual skills and continually expands the skill library through interaction with the real environment. MMG2Skill does not include a dedicated exploration stage for expanding skill coverage. VisualSkill does not include Replay and uses free exploration and guided exploration rather than multi-directional Exploration. Resource2Skill expands its skills by searching for additional resources, whereas our method expands the skill library through interaction with the real environment.

Table R1: Method comparisons.

MethodKAlgebraCelestiaChimeraXGRASS GISTeXstudioLean 4Avg
Baseline12.9020.2044.8312.7525.003.8519.92
RAG40.8621.2151.728.8220.833.8524.55
MMG2Skill48.3919.1950.5722.554.173.8524.79
VisualSkill52.6927.2758.6217.6510.425.1328.63
Resource2Skill29.0325.2556.3223.5310.423.8524.73
TutorialGUI64.5224.2463.2222.5531.253.8534.94

[W2] "Contributions of the proposed mechanisms"

(1) "RAG baseline"

Using the same tutorials, RAG improves the average success from 19.92% to 24.55%, while TutorialGUI reaches 34.94%, exceeding RAG by 10.39 percentage points (Table R1).

(2) "Table 2 changes both procedural quality and visual references"

Procedural quality and visual references both change solely through Replay, so their joint changes do not affect the assessment of Replay's contribution. In addition, they are tightly coupled: a new procedure produces a new reference image, so we cannot ablate them separately.

(3) "Exploration increases the number of skills"

The purpose of Exploration is to expand skill coverage, so the increase in the number of skills is itself part of Exploration's contribution.

[W3] "Robustness of admitted skills"

We tested all 190 explicitly parameterized skills using four new, distinct parameter inputs per skill. Of 760 executions with new parameters, 746 passed verification (98.16%; Table R2), showing high robustness to different inputs.

Table R2: Verification results for parameterized skills under parameter changes.

MetricKAlgebraCelestiaChimeraXGRASS GISTeXstudioLean 4Total
Parameterized skills9737972119190
Verified / Changed-input runs34/3627/28145/148383/38882/8475/76746/760
Pass rate (%)94.4496.4397.9798.7197.6298.6898.16

[W4] "Generalizes to genuinely new applications"

We note that TutorialGUI itself is a training-free approach and does not rely on any data from the benchmark. Therefore, all tasks within the benchmark can be regarded as unseen tasks.

We would like to clarify the definition of generalization in our scenario: like MMG2Skill, VisualSkill, and Resource2Skill, TutorialGUI relies on an application's tutorials and interactive environment. Generalization to new software therefore assumes access to its tutorials and an interactive environment, rather than requiring adaptation without any resources.

[Q1] "Skill construction cost"

Across six applications, skill library construction costs 107.072M tokens. Table R3 reports the total input and output tokens consumed by Replay and Exploration during construction: 103.832M and 3.240M, respectively. Construction is needed only once per application.

Table R3: Skill library construction costs (million tokens, including input and output).

MetricKAlgebraCelestiaChimeraXGRASS GISTeXstudioLean 4Total
Replay8.52919.55633.97820.70210.05811.009103.832
Exploration0.7120.3200.3630.7750.5610.5093.240
Construction Total9.24119.87634.34121.47710.61911.518107.072

[Q2] "Drop in performance after Exploration round R5"

Through qualitative analysis, we found that the decline in performance on KAlgebra after R5 was mainly caused by Selector errors.

Reviewer_nULZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents TutorialGUI, a training-free framework that leverages official software tutorials to improve GUI agents' performance on specialized applications requiring domain-specific knowledge and operational experience.

The framework first uses a vision-language model (VLM) to extract Knowledge Cards containing application-specific information and candidate skills describing operational goals, preconditions, parameters, and procedures. These skills are organized into a graph capturing execution dependencies and exploration relationships.

The core contribution is an iterative Replay–Exploration pipeline designed to address two limitations of software tutorials: incomplete procedures and limited coverage.

  • Replay: Executes candidate skills in the actual GUI environment, resolves missing steps or prerequisites, and uses a VLM to verify whether the intended outcomes were achieved. Successfully verified skills are refined and added to the library.
  • Exploration: Expands the skill library by discovering new candidate skills from previously verified skills, application observations, and relevant documentation. Exploration proceeds in three directions: forward (downstream operations), backward (prerequisite operations), and lateral (related or alternative operations). Newly discovered candidates undergo Replay verification before being admitted.

The resulting library of verified skills and Knowledge Cards can be retrieved by different GUI agents at inference time without updating model parameters.

The authors evaluate TutorialGUI on ScienceBoard, comprising 169 tasks across six scientific applications. They report an average improvement of 13.42 percentage points in task success across three foundation models compared with their respective unaugmented baselines.

Critical strengths
  • The paper is well-motivated, and mostly easy to follow.

  • The experimental results are promising, with consistent improvements across three model families. The evaluation includes genuinely specialized scientific applications, such as ChimeraX and GRASS GIS, demonstrating the potential of tutorial-derived skills and knowledge to improve GUI-agent performance in unfamiliar, domain-specific software.

Critical weaknesses
  • Limited novelty and missing comparisons with closely related methods. The proposed pipeline has substantial overlap with existing training-free, skill-based approaches such as VisualSkill and Resource2Skill. However, the paper does not experimentally compare against these methods, focusing instead on unaugmented foundation models and training-based agents such as ScaleCUA and SEAgent. This makes it difficult to assess the novelty and added value of the proposed Replay–Exploration pipeline relative to existing approaches.

  • Unclear source of performance improvements. It is difficult to determine whether the gains come primarily from acquiring new operational capabilities through Replay–Exploration or simply from retrieving relevant tutorial instructions at inference time. The latter could potentially be achieved through a simpler retrieval-augmented approach. A baseline that directly retrieves relevant tutorial content, without skill construction or exploration, would help isolate the contribution of the proposed pipeline.

  • Questions about benchmark suitability. While ChimeraX and GRASS GIS are convincing examples of specialized software, the remaining applications are less clearly suited to isolating the benefits of application-specific GUI knowledge. KAlgebra and Lean 4 require mathematical reasoning beyond GUI operation, TeXstudio combines conventional text editing with LaTeX knowledge, and Celestia has relatively straightforward navigation controls. This makes it difficult to determine whether task failures and improvements are driven by GUI-specific knowledge, domain reasoning, or other factors.

  • Uneven performance improvements across applications. KAlgebra alone accounts for approximately 57% of GPT-5.5's total reported average improvement, while the method produces no improvement on Lean 4 and only a modest gain on Celestia (+4.04 percentage points). The paper provides limited analysis of why the method is substantially more effective on some applications than others, raising questions about its generalizability across specialized software.

Questions for authors
  1. Could the authors compare TutorialGUI against closely related training-free, skill-based methods such as VisualSkill and Resource2Skill on the same benchmark? What specifically distinguishes the proposed Replay–Exploration pipeline from these approaches?

  2. How much of the observed performance improvement comes from acquiring genuinely new operational capabilities through Replay–Exploration, versus retrieving relevant tutorial instructions that the executor could already follow? Could the authors evaluate a simpler retrieval-augmented baseline using the same tutorial sources to isolate the contribution of the proposed pipeline?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nULZ

We sincerely appreciate the reviewer for recognizing our motivation and the value of our experiments on scientific software. We have added comparisons with related methods and a RAG baseline using the same tutorial sources, Knowledge Card and Skills ablations, and an analysis of uneven performance improvements.

[W1 & Q1] "Limited novelty and missing comparisons with closely related methods"

(1) "Comparisons with closely related methods"

We have added comparisons with MMG2Skill, VisualSkill, and Resource2Skill in Table R1. TutorialGUI outperforms all three methods, achieving improvements of 10.15, 6.31, and 10.21 percentage points, respectively. All comparisons use the same configuration: GPT-5.5, the official evaluator, and the same evaluation environment. We run each task three times and report the average performance. We will add these results to Table 1 of the revised paper.

Table R1

MethodsKAlgebraCelestiaChimeraXGRASS GISTeXstudioLean 4Avg
Baseline12.9020.2044.8312.7525.003.8519.92
RAG40.8621.2151.728.8220.833.8524.55
MMG2Skill48.3919.1950.5722.554.173.8524.79
VisualSkill52.6927.2758.6217.6510.425.1328.63
Resource2Skill29.0325.2556.3223.5310.423.8524.73
Candidate56.9917.1749.436.8620.833.8525.85
Replayed59.1425.2557.478.8225.003.8529.92
Card-only45.1620.2055.1710.7825.002.5626.48
Skill-only61.2921.2162.0715.6925.003.8531.52
TutorialGUI64.5224.2463.2222.5531.253.8534.94

(2) "What specifically distinguishes the proposed Replay–Exploration pipeline"

We attribute TutorialGUI's advantages to its Replay–Exploration loop, which refines individual skills and continually expands the skill library through interaction with the real environment. MMG2Skill does not include a dedicated exploration stage for expanding skill coverage. VisualSkill does not include Replay and uses free exploration and guided exploration rather than multi-directional Exploration. Resource2Skill expands its skills by searching for additional resources, whereas our method expands the skill library through interaction with the real environment.

[W2 & Q2] "Unclear source of performance improvements"

We have added a RAG baseline using the same tutorials, with results reported in Table R1. RAG increases the average success rate from 19.92% to 24.55%, while TutorialGUI achieves 34.94%, outperforming RAG by a further 10.39 percentage points. These results show that TutorialGUI's gains come not only from retrieving tutorial content, but also from the more complete, execution-verified skills with broader coverage constructed through Replay–Exploration.

[W3] "GUI-specific knowledge, domain reasoning, or other factors"

ScienceBoard evaluates actual task execution in scientific software, involving both domain knowledge and GUI knowledge. Our method provides both of them. Specifically, Knowledge Cards provide domain concepts and application knowledge, while skills provide procedures for GUI operations.

We add a new ablation for domain knowledge and GUI knowledge in Table R1 (using the final Knowledge Cards and complete Skill Library after Exploration). The average success rates for Baseline, Card-only, Skill-only, and their combination are 19.92%, 26.48%, 31.52%, and 34.94%, respectively. This verifies the effectiveness of each type of knowledge and their complementarity.

[W4] "Uneven performance improvements across applications"

We thank the reviewer for raising this point. We believe that uneven performance improvements are related to multiple factors, including differences in task difficulty across applications, model capabilities in their respective domains, and tutorial quality. For example, Lean 4 task A-05 involves infinite cardinals and cofinality in set theory. Such tasks are challenging and require both specialized mathematical knowledge and formal proof construction. In addition, Table R1 also shows that all methods exhibit uneven performance improvements. As TutorialGUI is a general method, we do not focus on resolving performance bottlenecks of specific applications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.