- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TutorialGUI, a training-free framework that converts software tutorials into reusable skills and Knowledge Cards. A Replay–Exploration loop refines and verifies skills through GUI execution and expands the library with additional skills. The resulting resources are retrieved to guide downstream agents. Experiments on six ScienceBoard applications demonstrate improved task success across three executor models.
Critical strengths
- The proposed framework improves average task success over the corresponding baseline agents.
- It equips GUI agents with specialized, reusable skills without updating model parameters.
Critical weaknesses
- Limited baseline comparisons. Closely related methods such as MMG2Skill [1] and VisualSkill [2] are not evaluated. Comparisons with these methods and a strong GUI agent under matched conditions would clarify the method’s added value.
- Confounded ablations. Replay changes both procedures and images, while Exploration adds skills and interactions. Fixed-modality ablations and budget-matched comparisons with simpler exploration would help isolate their contributions.
- Unclear generalization scope. Each application receives a dedicated skill library, while task novelty relative to construction examples is unclear. Tests on new inputs, initial states, and skill combinations would better establish generalization.
- Missing costs and budgets. Construction costs, round-level exploration budgets, and downstream execution limits are insufficiently reported.
Questions for authors
See Weakness.
Response to Reviewer Z9ev
We sincerely appreciate the reviewer for the constructive feedback and for recognizing the performance gains and skill augmentation without parameter updates. Below, we address your concerns with additional comparisons, ablations, and details of the experimental protocol.
[W1] "Limited baseline comparisons"
We have added comparisons with MMG2Skill, VisualSkill, and Resource2Skill in Table R1. TutorialGUI outperforms all three methods, achieving improvements of 10.15, 6.31, and 10.21 percentage points, respectively.
All comparisons use the same configuration: GPT-5.5, the official evaluator, and the same evaluation environment. We run each task three times and report the average performance. We will add these results to Table 1 of the revised paper.
We attribute TutorialGUI's advantages to its Replay–Exploration loop, which refines individual skills and continually expands the skill library through interaction with the real environment. MMG2Skill does not include a dedicated exploration stage for expanding skill coverage. VisualSkill does not include Replay and uses free exploration and guided exploration rather than multi-directional Exploration. Resource2Skill expands its skills by searching for additional resources, whereas our method expands the skill library through interaction with the real environment.
Table R1: Method comparisons.
[W2] "Confounded ablations"
(1) "Fixed-modality ablations"
Regarding the Replay ablation study, we would like to clarify that the procedure and reference image obtained during Replay are tightly coupled: a new procedure produces a new reference image. Therefore, we cannot ablate them separately.
(2) "Budget-matched comparisons with simpler exploration"
We have also added a comparison with budget-matched simpler exploration, which replaces three-direction Exploration with free exploration, i.e., the model freely generates new candidates from anchor skills without any directional guidance. As shown in Table R2, TutorialGUI achieves an average success rate of 34.94%, outperforming Simple Exploration (30.78%) by 4.16 percentage points.
Table R2: Simple Exploration comparison.
[W3] "Task novelty relative to construction examples"
We note that TutorialGUI itself is a training-free approach and does not rely on any tasks from the benchmark during skill construction. Therefore, all tasks within the benchmark can be regarded as unseen tasks.
[W4] "Missing costs and budgets"
(1) "Construction costs"
Table R3 reports the total input and output tokens consumed by Replay and Exploration during skill library construction, totaling 107.072M tokens.
Table R3: Skill library construction costs (million tokens, including input and output).
(2) "round-level exploration budgets"
Please refer to Appendices A.1 and A.4.6 (Exploration Candidate-Generation Prompt). Each Exploration round stops when the target number of newly admitted verified skills is reached. Table R3 reports the total Exploration token consumption during construction.
(3) "downstream execution limits"
Downstream evaluation follows the
stepslimits in ScienceBoard's official task configurations, ranging from 5 to 20 interaction rounds per task. Please refer to ScienceBoard for the evaluation configurations of individual applications and tasks.