acceptodds
Under review as a conference paper at ICLR 2027

Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

Abstract

Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging. Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details. We thus introduce Work2Skill, a framework that captures human GUI experience for reusable agent skill induction. WorkRecorder provides the framework's infrastructure for collecting this experience, aligning screen recordings, mouse and keyboard events, and available accessibility-tree information. Using WorkRecorder, we construct WorkDemo-350, the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation. Its 350 demonstrations are screened for completeness and task success. We then propose an agentic skill-induction method that inspects evidence on demand to produce reusable skills capturing workflow rules, step dependencies, and completion checks. When the same four execution models are equipped with skills induced by each method under shared execution settings, Work2Skill achieves the highest scores and success rates. Mean score improvements over the corresponding No Skill baseline are 4.67–10.54 points on MyPCBench Bounded Action and 3.11–7.50 points on a 286-task subset of OSWorld-Verified.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Skill Induction, Human Demonstrations, GUI Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xAS4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Work2Skill, a framework for converting human GUI demonstrations into reusable skills for computer-use agents. It includes WorkRecorder, which synchronizes screen recordings, mouse/keyboard events, and available accessibility-tree information, and WorkDemo-350, a curated dataset of 350 successful demonstrations across MyPCBench and OSWorld-Verified. The proposed agentic skill-induction method selectively inspects multimodal evidence and iteratively generates reusable procedural guidance in the Agent Skills format. Experiments across four execution models and two benchmarks demonstrate improvements in downstream task performance over several baselines.

Critical strengths
  1. The paper investigates the challenge of transferring procedural knowledge from human GUI demonstrations to computer-use agents, with a clear focus on reusable skills and downstream task execution.
  2. The idea of active exploring skill based on current task state is intuitive.
  3. The experiments is comprehensive, evaluating performance across two benchmarks and four execution models, with additional experiments on task-instance generalization, cross-model transfer, and evidence ablations.
Critical weaknesses
  1. This work involves two closely related works: ShowUI-Aloha and Demo2Tutorial. A more detailed comparison with them preferably in a table, would help clarify Work2Skill's methodological novelty and key differences.
  2. In the experiment part, ShowUI-Aloha retains only action descriptions, while Demo2Tutorial is reduced to text-only guidance. Although a unified interface is reasonable, removing essential components of the original methods may underestimate their capabilities.
  3. The contributions of WorkRecorder and adaptive evidence exploration are not fully isolated. From my understanding, active & adaptive exploration is one key contribution of the work, which should be investigated seriously. Although Figure 5(b) compares induction prompts, it does not directly evaluate adaptive versus static evidence exploration under identical inputs and computational budgets.
Questions for authors

See weakness.

Reviewer_2iBnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how human GUI demonstrations can be converted into reusable skills for computer-use agents. It introduces WorkRecorder and WorkDemo-350 to collect synchronized visual, action, and accessibility evidence, together with an agentic induction method that inspects this evidence and produces task-matched skill files. Experiments on MyPCBench and OSWorld-Verified show improvements over No Skill, ShowUI-Aloha, and Demo2Tutorial under the proposed evaluation protocol.

Critical strengths
  1. The motivation is reasonable: in settings where human demonstrations are available, abstracting them into skills provides a natural way to preserve and reuse procedural knowledge.

  2. The paper constructs a relatively high-quality dataset with human demonstrations, which may also benefit subsequent work in this area.

  3. Under the current experimental setting, the authors provide fairly detailed experiments demonstrating the effectiveness of the proposed method.

Critical weaknesses
  1. My main concern is the dataset and evaluation design. Each task instance is paired one-to-one with its own human demonstration, which neither fully reflects the motivation of reusable skill induction nor provides a sufficiently strong test of skill quality. The main experiments evaluate skills on exactly the demonstrated tasks. Although the paper includes some generalization experiments, they receive relatively limited emphasis, and the perturbations remain close to the original tasks. A more convincing main setting would be one-to-many, where a skill induced from one demonstration must support multiple related but distinct downstream tasks.
  2. This design also raises a potential leakage issue. The concern is not limited to fixed coordinates, timestamps, or other explicit instance-specific values. A generated skill may closely reproduce the entire successful workflow rather than capture a genuinely reusable procedure. Under the current task-matched evaluation, these two possibilities are difficult to distinguish, which weakens the claim that the method induces reusable skills.
  3. The baseline coverage is somewhat limited. ShowUI-Aloha and Demo2Tutorial can be reasonably adapted to the shared text-guidance protocol, but they are not fully representative of more explicit GUI-skill frameworks. Methods such as OmegaUse-SOP are conceptually more direct comparisons, while GPA and CUA-Skill also provide relevant structured workflow or skill abstractions. The paper should either include a feasible comparison with a more direct GUI-skill method.
Questions for authors

Since the paper’s contribution rests substantially on the dataset and task formulation, could the authors explain why the one-demonstration–one-task protocol is an appropriate primary evaluation of reusable skill induction? A small-scale experiment in which each induced skill is evaluated on multiple related tasks could provide additional evidence of generalization. However, because this concern originates from the overall task design, I am not sure that it can be fully resolved through the rebuttal alone.

2 comments

After reviewing the authors’ response and additional experiments, I find that the task-paired evaluation design has a reasonable basis. The Jitter and cross-task evaluations with frozen skills also partially address my concern about leakage by demonstrating reuse under changed task requirements. I have therefore raised my score to 3.

AuthorsOpenReview ↗

Thank You for Your Thoughtful Review

Thank you for taking the time to read our response and consider the additional experiments. We sincerely appreciate your reconsideration and your explanation of how the new results informed your updated assessment.

Your feedback helped us strengthen the paper. We’re grateful for the opportunity to address your questions and discuss our work with you.

Reviewer_ARm1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Work2Skill, a framework designed to capture human desktop GUI experience and induce reusable agent skills for computer-use tasks. The framework consists of three main components: WorkRecorder, WorkDemo-350, and an Agentic Skill Induction method that uses an on-demand evidence inspection agent to write and revise procedural skills. Extensive experiments across various execution models and benchmarks show that the induced skills improve downstream execution scores and success rates compared to several baselines.

Critical strengths
  1. WorkRecorder provides a well-thought-out system for aligning multimodal evidence (screen recordings, input actions, and accessibility tree data), which helps mitigate the ambiguity of raw coordinates.

  2. WorkDemo-350 offers a standardized collection of multimodal demonstrations across recognized benchmarks (MyPCBench and OSWorld-Verified), facilitating controlled comparison of skill induction methods.

Critical weaknesses
  1. Learning GUI agent policies or reusable procedural skills from human demonstrations, videos, or tutorials is an increasingly crowded research area. Several recent works already explore converting human demonstrations into semantic action traces, multimodal tutorials, or skills. The core idea of abstracting human GUI demonstrations into reusable agent guidance lacks sufficient conceptual novelty over these concurrent and prior studies.

  2. The "Agentic Skill Induction" relies largely on standard tool-using agent loops combined with specific prompt engineering. The pipeline structure and mechanisms do not introduce fundamentally new learning algorithms or architectural breakthroughs beyond leveraging powerful base LLMs to read structured logs and write Markdown files.

  3. The paper reads more like an engineering system report or a software release package rather than a study offering fundamental algorithmic or theoretical insights. The underlying mechanism relies heavily on off-the-shelf multimodal LLMs coupled with standard prompt-based tool-use loops to inspect logs and write Markdown files, lacking profound scientific novelty or reusable algorithmic frameworks for the research community.

  4. While the paper demonstrates improvements on specific benchmark subsets, real-world desktop environments exhibit high stochasticity, dynamic UI changes, and application updates. It remains questionable whether these induced static Markdown skills can robustly handle out-of-distribution application states or unexpected software updates without frequent manual curation or brittle heuristics.

Questions for authors

See the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.