acceptodds
Under review as a conference paper at ICLR 2027

Agentic Skill Self-Creation via Multimodal Proactive Exploration

Abstract

Agentic skills, which encode workflows, application, function specific conventions, and instructions, have become an indispensable component of LLM-based computer-use agents. However, whether written by humans or generated automatically, skills are often incomplete: their creators cannot easily anticipate which details an agent lacks, such as the location of a specific button or differences across application versions. Existing skill refinement pipelines also remain inefficient, as they typically require the AI agent to roll out a large number of task trajectories before invoking a separate refinement module to extract lessons from these trajectories. In this paper, we argue that the AI agent should take a more proactive role in creating and refining skills, because it is best positioned to identify what information is missing and can actively gather that information by interacting and exploring and searching resources, whereas prior work have much more limited access to such information. We therefore propose ProactSkill, task-specific agentic multimodal skill self-creation pipeline that, emulating human researchers, enables the AI agent to create skills for itself by searching application documentation and online resources, and to continuously refine these skills by exploring and interacting with the application during task execution. The created skills are multimodal, so they incorporate visual information throughout the pipeline to make the resulting skills more informative and actionable. We evaluate ProactSkill on agentic benchmarks and show that our proactive skill creation pipeline produces higher quality skills than existing skill creation baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI agents skills, Computer-use agents, multimodal agents and skills, LLMs, skill acquisition, skill refinement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wUinOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PROACTSKILL, a task-specific multimodal skill creation and refinement pipeline for computer-use agents. Given a task instruction, the system searches application documentation, web pages, and YouTube tutorials to construct a skill containing textual instructions and visual references. It then explores a cloned virtual machine before execution, converts the skill into a checklist, and triggers additional exploration and skill refinement when execution stalls. The method is evaluated on OSWorld tasks involving GIMP and three LibreOffice applications. The problem is practically important, and proactive information acquisition is a promising direction. However, the current evaluation contains substantial confounding factors, making it difficult to attribute the reported improvements to the proposed skill-creation method. The central claims concerning multimodality and online refinement are also insufficiently supported. I therefore lean toward rejection in the current form.

Critical strengths
  1. The paper addresses an important practical problem. Computer-use agents frequently fail because application guides omit version-specific interface details, dialog boxes, or intermediate actions. Dynamically constructing task-specific instructions is a useful research direction.

  2. The appendix provides substantial implementation detail. The inclusion of prompts, tool definitions, and refinement rules makes the overall system easier to understand.

Critical weaknesses
  1. Confounded comparison. The evaluated conditions use different prompts, tools, verification mechanisms, and termination rules. In particular, PROACTSKILL benefits from checklist tracking and completion verification that are disabled or different in the baselines. The reported gains therefore cannot be attributed solely to skill quality.

  2. The main claims are not fully supported. Post-refinement accuracy is slightly lower than initial-skill accuracy (80.6% vs. 81.4%), so refinement does not improve success rate. In addition, no matched text-only ablation is provided, making it impossible to isolate the contribution of multimodal content.

  3. Incomplete cost accounting. The low costs in Table 2 cover only execution with an existing skill, excluding search, skill creation, VM cloning, and exploration. The reported 124-minute and $7.37 up-front cost substantially changes the practical comparison, especially because reuse of these task-specific skills is not evaluated.

  4. Limited statistical evidence. The evaluation uses only ten tasks per domain and the corrupted-skill experiment contains only 12 tasks with large, overlapping error bars.

  5. Risk of test-time leakage. The system searches the web using the full benchmark instruction and explores a clone of the evaluation environment. Without releasing search queries and retrieved sources, it is unclear whether the method learns reusable skills or retrieves task-specific solutions.

Questions for authors
  1. Which support mechanisms are enabled in each Table 1 condition? Please explicitly list the status of checklist tracking, coordinate verification, completion judging, the save gate, and stuck recovery.
  2. Do the initial-skill and refined-skill evaluations use the completion judge? Does the no-skill condition use the same judge?
  3. Why does post-refinement accuracy fall below initial-skill accuracy? Is the difference statistically significant?
  4. Can the authors provide a strictly matched text-only PROACTSKILL baseline?
  5. Did any search result contain OSWorld, the benchmark instruction, or an existing solution?
  6. What are the end-to-end time and cost from receiving a task to completing its first execution?
2 comments

Thank you for the detailed review. We re-ran every condition after two fixes. Our cost logging had missed helper-model and skill-creation calls, and those calls sometimes hit their output-token limit before finishing, so we raised the limits and skill creation now completes reliably. The runs cover two trials on the same 35 OSWorld tasks:

ConditionSuccessCost / run
No skill49.7 ± 8.4%$1.06 ± 0.09
VisualSkill (original agent)56.9 ± 7.4%$2.62 ± 0.50
ProactSkill, initial skill62.6 ± 7.2%$1.04 ± 0.06
ProactSkill, refined skill78.3 ± 6.6%$1.00 ± 0.06

Confounded comparison. The no-skill, initial-skill and refined-skill conditions all use the same plain executor, with checklist tracking, the completion judge, the save gate, click verification and stuck recovery all turned off. These mechanisms are on only during the exploration run, which in revision we will label separately. VisualSkill used its own agent, so we also ran its guides in our plain executor. There it scores 51.1%, no better than no skill (+1.4 points, p = 1.0), and 27.1 points below the refined skill (p < 0.001).

Refinement and text-only: In the re-run, refinement raises success from 62.6% to 78.3%. A matched text-only ProactSkill (trial 1, same 35 tasks) scores 79.7% vs. 76.9% for the full version (not significant).

Test-time leakage We audited all 307 searches and fetched pages in two trials of skill-creation runs. Two GIMP tasks were flagged. For an infeasible task (58d3eeeb, flagged in both trials), the agent looked up how OSWorld scores infeasibility. For e2dd0213 (trial 1 only), one query named the benchmark and the task ID. No condition solved either task, and e2dd0213 is not among the 35 reported tasks.

Time and Cost: The first execution with the refined skill’s cost (1.00 ± 0.06 in dollars) and time ( 9 ± 1 min). A one time cost: the exploration stage, in which the agent explores a copy of the environment and refines the skill costs $9.61 ± 0.51 and takes 181 ± 7 minutes per task.

Thank you for the corrected runs and the search-only baseline. My main reason for remaining below acceptance is that the baseline did not receive the same GUI exploration findings used to refine the skill. Giving the same executor those findings directly would help isolate the benefit of turning them into a skill. The new results justify raising my rating, and my final rating is weak rejection.

Reviewer_buAnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProactSkill, a multimodal skill self-creation framework that constructs task-specific skills from web resources and YouTube tutorials, and refines them through proactive GUI exploration when execution gets stuck. Experiments on 40 OSWorld tasks show that the generated skills outperform VisualSkill and the no-skill baseline.

Critical strengths

The paper presents a practical approach to constructing task-specific multimodal skills from web resources and video tutorials without relying on pre-collected execution trajectories. The initial skills on OSWorld substantially outperforming VisualSkill and the no-skill baseline, showing the potential of task-specific skill construction.

Critical weaknesses
  1. The paper has practical engineering value, but its novelty is limited, as the proposed framework primarily integrates existing techniques rather than introducing a fundamentally new skill creation or optimization mechanism. Similarly, methods such as [1,2] have already explored closely related ideas. Although ProactSkill combines these capabilities into a unified online pipeline, the paper does not establish a sufficiently distinct algorithmic contribution beyond this integration, particularly since its proactive refinement stage does not improve task success over the initially constructed skills.
  2. The experiments do not clearly demonstrate the effectiveness of proactive exploration and skill refinement. The initial skill achieves 81.4% success, while the post-refinement skill achieves only 80.6%, and the exploration episode achieves 71.0%. These results do not provide convincing evidence that proactive exploration and skill refinement improve task success.

Reference

[1] VisualSkill: Multimodal Skills for Computer-Use Agents, Jiang et al., 2026.

[2] Agent S: An Open Agentic Framework that Uses Computers Like a Human, Agashe et al., 2025.

Questions for authors

Q1. Does the use of VisualSkill guides affect the fairness of the comparison? Section 3.1 states that ProactSkill can use existing application guides created by VisualSkill as additional context during skill construction. Could the authors evaluate ProactSkill without this additional information to ensure a fair comparison with VisualSkill?

Q2. What is the contribution of multimodal information to skill construction? The paper emphasizes multimodal skill construction as an important contribution, but does not adequately evaluate the individual contributions of visual information, such as images and YouTube videos. Could the authors provide ablation studies to quantify the effectiveness of different information modalities in skill construction?

Q3. Why does skill refinement fail to improve task success? Table 1 shows that the Initial Skill achieves 81.4% success, while the Refined Skill achieves only 80.6%. Could the authors analyze it?

Q4. The paper claims that Refined Skills are more efficient to execute, but the comparison does not account for the full cost of Skill Creation and Exploration. Could the authors report the total time and cost per task, from Skill Creation to final execution, and conduct cross-task skill reuse experiments to show how many reuses are needed to offset the initial exploration cost?

2 comments

Thank you for the review.

We re-ran every condition after two fixes. Our cost logging had missed helper-model and skill-creation calls, and those calls sometimes hit their output-token limit before finishing, so we raised the limits and skill creation now completes reliably. The runs cover two trials on the same 35 OSWorld tasks. These numbers replace Tables 1–3:

ConditionSuccessCost / run
No skill49.7 ± 8.4%$1.06 ± 0.09
VisualSkill (original agent)56.9 ± 7.4%$2.62 ± 0.50
ProactSkill, initial skill62.6 ± 7.2%$1.04 ± 0.06
ProactSkill, refined skill78.3 ± 6.6%$1.00 ± 0.06

Novelty We agree that each component has precedents, and we do not claim a new optimizer. What is new is building a skill for each task at test time from web pages and video tutorials written for people, and refining it within the same episode by exploring a live copy of the environment when execution stalls. By contrast, VisualSkill builds one general guide per application offline, and Agent S uses search for planning without producing a reusable skill.

**Refinement: ** In the re-run, refinement raises success from 62.6% to 78.3%. The exploration episode itself scores lower (55.4%) because it spends its action budget on probing the interface rather than finishing the task.

VisualSkill guides We ran ProactSkill without the VisualSkill guides (trial 1, same 35 tasks). Its refined skill scores 85.4%, versus 76.9% with the guides (+8.6 points, not significant), and its initial skill scores 79.7% versus 59.7%.

Multimodal contribution We ran a text-only ProactSkill, the same pipeline with all images removed from skills and guides (1 trial, same 35 tasks). Its refined skill scores 79.7%, versus 76.9% for the full version (+2.9 points, CI [−8.6, +14.3], not significant), so we find no measurable benefit from visual content on success rate. The gain comes from building and refining task-specific skills, which text-only skills still deliver (+31.4 points over no skill, p = 0.003).

Total cost and reuse Skill creation costs $4.29 and exploration plus refinement costs $6.32, for $10.61 per task, paid once. After that, each run costs $1.00, compared with $1.06 without a skill, so measured as cost per successful task the skill pays for itself after about 16 reuses (about 150 on dollars alone).

Thank you for the detailed response and the substantial effort in re-running the experiments and providing additional ablations. I appreciate that the new results address several of my concerns and provide stronger evidence for the effectiveness of skill refinement. However, I still have reservations about the limited methodological novelty and the unclear contribution of multimodal information. Therefore, I maintain my original rating.

Reviewer_ojCbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposes ProactSkill, which lets GUI agents search online docs and youtube videos to create task-specific skills. the agent can also explore the UI and update its skills when it gets stuck. experiments on OSWorld show a pretty large improvement over existing baselines

Critical strengths
  1. the idea is practical and the reported improvement is quite large, from around 54% to 81% success rate.
  2. I do like that they also test the generated skills with a fresh agent, without the extra execution tools.
Critical weaknesses
  1. the main comparison is not quite fair. their method searches web pages and videos for each task, while the baselines dont have access to the same information. there is no search-only baseline that gives the agent the retrieved information without creating a skill. so its unclear how much of the gain actually comes from skill creation.
  2. the refinement part is not very convincing. the initial skills already get 81.4%, while refined skills get 80.6%. the corrupted-skill experiment is also very small and shows little improvement over SkillRevise. given the large exploration cost, its hard to see what the refinement stage really adds
Questions for authors
  1. can you compare against a simple search-only baseline using the same retrieved information but without skill creation? i think this is important to support the main contribution.
  2. why does refinement not improve the success rate over the initial skills? is there any evidence that its extra cost pays off in practice?
1 comment

Thank you for the review. We re-ran every condition after two fixes. Our cost logging had missed helper-model and skill-creation calls, and those calls sometimes hit their output-token limit before finishing, so we raised the limits and skill creation now completes reliably. The runs cover two trials on the 35 OSWorld tasks:

ConditionSuccessCost / run
No skill49.7 ± 8.4%$1.06 ± 0.09
VisualSkill (original agent)56.9 ± 7.4%$2.62 ± 0.50
ProactSkill, initial skill62.6 ± 7.2%$1.04 ± 0.06
ProactSkill, refined skill78.3 ± 6.6%$1.00 ± 0.06

Search-only baseline: Thank you for the suggestion, and we added this baseline: the agent receives, pasted into its prompt, the same retrieved web and YouTube research that ProactSkill obtained for that task (taken from the creation transcript, no re-search), with no skill, no images, no checklist but with the VisualSkill application guide ProactSkill also received. ProactSkill outperforms the search only baseline.

ConditionSuccess
No skill49.7 ± 8.4%
Search-only (web research + VisualSkill guide)52.6 ± 8.1%
ProactSkill, initial skill62.6 ± 7.5%
ProactSkill, refined skill79.7 ± 6.5%

Refinement: In the corrected re-run, refinement raises success from 62.6% to 78.3%.

Does the cost pay off? Building and refining a skill costs 10.61 dollars on average per task, paid once. After that, each run costs 1.00 dollars, compared with 1.06 dollars without a skill, and it succeeds more often. So the revision says plainly that refinement pays off for repeated tasks, not one-off ones.

Corrupted-skill experiment: We re-ran it over two trials on skills confirmed broken before repair: ProactSkill repairs 14/29 (48%, 95% CI 31–66%) and SkillRevise 11/29 (38%, CI 23–56%, paired p = 0.37). The experiment is meant to show that exploration can repair a broken skill without a recorded failure. On the 8 tasks broken in both trials, mostly GIMP, both methods rarely succeed (3/16 vs. 2/16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.