- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PROACTSKILL, a task-specific multimodal skill creation and refinement pipeline for computer-use agents. Given a task instruction, the system searches application documentation, web pages, and YouTube tutorials to construct a skill containing textual instructions and visual references. It then explores a cloned virtual machine before execution, converts the skill into a checklist, and triggers additional exploration and skill refinement when execution stalls. The method is evaluated on OSWorld tasks involving GIMP and three LibreOffice applications. The problem is practically important, and proactive information acquisition is a promising direction. However, the current evaluation contains substantial confounding factors, making it difficult to attribute the reported improvements to the proposed skill-creation method. The central claims concerning multimodality and online refinement are also insufficiently supported. I therefore lean toward rejection in the current form.
Critical strengths
-
The paper addresses an important practical problem. Computer-use agents frequently fail because application guides omit version-specific interface details, dialog boxes, or intermediate actions. Dynamically constructing task-specific instructions is a useful research direction.
-
The appendix provides substantial implementation detail. The inclusion of prompts, tool definitions, and refinement rules makes the overall system easier to understand.
Critical weaknesses
-
Confounded comparison. The evaluated conditions use different prompts, tools, verification mechanisms, and termination rules. In particular, PROACTSKILL benefits from checklist tracking and completion verification that are disabled or different in the baselines. The reported gains therefore cannot be attributed solely to skill quality.
-
The main claims are not fully supported. Post-refinement accuracy is slightly lower than initial-skill accuracy (80.6% vs. 81.4%), so refinement does not improve success rate. In addition, no matched text-only ablation is provided, making it impossible to isolate the contribution of multimodal content.
-
Incomplete cost accounting. The low costs in Table 2 cover only execution with an existing skill, excluding search, skill creation, VM cloning, and exploration. The reported 124-minute and $7.37 up-front cost substantially changes the practical comparison, especially because reuse of these task-specific skills is not evaluated.
-
Limited statistical evidence. The evaluation uses only ten tasks per domain and the corrupted-skill experiment contains only 12 tasks with large, overlapping error bars.
-
Risk of test-time leakage. The system searches the web using the full benchmark instruction and explores a clone of the evaluation environment. Without releasing search queries and retrieved sources, it is unclear whether the method learns reusable skills or retrieves task-specific solutions.
Questions for authors
- Which support mechanisms are enabled in each Table 1 condition? Please explicitly list the status of checklist tracking, coordinate verification, completion judging, the save gate, and stuck recovery.
- Do the initial-skill and refined-skill evaluations use the completion judge? Does the no-skill condition use the same judge?
- Why does post-refinement accuracy fall below initial-skill accuracy? Is the difference statistically significant?
- Can the authors provide a strictly matched text-only PROACTSKILL baseline?
- Did any search result contain OSWorld, the benchmark instruction, or an existing solution?
- What are the end-to-end time and cost from receiving a task to completing its first execution?
2 comments
Thank you for the corrected runs and the search-only baseline. My main reason for remaining below acceptance is that the baseline did not receive the same GUI exploration findings used to refine the skill. Giving the same executor those findings directly would help isolate the benefit of turning them into a skill. The new results justify raising my rating, and my final rating is weak rejection.
Thank you for the detailed review. We re-ran every condition after two fixes. Our cost logging had missed helper-model and skill-creation calls, and those calls sometimes hit their output-token limit before finishing, so we raised the limits and skill creation now completes reliably. The runs cover two trials on the same 35 OSWorld tasks:
Confounded comparison. The no-skill, initial-skill and refined-skill conditions all use the same plain executor, with checklist tracking, the completion judge, the save gate, click verification and stuck recovery all turned off. These mechanisms are on only during the exploration run, which in revision we will label separately. VisualSkill used its own agent, so we also ran its guides in our plain executor. There it scores 51.1%, no better than no skill (+1.4 points, p = 1.0), and 27.1 points below the refined skill (p < 0.001).
Refinement and text-only: In the re-run, refinement raises success from 62.6% to 78.3%. A matched text-only ProactSkill (trial 1, same 35 tasks) scores 79.7% vs. 76.9% for the full version (not significant).
Test-time leakage We audited all 307 searches and fetched pages in two trials of skill-creation runs. Two GIMP tasks were flagged. For an infeasible task (58d3eeeb, flagged in both trials), the agent looked up how OSWorld scores infeasibility. For e2dd0213 (trial 1 only), one query named the benchmark and the task ID. No condition solved either task, and e2dd0213 is not among the 35 reported tasks.
Time and Cost: The first execution with the refined skill’s cost (1.00 ± 0.06 in dollars) and time ( 9 ± 1 min). A one time cost: the exploration stage, in which the agent explores a copy of the environment and refines the skill costs $9.61 ± 0.51 and takes 181 ± 7 minutes per task.