- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a new way to create skills based on demonstrations, collected from human traces (unspecified), using a method that automatically builds skill graph. They show that using those skills improve performance on two specialized benchmarks, WebArena-Verified’s hard split, ClawBench, and a tiny subset of osworld.
Critical strengths
- Figure 2 is a good figure to quickly understand how the method works
Critical weaknesses
- Benchmarks choices: I would expect the method to run on the full webarena (or webarena-verified), not just the verified hard subset. Similarly, for the computer-use portion, OSWorld-verified would be a much more general choice, and, although ClawBench could be useful, it should only be reported as an auxiliary result once we know the performance on more common benchmarks like full OSWorld and full WebArena (verified version or original version, either work). Subsets are not sufficient, and should only be considered for ablation studies if resources are insufficient.
- Lacks baselines: The only baseline they have is the same model without tools. However, they do not explore Agentic Workflow Memory, Agentic Skill Induction, and SkillWeaver, which are all well established methods that are 1+ years old.
- Vendor locking: only shown on two specific proprietary models, sonnet 4.6 and qwen 3.7 plus (not the open version), which means it may not transfer to other newer models once those two models are eventually deprecated. Showing transfer is crucial.
Questions for authors
Why did you choose sonnet 4.6 and qwen 3.7 plus over open-weight models?
2 comments
Your reply makes no sense: "Full WebArena and OSWorld evaluations would support broader benchmark-level conclusions. The current paper focuses on within-setting comparisons and does not extrapolate its success rates to those complete benchmarks"
But you ARE using subsets/derivatives of webarena and osworld. So what exactly do you mean by "within-setting comparisons"?
Agent Workflow Memory, Agentic Skill Induction, and SkillWeaver are relevant comparisons for positioning BrowserBC. Our existing experiments establish the above design properties and matched-guidance benefits; they do not rank BrowserBC against those systems.
And why not? they are exactly the kind of comparisons that would make your method stronger or weaker. Why can't you run them?
Response to Reviewer v5jX
Thank you for the feedback and for highlighting the clarity of Figure 2. We address benchmark coverage, baselines, and model transfer below.
1. Benchmark coverage
The experiments address complementary aspects of procedural guidance. WebArena-Hard contains 258 tasks across five task groups, with improvements in every group reported in Table 2. ClawBench contains 152 live-site tasks, evaluated under the disclosed rubric-informed, matched-guidance protocol. The former measures execution in self-hosted environments; the latter examines workflows on live production websites.
The desktop study serves a different purpose. Its 30 cases are constructed Ubuntu cases inspired by OSWorld, rather than a sampled subset of the OSWorld benchmark. Section 4.5 and Appendix D describe fixed initial states, independent state-based verifiers, and both successful and unsuccessful uses of guidance. The base agent solves 11 cases, guidance enables six additional successes, and 13 remain unsolved. This provides a diagnostic view of procedural assistance and execution failures beyond the browser.
Full WebArena and OSWorld evaluations would support broader benchmark-level conclusions. The current paper focuses on within-setting comparisons and does not extrapolate its success rates to those complete benchmarks. ClawBench and the desktop cases contribute evidence about different execution settings within this scope.
2. Baselines and evidence for the design
One clarification is important: the base agent has the same browser tools as BrowserBC. Section 4.1 holds the executor, action interface, and step budget fixed; skill-on adds demonstration-derived procedural guidance. Thus, the main comparison measures the benefit of that guidance with the execution setup controlled.
The submission also includes three comparisons beyond skill-on versus skill-off:
These studies provide evidence for library organization, adaptive execution, and the value of accumulated skills. Agent Workflow Memory, Agentic Skill Induction, and SkillWeaver are relevant comparisons for positioning BrowserBC. Our existing experiments establish the above design properties and matched-guidance benefits; they do not rank BrowserBC against those systems.
3. Model choice and transfer
BrowserBC stores procedural knowledge as external natural-language cards. The executor consumes those cards through its context, without requiring access to or modification of its weights. The two tested API models therefore allow us to examine the separation between skill production and skill execution.
Section 4.3 and Figure 3b explicitly cross Sonnet-4.6 and Qwen-3.7-plus as producers and consumers. Sonnet-produced skills improve Qwen from 53% to 77%, and Qwen-produced skills improve Sonnet from 60% to 70%. This is direct evidence of transfer across two model families, rather than an assumption based on the textual format alone.
Open-weight evaluation would extend this evidence and improve reproducibility. The present study establishes that skills can be consumed by a different tested model from the one that produced them. Compatibility with future models remains dependent on their ability to interpret the guidance and execute the required actions; the framework's weight-independent interface makes such reuse possible without establishing identical performance across all executors.