acceptodds
Under review as a conference paper at ICLR 2027

Scalable Behaviour Cloning on Browser Using via Skill Distillation

Abstract

Browser demonstrations encode procedural knowledge about where to navigate, which information to preserve, and how to verify task completion. Reusing this knowledge requires separating the procedure from the incidental details of a recorded session. We present BrowserBC, a framework that distills browser trajectories into structured natural-language skills and organizes them in a skill graph. Each skill captures applicability conditions, execution steps, progress checks, completion evidence, and recovery guidance. The executor interprets these instructions against the live page, enabling skills to be produced and consumed by different models. Evaluations on WebArena-Hard and ClawBench show improved task completion under task-family and rubric-informed, case-matched guidance, respectively, with shorter interactions on WebArena-Hard. Cross-model experiments demonstrate skill reuse across executors, while real-world scenario tests show benefits from greater skill availability and graph organization in both success and token efficiency. Desktop cases illustrate how procedural guidance supports navigation and completion checks while revealing persistent execution difficulties. Together, these findings support skill distillation as a way to accumulate and share procedural knowledge across agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Behaviour Cloning, Browser Using, Skill Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v5jXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new way to create skills based on demonstrations, collected from human traces (unspecified), using a method that automatically builds skill graph. They show that using those skills improve performance on two specialized benchmarks, WebArena-Verified’s hard split, ClawBench, and a tiny subset of osworld.

Critical strengths
  • Figure 2 is a good figure to quickly understand how the method works
Critical weaknesses
  • Benchmarks choices: I would expect the method to run on the full webarena (or webarena-verified), not just the verified hard subset. Similarly, for the computer-use portion, OSWorld-verified would be a much more general choice, and, although ClawBench could be useful, it should only be reported as an auxiliary result once we know the performance on more common benchmarks like full OSWorld and full WebArena (verified version or original version, either work). Subsets are not sufficient, and should only be considered for ablation studies if resources are insufficient.
  • Lacks baselines: The only baseline they have is the same model without tools. However, they do not explore Agentic Workflow Memory, Agentic Skill Induction, and SkillWeaver, which are all well established methods that are 1+ years old.
  • Vendor locking: only shown on two specific proprietary models, sonnet 4.6 and qwen 3.7 plus (not the open version), which means it may not transfer to other newer models once those two models are eventually deprecated. Showing transfer is crucial.
Questions for authors

Why did you choose sonnet 4.6 and qwen 3.7 plus over open-weight models?

2 comments
AuthorsOpenReview ↗

Response to Reviewer v5jX

Thank you for the feedback and for highlighting the clarity of Figure 2. We address benchmark coverage, baselines, and model transfer below.

1. Benchmark coverage

The experiments address complementary aspects of procedural guidance. WebArena-Hard contains 258 tasks across five task groups, with improvements in every group reported in Table 2. ClawBench contains 152 live-site tasks, evaluated under the disclosed rubric-informed, matched-guidance protocol. The former measures execution in self-hosted environments; the latter examines workflows on live production websites.

The desktop study serves a different purpose. Its 30 cases are constructed Ubuntu cases inspired by OSWorld, rather than a sampled subset of the OSWorld benchmark. Section 4.5 and Appendix D describe fixed initial states, independent state-based verifiers, and both successful and unsuccessful uses of guidance. The base agent solves 11 cases, guidance enables six additional successes, and 13 remain unsolved. This provides a diagnostic view of procedural assistance and execution failures beyond the browser.

Full WebArena and OSWorld evaluations would support broader benchmark-level conclusions. The current paper focuses on within-setting comparisons and does not extrapolate its success rates to those complete benchmarks. ClawBench and the desktop cases contribute evidence about different execution settings within this scope.

2. Baselines and evidence for the design

One clarification is important: the base agent has the same browser tools as BrowserBC. Section 4.1 holds the executor, action interface, and step budget fixed; skill-on adds demonstration-derived procedural guidance. Thus, the main comparison measures the benefit of that guidance with the execution setup controlled.

The submission also includes three comparisons beyond skill-on versus skill-off:

  • Flat versus graph-organized library: Success increases from 83.2% to 89.4%, while mean tokens decrease from 115.4K to 109.8K, on the same 113 GitLab cases (§4.4, Figure 4b).
  • Always-follow versus adaptive execution: Success increases from 77.5% to 81.4% when the executor can override a skill based on live observations (§4.3).
  • Skill availability: Figure 4a reports performance at five availability levels on a fixed evaluation set, with success increasing from 54.0% to 89.4%.

These studies provide evidence for library organization, adaptive execution, and the value of accumulated skills. Agent Workflow Memory, Agentic Skill Induction, and SkillWeaver are relevant comparisons for positioning BrowserBC. Our existing experiments establish the above design properties and matched-guidance benefits; they do not rank BrowserBC against those systems.

3. Model choice and transfer

BrowserBC stores procedural knowledge as external natural-language cards. The executor consumes those cards through its context, without requiring access to or modification of its weights. The two tested API models therefore allow us to examine the separation between skill production and skill execution.

Section 4.3 and Figure 3b explicitly cross Sonnet-4.6 and Qwen-3.7-plus as producers and consumers. Sonnet-produced skills improve Qwen from 53% to 77%, and Qwen-produced skills improve Sonnet from 60% to 70%. This is direct evidence of transfer across two model families, rather than an assumption based on the textual format alone.

Open-weight evaluation would extend this evidence and improve reproducibility. The present study establishes that skills can be consumed by a different tested model from the one that produced them. Compatibility with future models remains dependent on their ability to interpret the guidance and execute the required actions; the framework's weight-independent interface makes such reuse possible without establishing identical performance across all executors.

Your reply makes no sense: "Full WebArena and OSWorld evaluations would support broader benchmark-level conclusions. The current paper focuses on within-setting comparisons and does not extrapolate its success rates to those complete benchmarks"

But you ARE using subsets/derivatives of webarena and osworld. So what exactly do you mean by "within-setting comparisons"?

Agent Workflow Memory, Agentic Skill Induction, and SkillWeaver are relevant comparisons for positioning BrowserBC. Our existing experiments establish the above design properties and matched-guidance benefits; they do not rank BrowserBC against those systems.

And why not? they are exactly the kind of comparisons that would make your method stronger or weaker. Why can't you run them?

Reviewer_BABbOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BrowserBC, a framework for converting browser demonstrations into reusable natural-language procedural skills. The key innovation is the "skill card" representation, which encodes applicability conditions, execution steps, progress milestones, terminal evidence, failure modes, and recovery strategies.

These skills are organized in a skill graph and used to condition agent execution, improving task completion rates and reducing interaction length. Evaluations on WebArena-Hard, ClawBench, and real GitLab cases show significant improvements (+20.9%, +35.5%, and up to +35.4% respectively), with evidence of cross-model skill transfer.

Critical strengths
  1. Well-motivated practical problem. Web agents need reliable execution procedures and graceful recovery. Skill reuse is essential for scalability.

  2. Thoughtful skill representation. Skill cards capture preconditions, execution steps, progress evidence, and recovery guidance - enabling both distillation and flexible reuse.

  3. Comprehensive evaluation. WebArena-Hard (258 tasks), ClawBench (152), GitLab (113), Desktop (30). Results: +20.9%, +35.5%, up to +35.4% improvement.

  4. Strong empirical results. Interaction efficiency: −27.3% mean tool calls. Cross-model transfer: Sonnet skills 77% on Owen; Owen 80% on Sonnet.

Critical weaknesses
  1. Computational costs not analyzed. Cost of skill distillation, library size, retrieval latency are not reported. Paper mentions "total cost includes distillation, storage" but provides no numbers.

  2. Limited baseline comparisons. No comparison with in-context learning or alternative skill distillation methods. Missing comparison with learned skill libraries.

  3. Distillation methodology underspecified. Implementation vague. For WebArena-Hard, unclear if skills are task-specific. Annotation overhead unknown.

  4. Generalization claims need evidence. All evaluations on website/browser tasks. No evaluation on mobile, command-line, or APIs. Cross-model transfer tested only on Claude pairs.

Questions for authors
  1. What is total cost (LLM calls, tokens, wall-clock time) to build skill library of ~10 skills for WebArena-Hard? How does this amortize over 258 tasks?

  2. Can you provide ablation studies on skill card fields by importance? (e.g., -X% success if preconditions omitted, -Y% if recovery strategies omitted)

  3. For GitLab study, correlation between skill library size and success rate? Is there a saturation point?

  4. How does BrowserBC compare to few-shot in-context learning baseline using same human trajectories without explicit skill cards?

  5. Can you test cross-family generalization? (e.g., Sonnet skills on GPT-4o)

3 comments
AuthorsOpenReview ↗

Response to Reviewer BABb (1/2): Transfer and library growth

Thank you for recognizing the practical motivation, skill representation, and empirical results. Several of your questions are addressed by experiments already included in the submission; we summarize the relevant results below.

1. Cross-family transfer

The transfer study includes Claude-Sonnet-4.6 and Qwen-3.7-plus, so it already tests two model families. Figure 3b abbreviates Qwen-3.7-plus as Qwen-3.7. The six configurations are:

Skill sourceQwen executorSonnet executor
No skill53%60%
Qwen-distilled57%70%
Sonnet-distilled77%80%

Both cross-family directions improve performance: Sonnet → Qwen gives 53% → 77%, and Qwen → Sonnet gives 60% → 70%. The study uses the same source trajectories and varies skill producer and consumer separately (§4.3). It demonstrates reuse across these two families on 30 tasks; the reported percentages are rounded.

2. Library size and saturation

Section 4.4 and Figure 4a examine skill availability on a fixed set of 113 GitLab cases:

Available skill fraction0%25%50%75%100%
Success rate54.0%62.8%76.1%87.6%89.4%
Mean tokens146.5K133.4K118.4K120.6K109.8K

Success increases at every tested fraction. The gain from 75% to 100% is 1.8 percentage points, indicating diminishing marginal gains near full availability in this setting. The curve suggests diminishing returns within the tested range, although it does not establish a saturation point. The fractions denote available skills, not the amount of training data.

Figure 4b additionally shows that library organization matters: the full graph achieves 89.4% versus 83.2% success for a flat library, with mean tokens decreasing from 115.4K to 109.8K.

3. Evaluation beyond browser tasks

Section 4.5 and Appendix D include 30 controlled Ubuntu desktop cases covering settings, file and text editing, terminal commands, application configuration, dialogs, and office applications. The base agent solves 11 cases, matched guidance enables six additional successes, and 13 remain unsolved.

These cases show how the same procedural interface applies outside the browser, including to terminal and filesystem operations. For example, the archive-extraction case illustrates how establishing the correct working directory and checking the output location turns a valid command into successful task completion. The study is diagnostic; its scope does not include mobile or API benchmarks.

AuthorsOpenReview ↗

Response to Reviewer BABb (2/2): Method and efficiency

4. Distillation procedure and task specificity

Sections 3.3–3.6 and Appendix A.1 describe four stages: normalize and segment trajectories into evidence units; distill these into structured cards; consolidate and organize cards in a library; and supply selected cards as context to the executor. Appendix B provides the trajectory and skill schemas. Cards express applicability, procedures, progress and completion evidence, and recovery guidance, while the executor resolves concrete actions from the live state.

The guidance protocols are task-family matching on WebArena-Hard and one rubric-informed, case-level or domain-level matched card per subsequent ClawBench run. The executor is instructed to prioritize live observations when they conflict with a card. These details specify the supervision and execution interface; the paper does not quantify the human effort involved in constructing and checking the library.

5. Field ablations and raw-trajectory prompting

The current design comparisons examine graph organization and execution-time adaptation. Besides the flat-versus-graph result above, the always-follow variant achieves 77.5%, compared with 81.4% when the executor can adapt guidance to live observations (§4.3).

These results support the organization and execution choices. The individual contribution of each card field, and the advantage of cards over raw-trajectory few-shot prompting with identical demonstrations, remain distinct questions. Accordingly, §4.6 interprets the main comparison as the benefit of supplying demonstration-derived guidance, rather than attributing the full gain to card structure alone. The field definitions and case studies explain how the representation is used without assigning unsupported numerical effects to individual fields.

6. Cost and amortization

The measured efficiency gains concern online execution. Mean WebArena-Hard tool calls decrease from 31.2 to 22.7, and the full GitLab library reduces mean token usage from 146.5K to 109.8K relative to skill-off.

For a fixed library reused across tasks, amortized cost consists of construction cost divided by the number of uses, plus per-task execution and any additional retrieval or storage costs. The submission does not report the construction-time LLM calls, tokens, or wall-clock time needed to calculate that total for 258 tasks. The supported conclusion is therefore improved online interaction efficiency. The distinction between this measurement and total system cost is already noted in §4.3 and §4.6.

Reviewer_BABbOpenReview ↗

Thanks it's clear

Thanks it's clear for me now. Will icnrease the score.

Reviewer_JnQUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Tackling the problem of leveraging demonstrations as skills for LLM agents, this work proposes a framework named BrowserBC, which converts browser trajectories into structured natural-language skills. It first cleans demonstration trajectories and segments them into sub-procedures, preserving task context, actions, feedback, and observed outcomes. Then, an LLM-based distiller converts this into skill cards, specifying applicability conditions, execution steps, progress checks, completion evidence, and recovery strategies, while eliminating session-specific details such as coordinates and DOM selectors. To maintain the skill library, the authors merge skills with overlaps and organize them into a skill graph representing dependencies, specializations, and alternative procedures. During the inference/test time, the LLM agent is given matched procedural guidance and grounds their instructions in the live browser state. The authors show that their framework can improve performance on 258 WebArena-Hard tasks and 152 ClawBench tasks. Additional experiments suggest that distilled skills can be reused across different executor models.

Critical strengths

None

Critical weaknesses
  • Skill learning from demonstrations (and leveraging them) for LLM agents is a topic that has been extensively explored in the past, and this work needs clearer positioning regarding its contributions and novelty. Specifically, this work claims 3 contribution points (L096), where contribution 3 is about their empirical results.
    Regarding contribution 1, Agent Workflow Memory ("Agent Workflow Memory", Wang et al., ICML 2025) induces reusable natural-language workflows from web navigation trajectories, AutoGuide ("AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents", Fu et al., NeurIPS 2024) retrieves relevant conditional natural-language guidelines (extracted from trajectories) based on applicability contexts, ICAL ("VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought", Sarch et al., NeurIPS 2024) abstracts state changes, temporal subgoals, task relationships, etc. (which are relevant to "progress"), AutoManual ("AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning", Chen et al., NeurIPS 2024) manages procedural knowledge in a structured manner, and ExpeL ("ExpeL: LLM Agents Are Experiential Learners", Zhao et al., AAAI 2024) extracts knowledge from success and failure trajectories.
    Regarding contribution 2, Agent Workflow Memory (Wang et al., ICML 2025) has a trajectory -> reusable workflow/skill -> future execution pipeline, ICAL (Sarch et al., NeurIPS 2024) performs evidence/trajectory abstraction and skill distillation, Synapse ("Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control", Zheng et al., ICLR 2024), and AutoManual (Chen et al., NeurIPS 2024) tests cross-model knowledge transfer. I believe there are also papers on treating skills as nodes in graphs (e.g., SkillTracer ("SkillTracer: Structural Failure Attribution and Refinement of Agentic Skills in Long-Horizon Web Tasks", Li et al., KDD 2026)).
    While I'm not suggesting this work has no new contributions compared to existing work, it is largely missing many relevant papers on the same or similar problems and comparisons with them for clearer positioning of this work.
  • The evaluation presented in this work does not seem to be a proper held-out evaluation for testing generalization. For WebArena-Hard, the authors state "WebArena-Hard skills are distilled from demonstrations associated with the same task family." (L291). Regarding ClawBench, they state "ClawBench uses an iterative protocol: cases are first run without the skill library, successful demonstrations are distilled, and later runs receive a matched case-level or domain-level skill." (L292) and "Distillation inputs include the task description, evaluation rubric, verified human demonstrations, and available completion evidence. Each subsequent ClawBench run receives one rubric-informed, matched card." (L294). These imply that the evaluation of BrowserBC involves privileged information about the tasks (especially for the ClawBench results) and demonstrations for the same task distribution. Besides, it is compared with a naive baseline without skills and access to the privileged information, which is not very helpful for demonstrating the empirical impact of this work, considering the large volume of existing work in this domain.
Questions for authors

Please refer to the Critical Weaknesses section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer JnQU

Thank you for the detailed review and the relevant literature pointers. We address the contribution and evaluation protocol below, with reference to the evidence in the submitted paper.

1. Contribution and relationship to prior work

BrowserBC builds on the established direction of reusing agent experience. We agree that Agent Workflow Memory, AutoGuide, ICAL, Synapse, and AutoManual deserve more direct discussion in our positioning. Our contribution lies in the design and evaluation of a browser procedural-knowledge framework that connects evidence abstraction, structured skill cards, graph-based organization, and execution grounded in live observations.

The central design choice is to retain more than an action sequence: each card connects applicability and execution steps to progress checks, completion evidence, and recovery guidance. The executor interprets these instructions against the current page. The paper evaluates this framework through several complementary comparisons:

  • Library organization: On 113 GitLab cases, the graph-organized library achieves 89.4% success versus 83.2% for a flat library, while mean token usage decreases from 115.4K to 109.8K (§4.4, Figure 4b).
  • Adaptive execution: Allowing the executor to override guidance based on live observations achieves 81.4% success, compared with 77.5% for the always-follow variant (§4.3).
  • Cross-model reuse: Using the same source trajectories, Sonnet-produced skills improve the Qwen executor from 53% to 77%, while Qwen-produced skills improve the Sonnet executor from 60% to 70% (§4.3, Figure 3b).

These comparisons support specific aspects of BrowserBC beyond the main skill-on/skill-off improvement: organizing procedural knowledge, adapting it during execution, and separating its producer from its consumer. This is the contribution we intend to emphasize, rather than priority for natural-language skill learning or graph representations in isolation. Direct comparisons with the cited systems would establish their relative performance; the present results establish these properties within BrowserBC.

2. What the evaluation establishes

The main experiments evaluate execution with matched procedural guidance. WebArena-Hard uses demonstrations associated with the evaluated task family; ClawBench uses rubric-informed, case-level or domain-level matched cards. This scope is stated in §4.1, Appendix A.1, and §4.6.

The question is practically relevant to repeated workflows: when procedural experience is available, can an agent use it to complete tasks through live interaction? Both conditions use the same executor, action interface, and step budget. Under the reported guidance protocols, success increases from 156/258 to 210/258 on WebArena-Hard and from 50/152 to 104/152 on ClawBench. Mean WebArena-Hard tool calls also decrease from 31.2 to 22.7.

Your distinction between this setting and held-out generalization is important. The results support the utility of matched guidance; they do not establish transfer to unseen task families or websites. In particular, ClawBench's rubric access is part of its supervision, and resetting the browser or removing stale literals does not create a held-out split. Appendix A.1.3 and §4.6 explicitly make these distinctions.

The additional studies address separate questions. Figure 3b tests reuse across executors, Figure 4b tests library organization, and the always-follow comparison tests whether guidance should remain adaptable during execution. Together, they provide evidence for BrowserBC as a framework for organizing and applying procedural experience, with its main performance results interpreted under the disclosed supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.