acceptodds
Under review as a conference paper at ICLR 2027

Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents

16 views

Abstract

Frontier web agents using closed models have high inference and computational costs. Smaller open models can be an alternative, but to be performant, they require training supervision from a human or teacher model. Token-level teacher distillation is not an option for closed teachers as it needs access to the model's logits. The standard approach is to imitate teacher demonstrations offline, but it trains the student on states it has never visited, and in a style it does not write in. We introduce **WebKD**, where the teacher is the editor of the student's failures. Given a failed episode, the teacher finds the first point of mistake, and rewrites the student's reasoning traces, harness states, and actions from that point onward. The editing reveals that the teacher's corrections can be more surprising to the student than the student's own text. WebKD instead aims to edit minimally. In the data space, the teacher writes each correction in the student's style. In the objective space, the reasoning is weighted by its contribution to the action. A role-asymmetric KL anchor allows the corrected action to move freely while keeping the reasoning near the student's prior. On the TimeWarp benchmark, WebKD students gain about 5.5–10.7 pp over behavior cloning, outcome-reward RL, and teacher editing baselines. The performance transfers to unseen versions and to the live web in the OnlineMind2Web benchmark. WebKD opens a new way of distilling black-box models to improve the capabilities of open-weight web agents.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Web Agents, Knowledge Distillation, Black Box Distillation, Small Language Model Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rZMKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to leverage frontier closed-source models to teach smaller models to master web tasks. To achieve both on-policy and dense token-level supervision without confusing the student model by forcing it to directly copy the teacher's surface style, the authors devise a carefully designed method called WebKD. Starting from SFT as a warmup, the teacher only rewrites failed trajectories in the student's style, and the rewritten records are optimized by reweighting the token-level loss with a KL anchor, which is a more moderate and smoother practice. Experiments across relatively new open-source model series and scales show its effectiveness compared with other methods like SFT or RL alone.

Critical strengths
  1. The overall motivation, combining i) frontier models' capability, ii) an on-policy nature, and iii) avoiding confusing the student into copying the surface style of the teacher, is clear and reasonable.
  2. The overall method is well-designed and well-motivated.
  3. The analysis not only cares about overall performance, but also provides informative breakdowns, e.g., comparison across task difficulties and unseen environments.
Critical weaknesses

W1 Limited training data: the training corpus is quite small, so it is unclear whether the small dataset causes overfitting and how the performance gap among SFT, RL, and WebKD would change when the training data scales up.

W2 Limited discussion on generalization vs. data scaling: distillation may sacrifice some out-of-distribution generalization compared to RL, while its main strength lies in sample efficiency. Since the current training data is very limited, I would be eager to see more generalization analysis as data scales, so that each method (SFT, RL, and WebKD) can show its full potential.

W3 (Minor point) Although the method is effective overall, the four-stage pipeline feels a bit heuristic-heavy and design-heavy to me. I wonder whether WebKD could be simplified into a more direct paradigm for leveraging teachers, e.g., directly skipping the initial SFT stage, letting the base student roll out and the teacher rewrite on the fly, and training iteratively.

Questions for authors

Please see my above weaknesses

2 comments
AuthorsOpenReview ↗

Rebuttal by the Authors

We sincerely thank the reviewer for the thoughtful review and for recognizing the motivation, method, and analysis of our work. The reviewer's comments helped us strengthen our work by adding discussions on the training data, generalization, data scaling, and the design choices behind our pipeline.


W1, W2. Training corpus size, generalization and overfitting, data scaling.


Corpus Size. We agree with the reviewer that 15 failed trajectories may look small, but we would like to provide additional context. As we correct each step independently, the actual training set consists of 392 step-level samples, not the 15 trajectories. Failure sets are also generally small by design, which is not specific to our method or benchmarks. WebKD starts from the SFT seed, which is trained on the same 128 training tasks. The 15 failures are cases that SFT could not fix. Most post-training method that learns from a strong seed’s leftover failures will have a corpus of similar size. Even with only 392 samples from 15 trajectories, WebKD improves TimeWarp performance by a substantial 5.5-10.7 pp.


Generalization and Overfitting. Thank you for raising this point! We do believe there is a lower risk of overfitting, as WebKD has been evaluated on unseen tasks (Tab. 1, TimeWarp's evaluation splits), versions (Tab. 1, TimeWarp's unseen versions), and benchmarks (Fig. 4, unseen benchmark, OnlineMind2Web). In all three scenarios, WebKD outperforms the remaining baselines, strongly hinting at generalization rather than overfitting.

We think the strongest evidence for out-of-distribution generalization would come from the live web benchmark, OnlineMind2Web, where WebKD improves over BC by up to +6.0 percentage points. However, these numbers may appear low, since all baseline scores are <8%. We provide additional evidence of generalization by evaluating across several domains on another live web benchmark, WebVoyager. Here, WebKD often outperforms the second-best baseline by as much as 10 pp.

MethodAppleCambridgeHuggingFace
Zero-Shot3.973.173.25
BC-TW (SFT)6.3512.704.07
mGRPO (RL)6.354.763.25
WebKD11.9023.0210.57

Scaling. Thank you for suggesting the scaling experiments, as this helped validate the efficacy of WebKD across varying corpus sizes. Given the limited number of failure trajectories, we resampled TimeWarp v6 training rollouts with different seeds until we reached 100 unique failures. We then train an SFT baseline (TimeWarp-BC), an RL baseline (mGRPO), and WebKD on varying corpus sizes. We report the success rate on TimeWarp v6 below:

Method25%50%75%100%
BC-TW (SFT)22.0123.3027.5130.74
mGRPO (RL)24.6026.8629.4531.72
WebKD27.5133.6639.1642.72

WebKD leads across all corpus sizes, and its lead increases with more data. Between 25% and 100% of the corpus, WebKD gains 15.2 percentage points, compared with 8.7 for BC and 7.1 for GRPO. This analysis also prompted us to investigate whether increasing the number of training epochs could help the BC-TW seed, which flattens at 25% compared to WebKD's peak at 35% (Fig. 9).


We have added these to the revised manuscript: corpus size in Section C.3, scaling in Fig. 7, and generalization in Section 4.2, Fig. 6.


W3. Pipeline design

We agree that the four-stage pipeline is a bit design-heavy. We also felt the same way while building our method, as the design choices kept piling up. We had experimented with some of the mentioned approaches, e.g., combining the base student rollouts with the teacher edits into a single-phase training. However, this performed similar to SFT, which can be attributed to the different nature of the training data in the SFT and teacher corpora, and to training on both simultaneously.

We really appreciate your suggestions to skip SFT and to use iterative training. Ablating the pipeline (Tab 4 of the revised version), we found that WebKD strongly depends on the SFT seed, and that multiple iterations of WebKD provide minimal gains.

VariantS.R. ↑
Zero-Shot0.97
WebKD (1 round)41.10
↳ w/o SFT seed (1 round)4.21
↳ w/o SFT seed (2 rounds)9.06
WebKD (2 rounds)41.75

The corpus scaling and pipeline ablations also inspired another question: "What if we train the SFT seed for more epochs?" This also didn't work, as increasing the number of training epochs doesn't help the SFT seed, which flattens at 25% compared to WebKD's peak at 35% (Fig. 9).


We sincerely thank you and strongly believe that your review has helped us shape our work into a much stronger paper.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

We thank you again for your feedback on our work! Your suggestions were very valuable in improving our submission. As we are near the end of the discussion period, we would greatly appreciate your feedback on our rebuttal.

We tried our best to address your concerns on (W1/W2) training corpus size, generalization and overfitting, and data scaling, and (W3) pipeline design.

We would be really grateful to know whether these address your concerns. We are more than happy to address any remaining questions or run further experiments if that would be helpful!

Best regards,

The Authors

Reviewer_Don5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WEBKD, which improves web agents by leveraging teacher model to correct mistakes in the students's style. The approach starts with behavior cloning on teacher rollouts, then performs student rollout, asks a teacher to identify the mistakes of the student, and rewrite the subsequent trajectory. Instead of directly replacing the trajectory with teacher-generated sequences, WEBKD encourages teacher to edit in the student's style. The method also introduces action-informative token weighting, which conditions on the student's likelihood of the corrected action when individual tokens are removed, and a role-asymmetric KL penalty. Applied to the TimeWrap benchmark, the method achieved 5.5-10.7pp over baseline methods, and generalizes to unseen benchmarks.

Critical strengths
  1. A interesting approach to address the student-teacher mismatch. By identifing the failure and editing in the student's style, the method controls the perplexity level, and makes leraning smoother for the student.
  2. Great empirical gains across model families. Meanwhile the ablation shows isolated effects of teacher module, the style of editing, and correction strategies.
Critical weaknesses
  1. The generalization result is weak. Though WEBKD achieves substantial gains on TimeWarp v6, the generalization result on Online-Mind2Web remain below 8%. This raises questions about whether this method is effective in challenging, long-horizon web interactions.
  2. The action-informative token weighting is not properly justified. Token deletion can change the semantics and coherence of a sequence, and the resulting likelihood difference may reflect distributional artifacts rather than the token’s causal contribution to the action. It would be great if there could be an ablation with other weighting alternatives, like uniform weighting.
Questions for authors
  1. How robust is the proposed action-informative token weighting method?
  2. What is the cost of the teacher-based edition, like API usage, total training cost, against the resulting performance gains?
  3. How should the low absolute success rates on live websites be interpreted?
2 comments
AuthorsOpenReview ↗

Rebuttal by the Authors

We sincerely thank the reviewer for the thoughtful review, for finding our approach interesting, and for recognizing the empirical gains. The reviewer's comments helped us strengthen our work on the generalization results, clarify action-informative token weighting, and help interpret the live web results.


W1/Q3. Generalization and interpreting live web results

We appreciate that you raised this concern. We first clarify that the low success rate on live web Online-Mind2Web (OM2W) doesn’t come from the training method or the nature of the live web. It is because (a) the evaluated models, Qwen-3.5 4B and 9B, are on the smaller side, and (b) these models are not explicitly trained on the tasks from OM2W. Small models tend to perform worse on unseen benchmarks, which is why the OM2W baselines stay below 4.5%. However, even at these lower scores, WebKD outperforms the other baselines, improving BC by +2.6 and +6.0 percentage points for the Qwen 4B and 9B models, respectively.

We do agree that the OM2W scores are relatively low, and your comment helped us further investigate the failures. For TimeWarp, most of WebKD's failures come from wrong answers (24.3%), followed by being on the wrong pages, unsupported answers, and getting stuck in loops. However, the two most dominant failures for live web are getting stuck in a loop (45.5%) and blocked webpages (22.2%), e.g., Cloudflare verification or CAPTCHA. Blocked pages are a common issue on the live web and are not due to model or harness failure. This somewhat contributes to the low OM2W scores.

To further validate the efficacy of WebKD on the live web, we select three domains from another live web benchmark, WebVoyager, where the agents are highly unlikely to encounter blockers. The results show higher success rates for WebKD across all domains, while outperforming the other baselines.

MethodAppleCambridgeHuggingFace
Zero-Shot3.973.173.25
BC-TW6.3512.704.07
mGRPO6.354.763.25
WebKD11.9023.0210.57

We have added these in Sections 4.2 and D.2 and Figures 6, 10, and 11 in the revised manuscript.


W2/Q1. Action-informative token weighting

Thank you for pointing this out! We would first clarify that uniform weighting is in Tab. 3 as LCEL_{CE} (App. C.2), beside three other weightings: grounding, teacher self-consistency, and student perplexity. Action-informative weighting outperforms uniform weighting in both columns: 41.10 against 37.43 for uniform with the anchor, and 38.51 against 36.25 without.

We also add context to the deletion artifacts. We did not intend ΔsΔ_s to be read as a causal effect. It is the student's own pointwise mutual information between a span and the corrected action (Remark 1). Spans are whole sentences or plan lines, never tokens, so a deletion leaves well-formed text (App. B.5). Weights are floored at φ = 0.1, so a mis-scored span is down-weighted, never removed. This adds to the robustness of action-informative weighting.

We agree that this required additional clarification, and have added this discussion in Section 3.3.1 (Lines 359:362).


W3/Q2 Cost of API and training cost.

We thank the reviewer for this suggestion, as it helped us establish the effectiveness of WebKD in terms of cost vs. performance. Below, we present the cost in terms of total teacher tokens, API cost, and H200 GPU hours used for a single training run:

MethodTeacher TokensAPI CostGPU Hours
BC-TW (P1)21.8 M$143–15616.75
+ WebKD (P2-P4)+4.5 M+$27–32+2.32

WebKD's correction stages add less than a fifth of the SFT seed’s API cost and about a seventh of its GPU hours.

The cost analysis has been added in Table 5 of the revised manuscript.


We express our gratitude to the reviewer and strongly believe that the review has helped us shape our work into a much stronger paper.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

We thank you again for your feedback on our work! Your suggestions were very valuable in improving our submission. As we are near the end of the discussion period, we would greatly appreciate your feedback on our rebuttal.

We tried our best to address your concerns on (W1/Q3) generalization and interpreting live web results, (W2/Q1) action-informative weighting, and (W3/Q2) API and training cost.

We would be really grateful to know whether these address your concerns. We are more than happy to address any remaining questions or run further experiments if that would be helpful!

Best regards,

The Authors

Reviewer_jyF5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes WebKD, a black-box knowledge distillation framework for web agents that leverages teacher corrections of student failure trajectories. The method combines student-style alignment, action-informative token weighting, and role-asymmetric KL regularization to improve the task performance of smaller web agents.

Critical strengths
  • The paper addresses the limitation that closed-source teacher models do not provide access to logits by proposing a text-based correction approach for knowledge distillation.

  • By incorporating student-style alignment, action-informative token weighting, and KL regularization, the proposed method explicitly accounts for distributional differences between teacher and student models.

  • The paper evaluates the proposed method across multiple student models and web environments, with ablation studies demonstrating the effectiveness of its main components.

Critical weaknesses

Weakness 1

A key design of the proposed method is to have the teacher correct the student's reasoning, plans, memory, and actions starting from the first erroneous step. However, this mechanism may introduce inconsistencies between states and actions. According to P4 in Section 3.3, the teacher uses the observations from the student's original execution trajectory when correcting each step, and the corrections are generated through independent teacher calls. The concern is that once the teacher modifies an action at a particular step, the subsequent web states may also change. Consequently, the observations in the original failed trajectory may no longer correspond to the corrected action sequence.

Weakness 2

Although the proposed method achieves promising performance across multiple student models and web environment versions, the scale of teacher-corrected training data is relatively limited. According to Appendix C.3, the authors use only 15 failed training trajectories in the main training configuration, resulting in 392 step-level training records after correction and filtering.

Questions for authors
  • How do the authors ensure consistency between corrected actions and subsequent observations when teacher edits are generated independently at the student's original states?
  • Could the authors provide further evidence of the method's scalability and generalization given the limited number of teacher-corrected training trajectories?
2 comments
AuthorsOpenReview ↗

Rebuttal by the Authors

We sincerely thank the reviewer for the thoughtful review and for recognizing the motivation, approach, and effectiveness of our work. The reviewer’s comments have helped us strengthen our discussion on independent edits vs. off-policy takeover, training corpus size, scalability, and generalization using another benchmark.


W1/Q1. How is consistency ensured between corrected actions and subsequent observations?

Thank you for the careful observation! We agree that each step after the first error is corrected independently, but this was an intentional design choice. By doing so, each correction becomes its own training example, which also counters the smaller corpus size of 15 training trajectories by converting it into 392 step-level training samples.

Corrected actions are also never carried forward to later steps, so every training input is a state the student has actually reached. The alternative is to execute the teacher’s correction and let it rewrite every later step. This is essentially the off-policy takeover baseline that we evaluated in Tab. 1, 2, and Fig. 4, where it empirically underperforms WebKD’s independent edits.

We do appreciate the reviewer raising this concern, as it helped us investigate further the impact of off-policy takeover. When the agent drifts off-policy, the training data contains more states the agent is less likely to visit, and training on it can affect performance in the following ways:

  • Distribution shift. In the takeover corpus, 23.9% of the training steps are on pages the student never visits. In 54.9% of the steps, the action history is one the student did not produce. Both shares are 0% for WEBKD, because every visited state in its corpus is the student’s own.
  • Failure mode. Looping until the step runs out is the dominant failure mode for takeover, as it gets stuck in loops in 44.7% of v6 episodes, compared to WEBKD’s 25.2%.

We have added Section D4, which includes this discussion, to our revised manuscript.


W2/Q2. Limited training data, scalability, and generalization


Corpus Size. We agree with the reviewer that 15 failed trajectories may look small, but we would like to provide additional context. As we correct each step independently, the actual training set consists of 392 step-level samples, not the 15 trajectories. Failure sets are also generally small by design, which is not specific to our method or benchmarks. WebKD starts from the SFT seed, which is trained on the same 128 training tasks. The 15 failures are cases that SFT could not fix. Most post-training method that learns from a strong seed’s leftover failures will have a corpus of similar size. Even with only 392 samples from 15 trajectories, WebKD improves TimeWarp performance by a substantial 5.5-10.7 pp.


Scaling. We thank the reviewer for suggesting the scaling experiments, as this helped validate the efficacy of WebKD across varying corpus sizes. Given the limited number of failure trajectories, we resampled TimeWarp v6 training rollouts with different seeds until we reached 100 unique failures. We then train an SFT baseline (TimeWarp-BC), an RL baseline (mGRPO), and WebKD on varying corpus sizes. We report the success rate on TimeWarp v6 below:

Method25%50%75%100%
BC-TW22.0123.3027.5130.74
mGRPO24.6026.8629.4531.72
WebKD27.5133.6639.1642.72

WebKD leads across all corpus sizes, and its lead increases with more data. Between 25% and 100% of the corpus, WebKD gains 15.2 percentage points, compared with 8.7 for BC and 7.1 for GRPO. This analysis also prompted us to investigate whether increasing the number of training epochs could help the BC-TW seed, which flattens at 25% compared to WebKD's peak at 35% (Fig. 9).


Generalization. We would like to first highlight the current generalization results on unseen TimeWarp versions and the live web benchmark OnlineMind2Web. For unseen TimeWarp versions, WebKD outperforms the strongest baseline by 3.0–10.6pp (Tab. 1). On the Qwen-3.5 9B model, WebKD achieves a +4 pp gain over the strongest baseline on OnlineMind2Web (Fig. 4).

We provide additional evidence for generalization by running evaluation across several domains on another live web benchmark, WebVoyager. WebKD often outperforms the second-best baseline by as much as 10 percentage points.

MethodAppleCambridgeHuggingFace
Zero-Shot3.973.173.25
BC-TW6.3512.704.07
mGRPO6.354.763.25
WebKD11.9023.0210.57

We have added these to the revised manuscript: corpus size in Section C.3, scaling in Fig. 7, and WebVoyager results in Fig. 6.

We sincerely appreciate your review and believe it helped us shape our work into a much stronger paper.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

We thank you again for your feedback on our work! Your suggestions were very valuable in improving our submission. As we are near the end of the discussion period, we would greatly appreciate your feedback on our rebuttal.

We tried our best to address your concerns on (W1/Q2) ensuring consistency between corrected actions and subsequent observations, and (W2/Q1) training data, scalability, and generalization

We would be really grateful to know whether these address your concerns. We are more than happy to address any remaining questions or run further experiments if that would be helpful!

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviewers' feedback and authors response

Dear Reviewers and Area Chairs,

We sincerely thank all the reviewers for their constructive feedback! We are grateful that the reviewers found several strengths in our submission, which affirm the value of our research. The comments have been invaluable in helping us improve our manuscript.


Strengths Highlighted by Reviewers

  • Clear motivation using closed teachers without logit access (jyF5), while staying on-policy and not making the student copy the teacher’s surface style (rZMK).
  • Well-designed method for student–teacher mismatch: editing in the student’s style keeps perplexity down and makes learning smoother (Don5). It explicitly accounts for teacher–student distribution differences (jyF5) and is overall well-designed (rZMK).
  • Great empirical gains across model families (Don5).
  • Evaluation across multiple student models and web environments (jyF5).
  • Ablations isolating the main components: the teacher module, editing style and correction strategies (Don5, jyF5).
  • Informative breakdowns by task difficulty and unseen environments (rZMK).

Key Changes in the Revision

  1. Corpus scaling (Fig. 7): WebKD leads at every dataset size, and its lead grows with data (42.7% vs. 31.7%).
  2. New live-web benchmark (Fig. 6): WebKD beats all baselines on WebVoyager, by up to ~10 pp.
  3. Failure analysis (Sec. D.2): On TimeWarp, WebKD’s most common failures are wrong answers (24.3%). On the live web, looping dominates (45.5%).
  4. Independent edits vs. takeover (Sec. D.4): takeover trains on unvisited states and loops more (44.7% vs. 25.2%), resulting in lower performance.
  5. Cost analysis (Tab. 5): WebKD adds ~1/5 of the SFT seed's API cost and ~1/7 of its GPU hours.
  6. Pipeline ablations (Tab. 4): The SFT seed is essential (41.1% vs. 4.2%), and extra rounds add little.
  7. Longer SFT training (Fig. 9): SFT plateaus at ~25%, vs. WebKD's ~35% for Qwen-3.5 9B.

We have already added the changes to the revised manuscript. We again thank all the reviewers for their critiques of our work. Their feedback led directly to these revisions, and we believe our paper is much stronger as a result. We are more than happy to clarify anything further!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.