- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to leverage frontier closed-source models to teach smaller models to master web tasks. To achieve both on-policy and dense token-level supervision without confusing the student model by forcing it to directly copy the teacher's surface style, the authors devise a carefully designed method called WebKD. Starting from SFT as a warmup, the teacher only rewrites failed trajectories in the student's style, and the rewritten records are optimized by reweighting the token-level loss with a KL anchor, which is a more moderate and smoother practice. Experiments across relatively new open-source model series and scales show its effectiveness compared with other methods like SFT or RL alone.
Critical strengths
- The overall motivation, combining i) frontier models' capability, ii) an on-policy nature, and iii) avoiding confusing the student into copying the surface style of the teacher, is clear and reasonable.
- The overall method is well-designed and well-motivated.
- The analysis not only cares about overall performance, but also provides informative breakdowns, e.g., comparison across task difficulties and unseen environments.
Critical weaknesses
W1 Limited training data: the training corpus is quite small, so it is unclear whether the small dataset causes overfitting and how the performance gap among SFT, RL, and WebKD would change when the training data scales up.
W2 Limited discussion on generalization vs. data scaling: distillation may sacrifice some out-of-distribution generalization compared to RL, while its main strength lies in sample efficiency. Since the current training data is very limited, I would be eager to see more generalization analysis as data scales, so that each method (SFT, RL, and WebKD) can show its full potential.
W3 (Minor point) Although the method is effective overall, the four-stage pipeline feels a bit heuristic-heavy and design-heavy to me. I wonder whether WebKD could be simplified into a more direct paradigm for leveraging teachers, e.g., directly skipping the initial SFT stage, letting the base student roll out and the teacher rewrite on the fly, and training iteratively.
Questions for authors
Please see my above weaknesses
2 comments
Follow-up on our rebuttal
Dear Reviewer,
We thank you again for your feedback on our work! Your suggestions were very valuable in improving our submission. As we are near the end of the discussion period, we would greatly appreciate your feedback on our rebuttal.
We tried our best to address your concerns on (W1/W2) training corpus size, generalization and overfitting, and data scaling, and (W3) pipeline design.
We would be really grateful to know whether these address your concerns. We are more than happy to address any remaining questions or run further experiments if that would be helpful!
Best regards,
The Authors
Rebuttal by the Authors
We sincerely thank the reviewer for the thoughtful review and for recognizing the motivation, method, and analysis of our work. The reviewer's comments helped us strengthen our work by adding discussions on the training data, generalization, data scaling, and the design choices behind our pipeline.
W1, W2. Training corpus size, generalization and overfitting, data scaling.
Corpus Size. We agree with the reviewer that 15 failed trajectories may look small, but we would like to provide additional context. As we correct each step independently, the actual training set consists of 392 step-level samples, not the 15 trajectories. Failure sets are also generally small by design, which is not specific to our method or benchmarks. WebKD starts from the SFT seed, which is trained on the same 128 training tasks. The 15 failures are cases that SFT could not fix. Most post-training method that learns from a strong seed’s leftover failures will have a corpus of similar size. Even with only 392 samples from 15 trajectories, WebKD improves TimeWarp performance by a substantial 5.5-10.7 pp.
Generalization and Overfitting. Thank you for raising this point! We do believe there is a lower risk of overfitting, as WebKD has been evaluated on unseen tasks (Tab. 1, TimeWarp's evaluation splits), versions (Tab. 1, TimeWarp's unseen versions), and benchmarks (Fig. 4, unseen benchmark, OnlineMind2Web). In all three scenarios, WebKD outperforms the remaining baselines, strongly hinting at generalization rather than overfitting.
We think the strongest evidence for out-of-distribution generalization would come from the live web benchmark, OnlineMind2Web, where WebKD improves over BC by up to +6.0 percentage points. However, these numbers may appear low, since all baseline scores are <8%. We provide additional evidence of generalization by evaluating across several domains on another live web benchmark, WebVoyager. Here, WebKD often outperforms the second-best baseline by as much as 10 pp.
Scaling. Thank you for suggesting the scaling experiments, as this helped validate the efficacy of WebKD across varying corpus sizes. Given the limited number of failure trajectories, we resampled TimeWarp v6 training rollouts with different seeds until we reached 100 unique failures. We then train an SFT baseline (TimeWarp-BC), an RL baseline (mGRPO), and WebKD on varying corpus sizes. We report the success rate on TimeWarp v6 below:
WebKD leads across all corpus sizes, and its lead increases with more data. Between 25% and 100% of the corpus, WebKD gains 15.2 percentage points, compared with 8.7 for BC and 7.1 for GRPO. This analysis also prompted us to investigate whether increasing the number of training epochs could help the BC-TW seed, which flattens at 25% compared to WebKD's peak at 35% (Fig. 9).
We have added these to the revised manuscript: corpus size in Section C.3, scaling in Fig. 7, and generalization in Section 4.2, Fig. 6.
W3. Pipeline design
We agree that the four-stage pipeline is a bit design-heavy. We also felt the same way while building our method, as the design choices kept piling up. We had experimented with some of the mentioned approaches, e.g., combining the base student rollouts with the teacher edits into a single-phase training. However, this performed similar to SFT, which can be attributed to the different nature of the training data in the SFT and teacher corpora, and to training on both simultaneously.
We really appreciate your suggestions to skip SFT and to use iterative training. Ablating the pipeline (Tab 4 of the revised version), we found that WebKD strongly depends on the SFT seed, and that multiple iterations of WebKD provide minimal gains.
The corpus scaling and pipeline ablations also inspired another question: "What if we train the SFT seed for more epochs?" This also didn't work, as increasing the number of training epochs doesn't help the SFT seed, which flattens at 25% compared to WebKD's peak at 35% (Fig. 9).
We sincerely thank you and strongly believe that your review has helped us shape our work into a much stronger paper.