- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents a step-by-step recipe for training a better computer-use / web agent (CUA). It has four parts: a training dataset (live-browser-set, built with a three-part collection recipe , , ), a new evaluation benchmark (BrowserBench), and two fine-tuning stages (SFT with general, UI, and recovery portions of data and training schedule, then RL with a proposed GRPO variant). Experiments show gains over baselines on the tested benchmarks, plus studies of the SFT data mixture / schedule and ablations of the GRPO variant.
Critical strengths
- Interesting content and large coverage: The paper covers a training set, a benchmark, and two fine-tuning stages, with detailed theoratical description of finetining and demostration of gains over baselines on the tested benchmarks. This is an interesting combined recipe for training CUA agents.
- Useful analysis: the study of SFT data mixture / schedule and the ablation of the GRPO variant show practical information for anyone training similar agents.
Critical weaknesses
Some of these suggestions could go in the appendix if the main body becomes too long. I am willing to raise my score if these can be substantially addressed.
Not very easy to understand:
- Sections 5 and 6 are confusing with all the symbols. It seems some symbols are not properly introduced before use.
- The introduction is organized in a strange way (BrowserBench is mentioned at the beginning, before the problem formulation, etc.). It probably needs a rewrite.
Ablation results do not seem to report statistical significance
- Improvement over the baseline is shown on the tested benchmarks, but due to the highly dynamic nature of computer use, it seems hard to find the results convincing without repeated trials.
The main results (Table 2) could benefit from more explanation / model coverage / benchmark coverage
- No specialized CUA models are included in this table. The explanation for this omission is that frontier models usually outperform specialized CUA models, but this claim is not supported by evidence or citations.
- Frontier model coverage is also not ideal (Anthropic, Google, etc.). If resources are a constraint, the authors could cite public results. This is especially needed since a new benchmark (BrowserBench) is introduced as well.
- Some standard CUA benchmarks are not tested: WebArena? Mobile benchmarks if possible? OSWorld if possible?
Would be nice to have:
- The data collection process needs more coverage in the main text.
- No code / data release at all, despite being promised in the appendix.
Missing some benchmark / citation coverage
- Related work on web / CUA / GUI agents? SFT training of other web / CUA / GUI agents? Offline / online RL for web agents?
- The 4 major components of the paper each need their own discussion of prior work, so readers can better understand their novelty. Perhaps the related work section should be reorganized around them.
- For example, prior work related to the live-browser-set's data collection recipe (D_g, D_u, D_r) is not discussed at all.
Questions for authors
- How many runs are behind Table 2 and the ablations? Could you report mean and standard deviation over repeated trials?
- What supports the claim that frontier models outperform specialized CUA models? Could you add a few specialized CUA models and more frontier models to Table 2 (public numbers are fine if compute is limited)?
- Based on your setup, is it possible / sensible to add results on standard CUA benchmarks such as WebArena, and confirm whether code and data will be released as the appendix states?
2 comments
Thank You for Your Support and Feedback
Dear Reviewer JxRX,
Thank you very much for your insightful guidance throughout the review process and for raising your score to support our work! We also sincerely appreciate your two constructive follow-up suggestions, and we will incorporate both into our final manuscript.
Best regards, The Authors
Thank you
Thank you to the authors for addressing my concerns. I am willing to raise my score because most comments were substantially addressed, but I do want to mention 2 lingering concerns.