- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors proposed a novel GUI agents' trajectory generation method, which preserve the action sequence and re-ground the targets from pixels on the live interface. They found training on a few workflows could achieve a high success rate on benchmarks such as MiniWoB++ and WebArena.
Critical strengths
- The overall logic is clear and the problem is important and sound.
- The performance seems strong based on the experiments.
Critical weaknesses
- The writing needed to be improved.
The sentence in Figure 1 has grammatical errors “ A computer-use agent completes the task with expensive and delay”.
“A agent treats both as unknown” L060 -> “An agent …”
“Procedure failures are a planning problem and belong upstream” -> “belong to the upstream”
The current citation style in l065-073 is wierd, it should be like (Leotta et al. 2016, Choudhary et al. 2011, Leotta et al. 2015, Nass et al. 2023) instead of the current form.
“gaming where user does not have access for DOM become challenge for above solutions” -> “become a challenge” / “become challenging”
These sentences have grammatical errors “However, different task with different GUI will not depend on same setting of hard-coded design. Implement heavy weight on text (OCR) over GUI without text will under perform the performance while same setting could performed well under rich text screenshot”.
I highly recommend the authors to go through the paper and proofread the sentences. The current paper contains a lot of grammatical errors and hinder reading. A few more: L151 “They do not measures end-to-end task success on public benchmarks …”
This sentence is difficult to understand “The conclusion is a boundary on the claim rather than a caveat about it”
- The authors need justification for their methodological choices.
How the number 9 is determined in L184-185, and similarly, how intersection-over-union of 0.5, and how centers lies within 12 pixels are determined in L217-219.
Similarly, how the exact coefficients was determined in L243-248. The underlying rationale is clear, but how the exact number is determined is less disclosed.
Why the removal of webshop font size tasks are fair? This needs more explanation.
The authors need to specify what the error bar represents in Figure 3.
I think the p value in Table 2 should be compared to the strongest baseline for each task, rather than compared in a fixed manner with hybrid. A more formal way should be to detect the main effects of techniques, and then perform post-hoc comparisons with appropriate adjustments.
I think the authors could better explain “the frozen resolver transfers to unseen websites, unseen layouts, and unseen tasks within a represented perturbation vocabulary. It does not learn a universally perturbation-invariant notion of a target”, especially around how the current results could support these implications.
Questions for authors
- How they determined the methodological choices for their methods and experiments?