- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper uses computer use agents or GUI agents to run comparative task level evals on OSWorld. The first experiment is running the same family of model across sizes and going deeper than the aggregate improvement and seeing whether the smaller models success is a subset of the larger models; the results however show that a significant share no overlap. Further reruns show more such swaps showing evals are stochastic. There is a an LLM labelled taxonomy that is given which classifies failure types with most being premature ending or incorrect stuck loops. They also evaluate reflective recovery where GUI agents check task completion, generate feedback from prior attempts and retry. This improves overall success modestly but incorrect completion judgments can also trigger unnecessary retries that destroy previously successful states.
Critical strengths
- Rather than relying solely on aggregate success rates, the paper measures individual task improvements and regressions, alongside same-configuration reruns that reveal substantial stochastic variation.
- The paper demonstrates that recovery mechanisms can inadvertently discard previously successful states when completion checkers incorrectly judge tasks as incomplete.
Critical weaknesses
- The paper primarily demonstrates phenomena that are already well established in existing literature - no novelty or significance of the work can be found.
- The experiments rely on a single benchmark, a narrow selection of models and weak ones at that (4B and 9B).
- The reflection experiments do not isolate the benefits of reflection from simply allowing additional attempts while the FMART comparisons involve different controllers and execution budgets.
- No solution for the problematic phenomena has been provided.
- The taxonomy of errors found are synthetic which is fine but the results of the tagging are common knowledge again given that endless looping and premature ending are common llm artifacts.
Questions for authors
What better benchmarking practices or methods do you suggest can over come this? Do you have AI measurement robustness ideas that could make future evals better suited for this? maybe different metrics? General Practices?