- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studys the sources of failure in CUAs and the reliability of automatic diagnosis. Based on a constructed corpus containing 1,712 execution trajectories, the authors manually induct failure diagnosis descriptions, establish and finalize a failure classification system comprising 6 major categories and 18 subcategories, and design a three-stage LLM-based diagnostic process. The authors conducted their analysis on 3,825 execution traces, covering four agent frameworks, 2 execution models (Qwen 3.6 Flash and GLM 5.3 Flash), and different versions of OSWorld and ClawBench.
Critical strengths
-
The paper systematically analyzes the failures of CUA, distinguishing between sources of failure such as agents, the runtime environment, and evaluators.
-
Through sensitivity experiments, the authors demonstrate the impact of log normalization, screenshot information, and judge prompts on diagnostic results, which is valuable for understanding the limitations of automated diagnosis and designing future evaluations.
Critical weaknesses
-
The main experiment analyzes only two models: Qwen 3.6 Flash and GLM 5.3 Flash. Testing multiple agent frameworks does not establish whether the reported dominant failure patterns persist for frontier CUAs or mainly reflect limitations of the selected backbones.
- As an analytical paper, it would be more valuable to discuss why current strong CUA systems still fail on difficult tasks and in real-world applications.
-
For the more challenging OSWorld 2.0 benchmark, the authors used only 30 out of 108 tasks. Although they describe the selection process, they do not adequately explain why this subset is sufficient to represent the key failure mechanisms in long horizon tasks.
-
The paper lacks comparative experiments involving the judge backbone (other than Gemini 3.5 Flash-Lite); it remains unclear whether the primary attribution results and failure distributions remain stable when the diagnostic model is replaced.
Questions for authors
See weakness above.