acceptodds
Withdrawn from ICLR 2027

Why Do Computer-Using Agents Fail? A Trace-Grounded Taxonomy and Its Reliability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Computer-using agents (CUAs) are increasingly evaluated on graphical interfaces and live production websites, yet benchmarks typically report only a single terminal reward. This conflates failure origins across the model, agent scaffold, benchmark evaluator, execution harness, and task specification. We introduce a frozen CUA failure taxonomy—6 families and 18 categories—built by grounded-theory analysis over a -trace construction corpus, and apply it through a three-stage judging pipeline that decouples execution outcome from task success and attributes failures across system components. We evaluate on a disjoint held-out corpus of traces produced by a Harbor-based framework running four unmodified agent CLIs across two generations of OSWorld and ClawBench respectively. Category-level classification reaches Krippendorff's on the construction set and holds between and across the 26 held-out populations with and two blind annotators agree with the pipeline at –, against with each other over a validation set. The diagnostic infrastructure proves to be part of the measurement: trace normalization shifts attribution by percentage points, withholding screenshots raises the agent share by , and one revision of the judge's prompt moves it by roughly twenty—each exceeding the judge's own sampling noise.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
CUA, Computer Using Agents, Agents, LLM, Benchmark, Dataset, OSWorld, ClawBench, harbor, Taxonomy, Classify

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DDvmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studys the sources of failure in CUAs and the reliability of automatic diagnosis. Based on a constructed corpus containing 1,712 execution trajectories, the authors manually induct failure diagnosis descriptions, establish and finalize a failure classification system comprising 6 major categories and 18 subcategories, and design a three-stage LLM-based diagnostic process. The authors conducted their analysis on 3,825 execution traces, covering four agent frameworks, 2 execution models (Qwen 3.6 Flash and GLM 5.3 Flash), and different versions of OSWorld and ClawBench.

Critical strengths
  • The paper systematically analyzes the failures of CUA, distinguishing between sources of failure such as agents, the runtime environment, and evaluators.

  • Through sensitivity experiments, the authors demonstrate the impact of log normalization, screenshot information, and judge prompts on diagnostic results, which is valuable for understanding the limitations of automated diagnosis and designing future evaluations.

Critical weaknesses
  • The main experiment analyzes only two models: Qwen 3.6 Flash and GLM 5.3 Flash. Testing multiple agent frameworks does not establish whether the reported dominant failure patterns persist for frontier CUAs or mainly reflect limitations of the selected backbones.

    • As an analytical paper, it would be more valuable to discuss why current strong CUA systems still fail on difficult tasks and in real-world applications.
  • For the more challenging OSWorld 2.0 benchmark, the authors used only 30 out of 108 tasks. Although they describe the selection process, they do not adequately explain why this subset is sufficient to represent the key failure mechanisms in long horizon tasks.

  • The paper lacks comparative experiments involving the judge backbone (other than Gemini 3.5 Flash-Lite); it remains unclear whether the primary attribution results and failure distributions remain stable when the diagnostic model is replaced.

Questions for authors

See weakness above.

Reviewer_7NyFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why computer-using agents (CUAs) fail. A benchmark’s single terminal reward mixes failures caused by the model, agent scaffold, benchmark evaluator, execution harness, and task specification. The authors defined 6 families and 18 categories of failure taxonomy by analyzing the traces, and then apply it through a three-stage judging pipeline that attributes each failure to a system component. The judging pipeline first normalizes trace, then diagnoses failure with 3 independent passes and voting, and finally classifies failures into the taxonomy.

Critical strengths
  1. The taxonomy definition and analysis are useful for future research. The analysis that diagnosis results depend on how the judge is set up and quantifies it against resampling noise is particularly interesting.
Critical weaknesses
  1. The paper assign one failture label per trajectory when several causes may exist. Failure attribution for a trajectory (especially when it gets long) can reasonably be a multi-class classification instead of single class. The authors also noted this where annotator records multiple catogories (D.6, "The second annotator recorded multiple categories on 39 of those 183 verdicts") but the authors choose to only score the first-listed label. It is also well known that a single wrong step often triggers a cascade of subsequent failures of different types. This raised a concern of whether 1) we should treat this problem as multi-class failure attribution for a trajectory, or 2) perform single class failure attribution on step-level. The latter is actually more accurate and fine-grained but requires more efforts in annotation.
Questions for authors

Please see weaknesses. Would the author prefer evaluating multi-class failure attribution or step-level failure attribution and why?

Reviewer_CrvzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the causes of failures in computer-using agents and proposes a taxonomy consisting of six failure families and 18 categories. The authors develop a three-stage LLM-based judging pipeline that separates benchmark outcomes from causal attribution and failure classification. They evaluate the approach using 3,825 held-out trajectories across OSWorld and ClawBench, examining both the transferability of the taxonomy and the sensitivity of diagnostic outcomes to changes in the evaluation infrastructure.

The problem is important, and the investigation of diagnostic reliability is interesting. However, I have several concerns regarding the validity of the failure attribution, the interpretation of the reliability results, and the novelty of the proposed taxonomy.

Critical strengths
  • The paper addresses an important problem: benchmark-level success or failure does not necessarily identify which component of an agentic system is responsible.
  • The paper does the failure diagnostics and taxonomy for CUA agents which is relatively underexplored as of now.
  • The authors demonstrate that prompt revisions, screenshot availability, and trace normalization can substantially affect failure attribution.
Critical weaknesses
  • Since the technique is highly sensitive to prompt revisions, screenshots and other steps, what are the key takeaways? How can the authors claim generalizability and validity of the results if the framework varies its results by 20 points in some cases. All the numbers should then have error bars.
  • Unclear novelty: The paper does not discuss AgentRx https://arxiv.org/abs/2602.02475 (accepted to EMNLP) which also introduces a grounded failure taxonomy and an automated, evidence-based framework for diagnosing agent trajectories. The authors should clarify how their taxonomy and diagnostic pipeline differ from AgentRx, beyond the choice of evaluation domains and scale. The authors also do not compare against baselines they mention like the CUADebug work - https://arxiv.org/abs/2608.02643
  • Human validation is limited, particularly on long-horizon tasks: they only validate on 118 trajectories out of 3,825 ones with relatively low agreement scores.
  • Terms and implementation details unclear in several instances: unclear exactly what step localized means, how the attribution is done (is it majority voting?), detailed benchmark description missing, the authors say "did not read the trajectories cold" -- was there some abstracted version of the trajectories used for annotation?
Questions for authors

Please check the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.