acceptodds
Under review as a conference paper at ICLR 2027

Running the Gauntlet: Hard Agentic Tasks

Abstract

As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities. However, current benchmarks are typically built on popular applications with relatively simple tasks and focus on a narrow set of capabilities while overlooking broader dimensions, resulting in saturated performance on modern agents and failing to probe their limitations. To this end, we introduce GauntletBench, a web-based benchmark for evaluating agent generalisation in challenging scenarios, focusing on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning), across five less-covered professional applications (Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Designer), each with 27 vision-intensive tasks (135 in total). Our benchmark provides a modular pipeline that comprises an environment compatible with both open- and closed-source agent frameworks, a controlled web-based application, a well-structured task suite, and an automated evaluation engine with diverse metrics. Contrary to widespread expectations, our empirical results reveal that frontier agentic systems remain far from achieving human-level performance. Even the state-of-the-art agent achieves only a 28.2% success rate on our GauntletBench, highlighting the limitations in these overlooked capabilities and generalisation. By comparison, non-expert human annotators achieve over 80% success on our challenging yet feasible tasks, revealing the substantial gap between current agent capabilities and those required for complex real-world scenarios.

Primary area:
datasets and benchmarks
Keywords:
Agent Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_P6rvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GauntletBench, a benchmark of 135 tasks across five web applications: video editing, workflow construction, 3D modelling, flight analysis, and circuit design. It targets temporal perception, graphical understanding, and 3D reasoning through interactive tasks. The benchmark combines application-specific objective evaluators, MLLM-based progress scoring, and token/step measurements. The reported evaluation covers 14 agent settings and human participants, with a highest agent success rate of 28.2% versus 80.1% for humans. The paper also presents model-scale and reasoning ablations and failure analyses.

Critical strengths
  1. Useful task coverage. The benchmark brings together visually intensive, stateful interfaces involving timelines, graphs, spatial layouts, and circuits. This task distribution can broaden evaluation of GUI agents beyond common navigation and form-filling workflows.

  2. Inspectable task and evaluation design. The appendix documents task instructions and reference states extensively. Application-specific checks, including graph matching and circuit functional equivalence, provide a useful foundation for automated evaluation.

Critical weaknesses
  1. Unresolved benchmark and evaluation validity. The main results and failure analysis use different task populations (27 versus 20 tasks per application), and Table 6’s pooled token mean is inconsistent with its equally sized groups. The stated evaluators also raise validity concerns: 3D task 26 permits multiple layouts despite reference-coordinate matching, while Circuit task 25 provides its answer in the prompt without described construction verification. These issues undermine confidence in the scores and require a task/evaluator audit.

  2. Limited diagnostic insight. The analysis does not convincingly separate the claimed temporal, graphical, and 3D capability deficits from grounding, interface complexity, or interaction length. Higher partial completion does not establish that stronger agents are limited by execution rather than understanding (§3.4), and cross-system comparisons do not establish the benefit of finer-grained actions (§3.2). The paper provides useful descriptive observations, but its explanations of failure remain insufficiently validated.

Questions for authors
  1. Experimental clarification: Which task versions, agent configurations, and valid run counts underlie the main results, ablations, human evaluation, and failure analysis? Please reconcile their populations and statistics using a consistent run manifest.

  2. Evaluation validation: How does the implementation accept alternative valid layouts in 3D task 26 and verify circuit construction in measurement tasks such as Circuit task 25? What does an independent audit find about incorrect evaluator decisions, and how do corrections affect the reported scores?

  3. Specification clarification: Is the Video Editor timing tolerance 100 ms (§2.4) or 1 s (Appendix E)? For Video task 26, should clips be divided into quarters or thirds? Please clarify which specifications were used in evaluation.

6 comments
AuthorsOpenReview ↗

Response to Reviewer P6rv (Part 1/2)

We sincerely appreciate your time and effort in reviewing our manuscript. Below are our responses to your valuable comments.


Clarification of Inconsistent Results (W1 / Q1)

We sincerely apologise for these inconsistencies and thank the reviewer for bringing them to our attention. We would first like to emphasise that all main experimental results (Tables 2–4 and Figures 3–5) were obtained using the full 135-task benchmark, with three independent runs per agent. The reported inconsistencies arose from outdated descriptions and appendix analyses based on an earlier 100-task version, which we clarify below.

Specifically, the human evaluation already covered all 135 tasks (405 completions in total), and only the reported count was incorrect. The failure analysis (Section 3.4 and Appendix D.2–D.3) has been updated to the full 135-task benchmark with the 14 agents in Table 2, yielding 4,698 trajectories (4,266 failures). The main findings remain unchanged, with 97.0% of failures falling within the PR = 1–3 range.

Regarding the pooled statistics in Tables 6–9, some overall values were carried over from the earlier benchmark version with unequal difficulty-group sizes. We have recomputed all pooled means and standard deviations using the updated evaluation data across the three equally sized difficulty groups (45 tasks each). The corrected values are consistent across tables, and the main conclusions remain unchanged. For clarity, we summarise the evaluation settings for each experiment below:

ResultTasksAgents / configsRunsTrajectories
Table 2, Fig. 3–5, Table 101351434,698 (10 vision agents × 405 + 4 text-only agents × 162)
Tables 3–4, 6–9 (ablations)1356 + 6 configs3
Table 5 (judge agreement)5 trajectories per application4 judges325
Sec. 3.4 / App. D.2–D.31351434,698
Human (Table 2)13515 annotators1405

Task and Evaluator Validation (W1 / Q2)

We thank the reviewer for identifying these issues. We have audited the affected tasks and revised their evaluation criteria to ensure that valid solutions are correctly assessed.

3D Modeller Task 26. We acknowledge that the original evaluator relied on reference-coordinate matching despite the task allowing multiple valid layouts. Our audit of all 30 runs identified 7 valid solutions incorrectly rejected due to differences in leg placement. We have replaced the position matching with geometric contact constraints, while retaining exact checks for other properties (Appendix E). This correction increases the success rates of Gemini-3.1-Pro (30.9% → 34.6%), Gemini Enterprise (59.3% → 63.0%), and Claude-Opus-4.6 (14.8% → 16.0%).

Circuit Designer Task 25. We acknowledge that the example output inadvertently revealed the expected answer (62.5 Hz), which has been replaced with a neutral example (Appendix G.5). Importantly, no agent successfully completed this task in any run when the constructed circuit was verified. We also strengthened the evaluation of analogue measurement tasks to require that simulating the agent’s constructed circuit reproduces the reported value, rather than checking the reported value alone (Appendix E). This updates the success rates of Gemini-3.1-Pro (22.2% → 27.2%), GPT-5.4 CU (7.4% → 1.2%), and Gemini Enterprise (6.2% → 4.9%).

Overall, these corrections improve the validity of our task evaluation without changing the relative ranking of agents or the main conclusions.

AuthorsOpenReview ↗

Response to Reviewer P6rv (Part 2/2)

Failure Attribution and Interpretation (W2)

We thank the reviewer for these observations. We agree that the original analyses provide useful descriptive insights but do not fully establish the underlying causes of agent failures. We have strengthened the failure analysis and refined the corresponding interpretations.

Execution vs. understanding (§3.4). We agree that partial progress alone cannot distinguish execution failures from perception and reasoning limitations. To address this, we conduct a fine-grained analysis of failed trajectories, distinguishing perception, reasoning, interface operation, termination, and protocol errors. As shown in the table below, among 500 failed trajectories from the four strongest agents, interface operation accounts for 57.2% of failures, while perception (9.0%) and reasoning (25.8%) together account for 34.8%. These findings provide more direct evidence that both execution and the targeted perception and reasoning capabilities contribute to agent failures. We have incorporated this fine-grained failure analysis into Appendix D.5 of our revised manuscript.

Error TypePerceptionReasoningInterface operationTerminationProtocol
All (n = 500)9.025.857.27.80.2
3D Modeller2.048.049.01.00.0
Flight Analyser22.028.026.023.01.0
Video Editor9.018.070.03.00.0
Circuit Designer7.024.062.07.00.0
Workflow Builder5.011.079.05.00.0

Finer-grained actions (§3.2). We acknowledge that comparing Claude-Opus-4.6 CU and Gemini Enterprise cannot establish the causal benefit of finer-grained actions, as the systems differ in both their underlying models and execution frameworks. We have therefore revised §3.2 to retain the descriptive observation while removing the unsupported causal interpretation.


Clarification of Video Editor (Q3)

We thank the reviewer for identifying these inconsistencies. We clarify that the evaluation consistently followed the correct specifications, and the discrepancies were limited to textual errors. These corrections have been incorporated into the revised manuscript and do not affect the reported evaluation results.

Timing tolerance. The evaluator consistently uses a tolerance of ±1 s for start times, durations, and trim offsets, as specified in Appendix E. The “100 ms” stated in §2.4 and Figure 2 was outdated and has been corrected.

Video Editor Task 26. The correct specification is to divide clips into thirds, consistent with the task steps, segment labels (T1–T3, W1–W3), and ground truth. The reference to “quarters” in the task goal was a typo. We have also corrected an outdated clip name in Step 3.


Thank you once again for your time and thoughtful review. We will incorporate your feedback and improve our work in future versions.

Thanks for the rebuttal and most of my concerns are addressed.

  • For Circuit Designer task 25, why does Gemini’s score increase after adding construction and simulation checks?

  • The claim in §3.3 that additional reasoning generally improves performance should be narrowed. In Table 4, only Gemini improves while both GPT and Claude decrease.

Reviewer_rQXjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GauntletBench, a web-based benchmark for evaluating the generalization of computer-use agents in visually intensive professional applications. The benchmark contains five controlled applications—a video editor, workflow builder, 3D modeller, flight analyser, and circuit designer—with 135 tasks targeting temporal perception, graphical understanding, and 3D reasoning. The authors provide a modular interaction environment, application-specific objective evaluators, an MLLM-based partial-progress judge, and efficiency metrics. Experiments across open-source models, API-based MLLMs, and closed-source agent frameworks show that current systems remain far below human performance: the strongest agent achieves only a 28.2% success rate, compared with 80.1% for human participants. Ablations suggest that visual inputs, model scale, and extended reasoning can help, while error analysis identifies grounding failures, premature stopping, protocol violations, and long-horizon execution drift as major bottlenecks. Overall, this is a timely and potentially valuable benchmark that exposes important limitations of modern agents, although the current manuscript would benefit from substantially stronger internal consistency, human-baseline validation, and judge reliability analysis.

Critical strengths
  • The benchmark addresses a clear and important gap in agent evaluation. GauntletBench moves beyond common consumer-web tasks and evaluates temporal, graphical, and spatial reasoning in closed-loop professional interfaces. These capabilities are increasingly relevant but remain underrepresented in existing agentic benchmarks.

  • The environment and evaluation infrastructure are thoughtfully designed. The modular agent interface, deterministic self-hosted applications, standardized task format, and support for both open-source and closed-source frameworks improve reproducibility and make the benchmark practically useful for future comparisons.

  • The multi-level evaluation provides more diagnostic information than success rate alone. Combining strict objective success, partial-progress scoring, token consumption, interaction steps, and monetary cost gives a more complete picture of agent behavior. The application-specific evaluators are also more appropriate than naive exact-match evaluation because they account for timing tolerances, graph equivalence, rotational symmetry, and circuit-level functional equivalence.

  • The empirical analysis is broad and yields useful qualitative insights. The paper compares a diverse set of models and agent frameworks, studies task difficulty, model scale, reasoning effort, and visual ablations, and identifies meaningful failure modes. In particular, the distinction between early perceptual failures in weaker agents and late execution failures in stronger agents is valuable for guiding future research.

Critical weaknesses
  • The manuscript contains several significant internal inconsistencies. The paper alternates between 20 and 27 tasks per application, reports 14 agents in the main evaluation but uses 15 agents in the appendix’s trajectory count, gives incompatible human-task totals, and states that no API-based model exceeds 13.2% despite Table 2 reporting higher results for Claude-Opus-4.6 and Gemini-3.1-Pro. These discrepancies undermine confidence in the reported experimental accounting.
  • The external validity and statistical rigor remain limited. Although the five applications are more challenging than typical web benchmarks, they are still deterministic, self-hosted approximations of professional software and cover a relatively small set of domains. The main results also provide limited uncertainty reporting or significance analysis despite substantial variance across tasks and runs. As a result, it is not always clear how broadly the measured capability gaps will generalize to real professional deployments.
Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer rQXj

We sincerely appreciate your time and effort in reviewing our manuscript. Below are our responses to your valuable comments.


Clarification of Inconsistent Results (W1)

We sincerely apologise for these inconsistencies and thank the reviewer for bringing them to our attention. We would first like to emphasise that all main experimental results (Tables 2–4 and Figures 3–5) were obtained using the full 135-task benchmark, with three independent runs per agent. The reported inconsistencies arose from outdated descriptions and appendix analyses based on an earlier 100-task version, which we clarify below.

Specifically, the main evaluation includes 14 agents, while the earlier appendix analysis included an additional agent (OpenAI o3-pro), which was excluded from the final evaluation due to its limited performance relative to its high evaluation cost. The human evaluation already covered all 135 tasks (405 completions in total), and only the reported count was incorrect. The appendix trajectory analysis has been updated to the full 135-task benchmark with the 14 agents, yielding 4,698 trajectories (4,266 failures). Additionally, the statement that no API-based model exceeds 13.2% was outdated; the correct highest success rate is 19.5% (Gemini-3.1-Pro), as reported in Table 2. We have corrected these inconsistencies in the revised manuscript. For clarity, we summarise the evaluation settings for each experiment below:

ResultTasksAgentsRunsTrajectories
Table 2, Fig. 3–5, Table 101351434,698 (10 vision agents × 405 + 4 text-only agents × 162)
Tables 3–4 (scale / reasoning ablations)1353SR/PR - 135, tokens/steps - 45
Table 5 (judge agreement)5 trajectories per application3 judge runs25
Appendix D.2–D.31351434,698
Human (Table 2)13515 annotators1405

External Validity and Statistical Robustness (W2)

We thank the reviewer for these important concerns. We clarify our benchmark design choices and provide additional uncertainty analyses below.

External validity. Our self-hosted applications are a deliberate design choice to evaluate agent generalisation to unfamiliar interfaces, rather than performance on potentially familiar professional software. Importantly, these applications preserve the core functionalities of their real-world counterparts, including video editing, workflow automation, 3D modelling, flight analysis, and circuit design. Circuit Designer is further adapted from the open-source CircuitJS1 simulator. Moreover, deterministic environments ensure reproducible and consistent comparisons across agents. While five applications cannot cover all professional domains, they span our three targeted capability families, and our extensible framework supports additional applications and tasks (§2.3). We are also developing GauntletBench v2 with more challenging and realistic tasks, with examples provided in the new Appendix K.

Statistical robustness. All results are averaged over three independent runs. Beyond the standard deviations already reported for ablation studies (Tables 3–4, 6–9), we additionally provide per-agent and per-application standard deviations for both success rate and progress rate in the new Appendix J (Table 14). The strongest agent, Claude-Opus-4.6 CU, achieves 28.2 ± 1.2% success (26.7–29.6% across runs), compared with 80.1% for humans. This substantial gap remains consistent across runs, supporting the robustness of our main conclusion. We also acknowledge that differences between the strongest agents, such as Claude-Opus-4.6 CU (28.2 ± 1.2%) and Gemini Enterprise (26.2 ± 3.3%), are small relative to run-to-run variability and should not be interpreted as statistically significant.


Thank you once again for your time and thoughtful review. We will incorporate your feedback and improve our work in future versions.

Dear Reviewer rQXj,

Thanks a lot for your valuable efforts in reviewing our manuscript. Your insightful comments have greatly helped us improve the clarity and rigour of our paper, including explaining the inconsistent results and the design of our benchmark.

Just a kind reminder that the discussion stage is closing soon. If there are any unclear explanations, we are more than willing to provide further clarification.

Best regards,
Authors

Thank you for your comments. I have raised my score.

Thank you for your response. We are pleased that our provided experiments and explanations have addressed your concerns. We sincerely appreciate your encouraging feedback and will incorporate all these updates into the next version of the paper.

Best regards, The Authors

Reviewer_15tiOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GauntletBench, a benchmark for testing how well computer-use agents handle visually demanding tasks in professional software. It spans five web-based applications (video editing, workflow building, 3D modeling, flight analysis, and circuit design) and includes 135 tasks that probe temporal perception, graphical understanding, and 3D reasoning.

  • To support this, the authors built controlled environments along with automated evaluators tailored to each application.
  • The evaluation is fairly broad, covering open-source models, proprietary MLLMs, and dedicated computer-use agents.
  • The results are striking: the best agent succeeds on only 28.2% of tasks, compared to 80.1% for human participants.
  • Overall, the findings make a convincing case that current agents still have a long way to go on visually grounded, multi-step interactions.

<<< UPDATES >>>

i overall think this professional graphical application as an under-explored area in CUA, maybe OSWorld Science is similar, but apparently this benchmark has more focus on knowledge work than science. i appreciate the engineering efforts put into this work, building 135 tasks with automated graders is hard and good contributions to model creators and CUA researchers. my original concern is what to interpret from the low performance -- does that mean models have bad temporal perception or graphical understanding or models fail to engage in long-horizon knowledge work.

after the rebuttal, i think most of my concerns are well addressed. i still think difficulty levels may correlate with interaction complexity or simply task length? there is no controled experiment for this. i maintain my positive ratings and raise the confidence.

Critical strengths
  1. Meaningful expansion of agent evaluation. The benchmark covers professional graphical applications that are relatively underrepresented in existing computer-use benchmarks. Tasks involving timelines, node graphs, circuit layouts, and 3D scenes introduce interaction and perception demands that are difficult to capture with conventional web navigation benchmarks. This makes the benchmark a potentially useful addition to the evaluation landscape.

  2. Substantial environment and evaluation engineering. Building five interactive applications, 135 tasks, and application-specific objective evaluators is a meaningful contribution. I appreciate the emphasis on reproducible environments, objective success criteria, and compatibility with different agent frameworks. The broad evaluation across model families, together with difficulty and progress analyses, provides useful empirical evidence about current agent limitations.

Critical weaknesses
  1. The evaluation does not sufficiently disentangle the targeted capabilities from general agent execution failures. The benchmark is motivated by temporal perception, graphical understanding, and 3D reasoning, but low task success can also result from UI grounding, unfamiliarity with application controls, action formatting, or long-horizon execution failures. The authors' own error analysis identifies protocol violations, premature stopping, and incorrect UI actions. Without more targeted controls, it is difficult to determine whether the reported performance gaps specifically reflect the three proposed capabilities. I would particularly like to see controlled comparisons that separate perception and reasoning from interface operation.

  2. Important experimental and reporting inconsistencies weaken confidence in the quantitative conclusions. The paper describes 135 tasks and reports averages over 27 tasks per application, but other sections still refer to 20 tasks per application. Appendix A reports 300 human evaluations, while the main results present human performance alongside agent results across 135 tasks. Appendix D also refers to 4,500 trajectories computed using 20 tasks per application. These differences need to be reconciled. Additionally, the human participants had prior experience with the relevant software categories and received demonstration videos, complicating the interpretation of the human-agent gap as a direct measure of generalization.

Questions for authors
  • I had trouble working out exactly how many tasks each agent and the human baseline were evaluated on. The main text describes a 135-task benchmark, Appendix A mentions 300 human evaluations, and Appendix D refers to 4,500 trajectories based on 20 tasks per application. Could the authors explain how these numbers fit together, and confirm whether all reported scores come from the same task set?
  • The paper attributes the low success rates mainly to weaknesses in visual and spatial reasoning, but it’s not entirely clear these are the deciding factors rather than more general computer-use or execution errors. Additional experiments would help separate the two, for example runs where the relevant visual information is given to the agent directly, or where the interaction requirements are simplified.
  • The human participants watched demonstration videos and already had experience with their assigned application categories. Did the agents receive any comparable onboarding or guidance? It would also help to know how much the human-agent gap depends on these differences, and on the step and time budgets each side was given.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 15ti (Part 1/2)

We sincerely appreciate your time and effort in reviewing our manuscript. Below are our responses to your valuable comments.


Disentangling the Targeted Capabilities from General Failures (W1 / Q2)

We thank the reviewer for this important observation. We agree that end-to-end task success reflects not only perception and reasoning capabilities but also the reliability of UI interaction. As UI interaction is inherently intertwined with perception and reasoning in agentic workflows, we conduct three complementary analyses to better disentangle and quantify the underlying causes of agent failures.

Impact of execution frameworks. We compare the same underlying models using the REAL-based harness and their native computer-use frameworks (Table 2). GPT-5.4 improves from 3.7% to 6.2%, while Claude-Opus-4.6 improves from 17.5% to 28.2%. Despite these improvements, agents remain far below human performance (80.1%), suggesting persistent limitations in their underlying perception and reasoning capabilities.

Impact of task difficulty. As part of our benchmark design (§2.3), we categorise tasks into three difficulty levels, with increasing demands on our targeted capabilities: temporal perception, graphical understanding, and 3D reasoning. Within each application, tasks share the same interface, yet Claude-Opus-4.6 CU drops from 50.4% (easy) to 23.0% (medium) and 11.1% (hard), with a similar trend for Gemini Enterprise (45.9% → 19.3% → 13.3%) (Fig. 3). These declines suggest that agents face increasing challenges in the perception and reasoning capabilities targeted by our benchmark, beyond basic UI interaction.

Fine-grained failure analysis. To directly identify the sources of agent failures, we analyse 500 failed trajectories from the four strongest agents across five applications, categorising errors into perception, reasoning, interface operation, termination, and protocol violations. As shown in the table below, while interface operation accounts for 57.2% of failures, perception (9.0%) and reasoning (25.8%) together contribute 34.8%. Notably, these two categories account for 50% of failures in both 3D Modeller and Flight Analyser. These findings confirm that, beyond general UI execution challenges, our benchmark also captures substantial limitations in the targeted perception and reasoning capabilities. We have incorporated this fine-grained failure analysis into Appendix D.5 of our revised manuscript.

Error TypePerceptionReasoningInterface operationTerminationProtocol
All (n = 500)9.025.857.27.80.2
3D Modeller2.048.049.01.00.0
Flight Analyser22.028.026.023.01.0
Video Editor9.018.070.03.00.0
Circuit Designer7.024.062.07.00.0
Workflow Builder5.011.079.05.00.0
AuthorsOpenReview ↗

Response to Reviewer 15ti (Part 2/2)

Clarification of Inconsistent Results (W2 / Q1)

We sincerely apologise for these inconsistencies and thank the reviewer for bringing them to our attention. We would first like to emphasise that all main experimental results (Tables 2–4 and Figures 3–5) were obtained using the full 135-task benchmark, with three independent runs per agent. The reported inconsistencies arose from outdated descriptions and appendix analyses based on an earlier 100-task version, which we clarify below.

Specifically, the human evaluation already covered all 135 tasks (405 completions in total), and only the reported count was incorrect. The appendix trajectory analysis was updated from the earlier 100-task version to the full 135-task benchmark, yielding 4,698 trajectories (4,266 failures). We have corrected these inconsistencies in the revised manuscript. For clarity, we summarise the evaluation settings for each experiment below:

ResultTasksAgentsRunsTrajectories
Table 2, Fig. 3–5, Table 101351434,698 (10 vision agents × 405 + 4 text-only agents × 162)
Tables 3–4 (scale / reasoning ablations)1353SR/PR - 135, tokens/steps - 45
Table 5 (judge agreement)5 trajectories per application3 judge runs25
Appendix D.2–D.31351434,698
Human (Table 2)13515 annotators1405

Onboarding and Interaction Budgets (W2 / Q3)

Comparable experience and onboarding. Our human participants were not domain experts (e.g., professional video editors, circuit engineers, or air-traffic analysts) but had computer science backgrounds and general familiarity with the relevant software categories. Importantly, neither humans nor agents had prior experience with our benchmark applications. Both received comparable task-agnostic guidance: agents were provided with detailed application descriptions and interaction instructions (Appendix F.2), while humans received equivalent information through written instructions and short demonstration videos illustrating basic UI operations. Both also received identical task instructions without benchmark-specific hints. Thus, the evaluation provides a reasonably fair comparison of human and agent generalisation to unfamiliar applications.

Interaction budgets. Humans were given a 40-minute limit per task without retries, while agents were allowed up to 100 interaction steps. No human attempt reached the 40-minute limit, and the median human completion time was 115 seconds. Both budgets were designed to be sufficient for completing the benchmark tasks under efficient execution. To further assess whether these limits contribute to the human–agent gap, we analyse agent trajectories that exhaust their step budgets. These failures achieve an average progress rate of 2.04, compared with 1.84 for self-terminated failures and 4.56 for successful runs. Notably, for the strongest agent (Claude-Opus-4.6 CU), budget-exhausted failures show substantially lower progress than self-terminated failures (2.51 vs. 3.69). These results suggest that agents reaching their step limits are generally struggling to complete tasks rather than being close to success, making interaction budgets unlikely to fully explain the human–agent performance gap.


Thank you once again for your time and thoughtful review. We will incorporate your feedback and improve our work in future versions.

Dear Reviewer 15ti,

Thanks a lot for your valuable efforts in reviewing our manuscript. Your insightful comments have greatly helped us improve the clarity and rigour of our paper, including clarifying the disentanglement of targeted capabilities, explaining inconsistent results, and providing additional details on the experimental budget.

Just a kind reminder that the discussion stage is closing soon. If there are any unclear explanations, we are more than willing to provide further clarification.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.