- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces GauntletBench, a benchmark of 135 tasks across five web applications: video editing, workflow construction, 3D modelling, flight analysis, and circuit design. It targets temporal perception, graphical understanding, and 3D reasoning through interactive tasks. The benchmark combines application-specific objective evaluators, MLLM-based progress scoring, and token/step measurements. The reported evaluation covers 14 agent settings and human participants, with a highest agent success rate of 28.2% versus 80.1% for humans. The paper also presents model-scale and reasoning ablations and failure analyses.
Critical strengths
-
Useful task coverage. The benchmark brings together visually intensive, stateful interfaces involving timelines, graphs, spatial layouts, and circuits. This task distribution can broaden evaluation of GUI agents beyond common navigation and form-filling workflows.
-
Inspectable task and evaluation design. The appendix documents task instructions and reference states extensively. Application-specific checks, including graph matching and circuit functional equivalence, provide a useful foundation for automated evaluation.
Critical weaknesses
-
Unresolved benchmark and evaluation validity. The main results and failure analysis use different task populations (27 versus 20 tasks per application), and Table 6’s pooled token mean is inconsistent with its equally sized groups. The stated evaluators also raise validity concerns: 3D task 26 permits multiple layouts despite reference-coordinate matching, while Circuit task 25 provides its answer in the prompt without described construction verification. These issues undermine confidence in the scores and require a task/evaluator audit.
-
Limited diagnostic insight. The analysis does not convincingly separate the claimed temporal, graphical, and 3D capability deficits from grounding, interface complexity, or interaction length. Higher partial completion does not establish that stronger agents are limited by execution rather than understanding (§3.4), and cross-system comparisons do not establish the benefit of finer-grained actions (§3.2). The paper provides useful descriptive observations, but its explanations of failure remain insufficiently validated.
Questions for authors
-
Experimental clarification: Which task versions, agent configurations, and valid run counts underlie the main results, ablations, human evaluation, and failure analysis? Please reconcile their populations and statistics using a consistent run manifest.
-
Evaluation validation: How does the implementation accept alternative valid layouts in 3D task 26 and verify circuit construction in measurement tasks such as Circuit task 25? What does an independent audit find about incorrect evaluator decisions, and how do corrections affect the reported scores?
-
Specification clarification: Is the Video Editor timing tolerance 100 ms (§2.4) or 1 s (Appendix E)? For Video task 26, should clips be divided into quarters or thirds? Please clarify which specifications were used in evaluation.
6 comments
Response to Reviewer P6rv (Part 2/2)
Failure Attribution and Interpretation (W2)
We thank the reviewer for these observations. We agree that the original analyses provide useful descriptive insights but do not fully establish the underlying causes of agent failures. We have strengthened the failure analysis and refined the corresponding interpretations.
Execution vs. understanding (§3.4). We agree that partial progress alone cannot distinguish execution failures from perception and reasoning limitations. To address this, we conduct a fine-grained analysis of failed trajectories, distinguishing perception, reasoning, interface operation, termination, and protocol errors. As shown in the table below, among 500 failed trajectories from the four strongest agents, interface operation accounts for 57.2% of failures, while perception (9.0%) and reasoning (25.8%) together account for 34.8%. These findings provide more direct evidence that both execution and the targeted perception and reasoning capabilities contribute to agent failures. We have incorporated this fine-grained failure analysis into Appendix D.5 of our revised manuscript.
| Error Type | Perception | Reasoning | Interface operation | Termination | Protocol |
|---|---|---|---|---|---|
| All (n = 500) | 9.0 | 25.8 | 57.2 | 7.8 | 0.2 |
| 3D Modeller | 2.0 | 48.0 | 49.0 | 1.0 | 0.0 |
| Flight Analyser | 22.0 | 28.0 | 26.0 | 23.0 | 1.0 |
| Video Editor | 9.0 | 18.0 | 70.0 | 3.0 | 0.0 |
| Circuit Designer | 7.0 | 24.0 | 62.0 | 7.0 | 0.0 |
| Workflow Builder | 5.0 | 11.0 | 79.0 | 5.0 | 0.0 |
Finer-grained actions (§3.2). We acknowledge that comparing Claude-Opus-4.6 CU and Gemini Enterprise cannot establish the causal benefit of finer-grained actions, as the systems differ in both their underlying models and execution frameworks. We have therefore revised §3.2 to retain the descriptive observation while removing the unsupported causal interpretation.
Clarification of Video Editor (Q3)
We thank the reviewer for identifying these inconsistencies. We clarify that the evaluation consistently followed the correct specifications, and the discrepancies were limited to textual errors. These corrections have been incorporated into the revised manuscript and do not affect the reported evaluation results.
Timing tolerance. The evaluator consistently uses a tolerance of ±1 s for start times, durations, and trim offsets, as specified in Appendix E. The “100 ms” stated in §2.4 and Figure 2 was outdated and has been corrected.
Video Editor Task 26. The correct specification is to divide clips into thirds, consistent with the task steps, segment labels (T1–T3, W1–W3), and ground truth. The reference to “quarters” in the task goal was a typo. We have also corrected an outdated clip name in Step 3.
Thank you once again for your time and thoughtful review. We will incorporate your feedback and improve our work in future versions.
Thanks for the rebuttal and most of my concerns are addressed.
-
For Circuit Designer task 25, why does Gemini’s score increase after adding construction and simulation checks?
-
The claim in §3.3 that additional reasoning generally improves performance should be narrowed. In Table 4, only Gemini improves while both GPT and Claude decrease.
Response to Reviewer P6rv (Part 1/2)
We sincerely appreciate your time and effort in reviewing our manuscript. Below are our responses to your valuable comments.
We sincerely apologise for these inconsistencies and thank the reviewer for bringing them to our attention. We would first like to emphasise that all main experimental results (Tables 2–4 and Figures 3–5) were obtained using the full 135-task benchmark, with three independent runs per agent. The reported inconsistencies arose from outdated descriptions and appendix analyses based on an earlier 100-task version, which we clarify below.
Specifically, the human evaluation already covered all 135 tasks (405 completions in total), and only the reported count was incorrect. The failure analysis (Section 3.4 and Appendix D.2–D.3) has been updated to the full 135-task benchmark with the 14 agents in Table 2, yielding 4,698 trajectories (4,266 failures). The main findings remain unchanged, with 97.0% of failures falling within the PR = 1–3 range.
Regarding the pooled statistics in Tables 6–9, some overall values were carried over from the earlier benchmark version with unequal difficulty-group sizes. We have recomputed all pooled means and standard deviations using the updated evaluation data across the three equally sized difficulty groups (45 tasks each). The corrected values are consistent across tables, and the main conclusions remain unchanged. For clarity, we summarise the evaluation settings for each experiment below:
We thank the reviewer for identifying these issues. We have audited the affected tasks and revised their evaluation criteria to ensure that valid solutions are correctly assessed.
3D Modeller Task 26. We acknowledge that the original evaluator relied on reference-coordinate matching despite the task allowing multiple valid layouts. Our audit of all 30 runs identified 7 valid solutions incorrectly rejected due to differences in leg placement. We have replaced the position matching with geometric contact constraints, while retaining exact checks for other properties (Appendix E). This correction increases the success rates of Gemini-3.1-Pro (30.9% → 34.6%), Gemini Enterprise (59.3% → 63.0%), and Claude-Opus-4.6 (14.8% → 16.0%).
Circuit Designer Task 25. We acknowledge that the example output inadvertently revealed the expected answer (62.5 Hz), which has been replaced with a neutral example (Appendix G.5). Importantly, no agent successfully completed this task in any run when the constructed circuit was verified. We also strengthened the evaluation of analogue measurement tasks to require that simulating the agent’s constructed circuit reproduces the reported value, rather than checking the reported value alone (Appendix E). This updates the success rates of Gemini-3.1-Pro (22.2% → 27.2%), GPT-5.4 CU (7.4% → 1.2%), and Gemini Enterprise (6.2% → 4.9%).
Overall, these corrections improve the validity of our task evaluation without changing the relative ranking of agents or the main conclusions.