- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Agentic-WorkBench, a benchmark of 115 tasks across 11 work domains for evaluating multimodal agents in stateful workflows. It emphasizes visual grounding, sensemaking during execution, and realization of the final outcome. Experiments compare nine model configurations, examine harness effects, and analyze performance across visual demands and work contexts. A visual-access ablation and human validation of task graders provide additional evaluation.
Critical strengths
-
The benchmark addresses a relevant evaluation problem: using visual information throughout multi-step workflows involving tools, services, and final artifacts. Its coverage of different work domains makes it potentially useful for evaluating practical agents.
-
The evaluation goes beyond a single leaderboard. Repeated trials, harness comparisons, a visual-access ablation, and human validation of graders provide useful checks on the reported results.
Critical weaknesses
-
The distinction from existing benchmarks and the resulting insights are limited. The paper identifies cross-stage visual dependency as its main distinction, but Table 1 and the discussion do not sufficiently establish what important capabilities this benchmark evaluates beyond existing multimodal, long-horizon workflow benchmarks. The analysis is also mostly descriptive. Visual-function labels overlap across tasks, and performance differences may reflect general task difficulty, domain, or tool requirements. Therefore, low success on tasks labeled as requiring state tracking does not establish that state tracking itself is the bottleneck. The visual-access ablation supports the value of images, but does not isolate cross-stage visual dependency.
-
Task construction and grading are not documented in enough detail. Section 3.2 outlines the construction process, but does not clearly explain how candidate tasks were sourced and authored, how expert review was organized, or which filtering decisions reduced 245 candidates to 115 tasks. Appendix E also leaves the visual judge, scoring prompts, combination of grading signals, and binary success thresholds insufficiently specified. The human validation is helpful, but does not replace these implementation details. These omissions make it difficult to assess reproducibility and the consistency of evaluation across tasks.
-
Several comparisons are based on small task subsets. The benchmark divides 115 tasks across 11 domains, seven objectives, and multiple visual functions. Some domains contain only five to nine tasks, so a few outcomes can substantially change the reported success rates. Without uncertainty estimates for these subgroup comparisons, it is difficult to judge how robust the domain-level findings and model rankings are.
-
The harness analysis does not isolate the reasons for improvement. Section 4.3 attributes gains to tool routing, guided visual inspection, and session organization, but these components change together. Representative traces provide examples, but do not establish which mechanisms explain the gains. This limits the actionable insight beyond showing that performance depends on the execution harness.
Questions for authors
-
Could you clarify the benchmark’s concrete advantages over the closest existing work? For example, compared with Claw-Eval and WildClawBench, what visual workflow dependencies are missing from their tasks but systematically covered here? Compared with VisualWebArena and OSWorld 2.0, how does the proposed observation–action–verification loop differ from visually guided interaction and feedback? One or two side-by-side task examples would help establish the additional evaluation value beyond differences in tools or work domains.
-
How do you distinguish failures in specific visual functions from general task difficulty? For example, are state-tracking tasks also longer, more tool-intensive, or more concentrated in difficult domains? Could matched task comparisons or systematic failure annotations support the claimed bottlenecks?
-
Could you provide more construction and grading details—for example, whether tasks were adapted from real workflows or written from scratch, why candidate tasks were excluded, and how visual-judge scores determine binary success?
-
How stable are the subgroup findings and harness improvements? For example, do domain-level differences remain under task resampling, and which harness components account for the reported gains?
Thank you to the authors for the detailed rebuttal and for addressing my concerns regarding the evaluation methodology.
The additional explanations of the task-specific scoring rubrics, judge model configuration, evaluation prompts, and score aggregation procedures substantially improve the clarity and reproducibility of the benchmark. In particular, the two worked examples and the clarification that VLM-based judging accounts for only 28% of the total nominal rubric weight help address my concerns about grading transparency. The reported human evaluation results also provide useful evidence supporting the reliability of the evaluation procedure.
I appreciate the authors' efforts in responding to the review and strengthening the evaluation documentation. The rebuttal has satisfactorily addressed my primary concerns, and I believe the benchmark offers a valuable contribution to evaluating agentic systems on complex, real-world workflows.
I am therefore updating my recommendation from 2 (Weak Rejection) to 3 (Weak Acceptance).