acceptodds
Under review as a conference paper at ICLR 2027

Agentic-WorkBench: Evaluating Multimodal Agents on Complex Visual Workflows

Abstract

Multimodal work often requires agents to acquire visual evidence, use it to guide successive actions, and verify the resulting artifacts or service states. Existing work-agent evaluations leave open how reliably visual information can support this full workflow. We introduce Agentic-WorkBench, a benchmark of 115 tasks across eleven work domains in controlled, stateful environments. The benchmark targets workflow-level visual intelligence, organized into visual grounding, sensemaking, and realization, with observations that may become available only during execution. Across nine model configurations, the strongest achieves 64.0% graded completion, solves 41.7% of tasks within three trials, and succeeds in all three on only 22.6%. The lowest success rates concentrate on visual sensemaking demands, and success varies substantially across work domains and objectives. Harness effects vary across models, and greater resource use does not consistently accompany higher success. These results identify a gap between partial progress and repeatable completion, motivating agents that maintain visual state and use feedback throughout practical work.

Primary area:
datasets and benchmarks
Keywords:
Multimodal-LLMs, Agent, Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_u2ziOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Agentic-WorkBench, a benchmark of 115 tasks across 11 work domains for evaluating multimodal agents in stateful workflows. It emphasizes visual grounding, sensemaking during execution, and realization of the final outcome. Experiments compare nine model configurations, examine harness effects, and analyze performance across visual demands and work contexts. A visual-access ablation and human validation of task graders provide additional evaluation.

Critical strengths
  1. The benchmark addresses a relevant evaluation problem: using visual information throughout multi-step workflows involving tools, services, and final artifacts. Its coverage of different work domains makes it potentially useful for evaluating practical agents.

  2. The evaluation goes beyond a single leaderboard. Repeated trials, harness comparisons, a visual-access ablation, and human validation of graders provide useful checks on the reported results.

Critical weaknesses
  1. The distinction from existing benchmarks and the resulting insights are limited. The paper identifies cross-stage visual dependency as its main distinction, but Table 1 and the discussion do not sufficiently establish what important capabilities this benchmark evaluates beyond existing multimodal, long-horizon workflow benchmarks. The analysis is also mostly descriptive. Visual-function labels overlap across tasks, and performance differences may reflect general task difficulty, domain, or tool requirements. Therefore, low success on tasks labeled as requiring state tracking does not establish that state tracking itself is the bottleneck. The visual-access ablation supports the value of images, but does not isolate cross-stage visual dependency.

  2. Task construction and grading are not documented in enough detail. Section 3.2 outlines the construction process, but does not clearly explain how candidate tasks were sourced and authored, how expert review was organized, or which filtering decisions reduced 245 candidates to 115 tasks. Appendix E also leaves the visual judge, scoring prompts, combination of grading signals, and binary success thresholds insufficiently specified. The human validation is helpful, but does not replace these implementation details. These omissions make it difficult to assess reproducibility and the consistency of evaluation across tasks.

  3. Several comparisons are based on small task subsets. The benchmark divides 115 tasks across 11 domains, seven objectives, and multiple visual functions. Some domains contain only five to nine tasks, so a few outcomes can substantially change the reported success rates. Without uncertainty estimates for these subgroup comparisons, it is difficult to judge how robust the domain-level findings and model rankings are.

  4. The harness analysis does not isolate the reasons for improvement. Section 4.3 attributes gains to tool routing, guided visual inspection, and session organization, but these components change together. Representative traces provide examples, but do not establish which mechanisms explain the gains. This limits the actionable insight beyond showing that performance depends on the execution harness.

Questions for authors
  1. Could you clarify the benchmark’s concrete advantages over the closest existing work? For example, compared with Claw-Eval and WildClawBench, what visual workflow dependencies are missing from their tasks but systematically covered here? Compared with VisualWebArena and OSWorld 2.0, how does the proposed observation–action–verification loop differ from visually guided interaction and feedback? One or two side-by-side task examples would help establish the additional evaluation value beyond differences in tools or work domains.

  2. How do you distinguish failures in specific visual functions from general task difficulty? For example, are state-tracking tasks also longer, more tool-intensive, or more concentrated in difficult domains? Could matched task comparisons or systematic failure annotations support the claimed bottlenecks?

  3. Could you provide more construction and grading details—for example, whether tasks were adapted from real workflows or written from scratch, why candidate tasks were excluded, and how visual-judge scores determine binary success?

  4. How stable are the subgroup findings and harness improvements? For example, do domain-level differences remain under task resampling, and which harness components account for the reported gains?

Reviewer_N14tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Agentic-WorkBench, a benchmark of 115 tasks spanning eleven work domains, seven work objectives, and 101 tools in controlled, stateful service environments. It frames workflow-level visual intelligence as the connected use of visual grounding, sensemaking during execution, and realization of a final artifact or service state. Tasks are curated from 245 candidates and annotated with eleven overlapping visual functions and an evaluation-independent Easy/Hard partition. Nine canonical model configurations are evaluated using three trials per task, with partial completion, first-trial success, at-least-once success, and all-three-trial success reported separately. GPT-5.6 Sol obtains 64.0% completion, 41.7% Pass@3, and 22.6% Pass^3. Additional experiments examine alternative harnesses, larger turn budgets, work domains, visual demands, and estimated token cost. The appendices provide a Kimi visual-access ablation, a human feasibility check, and human validation of the graders. The results support the benchmark’s difficulty and the value of visual access for a subset of tasks; the interpretation of particular visual functions as underlying failure mechanisms remains less directly established.

Critical strengths

The benchmark targets a consequential integration problem and demonstrates measurable visual dependence. The tasks combine evidence acquisition, service interaction, and production or verification of an outcome, which makes the evaluation useful for studying whether perception can support an entire work process. The 115 tasks cover eleven domains and seven objectives rather than a single interface.

Critical weaknesses

The scoring implementation is underspecified. Appendix E states that completion scores combine task-specific rubrics with a visual LLM judge, but the paper does not identify the judge model, provide its evaluation prompt, or explain how the rubric and visual scores are weighted and normalized. Without these details, it is difficult to reproduce the results or assess how different grading choices might affect the reported scores.

Questions for authors
  1. Could you provide the complete grading configuration: judge model/version and prompt, representative task rubrics, score-combination rules, binary success thresholds, and whether judging is repeated or deterministic?

  2. Could you provide a fully worked task example and its grading specification, showing the objective checks, visual-judge model and prompt, score-combination and normalization rules, and exact conditions for binary success?

1 comment

Thank you to the authors for the detailed rebuttal and for addressing my concerns regarding the evaluation methodology.

The additional explanations of the task-specific scoring rubrics, judge model configuration, evaluation prompts, and score aggregation procedures substantially improve the clarity and reproducibility of the benchmark. In particular, the two worked examples and the clarification that VLM-based judging accounts for only 28% of the total nominal rubric weight help address my concerns about grading transparency. The reported human evaluation results also provide useful evidence supporting the reliability of the evaluation procedure.

I appreciate the authors' efforts in responding to the review and strengthening the evaluation documentation. The rebuttal has satisfactorily addressed my primary concerns, and I believe the benchmark offers a valuable contribution to evaluating agentic systems on complex, real-world workflows.

I am therefore updating my recommendation from 2 (Weak Rejection) to 3 (Weak Acceptance).

Reviewer_uEJzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Agentic-WorkBench, 115 tasks across 11 domains in stateful simulated services. It targets visual grounding, sensemaking, and realization within full workflows. Nine models are evaluated over three trials. The best reaches 64.0% Completion but only 22.6% Pass^3, with failures concentrated in visual state tracking and feedback adaptation.

Critical strengths
  • It addresses a real gap: visual evidence must be acquired mid-execution and constrain verifiable outcomes, unlike GUI or text-centric work benchmarks.
  • The reporting is reliability-aware (Pass@3 vs. Pass^3, harness and cost analyses), and the graders are validated against humans (κ = 0.71).
Critical weaknesses
  • The central claim of full-workflow visual dependency is weakly validated. This dependency is the main differentiator from prior work (Table 1, last column), yet the only direct evidence is a single-trial text-only ablation on one model.
  • 19 of Kimi-K3’s 32 successes (about 60%) survive without images.
  • Nothing is shown for the 83 tasks that fail in both conditions, or for the individual stages.
  • Visual-function labels are assigned from “metadata, prompts, tags, and tools” (App. C). This conflicts with the expert annotation described in Sec. 3.2, and no inter-annotator agreement is reported.
  • “Visual content understanding” is central in 114 of 115 tasks, so it does not discriminate between tasks.
  • Limited statistical resolution and a model–harness confound. Table 2 and Figs. 4–6 report no confidence intervals, and many highlighted differences amount to one or two tasks:
  • Luna 23.3% vs. Sol 20.0% on state tracking is 7 vs. 6 of 30 tasks.
  • Harness effects of ±0.9 pts correspond to a single task.
  • Domain slices contain only 5–17 tasks.

In addition, each model runs in its own “canonical” configuration, but Sec. 4.3 shows that the harness alone can shift Pass@3 by up to 20.9 pts. The main leaderboard therefore mixes model and harness effects.

Questions for authors
  • Can the text-only ablation be extended to more models and at least 3 trials, with a per-task report of whether images are necessary? A stage-wise ablation, such as withholding only images acquired during execution, would directly support the three-stage framing.
  • Which model is the visual LLM judge for Completion? Has self-preference bias toward same-family models been checked?
  • How exactly were the visual-function labels assigned (expert or heuristic), and what is the inter-annotator agreement?
  • Which harness does each canonical (Native/Batch) configuration use? Please add task-bootstrap CIs to Table 2 and to the per-slice analyses.
  • Will the environments, tasks, graders, and visual materials be released, and under what licenses (e.g., the museum artworks)? For the human feasibility check, what are the binary pass rate and the inter-rater agreement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.