acceptodds
Withdrawn from ICLR 2027

AgenticVBench-Omni: Evaluating Omnimodal Agents on Economically Valuable Work

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce AgenticVBench-Omni, a benchmark that tests whether omnimodal agents can turn video, audio, images, and documents into economically valuable work. Its 45 tasks span eight families and work activities linked to 10 occupations, requiring precise records, quantitative annotations, editable assets, or completed environment states. Designed around complex professional workflows, the tasks require sustained evidence gathering, cross-modal reasoning, and tool use while preserving temporal, spatial, and identity relationships. Each task specifies an output contract and uses a fully deterministic verifier to compute scores, without rubric-based grading by a model. Across five frontier agents evaluated three times per task, the strongest agent achieves a mean score of just 0.283, and 60% of attempts score below 0.10. Verifier outputs and execution traces show that agents recover partial information but struggle with precise localization, complete coverage, accurate reconstruction, and self-verification of the final deliverable.

Primary area:
datasets and benchmarks
Keywords:
Omnimodal reasoning, multimodal agents, agentic benchmarks, economically valuable work, long-horizon reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_U7j6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a benchmark for the quality of agent deliverables, measuring the agent's ability to complete long-process, high-precision tasks end-to-end, and uses objective standards for scoring.

Critical strengths

The paper moves beyond conventional question-answering benchmarks by requiring agents to produce structured records, quantitative annotations, and editable artifacts from multimodal evidence. The 45 tasks spanning eight families provide a useful setting for evaluating end-to-end task completion rather than isolated perception or reasoning abilities.

Critical weaknesses

The main problem with this paper is that a sufficient logical loop has not yet been formed between the proposal of the benchmark, the construction of the task, and the subsequent experimental verification.

  1. The benchmark fails to provide a reasonable metric for the problems posed. The introduction emphasizes that existing multimodal benchmarks primarily focus on perception and reasoning abilities, failing to adequately assess an agent's capacity for continuous evidence gathering, cross-modal reasoning, and sophisticated tool usage. However, this paper does not adequately demonstrate whether the constructed task can systematically measure these abilities, nor does it prove that these abilities are the primary factors contributing to the task's difficulty.
  2. The experiment failed to systematically explain the source of the task difficulty. The experimental design and related analysis in this paper focus on the agent's performance itself, failing to demonstrate the key aspects of the benchmark's assessment of the agent's existing capabilities, nor does it analyze, based on the benchmark performance, "which aspects of the agent's existing capabilities are insufficient and how this leads to the agent's inadequate delivery capabilities that this paper seeks to investigate."
Questions for authors
  1. Why is the threshold for Ablation set to 0.15, while the target threshold for Development Agent is 0.10? The article uses a Development Agent score below 0.10 as the difficulty calibration target for the task development phase. However, I observed that the threshold for the Ablation part is 0.15. Why does the agent score improve under more lenient conditions after ablating one of the modalities?
  2. Why are the threshold used during review and validation differs significantly from the performance in subsequent tests? The article mentions using a state-of-the-art strong model when employing the Development Agent. What exactly does this mean? My main question is that you limited its score to no more than 0.1 when building the benchmark, yet in subsequent tests, some models still scored significantly higher than 0.1? Were the models in subsequent tests stronger than those used when building the benchmark?
  3. What is the relationship between agent behavior analysis and the core research question of the paper? Figures 6 and 7 analyze behaviors such as Operation Grouping, Saved Scripts, and Sub-agent Delegation. However, these experiments primarily demonstrate the differences in execution strategies among different agents. Could the authors further explain the direct significance of these analyses for validating the multimodal evidence integration and professional work delivery capabilities emphasized by the Benchmark?
1 comment

We thank the reviewer for these questions. They go to the core of the paper: whether the benchmark measures what it says, and why agents fail. We agree the paper did not close this loop, and we try to close it here.

On whether the tasks measure the stated abilities: "This paper does not adequately demonstrate whether the constructed task can systematically measure these abilities, nor does it prove that these abilities are the primary factors contributing to the task's difficulty."

Response: We answer in two steps: which tasks need each ability, and what happens to the score when that ability cannot be used.

AbilityTasks that need itHow we tested itAverage score
Gathering evidence over a long runAll 45One frame only0.014
Combining picture, sound and documents11 need sound with video; 26 have documentsPicture only, or sound only0.084
Using toolsAll 45Frames, no tools0.004
All three availableFull input with tools0.478

When any one of the three is taken away, the score falls from 0.48 to below 0.09. So each ability is needed, and together they account for almost all of the score.

On the source of difficulty: "Which aspects of the agent's existing capabilities are insufficient and how this leads to the agent's inadequate delivery capabilities."

Response: With everything available, the best agents still score only about 0.28. The paper's data show where the score is lost, step by step:

StepWhat the two best agents achieve
Find the things that should be reportedabout 90% found
Give each one the right time or positionabout a third correct
Hand in a finished filea file is written in almost every run

So the weak point of the best agents is precise measurement, not finding and not writing the file. The weaker agents fail earlier: they more often hand in nothing usable or miss most items. The new runs agree: when an agent is asked to answer without the video, it gives the right number of items with the wrong times.

On the thresholds: "Why does the agent score improve under more lenient conditions after ablating one of the modalities?"

Response: It does not, and we are sorry the figure suggested this. Scores with a modality removed are far lower than with the full input (0.084 against 0.478, see the table above). The two numbers in Figure 3 are not pass marks for agents. 0.10 is a recommended target for the full input while building a task; the hard requirement is that the task is long. 0.15 is not a condition for accepting a task; that sentence and figure entry were written in error and are corrected.

On scores above the development target: "Were the models in subsequent tests stronger than those used when building the benchmark?"

Response: Yes. Tasks were built and tested with the agents available at the time, through Claude Code, Codex and Antigravity. The agents in the paper are a newer generation. The target is also a recommendation, not a limit. Even so, 60% of all attempts in the paper still score below 0.10.

On the behaviour analysis: "Could the authors further explain the direct significance of these analyses for validating the multimodal evidence integration and professional work delivery capabilities?"

Response: Each behaviour in Figures 6 and 7 is one of the three abilities in action.

Behaviour in the paperAbility it showsWhat we see
Grouping several operations in one stepGathering and combining evidenceIn the mixing task the agent turns the narration into text, looks at the frames it points to, and measures from them. Without the narration it takes as many steps (331 against 325) but rebuilds none of the edits
Saving and reusing scriptsUsing toolsAgents build their own measuring tools and reuse them across the video
Handing work to sub-agentsGathering evidence over a long runWhen this finishes, Claude Opus 5.5 beats GPT-6-Astra on all 12 such tasks (0.40 against 0.21). When the run ends while waiting for sub-agents, it scores 0.01 against 0.22 (7 tasks)

The last row is the clearest link from behaviour to score, and it explains why the two leading agents end up tied.

We hope these answers address the reviewer's concerns. We will reorganise Section 3 around this chain of reasoning, and we welcome further questions.

Reviewer_Qm7kOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

AgenticVBench-Omni evaluates whether agents can turn multimedia evidence into specified work products. Its 45 tasks span 8 families and require structured records, quantitative annotations, editable assets, or an environment state. Task-specific deterministic verifiers score the outputs. The paper evaluates five model–harness combinations three times per task; the highest mean score is <0.3. Trace analysis examines incomplete delivery, localization errors, and ineffective self-checks. The intended contribution is a benchmark for detailed work across domains, together with empirical evidence about where current agents fail.

Critical strengths
  1. I find the focus on complete work products somewhat motivated. the paper connect media evidence to explicit output requirements. Reconstructing a mixing session or linking runway operations to callsigns requires carrying details from the source into a usable artifact.
  2. The verifier-based case studies are informative. the paper shows why an agent's self-check can succeed while its deliverable is still wrong: reopening a mixing project preserves incorrect settings, and consistent bone lengths do not establish correct joint positions.
Critical weaknesses

  1. The scoring rules are difficult to assess. This is one of my main concern. Sections 2 give example tolerances and describe categories of checks, but do not specify each task's matching rules, component weights, completeness penalties, or conditions that force a zero score.
    1. In Table 1 then averages these heterogeneous scores. while the paper says task packages include verifiers, but the supplied pdf provides no access route to them, and I do not find any missing definitions within the paper. I therefore cannot assess how partial correctness maps to the aggregate score or whether equivalent valid outputs are accepted.
  2. The evidence for professional precision is limited. Also in Sec2, it describes manual evidence checks and documents an independent pass on one sports task. The paper does not summarize their coverage, reviewer expertise, or disagreements across the eight families. A full-reward oracle establishes consistency with the verifier; it does not independently show that an answer is recoverable at the required precision or that a tolerance fits the intended work activity.
  3. The extent of cross-modal dependence remains unclear. fig3 describe no-media, single-frame, and single-modality controls, but give targets rather than achieved scores or coverage. Moreover, a restricted-input target of at most 0.15 does not require performance to fall relative to full input? especially when development-agent scores are themselves targeted below 0.10. These thresholds alone cannot establish modality dependence.
    1. Also, task selection makes the difficulty result hard to contextualize. This work targets development-agent scores below 0.10, while low evaluation scores are a headline result. Constructing a difficult benchmark seems to be reasonable, but the development agents, candidate rejection or revision rates, and overlap with evaluated systems are unspecified.

Also, please check my questions.

Questions for authors
  1. Is there a review artifact with all 45 task specifications, scoring definitions or executable verifiers, and per-task results? Which components can be released for reuse?
  2. What human-validation coverage supports the requested precision across families, and what did independent judgments show before adjudication?
1 comment

We are grateful for the reviewer's detailed and specific comments, which guided most of our new material. We address each point below.

On scoring rules and access: "The supplied pdf provides no access route to them."

Response: We apologise for this. The scoring code for every task is now public, and each task page explains how the work is scored. Please see [Artifact] in the General Response.

On how task scores are combined: "I therefore cannot assess how partial correctness maps to the aggregate score or whether equivalent valid outputs are accepted."

Response: Every task gives a score from 0 to 1: the share of checked items that are right. Table 1 is the plain average of these scores, with every task counting the same. A different answer that is equally correct gets the same score, because the code compares values within a stated margin and not the way they are written.

On what can be reused: "Which components can be released for reuse?"

Response: All 45 tasks, their scoring code and the scores are released. The media are released for 39 tasks; for the other 6 the source licence does not allow us to redistribute them.

On human validation: "The paper does not summarize their coverage, reviewer expertise, or disagreements across the eight families."

Response: Each task was built by a contributor with experience in its field. On average, our contributors have five years of experience. We also counted the checks recorded for each task:

CheckTasks
Answer key checked by a second person10
Tested that the answer can be worked out from the media alone15
Error margin set from measured human error5

Where a second person checked the answer key, the two agreed closely: in one task they agreed on 46 of 46 items; in another, 23 of 24 proposed items were kept.

On cross-modal dependence: "These thresholds alone cannot establish modality dependence."

Response: The reviewer is right, and we apologise: the sentence about 0.15 in Section 2.4 and the entry in Figure 3 were written in error. 0.15 is not a condition for accepting a task. We have corrected both. A threshold cannot prove that a task needs its media, so we now measure it directly; please see [Analysis] in the General Response.

On development agents, rejection and revision: "The development agents, candidate rejection or revision rates, and overlap with evaluated systems are unspecified."

Response: Development agents. While building tasks we tested them with three agents: Claude Code, Codex and Antigravity.

Rejection and revision. Every accepted task was sent back for changes at least once before it was accepted.

Submitted tasksCount
Submitted56
Accepted45
Not accepted11

The review rounds below are for the 22 accepted tasks whose review was tracked round by round.

Rounds of requested changes before acceptanceAccepted tasks
111
2 to 33
4 to 55
6 to 73

Overlap with evaluated agents. This does not bias the results, because the evaluation is not done by an agent. A task's score comes from code that compares the work with a fixed answer key. No agent takes part in scoring.

We hope these responses address the reviewer's concerns, and we are glad to provide more detail.

Reviewer_ErMrOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgenticVBench-Omni, a benchmark that tests whether ominmodal agents can work on economically valuable tasks that involve video, audio, images, and documentation. The dataset includes 45 data points. With this dataset, the paper evaluates GPT-6-Astra, Claude Opus 5.5, Claude Opus 5, Gemini 3.8 Flash, DeepSeek 4.1 Flash and finds that GPT-6-Astra and Claude Opus 5.5 significantly outperforms other models; but all these models struggle on this new benchmark. Verifier outputs and traces show that agents recover partial information but struggle with precise localization, complete coverage, accurate reconstruction, and self-verification of the final deliverable.

Critical strengths
  1. The data points curated by the paper are valuable. They focus on the multi-modality aspect of economically valuable work and are challenging to identify limitations of cutting-edge models and agent harnesses.
  2. I like the evaluation the work did that go beyond task performance and cover aspects like time consumption and working style.
Critical weaknesses
  1. My main concern about is the work is the lack of scientific contribution to the field. While I acknowledge that the data points are very valuable, the methodology for collecting these datasets is similar to prior work like GDPval, APEX, APEX Agents, Agent Last Exam, etc., with a combination of model generation and human review. Besides a new set of task instances that cover new domains, I didn't feel that I learn too much from this work.
  2. The paper selects "dataset and benchmark" as primary arear but the reproductibility seems to be weak. The submission does come with data points for us to review.
  3. The paper's core contribution is that scoring is fully deterministic without rubric-based grading. But for task like operations and scientific observation, I think passing the verifiable checks does not equal to delivering high-quality task outcome. The current evaluation pipeline seems to oversimplify the nuances.
Questions for authors
  1. While I understand collecting this type of datasets is expensive, I think 45 tasks is quite small. Given that the confidence intervals in Table 1 are wide (e.g. [0.218, 0.354]), is this enough to draw reliable conclusions about frontier models? How sensitive are rankings to individual tasks?
2 comments

We sincerely thank the reviewer for recognising the value of the tasks and of the analysis beyond task scores. We address each point below.

On the scientific contribution: "Besides a new set of task instances that cover new domains, I didn't feel that I learn too much from this work."

Response: Our main contribution is that every task is scored in a fully deterministic way. GDPval and similar benchmarks rely on human experts or rubric-based grading to judge the result. We take the expert's opinion and turn it into a verifier that code can run, so the same work always gets the same score. To our knowledge this has not been done for multimodal professional tasks of this size.

The second contribution is the length of the tasks. A single run can take more than 1,000 steps (up to 1,166 tool calls in our runs). Building tasks this long that can still be checked exactly is a step forward for long-horizon evaluation.

On reproducibility: "The reproductibility seems to be weak."

Response: Thank you for raising this. Everything is now available; please see [Artifact] in the General Response.

On deterministic checks: "Passing the verifiable checks does not equal to delivering high-quality task outcome."

Response: We agree that the two are not the same, and our checks are built from expert judgement for this reason. An expert in the field first writes down what good work looks like for the task. We then turn each opinion into a check that code can run. Take making a film trailer as an example. A vague rule would be "the trailer covers all the important plot points". Instead, the expert names them: plot point 1, plot point 2, plot point 3. The check is then simply whether each one is there. This is how every expert opinion becomes a deterministic check. It tells us how completely and how well the task was done, and it gives the same reward for the same work every time. We have added Appendix E to the paper, which describes this conversion step by step, with a worked example from one of our tasks.

On the number of tasks and the wide intervals: "Is this enough to draw reliable conclusions about frontier models? How sensitive are rankings to individual tasks?"

Response: Thank you for this question. We answer its two parts in turn.

Is 45 tasks enough? Enough to separate the agents into three groups, not enough to order agents inside a group. The table compares agents on the same 45 tasks. When the interval does not include 0, the gap is reliable.

ComparisonGap in mean score95% intervalReliable?
GPT-6-Astra vs. Claude Opus 5.5+0.005-0.061 to +0.071No: tied
GPT-6-Astra vs. Claude Opus 5+0.177+0.133 to +0.225Yes
Claude Opus 5.5 vs. Claude Opus 5+0.172+0.106 to +0.243Yes
Claude Opus 5 vs. Gemini 3.8 Flash+0.049+0.006 to +0.104Yes
Gemini 3.8 Flash vs. DeepSeek 4.1 Flash-0.001-0.021 to +0.017No: tied

How sensitive are the rankings to individual tasks? We removed each task in turn and ranked the agents again, and did the same for each task family.

What we removedTimes the three groups changedTimes the top agent changed
One task (45 cases)05
One task family (8 cases)03

So the three groups never change, whichever task or family is removed. Only the order of the two tied leaders can flip. We will state in the paper that the two leaders are tied and report the groups.

How does this compare with other benchmarks? Our tasks are longer than those of the benchmarks the reviewer names, and those benchmarks also report sizeable variation in their scores.

BenchmarkAgent steps per taskTasksTop scoreSpread of scores across tasks (SD)95% interval of the mean
GDPval (independent replication)capped at 250 turns22047.347 points± 6.2 points
APEX-Agentscapped at 250 steps48024.043 points± 3.8 points
Oursno cap; median 259, longest third 6754528.324 points± 6.9 points

All scores are on a 0 to 100 scale. Two things follow. First, our scores are less spread out from task to task than in the other two benchmarks, because each task gives partial credit instead of pass or fail. Second, our interval is about as wide as GDPval's, even though we have fewer tasks. More tasks would narrow it further, and we plan to add them.

The median run in our benchmark is already longer than the step cap of the other two.

We hope these responses address the reviewer's concerns, and we welcome further discussion.

I thank the authors for the detailed response. While I acknowledge the efforts the authors made, I am unsure whether I buy the argument on scientific contribution.

Building long-horizon, checkable tasks is already a problem a lot of work trying to measure (e.g., Terminal Bench, Agent Last Exam) and is the area where a lot of data companies are selling data on. There are still a lot of unsolved problems for agent eval but this specific angle does not to have much scientific contribution to me.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

General Response

We sincerely thank all reviewers for the time and care they put into our paper. Your feedback has made the work clearer and stronger. Below we list what we added in response, followed by a reply to each reviewer.

[Artifact] The benchmark is now open for inspection. In response to Reviewers ErMr and Qm7k, we released an anonymous site, repository and dataset. The site has one page per task with the instruction, the media, the scores and a sample run step by step. The repository has all 45 tasks and their scoring code.

[Analysis] The tasks need their video and audio. In response to Reviewers Qm7k and U7j6. We tested this twice.

Round 1: when each task was submitted. This was required for every task. The contributor had to run an agent on the task with the media removed. If the agent still scored well, the answer was leaking from memory, and the task was sent back.

Round 2: for this response. We sampled three tasks and ran them again with the two best agents. Each agent did every task with the full input, and then with part of the input removed.

Average scoreRound 1 (at submission)Round 2 (new, 3 sampled tasks)
No media at all0.0030.000
Any part of the input removed0.0070.036
Full input0.100.478

Both rounds say the same thing: without the media, agents score close to zero. Round 2 adds the comparison that Round 1 could not give: the same agent, on the same task, scores far higher once it has the full input.

Round 2 is a sample of three tasks because of our compute budget. We will run the remaining tasks and report all of them in one table in the final version.

[Experiments] An agent must choose what to look at. In response to Reviewer U7j6. With tools, an agent looked at 133 megapixels of images and scored 0.82. Given about the same amount (129 megapixels) as small frames taken once per second, with no tools, it scored 0. Evenly spaced small frames miss the few moments and the fine detail that the answer depends on.

[Clarification] The 0.10 and 0.15 numbers in Figure 3. In response to Reviewers Qm7k and U7j6. 0.10 is a recommended target while building a task, not a hard limit. The hard requirement is that the task is long horizon. 0.15 is not a condition for accepting a task either.

We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.