acceptodds
Under review as a conference paper at ICLR 2027

Agentic-MME : What Agentic Capability Really Brings to Multimodal Intelligence?

Abstract

Multimodal Large Language Models (MLLMs) are rapidly evolving from passive observers into active agents. These agents increasingly solve problems through Visual Expansion, actively invoking visual tools to transform images. Furthermore, as real-world tasks often require information beyond visual content, modern systems combine these visual operations with Knowledge Expansion via open-web search. However, existing evaluations fall short in three critical aspects. (i) Lack flexibility and comprehensiveness in tool integration that supports heterogeneous tool interfaces. (ii) Test image tool use and web search separately, leaving their synergy unexplored. (iii) Evaluate primarily by the correctness of the final answer. Consequently, they cannot verify whether the tools were actually invoked, applied correctly, or used efficiently. To answer what agentic capability truly brings to multimodal intelligence, we introduce Agentic-MME, a process-verified benchmark for Multimodal Agentic Capabilities. Agentic-MME contains 418 real-world tasks across 6 domains and 3 difficulty levels designed to evaluate capability synergy, featuring over 2,000 stepwise checkpoints that average more than 10 person-hours of manual annotation per task. Each task is paired with (i) a unified evaluation framework that supports both sandboxed code execution and structured tool APIs, and (ii) a human reference trajectory annotated with stepwise checkpoints along dual-axis: S-axis and V-axis. To enable true process-level verification, we move beyond final answers by auditing fine-grained intermediate states. We also quantify efficiency via an overthinking metric relative to human trajectories. Experimental results show that the best model Gemini3-pro achieves an overall accuracy of 56.3% and this score falls significantly to 23.0% on Level-3 tasks, underscoring the difficulty of real-world multimodal agentic problem solving.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hoUDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agentic-MME introduces a process-level benchmark for evaluating multimodal agents that combine visual manipulation with web retrieval. It includes 418 tasks across three difficulty levels and uses V-axis and S-axis checkpoints to assess visual operations and search behavior. Results show that current models remain well behind human solvers, particularly on Level 3 tasks.

Critical strengths
  1. Relevant research problem. The benchmark examines how multimodal agents combine visual manipulation and web retrieval, an important setting for multi-tool agents.
  2. Evaluation beyond final answers. The V-axis and S-axis checkpoints assess intermediate visual and search behaviors, helping diagnose where agents fail.
  3. Rich process annotations. The benchmark contains 418 tasks across six domains and three difficulty levels, with more than 2,000 checkpoints and human reference trajectories.
  4. Broad experimental coverage. The paper evaluates multiple models under code-generation (Gen) and structured function-calling (Atm) interfaces, and studies different tool configurations and guidance settings.
Critical weaknesses
  1. Level 3 does not isolate visual–knowledge synergy. The paper states that Level 3 includes both advanced global image analysis and tasks involving interaction between visual evidence and external knowledge. Thus, the overall Level 3 score cannot be attributed solely to synergy. The tool ablations provide some evidence, but cover only two models and do not include significance testing.

  2. The oracle study supports a narrower conclusion than the paper claims. The task-filtering procedure shows that Gemini 3 Pro can solve retained tasks when given correct intermediate evidence. This does not establish that other models fail mainly because of planning or tool execution. The paper also does not clarify whether the three human baseline solvers overlap with the task designers or verifiers.

  3. Process-score validation remains limited. Table 7 compares aggregate Acc, S, and V scores across three judges and human experts, but does not report checkpoint-level agreement. The visual scoring rule also marks a checkpoint as passed if any generated image passes; this verifies that the evidence appears in an artifact, but does not by itself establish that the agent uses that evidence in subsequent reasoning.

Questions for authors
  1. Question: Could the authors report results separately for different Level-3 task types and provide statistical tests for the observed interaction effects?

  2. Question: Could the authors clarify whether the human solvers overlap with the designers or verifiers, and qualify the conclusions drawn from the oracle-guided test?

  3. Question: Could the authors report checkpoint-level agreement and further validate whether passing the visual-evidence check reflects the agent’s subsequent use of that evidence?

6 comments

Dear Reviewer hoUD,

We hope you are doing well. As the discussion period nears its close, we wanted to kindly follow up and thank you again for your encouraging evaluation and deeply insightful suggestions. Your observations regarding Level-3 task isolation, generation-level failure attribution, and checkpoint-level validation helped us bring out the strongest diagnostic insights of Agentic-MME, all of which we have detailed in our response above.

Thank you once again for your thoughtful guidance and support. If our responses have addressed your questions to your satisfaction, we would sincerely appreciate it if you would consider raising your rating in support of our paper. Please let us know if there is anything else we can clarify!

Best regards, The Authors

Dear Reviewer hoUD,

We wanted to kindly follow up again to thank you for your encouraging review and insightful suggestions. We hope our Level-3 subgroup analyses with significance tests, multi-generation failure attribution, and checkpoint-level agreement results have fully addressed your questions. Should you feel that our responses warrant a higher rating, we would deeply appreciate your support during the remaining discussion window.

Thank you again for your valuable time and guidance!

Best regards, The Authors

Dear Reviewer hoUD,

With the discussion period closing today (November 18 at 11:59pm AoE), we wanted to reach out briefly to thank you once more for your constructive feedback and support. Please feel free to let us know if there is anything else we can clarify before the window closes—and if you feel our additional analyses and checkpoint-level validations have fully resolved your questions, we would truly appreciate your considering a higher rating.

Thank you again for your valuable time and guidance!

Best regards, The Authors

Reviewer_3LwdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Agentic-MME is a process-verified benchmark for multimodal agents that measures how well models solve problems by combining active visual manipulation (such as cropping, rotating, and enhancing) with open-web search. It contains 418 real-world tasks spanning 6 domains and 3 difficulty levels, each paired with a human-annotated reference trajectory and over 2,000 stepwise checkpoints that verify intermediate behavior along two axes: the S-axis for search strategy and the V-axis for visual evidence. An Overthink metric additionally quantifies tool-use efficiency, and a unified evaluation harness supports both sandboxed code execution and structured function calling. Experiments show that the best model, Gemini 3 Pro, reaches only 56.3% overall accuracy and drops to between 23.0% and 33.3% on the hardest Level-3 tasks, far below the human baseline of 93.8%. Tools substantially improve performance on hard tasks, but the bottleneck for current models lies in multi-step planning and reliable tool execution rather than knowledge or perception.

Critical strengths

The paper’s central contribution is moving evaluation from whether the answer is correct to whether the process is correct. Every task comes with a human-annotated reference trajectory and stepwise checkpoints: the S-axis checks whether search actually retrieved the expected intermediate information, and the V-axis is further split into V_tool (whether the required tool was invoked at the right step) and V_true (whether the generated intermediate image truly contains the needed evidence), complemented by an Overthink metric for redundant calls. This makes failures attributable to specific stages, such as skipping tool use, cropping the wrong region, issuing poor search queries, or looping redundantly, all of which are conflated in final-answer-only benchmarks. The fine-grained seven-category error analysis in the appendix is a direct product of this mechanism.

The authors invest heavily in data construction to ensure validity. The backward-drafting mechanism first has a state-of-the-art model describe the raw image and then targets details it overlooks or hallucinates, guaranteeing that visual tools are necessary; multiple independent solvers reaching consensus ensures solvability; and step-wise oracle testing feeds ground-truth intermediate observations to the model and keeps only tasks it can then solve, so that failures reflect planning and execution rather than perceptual limits. A series of validation experiments supports these design choices: removing images drives accuracy to near zero, indicating no data leakage; ablations show the full toolset outperforms any restricted setting; and on Level-3 tasks, image tools and search exhibit a super-additive effect, confirming that the hardest tasks genuinely require both capabilities working together.

Critical weaknesses

The paper’s biggest problem is that its empirical conclusions rest on thin statistical ground. Once 418 tasks are split by 3 levels, 6 domains, and 2 interfaces, individual cells become very small: Level 3 contains only about 81 tasks, and the Level-1 S-axis scores in Table 3 are almost all integer multiples of 1/14 (21.4, 28.6, 42.9, 57.1, etc.), suggesting that this column rests on only a dozen or so search checkpoints and carries little statistical meaning. Yet the paper draws fairly strong inferences from such numbers, for example the super-additive synergy claim on Level 3, which actually corresponds to a difference of only a handful of tasks. No variance or repeated runs are reported, even though both web search results and LLM judges are non-deterministic. Several numbers are also internally inconsistent: the abstract says Gemini 3 Pro drops to 23.0% on Level 3 while the main text uses 33.3%, and the Qwen3-VL-235B Full baseline of 42.85% in Tables 4 and 5 does not match its best score of 34.9% in Table 3, with no explanation given. Together these issues make it hard to judge how robust the main findings really are.

Questions for authors

See Weaknesses

1 comment

Dear Reviewer 3Lwd,

Thank you once again for your thoughtful follow-up and for confirming that our rebuttal has addressed your concerns. We sincerely appreciate your constructive feedback and continued support.

Throughout the rebuttal period, we have made every effort to respond rigorously to the reviewers' concerns, including conducting additional full-benchmark repeated experiments, providing statistical variance analyses, clarifying numerical discrepancies, and refining our conclusions to ensure greater scientific precision.

Your feedback has been particularly valuable in helping us strengthen the empirical rigor and presentation of our work. We are encouraged that these efforts have satisfactorily resolved your concerns.

As the discussion period draws to a close, we would be especially grateful if you could consider offering stronger support for our work in the final deliberation, should you feel that our additional evidence and clarifications warrant it. We believe that the substantial human annotation effort, comprehensive process-level evaluation framework, and extensive new experimental validation together make a meaningful contribution to the community.

We fully respect your assessment and sincerely appreciate your time, fairness, and consideration.

Best regards,
The Authors

Reviewer_WJYsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Agentic-MME, a benchmark for evaluating multimodal agents that use visual manipulation tools and web search. It contains 418 tasks across six domains and three difficulty levels, with over 2,000 human-annotated process checkpoints. The benchmark evaluates final-answer accuracy alongside search behavior, visual tool use, intermediate visual evidence, and interaction efficiency. It supports both code execution and structured tool calls. Experiments compare multiple models and include tool ablations, image-removal tests, oracle guidance, human–judge comparisons, and a seven-category failure analysis.

Critical strengths
  1. Substantial human annotation provides a valuable resource for diagnosing agent behavior. The benchmark includes human reference trajectories, intermediate visual evidence, and over 2,000 process checkpoints. These annotations support analysis of where an agent’s workflow breaks down. The seven-category failure taxonomy distinguishes problems such as missing searches, ineffective queries, incorrect visual operations, execution errors, and failures to interpret relevant visual evidence. This diagnostic value is a substantial contribution.

  2. The benchmark is supported by a broad set of validation experiments. The paper includes tool-availability ablations, image-removal experiments, human–judge comparisons, and oracle guidance. In particular, stepwise guidance improves Gemini 3 Flash from 52.24% to 76.21% accuracy, supporting the usefulness of the annotated reference processes. Together, these experiments provide meaningful evidence for the benchmark’s task design and the practical value of its annotations.

Critical weaknesses
  1. The validity of process scores for alternative correct solutions remains insufficiently established. A task may admit several reasonable solution strategies, and an agent can reach the correct answer using intermediate evidence that differs from the reference trajectory. Checkpoints tied to expected intermediate information could therefore penalize valid alternative strategies. The aggregate human–judge comparison in Table 7 does not establish how frequently this happens: similar overall scores can conceal systematic disagreements on particular types of trajectories. Because process evaluation is a central contribution, this uncertainty limits confidence in the interpretation of the reported process scores and associated diagnostic conclusions.

  2. The scope of the findings across stronger vision models remains unclear. Task construction partly targets visual details that a reference model misses in the original image but can recognize after manipulation. Improvements in direct visual understanding could therefore change the necessity of particular tool operations, the difficulty distribution, and the dominant failure modes. Additional evidence from newer representative models would help establish how broadly the conclusions about tool benefits and remaining capability gaps apply.

Questions for authors
  1. Alternative correct solutions: How does the evaluator handle trajectories that produce the correct final answer using a different strategy or different intermediate evidence from the reference? Could you provide a targeted human audit of such trajectories, reporting how often valid alternative steps are rejected by the process evaluator? This would help determine whether the metrics reliably assess successful evidence acquisition across different solution strategies.

  2. Newer models and the stability of the findings: Do you have results for newer vision-capable models, such as Claude Opus 5.5 or GPT Astra, where available? A comparison of no-tool and tool-augmented performance on the same tasks, with documented input settings and controlled tool budgets, would be particularly informative. Do the gains from tools, the difficulty across levels, and the main failure patterns persist for these models?

5 comments

Dear Reviewer WJYs,

We hope this message finds you well. As the author–reviewer discussion period approaches its end, we wanted to gently follow up and express our deepest gratitude once again for your sharp, constructive review. Your thoughtful insights regarding alternative valid solution paths and generalization to next-generation frontier models genuinely elevated the rigor of our benchmark, and we have incorporated dedicated analyses and full evaluations in our response above to address every point you raised.

We truly appreciate your valuable time and expertise. If our responses and new experimental results have satisfactorily resolved your concerns, we would be deeply grateful if you would consider raising your score. Please also let us know if any further clarification would be helpful.

Warm regards, The Authors

Dear Reviewer WJYs,

We wanted to send a brief, polite follow-up to thank you again for your time and thoughtful review. We truly hope our detailed responses, human audits on alternative solutions, and full evaluations on GPT-6.1-Sol, GPT-6-Astra, and Claude Opus 5.5 have resolved your concerns. If you find our rebuttal satisfactory, we would be sincerely grateful if you would consider updating your score during the remaining discussion period.

Thank you once again for your time and consideration!

Warm regards, The Authors

Dear Reviewer WJYs,

As the author–reviewer discussion period concludes today (November 18 at 11:59pm AoE), we wanted to send one brief final note to thank you again for your insightful review. We remain available right up until the deadline should you have any last-minute questions about our human audits or new model evaluations, and we would be deeply grateful if you would consider updating your score if our responses have addressed your concerns.

Thank you very much once again for your time and consideration!

Warm regards, The Authors

Comments on the paper

1 comment

Dear reviewers, the authors have posted their responses to your reviews. Please have a look and discuss ASAP as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.