- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Agentic-MME introduces a process-level benchmark for evaluating multimodal agents that combine visual manipulation with web retrieval. It includes 418 tasks across three difficulty levels and uses V-axis and S-axis checkpoints to assess visual operations and search behavior. Results show that current models remain well behind human solvers, particularly on Level 3 tasks.
Critical strengths
- Relevant research problem. The benchmark examines how multimodal agents combine visual manipulation and web retrieval, an important setting for multi-tool agents.
- Evaluation beyond final answers. The V-axis and S-axis checkpoints assess intermediate visual and search behaviors, helping diagnose where agents fail.
- Rich process annotations. The benchmark contains 418 tasks across six domains and three difficulty levels, with more than 2,000 checkpoints and human reference trajectories.
- Broad experimental coverage. The paper evaluates multiple models under code-generation (Gen) and structured function-calling (Atm) interfaces, and studies different tool configurations and guidance settings.
Critical weaknesses
-
Level 3 does not isolate visual–knowledge synergy. The paper states that Level 3 includes both advanced global image analysis and tasks involving interaction between visual evidence and external knowledge. Thus, the overall Level 3 score cannot be attributed solely to synergy. The tool ablations provide some evidence, but cover only two models and do not include significance testing.
-
The oracle study supports a narrower conclusion than the paper claims. The task-filtering procedure shows that Gemini 3 Pro can solve retained tasks when given correct intermediate evidence. This does not establish that other models fail mainly because of planning or tool execution. The paper also does not clarify whether the three human baseline solvers overlap with the task designers or verifiers.
-
Process-score validation remains limited. Table 7 compares aggregate Acc, S, and V scores across three judges and human experts, but does not report checkpoint-level agreement. The visual scoring rule also marks a checkpoint as passed if any generated image passes; this verifies that the evidence appears in an artifact, but does not by itself establish that the agent uses that evidence in subsequent reasoning.
Questions for authors
-
Question: Could the authors report results separately for different Level-3 task types and provide statistical tests for the observed interaction effects?
-
Question: Could the authors clarify whether the human solvers overlap with the designers or verifiers, and qualify the conclusions drawn from the oracle-guided test?
-
Question: Could the authors report checkpoint-level agreement and further validate whether passing the visual-evidence check reflects the agent’s subsequent use of that evidence?
6 comments
Dear Reviewer hoUD,
We wanted to kindly follow up again to thank you for your encouraging review and insightful suggestions. We hope our Level-3 subgroup analyses with significance tests, multi-generation failure attribution, and checkpoint-level agreement results have fully addressed your questions. Should you feel that our responses warrant a higher rating, we would deeply appreciate your support during the remaining discussion window.
Thank you again for your valuable time and guidance!
Best regards, The Authors
Dear Reviewer hoUD,
With the discussion period closing today (November 18 at 11:59pm AoE), we wanted to reach out briefly to thank you once more for your constructive feedback and support. Please feel free to let us know if there is anything else we can clarify before the window closes—and if you feel our additional analyses and checkpoint-level validations have fully resolved your questions, we would truly appreciate your considering a higher rating.
Thank you again for your valuable time and guidance!
Best regards, The Authors
Dear Reviewer hoUD,
We hope you are doing well. As the discussion period nears its close, we wanted to kindly follow up and thank you again for your encouraging evaluation and deeply insightful suggestions. Your observations regarding Level-3 task isolation, generation-level failure attribution, and checkpoint-level validation helped us bring out the strongest diagnostic insights of Agentic-MME, all of which we have detailed in our response above.
Thank you once again for your thoughtful guidance and support. If our responses have addressed your questions to your satisfaction, we would sincerely appreciate it if you would consider raising your rating in support of our paper. Please let us know if there is anything else we can clarify!
Best regards, The Authors