- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
M³-Bench evaluates multimodal MCP workflows involving dependent steps and potentially parallel tool calls. It provides 211 cases over 27 MCP servers, with reference trajectories constructed through an Executor–Judge pipeline and human verification. The proposed evaluator aligns serialized calls using sentence-embedding similarity and tool-bucketed Hungarian matching, then measures coverage, argument similarity, and workflow structure. Additional model judges assess task completion and information grounding. The paper evaluates a broad range of MLLMs and analyzes tool-use errors and metric sensitivity.
Critical strengths
-
The benchmark brings together several practically relevant aspects of agentic tool use: visual grounding, multi-hop dependencies, parallelizable operations, and persistence of intermediate resources. The 211 cases span 27 MCP servers, and their reference trajectories contain 644 steps and 1,337 calls. This gives the study a concrete executable basis rather than relying only on hypothetical tool descriptions. The distinction between malformed calls, incorrect tool selection, invalid arguments, and backend resource failures also supports useful diagnosis beyond a single end-task success score.
-
The evaluator makes call correspondences inspectable and includes several useful validation efforts. Canonical JSON serialization, same-tool bucketing, and one-to-one Hungarian matching address ordering and correspondence issues explicitly. The paper also includes human refinement of references, a 30-participant study, threshold and encoder analyses, vision-specific diagnostics, and 5,000 trajectory-level bootstrap resamples for the original 14-model leaderboard. These analyses are valuable, even though they do not fully resolve the correctness and interpretation concerns below. Reporting both trajectory metrics and task-completion judgments is also useful because similar workflows and successful outcomes are related but distinct evaluation targets.
Critical weaknesses
-
The central call-matching primitive is not sufficiently validated as a measure of functional correctness. Similarity between serialized calls can remain high when a resource identifier, coordinate, unit, or numerical value changes enough to make the call incorrect. Conversely, different but valid execution-specific resource names can reduce textual agreement. These matches affect coverage and downstream structural scores, so errors can propagate beyond Argument Similarity alone. Appendix B.4's 40-pair calibration against strong/medium/weak correlation labels is a useful first check, but correlation judgments do not directly establish agreement on these consequential cases. The claim that four different encoders produce scores identical to four decimal places also needs clearer intermediate evidence, especially for a metric that directly averages embedding similarities. I regard this as a reproducibility question, not evidence of misconduct.
-
The structural metrics do not yet clearly separate workflow correctness from agreement with one preferred organization of steps. Step Coherence penalizes serializing independent calls that the reference places in one parallel step. Merge Purity penalizes combining reference steps, while the stated Order Consistency comparison includes pairs from different reference steps without explicitly restricting them to annotated causal dependencies. Some such penalties may reasonably describe scheduling efficiency or reference agreement, but they need not imply an invalid execution. Since the reported structural scores are also weighted by recall, lower coverage can reduce several scores simultaneously. This makes it difficult to infer how much of a model's apparent structural weakness reflects causal errors, valid alternative schedules, or missing calls. The existing sensitivity and ranking-stability an
Questions for authors
-
Functional validity of matching. Could you audit a small, explicitly sampled set of aligned call pairs, including incorrect identifiers/coordinates/numbers and valid run-specific resource renamings? Please compare the evaluator's decisions with execution outcomes where available, or with a clearly defined manual correctness judgment, and report the number and types of disagreements. A few representative false matches and missed valid matches would be more informative here than another correlation summary.
-
Valid alternative schedules. Could you score three simple variants of existing valid references: permutation within a parallel step, serial execution of the same independent calls, and reordering of independent branches across steps? Please distinguish intended efficiency penalties from correctness penalties. Reporting raw structural scores alongside their recall-covered versions on these examples would help clarify what each diagnostic measures. This can use a small set of existing trajectories rather than new model runs.
-
Encoder reproducibility. Please specify the exact encoder used in the released evaluator and its normalization/input settings. For several fixed serialized call pairs, could you provide the cosine similarities and resulting match decisions under each of the four encoders? This would explain whether the reported identical four-decimal results arise from stable assignments, score aggregation, or another documented property of the implementation.
-
Outcome-judge validation. The four-judge trimmed mean is a reasonable attempt to reduce individual judge noise. Could you identify the exact judge models and versions, summarize their disagreement on the existing outputs, and explain whether the reference-preference study in Appendix B.4 also validates the Task Completion and Information Grounding scores? If not, a small manually checked sample of judge disagreements would help establish how these outcome metrics complement trajectory similarity, without requiring a large new human study.
-
Coverage and scope. The ecosystem exposes 232 tools, while the reference trajectories exercise 69. Could you summarize how the 211 cases are distributed by actual tool use, hop count, and parallel-branch count, and clarify how the unused tools serve as distractors? A compact breakdown and a few examples would make the multi-hop/multi-threaded coverage claim easier to assess. Please qualify broad capability conclusions where the number of cases is small; adding new servers or substantially enlarging the dataset is not necessary for this clarification.
4 comments
Thank you for the detailed response. Your analysis of call matching and the examples of valid alternative schedules addressed my main concerns about how to interpret the benchmark scores. I will raise my overall rating from 2 to 3.
We sincerely thank the reviewer for the thoughtful engagement and for raising the score, and we will incorporate these clarifications on call matching and alternative schedules into the revision.
Thank you for the detailed clarifications and corrections. I accept several of the explanations and appreciate the authors’ efforts to address the issues.
However, my concern extends beyond individual discrepancies. The original submission contained numerous basic consistency problems across metric definitions, stated bounds, aggregation, and per-task reporting. Their breadth raises concerns about the quality control and traceability of the evaluation as a whole. For a benchmark paper, the reliability of the scoring pipeline and reported results is central to the contribution.
Although the responses address several specific errors, the evidence currently provided has not restored sufficient confidence that the evaluation and reporting have been comprehensively checked. Together with the remaining concerns about metric validity, this leaves my overall assessment unchanged. I will therefore maintain my current score of clear rejection.
Weaknesses
W1. Matching as functional correctness
We greatly thank the reviewer for the careful assessment of the matching primitive.
We linked every alignment decision to the result the tool actually returned (Q1). Matching follows functional validity: 91.1% of strong matches execute without error, against 57.9% of unmatched calls. Encoder settings are given in Q3.
W2. Structural metrics vs. one preferred schedule
We thank the reviewer for the clear distinction between efficiency and correctness.
The reference is not one preferred organization among many. The Judge builds it by "merging operations that could run concurrently" (Sec. 3.1), which places each call in the earliest step its inputs allow; for given dependencies this form is unique. Correctness is measured by Recall, Precision, and ArgSim, which ignore steps; the structural metrics measure how closely the execution follows this earliest-step form. Q2 scores the requested variants on all 211 references.
This is intended: recall weighting keeps "weak systems that match only a few easy MCP tools from inflating structure-aware metrics" (Sec. 4.3). With full coverage, as in the Q2 variants, raw and recall-covered scores coincide.
Questions
Q1. Functional validity of matching
We thank the reviewer for the concrete audit proposal.
Across 14 evaluated models, we looked up each predicted call's actual tool result in the stored runs:
The higher the similarity, the more often the call works. At the model level, the share of successful calls in Fig. 5 ranks models consistently with ArgSim in Table 2 (Spearman ρ = 0.79). We will add a sample of aligned pairs with their execution results to the appendix.
Q2. Valid alternative schedules
We thank the reviewer for the well-designed test.
We built the variants from all 211 references, keeping every call, so each matches itself with similarity 1:
Permutation within a step costs nothing. Serial execution needs about twice as many rounds, and only Step Coherence records this, the efficiency penalty it is defined for: "unnecessary splits of operations that could have been executed within a single multi-threaded step" (Sec. 4.3). Reordering an independent branch across steps keeps every call matched, so Recall, Precision, and ArgSim again stay at 1, and only the structural scores record the delay. Because coverage is full, raw and recall-covered scores are identical here.
Q3. Encoder reproducibility
We thank the reviewer for this reproducibility question.
The released evaluator uses BGE-small-en-v1.5 through SentenceTransformers. Arguments are serialized with keys sorted at every nesting level, and cosine similarity is computed only between calls to the same tool (Sec. 4.1). Part of the stability across encoders follows from the data: 27% of matched pairs have identical serialized arguments, which every encoder scores 1. We will add per-encoder similarities and match decisions for a fixed set of call pairs to the appendix.
Q4. Outcome-judge validation
We thank the reviewer for the thoughtful question.
On the 2,073 trajectories scored by GPT-5-mini, Gemini 2.5 Flash, DeepSeek-Chat, and Grok-4-Fast-Reasoning, the judges agree on the leaderboard: each judge alone ranks the models almost as the trimmed mean does (Spearman ρ = 0.90–0.97). Per trajectory, two judges differ by 0.17 on average, and the trimmed mean drops the highest and lowest of the four. App. B.4 covers the references and thresholds; for the judge scores, the independent check is our deterministic matching, and InfoGrnd ranks the 18 models in Table 2 like Recall does (ρ = 0.89).
Q5. Coverage and scope
We thank the reviewer for the helpful suggestion.
191 of 211 cases are multi-hop, 183 contain at least one parallel step, and 163 are both. A case uses 4.1 distinct tools on average, and 89% of cases span two or three servers. The unused tools are real distractors: at every step the planning prompt contains "[the full tool catalog is appended programmatically]" (App. B.2), so the model chooses among all 232 tools, and a wrong choice appears as an unmatched call or an Unknown Tool Invocation (Fig. 5). We will state case counts next to per-task conclusions.