acceptodds
Under review as a conference paper at ICLR 2027

M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark

Abstract

We present M-Bench, the first benchmark for evaluating multimodal tool use under the Model Context Protocol. The benchmark targets realistic, multi-hop and multi-threaded workflows that require visual grounding and textual reasoning, cross-tool dependencies, and persistence of intermediate resources across steps. We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching to obtain auditable one-to-one correspondences. On top of this alignment, we report interpretable metrics that decouple semantic fidelity from workflow consistency. The benchmark spans 26 servers with 228 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification; an auxiliary four large language models (LLMs) judge ensemble reports end-task Task Completion and information grounding. Evaluations of representative state-of-the-art Multimodal LLMs (MLLMs) reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structure consistency, underscoring the need for methods that jointly reason over images, text, and tool graphs. Our Benchmark's anonymous repository is at https://anonymous.4open.science/r/M3-Bench-E8848/

Primary area:
datasets and benchmarks
Keywords:
Multimodal LLM Agents, Model Context Protocol, Multi-Hop, Multi-Threaded

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1Mf4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

M³-Bench evaluates multimodal MCP workflows involving dependent steps and potentially parallel tool calls. It provides 211 cases over 27 MCP servers, with reference trajectories constructed through an Executor–Judge pipeline and human verification. The proposed evaluator aligns serialized calls using sentence-embedding similarity and tool-bucketed Hungarian matching, then measures coverage, argument similarity, and workflow structure. Additional model judges assess task completion and information grounding. The paper evaluates a broad range of MLLMs and analyzes tool-use errors and metric sensitivity.

Critical strengths
  1. The benchmark brings together several practically relevant aspects of agentic tool use: visual grounding, multi-hop dependencies, parallelizable operations, and persistence of intermediate resources. The 211 cases span 27 MCP servers, and their reference trajectories contain 644 steps and 1,337 calls. This gives the study a concrete executable basis rather than relying only on hypothetical tool descriptions. The distinction between malformed calls, incorrect tool selection, invalid arguments, and backend resource failures also supports useful diagnosis beyond a single end-task success score.

  2. The evaluator makes call correspondences inspectable and includes several useful validation efforts. Canonical JSON serialization, same-tool bucketing, and one-to-one Hungarian matching address ordering and correspondence issues explicitly. The paper also includes human refinement of references, a 30-participant study, threshold and encoder analyses, vision-specific diagnostics, and 5,000 trajectory-level bootstrap resamples for the original 14-model leaderboard. These analyses are valuable, even though they do not fully resolve the correctness and interpretation concerns below. Reporting both trajectory metrics and task-completion judgments is also useful because similar workflows and successful outcomes are related but distinct evaluation targets.

Critical weaknesses
  1. The central call-matching primitive is not sufficiently validated as a measure of functional correctness. Similarity between serialized calls can remain high when a resource identifier, coordinate, unit, or numerical value changes enough to make the call incorrect. Conversely, different but valid execution-specific resource names can reduce textual agreement. These matches affect coverage and downstream structural scores, so errors can propagate beyond Argument Similarity alone. Appendix B.4's 40-pair calibration against strong/medium/weak correlation labels is a useful first check, but correlation judgments do not directly establish agreement on these consequential cases. The claim that four different encoders produce scores identical to four decimal places also needs clearer intermediate evidence, especially for a metric that directly averages embedding similarities. I regard this as a reproducibility question, not evidence of misconduct.

  2. The structural metrics do not yet clearly separate workflow correctness from agreement with one preferred organization of steps. Step Coherence penalizes serializing independent calls that the reference places in one parallel step. Merge Purity penalizes combining reference steps, while the stated Order Consistency comparison includes pairs from different reference steps without explicitly restricting them to annotated causal dependencies. Some such penalties may reasonably describe scheduling efficiency or reference agreement, but they need not imply an invalid execution. Since the reported structural scores are also weighted by recall, lower coverage can reduce several scores simultaneously. This makes it difficult to infer how much of a model's apparent structural weakness reflects causal errors, valid alternative schedules, or missing calls. The existing sensitivity and ranking-stability an

Questions for authors
  1. Functional validity of matching. Could you audit a small, explicitly sampled set of aligned call pairs, including incorrect identifiers/coordinates/numbers and valid run-specific resource renamings? Please compare the evaluator's decisions with execution outcomes where available, or with a clearly defined manual correctness judgment, and report the number and types of disagreements. A few representative false matches and missed valid matches would be more informative here than another correlation summary.

  2. Valid alternative schedules. Could you score three simple variants of existing valid references: permutation within a parallel step, serial execution of the same independent calls, and reordering of independent branches across steps? Please distinguish intended efficiency penalties from correctness penalties. Reporting raw structural scores alongside their recall-covered versions on these examples would help clarify what each diagnostic measures. This can use a small set of existing trajectories rather than new model runs.

  3. Encoder reproducibility. Please specify the exact encoder used in the released evaluator and its normalization/input settings. For several fixed serialized call pairs, could you provide the cosine similarities and resulting match decisions under each of the four encoders? This would explain whether the reported identical four-decimal results arise from stable assignments, score aggregation, or another documented property of the implementation.

  4. Outcome-judge validation. The four-judge trimmed mean is a reasonable attempt to reduce individual judge noise. Could you identify the exact judge models and versions, summarize their disagreement on the existing outputs, and explain whether the reference-preference study in Appendix B.4 also validates the Task Completion and Information Grounding scores? If not, a small manually checked sample of judge disagreements would help establish how these outcome metrics complement trajectory similarity, without requiring a large new human study.

  5. Coverage and scope. The ecosystem exposes 232 tools, while the reference trajectories exercise 69. Could you summarize how the 211 cases are distributed by actual tool use, hop count, and parallel-branch count, and clarify how the unused tools serve as distractors? A compact breakdown and a few examples would make the multi-hop/multi-threaded coverage claim easier to assess. Please qualify broad capability conclusions where the number of cases is small; adding new servers or substantially enlarging the dataset is not necessary for this clarification.

4 comments

Weaknesses

W1. Matching as functional correctness

We greatly thank the reviewer for the careful assessment of the matching primitive.

We linked every alignment decision to the result the tool actually returned (Q1). Matching follows functional validity: 91.1% of strong matches execute without error, against 57.9% of unmatched calls. Encoder settings are given in Q3.

W2. Structural metrics vs. one preferred schedule

We thank the reviewer for the clear distinction between efficiency and correctness.

The reference is not one preferred organization among many. The Judge builds it by "merging operations that could run concurrently" (Sec. 3.1), which places each call in the earliest step its inputs allow; for given dependencies this form is unique. Correctness is measured by Recall, Precision, and ArgSim, which ignore steps; the structural metrics measure how closely the execution follows this earliest-step form. Q2 scores the requested variants on all 211 references.

lower coverage can reduce several scores simultaneously

This is intended: recall weighting keeps "weak systems that match only a few easy MCP tools from inflating structure-aware metrics" (Sec. 4.3). With full coverage, as in the Q2 variants, raw and recall-covered scores coincide.

Questions

Q1. Functional validity of matching

We thank the reviewer for the concrete audit proposal.

Across 14 evaluated models, we looked up each predicted call's actual tool result in the stored runs:

Alignment decisionCallsExecuted without error
Strong match (sim ≥ 0.8)3,44491.1%
Weak match (0.6 ≤ sim < 0.8)1,06286.3%
Unmatched15,44157.9%

The higher the similarity, the more often the call works. At the model level, the share of successful calls in Fig. 5 ranks models consistently with ArgSim in Table 2 (Spearman ρ = 0.79). We will add a sample of aligned pairs with their execution results to the appendix.

Q2. Valid alternative schedules

We thank the reviewer for the well-designed test.

We built the variants from all 211 references, keeping every call, so each matches itself with similarity 1:

VariantRecallPrecArgSimStepCohMergePurOrdCons
Permutation within steps1.0001.0001.0001.0001.0001.000
Serial execution1.0001.0001.0000.4861.0001.000

Permutation within a step costs nothing. Serial execution needs about twice as many rounds, and only Step Coherence records this, the efficiency penalty it is defined for: "unnecessary splits of operations that could have been executed within a single multi-threaded step" (Sec. 4.3). Reordering an independent branch across steps keeps every call matched, so Recall, Precision, and ArgSim again stay at 1, and only the structural scores record the delay. Because coverage is full, raw and recall-covered scores are identical here.

Q3. Encoder reproducibility

We thank the reviewer for this reproducibility question.

The released evaluator uses BGE-small-en-v1.5 through SentenceTransformers. Arguments are serialized with keys sorted at every nesting level, and cosine similarity is computed only between calls to the same tool (Sec. 4.1). Part of the stability across encoders follows from the data: 27% of matched pairs have identical serialized arguments, which every encoder scores 1. We will add per-encoder similarities and match decisions for a fixed set of call pairs to the appendix.

Q4. Outcome-judge validation

We thank the reviewer for the thoughtful question.

On the 2,073 trajectories scored by GPT-5-mini, Gemini 2.5 Flash, DeepSeek-Chat, and Grok-4-Fast-Reasoning, the judges agree on the leaderboard: each judge alone ranks the models almost as the trimmed mean does (Spearman ρ = 0.90–0.97). Per trajectory, two judges differ by 0.17 on average, and the trimmed mean drops the highest and lowest of the four. App. B.4 covers the references and thresholds; for the judge scores, the independent check is our deterministic matching, and InfoGrnd ranks the 18 models in Table 2 like Recall does (ρ = 0.89).

Q5. Coverage and scope

We thank the reviewer for the helpful suggestion.

Steps per case12–34–5
Cases2011081
Max parallel calls in one step123456≥7
Cases28484441241313

191 of 211 cases are multi-hop, 183 contain at least one parallel step, and 163 are both. A case uses 4.1 distinct tools on average, and 89% of cases span two or three servers. The unused tools are real distractors: at every step the planning prompt contains "[the full tool catalog is appended programmatically]" (App. B.2), so the model chooses among all 232 tools, and a wrong choice appears as an unmatched call or an Unknown Tool Invocation (Fig. 5). We will state case counts next to per-task conclusions.

Thank you for the detailed response. Your analysis of call matching and the examples of valid alternative schedules addressed my main concerns about how to interpret the benchmark scores. I will raise my overall rating from 2 to 3.

We sincerely thank the reviewer for the thoughtful engagement and for raising the score, and we will incorporate these clarifications on call matching and alternative schedules into the revision.

Thank you for the detailed clarifications and corrections. I accept several of the explanations and appreciate the authors’ efforts to address the issues.

However, my concern extends beyond individual discrepancies. The original submission contained numerous basic consistency problems across metric definitions, stated bounds, aggregation, and per-task reporting. Their breadth raises concerns about the quality control and traceability of the evaluation as a whole. For a benchmark paper, the reliability of the scoring pipeline and reported results is central to the contribution.

Although the responses address several specific errors, the evidence currently provided has not restored sufficient confidence that the evaluation and reporting have been comprehensively checked. Together with the remaining concerns about metric validity, this leaves my overall assessment unchanged. I will therefore maintain my current score of clear rejection.

Reviewer_M2rgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces M³-Bench, a benchmark for multimodal, multi-hop, and multi-threaded tool use under the Model Context Protocol. It contains 211 task instances involving an ecosystem of 27 MCP servers and 232 available tools, with reference trajectories exercising 69 tools. Reference trajectories are constructed through an Executor–Judge pipeline with human verification. The proposed evaluation framework combines tool-specific Hungarian matching over embedded call signatures, coverage and trajectory-structure metrics, and four-LLM-judge assessments of task completion and information grounding. The experiments compare multiple multimodal models and analyze their performance across tasks and tool-call failure categories.

Critical strengths
  1. Relevant evaluation setting. The paper addresses an important problem beyond isolated function calling: translating visual information into multi-step tool interactions while managing dependencies, parallel operations, and intermediate resources.

  2. Diverse task and tool coverage. The benchmark connects image-conditioned tasks with heterogeneous services, including image processing, geographic information, knowledge retrieval, and office automation. This provides a useful setting for studying interactions between perception and tool orchestration.

  3. Inspectable evaluation components and concrete failure cases. Explicit call correspondences and detailed trajectory examples make aspects of the evaluation easier to inspect. The distinction between malformed calls, nonexistent tools, invalid arguments, and unavailable resources is useful for understanding interface-level failures.

Critical weaknesses

1. Limitations of the evaluation metrics

1.1 The indirect similarity-based pipeline is not sufficiently validated against direct semantic judgment

The proposed pipeline—serialization, sentence encoding, similarity thresholding, and Hungarian matching—uses textual similarity as a proxy for argument correctness. However, small differences in coordinates, quantities, entity identifiers, or file paths can substantially change a tool call’s behavior, even when its serialized representation remains highly similar. Conversely, functionally equivalent calls may differ in their surface representations.

A context-aware LLM judge could directly consider task constraints and execution feedback, whereas the proposed proxy may discard some of this information. Nevertheless, the paper provides no direct comparison establishing the accuracy trade-off between these approaches. Determinism and auditability are valuable, but neither establishes correctness. Evaluation against independent human labels or execution-based validation is needed to show that the proposed pipeline does not sacrifice judgment accuracy for reproducibility. (Section 4.1, Figure 9, Appendix B.4.)

1.2 Argument Similarity is non-monotonic

Equation (1) averages only matches exceeding the strong threshold of 0.8, while Recall uses the weaker threshold of 0.6. This creates a counterintuitive failure mode. Suppose two fixed matches have similarities of 0.99 and 0.81; ArgSim is then 0.90. If the second similarity decreases to 0.79, both calls remain matched and Recall is unchanged, but the second call is excluded from the ArgSim average, increasing the score to 0.99.

Thus, worsening a call’s similarity can improve the reported argument-quality metric. This follows directly from the definition, and recall weighting does not resolve it. The metric needs a fixed denominator, an explicit penalty for lower-confidence matches, or joint reporting of strong-match coverage and similarity. (Equation 1.)

1.4 The auxiliary judge metrics do not sufficiently establish final task correctness

The Task Completion rubric allocates up to 60% of the score to planning and process, and only 40% to the final result. It therefore measures overall process quality rather than task completion alone. Meanwhile, Information Grounding primarily measures coverage of reference steps, rather than whether the final claims are supported by tool outputs.

A model could execute the expected steps while misinterpreting returned information, writing incorrect data, or producing an invalid deliverable. Step coverage does not reliably distinguish these cases. The paper needs stronger, independent checks of final facts, environment states, or artifact contents to establish that the task was actually completed correctly. (Section 4.4, Appendix B.2.)

2. Bias introduced by a fixed reference trajectory

2.1 Agreement with a reference schedule is not equivalent to causal correctness

The structural metrics primarily compare step assignments and ordering rather than actual dependency edges. Consider a task with only the dependency A→CA\rightarrow C, while BB is independent. Both schedules

\quad\text{and}\quad \{A\}\rightarrow\{B,C\}$$ are causally valid and, assuming equal-duration calls, complete in two rounds. Nevertheless, the second schedule is penalized by Step Coherence and Merge Purity when the first is used as the reference. Order Consistency similarly does not restrict comparisons to calls connected by actual dependencies. Consequently, legitimate reordering of independent operations can be treated as an ordering error. These metrics measure conformity to a particular reference schedule more directly than causal validity or scheduling efficiency. Evaluation against dependency constraints or sets of valid schedules would better support the paper’s stated objective. *(Equations 2–4.)* #### 2.2 Tool-specific matching can undervalue functionally equivalent solutions When a task does not explicitly require a particular tool, alternative tool combinations or retrieval paths may produce equally correct results. Tool-name bucketing nevertheless treats such alternatives as missing reference calls and potentially unnecessary predicted calls, reducing both Recall and Precision. Curating tools with limited functional overlap reduces this ambiguity but does not establish that each reference solution is unique. The paper should distinguish tools mandated by the task from tools merely chosen by the reference author, and evaluate how the metrics handle valid alternative solutions. Otherwise, the benchmark risks measuring adherence to a prescribed implementation rather than general tool-planning ability. *(Sections 3.2 and 4.1.)* #### 2.4 Necessary exploration and recovery may be penalized as redundancy Reference trajectories are cleaned by removing failed or additional calls. In a live environment, however, transient errors, unavailable resources, or changing search results may require retries, query reformulation, or additional verification. Such actions may be necessary for successful execution even when absent from an idealized reference. Step-agnostic matching gives credit to later corrected calls, but Precision can still penalize the preceding attempts. The evaluation does not sufficiently distinguish model-induced waste from reasonable recovery required by environmental feedback. Therefore, lower Precision cannot be straightforwardly interpreted as poorer planning or efficiency without identifying the source and necessity of additional calls. *(Sections 3.1 and 4.1–4.2.)* ### 3. The multimodal claims are not adequately isolated experimentally #### 3.1 The experiments do not establish visual grounding as the principal bottleneck The introduction identifies visual grounding as a central challenge, but the reported scores jointly depend on perception, external tool quality, formatting compliance, argument construction, and planning. Poor performance in this combined setting does not identify which component caused the failure. The paper lacks key controls, such as removing or replacing the image, substituting a complete human-written image description, or providing oracle OCR, entity labels, and bounding boxes. These experiments would distinguish failures to obtain correct visual information from failures to use that information downstream. Without them, the results support an end-to-end evaluation of image-conditioned tool workflows, but not the more specific attribution of performance gaps to visual grounding made in the introduction and analysis. *(Introduction, lines 41–45; Sections 5.2–5.3.)* ### 4. Internal inconsistencies in the reported results #### 4.1 Several scores violate the stated upper bound Section 4.3 explicitly states that recall-covered metrics satisfy $F^{cov}\leq Recall$. However, the appendix reports: | Location | Model | Recall | Reported alignment metric | |---|---|---:|---:| | Table 18 | Qwen3-VL-235B-A22B | 0.055 | ArgSim = 0.063 | | Table 19 | Claude 4.5 Sonnet | 0.489 | StepCoh = 0.506 | | Table 32 | Claude 4.5 Haiku | 0.285 | StepCoh = 0.322 | These discrepancies, particularly the last difference of 0.037, cannot be explained by the approximately ±0.001 numerical tolerance mentioned in the paper. The authors should clarify whether these tables report raw or recall-covered metrics, whether aggregation differs from the stated formula, and provide a consistent computation procedure. *(Section 4.3; Tables 18, 19, and 32.)* #### 4.2 Overall Task Completion scores cannot be reconciled with the task-level results under the stated aggregation The paper defines overall Task Completion as the mean of instance-level scores. However: | Model | Maximum TaskComp across all appendix task categories | Overall TaskComp in Table 2 | |---|---:|---:| | GPT-5 Nano | 0.502 | 0.551 | | GPT-5 Mini | 0.571 | 0.652 | | GPT-5 | 0.653 | 0.671 | If the main and appendix tables summarize the same evaluation instances and judgments, with the task categories forming the corresponding partition, the overall mean cannot exceed every category mean. Changing category weights cannot explain this discrepancy. Additionally, some judge-score pairs are repeated across many different tasks. For example, Qwen2.5-VL’s TaskComp/InfoGrnd pair of 0.303/0.388 appears in 17 task categories. Repetition alone does not establish an error, but together with the aggregation inconsistency, it warrants clarification through per-instance judge outputs and the table-generation procedure. *(Section 4.4; Table 2; Tables 18–45.)* #### 4.3 Further inconsistencies remain in the ablations, aggregate scores, and experimental coverage - **Human-verification ablation:** Table 3 reports Gemini 2.5 Pro ArgSim values of 0.312 and 0.413, a difference of approximately 0.101, despite describing the changes as marginal. The human-verified value also differs from the main-table value of 0.433. Any differences in evaluation settings should be explained. - **Average Score:** For GPT-5, the arithmetic mean of the eight displayed metrics is approximately 0.506, whereas the reported Avg Score is 0.482. This may reflect a different aggregation scheme, but that scheme needs an explicit definition. - **Encoder ablation:** The claim that four different encoders produce identical scores to four decimal places requires supporting configurations, similarities, and matching records, particularly because ArgSim directly averages continuous cosine similarities. - **Model coverage:** The main table contains 18 models, while the conclusion still refers to 14, and the ranking-stability analysis covers only the original model set. The scope of each analysis should be stated consistently. These issues need not share a common cause, but collectively reduce confidence in the traceability and reproducibility of the reported evaluation.
Questions for authors

see in Weaknesses

5 comments

Weaknesses

W1.1 Validation of similarity-based matching

We thank the reviewer for this thoughtful comparison.

We did not use an LLM judge for alignment because judges "rarely check whether argument objects are semantically correct" (Sec. 4.1). Our matching is checked both ways the reviewer asks. Execution: every predicted call is replayed on the real tools (Sec. 5.3); 91.1% of strong matches execute without error, against 57.9% of unmatched calls, and the call success rate (Fig. 5) ranks models like ArgSim (Spearman ρ = 0.79). Human labels: 30 MCP researchers labeled call pairs, and our thresholds gave "the closest agreement with human labels" (App. B.4). The LLM judges also agree: InfoGrnd and Recall rank the 18 models alike (ρ = 0.89).

W1.2 Non-monotonic ArgSim

We thank the reviewer for the precise boundary case.

At 0.79 the pair is still matched, and the ArgSim in Table 2 still counts it: it is the recall-covered score, the sum of matched similarities divided by the number of reference calls. In the example it falls from 0.90 to 0.89; lowering a similarity can only lower it. We will write Eq. 1 this way and report the strong-match average only as a diagnostic, next to its coverage.

W1.4 Judge metrics and final correctness

We thank the reviewer for the close reading of the judge prompts.

The two judge scores are auxiliary; the six main metrics are computed deterministically on the tool calls. References are checked for correct results: the Judge rejects any trajectory where tool output "is clearly wrong, but the call chain proceeds smoothly" (App. B.2). Misread information is also caught downstream, since in a multi-hop task it becomes the next call's argument. In App. E, Claude 4.5 Sonnet reads a Tokyo landmark as Xining, searches "Yakult building Xining China", and "this misidentification caused the subsequent geolocation and air-quality retrieval steps to fail."

W2.1 Reference schedule vs. causal correctness

We thank the reviewer for the concrete example.

The reference is not an arbitrary schedule. The Judge builds it by "merging operations that could run concurrently" (Sec. 3.1), which puts every call in the earliest step its inputs allow; for given dependencies this form is unique. For A→C with independent B it is {A, B}, {C}. The second schedule delays B, and StepCoh and MergePur give partial credit for that; Recall, Precision, ArgSim, and Order Consistency stay at 1.

W2.2 Alternative tool choices

We thank the reviewer for raising alternative solutions.

The tools are fixed by the task, not the reference author: "a human instruction fixes intent and permissible tools" (Sec. 3.1), e.g., Items Price: "Use Amazon MCP (search_products) to retrieve current prices" (App. B). The registry is curated for "minimal functional overlap" (Sec. 3.2), so equivalent substitutes are rare by design.

W2.4 Exploration and recovery

We thank the reviewer for this realistic point.

Recovery is credited: "early-wrong, later-correct fixes" still match (Sec. 4.1), so a successful retry restores Recall. Only Precision counts the failed attempt, which is its purpose: models "may make unnecessary calls, resulting in additional waste of resources" (Sec. 4.2). Environment failures are reported separately: Fig. 5 gives calls where "the call is valid yet the backend cannot locate the resource" their own category, only 2.4% of calls for GPT-5 and 2.7% for Grok 4.

W3.1 Isolating visual grounding

We thank the reviewer for the careful attention to attribution.

App. B.1 scores the visual part of each trajectory separately. Visual Reasoning Ability checks, task by task, whether a model's vision calls are at least as good as its non-visual planning calls; no model passes in half of the tasks (best: Grok 4, 0.455). Perception Recall checks whether the needed OCR, detection, or image-processing tools are called at all, and Grounding Similarity scores the calls that turn image content into tool arguments. Difficulty also follows the image: grouping the 28 tasks by image type, 12 of 14 models score below their own average on specialist domain imagery, against 1 of 14 on chart readout.

To separate obtaining the visual information from using it, we ran the suggested controls with five models under the standard protocol.

Recall change vs. image givenGPT-5GPT-5 MiniGPT-5 NanoClaude 4.5 SonnetClaude 4.5 Haiku
Image removed−0.24−0.14−0.28−0.19−0.53
Image + oracle visual facts+0.00+0.00−0.01+0.02+0.01

Removing the image lowers Recall for every model (all 95% intervals exclude 0), and many models reply that no image was attached and call no tool, as stated in Sec. 1: "multimodal tool invocation hinges on correctly interpreting the image before any tool can be parameterized". Adding oracle facts (names, text, and numbers visible in the image) leaves Recall unchanged: models already read what they need from the image, and the remaining gap lies in planning and argument construction, which the trajectory metrics score separately.

W4.1 Upper bound in per-task tables

We thank the reviewer for cross-checking the appendix.

In Table 2, every alignment score of every model is at or below its Recall. Per trajectory the bound is exact (F^cov = r·F ≤ r) and holds for any aggregate that weights Recall and F^cov alike, as Table 2 does. We regenerated the per-task tables from the stored per-call logs with this weighting: no regenerated alignment score exceeds its task's Recall, and the revised appendix uses these tables.

W4.2 Per-task judge scores

We thank the reviewer for the careful reading.

In most rows of the per-task tables, the two judge columns repeat the model-level score rather than a per-task mean, which explains the repeated pairs; these columns are not a partition of the overall score. Recomputed from the stored per-instance judge outputs, per-task TaskComp ranges over 0.25–0.96 for GPT-5, 0.20–0.84 for GPT-5 Mini, and 0.14–0.82 for GPT-5 Nano, which contain the Table 2 values. The revised appendix reports them.

W4.3 Table 3, Avg Score, encoders, model coverage

We thank the reviewer for the thorough check.

Table 3 is a separate recomputation for two models (Sec. 6) that tests whether the comparison between models depends on the human pass. It does not: in Table 2 and both columns of Table 3, GPT-5 leads Gemini 2.5 Pro on every metric except Precision. The revised Table 3 takes its human-verified column from the Table 2 run.

Avg Score is the mean of the seven columns other than TaskComp; for GPT-5 this gives 0.482.

Encoders: the released evaluator uses BGE-small-en-v1.5 on arguments serialized with sorted keys, comparing only calls to the same tool (Sec. 4.1). We will add per-encoder similarities and match decisions for sample call pairs.

Model coverage: we reran the 5,000-resample bootstrap on all 18 models in Table 2; rankings stay stable (mean Kendall's τ = 0.953; top model kept in 85.1% of resamples). The conclusion will use the current model count.

Thank you for the clarifications and additional experiments. The Avg Score explanation resolves my arithmetic concern, and I appreciate the proposed table corrections. My remaining concerns are:

  1. ArgSim definition versus implementation. The formula in the response differs from Equation (1) followed by recall weighting. In my example, Recall remains 1, so weighting cannot prevent the original score from increasing from 0.90 to 0.99. The new formula fixes this, but which formula generated Table 2? Please clarify whether this is a manuscript error or an implementation change requiring recomputation.

  2. Correctness validation. Successful execution does not imply task-correct arguments, and checking reference trajectories does not verify predicted final outputs. Is there independent validation of task-conditioned argument correctness and final deliverable correctness, beyond execution success and reference-step coverage?

  3. Structural metrics. A canonical earliest-step schedule is not necessarily the only causally valid, equally efficient schedule. For A→C→D with independent B, both {A,B}→{C}→{D} and {A}→{C}→{B,D} finish in three rounds. Yet Equation (4) gives the latter Order Consistency = 0.75 despite no dependency violation. Please address this example or narrow the interpretation to reference-schedule conformity.

Reviewer_y5DHOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

M3BenchM^3Bench evaluates multimodal agents on multi-hop and multi-threaded workflows using MCP tools. It provides 211 cases with reference trajectories constructed through an Executor–Judge pipeline and human verification. The evaluation aligns calls through tool-bucketed Hungarian matching over sentence-embedding similarities and reports detection, argument, structural, and LLM-judge metrics.

Critical strengths
  1. The benchmark offers useful image-conditioned workflows involving heterogeneous tools, cross-call dependencies, and persistent intermediate artifacts.
  2. Separating call coverage, argument similarity, workflow organisation, and end-task quality is a valuable diagnostic objective. Deterministic call correspondences also make scoring auditable.
  3. The paper includes detailed trajectory examples, a 30-participant preference study, threshold calibration, and 5,000 bootstrap resamples for the original fourteen-model leaderboard. These validation efforts deserve credit.
Critical weaknesses
  1. The structural metrics conflate reference scheduling with dependency correctness. Equations (2)–(4) operate on step partitions and indices rather than explicit dependency relations. Consider a DAG A→B→CA\rightarrow B\rightarrow C with an independent call DD. Both reference steps {A,D},{B},{C}\{A,D\},\{B\},\{C\} and predicted steps {A},{B},{C,D}\{A\},\{B\},\{C,D\} satisfy every dependency and require the minimum three steps. With exact call matches, the proposed metrics nevertheless yield StepCoh =0.75=0.75, MergePur ≈0.6845\approx0.6845, and OrdCons =0.75=0.75. Thus, valid alternative schedules can be interpreted as structural failures. Dependency validity and scheduling efficiency should be evaluated separately.
  2. Argument Similarity has a non-monotonic selection effect. With weak/strong thresholds of 0.6/0.8, two fixed matches with similarities {1,0.81}\{1,0.81\} obtain Recall =1=1 and ArgSim =0.905=0.905. Reducing the second similarity to 0.79 leaves Recall unchanged but raises ArgSim to 1 because the lower-scoring pair is excluded. Recall-covered aggregation does not fix this example. A conditional average over strong matches requires accompanying coverage information and should not directly represent overall argument fidelity. The forty-pair correlation calibration does not establish execution-level validity of this score.
Questions for authors
  1. Are the two schedules in the first counterexample considered equally valid? Are explicit dependency edges available and used by the evaluator?
  2. How should the increase in ArgSim after decreasing a matched similarity be interpreted? Has argument scoring been validated against functional correctness?
  3. What explains the metric-bound violation and the discrepancies between Tables 2 and 3? What is the exact Avg Score aggregation?
2 comments

Weaknesses

W1. Structural metrics conflate reference scheduling with dependency correctness (A→B→C + independent D counterexample)

We greatly thank the reviewer for the carefully worked counterexample.

Dependency validity and scheduling efficiency should be evaluated separately.

M³-Bench already does this. Recall, Precision, and ArgSim ignore step boundaries, so the reviewer's predicted schedule receives 1 on all three. Only the three structural metrics register that D moved from A's step into C's, and they give partial scores (0.75, 0.68, 0.75), not a failure.

valid alternative schedules can be interpreted as structural failures

The structural metrics describe how calls are organized, and deciding what to run in parallel is part of the task: a model "may unnecessarily split operations that could run in a single step" or "collapse causally distinct multi-hop subgoals into one step" (Sec. 4.3). The comparison is against a well-defined form, not an arbitrary schedule: the Judge builds each reference by "merging operations that could run concurrently" (Sec. 3.1), which places every call in the earliest step its inputs allow; for given dependencies this form is unique. D is ready in the first step, so the reference runs it there.

On all 211 references, permuting calls within steps leaves every metric at 1, and fully serial execution keeps Recall, Precision, ArgSim, Merge Purity, and Order Consistency at 1 while Step Coherence falls to 0.486. A valid reorganization moves only the scheduling scores.

W2. Argument Similarity is non-monotonic: lowering a match from 0.81 to 0.79 raises ArgSim from 0.905 to 1

We thank the reviewer for the precise threshold analysis.

Reducing the second similarity to 0.79 leaves Recall unchanged but raises ArgSim to 1

The ArgSim column of Table 2 is the recall-covered score: per trajectory, the sum of matched similarities divided by the number of reference calls. In the reviewer's example it falls from 0.905 to (1 + 0.79)/2 = 0.895; lowering a similarity can only lower it. The released evaluator computes this form. We will write Eq. 1 this way and keep the strong-match average only as a diagnostic, reported with its coverage.

The forty-pair correlation calibration does not establish execution-level validity

We linked every alignment decision to the result the tool actually returned. Across 14 evaluated models, 91.1% of strong matches execute without error, against 86.3% of weak matches and 57.9% of unmatched calls. At the model level, the share of successful calls in Fig. 5 ranks models consistently with ArgSim (Spearman ρ = 0.79).

Questions

Q1. Are the two schedules in the counterexample equally valid? Are explicit dependency edges available and used by the evaluator?

We greatly thank the reviewer for the precise question.

Both schedules respect every dependency, and both receive full Recall, Precision, and ArgSim; they differ only in step organization, which the structural metrics score partially (W1).

On dependency edges: the reference stores dependencies as step depth rather than as a separate edge list. Calls in the same step have no dependency edge between them (Definition 2), and in the earliest-step form a call sits in step kk only if it uses a result from step k−1k-1, so a call's step index is the length of the longest dependency chain ending at it. Step Coherence, Merge Purity, and Order Consistency all read this depth directly from the reference (Eqs. 2–4).

Q2. How to interpret the ArgSim increase; has argument scoring been validated against functional correctness?

We thank the reviewer for the clear question.

The reported ArgSim decreases in this example, from 0.905 to 0.895, and the execution check in W2 ties matching to functional correctness, complementing the human calibration in App. B.4.

Q3. Metric-bound violation, Table 2 vs. Table 3 discrepancies, and the exact Avg Score aggregation

We greatly thank the reviewer for cross-checking the tables so carefully.

What explains the metric-bound violation

In Table 2, all four alignment scores of every model are at or below its Recall. For a single trajectory the bound is exact, Fmcov=rmFm≤rmF^{\mathrm{cov}}_m = r_m F_m \le r_m, and it holds for any aggregate that pools Recall and FcovF^{\mathrm{cov}} with the same call weights, as Table 2 does. We regenerated the per-task appendix tables from the stored per-call logs with this weighting, which guarantees the bound: in every regenerated row, no alignment score exceeds its task's Recall. The revised appendix uses these tables and revises the ±0.001 note.

the discrepancies between Tables 2 and 3

Table 3 is a separate recomputation for two models (Sec. 6: "we recompute all trajectory metrics") that tests whether the comparison between models depends on the human pass. It does not: in Table 2 and both columns of Table 3, GPT-5 leads Gemini 2.5 Pro on every metric except Precision. The revised Table 3 takes its human-verified column from the Table 2 run, so the two tables share one set of numbers.

What is the exact Avg Score aggregation?

Avg Score is the arithmetic mean of seven columns of Table 2: Recall, Precision, ArgSim, StepCoh, OrdCons, MergePur, and InfoGrnd. TaskComp is reported next to it but not averaged. For GPT-5, (0.627+0.264+0.583+0.502+0.290+0.453+0.657)/7=0.482(0.627+0.264+0.583+0.502+0.290+0.453+0.657)/7 = 0.482, the reported value. We will state this definition in the caption of Table 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

With Gratitude to the AC and Reviewers: Summary of the Discussion

We thank the AC and all three reviewers for their careful reviews. We replied to each review point by point, quoting each claim before responding and citing the submitted paper wherever it already gave the answer. During the rebuttal we added an execution-linked check of the matching, alternative-schedule tests on all 211 references, an image-removal control on five models, and a ranking bootstrap on all 18 models.

1. Valid alternative schedules (y5DH W1/Q1, 1Mf4 W2/Q2, M2rg W2.1). The reviewers asked whether a causally valid schedule that differs from the reference is scored as a failure. It is not: Recall, Precision, and ArgSim ignore step boundaries, so they stay at 1 for any reorganization of correct calls, and only the structural diagnostics give partial scores. We scored the requested variants on all 211 references: permuting calls within steps leaves every metric at 1, and fully serial execution lowers only Step Coherence (0.486), the efficiency penalty it is defined for. The reference is the earliest-step schedule the Judge builds by "merging operations that could run concurrently" (Sec. 3.1), and the revision describes the structural metrics as agreement with that schedule, as M2rg suggested.

2. Matching vs. functional correctness (y5DH W2/Q2, 1Mf4 W1/Q1, M2rg W1.1). We linked every alignment decision to the result the tool actually returned, across 14 models: 91.1% of strong matches execute without error, against 86.3% of weak matches and 57.9% of unmatched calls, and the call success rate ranks models like ArgSim (Spearman ρ = 0.79). The four outcome judges agree on the leaderboard (ρ = 0.90–0.97 for each judge against the trimmed mean), and InfoGrnd ranks the 18 models like Recall (ρ = 0.89).

3. Formulas and table reporting (y5DH W2/Q3, M2rg W1.2/W4). The ArgSim column of Table 2 is the recall-covered score, which can only fall when a similarity falls (0.905 → 0.895 in y5DH's example); the revision states this form next to Eq. 1. Every alignment score in Table 2 is at or below its model's Recall, the Avg Score definition is added to the caption, and the per-task tables were regenerated from the stored per-call logs. The evaluator and the model rankings are unchanged, and the rankings are stable under 5,000 bootstrap resamples on all 18 models (mean Kendall's τ = 0.953).

4. Isolating visual grounding (M2rg W3.1). We ran the suggested control on five models: removing the image lowers Recall for every model, with all 95% intervals excluding 0.

Where the discussion ended. Reviewer 1Mf4 wrote that "Your analysis of call matching and the examples of valid alternative schedules addressed my main concerns about how to interpret the benchmark scores" and raised the rating from 2 to 3. Reviewer M2rg wrote that the Avg Score explanation "resolves my arithmetic concern" and accepted several further explanations; we answered each of the three follow-up questions, and the final comment raises no new specific issue. The remaining concern is general confidence in the evaluation, which we answered by explaining that every trajectory metric is computed by the released deterministic evaluator from stored per-call logs, the same logs that let us answer each question with exact numbers.

We are grateful to the reviewers for two rounds of careful engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.