acceptodds
Under review as a conference paper at ICLR 2027

M³AgentBench: Benchmarking Multi-Turn Agent–User Interaction in Multimodal Tasks

Abstract

Real-world agents must continue working as users intervene after the initial request, yet current multi-turn evaluations often focus on specific interaction phenomena rather than systematically diagnosing how agents respond to different forms of intervention. We introduce M³AgentBench, a benchmark for Multimodal, Multi-Turn, and Multi-Intervention agent evaluation. It contains 60 human-curated multimodal workflows across six domains and covers five representative user interventions: Clarification, Incremental, Revision, Topic Switching, and Adversarial Pressure. Each task requires persistent tool use and produces verifiable artifacts, while state-constrained user simulation and process- and result-aware grading enable reproducible evaluation of both interaction behavior and final outcomes. We evaluate frontier agents and further perform interaction-conditioned failure analysis to diagnose how agents fail beyond aggregate scores. Our study is designed to reveal whether similar overall performance can conceal distinct failure mechanisms and whether correct final artifacts can coexist with flawed interaction processes.

Primary area:
datasets and benchmarks
Keywords:
Agent, Benchmark, Interaction, Multimodal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_HXxGOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces M³AgentBench (Multimodal, Multi-turn, Multi-intervention), a benchmark designed to evaluate how agents handle various forms of user intervention—including clarification, incremental requests, revisions, topic switching, and adversarial pressure—within dynamic, multi-turn workflows. It addresses a key limitation of existing benchmarks, which primarily focus on final task success rather than agents’ ability to adapt to evolving user needs and interactions.

Critical strengths

1 By explicitly incorporating dynamic user interventions as a core evaluation dimension, the benchmark captures critical challenges in real-world human-agent interactions, such as incomplete information, evolving requirements, and topic shifts. This perspective offers valuable insights for developing more practical and robust agents. 2 Separating evaluation into interaction process (Process) and final outcome (Result) enables a more nuanced diagnosis of agent behavior. 3 The authors establish a structured framework that makes inherently open-ended and subjective multimodal, multi-turn interactions more standardized, measurable, and reproducible.

Critical weaknesses

The benchmark contains only 60 workflow tasks, which is relatively small and may limit the statistical power of evaluations, particularly when comparing large-scale frontier models. Moreover, its heavy reliance on manual task design and curation poses significant challenges for cost-effective, automated scaling in the future.

Questions for authors

Given the substantial manual effort required to construct the current 60 tasks, have the authors explored, or do they plan to develop, an LLM-driven pipeline to automatically generate, validate, and scale the benchmark to hundreds or thousands of new workflows based on the existing M³ specifications?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HXxG

1. Task scale and statistical power (Weakness 1)

The 60 long-horizon workflows are sufficient to support controlled comparisons under the intervention conditions covered in this paper. Similar benchmarks also use dozens of workflows that can be executed end to end. WildClawBench [1] contains 60 tasks, averaging approximately 8 minutes per task and totaling approximately 8 hours across all tasks. ClawProBench [2] uses 68 fixed, real-world-style scenarios to evaluate agent reliability. In our evaluation, the cumulative task runtime for one round of all 60 tasks across 15 models was approximately 5 hours.

We broaden coverage through intervention types and their combinations. EgoBench [3] organizes its evaluation into three modes: static, dynamic easy, and dynamic hard. M³AgentBench explicitly distinguishes five types of interventions: clarification, incremental, revision, topic switching, and adversarial pressure. Across 30 compound tasks, we cover 17 distinct intervention sequences. This allows us to examine how requirements evolve within a workflow and whether agents can retain prior constraints and resume the main task after an intervening question. Our existing automated construction pipeline can generate candidate tasks for specified combinations of interventions and run static checks, providing a path toward new combinations and more open-ended scenarios.

[1] Ding, Shuangrui, et al. “WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation.”

[2] Xiao, YuanHang. “ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts.”

[3] Liu, Yunqi, et al. “EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents.”

2. Automated construction and scaling (Weakness 1, Question 1)

We have piloted an automated approach to generating and screening candidate workflows according to the M³ specifications in an office setting. The pipeline can generate task specifications, asset descriptions, and grading criteria for specified intervention combinations and compile task configurations and grading code. For example, we generated 10 candidates for each of the 10 slots in the office setting, and all 100 candidates passed the static checks, which covered interaction timelines, deliverable verifiability, configuration parsing, and grading code compilation.

Candidate tasks still require manual refinement and end-to-end trial runs before they can enter the formal evaluation set. When scaling to hundreds or thousands of tasks, we can use the same generation and screening pipeline and assess the cost of scaling based on the end-to-end pass rate, the retention rate after human review, and the human time required per qualifying task. Scaling may require more manual annotation or validation to ensure evaluation accuracy. We leave this work for the future and will also open-source the relevant assets soon so that the community can maintain and develop them together.

AuthorsOpenReview ↗

Response to Reviewer HXxG

Dear Reviewer HXxG,

Thank you for taking the time to review our response. We noticed the update to your rating and would greatly appreciate any further feedback on the concerns that remain.

In particular, we would be grateful to know whether our response on the benchmark’s scale, statistical power, or automated expansion left any questions unresolved. Your feedback would help us clarify these points and improve the paper.

Best regards,
The Authors

Reviewer_WuR2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces M³ AgentBench, a benchmark for evaluating multimodal agents in multi-turn user interactions. It contains 60 workflows across six domains and five intervention types, including clarification, incremental requests, revision, topic switching, and adversarial pressure. The benchmark evaluates both interaction processes and final task outcomes using state-constrained user simulation. Experiments on 15 models reveal substantial performance gaps and show that successful task completion does not necessarily imply appropriate interaction behavior.

Critical strengths
  1. The paper addresses the important challenge of evaluating how agents respond to evolving user requirements during multi-turn interactions.
  2. The benchmark covers five intervention types and evaluates both interaction processes and final outcomes.
  3. Easy to follow.
Critical weaknesses
  1. Several existing benchmarks, including AppWorld-UL, OSWorld 2.0, and EgoBench, already investigate multi-turn agent-user interactions. The paper's primary contribution appears to be combining existing evaluation dimensions rather than introducing fundamentally new challenges.
  2. The benchmark contains only 60 workflows, with 10 tasks per domain, which may be insufficient to support robust conclusions about general agent capabilities.
  3. The benchmark relies on LLM-based user simulation and grading, both using DeepSeek-V4-Flash. The paper provides limited quantitative validation of grading reliability or agreement with human judgments.
Questions for authors

please see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer WuR2 [1/2]

We thank the reviewer for the valuable feedback. Below, we address the concerns raised.

1. Differences from existing benchmarks (Weakness 1)

M³AgentBench tests whether agents can handle successive interventions within the same multimodal workspace. AppWorld-UL examines clarification, confirmation, and infeasible requests in tool use. OSWorld 2.0 covers mid-workflow updates and proactive clarification in long-horizon GUI workflows. EgoBench evaluates tool-call coverage and the final database state in egocentric video tasks. We explicitly combine different intervention types and use the Process score to assess whether agents respond to revisions in time, retain unaffected content, and resume the main task after a topic switch, among other behaviors. This provides a clearer taxonomy of interaction types and clearer evaluation dimensions for the scenarios agents encounter in multimodal tasks.

Process scoring provides evidence beyond deliverable scoring. In our failure analysis of an evaluation subset, of the 64 trajectories with full Result scores, 26 still lost Process points (40.6%), spanning 11 models and 12 tasks. For example, an agent may ultimately deliver a satisfactory file but may have already submitted a deliverable before obtaining a necessary clarification. Process scoring can identify such timing errors. The Process score is tied to the interaction type itself, allowing it to capture in finer detail whether agents behave as expected across interaction types. Such interactions are more common in multimodal tasks.

2. Task scale and scope of conclusions (Weakness 2)

The 60 long-horizon workflows are sufficient to support controlled comparisons under the intervention conditions covered in this paper. Similar benchmarks also use dozens of workflows that can be executed end to end. WildClawBench [1] contains 60 tasks, averaging approximately 8 minutes per task and totaling approximately 8 hours across all tasks. ClawProBench [2] uses 68 fixed, real-world-style scenarios to evaluate agent reliability. In our evaluation, the cumulative task runtime for one round of all 60 tasks across 15 models was approximately 5 hours.

We broaden coverage through intervention types and their combinations. EgoBench [3] organizes its evaluation into three modes: static, dynamic easy, and dynamic hard. M³AgentBench explicitly distinguishes five types of interventions: clarification, incremental, revision, topic switching, and adversarial pressure. Across 30 compound tasks, we cover 17 distinct intervention sequences. This allows us to examine how requirements evolve within a workflow and whether agents can retain prior constraints and resume the main task after an intervening question. Our existing automated construction pipeline can generate candidate tasks for specified combinations of interventions and run static checks, providing a path toward new combinations and more open-ended scenarios.

[1] Ding, Shuangrui, et al. “WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation.”

[2] Xiao, YuanHang. “ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts.”

[3] Liu, Yunqi, et al. “EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents.”

AuthorsOpenReview ↗

Response to Reviewer WuR2 [2/2]

3. Grading reliability and same-family judge bias (Weakness 3)

To verify the reliability of the Process and Result scores, 4 annotators independently reviewed 12 long trajectories across six domains, covering single and compound interventions, with at least two annotators per trajectory. Human judgments agreed with the automated grading used in the paper on 107 of 118 scoring items, for an overall agreement of 90.7%, with 86.4% (51/59) for Process and 94.9% (56/59) for Result.

To examine whether using DeepSeek as both the user simulator and the judge affects the model rankings, we fixed the interaction trajectories and deliverables, masked the names of the evaluated models, and re-evaluated them using GPT-6.1 Terra and Gemini 3.8 Flash. Across all tasks, the three judges produced identical Overall rankings for all 15 models. The three DeepSeek models still ranked in the top three, which may indicate that they perform these multimodal tasks better across different interaction types and scenarios. For example, the top-ranked model scored higher than Hy4-preview, the strongest non-DeepSeek model, on both Process and Result under all three judges. This suggests that differences in judge preferences had little effect on the performance ranking of the model trajectories.

Evaluated modelDeepSeek (original)GPT-6.1 TerraGemini 3.8 Flash
DS-V4-Flash-V-Exp52.3446.3547.51
DS-V4-Flash49.6445.0745.69
DS-V4-Pro48.7543.2744.19
Hy4-preview46.9842.4643.94
GPT-5.545.9340.9942.33
Claude-Opus-545.8140.8241.99
GPT-5.6-Sol44.8340.5641.93
GLM-5.3-Flash44.2940.3540.94
Kimi-K343.7939.8540.70
GPT-5.6-Luna40.1637.3338.68
GLM-5.340.0337.1137.89
Claude-Sonnet-539.3836.3036.76
Qwen3.8-max38.9734.5435.44
Gemini-3.1-Pro36.2033.2334.59
MiniMax-M335.5829.3530.52
Reviewer_ZU1AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a benchmark for evaluating how multimodal agents respond to or adapt to user requests under different interventions: clarification, incremental, revision, topic switching, and adversarial pressure. The benchmark has 60 stateful multimodal workflows across six domains, with state-constrained multi-turn interaction and process- and result-aware evaluation. The authors evaluated 15 models and provided analysis of vision-interface robustness and interaction-conditioned failure analysis, in addition to model ranking, in the proposed benchmark. Although the benchmark is interesting, I still have some concerns about the design choices, human validation, and whether the final proposed benchmark matches reality.

Critical strengths

The paper proposes a benchmark to address the important problem of how to evaluate multimodal agents when they respond to or adapt to user requests under different interventions. Therefore, the proposed benchmark could help evaluate frontier models on these complex tasks.

Critical weaknesses
  1. Evaluation bias and benchmark realism: The paper heavily relies on the DeepSeek-V4 model family for both the user simulator and the judge. This creates two significant concerns: (1) DeepSeek models achieve the highest scores in Table 2, but DeepSeek-V4-Flash is also used as the LLM judge. This raises concerns about potential bias toward models from the same family. (2) Using a single model may limit the diversity of the simulated user behavior. Also, how do the authors validate that their simulated user interactions realistically represent human behavior?

  2. Human evaluation. The paper mentions that humans are involved in task selection, refinement, and auditing, but it does not report any human evaluation. For example, it is important to analyze human assessments of the task's realism and the benchmark's grading process and criteria. Also, the paper does not report any agreement analysis between human judgments and the LLM judge.

Since the paper proposes a benchmark to address this multimodal, multi-turn, and multi-intervention problem in LLM workflows, it is important to show a fair evaluation and that the proposed scenarios match reality. In particular, for the second, well-designed human evaluation analysis is critical for assessing realism.

Questions for authors
  1. How many human annotators or experts participated in Stages 4, 5, and 6? Were they the same individuals across stages? Did the authors independently validate the tasks and grading rubrics? If multiple humans were involved, was inter-annotator agreement measured?
  2. The authors describe 60 "real-world-like" workflows, initially constructed with LLM assistance. How were these scenarios selected, and what evidence supports their representativeness of real-world agent-user interactions?
  3. How was the LLM-based user simulator validated? Using a single model (DeepSeek) may limit the diversity of user behavior in the simulation. Have the authors considered using a mixture of different model families for user simulation or validating the simulated interactions against real human interactions?
  4. Are the visual assets used in the benchmark real-world images and screenshots, synthetically generated, or a mixture of both? It would be helpful to clarify how these materials were collected or generated and whether their realism was validated.
  5. Why does the DeepSeek model achieve the highest scores? Could the authors validate the results using independent judges from other model families to assess whether the rankings remain consistent?
  6. Why use shared vision as the primary evaluation instead of each model's native multimodal capabilities? Are the main conclusions robust to native vision across all 15 models, rather than just four?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZU1A [1/2]

We thank the reviewer for the constructive feedback. Below, we address the comments raised.

1. Judge bias (Weakness 1&3, Question 5)

To examine whether using DeepSeek as both the user simulator and the judge affects the model rankings, we fixed the interaction trajectories and deliverables, masked the names of the evaluated models, and re-evaluated them using GPT-6.1 Terra and Gemini 3.8 Flash. Across all tasks, the three judges produced identical Overall rankings for all 15 models. The three DeepSeek models still ranked in the top three, which may indicate that they perform these multimodal tasks better across different interaction types and scenarios. For example, the top-ranked model scored higher than Hy4-preview, the strongest non-DeepSeek model, on both Process and Result under all three judges. This suggests that differences in judge preferences had little effect on the performance ranking of the model trajectories.

Evaluated modelDeepSeek (original)GPT-6.1 TerraGemini 3.8 Flash
DS-V4-Flash-V-Exp52.3446.3547.51
DS-V4-Flash49.6445.0745.69
DS-V4-Pro48.7543.2744.19
Hy4-preview46.9842.4643.94
GPT-5.545.9340.9942.33
Claude-Opus-545.8140.8241.99
GPT-5.6-Sol44.8340.5641.93
GLM-5.3-Flash44.2940.3540.94
Kimi-K343.7939.8540.70
GPT-5.6-Luna40.1637.3338.68
GLM-5.340.0337.1137.89
Claude-Sonnet-539.3836.3036.76
Qwen3.8-max38.9734.5435.44
Gemini-3.1-Pro36.2033.2334.59
MiniMax-M335.5829.3530.52

2. Human validation of grading and benchmark construction personnel (Weakness 2&3, Question 1)

To verify the reliability of the Process and Result scores, we asked 4 annotators to independently review 12 long trajectories across all six domains, covering single and compound interventions, with at least two annotators per trajectory. Given the difficulty of evaluating these long trajectories, the annotation took more than 20 hours in total. Human judgments agreed with the automated grading used in the paper on 107 of 118 scoring items, for an overall agreement of 90.7%, with 86.4% (51/59) for Process and 94.9% (56/59) for Result. For example, for a chart analysis trajectory in which the requirements were revised midway through the task, human and automated grading agreed on every Process and Result scoring item.

The same primary reviewer conducted the full review for Stages 4–6 of benchmark construction. For each task, this reviewer checked the candidate task materials and grading rubrics and reviewed the pilot trajectories, covering all 60 workflows. The full annotation process took more than 100 hours. We asked 3 additional reviewers to review and annotate 12 tasks across all six domains, with at least two reviewers annotating each task. The results show that all multimodal assets supported the key facts, 92% of the grading rubrics covered the key success conditions, all information disclosures and state transitions by the user simulator passed the checks, and 92% of the responses passed the naturalness check. These annotations indicate that the current benchmark has a high level of reliability and that its scenarios are realistic.

3. Scenario selection and visual assets (Weakness 3, Question 2&4)

We drew the largest share of our scenario and requirement cues from real conversation data (internal data), then selected tasks based on predefined domains and intervention combinations. The final benchmark has 10 workflows in each of the six domains, with 30 compound tasks covering 17 ordered intervention combinations. The visual assets include real images, screenshots (e.g., of task authoring interfaces and renderings of accompanying data tables), and assets generated with ChatGPT. We created the assets by hand to suit the characteristics and context of each task, avoiding unchecked, large-scale generation via an API. After the assets were created and checked by the reviewer, 3 additional reviewers reviewed and annotated 12 tasks, as described in the previous response. The results show that all multimodal assets supported the key facts. This indicates that the scenarios are realistic and that the visual assets support each task.

AuthorsOpenReview ↗

Response to Reviewer ZU1A [2/2]

4. Simulated users (Weakness 1&3, Question 3)

To test how sensitive the results are to the choice of user simulator, we held the vision model (Gemini 3.1 Pro) and the judge (DeepSeek-V4-Flash-0731) fixed across all 60 tasks. We replaced the DeepSeek simulator in turn with models from three other families (i.e., Qwen3.8-27B, Gemini 4 Argon, and Claude Haiku 5.5) and tested three randomly selected models from different providers. The Overall ranking of the three agents was identical across all four simulators. The mean scores for each condition are shown below. The differences between simulated and real users in intervention timing, information disclosure, and interaction style still need to be validated through comparisons with real users.

User simulatorDS-V4-FlashGLM-5.3-FlashGPT-5.6-Luna
DS-V4-Flash-0731 (paper baseline)49.4444.9340.90
Qwen3.8-27B45.0043.8538.80
Gemini 4 Argon44.2043.8042.67
Claude Haiku 5.546.3543.5339.63

5. Shared vision and native vision (Question 6)

Shared vision allows the main evaluation to compare agents' workflow performance using the same vision tools. We used a shared vision component in the main evaluation to compare, under uniform vision conditions, which information the 15 agents choose to inspect and how they use visual evidence in subsequent interactions and task execution. Our native vision evaluation on four models (Figure 5) shows that after the switch, 5 of the 6 pairwise comparisons retained their original order. The highest- and lowest-scoring models remained the same, and only GLM-5.3-Flash and GPT-5.6-Luna switched places. This suggests that the main performance gaps among the four models remain visible in both settings, although closely matched models may swap positions (possibly reflecting differences in their multimodal capabilities). Different evaluation dimensions can lead to some differences in the results, while the conclusions from evaluating the models as agents under fair vision conditions still hold. However, even a single run per task for each of the 15 models would require 900 complete task runs. In a future revision, we will clarify the scope of the main table and also report the rankings of all models under native vision.

Dear Authors,

Thanks for your response. The human validation with 4 annotators is particularly useful and strengthens the paper. However, I still have some concerns regarding a potential evaluation bias. Do you have any insight into why DeepSeek models are particularly good on the proposed benchmark?

Best,

Reviewer ZU1A

AuthorsOpenReview ↗

Response to Reviewer ZU1A

Dear Reviewer ZU1A,

Thank you for this valuable follow-up question.

As noted in our previous response, re-evaluating the fixed trajectories and deliverables with judges from other model families yielded the same model ranking. Replacing the user simulator also preserved the ranking among these agents tested. These results suggest that DeepSeek’s lead is not solely attributable to a preference of the original judge.

To better understand this lead, we compared model scores and trajectories on the benchmark tasks. DeepSeek performed better on both Process and Result scores, indicating stronger performance in responding to user interventions and advancing multi-turn tasks, as well as in producing the final deliverables. The differences were particularly apparent in chart analysis and code/UI tasks, where agents must incorporate new or revised requirements into existing deliverables.

For example, in a multi-turn UI update task, the agent had to modify an existing project according to visual specifications provided over several turns while preserving functions such as filtering and reservations. DeepSeek’s deliverables scored higher on these functional preservation checks. On this task, DeepSeek averaged 76/100 across all runs, compared with 26 for Claude-Opus-5 and 46 for GPT-5.6-Sol. In a playback-control task, DeepSeek’s deliverables also performed better on consistency checks across the playhead, controller, and display, where both Opus and GPT-Sol lost points. These cases suggest differences in how the models integrate visual information, respond to multi-turn requirements, and maintain functional deliverables.

Together, these observations provide trajectory-level evidence that DeepSeek can carry evolving requirements through to the final deliverable, particularly by maintaining functional and state consistency across revisions. The cross-judge results further support this interpretation.

We also acknowledge that the limited task set may introduce bias. Expanding it would require substantial construction, annotation, and evaluation efforts, including hundreds or even thousands of hours of annotation. We leave this expansion to future work and plan to release the relevant resources soon so that the community can help maintain and develop them. Thank you again for your constructive feedback.

Best,

The Authors

Reviewer_u6AGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces M3 AgentBench, a benchmark for evaluating multimodal agents under multi-turn user interactions. It contains 60 human-curated workflows across six domains and covers five types of user interventions: Clarification, Incremental, Revision, Topic Switching, and Adversarial Pressure. The benchmark uses a state-constrained interaction protocol and evaluates both interaction behavior (Process) and final task outcomes (Result). The authors evaluate 15 frontier models under a unified agent harness and conduct additional analyses of intervention types, visual capabilities, and process-result discrepancies. The results show that current agents still struggle with multi-turn interactions and that correct final outputs do not necessarily imply correct interaction behavior.This paper introduces M3 AgentBench, a benchmark for evaluating multimodal agents under multi-turn user interactions. It contains 60 human-curated workflows across six domains and covers five types of user interventions: Clarification, Incremental, Revision, Topic Switching, and Adversarial Pressure. The benchmark uses a state-constrained interaction protocol and evaluates both interaction behavior (Process) and final task outcomes (Result). The authors evaluate 15 frontier models under a unified agent harness and conduct additional analyses of intervention types, visual capabilities, and process-result discrepancies. The results show that current agents still struggle with multi-turn interactions and that correct final outputs do not necessarily imply correct interaction behavior.

Critical strengths
  1. A meaningful benchmark design for multi-turn agent-user interaction. The paper addresses an important limitation of existing agent benchmarks, which often focus primarily on final task completion. By explicitly defining five types of user interventions, M3 AgentBench provides a structured way to evaluate how agents handle evolving requirements, revisions, clarification, task switching, and adversarial requests. The combination of six task domains and persistent multimodal workspaces makes the benchmark relevant to practical agent applications.

  2. Useful process-level and result-level evaluation. The separation of Process and Result scores is a valuable design choice. As demonstrated in Figure 6, 43 out of 80 trajectories receiving full result credit still lose process points. This shows that correct final artifacts do not necessarily imply correct interaction behavior. The evaluation across 15 models and the additional comparison across four agent harnesses also provide useful empirical observations.

Critical weaknesses
  1. Limited benchmark scale and task coverage. Although the benchmark covers six domains and five intervention types, it contains only 60 manually curated workflows, with just 10 tasks per domain. This is relatively small for drawing broad conclusions about the capabilities of multimodal agents. Moreover, the five intervention types are evaluated through predefined scenarios and state-constrained user simulation. It remains unclear whether the observed results generalize to more diverse, naturally occurring user interactions. A larger task collection and more diverse intervention combinations would make the benchmark more convincing.

  2. Insufficient validation of evaluation reliability. The benchmark relies on a combination of rule-based and LLM-based grading, with DeepSeek-V4-Flash serving as both the user simulator and the judge. This raises concerns about potential evaluator bias and the reliability of model-based scoring. The paper does not provide sufficient independent human validation of the grading results or agreement analysis between different evaluators. In addition, each model is evaluated only twice, without confidence intervals or a detailed analysis of run-to-run variance. Given the relatively small score differences among several leading models, stronger statistical validation is needed to support the reported rankings.

Questions for authors
  1. Could the authors provide independent human evaluation of the Process and Result graders? How well do the LLM-based scores agree with human judgments?

  2. Since DeepSeek-V4-Flash is used for both user simulation and evaluation, have the authors investigated potential evaluator bias by using an independent judge model?

  3. Given that each model is evaluated only twice, would the relative rankings remain stable across more independent runs? Could the authors provide confidence intervals for the main results?

  4. How representative are the 60 manually curated workflows of real-world multimodal agent interactions? Do the findings generalize to less structured and more naturally occurring user interventions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer u6AG [1/2]

We thank the reviewer for the valuable feedback.

1. Task scale, coverage, and generalizability (Weakness 1, Question 4)

The 60 long-horizon workflows are sufficient to support controlled comparisons under the intervention conditions covered in this paper. Similar benchmarks also use dozens of workflows that can be executed end to end. WildClawBench [1] contains 60 tasks, averaging approximately 8 minutes per task and totaling approximately 8 hours across all tasks. ClawProBench [2] uses 68 fixed, real-world-style scenarios to evaluate agent reliability. In our evaluation, the cumulative task runtime for one round of all 60 tasks across 15 models was approximately 5 hours.

We broaden coverage through intervention types and their combinations. EgoBench [3] organizes its evaluation into three modes: static, dynamic easy, and dynamic hard. M³AgentBench explicitly distinguishes five types of interventions: clarification, incremental, revision, topic switching, and adversarial pressure. Across 30 compound tasks, we cover 17 distinct intervention sequences. This allows us to examine how requirements evolve within a workflow and whether agents can retain prior constraints and resume the main task after an intervening question. Our existing automated construction pipeline can generate candidate tasks for specified combinations of interventions and run static checks, providing a path toward new combinations and more open-ended scenarios.

[1] Ding, Shuangrui, et al. “WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation.”

[2] Xiao, YuanHang. “ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts.”

[3] Liu, Yunqi, et al. “EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents.”

2. Human annotation and independent judges (Weakness 2, Question 1&2)

To verify the reliability of the Process and Result scores, we asked 4 annotators to independently review 12 long trajectories across six domains, with at least two annotators per trajectory. Given the difficulty of evaluating these long trajectories, we estimate that annotation took more than 20 hours in total. Human judgments agreed with the automated grading used in the paper on 107 of 118 scoring items, for an overall agreement of 90.7%, with 86.4% (51/59) for Process and 94.9% (56/59) for Result.

To examine whether using DeepSeek as both the user simulator and the judge affects the model rankings, we fixed the interaction trajectories and deliverables, masked the names of the evaluated models, and re-evaluated them using GPT-6.1 Terra and Gemini 3.8 Flash. Across all tasks, the three judges produced identical Overall rankings for all 15 models. The three DeepSeek models still ranked in the top three, which may indicate that they perform these multimodal tasks better across different interaction types and scenarios. For example, the top-ranked model scored higher than Hy4-preview, the strongest non-DeepSeek model, on both Process and Result under all three judges. This suggests that differences in judge preferences had little effect on the performance ranking of the model trajectories.

Evaluated modelDeepSeek (original)GPT-6.1 TerraGemini 3.8 Flash
DS-V4-Flash-V-Exp52.3446.3547.51
DS-V4-Flash49.6445.0745.69
DS-V4-Pro48.7543.2744.19
Hy4-preview46.9842.4643.94
GPT-5.545.9340.9942.33
Claude-Opus-545.8140.8241.99
GPT-5.6-Sol44.8340.5641.93
GLM-5.3-Flash44.2940.3540.94
Kimi-K343.7939.8540.70
GPT-5.6-Luna40.1637.3338.68
GLM-5.340.0337.1137.89
Claude-Sonnet-539.3836.3036.76
Qwen3.8-max38.9734.5435.44
Gemini-3.1-Pro36.2033.2334.59
MiniMax-M335.5829.3530.52

The human evaluation shows that both the Process and Result scores agree closely with human judgments. The two judges from different model families reproduced the original judge's full ranking on the paired samples.

AuthorsOpenReview ↗

Response to Reviewer u6AG [2/2]

3. Repeated runs and confidence intervals (Weakness 2, Question 3)

The mean Overall scores were close across the two runs. On the 55 tasks common to both runs, the median absolute difference in mean Overall scores between the two runs across 15 models was 0.35 points. The table below lists the Overall scores from the main table in the paper alongside the 95% t confidence intervals for the mean scores over the first two runs.

ModelOverall95% CI
DS-V4-Flash-V-Exp52.34[50.13, 53.94]
DS-V4-Flash49.64[47.12, 52.16]
DS-V4-Pro48.75[46.22, 51.28]
Hy4-preview46.98[45.18, 48.78]
Claude-Opus-545.93[45.07, 47.45]
GPT-5.6-Sol45.81[42.98, 47.64]
GLM-5.3-Flash44.83[42.22, 47.44]
GPT-5.544.29[43.59, 46.49]
Kimi-K343.79[41.60, 45.98]
GPT-5.6-Luna40.16[38.63, 43.55]
Qwen3.8-max40.03[37.78, 42.28]
Claude-Sonnet-539.38[37.36, 41.41]
GLM-5.338.97[38.97, 38.97]
Gemini-3.1-Pro36.20[31.56, 40.06]
MiniMax-M335.58[31.17, 37.53]

The Overall ranking remained relatively stable in the additional run. We used two runs for the main evaluation because of cost. One additional run for each of the 15 models would require 900 interaction trajectories. Based on archived usage, we estimate that this would take over 250 hours of cumulative task runtime and cost over $1,500 in model and image-reading fees. To balance cost and evaluation coverage, we randomly selected 7 models, about half of the models in the benchmark, spanning stronger and weaker models, and conducted an additional run. The Spearman ρ between the previous mean scores and the new scores was 0.96, and the mean absolute difference was 0**.83 points**. The vast majority of models maintained their relative order, while Claude-Opus-5 and GPT-5.6-Sol swapped positions. The two models were already close in performance, including on the metrics originally reported in the paper. These results show that the results in the original paper are representative and that the relative rankings are generally stable.

ModelOverallAdditional RunScore differenceRank change
DS-V4-Flash-V-Exp52.3452.95+0.611 → 1
Claude-Opus-545.9345.28−0.652 → 3
GPT-5.6-Sol45.8146.80+0.993 → 2
GPT-5.544.2943.80−0.494 → 4
GPT-5.6-Luna40.1638.30−1.865 → 5
Gemini 3.1 Pro36.2036.98+0.786 → 6
MiniMax-M335.5836.03+0.457 → 7

Comments on the paper

3 comments
AuthorsOpenReview ↗

[For reviewers] A Gentle Reminder

Dear Reviewers,

Thank you again for your time and feedback. As the discussion period will close in a few hours, we would be grateful if you could let us know whether any concerns remain or whether any part of our responses would benefit from further clarification. We would be happy to address them before the deadline.

Best regards,
The Authors

AuthorsOpenReview ↗

[For reviewers] A Gentle Reminder

Dear Reviewers,

As the discussion period will close in approximately two hours, we wanted to follow up once more. We would appreciate any feedback on whether our responses have addressed your concerns, and we are available to clarify any remaining questions before the deadline.

Thank you for your time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Global Response

Dear ACs, SACs, and PCs,

We sincerely thank the reviewers and area chairs for their time and constructive feedback. We provide a concise overview of our contributions, the reviewers’ main concerns, and our responses during the rebuttal.

Our Core Contributions

  • A benchmark for interactive multimodal agents. M³AgentBench contains 60 long-horizon workflows across six domains, where agents operate in persistent multimodal workspaces and respond to user interventions over multiple turns.

  • A systematic framework for user interventions. We define five representative intervention types: Clarification, Incremental, Revision, Topic Switching, and Adversarial Pressure, and combine them in workflows with successive interventions.

  • Process- and result-aware evaluation. We assess both interaction behavior and final deliverables. Our analysis shows that 43 of 80 trajectories with full Result credit still lost Process points, demonstrating that correct final artifacts can coexist with interaction errors.

  • A diagnostic study of frontier agents. We evaluate 15 models and analyze their performance across domains, intervention types, visual interfaces, and interaction-conditioned failure patterns.

Reviewer Concerns and Our Responses

  • Grading reliability and human validation (u6AG, ZU1A, WuR2). We conducted an additional human evaluation in which four annotators independently reviewed 12 long trajectories across six domains. Human judgments agreed with automated grading on 107 of 118 scoring items (90.7%), including 86.4% for Process and 94.9% for Result. Reviewer ZU1A noted that this validation was particularly useful and strengthened the paper.

  • Potential judge bias and the use of DeepSeek (u6AG, ZU1A, WuR2). To test whether the original judge favored DeepSeek models, we fixed the trajectories and deliverables, masked the evaluated model names, and used two judges from other model families. Both judges keeped the original Overall ranking of all 15 models.

  • User simulator and interaction realism (ZU1A). To examine sensitivity to the simulator, we replaced DeepSeek with models from three other families while keeping the vision component and judge fixed. The Overall ranking of the three tested agents remained unchanged across all four simulators. We also clarified the human review and refinement of task materials and interaction protocols.

  • Statistical stability and benchmark scale (u6AG, WuR2, HXxG). We provided further repeated-run analysis and conducted an additional run for seven models across different performance levels. The additional-run scores and previously reported scores had a Spearman correlation of 0.96, with most models retaining their relative order. We also clarified how the benchmark’s five intervention types and their combinations provide coverage across different interaction conditions.

Final Note

The rebuttal added human validation, independent-judge and simulator comparisons, and repeated-run analysis, alongside clarifications of the benchmark’s design and scope. We believe these additions strengthen the evidence for M³AgentBench as a framework for diagnosing how multimodal agents respond to user interventions. We plan to release the benchmark and evaluation resources to support further research.

Thank you again for your consideration.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.