- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces M³AgentBench (Multimodal, Multi-turn, Multi-intervention), a benchmark designed to evaluate how agents handle various forms of user intervention—including clarification, incremental requests, revisions, topic switching, and adversarial pressure—within dynamic, multi-turn workflows. It addresses a key limitation of existing benchmarks, which primarily focus on final task success rather than agents’ ability to adapt to evolving user needs and interactions.
Critical strengths
1 By explicitly incorporating dynamic user interventions as a core evaluation dimension, the benchmark captures critical challenges in real-world human-agent interactions, such as incomplete information, evolving requirements, and topic shifts. This perspective offers valuable insights for developing more practical and robust agents. 2 Separating evaluation into interaction process (Process) and final outcome (Result) enables a more nuanced diagnosis of agent behavior. 3 The authors establish a structured framework that makes inherently open-ended and subjective multimodal, multi-turn interactions more standardized, measurable, and reproducible.
Critical weaknesses
The benchmark contains only 60 workflow tasks, which is relatively small and may limit the statistical power of evaluations, particularly when comparing large-scale frontier models. Moreover, its heavy reliance on manual task design and curation poses significant challenges for cost-effective, automated scaling in the future.
Questions for authors
Given the substantial manual effort required to construct the current 60 tasks, have the authors explored, or do they plan to develop, an LLM-driven pipeline to automatically generate, validate, and scale the benchmark to hundreds or thousands of new workflows based on the existing M³ specifications?
2 comments
Response to Reviewer HXxG
Dear Reviewer HXxG,
Thank you for taking the time to review our response. We noticed the update to your rating and would greatly appreciate any further feedback on the concerns that remain.
In particular, we would be grateful to know whether our response on the benchmark’s scale, statistical power, or automated expansion left any questions unresolved. Your feedback would help us clarify these points and improve the paper.
Best regards,
The Authors
Response to Reviewer HXxG
1. Task scale and statistical power (Weakness 1)
The 60 long-horizon workflows are sufficient to support controlled comparisons under the intervention conditions covered in this paper. Similar benchmarks also use dozens of workflows that can be executed end to end. WildClawBench [1] contains 60 tasks, averaging approximately 8 minutes per task and totaling approximately 8 hours across all tasks. ClawProBench [2] uses 68 fixed, real-world-style scenarios to evaluate agent reliability. In our evaluation, the cumulative task runtime for one round of all 60 tasks across 15 models was approximately 5 hours.
We broaden coverage through intervention types and their combinations. EgoBench [3] organizes its evaluation into three modes: static, dynamic easy, and dynamic hard. M³AgentBench explicitly distinguishes five types of interventions: clarification, incremental, revision, topic switching, and adversarial pressure. Across 30 compound tasks, we cover 17 distinct intervention sequences. This allows us to examine how requirements evolve within a workflow and whether agents can retain prior constraints and resume the main task after an intervening question. Our existing automated construction pipeline can generate candidate tasks for specified combinations of interventions and run static checks, providing a path toward new combinations and more open-ended scenarios.
2. Automated construction and scaling (Weakness 1, Question 1)
We have piloted an automated approach to generating and screening candidate workflows according to the M³ specifications in an office setting. The pipeline can generate task specifications, asset descriptions, and grading criteria for specified intervention combinations and compile task configurations and grading code. For example, we generated 10 candidates for each of the 10 slots in the office setting, and all 100 candidates passed the static checks, which covered interaction timelines, deliverable verifiability, configuration parsing, and grading code compilation.
Candidate tasks still require manual refinement and end-to-end trial runs before they can enter the formal evaluation set. When scaling to hundreds or thousands of tasks, we can use the same generation and screening pipeline and assess the cost of scaling based on the end-to-end pass rate, the retention rate after human review, and the human time required per qualifying task. Scaling may require more manual annotation or validation to ensure evaluation accuracy. We leave this work for the future and will also open-source the relevant assets soon so that the community can maintain and develop them together.