acceptodds
Under review as a conference paper at ICLR 2027

MS-Eval: Benchmarking Sub-Agent Delegation and Orchestration in MCP-Intensive Working Scenarios

Abstract

LLM agents are increasingly integrated into real-world works, yet they must handle diverse domain-specific tools and complex multi-sub-task problems that sequential tool invocation resolves inefficiently and with severe context bloat. The Main-Agent Delegation and Sub-Agent Execution (MAD-SAE) paradigm addresses these issues by delegating sub-tasks to specialized sub-agents for parallel execution in isolated contexts, but its delegation and orchestration quality remains unquantified. We introduce MS-Eval, a unified evaluation kit comprising an MCP-intensive benchmark, a controlled execution harness, tailored delegation and orchestration metrics, and the MSQ index for comprehensive assessment. Evaluating four MAD-SAE variants and eight frontier models reveals that (1) sub-agent specialization (Spec) is the best practical practice; (2) open-source models are largely competitive with few exceptions; and (3) MAD-SAE generally raises cost with contingent payoff, benefiting most open-source models in both performance and efficiency while leaving closed-source models with higher cost and degraded performance. Our case study further traces these gains to better single-agent solutions, effective context offloading, and genuine sub-task parallelism, whereas unnecessary or redundant delegation turns the same cost into pure overhead.

Primary area:
datasets and benchmarks
Keywords:
evaluation, benchmark, multi-agent system, multi-agent orchestration, sub-agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZjBHOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MS-Eval, a benchmarking and evaluation toolkit for investigating Main-Agent Delegation and Sub-Agent Execution (MAD-SAE) in MCP-intensive, long-horizon work scenarios. The toolkit includes a benchmark of 166 tasks drawn from eight Toolathlon-Gym domains, manually annotated delegation priors, a controlled Kimi-Code-based execution harness, metrics for delegation appropriateness, and a unified MAD-SAE Quality (MSQ) index that combines delegation matching with performance, efficiency, and API cost relative to a single-agent baseline. The authors compare general, specialized, plan-first, and oracle delegation variants, as well as eight frontier models. Their results indicate that the specialized variant is the strongest practical configuration, that several open-source models outperform the evaluated closed-source models under this setup, and that delegation has conditional benefits: context offloading and parallel execution can improve success or efficiency, while unnecessary delegation, failure propagation, and execution bloat can turn the additional cost into overhead. The paper offers a useful empirical framework for analyzing sub-agent orchestration, although the validity of its conclusions depends substantially on the manually constructed delegation priors, the prompt/harness design, and the chosen evaluation metrics.

Critical strengths

1.The paper studies sub-agent delegation in realistic MCP-intensive workflows rather than only short-horizon reasoning or coding tasks, which is relevant to the emerging design of tool-using work agents.

2.MS-Eval separates delegation appropriateness from downstream orchestration utility and considers task success, critical-path steps, and API cost instead of reporting only a final success rate.

3.The 166 tasks cover eight working domains and include multi-wave, read-only, writable, and read-write workflows. The delegation-prior representation is potentially useful for future research on delegation planning and routing.

Critical weaknesses

1.The split between D_need and D_no-need, as well as the task-level delegation plans, are manually constructed by the authors, but the paper does not report the number of annotators, independent annotation procedures, inter-annotator agreement, or external validation. Consequently, UDR, ODR, sub-agent hit scores, and the Oracle result may largely reflect the authors’ particular decomposition choices.

2.The SA and MAD-SAE comparisons are confounded by different prompts and capabilities. MAD-SAE receives additional delegation norms, role descriptions, few-shot examples, and Agent/AgentSwarm tools, while SA does not. The fact that 305 of the 625 beneficial cases are attributed to Main-Agent-Only behavior further suggests that a substantial part of the gain may come from changed prompting or execution instructions rather than actual sub-agent collaboration.

3.Variant comparisons are primarily conducted with DeepSeek-V4-Flash-0731, while the eight-model comparison uses only Spec. Thus, the claim that Spec is the best practical practice and the observed open-source/closed-source contrast may be specific to the selected model and not a general interaction pattern.

4.The paper does not report repeated independent runs, random seeds, confidence intervals, standard errors, or significance tests. Given the stochasticity of long-horizon LLM trajectories and changing API/model snapshots, aggregate percentages from apparently single evaluations may not be sufficiently reliable.

Questions for authors

Please explain the above weaknesses.

Reviewer_6SZrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MS-Eval is an evaluation framework for sub-agent delegation and orchestration. It combines 166 tasks adapted from Toolathlon-Gym and annotated with reference delegation plans, a controlled execution harness built on Kimi-Code, and evaluation metrics. The authors evaluate four orchestration variants on DeepSeek and the Spec variant across eight models. The main score, MSQ, is the product of agreement with the reference plans and the average utility across success, cost, and critical steps relative to a single-agent baseline. The paper also reports these outcomes separately and analyzes cases where the multi-agent configuration improves or worsens results relative to the single-agent baseline.

Critical strengths
  • The paper reports changes in task success, cost, and critical steps relative to the single-agent baseline across the full benchmark. This lets readers assess the practical trade-offs without relying solely on MSQ.

  • The appendix documents the execution setup in detail, including model settings, cost calculations, agent prompts, and an example task with its reference plan.

  • The case analysis separates main-agent-only outcomes from those involving delegation and gives concrete examples of context offloading, parallel execution, error propagation, and unnecessary work.

Critical weaknesses
  1. The paper does not establish why the reference plans should define an upper bound on orchestration quality. Appendix B.2 allows different agent counts and specializations through functional matching. However, the paper does not establish whether deviations from the reference allocation of work between the main agent and sub-agents correspond to worse practical outcomes. Oracle is evaluated only on DeepSeek. It achieves high reference agreement and fewer critical steps, but no improvement in full-benchmark success or cost over the single-agent baseline.

  2. The MSQ ranking is difficult to interpret as a measure of practical benefit. DeepSeek Oracle ranks well above Kimi, although Kimi improves both full-benchmark success and cost over its own single-agent baseline, while Oracle does not. Oracle does reduce critical steps, but its much higher score also depends on matching the plan supplied to it. Why should reference agreement receive this much weight? I would like to see whether it predicts better task outcomes before accepting it as a multiplier in the headline score.

  3. The utility comparisons and case analysis need clearer treatment of task selection and run-to-run variability. In Table 1, utilities are computed on configuration-dependent subsets of tasks where delegation occurred, with cost and steps further conditioned on success. Comparisons across configurations therefore reflect different task selections, although the full-benchmark results partly address this issue. The paper also does not specify the number of repeated runs or report uncertainty estimates. This matters particularly for small success-rate differences and for interpreting the 305 beneficial cases with no sub-agent execution: without repeated runs, it is unclear how much of the observed change is reproducible.

  4. The claim that Spec is the best practice rests on the variant comparison using DeepSeek alone. For the other seven models, Spec is the only multi-agent variant evaluated, so this choice has not been validated across models. In particular, the regressions for Claude and GPT may depend on the selected variant. These experiments support conclusions about Spec in this setup, but not about how those models would perform with Gen or Plan-1st. The conclusions should therefore be scoped to the tested agent configurations, rather than presented as general claims about the models’ delegation and orchestration capabilities.

Questions for authors
  1. How were the annotated delegation priors validated, and what evidence supports treating them as an upper bound on orchestration quality?

  2. Does higher reference agreement predict better success, cost, or critical-step outcomes? How stable is the ranking when all configurations are compared on the same tasks and the agreement multiplier is removed?

  3. Do the reference plans improve practical outcomes when supplied to models other than DeepSeek?

  4. How many independent runs were performed for each task and configuration? Can you report uncertainty estimates to assess whether the small success-rate differences and the improvements without sub-agent execution are reproducible?

Reviewer_1U3POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MS-Eval, a benchmark designed to evaluate sub-agent delegation and orchestration in MCP-intensive working scenarios. MS-Eval consists of 166 tasks across eight real-world domains and systematically evaluates different delegation strategies and frontier models in terms of delegation appropriateness, task performance, execution efficiency, and cost. Experimental results suggest that specialized sub-agents can improve delegation quality, while the actual benefits of multi-agent collaboration depend heavily on the appropriateness of task delegation.

Critical strengths

Relevant and meaningful research problem. The paper investigates when and how agents should delegate tasks and whether multi-agent collaboration provides practical benefits, rather than focusing solely on task success rates.

Comprehensive evaluation framework. The proposed benchmark combines diverse working scenarios, annotated delegation priors, and multiple evaluation dimensions, including performance, efficiency, and cost, enabling a systematic assessment of multi-agent orchestration.

Insightful empirical analysis. The authors compare multiple delegation strategies and frontier models and analyze factors such as context offloading, parallel execution, and redundant delegation, providing useful insights into the benefits and limitations of multi-agent collaboration.

Critical weaknesses

Insufficient validation of delegation annotations. The authors use manually annotated delegation priors as reference standards for 166 tasks but do not adequately report inter-annotator agreement or investigate whether the F1 hit metric unfairly penalizes alternative yet equally valid delegation strategies.

Limited justification for the MSQ metric. The proposed MSQ multiplies F1 hit by the arithmetic mean of three equally weighted utility metrics. However, the paper provides insufficient justification for this aggregation design and lacks sensitivity analyses or comparisons with alternative metric formulations.

Insufficient statistical robustness of experimental results. Although the paper evaluates multiple delegation strategies and eight frontier models, it does not adequately report variance across repeated runs, confidence intervals, or statistical significance tests, making it difficult to assess the reliability of the observed performance differences.

Limited generalizability and fairness of model comparisons. The experiments primarily rely on 166 tasks derived from Toolathlon-Gym, without validation in independent environments. Furthermore, differences in model inference configurations and API pricing may affect the fairness of comparisons regarding delegation capabilities and cost-effectiveness.

Questions for authors

Reliability of delegation annotations: How do the authors validate the reliability of manually annotated delegation priors? How does MS-Eval avoid unfairly penalizing alternative yet equally valid delegation strategies for the same task?

Validity of the evaluation metric: What motivates the multiplicative formulation of MSQ using F1 hit and three equally weighted utility metrics? Could the authors provide sensitivity analyses to demonstrate the stability of model rankings under different weighting schemes?

Robustness of experimental results: Could the authors report results from multiple independent runs, along with confidence intervals and statistical significance tests, to establish the reliability of performance differences across delegation strategies and models?

Generalizability of the conclusions: Could the authors validate their main findings using alternative agent harnesses or independent task environments? Additionally, could they control for the number of sub-agent roles, tool permissions, and inference budgets to better isolate the benefits of sub-agent specialization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.