- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents MS-Eval, a benchmarking and evaluation toolkit for investigating Main-Agent Delegation and Sub-Agent Execution (MAD-SAE) in MCP-intensive, long-horizon work scenarios. The toolkit includes a benchmark of 166 tasks drawn from eight Toolathlon-Gym domains, manually annotated delegation priors, a controlled Kimi-Code-based execution harness, metrics for delegation appropriateness, and a unified MAD-SAE Quality (MSQ) index that combines delegation matching with performance, efficiency, and API cost relative to a single-agent baseline. The authors compare general, specialized, plan-first, and oracle delegation variants, as well as eight frontier models. Their results indicate that the specialized variant is the strongest practical configuration, that several open-source models outperform the evaluated closed-source models under this setup, and that delegation has conditional benefits: context offloading and parallel execution can improve success or efficiency, while unnecessary delegation, failure propagation, and execution bloat can turn the additional cost into overhead. The paper offers a useful empirical framework for analyzing sub-agent orchestration, although the validity of its conclusions depends substantially on the manually constructed delegation priors, the prompt/harness design, and the chosen evaluation metrics.
Critical strengths
1.The paper studies sub-agent delegation in realistic MCP-intensive workflows rather than only short-horizon reasoning or coding tasks, which is relevant to the emerging design of tool-using work agents.
2.MS-Eval separates delegation appropriateness from downstream orchestration utility and considers task success, critical-path steps, and API cost instead of reporting only a final success rate.
3.The 166 tasks cover eight working domains and include multi-wave, read-only, writable, and read-write workflows. The delegation-prior representation is potentially useful for future research on delegation planning and routing.
Critical weaknesses
1.The split between D_need and D_no-need, as well as the task-level delegation plans, are manually constructed by the authors, but the paper does not report the number of annotators, independent annotation procedures, inter-annotator agreement, or external validation. Consequently, UDR, ODR, sub-agent hit scores, and the Oracle result may largely reflect the authors’ particular decomposition choices.
2.The SA and MAD-SAE comparisons are confounded by different prompts and capabilities. MAD-SAE receives additional delegation norms, role descriptions, few-shot examples, and Agent/AgentSwarm tools, while SA does not. The fact that 305 of the 625 beneficial cases are attributed to Main-Agent-Only behavior further suggests that a substantial part of the gain may come from changed prompting or execution instructions rather than actual sub-agent collaboration.
3.Variant comparisons are primarily conducted with DeepSeek-V4-Flash-0731, while the eight-model comparison uses only Spec. Thus, the claim that Spec is the best practical practice and the observed open-source/closed-source contrast may be specific to the selected model and not a general interaction pattern.
4.The paper does not report repeated independent runs, random seeds, confidence intervals, standard errors, or significance tests. Given the stochasticity of long-horizon LLM trajectories and changing API/model snapshots, aggregate percentages from apparently single evaluations may not be sufficiently reliable.
Questions for authors
Please explain the above weaknesses.