- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
Paper introduces E-Bench, a benchmark for evaluating multi-step tool-use agents in stateful product environments. It contains 323 synthetic tasks across three domains: Honor of Kings, QQ Music, and Tencent Meeting. First, graph-guided database filling is used to generate consistent product environments. Then, a privileged generator with SQL and code execution access creates tasks that require agents to retrieve hidden information and perform multiple tool calls. Task success is evaluated through deterministic database-state differences.
The authors evaluate 11 LLMs under two settings, E-Bench and E-Bench-Code, where the latter provides additional code execution capabilities. The results show that existing models still struggle with reliable multi-step tool use, while code execution generally improves both task success rates and execution efficiency.
Critical strengths
-
The paper addresses an important problem in evaluating tool-use agents. Rather than focusing on individual API calls or static question answering, E-Bench requires agents to interact with a stateful environment and perform multiple operations to complete a task.
-
The per-domain and per-capability analyses help explain these differences and make the benchmark more informative than a single leaderboard.
Critical weaknesses
-
All three environments are synthetic and built around structured databases with domain-specific tools. Many tasks follow similar patterns involving data retrieval, filtering, aggregation, and database modifications. These are important capabilities, but they may not fully represent the range of challenges faced by general-purpose agents. e.g., ambiguous instructions, unexpected tool failures, changing environments, or interactions across multiple services.
-
The authors use a strong LLM to generate tasks, followed by validation with three strong models and difficulty filtering using a weak baseline. The process may favor tasks that align with strong model's capabilities and reasoning patterns. Paper does not provide sufficient analysis of how they affect the final task distribution.
Questions for authors
-
Have the authors performed any human evaluation of task realism or compared the generated tasks with real user workflows? This would help assess how representative the synthetic benchmark is of practical agent tasks.
-
Could the authors provide statistics on the number of tasks generated and rejected at each validation stage?
-
How does the benchmark handle tasks with multiple valid outcomes?
-
Whether generating tasks with different models or random seeds produces similar benchmark difficulty and model rankings?