acceptodds
Under review as a conference paper at ICLR 2027

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

Abstract

Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to this capability as **multi-step tool use**. Existing benchmarks have advanced tool-use agent evaluation, but often focus on isolated API calls, short trajectories, or settings that are difficult to scale or control. We introduce , a ***fully synthetic*** benchmark with state-changing tasks across three product domains: *Honor of Kings*, *QQ Music*, and *Tencent Meeting*. decouples *environment synthesis* from *task synthesis*: graph-guided database filling builds reusable, orphan-free product environments, while generator-solver asymmetry creates tasks with both an *information gap* and a *tool gap*, requiring agents to discover hidden data and compose multiple tool calls before changing state. Outcomes are graded deterministically by database-state diffs. Since both environments and tasks are synthetic, is controllable at the environment level and scalable at the task level. Benchmarking cutting-edge LLMs shows that multi-step tool use remains challenging: Pass stays below for the strongest models, and even with code execution in -Code extension, reliability (Pass) remains below .

Primary area:
datasets and benchmarks
Keywords:
Tool-use Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2AZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Paper introduces E-Bench, a benchmark for evaluating multi-step tool-use agents in stateful product environments. It contains 323 synthetic tasks across three domains: Honor of Kings, QQ Music, and Tencent Meeting. First, graph-guided database filling is used to generate consistent product environments. Then, a privileged generator with SQL and code execution access creates tasks that require agents to retrieve hidden information and perform multiple tool calls. Task success is evaluated through deterministic database-state differences.

The authors evaluate 11 LLMs under two settings, E-Bench and E-Bench-Code, where the latter provides additional code execution capabilities. The results show that existing models still struggle with reliable multi-step tool use, while code execution generally improves both task success rates and execution efficiency.

Critical strengths
  1. The paper addresses an important problem in evaluating tool-use agents. Rather than focusing on individual API calls or static question answering, E-Bench requires agents to interact with a stateful environment and perform multiple operations to complete a task.

  2. The per-domain and per-capability analyses help explain these differences and make the benchmark more informative than a single leaderboard.

Critical weaknesses
  1. All three environments are synthetic and built around structured databases with domain-specific tools. Many tasks follow similar patterns involving data retrieval, filtering, aggregation, and database modifications. These are important capabilities, but they may not fully represent the range of challenges faced by general-purpose agents. e.g., ambiguous instructions, unexpected tool failures, changing environments, or interactions across multiple services.

  2. The authors use a strong LLM to generate tasks, followed by validation with three strong models and difficulty filtering using a weak baseline. The process may favor tasks that align with strong model's capabilities and reasoning patterns. Paper does not provide sufficient analysis of how they affect the final task distribution.

Questions for authors
  1. Have the authors performed any human evaluation of task realism or compared the generated tasks with real user workflows? This would help assess how representative the synthetic benchmark is of practical agent tasks.

  2. Could the authors provide statistics on the number of tasks generated and rejected at each validation stage?

  3. How does the benchmark handle tasks with multiple valid outcomes?

  4. Whether generating tasks with different models or random seeds produces similar benchmark difficulty and model rankings?

Reviewer_5PjnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

E-Bench is a synthetic benchmark for evaluating agents that must gather information from stateful environments, coordinate multiple tool calls, and make exact changes to product state. It contains 323 tasks across Honor of Kings, QQ Music, and Tencent Meeting. The authors generate reusable database-backed environments, synthesize tasks that require discovering hidden state, and grade outcomes by comparing database changes with a recorded target. E-Bench-Code adds code execution over the same domain tools. Across 11 models, the strongest base-setting result is 73.79% Avg@3, while the highest Pass3 is 58.82%. Code execution improves Avg@3 for every evaluated model; Opus-4.8 reaches 81.11% Avg@3, but Pass3 remains below 70%. The results suggest that tool-heavy state changes remain unreliable, and that code execution helps most on retrieval and computation tasks. The evidence is from the benchmark’s own synthetic environments; it does not establish performance on live products.

Critical strengths
  • A clear, scalable construction and evaluation design. The paper separates environment generation from task generation and grades outcomes with database-state diffs. This provides a concrete way to evaluate state-changing tool use without relying on an LLM judge.

  • A useful reliability and code-execution analysis. The paper reports multiple success metrics, interaction costs, and results across three domains. The E-Bench-Code comparison helps show where code execution improves performance and where reliability remains low.

Critical weaknesses
  • Limited evidence of external validity. The environments and tasks are synthetic and cover only three product domains. The results therefore support conclusions about these benchmark settings, but provide limited evidence that the tasks represent use of real product systems or predict agent performance on them.

  • Task validity and reproducibility are difficult to assess from the paper. The paper describes LLM generation and model-based validation, but reports little independent human auditing of whether prompts express the intended state changes. It also does not clearly establish what artifacts will be released to reproduce the environments, tasks, and model evaluations.

Questions for authors
  • Did human reviewers independently audit a sample of tasks for ambiguity and for agreement between the user-facing request and its target database diff? If so, what were the audit procedure and error rate?

  • Will you release the environment databases, tasks, tool implementations, generation and validation code, and evaluation harness, along with enough model and run configuration to reproduce the results?

Reviewer_RotpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces E-bench, which is a fully synthetic benchmark for evaluating tool-use agents in multi-step real-world product domains. The paper discusses how the environment is first built, and then the tasks are generated using a powerful generator that has access to the entire environment and the database. The generated tasks are then evaluated using several LLMs but in two different settings: with and without access to code_exec tool. The different strengths of the generator and solver enables the understanding on the impact of information gap and tool gap. The results show that tool-use agents have still not saturated the multi-step real-world tasks. Several valuable ablations have been presented that show the impact of having code_exec access, and across domains and models, how the "best" agent changes based on what is available.

Critical strengths
  • The setting is very good, and the results are valuable. The impact of having access to code_exec tool and how different agents use it across different domains, is interesting to observe.
  • The main valuable contribution imo is the analysis like the impact of information and tool gap and how different agents leverage the available tools itself.
Critical weaknesses
  • The benchmark itself is not very valuable contribution. The paper is written in a way that E-bench is the main contribution and when evaluating the different agents on this dataset, several findings have been made. I would strongly recommend to consider writing it as the contribution being the findings on impact of information and tool gap (access to code_exec) on different agents and domains, and make the dataset release as a secondary contribution. That opens up more datasets to also evaluate AND strengthens the paper significantly imo.
  • If the benchmark is considered the main contribution, then it opens up lots of comparisons with several existing tool-use benchmarks (ToolBench, WorkBench, BFCL, APIBench, ToolSandbox, TauBench, etc. and many others). And opens up the need to clarify the novelty of this benchmark. For that purpose, it would then be important to have a comparison table that with other existing benchmarks which is currently missing.
  • No related work in the paper.
Questions for authors
  • typo in Fig. 2(b)->interaction
  • do all tasks within a domain have the same starting state?
  • is the same analysis possible to be done using other benchmarks? or E-bench is set up uniquely to support this analysis? If it is doable with other benchmarks, you should consider doing that.. maybe something like WorkBench or TauBench supports these analyses?
  • typo, footnote 1 in page 1: solver

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.