- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
BizBench is a 50-task benchmark for long-horizon business agents, reconstructed from procedures at a payments and financial-technology company. Its environment combines 789 knowledge documents, discoverable business capabilities, business records, and a user simulator. Tasks link operations through approval requirements, shared resources, timing constraints, and commitments. A deterministic evaluator checks requested outcomes, execution requirements, and protected objects or prohibited side effects using both states and histories, while allowing alternative valid strategies. Experiments with ten models examine strict success, repeated-run reliability, information acquisition, and tool-use behavior. The study shows how final goal attainment can conceal intermediate violations and provides evidence about the roles of customer information and document retrieval.
Critical strengths
-
The benchmark captures a consequential gap in outcome-only evaluation for agents. Scoring against business requirements instead of one prescribed trajectory preserves room for different valid strategies.
-
The environment construction and experiments provide a useful research resource. Every task has a passing reference and a validated alternative, and the validation protocol includes targeted violations and feasibility-changing conditions. The empirical analyses examine customer-information disclosure, relevant-document access, repeated database reads, and simulator sensitivity. These analyses help explain why agents fail and suggest concrete directions for improving information gathering and execution
Critical weaknesses
The main limitation is coverage across business domains. The benchmark is reconstructed from one payments company's procedures, and Appendix E explicitly states that the current release focuses on payment-related workflows. Although the tasks include linked activities such as account applications, insurance settlement, and delivery arrangements, these remain within the same reconstructed business setting. The experiments therefore do not establish how readily the construction process and history-based checks transfer to other settings, such as procurement, HR operations, or IT service management. It would help to distinguish the reusable components from the domain-specific rules, tools, and evaluator predicates that would need to be rebuilt. This limits the breadth of the conclusions without detracting from the value of the current benchmark.
Questions for authors
I have no questions for authors
Dear Reviewer 8PGX,
Thank you for recognizing the value of the benchmark’s construction and history-based evaluation. We clarify why the current release focuses on payments and how its methodology relates to other domains.
BizBench is grounded in real-world business operations, with insights drawn from more than a decade of accumulated business data and operational experience. We focus on payment workflows because this is the domain for which we have the necessary historical records, procedures, and operational knowledge.
The underlying service workflows and technical architecture address challenges common to long-horizon enterprise agents: identifying applicable rules, gathering information, obtaining authorization, coordinating shared resources, handling delayed events, and maintaining valid execution through irreversible actions. Related requirements also arise in procurement [1], HR operations, and IT service management [2].
Extending BizBench to these domains requires access to representative historical business data and well-documented operational procedures, which are not currently available to us and therefore fall beyond the scope of this work. Nevertheless, we hope BizBench will provide valuable insights to the research community and encourage further extensions through contributions from researchers and practitioners with deep domain expertise, enabling the incorporation of grounded scenarios across a broader range of industries. Over time, this expanding collection of realistic business workflows could serve as an important foundation for developing and evaluating the next generation of reliable AI agents.
Thank you again for highlighting the importance of broader domain coverage. We hope this response addresses your concern and clarifies both the current scope of BizBench and the opportunities for extending it to other domains.
Best,
Authors
[1] Frank F. Xu et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv, 2024. https://arxiv.org/abs/2412.14161
[2] Léo Boisvert et al. WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks. arXiv, 2024. https://arxiv.org/abs/2407.05291