acceptodds
Under review as a conference paper at ICLR 2027

BizBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business

Abstract

Existing evaluations of long-horizon agents often emphasize final task success while overlooking the validity of intermediate decisions. Real-world business applications, however, demand stricter assessment: an agent must behave correctly not only at the endpoint, but throughout the entire execution process. This requirement is particularly critical in high-stakes domains such as payments, where invalid intermediate actions can cause financial loss, regulatory violations, or irreversible downstream consequences even when the final goal is eventually achieved. Consequently, agents that perform well on existing benchmarks may still fail in deployment, as outcome-centric evaluation provides limited insight into the validity and downstream consequences of intermediate decisions. In this work, we aim to narrow the gap between offline evaluation of long-horizon agents and their deployment in real-world business systems. We introduce BizBench, a benchmark derived from real-world business procedures at a leading payments company. We encode these procedures in an executable environment with aligned policies, states, and tools; compose long-horizon tasks with resource, temporal, and commitment constraints, requiring agents to reconcile knowledge conflicts and identify outdated guidance; and replay reference traces to verify joint feasibility of goals and constraints. The resulting benchmark comprises 100 executable tasks and evaluates both final task success and execution validity through deterministic checks over outcomes and execution histories, while permitting multiple legitimate solution strategies. Extensive experiments on BizBench show that even state-of-the-art models struggle on BizBench, with some runs extending beyond 200 steps of tool use and document retrieval. Our results reveal a substantial gap between successful task completion and sustained valid execution. BizBench provides a testbed for developing long-horizon agents that can coordinate information gathering, planning, and action to complete business tasks while maintaining valid execution throughout.

Primary area:
datasets and benchmarks
Keywords:
Long-Horizon Tasks, Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8PGXOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

BizBench is a 50-task benchmark for long-horizon business agents, reconstructed from procedures at a payments and financial-technology company. Its environment combines 789 knowledge documents, discoverable business capabilities, business records, and a user simulator. Tasks link operations through approval requirements, shared resources, timing constraints, and commitments. A deterministic evaluator checks requested outcomes, execution requirements, and protected objects or prohibited side effects using both states and histories, while allowing alternative valid strategies. Experiments with ten models examine strict success, repeated-run reliability, information acquisition, and tool-use behavior. The study shows how final goal attainment can conceal intermediate violations and provides evidence about the roles of customer information and document retrieval.

Critical strengths
  1. The benchmark captures a consequential gap in outcome-only evaluation for agents. Scoring against business requirements instead of one prescribed trajectory preserves room for different valid strategies.

  2. The environment construction and experiments provide a useful research resource. Every task has a passing reference and a validated alternative, and the validation protocol includes targeted violations and feasibility-changing conditions. The empirical analyses examine customer-information disclosure, relevant-document access, repeated database reads, and simulator sensitivity. These analyses help explain why agents fail and suggest concrete directions for improving information gathering and execution

Critical weaknesses

The main limitation is coverage across business domains. The benchmark is reconstructed from one payments company's procedures, and Appendix E explicitly states that the current release focuses on payment-related workflows. Although the tasks include linked activities such as account applications, insurance settlement, and delivery arrangements, these remain within the same reconstructed business setting. The experiments therefore do not establish how readily the construction process and history-based checks transfer to other settings, such as procurement, HR operations, or IT service management. It would help to distinguish the reusable components from the domain-specific rules, tools, and evaluator predicates that would need to be rebuilt. This limits the breadth of the conclusions without detracting from the value of the current benchmark.

Questions for authors

I have no questions for authors

1 comment

Dear Reviewer 8PGX,

Thank you for recognizing the value of the benchmark’s construction and history-based evaluation. We clarify why the current release focuses on payments and how its methodology relates to other domains.

About Domain coverage and transferability

BizBench is grounded in real-world business operations, with insights drawn from more than a decade of accumulated business data and operational experience. We focus on payment workflows because this is the domain for which we have the necessary historical records, procedures, and operational knowledge.

The underlying service workflows and technical architecture address challenges common to long-horizon enterprise agents: identifying applicable rules, gathering information, obtaining authorization, coordinating shared resources, handling delayed events, and maintaining valid execution through irreversible actions. Related requirements also arise in procurement [1], HR operations, and IT service management [2].

Extending BizBench to these domains requires access to representative historical business data and well-documented operational procedures, which are not currently available to us and therefore fall beyond the scope of this work. Nevertheless, we hope BizBench will provide valuable insights to the research community and encourage further extensions through contributions from researchers and practitioners with deep domain expertise, enabling the incorporation of grounded scenarios across a broader range of industries. Over time, this expanding collection of realistic business workflows could serve as an important foundation for developing and evaluating the next generation of reliable AI agents.

Thank you again for highlighting the importance of broader domain coverage. We hope this response addresses your concern and clarifies both the current scope of BizBench and the opportunities for extending it to other domains.

Best,

Authors


[1] Frank F. Xu et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv, 2024. https://arxiv.org/abs/2412.14161

[2] Léo Boisvert et al. WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks. arXiv, 2024. https://arxiv.org/abs/2407.05291

Reviewer_UkyBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BizBench is a benchmark for long-horizon agents built from real business procedures at a large payments company. Its core contribution is a thoroughly designed set of 50 tasks where agents have to find the right rules among outdated and conflicting documents, work with a LLM simulated customer for approvals for its actions, and handle linked workflows with shared funds and deadlines. Information and relevant tools only shows up as events happen, so the benchmark tests how agents act when they have to commit to irreversible steps without the full picture in view. Scoring checks the whole execution history/trajectory, not just the end state, which catches reward hacky completions where the agent reaches the goal by breaking rules along the way. Experts reviewed each task to make sure it was valid and solvable, so there are no impossible tasks. Even the best model only reaches 64%, and many agents complete goals through steps that wouldn’t be acceptable in a real business and this benchmark can be used to measure such behavior.

Critical strengths
  1. Task design and validation: The task design is the core strength of this benchmark. The long horizon tasks mimic real business scenarios, which are messy and full of outdated and contradictory information, and where one wrong intermediate step can fail the whole task even if the goal is reached. The authors back this with robust checks, including replayed reference solutions, validated alternatives, violation tests, and expert review, which builds trust that the tasks are valid and solvable.
  2. Evaluation that separates outcome from process: Splitting success into goal (G), process (P), and protection (F), along with intermediate failure and strict success, helps isolate where agents actually fail and shows agents reaching goals through invalid steps. The long horizon and budget sensitivity analysis adds to this by showing that much of the agents’ extra effort goes into repeated or misdirected lookups rather than progress.
Critical weaknesses
  1. Analysis of agent failures could go deeper: The failure analysis mostly focuses on runs that reach the goal but break rules, and the detailed trace analysis is done for a few open models, mainly DeepSeek V4.1Flash. It would be helpful to see more on why stronger models like Claude Opus 5 and GPT-6 Sol fail the goal. The look-ahead planning cases (Q03 routes, Q24, Q34) also seem to be constructed from the task terms rather than observed model runs, so it’s less clear how agents actually handle these situations. 2., User interaction could be analyzed more: The full disclosure ablation gives a larger gain (43.3% to 70.0%) than the gold-doc ablation (50.0%), which suggests interacting with the customer is a major bottleneck. It would help to know whether agents ask the right questions or irrelevant ones, and whether the simulator ever rejects requests or approves things it shouldn’t.
Questions for authors

A. Clarifications (details about the setup)

  1. Agent tools: Could you list the 9 always-available agent tools with a short description of each? Is there a tool for messaging the customer?
  2. Golden references: How were the Golden reference traces produced, and how was the designated trace chosen when a task had more than one valid reference?
  3. Gold-doc package: How was the “revised Gold-doc package” in the ablation (Table 2) built, and what was revised?

B. Additional analysis (can be done on existing runs)

  1. Simulator behavior: How often does the user simulator decline an approval or push back on a request? Does it ever approve requests it shouldn’t, like ones outside the customer’s fee cap?
  2. Goal failures: Could you share more on why runs fail when the goal isn’t reached, ideally with failure categories across models?
  3. Look-ahead tasks: Section 4.5 mentions that annotations record consequential decisions for each task. Could you use these to report how agents behaved at decisions that require look-ahead, such as time-limited or irreversible steps? For eg: how does strict success on tasks with these decisions compare to the rest? do agents tend to commit too early or wait too long? etc...
  4. Question efficiency: Could you report how efficient the agent’s questions to the user are? Like, how many questions were needed to get the required facts? How many were redundant or irrelevant? etc...

C. Suggestions for future work (not required for this submission)

  1. Budget-aware runs: The tool-call cap in Table 3 is applied retrospectively and only for DeepSeek V4.1Flash. It would be interesting to see results with the agent aware of the budget, and for stronger models like GPT6 Sol.
  2. Memory: Agents seem to rely only on the context window. Testing a memory or note-taking tool could show whether it reduces repeated reads or helps keep track of customer terms. Humans probably would take notes for such business decisions so would be an interesting setting to compare to.
  3. Planning approaches: Comparing planning approaches, like plan-first prompting versus a step-by-step loop, would help show which strategies work best on these tasks.
Reviewer_fa4eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BizBench tests whether business agents can complete customer requests while following rules throughout execution. It contains 50 payment-related tasks, with 789 documents and 601 discoverable business capabilities. A deterministic evaluator checks task completion, process requirements, and whether unrelated records are changed. Experiments on ten models show cases where agents complete the requested goal but violate requirements along the way.

Critical strengths
  1. Authorization, timing, and shared-resource constraints make the tasks useful. Agents need to manage dependencies between operations, beyond simply finding and calling the right tools.
  2. The construction includes replaying reference solutions, checking valid alternatives, and testing targeted violations. These checks provide useful evidence that the tasks can be completed and that the evaluator catches the intended errors.
Critical weaknesses
  1. The difference from existing benchmarks needs to be clearer. Which requirements and failure types are not covered before? Comparing mainly with final-outcome evaluation does not fully answer this question.
  2. Documents, states, tools, and tasks are partly synthesized. Independent checks on actual model trajectories are therefore important. Agreement on task design does not show how often the evaluator accepts an invalid execution or rejects a valid one.
  3. The process score combines blocked unauthorized attempts with violations that actually take place. A rejected call and an executed reserve breach have different consequences.
  4. The leading models have different numbers of trials. Claude Opus 5 has one run per task, while GPT-6 Sol has three.
  5. The figures mainly show overall scores and resource use. The appendix examples are helpful, but the reference trajectory figure is hard to read.

After rebuttal: The trajectory audit and the breakdown of blocked attempts versus actual violations address my main concerns. The leading models still have unequal numbers of runs, so their ranking remains uncertain. I keep my rating at 3.

Questions for authors

see weaknesses

3 comments

Dear Reviewer fa4e,

Thank you for recognizing the value of the business constraints and construction checks. Your comments prompted us to clarify the contribution and examine evaluator judgments and execution violations in the saved trajectories.

Differentiation from existing benchmarks

BizBench evaluates linked business workflows in which spending shared funds can prevent a later obligation from being fulfilled, and waiting for information can close an action window. It checks both task completion and business requirements throughout execution. The table compares these features with the closest benchmarks.

BenchmarkCustomer interactionBusiness-rule retrievalResource constraintsExpiring action windowsSuccess checks
τ-bench✓—Payment balances—Final state and responses
τ²-bench✓—Payment balances in retail/airline—Final state, responses, and required handoffs
τ-Knowledge✓✓Account balances and withdrawal limits—Final state
ToolSandbox✓———Intermediate/final milestones and prohibited events
WorkArena++—✓ (L3)Budget allocation—State or sequential subtask checks
AutomationBench—✓——Final state
BizBench✓✓Shared funds across workflows✓Outcomes, continuous constraints, and protected objects

✓ denotes explicit coverage; — denotes a feature not specified in the cited paper. Business-rule retrieval means finding policies during execution. Expiring action windows means that waiting for an event or information can make an action unavailable within the simulated workflow. *Original τ²-bench release: retail/airline check final database states and required responses; telecom checks final states, with 20 of 114 tasks also requiring a handoff to a human agent.

These benchmarks cover different parts of this setting. τ-Knowledge combines customer interaction, retrieval of unstructured business knowledge, and state-changing fintech workflows, while its primary task reward is based on the final database state. ToolSandbox evaluates intermediate milestones and prohibited events, and WorkArena++ supports knowledge-base retrieval and sequential validation in enterprise web tasks. BizBench brings these requirements together in linked payment workflows with customer approvals, shared funds across workflows, delayed events, and irreversible actions. Its evaluator checks both the requested outcome and whether process and protection requirements hold over the full execution history. We will add this comparison and example to Section 2 in our final version.

Evaluator validation on model trajectories

We conducted a manual review of 100 randomly sampled trajectories, with 10 trajectories sampled from the evaluation results of each of the 10 models. The review cross-checked evaluator judgments against task requirements, tool-call logs, operation receipts, and state changes. Three human experts familiar with the task-specific rules independently reviewed the trajectories, and majority voting determined the review label. We found no case in which the verifier rejected a trajectory that the reviewers judged valid. This outcome is consistent with our construction and validation protocol: before model evaluation, all 50 task definitions in the final benchmark and their acceptance checks underwent review by business-domain experts, and every task passed reference replay, validated-alternative, and targeted-violation checks.

We did identify one false acceptance among the 100 audited trajectories, caused by a task-specific allocation requirement. In that task, the total amount of funding was correct, but the customer had earmarked separate amounts for designated operating wallets; the execution did not deliver those amounts to the required wallets. The verifier checked the aggregate funding total but omitted this allocation-level destination check, and therefore accepted the invalid execution. We added the missing check and corrected the verifier. This localized omission concerned one task-specific cross-wallet condition; it was not a general failure to evaluate execution histories or intermediate constraints.

Taken together, the audit found no observed false rejection and one false acceptance among the 100 reviewed trajectories. The false acceptance was caused by a localized, task-specific omission that has now been corrected, supporting the validity of the evaluator while showing the value of auditing actual model executions.

Blocked attempts versus executed violations

Blocked unauthorized attempts and executed violations are not equivalent. A blocked attempt is stopped before any business state changes, whereas an actual violation records an action or state path that has already breached a scored requirement. The binary Process score measures compliance with execution requirements, not the severity of resulting harm; we therefore count both as process failures when applicable but report them separately below.

  • B — blocked unauthorized attempt: the agent attempted execution without valid approval, and the tool rejected it before any transfer or posting. The task requires approval before attempting execution, so this remains a process failure even if approval is obtained later.
  • A — actual violation: an executed action, posted event, or recorded state path violated a scored process or protection requirement. For example, a balance fell below a required reserve. Such a violation can occur without a financial loss.

We examined 1,300 saved evaluations. Each trajectory is counted at most once per category, and B and A can overlap. “A involving P” counts actual process violations; the remaining A cases concern protection. Missing actions, other rejected calls, and scoring-reconstruction issues are recorded separately from A.

ModelNB, n (%)A, n (%)A involving PB ∩ A
Claude Opus 5500 (0.0)0 (0.0)00
GPT-6 Sol1500 (0.0)8 (5.3)80
GLM-5.3-Flash1502 (1.3)12 (8.0)120
DeepSeek-V4.1-Flash1501 (0.7)9 (6.0)80
DeepSeek-V4-Pro1502 (1.3)7 (4.7)70
Muse Spark 1.3500 (0.0)1 (2.0)10
Qwen3.8-27B15017 (11.3)13 (8.7)131
DeepSeek-V4-Flash15032 (21.3)26 (17.3)252
Qwen3.5-397B-A17B15028 (18.7)28 (18.7)265
Qwen3.5-9B15088 (58.7)19 (12.7)1915

For example, one DeepSeek-V4.1-Flash run attempted execution before approval and was blocked without changing the balance. Another reduced a EUR wallet to zero despite a required minimum of 1,000 minor units (EUR 10). The second run had already breached the reserve, even if funds were later restored.

For GLM-5.3-Flash, the P and Strict baselines use its completed three-run evaluation; the other baselines are the scores reported in Table 1. To measure the effect of B, we identify from the saved trajectories which failures are attributable solely to the blocked-authorization penalty. The "without B" scores remove only that contribution from the baseline, with all other requirements held fixed.

ModelP: original → without B (%)Strict: original → without B (%)Goal reached; failed Strict because of A
Claude Opus 596.0 → 96.064.0 → 64.00
GPT-6 Sol94.7 → 94.761.3 → 61.35
GLM-5.3-Flash79.3 → 80.743.3 → 44.06
DeepSeek-V4.1-Flash92.0 → 92.743.3 → 44.04
DeepSeek-V4-Pro85.3 → 86.739.3 → 40.02
Muse Spark 1.394.0 → 94.038.0 → 38.00
Qwen3.8-27B74.0 → 82.732.7 → 38.03
DeepSeek-V4-Flash53.3 → 69.35.3 → 6.02
Qwen3.5-397B-A17B46.7 → 58.70.7 → 0.70
Qwen3.5-9B18.7 → 42.70.0 → 0.00

Among the 38 runs that reached the goal but failed Strict, 12 fall into blocked authorization attempts, 22 into actual violations, and 4 into other rejected calls. These categories are mutually exclusive within this subset. Removing B raises Qwen3.8-27B’s P from 111/150 to 124/150 and Strict from 49/150 to 57/150. For GLM-5.3-Flash, removing B raises P from 119/150 to 121/150 and Strict from 65/150 to 66/150. One of the two blocked evaluations still fails Strict because its goal requirements remain unmet. For DeepSeek-V4.1-Flash and DeepSeek-V4-Pro, removing B rescues one Strict evaluation each. GPT-6 Sol has no B cases, and all five of its goal-reaching Strict failures involve actual violations. Thus, blocked attempts can affect Process without producing a business side effect or necessarily changing Strict, whereas actual violations concern requirements already breached in the realized execution. We therefore retain B and A as separate diagnostics.

Unequal numbers of trials

Claude Opus 5 is particularly expensive to evaluate on these long-horizon tasks. Given our evaluation budget, we ran it once per task and GPT-6 Sol three times. Both use the same 50 tasks and evaluation conditions. Table 1 reports success over all individual runs: 32/50 (64.0%) for Claude and 92/150 (61.3%) for GPT. Every GPT run contributes equally to its score. We interpret the 2.7 percentage-point gap as descriptive; repeated-run reliability is reported separately in Figure 2 for models with three runs per task.

We have also completed two additional runs per task for GLM-5.3-Flash, bringing its evaluation to 150 runs, with a three-run Strict success rate of 43.3% (65/150), compared with 46.0% (23/50) in the original single-run evaluation.

Figure informativeness and trajectory readability

We will enlarge the reference-trajectory figure’s labels, shorten the text inside the figure, and move individual tool-call details into its caption. The figure will emphasize the dependencies between funding, approval, and the review deadline. The appendix will retain the two execution examples discussed above: a blocked attempt before approval and a payment that breaches the reserve before the balance is restored.

Thank you again for your careful review. We hope these clarifications and additional analyses address your concerns about BizBench’s contribution, evaluation, and presentation.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.