acceptodds
Under review as a conference paper at ICLR 2027

ORGWorld: Benchmarking AI Agents as Collaborative Coworkers in Realistic Company Environments

Abstract

Can an agent carry out a full day of organizational work correctly without supervision? Such work requires an agent to infer obligations, reprioritize ongoing work as conditions change, and coordinate with colleagues under time constraints. Existing agent benchmarks provide limited evidence of whether agents can reliably fulfill organizational responsibilities under these combined demands. We introduce , a benchmark of 56 simulated company environments, each a full workday built around nine business systems, a company handbook, and 15 simulated colleagues. Each workday combines routine operational responsibilities with a cross-department project whose subtasks only colleagues can complete, requiring agents to carry company procedures through dependent steps and maintain progress across responsibilities as new events arrive. We combine expert review and solvability checks with programmatic verification of business outcomes, prohibited actions, and temporal constraints. A workday is solved only when all required responsibilities are completed and no prohibited action occurs. Across 12 frontier models, the strongest solves just 48.2% of workdays, while no other model exceeds 20%. These results reveal a substantial gap between current frontier agents and the reliability required to carry out organizational work autonomously.

Primary area:
datasets and benchmarks
Keywords:
AI coworkers, human-agent collaboration, tool use, long-horizon tasks, agentic agents, benchmark, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6qdqOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ORGWorld, which is a benchmark for evaluating AI agents on complete organizational responsibilities over a simulated workday.

The environment combines two tracks, the "assistance track" requiring agents to discover and execute business workflows, including responsibilities that arise without explicit requests, and the "coordination track" requiring agents to delegate subtasks, monitor colleagues, follow up on delays, and propagate information across dependencies under deadlines.

Evaluation relies on programmatic checks for required outcomes, intermediate states, timing constraints, prohibited actions, and unrequested modifications. The authors evaluate 12 frontier models, finding that the strongest completes only 48.2% of environments despite satisfying 92.9% of individual checks.

Critical strengths

Strengths:

S1. The paper's organization paper demands are well motivated and solid.

S2. Figure 3 which presents the taxonomy of the 56 environments is very infomrative and showcasing the domain of this benchmark.

S3. The benchmark stats seem to represent the domain.

S4. I appreciate evaluating the benchmark with frontier models

Critical weaknesses

Weaknesses:

W1. In today's world, constructing a benchmark is easier and staling fast, this is why many benchmarks are out there and no one is using them. To have a good benchmark paper, the first part should be not what you constructed, but a clear comparison emphasizing the gaps in the alternatives. See Q1. The current status in my opinion is overclaiming (no benchmark provides the similar properties) without proving it via rigorous comparison. In addition, I think that [1,2,3,4] must be cited and compared (as some benchmarks in Appendix A present stale statistics and old benchmarks) - and even more benchmarks (since its only about daily organizational tasks, its about evaluation properties as policy adherence).

W2. "whether an agent can carry out a full day of organizational work correctly without supervision." to do so we can think about SOP which is automation of existing workflow (which is clearly not in the assistance track), or about proactive agents (which has a lot of literature to refer to) - which is uncited.

W3. I'm missing rigorous root cause analysis to where the agents fails with examples. Claiming "Claude Fable 5.1, passes only 48.2% of environments" is a strong claim about the benchmark quality, but i'm expecting to get the information why didn't these frontier models succeed more. The "work that nobody announces" is not clear and should be elaborated why isnt it a benchmark simulator flaw rather than Frontier model missguidance/missunderstanding. For example the devices example, imo its shouting missguidance rather than inability, or more of proactivity dimension which is not the case in closed organization procedure.

W4. While the benchmark provides both strict Pass@1 and a relaxed Check Pass Rate, the latter treats each failed requirement, triggered violation, or unrequested modification as a single failed check, without distinguishing their severity. In realistic enterprise workflows, a minor incomplete step and a serious policy violation (e.g., exposing confidential information) may have substantially different consequences that should be represented in such simulation. In addition, its unclear whether prohibited intermediate actions that are subsequently reversed are detected by the verifier. To understand the benchmark difficulty further and to take conclusions for future development, it should clarify the coverage of violation checks and consider reporting separate metrics for task completion, policy compliance, and violation severity, rather than relying on an aggregated check pass rate.

W5. Although the evaluation was done using frontier models, the agentic task here is more complex and should be tested with agents that are suitable for enterprise workflows rather than using simple scaffold (e.g., Hermes, Manus, Cuga, OpenClaw..) powered by these frontier model. Thier harness might be the solution to all of the failures.

W6. "All models receive the same 106 tools" means context overload, the tool dicrimination task is really hard and well researched, did you try anything to facilitate this bottleneck?

I'm willing to reassess my score if you'll refer to my concerns and showcase actual agents performance on your benchmark.

[1] EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge [2] ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents [3] OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation [4] WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks

Questions for authors

Questions:

Q1. You're claiming: "Existing agent benchmarks provide limited evidence of whether agents can reliably fulfill organizational responsibilities under these combined demands.", to claim such strong claim, you must present rigorous comparison with the existing benchmarks, and it must be in main paper, as to get the motivation clear for "why did you construct the benchmark as you did", what was done differently and why and many questions that should be the standard for presenting a new benchmark -> Why is it more useful than the exsitings ones. I do see that appendix A has related work, yet its In addition "τ -bench 2 domains" they added 2 more domains throughout the years (telekom and banking), so the table is stale.

Q2. Can you please explain how does the "Violation Check" is actually done.

4 comments
AuthorsOpenReview ↗

Thank you for the related benchmarks and the root-cause request: comparison with [1–4], SOP and proactive agents, and root-cause analysis (1/2)

W1 / Q1. We agree that this comparison belongs in the main text. In the revision, we have added a comparison table to the main text (Table 1 in Section 2), covering the four benchmarks you cited [1–4], the policy-adherence benchmark τ-bench, and the workplace benchmarks cited in our paper. Each property in the table appears in some prior benchmark, but none has all of them. We have also cited and discussed [1–4], stated our contribution more precisely in light of this comparison, namely bringing these elements together within one simulated workday, and updated the description of τ-bench in Appendix A to include the telecom domain of τ²-bench and the banking domain of τ-Knowledge.

W2. Thank you; these two lines of work indeed belong in the main text. Our discussion of proactive agents (ProactiveBench), previously in Appendix A, is now in the main text (Section 2), and we have added PROBE, in which agents find and resolve bottlenecks hidden across workplace data sources. For SOP automation, we have cited SOP-Bench, where each task arrives as an explicit request together with an expert-written procedure.

Our assistance track differs from both lines of work in two ways.

  • First, in real offices most procedures live in a shared handbook, and relying only on hand-written SOPs would limit the kinds of work an agent can take on; the agent is therefore told where the 29-document company handbook is and finds the section that applies to each piece of work; for example, when a vendor asks to change its bank details, it has to find the finance rule that requires the Finance Manager to confirm a call-back first.
  • Second, part of the work is never requested by anyone, and the agent has to notice it itself, which ORGWorld shares with proactive agents. But where those settings ask the agent to judge what might help a user, here the work is a duty the company has already defined; and while in PROBE each bottleneck is resolved by exactly one action, in ORGWorld the agent must carry the work through a multi-step procedure, meeting preconditions, waiting for colleagues' replies or approvals, and recovering from mistakes, while it reprioritizes under time constraints and keeps several responsibilities moving. For example, when a due date in a record arrives, nobody reminds the agent, but the handbook says what must be done.

W3. We traced every failure of all 12 models, on both tracks, to its root cause, 1,938 in total, and grouped them into six types:

  • skipping work that was due (618, 32%), such as a new spreadsheet row that nobody announces and the agent never notices;
  • acting at the wrong time (362, 19%), such as following up too late after a colleague misses a promised time;
  • using the wrong method, so the work is not done (348, 18%), such as assigning a subtask to the wrong colleague;
  • stopping halfway (257, 13%), as in the device example below;
  • getting the content wrong (228, 12%), such as a wrong amount, date, or recipient;
  • doing something forbidden (125, 6%), such as changing a record marked Hold.

Most failures on unannounced work come from the model not acting on information it could already see. Of the 1,254 root causes on the assistance track, 452 (36%) concern unannounced work, that is, new rows in spreadsheets and due dates in records. To check whether they come from the simulator, we looked at three conditions in each case: the rule is in the handbook, the agent could see the row or the date, and at least one other model handled the same case correctly. All three hold for 179 of the 188 root causes on new rows (95%) and for 209 of the 264 on due dates (79%). The handbook also tells the agent where to look, listing these spreadsheets and asking it to check them and to keep track of the dates it sees.

We designed unannounced work to reflect something common in real offices, where colleagues sometimes update a shared file without telling anyone; ClawMark finds such unannounced changes hard for current models, with its checks on noticing them failing at 56.5%, nearly twice its overall failure rate (31.6%). We have made this explanation explicit in the revision (Appendix J).

Thank you for raising the device example (environment 122). We intended this case to test whether the agent completes a procedure written in the handbook. The HR section of the handbook asks the agent to post an announcement in #it when a new employee joins, and the IT section, which the agent also handles, asks it to assign a laptop once that announcement appears. Nine models posted the announcement and stopped, although seven of them had read the IT section, while three models and the human experts assigned the laptop. The handbook uses the same pattern when an employee leaves (post in #it, then revoke access), and 8 of the 12 models completed that case in environment 123. We have explained this example more clearly in the revision (Section 4.3 and Appendix J).

AuthorsOpenReview ↗

Severity-graded violations, how the violation checks work, results with an enterprise agent framework, and narrowing the tools (2/2)

W4 / Q2. On severity: we agree that a minor omission and a serious violation should not count the same, and we thank you for suggesting separate metrics. We graded all 419 violation checks by severity, with 202 high (for example, acting without a required approval or disclosing confidential data), 116 medium, and 101 low; about a third of the violations that models actually triggered are high severity. In the revision, Table 3 now reports task completion (the share of required checks passed), policy compliance (the share of runs with no violation), and the number of high-severity violations next to Pass@1. They rank models differently; for example, GPT-6 Astra is third on task completion but eleventh on compliance.

On violations that happen during the day but are undone before it ends: 266 of the 419 violation checks (63.5%) concern actions that cannot be undone, such as sending a message or creating a payment, so they always remain visible at the end of the day. For the other 153, we replayed all 672 runs (111,278 tool calls) step by step and evaluated every violation check again after each tool call that changes data. 5 violations appeared during a run and were undone later; none of them involved holds, disputes, approvals, payments, or confidential data, and none changes any Pass@1 result. We have added this replay analysis to the revision (Appendix F).

On how the violation checks work (Q2): a violation is an action that the handbook forbids. The 419 violation checks cover seven kinds of action: acting beyond one's authority or without a required approval (82), disclosing confidential information (80), acting on records marked Hold or Dispute (29), creating duplicate records or payments (25), messaging someone who should not receive the message (25), changing a record wrongly (91), and taking an unneeded step (87).

Each check is a fixed Python test on the state of the systems that targets one such action. It passes at the start of the day, and if the forbidden action has happened by the end of the day, it counts as one failed check. Checks on messages look at every message sent that day, including the channel it was posted in and everyone who received it, even as a copy. We also compare every record at the start and at the end of the day, and each change that nobody asked for counts as one failed check. We have described these checks in detail in the revision (Appendix F).

W5. Thank you for suggesting enterprise agent frameworks; we tested CUGA v0.3.2, IBM's open-source agent and, among the four frameworks you named, the one built for enterprise workflows. We ran it with DeepSeek-V4.1-Flash, the same model as our simple baseline (the agent loop used for every model in the paper), on 10 environments drawn at random with the seed "Dev", keeping the paper's setup as far as CUGA allows, with two main changes:

  • each new email, message, or ticket reaches CUGA as a new request, since it handles one request at a time;
  • we added a reminder tool.

CUGA passes 38.2% of the checks (144 of 377), while our baseline passes 76.6% on average over three runs; CUGA scores lower in every one of the 10 environments and passes neither track in any of them. The traces suggest that much of this gap comes from information lost between CUGA's modules, because the module that reads a handbook requirement is not the one that carries out the action. For example, when an employee asks for a password reset, the handbook requires a reply by email and an entry in the Reset Log sheet; CUGA sent the reply but dropped the log entry, while our baseline did both in all three runs, and logging an entry in a spreadsheet is the step CUGA drops most often (11 times).

CUGA is also far more expensive. Because each module makes its own model call, it makes 14 times as many model calls as our baseline (10,883 vs. 758), costing about 52 times as much and taking about 37 times as long. In this setting, the heavier harness did not reduce the failures, and we have included this comparison in the revision (Section 4.2 and Appendix I).

W6. We agree that 106 tools take up a lot of context. Tool choice does not appear to be a main source of failure, since fewer than 3.5% of tool calls return an error and none of the six failure types in W3 is about choosing the wrong tool. CUGA also lets us look at one way to narrow the tools, picking the systems relevant to each request before handling it. In 17 pieces of work, the systems it picked left out the one that the handbook required, and CUGA completed none of them, while our baseline completed 12, 15, and 16 of the same 17. Narrowing the tools therefore needs care in long workdays, where the systems a piece of work needs may only become clear later, and we have added this discussion to the revision (Appendix I).

I appreciate the effort and believe several of my concerns have been addressed, particularly the related work comparison.

Two concerns remain:

  • W5: If I get it correctely the enterprise agent experiment (The CUGA comparison) changes the execution setting to request-driven workflows, making it difficult to isolate the effect of the agent harness. The results do not necessarily generalize to other agent architectures which I still encourage checking rather ones without an harness.
  • W6: The evidence of low tool call error rate does not rule out tool selection difficulties, since successful calls may still target incorrect tools, and relevant tools may never be selected.

Therefore, I will maintain my positive score of 3.

AuthorsOpenReview ↗

Hermes in the same execution setting passes a similar number of environments; fewer tools to choose from change little

Thank you for the quick reply and for acknowledging the comparison. We ran new experiments for both remaining points; the table summarizes them (DeepSeek-V4.1-Flash, the same 10 environments, three runs each; check pass rate as mean ± standard deviation).

SettingToolsCheck pass rate (%)Environments passed (of 10)
Our baseline, all tools10676.6 ± 8.10, 1, 1
Our baseline, relevant tools only4281.3 ± 4.40, 2, 0
Hermes, tool search on (default)10682.4 ± 5.00, 0, 0
Hermes, tool search off10679.9 ± 2.22, 0, 0

W5. Agreed. We ran Hermes Agent in the same execution setting as our baseline, keeping its harness and turning off only the tools that reach outside the simulated company, such as web search and the terminal.

Hermes passes slightly more checks than our baseline but a similar number of environments, and the shares of its failure types are close to the baseline's. For example, on the assistance track, unannounced work accounts for 35% of Hermes' root causes and 37% of the baseline's. These runs suggest that a richer agent harness helps with individual steps, while the long-horizon challenge of getting every task right over a whole workday remains.

W6. Agreed. We first counted calls to clearly wrong tools: across all 672 runs of the main experiment, only 60 of the 23,187 write calls (0.26%) went to a system that none of the day's tasks uses, such as creating an invoice in QuickBooks on a day with no billing work. We then tested whether narrowing the tools helps, in two ways.

  • For each environment, we built an oracle solution that reaches full marks using only 17.6 tools on average. The oracle knew which emails and records mattered and opened them directly, while the model under evaluation has to browse its inbox, channels, and spreadsheets to discover what work has arrived. We therefore gave the model these tools plus 24.6 read-only tools on average for gathering information, such as reading emails, listing spreadsheet rows, or searching customer records. With the toolkit cut from 106 to 42 tools, the model passes slightly more checks, while the number of environments passed stays about the same.
  • Hermes has a built-in tool search for settings with many tools, on by default: the model searches for the tools it needs by name or description and then calls them. Turning tool search on or off makes no clear difference: the check pass rate differs by less than the variation between repeated runs.

In both cases, reducing the number of tools the model chooses from changes the check pass rate by a few points at most and leaves the number of environments passed about the same, which suggests that tool selection is not the main source of the failures. We will add these experiments to the paper.

Reviewer_neCZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ORGWorld evaluates agents over a simulated parallel workday in which they must discover obligations, follow an enterprise handbook, coordinate across departments, follow up with coworkers, and meet deadlines. Its 56 environments use nine business systems, 106 tools, 15 scripted coworkers, 2,157 required checks, and 419 prohibited-action checks. Twelve models are evaluated with strict complete-day success and finer-grained metrics. Expert solvability checks and targeted analyses support the benchmark, including a limited parallel-versus-isolated ablation.

Critical strengths
  1. The benchmark measures a consequential combination of proactive discovery, procedural compliance, cross-system state changes, and sustained follow-through. The checks cover ordering, deadlines, intermediate states, and prohibited actions rather than only final artifacts (Sections 2.1–2.4). Seven experts reviewed the environments and completed all 56 using the same tools; the reference scheduling checks provide additional solvability evidence (Section 2.5; Appendix B).

  2. The analysis identifies concrete gaps beyond reporting a low pass rate. The best model satisfies 92.9% of checks but passes 48.2% of complete workdays (Table 2), and unannounced obligations account for a disproportionate share of failures. The HR-to-IT example, in which most models send the handoff but fail to complete the ensuing setup, makes the follow-through issue especially clear (Sections 3.2–3.3).

Critical weaknesses
  1. The collaborative-coworker interpretation is not yet well validated beyond fixed organizational protocols. Coworkers follow four scripted response patterns, handoffs require specified fact tokens, and out-of-protocol communication is not interpreted (pp. 5, 14–18). Meanwhile, extra follow-ups are not penalized: the top models send 1.43–1.79 times the minimum requests, including 67 reminders to punctual coworkers for GPT-6 Astra (pp. 22–23). The benchmark therefore convincingly tests scheduling and protocol compliance, but the stability of rankings under semantically equivalent communication and meaningful communication costs remains uncertain. The paper acknowledges the scripted scope; the concern is the missing robustness evidence.

  2. Reliability and parallel-interference conclusions have limited repeatability evidence. Each model–environment pair is run once (p. 7); cross-environment confidence intervals do not estimate within-environment sampling variance. The parallel ablation covers only Kimi-K3 and DeepSeek-V4.1-Flash, and DeepSeek's coordination pass rate improves rather than declines under parallelism (Table 7, p. 25). Thus the observed assistance degradation is informative for those configurations, but broader conclusions about strong models or collaboration in general need restraint.

Questions for authors
  1. Do key rankings and failure categories survive semantic paraphrases of handoff tokens or reasonable changes in message placement, with business dependencies held fixed? A blind audit separating business failures from protocol-matching failures would help assess construct validity.

  2. Can the best model and one or two representative models be repeated on a paired environment subset, including isolated/parallel conditions? Please distinguish across-environment uncertainty from repeated-run variance.

  3. Would a generic obligation/dependency tracker and timer scaffold resolve much of the gap, and how does performance change when unnecessary follow-ups are recorded as a separate cost metric? These comparisons would clarify the capabilities and operational behavior measured.

2 comments
AuthorsOpenReview ↗

Thank you for the careful reading and concrete experiments: robustness to paraphrased handoffs and protocol-matching failures (1/2)

W1 / Q1. You are right that the colleagues are scripted, and the revision now states clearly (Section 5) that the coordination track tests coordination under a fixed protocol. The rules of this protocol are all written in the materials the agent receives. The CEO's brief asks it to pass each result on in the exact words it was given (Appendix B), and the handbook asks it to tag colleagues with a Slack mention and to keep follow-ups in the original thread. The agent read this handbook page in 670 of the 672 runs. In all 672 runs, 98.3% of the 2,357 messages that pass on a result from an upstream colleague carry the token word for word.

Most coordination failures start in follow-ups or in dispatching subtasks. For each of the 391 failed coordination tracks, we found the first failure that decided the outcome; it lies in follow-ups for 236 (60%), in dispatching subtasks for 119 (30%), and in passing on results, reporting to the CEO, or the project's final delivery for 36 (9%).

When we separate business failures from protocol-matching failures, only 35 of the 391 (9%) fail because a message did not follow the protocol format. In 4 of these, a token was paraphrased; the rest are a dispatch with the wrong wording or format, a missing Slack mention, a reply in the wrong thread, or a CEO report split over two messages. The other 356 fail in the work itself (follow-ups, dispatches, handoffs, or deliveries).

If paraphrased tokens are accepted, all rankings stay the same (Kendall τ = 1.0 for the main ranking and for the coordination track alone). This assumes that the later steps of a project would then succeed. If we also accept every protocol failure, τ is 0.93 for the main ranking and 0.88 for the coordination track, and the same three models stay on top. Follow-ups remain the largest failure category and dispatching the second, together still more than 90% of the failures. We have added this analysis to the revision (Appendix K).

AuthorsOpenReview ↗

Repeated runs, a generic tracker, and the cost of unnecessary follow-ups (2/2)

W2 / Q2. We repeated four models on 10 of the 56 environments, drawn at random with the seed "Dev": Claude Fable 5.1, the strongest model in the paper, plus MiMo-V2.6-Pro, DeepSeek-V4.1-Flash, and MiniMax-M3. Each model ran three times with both tracks concurrently (the main setting) and with each track alone.

For Fable 5.1, run-to-run variation is small compared with the differences between environments. Over three concurrent runs, Fable 5.1 passes both tracks in 43.3% ± 5.8 pp of the 10 environments (mean ± SD; 5/10, 4/10, 4/10), while the standard error across the 10 environments is 13.2 pp. Two environments pass in all three runs, 4 fail in all three, and 4 vary.

The table shows in how many of the 10 environments each track passes all of its checks (mean of three runs).

ModelAssistance, isolatedAssistance, concurrentCoordination, isolatedCoordination, concurrent
Claude Fable 5.16.04.79.08.7
MiMo-V2.6-Pro1.71.72.32.0
DeepSeek-V4.1-Flash4.02.31.71.7
MiniMax-M34.70.75.32.7

When the two tracks run together, performance drops overall. Of the 8 comparisons (4 models × 2 tracks), 6 drop, 2 stay the same, and none rises. The strongest model, Fable 5.1, drops on both tracks. In the submitted version, DeepSeek-V4.1-Flash's coordination pass rate rose when the tracks ran together (from 4 to 9 of the 56 environments, i.e., 7.1% to 16.1%). Repeated three times on these 10 environments, this rise does not recur (1.7 environments on average in both conditions). We have added these repeated runs to the paper (Section 4.3, Table 4, and Appendix H) and limited the claim about concurrency to the models we tested.

Q3. In our test, a generic to-do list and reminder made no clear difference. We gave our baseline harness two generic tools, a to-do list and a reminder, and changed nothing else. With DeepSeek-V4.1-Flash on the same 10 environments, the model passes 81.6% of the checks with the tools, compared with 67.9%, 77.9%, and 84.0% in three runs without them. We have added this test to the revision (Appendix I).

Beyond these two simple tools, we also tested a fuller form of tracking, an enterprise agent framework, CUGA v0.3.2 (IBM's open-source agent for enterprise workflows), whose planner tracks sub-tasks; we also gave it a reminder tool. With the same model on the same 10 environments, it passes 38.2% of the checks, against 76.6% on average for our baseline.

The traces suggest that much of this gap comes from information lost between CUGA's modules. CUGA passes each request through a chain of modules (task decomposition, plan control, API shortlisting, code planning, coding, and reflection), and the module that reads a handbook requirement is not the one that carries out the action. For example, when an employee asks for a password reset, the handbook requires a reply by email and an entry in the Reset Log sheet; CUGA sent the reply but dropped the log entry, while our baseline did both in all three runs. Logging an entry in a spreadsheet is the step CUGA drops most often, 11 times in total.

We count a follow-up as unnecessary in two cases:

  • the colleague is not yet due (they have not started, or their promised time has not passed);
  • the colleague has already delivered, but the agent still asks them to confirm or add something.

Across all 56 environments, the number of environments with more than one unnecessary follow-up is 14 for Claude Fable 5.1, 19 for Claude Opus 5.5, and 38 for GPT-6 Astra.

If the coordination track of an environment passes only when it has at most one unnecessary follow-up, the ranking changes clearly. Opus 5.5 and GPT-6 Astra, previously tied for first, drop to 58.9% (second) and 30.4% (tied for fourth), while Fable 5.1, previously third, drops only to 69.6% and becomes first. GPT-6 Astra's high pass rate thus comes with many unnecessary follow-ups, while Fable 5.1 coordinates nearly as well (50 of the 56 environments pass, against 51 for Astra) with far fewer. On the main ranking, with both tracks, Fable 5.1 is still first. We have added this cost metric to the paper (Section 4.3 and Appendix K).

Reviewer_Bb8TOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors proposed a benchmark to evaluate whether AI agents can work as employees in a simulated company environment. It addresses a key limitation of existing benchmarks that mostly evaluate individual tasks. In contrast, the real organizational work requires agents to handle multiple responsibilities, discover tasks, adapt to new events, and coordinate with colleagues. It contains 56 simulated workday environments with nine enterprise systems, 106 tools, a company handbook, and 15 simulated colleagues. Each environment contains two concurrent tracks: an assistance track, where agents handle incoming requests and discover unannounced tasks, and a coordination track, where agents assign work to colleagues, follow up, and manage deadlines.

Evaluations use programmatic checks and expert reviews to validate the benchmark and evaluate agent performance. Authors evaluated 12 frontier models on all 56 environments, and found that agents struggle with discovering unannounced work, following the correct procedures, and completing downstream steps.

Critical strengths
  • The benchmark is more realistic, combines task discovery, changing priorities, colleague coordination and deadlines into one environment. Tasks are also long horizon and require multi-step execution across different systems.

  • Evaluataion includes programmatic check of both outcome and intermmediate states, action ordering, deadline and prohibited actions

  • All environments are valid and solvable. Human experts reviewed all 56 environments and they were able to complete them using the available tools.

Critical weaknesses
  • Analysis on concurrent vs isolated runs is limited. The ablation comparing isolated and concurrent tracks is performed only for Kimi-K3 and DeepSeek-V4.1-Flash, it is unclear where same performance drops occur for stronger models.

  • Overall, authors have done meaningful benchmark quality control. However, there are several unknown questions, like how are 127 task types selected and are they representative of a real org? how realistic and complex are company policies? how diverse are the organizations?

Questions for authors
  • Can you extend the concurrent vs. isolated tracks ablation to stronger models?

  • Can you provide more details about the human validation process, including expert backgrounds, reviewers per environment, and agreement between human judgments and programmatic checks?

  • Would the release include everything needed to reproduce the exact 56 environments and their evaluation results, or just the eval environments?

1 comment
AuthorsOpenReview ↗

Thank you for the encouraging assessment and the stronger-model ablation: ablation on stronger models, how tasks and policies were built, human validation, and full release

W1 / Q1. We added the strongest model in the paper, Claude Fable 5.1, and two more models, MiMo-V2.6-Pro and MiniMax-M3, and reran DeepSeek-V4.1-Flash, all on 10 of the 56 environments drawn at random with Python's random.sample and the seed "Dev". Each model ran three times in each condition, that is, with both tracks concurrently (the setting of the paper's main experiment), the assistance track alone, and the coordination track alone. The table shows in how many of the 10 environments each track passes all of its checks (mean of three runs).

ModelAssistance, isolatedAssistance, concurrentCoordination, isolatedCoordination, concurrent
Claude Fable 5.16.04.79.08.7
MiMo-V2.6-Pro1.71.72.32.0
DeepSeek-V4.1-Flash4.02.31.71.7
MiniMax-M34.70.75.32.7

When the two tracks run together, performance drops overall. Of the 8 comparisons (4 models × 2 tracks), 6 drop, 2 stay the same, and none rises. The strongest model, Fable 5.1, drops on both tracks. We have added these repeated runs to the paper (Section 4.3, Table 4, and Appendix H).

W2. On how the task types were selected and whether they are representative: the 127 task types are routine procedures, defined in the handbook, of seven departments that almost every company has, namely finance (26), HR (24), sales (17), marketing (16), operations (16), support (16), and IT (12). We kept a task type only if its result can be checked in the systems. Each type can be mapped to a process in the Process Classification Framework of APQC, a widely used, industry-neutral list of business processes. For revenue accounting, accounts payable and expense reimbursement, recruiting, and onboarding, our task types cover every process that APQC lists.

On how realistic and complex the policies are: the handbook has 29 documents with about 18,600 words and 127 rules; 38 of the rules require waiting for a named person to approve or confirm before continuing, and 56 involve two or more departments. For example, one spending rule lets the agent approve up to USD 2,000, the department manager up to USD 10,000, and the CEO anything above that, with four exceptions (wire transfers, salaries, customer discounts, and payroll payments). As in a real company, the policies live in a shared handbook, and the agent finds the sections that apply to each piece of work, whereas each policy in τ-bench's original domains is about 1,000 words and is given in the system prompt.

Most rules follow common business controls (sources in Appendix C): the approval limits follow the COSO internal-control framework; the call-back check before changing a vendor's bank details is what the FBI and FinCEN recommend against business email compromise; and granting and revoking system access from HR records follows NIST controls AC-2 and PS-4.

On how diverse the organizations are: no two of the 56 environments have the same combination of work. On average, the assistance track of an environment contains about 7 pieces of work (3 to 14), each of one task type, and the coordination track is one cross-department project with 3 to 7 subtasks (4.6 on average). For example, one environment has a new hire to onboard; another has a sales lead that becomes a discounted order. The same task can also play out differently. For example, an expense claim needs a manager's approval in one environment and exceeds the meal limit in another. We have added these details to the revision (Appendices C and E).

Q2. The seven experts work at technology companies and research institutes. Three hold PhDs, three are PhD students, and one holds a master's degree, all in engineering. In their own jobs, they all go through routine procedures like those in the handbook, such as expense reimbursement, purchase requests, onboarding, and IT access requests; three of them also run their own companies, and some procedures in the handbook draw on how they manage them.

The three experts with PhDs reviewed all 56 environments, and each of the other four reviewed 14 environments in depth, so every environment had four reviewers. The experts' degrees, the review assignment, and the agreement below are now in Appendix B.

The experts agree with the programmatic verdict for 149 of the 154 checks (96.8%) that Claude Fable 5.1 fails. They compared each check with the handbook and with what the agent did. We will extend this review to the checks of all models.

Q3. We will release everything needed to reproduce the reported numbers, namely the 56 environment files exactly as evaluated, the handbook, the nine simulated systems, the agent harness, and the scoring code, as well as the full trajectories and per-check scores of all 12 models in the main experiment and of all ablation runs. With these, anyone can check every number in the paper without running any model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.