- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ORGWorld, which is a benchmark for evaluating AI agents on complete organizational responsibilities over a simulated workday.
The environment combines two tracks, the "assistance track" requiring agents to discover and execute business workflows, including responsibilities that arise without explicit requests, and the "coordination track" requiring agents to delegate subtasks, monitor colleagues, follow up on delays, and propagate information across dependencies under deadlines.
Evaluation relies on programmatic checks for required outcomes, intermediate states, timing constraints, prohibited actions, and unrequested modifications. The authors evaluate 12 frontier models, finding that the strongest completes only 48.2% of environments despite satisfying 92.9% of individual checks.
Critical strengths
Strengths:
S1. The paper's organization paper demands are well motivated and solid.
S2. Figure 3 which presents the taxonomy of the 56 environments is very infomrative and showcasing the domain of this benchmark.
S3. The benchmark stats seem to represent the domain.
S4. I appreciate evaluating the benchmark with frontier models
Critical weaknesses
Weaknesses:
W1. In today's world, constructing a benchmark is easier and staling fast, this is why many benchmarks are out there and no one is using them. To have a good benchmark paper, the first part should be not what you constructed, but a clear comparison emphasizing the gaps in the alternatives. See Q1. The current status in my opinion is overclaiming (no benchmark provides the similar properties) without proving it via rigorous comparison. In addition, I think that [1,2,3,4] must be cited and compared (as some benchmarks in Appendix A present stale statistics and old benchmarks) - and even more benchmarks (since its only about daily organizational tasks, its about evaluation properties as policy adherence).
W2. "whether an agent can carry out a full day of organizational work correctly without supervision." to do so we can think about SOP which is automation of existing workflow (which is clearly not in the assistance track), or about proactive agents (which has a lot of literature to refer to) - which is uncited.
W3. I'm missing rigorous root cause analysis to where the agents fails with examples. Claiming "Claude Fable 5.1, passes only 48.2% of environments" is a strong claim about the benchmark quality, but i'm expecting to get the information why didn't these frontier models succeed more. The "work that nobody announces" is not clear and should be elaborated why isnt it a benchmark simulator flaw rather than Frontier model missguidance/missunderstanding. For example the devices example, imo its shouting missguidance rather than inability, or more of proactivity dimension which is not the case in closed organization procedure.
W4. While the benchmark provides both strict Pass@1 and a relaxed Check Pass Rate, the latter treats each failed requirement, triggered violation, or unrequested modification as a single failed check, without distinguishing their severity. In realistic enterprise workflows, a minor incomplete step and a serious policy violation (e.g., exposing confidential information) may have substantially different consequences that should be represented in such simulation. In addition, its unclear whether prohibited intermediate actions that are subsequently reversed are detected by the verifier. To understand the benchmark difficulty further and to take conclusions for future development, it should clarify the coverage of violation checks and consider reporting separate metrics for task completion, policy compliance, and violation severity, rather than relying on an aggregated check pass rate.
W5. Although the evaluation was done using frontier models, the agentic task here is more complex and should be tested with agents that are suitable for enterprise workflows rather than using simple scaffold (e.g., Hermes, Manus, Cuga, OpenClaw..) powered by these frontier model. Thier harness might be the solution to all of the failures.
W6. "All models receive the same 106 tools" means context overload, the tool dicrimination task is really hard and well researched, did you try anything to facilitate this bottleneck?
I'm willing to reassess my score if you'll refer to my concerns and showcase actual agents performance on your benchmark.
[1] EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge [2] ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents [3] OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation [4] WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks
Questions for authors
Questions:
Q1. You're claiming: "Existing agent benchmarks provide limited evidence of whether agents can reliably fulfill organizational responsibilities under these combined demands.", to claim such strong claim, you must present rigorous comparison with the existing benchmarks, and it must be in main paper, as to get the motivation clear for "why did you construct the benchmark as you did", what was done differently and why and many questions that should be the standard for presenting a new benchmark -> Why is it more useful than the exsitings ones. I do see that appendix A has related work, yet its In addition "τ -bench 2 domains" they added 2 more domains throughout the years (telekom and banking), so the table is stale.
Q2. Can you please explain how does the "Violation Check" is actually done.
4 comments
Severity-graded violations, how the violation checks work, results with an enterprise agent framework, and narrowing the tools (2/2)
W4 / Q2. On severity: we agree that a minor omission and a serious violation should not count the same, and we thank you for suggesting separate metrics. We graded all 419 violation checks by severity, with 202 high (for example, acting without a required approval or disclosing confidential data), 116 medium, and 101 low; about a third of the violations that models actually triggered are high severity. In the revision, Table 3 now reports task completion (the share of required checks passed), policy compliance (the share of runs with no violation), and the number of high-severity violations next to Pass@1. They rank models differently; for example, GPT-6 Astra is third on task completion but eleventh on compliance.
On violations that happen during the day but are undone before it ends: 266 of the 419 violation checks (63.5%) concern actions that cannot be undone, such as sending a message or creating a payment, so they always remain visible at the end of the day. For the other 153, we replayed all 672 runs (111,278 tool calls) step by step and evaluated every violation check again after each tool call that changes data. 5 violations appeared during a run and were undone later; none of them involved holds, disputes, approvals, payments, or confidential data, and none changes any Pass@1 result. We have added this replay analysis to the revision (Appendix F).
On how the violation checks work (Q2): a violation is an action that the handbook forbids. The 419 violation checks cover seven kinds of action: acting beyond one's authority or without a required approval (82), disclosing confidential information (80), acting on records marked Hold or Dispute (29), creating duplicate records or payments (25), messaging someone who should not receive the message (25), changing a record wrongly (91), and taking an unneeded step (87).
Each check is a fixed Python test on the state of the systems that targets one such action. It passes at the start of the day, and if the forbidden action has happened by the end of the day, it counts as one failed check. Checks on messages look at every message sent that day, including the channel it was posted in and everyone who received it, even as a copy. We also compare every record at the start and at the end of the day, and each change that nobody asked for counts as one failed check. We have described these checks in detail in the revision (Appendix F).
W5. Thank you for suggesting enterprise agent frameworks; we tested CUGA v0.3.2, IBM's open-source agent and, among the four frameworks you named, the one built for enterprise workflows. We ran it with DeepSeek-V4.1-Flash, the same model as our simple baseline (the agent loop used for every model in the paper), on 10 environments drawn at random with the seed "Dev", keeping the paper's setup as far as CUGA allows, with two main changes:
- each new email, message, or ticket reaches CUGA as a new request, since it handles one request at a time;
- we added a reminder tool.
CUGA passes 38.2% of the checks (144 of 377), while our baseline passes 76.6% on average over three runs; CUGA scores lower in every one of the 10 environments and passes neither track in any of them. The traces suggest that much of this gap comes from information lost between CUGA's modules, because the module that reads a handbook requirement is not the one that carries out the action. For example, when an employee asks for a password reset, the handbook requires a reply by email and an entry in the Reset Log sheet; CUGA sent the reply but dropped the log entry, while our baseline did both in all three runs, and logging an entry in a spreadsheet is the step CUGA drops most often (11 times).
CUGA is also far more expensive. Because each module makes its own model call, it makes 14 times as many model calls as our baseline (10,883 vs. 758), costing about 52 times as much and taking about 37 times as long. In this setting, the heavier harness did not reduce the failures, and we have included this comparison in the revision (Section 4.2 and Appendix I).
W6. We agree that 106 tools take up a lot of context. Tool choice does not appear to be a main source of failure, since fewer than 3.5% of tool calls return an error and none of the six failure types in W3 is about choosing the wrong tool. CUGA also lets us look at one way to narrow the tools, picking the systems relevant to each request before handling it. In 17 pieces of work, the systems it picked left out the one that the handbook required, and CUGA completed none of them, while our baseline completed 12, 15, and 16 of the same 17. Narrowing the tools therefore needs care in long workdays, where the systems a piece of work needs may only become clear later, and we have added this discussion to the revision (Appendix I).
I appreciate the effort and believe several of my concerns have been addressed, particularly the related work comparison.
Two concerns remain:
- W5: If I get it correctely the enterprise agent experiment (The CUGA comparison) changes the execution setting to request-driven workflows, making it difficult to isolate the effect of the agent harness. The results do not necessarily generalize to other agent architectures which I still encourage checking rather ones without an harness.
- W6: The evidence of low tool call error rate does not rule out tool selection difficulties, since successful calls may still target incorrect tools, and relevant tools may never be selected.
Therefore, I will maintain my positive score of 3.
Hermes in the same execution setting passes a similar number of environments; fewer tools to choose from change little
Thank you for the quick reply and for acknowledging the comparison. We ran new experiments for both remaining points; the table summarizes them (DeepSeek-V4.1-Flash, the same 10 environments, three runs each; check pass rate as mean ± standard deviation).
| Setting | Tools | Check pass rate (%) | Environments passed (of 10) |
|---|---|---|---|
| Our baseline, all tools | 106 | 76.6 ± 8.1 | 0, 1, 1 |
| Our baseline, relevant tools only | 42 | 81.3 ± 4.4 | 0, 2, 0 |
| Hermes, tool search on (default) | 106 | 82.4 ± 5.0 | 0, 0, 0 |
| Hermes, tool search off | 106 | 79.9 ± 2.2 | 2, 0, 0 |
W5. Agreed. We ran Hermes Agent in the same execution setting as our baseline, keeping its harness and turning off only the tools that reach outside the simulated company, such as web search and the terminal.
Hermes passes slightly more checks than our baseline but a similar number of environments, and the shares of its failure types are close to the baseline's. For example, on the assistance track, unannounced work accounts for 35% of Hermes' root causes and 37% of the baseline's. These runs suggest that a richer agent harness helps with individual steps, while the long-horizon challenge of getting every task right over a whole workday remains.
W6. Agreed. We first counted calls to clearly wrong tools: across all 672 runs of the main experiment, only 60 of the 23,187 write calls (0.26%) went to a system that none of the day's tasks uses, such as creating an invoice in QuickBooks on a day with no billing work. We then tested whether narrowing the tools helps, in two ways.
- For each environment, we built an oracle solution that reaches full marks using only 17.6 tools on average. The oracle knew which emails and records mattered and opened them directly, while the model under evaluation has to browse its inbox, channels, and spreadsheets to discover what work has arrived. We therefore gave the model these tools plus 24.6 read-only tools on average for gathering information, such as reading emails, listing spreadsheet rows, or searching customer records. With the toolkit cut from 106 to 42 tools, the model passes slightly more checks, while the number of environments passed stays about the same.
- Hermes has a built-in tool search for settings with many tools, on by default: the model searches for the tools it needs by name or description and then calls them. Turning tool search on or off makes no clear difference: the check pass rate differs by less than the variation between repeated runs.
In both cases, reducing the number of tools the model chooses from changes the check pass rate by a few points at most and leaves the number of environments passed about the same, which suggests that tool selection is not the main source of the failures. We will add these experiments to the paper.
Thank you for the related benchmarks and the root-cause request: comparison with [1–4], SOP and proactive agents, and root-cause analysis (1/2)
W1 / Q1. We agree that this comparison belongs in the main text. In the revision, we have added a comparison table to the main text (Table 1 in Section 2), covering the four benchmarks you cited [1–4], the policy-adherence benchmark τ-bench, and the workplace benchmarks cited in our paper. Each property in the table appears in some prior benchmark, but none has all of them. We have also cited and discussed [1–4], stated our contribution more precisely in light of this comparison, namely bringing these elements together within one simulated workday, and updated the description of τ-bench in Appendix A to include the telecom domain of τ²-bench and the banking domain of τ-Knowledge.
W2. Thank you; these two lines of work indeed belong in the main text. Our discussion of proactive agents (ProactiveBench), previously in Appendix A, is now in the main text (Section 2), and we have added PROBE, in which agents find and resolve bottlenecks hidden across workplace data sources. For SOP automation, we have cited SOP-Bench, where each task arrives as an explicit request together with an expert-written procedure.
Our assistance track differs from both lines of work in two ways.
W3. We traced every failure of all 12 models, on both tracks, to its root cause, 1,938 in total, and grouped them into six types:
Most failures on unannounced work come from the model not acting on information it could already see. Of the 1,254 root causes on the assistance track, 452 (36%) concern unannounced work, that is, new rows in spreadsheets and due dates in records. To check whether they come from the simulator, we looked at three conditions in each case: the rule is in the handbook, the agent could see the row or the date, and at least one other model handled the same case correctly. All three hold for 179 of the 188 root causes on new rows (95%) and for 209 of the 264 on due dates (79%). The handbook also tells the agent where to look, listing these spreadsheets and asking it to check them and to keep track of the dates it sees.
We designed unannounced work to reflect something common in real offices, where colleagues sometimes update a shared file without telling anyone; ClawMark finds such unannounced changes hard for current models, with its checks on noticing them failing at 56.5%, nearly twice its overall failure rate (31.6%). We have made this explanation explicit in the revision (Appendix J).
Thank you for raising the device example (environment 122). We intended this case to test whether the agent completes a procedure written in the handbook. The HR section of the handbook asks the agent to post an announcement in #it when a new employee joins, and the IT section, which the agent also handles, asks it to assign a laptop once that announcement appears. Nine models posted the announcement and stopped, although seven of them had read the IT section, while three models and the human experts assigned the laptop. The handbook uses the same pattern when an employee leaves (post in #it, then revoke access), and 8 of the 12 models completed that case in environment 123. We have explained this example more clearly in the revision (Section 4.3 and Appendix J).