- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents WorkWorlds as an infrastructure for evaluating AI agents on workplace tasks. The authors argue that task-first benchmarks can make information search easier by selecting relevant context before the agent begins working. WorkWorlds addresses this by constructing an organization with its records, history, and permissions already in place. Agents are then assigned tasks within this existing workplace and must locate relevant information using the resources available to the employee they represent.
The authors implement WorkWorlds in a synthetic pharmaceutical company with eight tasks across six employee seats. They first verify that the same organizational state can be reproduced consistently and then conduct 192 matched evaluations with three agents. The experiments compare full employee-visible workspaces with task-curated subsets and show that curation increases sufficient-evidence access from 72.8% to 90.4% and criterion pass rates from 68.0% to 76.7%. Pass rates remain similar once sufficient evidence is accessed, suggesting that much of the measured difference occurs during information localization. The authors conclude that the information environment is an important part of what workplace benchmarks measure.
Critical strengths
The experimental setup is clear . The authors keep the underlying workplace and task fixed across both conditions, and they use the same agent configuration and grader while changing only the information available to the agent. The curated packets are prepared by practitioners who have not seen the rubric, which helps reduce potential bias. Sufficient-evidence access increases from 72.8% to 90.4%, with improvements across all three agents. This design helps isolate the effect of context selection and provides useful evidence about its impact on benchmark results. I think the paper is overall clear
The authors provide several checks on the reliability of their measurements. The human audit of the LLM grader shows 93.8% agreement across 240 sampled executions, and they also conduct leave-one-task-out and per-curator sensitivity analyses. The paper reports less favorable results, such as the performance decline on validation defense, and acknowledges the uncertainty associated with having only eight task clusters. These checks increase confidence in the reported findings while making their limitations clear.
Critical weaknesses
I think the paper could more clearly distinguish its contribution from prior work. APEX-Agents already constructs professional worlds before defining tasks, Workspace-Bench evaluates information search in large workspaces, and EnterpriseBench supports role-based access control. WorkWorlds adds a consistent way to construct employee-specific environments across organizational revisions and dates, but I am not sure what new evaluation capabilities this provides. The authors validate the consistency of these environments, while the main agent experiment focuses on context curation using one revision and one date. It remains unclear whether the proposed infrastructure enables evaluation insights that existing benchmarks cannot provide. I feel the novelty of WorkWorlds somewhat limited, as the experiments do not clearly demonstrate what new evaluation insights its additional features provide.
I think the experimental scope is somewhat limited. The main results come from only eight tasks in one pharmaceutical company, with 192 runs representing repeated evaluations of those tasks. For four tasks, the full workspace contains 775 files while the curated packets contain only 7–10 files. I think it will be great to test more tasks to make sure the generalization is valid.
Questions for authors
The paper identifies revision and date materialization and cross-assignment consistency as key features of WorkWorlds. Section 5.1 validates these features at the construction level, but the main agent experiment uses one revision and one date, with each task evaluated from a single employee seat. Could the authors clarify what new evaluation capabilities these features provide beyond existing benchmarks? In particular, I would like to understand whether the matched curation experiment could also be conducted using APEX-Agents or Workspace-Bench, and what WorkWorlds uniquely enables in this setting. Also, I wonder if possible to increase tasks as I think eight tasks are not enough