WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks

Many knowledge-work benchmarks are constructed around individual tasks, with the context needed for each task selected together with or after the task has been specified. This design measures performance on workplace-like tasks in an environment assembled for the task. When task specification guides which context is selected, the evaluation can encode task information into the environment and pre-complete part of the information-localization work that workplace performance normally requires. We introduce WorkWorlds, an evaluation infrastructure that separates organizational state from task specification. A world first fixes a revision, date, and employee seat and materializes the organizational state that employee can access; tasks are introduced only afterward. We implement WorkWorlds in a primary synthetic pharmaceutical company with 8 measured tasks across 6 employee seats, and construct additional organizational worlds. Across 192 matched evaluations, task-level curation increased evidence access by 15.9 percentage points (74.5% to 90.4%) and criterion pass by 11.2 percentage points (68.2% to 79.4%), while pass conditional on evidence access remained nearly unchanged (84.5% vs. 84.7%). Most of the measured difference occurred before the agent reached sufficient evidence.