acceptodds
Withdrawn from ICLR 2027

WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many knowledge-work benchmarks are constructed around individual tasks, with the context needed for each task selected together with or after the task has been specified. This design measures performance on workplace-like tasks in an environment assembled for the task. When task specification guides which context is selected, the evaluation can encode task information into the environment and pre-complete part of the information-localization work that workplace performance normally requires. We introduce WorkWorlds, an evaluation infrastructure that separates organizational state from task specification. A world first fixes a revision, date, and employee seat and materializes the organizational state that employee can access; tasks are introduced only afterward. We implement WorkWorlds in a primary synthetic pharmaceutical company with 8 measured tasks across 6 employee seats, and construct additional organizational worlds. Across 192 matched evaluations, task-level curation increased evidence access by 15.9 percentage points (74.5% to 90.4%) and criterion pass by 11.2 percentage points (68.2% to 79.4%), while pass conditional on evidence access remained nearly unchanged (84.5% vs. 84.7%). Most of the measured difference occurred before the agent reached sufficient evidence.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
AI agents, agent evaluation, benchmarks, workplace tasks, organizational environments

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_hwBAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents WorkWorlds as an infrastructure for evaluating AI agents on workplace tasks. The authors argue that task-first benchmarks can make information search easier by selecting relevant context before the agent begins working. WorkWorlds addresses this by constructing an organization with its records, history, and permissions already in place. Agents are then assigned tasks within this existing workplace and must locate relevant information using the resources available to the employee they represent.

The authors implement WorkWorlds in a synthetic pharmaceutical company with eight tasks across six employee seats. They first verify that the same organizational state can be reproduced consistently and then conduct 192 matched evaluations with three agents. The experiments compare full employee-visible workspaces with task-curated subsets and show that curation increases sufficient-evidence access from 72.8% to 90.4% and criterion pass rates from 68.0% to 76.7%. Pass rates remain similar once sufficient evidence is accessed, suggesting that much of the measured difference occurs during information localization. The authors conclude that the information environment is an important part of what workplace benchmarks measure.

Critical strengths

The experimental setup is clear . The authors keep the underlying workplace and task fixed across both conditions, and they use the same agent configuration and grader while changing only the information available to the agent. The curated packets are prepared by practitioners who have not seen the rubric, which helps reduce potential bias. Sufficient-evidence access increases from 72.8% to 90.4%, with improvements across all three agents. This design helps isolate the effect of context selection and provides useful evidence about its impact on benchmark results. I think the paper is overall clear

The authors provide several checks on the reliability of their measurements. The human audit of the LLM grader shows 93.8% agreement across 240 sampled executions, and they also conduct leave-one-task-out and per-curator sensitivity analyses. The paper reports less favorable results, such as the performance decline on validation defense, and acknowledges the uncertainty associated with having only eight task clusters. These checks increase confidence in the reported findings while making their limitations clear.

Critical weaknesses

I think the paper could more clearly distinguish its contribution from prior work. APEX-Agents already constructs professional worlds before defining tasks, Workspace-Bench evaluates information search in large workspaces, and EnterpriseBench supports role-based access control. WorkWorlds adds a consistent way to construct employee-specific environments across organizational revisions and dates, but I am not sure what new evaluation capabilities this provides. The authors validate the consistency of these environments, while the main agent experiment focuses on context curation using one revision and one date. It remains unclear whether the proposed infrastructure enables evaluation insights that existing benchmarks cannot provide. I feel the novelty of WorkWorlds somewhat limited, as the experiments do not clearly demonstrate what new evaluation insights its additional features provide.

I think the experimental scope is somewhat limited. The main results come from only eight tasks in one pharmaceutical company, with 192 runs representing repeated evaluations of those tasks. For four tasks, the full workspace contains 775 files while the curated packets contain only 7–10 files. I think it will be great to test more tasks to make sure the generalization is valid.

Questions for authors

The paper identifies revision and date materialization and cross-assignment consistency as key features of WorkWorlds. Section 5.1 validates these features at the construction level, but the main agent experiment uses one revision and one date, with each task evaluated from a single employee seat. Could the authors clarify what new evaluation capabilities these features provide beyond existing benchmarks? In particular, I would like to understand whether the matched curation experiment could also be conducted using APEX-Agents or Workspace-Bench, and what WorkWorlds uniquely enables in this setting. Also, I wonder if possible to increase tasks as I think eight tasks are not enough

Reviewer_9ZLUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary
  1. WorkWorlds proposes a framework for evaluating workplace agents that constructs work environments based on organizational revision, date, and employee identity, allowing different tasks to share a consistent organizational state.
  2. It first fixes the employee-visible base environment, then adds primarily human-authored tasks and grading rubrics; the infrastructure handles environment materialization, access control, grading-information isolation, execution, and validation.
  3. The authors define 8 tasks, 6 employee seats, and 72 grading criteria in a fictional pharmaceutical company, PharmaCo, comparing the full employee-visible environment with subsets of materials selected by experts based on each task.
  4. Across 192 runs, manual curation increased sufficient-evidence access by 17.6 percentage points and criterion pass rates by 8.7 percentage points, demonstrating that how information is presented affects evaluation results.
Critical strengths

Check above.

Critical weaknesses
  1. The novelty and reuse value of the general-purpose infrastructure have not been sufficiently validated.
  2. The study demonstrates that the environment affects scores, but does not establish that the resulting benchmark is better. Higher pass rates after material curation do not imply greater evaluation validity, and a more difficult full environment does not automatically make it more realistic.
  3. Limited experimental coverage and openness make generalization and reproducibility difficult to assess. Most grading relies on an LLM judge, making it difficult for external teams to fully rerun the environment, assess task representativeness, or evaluate broader reusability.
Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.