acceptodds
Under review as a conference paper at ICLR 2027

DWBench: Evaluating AI Agents as Digital Workers over Enterprise Graph Data

Abstract

Enterprise knowledge work is broader than writing code: a digital worker must locate the right thread in a noisy inbox, reconcile facts scattered across emails, files, meetings, and chats, then produce a report, spreadsheet, or slide deck. We present DWBench, an agent-interface-agnostic benchmark for grounded retrieval and Office-artifact generation over ego-centric, permission-scoped enterprise graphs. We release two synthetic enterprise tenants (biotechnology and financial advisory) comprising 80 acting identities and 6,110 content units across email, chat, meetings, and 15–16 file types per tenant; 474 access-safe tasks spanning single-fact retrieval, multi-hop reasoning, Word, Excel, and PowerPoint production; content assertions linked to supporting records; and a common Model Context Protocol environment. We evaluate GitHub Copilot (GHCP), OpenCode, as well as a reference agent (RefAgent) across 13 model settings. The experiments comprise 15K executions across three harnesses, two tenants, and five task types using frontier models, including GPT-6-Astra, GPT-5.6-Sol, and Opus-5. They include paired-harness comparisons and model-scaling studies. Results show that mean assertion pass rate varies with the model, harness, and task type: Word generation attains mean assertion pass rates of 87.8–92.8%, whereas multi-hop search remains a bottleneck at 54.9–60.1%. GHCP achieves the highest mean assertion pass rate, exceeding OpenCode and RefAgent by 1.5 and 2.2 percentage points, respectively, while significant GHCP advantages over OpenCode in task completion rate at the 0.75 assertion threshold (a query is complete when at least 75% of its assertions are satisfied) are concentrated in specific models. Our findings provide reproducible diagnostics of tool-trace depth, model scaling, and evaluation robustness across enterprise environments.

Primary area:
datasets and benchmarks
Keywords:
enterprise agents, benchmarks, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XUgvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DWBench evaluates agents on retrieving enterprise information and producing Office artifacts. It contains two synthetic enterprises, 80 employees, 6,110 content units, and 474 tasks covering single-record retrieval, multi-hop reasoning, and Word, Excel, and PowerPoint generation. A shared MCP interface provides identity-scoped enterprise search, while content assertions and separate visual rules evaluate outputs. The study compares three agent harnesses and 11 main model configurations, with threshold sensitivity analyses, cross-model re-judging, and human validation. Multi-hop tasks have lower completion rates, Word tasks have higher completion rates, and harness differences depend on the model.

Critical strengths

1.The task design has practical value. DWBench combines enterprise information retrieval with Word, Excel, and PowerPoint generation. Agents must retrieve information from sources such as emails, chats, and meetings before completing office tasks. This design covers a relatively complete workflow and helps assess agents’ practical capabilities in enterprise settings. 2.The experiments are comprehensive and include validation of scoring reliability. The paper compares three agent harnesses and multiple model configurations, with additional threshold sensitivity analyses, cross-model re-evaluation, and human validation.

Critical weaknesses

1.The task-completion metric insufficiently assesses the functionality of generated Office files. Completion mainly depends on whether text extracted from an artifact satisfies content assertions. Appendix states that structural checks, including formulas and charts, do not contribute to task completion. Therefore, a high completion rate does not establish that generated files meet practical requirements, particularly for spreadsheet calculations and structure. 2.The reliability of visual scoring remains limited. Agreement between the two visual judges is low. After controlling for artifact format, the correlation between visual scores and human quality ratings is only 0.14, with a confidence interval that includes zero. Although the authors acknowledge that these scores serve only as supplementary diagnostics, these findings limit their usefulness for comparing artifact quality.

Questions for authors
  1. Appendix A.4.5 states that structural checks, including formulas and charts, do not contribute to task completion. Among artifacts classified as complete, how often did human reviewers identify functional or structural problems that made them unsuitable for the requested task?
  2. Given the low inter-judge agreement (κ = 0.29) and weak correlation with human quality ratings after controlling for artifact format (ρ = 0.14), which reported model comparisons in visual quality are also supported by human evaluation?
4 comments
AuthorsOpenReview ↗

Follow-up to Reviewer XUgv

Thank you again for your review. We would like to check whether our response and the revised manuscript address your concerns, in particular your question on whether content-complete artifacts are usable in practice (Q1), for which we added the human ratings in Table A.17, and the visual scores (W2). If anything remains unclear, we would be glad to clarify before the public discussion closes on 18 November.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your time reviewing our paper. As the discussion period closes on 18 November and we have not yet heard back, we would like to check whether you have any remaining concerns we can address. We have responded to each point you raised and revised the paper following your suggestions. We would be grateful if you could take these into account in your final assessment.

Thank you for the detailed rebuttal and the additional clarifications. I appreciate the authors' efforts to address my concerns. The clarification regarding the distinction between task completion and visual quality is helpful, and the additional human evaluation of 73 content-complete documents provides useful supporting evidence. I also appreciate the authors' acknowledgment of the limitations of automated visual quality assessment and their efforts to clarify its role in the benchmark.

However, my main concerns regarding evaluation validity remain. First, although the reported human evaluation results are encouraging, they are based on content-complete documents and do not fully establish whether the task completion metrics reliably capture the functional correctness and practical usability of generated Office artifacts across the broader benchmark. Second, the relatively weak agreement between automated visual assessments and human judgments remains a concern. While visual quality is inherently subjective, this does not fully address whether the proposed metrics can support reliable comparisons between different agents.

Overall, I recognize the practical value of DWBench and the improvements made in the rebuttal. Nevertheless, I believe that the validity and reliability of the evaluation methodology are central to the contribution of a benchmark paper, and the current evidence does not sufficiently resolve my concerns. Therefore, I will maintain my original score of 2.

AuthorsOpenReview ↗

Response to Reviewer XUgv’s Follow-up Concerns Regarding Evaluation Validity, Human Validation, and Visual-Quality Reliability

Thank you for the thoughtful follow-up and for recognizing the practical value of DWBench. We would like to clarify that the revised evaluation provides three complementary forms of validity evidence.

First, the human usability study covers both content-complete and content-incomplete outputs, rather than only the 73 complete artifacts. Human raters inspected 90 eligible original Office files, comprising 73 content-complete and 17 content-incomplete outputs, with 177 valid eyes-on ratings. Content-complete outputs receive substantially higher human ratings—4.46/5 across 145 ratings, compared with 3.66/5 across 32 ratings for incomplete outputs. Moreover, 61 of the 73 complete artifacts receive ratings of 4 or 5 from both raters, and none receives a rating of 2 or below from both. This directly shows that the task-completion metric distinguishes outputs that humans find more complete and practically usable.

Second, the primary content-completion evaluation has strong independent reliability. Human evaluators agree with the automatic assertion verdicts on 90.0% of all ratings and 92.3% of decisive Pass/Fail judgments (κ = 0.71). An evaluator from a different model family independently agrees on 92.2% of task-completion outcomes. For content-complete artifacts, human-corrected format verdicts also agree with the deterministic structural checks on 560 of 562 ratings. Together, these results support the reliability of the source-grounded semantic and structural evaluation used for our primary model and harness comparisons.

Third, the automated visual evaluation provides additional convergent evidence. Its two-judge mean is positively correlated with human eyes-on artifact quality (ρ = 0.34, 95% CI [0.10, 0.56]). The two independent VLM judges also reproduce the same leading model on Word—GPT-6-Astra—and the same complete ordering on PowerPoint—Claude-Opus-5, GPT-6-Astra, then Grok-4.6. They differ only on Excel, where the score intervals overlap and we therefore make no winner claim. Thus, despite differences in absolute failure counts, the principal visual comparison patterns are independently reproduced across judges on two of the three artifact formats.

DWBench deliberately uses a multi-view evaluation rather than collapsing factual correctness, structure, usability, and presentation into one opaque score. Task completion measures source-grounded semantic fulfillment; deterministic checks assess requested structure; VLM judges assess presentation; and human raters directly assess usability and evaluator correctness. The agreement observed across these independent views provides a transparent and well-supported basis for comparing digital-worker agents.

We agree that future work can further extend formula- and interaction-specific testing. However, given the direct human separation between complete and incomplete outputs, strong human and cross-model agreement on assertion judgments, near-perfect agreement on structural checks, and independently reproduced visual patterns, we respectfully hope you may reconsider whether the remaining extension opportunities warrant maintaining the original score of 2.

Reviewer_pD6LOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DWBench, a benchmark and evaluation platform for assessing AI agents as digital workers over ego-centric, permission-scoped enterprise knowledge graphs. The core scenario is a "search-then-produce" loop: an agent must retrieve grounded facts from heterogeneous enterprise data (emails, files, meetings, chat) and synthesise them into a textual response or an editable Office artifact (Word, Excel, PowerPoint). The paper contributes: (1) a formalisation of this evaluation setting along six dimensions absent in combination from prior benchmarks; (2) an agent-agnostic MCP enterprise-search environment, a two-stage data-generation pipeline yielding two synthetic tenants (BioNova, Ashcroft) with 474 tasks and machine-checkable content assertions, and a multi-view evaluator combining content assertion scoring and visual presentation judges; (3) a 15K-execution empirical study comparing three agent harnesses (GitHub Copilot, OpenCode, RefAgent) across 11 model-effort configurations using frontier models (Claude-Opus-5, GPT-6-Astra, Grok-4.6). Results show that Word generation is strongest (87.8–92.8% mean assertion pass rate) while multi-hop retrieval remains the bottleneck (54.9–60.1%), and that GHCP edges out OpenCode and RefAgent overall.

Critical strengths

Strength 1: Agent-agnostic MCP environment with principled permission scoping. Publishing the enterprise-search environment as a single MCP server with identity-bound access controls is the paper's strongest technical contribution. It separates the evaluation environment cleanly from the agent under test, ensures a fair common interface, and models a real constraint (ego-centric communication scoping: only 8.1–13.3% of the communications graph is visible per user) that prior benchmarks ignore. This design enables matched comparisons of commercial and open-source harnesses that are otherwise difficult to achieve, and the community would benefit from adopting it.

Strength 2: Scale and multi-view evaluation - Running 15K executions across 3 harnesses, 2 tenants, 5 task types, and 11 model configurations, and combining content assertion scoring, threshold sensitivity analysis, visual presentation evaluation, human validation, and inter-model re-judging in one study is ambitious and thorough. Most prior agent benchmarks rely on a single evaluator. The multi-view approach increases confidence that the results are not artefacts of a single evaluation design choice.

Critical weaknesses

Weakness 1:There is a fundamental reproducibility gap due to GHCP. This emanates from three consequential challenges with the soundness of the paper as follows:

a) Reproducibility of GHCP results. GitHub Copilot (GHCP v1.0.81-14) is a proprietary, cloud-hosted commercial system that can change behaviour without notice. Results attributed to GHCP, which include its aggregate 1.5–2.2 percentage-point advantage over other harnesses, cannot be reproduced by independent researchers at a fixed point in time. For a benchmark paper whose primary purpose is to enable community replication and comparison, this is a fundamental soundness issue.

b) LLM-generated tasks and assertions, judged by an LLM. The task authoring pipeline uses an LLM to write tasks and assertions from tenant materials, and a separate LLM judge (GPT-5.4 mini) to evaluate outputs. While the gold/bluff calibration step (Appendix A.3) and the 92.2% inter-model agreement between GPT-5.4-mini and Haiku-4.5 are positive, the entire measurement chain rests on LLM reliability. Systematic biases shared between the authoring model and the judge model could inflate or deflate task difficulty in ways that are hard to detect. No human assessment of task quality or assertion precision/recall is reported in the main text.

c) Task set size and tenant coverage. With 474 tasks spread across 5 task types and 2 tenants, individual task-type buckets are small (e.g., 34 Word tasks per tenant). The large standard deviations in Table 2 reflect this, and confidence intervals on completion rates will be wide. The two tenants (biotech and financial advisory) are both synthetically generated and do not characterise how well the benchmark represents the diversity of real enterprise deployments. Generalisability claims are consequently limited.

From these points, the inclusion of a closed, versioned commercial system (GHCP v1.0.81-14) as one of the three primary evaluation arms makes a significant portion of the results irreproducible. GHCP achieves the highest mean assertion pass rate in the paper, making this not a peripheral concern. A benchmark paper at ICLR should either (a) restrict its primary harness comparison to reproducible systems, or (b) provide a detailed protocol for accessing GHCP at a specific versioned snapshot. Without this, the harness comparison results cannot be verified by PC members, reviewers, or the community.

Weakness 2: The findings are largely confirmatory with limited contribution. The paper claims its main empirical findings that word generation is easiest, multi-hop retrieval is hardest, larger models outperform smaller ones, and GHCP marginally beats open-source harnesses. These do not constitute new scientific knowledge about agent behaviour. They are expected given the task design and the model capability literature. For a benchmark paper to meet the ICLR bar, it should either reveal something unexpected or enable future work to test specific hypotheses. DWBench's contribution is closer to infrastructure provisioning than to scientific insight. Deeper failure analysis (why does multi-hop retrieval fail? is it search recall, reasoning, or assertion phrasing?) would significantly strengthen the paper's scientific contribution.

Questions for authors

Background The benchmark overview in Figure 1 is effective, and the three-component framework (environment/data pipeline/evaluator) is clearly structured. However, substantial methodological detail is deferred to appendices, including the assertion authoring process (Appendix A.3), the MCP implementation (Appendix A.4), and the visual judge protocol (Appendix A.6). Reviewers cannot fully assess soundness from the main text alone. Additionally, the paper's claim of three contributions does not list the third contribution explicitly in the main text, and as such, it must be inferred from the experiments section. Table 2 is well-designed, but the associated textual discussion of the results is brief given the complexity of the model × harness × tenant × task-type space.

The benchmark design itself — particularly the MCP-based agent-agnostic interface and the gold/bluff assertion validation procedure — is genuinely useful and informs how future enterprise agent benchmarks should be designed. The task completion rates and per-type assertion pass rates across frontier models provide a concrete snapshot of where current agents succeed and fail. However, the headline results (multi-hop is harder, larger models do better, commercial harnesses edge out open-source ones) are unsurprising and do not significantly deepen scientific understanding. The paper would be considerably more informative if the failure analysis i.e. why multi-hop at 54.9–60.1% remains a bottleneck, were carried out and reported in the main text.

My questions to the authors

  1. Reproducibility: GHCP is a closed commercial system that changes without notice. How do you plan to ensure that GHCP results reported in Table 2 can be reproduced by a third party? Is the exact GHCP snapshot archived and accessible?

  2. Soundness Challenge: The task and assertion authoring pipeline relies on an LLM, and evaluation uses an LLM judge. To what extent have you characterised task quality independently of the authoring model? Specifically: what is the human assessment of task answerability and assertion precision/recall, and how many tasks were filtered out or corrected after LLM authoring?

  3. Clarification: Multi-hop retrieval achieves a 54.9–60.1% assertion pass rate, the lowest of all task types. Is this bottleneck attributable primarily to (a) search recall failure (the agent does not find all necessary records), (b) reasoning failure (the agent finds records but fails to correctly reconcile them), or (c) assertion phrasing artefacts (assertions are harder to satisfy even with correct answers)? A breakdown would substantially sharpen the scientific contribution.

  4. Reproducibility: The frontier models used (GPT-5.4–6-Astra, Claude-Opus-4.7–5, Grok-4.5–4.6) are API-accessed commercial models. How do you handle the fact that these model versions are not static and may already differ from those used during the study? Is there a protocol for re-running the benchmark against future model versions in a comparable way?

  5. Clarification: The paper lists three contributions in the introduction, but only two are explicitly enumerated. What is the third contribution, and why is it not stated as clearly as the first two?

2 comments
AuthorsOpenReview ↗

Follow-up to Reviewer pD6L

Thank you again for your review. We would like to check whether our response and the revised manuscript address your concerns, in particular the multi-hop failure breakdown you suggested (Q3), now in the main text, and the reproducibility of the GHCP results (W1a). If anything remains unclear, we would be glad to clarify before the public discussion closes on 18 November.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your time reviewing our paper. As the discussion period closes on 18 November and we have not yet heard back, we would like to check whether you have any remaining concerns we can address. We have responded to each point you raised and revised the paper following your suggestions. We would be grateful if you could take these into account in your final assessment.

Reviewer_fvPDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DWBench which is benchmark that simulates enterprise interactions and workflows for two synthetic companies Ashcroft and Bionova. It uses a synthetic data generator based on LLMs with human validation of some samples tasks. The tasks combine personal information, data retrieval from multiple data sources and artifact generation (like excel sheet, word and powerpoint).

The authors have shown where models fail and succeed accross ablation studies related to long horizon tasks and number of file type across three harnesses GitHub Copilot (GHCP), OpenCode, and a reference agent (RefAgent)

Critical strengths
  • The benchmark includes an interesting mix of deep research and artifact generation which extends nicely to some existing benchmarks that only do deep research DRBench.
  • the paper is well-written and has a comprehensive set of ablation studies that allows us to look at how the harnesses perform across some different confounding factors.
  • i like the simulation idea the authors proposed to generate interactions in an enterprise as that can help create more realistic long horizon tasks then if the workflows are generated without such simulator.
  • the permission-based data access is a nice addition.
Critical weaknesses
  • There is no public web search which is important for creating artifacts that require web search. This is something that is present in DRBench but not here and i think it is an important part of a holistic enterprise benchmarks.
  • It is not clear whether the generated synthetic examples realistically represent enterprise workflows. There needs to be some actual held-out real world usecases where expert humans would look at the generated artifacts and judge whether they are useful.
  • the explanation in the way the data is generated is incomplete. How was each user represented and what are the llm prompts used to generate the data accross a simulated timeline in an enterprise? there is no code nor a step-by-step explanation of the data generation process -Figure 1 doesn't look nice and is not easy to follow. For example "representative request" is only shown in the figure and is not explained anywhere else, what does representative request mean in this case?
  • The contribution is limited in that the benchmark basically combines different aspects of what makes an enterprise task from existing benchmarks. It is not clear what new insights this paper adds that existing benchmarks do not have. The results showing where the harnesses fail are obvious - for example, more file types in a task leads to worse model performance and that has been shown multiple times in existing benchmarks. So the question becomes, what does this paper tell us about how we can improve existing agents so that they perform better in the real world.
  • There are many ablation studies that are missing which are necessary to isolate confounding factors. For example it is not clear whether performance is being hurt by eithe retrieval or reasoning, or which parts of the harness is the actual bottleneck - is it the planning, the tool execution or something else? Why is it important to have permission scoping in the tasks, what kind of challenge does it add? since the simulated workflows have some temporal aspect where information can get updated over time, how do agents perform when new content get updated or when conflicting information start to appear?
  • I found the results in Table 2 to have very high variance across tasks, this makes me wonder whether it is because there are just so many ways we can create an artifact.
Questions for authors

Please see the weaknesses above

2 comments
AuthorsOpenReview ↗

Follow-up to Reviewer fvPD

Thank you again for your review. We would like to check whether our response and the revised manuscript address your concerns, in particular the findings beyond aggregate rankings and the trace analysis of where agents fail (W4, W5), and the step-by-step account of data generation (W3). If anything remains unclear, we would be glad to clarify before the public discussion closes on 18 November.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your time reviewing our paper. As the discussion period closes on 18 November and we have not yet heard back, we would like to check whether you have any remaining concerns we can address. We have responded to each point you raised and revised the paper following your suggestions. We would be grateful if you could take these into account in your final assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Note to the Area Chair

Dear Area Chair,

Thank you for handling our submission. We have posted our response and the revised manuscript and have not yet heard back from the reviewers. As the public discussion closes on 18 November, we would be grateful if you could encourage them to take a look.

For convenience, this is where the revision addresses the main concerns in the reviews:

  • Findings beyond aggregate rankings (fvPD W4, pD6L W2). Harness sensitivity depends on the model family: across the three harnesses, task completion varies by 5.7–11.8 points for every GPT setting but by at most 3.2 points for every Claude and Grok setting (Table B.2). In mean assertion pass rate, Opus-5 leads GPT-6-Astra on single-fact retrieval in all six tenant–harness cells yet trails it on multi-hop in five (Table 2).
  • Why multi-hop fails (pD6L Q3, fvPD W5). The failure analysis that Reviewer pD6L asked for is now in the main text: across 4,323 multi-hop runs, 77.8% of non-completions never surface all supporting records (§4.2, Table C.2).
  • Reproducibility of the GHCP arm (pD6L W1a). The main conclusions do not rely on GHCP. With RefAgent and OpenCode alone, harness gaps still reach 11.4 points for GPT settings against at most 3.2 for Claude and Grok, and the Opus-5/GPT-6-Astra reversal remains in three of the four cells (see our response to W1a). All run trajectories will be released with the code so that every reported result can be re-scored.
  • Validity of LLM-authored tasks and of the content metric (pD6L W1b, XUgv W1 and Q1, fvPD W2). The blinded human study is now more prominent in §4.1. In the new Table A.17, none of the 73 content-complete artifacts is rated 2 or below by both raters.
  • Transparency (fvPD W3, pD6L Q2). §2.2 and Appendices A.3 and A.4.1 give the generation workflow, prompt skeletons and task-filtering counts.

We are happy to answer any further questions.

Best regards,
The authors

AuthorsOpenReview ↗

Please help to encourage reviewers for discussion

Dear ACs, SACs, and PCs,

Hope this message finds you well. Many thanks for your efforts in organizing review. We have responded to reviewers and provided the revised paper for more than five days. Reviewers haven’t give us any feedback yet. Could you please encourage them to respond us? We really appreciate the chance to directly discuss with reviewers! Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.