- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DWBench evaluates agents on retrieving enterprise information and producing Office artifacts. It contains two synthetic enterprises, 80 employees, 6,110 content units, and 474 tasks covering single-record retrieval, multi-hop reasoning, and Word, Excel, and PowerPoint generation. A shared MCP interface provides identity-scoped enterprise search, while content assertions and separate visual rules evaluate outputs. The study compares three agent harnesses and 11 main model configurations, with threshold sensitivity analyses, cross-model re-judging, and human validation. Multi-hop tasks have lower completion rates, Word tasks have higher completion rates, and harness differences depend on the model.
Critical strengths
1.The task design has practical value. DWBench combines enterprise information retrieval with Word, Excel, and PowerPoint generation. Agents must retrieve information from sources such as emails, chats, and meetings before completing office tasks. This design covers a relatively complete workflow and helps assess agents’ practical capabilities in enterprise settings. 2.The experiments are comprehensive and include validation of scoring reliability. The paper compares three agent harnesses and multiple model configurations, with additional threshold sensitivity analyses, cross-model re-evaluation, and human validation.
Critical weaknesses
1.The task-completion metric insufficiently assesses the functionality of generated Office files. Completion mainly depends on whether text extracted from an artifact satisfies content assertions. Appendix states that structural checks, including formulas and charts, do not contribute to task completion. Therefore, a high completion rate does not establish that generated files meet practical requirements, particularly for spreadsheet calculations and structure. 2.The reliability of visual scoring remains limited. Agreement between the two visual judges is low. After controlling for artifact format, the correlation between visual scores and human quality ratings is only 0.14, with a confidence interval that includes zero. Although the authors acknowledge that these scores serve only as supplementary diagnostics, these findings limit their usefulness for comparing artifact quality.
Questions for authors
- Appendix A.4.5 states that structural checks, including formulas and charts, do not contribute to task completion. Among artifacts classified as complete, how often did human reviewers identify functional or structural problems that made them unsuitable for the requested task?
- Given the low inter-judge agreement (κ = 0.29) and weak correlation with human quality ratings after controlling for artifact format (ρ = 0.14), which reported model comparisons in visual quality are also supported by human evaluation?
4 comments
Official Comment by Authors
Thank you again for your time reviewing our paper. As the discussion period closes on 18 November and we have not yet heard back, we would like to check whether you have any remaining concerns we can address. We have responded to each point you raised and revised the paper following your suggestions. We would be grateful if you could take these into account in your final assessment.
Thank you for the detailed rebuttal and the additional clarifications. I appreciate the authors' efforts to address my concerns. The clarification regarding the distinction between task completion and visual quality is helpful, and the additional human evaluation of 73 content-complete documents provides useful supporting evidence. I also appreciate the authors' acknowledgment of the limitations of automated visual quality assessment and their efforts to clarify its role in the benchmark.
However, my main concerns regarding evaluation validity remain. First, although the reported human evaluation results are encouraging, they are based on content-complete documents and do not fully establish whether the task completion metrics reliably capture the functional correctness and practical usability of generated Office artifacts across the broader benchmark. Second, the relatively weak agreement between automated visual assessments and human judgments remains a concern. While visual quality is inherently subjective, this does not fully address whether the proposed metrics can support reliable comparisons between different agents.
Overall, I recognize the practical value of DWBench and the improvements made in the rebuttal. Nevertheless, I believe that the validity and reliability of the evaluation methodology are central to the contribution of a benchmark paper, and the current evidence does not sufficiently resolve my concerns. Therefore, I will maintain my original score of 2.
Response to Reviewer XUgv’s Follow-up Concerns Regarding Evaluation Validity, Human Validation, and Visual-Quality Reliability
Thank you for the thoughtful follow-up and for recognizing the practical value of DWBench. We would like to clarify that the revised evaluation provides three complementary forms of validity evidence.
First, the human usability study covers both content-complete and content-incomplete outputs, rather than only the 73 complete artifacts. Human raters inspected 90 eligible original Office files, comprising 73 content-complete and 17 content-incomplete outputs, with 177 valid eyes-on ratings. Content-complete outputs receive substantially higher human ratings—4.46/5 across 145 ratings, compared with 3.66/5 across 32 ratings for incomplete outputs. Moreover, 61 of the 73 complete artifacts receive ratings of 4 or 5 from both raters, and none receives a rating of 2 or below from both. This directly shows that the task-completion metric distinguishes outputs that humans find more complete and practically usable.
Second, the primary content-completion evaluation has strong independent reliability. Human evaluators agree with the automatic assertion verdicts on 90.0% of all ratings and 92.3% of decisive Pass/Fail judgments (κ = 0.71). An evaluator from a different model family independently agrees on 92.2% of task-completion outcomes. For content-complete artifacts, human-corrected format verdicts also agree with the deterministic structural checks on 560 of 562 ratings. Together, these results support the reliability of the source-grounded semantic and structural evaluation used for our primary model and harness comparisons.
Third, the automated visual evaluation provides additional convergent evidence. Its two-judge mean is positively correlated with human eyes-on artifact quality (ρ = 0.34, 95% CI [0.10, 0.56]). The two independent VLM judges also reproduce the same leading model on Word—GPT-6-Astra—and the same complete ordering on PowerPoint—Claude-Opus-5, GPT-6-Astra, then Grok-4.6. They differ only on Excel, where the score intervals overlap and we therefore make no winner claim. Thus, despite differences in absolute failure counts, the principal visual comparison patterns are independently reproduced across judges on two of the three artifact formats.
DWBench deliberately uses a multi-view evaluation rather than collapsing factual correctness, structure, usability, and presentation into one opaque score. Task completion measures source-grounded semantic fulfillment; deterministic checks assess requested structure; VLM judges assess presentation; and human raters directly assess usability and evaluator correctness. The agreement observed across these independent views provides a transparent and well-supported basis for comparing digital-worker agents.
We agree that future work can further extend formula- and interaction-specific testing. However, given the direct human separation between complete and incomplete outputs, strong human and cross-model agreement on assertion judgments, near-perfect agreement on structural checks, and independently reproduced visual patterns, we respectfully hope you may reconsider whether the remaining extension opportunities warrant maintaining the original score of 2.
Follow-up to Reviewer XUgv
Thank you again for your review. We would like to check whether our response and the revised manuscript address your concerns, in particular your question on whether content-complete artifacts are usable in practice (Q1), for which we added the human ratings in Table A.17, and the visual scores (W2). If anything remains unclear, we would be glad to clarify before the public discussion closes on 18 November.