acceptodds
Under review as a conference paper at ICLR 2027

OfficeBench-Pro: Benchmarking Agents for Professional Office Work

Abstract

Professional Office work requires factual accuracy, analytical soundness, and aesthetic quality to jointly serve a workplace objective, yet existing benchmarks only partially capture their interdependence. We introduce , comprising expert-authored and synthesized tasks across professional domains. Agents create or revise Office deliverables from heterogeneous sources, with native structures essential to analysis or delivery. Our synthesis framework combines source-grounded design, execution-guided evolution, and independent task and rubric review. We adapt Agent-as-a-Judge to trace source evidence, test required native behavior, and visually inspect rendered outputs. Across six models under a shared execution protocol, the highest overall scores are 51.3% on expert tasks and 50.8% on synthesized tasks; no model leads all three quality dimensions in either collection. Qualitative analyses of archived submissions reveal unsupported premises propagated across deliverables, broken calculation dependencies, and visual organization that obscures analytical relationships.

Primary area:
datasets and benchmarks
Keywords:
Agent, Agent-as-a-Judge, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1SZhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new benchmark for agentic systems on professional office work - complex multistep tasks on files and applications that professionals would use in their day to day workflows. The benchmark contains 130 tasks curated by experts and 47 synthetic tasks. The tasks consist of a number of input files, instructions, and a rubric for scoring the output that is not made available to the agent during execution. The outputs are evaluated for factual accuracy, analytical soundness and aesthetic quality. The paper also introduces a framework that the authors used to generate the syntethic tasks along with the rubrics needed to grade the tasks.

The results are reported across 6 different models, and the best models perform at around 50% on both expert curated (51.3%) and synthetitc tasks (50.8%). The expert curated tasks are curated by professionals in 7 different domains. Quality control is dependent on self-checks by the author and a spot-check review by a QC team. Issues identified are iterated on till the task passes. For synthetic tasks, the authors use JobBench files and a multi-agent framework to propose new tasks and corresponding rubrics - first, the workspace with the files is checked for feasibility of creating tasks, then a task designer checks that the inputs are appropriate and whether the files can be used to create tasks that use native Office document structures. For the workspaces that pass, the designer would use the resources to generate the task, potentially adapting the input files. The rubric designer independently creates the acceptance criteria across the 3 axes mentioned earlier, and an independent rubric reviewer checks the coverage, fairness and support for the created rubrics.

The paper distinguishes between an execution stage and an evaluation stage, and introduces an evolution analyst that effectively uses the execution logs to cross check instruction, rubrics, and all the evidence from execution and evaluation to strengthen/harden the task and the rubrics. The evolution process is bounded, (up to 4 revisions, giving us 5 versions including the original). The evaluation itself uses an Agent-as-a-Judge based system that relies on OfficeCLI to parse, render and interact with the documents, and uses criterion based verification (where the judge model uses the CLI to check that the requirements are met).

The headline results show that Claude Fable 5 performs the best in aggregate on both types of tasks. Fable does the best on factual tasks, Opus doing better on reasoning, and Kimi K3 doing best on aesthetics. The authors also run a human alignment study on the judge outputs for aesthetic agreement, on ~2.6k pairs. The harness used is Claude Code (though OpenClaw and Codex are also explored), each model runs each task once, and is provided a time limit (but no token limit).

Critical strengths
  • The main strength of this work is the data - the expert task curation, providing us with new tasks that are representative of professional work. This is particularly useful as we try to adapt agents for more complex, real world tasks that people care about daily. The synthetic task curation framework is useful too, especially since the rubrics are validated by experts. Being familiar with similar data collection processes, this can be a painful and expensive process to complete. This is the main point that is making me lean towards accept.

  • The rigor in testing the validity of the evaluator judge is another strong point - the budget is held fixed, and for the actual human analysis, two humans look at both the rendered versions, the native files themselves and the agreement scores are reported based on the evidence used to judge

Critical weaknesses
  • Primary weakness is the fact that there was only one run per model per task - agentic systems are very prone to errors, and a single run is not very representative of the model's capabilities. Additionally, the execution protocol is not very detailed - is the agent provided with a system prompt or does it just use the default harness prompt? What level of reasoning is selected? Do we have a sense of how many tokens each model consumed to solve the same problem? While a time based bounding makes sense, due to difference in deployments, it is entirely possible that latency could have caused one model to effectively have much fewer turns. Maybe an accounting of how many tasks hit the 2 hour cap ( from page 7) would help here. Reading the paper, I also don't know the cost of running this benchmark.
  • I'd ideally prefer to know if the agent that ran these tasks were provided with any skills or tools, for example, anthropic has official docx/pptx skills for working with office documents.
  • Another weakness stems from the review process for curating the tasks - these exercises usually involve a lot of back and forth with the experts, and generally speaking, it is good practice to have rubrics independently verified by another expert; however it is not clear from the paper if the quality control team are domain experts;
Questions for authors
  • What skills and prompts were the models operating with? [clarification]
  • Can we run multiple runs for at least a subset to see run-to-run variance and whether that changes rankings? The numbers are really close together, that this is essential.
  • Do we have any analysis of how many tokens each model takes to complete the task? More so than time, this is a better indicator of model capability.
  • The reproducibility statement only talks about how the scoring/evaluation was done - but does not mention anything about access. I would have greatly preferred to actually look at some sample tasks and see if they actually are complex, domain-specific tasks that professionals would use on a day to day basis. I also don't see the dataset linked in supplementary material. [clarification]

To sum up, I'd like to see more thorough analysis on how consistent the headline numbers are - a single run is not sufficient evidence to claim these rankings. This

Reviewer_ae2wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Professional tasks are an important topic, but often datasets do not contain artifacts and tasks that represent the detail-oriented nature of real work. A lot of existing datasets/benchmarks necessitate the use of office files, but do not fully utilize (necessitate fully utilizing) the complexity of these files for the task. An example would be utilizing the visible cell value of a spreadsheet vs. needing to amend the formula underlying that value.

The authors present officebench-pro as a response to this gaps. This dataset requires native office structure as an admission requirement, focusing on office files. It requires quality requirements are explicitly connected within the task. Many of the tasks require outputs of multiple office formats, but those outputs must also usually be coordinated. The dataset itself is a combination of professionally authored tasks and synthesized tasks. n=177

Using this dataset, authors provide additional insight findings:

  • creating native objects is easier for agents than preserving dependencies in the task
  • presentations can be aesthetic but have bad content
  • different support structures (like source maps or visual review) affect different requirements (like reasoning or aethetics, respectively)

It is not clear that the dataset will be made public.

Critical strengths

OfficeBench-Pro: this is a novel concept of a dataset that goes further into task complexity to better be in domain of real world tasks. It takes a deeper dive into office files, utilizing the complexity of both the files within the scope as well as pushing task definitions to be complex and rich with dependencies, which mimic more real world tasks.

Critical weaknesses

Lack of quality proof for the data: though the data was authored by experts (or reviewed/edited by experts for the synthetic ones), there is no proof of the quality of the dataset. Furthermore, there is no proof that the dataset closes a gap when compared with others in terms of the exact motivations of the authors. There is limited quantitative evaluation of the claims, such as the task complexity and document interdependence.

Questions for authors

Will the authors make the full dataset available for the field?

Reviewer_7iBEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OfficeBench-Pro, a benchmark for evaluating agents on professional Office tasks.

The benchmark contains 130 expert-authored and 47 synthesized tasks, with 147 tasks requiring multiple Office output formats. The authors propose a task synthesis and evolution framework and an Office Agent-as-a-Judge framework.

Across six models, the highest overall scores are 51.3% on expert-authored tasks and 50.8% on synthesized tasks. Additional analyses reveal shortcomings in evidence use, native functionality, and professional presentation

Critical strengths

The critical strength of this paper is the investment in task construction and quality control. The expert-authored collection involves 56 professionals, and all 47 synthesized tasks undergo expert review.

Critical weaknesses
  • The lack of details in 47 synthezied tasks make the results unconvincing. The paper reports neither the total number of synthesized tasks nor the rejection rate of each stage, leaving the framework's scalability unverifiable.
  • The experiments are not sound enough. The main experiment uses a single run per model per task. The analysis of alternative agent frameworks are based on a smaller subset, weakening the conclusions.
Questions for authors
  • Please provide full data for the synthesis pipeline: initial counts, per-stage rejection rates, and whether the 47 tasks carry selection bias in difficulty or domain.
  • Can you report variance from a repeatedly executed subset? Gaps like 51.3% vs. 49.2% may lie within run-to-run variance.
  • The experiments on alternative agent frameworks could be expanded to more tasks.
  • The paper omits cost–performance trade-off analysis (e.g, token consumption / execution time / estimated dollar cost).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.