- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a new benchmark for agentic systems on professional office work - complex multistep tasks on files and applications that professionals would use in their day to day workflows. The benchmark contains 130 tasks curated by experts and 47 synthetic tasks. The tasks consist of a number of input files, instructions, and a rubric for scoring the output that is not made available to the agent during execution. The outputs are evaluated for factual accuracy, analytical soundness and aesthetic quality. The paper also introduces a framework that the authors used to generate the syntethic tasks along with the rubrics needed to grade the tasks.
The results are reported across 6 different models, and the best models perform at around 50% on both expert curated (51.3%) and synthetitc tasks (50.8%). The expert curated tasks are curated by professionals in 7 different domains. Quality control is dependent on self-checks by the author and a spot-check review by a QC team. Issues identified are iterated on till the task passes. For synthetic tasks, the authors use JobBench files and a multi-agent framework to propose new tasks and corresponding rubrics - first, the workspace with the files is checked for feasibility of creating tasks, then a task designer checks that the inputs are appropriate and whether the files can be used to create tasks that use native Office document structures. For the workspaces that pass, the designer would use the resources to generate the task, potentially adapting the input files. The rubric designer independently creates the acceptance criteria across the 3 axes mentioned earlier, and an independent rubric reviewer checks the coverage, fairness and support for the created rubrics.
The paper distinguishes between an execution stage and an evaluation stage, and introduces an evolution analyst that effectively uses the execution logs to cross check instruction, rubrics, and all the evidence from execution and evaluation to strengthen/harden the task and the rubrics. The evolution process is bounded, (up to 4 revisions, giving us 5 versions including the original). The evaluation itself uses an Agent-as-a-Judge based system that relies on OfficeCLI to parse, render and interact with the documents, and uses criterion based verification (where the judge model uses the CLI to check that the requirements are met).
The headline results show that Claude Fable 5 performs the best in aggregate on both types of tasks. Fable does the best on factual tasks, Opus doing better on reasoning, and Kimi K3 doing best on aesthetics. The authors also run a human alignment study on the judge outputs for aesthetic agreement, on ~2.6k pairs. The harness used is Claude Code (though OpenClaw and Codex are also explored), each model runs each task once, and is provided a time limit (but no token limit).
Critical strengths
-
The main strength of this work is the data - the expert task curation, providing us with new tasks that are representative of professional work. This is particularly useful as we try to adapt agents for more complex, real world tasks that people care about daily. The synthetic task curation framework is useful too, especially since the rubrics are validated by experts. Being familiar with similar data collection processes, this can be a painful and expensive process to complete. This is the main point that is making me lean towards accept.
-
The rigor in testing the validity of the evaluator judge is another strong point - the budget is held fixed, and for the actual human analysis, two humans look at both the rendered versions, the native files themselves and the agreement scores are reported based on the evidence used to judge
Critical weaknesses
- Primary weakness is the fact that there was only one run per model per task - agentic systems are very prone to errors, and a single run is not very representative of the model's capabilities. Additionally, the execution protocol is not very detailed - is the agent provided with a system prompt or does it just use the default harness prompt? What level of reasoning is selected? Do we have a sense of how many tokens each model consumed to solve the same problem? While a time based bounding makes sense, due to difference in deployments, it is entirely possible that latency could have caused one model to effectively have much fewer turns. Maybe an accounting of how many tasks hit the 2 hour cap ( from page 7) would help here. Reading the paper, I also don't know the cost of running this benchmark.
- I'd ideally prefer to know if the agent that ran these tasks were provided with any skills or tools, for example, anthropic has official docx/pptx skills for working with office documents.
- Another weakness stems from the review process for curating the tasks - these exercises usually involve a lot of back and forth with the experts, and generally speaking, it is good practice to have rubrics independently verified by another expert; however it is not clear from the paper if the quality control team are domain experts;
Questions for authors
- What skills and prompts were the models operating with? [clarification]
- Can we run multiple runs for at least a subset to see run-to-run variance and whether that changes rankings? The numbers are really close together, that this is essential.
- Do we have any analysis of how many tokens each model takes to complete the task? More so than time, this is a better indicator of model capability.
- The reproducibility statement only talks about how the scoring/evaluation was done - but does not mention anything about access. I would have greatly preferred to actually look at some sample tasks and see if they actually are complex, domain-specific tasks that professionals would use on a day to day basis. I also don't see the dataset linked in supplementary material. [clarification]
To sum up, I'd like to see more thorough analysis on how consistent the headline numbers are - a single run is not sufficient evidence to claim these rankings. This