- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In this paper, authors present ProfessionalBench, a benchmark to test agents' capabilities on real-world professional workflows. They argue that there're certain professional responsibilities that need to be undertaking during the task solutions (e.g. planning). These responsibilities might be resolved in advance which defeats the point of the benchmark. The authors present a Know-Act-Close (KAC) framework for responsibility preservation. Each task is evaluated with dense rubrics. They evaluate frontier models on this benchmark and show that even the best current models struggle with solving end-to-end professional tasks.
Critical strengths
Knowledge work benchmark sourced by human experts for real-world valuable tasks in 3 valuable domains: healthcare, finance and law.
Critical weaknesses
- No error bars and only one trial per model. In workflow compression experiments, the largest difference in Outcome Score is 4%, without standard deviation it's even clear if it's significant.
- Repeated evaluation also demonstrate that the tasks are reproducible and if a task is stochastic in its nature, several trial runs for both reference and model solutions should ensure reliability.
- The fact that the frontier agents have low pass rates on this benchmark doesn't prove that the agents are weak or benchmark is hard unless all confounds are removed: cheating, task integrity, verifier-instruction alignment, reproducibility, adequate constraints etc.
Questions for authors
Professional units that were used for task instantiation, how were they created? Why specifically these?
- What is the difference between Outcome Score and Professional Pass? Outcome Score assigns partials credit and Professional Score is just binary (all pass = 1)? Why in the fig. 4 y-axis are called "Performance Score" and not "Outcome Score" which I assume is shown there?
- Evaluation protocol:
- Why different models have different setups "Codex used high reasoning effort with web search disabled. DeepSeek and GLM used text-only Anthropic-compatible routes. For healthcare and law GLM runs, Claude Code’s Read tool was disabled, while Bash, Python, PDF extraction, and OCR remained available." (app. C.2)? does it mean that other models except Codex used web search and for other models except GLM in law and healthcare Claude Code’s Read tool was enabled? What is the reason for this?
- "Each task-model pair was executed once." I assume it was done with sampling decoding? So the scores might not be deterministic.
- Does the benchmark have the oracle solution that demonstrates that it's solvable withing its constraints?
- Were there any analysis on whether the tasks are cheatable/prone to reward hacking (e.g. an agent can see the answer in the environment files)? Are there any false positives/negatives?
- Will the benchmark, code, agent traces be available for the analysis?
6 comments
W3 & Q4 & Q5: Task Solvability, Integrity, and Reward Hacking
Thank you for pointing out that a low pass rate alone cannot establish task difficulty; it must be interpreted alongside task solvability, input completeness, verifier reliability, and potential reward hacking. We agree and summarize our quality-control measures and validation evidence across five dimensions: Cheating, Task Integrity, Verifier–Instruction Alignment, Reproducibility, and Adequate Constraints.
| Factor | Available Evidence |
|---|---|
| Cheating | Reference materials are mounted only during the scoring stage, while the Solver runs in an isolated container, thereby limiting its access to reference answers. |
| Task Integrity | We constructed 435 task packages, 8,308 inputs, and 1,880 deliverable requirements; performed hash checks on all inputs; and verified that all 435/435 task packages could be loaded. |
| Verifier–Instruction Alignment | The Rubric aligns acceptance criteria, evidence targets, dependencies, and critical thresholds. Submissions automatically judged as both passing and failing are included in the expert review. |
| Reproducibility | We provide the Harness, environment versions, and execution entry points, and examine result stability through repeated runs and cross-Judge evaluations. |
| Adequate Constraints | Tasks are constructed around professional roles, business objectives, responsibility boundaries, and completion criteria, ensuring alignment among input materials, execution requirements, and final deliverables. |
Regarding task solvability. Domain experts participated in task design, reference-solution authoring, and Rubric development. All candidate task packages underwent automated validation, independent expert review, and end-to-end testing. Furthermore, reference deliverables for all 435 tasks passed three independent acceptance rounds, with every reference solution scoring 100 and achieving 100% Professional Pass in each round.
Regarding false positives and false negatives. Experts reviewed all 630 main-experiment runs, of which 610 agreed with the automatic Professional Pass judgments (96.8%). Domain-level results are reported below:
| Domain | Number of Tasks in Benchmark | Number of Rubric Items | Main-Experiment Tasks | Expert-Reviewed Runs | Expert Agreement |
|---|---|---|---|---|---|
| Finance | 141 | 4,232 | 30 | 210 | 209/210(99.5%) |
| Law | 167 | 6,029 | 30 | 210 | 197/210(93.8%) |
| Healthcare | 127 | 4,848 | 30 | 210 | 204/210(97.1%) |
| Overall | 435 | 15,109 | 90 | 630 | 610/630(96.8%) |
Using expert judgments as reference labels, we further decomposed the 20 disagreements into 12 false acceptances (FP) and 8 false rejections (FN). Each expert judgment was paired with the automatic judgment from the same evaluation version, with the corresponding review records retained.
| Automatic Judgment / Expert Judgment | Expert: Pass | Expert: Fail |
|---|---|---|
| Automatic Professional Pass | TP=117 | FP=12 |
| Automatic Fail | FN=8 | TN=493 |
According to the table above, the false rejection rate is FN/(TP+FN) = 8/(117+8) = 6.4%.
Q1: Construction and Coverage of Professional Units
Thank you for your interest in the sources and selection criteria of Professional Units. We organize the task space into five levels: Domain, Subdomain, Professional Unit, Anchor Workflow, and Specific Task. Professional Units are constructed with reference to the O*NET occupational classification, industry standards, and domain-expert knowledge, with experts validating the responsibility boundaries and representative workflows of each unit.
The following table reports the number of Professional Units in each domain and the coverage of instantiated tasks:
| Domain | Subdomains | Professional Units | Instantiated Tasks | Unit Coverage in the 90-Task Main Set |
|---|---|---|---|---|
| Finance | 12 | 41 | 141 | 27/41 |
| Law | 10 | 28 | 167 | 21/28 |
| Healthcare | 8 | 58 | 127 | 24/58 |
| Overall | 30 | 127 | 435 | 72/127 |
The following table illustrates the correspondence among Professional Units, Anchor Workflows, and representative tasks across the three domains, using the same task IDs as the intervention-task mapping above.
| Domain | Professional Unit | Typical Anchor Workflow | Paper Task ID | Corresponding Task Description |
|---|---|---|---|---|
| Finance | F11.1, Financial Planner | Retirement/Education Goal Assessment and Planning | F_T18 | School-district home purchase and mortgage repayment planning. Taking into account the child's school enrollment window and the household's housing and financial situation, verify mortgage eligibility and down-payment feasibility; compare the timing of the purchase, financing methods, and repayment arrangements; and develop a plan that balances interest costs, loan term, and liquidity. |
| Law | L04.03, Data Compliance/Cybersecurity | Personal Information Processing Relationships, Legal Basis, and Impact Assessment | L_T8 | Compliance review for medical data disclosure. For the three stages of a pancreatic cancer early-screening project, verify the data-processing relationships, degree of data anonymization, and scope of authorization and impact assessments; analyze cross-border disclosure requirements; and provide a go/no-go decision and prerequisite remediation conditions for each stage. |
| Healthcare | H.clinical.16, Endocrinology and Metabolism Physician | Selection of Specialist Examinations, Integration of Results, and Differential Diagnosis | H_T12 | Endocrinology inpatient admission assessment. Integrate the medical history and laboratory materials, prioritize screening for acute metabolic risks, and then arrange staged differential diagnostic examinations for diabetes classification, polydipsia and polyuria, and related syndromes, specifying the examination sequence, safety conditions, and diagnostic boundaries. |
To assess the coverage of existing professional benchmarks under our classification, we mapped tasks from multiple Finance, Law, and Healthcare benchmarks to the same Professional Unit space. The following table summarizes their Unit coverage; complete results are provided in Table 10 of the main paper:
| Benchmark | Audited Tasks | Finance Unit Coverage | Law Unit Coverage | Healthcare Unit Coverage | Overall Unit Coverage |
|---|---|---|---|---|---|
| GDPval | 220 | 19/41(46.3%) | 4/28(14.3%) | 6/58(10.3%) | 29/127(22.8%) |
| APEX-v1-extended | 100 | 14/41(34.1%) | 8/28(28.6%) | 14/58(24.1%) | 36/127(28.3%) |
| APEX-Agents-v1.1 | 240 | 18/41(43.9%) | 15/28(53.6%) | — | 33/127(26.0%) |
| ALE | 157 | 8/41(19.5%) | 5/28(17.9%) | 10/58(17.2%) | 23/127(18.1%) |
| PRBench | 1,100 | 33/41(80.5%) | 26/28(92.9%) | — | 59/127(46.5%) |
| APEX-Accounting | 10 | 3/41(7.3%) | — | — | 3/127(2.4%) |
| Finance Agent v2 | 27 | 2/41(4.9%) | — | — | 2/127(1.6%) |
| FinGAIA | 150 | 16/41(39.0%) | — | — | 16/127(12.6%) |
| Harvey LAB | 2,010 | — | 25/28(89.3%) | — | 25/127(19.7%) |
| LegalAgentBench | 300 | — | 7/28(25.0%) | — | 7/127(5.5%) |
| MedAgentBench | 300 | — | — | 3/58(5.2%) | 3/127(2.4%) |
| HealthAgentBench | 54 | — | — | 5/58(8.6%) | 5/127(3.9%) |
| HealthBench Professional | 525 | — | — | 33/58(56.9%) | 33/127(26.0%) |
For example, PRBench covers 59/69 Units across Finance and Law, while HealthBench Professional covers 33/58 Healthcare Units. These results reveal differences in occupational-responsibility coverage and help identify underrepresented work types.
Thank you for carefully reviewing the manuscript and providing detailed comments on result stability, task solvability, and evaluation transparency. Below, we respond to your comments point by point in the hope of helping you better understand the paper and addressing your questions.
Thank you for pointing out that the original experiments lacked repeated-run statistics. In the main experiment, each task–model configuration was executed once, yielding 630 runs. To assess execution variability, we added two independent runs for GPT-5.6 Sol, Claude Opus 5, Qwen3.8-Max, and DeepSeek V4.1 Flash on the original task set, and report the three-run averages and standard deviations.
Run 1 corresponds to the original main-experiment result, while Runs 2 and 3 are independent repeats. Score avg@3 averages each task's three scores before averaging across tasks, while Pass avg@3 averages the three run-level pass rates. Mean within-task SD is the average task-wise sample standard deviation, and Across-run Score SD is the sample standard deviation of the three run-level mean scores.
All runs used the same task set, execution configuration, and evaluation version, with the environment reset before each run. Average scores are generally close to the original results, although variability differs across models and domains. For example, GPT-5.6 Sol is stable in Healthcare (66.91, 66.31, and 66.88), whereas DeepSeek V4.1 Flash shows greater variation in Law (50.32–55.45). These results motivate reporting both task-level and run-level variability rather than relying on a single run.
We additionally evaluated the submissions of all seven models using two additional judges, GPT-6.1 Sol and Opus 5.5. The table reports the corresponding Score, Professional Pass rate, and Score ranking (in parentheses).
Across the three judges, GPT-5.6 Sol and Claude Opus 5 exchange the top two positions, while Kimi K3 consistently ranks third. The Score SD across judges ranges from 1.32 to 2.84. Thus, although individual scores and some rankings vary, the overall performance tiers remain largely consistent.