acceptodds
Under review as a conference paper at ICLR 2027

ProfessionalBench: Benchmarking AI Agents on Real-World Professional Workflows

19 views

Abstract

As AI agents handle files, use tools, and produce complex deliverables, evaluating their ability to perform professional work becomes increasingly important. Recent benchmarks increasingly adopt workflow-based tasks to improve task realism. However, workflows may undergo Workflow Compression during task construction: responsibilities that belong to the designated role are omitted or resolved in advance, preventing direct evaluation of whether agents can fulfill them and where they fall short. To capture real professional workflows, we introduce ProfessionalBench, spanning 30 subdomains across finance, law, and healthcare. Experts compare tasks against source workflows and use a Know–Act–Close (KAC) formulation to preserve interdependent responsibilities for professional judgment, execution, and closure. Expert-reviewed Dense Rubrics, aligned with KAC responsibilities, support hybrid verification of requirement completion and deliverability, while trajectory analysis examines the failure mechanisms behind unmet requirements. We evaluate seven frontier agents on ProfessionalBench. Results reveal that even the strongest agents achieve pass rates below 40%. Workflow Compression interventions show thatomitting or pre-resolving responsibilities increases mean scores, with larger gains from compressing Act (+4.1 points) and Close (+3.7) than Know (+2.2). Trajectory-based diagnosis finds failures across all three responsibilities, with the largest shares attributed to Act in finance, Know in law, and Close in healthcare. Together, these findings show that preserving role-specific responsibilities exposes substantial professional headroom and identifies where agents need to improve.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Professional Work, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_prFMOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, authors present ProfessionalBench, a benchmark to test agents' capabilities on real-world professional workflows. They argue that there're certain professional responsibilities that need to be undertaking during the task solutions (e.g. planning). These responsibilities might be resolved in advance which defeats the point of the benchmark. The authors present a Know-Act-Close (KAC) framework for responsibility preservation. Each task is evaluated with dense rubrics. They evaluate frontier models on this benchmark and show that even the best current models struggle with solving end-to-end professional tasks.

Critical strengths

Knowledge work benchmark sourced by human experts for real-world valuable tasks in 3 valuable domains: healthcare, finance and law.

Critical weaknesses
  • No error bars and only one trial per model. In workflow compression experiments, the largest difference in Outcome Score is 4%, without standard deviation it's even clear if it's significant.
  • Repeated evaluation also demonstrate that the tasks are reproducible and if a task is stochastic in its nature, several trial runs for both reference and model solutions should ensure reliability.
  • The fact that the frontier agents have low pass rates on this benchmark doesn't prove that the agents are weak or benchmark is hard unless all confounds are removed: cheating, task integrity, verifier-instruction alignment, reproducibility, adequate constraints etc.
Questions for authors

Professional units that were used for task instantiation, how were they created? Why specifically these?

  • What is the difference between Outcome Score and Professional Pass? Outcome Score assigns partials credit and Professional Score is just binary (all pass = 1)? Why in the fig. 4 y-axis are called "Performance Score" and not "Outcome Score" which I assume is shown there?
  • Evaluation protocol:
    • Why different models have different setups "Codex used high reasoning effort with web search disabled. DeepSeek and GLM used text-only Anthropic-compatible routes. For healthcare and law GLM runs, Claude Code’s Read tool was disabled, while Bash, Python, PDF extraction, and OCR remained available." (app. C.2)? does it mean that other models except Codex used web search and for other models except GLM in law and healthcare Claude Code’s Read tool was enabled? What is the reason for this?
    • "Each task-model pair was executed once." I assume it was done with sampling decoding? So the scores might not be deterministic.
  • Does the benchmark have the oracle solution that demonstrates that it's solvable withing its constraints?
  • Were there any analysis on whether the tasks are cheatable/prone to reward hacking (e.g. an agent can see the answer in the environment files)? Are there any false positives/negatives?
  • Will the benchmark, code, agent traces be available for the analysis?
6 comments

Thank you for carefully reviewing the manuscript and providing detailed comments on result stability, task solvability, and evaluation transparency. Below, we respond to your comments point by point in the hope of helping you better understand the paper and addressing your questions.

W1 & W2: Repeated Execution, Repeated Scoring, and Stability

Thank you for pointing out that the original experiments lacked repeated-run statistics. In the main experiment, each task–model configuration was executed once, yielding 630 runs. To assess execution variability, we added two independent runs for GPT-5.6 Sol, Claude Opus 5, Qwen3.8-Max, and DeepSeek V4.1 Flash on the original task set, and report the three-run averages and standard deviations.

Run 1 corresponds to the original main-experiment result, while Runs 2 and 3 are independent repeats. Score avg@3 averages each task's three scores before averaging across tasks, while Pass avg@3 averages the three run-level pass rates. Mean within-task SD is the average task-wise sample standard deviation, and Across-run Score SD is the sample standard deviation of the three run-level mean scores.

DomainModelRun 1 ScoreRun 1 PassRun 2 ScoreRun 2 PassRun 3 ScoreRun 3 PassScore avg@3Pass avg@3Mean within-task SDAcross-run Score SD
FinanceGPT-5.6 Sol60.3133.33%60.8536.67%60.8733.33%60.6834.44%3.590.32
FinanceClaude Opus 563.5833.33%62.7930.00%63.7436.67%63.3733.33%3.710.51
FinanceQwen3.8-Max53.8726.67%53.2330.00%52.0826.67%53.0627.78%4.150.91
FinanceDeepSeek V4.1 Flash52.046.67%49.6610.00%51.6413.33%51.1110.00%3.951.27
LawGPT-5.6 Sol63.1830.00%67.5833.33%62.9733.33%64.5832.22%3.272.60
LawClaude Opus 560.8126.67%64.6526.67%62.9830.00%62.8127.78%3.321.93
LawQwen3.8-Max42.4110.00%39.7610.00%42.1713.33%41.4511.11%2.201.47
LawDeepSeek V4.1 Flash51.9613.33%50.3213.33%55.4513.33%52.5813.33%3.882.62
HealthcareGPT-5.6 Sol66.9133.33%66.3133.33%66.8833.33%66.7033.33%0.770.34
HealthcareClaude Opus 564.4636.67%65.0536.67%61.9543.33%63.8238.89%3.021.65
HealthcareQwen3.8-Max58.9223.33%58.7526.67%59.3426.67%59.0025.56%0.710.30
HealthcareDeepSeek V4.1 Flash55.1323.33%55.4923.33%56.4026.67%55.6724.44%1.520.65

All runs used the same task set, execution configuration, and evaluation version, with the environment reset before each run. Average scores are generally close to the original results, although variability differs across models and domains. For example, GPT-5.6 Sol is stable in Healthcare (66.91, 66.31, and 66.88), whereas DeepSeek V4.1 Flash shows greater variation in Law (50.32–55.45). These results motivate reporting both task-level and run-level variability rather than relying on a single run.

We additionally evaluated the submissions of all seven models using two additional judges, GPT-6.1 Sol and Opus 5.5. The table reports the corresponding Score, Professional Pass rate, and Score ranking (in parentheses).

Evaluated AgentGPT-5.6 Sol ScoreGPT-5.6 Sol PassGPT-6.1 Sol ScoreGPT-6.1 Sol PassOpus 5.5 ScoreOpus 5.5 PassAverage Score Across Three JudgesScore SD Across Judges
GPT-5.6 Sol63.46(1)32.2%66.09(1)33.3%64.68(2)32.2%64.741.32
Claude Opus 562.95(2)32.2%60.30(2)32.2%65.29(1)30.0%62.852.50
Kimi K356.26(3)18.9%58.62(3)20.0%61.91(3)20.0%58.932.84
DeepSeek-V4.1-Flash53.04(4)14.4%55.90(5)16.67%56.27(4)15.6%55.071.77
Qwen3.8-Max51.73(5)20.0%56.95(4)18.9%55.82(5)20.0%54.832.75
GLM-5.248.32(6)12.2%53.41(6)14.4%49.89(7)12.2%50.542.61
Gemini 3.8 Flash47.09(7)13.3%47.24(7)12.2%50.53(6)15.6%48.291.94

Across the three judges, GPT-5.6 Sol and Claude Opus 5 exchange the top two positions, while Kimi K3 consistently ranks third. The Score SD across judges ranges from 1.32 to 2.84. Thus, although individual scores and some rankings vary, the overall performance tiers remain largely consistent.

W3 & Q4 & Q5: Task Solvability, Integrity, and Reward Hacking

Thank you for pointing out that a low pass rate alone cannot establish task difficulty; it must be interpreted alongside task solvability, input completeness, verifier reliability, and potential reward hacking. We agree and summarize our quality-control measures and validation evidence across five dimensions: Cheating, Task Integrity, Verifier–Instruction Alignment, Reproducibility, and Adequate Constraints.

FactorAvailable Evidence
CheatingReference materials are mounted only during the scoring stage, while the Solver runs in an isolated container, thereby limiting its access to reference answers.
Task IntegrityWe constructed 435 task packages, 8,308 inputs, and 1,880 deliverable requirements; performed hash checks on all inputs; and verified that all 435/435 task packages could be loaded.
Verifier–Instruction AlignmentThe Rubric aligns acceptance criteria, evidence targets, dependencies, and critical thresholds. Submissions automatically judged as both passing and failing are included in the expert review.
ReproducibilityWe provide the Harness, environment versions, and execution entry points, and examine result stability through repeated runs and cross-Judge evaluations.
Adequate ConstraintsTasks are constructed around professional roles, business objectives, responsibility boundaries, and completion criteria, ensuring alignment among input materials, execution requirements, and final deliverables.

Regarding task solvability. Domain experts participated in task design, reference-solution authoring, and Rubric development. All candidate task packages underwent automated validation, independent expert review, and end-to-end testing. Furthermore, reference deliverables for all 435 tasks passed three independent acceptance rounds, with every reference solution scoring 100 and achieving 100% Professional Pass in each round.

Regarding false positives and false negatives. Experts reviewed all 630 main-experiment runs, of which 610 agreed with the automatic Professional Pass judgments (96.8%). Domain-level results are reported below:

DomainNumber of Tasks in BenchmarkNumber of Rubric ItemsMain-Experiment TasksExpert-Reviewed RunsExpert Agreement
Finance1414,23230210209/210(99.5%)
Law1676,02930210197/210(93.8%)
Healthcare1274,84830210204/210(97.1%)
Overall43515,10990630610/630(96.8%)

Using expert judgments as reference labels, we further decomposed the 20 disagreements into 12 false acceptances (FP) and 8 false rejections (FN). Each expert judgment was paired with the automatic judgment from the same evaluation version, with the corresponding review records retained.

Automatic Judgment / Expert JudgmentExpert: PassExpert: Fail
Automatic Professional PassTP=117FP=12
Automatic FailFN=8TN=493

According to the table above, the false rejection rate is FN/(TP+FN) = 8/(117+8) = 6.4%.

Q1: Construction and Coverage of Professional Units

Thank you for your interest in the sources and selection criteria of Professional Units. We organize the task space into five levels: Domain, Subdomain, Professional Unit, Anchor Workflow, and Specific Task. Professional Units are constructed with reference to the O*NET occupational classification, industry standards, and domain-expert knowledge, with experts validating the responsibility boundaries and representative workflows of each unit.

The following table reports the number of Professional Units in each domain and the coverage of instantiated tasks:

DomainSubdomainsProfessional UnitsInstantiated TasksUnit Coverage in the 90-Task Main Set
Finance124114127/41
Law102816721/28
Healthcare85812724/58
Overall3012743572/127

The following table illustrates the correspondence among Professional Units, Anchor Workflows, and representative tasks across the three domains, using the same task IDs as the intervention-task mapping above.

DomainProfessional UnitTypical Anchor WorkflowPaper Task IDCorresponding Task Description
FinanceF11.1, Financial PlannerRetirement/Education Goal Assessment and PlanningF_T18School-district home purchase and mortgage repayment planning. Taking into account the child's school enrollment window and the household's housing and financial situation, verify mortgage eligibility and down-payment feasibility; compare the timing of the purchase, financing methods, and repayment arrangements; and develop a plan that balances interest costs, loan term, and liquidity.
LawL04.03, Data Compliance/CybersecurityPersonal Information Processing Relationships, Legal Basis, and Impact AssessmentL_T8Compliance review for medical data disclosure. For the three stages of a pancreatic cancer early-screening project, verify the data-processing relationships, degree of data anonymization, and scope of authorization and impact assessments; analyze cross-border disclosure requirements; and provide a go/no-go decision and prerequisite remediation conditions for each stage.
HealthcareH.clinical.16, Endocrinology and Metabolism PhysicianSelection of Specialist Examinations, Integration of Results, and Differential DiagnosisH_T12Endocrinology inpatient admission assessment. Integrate the medical history and laboratory materials, prioritize screening for acute metabolic risks, and then arrange staged differential diagnostic examinations for diabetes classification, polydipsia and polyuria, and related syndromes, specifying the examination sequence, safety conditions, and diagnostic boundaries.

To assess the coverage of existing professional benchmarks under our classification, we mapped tasks from multiple Finance, Law, and Healthcare benchmarks to the same Professional Unit space. The following table summarizes their Unit coverage; complete results are provided in Table 10 of the main paper:

BenchmarkAudited TasksFinance Unit CoverageLaw Unit CoverageHealthcare Unit CoverageOverall Unit Coverage
GDPval22019/41(46.3%)4/28(14.3%)6/58(10.3%)29/127(22.8%)
APEX-v1-extended10014/41(34.1%)8/28(28.6%)14/58(24.1%)36/127(28.3%)
APEX-Agents-v1.124018/41(43.9%)15/28(53.6%)—33/127(26.0%)
ALE1578/41(19.5%)5/28(17.9%)10/58(17.2%)23/127(18.1%)
PRBench1,10033/41(80.5%)26/28(92.9%)—59/127(46.5%)
APEX-Accounting103/41(7.3%)——3/127(2.4%)
Finance Agent v2272/41(4.9%)——2/127(1.6%)
FinGAIA15016/41(39.0%)——16/127(12.6%)
Harvey LAB2,010—25/28(89.3%)—25/127(19.7%)
LegalAgentBench300—7/28(25.0%)—7/127(5.5%)
MedAgentBench300——3/58(5.2%)3/127(2.4%)
HealthAgentBench54——5/58(8.6%)5/127(3.9%)
HealthBench Professional525——33/58(56.9%)33/127(26.0%)

For example, PRBench covers 59/69 Units across Finance and Law, while HealthBench Professional covers 33/58 Healthcare Units. These results reveal differences in occupational-responsibility coverage and help identify underrepresented work types.

Reviewer_5QegOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProfessionalBench introduces 435 professional workflow tasks spanning finance, law, and healthcare, organized into 30 subdomains, 127 professional units, and 355 anchor workflows. Its central design principle is to preserve Know–Act–Close responsibilities that benchmark construction might otherwise omit or pre-resolve. Expert-reviewed dense rubrics combine artifact/state checks with an agent verifier. Seven model–harness configurations are evaluated on 90 tasks; a separate 10-task intervention compares responsibility-compressed variants. The paper reports low professional pass rates despite appreciable partial completion, and uses trajectory analysis to localize failure modes.

Critical strengths
  1. The responsibility-preservation perspective is useful and concrete. The KAC formulation links task sourcing, deliverables, verification, and diagnosis rather than treating longer tool trajectories as sufficient evidence of professional realism (Sections 3.1–3.3; task examples in Appendix A). This could improve how professional benchmarks are designed.

  2. The evaluation distinguishes requirement completion from professional acceptance and checks artifacts and business states against explicit gates (Sections 3.5–4). Appendix B reports blinded expert comparisons, including 92.1% rubric agreement and 96.9% expert-confirmed acceptance among reviewed automatic passes. Although the reporting needs strengthening, this is useful validation beyond unexamined LLM judging.

Critical weaknesses
  1. The evidence supporting professional acceptance is insufficiently specified. Appendix B does not report the number of reviewed runs/criteria, expert counts and domain coverage, sampling scheme, uncertainty, or expert disagreement. Deliverable-level validation examines automatic passes, so the 96.9% figure does not estimate false rejections. Since the headline concerns pass rates below 40%, this distinction is consequential. It is also important to assess possible judge–solver dependence because GPT-5.6 Sol is both a tested configuration and the open-ended verifier (Section 4.1).

  2. The main empirical support for Workflow Compression is limited. Section 5.1 uses 10 tasks, but does not provide sufficient task-level intervention details, sampling composition, repeated-run uncertainty, or numerical results for the shared-rubric comparison asserted on p. 8. The reported 2.2–4.1 point gains therefore do not yet establish how much comes from bypassing particular responsibilities versus changes in information, scaffolding, or scoring scope. This matters because compression, rather than benchmark scale alone, is the principal differentiating claim.

Questions for authors
  1. Please provide the Appendix B sample sizes and sampling design, domain-stratified human/verifier confusion matrices including automatic failures, and uncertainty estimates. How sensitive are pass rates and model ordering to judge choice and the 0.85 acceptance threshold?

  2. For the compression intervention, can you provide the exact 10 tasks, modifications and rubric mappings, paired task/model scores with uncertainty, and the shared-criterion results mentioned on p. 8? These would help distinguish responsibility-specific effects from scoring or information changes.

  3. Please reconcile the abstract's claim that Know is the largest failure share in law with Figure 5 and Section 5.2, which report Act at 39.1% versus Know at 29.2%. This appears to be a correctable reporting inconsistency.

5 comments

We sincerely thank the reviewers for carefully reviewing our work and providing valuable feedback. We respond to each comment point by point below.

W1 & Q1: Scope of Expert Review, Acceptance Agreement, and False Rejections

Thank you for pointing out the ambiguity in our description of expert review and acceptance. In the original manuscript, automatic passes referred to runs in which the automatic scoring procedure completed successfully and generated a valid score, rather than submissions that passed professional acceptance. Only Professional Pass indicates that a submission meets the specified professional acceptance criteria. We apologize for the ambiguity and have revised the terminology accordingly.

We would also like to clarify two expert-review metrics that may be easily confused:

  • 92.1% Rubric Review Acceptance Rate: The proportion of rubric items accepted by experts during the final expert review, reflecting the expert validation of the task scoring criteria.

  • 96.8% Professional Pass Agreement: The agreement rate between the automatic Professional Pass decisions and expert judgments across all 630 runs (90 tasks × 7 Agent configurations) in the main experiment, corresponding to 610/630.

Fourteen domain experts participated in professional framework calibration and result verification: 4 in Finance, 5 in Law, and 5 in Healthcare. All had relevant domain expertise and industry experience.

For the main experiment, experts selected 90 representative tasks from the 435-task benchmark (30 per domain), covering 26/30 subdomains and 72/127 Professional Units. Expert review then covered all 630 runs across the seven Agent configurations.

DomainTasks in BenchmarkRubric ItemsMain Experiment TasksRuns Reviewed by ExpertsExpert Agreement
Finance1414,23230210209/210 (99.5%)
Law1676,02930210197/210 (93.8%)
Healthcare1274,84830210204/210 (97.1%)
Overall43515,10990630610/630 (96.8%)

To address the concern about False Rejections, we constructed a confusion matrix from the same 630 runs, treating expert-confirmed Professional Pass as the positive class:

Automatic Decision / Expert JudgmentExpert: PassExpert: Fail
Automatic Professional PassTP=117FP=12
Automatic FailFN=8TN=493

The False Rejection Rate is therefore FN/(TP+FN) = 8/(117+8) = 6.4%: the automatic evaluator incorrectly rejected 8 of the 125 submissions that experts judged to pass. Together, the agreement rate and confusion matrix quantify both overall agreement and false rejections.


Q1: Sensitivity to Judge Selection, Evaluator–Solver Dependence, and Acceptance Thresholds

Thank you for noting the potential bias from using GPT-5.6 Sol as both solver and Judge. We therefore added two Judges and conducted sensitivity analyses across Judge models and acceptance thresholds.

(1) Judge Model Sensitivity Analysis

We added GPT-6.1 Sol and Claude Opus 5.5 as additional Judges and used them to re-evaluate all responses from the seven models. The table reports each Judge's Outcome Scores and Professional Pass Rates, with parenthetical numbers indicating model rankings.

Evaluated AgentGPT-5.6 Sol ScoreGPT-5.6 Sol PassGPT-6.1 Sol ScoreGPT-6.1 Sol PassOpus 5.5 ScoreOpus 5.5 PassAverage Score Across Three JudgesInter-Judge Score SD
GPT-5.6 Sol63.46 (1)32.2%66.09 (1)33.3%64.68 (2)32.2%64.741.32
Claude Opus 562.95 (2)32.2%60.30 (2)32.2%65.29 (1)30.0%62.852.50
Kimi K356.26 (3)18.9%58.62 (3)20.0%61.91 (3)20.0%58.932.84
DeepSeek-V4.1-Flash53.04 (4)14.4%55.90 (5)16.67%56.27 (4)15.6%55.071.77
Qwen3.8-Max51.73 (5)20.0%56.95 (4)18.9%55.82 (5)20.0%54.832.75
GLM-5.248.32 (6)12.2%53.41 (6)14.4%49.89 (7)12.2%50.542.61
Gemini 3.8 Flash47.09 (7)13.3%47.24 (7)12.2%50.53 (6)15.6%48.291.94

Note: Inter-Judge Score SD represents the standard deviation of the average Outcome Score obtained by the same evaluated Agent across the three Judges.

Judge choice affects some scores and rankings: GPT-5.6 Sol and Claude Opus 5 exchange the top two positions, while Kimi K3 remains third under all three Judges. Nevertheless, the top-two set remains unchanged, indicating that the leading performance tier is robust to Judge selection despite some evaluator-dependent variation.

(2) Acceptance Threshold Sensitivity Analysis

Using the scoring definition in §3.5, Equation (1), we varied only the final acceptance threshold while keeping the positive and negative weights and key-requirement handling fixed. Parenthetical numbers indicate model rankings.

Model0.750.800.850.900.95
GPT-5.6 Sol38.89% (2)36.67% (2)32.22% (1)17.78% (1)13.33% (1)
Claude Opus 542.22% (1)37.78% (1)32.22% (1)16.67% (2)12.22% (3)
Qwen3.8-Max31.11% (4)27.78% (3)20.00% (3)16.67% (2)13.33% (1)
Kimi K333.33% (3)26.67% (4)18.89% (4)11.11% (4)4.44% (6)
DeepSeek V4.1 Flash27.78% (5)20.00% (5)14.44% (5)7.78% (6)6.67% (5)
Gemini 3.8 Flash23.33% (6)18.89% (6)13.33% (6)10.00% (5)7.78% (4)
GLM-5.222.22% (7)15.56% (7)12.22% (7)4.44% (7)4.44% (6)

Across thresholds 0.75–0.90, GPT-5.6 Sol and Claude Opus 5 remain the top two models. At 0.95, Claude Opus 5 drops to third, showing that stricter thresholds can alter relative rankings. We retain 0.85 as the default and report this analysis to make the effect of threshold choice explicit.

W2 & Q2: Intervention Settings, Paired Results, and Shared Acceptance Criteria

Thank you for raising concerns about intervention design and cross-condition comparability. We clarify the task mapping, the support introduced in each variant, and the criterion-level results under the shared Rubric.

(1) Intervention Tasks and Paired Experimental Settings

The intervention study used 30 original tasks (10 per domain), each evaluated under four paired conditions: Unified, Know, Act, and Close. The following table lists all task IDs, professional roles, and task summaries; roles are derived from the original task descriptions.

Domain / Fig. 20 IDProfessional RoleTask Description
Finance F_T6Investment Valuation ReviewerVerify the valuation parameters, sources, and valuation results of an investee company.
Finance F_T7Market Risk Management SpecialistReview the month-end VaR of a three-asset portfolio and prepare approval materials.
Finance F_T11Carbon Trading Compliance ReviewerVerify the carbon allowance auction calendar and related arrangements.
Finance F_T12Money Market Fund Eligibility ReviewerVerify money market fund eligibility requirements and conclusions based on the provided materials.
Finance F_T18Financial PlannerDevelop a home purchase and repayment plan under school enrollment timing and funding constraints.
Finance F_T19Cost AuditorVerify supporting cost documentation and complete sampling reviews.
Finance F_T23Auto Loan Risk ReviewerVerify lease-based vehicle purchase financing and repayment records.
Finance F_T24Group TreasurerDevelop and verify a group-wide cash management plan.
Finance F_T25Financial Data ReviewerClean crude oil futures data and verify the mapping of trading timestamps.
Finance F_T26Insurance Claims ReviewerCalculate insurance compensation according to Huiminbao policy terms and verify the review results.
Law L_T1Litigation Research LawyerAnalyze defense arguments and procedural strategies in a data breach lawsuit.
Law L_T2International Law CounselFormulate a legal opinion on a high-seas fisheries enforcement dispute.
Law L_T3Cross-Border Trade Legal CounselAnalyze the legal risks and corresponding responses concerning seafood imports into China.
Law L_T4Criminal Defense LawyerDraft a criminal defense statement based on the facts of a fraud case.
Law L_T7Securities Compliance CounselReview year-end closing and compliance issues concerning soft-dollar arrangements.
Law L_T8Data Compliance CounselAnalyze medical data processing relationships, lawful bases, and data protection impact assessment requirements.
Law L_T9IPO Due Diligence LawyerConduct specialized legal due diligence and identify risks prior to an IPO.
Law L_T10Contract Review LawyerReview a smart customer service contract and propose revisions.
Law L_T11Labor LawyerAnalyze an overtime pay dispute and provide legal advice on protecting the relevant rights.
Law L_T12Competition Law LawyerInterpret platform antitrust guidelines and analyze their applicability.
Healthcare H_T2CardiologistReview a right ventricular cardiomyopathy case and the associated clinical judgments.
Healthcare H_T3Pathology Consultation PhysicianAssess a finger mass based on limited pathological images and clinical information.
Healthcare H_T5Gastroenterology PhysicianAnalyze the diagnosis of a gastric mass and the preoperative treatment plan.
Healthcare H_T9Neurology PhysicianReview follow-up evidence and subsequent decisions concerning NIID.
Healthcare H_T10ObstetricianDevelop and verify emergency management procedures for obstetric hemorrhage.
Healthcare H_T12EndocrinologistIntegrate examination results to produce an admission assessment and differential diagnosis.
Healthcare H_T15Multidisciplinary Consultation PhysicianJointly review the underlying cause of seizures and the supporting evidence.
Healthcare H_T16Maternal-Fetal Medicine and Gynecologic Oncology ConsultantReview referral decisions and the corresponding management conditions.
Healthcare H_T25Clinical PhysicianAnalyze an acute monoarthritis case and its clinical rationale.
Healthcare H_T29ICU PhysicianDevelop and verify a joint ICU day-one management plan and its key requirements.

Per-task results and mappings across all four conditions are provided in Appendix I, Figures 19–20.

Reviewer_RHKqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose ProfessionalBench to evaluate the ability of AI agents on real-world professional tasks. They argue that current benchmarks may suffer from workflow compression where some important steps are pre-resolved/ omitted during task construction. So, they propose the Know-Act-Close (KAC) framework.

KAC divides professional tasks into three components,

  1. Know where agent explores evidences to make decisions
  2. Act where agent plans and acts based on available evidences
  3. Close where agent verifies outcomes, resolve remaining issues and complete the task Authors use dKAC during benchmark construction: domain experts compare each task against its original professional workflow, identify pre-resolved/ omitted responsibilities, and modify the task to preserve them and their dependencies.

The benchmark is reasonably big and diverse (435 tasks across finance, law and healthcare), and evaluations include 7 frontier agents on 90 tasks. Performance analyses used expert designed rubrics and hybrid verification, combining programmatic checks with agent-based evaluation. Authors also conducted controlled Workflow Compression experiments on 10 tasks, showing that pre-resolving Know, Act, or Close responsibilities increases measured performance.

Critical strengths
  • A reasonable benchmark with 435 tasks and expert-reviewed rubrics could be a useful resource for future research.
Critical weaknesses
  • The main claim that existing professional benchmarks omit important workflow steps need more evidence. Having a comparison table with examples of what are omitted from recent benchmarks would help better understand the siginficance of the proposed benchmark. For example, a very important benchmark (EnterpriseOpsGym) is not even discussed in the paper which is a high-quality benchmark with over 1100 human-written tasks and uses rubrics-based evaluation. ALE and Workflow-GYM also study real professional workflows and structured task construction. Can you discuss what responsibilities are actually missing in these benchmarks and what additional capabilities KAC measures?

  • One experimental limitation (not a very strong weakness as cost of running evals is a legitimate issue): the main agent evaluation covers 90 tasks and workflow compression analyses uses 10 tasks, while the benchmark contains 435 tasks. I think at least for 1-2 subset of models, evaluating on full 435 tasks would be a good addition to the current paper.

Questions for authors
  • What is the evidence for Workflow Compression in the recent benchmarks? Can you audit representative tasks from EnterpriseOps-Gym, ALE, and Workflow-GYM?

  • EnterpriseOps-Gym already compares independent planning against supplied human plans and uses state-based verification. What new capability or failure does KAC identify that such existing evaluations cannot measure? Can the authors provide a direct example?

3 comments

We sincerely thank the reviewer for carefully reviewing our paper and providing valuable feedback. Our point-by-point responses are provided below.

W1 & Q1: Comparisons with Recent Closely Related Benchmarks

We thank the reviewer for noting the limited discussion of recent professional-workflow benchmarks. We have added a systematic comparison and audited how well these benchmarks preserve professional responsibilities.

(1) Workflow Compression Audit Results

Using a unified standard, we audited 1,486 tasks from ALE, APEX-Agents-v1.1, GDPval, PRBench, and EnterpriseOps-Gym, alongside 90 ProfessionalBench tasks (1,576 in total). The table summarizes the expert-reviewed results:

BenchmarkTasksKnow Retention (Confirmed)Act Retention (Confirmed)Close Retention (Confirmed)Joint K/A/C Retention (Confirmed)Undetermined
ALE15754.1%49.7%52.9%26.8%21.0%
APEX-Agents-v1.124045.4%15.0%30.0%6.7%0.0%
GDPval22063.2%82.3%71.8%45.0%0.9%
PRBench22084.5%75.9%48.2%33.2%0.0%
EnterpriseOps-Gym64999.2%19.11%37.3%12.0%0.0%
ProfessionalBench90100.0%100.0%100.0%100.0%0.0%

Across the five external benchmarks, joint K/A/C retention ranges from 6.7% to 45.0%, indicating substantial gaps in end-to-end responsibility preservation. Initial and expert-reviewed results are reported in Appendix H.2, Table 13(a)–(b).

(2) Examples of Workflow Compression

To complement the aggregate results, we present representative cases from GDPval, ALE, Workflow-GYM, and EnterpriseOps-Gym, showing which responsibilities are pre-completed by task inputs and how they can be restored (Appendix H.3, Figures 15–18).

CaseContent Provided in AdvanceDirection for Responsibility RestorationAppendix H.3
GDPval Warehouse AnalysisProvides the mean of 23.02 days and the total cost of 82,567, thereby pre-completing two Act-related calculations.Calculate and verify these values from the original records.Figure 15
ALE Protein AnnotationSpecifies the protein family and functional conclusion, pre-completing part of the Know assessment.Infer the family and function from domain and GO evidence.Figure 16
Workflow-GYM Blender 42Provides a 40-step route, pre-completing part of the Act planning.Replace the prescribed steps with geometric goals and require autonomous planning.Figure 17
EnterpriseOps-Gym Spreadsheet AuditSpecifies the Close conclusions that verification has succeeded and the audit is ready, while the SQL queries retrieve only replies and metadata.Inspect the actual formulas and data before confirming.Figure 18

W2: Evaluation Scope and Representativeness

We appreciate the reviewer's understanding of the evaluation costs. Since the original 90-task evaluation remains limited, we expanded the study to examine whether the findings hold at a larger scale.

The original main experiment covered 90 tasks (30 per domain) with seven Agent configurations. The intervention study covered 30 tasks (10 per domain), evaluated by GPT-5.6 Sol, Claude Opus 5, and Gemini 3.8 Flash under four conditions (Unified, Know, Act, and Close), totaling 360 trials.

We then evaluated all 435 ProfessionalBench tasks with two additional Agent configurations—GPT-5.6 Sol + Codex and DeepSeek V4.1 Flash + Claude Code—and compared the original 30-task subset with the full set across all three domains:

AgentDomainTotal TasksOriginal 30-Task ScoreOriginal 30-Task Pass RateFull-Set ScoreFull-Set Pass RateFull-Set Avg. Cost (USD/task)Full-Set Avg. Delay (min/task)
GPT-5.6 Sol + CodexFinance14160.3133.3%57.2429.1%2.136.2
GPT-5.6 Sol + CodexLaw16763.1830.0%55.4828.1%5.2212.75
GPT-5.6 Sol + CodexHealthcare12766.9133.3%62.4331.5%1.9610.26
DeepSeek V4.1 Flash + Claude CodeFinance14152.046.7%52.716.4%0.3218.30
DeepSeek V4.1 Flash + Claude CodeLaw16751.9613.3%47.2212.0%1.0235.51
DeepSeek V4.1 Flash + Claude CodeHealthcare12755.1323.3%51.0922.0%0.4210.74

For both configurations, Outcome Scores and Pass Rates shift somewhat, but overall pass-rate levels remain broadly consistent between the 30-task subset and full evaluation, supporting the robustness of our findings at larger scale.

We are finalizing the task packages and de-identification before releasing all 435 tasks, together with their classifications, package versions, and scoring criteria, to support further analysis and reproducibility.

Q2: What Does KAC Add Beyond Human-Provided Plans and Final-State Verification?

Thank you for raising this question. KAC complements existing planning, execution, and final-state verification methods by providing a unified framework for mapping professional responsibilities to acceptance evidence.

(1) Rationale and Positioning of the KAC Framework

Prior work has studied planning support, long-horizon execution, and policy adherence:

WorkPrimary FocusComparison and Verification Methods
EnterpriseOps-GymPlanning and tool execution in enterprise tasks.Compares autonomous execution, model-generated plans, and human planning support, with business-state verification.
Workflow-GYMLong-horizon GUI workflows in professional software.Compares execution with no tutorial, step-by-step text, and video guidance.
HANDBOOK.mdInstruction following under long SOP constraints.Checks required/prohibited actions and execution/compliance claims.

Our focus differs: beyond task success, we distinguish responsibilities fulfilled by task inputs or external support from those the Agent must perform independently.

KAC maps professional responsibilities to three interrelated components: evidence acquisition and assessment (Know), decision-making and action execution (Act), and outcome verification and delivery (Close). It specifies responsibilities, dependencies, and acceptance evidence rather than merely dividing tasks into stages (§3.2; Appendix H.1, Table 12).

The distinction is summarized below:

Existing Evaluation SettingQuestion Already AddressedAdditional Question Addressed by KAC
Autonomous planning vs. human-provided plansHow does providing a plan affect success?Which judgments does the plan supply, and what must the Agent still do independently?
Database or interface final-state verificationHas the business state reached the target?Does acceptance cover all responsibilities, and is the completion claim evidence-backed?
Long-form SOP and policy adherenceAre required actions taken and prohibited actions avoided?How do rules govern evidence gathering, decision-making, verification, and delivery?
Rubric-based scoring and pass/fail decisionsDoes the task meet predefined requirements and pass the threshold?Which responsibilities remain unmet, and at what stage?

No new verifier is required: existing verifiers can implement these checks. KAC clarifies what to check and links each responsibility to acceptance evidence.

(2) What Additional Issues Can KAC Reveal?

We map responsibilities to verifiable acceptance criteria and use rubric items and execution traces to diagnose failures. In the 90-task main experiment, primary failure types were:

CategoryMain Failure TypesOverall Proportion in Figure 5
KnowProfessional knowledge gaps, evidence selection/interpretation errors, hallucinations/fabrication.25.2%
ActStrategy/reasoning and implementation errors, boundary/safety violations.38.6%
CloseIncomplete/early termination, communication/formatting errors, unusable outputs.34.9%
InfrastructureTimeouts/resource limits, Harness/environment errors.1.3%

Each run receives one primary diagnosis. Act and Close account for 73.5% of failures, highlighting execution and delivery challenges beyond knowledge and evidence assessment. Two cases illustrate the value of responsibility-level checks:

CaseOriginal Verification ResultsIssues Identified by Responsibility-Level Checks
EnterpriseOps-Gym HR Notification Acknowledgment (HRC0000713)The five original SQL checks focus mainly on business states. In a constructed SQLite replay, all six submissions passed all five checks.Four of the six submissions nevertheless had unsupported or omitted confirmations, claim-record contradictions, or references to outdated states. Evidence-claim consistency checks can reveal whether confirmations are supported by retrieved records.
ProfessionalBench Home Purchase Planning (F_T18)Under the Act-supported condition, the task scored 87.03 and passed the Professional Pass threshold.Reviewing historical Judge verdicts item by item revealed inconsistent total-interest figures in the report and review record (RMB 22,447 vs. RMB 24,112), and mixed payment conventions in the JSON. Under Know and Close, evidence for the disbursement timing was also missing.

These cases show that a passing score can mask issues affecting deliverable usability. Responsibility-level checks expose issues beyond the final score.

AuthorsOpenReview ↗

A sincere and kind reminder to Reviewer RHKq

Dear Reviewer RHKq,

Thank you very much for taking the time to review our work and for providing such insightful feedback.

As the review discussion period is drawing to a close, we would like to ensure that we have fully addressed your concerns. We have responded to each of your comments and supplemented the manuscript with additional experiments and clarifications. If you have any further questions or comments, please do not hesitate to let us know. We greatly value your feedback and will make every effort to address any remaining concerns and further improve our work.

If our responses and the newly added supporting evidence have adequately addressed your main concerns, we would sincerely appreciate it if you could reconsider your assessment in light of these updates and consider raising your score.

Thank you once again for your careful and thoughtful review.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.