acceptodds
Under review as a conference paper at ICLR 2027

WORKBank: Auditing AI Agent Readiness Against Worker Desire across the U.S. Workforce

Abstract

The rapid rise of AI agents is reshaping the labor market, raising concerns about job displacement, diminished human agency, and overreliance on automation. Yet these debates proceed without answering a fundamental question: which tasks should AI agents automate, and which present opportunities for augmentation? Existing approaches, from agent benchmarks to exposure studies, measure what agents can do but lack the evaluative apparatus to distinguish technical feasibility from social acceptability and what workers want. We address this gap by introducing a task-level auditing framework that contrasts worker desire with technological capability across the automation-augmentation spectrum. Our framework introduces the Human Agency Scale (HAS, H1-H5), a shared evaluation language that moves beyond binary automation framing to capture the full range of agent readiness. Instantiating this framework at scale, we construct WORKBank—comprising data from 1,500 domain workers across 104 occupations and capability assessments from 52 AI experts—as the first evaluation resource to jointly measure worker desire and AI readiness across the U.S. workforce. We demonstrate three evaluative use cases: auditing worker-centered automation demand across sectors, identifying mismatches between the desire-capability landscape and current AI research and investment, and informing collaborative agent design from HAS profiles. Critically, 31.6% of tasks fall in a high-capability but low-desire zone, while 12.7% represent high-desire but low-capability opportunities that current R&D underserves. All data, codebook, and the HAS specification are released publicly to support longitudinal tracking of AI agent readiness across the workforce.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI agents, automation and augmentation, real-world work, suvey-based audit

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LJ49OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents WORKBank, a dataset of work tasks created by polling workers in a wide variety of domains on tasks in their job. They ask 1,500 workers across 104 occupations and 844 tasks how much they want a particular task that makes up their work to be automated on a 1-5 scale. They also ask the study participants the extent to which they believe a human is required in carrying out a task. They call this scale a human agency scale (HAS), which uses values from H1 (no human involvement required) to H5 (human involvement essential).

They also ask 52 AI experts whether AI was already good enough to automate a given task, using a 1 to 5 scale which measures "To what extent do current AI systems support automating this task?". The experts also provide HAS scores for the tasks.

They also find that 31.6% of tasks are in the Red Light category, meaning capable of automation but not wanted. 12.7% are wanted for automation but not capable of it. This implies that ability to automate currently outstrips the appetite for automation. They also find that workers prefer higher levels of agency than experts would predict to be necessary in many of the tasks.

Additionally they show that 41% of Y Combinator company–task mappings fall in areas that are either Low Priority (low desire and low capability) or Red Light (low desire, high capability).

Critical strengths

A strength of the work is that it is the first study of its kind to try to quantitatively establish which jobs and tasks in the economy present opportunities for AI automation from the workers' perspective, as well as the extent to which AI is already capable of automating such tasks. To my knowledge, no other such work of this scale exists.

  • The work's strengths also include a large sample of workers, specifically 1,500 across 104 occupations.
  • The authors do a good job of documenting their full process, including every survey question, the LLM prompts and the full regression results.
  • The data is shared in the supplementary material, with every individual rating provided in full.
  • The data supports the main claim made by the authors, that more tasks are Red Light than R&D Opportunity.
Critical weaknesses

In an analysis of the supplementary data, I found a few issues that were concerning to me. In particular, in a blind check of 240 sampled workers' descriptions of their own jobs, 24% didn't match the occupation that they claimed. I also compared non-Prolific and Prolific workers and found that when one looks only at non-Prolific workers, the gap between the Red Light and R&D Opportunity categories shrinks from about 16 percentage points to roughly 3 to 6 points on the same tasks. This means that if you discount the Prolific workers as being less representative samples of professionals in the industry, which I somewhat expect from Prolific workers, it becomes much harder to separate these two categories. More concerningly, in examining the experts' views, I found that the experts agree only slightly more than chance on their capability ratings, providing the same score only 29% of the time where random chance would be 24.5%.

Given how noisy the data is, the lack of confidence intervals on the paper's headline figures is also a weakness, as they are very informative for someone trying to draw conclusions from the data.

Questions for authors

Clarifications:

  1. How were the workers' occupations checked? When I checked a sample of 240 workers in the data, I found that roughly 24% of the given occupations didn't match the person's own description of their job.

Requests for new work:

  1. To what extent do the zone shares hold if you analyse the Prolific and non-Prolific workers separately? My initial analysis found a substantial difference in the gap between the Red Light and R&D Opportunity categories between the Prolific and non-Prolific workers.
  2. Can you add confidence intervals for all of the critical numbers given in the paper, such as the zone shares and the agency figures?
  3. Additionally, while this cannot be covered in the time offered by the review, I think improving expert agreement would substantially improve the credence I can give to the data and subsequent analysis based on that data.
5 comments

We thank the reviewer for the careful analysis of our supplementary data, for acknowledging the novelty and scale of the study, and the efforts we made on data collection and documentation. Your review led to several new analyses that make the paper stronger and we address each of your concerns below.

How were the workers' occupations checked? When I checked a sample of 240 workers in the data, I found that roughly 24% of the given occupations didn't match the person's own description of their job.

Occupations are self-reported on the O*NET taxonomy: participants first choose a sector and then an occupation from a second menu. Additional checks depended on the platform: on Prolific, participants were screened so that the sector matched the industry in their Prolific profile; on Upwork we checked the match between the occupation in their profile and ours; for LinkedIn/word-of-mouth we did not filter, since the survey was distributed through personal networks. We add this to Appendix D.4.

We have the task-familiarity filter (Section 3.3) as additional safeguard. Participants only rate tasks they confirm they perform, and compensation does not depend on how many tasks they rate, so there is no incentive to claim familiarity.

After reading your review, we went through all participants and compared their interview reflections with their self-reported occupation. Of the 1,500 participants, 134 provided no ratings (none of the tasks shown matched what they do, likely because of a wrong occupation selection), leaving 1,366 workers whose ratings contribute to the WORKBank task-level results. We state this explicitly in the revised paper.

To what extent do the zone shares hold if you analyse the Prolific and non-Prolific workers separately? My initial analysis found a substantial difference in the gap between the Red Light and R&D Opportunity categories between the Prolific and non-Prolific workers.

Thank you for highlighting this. We recomputed desire separately for Prolific and non-Prolific (Upwork + Linkedin/word of mouth) respondents, holding expert capability fixed, on the same 514 tasks rated by both groups:

Recruitment SourceGreen LightRed LightR&D OpportunityLow Priority
Prolific40.9%35.8%12.3%11.1%
Non-Prolific56.0%20.6%16.7%6.6%

Participants recruited from Prolific have lower automation desire in general compared to participants recruited from other sources. At the same time, this additional analysis reveals the core findings of the paper remain robust. In both groups, Red Light is at least as large as R&D Opportunity in the point estimates (35.8 vs 12.3; 20.6 vs 16.7). The qualitative pattern in the threshold-robustness analysis (Appendix B.1) is also unchanged. We include these results in Appendix D.4 Recruitment Sources.

Which group is more "representative"? We do not think either should be treated as ground truth, and this is why we deliberately recruited from several platforms.The difference we see is a useful finding about how measured desire varies with who is asked, which is also why WORKBank releases the recruitment source per worker so that users can stratify or reweight.

Can you add confidence intervals for all of the critical numbers given in the paper, such as the zone shares and the agency figures?

Thanks for the suggestion. We revise the manuscript to report:

  • The 95% confidence interval (N=10,000) for the percentage of tasks with mean desire > 3: 46.1% [40.6, 51.7]
  • The 95% confidence interval (N=10,000) for automation desire-capability landscape zone share:
    • Automation “Green Light” Zone: 43.1% [37.8, 48.8]
    • Automation “Red Light” Zone: 31.6% [27.0, 36.3]
    • R&D Opportunity Zone: 12.7% [9.6, 16.0]
    • Low Priority Zone: 12.6% [9.5, 15.9]
  • We update Figure 5 and Figure 8 to include the 95% confidence interval (N=10,000) for the distribution of worker-desired HAS levels (Hw(t)H_w(t)) and expert-assessed feasible HAS levels (He(t)H_e(t)) by occupations.

Given that the authors say that they intend to release the information, this seems concerning as the information is fairly sensitive in itself, and particularly so in relation to the automation of their work. It's also unclear if participants are aware of the extent of the information that would be revealed and whether they consented to this with full knowledge. I don't expect the risks of this to be high, but I don't think it follows good principles of data management and there is little advantage in releasing the personal information.

Thank you for raising this and we want to highlight that we take participant privacy seriously. All participants gave informed consent, and the study was approved by our institution's IRB.

In the public release, we will remove all interview transcripts and ZIP codes, so demographic fields will be linked only to scalar ratings. The demographic fields that remain are useful for stratified analyses like the ones above, and, with ZIP codes removed. Since each combination is shared by many workers within the same occupation, the deidentification risk is low.

Hi Reviewer LJ49,

Thank you again for your careful review! We have posted our rebuttal and revised the manuscript to clarify the quality control and include confidence intervals to our figures (changes marked in blue).

We would greatly appreciate it if you could take a look at our rebuttal and reconsider your assessment. Please let us know if any concerns remain or if further clarification would be helpful!

Best regards,

Submission 3247 Authors

Reviewer_px8oOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the mismatch between workers’ preferences for automation and experts’ judgments about AI task feasibility. It introduces the Human Agency Scale and reports a survey covering 1,500 workers across 104 occupations, alongside assessments from 52 AI experts. The resource may help characterize where automation is desired, undesired, or not yet technically feasible. However, technical readiness is based on expert judgments rather than direct execution by current AI agents, so the paper primarily provides a workforce-oriented audit framework rather than an empirical assessment of deployed agent capability.

Critical strengths
  1. The paper addresses an important but under-measured question: technical automability and workers’ desire for automation are not the same thing. This framing can inform more responsible agent deployment.
  2. The breadth of the survey and occupation-level organization provide a potentially useful resource for studying human–AI work allocation.
Critical weaknesses
  1. Expert-rated feasibility is only a proxy for actual agent capability. Without validating these judgments against standardized task execution by contemporary agents, the central capability–preference mismatch may not accurately reflect what systems can do in practice.
  2. Somewhat limited methodological novelty and broader impact. The main contribution is primarily descriptive and resource-oriented, with somewhat limited methodological novelty from a machine-learning perspective. The generalizability across worker populations, occupations, and evolving model capabilities remains unclear. For a conference like ICLR, the broader implications and insights for the agent research community could be more substantial.
Questions for authors
  1. How were workers recruited and weighted, and what analyses establish that the sample represents the occupations and worker groups discussed in the paper?
  2. Have the expert feasibility ratings been compared with standardized evaluations of actual agents on a subset of the same tasks? If not, how should readers interpret “AI readiness” beyond expert perception?
4 comments

We thank the reviewer for the insightful review and for recognizing that the paper addresses an important but under-measured question and that WORKBank can inform more responsible agent deployment.

Expert-rated feasibility is only a proxy for actual agent capability. Without validating these judgments against standardized task execution by contemporary agents, the central capability–preference mismatch may not accurately reflect what systems can do in practice.

Have the expert feasibility ratings been compared with standardized evaluations of actual agents on a subset of the same tasks? If not, how should readers interpret “AI readiness” beyond expert perception?

We agree that expert ratings are a proxy, and we revise the manuscript to say so more clearly. We would like to explain why we believe it was the right design choice and why our conclusions hold up.

Why expert ratings. The goal of WORKBank is to audit readiness across the computer-based workforce at the task level. Executing hundreds of open-ended, multi-step tasks across 104 occupations with contemporary agents, with standardized grading by domain specialists, was not available at the time of our study. As shown in Table 1, existing benchmarks cover few occupations (GDPval 44, Remote Labor Index 9, APEX paper 4, with limiting data points capture different tasks within the occupations). Expert estimates are the only way to put the whole workforce on the same capability scale today. Our protocol requires ≥2 independent experts per task, with additional review when ratings diverge (SD ≤ 1; Appendix B.2).

Construct validity of the expert ratings. We ran three sanity checks on whether the expert capability ratings behave as expected (added into Appendix B.2).

  1. Checking against independent worker ratings. We compute the correlation between expert-assessed capability and worker-reported physical action/interpersonal communication/uncertainty/domain expertise. Task-level worker ratings of each of these features correlate negatively with expert-assessed capability.
Worker-rated Task FeatureSpearman ρ\rho with Ae(t)A_e(t)pp-value
Physical action-0.199<10−8<10^{-8}
Interpersonal communication-0.250<10−12<10^{-12}
Involved uncertainty-0.300<10−15<10^{-15}
Domain expertise-0.20110−810^{-8}
  1. Checking average across different O*NET work activities. Tasks mapped to information-processing activities (n = 380) score 0.44 (95% confidence interval [0.26, 0.61]) points higher than tasks mapped to interpersonal activities such as coaching, conflict resolution or consultation (n = 130).
  2. Comparison with standardized agent evaluations on a small subset of occupations. As a coarse domain-level check, we grouped our occupations to match publicly available agent benchmarks on APEX (https://www.mercor.com/apex/) and compared the expert capability ratings with the best reported agent score:
Agent BenchmarkMapped WORKBank OccupationsAverage Ae(t)A_e(t)Best Agent Score on the APEX Benchmark
APEX-AccountingAccountants and Auditors, Bookkeeping Clerks, Tax Preparers4.2271.5%
FrontierCodeProgrammers, Software QA Testers, Web Developers, Systems Engineers/Architects3.7461.1%
τ³-BankingCustomer Service Representatives, Credit Counselors3.7659.8%
Physical-Engineering's Last ExamMechanical, Civil and Energy Engineers, Drafters, Mechanical Technologists3.0548.3%
Harvey LABLawyers, Judicial Law Clerks, Legal Secretaries3.5227.4%

The domain ordering agrees (Spearman ρ = 0.80, n = 5, exact permutation p = 0.13); but with only five domains in APEX public release this is suggestive, not conclusive. We hope WORKBank would inspire more agent benchmarks on measuring AI agents capability on tasks that workers desire AI to help.

How to interpret "AI readiness." WORKBank is the expert-perceived level at which current systems support a task as of early 2025, so the zones should be read as a prioritization lens (where to invest, evaluate, or hold back). WORKBank enables selecting tasks from the Green Light and R&D Opportunity zones to build execution-based evaluation as we discuss in "Broader Utilities of WORKBank" (Section 4.3). We revise Section 5 to clarify this.

Somewhat limited methodological novelty and broader impact.

We respectfully disagree and would like to highlight the novelty of this work:

  1. A new measurement object. To our knowledge, WORKBank is the first resource that elicits worker desire and expert capability on the same tasks, with the same rubrics, at scale (104 occupations). Prior capability benchmarks omit desire, and prior worker surveys (e.g., Armstrong et al., 2024) elicit general attitudes only.
  2. The Human Agency Scale (Section 3.2). HAS goes beyond the automation/no-automation view, defining five levels along team dynamics and required human involvement. It lets worker preference and expert-judged feasibility be compared on one scale, which gives results that a binary framing cannot, e.g., 47.5% of tasks where workers prefer more human agency than experts consider necessary, and H3 as the dominant desired level in 45.2% of occupations (Figure 5).
  3. A survey design that grounds ratings (Section 3.3). The audio-based mini-interview, task-familiarity filter, and guided consideration of enjoyment and job security are designed to elicit well-calibrated, experience-based responses at a scale that is hard to reach with text surveys. Our mixed-effects analysis (Appendix B.3) shows desire reflects task properties, not only demographics.

Broader impact for the agent research community. WORKBank offers concrete levers for agent research:

  1. Guiding benchmark curation, by pointing to tasks with high worker desire that lack benchmarks;
  2. Informing human–agent collaboration design, since HAS profiles show where agents should be built to coordinate with humans (H3) vs. act autonomously (H1–H2); and
  3. Providing empirical grounding for AI policy and workforce preparation, by identifying automation-prone occupational tasks and the mismatch between worker desire and AI agent capability.

In Section 4.3 “Broader Utilities of WORKBank”, we list a few examples that leverage WORKBank, including benchmarks like JobBench [1], working report from International Labour Organization [2], and so on.

[1] Jobbench: Aligning agent work with human will., Li et al., 2026, https://arxiv.org/abs/2605.26329

[2] Revisiting occupational segregation and the valuation of women’s work, Liepmann and Hegewisch, 2025, https://www.ilo.org/publications/revisiting-occupational-segregation-and-valuation-women%E2%80%99s-work

How were workers recruited and weighted, and what analyses establish that the sample represents the occupations and worker groups discussed in the paper?

Recruitment. Workers were recruited from Prolific (with screening on employment, US nationality, English, computer use for ≥25% of work, and industry matching), Upwork (US-based freelancers), and LinkedIn/word of mouth (US-based) (Appendix D.4, Table 4). Participants chose their sector and then their O*NET occupation.

Representativeness. We compare sector and demographic profiles against BLS data (Appendix D.2, D.3; Figures 6–7): the sector distribution is close to BLS for the 104 occupations, with a tendency toward younger participants. We do not apply sampling weights: results are descriptive and the sample is not a probability sample, which we will state plainly.

Hi Reviewer px8o,

Thank you again for your thoughtful feedback! We have posted our rebuttal addressing the concerns raised in the review and revised the manuscript to incorporate your suggestions (changes marked in blue).

We would greatly appreciate it if you could take a look at our rebuttal and reconsider your assessment. Please let us know if any concerns remain or if further clarification would be helpful!

Best regards,

Submission 3247 Authors

After considering the comments from the other reviewers, I have decided to maintain my original score.

Reviewer_wkuXOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends the ongoing scientific agenda around the impact of AI on jobs by aligning both AI capabilities and worker desire for AI. This is a nice extension of the existing literature that focuses only on the capabilities dimension. The results primarily are descriptive characterizations of what work falls into what quadrant of the resulting 2-dimensional characterization. The work also makes methodological contributions with the HAS scale to capture the range of fully automated to critical human involvement in work tasks, and the workbank dataset will be valuable to future research.

Critical strengths
  1. The central contribution of adding the worker desire spectrum to the AI capability spectrum is an important contribution to one of the pressing questions of the time, which is how will AI impact work and jobs. The resulting workbank dataset -again, which aligns worker desire and AI capability on the same tasks- will help move the rest of the field forward.
  2. Breadth of the work is solid - 1500 workers from 104 occupations across different channels, and 52 AI experts is a lot. This type of data collection is non-trivial and often you see much smaller numbers.
  3. HAS scale: the underlying concept of it isn't terribly novel, but it's a nice contribution in the context of the work, and I appreciate the quantification so that we can say things like that H3 is the most desired level.
  4. I appreciate the threshold testing for robustness. There is a tendency to use/interpret quadrant frameworks like this as super robust when in fact many data points in two quadrants may be quite similar. So it's good to see some robustness checking here.
Critical weaknesses

Few minor method considerations:

  1. Operationalization of 'desire' among workers: a lot rides on the worker desire measure, which kind of comes down to a question just asking people how much of a task they would want AI to do. This seem fine, but as a minor point, I'd be careful not to over-generalize too much from this. The question protocol seems pretty thoughtful, but it may be hard for people to truly think it all through and then have to sum it all up in a response or two.
  2. AI capabilities from expert estimates: again this is nicely done, but there is a difference in using estimates versus actual empirical evidence. Benchmarks are mentioned, but unless I missed it, there is no direct comparison of the expert assessments to benchmark data. Also, there are papers that have used empirical evidence to assess AI capabilities on ONET activities (see Tomlinson et al.).
Questions for authors
  1. Is there a stronger claim for the expert estimates of AI capabilities that could be made by aligning the expert estimates with empirical evidence (as I noted above, perhaps I missed this)?
  2. How were the 104 occupations selected and how well do you think they generalize to a broader range of jobs (e.g., the 900+ jobs in ONET)? I assume the 104 are those in Figure 8 in the supplementary materials, but how were those 104 chosen?
3 comments

Thank you for your positive and thoughtful review, and for recognizing the value of aligning worker desire with AI capability, the breadth of our data collection, the HAS scale, and our robustness checks.

Operationalization of 'desire' among workers: a lot rides on the worker desire measure, which kind of comes down to a question just asking people how much of a task they would want AI to do. This seem fine, but as a minor point, I'd be careful not to over-generalize too much from this.

We totally agree with your point here. The workers' voice is a component that is largely missing from current AI development even though workers are among the populations most affected by AI agents; our goal is to make it measurable, not to claim it is a complete or fixed picture of preferences.

We make the elicitation thoughtful: workers begin with an audio-based mini-interview about their own work and views of AI, may only rate tasks they confirm they have done, and are asked to consider job security and enjoyment before rating desire. We note in Limitations that "despite prompts to reflect on job-loss and enjoyment concerns, workers may still withhold honest feedback due to job-security concerns; we treat such concerns as critical input for responsible deployment." We revised the Limitations part to further clarify that desire in WORKBank is a stated preference on a single Likert item per task, elicited at one point in time (Jan–May 2025), and may shift as workers gain more experience with AI agents.

AI capabilities from expert estimates: again this is nicely done, but there is a difference in using estimates versus actual empirical evidence. Benchmarks are mentioned, but unless I missed it, there is no direct comparison of the expert assessments to benchmark data.

Is there a stronger claim for the expert estimates of AI capabilities that could be made by aligning the expert estimates with empirical evidence (as I noted above, perhaps I missed this)?

Thank you for the pointer. To the best of our knowledge, no benchmark or usage dataset covered the full set of tasks we audit at the time of the study, and a study in 2026 shows that current AI agent benchmark coverage remains concentrated in programming [1]. In fact, we see WORKBank as a way to direct such evidence collection: as discussed in "Broader Utilities of WORKBank" (Section 4.3), WORKBank can inform benchmark curation by identifying tasks where workers have high automation desire but benchmarks do not exist and we already see work developing benchmark based on WORKBank.

Construct validity of the expert ratings. We ran three sanity checks on whether the expert capability ratings behave as expected (added into Appendix B.2).

  1. Checking against independent worker ratings. We compute the correlation between expert-assessed capability and worker-reported physical action/interpersonal communication/uncertainty/domain expertise. Task-level worker ratings of each of these features correlate negatively with expert-assessed capability.
Worker-rated Task FeatureSpearman ρ\rho with Ae(t)A_e(t)pp-value
Physical action-0.199<10−8<10^{-8}
Interpersonal communication-0.250<10−12<10^{-12}
Involved uncertainty-0.300<10−15<10^{-15}
Domain expertise-0.20110−810^{-8}
  1. Checking average across different O*NET work activities. Tasks mapped to information-processing activities (n = 380) score 0.44 (95% confidence interval [0.26, 0.61]) points higher than tasks mapped to interpersonal activities such as coaching, conflict resolution or consultation (n = 130).
  2. Comparison with standardized agent evaluations on a small subset of occupations. As a coarse domain-level check, we grouped our occupations to match publicly available agent benchmarks on APEX (https://www.mercor.com/apex/) and compared the expert capability ratings with the best reported agent score:
Agent BenchmarkMapped WORKBank OccupationsAverage Ae(t)A_e(t)Best Agent Score on the APEX Benchmark
APEX-AccountingAccountants and Auditors, Bookkeeping Clerks, Tax Preparers4.2271.5%
FrontierCodeProgrammers, Software QA Testers, Web Developers, Systems Engineers/Architects3.7461.1%
τ³-BankingCustomer Service Representatives, Credit Counselors3.7659.8%
Physical-Engineering's Last ExamMechanical, Civil and Energy Engineers, Drafters, Mechanical Technologists3.0548.3%
Harvey LABLawyers, Judicial Law Clerks, Legal Secretaries3.5227.4%

The domain ordering agrees (Spearman ρ = 0.80, n = 5, exact permutation p = 0.13); but with only five domains in APEX public release this is suggestive, not conclusive. We hope WORKBank would inspire more agent benchmarks on measuring AI agents capability on tasks that workers desire AI to help.

To give readers more confidence in the expert ratings, we additionally report the following on the released data. On the capability rating, each task has ≥2 expert ratings (mean 2.9); pairwise exact agreement is 53.3% and agreement within one scale point is 87.9%.

[1] How Well Does Agent Development Reflect Real-World Work?, Wang et al., 2026, https://arxiv.org/abs/2603.01203

How were the 104 occupations selected and how well do you think they generalize to a broader range of jobs (e.g., the 900+ jobs in ONET)? I assume the 104 are those in Figure 8 in the supplementary materials, but how were those 104 chosen?

We source tasks from O*NET 29.2 (18,796 task statements, 923 occupations; Appendix C.1) and the occupations are filtered in two stages.

  1. Filtering to tasks that can be completed on a computer: Because we focus on digital AI agents, we filter to tasks that can be completed on a computer, using four criteria: (1) the occupation mainly involves computer work (judged by gpt-4o); (2) the task can be finished on a computer (gpt-4o); (3) the task has a Core/Supplementary annotation; and (4) the task is performed more than monthly. We manually checked all filtered tasks. This leaves 2,131 tasks across 287 occupations.
  2. Filtering to occupations that have enough respondents. Between January and May 2025 we distributed the survey on Prolific, Upwork and LinkedIn/word of mouth, and 1,676 workers participated. We kept occupations with ≥10 participants so that task-level means are not driven by a handful of people, giving 1,500 workers in 104 occupations.

The 104 occupations are therefore the computer-based occupations for which we obtained adequate responses, and not a random sample.

Regarding representativeness: the 104 occupations cover 13 sectors and their sector distribution closely tracks BLS workforce statistics (Appendix D.2, Figure 6), but we do not claim they represent all occupations. We state in the Limitations that “While our findings may not cover the full picture of AI agents for the workplace, our multi-platform recruitment (Upwork, Prolific, LinkedIn) closely matching U.S. workforce sector distributions across the 13 sectors represented in our database.” Because the survey interface and protocol are released, extending coverage to additional occupations is straightforward.

Comments on the paper

1 comment

We thank all reviewers for their time and constructive feedback. We are encouraged that the reviewers recognized the following strengths of our work:

  1. An important, under-measured question (wkuX, px8o, LJ49): “The paper addresses an important but under-measured question”, contributing “to one of the pressing questions of the time, which is how will AI impact work and jobs”. It is “the first study of its kind to try to quantitatively establish which jobs and tasks in the economy present opportunities for AI automation from the workers' perspective, as well as the extent to which AI is already capable of automating such tasks.”
  2. Scale and breadth (wkuX, px8o, LJ49): 1,500 workers across 104 occupations / 13 sectors, 844 tasks, and 52 AI experts.
  3. The Human Agency Scale (wkuX, px8o) provides a shared vocabulary that moves beyond the binary automation/no-automation framing.
  4. Robustness and transparency (wkuX, LJ49): threshold-sensitivity analysis (Appendix B.1), mixed-effects regression (Appendix B.3), full documentation of survey questions, prompts as well as the released data and codebook.

We revise the manuscript to address the concerns:

  1. Expert-estimated capability vs. empirical agent evidence.
    • There is no agent benchmark covering the 844 tasks / 104 occupations that WORKBank spans. In fact, a recent audit of 43 agent benchmarks (72,342 tasks) finds that agent evaluation remains heavily programming-centric relative to the distribution of human labor [1, Wang et al., 2026]. This is precisely the gap WORKBank is designed to fill. We therefore see our expert ratings as a scalable, task-level readiness estimate, not a replacement for execution-based evaluation.
    • We ran three sanity checks and found the expert capability ratings behave as expected when checking against independent worker ratings, comparing ratings across different O*NET activities, and comparing with standardized agent evaluations on a small subset of occupations.
  2. New analyses added. We added (i) 95% confidence intervals for the main result numbers, (ii) a recruitment-source stratified analysis, (iii) expert-agreement statistics. We further clarified how to use WORKBank and its limitations in Section 5.

After the anonymity period, we will release WORKBank with interview transcripts and ZIP codes removed, so demographic fields will be linked only to scalar ratings. Since each combination is shared by many workers within the same occupation, the deidentification risk is low.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.