- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents WORKBank, a dataset of work tasks created by polling workers in a wide variety of domains on tasks in their job. They ask 1,500 workers across 104 occupations and 844 tasks how much they want a particular task that makes up their work to be automated on a 1-5 scale. They also ask the study participants the extent to which they believe a human is required in carrying out a task. They call this scale a human agency scale (HAS), which uses values from H1 (no human involvement required) to H5 (human involvement essential).
They also ask 52 AI experts whether AI was already good enough to automate a given task, using a 1 to 5 scale which measures "To what extent do current AI systems support automating this task?". The experts also provide HAS scores for the tasks.
They also find that 31.6% of tasks are in the Red Light category, meaning capable of automation but not wanted. 12.7% are wanted for automation but not capable of it. This implies that ability to automate currently outstrips the appetite for automation. They also find that workers prefer higher levels of agency than experts would predict to be necessary in many of the tasks.
Additionally they show that 41% of Y Combinator company–task mappings fall in areas that are either Low Priority (low desire and low capability) or Red Light (low desire, high capability).
Critical strengths
A strength of the work is that it is the first study of its kind to try to quantitatively establish which jobs and tasks in the economy present opportunities for AI automation from the workers' perspective, as well as the extent to which AI is already capable of automating such tasks. To my knowledge, no other such work of this scale exists.
- The work's strengths also include a large sample of workers, specifically 1,500 across 104 occupations.
- The authors do a good job of documenting their full process, including every survey question, the LLM prompts and the full regression results.
- The data is shared in the supplementary material, with every individual rating provided in full.
- The data supports the main claim made by the authors, that more tasks are Red Light than R&D Opportunity.
Critical weaknesses
In an analysis of the supplementary data, I found a few issues that were concerning to me. In particular, in a blind check of 240 sampled workers' descriptions of their own jobs, 24% didn't match the occupation that they claimed. I also compared non-Prolific and Prolific workers and found that when one looks only at non-Prolific workers, the gap between the Red Light and R&D Opportunity categories shrinks from about 16 percentage points to roughly 3 to 6 points on the same tasks. This means that if you discount the Prolific workers as being less representative samples of professionals in the industry, which I somewhat expect from Prolific workers, it becomes much harder to separate these two categories. More concerningly, in examining the experts' views, I found that the experts agree only slightly more than chance on their capability ratings, providing the same score only 29% of the time where random chance would be 24.5%.
Given how noisy the data is, the lack of confidence intervals on the paper's headline figures is also a weakness, as they are very informative for someone trying to draw conclusions from the data.
Questions for authors
Clarifications:
- How were the workers' occupations checked? When I checked a sample of 240 workers in the data, I found that roughly 24% of the given occupations didn't match the person's own description of their job.
Requests for new work:
- To what extent do the zone shares hold if you analyse the Prolific and non-Prolific workers separately? My initial analysis found a substantial difference in the gap between the Red Light and R&D Opportunity categories between the Prolific and non-Prolific workers.
- Can you add confidence intervals for all of the critical numbers given in the paper, such as the zone shares and the agency figures?
- Additionally, while this cannot be covered in the time offered by the review, I think improving expert agreement would substantially improve the credence I can give to the data and subsequent analysis based on that data.
5 comments
Can you add confidence intervals for all of the critical numbers given in the paper, such as the zone shares and the agency figures?
Thanks for the suggestion. We revise the manuscript to report:
- The 95% confidence interval (N=10,000) for the percentage of tasks with mean desire > 3: 46.1% [40.6, 51.7]
- The 95% confidence interval (N=10,000) for automation desire-capability landscape zone share:
- Automation “Green Light” Zone: 43.1% [37.8, 48.8]
- Automation “Red Light” Zone: 31.6% [27.0, 36.3]
- R&D Opportunity Zone: 12.7% [9.6, 16.0]
- Low Priority Zone: 12.6% [9.5, 15.9]
- We update Figure 5 and Figure 8 to include the 95% confidence interval (N=10,000) for the distribution of worker-desired HAS levels () and expert-assessed feasible HAS levels () by occupations.
Given that the authors say that they intend to release the information, this seems concerning as the information is fairly sensitive in itself, and particularly so in relation to the automation of their work. It's also unclear if participants are aware of the extent of the information that would be revealed and whether they consented to this with full knowledge. I don't expect the risks of this to be high, but I don't think it follows good principles of data management and there is little advantage in releasing the personal information.
Thank you for raising this and we want to highlight that we take participant privacy seriously. All participants gave informed consent, and the study was approved by our institution's IRB.
In the public release, we will remove all interview transcripts and ZIP codes, so demographic fields will be linked only to scalar ratings. The demographic fields that remain are useful for stratified analyses like the ones above, and, with ZIP codes removed. Since each combination is shared by many workers within the same occupation, the deidentification risk is low.
Hi Reviewer LJ49,
Thank you again for your careful review! We have posted our rebuttal and revised the manuscript to clarify the quality control and include confidence intervals to our figures (changes marked in blue).
We would greatly appreciate it if you could take a look at our rebuttal and reconsider your assessment. Please let us know if any concerns remain or if further clarification would be helpful!
Best regards,
Submission 3247 Authors
We thank the reviewer for the careful analysis of our supplementary data, for acknowledging the novelty and scale of the study, and the efforts we made on data collection and documentation. Your review led to several new analyses that make the paper stronger and we address each of your concerns below.
Occupations are self-reported on the O*NET taxonomy: participants first choose a sector and then an occupation from a second menu. Additional checks depended on the platform: on Prolific, participants were screened so that the sector matched the industry in their Prolific profile; on Upwork we checked the match between the occupation in their profile and ours; for LinkedIn/word-of-mouth we did not filter, since the survey was distributed through personal networks. We add this to Appendix D.4.
We have the task-familiarity filter (Section 3.3) as additional safeguard. Participants only rate tasks they confirm they perform, and compensation does not depend on how many tasks they rate, so there is no incentive to claim familiarity.
After reading your review, we went through all participants and compared their interview reflections with their self-reported occupation. Of the 1,500 participants, 134 provided no ratings (none of the tasks shown matched what they do, likely because of a wrong occupation selection), leaving 1,366 workers whose ratings contribute to the WORKBank task-level results. We state this explicitly in the revised paper.
Thank you for highlighting this. We recomputed desire separately for Prolific and non-Prolific (Upwork + Linkedin/word of mouth) respondents, holding expert capability fixed, on the same 514 tasks rated by both groups:
Participants recruited from Prolific have lower automation desire in general compared to participants recruited from other sources. At the same time, this additional analysis reveals the core findings of the paper remain robust. In both groups, Red Light is at least as large as R&D Opportunity in the point estimates (35.8 vs 12.3; 20.6 vs 16.7). The qualitative pattern in the threshold-robustness analysis (Appendix B.1) is also unchanged. We include these results in Appendix D.4 Recruitment Sources.
Which group is more "representative"? We do not think either should be treated as ground truth, and this is why we deliberately recruited from several platforms.The difference we see is a useful finding about how measured desire varies with who is asked, which is also why WORKBank releases the recruitment source per worker so that users can stratify or reweight.