acceptodds
Under review as a conference paper at ICLR 2027

The Open Source Economic Index of AI Adoption and Capability

Abstract

We work towards measuring both AI adoption and the capability of AI to perform discrete labor tasks across various occupations. To measure adoption, we develop an open-source economic index that uses publicly available user-LLM chat data and ONET tasks to replicate studies produced by leading AI labs, finding that occupations in the finance, computer science, and arts sectors are those with the highest adoption rates. To measure capabilities, we build a system that generates benchmark scenarios grounded in ONET occupations, tasks, and model-context-protocol (MCP) servers. We test Kimi-k2.5 with an OpenAI agents SDK harness on scenarios across 9 occupations that appear frequently in our index, finding that AI correctly executes high-level workflows but often errs in the granular details (such as specific tool calls used).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Economic Index, Synthetic LLM Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_QvPwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper combines an occupational-task usage index derived from public WildChat conversations with a synthetic tool-use benchmark built from O*NET tasks and MCP tools. It maps 668,380 conversations to at least one occupational task, and evaluates capability over nine occupations, 130 tasks, 100 servers, and more than 1,000 tools, primarily with Kimi-k2.5. Single-turn and simulated multi-turn settings are scored for tool use, workflow completion/autonomy, and grounding. The intended contribution is an inspectable bridge between observed task mentions and agents' potential professional capabilities.

Critical strengths
  1. Public conversation data, explicit O*NET mappings, and detailed prompts make the task-usage analysis more inspectable than an index based entirely on proprietary interactions (Sections 2–3; mapping/filtering prompts in the appendix). The distinction between task mentions and successful execution is also explicitly recognized in parts of the discussion.

  2. Separating workflow progress from tool-call correctness and grounding provides useful diagnostic information. The paper includes a human check of 50 trajectories: reported tool-call agreement is 86% and workflow exact agreement is 70.8% (Appendix A, p. 13). The observed decline in tool accuracy and grounding under simulated multi-turn interaction is worth investigating, even though the interpretation of the aggregate comparison needs revision.

Critical weaknesses
  1. The central adoption–capability comparison does not use commensurate measures. Figure 1's usage statistic is the fraction of generated scenarios whose O*NET tasks appear at least four times in the index; capability is a synthetic-scenario completion fraction (p. 2). Neither the denominator nor the population is shared, and task mention coverage in selected English WildChat conversations is not an occupational adoption rate. The filtering also permits mappings based on shared core skills across domains. Consequently, the evidence supports task coverage within this sample, but does not by itself establish an economic adoption gap, productivity impact, or occupational replacement risk.

  2. The evidence for collaboration gains and realistic capability remains difficult to interpret. Figure 9 compares single-turn Workflow Completion with multi-turn Autonomy under different rubrics, while restricting to scenarios successfully executed in both modes (p. 9; Tables 6 and 8). This cannot isolate an improvement in the same completion outcome. In addition, tool simulation may fabricate plausible missing values, and the workflow-informed user simulator can guide the agent (appendix prompts, pp. 28–32). Human agreement on scoring does not establish that these simulated trajectories correspond to valid professional outcomes; grounding is particularly weakly validated.

Questions for authors
  1. Can the paired single-turn/multi-turn scenarios be rescored with the same completion rubric, including all agent failures in the denominator? Please report generated, executed, excluded, failed, and timed-out counts by mode so the comparison is interpretable.

  2. How stable are Figure 1 and occupational rankings to the four-mention cutoff, skill-based mapping rule, and sampling choices? Please either justify a common estimand for adoption and capability or limit the economic claims to WildChat task coverage and the selected simulated scenarios.

  3. Can a representative subset be calibrated against real MCP execution or blind expert assessment of final outcomes, including missing-data handling and grounding? This would directly test whether the simulator and judges measure the claimed professional capability.

2 comments

We thank the reviewer for pushing on the robustness of our comparisons.

1. Single- vs. multi-turn comparison

Rescoring with the same completion rubric. As the reviewer points out, the workflow-informed user simulator can guide the agent in the collaborative setting. For this reason it does not make sense to use the single-turn rubric for multi-turn trajectories: it would not account for the fact that the agent gets multiple guided turns to attempt the tool workflow, and would credit completions achieved through user steering, conflating agent capability with simulated-user assistance. Autonomy is defined as workflow completion net of steering (score 5 = completed end-to-end except for essential withheld inputs), so it is a strictly harder criterion, and comparing single-turn Workflow Completion against multi-turn Autonomy (Figure 9) is conservative with respect to multi-turn performance. We will emphasize this in the updated draft.

Counts by mode. Both modes use the same 4,654 scenarios.

Single-turnMulti-turn
Generated4,6544,654
Excluded before execution (malformed scenario)36
Executed4,6514,648
Timed out (1,200 s wall-clock)3541,279
Other harness/API errors73169
Completed and scored4,2243,200
  Perfect (score 5)2,5812,223
  Partial (score 2–4)1,380811
  Did not complete (score 1)2180
  Judge output unparseable24286
% perfect, matched completed scenarios (n = 3,050; as in paper)62.169.3
% perfect, all executed, failures counted as not perfect (n = 4,645)55.547.8

With failures in the denominator, the multi-turn advantage does not hold. However given that most of the additional failures are wall clock timeouts from additional required LLM churn, we do not view this as a particularly interesting finding. however we are happy to report it for clarity.

Simulated tools and fabricated values. The simulator validates calls against each tool's input schema (missing, unknown, or mistyped arguments are rejected), but it does not check argument values, and it may fabricate plausible content in its responses. Value correctness is instead enforced by Tool Call Accuracy, which compares each call's arguments against the answer key; in multi-turn scenarios the answer key contains the true withheld values, so an agent that guesses rather than asks scores 0 on that call. This is consistent with the lower multi-turn tool-call accuracy (0.43 vs. 0.56).

2. Stability of Figure 1 and the index

Index-level results are stable.

  • Skills: the top-5 skills are identical for importance thresholds from 3.0 to 4.0, and under 1,000 bootstrap resamples of WildChat conversations the top-10 skills are identical and in the same order in 100% of resamples.
  • Sectors and occupations: sector ordering is stable (Kendall τ = 0.99, 95% CI [0.97, 1.00]) and the top-20 occupations overlap 20/20.
  • Mapping rule: alternative rules leave results largely unchanged (majority-vote assignment: 9/10 top skills, top-20 occupation overlap 15/20; omitting the LLM relevance filter: 10/10, 19/20).
  • Depth: our depth estimate stays below prior work's 4% for every threshold t ≥ 4.

Figure 1. Usage is monotone in the mention cutoff N by construction, so rather than fix N we report each occupation's crossover N, i.e. the smallest cutoff at which capability exceeds usage. The mean crossover is N = 2.2 for autonomous and N = 2.7 for augmentative capability (median 2 and 1). Capability exceeds WildChat task coverage once a task must appear in more than ~2–3 conversations, in 17 of 18 occupation-mode pairs (the exception is the augmentative series for Journalists, N = 11). Capability is also essentially independent of coverage: success rates are similar for covered and uncovered scenarios (60.6% vs. 63.1% autonomous; 70.3% vs. 68.7% augmentative).

The ranking of occupations by usage is less stable across N (Kendall τ vs. N = 4: 0.34–0.87), so we will (i) relabel the usage axis as WildChat task coverage of our simulated scenarios, (ii) report crossover N in place of a single cutoff, and (iii) add bootstrap CIs and per-occupation n (Graphic Designers: n = 10).

3. Calibration against real execution and expert assessment

We already provide a comparison with human judges on a subset of the outcomes in Appendix A: the tool-call judge agrees with humans as often as two humans agree with each other (κ = 0.66 vs. 0.61), and workflow completion shows strong agreement with no bias (quadratic-weighted κ = 0.79). We agree grounding is the least validated dimension (agreement across independent frontier judges is low) and will state this as a limitation. The number of functional MCP servers available on Smithery is limited: of 627 connectable servers, 416 had at least one tool returning quality output, and only about a third of tools did so overall (see Appendix B). Given this, we do not think it is feasible to validate our scenarios with MCP server output. That said, we do ground all generated MCP calls in crawled MCP server documentation, so no tool schemas or descriptions are AI generated, only the responses.

Reviewer_1cgtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an analysis of the usage and capabilities of AI across ONET tasks. The contribution fall into the research line of the impact of AI on jobs and work, and specifically here the authors contribute an open source version of such analyses by using open source data (WildChat).The paper complements the analysis of usage for work with an AI capabilities assessment, using LLM as a judge with human labeling for validation.

Results characterize the distribution of work done with AI (in the WildChat dataset) to the distribution of workers in the U.S., highlighting some notable differences such as a far higher percent of design work in the AI usage dataset than in occupations, and the opposite for food workers.

Critical strengths
  1. Connecting AI usage and capabilities for the same tasks is genuinely useful, and avoids just looking at capabilities or exposure alone.
  2. The open source nature of the work supports reproducability, something very needed given that a lot of the similar work comes from model labs and other closed environments.
Critical weaknesses
  1. WildChat as a data source. It's dated now (which matters a lot given the pace of improvement in AI capabilities) and likely not representative of the broader workforce. I realize that's a super tough criticism given that a big motivation and contribution for the paper is to be open (and the "privatization of science" is a real issue), but it's just hard to know how much to take away from these results given the nature of the primary dataset.
  2. The capability benchmark covers only 9 occupations (correct?). There are more than 900 occupations in ONET, so this captures only a tiny fraction of all occupations.
Questions for authors
  1. Is there a way to feel better about generalizing from WildChat-bsaed task frequencies given the dated and likely skewed nature of these chat sessions?
  2. How much does the MCP server availability effect selection of the ONET tasks and occupations that are then included in the benchmark? Is that filtering a meaningful source of bias in any downstream results?
1 comment

Thank you for reviewing our submission

WildChat as a data source. It's dated now (which matters a lot given the pace of improvement in AI capabilities) and likely not representative of the broader workforce. I realize that's a super tough criticism given that a big motivation and contribution for the paper is to be open (and the "privatization of science" is a real issue), but it's just hard to know how much to take away from these results given the nature of the primary dataset.

We agree that Wildchat is dated at this point, but we think that the chat-index section of our paper still provides a valid contribution. Namely, AI labs such as Anthropic and OpenAI have released studies based on similar data, and our findings are one of the few to provide any corroboration on publicly available data. Additionally, the methodology is theoretically applicable to any updated publicly available chat data that is released down the line.

How much does the MCP server availability effect selection of the ONET tasks and occupations that are then included in the benchmark? Is that filtering a meaningful source of bias in any downstream results?

This is a valid concern for using real MCP servers to simulate agentic interaction. Our code provides an alternative: using the crawled MCP meta data and having another agent provide simulated tool outputs. This ensures that occupations have access to any MCP server, as well as all tools with documentation on smithery, which is a much larger set than all of the actual functioning tools (see appendix B).

Appendix B.1. may also be of interest to the reviewer: there we study how well various standardized software categories are covered by MCP servers. The simulated set provides decent coverage (49/137 categories), but we do believe a sandbox with tools of all 137 types could likely provide scenarios that are more difficult (for example, requiring the need to continually edit a powerpoint).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.