- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper combines an occupational-task usage index derived from public WildChat conversations with a synthetic tool-use benchmark built from O*NET tasks and MCP tools. It maps 668,380 conversations to at least one occupational task, and evaluates capability over nine occupations, 130 tasks, 100 servers, and more than 1,000 tools, primarily with Kimi-k2.5. Single-turn and simulated multi-turn settings are scored for tool use, workflow completion/autonomy, and grounding. The intended contribution is an inspectable bridge between observed task mentions and agents' potential professional capabilities.
Critical strengths
-
Public conversation data, explicit O*NET mappings, and detailed prompts make the task-usage analysis more inspectable than an index based entirely on proprietary interactions (Sections 2–3; mapping/filtering prompts in the appendix). The distinction between task mentions and successful execution is also explicitly recognized in parts of the discussion.
-
Separating workflow progress from tool-call correctness and grounding provides useful diagnostic information. The paper includes a human check of 50 trajectories: reported tool-call agreement is 86% and workflow exact agreement is 70.8% (Appendix A, p. 13). The observed decline in tool accuracy and grounding under simulated multi-turn interaction is worth investigating, even though the interpretation of the aggregate comparison needs revision.
Critical weaknesses
-
The central adoption–capability comparison does not use commensurate measures. Figure 1's usage statistic is the fraction of generated scenarios whose O*NET tasks appear at least four times in the index; capability is a synthetic-scenario completion fraction (p. 2). Neither the denominator nor the population is shared, and task mention coverage in selected English WildChat conversations is not an occupational adoption rate. The filtering also permits mappings based on shared core skills across domains. Consequently, the evidence supports task coverage within this sample, but does not by itself establish an economic adoption gap, productivity impact, or occupational replacement risk.
-
The evidence for collaboration gains and realistic capability remains difficult to interpret. Figure 9 compares single-turn Workflow Completion with multi-turn Autonomy under different rubrics, while restricting to scenarios successfully executed in both modes (p. 9; Tables 6 and 8). This cannot isolate an improvement in the same completion outcome. In addition, tool simulation may fabricate plausible missing values, and the workflow-informed user simulator can guide the agent (appendix prompts, pp. 28–32). Human agreement on scoring does not establish that these simulated trajectories correspond to valid professional outcomes; grounding is particularly weakly validated.
Questions for authors
-
Can the paired single-turn/multi-turn scenarios be rescored with the same completion rubric, including all agent failures in the denominator? Please report generated, executed, excluded, failed, and timed-out counts by mode so the comparison is interpretable.
-
How stable are Figure 1 and occupational rankings to the four-mention cutoff, skill-based mapping rule, and sampling choices? Please either justify a common estimand for adoption and capability or limit the economic claims to WildChat task coverage and the selected simulated scenarios.
-
Can a representative subset be calibrated against real MCP execution or blind expert assessment of final outcomes, including missing-data handling and grounding? This would directly test whether the simulator and judges measure the claimed professional capability.
2 comments
2. Stability of Figure 1 and the index
Index-level results are stable.
- Skills: the top-5 skills are identical for importance thresholds from 3.0 to 4.0, and under 1,000 bootstrap resamples of WildChat conversations the top-10 skills are identical and in the same order in 100% of resamples.
- Sectors and occupations: sector ordering is stable (Kendall τ = 0.99, 95% CI [0.97, 1.00]) and the top-20 occupations overlap 20/20.
- Mapping rule: alternative rules leave results largely unchanged (majority-vote assignment: 9/10 top skills, top-20 occupation overlap 15/20; omitting the LLM relevance filter: 10/10, 19/20).
- Depth: our depth estimate stays below prior work's 4% for every threshold t ≥ 4.
Figure 1. Usage is monotone in the mention cutoff N by construction, so rather than fix N we report each occupation's crossover N, i.e. the smallest cutoff at which capability exceeds usage. The mean crossover is N = 2.2 for autonomous and N = 2.7 for augmentative capability (median 2 and 1). Capability exceeds WildChat task coverage once a task must appear in more than ~2–3 conversations, in 17 of 18 occupation-mode pairs (the exception is the augmentative series for Journalists, N = 11). Capability is also essentially independent of coverage: success rates are similar for covered and uncovered scenarios (60.6% vs. 63.1% autonomous; 70.3% vs. 68.7% augmentative).
The ranking of occupations by usage is less stable across N (Kendall τ vs. N = 4: 0.34–0.87), so we will (i) relabel the usage axis as WildChat task coverage of our simulated scenarios, (ii) report crossover N in place of a single cutoff, and (iii) add bootstrap CIs and per-occupation n (Graphic Designers: n = 10).
3. Calibration against real execution and expert assessment
We already provide a comparison with human judges on a subset of the outcomes in Appendix A: the tool-call judge agrees with humans as often as two humans agree with each other (κ = 0.66 vs. 0.61), and workflow completion shows strong agreement with no bias (quadratic-weighted κ = 0.79). We agree grounding is the least validated dimension (agreement across independent frontier judges is low) and will state this as a limitation. The number of functional MCP servers available on Smithery is limited: of 627 connectable servers, 416 had at least one tool returning quality output, and only about a third of tools did so overall (see Appendix B). Given this, we do not think it is feasible to validate our scenarios with MCP server output. That said, we do ground all generated MCP calls in crawled MCP server documentation, so no tool schemas or descriptions are AI generated, only the responses.
We thank the reviewer for pushing on the robustness of our comparisons.
1. Single- vs. multi-turn comparison
Rescoring with the same completion rubric. As the reviewer points out, the workflow-informed user simulator can guide the agent in the collaborative setting. For this reason it does not make sense to use the single-turn rubric for multi-turn trajectories: it would not account for the fact that the agent gets multiple guided turns to attempt the tool workflow, and would credit completions achieved through user steering, conflating agent capability with simulated-user assistance. Autonomy is defined as workflow completion net of steering (score 5 = completed end-to-end except for essential withheld inputs), so it is a strictly harder criterion, and comparing single-turn Workflow Completion against multi-turn Autonomy (Figure 9) is conservative with respect to multi-turn performance. We will emphasize this in the updated draft.
Counts by mode. Both modes use the same 4,654 scenarios.
With failures in the denominator, the multi-turn advantage does not hold. However given that most of the additional failures are wall clock timeouts from additional required LLM churn, we do not view this as a particularly interesting finding. however we are happy to report it for clarity.
Simulated tools and fabricated values. The simulator validates calls against each tool's input schema (missing, unknown, or mistyped arguments are rejected), but it does not check argument values, and it may fabricate plausible content in its responses. Value correctness is instead enforced by Tool Call Accuracy, which compares each call's arguments against the answer key; in multi-turn scenarios the answer key contains the true withheld values, so an agent that guesses rather than asks scores 0 on that call. This is consistent with the lower multi-turn tool-call accuracy (0.43 vs. 0.56).