- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper describes TaskAtlas, a framework for discovering and organizing data-analysis task spaces via interviews with simulated analysts with diverse occupational backgrounds. It leverages O*NET occupational records to select occupations of relevance, construct analyst profiles, elicit structured analytical needs through adaptive multi-turn interviews, map these needs to an existing taxonomy, and group needs not covered by the existing taxonomy into candidate task types. The authors evaluate their framework in the domain of tabular text analysis and causal inference against direct generation, single-analyst baseline, and an adapted Elicitron method. It obtains higher external corroboration rates and semantic diversity while covering all 12 and 13 reference task categories in the respective domains. The former is defined as a judgment of LLM whether retrieved public-discussion snippet addresses the same underlying problems as a task. The paper also proposes two candidate task extensions: discovering attributes and generating structured columns from existing text for a predictive objective (TA++), detecting latent confounding with explicit consideration of identifiability (LCD). The primary contribution of the paper is the creation of a traceable, occupationally grounded workflow connecting simulated needs elicitation to task-space extension, supported by empirical evaluations of external corroboration and reference-task coverage.
Critical strengths
- Framework explicitly specifies occupation sources and maintains traceability from elicited needs to task types. TaskAtlas leverages O*NET records to construct simulated analysts, performs multi-turn interviews to ground analytical objectives, data, and expected outputs, and preserves associations between resulting task types and source needs. Such transparency of the task-space construction process enables researchers to examine the examples that contribute to a given category and better understand what they represent. With its ability to systematically integrate different occupational contexts, the framework provides a valuable methodological contribution to the design of data-analysis benchmarks by offering a way to make choices about which tasks to evaluate.
- Comparisons across two domains provide empirical support for generating diverse needs with external corroboration. On the paper’s retrieval and judging procedure, TaskAtlas obtains external corroboration rates of 81.6% in tabular text analysis and 68.0% in causal inference, outperforming direct generation, single analyst baseline, adapted Elicitron, and achieving higher semantic diversity than these approaches. The complementary outcomes indicate that the framework generates more diverse need descriptions while increasing the fraction supported by public discussions of related problems, providing evidence of its potential as an aid for task-space construction.
Critical weaknesses
- The evaluation gives limited independent validation that occupational grounding yields needs that truly reflect the associated work contexts. External corroboration is defined by an LLM assessing if retrieved snippets address the same underlying problem, while query rewriting eliminates role- and organization-specific details. This can provide valuable evidence that similar problems arise in public discussions, but it does not confirm the validity of the simulated connections between occupations, workflows, and analytical needs. The paper provides neither practitioner validation of these connections nor human validation of the crucial evidence-support judgments. As a result, the higher corroboration rates do not clearly distinguish improved alignment with actual occupational needs from descriptions that are easier to match to online discussions. This limits the evidence for the distinctive value of occupational grounding.
- The evaluation of the task-space extension does not adequately demonstrate the added value of the proposed task types. Reference-task coverage requires only one supporting need per category, and in the tabular text analysis, both the single-analyst baseline and Elicitron already achieve full coverage, while in causal inference, TaskAtlas covers one additional category relative to Elicitron. These results support the task coverage of established objectives but provide limited evidence of task discovery capabilities beyond the reference inventories. The two candidate extensions are mainly supported by illustrative needs and task formulations: TA++ explicitly combines open-schema extraction and column generation, while LCD targets an objective not separately represented in the chosen causal taxonomy. The paper does not provide sufficient comparative detail with existing formulations or independent expert assessment to establish what substantively new analytical objectives these extensions bring. Thus, the evidence is stronger for the value of the needs-generation workflow than for task-space extension as a central contribution.
Questions for authors
- Validation of occupational grounding and external corroboration (clarification request): Have the authors performed any human checks of the external-support labels or independent assessments of whether the generated needs are suitable for their associated occupations and work contexts? If so, please describe your sampling procedure, assessment criteria, and results, specifying what evidence you have that a similar problem was recognized in practice and what evidence you have that the need is appropriate for the simulated occupational context. This would be helpful information for evaluating the degree to which the higher corroboration rates in these data support the value of occupational grounding.
- Contribution of the candidate task extensions (clarification request): The paper presents TA++ and LCD as candidate task extensions. Could the authors clarify whether their contribution is primarily in the combination of existing capabilities, incorporation of established problems into the reference taxonomy, or inclusion of analytical objectives that are substantially different from existing formulations? Using the source needs and the closest existing task definitions, please explain what is added by objective-guided attribute discovery in TA++ and latent-common-cause assessment in LCD. This would be helpful information for evaluating the significance of the task-space extensions and the framework's demonstrated discovery capabilities.
4 comments
Response to Q2
[Contribution scope.] The submitted paper presents TA++ and LCD as candidate task extensions. TA++ primarily combines existing capabilities under the constraints of a downstream objective. LCD primarily incorporates latent-common-cause assessment explicitly into the reference taxonomy and specifies its output and identifiability requirements.*
[Source need and additional requirements of TA++.] In the interview with the simulated environmental restoration planner, P_19_2041_02, the analyst asks whether narratives from initial site assessments can predict subsequent restoration outcomes and which narrative characteristics are associated with those outcomes. This need specifies a prediction objective, while the relevant textual attributes remain to be discovered. The closest existing tasks are open-schema extraction in T4 and column generation in T12. TA++ adds the requirement that the downstream objective guide attribute-schema discovery and selection. The system must determine which narrative characteristics should become structured columns, generate values grounded in the source text for each record, and assess the utility of the resulting representation for the prediction objective. Different objectives may lead to different attribute selections for the same text-containing table.TA++ therefore brings attribute discovery, attribute-value extraction, and downstream utility into a single task specification. Its evaluation should assess both the faithfulness of attribute values and the predictive utility of the resulting representation on held-out data. The extension’s value lies in the objective’s constraints on the combination of existing capabilities and the resulting joint evaluation requirements.
[Source need and additional requirements of LCD.] In the facilities manager’s need, P_11_3013_00-Q4, the analyst observes that work-order volume and satisfaction vary together and asks whether factors such as occupant expectations influence both, as well as what additional data would be needed to answer this question. The original mapping associates this need with A1, D1, and D2, reflecting its connections to existing tasks. In the reference taxonomy, A1 concerns the causal direction between two variables, D1 concerns confounders, adjustment variables, and the fairness of comparisons, and D2 concerns the sensitivity of conclusions to changes in assumptions. LCD further specifies whether a particular pair of variables shares a latent common cause* as the direct analytical objective and requires an assessment of whether the available data and assumptions support that determination.
Specifically, LCD takes observational data and data-generating assumptions as inputs and returns yes, no, or not-identifiable, together with a confidence assessment and a diagnostic explanation. If mechanisms implying different conclusions about confounding satisfy the given assumptions and induce the same observational distribution, the correct output is not-identifiable. Evaluation should accordingly cover judgment accuracy in identifiable cases, correct withholding of judgment in non-identifiable cases, and the consistency of explanations with the assumptions and evidence.
LCD thus organizes this established causal-analysis problem into an explicit task within the reference taxonomy, specifying the question to be answered, the conditions supporting a judgment, and the expected outputs.
[Implications for the framework’s discovery capabilities.] These examples illustrate how TaskAtlas identifies analytical objectives that warrant combination or refinement from concrete occupational needs and develops them into candidate task specifications. TA++ highlights how prediction objectives constrain attribute discovery, and LCD highlights the identifiability and diagnostic-explanation requirements of latent-common-cause assessment. The source needs provide a traceable basis for these abstractions and help specify the inputs, expected outputs, and evaluation requirements for future benchmark construction.
Thank you for the clarifications
Thank you for the detailed responses and additional experiments! The human audit and controlled O*NET comparison meaningfully address my concern about occupational grounding.
The clarification of TA++ and LCD is also helpful. However, their value as task-space extensions remains supported mainly by illustrative needs and task definitions, rather than comparative evaluation or independent assessment. Thus, my original concern remains: the evidence is stronger for the needs-generation workflow than for task-space extension as a central contribution. I therefore maintain my current rating, while appreciating the authors’ efforts to strengthen the paper.
Clarifying the Novelty of TA++ and LCD
Thank you for your follow-up and for recognizing that the additional human audit and controlled O*NET comparison meaningfully address your concerns about occupational grounding. Regarding task-space extension, we would like to further clarify the novelty of TA++ and LCD.
We believe that TA++ and LCD contribute new task objectives and evaluation requirements. TA++ requires discovering and generating useful attributes guided by a downstream objective; text extraction or column generation alone is insufficient to fulfill this task. LCD requires determining whether two observed variables share a latent common cause and correctly distinguishing among “yes,” “no,” and “not identifiable” given the available data and assumptions.
These requirements constitute explicit new task formulations.
Response to Q1
Thank you for your careful review of our paper. We appreciate your thoughtful questions and constructive suggestions on validating occupational grounding and clarifying the contributions of TA++ and LCD as candidate task extensions.
[Human Audit Setup.] In each of the TA and Causal domains, we randomly sampled 20% of the needs from TaskAtlas and Elicitron for which automated evaluation had been completed, retaining all retrieved snippets associated with these needs. One annotator conducted the audit, with method names, LLM labels, and their explanations hidden. The audit comprised two assessments.(1) External support for the general problem. The annotator received the same information as the original judge and applied the
support,partial, andunrelatedcriteria in Appendix E.2 to determine whether each snippet provided evidence that a similar analytical problem had been encountered in practice. Such evidence included practitioners describing the problem, seeking solutions, or discussing relevant tools or methods. Following the original definition of Coverage@1, a need was counted as externally corroborated if at least one snippet was labeledsupport.(2) External support for the occupational context. The annotator further considered the occupational responsibilities, relevant data, and intended use of the results recorded in each need, examining whether the snippets provided evidence corresponding to the described work context. A need contributed to the occupational context support rate only if at least one snippet supported both the general problem and its occupational context.[Human Audit Results.] The table below reports need-level human–LLM agreement, human Coverage@1, and occupational context support rates. Human–LLM agreement was calculated based on whether a need had at least one snippet labeled
support. All metrics were computed on the same audited set within each method–domain combination.Need-level agreement between human and LLM judgments ranged from 92.6% to 95.3%. Using human labels, TaskAtlas’s Coverage@1 exceeded Elicitron’s by 18.1 and 10.3 percentage points in TA and Causal, respectively, preserving the relative ranking of the two methods in the audited samples. When occupational context support was additionally required, TaskAtlas achieved support rates of 53.5% and 40.2%, exceeding Elicitron by 15.7 and 11.1 percentage points, respectively. [Controlled Comparison of O*NET Grounding.] To further examine the contribution of occupational information, we compared conditions with and without ONET records in both domains. Both conditions used
claude-opus-4.7with 160 analysts per condition. We held the occupation list, persona-generation templates and procedure, adaptive interview protocol and budget, and downstream need extraction and evaluation rules fixed. The experimental variable was whether the corresponding ONET occupational records were provided during persona generation.The table below reports results for both domains.
Adding O*NET records increased Coverage@1 by 10.4 and 8.7 percentage points in TA and Causal, respectively, and increased Diversity by 0.03 and 0.04. Both domains covered one additional reference task category. These results support the incremental contribution of occupational information to external corroboration, diversity, and reference task coverage.