acceptodds
Under review as a conference paper at ICLR 2027

TaskAtlas: Discovering Data-Analysis Task Spaces from a Simulated Analyst Society

Abstract

Constructing data-analysis benchmarks requires discovering real-world analytical tasks. Existing efforts typically rely on expert-led task-space construction, which requires continued human effort and may underrepresent analytical tasks across diverse occupations. To address these issues, we introduce TaskAtlas, a framework for discovering data-analysis task spaces from an occupationally anchored simulated analyst society. TaskAtlas selects occupations relevant to the target analytical domain from public occupational records and uses their profiles to construct simulated analysts with diverse backgrounds and work contexts. It then employs an interviewer agent to elicit their analytical needs through multi-turn conversations, extracts tasks from these needs, and organizes them into an explicit task space. We evaluate TaskAtlas in tabular text analysis and causal inference, examining the authenticity of elicited needs and the coverage of the discovered task space. The results show that the elicited needs receive external support from real world and the task space recover established tasks in expert-curated taxonomy. More importantly, TaskAtlas enables new task formulations to emerge, including text augmentation for predictive data analysis (TA++) and latent-confounder detection (LCD), offering new directions for data-analysis research.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
data analysis benchmark, agent simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uj3LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper describes TaskAtlas, a framework for discovering and organizing data-analysis task spaces via interviews with simulated analysts with diverse occupational backgrounds. It leverages O*NET occupational records to select occupations of relevance, construct analyst profiles, elicit structured analytical needs through adaptive multi-turn interviews, map these needs to an existing taxonomy, and group needs not covered by the existing taxonomy into candidate task types. The authors evaluate their framework in the domain of tabular text analysis and causal inference against direct generation, single-analyst baseline, and an adapted Elicitron method. It obtains higher external corroboration rates and semantic diversity while covering all 12 and 13 reference task categories in the respective domains. The former is defined as a judgment of LLM whether retrieved public-discussion snippet addresses the same underlying problems as a task. The paper also proposes two candidate task extensions: discovering attributes and generating structured columns from existing text for a predictive objective (TA++), detecting latent confounding with explicit consideration of identifiability (LCD). The primary contribution of the paper is the creation of a traceable, occupationally grounded workflow connecting simulated needs elicitation to task-space extension, supported by empirical evaluations of external corroboration and reference-task coverage.

Critical strengths
  1. Framework explicitly specifies occupation sources and maintains traceability from elicited needs to task types. TaskAtlas leverages O*NET records to construct simulated analysts, performs multi-turn interviews to ground analytical objectives, data, and expected outputs, and preserves associations between resulting task types and source needs. Such transparency of the task-space construction process enables researchers to examine the examples that contribute to a given category and better understand what they represent. With its ability to systematically integrate different occupational contexts, the framework provides a valuable methodological contribution to the design of data-analysis benchmarks by offering a way to make choices about which tasks to evaluate.
  2. Comparisons across two domains provide empirical support for generating diverse needs with external corroboration. On the paper’s retrieval and judging procedure, TaskAtlas obtains external corroboration rates of 81.6% in tabular text analysis and 68.0% in causal inference, outperforming direct generation, single analyst baseline, adapted Elicitron, and achieving higher semantic diversity than these approaches. The complementary outcomes indicate that the framework generates more diverse need descriptions while increasing the fraction supported by public discussions of related problems, providing evidence of its potential as an aid for task-space construction.
Critical weaknesses
  1. The evaluation gives limited independent validation that occupational grounding yields needs that truly reflect the associated work contexts. External corroboration is defined by an LLM assessing if retrieved snippets address the same underlying problem, while query rewriting eliminates role- and organization-specific details. This can provide valuable evidence that similar problems arise in public discussions, but it does not confirm the validity of the simulated connections between occupations, workflows, and analytical needs. The paper provides neither practitioner validation of these connections nor human validation of the crucial evidence-support judgments. As a result, the higher corroboration rates do not clearly distinguish improved alignment with actual occupational needs from descriptions that are easier to match to online discussions. This limits the evidence for the distinctive value of occupational grounding.
  2. The evaluation of the task-space extension does not adequately demonstrate the added value of the proposed task types. Reference-task coverage requires only one supporting need per category, and in the tabular text analysis, both the single-analyst baseline and Elicitron already achieve full coverage, while in causal inference, TaskAtlas covers one additional category relative to Elicitron. These results support the task coverage of established objectives but provide limited evidence of task discovery capabilities beyond the reference inventories. The two candidate extensions are mainly supported by illustrative needs and task formulations: TA++ explicitly combines open-schema extraction and column generation, while LCD targets an objective not separately represented in the chosen causal taxonomy. The paper does not provide sufficient comparative detail with existing formulations or independent expert assessment to establish what substantively new analytical objectives these extensions bring. Thus, the evidence is stronger for the value of the needs-generation workflow than for task-space extension as a central contribution.
Questions for authors
  1. Validation of occupational grounding and external corroboration (clarification request): Have the authors performed any human checks of the external-support labels or independent assessments of whether the generated needs are suitable for their associated occupations and work contexts? If so, please describe your sampling procedure, assessment criteria, and results, specifying what evidence you have that a similar problem was recognized in practice and what evidence you have that the need is appropriate for the simulated occupational context. This would be helpful information for evaluating the degree to which the higher corroboration rates in these data support the value of occupational grounding.
  2. Contribution of the candidate task extensions (clarification request): The paper presents TA++ and LCD as candidate task extensions. Could the authors clarify whether their contribution is primarily in the combination of existing capabilities, incorporation of established problems into the reference taxonomy, or inclusion of analytical objectives that are substantially different from existing formulations? Using the source needs and the closest existing task definitions, please explain what is added by objective-guided attribute discovery in TA++ and latent-common-cause assessment in LCD. This would be helpful information for evaluating the significance of the task-space extensions and the framework's demonstrated discovery capabilities.
4 comments
AuthorsOpenReview ↗

Response to Q1

Thank you for your careful review of our paper. We appreciate your thoughtful questions and constructive suggestions on validating occupational grounding and clarifying the contributions of TA++ and LCD as candidate task extensions.

[Human Audit Setup.] In each of the TA and Causal domains, we randomly sampled 20% of the needs from TaskAtlas and Elicitron for which automated evaluation had been completed, retaining all retrieved snippets associated with these needs. One annotator conducted the audit, with method names, LLM labels, and their explanations hidden. The audit comprised two assessments.(1) External support for the general problem. The annotator received the same information as the original judge and applied the support, partial, and unrelated criteria in Appendix E.2 to determine whether each snippet provided evidence that a similar analytical problem had been encountered in practice. Such evidence included practitioners describing the problem, seeking solutions, or discussing relevant tools or methods. Following the original definition of Coverage@1, a need was counted as externally corroborated if at least one snippet was labeled support.(2) External support for the occupational context. The annotator further considered the occupational responsibilities, relevant data, and intended use of the results recorded in each need, examining whether the snippets provided evidence corresponding to the described work context. A need contributed to the occupational context support rate only if at least one snippet supported both the general problem and its occupational context.

[Human Audit Results.] The table below reports need-level human–LLM agreement, human Coverage@1, and occupational context support rates. Human–LLM agreement was calculated based on whether a need had at least one snippet labeled support. All metrics were computed on the same audited set within each method–domain combination.

MethodTA human–LLM agreementTA human Coverage@1TA occupational context support rateCausal human–LLM agreementCausal human Coverage@1Causal occupational context support rate
TaskAtlas95.3%80.3%53.5%94.1%66.7%40.2%
Elicitron93.8%62.2%37.8%92.6%56.4%29.1%

Need-level agreement between human and LLM judgments ranged from 92.6% to 95.3%. Using human labels, TaskAtlas’s Coverage@1 exceeded Elicitron’s by 18.1 and 10.3 percentage points in TA and Causal, respectively, preserving the relative ranking of the two methods in the audited samples. When occupational context support was additionally required, TaskAtlas achieved support rates of 53.5% and 40.2%, exceeding Elicitron by 15.7 and 11.1 percentage points, respectively. [Controlled Comparison of O*NET Grounding.] To further examine the contribution of occupational information, we compared conditions with and without ONET records in both domains. Both conditions used claude-opus-4.7 with 160 analysts per condition. We held the occupation list, persona-generation templates and procedure, adaptive interview protocol and budget, and downstream need extraction and evaluation rules fixed. The experimental variable was whether the corresponding ONET occupational records were provided during persona generation.

The table below reports results for both domains.

O*NET groundingTA Coverage@1 ↑TA Diversity ↑TA Task Coverage ↑Causal Coverage@1 ↑Causal Diversity ↑Causal Task Coverage ↑
✗70.4%0.7211/1258.7%0.7012/13
✓80.8%0.7512/1267.4%0.7413/13

Adding O*NET records increased Coverage@1 by 10.4 and 8.7 percentage points in TA and Causal, respectively, and increased Diversity by 0.03 and 0.04. Both domains covered one additional reference task category. These results support the incremental contribution of occupational information to external corroboration, diversity, and reference task coverage.

AuthorsOpenReview ↗

Response to Q2

[Contribution scope.] The submitted paper presents TA++ and LCD as candidate task extensions. TA++ primarily combines existing capabilities under the constraints of a downstream objective. LCD primarily incorporates latent-common-cause assessment explicitly into the reference taxonomy and specifies its output and identifiability requirements.*

[Source need and additional requirements of TA++.] In the interview with the simulated environmental restoration planner, P_19_2041_02, the analyst asks whether narratives from initial site assessments can predict subsequent restoration outcomes and which narrative characteristics are associated with those outcomes. This need specifies a prediction objective, while the relevant textual attributes remain to be discovered. The closest existing tasks are open-schema extraction in T4 and column generation in T12. TA++ adds the requirement that the downstream objective guide attribute-schema discovery and selection. The system must determine which narrative characteristics should become structured columns, generate values grounded in the source text for each record, and assess the utility of the resulting representation for the prediction objective. Different objectives may lead to different attribute selections for the same text-containing table.TA++ therefore brings attribute discovery, attribute-value extraction, and downstream utility into a single task specification. Its evaluation should assess both the faithfulness of attribute values and the predictive utility of the resulting representation on held-out data. The extension’s value lies in the objective’s constraints on the combination of existing capabilities and the resulting joint evaluation requirements.

[Source need and additional requirements of LCD.] In the facilities manager’s need, P_11_3013_00-Q4, the analyst observes that work-order volume and satisfaction vary together and asks whether factors such as occupant expectations influence both, as well as what additional data would be needed to answer this question. The original mapping associates this need with A1, D1, and D2, reflecting its connections to existing tasks. In the reference taxonomy, A1 concerns the causal direction between two variables, D1 concerns confounders, adjustment variables, and the fairness of comparisons, and D2 concerns the sensitivity of conclusions to changes in assumptions. LCD further specifies whether a particular pair of variables shares a latent common cause* as the direct analytical objective and requires an assessment of whether the available data and assumptions support that determination.

Specifically, LCD takes observational data and data-generating assumptions as inputs and returns yes, no, or not-identifiable, together with a confidence assessment and a diagnostic explanation. If mechanisms implying different conclusions about confounding satisfy the given assumptions and induce the same observational distribution, the correct output is not-identifiable. Evaluation should accordingly cover judgment accuracy in identifiable cases, correct withholding of judgment in non-identifiable cases, and the consistency of explanations with the assumptions and evidence.

LCD thus organizes this established causal-analysis problem into an explicit task within the reference taxonomy, specifying the question to be answered, the conditions supporting a judgment, and the expected outputs.

[Implications for the framework’s discovery capabilities.] These examples illustrate how TaskAtlas identifies analytical objectives that warrant combination or refinement from concrete occupational needs and develops them into candidate task specifications. TA++ highlights how prediction objectives constrain attribute discovery, and LCD highlights the identifiability and diagnostic-explanation requirements of latent-common-cause assessment. The source needs provide a traceable basis for these abstractions and help specify the inputs, expected outputs, and evaluation requirements for future benchmark construction.

Reviewer_uj3LOpenReview ↗

Thank you for the clarifications

Thank you for the detailed responses and additional experiments! The human audit and controlled O*NET comparison meaningfully address my concern about occupational grounding.

The clarification of TA++ and LCD is also helpful. However, their value as task-space extensions remains supported mainly by illustrative needs and task definitions, rather than comparative evaluation or independent assessment. Thus, my original concern remains: the evidence is stronger for the needs-generation workflow than for task-space extension as a central contribution. I therefore maintain my current rating, while appreciating the authors’ efforts to strengthen the paper.

AuthorsOpenReview ↗

Clarifying the Novelty of TA++ and LCD

Thank you for your follow-up and for recognizing that the additional human audit and controlled O*NET comparison meaningfully address your concerns about occupational grounding. Regarding task-space extension, we would like to further clarify the novelty of TA++ and LCD.

We believe that TA++ and LCD contribute new task objectives and evaluation requirements. TA++ requires discovering and generating useful attributes guided by a downstream objective; text extraction or column generation alone is insufficient to fulfill this task. LCD requires determining whether two observed variables share a latent common cause and correctly distinguishing among “yes,” “no,” and “not identifiable” given the available data and assumptions.

These requirements constitute explicit new task formulations.

Reviewer_xBYsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TaskAtlas interviews LLM-simulated analysts grounded in O*NET occupational records to elicit data-analysis needs and extend a task taxonomy. Interviews are converted into structured need records; needs not covered by the reference taxonomy are grouped into candidate task types. Experiments cover tabular text analysis and causal inference, with Direct-CoT, Single Analyst, and an adapted Elicitron as baselines. TaskAtlas reports external corroboration rates of 81.6% and 68.0% using web retrieval and an LLM judge, and covers all 12 and 13 reference categories, respectively. Text augmentation for predictive analysis and latent-confounder detection illustrate proposed extensions. The contribution concerns task discovery and specification, rather than algorithms for solving those tasks.

Critical strengths
  1. The framework makes the sources of proposed tasks inspectable. The documented recruitment run selects 160 occupations from 1,016 candidates, and need records retain analyst identifiers and supporting interview excerpts (Section 4 and Appendices A-D). This traceability is useful for examining and revising a benchmark's task inventory.
  2. The results provide encouraging evidence about the elicited needs. Compared with Elicitron, TaskAtlas has higher external corroboration in both domains: 81.6% versus 64.8% for tabular text analysis and 68.0% versus 58.3% for causal inference, along with higher embedding diversity (Table 3). The logistics example shows how follow-up questions specify the desired output and its intended use.
Critical weaknesses
  1. The external corroboration measure has not been calibrated against human judgments. Coverage@1 counts a need as supported when GPT-4o-mini accepts at least one retrieved snippet. Search-query rewriting removes company- and role-specific nouns, and a discussion of a solution can count as support (Appendices E.1-E.3). This can establish that a general problem occurs, but offers weaker evidence for the simulated occupational context. The authors acknowledge the metric's limits; a small manual audit is still needed to assess whether the reported gains reflect more credible needs under the intended interpretation.
  2. The comparison does not isolate the contribution of occupational grounding. Elicitron already uses diverse simulated users and contextual interviews, while Section 5 does not fully specify models, population sizes, or generation budgets for each method. Appendix C.2 leaves the archived TA backend model unresolved and does not verify sampling settings for every historical run. Appendix C.1 gives a prompt wrapper with a {protocol} placeholder, but omits the actual domain protocols. Profiles also contain generated workflow difficulties before the interview (Appendix B.1). Without controlling model capability, interview budget, and this profile content, the gains cannot confidently be attributed to occupational grounding.
  3. The evidence for useful task-space extensions is limited. Single Analyst and Elicitron also cover all 12 TA categories; in causal inference, TaskAtlas covers 13 categories against Elicitron's 12 (Table 4). Section 6 therefore carries much of the evidence for extending the reference inventories. TA++ explicitly combines T4 and T12, and LCD overlaps with established work on latent-confounded causal discovery. These connections do not rule out a valuable task formulation, but the paper should identify the operational requirement or evaluation target that existing formulations do not capture. The acknowledged non-identifiability of LCD is appropriate and is not itself a flaw. My weak-reject recommendation rests primarily on the unvalidated interpretation of the corroboration results and the unresolved attribution of gains to occupational grounding. The traceable pipeline is useful, but the current comparisons leave its added value uncertain. A focused human audit and a controlled comparison could materially improve my assessment; clarifying the distinct requirements of TA++ and LCD would strengthen the task-discovery contribution.
Questions for authors
  1. Could you audit a small sample of need-snippet pairs from TaskAtlas and Elicitron, blinded to method and judge label? Please distinguish support for the general problem from support for the occupational context. Also report the evaluated denominators and sampling rules behind Table 3; Appendix E.3 provides 516/632 for TaskAtlas TA, but the corresponding details for the other results are unclear.
  2. Which models, domain protocols, analyst counts, and generation budgets produced Tables 3-4? Please identify any settings that cannot be recovered. A focused comparison in one domain, holding the model, analyst count, interview budget, and profile-generation procedure fixed while varying O*NET grounding, would help test the claimed contribution. The comparison should also control how initial workflow difficulties are supplied.
  3. For each proposed extension, could you trace one example from its source interview through extracted attributes and grouping to the final task definition? What requirement makes TA++ more than an instance of T4/T12, and what distinguishes LCD's requested output or evaluation from prior latent-confounding work? For LCD, a relevant comparison is Kaltenpoth and Vreeken, Nonlinear Causal Discovery with Latent Confounders (ICML 2023): https://proceedings.mlr.press/v202/kaltenpoth23a.html
3 comments
AuthorsOpenReview ↗

Response to Q1

Thank you for taking the time to review our paper. We appreciate your detailed feedback and constructive suggestions on human validation, controlled comparisons, and the evidence supporting the proposed task extensions.

[Setup.] For each method–domain combination, we randomly sampled 20% of the needs from TaskAtlas and Elicitron for which automated evaluation had been completed. We conducted a blinded human audit of these needs and all their retrieved snippets. One annotator performed all assessments. Method names, LLM labels, and their explanations were hidden during annotation, and the annotator received the same need descriptions, occupational information, and retrieved snippets as the original judge. Each need–snippet pair was assessed along two dimensions. 1. Support for the general problem. Following the criteria in Appendix E.2, the annotator assessed whether the snippet described or addressed the same type of analytical problem as the need. 2. Support for the occupational context. The annotator assessed whether the snippet provided evidence matching the occupational responsibilities, data usage, or decision-making context described in the need.

Following the original definition of Coverage@1, we counted a need as externally corroborated at the general problem level if at least one snippet was labeled support. We also calculated need-level agreement between human and LLM judgments.

[Results.] Human and LLM Coverage@1 values in the table below were computed on the same audited samples.

MethodTA need-level agreementTA Coverage@1 (Human / LLM)TA differenceTA occupational context support rateCausal need-level agreementCausal Coverage@1 (Human / LLM)Causal differenceCausal occupational context support rate
TaskAtlas95.3%80.3% / 81.9%−1.6 pp53.5%94.1%66.7% / 68.4%−1.7 pp40.2%
Elicitron93.8%62.2% / 64.4%−2.2 pp37.8%92.6%56.4% / 58.1%−1.7 pp29.1%

Evaluated denominators and sampling rules for the original Table 3. In Table 3, # Needs denotes the total number of generated needs. Coverage@1 is calculated as the number of needs with at least one supporting snippet divided by the number of needs actually evaluated. For example, TaskAtlas generated 1,333 needs in TA. We randomly sampled up to four needs from each of 158 personas using a random seed of 42, yielding 632 evaluated needs. Of these, 516 had at least one supporting snippet, giving a Coverage@1 of 516/632, or 81.6%. The table below summarizes the counts and sampling rules for each method in both domains.

MethodTA supported / evaluated needsTA evaluation procedureCausal supported / evaluated needsCausal evaluation procedure
Direct-CoT948 / 1,809All needs evaluated796 / 1,822All needs evaluated
Single Analyst1,090 / 1,752All needs evaluated896 / 1,656All needs evaluated
Elicitron990 / 1,528All needs evaluated798 / 1,368All needs evaluated
TaskAtlas516 / 632Up to four needs per persona, seed 42592 / 871Up to four needs per persona, seed 42

[Analysis.] Human and LLM judgments of whether a need received support for the general problem agreed in 92.6%–95.3% of cases. TaskAtlas’s human Coverage@1 exceeded Elicitron’s by 18.1 and 10.3 percentage points in TA and Causal, preserving the relative ranking of the two methods in the audited samples. The occupational context support rates further clarify the scope of the retrieved evidence. For TaskAtlas, 53.5% and 40.2% of needs received occupational context support in the two domains, respectively, compared with 37.8% and 29.1% for Elicitron.

AuthorsOpenReview ↗

Response to Q2

[Original experimental settings.] Appendices A–E describe occupational recruitment, persona initialization, interviewing, need extraction, and external corroboration. (1) Models. In both TA and Causal, the generation model for Direct-CoT, Single Analyst, Elicitron, and TaskAtlas was claude-opus-4.7. (2)Domain-specific protocols. Interviews in both domains began with occupational responsibilities and recent work experiences, then progressively explored current difficulties, desired analytical capabilities, and concrete natural-language instructions. The TA protocol focused on workflows and difficulties involving tabular text, desired text-analysis capabilities, connections across information sources, and task prioritization. The Causal protocol focused on decision responsibilities and spreadsheet usage, the effects of actions or changes, causal drivers, the fairness of comparisons and potential common causes, and needs involving causal structure and effect estimation. Each need record included the analytical request, referenced data, expected output, usage frequency, stakes, and intended consumer. (3) Analyst settings. Direct-CoT generated needs directly, with analyst count marked as N/A. Single Analyst conducted interviews using a single generic analyst role. Elicitron generated multiple user profiles and performed experience simulation and interviews. The exact number of users in its original runs remains unconfirmed. TaskAtlas loaded 160 pregenerated personas, all of which proceeded to the interview stage. (4) Generation budgets. TaskAtlas interviews consisted of 10 main question–answer exchanges, followed by one closing question and response. The interview client used a temperature of 0.7 and a maximum output length of 32,768 tokens per call.

[Controlled comparison.] To directly assess the contribution of ONET grounding, we compared conditions with and without ONET records in both TA and Causal. All conditions used claude-opus-4.7, a temperature of 0.7, a maximum output length of 32,768 tokens per call, and 160 analysts per condition. Within each domain, we held the occupational roster, persona-generation template and procedure, interview protocol and budget, and downstream need-extraction and evaluation procedures fixed. We varied only whether O*NET occupational records were supplied during persona generation.

[Results.] The table below reports the comparison in both domains. Task Coverage is reported as the number of covered reference task types divided by the total number of reference task types.

O*NET groundingTA Coverage@1 ↑TA Diversity ↑TA Task Coverage ↑Causal Coverage@1 ↑Causal Diversity ↑Causal Task Coverage ↑
✗70.4%0.7211/1258.7%0.7012/13
✓80.8%0.7512/1267.4%0.7413/13

[Analysis.] Adding ONET records increased Coverage@1 by 10.4 and 8.7 percentage points in TA and Causal, respectively, and increased Diversity by 0.03 and 0.04. Both domains covered one additional reference task type, achieving full reference-task coverage. These results support the incremental contribution of ONET occupational records to external corroboration, semantic diversity, and reference-task coverage when occupational composition, analyst count, persona-generation procedure, and interview budget are held fixed.

AuthorsOpenReview ↗

Response to Q3

[Provenance and task abstraction.] The table below presents the provenance and task abstraction of the two examples, based on the saved interviews, need-extraction records, and task definitions in the paper.

StageTA++LCD
Source interviewEnvironmental restoration planner, P_19_2041_02Facilities manager, P_11_3013_00
Paraphrased original needCan narratives from initial site assessments predict subsequent restoration outcomes? Which narrative characteristics are associated with those outcomes?Work-order volume and satisfaction vary together. Could a third factor, such as occupant expectations, influence both? What additional data would be needed to answer this question?
Recorded needThe need appears in the interview and in novel_observationsExtracted as P_11_3013_00-Q4, with the original mapping associating it with A1, D1, and D2
Attribute summaryThe inputs are assessment narratives and a downstream prediction objective. The attributes relevant to that objective remain to be discovered. The outputs are an attribute schema, attribute values, and an augmented tableThe inputs are observations of two variables and data-generating assumptions. The target is a potential shared latent cause. The outputs are a confounding judgment, a confidence assessment, and a diagnostic explanation
Grouping rationaleAbstract needs with a specified objective, attributes to be discovered, and a requirement to construct a structured representation for prediction into a common taskAbstract needs to determine whether a specified variable pair shares a latent cause, and whether the available information supports that determination, into a common task
Final task definitionGiven a text-containing table XX, an objective gg, and optional context CC, produce an attribute schema SS, attribute values AA, and the augmented table [X;A][X;A]Given observational data DXY\mathcal D_{XY} and assumptions A\mathcal A, return yes, no, or not-identifiable, together with a confidence assessment and a diagnostic explanation

[Distinction between TA++ and T4/T12.] The submitted paper explicitly states that TA++ combines the open-schema extraction capability of T4 with the column-generation capability of T12. Its additional requirement is that the downstream objective gg guides attribute-schema discovery and selection, and that the resulting representation is evaluated against that objective. In the environmental restoration example, the analyst specifies the objective of predicting restoration outcomes, while the relevant textual attributes remain to be discovered. The system must determine which narrative characteristics should become structured columns, assign values grounded in the source material, and assess whether these attributes are useful for the prediction objective.

TA++ therefore connects attribute discovery, attribute-value extraction, and downstream utility within a single task specification. Its evaluation should assess both the faithfulness of attribute values to the source text and the predictive utility of the resulting representation on held-out data. We accordingly position TA++ as a compositional task extension guided by the downstream objective, distinguished by the objective’s constraints on schema selection and the explicit requirement to evaluate downstream utility.

[Distinction between LCD and Kaltenpoth and Vreeken.] LCD incorporates whether a confounding judgment for a specified variable pair is supported by the available data and assumptions into its task objective. NOCADILAC, proposed by Kaltenpoth and Vreeken, recovers a causal network containing latent confounders from multivariate observational data under a specified nonlinear causal model and structural conditions. Kaltenpoth and Vreeken, ICML 2023

LCD permits inputs containing observations of only two target variables and explicitly includes the available data-generating assumptions. The system must determine whether this information is sufficient to establish the presence of a shared latent cause and return yes, no, or not-identifiable, accompanied by a diagnostic explanation. This requirement introduces additional evaluation criteria. LCD should be evaluated on the accuracy of confounding judgments in identifiable cases, its ability to correctly return not-identifiable in non-identifiable cases, and the consistency of its diagnostic explanations with the specified assumptions and evidence.

Reviewer_R5jxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework for discovering data-analysis task spaces by interviewing simulated analysts. The authors evaluate the framework in tabular text analysis and causal inference, examining external corroboration, semantic diversity, and coverage of reference tasks.

Critical strengths
  1. The framework connects occupational profiles and interviews to analytical needs and task types, allowing each task to be traced back to its supporting evidence.
  2. The proposed method achieves the highest external corroboration and semantic diversity among the evaluated methods in both domains.
  3. The paper distinguishes analytical needs, task types, and task instances, suggesting new research directions for the data-analysis community.
Critical weaknesses
  1. Task Coverage measures whether generated needs support established reference categories. However, it does not directly assess the coherence, redundancy, granularity, or stability of newly induced categories.
  2. The comparisons do not establish how much improvement comes from O*NET records, persona diversity, generated contextual detail, or adaptive interviewing. In particular, profiles already contain LLM-generated initial workflow difficulties, which may influence the subsequently elicited needs. Controlled ablations and comparisons under matched computational budgets are needed.
  3. The paper does not report token costs, runtime, or the amount of human curation required, making it difficult to assess the framework’s practical scalability.
  4. The model used for the tabular-text run and the causal-specific recruitment setup are not clearly documented.
Questions for authors
  1. How closely do human judgments agree with the LLM-based corroboration labels?
  2. Under matched model and token budgets, how much do O*NET grounding and adaptive interviews contribute individually?
  3. Could the authors provide the complete induced taxonomies and evaluate the coherence and stability of the new categories?
  4. Could the authors provide the exact domain-specific protocols and model configurations, and report whether the findings remain consistent across repeated runs?
3 comments
AuthorsOpenReview ↗

Response to Q1&Q2

Thank you for your careful review of our paper and for recognizing the value of TaskAtlas. Your suggestions have helped us improve our evaluation.

Q1: How closely do human judgments agree with the LLM-based corroboration labels?

[Setup.] In each of the two domains, we randomly sampled 20% of the TaskAtlas requirements for which automated evaluation had been completed and conducted a blinded human review of these requirements and all their retrieved snippets. One annotator assessed them following the criteria in Appendix E.2. Method names, LLM labels, and their explanations were hidden during annotation, and the annotator was provided with the same information as the original judge.

[Results.] Following the original definition of Coverage@1, we counted a requirement as externally corroborated if at least one retrieved snippet was labeled support. Both human and LLM results were computed on the same sampled set. Requirements for which retrieval had completed but returned no snippets were retained in the denominator.

DomainRequirement-level agreementHuman Coverage@1LLM Coverage@1
Tabular text analysis95.3%80.3%81.9%
Causal inference94.1%66.7%68.4%

[Analysis.] Across the two domains, human and LLM judgments of whether a requirement was externally corroborated agreed in 95.3% and 94.1% of cases, respectively. Recomputing Coverage@1 using human labels resulted in decreases of 1.6 and 1.7 percentage points, respectively. These results indicate high requirement-level agreement between the automated evaluation and this human annotator on the sampled TaskAtlas requirements.

Q2: Under matched model and token budgets, how much do O*NET grounding and adaptive interviews contribute individually?

[Setup.] We conducted a 2×2 ablation study, independently varying whether persona construction used ONET records and whether interviews used adaptive follow-up questions. Conditions with ONET grounding generated persona backgrounds from occupation titles and the corresponding occupational records. The remaining conditions generated backgrounds from occupation titles alone. Fixed interviews followed a predefined question list, while adaptive interviews allowed follow-up questions based on the interviewee’s responses. All conditions used claude-opus-4.7.

[Results.] The table below reports results for both domains. TA denotes tabular text analysis, and Causal denotes causal inference. Task Coverage is reported as the number of covered reference task types divided by the total number of reference task types.

O*NET groundingAdaptive interviewsTA Coverage@1 ↑TA Diversity ↑TA Task Coverage ↑Causal Coverage@1 ↑Causal Diversity ↑Causal Task Coverage ↑
✗✗60.2%0.6810/1248.5%0.649/13
✓✗68.1%0.7111/1255.2%0.6711/13
✗✓70.4%0.7211/1258.7%0.7012/13
✓✓80.8%0.7512/1267.4%0.7413/13

[Analysis.] (1) Contribution of O*NET grounding. Under fixed interviews, adding ONET records increased Coverage@1 by 7.9 and 6.7 percentage points in TA and Causal, respectively. Under adaptive interviews, the corresponding gains were 10.4 and 8.7 percentage points. Semantic diversity and coverage of reference task types also increased in both domains. These results suggest that, with occupational composition held fixed and computational budgets matched, using ONET records to construct personas improves the external corroboration of elicited needs and broadens the analytical objectives covered.(2) Contribution of adaptive interviews. When persona backgrounds were generated from occupation titles alone, adaptive follow-up questions increased Coverage@1 by 10.2 percentage points in both domains. With O*NET grounding, the corresponding gains were 12.7 and 12.2 percentage points. These results suggest that, under the same budget, following up on interviewees’ responses further improves the external corroboration of elicited needs.

AuthorsOpenReview ↗

Response to Q3

Q3: Could the authors provide the complete induced taxonomies and evaluate the coherence and stability of the new categories?

[Setup.] The methods section of the original manuscript describes how the reference taxonomies are extended through attribute extraction, deterministic grouping, and category naming. Appendix F provides the reference taxonomies and category boundaries for both domains. The Emergent Tasks section presents the task definitions, inputs and outputs, and source scenarios for two candidate extensions, TA++ and LCD. To further evaluate the quality of the new categories, we conducted additional coherence and stability experiments and report the results in this response. For coherence, one annotator reviewed each requirement using the category definition, requirement text, and source interview excerpts to determine whether it aligned with the analytical goal and expected output of its assigned category. We first calculated the proportion of requirements matching the definition within each category and then averaged these proportions across categories. The evaluation used the automatically induced results before manual revision. For stability, we repeated the experiment three times to assess the consistency of requirement groupings across runs. This design draws on the practice of pairwise comparison of clustering results in clustering stability research. Specifically, we used Pairwise F1 to measure grouping agreement between two runs. Pairwise F1 has been used in clustering evaluation to compare pairs of instances assigned to the same cluster [1]. Values closer to 1 indicate greater consistency in grouping relationships between runs.

[1] Viswanathan, V., Gashteovski, K., Gashteovski, K., Lawrence, C., Wu, T., & Neubig, G. (2024). Large language models enable few-shot clustering. Transactions of the Association for Computational Linguistics, 12, 321–333.

[Results.] The table below reports the evaluation results for both domains. Coherence is the macro-average of the within-category proportions of requirements matching their category definitions. Stability is the mean Pairwise F1 across pairs of runs.

DomainCoherence ↑Stability (Pairwise F1) ↑
TA92.0%0.89
Causal88.5%0.85

[Analysis.] Coherence scores of 92.0% for TA and 88.5% for Causal indicate that, on average across categories, the human-reviewed requirements closely align with the analytical goals and expected outputs of their assigned categories. This supports the internal coherence of the new categories. The mean Pairwise F1 scores across three independent runs were 0.89 and 0.85, respectively, indicating that the induction process consistently preserves co-membership relationships among requirements under a fixed requirement set and model configuration. These results provide direct evidence of induction stability on the given requirement sets.

AuthorsOpenReview ↗

Response to Q4

Q4: Could the authors provide the exact domain-specific protocols and model configurations, and report whether the findings remain consistent across repeated runs?

[Setup.] Appendices A–E of the original manuscript describe occupational recruitment, persona initialization, the shared interview template, requirement extraction, and external corroboration evaluation. We further clarify the domain-specific protocols, model configuration, and repeated-run settings below.(1) Domain-specific protocols. In both domains, interviews begin with occupational responsibilities and recent work experiences, then progressively explore current challenges, desired analytical capabilities, and concrete natural-language instructions. Each requirement records the analytical request, relevant data, expected output, usage frequency, decision impact, and intended users of the results. The main topics covered in each domain are summarized below.

DomainTopics covered by the interview protocol
TAOccupational background and recent work products, workflows and challenges in processing tabular text, desired text analysis capabilities, linking information across sources, concrete analytical instructions, and task priorities
CausalDecision-making responsibilities and tabular data use cases, effects of actions or changes, how variables influence one another, fairness of comparisons and potential common causes, needs for causal structure discovery and effect estimation, concrete analytical instructions, and task priorities

(2) Model configuration. We used claude-opus-4.7 with a temperature of 0.7 and a maximum output length of 32,768 tokens per call. The run loaded 160 pregenerated personas, all of which participated in interviews.(3) Repeated runs. We fixed the occupation list, domain-specific protocols, model configuration, and evaluation rules, and independently ran the complete TaskAtlas generation pipeline three times in each domain. Each run regenerated the personas, interviews, requirements, and taxonomy results, and independently computed Coverage@1, Diversity, and Task Coverage. This experiment measures how variability in the generation process affects the final metrics under a fixed occupational composition.

[Results.] The table below reports the results of the three runs. Task Coverage is presented as the number of covered categories divided by the total number of reference categories.

RunTA Coverage@1 ↑TA Diversity ↑TA Task Coverage ↑Causal Coverage@1 ↑Causal Diversity ↑Causal Task Coverage ↑
Run 181.2%0.7512/1267.6%0.7413/13
Run 280.4%0.7412/1268.4%0.7413/13
Run 382.0%0.7512/1267.2%0.7313/13

[Analysis.] Across the three runs, Coverage@1 ranged from 80.4% to 82.0% for TA and from 67.2% to 68.4% for Causal, with maximum differences of 1.6 and 1.2 percentage points, respectively. The maximum difference in Diversity was 0.01 in both domains, and every run covered all reference task categories. These results indicate that TaskAtlas achieves good cross-run consistency in external corroboration rate, semantic diversity, and reference task coverage under the fixed configuration described above.

We thank the reviewer again for the careful review of our paper and the specific suggestions regarding evaluation reliability, component contributions, taxonomy quality, and reproducibility. We hope that the clarifications and additional experiments above address your concerns and help you further assess the contributions of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.