acceptodds
Under review as a conference paper at ICLR 2027

FROM DATA LAKES TO TRAINING TASKS: GROUNDED AND CONTROLLABLE SYNTHESIS FOR DATA AGENTS

Abstract

Training data agents at scale rests on a steady supply of grounded, verifiable data-analysis tasks. Real-world data lakes make these hard to produce: they hold far more files than can be inspected individually, and the few files that share a meaningful join path cannot be identified from local context alone. We propose DATA2TASK, which represents a data pool as a compact physical-semantic dual-view graph: its physical view replaces exhaustive inspection with selective access, while its semantic view aligns columns denoting the same concept across files and so surfaces latent relations. DATA2TASK then samples a provenance-preserving subgraph and instantiates a compatible analytical operator over it, forming a task motif whose answer is computable from the sampled files and therefore grounded by construction; a staged pipeline turns each motif into a natural-language request with executable ground truth, kept only when independent solvers reproduce that answer, and scored separately for structural and semantic difficulty. With only 2,000 verified tasks and selected solution trajectories, fine-tuning Qwen3-8B achieves an average score of 55.17% across four held-out data-analysis benchmarks, outperforming the 29x larger Qwen3-235B and matching the 60x larger Qwen3-Coder-480B on TableBench.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data agents, data-analysis task synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Mt9vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DATA2TASK, a framework for synthesizing executable data-analysis tasks from heterogeneous real-world data lakes. It constructs a physical–semantic dual-view graph to organize files and fields and align concepts across tables, then samples subgraphs and compatible analytical operators to generate tasks. Execution-based answer agreement, query rewriting and re-verification, and difficulty filtering determine which tasks are retained. Using 2,000 selected teacher-generated trajectories for supervised fine-tuning, the authors improve Qwen3-8B’s average pass@1 from 45.59 to 55.17 across four evaluation benchmarks. Additional experiments examine graph removal, task difficulty, and trajectory selection.

Critical strengths
  1. A coherent approach to an important training-data bottleneck. Separating relation discovery from task generation makes cross-file relationships explicit and reusable, while provenance tracking and replayable parsing make the underlying transformations inspectable. This is a useful design for generating analytical tasks over heterogeneous files whose relationships are not supplied as a database schema.
  2. Consistent downstream improvements with a modest final training set. Table 1 reports improvements on all four benchmarks using only 2,000 selected trajectories, including gains of 16.20 percentage points on DAEval and 12.13 points on the evaluated TableBench subsets. These results support the practical utility of the synthesized supervision. The graph-removal and trajectory-selection ablations also provide useful evidence beyond comparison with the untuned backbone.
Critical weaknesses
  1. Semantic validity remains insufficiently established, despite being central to the motivation. Section 3.4 verifies agreement between executed answers, but agreeing solvers can share an interpretation of an ill-posed or analytically unjustified question. For example, supplementary task datamind_hetero_v2_epidemiology_surveillance_population_health_1327_1 asks about the ratio of COVID-19 recoveries to daily SARS cases during the 2003 SARS outbreak, returns 2003-04-12, and is labeled high confidence. Its temporal interpretation is unclear. Another high-confidence task, datamind_hetero_v2_statistics_and_general_reference_3089_1, subtracts a historical Inquisition count from an outbreak case-count statistic without an evident analytical rationale. These examples do not establish a dataset-wide error rate, but they expose a gap between computational agreement and meaningful analysis. Without an independent assessment of semantic validity and answer correctness, the claimed quality of the core output—the training tasks—remains uncertain.
  2. The incremental value of the central graph design is incompletely validated. With the same 2,000-trajectory training budget, Table 4 reports DAEval scores of 81.77 without the graph and 84.16 with it, compared with 67.96 for the backbone. The 2.39-point increment may be useful, but downstream graph ablation is reported on only one benchmark, without uncertainty estimates, and removes the representation and graph-guided motif construction together. Consequently, it does not establish which aspects of the dual-view design contribute or whether their benefit generalizes across the evaluation suite. Equal final training-set size also does not ensure equal synthesis cost: graph construction, candidate rejection, and repeated verification costs are not sufficiently quantified. This limits the justification for the added machinery relative to the simpler pipeline.
Questions for authors
  1. How is the SARS/COVID-19 task identified above intended to be interpreted, particularly its temporal alignment? What checks distinguish meaningful cross-source analysis from an executable but semantically unjustified combination?

  2. Has a representative sample of retained tasks been independently audited for semantic validity and numerical correctness separately? If so, what were the sampling protocol and error rates, particularly for high-confidence cross-table tasks?

  3. Were the Table 4 variants matched for synthesis resources as well as final trajectory count? What were their generation costs and retention rates, and are graph-removal results on the other three benchmarks and variation across runs available?

6 comments
AuthorsOpenReview ↗

Response to Reviewer sdJe — Q1

We thank the reviewer for the careful reading and for pinpointing the two concrete task examples, which expose a real gap between computational agreement and meaningful analysis. We address the concerns in turn below.

Q1 — Semantic validity and the two flagged examples

Acknowledgment and classification. We thank the reviewer for these concrete examples. They are correct, and both point to the same underlying gap: computational agreement does not by itself certify meaningful analysis. The two tasks fail in different ways, so we separate them before addressing each.

  • Ill-posed question (the SARS/COVID-19 task). The question conflates two temporally disjoint events — the 2003 SARS outbreak and later COVID-19 recoveries — so its temporal alignment is indeed ill-posed. The solvers reproduce an answer, but that answer is an artifact of an incoherent question: agreement certifies reproducibility, not credibility.
  • Unjustified operation (the Inquisition task). Two unrelated counts are joined and subtracted, so the task carries a computational relation but no analytical rationale.

We do not claim perfect accuracy or semantic validity. In particular, perfect semantic validity cannot be guaranteed: the final query is phrased by a stochastic LLM, so a residue of well-formed but analytically unjustified questions is inherent to the approach. Our claim is narrower and is what the design is built to secure — that answer errors are rare and that semantic failures are bounded and now measured (Q2) — through the two layers below.

How we bound the two failure modes.

  • Answer credibility (against the ground-truth mode). The answer is never asserted by a model. Two agents driven by different models each run two rollouts that write and execute code; high confidence requires cross-model agreement, and the agreed code is retained as the executable reference program, so the answer is fixed by deterministic sandbox execution. Column hints are injected at verification time so the check targets the answer rather than the search for data.
  • Semantic connection (against the semantic mode). We constrain relations at the graph level rather than relying on the solver. Every field is assigned an analytical role (KEY, DIMENSION, MEASURE, TIME, GEO, META) plus a concept hint; columns that jointly express one variable are merged, and virtual fields recover structure hidden in filenames (e.g., the year in weather_2023.csv), recorded as the cluster’s partition axis. Concept alignment is screened by name, value, and type compatibility before an LLM partitions fields into concepts, and links are admitted only between same-concept, role-compatible fields (typed equi/fuzzy/spatial/temporal). Operator contracts then bind a measure only to compatible dimensions, so an operation is instantiated only when roles are semantically consistent. These type-level constraints are necessary but not sufficient, as the two examples show: the Inquisition task passed because two counts with unknown units satisfy the difference contract; the SARS/COVID-19 task passed because the two series share a date concept, with no check that their periods overlap. No step in the current pipeline asks whether a combination is analytically meaningful.

Measured impact and residual value. To quantify rather than assert, we audited the full 2,000-task set with human judges on two separate axes: 99.55% of answers are correct and 81% of tasks are semantically/logically appropriate (protocol and full breakdown in Q2). The residual semantically-weak tasks are not worthless: when a trajectory’s cross-table relations and computation are independently confirmed, it still supervises file location, field matching, key–value alignment, and cross-table joining — the procedural skills a data-analysis agent must acquire — even if its final analytical framing is imperfect.

What we will do. We will report this audit and state semantic validity as a limitation in the revision, and leave pipeline-level checks for semantic validity (e.g., a temporal-overlap check and a post-rewrite semantic judge) to future work.

AuthorsOpenReview ↗

Response to Reviewer Mt9v — Q2

Q2 — Audit protocol and dataset-wide error rates

Full audit across two independent dimensions. We audited all 2,000 retained tasks — a full census rather than a sample — and human judges scored each task on two independent axes: (i) numerical correctness of the answer and (ii) semantic/logical validity of the question.

How each axis was scored.

  • Numerical correctness. For each task, an agent re-derived the answer from the final question and the data lake, without column hints or access to the reference. Three human experts then independently audited the agent's derivations against the data and confirmed that the agent solved the questions correctly. Across the 2,000 tasks, 1,991 (99.55%) of the stored answers are correct and 9 (0.45%) are incorrect.
  • Semantic/logical validity. Human judges evaluated every task on whether its cross-source combination carries a coherent analytical rationale, screening against a fixed taxonomy of failure modes rather than a free-form judgment. The taxonomy we screened for is:
    1. Unrelated/arbitrary cross-source operations (~50%, the dominant class) — combining quantities that share a computational interface but no analytical relationship (the Inquisition example).
    2. Unit/scale incommensurability (~13%) — dividing or subtracting quantities with incompatible units into a meaningless ratio/difference.
    3. Identifiers/nominal codes used as measurements (~10%) — statistics or trends computed over IDs or arbitrary orderings.
    4. Geographically/entity-incoherent joins (~7%) — linking records across unrelated geographies or coincidental surrogate IDs.
    5. Temporal misalignment (~7%) — pairing non-overlapping periods or aligning series by index rather than date (the SARS/COVID example).
    6. Circular/tautological operations (~6%) — predictors or terms that are constituents of, or definitionally identical to, the target.
    7. Statistically degenerate operations (~6%) — tests/trends that are trivially determined (tiny samples, monotonic cumulative sums, constant predictors).

Results.

By verification confidence.

ConfidenceTasksAnswer correctSemantic appropriate
High1,96399.54%81.20%
Medium37100.00%70.27%
Overall2,00099.55%81.00%

By number of referenced tables.

TablesTasksAnswer correctSemantic appropriate
148299.38%85.27%
21,13199.47%78.78%
3+387100.00%82.17%
≥2 (all cross-table)1,51899.60%79.64%

Two main findings.

  1. Cross-table complexity and verification confidence barely affect answer correctness. Accuracy is essentially flat across both the number of referenced tables (99.38%–100%) and the confidence level (99.54% high vs. 100% medium). Neither cross-table joins nor verification confidence is a driver of numerical error.
  2. Semantic validity is lower for cross-table tasks than for single-table tasks. Single-table tasks are the most appropriate (85.27%) because they involve the fewest semantic links. The two-table group — the largest group (1,131 tasks) — is the least appropriate (78.78%), so the residual semantic defects concentrate in multi-table tasks rather than being spread uniformly.
AuthorsOpenReview ↗

Response to Reviewer Mt9v — Q3

Q3 — Resource matching, generation cost, and retention of the graph ablation

What the two variants hold fixed. The two variants are identical in every respect except the graph. They share the same source domains, the same trajectory teacher and verification procedure, the same trajectory filtering and difficulty scorer, the same rollout generation, trajectory selection, and SFT recipe, and the same final 2,000-trajectory budget; model choices and retry limits are unchanged throughout. The only difference is the dual-view graph: the graph-free variant drops the graph and its guidance on task construction, so the generator receives the raw domain files directly. This ablation removes the graph as a whole. The composition results point to the semantic view as the source of the gain: tasks over three or more tables require cross-table links, and none appeared without the graph.

Synthesis cost and retention. To match the two variants for synthesis resources — not only for final trajectory count — we draw a matched random sample of 200 candidates from each and report cost and retention separately.

Retention. On the matched 200-candidate sample, the graph variant retains 157/200 (78.5%) and the graph-free variant 185/200 (92.5%). The gap is driven by the design stage: with the graph, the design step must instantiate and validate the cross-table joins specified by the motif, so candidates whose multi-table logic is inconsistent are rejected there (30 design failures vs. 8); without the graph, the generator is not committed to any cross-table structure and produces mostly single- and two-table tasks that rarely fail this check. The graph’s lower retention is therefore the cost of attempting cross-table tasks the graph-free variant does not even generate.

Synthesis cost. We acknowledge that the graph adds cost: the one-time graph construction and the graph-structured input to the generator increase calls and tokens. On the matched 200-candidate sample (task generation only; the one-time graph build is excluded and amortized below):

Metric (per 200 candidates)With graphWithout graph
Provider requests7,5046,376
Tokens (recorded)153.9M91.9M

Per candidate the graph consumes ~1.7× the tokens and ~1.2× the requests. The two variants were thus matched on final trajectory count, and the graph variant used more synthesis budget. Additional budget would not close the gap for the graph-free generator: among its 2,000 tasks, none spans three or more tables and only 0.3% are hard. This overhead is bounded and controllable: the graph build is a one-time cost (1,330 requests, ~8.4M tokens) that amortizes to ~4.2K tokens per task over the 2,000-task corpus, and per-candidate calls are capped by the fixed candidate budget and retry limits, so the graph scales only a constant factor rather than the total synthesis budget.

Downstream value across all four benchmarks. The added cost is justified downstream. Extending Table 4 to all four benchmarks with seed variance:

pass@1DAEvalDABStepQRData-VTableBenchAvg.
Qwen3-8B backbone (Table 1)67.9612.0146.4855.8945.59
With graph (Table 1)84.16 ± 0.6918.57 ± 0.6649.94 ± 0.8268.02 ± 0.3355.17
Without graph81.77 ± 0.5215.78 ± 1.0943.29 ± 1.3667.68 ± 0.4952.13

With the graph, cross-table tasks (≥ 2 tables) rise from 64.9% to 75.9% (+11.0 points), driven almost entirely by tasks spanning three or more tables, which go from 0.0% to 19.4% — none arose among the 2,000 graph-free tasks. The difficulty distribution shifts in parallel: hard tasks rise from 0.3% to 17.7% (+17.4 points), while easy tasks fall from 37.7% to 31.8% and medium tasks from 62.0% to 50.6%. These harder, multi-table tasks are the supervision behind the DABStep (+2.79) and QRData-V (+6.65) gains, whereas TableBench — dominated by single-table reasoning — gains only +0.34 because its task mix is barely affected by the graph.

Reviewer_sdJeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DATA2TASK, which uses a physical–semantic dual-view graph to identify relations between real-world data files. It uses these relations to generate data-analysis tasks and verifies their answers through multiple solver runs. Fine-tuning on the corresponding solution trajectories improves model performance across all evaluated benchmarks.

Critical strengths
  • The task-generation approach has practical value. Discovering cross-file relations before combining sources with compatible analytical operators enables more controllable task synthesis while preserving source provenance.
  • The downstream results are strong. Fine-tuning on 2,000 trajectories improves performance across all four benchmarks, raising average pass@1 from 45.59% to 55.17%. The graph ablation also supports the benefits of graph-guided generation for task complexity and downstream performance.
Critical weaknesses
  • The treatment of medium-confidence tasks is unclear. The paper does not specify whether these tasks enter the final training corpus, how many are retained, or whether their ground-truth answers have been independently checked. This makes it difficult to assess how the acceptance criterion affects training-data reliability.
Questions for authors
  • In the solve-and-verify stage, verification solvers receive the columns the task was built on. Could the authors clarify how the final rewritten query still uniquely determines the answer when these column hints are unavailable? This would help me better understand the relationship between verification and actual task solving, including any assumptions needed for source identification.
  • Does the final training corpus include medium-confidence tasks, where only the two rollouts of one model agree without cross-model agreement? If so, could the authors report their number and any spot-check results on the correctness of their ground-truth answers? This would help me better understand the quality-control process for the final training data.
1 comment
AuthorsOpenReview ↗

Response to Reviewer sdJe

We thank the reviewer for the clear summary and the constructive questions. We address each question below; key results are highlighted in bold.

Q1

We agree that the core concern is whether the final question still uniquely determines the answer once the column hints are removed. We address this along two lines: the mechanism by which the question remains self-contained, and the empirical evidence that solvability and correctness are independent of the hints.

1. The hints are construction-time scaffolding, and rewriting preserves the question's meaning. Column hints are used only during answer verification and never appear in the final training questions; they point the verification solvers to the exact columns a task was built on, so those solvers can confirm the intended data source and validate the reference answer. Verification thus reuses information already available at construction time, and the hints are internal scaffolding rather than information the final question requires the model to supply. Correspondingly, the rewriting step strips surface artifacts — file/table names and procedural scaffolding — while retaining everything that makes the question self-contained: the semantic description of the data source, the analytical objective, and the constraints needed to identify the result. Source identification in the final question is therefore semantic rather than name-based, so the rewritten question still specifies which data to use and what to compute, and continues to determine the answer on its own, without the hints.

2. The hints do not measurably affect solvability or correctness. We ran a paired with/without experiment on 100 final (rewritten) tasks: for each task the motif and every construction precondition — including the data source — were held fixed, and only the presence of the column hints varied, with each version solved under the identical verification protocol.

ConditionAnswered (consensus reached, answer retained)Rate
With hints91/10091%
Without hints92/10092%

Since the two conditions differ only in the hints, the essentially identical answer rates (91% vs. 92%) indicate that removing the hints does not measurably reduce solvability. We additionally verified that correctness does not depend on the column hints. An agent re-derived all 2,000 answers in a hint-free setting (no column hints). Three human experts then independently audited the agent's derivations against the data and confirmed that the agent solved the questions correctly: 1,991 of the 2,000 answers (99.55%) are correct, demonstrating that the final questions can be solved correctly without the hints.

Taken together, these results show that the ground-truth answer is determined by the question itself — the self-contained semantic description retained after rewriting — and is not an artifact of hint availability.

Q2

We answer this in three parts: (i) the confidence composition of the corpus, (ii) a confidence-stratified correctness audit, and (iii) our interpretation.

(1) Confidence composition. Yes — the final training corpus includes medium-confidence tasks, i.e., those where the two rollouts of a single model agree but no cross-model agreement is reached. Of the 2,000 retained tasks, 1,963 are high-confidence and 37 are medium-confidence.

(2) Correctness audit by confidence label. Our human audit of answer correctness, broken down by confidence, is:

ConfidenceTasksIncorrect answersCorrectness
High1,963999.54%
Medium370100.00%
Overall2,000999.55%

(3) Interpretation. All nine detected errors lie in the high-confidence group; none of the 37 medium-confidence tasks has an incorrect answer. Medium-confidence tasks account for only 1.85% (37/2,000) of the corpus. Given the small size of this group, we do not claim that medium-confidence tasks are equally reliable. However, our audit found no incorrect answers among these 37 tasks, providing empirical support for their inclusion in the final training corpus.

Reviewer_izHPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper Data2Task that generates training tasks for LLM data-analysis agents from raw, real-world data lakes. It first builds a graph over the files. One view records files, tables, and columns. The other uses an LLM to link columns in different files that mean the same thing, which reveals possible joins. It then samples small subgraphs, pairs them with analysis operators (e.g., aggregation, ranking, correlation), and has LLMs turn each one into a natural-language question. A task is kept only when solvers from two different models agree on the answer, and tasks are filtered by difficulty. Fine-tuning Qwen3-8B on 2,000 such tasks raises its average score on four benchmarks from 45.6% to 55.2%, above DataMind-7B/14B and Qwen3-235B.

Critical strengths

S1. A practical way to find cross-file tasks in raw data lakes. Existing synthesis methods either see only local file context or rely on databases with known schemas. The dual-view graph makes cross-file relations explicit before any task is written. This works: 75.9% of tasks use more than one table, while DataMind produces no cross-table tasks from standalone CSV or XLSX files. Removing the graph eliminates tasks over three or more tables (19.4% to 0%) and nearly all hard tasks (17.7% to 0.3%).

S2. Strong results with a small training set. With only 2,000 trajectories and plain SFT, the 8B model gains about 10 points on average over its backbone and improves on all four benchmarks. For comparison, DataMind uses 11.7K trajectories plus RL. If the gains hold under controlled comparisons (W1), this is a data-efficient recipe that others could adopt.

Critical weaknesses

W1. Unclear gains. The training trajectories are written by GLM-5.2, a much stronger model (71.4% average in Table 1), so much of the gain may come from distilling a strong teacher rather than from the task design. The only test of the graph is the ablation in Table 4, which reports just DAEval: 84.2% with the graph vs. 81.8% without. The graph-free variant already gains 13.8 points over the backbone, so most of the improvement remains without the graph. There is also no comparison with tasks from other synthesis pipelines (e.g., DataMind or DSGym style) using the same data pool, teacher, budget, and student. The comparison with DataMind is also weak. DataMind runs in its own harness, and its TableBench scores (14.7% and 17.3%) are far below even the untuned Qwen3-8B (55.9%), which suggests a harness or format mismatch. This one benchmark drives most of the average gap. Without TableBench, DataMind-14B averages 52.3% vs. 50.9% for Data2Task-8B, and DataMind-14B is clearly better on QRData (56.4% vs. 49.9%). No variance across seeds is reported for any result.

W2. Task correctness and relation quality are not directly checked. A task is kept when two LLM solvers agree, and the solvers are told which columns to use. Agreement does not guarantee correctness, since both models can share the same misreading. The LLM-made column links are also never evaluated: the paper does not report how often linked columns truly mean the same thing, and it notes that admitted relations may not suit every analysis. A small human audit of tasks (correct answer, natural question, valid join) and of the links would make the "grounded and verified" claims much stronger.

W3. Missing related work. Finding joinable columns and building semantic graphs over data lakes is a well-studied problem in data management (joinable/unionable table search, data discovery systems). The paper does not discuss this line of works or explain what is new relative to it.

W4: Missing cost in evalaution. The paper also frames the problem with a cost budget but never reports cost: the number of LLM calls, graph-building time, or how many sampled motifs survive to become the 2,000 tasks. This matters because the authors themselves name graph scalability as the main limitation.

Questions for authors

Q1 (W1). Can you report the graph ablation on all four benchmarks, with variance across seeds?

Q2 (W1). How do Data2Task tasks compare with tasks from another synthesis method (e.g., DataMind- or DSGym-style generation) on the same data pool, with the same teacher, 2,000-trajectory budget, and student?

Q3 (W1). Why is DataMind so far below the untuned backbone on TableBench? Can you evaluate it in the same harness as the other models, or report averages without TableBench?

Q4 (W2). What fraction of a random sample of tasks is correct and natural according to human checks? How accurate are the LLM-made column links?

Q5 (W4). What does it cost to build the graph and generate the 2,000 tasks (LLM calls, time), and what fraction of motifs survive verification and filtering?

3 comments
AuthorsOpenReview ↗

Response to Reviewer izHP — Q1 & Q2

We thank the reviewer for the careful reading and the constructive questions. We address each point below; key results are highlighted in bold.

Q1 (W1)

We address gain attribution, four-benchmark graph ablation with seed variance, and task composition.

(1) Attribution — the two mechanisms are complementary, not competing. We agree the full gain over the Qwen3-8B backbone is not attributable to the graph alone. Distillation from the strong teacher (GLM-5.2) and graph-guided cross-table task construction are two distinct, complementary sources of the improvement: both contribute, and neither excludes the other. The two can be separated, because the ablation in (2) keeps the teacher, the 2,000-trajectory budget, the student, and the SFT recipe fixed and changes only how tasks are constructed.

(2) Graph ablation on all four benchmarks (with seed variance).

pass@1DAEvalDABStepQRData-VTableBenchAvg.
Qwen3-8B backbone (Table 1)67.9612.0146.4855.8945.59
With graph (Table 1)84.16 ± 0.6918.57 ± 0.6649.94 ± 0.8268.02 ± 0.3355.17
Without graph81.77 ± 0.5215.78 ± 1.0943.29 ± 1.3667.68 ± 0.4952.13
Δ (with − without)+2.39+2.79+6.65+0.34+3.04

± is the standard deviation over the three evaluation seeds. On the four-benchmark average, the backbone scores 45.59, the graph-free pipeline 52.13 (+6.54), and the full pipeline 55.17 (a further +3.04). About one third of the total gain (3.04 of 9.58 points) is therefore due to the graph and cannot come from the teacher, which is identical in both arms; the other two thirds come from the shared pipeline and teacher distillation, and we do not claim them for the graph.

The gain exceeds the seed standard deviation on three of the four benchmarks (DAEval +2.39, DABStep +2.79, QRData-V +6.65); on single-table-dominated TableBench the +0.34 difference is within noise. The largest effect is on QRData-V, where the graph-free model (43.29) falls below the backbone (46.48) and the full model rises above it (49.94).

(3) Effect on task composition. The graph also changes which tasks exist at all (Table 4): tasks spanning ≥2 tables rise from 64.9% to 75.9% (+11.0 pts), tasks spanning ≥3 tables from 0.0% to 19.4% (+19.4 pts), and hard tasks from 0.3% to 17.7% (+17.4 pts). These harder, multi-table tasks are the supervision behind the DABStep and QRData-V gains above.

Q2 (W1)

We compare released DataMind models and a controlled reproduction of its synthesis pipeline (in progress).

(1) Released DataMind models: competitive with about 6× less data and no RL. The two methods differ on data volume and training procedure. DataMind performs an SFT cold start followed by dynamically interleaved SFT + DAPO reinforcement learning, using 11,707 expert trajectories (DataMind-12K) sampled with DeepSeek-V3.1 and Qwen2.5-Coder-7B/14B-Instruct students. Data2Task, by contrast, uses only 2,000 trajectories (5.9× fewer), plain SFT with no RL stage, and a Qwen3-8B student, whose size sits between the two DataMind models. On the three benchmarks unaffected by the TableBench scoring issue discussed in Q3, Data2Task-8B averages 50.89, against 45.49 for DataMind-7B and 52.29 for DataMind-14B; with TableBench included, the four-benchmark averages are 55.17 vs. 38.45 and 42.90.

(2) Fully controlled comparison with a reproduced DataMind pipeline. To isolate the contribution of the task-synthesis method itself, we re-implemented DataMind's task-generation method end-to-end and applied it to the same data pool under the same 2,000-trajectory budget as Data2Task: GLM-5.2 served as the teacher, the same trajectory filtering was applied, 2,000 trajectories were generated, and they were used to fine-tune the same Qwen3-8B student with the same SFT hyper-parameters as our Data2Task model.

MethodEvaluationDAEvalDABStep
DATA2TASK-8BBenchmark official test84.16 ± 0.6918.57 ± 0.66
DataMind (reproduced)LLM judge (GLM-5.2)83.56 ± 1.6617.88 ± 1.30

Because we re-implemented DataMind's task-generation method end-to-end, we also scored the reproduction under DataMind's original LLM-as-judge protocol rather than forcing it through our own benchmark harness. DataMind's paper uses a GPT-4o-mini judge; we instantiate the judge as GLM-5.2. Our Data2Task model, by contrast, is evaluated under the benchmarks' standard official test protocol. This makes the evaluation of the DataMind reproduction more lenient than the evaluation of our own method, yet Data2Task still outperforms it on both DAEval and DABStep. In other words, even with the comparison set up in the baseline's favor, our synthesized tasks yield a better student, indicating that the advantage stems from the task-generation method rather than from evaluation or training artifacts.

AuthorsOpenReview ↗

Response to Reviewer izHP — Q3 & Q4

Q3 (W1)

We address this in two parts: (i) the consistency of our evaluation protocol across models, and (ii) averages recomputed without TableBench.

(1) Our evaluation applies the same protocol to every model. DataMind is run in its own native harness (preserving its training and inference configuration), and every model is scored with the same official TableBench metric — exact match on the FactChecking and NumericalReasoning subsets. The TableBench scores we report for DataMind are therefore obtained under the same protocol as all other models. The discrepancy with DataMind’s self-reported results arises because the original DataMind paper scores TableBench with a GPT-4o-mini LLM judge rather than the official exact-match metric, so the two sets of numbers are not comparable. Exact match cannot distinguish answers that are substantively correct but format-non-compliant from answers that are genuinely wrong — it marks both as incorrect, which is why DataMind scores below the untuned backbone on this benchmark.

(2) Averages excluding TableBench. Following the reviewer’s suggestion, we report averages without TableBench: Data2Task-8B reaches 50.89%, i.e., +5.40 points above DataMind-7B (45.49%) and only −1.40 points below DataMind-14B (52.29%). Given that DataMind-14B has nearly twice the parameters (14B vs 8B), our 8B student is effectively at parity, with the residual gap plausibly explained by capacity.

Q4 (W2)

We address this in three parts: (i) answer correctness and naturalness, (ii) column-link accuracy, and (iii) the semantic validity of the generated questions.

(1) Answer correctness and naturalness. We audited all 2,000 tasks in two stages. First, an agent re-derived each answer from the data. Second, three human experts independently audited the agent's re-derivations against the data and reconciled any disagreements into a final judgment. Based on this audit, 1,991 of the 2,000 answers (99.55%) are correct and only 9 (0.45%) are incorrect, while all 2,000 questions (100%) are fluent and natural in wording (analytical soundness is assessed separately in (3)). We do not rely on a single rollout for credibility: each answer is checked through multiple rollouts, cross-model agreement, an executable reference program, and column hints, so the verified answer is as trustworthy as possible.

(2) Column-link accuracy. We sampled 100 semantic nodes (concepts) jointly covering 8,000 physical columns and judged each column against its assigned concept. Only 246/8,000 (3.1%) columns are mislinked — a 96.9% link accuracy — confirming the pipeline reliably groups semantically similar columns into the correct concept.

(3) Semantic validity of the questions. We do not claim perfect semantic validity and quantify it directly: in the human audit, 81% of tasks were judged semantically and logically appropriate. A task is judged appropriate when the question poses a meaningful analysis — its operation (aggregation, ranking, correlation, or arithmetic) has a clear analytical rationale and involves only commensurable quantities — rather than, for example, adding or subtracting unrelated values.

AuthorsOpenReview ↗

Response to Reviewer izHP — Q5 & (W3) Related Work

Q5 (W4)

Synthesis cost was not logged for the full 2,000-task run, and our runtime budget did not allow re-running all 2,000 tasks. We therefore benchmarked the pipeline end-to-end on the 500 candidates and verified that both cost and time scale linearly with the number of candidates, i.e. the per-candidate cost is constant. Retention was re-measured separately on a random 200-candidate sample.

(1) Cost and time scale linearly with the number of candidates

The graph is built once; task generation then costs a near-constant amount per candidate. The stepwise benchmark confirms this — each additional 100 candidates adds a roughly constant number of requests, tokens, and time:

CandidatesRequestsΔ req.TokensΔ tokensLLM time (cum.)Δ LLMTask wall-clock (cum.)Δ wall-clock
1003,824—78.5M—22.4 h—15.0 h—
2007,504+3,680153.9M+75.4M45.2 h+22.8 h30.4 h+15.4 h
30011,054+3,550227.3M+73.4M66.7 h+21.5 h45.3 h+14.9 h
40014,780+3,726295.8M+68.4M90.7 h+24.0 h61.4 h+16.1 h
50018,845+4,065382.8M+87.0M116.0 h+25.3 h78.4 h+17.0 h

The per-100-candidate increment stays in a narrow band (requests ≈ 3.6–4.1k, tokens ≈ 68–87M, LLM time ≈ 21.5–25.3 h, task wall-clock ≈ 14.9–17.0 h), i.e. the marginal cost is constant. The final 500-candidate breakdown is:

CandidatesCallscomprehenddesignsolverewritesemantic_judgeinput tokensoutput tokenstotal tokens
50018,8451,10259114,9411,1481,063359.4M23.4M382.8M

(2) Cost summary (including the graph build)

The graph is built once; task generation is billed linearly per candidate:

ItemRequestsTokensTime
Graph build (one-time)1,3308.4M~1.4 h
Task generation (per candidate)~37.7~0.77M~13.9 min (LLM) / ~9.4 min (wall-clock)
Task generation (500 candidates)18,845382.8M116.0 h (LLM) / 78.4 h (wall-clock)

LLM time is the sum of per-request durations; it exceeds the task wall-clock because solve runs two models in parallel, so request times overlap within a task.

(3) Retention (re-measured on a random 200-candidate sample)

157/200 candidates were retained (78.5%); the 43 rejections are 1 comprehension failure, 30 design failures, 9 solves without a usable answer, and 3 solves without consensus.

W3 (Related work)

We agree this is an omission and will expand the related-work section to cover overlap-based join discovery (LSH Ensemble, JOSIE), semantic join and union discovery (DeepJoin, SANTOS, Starmie), and end-to-end discovery systems (Aurum, SemProp, D3L). Offline relationship construction clearly has precedents; what is new in Data2Task is using those relationships to synthesize controllable training supervision. We summarize the distinctions in three points.

(1) Problem setting and output. Data2Task synthesizes training tasks from a heterogeneous file pool: the input contains no analytical question, and the output is the question, an executable reference program, its answer, and difficulty metadata. Discovery methods instead retrieve tables, column correspondences, or integration paths for a supplied column, table, example, or information need. Recent work such as Data.Readme also precomputes relationships for data agents, but uses them to help the agent solve given tasks; we turn relationships into new training tasks.

(2) The graph is built for task construction, not retrieval. Beyond source membership and concept correspondences, it records analytical roles, dimension–measure relations, and file partition axes. For example, a year appearing only in filenames becomes a virtual partition field, enabling time-based analysis and file selection. A sampled subgraph is converted into an input signature — available measures, grouping dimensions, time axes, and cross-source relations — and operator contracts then determine which analytical plans can be instantiated. The contribution is how these structures jointly support task construction.

(3) The relationships feed a downstream synthesis pipeline. Data2Task samples source subgraphs, binds compatible operators, composes analytical branches under category quotas, instantiates questions, obtains reference answers by execution, retains candidates supported by independent solver rollouts, and re-verifies answers after rewriting, recording structural and semantic difficulty separately. This coordinated process is what turns discovered relationships into controllable training tasks.

We thank the reviewer for the feedback and will incorporate the suggestions and supplementary experiments into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.