- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces DATA2TASK, a framework for synthesizing executable data-analysis tasks from heterogeneous real-world data lakes. It constructs a physical–semantic dual-view graph to organize files and fields and align concepts across tables, then samples subgraphs and compatible analytical operators to generate tasks. Execution-based answer agreement, query rewriting and re-verification, and difficulty filtering determine which tasks are retained. Using 2,000 selected teacher-generated trajectories for supervised fine-tuning, the authors improve Qwen3-8B’s average pass@1 from 45.59 to 55.17 across four evaluation benchmarks. Additional experiments examine graph removal, task difficulty, and trajectory selection.
Critical strengths
- A coherent approach to an important training-data bottleneck. Separating relation discovery from task generation makes cross-file relationships explicit and reusable, while provenance tracking and replayable parsing make the underlying transformations inspectable. This is a useful design for generating analytical tasks over heterogeneous files whose relationships are not supplied as a database schema.
- Consistent downstream improvements with a modest final training set. Table 1 reports improvements on all four benchmarks using only 2,000 selected trajectories, including gains of 16.20 percentage points on DAEval and 12.13 points on the evaluated TableBench subsets. These results support the practical utility of the synthesized supervision. The graph-removal and trajectory-selection ablations also provide useful evidence beyond comparison with the untuned backbone.
Critical weaknesses
- Semantic validity remains insufficiently established, despite being central to the motivation. Section 3.4 verifies agreement between executed answers, but agreeing solvers can share an interpretation of an ill-posed or analytically unjustified question. For example, supplementary task datamind_hetero_v2_epidemiology_surveillance_population_health_1327_1 asks about the ratio of COVID-19 recoveries to daily SARS cases during the 2003 SARS outbreak, returns 2003-04-12, and is labeled high confidence. Its temporal interpretation is unclear. Another high-confidence task, datamind_hetero_v2_statistics_and_general_reference_3089_1, subtracts a historical Inquisition count from an outbreak case-count statistic without an evident analytical rationale. These examples do not establish a dataset-wide error rate, but they expose a gap between computational agreement and meaningful analysis. Without an independent assessment of semantic validity and answer correctness, the claimed quality of the core output—the training tasks—remains uncertain.
- The incremental value of the central graph design is incompletely validated. With the same 2,000-trajectory training budget, Table 4 reports DAEval scores of 81.77 without the graph and 84.16 with it, compared with 67.96 for the backbone. The 2.39-point increment may be useful, but downstream graph ablation is reported on only one benchmark, without uncertainty estimates, and removes the representation and graph-guided motif construction together. Consequently, it does not establish which aspects of the dual-view design contribute or whether their benefit generalizes across the evaluation suite. Equal final training-set size also does not ensure equal synthesis cost: graph construction, candidate rejection, and repeated verification costs are not sufficiently quantified. This limits the justification for the added machinery relative to the simpler pipeline.
Questions for authors
-
How is the SARS/COVID-19 task identified above intended to be interpreted, particularly its temporal alignment? What checks distinguish meaningful cross-source analysis from an executable but semantically unjustified combination?
-
Has a representative sample of retained tasks been independently audited for semantic validity and numerical correctness separately? If so, what were the sampling protocol and error rates, particularly for high-confidence cross-table tasks?
-
Were the Table 4 variants matched for synthesis resources as well as final trajectory count? What were their generation costs and retention rates, and are graph-removal results on the other three benchmarks and variation across runs available?
6 comments
Response to Reviewer Mt9v — Q2
Q2 — Audit protocol and dataset-wide error rates
Full audit across two independent dimensions. We audited all 2,000 retained tasks — a full census rather than a sample — and human judges scored each task on two independent axes: (i) numerical correctness of the answer and (ii) semantic/logical validity of the question.
How each axis was scored.
- Numerical correctness. For each task, an agent re-derived the answer from the final question and the data lake, without column hints or access to the reference. Three human experts then independently audited the agent's derivations against the data and confirmed that the agent solved the questions correctly. Across the 2,000 tasks, 1,991 (99.55%) of the stored answers are correct and 9 (0.45%) are incorrect.
- Semantic/logical validity. Human judges evaluated every task on whether its cross-source combination carries a coherent analytical rationale, screening against a fixed taxonomy of failure modes rather than a free-form judgment. The taxonomy we screened for is:
- Unrelated/arbitrary cross-source operations (~50%, the dominant class) — combining quantities that share a computational interface but no analytical relationship (the Inquisition example).
- Unit/scale incommensurability (~13%) — dividing or subtracting quantities with incompatible units into a meaningless ratio/difference.
- Identifiers/nominal codes used as measurements (~10%) — statistics or trends computed over IDs or arbitrary orderings.
- Geographically/entity-incoherent joins (~7%) — linking records across unrelated geographies or coincidental surrogate IDs.
- Temporal misalignment (~7%) — pairing non-overlapping periods or aligning series by index rather than date (the SARS/COVID example).
- Circular/tautological operations (~6%) — predictors or terms that are constituents of, or definitionally identical to, the target.
- Statistically degenerate operations (~6%) — tests/trends that are trivially determined (tiny samples, monotonic cumulative sums, constant predictors).
Results.
By verification confidence.
| Confidence | Tasks | Answer correct | Semantic appropriate |
|---|---|---|---|
| High | 1,963 | 99.54% | 81.20% |
| Medium | 37 | 100.00% | 70.27% |
| Overall | 2,000 | 99.55% | 81.00% |
By number of referenced tables.
| Tables | Tasks | Answer correct | Semantic appropriate |
|---|---|---|---|
| 1 | 482 | 99.38% | 85.27% |
| 2 | 1,131 | 99.47% | 78.78% |
| 3+ | 387 | 100.00% | 82.17% |
| ≥2 (all cross-table) | 1,518 | 99.60% | 79.64% |
Two main findings.
- Cross-table complexity and verification confidence barely affect answer correctness. Accuracy is essentially flat across both the number of referenced tables (99.38%–100%) and the confidence level (99.54% high vs. 100% medium). Neither cross-table joins nor verification confidence is a driver of numerical error.
- Semantic validity is lower for cross-table tasks than for single-table tasks. Single-table tasks are the most appropriate (85.27%) because they involve the fewest semantic links. The two-table group — the largest group (1,131 tasks) — is the least appropriate (78.78%), so the residual semantic defects concentrate in multi-table tasks rather than being spread uniformly.
Response to Reviewer Mt9v — Q3
Q3 — Resource matching, generation cost, and retention of the graph ablation
What the two variants hold fixed. The two variants are identical in every respect except the graph. They share the same source domains, the same trajectory teacher and verification procedure, the same trajectory filtering and difficulty scorer, the same rollout generation, trajectory selection, and SFT recipe, and the same final 2,000-trajectory budget; model choices and retry limits are unchanged throughout. The only difference is the dual-view graph: the graph-free variant drops the graph and its guidance on task construction, so the generator receives the raw domain files directly. This ablation removes the graph as a whole. The composition results point to the semantic view as the source of the gain: tasks over three or more tables require cross-table links, and none appeared without the graph.
Synthesis cost and retention. To match the two variants for synthesis resources — not only for final trajectory count — we draw a matched random sample of 200 candidates from each and report cost and retention separately.
Retention. On the matched 200-candidate sample, the graph variant retains 157/200 (78.5%) and the graph-free variant 185/200 (92.5%). The gap is driven by the design stage: with the graph, the design step must instantiate and validate the cross-table joins specified by the motif, so candidates whose multi-table logic is inconsistent are rejected there (30 design failures vs. 8); without the graph, the generator is not committed to any cross-table structure and produces mostly single- and two-table tasks that rarely fail this check. The graph’s lower retention is therefore the cost of attempting cross-table tasks the graph-free variant does not even generate.
Synthesis cost. We acknowledge that the graph adds cost: the one-time graph construction and the graph-structured input to the generator increase calls and tokens. On the matched 200-candidate sample (task generation only; the one-time graph build is excluded and amortized below):
| Metric (per 200 candidates) | With graph | Without graph |
|---|---|---|
| Provider requests | 7,504 | 6,376 |
| Tokens (recorded) | 153.9M | 91.9M |
Per candidate the graph consumes ~1.7× the tokens and ~1.2× the requests. The two variants were thus matched on final trajectory count, and the graph variant used more synthesis budget. Additional budget would not close the gap for the graph-free generator: among its 2,000 tasks, none spans three or more tables and only 0.3% are hard. This overhead is bounded and controllable: the graph build is a one-time cost (1,330 requests, ~8.4M tokens) that amortizes to ~4.2K tokens per task over the 2,000-task corpus, and per-candidate calls are capped by the fixed candidate budget and retry limits, so the graph scales only a constant factor rather than the total synthesis budget.
Downstream value across all four benchmarks. The added cost is justified downstream. Extending Table 4 to all four benchmarks with seed variance:
| pass@1 | DAEval | DABStep | QRData-V | TableBench | Avg. |
|---|---|---|---|---|---|
| Qwen3-8B backbone (Table 1) | 67.96 | 12.01 | 46.48 | 55.89 | 45.59 |
| With graph (Table 1) | 84.16 ± 0.69 | 18.57 ± 0.66 | 49.94 ± 0.82 | 68.02 ± 0.33 | 55.17 |
| Without graph | 81.77 ± 0.52 | 15.78 ± 1.09 | 43.29 ± 1.36 | 67.68 ± 0.49 | 52.13 |
With the graph, cross-table tasks (≥ 2 tables) rise from 64.9% to 75.9% (+11.0 points), driven almost entirely by tasks spanning three or more tables, which go from 0.0% to 19.4% — none arose among the 2,000 graph-free tasks. The difficulty distribution shifts in parallel: hard tasks rise from 0.3% to 17.7% (+17.4 points), while easy tasks fall from 37.7% to 31.8% and medium tasks from 62.0% to 50.6%. These harder, multi-table tasks are the supervision behind the DABStep (+2.79) and QRData-V (+6.65) gains, whereas TableBench — dominated by single-table reasoning — gains only +0.34 because its task mix is barely affected by the graph.
Response to Reviewer sdJe — Q1
We thank the reviewer for the careful reading and for pinpointing the two concrete task examples, which expose a real gap between computational agreement and meaningful analysis. We address the concerns in turn below.
Q1 — Semantic validity and the two flagged examples
Acknowledgment and classification. We thank the reviewer for these concrete examples. They are correct, and both point to the same underlying gap: computational agreement does not by itself certify meaningful analysis. The two tasks fail in different ways, so we separate them before addressing each.
We do not claim perfect accuracy or semantic validity. In particular, perfect semantic validity cannot be guaranteed: the final query is phrased by a stochastic LLM, so a residue of well-formed but analytically unjustified questions is inherent to the approach. Our claim is narrower and is what the design is built to secure — that answer errors are rare and that semantic failures are bounded and now measured (Q2) — through the two layers below.
How we bound the two failure modes.
weather_2023.csv), recorded as the cluster’s partition axis. Concept alignment is screened by name, value, and type compatibility before an LLM partitions fields into concepts, and links are admitted only between same-concept, role-compatible fields (typed equi/fuzzy/spatial/temporal). Operator contracts then bind a measure only to compatible dimensions, so an operation is instantiated only when roles are semantically consistent. These type-level constraints are necessary but not sufficient, as the two examples show: the Inquisition task passed because two counts with unknown units satisfy the difference contract; the SARS/COVID-19 task passed because the two series share a date concept, with no check that their periods overlap. No step in the current pipeline asks whether a combination is analytically meaningful.Measured impact and residual value. To quantify rather than assert, we audited the full 2,000-task set with human judges on two separate axes: 99.55% of answers are correct and 81% of tasks are semantically/logically appropriate (protocol and full breakdown in Q2). The residual semantically-weak tasks are not worthless: when a trajectory’s cross-table relations and computation are independently confirmed, it still supervises file location, field matching, key–value alignment, and cross-table joining — the procedural skills a data-analysis agent must acquire — even if its final analytical framing is imperfect.
What we will do. We will report this audit and state semantic validity as a limitation in the revision, and leave pipeline-level checks for semantic validity (e.g., a temporal-overlap check and a post-rewrite semantic judge) to future work.