acceptodds
Under review as a conference paper at ICLR 2027

Table-Zero: Self-Improvement for Complex Table Reasoning without Additional Human Supervision

Abstract

Complex table reasoning requires agents to combine evidence across multiple tables through multi-step reasoning, but high-quality training data for these tasks is costly to build. Self-improving reinforcement learning (RL) offers a promising alternative: a model generates its own training questions and learns from them without human labels. However, existing self-improving methods struggle in this setting: it is hard for a model to generate questions that require multiple tables, and the reference answers to such questions may be wrong. To address this, we introduce \method, a self-improvement framework that uses the table collection itself to choose which tables each question combines and to compute its answer. \method selects related tables through two intrinsic properties of tables, joinability and decomposability. A proposer writes a question and reference SQL over these tables, and executing the SQL supplies the reference answer. The proposer and the solver then improve in turn: the proposer is trained with DPO on its own checked questions, and the solver is trained with RL on the resulting tasks, for three iterations. Across seven datasets, \method achieves an average exact match of 32.3% with Qwen3-8B and 37.0% with Qwen3-14B, compared with 20.4% and 24.3% for , respectively. At both model sizes, it exceeds the average accuracy of adapted self-improvement baselines and RL trained on human-annotated tasks.

Primary area:
reinforcement learning
Keywords:
self-improvement, reinforcement learning, multi-table reasoning, question answering, SQL execution, synthetic training data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GVTaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TABLE-ZERO, a self-improvement framework for complex table reasoning using automatically generated training tasks. It exploits joinability and lossless table decomposition to organize task-generation contexts, while SQL execution provides reference answers. The framework alternates DPO training of a task proposer with GRPO training of a solver over three iterations. Experiments with Qwen3-8B and Qwen3-14B across seven datasets report higher average exact-match accuracy than prompting, adapted self-improvement baselines, and reinforcement learning on human-annotated tasks.

Critical strengths
  • The paper addresses a problem of practical importance: improving complex table reasoning without additional human task annotations. The proposed use of table structure to guide task generation is intuitive and well motivated.

  • The experiments demonstrate substantial improvements across multiple datasets. TABLE-ZERO achieves the highest average exact-match accuracy among the compared methods at both evaluated model sizes.

Critical weaknesses
  • The main experiments do not sufficiently disentangle the quality of the generated data from the effectiveness of the overall training framework. As stated in lines 925–928, TABLE-ZERO and the adapted baselines differ not only in task generation but also in solver optimization, including online filtering and difficulty weighting. Consequently, the performance gains cannot be clearly attributed to better training data. Moreover, Table 1 shows that TABLE-ZERO without proposer training already achieves substantial gains, while proposer training provides a comparatively smaller improvement. This suggests that the task-construction pipeline may account for much of the benefit, although its contribution is not isolated. I recommend comparisons under matched solver-training settings.

  • The human evaluation is limited to one expert assessing 15 generated tasks and 15 human-annotated tasks. This small sample provides limited evidence about the overall quality of the generated data. Evaluation by multiple annotators on a larger sample would make the findings more convincing.

  • Although DPO improves the usable-task yield, the absolute yield remains low: Figure 5 reports 111 usable tasks per 10,000 proposals after the third round, approximately 1.11%. This raises questions about the generation and filtering effort required to obtain a sufficiently large training pool. The paper would benefit from clarifying the practical implications of this low acceptance rate.

  • The paper does not clearly report downstream solver performance after each self-improvement iteration. Figure 5 shows improvements in proposer task yield across rounds, but these do not directly establish corresponding improvements in solver performance. I recommend reporting solver evaluation results using task pools generated by the proposer at each iteration, with comparable solver-training budgets.

  • The paper lacks a detailed comparison of computational efficiency across methods. Reporting wall-clock time for task generation, verification, proposer training, and solver training, together with the number of unique training tasks used, would help assess the practical cost of the reported gains. A common update budget alone does not establish comparable computational cost.

Questions for authors
  • How much of TABLE-ZERO’s improvement remains when the compared methods use the same solver optimization settings and comparable training budgets? Can the authors isolate the contribution of task construction from online filtering and difficulty weighting?

  • Can the authors provide a larger-scale evaluation of generated-task quality with multiple annotators, particularly regarding question–SQL consistency and whether answering genuinely requires multiple tables?

  • How does downstream solver performance change after each self-improvement iteration? Can the authors compare solvers trained on task pools generated at different proposer iterations under comparable training budgets?

  • Given the low absolute usable-task yield, what is the computational cost of obtaining the final training pools? Please report task counts and wall-clock time for generation, verification, proposer training, and solver training across methods.

  • Does TABLE-ZERO generalize to LLM families beyond Qwen? Have the authors evaluated a non-Qwen model to determine whether the observed improvements depend on the initialization’s SQL generation and tool-use capabilities?

2 comments

Thank you for your helpful review. We agree that the original experiments did not clearly separate the contribution of the generated task pools from differences in solver optimization. We have conducted additional controlled comparisons and expanded the task-quality audit.

Q1. How much improvement remains under matched solver-training settings?

We now compare Qwen3-8B solvers using the same online group filtering, difficulty weighting, and training budget: 75 updates, with 128 prompts and eight rollouts per prompt at each update. All methods use the same checkpoint-selection rule, selecting the checkpoint with the highest score on their training tasks.

Under this setup, average EM increases from 23.1 to 26.1 for Dr. Zero and from 23.0 to 24.7 for SPICE. Table-Zero with a frozen proposer achieves 30.2 EM, exceeding these baselines by 4.1 and 5.5 percentage points. Training the proposer raises average EM to 31.1 with GRPO and 32.3 with DPO.

These results show that the advantage persists after standardizing solver optimization. They also support your observation that much of the improvement is already present without proposer training, with DPO providing an additional 2.1 points. However, task-pool sizes remain different: Dr. Zero and SPICE retain 170 and 312 tasks within the generation budget, whereas Table-Zero reaches 1,000. The experiment therefore controls solver optimization but does not separate per-task quality from the amount of available training data.

Q2. Can the authors provide a larger evaluation of generated-task quality with multiple annotators?

We expanded the evaluation to cover both decomposability and joinability. For the untrained Qwen3-8B proposer and DPO rounds 1–3, we evaluated balanced 1,000-task pools containing 500 tasks from each construction route. We also conducted a broader audit of 8,717 deduplicated candidates from the corresponding generation runs, covering 329 source tables.

For the broader audit, we shuffled the questions and ran Claude Opus 5.5 twice with the same rubric and SQL access to the full tables. The judge assessed strict correctness and table necessity separately. Strict correctness required the reference SQL to answer the question as written and the stored label to contain the requested columns in the requested order, the correct rows, and the specified row order whenever applicable. For table necessity, the judge identified the smallest sufficient set of supplied fragments, counting a fragment only when answering required a column unique to it.

The two runs agreed on 98.5% of correctness judgments (Cohen’s κ = 0.97) and 99.5% of judgments about whether at least two tables were required. A third run adjudicated 171 entries with a disagreement or missing verdict. Returning extra columns was the most common correctness error.

The results for the balanced pools covering both construction routes are:

UntrainedDPO round 1DPO round 2DPO round 3
Tasks evaluated1,0001,0001,0001,000
Decomposability tasks500500500500
Joinability tasks500500500500
Strictly correct719 (71.9%)713 (71.3%)717 (71.7%)758 (75.8%)
Require ≥2 tables792 (79.2%)826 (82.6%)894 (89.4%)897 (89.7%)

Each percentage uses the corresponding 1,000-task pool as its denominator. Strict correctness and table necessity are assessed independently, so these rates do not indicate the proportion satisfying both criteria. From the untrained proposer to DPO round 3, strict correctness increases by 3.9 percentage points and the proportion requiring at least two tables increases by 10.5 percentage points.

We also expanded the human evaluation to 200 tasks, with one expert assessing 50 tasks from each stage. Initial judgments classified 16, 25, 37, and 39 tasks as strictly correct and 43, 46, 47, and 48 as requiring at least two tables, respectively. This provides a larger human evaluation than the original study, although it does not include multiple human annotators. We report these human judgments separately from the automated audit results.

Q3. How does downstream solver performance change across self-improvement iterations?

We additionally report Qwen3-8B solver performance across three iterations. In the self-improvement procedure, each iteration first updates the proposer with DPO, then generates a task pool with the updated proposer, and finally trains the solver on that pool. The base model achieves 20.38% average EM across the seven evaluation datasets.

Solver step within iterationIteration 1Iteration 2Iteration 3
1521.8830.2931.47
3025.8530.4931.33
4528.7330.8031.54
6030.0731.1631.64
7530.1931.2232.30

All values are average EM (%). At the 75-update checkpoint within each iteration, performance increases from 30.19% in iteration 1 to 31.22% in iteration 2 and 32.30% in iteration 3, corresponding to gains of 1.03 and 1.08 percentage points.

Q4. What is the computational cost of obtaining the training pools?

The 111 usable tasks per 10,000 proposals in Figure 5 is a held-out proposer-evaluation metric with stricter acceptance criteria than those used to construct solver-training pools. In addition to the existing checks, this evaluation uses an LLM judge to assess naturalness and rewriting sufficient to avoid leakage. It should therefore be distinguished from the acceptance rate during training-pool construction.

For the actual Qwen3-8B solver pools, the proposal counts are:

ProposerProposalsRetained training tasksAcceptance rate
Untrained92,7171,0001.08%
DPO round 148,8441,0002.05%
DPO round 241,2321,0002.43%
DPO round 341,4181,0002.41%

Round 3 therefore requires approximately 55% fewer proposals than the untrained proposer to obtain the same number of retained tasks. For comparison, under a generation budget of up to 120,000 proposals, Dr. Zero and SPICE retain only 170 and 312 tasks, respectively.

To quantify the cost beyond proposal counts, we report wall-clock runtimes for proposer training, generation, verification, and solver training. All runs use one eight-GPU H100 node. Values are in hours, and slash-separated entries correspond to iterations 1, 2, and 3. Each total is the sum of the listed stage runtimes.

MethodProposer trainingGenerationVerificationSolver trainingTotal
Dr. Zero, Qwen3-8B4.6 / 7.7 / 6.21.2 / 1.3 / 1.21.3 / 1.7 / 1.63.1 / 4.2 / 4.338.4
SPICE, Qwen3-8B6.3 / 11.7 / 10.32.1 / 2.3 / 2.21.1 / 0.8 / 0.83.1 / 3.6 / 3.347.6
Dr. Zero, Qwen3-14B7.8 / 9.0 / 8.92.2 / 2.3 / 2.21.0 / 2.1 / 2.04.7 / 4.5 / 4.451.1
SPICE, Qwen3-14B6.9 / 10.0 / 7.41.4 / 1.3 / 1.20.5 / 0.6 / 0.53.6 / 3.3 / 3.440.1
Table-Zero (w/o proposer training), Qwen3-8B–0.230.035.35.56
Table-Zero (w/o proposer training), Qwen3-14B–0.240.036.66.87
Table-Zero with DPO proposer, Qwen3-8B5.5 / 2.1 / 2.42.9 / 0.3 / 0.31.3 / 0.6 / 0.75.3 / 5.1 / 5.331.8
Table-Zero with DPO proposer, Qwen3-14B5.4 / 4.3 / 4.32.0 / 0.5 / 0.50.9 / 1.3 / 1.36.6 / 6.7 / 6.640.4

For Qwen3-8B, Table-Zero with DPO takes 31.8 hours across the three iterations, compared with 38.4 hours for Dr. Zero and 47.6 hours for SPICE. For Qwen3-14B, it takes 40.4 hours, compared with 51.1 hours for Dr. Zero and 40.1 hours for SPICE. Table-Zero therefore has lower total runtime than both baselines at 8B and than Dr. Zero at 14B, while its 14B runtime is close to SPICE’s.

For Table-Zero with DPO, generation and verification together account for 6.1 hours at 8B and 6.5 hours at 14B across the three iterations. Proposer and solver training account for the remaining 25.7 and 33.9 hours, respectively. Thus, although proposal acceptance remains low, training accounts for most of the measured runtime.

The frozen-proposer variants take 5.56 hours at 8B and 6.87 hours at 14B. These variants use a single generation and solver-training run, so their totals should be distinguished from those of the three-iteration methods. The reported timings describe the evaluated pipelines; they do not represent a comparison with equal numbers of retained training tasks.

Q5. Does Table-Zero generalize beyond Qwen?

Our current experiments evaluate Qwen3-8B and Qwen3-14B, covering two model sizes within the same family. We agree that these results do not establish generalization beyond Qwen. We will add experimental results for a non-Qwen model in the camera-ready version.

Reviewer_CayAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Table-Zero, a framework for improving table-reasoning agents using automatically generated training tasks. It organizes task construction through two properties of relational tables: joinability, which selects connected tables using known join relations, and decomposability, which partitions a table into losslessly reconstructable fragments. A proposer generates a question and reference SQL, whose execution supplies the answer label. The proposer is trained with DPO using validity and difficulty preferences, while the solver is trained with GRPO to retrieve relevant sources and answer the generated questions. The framework alternates these updates over three iterations. Experiments train Qwen3-8B and Qwen3-14B on MMQA sources and evaluate on MMQA plus six transfer datasets. Table-Zero achieves average exact match of 32.3% and 37.0%, respectively, compared with 20.4% and 24.3% for prompting, and exceeds the evaluated self-improvement baselines and human-annotated RL on average. The reported implementation uses SQL-derived join metadata and shared-key decomposition, avoiding additional task annotation while retaining inherited relational supervision.

Critical strengths
  1. A practical mechanism for constructing executable tasks over distributed tabular evidence. Joinability and decomposability address a concrete bottleneck in self-generated table QA: identifying related sources while retaining a reproducible basis for computing answer labels. In the decomposition route, verified reconstruction preserves the source data, and excluding the original table creates an environment in which the solver must work with distributed fragments. This provides a useful foundation for generating training tasks without additional human question–answer annotation.

  2. Substantial reported performance gains with broad transfer evaluation. The full method improves over prompting at both model sizes and on all six transfer datasets, spanning table, mixed table–passage, and passage-only reasoning. The no-proposer-training variant already achieves average EM of 30.2% and 35.1%, supporting the practical value of the task-construction and solver-training pipeline. These results provide meaningful evidence of utility beyond the training benchmark, even though further controls are needed to attribute the gains to individual components. 57412_Table_Zero_Self_Improvem

Critical weaknesses
  1. The comparisons limit attribution of the gains to the proposed task construction and adaptive self-improvement. Appendix C.1 states that Table-Zero and Human Annotated RL use online group filtering and difficulty weighting that are absent from the adapted Dr. Zero and SPICE solver runs. Although all solver runs have a common 75-update budget, task-pool sizes and total compute remain method-specific, and the reported checkpoints differ without a common selection rule. These comparisons support the performance of the complete configurations, but do not isolate the contribution of the proposed task source under matched solver optimization. The incremental benefit of proposer training is also supported by limited robustness evidence. DPO improves average EM over the untrained-proposer variant by approximately 2.1 and 1.9 percentage points, but the main comparison uses one training run per method and model setting. The five-source-partition study does not estimate independent training-run variability or provide fold-matched baseline comparisons. This limits confidence in the reproducibility and attribution of the additional self-improvement gains.

  2. Generated-task faithfulness and genuine multi-table necessity are insufficiently quantified. SQL execution establishes a program’s output, but does not establish that the program faithfully answers its paired natural-language question. Similarly, Appendix B.3 explicitly notes that lexical matching of SQL columns to fragments does not establish that each fragment is semantically necessary. The expert study provides encouraging evidence: 15 generated tasks receive a mean reference-answer correctness score of 3.00/3. However, this small sample, assessed by one expert, does not establish error rates across both construction routes and successive iterations. Section D describes auditing the curated MMQA pool, but its coverage of generated training pools is unclear. Furthermore, the expert questionnaire collects the smallest sufficient table set without reporting aggregate results for that criterion. These omissions leave uncertainty about how consistently the pipeline generates faithful questions requiring the intended multi-table reasoning.

Questions for authors
  1. Controlled comparison and robustness: What advantage remains when generated task sources are compared using the same solver optimizer, filtering, difficulty weighting, rollout budget, and checkpoint-selection rule? Please provide task-pool sizes and training costs by method and iteration, together with any available independent-run variability for the DPO versus untrained-proposer comparison.

  2. Experimental provenance: Which exact variants and checkpoints underlie Figure 3, Figure 5a, and Table 6? Please reconcile Section 4.3’s “without proposer training” description with the DPO labels and explain the matching displayed scores between Figure 5a’s s4 and Table 1’s no-proposer-training row.

  3. Generated-task validation: Does the Section D audit cover generated training tasks, and, if so, which pools and iterations? What evidence is available about question–SQL faithfulness and the proportion of tasks genuinely requiring at least two tables, separately for joinability and decomposability? Please also report the aggregate smallest-sufficient-table-set results from the expert study.

2 comments

Thank you for your careful review. We agree that the original comparisons did not sufficiently separate the contribution of the generated task pools from differences in solver optimization. We have therefore conducted a comparison under common solver-training settings and expanded the evaluation of generated-task quality. We also clarify the provenance of the diagnostic results below.

Q1. Controlled comparison, task-pool sizes, computational cost, and robustness

We now compare Qwen3-8B solvers using the same training configuration, including online group filtering, difficulty weighting, and a budget of 75 updates, with 128 prompts and eight rollouts per prompt at each update. We apply the same checkpoint-selection rule to every method, selecting the checkpoint with the highest score on its training tasks.

Under this setting, average EM increases from 23.1 to 26.1 for Dr. Zero and from 23.0 to 24.7 for SPICE. Table-Zero with a frozen proposer achieves 30.2 EM, retaining advantages of 4.1 and 5.5 percentage points, respectively. Training the proposer raises average EM to 31.1 with GRPO and 32.3 with DPO. Thus, Table-Zero’s advantage persists under common solver optimization, and substantial gains are already present without proposer training. DPO provides a further 2.1-point improvement over the frozen-proposer variant.

For the Qwen3-8B task pools, we targeted 1,000 training tasks per method, allowing each baseline up to 5 generation rounds of 24,000 proposals. Dr. Zero retained only 170 tasks and SPICE retained 312. Table-Zero obtained 1,000 tasks from 92,717 proposals with the untrained proposer and from 48,844, 41,232, and 41,418 proposals after DPO rounds 1–3, respectively. These differences reflect the methods’ ability to produce tasks that pass validation and difficulty filtering. The comparison therefore evaluates the resulting task pools under common solver optimization; it does not isolate per-task quality from pool size.

We also report the runtime breakdown below. All values are wall-clock hours on one eight-GPU H100 node. Slash-separated values correspond to iterations 1, 2, and 3; the frozen-proposer variants have a single generation and solver-training run. Each total is the sum of the listed stage runtimes.

MethodProposer trainingGenerationVerificationSolver trainingTotal
Dr. Zero, Qwen3-8B4.6 / 7.7 / 6.21.2 / 1.3 / 1.21.3 / 1.7 / 1.63.1 / 4.2 / 4.338.4
SPICE, Qwen3-8B6.3 / 11.7 / 10.32.1 / 2.3 / 2.21.1 / 0.8 / 0.83.1 / 3.6 / 3.347.6
Dr. Zero, Qwen3-14B7.8 / 9.0 / 8.92.2 / 2.3 / 2.21.0 / 2.1 / 2.04.7 / 4.5 / 4.451.1
SPICE, Qwen3-14B6.9 / 10.0 / 7.41.4 / 1.3 / 1.20.5 / 0.6 / 0.53.6 / 3.3 / 3.440.1
Table-Zero (w/o proposer training), Qwen3-8B–0.230.035.35.56
Table-Zero (w/o proposer training), Qwen3-14B–0.240.036.66.87
Table-Zero with DPO proposer, Qwen3-8B5.5 / 2.1 / 2.42.9 / 0.3 / 0.31.3 / 0.6 / 0.75.3 / 5.1 / 5.331.8
Table-Zero with DPO proposer, Qwen3-14B5.4 / 4.3 / 4.32.0 / 0.5 / 0.50.9 / 1.3 / 1.36.6 / 6.7 / 6.640.4

For Qwen3-8B, Table-Zero with DPO takes 31.8 hours, compared with 38.4 hours for Dr. Zero and 47.6 hours for SPICE, corresponding to reductions of 17.2% and 33.2%. For Qwen3-14B, Table-Zero with DPO takes 40.4 hours, which is 20.9% lower than Dr. Zero’s 51.1 hours and 0.7% higher than SPICE’s 40.1 hours. These are runtime comparisons of the evaluated pipelines, whose retained task-pool sizes differ.

Q2. Provenance of Figure 3, Figure 5(a), and Table 6

Section 4.3 correctly identifies the variant: Figure 3, Figure 5(a), and Table 6 concern Table-Zero without proposer training. Specifically, the s4 column in Figure 5(a) and the Qwen3-8B “without proposer training” row in Table 1 use the same step-75 solver checkpoint, which explains their identical scores. Across the five folds in Figure 5(a), s3 uses step 60 and the other folds use step 75, as described in Appendix D.1.

Figure 3(a–e) follows the frozen-proposer solver runs through training. Figure 3(f–g) compares their step-75 checkpoints with the released models under search and gold-source conditions, while Table 6 reports absolute EM under the gold-source condition. The DPO labels in Figure 5(a) and Table 6 are incorrect. We will correct them and the corresponding descriptions in Section 4.4 and Appendices C.2 and E.1.

Q3. Generated-task validation and audit coverage

Appendix D describes corrections to the curated MMQA questions and reference answers used for evaluation and Human-Annotated RL training. Table-Zero, Dr. Zero, and SPICE train on self-generated tasks, so these corrections do not modify their generated training questions or labels. The Appendix D audit therefore does not establish the quality of their self-generated training tasks.

To address this separately, we evaluated balanced 1,000-task pools containing 500 decomposability tasks and 500 joinability tasks for the untrained Qwen3-8B proposer and DPO rounds 1–3. We also conducted a broader audit of 8,717 candidates remaining after deterministic filtering and deduplication. These candidates came from the generation runs for the same four proposer stages and covered 329 source tables.

For the broader audit, we shuffled the questions and ran Claude Opus 5.5 twice with the same rubric and SQL access to the full tables. The audit assessed strict correctness and table necessity separately. Strict correctness required the reference SQL to answer the question as written and the stored label to contain the requested columns in the requested order, the correct rows, and the specified row order whenever applicable. For table necessity, the judge identified the smallest sufficient set of supplied fragments, counting a fragment only when answering required a column unique to it.

The two runs agreed on 98.5% of correctness judgments (Cohen’s κ = 0.97) and on 99.5% of judgments about whether at least two tables were required. A third run adjudicated 171 entries with a disagreement or missing verdict. Returning extra columns was the most common correctness error.

The results for the balanced pools covering both construction routes are:

UntrainedDPO round 1DPO round 2DPO round 3
Training tasks1,0001,0001,0001,000
Decomposability tasks500500500500
Joinability tasks500500500500
Strictly correct719 (71.9%)713 (71.3%)717 (71.7%)758 (75.8%)
Require ≥2 tables792 (79.2%)826 (82.6%)894 (89.4%)897 (89.7%)

Each percentage uses the corresponding 1,000-task pool as its denominator. Strict correctness and table necessity are assessed independently, so these rates do not indicate the proportion satisfying both criteria. From the untrained proposer to DPO round 3, strict correctness increases by 3.9 percentage points and the proportion requiring at least two tables increases by 10.5 percentage points.

To distinguish the two construction routes, we also report their results separately at the untrained and round-3 endpoints:

Construction routeCriterionUntrainedDPO round 3
DecomposabilityStrictly correct368 (73.6%)388 (77.6%)
DecomposabilityRequires ≥2 tables417 (83.4%)473 (94.6%)
JoinabilityStrictly correct351 (70.2%)370 (74.0%)
JoinabilityRequires ≥2 tables375 (75.0%)424 (84.8%)

Each route-specific percentage uses 500 tasks as its denominator. Both construction routes show higher strict correctness and multi-table necessity in round 3 than with the untrained proposer.

We also expanded the human evaluation to 200 tasks, with one expert assessing 50 tasks from each stage. Initial judgments marked 16, 25, 37, and 39 tasks as strictly correct and 43, 46, 47, and 48 as requiring at least two tables, respectively, for the untrained proposer and DPO rounds 1–3. We report these human judgments separately from the automated audit results.

Reviewer_5FvvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Table-Zero constructs multi-table question-answering tasks using existing join relationships or lossless decomposition of single tables. The proposer generates questions and reference SQL queries, with execution results serving as answer labels. The solver receives only the questions and answers them through retrieval and SQL execution. The two components are trained alternately for three rounds using DPO and GRPO, respectively. Experiments cover Qwen3-8B and Qwen3-14B across seven datasets, reporting average exact-match (EM) scores of 32.3% and 37.0%, respectively. These results outperform untrained models, adapted self-improvement baselines, and baselines trained on human-annotated data.

Critical strengths
  • The task construction is concrete and facilitates execution-based verification.
  • The evaluation covers multiple tasks, with careful attention to data isolation.
Critical weaknesses
  • The key comparisons are affected by multiple confounding factors. Appendices B.5 and C.1 explicitly state that Table-Zero uses online group filtering and difficulty weighting that are absent from Dr.Zero/SPICE. The number of rollouts, task pool sizes, and checkpoint selection procedures also differ. The proposer ablation similarly does not align the optimization objectives: GRPO rewards passing validation checks, whereas DPO additionally incorporates difficulty preferences. Consequently, the performance gaps in the main table cannot yet be attributed to the two task-construction strategies, and the differences in Figure 5 are insufficient to establish that DPO itself mitigates reward hacking. Although the authors disclose these limitations, direct experiments controlling for these factors are still missing.

  • The semantic quality of the generated tasks is insufficiently validated. Successful SQL execution does not guarantee that the query faithfully answers the natural-language question. Similarly, referencing two table fragments in SQL does not necessarily mean that the question requires both tables. The multi-table checks in Appendix B.3 rely primarily on column-name coverage. Appendix D audits the original MMQA question pool, while the only direct human evaluation of newly generated tasks consists of 15 questions assessed by a single expert. Moreover, the paper does not report the statistics on the "minimal sufficient table set" collected in the evaluation questionnaire. As a result, it remains difficult to assess the error rate of the generated labels and the proportion of tasks that genuinely require multiple tables.

Questions for authors
  • If the solver's optimization configuration, training budget, and checkpoint selection criteria are standardized, with only the task pool changed, how much of Table-Zero's advantage remains? On the proposer side, does DPO still outperform GRPO when GRPO is given the same quality and difficulty objectives?

  • Could the authors randomly sample generated tasks by construction strategy and training round, and report both the question–SQL consistency rate and the proportion of tasks that genuinely require at least two tables?

  • Section 4.3 states that the diagnostic analysis uses a variant with an "untrained proposer," yet Table 6 labels it as DPO. Furthermore, the s4 scores in Figure 5(a) match the untrained-proposer row in Table 1 exactly. Which variants and checkpoints do these results correspond to?

2 comments

Thank you for your helpful review. We agree that our original comparisons made it hard to see how much our newly generated tasks help on their own. To address this, we ran new experiments where all methods use the same training setup. We also checked 8,717 tasks generated by Table-Zero and employed a human expert to judge 200 of them.

Reply to Q1 and W1

Solver comparison

To make the comparison clearer, we ran new experiments. Together with the results in the paper, we now compare Qwen3-8B solvers trained with Dr. Zero, SPICE, Human-Annotated RL, and three Table-Zero variants. All methods start from the same MMQA source corpus and use the same training setup:

  • the same online group filtering and difficulty weighting;
  • 75 updates, each with 128 prompts and 8 rollouts per prompt;
  • the checkpoint with the highest score on the training tasks.

With this setting, mean exact match (EM) rises from 23.1 to 26.1 for Dr. Zero and from 23.0 to 24.7 for SPICE. Table-Zero still performs best.

MethodMean EM
SPICE24.7
Dr. Zero26.1
Table-Zero, frozen proposer30.2
Table-Zero, GRPO proposer31.1
Table-Zero, DPO proposer32.3

Even with a frozen proposer, Table-Zero reaches 30.2 EM, which is 4.1 points above Dr. Zero and 5.5 points above SPICE. Training the proposer helps more: it reaches 31.1 EM with GRPO and 32.3 with DPO.

Task-pool size

As mentioned above, we used the best checkpoint of each baseline and let it generate training tasks. Our goal was to collect 1,000 training tasks for each method. We allow at most 5 rounds of task generation by the proposer, where it writes 24,000 tasks each round.

Neither baseline reached the 1,000 goal.

MethodProposalsTraining tasksYield
Dr. Zero120,0001700.14%
SPICE120,0003120.26%
Table-Zero, frozen proposer92,7171,0001.08%
Table-Zero, DPO proposer41,4181,0002.41%

There are two reasons for this gap.

  1. More invalid tasks. The baselines write many more invalid tasks: 40.5% for Dr. Zero and 35.2% for SPICE, compared with at most 21.9% for the three Table-Zero variants.
  2. No learning signal. Almost all of their valid tasks have a pass rate of either 0% (the judge never solves them) or 100% (the judge always solves them). Under the setting in their papers, these tasks are not qualified to be included in their training set, giving no learning signal, so they are filtered out. Among valid, non-duplicate proposals, only 0.24% of Dr. Zero’s and 0.40% of SPICE’s became training tasks.

These results support the motivation in Section 1. When the proposer has to find related tables on its own, it struggles to write questions that need evidence from more than one table. Specifically, 85 of the 170 retained Dr. Zero tasks use only one table, and all 312 retained SPICE tasks use only one table. Table-Zero solves this by giving the proposer a set of connected tables before it writes the question. Table-Zero finds these tables either by following known join relationships or by splitting one table into pieces that can be joined back together (Section 3.1).

Proposer comparison

We trained a GRPO proposer with rewards for both task quality and difficulty (GRPO-QD). We compared it with DPO on the same held-out test set:

  • 239 unique prompts over 63 seed tables, where each prompt uses a different set of seed tables;
  • 20 proposals per prompt for each checkpoint (4,780 in total);
  • five attempts per proposal by the reference solver.

Note that the standard for a “usable task” here is much stricter than what a task needs to be used for training. On top of our existing checks, we add an LLM judge. It checks whether the question reads naturally and is rewritten enough to avoid leakage.

Usable tasks per 10,000 proposals, with matched training budgets:

RoundScored proposals (approx.)DPOGRPO-QD
141,0006756
261,0009082
382,00011196

In round 3, DPO produces 15 more usable tasks per 10,000 proposals than GRPO-QD.

Reply to Q2 (W2)

We expanded the evaluation of generated-task quality to cover both decomposability and joinability. For the untrained Qwen3-8B proposer and DPO rounds 1–3, we evaluated balanced 1,000-task pools containing 500 tasks from each construction route.

We also conducted a broader audit of decomposability candidates from the corresponding generation runs, which used prompts drawn from 416 training tables. Across these runs, 14,745 question–SQL pairs passed four deterministic checks:

  1. Question wording: we reject a pair if the question is not well formed or if it leaks the answer.
  2. SQL execution: we reject a pair if the reference SQL does not run.
  3. Unique answer: the question must have exactly one correct answer. We reject a pair if tied rows can change the result (for example, a top-3 query with a tie for third place), or if the SQL cuts the result with a LIMIT that the question does not ask for.
  4. Requested operation: each prompt asks the proposer for a specific operation, such as an aggregation or a comparison. We reject a pair if the SQL does not use that operation.

Within each pool, we removed entries with identical questions, identical SQL, or question-word overlap of at least 0.8. We audited all 8,717 remaining entries, which covered 329 source tables.

To make this audit fair, we shuffled the questions and ran Claude Opus 5.5 twice with the same rubric and SQL access to the full tables. Each run assessed strict correctness and table necessity separately. Strict correctness required the reference SQL to answer the question as written and the stored label to contain the requested columns in the requested order, the correct rows, and the specified row order whenever applicable. For table necessity, the judge identified the smallest sufficient set of supplied fragments, counting a fragment only when answering required a column unique to it.

The two runs agreed on 98.5% of correctness judgments (Cohen’s κ = 0.97) and on 99.5% of judgments about whether at least two tables were required. A third run adjudicated 171 entries with a disagreement on either criterion or a missing verdict. Returning extra columns was the most common correctness error.

The results for the balanced pools covering both construction routes are:

UntrainedDPO round 1DPO round 2DPO round 3
Training tasks1,0001,0001,0001,000
Decomposability tasks500500500500
Joinability tasks500500500500
Strictly correct719 (71.9%)713 (71.3%)717 (71.7%)758 (75.8%)
Require ≥2 tables792 (79.2%)826 (82.6%)894 (89.4%)897 (89.7%)

Each percentage uses the corresponding 1,000-task pool as its denominator. Strict correctness and table necessity are assessed independently, so these rates do not indicate the proportion satisfying both criteria.

Strict correctness ranges from 71.3% to 75.8%, reaching its highest level in DPO round 3. The proportion requiring at least two tables increases from 79.2% for the untrained proposer to 89.7% in round 3. The corresponding improvements are 3.9 and 10.5 percentage points.

In a separate human check, one expert assessed 50 tasks per stage, for 200 tasks in total. Initial judgments marked 16, 25, 37, and 39 tasks as strictly correct and 43, 46, 47, and 48 as requiring at least two tables, respectively, for the untrained proposer and DPO rounds 1–3. We report these human judgments separately from the automated audit results.

Reply to Q3

Which variants and checkpoints underlie Section 4.3, Table 6, and Figure 5(a)?

Section 4.3 correctly identifies the variant. Figure 3, Table 6, and Figure 5(a) all refer to Table Zero w/o proposer training.

Figure 5(a) and the Qwen3-8B “w/o proposer training” row in Table 1 use the same step-75 solver checkpoint. This is why their scores match exactly.

Figure 3(a–e) follows the frozen-proposer solver runs through training. Figure 3(f–g) compares their step-75 checkpoints with the released models under search and gold-source conditions. Table 6 reports absolute EM for the gold-source condition.

We will correct the DPO labels in Figure 5(a) and Table 6, together with the corresponding text in Section 4.4 and Appendices C.2 and E.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.