- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TABLE-ZERO, a self-improvement framework for complex table reasoning using automatically generated training tasks. It exploits joinability and lossless table decomposition to organize task-generation contexts, while SQL execution provides reference answers. The framework alternates DPO training of a task proposer with GRPO training of a solver over three iterations. Experiments with Qwen3-8B and Qwen3-14B across seven datasets report higher average exact-match accuracy than prompting, adapted self-improvement baselines, and reinforcement learning on human-annotated tasks.
Critical strengths
-
The paper addresses a problem of practical importance: improving complex table reasoning without additional human task annotations. The proposed use of table structure to guide task generation is intuitive and well motivated.
-
The experiments demonstrate substantial improvements across multiple datasets. TABLE-ZERO achieves the highest average exact-match accuracy among the compared methods at both evaluated model sizes.
Critical weaknesses
-
The main experiments do not sufficiently disentangle the quality of the generated data from the effectiveness of the overall training framework. As stated in lines 925–928, TABLE-ZERO and the adapted baselines differ not only in task generation but also in solver optimization, including online filtering and difficulty weighting. Consequently, the performance gains cannot be clearly attributed to better training data. Moreover, Table 1 shows that TABLE-ZERO without proposer training already achieves substantial gains, while proposer training provides a comparatively smaller improvement. This suggests that the task-construction pipeline may account for much of the benefit, although its contribution is not isolated. I recommend comparisons under matched solver-training settings.
-
The human evaluation is limited to one expert assessing 15 generated tasks and 15 human-annotated tasks. This small sample provides limited evidence about the overall quality of the generated data. Evaluation by multiple annotators on a larger sample would make the findings more convincing.
-
Although DPO improves the usable-task yield, the absolute yield remains low: Figure 5 reports 111 usable tasks per 10,000 proposals after the third round, approximately 1.11%. This raises questions about the generation and filtering effort required to obtain a sufficiently large training pool. The paper would benefit from clarifying the practical implications of this low acceptance rate.
-
The paper does not clearly report downstream solver performance after each self-improvement iteration. Figure 5 shows improvements in proposer task yield across rounds, but these do not directly establish corresponding improvements in solver performance. I recommend reporting solver evaluation results using task pools generated by the proposer at each iteration, with comparable solver-training budgets.
-
The paper lacks a detailed comparison of computational efficiency across methods. Reporting wall-clock time for task generation, verification, proposer training, and solver training, together with the number of unique training tasks used, would help assess the practical cost of the reported gains. A common update budget alone does not establish comparable computational cost.
Questions for authors
-
How much of TABLE-ZERO’s improvement remains when the compared methods use the same solver optimization settings and comparable training budgets? Can the authors isolate the contribution of task construction from online filtering and difficulty weighting?
-
Can the authors provide a larger-scale evaluation of generated-task quality with multiple annotators, particularly regarding question–SQL consistency and whether answering genuinely requires multiple tables?
-
How does downstream solver performance change after each self-improvement iteration? Can the authors compare solvers trained on task pools generated at different proposer iterations under comparable training budgets?
-
Given the low absolute usable-task yield, what is the computational cost of obtaining the final training pools? Please report task counts and wall-clock time for generation, verification, proposer training, and solver training across methods.
-
Does TABLE-ZERO generalize to LLM families beyond Qwen? Have the authors evaluated a non-Qwen model to determine whether the observed improvements depend on the initialization’s SQL generation and tool-use capabilities?
2 comments
Q3. How does downstream solver performance change across self-improvement iterations?
We additionally report Qwen3-8B solver performance across three iterations. In the self-improvement procedure, each iteration first updates the proposer with DPO, then generates a task pool with the updated proposer, and finally trains the solver on that pool. The base model achieves 20.38% average EM across the seven evaluation datasets.
| Solver step within iteration | Iteration 1 | Iteration 2 | Iteration 3 |
|---|---|---|---|
| 15 | 21.88 | 30.29 | 31.47 |
| 30 | 25.85 | 30.49 | 31.33 |
| 45 | 28.73 | 30.80 | 31.54 |
| 60 | 30.07 | 31.16 | 31.64 |
| 75 | 30.19 | 31.22 | 32.30 |
All values are average EM (%). At the 75-update checkpoint within each iteration, performance increases from 30.19% in iteration 1 to 31.22% in iteration 2 and 32.30% in iteration 3, corresponding to gains of 1.03 and 1.08 percentage points.
Q4. What is the computational cost of obtaining the training pools?
The 111 usable tasks per 10,000 proposals in Figure 5 is a held-out proposer-evaluation metric with stricter acceptance criteria than those used to construct solver-training pools. In addition to the existing checks, this evaluation uses an LLM judge to assess naturalness and rewriting sufficient to avoid leakage. It should therefore be distinguished from the acceptance rate during training-pool construction.
For the actual Qwen3-8B solver pools, the proposal counts are:
| Proposer | Proposals | Retained training tasks | Acceptance rate |
|---|---|---|---|
| Untrained | 92,717 | 1,000 | 1.08% |
| DPO round 1 | 48,844 | 1,000 | 2.05% |
| DPO round 2 | 41,232 | 1,000 | 2.43% |
| DPO round 3 | 41,418 | 1,000 | 2.41% |
Round 3 therefore requires approximately 55% fewer proposals than the untrained proposer to obtain the same number of retained tasks. For comparison, under a generation budget of up to 120,000 proposals, Dr. Zero and SPICE retain only 170 and 312 tasks, respectively.
To quantify the cost beyond proposal counts, we report wall-clock runtimes for proposer training, generation, verification, and solver training. All runs use one eight-GPU H100 node. Values are in hours, and slash-separated entries correspond to iterations 1, 2, and 3. Each total is the sum of the listed stage runtimes.
| Method | Proposer training | Generation | Verification | Solver training | Total |
|---|---|---|---|---|---|
| Dr. Zero, Qwen3-8B | 4.6 / 7.7 / 6.2 | 1.2 / 1.3 / 1.2 | 1.3 / 1.7 / 1.6 | 3.1 / 4.2 / 4.3 | 38.4 |
| SPICE, Qwen3-8B | 6.3 / 11.7 / 10.3 | 2.1 / 2.3 / 2.2 | 1.1 / 0.8 / 0.8 | 3.1 / 3.6 / 3.3 | 47.6 |
| Dr. Zero, Qwen3-14B | 7.8 / 9.0 / 8.9 | 2.2 / 2.3 / 2.2 | 1.0 / 2.1 / 2.0 | 4.7 / 4.5 / 4.4 | 51.1 |
| SPICE, Qwen3-14B | 6.9 / 10.0 / 7.4 | 1.4 / 1.3 / 1.2 | 0.5 / 0.6 / 0.5 | 3.6 / 3.3 / 3.4 | 40.1 |
| Table-Zero (w/o proposer training), Qwen3-8B | – | 0.23 | 0.03 | 5.3 | 5.56 |
| Table-Zero (w/o proposer training), Qwen3-14B | – | 0.24 | 0.03 | 6.6 | 6.87 |
| Table-Zero with DPO proposer, Qwen3-8B | 5.5 / 2.1 / 2.4 | 2.9 / 0.3 / 0.3 | 1.3 / 0.6 / 0.7 | 5.3 / 5.1 / 5.3 | 31.8 |
| Table-Zero with DPO proposer, Qwen3-14B | 5.4 / 4.3 / 4.3 | 2.0 / 0.5 / 0.5 | 0.9 / 1.3 / 1.3 | 6.6 / 6.7 / 6.6 | 40.4 |
For Qwen3-8B, Table-Zero with DPO takes 31.8 hours across the three iterations, compared with 38.4 hours for Dr. Zero and 47.6 hours for SPICE. For Qwen3-14B, it takes 40.4 hours, compared with 51.1 hours for Dr. Zero and 40.1 hours for SPICE. Table-Zero therefore has lower total runtime than both baselines at 8B and than Dr. Zero at 14B, while its 14B runtime is close to SPICE’s.
For Table-Zero with DPO, generation and verification together account for 6.1 hours at 8B and 6.5 hours at 14B across the three iterations. Proposer and solver training account for the remaining 25.7 and 33.9 hours, respectively. Thus, although proposal acceptance remains low, training accounts for most of the measured runtime.
The frozen-proposer variants take 5.56 hours at 8B and 6.87 hours at 14B. These variants use a single generation and solver-training run, so their totals should be distinguished from those of the three-iteration methods. The reported timings describe the evaluated pipelines; they do not represent a comparison with equal numbers of retained training tasks.
Q5. Does Table-Zero generalize beyond Qwen?
Our current experiments evaluate Qwen3-8B and Qwen3-14B, covering two model sizes within the same family. We agree that these results do not establish generalization beyond Qwen. We will add experimental results for a non-Qwen model in the camera-ready version.
Thank you for your helpful review. We agree that the original experiments did not clearly separate the contribution of the generated task pools from differences in solver optimization. We have conducted additional controlled comparisons and expanded the task-quality audit.
Q1. How much improvement remains under matched solver-training settings?
We now compare Qwen3-8B solvers using the same online group filtering, difficulty weighting, and training budget: 75 updates, with 128 prompts and eight rollouts per prompt at each update. All methods use the same checkpoint-selection rule, selecting the checkpoint with the highest score on their training tasks.
Under this setup, average EM increases from 23.1 to 26.1 for Dr. Zero and from 23.0 to 24.7 for SPICE. Table-Zero with a frozen proposer achieves 30.2 EM, exceeding these baselines by 4.1 and 5.5 percentage points. Training the proposer raises average EM to 31.1 with GRPO and 32.3 with DPO.
These results show that the advantage persists after standardizing solver optimization. They also support your observation that much of the improvement is already present without proposer training, with DPO providing an additional 2.1 points. However, task-pool sizes remain different: Dr. Zero and SPICE retain 170 and 312 tasks within the generation budget, whereas Table-Zero reaches 1,000. The experiment therefore controls solver optimization but does not separate per-task quality from the amount of available training data.
Q2. Can the authors provide a larger evaluation of generated-task quality with multiple annotators?
We expanded the evaluation to cover both decomposability and joinability. For the untrained Qwen3-8B proposer and DPO rounds 1–3, we evaluated balanced 1,000-task pools containing 500 tasks from each construction route. We also conducted a broader audit of 8,717 deduplicated candidates from the corresponding generation runs, covering 329 source tables.
For the broader audit, we shuffled the questions and ran Claude Opus 5.5 twice with the same rubric and SQL access to the full tables. The judge assessed strict correctness and table necessity separately. Strict correctness required the reference SQL to answer the question as written and the stored label to contain the requested columns in the requested order, the correct rows, and the specified row order whenever applicable. For table necessity, the judge identified the smallest sufficient set of supplied fragments, counting a fragment only when answering required a column unique to it.
The two runs agreed on 98.5% of correctness judgments (Cohen’s κ = 0.97) and 99.5% of judgments about whether at least two tables were required. A third run adjudicated 171 entries with a disagreement or missing verdict. Returning extra columns was the most common correctness error.
The results for the balanced pools covering both construction routes are:
Each percentage uses the corresponding 1,000-task pool as its denominator. Strict correctness and table necessity are assessed independently, so these rates do not indicate the proportion satisfying both criteria. From the untrained proposer to DPO round 3, strict correctness increases by 3.9 percentage points and the proportion requiring at least two tables increases by 10.5 percentage points.
We also expanded the human evaluation to 200 tasks, with one expert assessing 50 tasks from each stage. Initial judgments classified 16, 25, 37, and 39 tasks as strictly correct and 43, 46, 47, and 48 as requiring at least two tables, respectively. This provides a larger human evaluation than the original study, although it does not include multiple human annotators. We report these human judgments separately from the automated audit results.