- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces TABULARMATH, a benchmark for evaluating computational extrapolation in tabular learning. It takes 100 GSM8K word problems and 14 AIME competition problems as seeds, and compiles them via program-verified synthesis into 114 executable task families; each family generates 2,048 program-labeled rows, totaling 233,472 rows. Evaluation uses two data splits: random split measures in-distribution interpolation, while upper-output-tail split measures upper-output-tail prediction; the evaluated models are nine tabular methods and GPT-OSS-120B. The nine tabular methods are evaluated under row budgets of 32–2,048; GPT-OSS-120B is evaluated only at 32/64/128 rows due to context-length constraints. The core finding is that regression fit and rounded accuracy can be severely decoupled: tabular methods have higher median shifted , but low rounded accuracy under the shifted condition; GPT-OSS-120B has higher rounded accuracy under the shifted condition, but its median shifted is negative due to large errors; Increasing the number of rows mainly improves interpolation under the random split, with limited improvement in accuracy on the upper output tail.
The paper's contributions are the construction of executable task families, controlled row-budget and split comparisons, and a joint task-level analysis of regression fit and rounded accuracy.
Regarding validation, the paper uses an executable verifier to generate labels and sets up checks such as seed anchoring, valid generation, and deterministic execution; the authors claim that conditional label variance is zero for programs that pass the checks. Evidence supports that within the controlled setting of TABULARMATH, regression fit and rounded accuracy are indeed separated, and that at cap 128 on GSM8K under the upper-output-tail split, all methods lose accuracy. However, this upper-output-tail split constructs a joint shift, rather than pure output extrapolation; Based on the above evidence scope, conclusions should be limited to TABULARMATH's controlled executable tabular tasks and the evaluated models/configurations.
Critical strengths
- Program-verified synthesis improves internal validity and label reliability. The paper uses a generator to sample inputs and a verifier to compute and validate labels, incorporating checks such as seed anchoring, valid generation, and deterministic execution. On programs that pass the checks, labels are produced by a deterministic program rather than relying on human annotation or noisy observations. This makes prediction errors more attributable to the model itself rather than to data uncertainty.
- Joint evaluation of regression fit and rounded accuracy, with task-level analysis. The paper reports both median task-level and equal-task mean rounded accuracy, and analyzes their relationship on the same task. This reveals the risk that good regression fit does not imply correct answers, offering practical value for evaluating tabular learning.
Critical weaknesses
- Statistical uncertainty is not reported, so the robustness of model comparisons cannot be fully confirmed. At cap 128, which is used for full method comparison, each task has approximately 102 training rows and 26 queries; The resulting accuracy estimates may vary substantially. A single split seed also means that model rankings may hold only for the current sampling. The paper does not report variance across multiple runs, confidence intervals, or significance tests.
- The paper's framing and actual measurement do not fully match, making the conclusions difficult to generalize directly to computational extrapolation in general. The abstract emphasizes prediction when outputs exceed the training range, but the actual upper-output-tail split sorts by output , takes the lower 80% of outputs as the training set and the upper 20% as the query set; the authors explicitly acknowledge that this induces a joint input-output shift rather than pure output extrapolation. In addition, each family is a separate regression task, and evaluation does not train or test across families; different families have different computation rules. The benchmark therefore measures interpolation and upper-tail prediction on multiple independent computational tasks, rather than a unified computational extrapolation ability.
Questions for authors
- The paper does not specify the selection criteria for the 14 AIME problems and the 100 GSM8K problems. Can the authors explain how these problems were selected? Were they randomly sampled? Do they cover different difficulty levels and computation types?
- The paper adds generic features such as log-magnitude, sign, parity, and modular features to the input representation to expose numerical structure without supplying the computation. These are not variables directly provided by the original problems. Did the authors perform a feature engineering ablation, for example, retraining using only the primary numerical slots and removing all derived features, to show that the core separation phenomenon does not depend on a specific representation?
- The paper explicitly states that each family is a separate regression task. Do the authors claim that the average performance across the 114 independent families represents a unified computational extrapolation ability? If not, can the conclusions be explicitly limited to the average performance on multiple independent tasks, and can the authors discuss whether certain task types drive the overall results?
- The paper mentions at most 20 synthesis attempts, and the code shows that sampling tries at most 100 times and falls back to the base assignment upon failure. The paper does not report the pass rate of construction checks, the human modification rate, the sampling failure rate, or the fallback rate. Can the authors provide these statistics?
2 comments
Thank you for the detailed rebuttal and additional experiments. Several of my concerns have been addressed, particularly regarding statistical uncertainty and the evaluation scope.
However, the robustness across independent random splits remains insufficiently established, and the incomplete feature ablation and missing synthesis statistics still raise concerns about the reliability and reproducibility of the benchmark.
While I acknowledge these improvements, I believe the remaining concerns are important. Therefore, I maintain my original rating of 2 (Weak Rejection).
Thank you for your review. Shared evidence is in the general response posted above. Here we answer your specific points.
The review finds that the evidence "supports that within the controlled setting of TabularMath, regression fit and rounded accuracy are indeed separated". The submission already states the joint shift that the second critical weakness discusses, and the revision adds the analyses both weaknesses request.
W1: statistical uncertainty. The revision reports it: task-bootstrap 95% intervals (20,000 resamples; Section 3, App. B) for GPT-OSS-120B vs. TabPFN v2.5 (30.3 points [19.1, 41.6], Section 4.1), for all ten shifted-minus-random contrasts, every one below zero (Table 6), and per model (Fig. 1, right). For the single split seed, it adds two checks (App. I). Query noise is small: 0.35 points of standard error on TabPFN v2.5's shifted mean (0.63 on the random split), against 2.0 (3.7) across families, and the bootstrap includes both. Rankings are stable: TabPFN v2.5 ranks first among the tabular methods under shift at all seven row budgets, and the median Kendall's tau between the rankings at different budgets is 0.74 under shift (0.76 random). The budgets are nested subsamples rather than independent seeds, but the differences the conclusions rest on are far wider than the query noise: PySR exceeds TabPFN v2.5 under shift by +68.3 points [59.9, 76.5] (Wilcoxon p = 2.9e-15).
W2: framing versus measurement. The framing matches the measurement, and the revised abstract now states the joint shift explicitly. The submission's abstract asks about prediction "when outputs exceed the training range", and 96.2% of shifted GSM8K queries have targets above every training target. The accompanying input shift is, as the review notes, one "the authors explicitly acknowledge": Section 3 states, "The measured regime is upper-output-tail prediction under this joint shift." The revision measures it: 82% of shifted queries have no input outside its training range. Nor does it put exact answers out of reach: PySR averages 76.5% on in-range queries and 69.8% on the others, and no closed-form or symbolic baseline loses more than 9 points out of range (Table 7, App. J).
On independent tasks (W2, Q3): the submission states that "each family is a separate regression task" (Section 2), and no unified ability is claimed. The revision adopts the scope the review proposes: the conclusions concern averages over 114 independent computations, for these tasks and models (Section 5, Scope). Table 4 shows what drives the averages: 60% of TabPFN v2.5's shifted mean comes from the 14 linear families, and it averages 1.2% on the 49 non-polynomial ones, while PySR solves every linear, multiplicative and rational family.
Q1. The 100 GSM8K seeds were drawn uniformly at random from the GSM8K test split (1,319 problems; indices 7–1293), and the 14 AIME seeds are the AIME 2024 problems whose families yield at least 2,048 distinct rows during construction (App. L). They span two difficulty levels, grade-school and competition mathematics, and many computation types: the GSM8K families comprise 14 linear, 11 multiplicative, 19 quadratic, 7 rational and 49 non-polynomial computations (Table 4), and the AIME families 11 non-polynomial and one each linear, multiplicative and rational (App. L).
Q2. The separation also appears without the derived features: models fit on the primary slots alone show it. Degree-3 polynomial regression on the slots reaches a median shifted R² of 0.99 with 33.2% accuracy (Table 3). The slots also hold the information for exact answers: using them alone, PySR solves 72 of 100 families. The derived features are generic per-slot transforms plus two divisibility summaries, and none equals the rounded target on more than 29% of rows in any table. We did not rerun the tabular methods without them (App. F).
Q4. Fallbacks cannot inflate the tables. A fallback reuses the seed's inputs and so reproduces the seed row, and no released table contains the seed row twice: 27 of the 233,472 rows repeat within a table (in 11 tables), and none of them is the seed row (App. L). This corrects the submission's statement that the rows are deduplicated. The other rates the review asks for (construction-check pass rate, human modification rate, sampling failure rate) cannot be reconstructed: the synthesis logs were not retained, and the only records of the original runs are the released tables, code and task-level metrics (App. L).