acceptodds
Under review as a conference paper at ICLR 2027

TabularMath: Diagnosing Computational Extrapolation in Tabular Foundation Models

Abstract

Can a tabular predictor learn a calculation from examples and return its answers when outputs exceed the training range? TabularMath tests this question using 100 GSM8K word problems and 14 AIME competition problems, compiled into reusable computations with 233,472 program-labeled rows. Nine tabular methods and GPT-OSS-120B predict randomly held-out rows or learn from lower-output rows and predict held-out upper-output rows. We measure both regression fit and answer accuracy after rounding to integers. At a 128-row budget under output shift, TabPFN v2.5 achieves median but only 7.8% mean rounded accuracy on GSM8K. The gap occurs within individual computations: 33 families have and accuracy below 20%. Increasing the budget to 2,048 rows improves random-split prediction but leaves shifted accuracy near 8%. GPT-OSS-120B attains 39.2% shifted accuracy at 128 rows, despite negative median from large errors elsewhere. These results make precise answers and controlled error a joint target for computational generalization. We release the executable families and evaluation records.

Primary area:
datasets and benchmarks
Keywords:
tabular learning, tabular foundation models, computational extrapolation, systematic generalization, numerical reasoning, program-verified synthesis, regression, exact prediction, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GgZWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TABULARMATH, a benchmark for evaluating computational extrapolation in tabular learning. It takes 100 GSM8K word problems and 14 AIME competition problems as seeds, and compiles them via program-verified synthesis into 114 executable task families; each family generates 2,048 program-labeled rows, totaling 233,472 rows. Evaluation uses two data splits: random split measures in-distribution interpolation, while upper-output-tail split measures upper-output-tail prediction; the evaluated models are nine tabular methods and GPT-OSS-120B. The nine tabular methods are evaluated under row budgets of 32–2,048; GPT-OSS-120B is evaluated only at 32/64/128 rows due to context-length constraints. The core finding is that regression fit and rounded accuracy can be severely decoupled: tabular methods have higher median shifted R2R^2, but low rounded accuracy under the shifted condition; GPT-OSS-120B has higher rounded accuracy under the shifted condition, but its median shifted R2R^2 is negative due to large errors; Increasing the number of rows mainly improves interpolation under the random split, with limited improvement in accuracy on the upper output tail.

The paper's contributions are the construction of executable task families, controlled row-budget and split comparisons, and a joint task-level analysis of regression fit and rounded accuracy.

Regarding validation, the paper uses an executable verifier to generate labels and sets up checks such as seed anchoring, valid generation, and deterministic execution; the authors claim that conditional label variance is zero for programs that pass the checks. Evidence supports that within the controlled setting of TABULARMATH, regression fit and rounded accuracy are indeed separated, and that at cap 128 on GSM8K under the upper-output-tail split, all methods lose accuracy. However, this upper-output-tail split constructs a joint shift, rather than pure output extrapolation; Based on the above evidence scope, conclusions should be limited to TABULARMATH's controlled executable tabular tasks and the evaluated models/configurations.

Critical strengths
  1. Program-verified synthesis improves internal validity and label reliability. The paper uses a generator to sample inputs and a verifier to compute and validate labels, incorporating checks such as seed anchoring, valid generation, and deterministic execution. On programs that pass the checks, labels are produced by a deterministic program rather than relying on human annotation or noisy observations. This makes prediction errors more attributable to the model itself rather than to data uncertainty.
  2. Joint evaluation of regression fit and rounded accuracy, with task-level analysis. The paper reports both median task-level R2R^2 and equal-task mean rounded accuracy, and analyzes their relationship on the same task. This reveals the risk that good regression fit does not imply correct answers, offering practical value for evaluating tabular learning.
Critical weaknesses
  1. Statistical uncertainty is not reported, so the robustness of model comparisons cannot be fully confirmed. At cap 128, which is used for full method comparison, each task has approximately 102 training rows and 26 queries; The resulting accuracy estimates may vary substantially. A single split seed also means that model rankings may hold only for the current sampling. The paper does not report variance across multiple runs, confidence intervals, or significance tests.
  2. The paper's framing and actual measurement do not fully match, making the conclusions difficult to generalize directly to computational extrapolation in general. The abstract emphasizes prediction when outputs exceed the training range, but the actual upper-output-tail split sorts by output yy, takes the lower 80% of outputs as the training set and the upper 20% as the query set; the authors explicitly acknowledge that this induces a joint input-output shift rather than pure output extrapolation. In addition, each family is a separate regression task, and evaluation does not train or test across families; different families have different computation rules. The benchmark therefore measures interpolation and upper-tail prediction on multiple independent computational tasks, rather than a unified computational extrapolation ability.
Questions for authors
  1. The paper does not specify the selection criteria for the 14 AIME problems and the 100 GSM8K problems. Can the authors explain how these problems were selected? Were they randomly sampled? Do they cover different difficulty levels and computation types?
  2. The paper adds generic features such as log-magnitude, sign, parity, and modular features to the input representation to expose numerical structure without supplying the computation. These are not variables directly provided by the original problems. Did the authors perform a feature engineering ablation, for example, retraining using only the primary numerical slots and removing all derived features, to show that the core separation phenomenon does not depend on a specific representation?
  3. The paper explicitly states that each family is a separate regression task. Do the authors claim that the average performance across the 114 independent families represents a unified computational extrapolation ability? If not, can the conclusions be explicitly limited to the average performance on multiple independent tasks, and can the authors discuss whether certain task types drive the overall results?
  4. The paper mentions at most 20 synthesis attempts, and the code shows that sampling tries at most 100 times and falls back to the base assignment upon failure. The paper does not report the pass rate of construction checks, the human modification rate, the sampling failure rate, or the fallback rate. Can the authors provide these statistics?
2 comments

Thank you for your review. Shared evidence is in the general response posted above. Here we answer your specific points.

The review finds that the evidence "supports that within the controlled setting of TabularMath, regression fit and rounded accuracy are indeed separated". The submission already states the joint shift that the second critical weakness discusses, and the revision adds the analyses both weaknesses request.

W1: statistical uncertainty. The revision reports it: task-bootstrap 95% intervals (20,000 resamples; Section 3, App. B) for GPT-OSS-120B vs. TabPFN v2.5 (30.3 points [19.1, 41.6], Section 4.1), for all ten shifted-minus-random contrasts, every one below zero (Table 6), and per model (Fig. 1, right). For the single split seed, it adds two checks (App. I). Query noise is small: 0.35 points of standard error on TabPFN v2.5's shifted mean (0.63 on the random split), against 2.0 (3.7) across families, and the bootstrap includes both. Rankings are stable: TabPFN v2.5 ranks first among the tabular methods under shift at all seven row budgets, and the median Kendall's tau between the rankings at different budgets is 0.74 under shift (0.76 random). The budgets are nested subsamples rather than independent seeds, but the differences the conclusions rest on are far wider than the query noise: PySR exceeds TabPFN v2.5 under shift by +68.3 points [59.9, 76.5] (Wilcoxon p = 2.9e-15).

W2: framing versus measurement. The framing matches the measurement, and the revised abstract now states the joint shift explicitly. The submission's abstract asks about prediction "when outputs exceed the training range", and 96.2% of shifted GSM8K queries have targets above every training target. The accompanying input shift is, as the review notes, one "the authors explicitly acknowledge": Section 3 states, "The measured regime is upper-output-tail prediction under this joint shift." The revision measures it: 82% of shifted queries have no input outside its training range. Nor does it put exact answers out of reach: PySR averages 76.5% on in-range queries and 69.8% on the others, and no closed-form or symbolic baseline loses more than 9 points out of range (Table 7, App. J).

On independent tasks (W2, Q3): the submission states that "each family is a separate regression task" (Section 2), and no unified ability is claimed. The revision adopts the scope the review proposes: the conclusions concern averages over 114 independent computations, for these tasks and models (Section 5, Scope). Table 4 shows what drives the averages: 60% of TabPFN v2.5's shifted mean comes from the 14 linear families, and it averages 1.2% on the 49 non-polynomial ones, while PySR solves every linear, multiplicative and rational family.

Q1. The 100 GSM8K seeds were drawn uniformly at random from the GSM8K test split (1,319 problems; indices 7–1293), and the 14 AIME seeds are the AIME 2024 problems whose families yield at least 2,048 distinct rows during construction (App. L). They span two difficulty levels, grade-school and competition mathematics, and many computation types: the GSM8K families comprise 14 linear, 11 multiplicative, 19 quadratic, 7 rational and 49 non-polynomial computations (Table 4), and the AIME families 11 non-polynomial and one each linear, multiplicative and rational (App. L).

Q2. The separation also appears without the derived features: models fit on the primary slots alone show it. Degree-3 polynomial regression on the slots reaches a median shifted R² of 0.99 with 33.2% accuracy (Table 3). The slots also hold the information for exact answers: using them alone, PySR solves 72 of 100 families. The derived features are generic per-slot transforms plus two divisibility summaries, and none equals the rounded target on more than 29% of rows in any table. We did not rerun the tabular methods without them (App. F).

Q4. Fallbacks cannot inflate the tables. A fallback reuses the seed's inputs and so reproduces the seed row, and no released table contains the seed row twice: 27 of the 233,472 rows repeat within a table (in 11 tables), and none of them is the seed row (App. L). This corrects the submission's statement that the rows are deduplicated. The other rates the review asks for (construction-check pass rate, human modification rate, sampling failure rate) cannot be reconstructed: the synthesis logs were not retained, and the only records of the original runs are the released tables, code and task-level metrics (App. L).

Thank you for the detailed rebuttal and additional experiments. Several of my concerns have been addressed, particularly regarding statistical uncertainty and the evaluation scope.

However, the robustness across independent random splits remains insufficiently established, and the incomplete feature ablation and missing synthesis statistics still raise concerns about the reliability and reproducibility of the benchmark.

While I acknowledge these improvements, I believe the remaining concerns are important. Therefore, I maintain my original rating of 2 (Weak Rejection).

Reviewer_twqyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper creates a benchmark TabularMath to evaluate Tabular models and GPT-OSS 120B. They create a template for 100 GSM8k problems and then for each problem they create different variants by substituting values to variables. They have examples of count ranging from 32 to 2048 in multiples of two. They repeat this procedure for AIME dataset. They then compute accuracies for the output prediction in two settings. In the first setting, they have a randomised split of the rows. And in the second setting, they sort the output values and then train on the lower 80% examples and test on the upper 20% examples (shifted). They compare between different Tabular models and also GPT-OSS 120B. They evaluate with regression fit R^2 and accuracy. They find that Tabular models such as TabPFN v2.5 achieves R^2 of 1.00 but accuracy of 39.5% on GSM8k benchmark for 128 rows for randomised split. When they evaluate on 2048 rows, the accuracy for random-split improves but on shifted experiment, the accuracy remains near 8%.

Critical strengths
  • The paper creates a benchmark for evaluating tabular models on task of learning a calculation from examples
  • They create two setups: one that evaluates in a randomised split and another that evaluates if models are able to make prediction for numbers higher than seen in training/prompt.
Critical weaknesses
  • Tabular foundations are generally evaluated for regression setups. In this paper, they are used to predict numbers which are then rounded. A high R^2 could be useful for tabular prediction tasks. And the rounded accuracy may not be so relevant. The current experimental setup is thus not very informative.
  • They should compare with symbolic regression baselines. They cite PySR and SRBench but do not evaluate them.
  • Findings on the gpt model are not very reliable because of the risk of contamination with gsm8k and AIME dataset. We do not know what the gpt model has been trained on. The authors should also provide the prompt for the gpt model.
  • In terms of novelty, GSM Symbolic [1] has shown earlier that grade school math problems can be converted to templates and then new variants can be created by substituting values. Also PAL[2] show that with larger numbers the model accuracies decline.
  • The paper writing is unclear at multiple places which make it difficult to read. For examples, lines 34, 99-101, 221-223
  • There are no examples for AIME. Also, it will be useful to see some examples where GPT model make large mistakes for a negative R^2
  • For 128 example evaluation, the test set of 20% is very small in number. So the accuracy values can have variance. There are no significance tests.

[1] Mirzadeh, Iman, et al. "Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models." International Conference on Learning Representations. Vol. 2025. 2025. [2] Gao, Luyu, et al. "Pal: Program-aided language models." International conference on machine learning. PMLR, 2023.

Questions for authors
4 comments

Thank you for your review. Shared evidence is in the general response posted above. Here we answer your specific points.

"There are no significance tests." The revision adds them: task-bootstrap 95% intervals (20,000 resamples; Section 3, App. B) for GPT-OSS-120B vs. TabPFN v2.5 (Section 4.1: 30.3 points [19.1, 41.6]) and for all ten shifted-minus-random contrasts (Table 6). Every interval excludes zero, which is a test of each difference at the 5% level. On the 26 queries per family: query sampling adds 0.35 points of standard error to TabPFN v2.5's shifted mean, against 2.0 points across families, and the bootstrap includes both (App. I). The revision adds Wilcoxon tests; for GPT-OSS-120B vs. TabPFN v2.5 under shift, p = 3.1e-5.

Rounded accuracy. Regression scores suit noisy targets, and R² is reported throughout. TabularMath asks a different question: has the learner recovered a computation whose answers are exact? R² cannot answer it, because it is scale-free. The review's own summary shows this: TabPFN v2.5 "achieves R^2 of 1.00 but accuracy of 39.5%". An R² of 0.99 still allows an error of a tenth of the target's standard deviation, more than half a unit once that deviation exceeds 5 (Section 3). In 99 of the 114 families every target is an integer (Section 3), which is also why GSM8K and AIME are scored by exact match. Under shift, TabPFN v2.5 keeps a median R² of 0.76 yet answers no query correctly in 70 of 100 families.

Symbolic regression. Run as requested, on the original subsamples and query rows: PySR answers every shifted query in 72 of 100 GSM8K families (76.1%), TabPFN v2.5 in 2 (7.8%) (Table 3). The answers are recoverable from the numbers alone; tabular foundation models miss them because of their inductive bias.

GSM-Symbolic and PAL. Both study language models that read a word problem. In TabularMath no learner reads a problem: program-generated variation only builds exact-label tables, and the question is whether tabular learners can infer the computation from rows. Neither paper evaluates a tabular learner. GSM-Symbolic is now cited (Section 6).

Contamination and the prompt. The risk applies only to the language-model reference: the GSM8K seeds come from the GSM8K test split, AIME 2024 is public, and the prompt (now in App. G) names the task. The paper's conclusions rest on the tabular methods and the numbers-only baselines, which never see the problem text; on GPT-OSS-120B's 87 families, the numbers-only PySR solves 68 and the language model 32.

Presentation and examples. Lines 34, 99–101 and 221–223 are rewritten. For example, line 34 now reads: "To reproduce that rule, a predictor must return the exact answer—here, the target rounded to the nearest integer—including for rows whose answers are larger than any seen in training." Seven repetitive paragraphs in Sections 3–5 are removed or condensed. App. M works through an AIME family (2024-I-1: problem, slots, computation, example rows, and every method's result). The original runs kept task-level metrics only, not the language model's responses, so App. H shows its large mistakes by family: the ten families with its lowest R². On gsm8k-000734, for example, its R² is −229 while TabPFN v2.5 answers 88.5% of the queries exactly.

Thanks for your response.

  • The original submission didn't contain details of significance test including the bootstrap sampling.
  • I maintain that the current experimental setup is not very informative.Tabular foundations are generally evaluated for regression setups. And, in such setups, R^2 is a valid metric. Exact match is not a natural fit for these models.
  • Examples are needed to understand what mistakes the model makes. GPT-OSS 120B is an open weights model, so you could rerun the inference and share some sample responses.

I retain my scores.

Thank you for the follow-up. We answer the three points for the record.

Significance tests. The revision reports them: task-bootstrap 95% intervals (20,000 resamples; Section 3, App. B) for GPT-OSS-120B vs. TabPFN v2.5 and for all ten shifted-minus-random contrasts, every one excluding zero (Section 4.1, Table 6), and Wilcoxon tests.

R² and exact match. We agree that R² is a valid metric, and the paper reports it alongside rounded accuracy throughout. The conclusion also holds under R² alone: on the same shifted queries, PySR, which sees only the numbers, has a median R² of 1.00, against 0.76 for TabPFN v2.5 (Table 3). Exact match adds what R² cannot show. A degree-3 polynomial reaches a median shifted R² of 0.99 yet averages 33.2% shifted accuracy, while PySR, at nearly the same R², solves 72 of 100 families. In 99 of the 114 families every target is an integer, and GSM8K and AIME score answers by exact match, so whether a predictor returns the answer is the question the benchmark asks. Reporting both metrics shows which methods track the trend and which recover the computation.

Examples of GPT-OSS-120B's mistakes. The original runs kept task-level metrics, not responses, so a new inference run would produce new responses that could not be tied to the reported scores. The revision gives the examples the records support: App. H lists the ten families with GPT-OSS-120B's lowest shifted R², and App. M works one AIME family through every method. GPT-OSS-120B is a reference in the revision, and the conclusions about tabular models do not depend on it: on its 87 families, the numbers-only PySR solves 68 and the language model 32.

Thanks for your comments.

Significance test- Thanks for adding them in the revision

R² and exact match- The conclusions have changed from what the original paper was pitching. I believe this needs a fresh review.

Examples of GPT-OSS-120B's mistakes- Examples are needed to understand what happens beyond the numbers in the tables. Rerunning them may not reproduce the exact numbers. That is also useful signal. But rerunning can get you examples and help in error analysis.

I retain my scores.

Reviewer_BdY2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper converts 100 GSM8K and 14 AIME problems into task families... an LLM writes a generator and a verifier program for each problem, and each family gives 2048 rows where inputs are the columns and the program output is the target. Paper tests 9 tabular methods and GPT-OSS-120B on a random split and on an output shift split (train on lower 80% of y, test on top 20%), and reports rounded accuracy and r2. The paper find TabPFN v2.5 gets median r2 0.76 but only 7.8% accuracy under shift at 128 rows, more rows help the random split but not the shifted one, and GPT-OSS-120B gets 39.2% shifted accuracy with negative r2. Paper also reports AIME results where all methods are low.

Critical strengths

1- Building tabular tasks from math programs is nice idea. Labels come from an executed program so there is no label noise, and new rows or new ranges can be sampled when needed. Normal tabular benchmarks cant do this, so it can be useful for testing tabular foundation models on rule like data.

Critical weaknesses

1- Many findings are already known in literature.. so its hard for me to find the new knowledge after reading the paper. The executable families are from VeRA (Cheng et al. 2026) and numerical range shift on GSM8K is already studied in GSM-Ranges (Shrestha et al. 2025). TabPFN extrapolating flat at about the last observed value is shown in Zhang et al. 2025 (arXiv 2505.20003). Whats new is mostly putting VeRA families into tables and scoring with rounding. Please clarify.

2- All 9 tabular methods are trees or smooth models, and since the shift split sorts by y almost every test target is above the training max.. so trees cant get these right by design, and the near 0% numbers in Table 3 dont tell much. There is no linear or polynomial regression, no symbolic regression (PySR is cited but not run) and no LLM that writes a program from the rows. Examples in the paper are simple polynomials, so these simple baselines would likely solve many families exactly. On the other side GPT-OSS-120B gets key=value rows with names like price, discount pct, tax pct, GSM8K test is very likely in its training data, and its a reasoning model so it can just guess the formula. So the 39.2% vs 7.8% comparison doesnt really measure extrapolation from examples.. this is misleading.

Questions for authors
  1. Add linear and polynomial regression (also on log features), PySR, and an LLM that writes a python function from the training rows and then runs it.. Does the gap to GPT-OSS-120B still hold?

  2. Can you split results by computation type (polynomial, division, conditional, loops/search)? I suspect the 33 families with R2 > 0.8 and accuracy < 20% are mostly the simple polynomial ones.

  3. And please clarify all the novelties/new findings compared to the papers above - a clear distinguish b/w what was already known and what your paper has new. I am right now finding it very hard to see what's the novelty/new findings here.

2 comments

Thank you for your review. Shared evidence is in the general response posted above. Here we answer your specific points.

W1/Q3: what is known and what is new. Known: language-model accuracy drops when the numbers in a word problem change (GSM-Ranges, GSM-Symbolic), and TabPFN v2 extrapolates flat (Zhang et al.; our TabPFN v2 results agree, median shifted R² −1.34). None of these works evaluates a tabular learner on an exact computation. New:

  1. The resource the review credits ("Normal tabular benchmarks cant do this"): 114 tasks whose labels an executed program computes exactly, and construction code that compiles further problems into new tasks.
  2. Fit and exact answers come apart within the same computation. TabPFN v2.5 does not extrapolate flat (median shifted R² 0.76, whereas predicting the training maximum gives −1.67), yet it answers no shifted query correctly in 70 of 100 families; 33 families combine R² > 0.8 with accuracy below 20%.
  3. Sixteen times more rows raise its random-split accuracy by 22.5 points but its shifted accuracy by only 0.4 [−3.4, 3.9].
  4. A numbers-only method answers every shifted query in 72 of these 100 families (below), so the failure is one of inductive bias.

The verdict "Whats new is mostly putting VeRA families into tables" rests on "The executable families are from VeRA." They are not: the 114 families were built for this benchmark (Section 2), with the generator–verifier construction that VeRA also uses. One sentence of the submission's Related Work, "VeRA supplies executable task families", allowed the other reading and is rewritten; VeRA is concurrent work that synthesizes reasoning data for language models (Section 6).

W2/Q1: the requested baselines confirm the diagnosis. We ran OLS (raw and with log features), log-linear regression, polynomial regression (degree 2, degree 3, and over reciprocals) and PySR on the primary numeric slots, with the original subsamples and query rows (Table 3, App. F). The review predicted that "these simple baselines would likely solve many families exactly." They do: PySR answers every shifted query in 72 of 100 GSM8K families (76.1% mean accuracy), TabPFN v2.5 in 2 (7.8%). About 100 rows of numbers suffice for exact answers, so a tabular foundation model that misses them on the same rows has an inductive-bias gap, the one the submission's Section 5 already pointed to ("program induction, arithmetic modules, and hybrid predictors"). The benchmark is not thereby solved: 23 non-polynomial families and the AIME families (at most 18.3% for every method) remain open.

The review notes that trees "cant get these right by design". Correct, and the revision labels them a structural floor (Section 3). The conclusion rests on TabPFN v2.5 and the neural models, which are not bound by the training maximum and still fail.

W2: the GPT-OSS-120B comparison. The revision adopts the review's point: GPT-OSS-120B is a reference, not a method under test. The paper states that its prompt contains the column names and the task identifier, that the GSM8K seeds come from the test split, and that inferring the computation from column meanings is part of what this reference measures (Section 3, App. G). The review's question, "Does the gap to GPT-OSS-120B still hold?", now has a numbers-only answer: on GPT-OSS-120B's 87 families, PySR solves 68 and the language model 32. LLM program induction would test another language model, not a tabular learner; since the conclusion about tabular models no longer depends on any language model, we did not add it.

Q2: "mostly the simple polynomial ones"? Two-thirds are simple closed forms, and that is the point. Of the 33 gap families, 22 are closed-form (10 quadratic, 4 linear, 7 multiplicative, 1 rational) and 11 are not (Section 4.2; each family is typed by the simplest closed-form model that reproduces it under 5-fold cross-validation). A short fit recovers such computations from 102 rows, yet TabPFN v2.5 misses them: it averages 33.5% on linear families that least squares answers exactly.

The linear, polynomial, log-feature and symbolic baselines the review requested have all been run, and each sharpens the paper's conclusion.

Reviewer_BdY2OpenReview ↗

Reply to Author's Rebuttal

Thanks authors for running all baselines. I accept the VeRA correction (the 114 families are new). This fixes my soundness concern to some extent so I raise soundness from 1 to 2. I am not going higher because PySR answers every shifted query in 72/100 GSM8K family so the main finding is that tabular models lack the bias for exact formula extrapolation and symbolic regression has it which is expected and already tested by SRBench and AI Feynman.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviews

All three reviews credit the benchmark's design: labels computed by an executed program (BdY2, GgZW), queries beyond the training range (BdY2, twqy), and the joint analysis of fit and answer precision (GgZW). None identifies an error in a reported measurement. The critical weaknesses fall into three kinds, each checkable against the submission and the revision.

  1. A statement the submission contradicts. One novelty verdict rests on "The executable families are from VeRA". They are not: the 114 families were built for this benchmark (Section 2), with the generator–verifier construction that VeRA, concurrent work, also uses (Section 6). One Related Work sentence of the submission, "VeRA supplies executable task families", allowed that reading and is rewritten (Section 6).
  2. Limitations the submission itself states. The joint input–output shift is stated in the submission's Section 3 ("The measured regime is upper-output-tail prediction under this joint shift"), and the independence of the families in Section 2. The revision measures both: 96.2% of shifted queries have targets above every training target, while 82% keep every input inside its training range, and out-of-range inputs cost the numbers-only baselines little (Section 3, Table 7); Table 4 shows which computation types drive the averages, and Section 5 states the scope.
  3. Requested analyses, now added, sharpen the conclusion. Two reviews ask for significance tests and confidence intervals. The revision reports task-bootstrap 95% intervals for eleven paired contrasts, every one excluding zero, and Wilcoxon tests (Sections 3 and 4.1, Table 6, App. B). With the numbers alone, PySR answers every shifted query in 72 of 100 GSM8K families, TabPFN v2.5 in 2 (mean accuracy 76.1% vs. 7.8%; +68.3 points [59.9, 76.5], Wilcoxon p = 2.9e-15). The information is in the table and tabular foundation models do not extract it: an inductive-bias gap, calling for the remedies the submission's Section 5 already named ("program induction, arithmetic modules, and hybrid predictors"). On GPT-OSS-120B's 87 families, PySR solves 68 and the language model 32, so the conclusion no longer depends on the language model. The benchmark is not thereby solved: 23 of the 49 non-polynomial GSM8K families resist every closed-form and symbolic method, and no method exceeds 18.3% shifted accuracy on AIME.

What is new. Prior work studies language models reading word problems (GSM-Symbolic, GSM-Ranges, PAL) and the flat extrapolation of TabPFN v2 (Zhang et al.). None tests whether a tabular learner recovers an exact computation. TabPFN v2.5 does not extrapolate flat (median shifted R² 0.76, whereas predicting the training maximum for every query gives −1.67), yet it answers no shifted query correctly in 70 of 100 families, and sixteen times more rows raise its shifted accuracy by only 0.4 points.

What changed:

ChangeLocation
Closed-form and symbolic baselinesSection 4.2, Table 3, App. F
Results by computation typeSection 4.2, Table 4
Split stated as upper-tail prediction under a joint shift; in-range share and accuracyAbstract, Section 3, Table 7, App. J
GPT-OSS-120B as a reference; its prompt; GSM8K test-split sourceSections 3 and 5, App. G
Uncertainty: task-bootstrap intervals for every contrast, Wilcoxon tests, noise decomposition, rank stabilitySections 3 and 4.1, Table 6, Fig. 1, App. B, I
Paired scaling contrastSection 4.3
Tolerance accuracyApp. K
GPT-OSS-120B results by family; worked AIME familyApp. H, App. M
Seed selection and construction records; 27 repeated rows (corrects "deduplicated rows")Section 2, App. L
xRFM accuracy recomputed: the released code averaged it over all prediction–target pairs (no conclusion changes)Every table and figure with xRFM accuracy; App. F, I
Related work (GSM-Symbolic, Zhang et al., VeRA)Section 6
Four behaviors observed; scope; open problemsSection 5
Passages flagged as unclear, rewrittenSections 1 and 3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.