acceptodds
Under review as a conference paper at ICLR 2027

ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules

Abstract

Tabular foundation models such as TabPFN [Hollmann et al., 2025] and TabICL [Qu et al., 2026] already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics (RMSE, R2). This discards precisely the distributional information these models are designed to provide — a critical gap for high-stakes domains where not all kinds of errors are equally costly. We introduce ScoringBench, an open and extensible benchmark that evaluates tabular regression models under a comprehensive suite of proper scoring rules — including CRPS, interval score, energy score, and weighted CRPS — alongside standard point metrics. ScoringBench covers regression datasets from diverse domains, supports transparent community contributions via a git-based leaderboard, and provides two complementary ranking protocols: an ordinal Demšar/autorank approach and a magnitude-preserving z-score ranking approach. Evaluating several models — spanning in-context learners, fine-tuned foundation models, gradient-boosted trees, and MLPs — we find that model rankings shift substantially depending on the scoring rule: models that excel on point-estimate metrics can rank poorly on probabilistic ones, and the top-performing model under one proper scoring rule may rank noticeably lower under another. These results demonstrate that the choice of evaluation metric is not a technicality but a modelling decision — and, for applications where e.g. tail errors are disproportionately costly, a domain-specific requirement with direct consequences for model deployment.

Primary area:
datasets and benchmarks
Keywords:
tabular foundation models, probabilistic regression, proper scoring rules, benchmarking, distributional evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x1B5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides a benchmark for evaluating tabular foundation models. It criticizes existing benchmarks for focusing on point estimate metrics and instead proposes evaluating the predictive distributions using metrics such as CRPS. The paper evaluates models across 101 regression datasets and shows that model rankings can vary depending on the evaluation metric used.

Critical strengths

The paper provides a sound extension to existing benchmarks by incorporating distributional evaluations. The benchmark includes a comprehensive set of models evaluated across a relatively large set of datasets.

Critical weaknesses
  1. The paper has important results left in the appendix even though the current version is less than 8 pages. More of these should be moved to the main paper.
  2. The provided link, "hhttps://incredible-truffle-9624b2.netlify.app/", cannot be accessed because of the extra "h".
  3. The figures are not readable because the model names are long and difficult to recognize (e.g, finetune_tabpfn_realv2_5_is_90, crepes_xgb_difficulty_mondrian).
  4. The paper provides limited novelty, as extending evaluation beyond point estimates has already been explored and proposed in other papers. E.g: Izbicki, Rafael, and Pedro LC Rodrigues. "Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression."
  5. The result that different scoring rules can lead to different rankings is expected, and overall the paper does not seem to provide sufficiently novel or interesting findings for the ICLR community.
Questions for authors

Please see my comments in the Weaknesses section above. It would be great if the authors could comment on the novelty of this work compared with existing work on distributional evaluation for tabular foundation models.

1 comment

Concern 1 — Important results in the appendix

The revised main text makes best use of the 10 pages available for the rebuttal revision. It now reports paired win-rate comparisons (Section 5.1, Figure 2), dataset-level scoring-rule agreement (Section 5.2, Figure 3), and the β\beta-energy crossover analysis (Section 5.3, Figure 4). Unfortunately, some results had to still be put into the appendix due to the page limit.

Concern 2 — Broken leaderboard link

We corrected the extra “h” in the leaderboard URL. The revised manuscript uses https://incredible-truffle-9624b2.netlify.app/.

Concern 3 — Figure readability and model names

Figure 1 now places model labels in two vertically staggered columns and uses a larger font. It abbreviates the finetune_ prefix as ft_; the remaining model names are retained for consistency with the manuscript.

In Figure 3, we enlarged the colourbar tick labels and description and reduced the gaps between heatmaps. In Figure 2, we enlarged the model-pair and Holm-adjusted p-value annotations, arranged them on a single line, and moved them closer to the plots.

Concerns 4–5 and question — Novelty and empirical findings

We agree that evaluating tabular foundation models beyond point predictions is being investigated. Izbicki and Rodrigues (2026) evaluate CDE loss, log-likelihood, and CRPS; Landsgesell et al. (2026) include fine-tuning experiments and scoring-rule correlations. Both above works cite ScoringBench as concurrent work or use it as infrastructure. ScoringBench contributes an extensible benchmark across multiple scoring-rule families and examines how scoring-rule choice changes model competitiveness and dataset-specific model selection (Sections 5.1–5.3).

The revised manuscript adds direct evidence through paired win-rate comparisons across scoring rules, using exact McNemar tests with joint Holm correction (Section 5.1, Figure 2). Selected comparisons appear in the main text, and further results are provided in Appendix D.4. Section 5.3 and Figure 4 present complementary dataset-specific crossover analyses. The revised discussion distinguishes these effects from the generally high agreement in pooled rankings which is motivated by a decomposition of the covariances (occuring in the Pearson correlation) in the appendix.

The benchmark's value to the community is also reflected in its use by independent teams to evaluate new tabular foundation models. NVIDIA reports Kumo Tabular's ScoringBench results alongside TabArena, BeyondArena, and TALENT; the TabPFN-3.5, EXAONE Tabular 1.0, and Causilo technical reports also use ScoringBench:

This uptake supports its practical value as a community evaluation resource alongside the new empirical analyses.

Reviewer_wpCGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ScoringBench, a benchmark for evaluating probabilistic predictions from tabular foundation models using proper scoring rules. It covers 101 regression datasets and evaluates a range of tabular foundation models and conventional baselines using CRPS, energy scores, interval scores, weighted CRPS, and other metrics. The benchmark provides two ranking approaches, based on Autorank and per-dataset standardized scores, together with an extensible codebase and a continuously updated leaderboard.

The main empirical findings are that model rankings depend on the scoring rule, fine-tuning with different objectives can affect predictive performance, and tabular foundation models generally outperform conventional baselines on probabilistic metrics.

Critical strengths
  1. Broad and useful benchmark coverage.

The benchmark covers 101 regression datasets, a wide range of probabilistic evaluation metrics, and several recent tabular foundation models. This provides a useful reference for comparing predictive distributions rather than relying exclusively on point prediction metrics.

  1. Reproducible and extensible evaluation framework.

The publicly available implementation and continuously updated leaderboard are valuable. The benchmark also provides a consistent evaluation protocol and two complementary ranking methods. These features make it potentially useful for evaluating future tabular foundation models.

Critical weaknesses
  1. Limited novelty relative to closely related work.

The paper's main findings substantially overlap with existing studies. Landsgesell et al. (2026) already investigated how proper scoring rules affect model rankings and how alternative fine-tuning objectives influence probabilistic predictions from tabular foundation models. Izbicki and Rodrigues (2026) also conducted a systematic benchmark of tabular foundation models for conditional density estimation using multiple probabilistic metrics.

ScoringBench expands the number of datasets, models, and scoring rules and provides a living leaderboard. These are useful contributions. However, it is not sufficiently clear what new empirical insights are obtained from this expanded evaluation beyond those already established in the literature.

For a benchmark paper, broader coverage is valuable, but the paper should explain more concretely what the additional evaluations reveal that previous benchmarks could not.

  1. Insufficient evidence for the claimed importance of model ranking differences.

A central claim is that model rankings change substantially depending on the scoring rule. However, the reported results do not clearly establish how consequential these changes are among the strongest models.

For example, Tables 3, 4, and 6 contain the same four leading models under CRPS, the 90% interval score, and RMSE, although their ordering differs. This does not rule out important ranking changes under other metrics, but the paper should identify and analyze those cases more explicitly.

The interpretation of fine-tuning results also requires caution. Under CRPS, the CRLS-fine-tuned realTabPFNv2.5 has a mean rank of 11.08, compared with 18.21 for the original model. However, this difference does not exceed the critical difference shown by the reported Nemenyi analysis. Therefore, the observed rank improvement should not be presented as evidence of a statistically significant improvement.

These issues matter because scoring-rule-dependent model selection is the paper's central empirical motivation. The current presentation makes it difficult to distinguish practically meaningful differences from relatively small ranking changes.

Questions for authors
  1. What new empirical findings does ScoringBench establish beyond Landsgesell et al. (2026) and Izbicki and Rodrigues (2026)?

I understand the contribution of the larger benchmark and living leaderboard. However, I would appreciate a more concrete explanation of which previously unavailable comparisons or conclusions justify the expanded evaluation.

  1. Can the authors identify cases where changing the proper scoring rule leads to a substantial and statistically supported reversal between competitive models?

The leading models remain relatively stable across several reported metrics. A focused analysis of the most consequential ranking reversals would help establish the importance of evaluating multiple scoring rules.

  1. How should the comparison between foundation models and conventional baselines be interpreted given the 2,400-sample training cap?

The paper acknowledges that this setting may favor in-context learners. Does the sample-size ablation support the broader conclusion that foundation models outperform conventional baselines, or should that conclusion be restricted to the small-sample regime?

1 comment

Thank you for your feedback which helped us distinguish pooled ranking agreement from dataset-specific scoring-rule effects and state the limits of our comparisons. All references below are to the revised manuscript.

Q1 and novelty concern — Findings beyond prior distributional evaluations

We expanded Related Work to distinguish ScoringBench from Oreshkin et al. (2026), Izbicki and Rodrigues (2026), and Landsgesell et al. (2026). ScoringBench contributes an extensible benchmark and living leaderboard across multiple scoring-rule families, together with a broad, dataset-conditional comparison of the rankings induced by these rules.

ScoringBench includes cumulative distribution function (CDF)-based rules that admit predictive distributions with atoms as well as density-based rules that require a density. The revised paper adds dataset-level scoring-rule agreement alongside pooled agreement (Section 5.2, Figure 3) and model winner crossovers (Section 5.3, Figure 4). Appendix G provides the full per-dataset agreement matrices. These analyses examine variation that pooled leaderboard comparisons can conceal.

Q2 and ranking-sensitivity concern — Statistical support among competitive models

We agree that relatively stable pooled rankings do not establish large or practically consequential differences among leading models. Patton (2020) supplies a general theoretical motivation for examining scoring-rule sensitivity in forecast comparisons; it does not study tabular foundation models. Our contribution is a broader evaluation that explicitly examines dataset-level variation alongside pooled comparisons, without claiming that every ranking difference is practically consequential.

Section 5.1 and Figure 2 examine scoring-rule effects using paired dataset win-rate comparisons and exact McNemar tests with joint Holm correction. The main text presents selected examples; Appendix D.4 provides more examples, adjusted p-values, and analysis details. Section 5.3 and Figure 4 provide complementary descriptive examples of dataset-specific winner crossovers.

Fine-tuning concern — Comparing each variant with its own base model

We agree that an observed mean-rank improvement should not itself be presented as statistically significant when it does not exceed the Nemenyi critical difference although there is more powerful post-hoc tests than the Nemenyi test. We added paired per-dataset skill scores against each variant's own base model (Appendix E.3, Figure 12), together with exact paired sign tests of equal improvement and deterioration probabilities. All 66 variant–metric tests are jointly Holm-corrected.

Q3 — Scope of the 2,400-observation training cap

We agree that this cap limits comparisons with conventional baselines that may benefit from more data. Our results apply to the capped-data regime and do not establish relative performance at larger training sizes.

The sample-size ablation (Appendix E.1, Figure 11) compares total-dataset caps of 1,000, 2,000, and 8,000 observations with the default 3,000. Pearson correlations on Autorank positions remain high across these settings (median 0.99).

Reviewer_tJHHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ScoringBench, an open benchmark for tabular regression. Authors state that tabular foundation models emit full predictive distributions, while most tabular benchmarks score them almost exclusively with point metrics, like RMSE or R2R^2, discarding distributional information. This work proposes an evaluation harness that scores models under a suite of distributional metrics: CRPS, the β-energy score, CRLS, density power divergence, the interval score, quantile-weighted CRPS, the CDE/continuous-Brier loss, and pseudo-spherical scores, alongside point metrics and calibration diagnostics. 44 metrics overall. Authors score the models with a collection of 101 regression publicly available datasets, with a lightweight preprocessing. Authors also propose a ranking protocol pairing the Demsar/autorank procedure with a magnitude-preserving per-dataset z-score ranking across models. Finally authors share a repository with leaderboard that commits per-run parquet files, with a documented contribution workflow. Results show that model rankings depend on the chosen scoring rule, several fine-tuned realTabPFN variants rank ahead of the base models under CRPS. Pretrained in-context models clearly outperform gradient-boosted baselines on probabilistic metrics. CREPES-coupled models attain near-nominal coverage while the interval score and other proper scores degrade.

Critical strengths

The benchmark with code and leaderboard is a real contribution that outlives the paper's findings, and its construction shows statistical care where benchmarks usually do not. This approach brings significant value to distributional tabular evaluation. It scores four distinct native output types under a unified, comparable set of proper scoring rules, supported by analytic evaluation. As a result, the evaluation suite captures genuinely complementary sensitivities.

The protocol decisions are defensible and mostly explained: fold-averaging before ranking with a pseudoreplication justification, ordinal and magnitude-aware protocols side by side, four-level deduplication with explicit alias keys, target-cardinality filtering to keep genuine regression tasks, and a leaderboard that commits per-run artefacts, giving model submissions an auditable public snapshot. A benchmark's value is largely a public-good function of usability, coverage and auditability, and on those axes the submission stands on its own even where its empirical conclusions are thin.

The coverage-versus-proper-score contrast is quantified and sharpens cited prior work. CREPES coupling buys near-nominal coverage while proper scores degrade severely: crepes_catboost_difficulty ranks 13.4 on coverage deviation but 40.9 on CRPS, crepes_xgb_difficulty 14.2 versus 42.9. Wrapping TabICLv2 moves CRPS mean rank from 14.12 to 31.96 and 90% interval-score rank from 12.58 to 33.74. Notably the coverage-deviation leaderboard is not collinear with the rule leaderboards. Its leaders are mostly conformal-wrapped models sitting deep in the bottom tier on proper scores (the one exception among the top four, finetune_tabiclv2, is not conformal-wrapped and remains top-5 on CRPS). This is the practically important warning for deploying a TFM under asymmetric error costs. It also sharpens the already-cited Dheur & Ben Taieb (2023) in a specific and interesting way: here the base model was itself CRPS-trained, so post-hoc coverage optimisation damages the very objective the model was fit to.

Critical weaknesses

Section 3.2 presents the 101-dataset multi-repository curation as a design contribution while Oreshkin, Tavker & Efimov's TabRegSet-101 has the same dataset count, the same multi-repository sourcing pattern: UCI, Kaggle, PMLB, OpenML, KEEL, and CRPS-over-200-quantiles plus 95% coverage on overlapping model families.

The evidence presented in the paper does not test its core claims regarding proper-rule sensitivity and fine-tuning:

  1. Proper rules are stable, not shifting. The abstract claims rankings shift noticeably across proper scoring rules, yet no cross-rule rank-correlation matrix is provided. In the printed leaderboards (Tables 3, 4, 6), the top three models are identical, and the maximum shift between two proper rules CRPS vs. IS-90 is only 4.32 ranks. The largest shifts 10.54 for crepes_tabiclv2 and 10.23 for forest_diffusion_flow occur between point and proper metrics. The printed data therefore shows the large reversals occurring only across the point/proper and calibration boundaries, a distinction the paper fails to draw though the unprinted rule leaderboards (wCRPS, energy scores, DPD, pseudo-spherical) leave this open.
  2. Fine-tuning is tested against the wrong baseline. Because the authors use autorank's default setting, Akinshin’s γ\gamma is computed against the top overall model rather than the fine-tuned variants own base model. While one rule-matched signal exists (finetune_tabpfn_realv2_5_is_90 improves by 4.2 ranks from CRPS to IS-90), it still finishes 13th of 21 variants behind rule-mismatched rivals, and lacks a paired per-dataset test to rule out noise from the single fixed-seed CV configuration. Authors report 5 folds, n_repeats_cv = 1.
  3. Consequently, the paper’s headline claims and practical recommendation (choosing a scoring rule to match deployment costs) rest on irrelevant metrics rather than verified cross-rule disagreement.
Questions for authors

All five could change my assessment. Q1 and Q3 are the ones I would weight most.

Q1 Please test your core claim directly by reporting the cross-scoring-rule rank-correlation matrix (Spearman and Pearson) across the 51 models, restricted to the proper scoring rules. For the most divergent rule pairs, identify the specific models whose relative rankings reverse, supported by per-dataset paired tests and confidence intervals. Additionally, please provide the promised per-metric table of propriety classes and admissible output types (CRLS class is currently omitted in A.1).

Q2 Please clarify the exact scoring implementation and run a stratified sensitivity check:

(a) bin count, support and edge placement for the equal-width histogram, and the treatment of tied/duplicate quantiles and zero-width bins.

(b) reconcile §3.4 with A.1 — for quantile-native models (TabICLv2, ExaOneTabular, Nori) is CRPS/ESβ\text{ES}_\beta computed from native quantiles or from the monotone-interpolated histogram?

(c) Report bin-count sensitivity for DPD, pseudo-spherical, CDE, and PIT-KS, stratified by native output type (native histogram vs. interpolated quantiles/samples), since representation tracks model family.

(d) define the pseudo-spherical score, provide the exact formula, admissible class, and α\alpha convention for α∈{1.5,2,3}\alpha \in \{1.5, 2, 3\}, and properly attribute it. Cite Good 1971 and Gneiting & Raftery 2007.

(e) Appendix A.2 assigns a literal 0.5 on atoms (citing Czado et al. 2009). However, Czado et al. prescribe randomized PIT on [F(y−),F(y)][F(y^-), F(y)] or its non-randomized midpoint (F(y−)+F(y))/2(F(y^-) + F(y))/2, which equals 0.50.5 only for degenerate point-mass forecasts. Was literal 0.5 implemented by mistake, or the midpoint?

(f) Could authors point to the exact equation in Gneiting & Ranjan (2011) where the piecewise weights w(α)∈{(1−α)2,α(1−α),α2}w(\alpha) \in \{(1-\alpha)^2, \alpha(1-\alpha), \alpha^2\} appear.

(g) Appendix A.2 attributes sharpness to Tran et al. (2020), and the dispersion definition continues "using the notation above" without its own citation, yet the dispersion metric of Tran et al. is the coefficient of variation (cross-sample SD of predicted sigmas divided by their mean), whereas the paper's Dispersion is the un-normalized raw cross-sample standard deviation. Please confirm whether using raw cross-sample standard deviation instead of the coefficient of variation, and carrying the Tran et al. attribution over to dispersion, was intentional.

Q3 Please document the training setup and statistical evaluation for the ~20 realTabPFNv2.5 fine-tuned variants. State the fine-tuning data source, step count and schedule. Explicitly confirm whether any of the 101 benchmark datasets (or their CV folds) overlap with the fine-tuning or pre-training corpora of the evaluated foundation models.

Q4 Please clarify the following experimental design choices: (a) Per-model tuning budgets (defaults vs. HPO) and whether the top-10 rankings hold for n_repeats_cv greater than 1.

(b) Ranking direction (higher vs. lower is better) for sharpness and dispersion, and confirm whether train time is excluded from aggregate rankings.

(c) Table 1 includes both a base dataset and its synthetic bootstrap resampling: 344 mv together with BNG(mv). Note that the substring rule as specified cannot catch these: numeric prefixes are stripped in normalization, and the substring check is only applied to names longer than three characters. Please clarify whether synthetic resamplings of base datasets already in the suite are treated as distinct tasks and quantify their impact on mean ranks.

Q5 Please cite and differentiate your benchmark from TabRegSet-101 (Oreshkin et al., TMLR/ICML 2026).

3 comments
AuthorsOpenReview ↗

Response Part 1 of 3: Main concerns and Q1

Thank you for your careful review and for asking us to test scoring-rule agreement directly and compare fine-tuned variants with their own base models. Your questions also helped us clarify how the scores are computed. All section, figure, and table numbers below refer to the revised manuscript.

Main concerns — Ranking sensitivity and the fine-tuning baseline

The revised discussion distinguishes aggregate agreement from dataset-specific variation, with new analyses of scoring-rule correlations and model win rates (Sections 5.1–5.3). Patton (2020) develops the general forecasting argument that comparisons of misspecified forecasts can depend on the scoring rule; it is not a study of tabular foundation models. We examine this issue across a broader set of models, datasets, and rules without assuming that ranking differences must be large.

Autorank's default effect-size baseline does not directly assess fine-tuning against each variant's own base model. We therefore added paired per-dataset skill scores, win rates, and two-sided exact sign tests of equal improvement and deterioration probabilities (Appendix E.3, Figure 12). These tests concern win frequency rather than improvement magnitude and require no symmetry assumption (unlike the Wilcoxon test). Holm's correction covers all 66 variant–metric tests jointly. Running ScoringBench with n_repeats_cv=1 means that we run one-time repeated five-fold crossvalidation per model and dataset (i.e. we did not run repeated five-fold cross-validation per model and dataset). This was a development option which we will remove in the future.

Q1 — Cross-scoring-rule agreement and specific model comparisons

Figure 3 reports Pearson correlations on pooled Autorank mean ranks (mean off-diagonal correlation 0.968). For the benchmark average and the least-agreeing continuous-target dataset, we compute Pearson correlations on within-fold model ranks (equivalently, Spearman correlations on raw scores). Appendix G provides the per-dataset matrices. These figures exclude fine-tuned variants.

We measure agreement between two scoring rules by correlating their model rankings. First, for each dataset and CV fold, we rank models separately under each rule and calculate the Pearson correlation for every rule pair. For a dataset-level result, we average each pair's correlation across the five folds. For the benchmark-wide result, we first average each pair's correlation across datasets within each fold (using datasets where that pair is defined), then average across folds. The pooled Autorank result is calculated differently: Autorank aggregates each model's scores across folds to produce a mean rank for each rule, and we correlate those per-rule mean-rank vectors across models.

The scripts also calculate cell-wise standard errors across folds. The median cell-wise standard error is 0.0014 for the benchmark-average matrix and 0.0207 for the least-agreeing continuous-target dataset; their mean off-diagonal correlations are 0.901 and 0.805, respectively (see also subfigure titles). These standard errors are small relative to the observed departure from perfect agreement (correlation 1), supporting that the disagreement is not simply fold-to-fold variation.

Section 5.1 and Figure 2 present paired dataset win-rate comparisons across scoring rules, using exact McNemar tests with joint Holm correction and retaining native forecasts for CDF-based rules. The main text shows a compact selection; Appendix D.4 provides the full selected comparisons, adjusted p-values, and analysis details.

Section 5.3 and Figure 4 show how dataset-level winner changes can coexist with comparatively stable pooled summaries. They provide illustrative model and dataset examples alongside the broader crossover analysis.

Q1 — Propriety classes and admissible output types

We added Table 1 in Appendix A.1, listing propriety classes and required forecast information. For CRLS, we cite Juutilainen et al. (2012) and quote the regularity and integrability qualification from Gneiting and Raftery (2007).

AuthorsOpenReview ↗

Part 2 of 3: Q2

Q2(a) — Histogram construction, ties, and zero-width bins

The density view uses B=200B=200 equal-width bins over [min⁡(t−,q1),max⁡(t+,qK)][\min(t_-,q_1),\max(t_+,q_K)], where [t−,t+][t_-,t_+] is the training-target range and q1,qKq_1,q_K are the forecast's outermost nodes. Test outcomes do not determine the support. Native histogram outputs are not regridded.

In the native view, tied quantiles remain atoms and duplicate probability levels contribute zero mass. In the density view, quantile and sample forecasts are interpolated onto positive-width bins. Zero-width bins in native histograms are assigned zero density, and the remaining density mass is renormalized. Figure 14 in Appendix H illustrates the two views and their mappings from each output type.

Q2(b) — Native quantiles versus the interpolated density view

For quantile-native models, CRPS and the β\beta-energy scores are computed from native quantile forecasts. CDF-based rules use the native forecast, which may contain atoms. Density-based rules require a density, so quantile and sample forecasts use the interpolated density view; native histogram or density outputs retain their own representation. Appendix A.1 lists the admissible classes and required forecast information.

Datasets with discrete or ordinal targets are flagged and excluded from density-based scores and PIT-KS, but remain eligible for CDF-based scores.

You may find figure 14 helpful as an overview about how we deal with different model outputs (quantile models, native grid models or models which emit predictive samples).

Q2(c) — Bin-count sensitivity by output type

We corrected the quantile density view: unreported lower- and upper-tail masses are spread uniformly from the outermost quantiles to the support boundaries, without renormalizing the interior.

The added synthetic sensitivity study distinguishes native histograms, quantile forecasts, and sample forecasts. Native histograms are invariant to BB. At B=200B=200, 99- and 999-level quantile forecasts have less than 0.5% density-score excess. Sample forecasts retain finite-draw Monte Carlo error (about 3–15% for the tested draw counts), which changing the grid does not eliminate. PIT-KS uses the native view and is invariant to BB. Appendix H describes the study and regression test; Figure 15 reports the sensitivity results.

Q2(d) — Pseudo-spherical score

Appendix A.1 now gives the exact formula and cites Good (1971) and Gneiting and Raftery (2007). We use α>1\alpha>1 and require a predictive density f∈Lαf\in L^\alpha, so its normalizing norm is finite. The appendix specifies the loss-oriented affine transformation used in ScoringBench.

Q2(e) — PIT at atoms

The previous implementation assigned the midpoint (F^n(yn−)+F^n(yn))/2(\widehat F_n(y_n^-)+\widehat F_n(y_n))/2 at atoms, rather than a literal 0.5. We now implement the randomized PIT described by Czado et al. (2009), both in the code and in Appendix A.2:

un=F^n(yn−)+Vn(F^n(yn)−F^n(yn−)),Vn∼Uniform(0,1).u_n=\widehat F_n(y_n^-)+V_n\bigl(\widehat F_n(y_n)-\widehat F_n(y_n^-)\bigr),\qquad V_n\sim\mathrm{Uniform}(0,1).

For continuous forecasts this equals F^n(yn)\widehat F_n(y_n). Datasets with discrete or ordinal targets are excluded from PIT and density-based scoring-rule evaluations.

Q2(f) — Weighted-CRPS attribution

The general weighted-CRPS integral is given in Appendix A.1. The left-, center-, and right-weight functions are (1−α)2(1-\alpha)^2, α(1−α)\alpha(1-\alpha), and α2\alpha^2, respectively. These are the functions listed in Table 2 of Gneiting and Ranjan (2011); we do not introduce new weighting functions.

Q2(g) — Sharpness and dispersion definitions

We revised both diagnostics to match Tran et al. (2020). Sharpness is the root mean predictive variance, sha=N−1∑nvar(Fn)\mathrm{sha}=\sqrt{N^{-1}\sum_n\mathrm{var}(F_n)} (their Eq. (3)). Dispersion is the coefficient of variation of predicted standard deviations, Cv=SDN−1(σn)/μσC_v=\mathrm{SD}_{N-1}(\sigma_n)/\mu_\sigma (their Eq. (4)). Appendix A.2 clarifies that dispersion is diagnostic rather than a proper score and that its interpretation depends on whether prediction errors are heteroscedastic.

AuthorsOpenReview ↗

Part 3 of 3: Q3–Q5

Q3 — Fine-tuning setup, statistical evaluation, and data overlap

Appendix E.2 documents the fine-tuning setup. Fine-tuning uses each benchmark dataset's training split separately for every CV fold. We reserve 10% of that split for early stopping; the held-out test fold is not used for fine-tuning. The realTabPFN-v2.5 variants differ by their fine-tuning loss and early-stopping criterion.

Appendix E.3 and Figure 12 report paired per-dataset skill scores against each variant's own base model, two-sided exact paired sign tests, and joint Holm correction over all 66 variant–metric tests. We cannot audit the evaluated models' pretraining corpora or rule out overlap with benchmark test folds.

Q4(a) — Tuning budgets and repeated CV

We use model defaults and do not run benchmark-wide hyperparameter optimization, except where a model performs its own internal tuning. We use five-fold CV once per dataset (n_repeats_cv=1). Additional repeats would increase the already substantial runtime, particularly because fine-tuned variants are trained separately on each dataset and fold.

Q4(b) — Diagnostic ranking directions and training time

Sharpness is ranked lower-is-better. Dispersion is reported as higher-is-better in the diagnostic leaderboard, but higher values are not universally preferable: its interpretation depends on whether errors are heteroscedastic. Neither diagnostic is included in the proper-scoring-rule comparisons. Training time is reported separately and is excluded from aggregate model rankings.

Q4(c) — Duplicate datasets and their impact

Thank you for pointing out this pseudoreplication. We removed BNG(mv), resolved the duplicate between weather_izmir and Wizmir, and excluded Brazilian Houses because its target is effectively determined by its feature columns. These exclusions are documented in Appendix B.2.2 and leave 98 datasets.

Recomputing the aggregate rankings after these exclusions yielded the same overall model ordering, including the top 10. Rankings did change slightly following the separate correction to the representation of tail masses for quantile-based models, which we identified during the bin-count sensitivity analysis. We thank the reviewer for requesting this analysis.

Q5 — Comparison with TabRegSet-101

The count of 101 referred to an earlier ScoringBench candidate set; the revised manuscript reports 98 datasets after the additional duplicate and target-integrity filters. The collections partially overlap but differ in composition. ScoringBench draws from OpenML, PMLB, KEEL, and TALENT and does not source datasets from Kaggle, whereas 23% of TabRegSet-101 does.

TabRegSet-101 evaluates CRPS, 95% interval coverage, and point metrics. ScoringBench adds a broader suite of proper scoring rules and analyzes agreement between their induced rankings across datasets. We clarified this distinction in Related Work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their feedback. We are encouraged by ScoringBench's adoption by independent teams evaluating new tabular foundation models, and we appreciate the reviewers' feedback in helping us strengthen its empirical analysis and presentation. We have uploaded a revised manuscript and provided point-by-point responses under each review.

The main revisions are:

  • Direct analyses of scoring-rule sensitivity: paired model win-rate comparisons with exact McNemar tests and Holm correction (Section 5.1), dataset-level scoring-rule correlations alongside pooled agreement (Section 5.2), and dataset-specific winner crossovers (Section 5.3).
  • Paired fine-tuning evaluation: comparisons against each variant's own base model, with skill scores, win rates, and jointly Holm-corrected exact sign tests (Appendix E.3).
  • Implementation and benchmark clarification: corrected quantile-tail handling, a bin-count sensitivity study (Appendix H), clarified scoring-rule admissibility and diagnostics, and additional dataset exclusions yielding 98 datasets.

We also clarified the relationship to prior work, improved figure readability, and qualified the conclusions regarding pooled ranking changes, fine-tuning significance, and the capped training-data regime. All references in our responses refer to the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.