All five could change my assessment. Q1 and Q3 are the ones I would weight most.
Q1 Please test your core claim directly by reporting the cross-scoring-rule rank-correlation matrix (Spearman and Pearson) across the 51 models, restricted to the proper scoring rules. For the most divergent rule pairs, identify the specific models whose relative rankings reverse, supported by per-dataset paired tests and confidence intervals. Additionally, please provide the promised per-metric table of propriety classes and admissible output types (CRLS class is currently omitted in A.1).
Q2 Please clarify the exact scoring implementation and run a stratified sensitivity check:
(a) bin count, support and edge placement for the equal-width histogram, and the treatment of tied/duplicate quantiles and zero-width bins.
(b) reconcile §3.4 with A.1 — for quantile-native models (TabICLv2, ExaOneTabular, Nori) is CRPS/ESβ computed from native quantiles or from the monotone-interpolated histogram?
(c) Report bin-count sensitivity for DPD, pseudo-spherical, CDE, and PIT-KS, stratified by native output type (native histogram vs. interpolated quantiles/samples), since representation tracks model family.
(d) define the pseudo-spherical score, provide the exact formula, admissible class, and α convention for α∈{1.5,2,3}, and properly attribute it. Cite Good 1971 and Gneiting & Raftery 2007.
(e) Appendix A.2 assigns a literal 0.5 on atoms (citing Czado et al. 2009). However, Czado et al. prescribe randomized PIT on [F(y−),F(y)] or its non-randomized midpoint (F(y−)+F(y))/2, which equals 0.5 only for degenerate point-mass forecasts. Was literal 0.5 implemented by mistake, or the midpoint?
(f) Could authors point to the exact equation in Gneiting & Ranjan (2011) where the piecewise weights w(α)∈{(1−α)2,α(1−α),α2} appear.
(g) Appendix A.2 attributes sharpness to Tran et al. (2020), and the dispersion definition continues "using the notation above" without its own citation, yet the dispersion metric of Tran et al. is the coefficient of variation (cross-sample SD of predicted sigmas divided by their mean), whereas the paper's Dispersion is the un-normalized raw cross-sample standard deviation. Please confirm whether using raw cross-sample standard deviation instead of the coefficient of variation, and carrying the Tran et al. attribution over to dispersion, was intentional.
Q3 Please document the training setup and statistical evaluation for the ~20 realTabPFNv2.5 fine-tuned variants. State the fine-tuning data source, step count and schedule. Explicitly confirm whether any of the 101 benchmark datasets (or their CV folds) overlap with the fine-tuning or pre-training corpora of the evaluated foundation models.
Q4 Please clarify the following experimental design choices:
(a) Per-model tuning budgets (defaults vs. HPO) and whether the top-10 rankings hold for n_repeats_cv greater than 1.
(b) Ranking direction (higher vs. lower is better) for sharpness and dispersion, and confirm whether train time is excluded from aggregate rankings.
(c) Table 1 includes both a base dataset and its synthetic bootstrap resampling: 344 mv together with BNG(mv). Note that the substring rule as specified cannot catch these: numeric prefixes are stripped in normalization, and the substring check is only applied to names longer than three characters. Please clarify whether synthetic resamplings of base datasets already in the suite are treated as distinct tasks and quantify their impact on mean ranks.
Q5 Please cite and differentiate your benchmark from TabRegSet-101 (Oreshkin et al., TMLR/ICML 2026).
Concern 1 — Important results in the appendix
The revised main text makes best use of the 10 pages available for the rebuttal revision. It now reports paired win-rate comparisons (Section 5.1, Figure 2), dataset-level scoring-rule agreement (Section 5.2, Figure 3), and the β-energy crossover analysis (Section 5.3, Figure 4). Unfortunately, some results had to still be put into the appendix due to the page limit.
Concern 2 — Broken leaderboard link
We corrected the extra “h” in the leaderboard URL. The revised manuscript uses https://incredible-truffle-9624b2.netlify.app/.
Concern 3 — Figure readability and model names
Figure 1 now places model labels in two vertically staggered columns and uses a larger font. It abbreviates the
finetune_prefix asft_; the remaining model names are retained for consistency with the manuscript.In Figure 3, we enlarged the colourbar tick labels and description and reduced the gaps between heatmaps. In Figure 2, we enlarged the model-pair and Holm-adjusted p-value annotations, arranged them on a single line, and moved them closer to the plots.
Concerns 4–5 and question — Novelty and empirical findings
We agree that evaluating tabular foundation models beyond point predictions is being investigated. Izbicki and Rodrigues (2026) evaluate CDE loss, log-likelihood, and CRPS; Landsgesell et al. (2026) include fine-tuning experiments and scoring-rule correlations. Both above works cite ScoringBench as concurrent work or use it as infrastructure. ScoringBench contributes an extensible benchmark across multiple scoring-rule families and examines how scoring-rule choice changes model competitiveness and dataset-specific model selection (Sections 5.1–5.3).
The revised manuscript adds direct evidence through paired win-rate comparisons across scoring rules, using exact McNemar tests with joint Holm correction (Section 5.1, Figure 2). Selected comparisons appear in the main text, and further results are provided in Appendix D.4. Section 5.3 and Figure 4 present complementary dataset-specific crossover analyses. The revised discussion distinguishes these effects from the generally high agreement in pooled rankings which is motivated by a decomposition of the covariances (occuring in the Pearson correlation) in the appendix.
The benchmark's value to the community is also reflected in its use by independent teams to evaluate new tabular foundation models. NVIDIA reports Kumo Tabular's ScoringBench results alongside TabArena, BeyondArena, and TALENT; the TabPFN-3.5, EXAONE Tabular 1.0, and Causilo technical reports also use ScoringBench:
This uptake supports its practical value as a community evaluation resource alongside the new empirical analyses.