ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules

Tabular foundation models such as TabPFN [Hollmann et al., 2025] and TabICL [Qu et al., 2026] already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics (RMSE, R2). This discards precisely the distributional information these models are designed to provide — a critical gap for high-stakes domains where not all kinds of errors are equally costly. We introduce ScoringBench, an open and extensible benchmark that evaluates tabular regression models under a comprehensive suite of proper scoring rules — including CRPS, interval score, energy score, and weighted CRPS — alongside standard point metrics. ScoringBench covers regression datasets from diverse domains, supports transparent community contributions via a git-based leaderboard, and provides two complementary ranking protocols: an ordinal Demšar/autorank approach and a magnitude-preserving z-score ranking approach. Evaluating several models — spanning in-context learners, fine-tuned foundation models, gradient-boosted trees, and MLPs — we find that model rankings shift substantially depending on the scoring rule: models that excel on point-estimate metrics can rank poorly on probabilistic ones, and the top-performing model under one proper scoring rule may rank noticeably lower under another. These results demonstrate that the choice of evaluation metric is not a technicality but a modelling decision — and, for applications where e.g. tail errors are disproportionately costly, a domain-specific requirement with direct consequences for model deployment.