acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping

Abstract

While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a carefully tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight uncertainty-dependent stopping phenomenon as an important consideration in the design of tabular methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
tabular, regression

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_uZrJOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper evaluates many regression losses for training tabular deep learning models, with the goal of minimizing MSE on test data. It first benchmarks many losses with hyperparameter tuning on a vanilla multilayer perceptron (MLP), then evaluates the most promising ones on top non-foundational deep learning methods TabPack, TabM, and RealMLP. Quantile loss shows robust improvements across architectures. The authors then show that on heteroskedastic datasets, the best early stopping epoch depends on the label variance. Finally, they propose AdaQR, an adaptation of quantile loss that addresses this issue and achieves better practical results.

Critical strengths

The paper is well-written, addresses an important problem, and is experimentally rigorous (modern benchmark datasets, extensive hyperparameter tuning, and multiple backbones + optimizers). It benchmarks existing methods, then analyzes a weakness of them, and proposes a new method that improves on them. It also provides code.

Critical weaknesses

None

Questions for authors
  • What is the reference epoch in Figure 2? Would it make sense to use e.g. the first of the four displayed epochs as the reference? And how is the x-axis sorted in the homoskedastic case?
  • l. 1107: Why are you using ordinal encoding for categoricals? That seems bad, no?
  • Could the authors provide some context how big the loss-related improvements are relative to the inter-model differences? E.g., I'm lacking an intuition like "this gains X Elo/improvability on BeyondArena".

Minor comments:

  • I wonder if the advantage of quantile loss over MSE is mostly from reduced variation between the optimal stopping epochs per region. E.g., it could be interesting to check whether the cohort-based checkpoint selection in Table 3 could close the gap between MSE and quantile loss.
  • In the direction of more baselines, ISQF for quantile loss might be interesting (https://proceedings.mlr.press/v151/park22a/park22a.pdf).
  • Some of the extra space for the revision could be used to better explain the AdaQR loss: Is it a proper loss, i.e., does the distribution that minimize it in expectation have the correct mean? Why is the energy distance squared?
2 comments

Thank you for the review and the positive assessment of our work. Below, we address your questions and suggestions.

Q1

In each panel, the reference is the checkpoint with the lowest aggregate validation MSE, i.e., the global early-stopping checkpoint. Using the first displayed epoch would also be valid. We chose the global checkpoint to show which regions benefit from stopping earlier or later than the standard choice. Changing the reference would not alter the differences between the epoch curves.

The homoskedastic panel uses the same ordering as the heteroskedastic panel, based on the noise scale σ(x)\sigma(x) in the heteroskedastic counterpart. The features and noiseless target function are unchanged between the two versions; only the training noise scale becomes constant in the homoskedastic one. This preserves the same groups of test inputs across the two settings. We will clarify this in the caption.

Q2

Thank you for catching this error. We use one-hot encoding for categorical features, “ordinal encoding” in Appendix H.1 is a mistake in the description. We will fix it in the revised version of the paper.

Q3

To provide this context, we computed Elo-style ratings from our experimental results. These are internal ratings for our comparison, not official BeyondArena or TabArena leaderboard ratings.

BackboneMSEHL-GaussianQuantile loss
MLP100011451122
TabM-mini PLE-Q130813711406
RealMLP137214191519
TabPack1384—1492

In our ratings, changing from MSE to quantile loss gives +122 points on MLP and +98 on TabM. For comparison, moving from MLP to TabM with MSE gives +308 points, while moving from TabM to RealMLP with MSE gives +64. Thus, in this comparison, changing the objective has a smaller effect than the MLP to TabM transition, but can exceed the difference between two stronger backbones.

Minor comments

I wonder if the advantage of quantile loss over MSE is mostly from reduced variation between the optimal stopping epochs per region. E.g., it could be interesting to check whether the cohort-based checkpoint selection in Table 3 could close the gap between MSE and quantile loss.

Thank you for this suggestion. We also applied cohort-based checkpoint selection to MSE on the heteroskedastic synthetic dataset. With K = 5, the relative RMSE improvement is 0.457%, compared with 0.318% for QR in Table 3. However, this is much smaller than the 25.48% gap betweeen QR and MSE, so cohort-based selection does not close the gap in this experiment.

However, this test is an imperfect proxy for the role of early stopping mismatch. It only selects among checkpoints from an existing training run, with hyperparameters tuned for global early stopping. Moreover, selecting each checkpoint from a smaller validation subset introduces additional noise, which can obscure potential gains. The result therefore does not rule out better early stopping alignment as an important contributor to QR’s advantage over MSE.

In the direction of more baselines, ISQF for quantile loss might be interesting.

Thank you for pointing us towards ISQF. We will cite this work and discuss its connection to our quantile-based approaches in the revised paper.

Does the distribution that minimize it in expectation have the correct mean?

Why is the energy distance squared?

Ignoring quantile discretization, quantile loss is proportional to the Continuous Ranked Probability Score (CRPS), a standard loss for evaluating predicted distributions. In one dimension, CRPS⁡(F,y)=12D2(F,δy)\operatorname{CRPS}(F,y)=\tfrac12D^2(F,\delta_y). This explains why the squared energy distance appears in our formulation.

The core modification behind AdaQR is to replace the point target yy with a Gaussian distribution centered at yy, with an input-dependent smoothing scale σl\sigma_l. This can be viewed as adding zero-mean Gaussian noise to the target.

For fixed xx and σl\sigma_l, the expected AdaQR loss can therefore be written as $\mathbb E_Y L(F,Y,\sigma_l)=\mathbb E_{Y,Z}\operatorname{CRPS}(F,Y+\sigma_l Z) +\left(\lambda-\frac1{\sqrt{\pi}}\right)\sigma_l,$ where ZZ is a standard normal variable independent of YY.

CRPS is a proper scoring rule: its expected value is minimized by predicting the distribution of the target it is evaluated against. Since the last term above does not depend on FF, the optimal predictive distribution is the distribution Y+σlZY+\sigma_l Z. The added noise has zero mean, so this distribution has the same mean as YY. Thus, ignoring quantile discretization, AdaQR preserves the conditional mean at the optimum, although the distribution itself is smoothed.

Thank you for your clarifications and additional results, they look good!

Reviewer_9ALPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the impact of different objective functions on regression tasks in tabular deep learning. It provides a systematic evaluation of 18 different loss functions across 41 datasets and multiple neural backbones. A key contribution is the identification of a non-trivial challenge: the "uncertainty-dependent stopping" phenomenon. The authors demonstrate that in heteroskedastic settings, a single global early-stopping point forces a compromise between regions with different noise levels. To address this, they propose AdaQR (Adaptive Quantile Regression), which incorporates input-dependent label smoothing into quantile training. The method is theoretically grounded in Energy Distance and successfully outperforms standard Quantile Loss across several state-of-the-art tabular backbones (e.g., TabPack, TabM, RealMLP).

Critical strengths
  • Comprehensive Empirical Evaluation: The large-scale benchmark (41 datasets) provides a solid empirical foundation for the claims regarding the superiority of Quantile Loss over standard MSE in various contexts. It also is a great base as reasoning to implement AdaQR.

  • Insightful Problem Identification: The analysis of "uncertainty-dependent stopping" is a novel observation that moves the discussion from simple "accuracy gains" to understanding why certain models behave differently in heteroskedastic environments.

Critical weaknesses
  • Inconsistency in Evaluation Scope: While the initial benchmark is extensive (41 datasets), the evaluation of AdaQR is performed on a smaller subset (28 datasets). While the authors justify this by excluding "Tiny" datasets due to complexity, it remains unclear if these excluded datasets represent a significant portion of the "heteroskedastic" landscape where standard methods might still be sufficient.

  • Complexity vs. Practical Gain: AdaQR introduces an additional learnable parameter (σl\sigma_l) and a new hyperparameter (λ\lambda). The authors should clarify the "cost of complexity" specifically, whether the performance gains over standard Quantile Loss are significant enough to justify the added complexity in production environments where simplicity is often preferred.

Questions for authors
  • Regarding Scope: Could the authors clarify if the 13 "Tiny" datasets excluded from the AdaQR evaluation possess specific characteristics (e.g., low feature dimensionality or high sparsity) that would make them fundamentally different from the 28 included datasets?

  • Regarding Hyperparameters: Could the authors provide a sensitivity analysis for the λ\lambda parameter? It would be helpful to see if a wide range of λ\lambda values yields similar results, or if the performance of AdaQR is highly sensitive to this specific hyperparameter.

  • Regarding Methodology: Could the authors elaborate on the distinction between AdaQR and β\beta-NLL? Specifically, does the "smoothing" mechanism in AdaQR address the "stopping mismatch" more effectively than the "weighting" approach used in β\beta-NLL?

2 comments

Thank you for the review. Below, we would like to address the concerns you have raised regarding our paper.

W1 & Q1

For the main comparison, we excluded datasets with fewer than 1,000 training examples. The restriction is based only on training-set size, rather than feature dimensionality, sparsity, or estimated heteroskedasticity. Across the 13 Tiny datasets, the median validation set size is 73. This can make our hyperparameter optimization procedure, which uses 100 trials, unstable and prone to overfitting the validation set.

W2

As we show in Table 4 of the paper, compared with QR, AdaQR increases the median gains over MSE from 0.35% to 0.82% on RealMLP and from 0.39% to 0.97% on TabM. These gains do come with additional difficulty in hyperparameter tuning and gradient-based optimization. At inference, however, the σl\sigma_l output is not needed, and predictions are obtained by averaging the quantiles. Thus, AdaQR could serve as a drop-in improvement over QR at the deployment stage.

AdaQR also provides a concrete example of how the stopping mismatch identified in Section 4 can be partially mitigated during training, as illustrated in Figure 2. This is part of its motivation, alongside the predictive gains.

Q2

We thank you for the suggestion. We have conducted this analysis on six datasets. First, we tune hyperparameters for AdaQR using our usual protocol. We then vary the regularization parameter while keeping the remaining hyperparameters fixed and retrain the model. We also evaluate QR using the same hyperparameters (except for λ\lambda). Below, we report the mean and standard deviation of test RMSE. For convenience, we tune a rescaled parameter, Regularizer = 1−πλ1 - \sqrt{\pi}\lambda, over [0,1][0, 1]. Regularizer = 0 favors vanishing smoothing, for which the loss approaches QR up to a constant factor.

Regularizer / methodSberbank HousingSuperconductivityDiamondAirfoilHouse 16HFood Delivery
00.22983 ± 0.002239.610 ± 0.0540.12890 ± 0.000391.578 ± 0.05929916.4 ± 159.47.4742 ± 0.0058
0.010.23092 ± 0.001729.606 ± 0.0480.12870 ± 0.000391.556 ± 0.04929877.9 ± 181.87.4738 ± 0.0056
0.030.23046 ± 0.002239.620 ± 0.0450.12869 ± 0.000301.531 ± 0.03329870.3 ± 200.97.4712 ± 0.0055
0.10.23053 ± 0.002239.626 ± 0.0540.12884 ± 0.000361.554 ± 0.04329872.3 ± 201.07.4727 ± 0.0064
0.30.23089 ± 0.002719.602 ± 0.0440.12879 ± 0.000421.543 ± 0.05129894.0 ± 215.07.4712 ± 0.0056
0.50.23071 ± 0.002669.595 ± 0.0330.12888 ± 0.000551.524 ± 0.03829945.9 ± 185.87.4693 ± 0.0058
0.70.23049 ± 0.001719.600 ± 0.0430.12898 ± 0.000421.524 ± 0.04430028.1 ± 220.97.4625 ± 0.0056
0.90.23409 ± 0.003079.609 ± 0.0370.12923 ± 0.000681.497 ± 0.02930020.1 ± 169.77.4492 ± 0.0053
0.990.23566 ± 0.0014017.134 ± 0.2020.12986 ± 0.000651.470 ± 0.01730110.9 ± 165.27.4335 ± 0.0054
HPO-selected0.23057 ± 0.001709.607 ± 0.0370.12907 ± 0.000571.526 ± 0.02529886.8 ± 208.97.4470 ± 0.0055
QR (same HPs)0.23007 ± 0.001889.627 ± 0.0400.12897 ± 0.000421.535 ± 0.05029899.0 ± 185.97.4689 ± 0.0064

Across the tested Regularizer values from 0.01 to 0.7, mean RMSE varies by less than 0.6% on Sberbank, Superconductivity, Diamond, and House 16H. The preferred range is dataset-dependent: Airfoil and Food Delivery benefit from larger settings, while Superconductivity deteriorates sharply at 0.99. These results show a broad range with similar performance on several datasets, while also showing that tuning remains important. We will include this analysis in the revised version of the paper.

Q3

When analysing the relationship between data uncertainty and the optimal early-stopping epoch in Section 4, we found that regions with high data uncertainty begin to overfit earlier and therefore prefer earlier checkpoints. β\beta-NLL addresses a different optimization issue in Gaussian NLL: the gradient with respect to the predicted mean is (μ−y)/σ2(\mu-y)/\sigma^2, so samples assigned a large predicted variance can learn more slowly and underfit. β\beta-NLL reweights the loss to counteract this effect.

Quantile loss has no such explicit inverse-variance factor. For a fixed quantile level, its gradient with respect to the predicted quantile takes one value when the target is below the prediction and another when it is above. Nevertheless, our experiments show that high-uncertainty regions can still overfit before lower-uncertainty regions have finished learning. AdaQR introduces input-dependent target smoothing to mitigate this mismatch, as illustrated in Figure 2. Our stopping-time analysis compares AdaQR with QR; the comparison with β\beta-NLL in Table 4 concerns predictive performance.

Thank you for the detailed and thorough response. Your clarifications fully address the concerns I raised in my review. Excluding the Tiny datasets only because of training-set size and unstable HPO on very small validation sets, rather than because of their data characteristics, makes sense, and it does not bias the conclusions. The gains in Table 4, together with the fact that AdaQR adds no cost at inference, show that the extra training complexity is justified, which makes AdaQR a practical drop-in replacement for QR. The new sensitivity analysis is helpful: performance stays stable across a broad range of Regularizer values on most datasets, while tuning still matters in some cases. I appreciate that it will be included in the revised version. The distinction between AdaQR's smoothing, which targets the stopping mismatch, and β-NLL's reweighting, which targets underfitting of high-variance samples, is now clear. I appreciate the effort the authors put into the additional experiments, and I maintain my positive assessment of the paper.

Reviewer_gieBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a study of 18 regression objectives for deep tabular learning on 41 datasets. Authors demonstrated that quality loss consistently outperforms a well-tunned first choice for regression problems MSE baseline. Also, the paper propose a novel loss AdaQR. which uses input depended label smoothing to improve models performance.

Critical strengths
  • Important and understudied problem
  • Well written and easy to follow
  • Extensive number of datasets and strong benchmarking protocol
Critical weaknesses
  • No foundation models (current SOTA on tabular data)
  • AdaQR improvement are rather smalls
Questions for authors

Please see Weaknesses.

Also, given the relatively small improvements over standard quantile loss, how do the authors justify the additional complexity of AdaQR?

Have the authors considered ensembling models trained with multiple objectives, such as MSE, quantile loss, and HL-Gaussian?

1 comment

Thank you for the review. Below, we would like to address the concerns you have raised regarding our paper.

No foundation models (current SOTA on tabular data)

Our main benchmark focuses on conventional DNNs trained separately on each dataset. A thorough comparison of different regression objectives for SOTA TFMs would be computationally prohibitive because tuning the optimization settings and loss-specific hyperparameters under comparable budgets would require multiple full pretraining runs per objective.

However, quantile and binning-based objectives highlighted in our evaluation are widely used among the strongest models evaluated on the regression tracks of TabArena and BeyondArena, which indicates their superiority for TFMs as well. Kumo, which ranks first on both, uses quantile loss, as do TabAbax, Xiaomi-TabLDM, and TabICLv2. TabPFN-3.5, which ranks near the top on both, and LimiX-2 predict distributions over target bins. Mitra-v2, TabPFN v2, and TabPFN-2.5 also use binning-based regression losses.

AdaQR improvement are rather smalls

Also, given the relatively small improvements over standard quantile loss, how do the authors justify the additional complexity of AdaQR?

As we show in Table 4 of the paper, AdaQR more than doubles QR's median gains over MSE on RealMLP and TabM: from 0.35% to 0.82% on RealMLP and from 0.39% to 0.97% on TabM. We hypothesise that the additional difficulty of jointly optimizing the quantiles and smoothing scale may limit the gains on MLP and TabPack.

Have the authors considered ensembling models trained with multiple objectives, such as MSE, quantile loss, and HL-Gaussian?

We thank you for the suggestion. To test whether such ensembling would be beneficial, we evaluate ensembles comprising models trained using different loss functions. We use the MLP backbone on the BeyondArena datasets used in the paper, excluding the Large subset. Ensembles consist of six models, split equally among the participating objectives. For comparison, we also evaluate ensembles in which all models use the same objective. Below, we report the median relative improvement in RMSE over a six-model MSE ensemble for each of the seven ensemble types.

MethodGain (%)
MSE0.000%
Quantile loss0.406%
HL-Gaussian0.495%
MSE + quantile loss0.397%
HL-Gaussian + quantile loss0.683%
MSE + HL-Gaussian0.500%
MSE + quantile loss + HL-Gaussian0.681%

As shown in the table above, combining HL-Gaussian and quantile loss yields a larger median gain than using either objective alone. Adding MSE does not increase this median gain (0.681% versus 0.683%). We will include this result in the revised version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.