- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper evaluates many regression losses for training tabular deep learning models, with the goal of minimizing MSE on test data. It first benchmarks many losses with hyperparameter tuning on a vanilla multilayer perceptron (MLP), then evaluates the most promising ones on top non-foundational deep learning methods TabPack, TabM, and RealMLP. Quantile loss shows robust improvements across architectures. The authors then show that on heteroskedastic datasets, the best early stopping epoch depends on the label variance. Finally, they propose AdaQR, an adaptation of quantile loss that addresses this issue and achieves better practical results.
Critical strengths
The paper is well-written, addresses an important problem, and is experimentally rigorous (modern benchmark datasets, extensive hyperparameter tuning, and multiple backbones + optimizers). It benchmarks existing methods, then analyzes a weakness of them, and proposes a new method that improves on them. It also provides code.
Critical weaknesses
None
Questions for authors
- What is the reference epoch in Figure 2? Would it make sense to use e.g. the first of the four displayed epochs as the reference? And how is the x-axis sorted in the homoskedastic case?
- l. 1107: Why are you using ordinal encoding for categoricals? That seems bad, no?
- Could the authors provide some context how big the loss-related improvements are relative to the inter-model differences? E.g., I'm lacking an intuition like "this gains X Elo/improvability on BeyondArena".
Minor comments:
- I wonder if the advantage of quantile loss over MSE is mostly from reduced variation between the optimal stopping epochs per region. E.g., it could be interesting to check whether the cohort-based checkpoint selection in Table 3 could close the gap between MSE and quantile loss.
- In the direction of more baselines, ISQF for quantile loss might be interesting (https://proceedings.mlr.press/v151/park22a/park22a.pdf).
- Some of the extra space for the revision could be used to better explain the AdaQR loss: Is it a proper loss, i.e., does the distribution that minimize it in expectation have the correct mean? Why is the energy distance squared?
2 comments
Thank you for your clarifications and additional results, they look good!
Thank you for the review and the positive assessment of our work. Below, we address your questions and suggestions.
Q1
In each panel, the reference is the checkpoint with the lowest aggregate validation MSE, i.e., the global early-stopping checkpoint. Using the first displayed epoch would also be valid. We chose the global checkpoint to show which regions benefit from stopping earlier or later than the standard choice. Changing the reference would not alter the differences between the epoch curves.
The homoskedastic panel uses the same ordering as the heteroskedastic panel, based on the noise scale σ(x) in the heteroskedastic counterpart. The features and noiseless target function are unchanged between the two versions; only the training noise scale becomes constant in the homoskedastic one. This preserves the same groups of test inputs across the two settings. We will clarify this in the caption.
Q2
Thank you for catching this error. We use one-hot encoding for categorical features, “ordinal encoding” in Appendix H.1 is a mistake in the description. We will fix it in the revised version of the paper.
Q3
To provide this context, we computed Elo-style ratings from our experimental results. These are internal ratings for our comparison, not official BeyondArena or TabArena leaderboard ratings.
In our ratings, changing from MSE to quantile loss gives +122 points on MLP and +98 on TabM. For comparison, moving from MLP to TabM with MSE gives +308 points, while moving from TabM to RealMLP with MSE gives +64. Thus, in this comparison, changing the objective has a smaller effect than the MLP to TabM transition, but can exceed the difference between two stronger backbones.
Minor comments
Thank you for this suggestion. We also applied cohort-based checkpoint selection to MSE on the heteroskedastic synthetic dataset. With K = 5, the relative RMSE improvement is 0.457%, compared with 0.318% for QR in Table 3. However, this is much smaller than the 25.48% gap betweeen QR and MSE, so cohort-based selection does not close the gap in this experiment.
However, this test is an imperfect proxy for the role of early stopping mismatch. It only selects among checkpoints from an existing training run, with hyperparameters tuned for global early stopping. Moreover, selecting each checkpoint from a smaller validation subset introduces additional noise, which can obscure potential gains. The result therefore does not rule out better early stopping alignment as an important contributor to QR’s advantage over MSE.
Thank you for pointing us towards ISQF. We will cite this work and discuss its connection to our quantile-based approaches in the revised paper.
Ignoring quantile discretization, quantile loss is proportional to the Continuous Ranked Probability Score (CRPS), a standard loss for evaluating predicted distributions. In one dimension, CRPS(F,y)=21D2(F,δy). This explains why the squared energy distance appears in our formulation.
The core modification behind AdaQR is to replace the point target y with a Gaussian distribution centered at y, with an input-dependent smoothing scale σl. This can be viewed as adding zero-mean Gaussian noise to the target.
For fixed x and σl, the expected AdaQR loss can therefore be written as $\mathbb E_Y L(F,Y,\sigma_l)=\mathbb E_{Y,Z}\operatorname{CRPS}(F,Y+\sigma_l Z) +\left(\lambda-\frac1{\sqrt{\pi}}\right)\sigma_l,$ where Z is a standard normal variable independent of Y.
CRPS is a proper scoring rule: its expected value is minimized by predicting the distribution of the target it is evaluated against. Since the last term above does not depend on F, the optimal predictive distribution is the distribution Y+σlZ. The added noise has zero mean, so this distribution has the same mean as Y. Thus, ignoring quantile discretization, AdaQR preserves the conditional mean at the optimum, although the distribution itself is smoothed.