acceptodds
Withdrawn from ICLR 2027

TabCAM: Online Calibrated Regularization For Tabular Deep Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern tabular deep learning models are mostly built on overparameterized architectures, which require carefully calibrated regularization to ensure out-of-sample predictive accuracy and reliability. Standard regularization approaches rely on cross-validation via grid search to tune a small number of global penalty coefficients, which inevitably over-regularizes some parameters and under-regularizes others. Ideally, each model parameter should be assigned its own regularization strength, enabling the training to selectively suppress dataset-specific noise while preserving signals that capture true population-level structure. However, since the cost of grid search grows exponentially in the number of hyperparameters, per-parameter regularization is computationally infeasible under standard approaches. To address this bottleneck, we propose TabCAM (Tabular Calibrated Auto-Modeling), an add-on module for differentiable backbones that jointly estimates the model parameters and per-parameter regularization strengths inside a single training loop. Through a novel online construction of calibrated future targets, TabCAM adds minimal computational overhead beyond vanilla model training, making it particularly promising for future adoption with even more complex architectures where principled per-parameter regularization becomes increasingly valuable. Moreover, because TabCAM estimates the population conditional distribution during training, covariate-conditional uncertainty quantification is built in. On the 51-dataset TabArena benchmark, TabCAM ranks third among 16 non-foundation default methods and improves substantially under modest tuning while matching the runtime of vanilla model training. In simulations, TabCAM converges to the population minimizer and produces prediction intervals with near-nominal coverage that is more consistent and narrower than split-conformal baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
per-parameter regularization, online calibration, tabular deep learning, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1KazOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The manuscript proposes a method called TabCAM, which learns a separate regularization strength for each parameter of a neural network for tabular prediction. The method avoids computationally expensive hyperparameter search by learning these regularization strengths online during model training. To achieve this, TabCAM divides the data into two parts at selected stages of training. The first part is used to train the neural network, while the second part is used to correct the model’s predictions. Based on these corrected predictions, TabCAM creates pseudo- outcomes that are intended to represent future data. It then compares the model updates suggested by the training data and the pseudo- outcomes, and uses their difference to adjust the regularization strengths. For regression, the method starts from a Gaussian prediction model; for classification, it uses calibrated class probabilities. Experiments evaluate predictive performance, computational cost, learned regularization patterns, and uncertainty estimates.

Critical strengths
  • The online approximation avoids repeated bootstrap refitting and can be applied to several neural tabular architectures with modest overhead.
  • There is some evidence of portability and interesting learned structure. More specifically, the method works with MLP, TabM, and FT-Transformer, while simulations show stronger penalties for parameters associated with irrelevant features.
Critical weaknesses
  • Limited technical novelty. The decomposition, per-parameter gradient-matching objective, and alternating optimization closely follow Jiang and Liu as the authors have acknowledged in the manuscript. The major change is replacing bootstrap imputation with online calibrated pseudo-samples, while the calibration ingredients are closely related to Kuleshov et al. and Guo et al. This combination may be new, but it appears technically incremental.

  • Unconvincing empirical improvement. If the empirical performance of the proposed method is strong, then the 1st weakness would become relatively minor. However, TabCAM does not outperform traditional methods, not even to mention the recent tabular foundation-model baselines. Even against matched neural backbones, its improvements are generally around or below 1%, with some confidence intervals including no improvement. These results do not clearly demonstrate that the additional machinery provides a consequential practical advantage.

Questions for authors
  • How robust is the Gaussian working model?
  • PIT calibration addresses marginal calibration but does not guarantee conditional correctness. Can the authors comment on how this would influence the performance of the method?
  • Because pseudo-targets are generated from the current fitted model, can the coupled updates converge to a self-confirming but incorrect solution?
  • A gradient evaluated on an independent validation set is already an unbiased estimator of the population gradient under standard i.i.d. assumptions. Having a larger sample size usually leads to a more precise approximation of the population level gradient in the V term. Hence, why should the more indirect calibrated pseudo-sample gradient be used than directly using the calibration-set gradient? What are the advantages?
Reviewer_oiNVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TabCAM, a method that gives every parameter of a tabular neural network its own regularization strength and learns these strengths during training instead of tuning them by grid search. At each regularization update, the data are split in two: one part trains the model, and the other is used to generate synthetic “future” responses from the current model. The gap between how the model behaves on these synthetic responses and on the training data is used to adjust the regularization strengths. The same fitted model also yields prediction intervals for uncertainty quantification. Empirically, TabCAM ranks third among 16 non-foundation methods on TabArena at default settings (with overlapping confidence intervals), is more than 10x faster than the Auto-Modeling approach it builds on, and gives narrower prediction intervals than split-conformal baselines at similar coverage in simulation.

Critical strengths

Strength: TabCAM makes Auto-Modeling feasible at benchmark scale, with transparent evaluation.

  1. TabCAM is more than 10× faster than AM at roughly RealMLP’s training cost (Fig. 3). In the ridge simulation, it gets closer to the oracle than AM (risk ratio about 1.015 vs 1.11) while being about 46x faster (Fig. 5).
  2. As a result, it can be evaluated on all 51 TabArena datasets with the official toolkit and confidence intervals. It performs competitively among non-foundation methods (Tables 12, 14).
  3. The paper fully reports its settings and search spaces (Tables 1,2), which makes the results checkable, including the limitations noted below.
Critical weaknesses

Weakness 1: The λ-step’s link to population risk is not established.

  1. With the CDF and the empirical PIT correction, each future target reduces to the model’s own prediction plus a residual borrowed from a random other calibration point (Eq. 11; Appendix B.3 and H.2). However, the residual is unrelated to xix_i, so gQg_Q loses the information about where the model is wrong, which is what the true gap gradient depends on (Eq. 9). Averaging several draws pushes the target even closer to the model’s own prediction (Table 1 and Appendix F.2).
  2. The paper never checks that gQg_Q − gtrg_{tr} tracks the true gap, even in the ridge simulation where it is computable (Appendix. F.1).
  3. Prop. 1 gives a sufficient condition (App. A.1), but the paper states that exact matching can fail under λ ≥ 0 or θj\theta_j = 0 (App. A.2), and no result shows that the estimated λ-step (Eq. 10) approaches it.
  4. The λ-step solves k equations in k unknowns. It can fit gradient noise and is unstable as θj\theta_j → 0 (Appendix. A.2, Eq. 15). No variance analysis is given.
  5. Unless I’m missing something, TabCAM may be closer to noisy adaptive weight decay than to population-risk targeting.

Weakness 2: the gains are small and the advantage of TabCAM is unclear

  1. Table 10 shows that turning off the main contribution (calibration + λ-update) result in statistically similar results with p = 0.052 and +2.9% error, making the contribution less convincing
  2. In Appendix D, it shows that TabCAM is mostly RealMLP underneath since the preprocessing follows RealMLP-TD and the architecture reuses RealMLP parts. Yet RealMLP beats TabCAM on TabArena in Figure 2 and Table 14 (TabArena Lite).
  3. Missing a tuned weight-decay baseline on the same backbone and budget in the experiment in Table 10 and 14 to justify the need of per-parameter regularization.
  4. Overall, it’s hard to know if the gain is from TabCAM’s core contribution on per-parameter regularization or a strong RealMLP backbone model.
Questions for authors

Major questions:

  1. Why does gQg_Q estimate the population gradient? Is there empirical or theoretical explanation why should gQg_Q − gtrg_{tr} estimate ∇RP\nabla R_P − ∇Rn\nabla R_n? Can the authors show this numerically in the ridge model, where ∇RP\nabla R_P is known (Appendix F.1)? For example, report the cosine similarity between the two over training.
  2. How does TabCAM compare with tuned weight decay? Can the authors compare TabCAM with the same backbone plus standard (global or per-layer) weight decay, tuned with the same 100-trial budget (e.g., extending Tables 10, 14)? Without this, the gain from per-parameter λ cannot be separated from ordinary tuning.
  3. How far does λ move on TabArena? The paper shows that default lambda learning rate is 3e-6 with Adam and the simulation used 3e-2 (Appendix I.2). What is the initial λ on TabArena? How far do the λj move by the end of training? Do tuned configs with large lambda learning rate dominate the ensemble (Table 2)?
  4. Is the gain over the backbone significant on all 51 datasets? The online-calibration ablation has p = 0.052 on 9 datasets (Table 10). Can you re-run it on all 51 datasets, with dataset-level paired tests or bootstrap CIs.
  5. Does the UQ advantage survive a matched scale model? TabCAM’s scale model uses the simulator’s exact noise rule (Appendix B.4), while the split-conformal baselines are constant-width (Appendix F.2), so the high-noise coverage gap (Table 4) may come from this oracle information. Can the authors compare against normalized split conformal with the same s^\hat{s}(x), and against both methods learning σ\sigma(x) from data? Also clarify which methods use s^\hat{s}(x), since Appendix F.2 says scale calibration is “for both TabCAM and split conformal.”
  6. Are calibration residuals out-of-sample? The calibration split also trains θ\theta during warmup and in epochs without a λ-update (Algorithm 1, lines 2 and 10). Does this reuse bias the calibration residuals, and through them gQg_Q?

Minor questions:

  1. Default TabCAM ranks above RealMLP on full TabArena (Table 12) but below it on Lite (Table 14). Does this indicate the default advantage over RealMLP is within noise?
  2. Which λ types do tuned configs pick? The search space includes scalar-λ options. What fraction of tuned configs use each lambda_type?
  3. Figure 3 and Figure 11 list different CPUs (EPYC 7413 vs 7662) for what appear to be the same timing runs.
  4. Does vector-L2 TabCAM recover known per-coordinate optima? The ridge oracle test covers only a scalar λ
  5. What is the intended use case for TabCAM given recent progress in tabular foundation models? Fig. 10 shows foundation models such as TabPFN-2.6 and TabICLv2 ahead of TabCAM on TabArena. I’d be interested in the authors’ view on where from-scratch, per-parameter regularization still adds the most value. Is it regimes where foundation models struggle, such as large n, many features, or tight latency? Or is the main contribution the regularization idea itself, which could later carry over to fine-tuning those models? A short discussion of this positioning would help readers understand when to choose TabCAM.
Reviewer_xKgDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This research proposes TabCAM (Tabular Calibrated Auto-Modeling), a method for learning the regularization strength for each model parameter during training in tabular learning. It assigns an individual regularization coefficient to each model parameter and alternately updates the model parameters and regularization coefficients within a single training loop. Model parameters are trained to minimize the normalized empirical risk, while regularization coefficients are learned via gradient matching so that the normalized training gradient converges to the gradient calculated from the calibrated future target. This approach aims to adjust the regularization strength by approximating the unobservable population-risk gradient with the gradient of calibrated synthetic data.

Critical strengths
  • The proposed method is a backbone-agnostic regularization framework that is not dependent on any specific neural network architecture; it can be applied to various differentiable backbones without significantly altering the structure of existing models.
  • The main methodological contribution is the replacement of the computationally intensive bootstrap-based future observation construction used in existing auto-modeling approaches with an online future-target construction method that utilizes PIT calibration and inverse CDF.
Critical weaknesses
  • The primary objective of this paper is to improve generalization performance by efficiently learning the per-parameter regularization strength of a neural network. Therefore, to evaluate the practical value of the proposed method, it is important to compare it with existing regularization and hyperparameter optimization methods that share the same objective. However, current experiments primarily focus on comparing prediction performance on the TabArena benchmark and comparing computation time with the existing auto-modeling method. Although the Appendix presents ablation results against a global regularization coefficient and comparisons using the same backbone, direct comparisons with existing efficient adaptive regularization or gradient-based hyperparameter optimization methods are lacking. In particular, MacKay et al. (2019) and Lorraine et al. (2020), cited in the paper, also proposed methods for efficiently optimizing multiple regularization hyperparameters during training. Without comparing prediction performance and computational cost using the same backbone and training protocol as these prior studies, it is difficult to fully assess the relative superiority of TabCAM.

  • A significant portion of the methodology presented in this paper is based on concepts and methods proposed in previous studies. In particular, the fundamental approaches to population-risk decomposition, the duality function, and gradient matching in Section 2.1 are derived from the Auto-Modeling framework proposed by Jiang and Liu (2025), and the concept of learning per-parameter regularization strength also exists in prior research. Therefore, the core methodological contribution of this paper appears to lie in the online calibrated future-target construction proposed in Section 2.2. While this construction is significant in that it reduces the computational cost of existing bootstrap-based Auto-Modeling, it does not provide sufficient justification regarding how accurately the gradient calculated from the generated future target approximates the actual population-risk gradient. In particular, Proposition 1 in Appendix A assumes precise gradient matching conditions, and it is unclear how well these conditions are satisfied in the actual algorithm or how the approximation error affects the final learning results.

  • Section 3.1 describes the results of the TabArena benchmark with a focus on the Elo score, but the main text does not clearly explain which performance metrics are used for regression, binary classification, and multiclass classification, respectively. Although explanations of RMSE, ROC-AUC, and log-loss are provided in Appendix E.1, this basic evaluation information is essential for understanding the experimental results; therefore, it would be appropriate to explain it first in the main text.

  • Section 2.1 describes gradient-matching-based optimization based on Proposition 1, but the proposition itself is presented in Appendix A. Even if the detailed proof is placed in the appendix, presenting the content, key assumptions, and significance of the proposition in the main text would help readers understand the methodology. In the current structure, readers must repeatedly refer to various appendices to verify the main claims and their supporting evidence, making it difficult to follow the flow of the paper.

Questions for authors

Please address the concerns in the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.