acceptodds
Under review as a conference paper at ICLR 2027

DrTab: Diagnose, Revise, and Re-read the Table

Abstract

Tabular foundation models (TFMs) learn transferable prediction algorithms through large-scale pretraining and apply them to unseen tables through in-context learning (ICL). Recent studies further narrow the gap between pretrained models and specific downstream tasks through task-time fine-tuning. However, how much of this gap is correctable through adaptation, and how to diagnose this potential before parameter updates to guide adaptation effort, remain insufficiently characterized. To this end, under controlled gradient estimation error and local smoothness, we derive a lower bound on local improvement in the full-context objective from pre-update task signals, providing a theoretical basis for diagnosing adaptation potential. To put this theoretical insight into practice, we propose DrTab (Diagnose, Revise, and Re-read the Table). Grounded in our theoretical analysis, Diagnose uses information from the current training table to estimate the model's remaining potential for improvement on the task. Revise uses the diagnosis to determine whether to continue adaptation and to guide task-specific parameter updates, after which Re-read combines the revised model with the training table for in-context prediction. Experiments show that the pre-update gradient energy characterized by our theoretical analysis is strongly positively correlated with subsequent loss reduction, supporting its use as a diagnostic signal for adaptation potential. Furthermore, as an add-on adaptation framework for TFMs, DrTab demonstrates broader applicability across TFM backbones and achieves better tabular prediction performance than prior adaptation methods. Its performance gains on non-tabular out-of-distribution (OOD) tasks further demonstrate its effectiveness in enhancing TFM adaptation to new task types.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Foundation Models, Task-Time Adaptation, In-Context Learning, Parameter-Efficient Adaptation, Task-Specific Learning, Tabular Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kb7rOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper introduces DrTab, a framework for adapting tabular foundation models (TFMs) to downstream tasks through task-specific LoRA updates. The framework consists of three stages: Diagnose, Revise, and Re-read. Diagnose assesses whether further adaptation appears beneficial at the current model state; Revise proposes and evaluates parameter updates; and Re-read performs prediction using the last accepted model state.

The method proceeds as follows:

  1. Diagnose: The training dataset is divided into (K) folds. Each fold serves as a query set, with its labels hidden from the model’s input, while the remaining folds provide labeled context. The query labels are used to compute the loss and gradients. All (K) gradients are evaluated at the same parameter state and combined to estimate gradient strength, disagreement, and correction headroom. Revision proceeds when the headroom and gradient-consistency criteria are satisfied.

  2. Revise: The diagnostic gradients are reused to construct a candidate update. The candidate undergoes both surrogate-based screening and an acceptance check on an alternate partition of the training data. Accepted updates are retained; rejected updates trigger restoration of the previous parameters. The method then returns to Diagnose to decide whether to continue.

  3. Re-read: The last retained model state is used for prediction through the backbone’s native inference pipeline.

The main experiments cover five TFM backbones. DrTab improves their reported Elo scores on the full 51-dataset TabArena-v0.1 benchmark and generally improves or preserves predictive performance on the AutoML subset and OpenML-CTR23 collection.

Critical strengths
  1. Practical motivation. The paper addresses the relevant problem of deciding whether and how much to adapt a pretrained TFM to a downstream task. Automating these decisions could reduce the need for manual tuning.

  2. Interesting diagnostic formulation. Correction headroom combines gradient information, local geometry, and disagreement across folds into a signal intended to guide adaptation.

  3. Evaluation across multiple backbones. Testing five main TFM backbones provides evidence that the approach is applicable across different architectures.

Critical weaknesses
  1. The acceptance rule is incompletely specified. The paper describes assessing aggregate objective reduction and consistency across folds, but does not fully specify the thresholds or operational definition of consistency. This makes the procedure difficult to reproduce.

  2. The behavior after rejection is unclear. The method restores the previous parameters and reruns Diagnose. If the diagnostic data and computation are unchanged, this should reproduce the same diagnosis. The paper does not clearly explain what changes in the next proposal or what stopping rule prevents repeated rejection.

  3. [Major] The benefit over straightforward fine-tuning with early stopping is not established. Although the paper compares against existing adaptation methods, a controlled comparison with simple LoRA fine-tuning and validation-based early stopping is needed to isolate the value of the diagnostic machinery.

  4. The accuracy–runtime tradeoff is not established. The paper does not report sufficient runtime comparisons to determine whether its predictive gains justify the additional diagnostic and acceptance computations, or where DrTab lies on the accuracy–runtime Pareto frontier.

  5. The quantitative validity of correction headroom is not directly evaluated. The headroom ablation supports the component’s usefulness, but does not show whether its numerical estimates predict realized improvement. The gradient-energy analysis also does not evaluate the complete headroom formula.

  6. [Minor] Averaging raw RMSE across CTR23 datasets complicates interpretation. Because target scales differ across datasets, the aggregate can be dominated by tasks with larger target magnitudes. A scale-independent evaluation would help establish whether the conclusion holds broadly across tasks.

Questions for authors
  1. What is the exact acceptance/rejection rule in Revise? Please specify the proposal-screening threshold, the required aggregate objective improvement, and how consistency across folds is defined.

  2. What happens after a rejected update? With fixed folds and deterministic computation, restoring the same parameters should yield the same diagnosis. Does the method reduce the learning rate, advance to another candidate, change another part of the state, or terminate? Are the optimizer states also restored?

  3. How well does estimated headroom predict actual improvement? A scatter plot across benchmark datasets, with predicted headroom on the x-axis and realized gain on the y-axis, would be informative. Please report correlation coefficients and distinguish one-step gains from gains over the full adaptation trajectory. For a direct quantitative comparison, normalized held-out loss reduction would align with the headroom definition. Evaluating datasets where adaptation is rejected or stopped would also help assess missed improvement opportunities.

  4. Do the regression conclusions hold under scale-independent metrics? Could you report predictive (R^2), normalized RMSE, or relative error reductions on CTR23 alongside the current raw-RMSE aggregate?

  5. How does DrTab compare with simple fine-tuning and early stopping? A baseline using the same LoRA configuration, backbone, data access, and comparable tuning and compute budgets would help isolate the contribution of diagnosis, step guidance, and acceptance control.

  6. What is the accuracy–runtime tradeoff? Could you report total adaptation time and plot predictive performance against runtime across methods and compute budgets, using comparable hardware? This would clarify whether DrTab improves the Pareto frontier.

Reviewer_kh2GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DrTab is a task-time adaptation framework for tabular foundation models. It aggregates cross-fitted gradients to estimate correction headroom, controls LoRA updates using gradient strength and disagreement, and evaluates candidates on an alternate partition of the training table. The paper connects cross-fitted gradients to improvement in the full-context objective and evaluates the framework across several TFM backbones, MNIST, and German/French language identification.

Critical strengths
  1. I see independent value in the theoretical connection between reduced-context gradients and the full-context objective. Appendix A.2 explicitly handles overlapping complementary contexts.

  2. The evaluation covers five main backbones and binary classification, multiclass classification, and regression. Table 1 reports improvements over the corresponding Frozen models in Overall Elo.

  3. The held-out audit in Appendix D.4 is useful: audit rows are excluded from context construction, gradient computation, and adaptation decisions.

Critical weaknesses
  1. My main concern is whether the diagnostic controller improves on ordinary LoRA fine-tuning with a tuned learning rate and validation-based early stopping. This comparison should match the backbone, adapter configuration, and context–query supervision, with comparable data access, regularization, and tuning budgets.

    The Frozen comparisons show that the complete adaptation procedure can help. TuneTables and LoCalPFN provide additional adaptation comparisons, but differ from DrTab in more than diagnostic control. They therefore do not isolate the controller's contribution. Figure 5b includes a LoRA ablation, but does not clearly establish that it is a strong early-stopping baseline.

  2. Figure 5a shows that current-task gradients produce better local corrections than family-averaged gradients. This supports task-specific adaptation, which ordinary fine-tuning already uses.

    Figure 5b addresses diagnostic control more directly. However, the operational definitions of the ablated variants, their stopping rules, and their learning-rate selection are not sufficiently clear. The definition of "relative gain" and its relationship to absolute outer-test performance also need clarification.

    For Step Guidance, Appendix A.4 reports a configuration that scales the projected Adam displacement by a factor between 0.75 and 1. The reported ablation supports this rule under the tested configuration, but does not show whether it outperforms a suitable fixed shrinkage factor or conventional learning-rate schedule.

    The theory does not settle these empirical questions. As acknowledged in Appendix A.3, extending the initial-state probability guarantee across adaptive states requires additional error control. The audit is encouraging—51 of 63 accepted updates reduce the regularized audit objective—but it does not compare the controller with ordinary updates or evaluate rejected candidates. It therefore provides limited evidence about the quality of the controller's selection decisions.

  3. MNIST and German/French language identification are reasonable auxiliary transfer tasks. Improvements over Frozen TFMs show that DrTab can help on these tasks, but do not establish an advantage over ordinary LoRA adaptation using the same labeled support.

    The LLM/VLM comparisons do not answer this question either. Appendix E.2 specifies zero labeled demonstrations for those models, whereas the TFMs receive labeled task data. These results provide context, but do not isolate the benefit of DrTab's diagnostic mechanism.

Questions for authors
  1. How does DrTab compare with matched LoRA fine-tuning using a tuned learning rate and validation-based early stopping? Results on representative backbones and datasets, including a non-tabular setting, would address my main concern.

  2. What stopping and acceptance rules remain in each Figure 5b variant, and were their learning rates independently tuned? Please define relative gain and report the corresponding absolute outer-test results.

  3. Does Step Guidance outperform a tuned constant shrinkage factor or conventional learning-rate schedule under a comparable budget?

Reviewer_ZCiHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

DrTab aims to decide when and how much to adapt a tabular foundation model to a new dataset. It combines gradients from held-out folds to estimate the potential for further improvement. It then guides and checks LoRA updates, while keeping the training table as context for prediction. The paper gives a conditional guarantee for one-step improvement and evaluates the method across several backbones, classification and regression tasks, and image/text tasks represented as tables.

Critical strengths
  1. The experiments cover several backbones with both classification and regression.
  2. The ablations and audit of accepted updates are useful checks.
  3. Most figures are well presented.
Critical weaknesses
  1. The benefit over ordinary fine-tuning is still unclear. The adaptation baselines and "w/o All" ablation do not clearly isolate the benefit of the suggested method. A direct comparison with ordinary LoRA fine-tuning under a comparable budget is needed.

  2. In Figure 5b, removing Step Guidance or Headroom retains about 97% of the reported multiclass gain, with no uncertainty shown.

  3. Clarity and flow are a serious problem. The text is hard to follow throughout, especially in Sections 2-3. Key definitions are mainly left to the appendix. The explanation of Equations 5-8 should distinguish theory, design choices, and the actual algorithm. I also could not find the numerical headroom threshold or complete consistency rule. These gaps make the method difficult to understand and assess.

Questions for authors
  1. How does DrTab compare with ordinary LoRA fine-tuning using the same backbone, data, optimizer, and adapter capacity, with a comparable compute budget and validation-based selection?

  2. What exactly does "w/o All" retain?

  3. Can you explain one adaptation round, connecting Equations 5-8 to the actual updates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.