- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
This paper introduces DrTab, a framework for adapting tabular foundation models (TFMs) to downstream tasks through task-specific LoRA updates. The framework consists of three stages: Diagnose, Revise, and Re-read. Diagnose assesses whether further adaptation appears beneficial at the current model state; Revise proposes and evaluates parameter updates; and Re-read performs prediction using the last accepted model state.
The method proceeds as follows:
-
Diagnose: The training dataset is divided into (K) folds. Each fold serves as a query set, with its labels hidden from the model’s input, while the remaining folds provide labeled context. The query labels are used to compute the loss and gradients. All (K) gradients are evaluated at the same parameter state and combined to estimate gradient strength, disagreement, and correction headroom. Revision proceeds when the headroom and gradient-consistency criteria are satisfied.
-
Revise: The diagnostic gradients are reused to construct a candidate update. The candidate undergoes both surrogate-based screening and an acceptance check on an alternate partition of the training data. Accepted updates are retained; rejected updates trigger restoration of the previous parameters. The method then returns to Diagnose to decide whether to continue.
-
Re-read: The last retained model state is used for prediction through the backbone’s native inference pipeline.
The main experiments cover five TFM backbones. DrTab improves their reported Elo scores on the full 51-dataset TabArena-v0.1 benchmark and generally improves or preserves predictive performance on the AutoML subset and OpenML-CTR23 collection.
Critical strengths
-
Practical motivation. The paper addresses the relevant problem of deciding whether and how much to adapt a pretrained TFM to a downstream task. Automating these decisions could reduce the need for manual tuning.
-
Interesting diagnostic formulation. Correction headroom combines gradient information, local geometry, and disagreement across folds into a signal intended to guide adaptation.
-
Evaluation across multiple backbones. Testing five main TFM backbones provides evidence that the approach is applicable across different architectures.
Critical weaknesses
-
The acceptance rule is incompletely specified. The paper describes assessing aggregate objective reduction and consistency across folds, but does not fully specify the thresholds or operational definition of consistency. This makes the procedure difficult to reproduce.
-
The behavior after rejection is unclear. The method restores the previous parameters and reruns Diagnose. If the diagnostic data and computation are unchanged, this should reproduce the same diagnosis. The paper does not clearly explain what changes in the next proposal or what stopping rule prevents repeated rejection.
-
[Major] The benefit over straightforward fine-tuning with early stopping is not established. Although the paper compares against existing adaptation methods, a controlled comparison with simple LoRA fine-tuning and validation-based early stopping is needed to isolate the value of the diagnostic machinery.
-
The accuracy–runtime tradeoff is not established. The paper does not report sufficient runtime comparisons to determine whether its predictive gains justify the additional diagnostic and acceptance computations, or where DrTab lies on the accuracy–runtime Pareto frontier.
-
The quantitative validity of correction headroom is not directly evaluated. The headroom ablation supports the component’s usefulness, but does not show whether its numerical estimates predict realized improvement. The gradient-energy analysis also does not evaluate the complete headroom formula.
-
[Minor] Averaging raw RMSE across CTR23 datasets complicates interpretation. Because target scales differ across datasets, the aggregate can be dominated by tasks with larger target magnitudes. A scale-independent evaluation would help establish whether the conclusion holds broadly across tasks.
Questions for authors
-
What is the exact acceptance/rejection rule in Revise? Please specify the proposal-screening threshold, the required aggregate objective improvement, and how consistency across folds is defined.
-
What happens after a rejected update? With fixed folds and deterministic computation, restoring the same parameters should yield the same diagnosis. Does the method reduce the learning rate, advance to another candidate, change another part of the state, or terminate? Are the optimizer states also restored?
-
How well does estimated headroom predict actual improvement? A scatter plot across benchmark datasets, with predicted headroom on the x-axis and realized gain on the y-axis, would be informative. Please report correlation coefficients and distinguish one-step gains from gains over the full adaptation trajectory. For a direct quantitative comparison, normalized held-out loss reduction would align with the headroom definition. Evaluating datasets where adaptation is rejected or stopped would also help assess missed improvement opportunities.
-
Do the regression conclusions hold under scale-independent metrics? Could you report predictive (R^2), normalized RMSE, or relative error reductions on CTR23 alongside the current raw-RMSE aggregate?
-
How does DrTab compare with simple fine-tuning and early stopping? A baseline using the same LoRA configuration, backbone, data access, and comparable tuning and compute budgets would help isolate the contribution of diagnosis, step guidance, and acceptance control.
-
What is the accuracy–runtime tradeoff? Could you report total adaptation time and plot predictive performance against runtime across methods and compute budgets, using comparable hardware? This would clarify whether DrTab improves the Pareto frontier.