- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces OrbitTune, a fine-tuning strategy for tabular foundation models (TFMs) that improves robustness to arbitrary representations of the same supervised prediction task. Tabular datasets can be expressed using different numerical scales, categorical encodings, class-label identifiers, and regression target units without changing the underlying prediction problem. Standard fine-tuning optimizes a model on a single representation, potentially making it sensitive to arbitrary coordinate choices. OrbitTune addresses this by sampling invertible input-output transformations and applying them consistently to query features, labelled context features, context labels, and query targets. Predictions are subsequently decoded into the original output space before computing the supervised loss. This encourages the model to produce consistent semantic predictions across alternative task representations. Experimentally, the method is evaluated using LimiX-16M and TabPFN-2.5 on TabZilla, TabArena, and TALENT, alongside controlled experiments on 100 synthetic structural causal model tasks. Results show improvements in classification performance, regression performance, and calibration.
Critical strengths
The paper identifies an important property of tabular in-context learning: context labels and target representations are part of the model input, so augmentation should operate consistently on the complete supervised task rather than only on individual feature rows.
The proposed method seems architecture-agnostic. This makes it relatively easy to incorporate into existing fine-tuning pipelines.
Critical weaknesses
-
The central motivation is that equivalent representations of the same supervised task should lead to equivalent predictions after decoding. However, the paper primarily evaluates downstream predictive performance and training dynamics, without directly quantifying whether OrbitTune actually improves this consistency. A natural evaluation would compare predictions under the original and transformed task views, decode both into the same coordinate system, and measure their discrepancy. Without such an analysis, it remains unclear whether the improvements arise specifically from learning task-representation equivariance or from a more general regularization effect of stochastic augmentation.
-
Although the paper establishes conditional Bayes-equivariance properties for its selected task reparameterizations, it does not systematically investigate other fundamental symmetries of tabular data, particularly feature-column permutation invariance and support-example ordering invariance. Recent work such as ASPIRE (Brahmavar et al.) explicitly addresses permutation invariance over both feature and instance sets, while EquiTabPFN (Arbel et al., NeurIPS 2025) studies target-permutation equivariance and demonstrates the benefits of enforcing this property architecturally.
Questions for authors
-
The paper reports substantial improvements over standard full-parameter fine-tuning, but standard fine-tuning also shows considerable performance degradation on some benchmarks, particularly for LimiX-16M on TabArena. Were the optimization hyperparameters independently tuned for the baseline and OrbitTune, or were identical configurations imposed throughout?
-
Rest same as weakness
5 comments
Response to Reviewer cWnM (2/2)
Q1: Optimization settings and performance degradation under Base FT
R3: The main comparisons used identical optimization hyperparameters within each backbone, without independent tuning: AdamW, learning rate 10⁻⁵, weight decay 0.05, at most 1,000 updates, and validation-based early stopping after 10 evaluations without improvement (Appendix E, Table 3, and Appendix H.4). Base FT and OrbitTune update all trainable backbone parameters with the same pretrained initialization, learning-rate schedule, and optimization budget to evaluate OrbitTune task views under a shared protocol.
We additionally tuned Base FT on three TabArena datasets, using 50 Optuna trials per dataset. All three searches used learning rate [5 × 10⁻⁶, 2 × 10⁻⁴] and weight decay [5 × 10⁻⁶, 5 × 10⁻²]; these ranges include the original learning rate and weight decay. Configurations were selected by validation performance. The table reports held-out test AUC, with the original fixed configurations retained as references:
| Dataset | Zero-shot | Base FT, fixed | Base FT, tuned | OrbitTune, fixed |
|---|---|---|---|---|
| seismic-bumps | 0.784 | 0.748 | 0.768 | 0.796 |
| blood-transfusion-service-center | 0.733 | 0.704 | 0.726 | 0.750 |
| credit_card_clients_default | 0.787 | 0.774 | 0.780 | 0.789 |
Tuning improves Base FT by 0.020, 0.022, and 0.006 AUC, respectively, narrowing its gaps to OrbitTune. OrbitTune still exceeds tuned Base FT by 0.028, 0.024, and 0.009 AUC, and also exceeds zero-shot on all three datasets. Thus, tuning accounts for part of the original gaps but does not close them in these searches. These finite searches on three datasets do not establish a benchmark-wide advantage after independent tuning, and the reported point estimates alone do not establish statistical significance.
For LimiX-16M on TabArena, Table 2 reports classification Elo point estimates of 1012.58 for zero-shot, 777.73 for Base FT, 1174.09 for FT + CTGAN, and 1218.42 for OrbitTune. OrbitTune's higher estimates than zero-shot and CTGAN indicate benefits beyond recovering Base FT's degradation. These are local estimates, not official leaderboard scores, and their ordering alone does not establish pairwise statistical significance.
Small validation sets can make early stopping unreliable (Bühler et al. [1]; Appendix E), but this is only a possible contributor to degradation in our runs. The added searches measure sensitivity to learning rate and weight decay without identifying the cause of the remaining decline.
References
[1] Magnus Bühler, Lennart Purucker, and Frank Hutter. Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Models. EurIPS 2025 Workshop on AI for Tabular Data; arXiv:2601.04110, 2026.
Dear Reviewer cWnM,
Thank you again for the time and care you have devoted to reviewing our paper. As the discussion period draws to a close, we would be grateful if you could consider the additional results and clarifications provided in our rebuttal. We hope these additions help address the concerns you raised and provide useful context for your final assessment. Should any questions or concerns remain, we would be happy to provide further clarification.
Response to Reviewer cWnM (1/2)
We thank the reviewer for their careful evaluation and constructive feedback.
W1: Cross-view prediction consistency and regularization
R1: OrbitTune reduces mean decoded cross-view Jensen–Shannon (JS) divergence by 90.8% relative to Base FT in our added diagnostic on LimiX-16M (Zhang et al. [1]) / TabZilla classification tasks (McElfresh et al. [2]). For each model, we compare original-view predictions with predictions under joint input–output reparameterizations, decoded into the original class coordinates. We average JS first within each dataset, then equally across datasets. All three models undergo the same type of joint evaluation transformation; lower JS indicates greater consistency.
Across all TabZilla datasets included in this JS diagnostic, the mean paired difference, JS(OrbitTune) − JS(Base FT), is −0.03605, with a 95% dataset-level paired bootstrap confidence interval of [−0.04705, −0.00067]. Resampling datasets while retaining paired results yields an interval below zero, supporting lower mean divergence than Base FT. OrbitTune also has lower mean JS than zero-shot, but this interval concerns only Base FT.
Base FT increases mean cross-view sensitivity, while OrbitTune avoids that increase. Task-equivalent augmentation can regularize training and improve consistency; these effects are compatible. The diagnostic does not show that consistency causes predictive gains, and lower JS alone does not ensure correct predictions. Held-out benchmarks and component ablations assess predictive performance separately. Our theory establishes fixed-task semantic preservation and conditional Bayes equivariance, without guaranteeing exact equivariance of a finitely trained model.
OrbitTune uses decoded supervised loss without a pairwise consistency penalty. Transformed-view predictions are used only for this diagnostic; the main controlled evaluations use the original task view without additional OrbitTune transformations or prediction averaging at deployment.
W2: Feature-column and support-example ordering invariance
R2: ASPIRE (Brahmavar et al. [3]) treats feature–value pairs and support examples as sets for inference over heterogeneous tabular data; EquiTabPFN (Arbel et al. [4]) architecturally enforces target-permutation equivariance.
OrbitTune complements these approaches through task-level augmentation for fine-tuning existing pretrained TFMs. Each transformation is shared across query features, context features, context labels, and training query targets; predictions are decoded before computing supervised loss. This preserves query–context correspondence and semantic supervision without changing the backbone architecture.
Task equivalence defines valid training views rather than imposing architectural symmetry. Our transformation family (§2.2–2.3) covers numerical feature coordinates, nominal category IDs, class-label IDs, and regression target coordinates.
References
[1] Xingxuan Zhang et al. LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence. arXiv:2509.03505, 2025.
[2] Duncan McElfresh et al. When Do Neural Nets Outperform Boosted Trees on Tabular Data?. NeurIPS Datasets and Benchmarks Track, 2023.
[3] Shreyas Bhat Brahmavar, Yang Li, and Junier Oliva. Towards Universal Neural Inference. arXiv:2508.09100v1, 2025.
[4] Michael Arbel, David Salinas, and Frank Hutter. EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Network. NeurIPS, 2025.