acceptodds
Under review as a conference paper at ICLR 2027

OrbitTune: Input-Output Task-Orbit Symmetrization for Reliable Fine-Tuning of Tabular Foundation Models

Abstract

Fine-tuning tabular foundation models typically optimizes performance in a single representation of a downstream table, even though feature scales, category codes, class-label IDs, and target units can be arbitrary coordinates of the same prediction task. We propose OrbitTune, an input-output task-orbit objective that fine-tunes models across equivalent task views. OrbitTune consistently reparameterizes query features, context features, context labels, and query targets, and evaluates predictions after decoding to the original output coordinates. This preserves the semantic supervision while encouraging consistent predictions across task representations. We establish fixed-task semantic preservation and characterize Bayes equivariance under a task-orbit symmetric prior. On controlled structural causal model classification tasks, standard fine-tuning exhibits confidence saturation and loss concentration, whereas OrbitTune changes sample-level training dynamics and improves negative log-likelihood and calibration error. Targeted sample-removal and view-replacement ablations support the usefulness of samples that become more variable under OrbitTune. On TabZilla, OrbitTune improves mean accuracy, AUC, and macro-F1 over full-parameter fine-tuning and the evaluated augmentation baselines for both LimiX-16M and TabPFN-2.5. On TabArena, its two backbone variants attain the highest classification Elo point estimates among the compared methods. These results support task-level input-output reparameterization as a practical strategy for adapting tabular foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
TFM, finetune, data augmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cWnMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces OrbitTune, a fine-tuning strategy for tabular foundation models (TFMs) that improves robustness to arbitrary representations of the same supervised prediction task. Tabular datasets can be expressed using different numerical scales, categorical encodings, class-label identifiers, and regression target units without changing the underlying prediction problem. Standard fine-tuning optimizes a model on a single representation, potentially making it sensitive to arbitrary coordinate choices. OrbitTune addresses this by sampling invertible input-output transformations and applying them consistently to query features, labelled context features, context labels, and query targets. Predictions are subsequently decoded into the original output space before computing the supervised loss. This encourages the model to produce consistent semantic predictions across alternative task representations. Experimentally, the method is evaluated using LimiX-16M and TabPFN-2.5 on TabZilla, TabArena, and TALENT, alongside controlled experiments on 100 synthetic structural causal model tasks. Results show improvements in classification performance, regression performance, and calibration.

Critical strengths

The paper identifies an important property of tabular in-context learning: context labels and target representations are part of the model input, so augmentation should operate consistently on the complete supervised task rather than only on individual feature rows.

The proposed method seems architecture-agnostic. This makes it relatively easy to incorporate into existing fine-tuning pipelines.

Critical weaknesses
  1. The central motivation is that equivalent representations of the same supervised task should lead to equivalent predictions after decoding. However, the paper primarily evaluates downstream predictive performance and training dynamics, without directly quantifying whether OrbitTune actually improves this consistency. A natural evaluation would compare predictions under the original and transformed task views, decode both into the same coordinate system, and measure their discrepancy. Without such an analysis, it remains unclear whether the improvements arise specifically from learning task-representation equivariance or from a more general regularization effect of stochastic augmentation.

  2. Although the paper establishes conditional Bayes-equivariance properties for its selected task reparameterizations, it does not systematically investigate other fundamental symmetries of tabular data, particularly feature-column permutation invariance and support-example ordering invariance. Recent work such as ASPIRE (Brahmavar et al.) explicitly addresses permutation invariance over both feature and instance sets, while EquiTabPFN (Arbel et al., NeurIPS 2025) studies target-permutation equivariance and demonstrates the benefits of enforcing this property architecturally.

Questions for authors
  1. The paper reports substantial improvements over standard full-parameter fine-tuning, but standard fine-tuning also shows considerable performance degradation on some benchmarks, particularly for LimiX-16M on TabArena. Were the optimization hyperparameters independently tuned for the baseline and OrbitTune, or were identical configurations imposed throughout?

  2. Rest same as weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer cWnM (1/2)

We thank the reviewer for their careful evaluation and constructive feedback.

W1: Cross-view prediction consistency and regularization

R1: OrbitTune reduces mean decoded cross-view Jensen–Shannon (JS) divergence by 90.8% relative to Base FT in our added diagnostic on LimiX-16M (Zhang et al. [1]) / TabZilla classification tasks (McElfresh et al. [2]). For each model, we compare original-view predictions with predictions under joint input–output reparameterizations, decoded into the original class coordinates. We average JS first within each dataset, then equally across datasets. All three models undergo the same type of joint evaluation transformation; lower JS indicates greater consistency.

ModelMean joint JS ↓
Zero-shot0.00488
Base FT0.03970
OrbitTune (ours)0.00365

Across all TabZilla datasets included in this JS diagnostic, the mean paired difference, JS(OrbitTune) − JS(Base FT), is −0.03605, with a 95% dataset-level paired bootstrap confidence interval of [−0.04705, −0.00067]. Resampling datasets while retaining paired results yields an interval below zero, supporting lower mean divergence than Base FT. OrbitTune also has lower mean JS than zero-shot, but this interval concerns only Base FT.

Base FT increases mean cross-view sensitivity, while OrbitTune avoids that increase. Task-equivalent augmentation can regularize training and improve consistency; these effects are compatible. The diagnostic does not show that consistency causes predictive gains, and lower JS alone does not ensure correct predictions. Held-out benchmarks and component ablations assess predictive performance separately. Our theory establishes fixed-task semantic preservation and conditional Bayes equivariance, without guaranteeing exact equivariance of a finitely trained model.

OrbitTune uses decoded supervised loss without a pairwise consistency penalty. Transformed-view predictions are used only for this diagnostic; the main controlled evaluations use the original task view without additional OrbitTune transformations or prediction averaging at deployment.

W2: Feature-column and support-example ordering invariance

R2: ASPIRE (Brahmavar et al. [3]) treats feature–value pairs and support examples as sets for inference over heterogeneous tabular data; EquiTabPFN (Arbel et al. [4]) architecturally enforces target-permutation equivariance.

OrbitTune complements these approaches through task-level augmentation for fine-tuning existing pretrained TFMs. Each transformation is shared across query features, context features, context labels, and training query targets; predictions are decoded before computing supervised loss. This preserves query–context correspondence and semantic supervision without changing the backbone architecture.

Task equivalence defines valid training views rather than imposing architectural symmetry. Our transformation family (§2.2–2.3) covers numerical feature coordinates, nominal category IDs, class-label IDs, and regression target coordinates.

References

[1] Xingxuan Zhang et al. LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence. arXiv:2509.03505, 2025.

[2] Duncan McElfresh et al. When Do Neural Nets Outperform Boosted Trees on Tabular Data?. NeurIPS Datasets and Benchmarks Track, 2023.

[3] Shreyas Bhat Brahmavar, Yang Li, and Junier Oliva. Towards Universal Neural Inference. arXiv:2508.09100v1, 2025.

[4] Michael Arbel, David Salinas, and Frank Hutter. EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Network. NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer cWnM (2/2)

Q1: Optimization settings and performance degradation under Base FT

R3: The main comparisons used identical optimization hyperparameters within each backbone, without independent tuning: AdamW, learning rate 10⁻⁵, weight decay 0.05, at most 1,000 updates, and validation-based early stopping after 10 evaluations without improvement (Appendix E, Table 3, and Appendix H.4). Base FT and OrbitTune update all trainable backbone parameters with the same pretrained initialization, learning-rate schedule, and optimization budget to evaluate OrbitTune task views under a shared protocol.

We additionally tuned Base FT on three TabArena datasets, using 50 Optuna trials per dataset. All three searches used learning rate [5 × 10⁻⁶, 2 × 10⁻⁴] and weight decay [5 × 10⁻⁶, 5 × 10⁻²]; these ranges include the original learning rate and weight decay. Configurations were selected by validation performance. The table reports held-out test AUC, with the original fixed configurations retained as references:

DatasetZero-shotBase FT, fixedBase FT, tunedOrbitTune, fixed
seismic-bumps0.7840.7480.7680.796
blood-transfusion-service-center0.7330.7040.7260.750
credit_card_clients_default0.7870.7740.7800.789

Tuning improves Base FT by 0.020, 0.022, and 0.006 AUC, respectively, narrowing its gaps to OrbitTune. OrbitTune still exceeds tuned Base FT by 0.028, 0.024, and 0.009 AUC, and also exceeds zero-shot on all three datasets. Thus, tuning accounts for part of the original gaps but does not close them in these searches. These finite searches on three datasets do not establish a benchmark-wide advantage after independent tuning, and the reported point estimates alone do not establish statistical significance.

For LimiX-16M on TabArena, Table 2 reports classification Elo point estimates of 1012.58 for zero-shot, 777.73 for Base FT, 1174.09 for FT + CTGAN, and 1218.42 for OrbitTune. OrbitTune's higher estimates than zero-shot and CTGAN indicate benefits beyond recovering Base FT's degradation. These are local estimates, not official leaderboard scores, and their ordering alone does not establish pairwise statistical significance.

Small validation sets can make early stopping unreliable (Bühler et al. [1]; Appendix E), but this is only a possible contributor to degradation in our runs. The added searches measure sensitivity to learning rate and weight decay without identifying the cause of the remaining decline.

References

[1] Magnus Bühler, Lennart Purucker, and Frank Hutter. Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Models. EurIPS 2025 Workshop on AI for Tabular Data; arXiv:2601.04110, 2026.

Dear Reviewer cWnM,

Thank you again for the time and care you have devoted to reviewing our paper. As the discussion period draws to a close, we would be grateful if you could consider the additional results and clarifications provided in our rebuttal. We hope these additions help address the concerns you raised and provide useful context for your final assessment. Should any questions or concerns remain, we would be happy to provide further clarification.

Reviewer_FG99OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes OrbitTune, a fine-tuning method for tabular foundation models (TFMs) that applies invertible coordinate transformations to both input features and output labels/targets during training, and decodes the predictions back to the original space before computing the loss. The authors provide theoretical analysis and evaluate the method on synthetic SCM tasks as well as real-world benchmarks (TabZilla, TabArena, and TALENT) using LimiX-16M and TabPFN-2.5.

Critical strengths
  1. Well-written paper and simple, zero-overhead method: The paper is clearly written and easy to follow. The idea of regularizing TFM fine-tuning via input-output task-orbit transformations is intuitive and adds no extra inference cost at test time.
  2. Comprehensive experiments across a large set of benchmarks: The authors conduct extensive experiments across 100 synthetic SCM tasks and 190 real-world datasets from TabZilla, TabArena, and TALENT, showing consistent gains over standard full-parameter fine-tuning and data augmentation baselines.
Critical weaknesses
  1. Non-standard way of evaluating TFM: In standard out-of-box usage, TabPFN typically ensemble multiple forward passes across multiple random feature/class permutation, but authors state that they didn't (Appendix H.4, Section 3.5). The zero-shot evaluation would jump in performance with ensemble, and OrbitTune already have used whole dataset for fine-tuning so the return would be diminishing.
Questions for authors
  1. How do the results of Zero-shot and OrbitTune change when following the standard multi-pass ensemble?
  2. How does OrbitTune compare against other parameter-efficient fine-tuning methods (e.g. LoRA, CausalMixFT)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer FG99

We thank the reviewer for their careful evaluation and constructive feedback.

W1 & Q1: Evaluation protocol and inference-time ensembling

R1: OrbitTune's gains persist under each backbone's default inference pipeline. On TabZilla (McElfresh et al. [1]), OrbitTune improves mean accuracy over zero-shot by 2.6 and 3.5 percentage points for LimiX-16M (Zhang et al. [2]) and TabPFN-2.5 (Grinsztajn et al. [3]), respectively, and over full-parameter FT by 4.0 and 4.3 percentage points. Mean AUC and F1 also improve for both backbones.

We followed Section 3.5, except that all three methods used each backbone's default inference pipeline, including ensembling and preprocessing, with no additional OrbitTune task-view sampling at test time. The table reports means across datasets:

BackboneMethodPerformance: Acc.Performance: AUCPerformance: F1
LimiX-16MZero-shot0.8850.9380.836
LimiX-16MFull-parameter FT0.8710.9420.822
LimiX-16MOrbitTune (ours)0.9110.9560.874
TabPFN-2.5Zero-shot0.8830.9430.832
TabPFN-2.5Full-parameter FT0.8750.9460.848
TabPFN-2.5OrbitTune (ours)0.9180.9570.885

As anticipated, the default pipeline raises zero-shot accuracy by 3.1 percentage points for each backbone. OrbitTune's advantage narrows from 5.7 to 2.6 points for LimiX-16M and from 5.4 to 3.5 points for TabPFN-2.5, but remains positive across all three metrics. Because preprocessing and ensembling change together, this comparison does not isolate ensembling alone.

The new evaluation complements our controlled single-model protocol. OrbitTune samples shared task reparameterizations during training and decodes predictions before supervision (Section 2.3; Algorithm 1). It requires no additional task-view transformations or prediction averaging at deployment and is compatible with the backbone's existing ensemble.

Q2: Comparison with LoRA and CausalMixFT

R2: Our controlled comparisons establish gains over full-parameter FT with the backbone, initialization, and optimization settings fixed. Direct comparisons with LoRA and CausalMixFT remain open.

LoRA (Hu et al. [4]) has been explored in prior TFM fine-tuning work: Rubachev et al. [5] (Section 3) explicitly evaluate LoRA on TabPFNv2 and report small performance differences between full FT and the evaluated PEFT variants, with full FT offering faster convergence in their experiments. This supports our choice of full-parameter FT as a strong, simple baseline, but does not establish LoRA's relative performance on the LimiX-16M and TabPFN-2.5 backbones studied here. OrbitTune regularizes task views and is compatible in principle with LoRA's low-rank parameter updates; evaluating LoRA alone and OrbitTune + LoRA remains future work.

CausalMixFT (Bühler et al. [6]) uses structural causal models for data augmentation rather than parameter-efficient adaptation. Its source code was not publicly available at the time of our experiments. Without access to the authors' implementation, we were unable to reliably reproduce the results reported in the paper and therefore did not include CausalMixFT as a baseline (Appendix H.2). Our results support gains over evaluated baselines, including CTGAN and SMOTE.

References

[1] Duncan McElfresh et al. When Do Neural Nets Outperform Boosted Trees on Tabular Data?. NeurIPS Datasets and Benchmarks Track, 2023.

[2] Xingxuan Zhang et al. LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence. arXiv:2509.03505, 2025.

[3] Léo Grinsztajn et al. TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models. arXiv:2511.08667, 2025.

[4] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.

[5] Ivan Rubachev, Akim Kotelnikov, Nikolay Kartashev, and Artem Babenko. On Finetuning Tabular Foundation Models. arXiv:2506.08982, 2025.

[6] Magnus Bühler, Lennart Purucker, and Frank Hutter. Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Models. EurIPS 2025 Workshop on AI for Tabular Data; arXiv:2601.04110, 2026.

Dear Reviewer FG99,

Thank you again for the time and care you have devoted to reviewing our paper. As the discussion period draws to a close, we would be grateful if you could consider the additional results and clarifications provided in our rebuttal. We hope these additions help address the concerns you raised and provide useful context for your final assessment. Should any questions or concerns remain, we would be happy to provide further clarification.

Reviewer_ksqgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OrbitTune, a task-level augmentation method for fine-tuning tabular foundation models. Each update samples an invertible reparameterization of numerical features, nominal category identifiers, classification labels, or regression targets. The same transformation is applied to the query, context, and corresponding targets, and predictions are decoded before computing the loss. Test inference uses the original normalized view without ensembling.

The paper includes semantic-preservation and Bayes-equivariance analyses, training-dynamics diagnostics, and experiments with LimiX-16M and TabPFN-2.5 on synthetic and real classification and regression benchmarks.

Critical strengths
  1. Simple and model-agnostic method. OrbitTune requires no architectural changes and avoids inference-time ensembles. It provides a unified formulation for different feature types and both classification and regression.

  2. Well-motivated joint input–output construction. Transforming the complete query-context task, rather than individual samples alone, is well suited to tabular foundation models. The decoded loss clearly preserves the original prediction target.

  3. Useful empirical evaluation. The experiments cover two backbones and several benchmarks, with promising average improvements over full-parameter fine-tuning. The input/output ablations and training-dynamics analyses provide additional insight.

  4. Clear presentation and appropriate theoretical scope. The method is easy to follow, and the paper appropriately distinguishes semantic preservation and ideal Bayes equivariance from exact equivariance of a finitely trained model.

Critical weaknesses
  1. The effect of task-orbit symmetrization is not fully isolated from generic stochastic regularization. The current comparisons and ablations show useful gains, but a small aligned-versus-misaligned control would better identify the contribution of the task-level construction.

  2. Cross-view equivariance is not measured directly. The training-dynamics and calibration analyses are informative, but they support the method’s effectiveness more directly than its proposed mechanism.

  3. The conditions under which OrbitTune helps remain unclear. In particular, the strong output-only ablation calls for further analysis of when joint input–output reparameterization provides additional benefits and when the gains are limited.

Questions for authors
  1. Can the authors evaluate cross-view prediction consistency using existing checkpoints? On a representative subset of tasks, comparing zero-shot, Base FT, and OrbitTune through decoded predictions would directly test the proposed mechanism without requiring full retraining.

  2. Is sharing the same transformation between query and context important? A small comparison with independently sampled query/context transformations would help isolate the benefit of the shared task view.

  3. Which task properties are associated with larger gains? A reanalysis of existing per-dataset results could clarify when joint reparameterization outperforms output-only augmentation and when it provides little or no additional benefit.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ksqg (1/2)

We thank the reviewer for their careful evaluation and constructive feedback.

W1 & Q2: Task-orbit symmetrization and generic stochastic regularization

R1: Shared transformations preserve query–context correspondence in a common coordinate system, with decoding restoring the original target. On TabArena (Erickson et al. [1]) with LimiX-16M (Zhang et al. [2]), independently sampled query and context transformations caused persistent training-loss oscillations under otherwise identical settings and optimization budgets. Separately, Base FT + Mixup (Zhang et al. [3]) achieved 0.941 AUC, compared with 0.952 for OrbitTune.

These results support shared task views and show that the tested Mixup configuration did not match OrbitTune. However, independently sampled transformations also disrupt query–context correspondence, preventing a clean separation of task-orbit effects from generic regularization. OrbitTune's task-orbit inductive bias, stochastic regularization, and improved consistency can coexist.

W2 & Q1: Direct measurement of cross-view prediction consistency

R2: We evaluated zero-shot, Base FT, and OrbitTune checkpoints of LimiX-16M (Zhang et al. [2]) on TabZilla classification tasks (McElfresh et al. [4]). For each model, we measure Jensen–Shannon (JS) divergence between original-view predictions and predictions under joint input–output transformations, decoded into the original class coordinates. JS is averaged within datasets, then equally across datasets. All checkpoints undergo the same type of joint transformation, regardless of their training procedure.

ModelMean joint JS ↓
Zero-shot0.00488
Base FT0.03970
OrbitTune (ours)0.00365

OrbitTune lowers mean joint JS from 0.03970 to 0.00365, a reduction of 0.03605 (90.8%) relative to Base FT. The mean paired difference, JS(OrbitTune) − JS(Base FT), is −0.03605, with a 95% paired-bootstrap confidence interval of [−0.04705, −0.00067]. The bootstrap resamples datasets, keeping model results paired within each dataset, and covers all TabZilla tasks included in this JS diagnostic, not only binary tasks. The interval excludes zero and captures uncertainty across datasets, not independent training runs.

Relative to zero-shot, Base FT increases mean cross-view sensitivity, while OrbitTune maintains low divergence. This supports improved consistency under the evaluated transformations, without establishing exact equivariance or a causal link to predictive gains. Low JS alone does not ensure correct predictions; held-out benchmarks and component ablations assess predictive performance separately.

OrbitTune uses decoded supervised loss without a pairwise consistency penalty. This diagnostic requires only checkpoint forward passes. The main controlled predictive evaluations use the original task view without additional OrbitTune transformations or prediction averaging at deployment.

References

[1] Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS Datasets and Benchmarks Track, 2025.

[2] Xingxuan Zhang et al. LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence. arXiv:2509.03505, 2025.

[3] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond Empirical Risk Minimization. ICLR, 2018.

[4] Duncan McElfresh et al. When Do Neural Nets Outperform Boosted Trees on Tabular Data?. NeurIPS Datasets and Benchmarks Track, 2023.

AuthorsOpenReview ↗

Response to Reviewer ksqg (2/2)

W3 & Q3: Task-dependent benefits of joint input–output reparameterization

R3: Joint reparameterization improves on the strong Output-only baseline on average. Appendix E.2, Table 4 reports these dataset means on TabZilla (McElfresh et al. [1]) with LimiX-16M (Zhang et al. [2]):

Training variantAcc.AUCMacro-F1
Base FT0.8750.9430.822
Input-only0.8670.9490.848
Output-only0.9000.9440.856
Joint (OrbitTune)0.9110.9550.874

Input-only improves mean AUC by 0.006 and macro-F1 by 0.026 over Base FT, but reduces accuracy by 0.008. Adding input reparameterization to Output-only improves all three means: +0.011 accuracy, +0.011 AUC, and +0.018 macro-F1, supporting its incremental value in the joint formulation.

Exploratory dataset-level inspection suggests larger accuracy gains from output permutation on imbalanced datasets, with additional AUC gains from Joint over Output-only in both imbalance groups.

References

[1] Duncan McElfresh et al. When Do Neural Nets Outperform Boosted Trees on Tabular Data?. NeurIPS Datasets and Benchmarks Track, 2023.

[2] Xingxuan Zhang et al. LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence. arXiv:2509.03505, 2025.

Dear Reviewer ksqg,

Thank you again for the time and care you have devoted to reviewing our paper. As the discussion period draws to a close, we would be grateful if you could consider the additional results and clarifications provided in our rebuttal. We hope these additions help address the concerns you raised and provide useful context for your final assessment. Should any questions or concerns remain, we would be happy to provide further clarification.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Clarifying OrbitTune's Scope and Cross-View Consistency Results

Dear Area Chair and Reviewers,

We thank the reviewers for their careful evaluation and constructive feedback. We clarify the scope of our analysis and summarize the additional evidence below.

Our invariance analysis concerns task preservation during fine-tuning: the sampled reparameterizations preserve the underlying supervised problem. It does not guarantee inference-time prediction consistency of the trained model.

Our supplementary experiments show that OrbitTune improves inference-time cross-view consistency over standard fine-tuning under the evaluated transformations. This finding complements the predictive-performance results but does not establish that greater consistency causes better predictions. Our contribution remains the task-level fine-tuning framework and its analysis; improved inference-time consistency is supporting empirical evidence.

Area_Chair_kCEWOpenReview ↗

Prompt for Discussion

Hi Reviewers,

The authors have replied to each of your reviews. Can you please engage in discussion? Have your views changed or not?

Regards, AC

Thank you for the reminder. We welcome further discussion and would be happy to address any additional questions or concerns from the reviewers.

Dear Area Chair and Reviewers,

Thank you for your careful reading of our paper and for suggesting concrete ways to test OrbitTune. We hope the additional experiments and clarifications in our individual replies address the main concerns you raised.

  • Consistency across equivalent task views. Reviewers cWnM and ksqg asked whether equivalent representations of a task lead to similar predictions. On the TabZilla classification tasks we examined with LimiX-16M, the mean Jensen–Shannon divergence between predictions decoded into the same label space fell from 0.03970 with standard fine-tuning to 0.00365 with OrbitTune. This directly supports improved consistency under the tested transformations.

  • The choice of training transformations. Following Reviewer ksqg’s suggestion, we sampled transformations independently for the query and context. Training loss showed persistent oscillations in this control on LimiX-16M. A separate TabZilla ablation with the same backbone showed that adding input transformations to output-only augmentation improved mean accuracy, AUC, and macro-F1, with gains varying across datasets.

  • Default inference and baseline tuning. Following Reviewer FG99’s suggestion, we evaluated each backbone with its default preprocessing and ensembling. OrbitTune exceeded zero-shot mean accuracy on TabZilla by 2.6 percentage points for LimiX-16M and 3.5 points for TabPFN-2.5. In response to Reviewer cWnM, we also ran a separate hyperparameter search for standard fine-tuning on each of three TabArena datasets. Tuning narrowed the gap, and OrbitTune retained higher test AUC on all three.

We appreciate your time and would welcome any further questions.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.