OrbitTune: Input-Output Task-Orbit Symmetrization for Reliable Fine-Tuning of Tabular Foundation Models

Fine-tuning tabular foundation models typically optimizes performance in a single representation of a downstream table, even though feature scales, category codes, class-label IDs, and target units can be arbitrary coordinates of the same prediction task. We propose OrbitTune, an input-output task-orbit objective that fine-tunes models across equivalent task views. OrbitTune consistently reparameterizes query features, context features, context labels, and query targets, and evaluates predictions after decoding to the original output coordinates. This preserves the semantic supervision while encouraging consistent predictions across task representations. We establish fixed-task semantic preservation and characterize Bayes equivariance under a task-orbit symmetric prior. On controlled structural causal model classification tasks, standard fine-tuning exhibits confidence saturation and loss concentration, whereas OrbitTune changes sample-level training dynamics and improves negative log-likelihood and calibration error. Targeted sample-removal and view-replacement ablations support the usefulness of samples that become more variable under OrbitTune. On TabZilla, OrbitTune improves mean accuracy, AUC, and macro-F1 over full-parameter fine-tuning and the evaluated augmentation baselines for both LimiX-16M and TabPFN-2.5. On TabArena, its two backbone variants attain the highest classification Elo point estimates among the compared methods. These results support task-level input-output reparameterization as a practical strategy for adapting tabular foundation models.