TabFSA: Feature-to-Strategy Adaptation for Tabular In-Context Learning

Tabular foundation models (TFMs) enable in-context prediction across datasets, but target settings involving temporal transfer, unseen groups, or rare or unseen categorical levels may require targeted adaptation. We study how target data can improve their generalization to new prediction settings without full-model fine-tuning or pretraining a new backbone. We propose Feature-to-Strategy Adaptation (FSA), which uses support–query training tasks representing the target settings to learn a bank of low-rank expert adapters, each specialized to one task family. With the backbone and adapters frozen, controlled comparisons show that favorable adaptation strengths and allocations vary across tasks, and some weighted combinations outperform directly applying any one expert at maximum strength. These observations motivate learning the adaptation strategy from the current task context. FSA constructs a dual-view strategy feature from labeled support examples and unlabeled query inputs, comparing support and query through feature and sample views. We jointly learn this strategy feature and its mapping to adaptation strength and expert allocation by minimizing query prediction loss, enabling task-dependent composition without further parameter optimization on unseen tasks. We instantiate FSA as TabFSA with TabICLv2 and synthetic target training tasks. On BeyondArena, TabFSA raises aggregate Elo from 1178 to 1216 and improves average rank from 4.40 to 3.90. On 200 TALENT classification datasets, it records 136 wins, 15 ties, and 49 losses against the frozen backbone. Retraining FSA on frozen TabPFN also improves prediction, demonstrating applicability across backbones.