Accelerating Tabular Foundation Models via Residual-Aware Context Compression

Tabular foundation models (TFMs) enable in-context learning over tabular data by conditioning on a labeled training table without task-specific backbone updates. However, their inference cost grows with context size because each query attends to the training rows, leading to substantial latency and memory overhead on large datasets. In this paper, we study context compression for TFM inference and identify three fundamental issues when reducing the context: (1) insufficient preservation of predictive signals for the frozen TFM, (2) the overhead of incorporating supervision from omitted rows, and (3) the lack of a unified compression framework across frozen TFMs and downstream tasks. To address these issues, we propose Residual-Aware Context Compression (RCC), which compiles a large training table into a compact reusable artifact through a one-time offline process. RCC first selects an initial set of context rows, then uses prediction residuals under this context to identify additional rows with complementary predictive information. Afterwards, it fits a closed-form residual readout using the remaining rows to correct predictions under the compact context. Together, RCC preserves key information from the full context and generalizes across different TFMs and downstream tasks. We evaluate RCC on three frozen TFMs and 100 classification and 29 regression datasets. At $128\times$ compression, RCC retains 96.5%-97.5% of full-context classification macro-F1, with regression nRMSE increasing by 8.3-8.6 points, while achieving median single-query speedups of $2.56$-$15.61\times$. On 32 large-scale datasets with up to one million rows, our lightweight variant, RCC-lite, maintains comparable predictive performance at $128\times$ compression ratio, achieving single-query speedups of up to $1,514\times$ over the full-context baseline.