Interpretable by Design Tabular Prediction via In-Context Local Linear Learning

In-context learning (ICL) based foundation models have shifted how we make predictions on tabular data: instead of training dataset-specific models, they make predictions by attending to labeled context samples. Pretrained on many tabular tasks, these models achieve strong accuracy, but their attention-based decisions are difficult to interpret in terms of feature contributions. Post-hoc methods can assign feature importance scores, yet those scores do not determine the prediction. We introduce ${\mathrm{TICL}}^3$ (*$\mathbf{T}$abular In-Context $\mathbf{L}$ocal $\mathbf{L}$inear $\mathbf{L}$earning*), an interpretable-by-design predictor that uses a tabular foundation model to guide an explicit prediction rule. ${\mathrm{TICL}}^3$ first learns a shared linear model from the foundation model's predictions. For each query, it retrieves similar training rows in the foundation model's representation space and fits a second linear model to the residuals of the shared model in that neighborhood. The two models together make the final prediction, and their combined coefficients determine each feature's contribution to its query-specific linear score. Across 386 classification benchmark–dataset evaluations, ${\mathrm{TICL}}^3$ trails $TabICLv2$ by 1.1 percentage points and beats CatBoost on 266 of them. It improves average support F1 over MAPLE by 4.7 points and leads both real-data selected-feature utility metrics.