How Context Shapes Internal Representations in Tabular Foundation Models

Tabular foundation models make predictions by learning statistical structure from examples provided in context. However, little is known about how this structure is represented internally or how it varies with the amount of examples provided. We study this using controlled synthetic inference problems. These problems contain exact reference posteriors whose geometry varies along known properties, such as covariance orientation, correlation, skewness, and mode separation. Across TabPFN V2, V2.5, V3, and TabICL, these properties are recoverable from low-dimensional hidden-state subspaces. We discover that intervening along the same subspaces causally changes model predictions. Copying only the task-subspace component of a query state from one context into another reproduces 95–99% of the resulting change in the final prediction, while the remaining component reproduces almost none. Small encoded shifts produce substantial movement in the intended direction, which is increased when the model is provided with longer context. Larger requested shifts are only partially realized at intermediate depths. We find that more context makes the controlled settings easier to decode from hidden states. Finally, training short-context models to match long-context task coordinates successfully moves their representations toward the long-context targets, but does not consistently improve prediction beyond training on long-context outputs alone. Output-only training can even reduce prediction error while moving the task coordinates away from those targets. We conclude that representations can be both decodable and causally steerable without providing additional predictive value as training targets.