Row Geometry for Tabular In-Context Learning: What Do the Models Use?

Tabular foundation models perform in-context prediction by relating queries to a labeled support set, across tables whose columns differ in number, units, and meaning, so that relations between rows provide a shared structure. We study whether support-to-query prediction can be carried by these relations alone, and which part the models use. Our approach constructs the row Gram matrix of each task from the raw input features standardized with support statistics, and uses it as a row operator that propagates support labels to queries. This formulation connects tabular in-context learning with label propagation, admits several choices of geometry and propagation, and allows changing the row operator while the trained model stays fixed. Across six architectures, the Gram matrix can replace direct access to the raw input features, and changing only the row operator shows that these models rely on first-order propagation along correctly attached support–query relations. We further characterize the Gram matrix as the maximal orthogonal invariant of the normalized rows, show that it remains predictive for classical predictors on 25 OpenML tables, and find closely related internal row structure in frozen TabPFN and TabICL; adding it to TabPFN is neutral on natural tables and helps where the frozen model is weak. These results identify row geometry as a useful object for understanding and designing in-context learning over tabular data.