QuTab: A Single Mutable-State Transformer for Mechanistically Interpretable Tabular In-Context Learning

Tabular foundation models have recently emerged as a promising approach to prediction across diverse datasets, as they can adapt to new tables from labeled examples without dataset-specific training. They obtain expressive in-context learning capabilities by repeatedly updating a large grid of cell or row representations throughout the network, creating indirect information paths and making their internal information flow unlike the token-wise residual-stream computation studied in language models. Instead, they distribute the evolving computation across many states, complicating mechanistic analysis and limiting direct use of the interpretability toolkit developed for such architectures. We introduce QuTab, a 2B-parameter tabular foundation model in which the context table acts as read-only memory and all layerwise computation is carried by a single recurrent prediction state, which we call the query-label state. Context representations remain fixed throughout the Transformer, while the query-label state repeatedly attends to the context and updates itself. This concentrates the model's evolving computation into one residual trajectory, making layerwise information flow easier to isolate and analyze. We realize this architecture by introducing Directed Tabular Attention (DTA), a new attention mechanism that allows the query-label state to simultaneously attend to context cells, same-row query features, and its previous layers' hidden states. Adapted rotary positional encodings (RoPE) encode column, row, and label identity during this joint attention operation, a mechanism we call Structural RoPE. We demonstrate global probing, activation patching, and steering through this residual stream, and localize a sparse causal readout circuit. On TabArena-Lite, QuTab reaches 87.0% accuracy and 69.5% $R^2$, demonstrating that a model built around a single mechanistically accessible residual stream can recover much of the predictive strength of specialized table-native foundation models alongside favorable compute scaling.