- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes QuTab, a 2B parameter tabular foundation model for in context learning. Different from existing tabular Transformers which maintain many mutable cell- or row-wise representations updated across layers, QuTab enforces read-only context cell encodings. All recurrent computation is concentrated inside one single mutable query-label state per query example, forming a causal bottleneck for mechanistic interventions. Two core components are introduced: Directed Tabular Attention (DTA) which jointly attends to context cells, query self-features, and its own layer history within one shared Softmax; and Structural RoPE, an extended rotary positional embedding that encodes row, column and label structural identities.
Critical strengths
- Prior tabular ICL models yield good predictive performance, but their computation is distributed over hundreds of mutable cell/row states, making mechanistic causal interpretability extremely difficult. QuTab introduces an architectural constraint: freeze context cells and route all evolving prediction-related computation through one query-label residual bottleneck. This provides a well defined intervention site for interpretability techniques.
- Directed Tabular Attention unifies reading context, self features and layer history in a single Softmax, together with sink controllers to stabilize attention for large tables. Structural RoPE binds row/column/label identity into key-value representations, allowing attention heads to discriminate cells by structural position beyond raw feature values.
- The paper does not rely only on post-hoc attribution methods such as SHAP or static attention heatmaps. It performs global probing, held-out steering, component-level activation patching, and validates a sparse causal readout circuit with both counterfactual transfer and complement resample sufficiency ablation.
Critical weaknesses
- QuTab reaches 69.5% R^2, while TabICLv2 and TabPFN 2.6 achieve near 80%. The authors hypothesize this comes from the inductive bias of their sparse readout circuit: heads tend to selectively retrieve small subsets of context rows instead of broadly averaging many context targets to reduce variance for regression. Though the hypothesis is reasonable, more empirical investigation is desirable. For real world tabular use cases, regression is equally important as classification.
- All circuit discovery, steering and global probing causal evidence comes from synthetic controlled tables. It remains unclear whether the same sparse circuit behavior and steering transfer properties persist on real world physical tabular datasets.
- Since context cells are strictly read only, the model cannot execute iterative cell to cell communication among context entries. The paper mentions this limitation in Section 5, but it would benefit from small scale-controlled experiments demonstrating what kinds of tasks suffer performance degradation from this constraint.
Questions for authors
- QuTab reaches 69.5% R^2, while TabICLv2 and TabPFN 2.6 achieve near 80%. The authors hypothesize this comes from the inductive bias of their sparse readout circuit: heads tend to selectively retrieve small subsets of context rows instead of broadly averaging many context targets to reduce variance for regression. Though the hypothesis is reasonable, more empirical investigation is desirable. For real world tabular use cases, regression is equally important as classification.
- All circuit discovery, steering and global probing causal evidence comes from synthetic controlled tables. It remains unclear whether the same sparse circuit behavior and steering transfer properties persist on real world physical tabular datasets.
- Since context cells are strictly read only, the model cannot execute iterative cell to cell communication among context entries. The paper mentions this limitation in Section 5, but it would benefit from small scale-controlled experiments demonstrating what kinds of tasks suffer performance degradation from this constraint.
Response to Reviewer 8uqx
We thank you for taking the time to write a constructive and helpful review! It helped us to improve the paper.
We agree that the regression gap deserves further investigation. We have added row-mean removal and learned row-summary ablations, as discussed under “Regression gap” in the shared response and at L496–505 (Table 2, L508–516). These results provide more evidence for our hypothesis.
We have now tested the unchanged 14-head circuit on all 13 real TabArena-Lite regression datasets, comparing real context labels with permuted labels. Mean transfer is 98%, exceeding the random-circuit controls on every dataset (L1405–1409; Table A8, L1427).
We also want to point out that our aim for this paper is to show that the model is suitable for mechanistic interpretability; understanding how it solves particular real tables would be a follow-up. Additionally, like many other tabular foundation models, QuTab maps categorical text to numerical identifiers without processing its semantics. In this respect, real and synthetic tables look similar in terms of their representations inside the model (L1852–1859 and L1875–1877).
We believe that identifying such tasks would require a broad search and manual prediction error analysis across many datasets, which we had to defer due to the time and resources required. We do not think that this question can be assessed from a purely theoretical standpoint. We are open to performing additional small-scale controlled experiments as mentioned, but kindly ask if you could clarify which concrete experiment you have in mind.