acceptodds
Under review as a conference paper at ICLR 2027

QuTab: A Single Mutable-State Transformer for Mechanistically Interpretable Tabular In-Context Learning

Abstract

Tabular foundation models have recently emerged as a promising approach to prediction across diverse datasets, as they can adapt to new tables from labeled examples without dataset-specific training. They obtain expressive in-context learning capabilities by repeatedly updating a large grid of cell or row representations throughout the network, creating indirect information paths and making their internal information flow unlike the token-wise residual-stream computation studied in language models. Instead, they distribute the evolving computation across many states, complicating mechanistic analysis and limiting direct use of the interpretability toolkit developed for such architectures. We introduce QuTab, a 2B-parameter tabular foundation model in which the context table acts as read-only memory and all layerwise computation is carried by a single recurrent prediction state, which we call the query-label state. Context representations remain fixed throughout the Transformer, while the query-label state repeatedly attends to the context and updates itself. This concentrates the model's evolving computation into one residual trajectory, making layerwise information flow easier to isolate and analyze. We realize this architecture by introducing Directed Tabular Attention (DTA), a new attention mechanism that allows the query-label state to simultaneously attend to context cells, same-row query features, and its previous layers' hidden states. Adapted rotary positional encodings (RoPE) encode column, row, and label identity during this joint attention operation, a mechanism we call Structural RoPE. We demonstrate global probing, activation patching, and steering through this residual stream, and localize a sparse causal readout circuit. On TabArena-Lite, QuTab reaches 87.0% accuracy and 69.5% , demonstrating that a model built around a single mechanistically accessible residual stream can recover much of the predictive strength of specialized table-native foundation models alongside favorable compute scaling.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Tabular Foundation Models, In-Context Learning, Transformers, Activation Patching, Circuit Analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8uqxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes QuTab, a 2B parameter tabular foundation model for in context learning. Different from existing tabular Transformers which maintain many mutable cell- or row-wise representations updated across layers, QuTab enforces read-only context cell encodings. All recurrent computation is concentrated inside one single mutable query-label state per query example, forming a causal bottleneck for mechanistic interventions. Two core components are introduced: Directed Tabular Attention (DTA) which jointly attends to context cells, query self-features, and its own layer history within one shared Softmax; and Structural RoPE, an extended rotary positional embedding that encodes row, column and label structural identities.

Critical strengths
  1. Prior tabular ICL models yield good predictive performance, but their computation is distributed over hundreds of mutable cell/row states, making mechanistic causal interpretability extremely difficult. QuTab introduces an architectural constraint: freeze context cells and route all evolving prediction-related computation through one query-label residual bottleneck. This provides a well defined intervention site for interpretability techniques.
  2. Directed Tabular Attention unifies reading context, self features and layer history in a single Softmax, together with sink controllers to stabilize attention for large tables. Structural RoPE binds row/column/label identity into key-value representations, allowing attention heads to discriminate cells by structural position beyond raw feature values.
  3. The paper does not rely only on post-hoc attribution methods such as SHAP or static attention heatmaps. It performs global probing, held-out steering, component-level activation patching, and validates a sparse causal readout circuit with both counterfactual transfer and complement resample sufficiency ablation.
Critical weaknesses
  1. QuTab reaches 69.5% R^2, while TabICLv2 and TabPFN 2.6 achieve near 80%. The authors hypothesize this comes from the inductive bias of their sparse readout circuit: heads tend to selectively retrieve small subsets of context rows instead of broadly averaging many context targets to reduce variance for regression. Though the hypothesis is reasonable, more empirical investigation is desirable. For real world tabular use cases, regression is equally important as classification.
  2. All circuit discovery, steering and global probing causal evidence comes from synthetic controlled tables. It remains unclear whether the same sparse circuit behavior and steering transfer properties persist on real world physical tabular datasets.
  3. Since context cells are strictly read only, the model cannot execute iterative cell to cell communication among context entries. The paper mentions this limitation in Section 5, but it would benefit from small scale-controlled experiments demonstrating what kinds of tasks suffer performance degradation from this constraint.
Questions for authors
  1. QuTab reaches 69.5% R^2, while TabICLv2 and TabPFN 2.6 achieve near 80%. The authors hypothesize this comes from the inductive bias of their sparse readout circuit: heads tend to selectively retrieve small subsets of context rows instead of broadly averaging many context targets to reduce variance for regression. Though the hypothesis is reasonable, more empirical investigation is desirable. For real world tabular use cases, regression is equally important as classification.
  2. All circuit discovery, steering and global probing causal evidence comes from synthetic controlled tables. It remains unclear whether the same sparse circuit behavior and steering transfer properties persist on real world physical tabular datasets.
  3. Since context cells are strictly read only, the model cannot execute iterative cell to cell communication among context entries. The paper mentions this limitation in Section 5, but it would benefit from small scale-controlled experiments demonstrating what kinds of tasks suffer performance degradation from this constraint.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 8uqx

We thank you for taking the time to write a constructive and helpful review! It helped us to improve the paper.

W1 / Q1. “Though the hypothesis is reasonable, more empirical investigation is desirable.”

We agree that the regression gap deserves further investigation. We have added row-mean removal and learned row-summary ablations, as discussed under “Regression gap” in the shared response and at L496–505 (Table 2, L508–516). These results provide more evidence for our hypothesis.

W2 / Q2. “It remains unclear whether the same sparse circuit behavior and steering transfer properties persist on real world physical tabular datasets.”

We have now tested the unchanged 14-head circuit on all 13 real TabArena-Lite regression datasets, comparing real context labels with permuted labels. Mean transfer is 98%, exceeding the random-circuit controls on every dataset (L1405–1409; Table A8, L1427).

We also want to point out that our aim for this paper is to show that the model is suitable for mechanistic interpretability; understanding how it solves particular real tables would be a follow-up. Additionally, like many other tabular foundation models, QuTab maps categorical text to numerical identifiers without processing its semantics. In this respect, real and synthetic tables look similar in terms of their representations inside the model (L1852–1859 and L1875–1877).

W3 / Q3. “The paper mentions this limitation in Section 5, but it would benefit from small scale-controlled experiments demonstrating what kinds of tasks suffer performance degradation from this constraint.”

We believe that identifying such tasks would require a broad search and manual prediction error analysis across many datasets, which we had to defer due to the time and resources required. We do not think that this question can be assessed from a purely theoretical standpoint. We are open to performing additional small-scale controlled experiments as mentioned, but kindly ask if you could clarify which concrete experiment you have in mind.

Reviewer_LmR7OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces QuTab, a 2B-parameter tabular foundation model designed to facilitate mechanistic interpretability through a single mutable query-label state. Unlike conventional tabular Transformers that repeatedly update cell-level or row-level representations, QuTab maintains immutable cell representations and concentrates recurrent prediction-side computation into one residual stream per query. The architecture combines Directed Tabular Attention (DTA), Structural RoPE, and history attention to support structured retrieval from context cells, query features, and previous hidden states.

The authors evaluate both predictive performance and mechanistic interpretability. QuTab achieves 87.0% classification accuracy and 69.5% regression R² on TabArena-Lite. The interpretability analysis includes global probing across independent tables, activation steering, full-state and component-level activation patching, and the discovery of a 14-head sparse readout circuit. The identified circuit demonstrates strong counterfactual transfer and normalized sufficiency on controlled regression tasks. The paper also provides an analytical study of computational scaling.

Critical strengths

1. A meaningful architectural approach to mechanistically interpretable tabular foundation models.

The paper addresses an important and relatively underexplored problem: how to design tabular foundation models whose internal computations can be studied through a well-defined residual-stream interface.

The proposed separation between immutable cell representations and a single evolving query-label state is conceptually appealing. It reduces the number of dynamically updated representations while preserving direct access to individual cells. Together with DTA and Structural RoPE, this provides a flexible mechanism for structured memory access.

I find the motivation compelling, particularly because interpretability is incorporated into the architectural design rather than considered exclusively through post-hoc analysis.

2. Extensive and carefully designed mechanistic experiments.

The interpretability evaluation is a major strength of the paper. The authors go beyond correlational analysis by combining global probing, held-out steering, activation patching, and component-level causal interventions.

The appendices provide important methodological details and controls, including independent train/validation/test splits for global probing, shuffled-target controls, randomized steering directions, and held-out evaluations of the selected sparse circuit.

In particular, the 14-head circuit achieves 95.9% counterfactual transfer, 0.990 normalized donor-output fidelity, and 0.909 normalized sufficiency on the held-out additive–multiplicative contrast. The authors also evaluate its behavior under alternative target laws, unrelated tables, and changed queries. These experiments provide meaningful causal evidence and help establish the scope of the proposed readout mechanism.

The additional classification experiments in Appendix A.7 further strengthen the evidence that the residual-stream interface supports multiple mechanistic analysis techniques across prediction tasks.

Critical weaknesses

1. Insufficient ablation studies and incomplete characterization of causal information flow.

QuTab combines several architectural components, including the single mutable-state design, Structural RoPE, explicit structural carriers, and history attention. However, the paper lacks controlled ablation studies that isolate their individual contributions to predictive performance and mechanistic interpretability. Consequently, it remains unclear which components are essential to the reported interpretability benefits.

Furthermore, the interpretation of layerwise activation patching could be better justified. Since historical K/V states remain accessible through MoDA, replacing the current query-label state may not fully isolate the influence of earlier computations. Without clarifying how historical states are handled during patching, it is difficult to determine to what extent the observed late-layer recovery reflects the causal contribution of the current residual state rather than information retained through history attention.

2. Insufficient empirical validation of predictive and computational trade-offs.

QuTab exhibits a substantial regression performance gap compared with TabPFNv2.6 and TabICLv2. Although the authors hypothesize that sparse target retrieval may limit the broader context aggregation required for regression, this explanation is not directly supported by controlled experiments. As a result, the relationship between the proposed architecture, its learned retrieval behavior, and the observed regression degradation remains unclear.

Additionally, the efficiency analysis primarily relies on theoretical attention complexity and forward FLOP estimates, without empirical comparisons of inference latency, throughput, or peak GPU memory usage. While the theoretical scaling properties are promising, the practical computational advantages of QuTab over existing tabular foundation models remain insufficiently demonstrated.

Questions for authors

Q1. Contribution of individual architectural components.

Could the authors provide controlled ablations of the single mutable-state constraint, Structural RoPE, explicit structural carriers, and history attention? In particular, how would removing the label carrier or replacing Structural RoPE with simpler structural embeddings affect global probing, steering, and circuit-level causal intervention results?

Q2. Interpretation of layerwise activation patching under history attention.

Appendix A.5 reports that full-state activation patching becomes substantially more effective in later layers. However, since QuTab retains historical K/V projections of earlier query-label states, patching the current residual state may leave prediction-relevant information accessible through the history branch.

Could the authors clarify how previously stored history K/V states are handled during full-state patching? Additionally, would the layerwise recovery curves change if the current query-label state and the previously stored history K/V were jointly patched from the donor run?

Q3. Understanding the regression performance gap.

QuTab exhibits a substantial regression performance gap compared with TabPFNv2.6 and TabICLv2. In Appendix D.3, the authors hypothesize that sparse target retrieval may limit the broader context aggregation needed for regression. However, this hypothesis has not been directly validated.

Could the authors provide additional experimental evidence supporting the relationship between selective context-label retrieval and regression performance?

Q4. Practical inference efficiency.

Appendix E.2 provides a theoretical analysis of attention complexity and forward FLOPs, demonstrating favorable scaling with context size under fixed query and feature counts. However, theoretical computational savings do not necessarily translate into improved inference efficiency.

Could the authors provide empirical comparisons of inference latency, throughput, and peak GPU memory usage against representative tabular foundation models under comparable hardware and input configurations? Such results would help establish whether the proposed architecture offers practical efficiency advantages beyond its theoretical complexity properties.

1 comment
AuthorsOpenReview ↗

Response to Reviewer LmR7

We thank you for taking the time to write a constructive and helpful review! It helped us to improve the paper.

W1 / Q1. “it remains unclear which components are essential to the reported interpretability benefits.”

We have now added predictive ablations at L490–505 (Table 2, L508–516). The information flow and direct cell access are what make the model accessible for mechanistic interpretability. The other components mainly help retain predictive performance under this constraint. Replacing the label rotation and carrier with an additive target embedding changes mean regression R2R^2 from 64.53% to 62.24% (L2660–2665; Table 2, L508–516). An additional reason to retain Structural RoPE over this simpler additive approach is that it keeps content, row, column, and label in separate parts of its attention vectors. We can therefore inspect why a head reads a particular cell—whether its attention score responds to the cell's content, row, column, or label (L2671–2675; Figure 12, L1670–1673). We have also worked on explaining the information flow more clearly throughout the paper.

Q2. “Could the authors clarify how previously stored history K/V states are handled during full-state patching?”

The original intervention retained recipient history. We now also patch the donor history, as requested, and describe the result under “History-aware patching” in the shared response and at L1184–1187 and L1198–1199 (Table A5, L1189–1195).

W2 / Q3. “Could the authors provide additional experimental evidence supporting the relationship between selective context-label retrieval and regression performance?”

We prioritized this question in our revision. We added the row-mean removal and stronger row-summary experiments to investigate this concern, as discussed in the shared response under “Regression gap” and in the paper at L496–505 and L2850–2854.

W2 / Q4. “Could the authors provide empirical comparisons of inference latency, throughput, and peak GPU memory usage”

We compare QuTab with TabPFNv2, TabICLv2, TabFM, and EXAONE Tabular on the same TabArena-Lite fold-0 datasets and one RTX PRO 6000 Blackwell GPU, with at most 10,240 total rows, 100 features, and 200 queries. We report one-query latency, throughput for up to 200 queries, and peak GPU memory (L3053–3060; Table F3, L3063–3071).

QuTab reaches 0.120 s one-query latency, 435 queries/s with up to 200 queries, and 8.43 GiB peak GPU memory. These values are in the same general range as the comparison models. QuTab is close to the fastest model at one query, but trails three of the four comparisons in throughput at up to 200 queries. This shows that our favorable row scaling holds only for fixed query counts in the regime where Q << N. Workloads with many queries can be less favorable for QuTab.

Reviewer_bbKXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes QuTab, a 2B-parameter tabular in-context learner. Each cell is encoded once and never updated. All recurrent computation happens in a single query-label state per query row. This state reads the fixed cells, its own row's features, and its earlier-layer states through one joint softmax, which the authors call Directed Tabular Attention. Random-phase rotations, called Structural RoPE, encode row, column and label identity. The authors argue this gives a clean residual stream for mechanistic analysis. They show probing, steering, activation patching and a 14-head readout circuit, mostly on synthetic regression tables. On TabArena-Lite the model reaches 87.0% accuracy and 69.5% R2.

Critical strengths

The design is clean, and the released code has a well-built set of interpretability hooks. Because cells never change, statements about which cell a head reads refer to the actual input, which is useful in practice.

The intervention experiments are carefully set up, with held-out pairs, random controls and sufficiency tests. The authors also report negative results openly, for example that the circuit fails when the query changes.

Critical weaknesses
  1. The main claim is not convincingly supported. A single per-query stream that no other state reads already exists in TabICL and TabPFN, because test rows only attend to training rows. Table 1 marking TabICLv2 as lacking a causal bottleneck is therefore questionable. QuTab's own state is also not a single bottleneck. Every block reads a cache of earlier-layer states, and some heads put over 90% of their attention there. In the released code, "full-state" patching replaces only the residual and leaves this cache untouched. Finally, all baseline entries in Table 1 come from other papers with different protocols. TabICLv2 is open, so the same experiments could have been run on it, and without that the comparison does not show that the architecture is what makes the analysis work.

  2. Novelty is limited. Queries attending to a fixed context representation is the core design of Attentive and Transformer Neural Processes and of Perceiver IO. Reusing one KV memory across layers resembles YOCO. The history branch, the attention sink and the Fourier embedding are taken from MoDA, gpt-oss and TabFM. Structural RoPE with an unrotated query is essentially a fixed random identity code. Closely related 2026 work is not discussed: KernelICL makes TabPFN and TabICL predictions explicit weighted averages of training labels with little loss in accuracy, and "How TabPFN Learns" already finds causal retrieval heads in TabPFN.

  3. Predictive performance is weak for a 2B model, and nothing explains why. Regression R2 is more than 10 points below every table-native baseline, with large gaps on concrete, airfoil and QSAR-TID-11. There are no ablations of any proposed component and no compute-matched comparison against a standard model. Baseline numbers are copied from other papers, the evaluation uses a single split, and wide datasets are cut to 100 random features. The abstract claims favorable compute scaling, but Appendix E.2 shows TabICLv2 is cheaper across the entire range QuTab supports.

  4. Some mechanistic evidence is internally inconsistent or confounded. Replacing the whole state before layer 20 has almost no effect, yet steering at layers 4 to 8 recovers about 45% of the same gap. The steering vectors are as large as the residual itself, so the effect may simply be off-distribution. The probe for the product ab is taken at the last layer, where ab equals the prediction minus a query feature, so it need not reflect a real intermediate. On the code side, categorical features are encoded by order of appearance, which the paper does not mention and which makes predictions depend on row order. The label permutation seed also differs from the paper, and no checkpoints or experiment scripts are provided, so none of the results can be reproduced.

Questions for authors
  1. Can you run the same probing, steering, patching and circuit tests on TabICLv2's test-row stream? This would directly test whether the architecture is the reason the analysis works.
  2. What happens in Table 4 if the history cache is patched together with the residual?
  3. How do you reconcile weak full-state patching with strong steering at the same layers? What does single-layer steering give at a matched total perturbation?
  4. Please add ablations of the main components and a compute-matched baseline trained on the same prior. Which component causes the regression gap?
  5. Were the baseline results obtained with ensembling or with all features? How does QuTab do with all features, with ensembling, and over several splits?
4 comments
AuthorsOpenReview ↗

Response to Reviewer bbKX, part 1

We thank you for taking the time to write a constructive and helpful review! It helped us to improve the paper.

W1. “A single per-query stream that no other state reads already exists in TabICL and TabPFN, because test rows only attend to training rows.”

We agree that TabICLv2 has a test-row stream that reads training rows; our original presentation did not make clear why that alone does not satisfy the property in Table 1. We define a causal bottleneck as “one evolving prediction state through which all prediction-relevant information must pass” (L361–362). QuTab has this property because its “context and self-feature representations are encoded once,” while “Only the query-label state receives recurrent DTA and SwiGLU updates” (L901–904). Even information saved in history reaches the prediction only when the current query-label state reads it (L904–906). TabICLv2 instead “evolves multiple row representations during dataset-level computation” (L900–901). Those context rows can therefore compute relationships before the test-row readout, outside the proposed intervention site; TabPFNv2 likewise updates multiple feature-group states (L898–899). This is why Table 1 marks the architectural bottleneck as not fulfilled for these models (L341–350).

W1. “Every block reads a cache of earlier-layer states, and some heads put over 90% of their attention there.”

We address this argument and the new no-history ablation together under “Causal bottleneck and history” in the shared response. The cache retains earlier computations from the same query state and, as such, does not introduce an independently evolving computation stream. Information still has to flow through the single query state; the model can only temporarily “forget” information and read it back later.

Q2. “What happens in Table 4 if the history cache is patched together with the residual?”

We agree that the original experiment left the influence of retained history unclear. We have now performed the joint intervention, as described under “History-aware patching” in the shared response and at L1184–1187 and L1198–1199 (Table A5, L1189–1195). This joint intervention improves or at least retains the previously reported patching results.

Q1. “Can you run the same probing, steering, patching and circuit tests on TabICLv2's test-row stream?”

We acknowledge that it is possible to run experiments on probing, steering, patching and circuit analysis on TabICLv2. However, differences in probing or intervention results would not by themselves isolate the effect of the architecture, because the different models may implement different ways to solve the task in their weights. We had to defer these experiments due to the required time and compute. The presented experiments establish mechanistic accessibility in QuTab. We explain that QuTab’s architectural properties make the interventions well defined (L361–367; L901–907) and allow a seamless application of the toolkit developed for large language models (L68–71, L79–85). Existing interpretability research on tabular foundation models has often found difficulties applying these techniques (L371–374, L383–386, L411–413). A more direct comparison will be future work.

W2. “Queries attending to a fixed context representation is the core design of Attentive and Transformer Neural Processes and of Perceiver IO.”

We have now included them in the related work. They use computation in context or latent representations before the target read, whereas our focus is a fixed cell representation and a single query state for table prediction (L125–130).

W2. “Reusing one KV memory across layers resembles YOCO. [...] Structural RoPE with an unrotated query is essentially a fixed random identity code.”

We have included YOCO at L128–130. YOCO is a memory optimization, and its cache is produced by an evolving token stack; QuTab’s cell representations remain fixed through depth, making reuse an information-flow restriction and compute optimization. We already credit MoDA, the attention sink, and the adopted embeddings. For RoPE, the carrier identities are fixed codes, but the bound pathway couples structure to content and the regression-label encoding is continuous and target-dependent (L1942–1947, L1977–1995, and L2000–2006).

W2. “Closely related 2026 work is not discussed”

We agree that KernelICL is a useful approach to increasing explainability, and we have added it to the related work. Our mechanistic interpretability focus differs in that we are interested in understanding how the model processes a table and computes its predictions. We also could not find “How TabPFN Learns”. Could you please share a link? We would be happy to take a look and include it in our related work if relevant.

AuthorsOpenReview ↗

Response to Reviewer bbKX, part 2

W3. “Predictive performance is weak for a 2B model, and nothing explains why.”

We made understanding the regression gap a priority for the rebuttal. We discuss the new evidence under “Regression gap” in the shared response. We ran these ablations on a smaller 169M model, which already performs reasonably well. For previous tabular foundation models, it has been found that increasing size does not automatically improve every metric: the deeper TabICLv2 ablation, for example, showed no clear log-loss gain (Qu et al., Appendix C). In our setup, the larger model reaches comparable scores with fewer training tables under the respective schedules, showing a useful benefit from scaling: the original 2B model reaches the 169M model's 5M-table classification accuracy (85.76%) and regression R2R^2 (64.53%) after 2,001,920 and 3,001,600 tables, respectively, and the corresponding training-loss levels (0.64024 and 0.40992) after 2,316,800 and 2,945,280 tables.

We also want to point out that parameter count alone does not equal compute per table. At 10,240 total rows, 100 features, and 200 queries, QuTab requires 55.24 analytical TFLOPs versus TabFM’s 85.45, despite QuTab’s approximately 2B parameters (Table F2, L2997–3009).

Q4. “Please add ablations of the main components and a compute-matched baseline trained on the same prior. Which component causes the regression gap?”

We have now included several ablation experiments at L490–505 (Table 2, L508–516). We could not train a new compute-matched baseline within the rebuttal period. However, TabICLv2 does provide a comparison using the same GraphSCM prior: it saw approximately 35M tables per model, versus our 23M classification and 12M regression tables. Thus, QuTab has less training data exposure in this comparison (Qu et al., Section 4.1; L2466–2472).

Q5. “Were the baseline results obtained with ensembling or with all features? How does QuTab do with all features, with ensembling, and over several splits?”

We have added bagging and evaluation across all official splits, as described under “Predictive evaluation” in the shared response. Figure 6 (L473–477) now shows QuTab with one and eight estimators and gives the reference-model counts directly: TabPFNv2.6, TabICLv2, and ConTextTab use eight estimators; FlexTab uses one. For feature handling, FlexTab’s released implementation randomly subsamples wide tables to 499 input features. The random subsampling to 100 features we use for our model also has precedent in Tan et al., Appendix F.1.

We additionally evaluated QuTab with a 2,048-column cap, which covers all features in the TabArena datasets. It obtains 87.01% accuracy and 68.02% R² at the same 10,240-context-row cap (Table E2, L2628–2635).

W3. “The abstract claims favorable compute scaling, but Appendix E.2 shows TabICLv2 is cheaper across the entire range QuTab supports.”

We agree that the wording in the abstract could be misleading. We have therefore removed the claim, as it is not central to the paper. We would still like to clarify that the analysis at L2954–2969 and L2970–2973 concerns theoretical attention complexity and analytical forward FLOPs. Although TabICLv2 is cheaper within QuTab’s supported range, QuTab is cheaper per table than some other tabular foundation models in that range despite its 2B parameters, as illustrated by the TabFM comparison above.

AuthorsOpenReview ↗

Response to Reviewer bbKX, part 3

Q3. “How do you reconcile weak full-state patching with strong steering at the same layers? What does single-layer steering give at a matched total perturbation?”

We understand why the original presentation made these results appear inconsistent. We steer across five layers, whereas the patching intervention acts at one layer. We have now added matched-budget single-layer steering: recovery is 0.154–0.254 across L4–L8 individually, compared with 0.455 jointly. The individual effects are weaker and less consistently above controls (L1069–1075; Table A3, L1094–1106). These results show that the effect is stronger with multi-layer steering.

W4. “The steering vectors are as large as the residual itself, so the effect may simply be off-distribution.”

We agree that the steering vector strength is large. However, it was selected through validation, and the learned direction outperforms shuffled and random controls of matched magnitude, which argues against perturbation size alone explaining the effect (L1061–1067 and Table A2, L1081–1091).

W4. “The probe for the product ab is taken at the last layer”

We agree that the very last layer can give a trivial readout because of its relationship to the prediction. Because of this, we report probing for all layers, not only the last one (Figure 4a, L399–401; Table A1, L973–983).

W4. “categorical features are encoded by order of appearance, which the paper does not mention [...] no checkpoints or experiment scripts are provided”

We apologize for omitting the appearance-order convention. We document it and credit TabFM at L1852–1859. We have also corrected the class-label permutation in the supplementary code to use seed 0, matching the paper (L2588–2590). The supplementary material includes the inference implementation, documented interpretability hooks, and an example script (examples/interpretability.py) showing decoder-lens readouts, steering, head patching, and attention summaries. The trained checkpoints will be provided with the full release, as stated in the reproducibility statement at L549–551.

I appreciate the authors' detailed response and the additional experiments. The inclusion of history-aware patching, single-layer steering controls (matched for perturbation budget), component ablation (specifically the row-mean/row-summary experiments, which help explain the performance gap), as well as bagging and multi-split evaluation, addresses most of my core concerns. The authors' clarification regarding the "causal bottleneck"—where the context representation remains fixed while only the query-label state is updated recursively—is also much clearer.

While a direct mechanistic comparison with TabICLv2 and the inclusion of compute-matched baselines are still missing, I do not believe this detracts from the paper's primary contributions; these can be addressed in future work. I maintain my original positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviews

We thank all reviewers for their careful reading and constructive feedback. Our paper presents a “meaningful architectural approach to mechanistically interpretable tabular foundation models” with “extensive and carefully designed” mechanistic experiments (Rev. LmR7). The design is “clean,” with “a well-built set of interpretability hooks” (Rev. bbKX), and read-only cells provide “a well defined intervention site for interpretability techniques” (Rev. 8uqx). We appreciate both this recognition and the reviewers’ concerns about causal information flow, ablations, predictive trade-offs, and the scope of our mechanistic evidence.

To address the central concerns, we added architecture ablations to investigate the regression gap and performed several follow-up experiments to strengthen our mechanistic results and predictive evaluation. We summarize the overall additions here and address the individual reviewer concerns in separate replies.

Regression gap. We have now added ablations for several components, with a particular focus on why regression performs worse. Removing the row-mean key in the smaller ablation model reduces R² from 64.53% to 50.74%, while replacing it with a stronger row summary raises R² to 68.99%. Together, these results provide empirical evidence that the model benefits from a stronger inductive bias towards row summaries (L496–505; Table 2, L508–516). This builds on our current hypothesis that the model performs more selective reads of context labels, providing less of the broad averaging that can benefit regression (L2837–2849).

Causal bottleneck and history. We also performed an ablation in which we removed the history route. As with the incremental performance gains reported for MoDA in language models, removing history only slightly reduces our predictive performance: mean regression R² changes from 64.53% to 64.13% (Table 2, L508–514; L2648–2650).

We would also like to address the architectural concerns about whether the history path invalidates our causal bottleneck and whether other tabular foundation models already provide the same restriction. The history is formed by projections of earlier states of the same query-label state. These cached entries do not undergo independent computation, and any information retrieved from them must flow back through the current query state. History therefore allows the model to recover information it computed earlier, while the recurrent computation still occurs within the same query stream.

TabPFNv2 and TabICLv2 can update their context states during inference: feature-group states in TabPFNv2 and row states during TabICLv2’s dataset-level computation. QuTab’s encoded context cells remain fixed after their initial encoding. Only the query-label state receives recurrent updates, giving us a defined state on which to intervene and through which the retrieved context information must pass before reaching the output (L361–367, L901–907).

History-aware patching. As requested, we added a patching condition that also patches the history KV. This increases recovery or leaves it approximately unchanged at the tested layers, preserving the overall pattern (L1184–1187; Table A5, L1189–1195; L1198–1199).

Predictive evaluation. We added bagging and evaluation across all official splits. Eight-estimator bagging improves accuracy/R² to 87.41%/70.58%, and the multi-split results remain close to the original results (L2592–2595, L2601–2604, and L2864–2869).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.