acceptodds
Under review as a conference paper at ICLR 2027

How Context Shapes Internal Representations in Tabular Foundation Models

Abstract

Tabular foundation models make predictions by learning statistical structure from examples provided in context. However, little is known about how this structure is represented internally or how it varies with the amount of examples provided. We study this using controlled synthetic inference problems. These problems contain exact reference posteriors whose geometry varies along known properties, such as covariance orientation, correlation, skewness, and mode separation. Across TabPFN V2, V2.5, V3, and TabICL, these properties are recoverable from low-dimensional hidden-state subspaces. We discover that intervening along the same subspaces causally changes model predictions. Copying only the task-subspace component of a query state from one context into another reproduces 95–99% of the resulting change in the final prediction, while the remaining component reproduces almost none. Small encoded shifts produce substantial movement in the intended direction, which is increased when the model is provided with longer context. Larger requested shifts are only partially realized at intermediate depths. We find that more context makes the controlled settings easier to decode from hidden states. Finally, training short-context models to match long-context task coordinates successfully moves their representations toward the long-context targets, but does not consistently improve prediction beyond training on long-context outputs alone. Output-only training can even reduce prediction error while moving the task coordinates away from those targets. We conclude that representations can be both decodable and causally steerable without providing additional predictive value as training targets.

Primary area:
interpretability and explainable AI
Keywords:
Tabular Foundation Models, Interpretability, In-Context Learning, Representation Learning, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_orfqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors consider the geometry of activation centroids as they relate to in-context learning in a special kind of transformer called a "Tabular Foundation Model". The test various tasks that are composed of a "context" which are pairs of inputs and their corresponding outputs. Given this context, the model task is to predict a distribution over the outputs for separate sets of inputs called "queries". These posteriors can then be used to calculate a scalar quantity-- which becomes the models "predicted property". The tasks they use have exactly known ground truth posteriors for a given query, which allows model performance to be well measured and calibrated. The authors find activation vector centroids for various control parameters in a given task family, and then project activations onto subspaces defined by the principal components of these centroids. They show that these centroids become more clustered as context accrues. Through causal interventions, they find that the organization of activations in these relatively low dimensional subspaces is tied to model predictions, but qualify this as a local phenomena. They additionally do a supervised fine tuning experiment to teach a model with low context to use representations in the task subspace that are similar to a teacher model with more context. They succeed in somewhat aligning the representation, but provide evidence that this does not aid the model in the prediction task more than only training it to mimic the outputs of the teacher model.

Critical strengths
  1. The authors undertake an important qustion, and appear to have a well-motivated testing regime. The paper separates three things that are often conflated when dealign with in-context task representations: whether something is decodable from hidden states, whether intervening on it changes predictions, and whether it is useful as a training target. The synthetic task families have exactly known posteriors, so there is a ground truth for what the model should be encoding and the predicted distribution can be scored against an exact reference. The progression from decoding to causal intervention to supervision is a sensible order in which to address the question.

  2. Training a short-context student against the same frozen model with more context is a clean way to ask whether a decodable and causally effective representation is also worth learning. The comparison is run against a reasonable output-only training objective and the result (guided training moves the student’s task coordinates toward the teacher’s but does not improve prediction beyond output matching) is a useful negative result to build off of.

Critical weaknesses

The paper does not stand on its own; a substantial rewrite is needed. I found the manuscript nearly unintelligible without outside help (an LLM) to read the appendices and related work and fill in definitions. Concretely: The introduction quotes numerical results and metrics (realized rotation in degrees, percent of a Hellinger gap recovered, “31 of 32 model–task–depth conditions”) before any of the underlying quantities, interventions, or metrics are defined. Readers cannot evaluate these claims, and the qualitative points are buried among them. Terms are often used before they are defined, or never defined: “independently calibrated centroid in task coordinates” and “independently calibrated task subspace” (Sections 4 and 5), “scope,” “late single-block,” “matched model–task comparisons.” All these terms appear in contexts where they mean very specific things-- but those things have not been (sometime they never are) clearly stated. On th eother side of things-- we have some definitions that come well before it makes sense. For example, c(τ) is used only in the intervention experiments but is defined much earlier in the text. Defining this well beofre using it creates friciton in understanding what is going on (though, not quite as bad as the reverse).

There is also no description of the model architectures or of how the four models differ from a standard GPT-2-style transformer. Figure 1 assumes the attention structure is already known. Much of the task structure lives in the appendix. The main text often leans on appendix figures and tables for its central evidence, and appendix figures are referenced inconsistently (sometimes labeled as appendix figures, sometimes not).

Section 6’s paragraph beginning “In the six-seed Gaussian comparison in Figure 21” is representative: two experimental cohorts, three counts with unstated denominators, and four undefined terms in five sentences. Table 2 is overloaded. Five columns with [min, max] brackets that are never discussed obscure a point that two columns, or better) a figure, would make.

Suggested remedies: a thorough appendix on architectures and tasks, with a short main-text statement of only the differences that matter for the interventions; shorten the intro significantly, keeping it high level; bring some of the patching evidence into the main text, and convert the dense tables to figures. A thorough clarity over the entire main text is also warranted.

Questions for authors

Why choose tables to represent the information in tables 1 and 2? I found it quite hard to tease out the point of the tables without careful analysis. A figure will help the reader.

The conclusion, as written, stays rather narrow in scope. The manuscript cites the implicit Bayesian inference paper (Xie et al.) and belief-state geometry (Shai et al.) but never says clearly what its findings add to those accounts in-context learning and geometry. Can the authors perhaps provide a more ambitious interpretation of how their results support or draw into question other recent publications about these topics?

I believe the context-length dependent results (like fig 3) are measured with respect to the basis frozen at the longest context. Is that the case?

Followup if the above is true: it might be the case that the finding that same-tau states cluster more with context is actually a statement about organization in the long-context subspace being different than organization in the short-context subspace. It is not clear whether short-context states fail to cluster because they are noisier, because they are systematically displaced from the long-context geometry, or because the model is encoding something about context size itself. Recalibrating the basis at each length, or showing the result is robust to doing so, feels like a potentially lacking control experiment to separate these things. Has this been checked or reported already in the appendix? If not, would the authors consider checking this?

What is meant by "The two-dimensional projections in Figure 2 are fitted separately for visualization"? What are they fitted to? And, why not just use the first few PC's of the m_a centroids or c(tau_a)?

1 comment

Thank you for your detailed review. We greatly appreciate your time and feedback.

We will take our time to improve upon the weaknesses you have noted.

To address your questions specifically:

  1. Tables and figures. While preparing the manuscript we chose the tables, because we felt like it was the most direct way to communicate the results, but after revisiting them following your review we agree that the current tables make the central comparisons harder to decipher than necessary. In our next iteration, we will show the predictive error and the representation distance side by side. We will move the full numerical results to the appendix.

  2. Bayesian interpretation. As we revise this work we will take special heed to this comment. We believe that our results are compatible with Bayesian interpretations of ICL, but our results do not establish that connection as they stand. Our goal moving forward is to better unify our approach with the groundwork laid out by Xie et al., Shai et al. and others.

For example, our manuscript does not seek to reconstruct a Bayesian belief state in the sense that it is done in Shai et al., nor do we establish that TFM models follow a Bayesian updating rule a la Xie et al.. One finding that we would like to highlight is that although we were able to shift outputs by intervening on the representations we identify, we did not observe consistent improvements in prediction quality (beyond training on the long context predictions alone) when using them during the learning process. We will investigate this further through additional experiments.

We plan to make these connections more explicit in future renditions of the project. In particular, our goal is to connect our work more directly to Xie et al., as we believe it directly connects to the setup we employ. Namely, if we take the interpretation that TFMs learn a prior over the (largely or wholly synthetic) data generating processes present in their pre-training data, and that during inference the in-context data is used to perform a belief update that facilitates the generation of posterior predictive distributions aligned with the context, then we believe that the key question we seek to address is essentially: “to what extent can we update the prior learned by the model by fine-tuning against internal representations?” Our project builds toward that question, but we are missing some connections that we will add moving forward.

  1. Context-length comparison. Yes, for Figure 3 and the context-length sweep in Appendix B.4, the task basis is calibrated against the longest context condition (eight independent 4,096-row contexts) and then held fixed when evaluating shorter contexts. We did test calibration against a shorter (512 row) context in an adjacent experiment as well (Figure 4 in the appendix), and we saw similar results there, but we need to strengthen this analysis and clarify this scope in the manuscript.

  2. Recalibration. We agree that the fixed-basis design we implement does not fully distinguish the reductions in variability from the changes in representational geometry relative to the long-context basis. We will add at least one test where we recalibrate at each context length as you suggest.

  3. Two-dimensional projections. In Figure 2, we separately fit a visualization projection rather than displaying the first two dimensions of the intervention basis. That said, your recommendation is well taken and we agree that using the PCs of the calibration centroids could make the relationship between the visualization and the experiments clearer. Thank you for the suggestion.

Thank you again for your careful reading and detailed guidance.

Reviewer_N91DOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the computational representations underlying in-context learning (ICL) in Tabular Foundation Models. It finds that these models produce intermediate results required for task completion within a low-dimensional space, and subsequent causal intervention experiments confirm that these intermediate results indeed participate in the computation.

Critical strengths
  • Section 6 reveals some interesting phenomena and has the potential to motivate deeper discussions. For example, can explicitly enforcing certain computational steps that we believe to be effective during training yield better results than allowing the model to develop such computational steps naturally?
Critical weaknesses
  1. Sections 4–5 essentially fall into the category of so-called "Slopterpretability," namely:

    1. Carves off a narrow question: e.g., what is the mechanism behind refusal in multi-turn jailbreaks? (not a real submitted paper)
    2. Performs interpretability analysis using off-the-shelf tools, like circuit discovery or SAEs
    3. Reports the results and draws conclusions, sometimes with exaggeration

    (Quoted from https://gregdurrett.github.io/colm2026-blog/ai-papers.html)

    My criticism of this research paradigm is that it does not provide substantial insights or generalizable knowledge. We should be cautious about the fact that this paradigm essentially uses one black box—the decoded steering vector (note that we still lack effective tools for understanding what steering vectors actually do)—to explain another black box. It does not produce human-understandable computational principles (considering that mechanistic interpretability aims to reduce a computational process to human-understandable pseudocode, merely identifying a steering vector or a low-dimensional space and demonstrating its clear association with a particular computation is obviously not an ideal form of reduction), nor does it produce more generalizable knowledge. Moving beyond steering vectors toward representational manifolds is a promising attempt.

  2. Major concerns regarding the task design

    • Why use Tabular Foundation Models? It is conceivable that the types of computation investigated in this paper (although I also have reservations about these computations), including estimating a rotation mapping, could also be simulated by ordinary LLMs. Furthermore, representational manifolds such as those shown in Figure 2 have already been widely observed in ordinary LLMs. I do not see any additional reason why this particular class of models is necessary.

    • Similarly, why use these synthetic datasets? I can imagine that synthetic data may provide ground-truth "internal task representations," but the paper seems to define an "internal task representation" not as "an abstract representation of the task currently being performed," but rather as "a representation of an intermediate variable required by the task." Such intermediate-variable representations have already been observed in more realistic tasks, such as arithmetic addition [1].

    • More seriously, as shown in Appendix B.1, for example around line 895, the inputs explicitly contain the very task-related latent variables that you hope to discover in the hidden space. This makes it highly possible that your investigation is not revealing the essence of the model's computation, but merely demonstrating that the model can read the answer directly from its input, thereby undermining the causal attribution of the entire experiment. This is a serious issue. I believe that only the 3D Gaussian Rotation task is free from this problem.

    Therefore, the aforementioned issues, including the unjustified deliberate restriction of the investigation's scope and the flaws in the task design, substantially undermine the reliability of the paper.

  3. Concerns regarding conceptual novelty: It should be noted that LLMs have already been shown to perform ICL using low-rank spaces [2].

  4. (I could copy this weakness verbatim into every paper in my review batch. How unfortunate.) The paper appears to be AI-generated without sufficient manual refinement, with inadequate attention paid to human readability. It retains numerous characteristics of AI-generated papers that make them difficult to read, and extensive revisions throughout the manuscript are necessary.

[1] https://arxiv.org/abs/2505.05145

[2] https://proceedings.iclr.cc/paper_files/paper/2026/hash/105112d52254f86d5854f3da734a52b4-Abstract-Conference.html

Questions for authors

See Weaknesses.

1 comment

Thank you for your comments.

We chose to study tabular foundation models not only because they are growing in popularity, but specifically because they allow us to relate in-context examples directly to known reference posteriors, while maintaining direct control over the provided context. The synthetic families were chosen to let us vary distributional properties selectively. Related questions have been studied in the context of language models (and great work has been produced in that area), and we do not assume that the phenomena we study are unique to TFMs.

We believe that your novelty concerns are fair, particularly considering our communicative shortcomings, and we will reevaluate the novelty of this project in light of the work you cite. In particular, our aim is to explore how using task representations as learning targets relates to Bayesian interpretations of ICL. Your review has made it clear to us that we are far from achieving that goal, but we intend to continue to pursue it, and appreciate your overall candor. We are committed to improvement and revision on all fronts you have identified.

We would also like to note that we agree that the current presentation makes it difficult to parse the differences between the experiments and the questions they address. To address the concerns you highlight in point (2), we have included some clarifications below.

In the scalar decoding and steering experiments (in Appendix B.1), the control parameter is supplied as an input feature. Those experiments do not establish that the TFMs we test infer this parameter from the context alone. They were intended to test how a supplied parameter is represented, and whether we can identify and then intervene on a subspace associated with said parameter to affect predictions. We will make this limitation more prominent during revisions.

We include a separate context only patching experiment in Section 5 and Appendix B.7. There, we vary the context while holding the query fixed. Specifically, for the scalar tasks, we fill the context with examples generated using the same simulator parameters, so that the task conditions are conveyed through the in-context examples, rather than being supplied as a column. We will update the manuscript to make this distinction much clearer.

In the context only experiment, we replace either the full query state, the task subspace component, or the residual component with the associated source state or component. For some of these conditions (e.g. copying the task component after the final block of the model), we observed a substantial (>=94%) reduction in the squared Hellinger distance to the original source-context predictions. We interpret this as a suggestion that under these conditions, the subspace carries much of the predictive effect of changing the context. This does not, however, identify any sort of full inference algorithm, but we believe it partially addresses the input concerns you have mentioned. That being said, further clarification is required in the manuscript and we will work to address this weakness.

Thank you again for your review.

Reviewer_unmROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how context impacts internal representations formed in TFMs. The paper claims that it is possible to decode task characteristics from the residual stream and steer predictions based on these insights. These findings aim to improve our understanding of the inference mechanisms of (tabular) ICL models.

Critical strengths

The strongest point about this paper is the chosen research question, which answered thoroughly, could impact the field of TFMs and inform future generations of TFMs architectures and pre-training setups.

Critical weaknesses

My main concern is structure, writing, and a lack of clarity (all of which are necessary to assess the paper extensively):

  1. The paper starts really well and motivates the need for this research. However, already at the end of the intro, the paper dives into technicalities of the experiments that require more background or only make sense later (e.g., what is a Gaussian rotation? l078).
  2. The related work section is one paragraph and does not embed the proposed work in the context of prior findings. It also lacks prior relevant works (e.g. Balef et al, ICML 2026; Ye et al. NeurIPS'25) and mostly references non-archival papers. Furthermore, it also lacks a short explanation of the state-of-the-art in tabular predictions and TFM architectures (which would be needed to follow the remainder paper).
  3. Section 3 tries to provide all information relevant for the experiment, but is really hard to follow (e.g. sentences in l141-144 seem to contradict each other). Furthermore, Sections 4-6 report results from experiments, but lack an introduction telling the reader why each experiment is needed (e.g. Section 4 starts with results in line 206).
  4. Overall and most importantly, the paper is not self-contained and not meant to be read linearly (by a human) as key definitions, notation, results and their interpretation are spread across sections.

Overall, the lack of clarity makes it difficult to assess the central claims, the validity of the empirical analysis, and the conclusions and findings in this work, and the paper as such cannot be accepted. Since I believe this research direction is very interesting for the community, I recommend revising the manuscript and re-submit to the next deadline.

Questions for authors

None

1 comment

Thank you for your comments and feedback. We agree that the manuscript needs to be restructured and rewritten to make the contents more digestible. We will work to clarify experimental design and make the conclusions easier to assess.

We also appreciate the references you suggested. We will update and reorganize the related work (by condensing the extended related work in the appendix and moving it into the main body of the text among other things) to clarify how this study relates to existing work on tabular foundation models and their internal representations. Thank you for identifying the potential of the research question. We will continue to work on it, while making the presentation changes required to better communicate it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.