- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors consider the geometry of activation centroids as they relate to in-context learning in a special kind of transformer called a "Tabular Foundation Model". The test various tasks that are composed of a "context" which are pairs of inputs and their corresponding outputs. Given this context, the model task is to predict a distribution over the outputs for separate sets of inputs called "queries". These posteriors can then be used to calculate a scalar quantity-- which becomes the models "predicted property". The tasks they use have exactly known ground truth posteriors for a given query, which allows model performance to be well measured and calibrated. The authors find activation vector centroids for various control parameters in a given task family, and then project activations onto subspaces defined by the principal components of these centroids. They show that these centroids become more clustered as context accrues. Through causal interventions, they find that the organization of activations in these relatively low dimensional subspaces is tied to model predictions, but qualify this as a local phenomena. They additionally do a supervised fine tuning experiment to teach a model with low context to use representations in the task subspace that are similar to a teacher model with more context. They succeed in somewhat aligning the representation, but provide evidence that this does not aid the model in the prediction task more than only training it to mimic the outputs of the teacher model.
Critical strengths
-
The authors undertake an important qustion, and appear to have a well-motivated testing regime. The paper separates three things that are often conflated when dealign with in-context task representations: whether something is decodable from hidden states, whether intervening on it changes predictions, and whether it is useful as a training target. The synthetic task families have exactly known posteriors, so there is a ground truth for what the model should be encoding and the predicted distribution can be scored against an exact reference. The progression from decoding to causal intervention to supervision is a sensible order in which to address the question.
-
Training a short-context student against the same frozen model with more context is a clean way to ask whether a decodable and causally effective representation is also worth learning. The comparison is run against a reasonable output-only training objective and the result (guided training moves the student’s task coordinates toward the teacher’s but does not improve prediction beyond output matching) is a useful negative result to build off of.
Critical weaknesses
The paper does not stand on its own; a substantial rewrite is needed. I found the manuscript nearly unintelligible without outside help (an LLM) to read the appendices and related work and fill in definitions. Concretely: The introduction quotes numerical results and metrics (realized rotation in degrees, percent of a Hellinger gap recovered, “31 of 32 model–task–depth conditions”) before any of the underlying quantities, interventions, or metrics are defined. Readers cannot evaluate these claims, and the qualitative points are buried among them. Terms are often used before they are defined, or never defined: “independently calibrated centroid in task coordinates” and “independently calibrated task subspace” (Sections 4 and 5), “scope,” “late single-block,” “matched model–task comparisons.” All these terms appear in contexts where they mean very specific things-- but those things have not been (sometime they never are) clearly stated. On th eother side of things-- we have some definitions that come well before it makes sense. For example, c(τ) is used only in the intervention experiments but is defined much earlier in the text. Defining this well beofre using it creates friciton in understanding what is going on (though, not quite as bad as the reverse).
There is also no description of the model architectures or of how the four models differ from a standard GPT-2-style transformer. Figure 1 assumes the attention structure is already known. Much of the task structure lives in the appendix. The main text often leans on appendix figures and tables for its central evidence, and appendix figures are referenced inconsistently (sometimes labeled as appendix figures, sometimes not).
Section 6’s paragraph beginning “In the six-seed Gaussian comparison in Figure 21” is representative: two experimental cohorts, three counts with unstated denominators, and four undefined terms in five sentences. Table 2 is overloaded. Five columns with [min, max] brackets that are never discussed obscure a point that two columns, or better) a figure, would make.
Suggested remedies: a thorough appendix on architectures and tasks, with a short main-text statement of only the differences that matter for the interventions; shorten the intro significantly, keeping it high level; bring some of the patching evidence into the main text, and convert the dense tables to figures. A thorough clarity over the entire main text is also warranted.
Questions for authors
Why choose tables to represent the information in tables 1 and 2? I found it quite hard to tease out the point of the tables without careful analysis. A figure will help the reader.
The conclusion, as written, stays rather narrow in scope. The manuscript cites the implicit Bayesian inference paper (Xie et al.) and belief-state geometry (Shai et al.) but never says clearly what its findings add to those accounts in-context learning and geometry. Can the authors perhaps provide a more ambitious interpretation of how their results support or draw into question other recent publications about these topics?
I believe the context-length dependent results (like fig 3) are measured with respect to the basis frozen at the longest context. Is that the case?
Followup if the above is true: it might be the case that the finding that same-tau states cluster more with context is actually a statement about organization in the long-context subspace being different than organization in the short-context subspace. It is not clear whether short-context states fail to cluster because they are noisier, because they are systematically displaced from the long-context geometry, or because the model is encoding something about context size itself. Recalibrating the basis at each length, or showing the result is robust to doing so, feels like a potentially lacking control experiment to separate these things. Has this been checked or reported already in the appendix? If not, would the authors consider checking this?
What is meant by "The two-dimensional projections in Figure 2 are fitted separately for visualization"? What are they fitted to? And, why not just use the first few PC's of the m_a centroids or c(tau_a)?
Thank you for your detailed review. We greatly appreciate your time and feedback.
We will take our time to improve upon the weaknesses you have noted.
To address your questions specifically:
Tables and figures. While preparing the manuscript we chose the tables, because we felt like it was the most direct way to communicate the results, but after revisiting them following your review we agree that the current tables make the central comparisons harder to decipher than necessary. In our next iteration, we will show the predictive error and the representation distance side by side. We will move the full numerical results to the appendix.
Bayesian interpretation. As we revise this work we will take special heed to this comment. We believe that our results are compatible with Bayesian interpretations of ICL, but our results do not establish that connection as they stand. Our goal moving forward is to better unify our approach with the groundwork laid out by Xie et al., Shai et al. and others.
For example, our manuscript does not seek to reconstruct a Bayesian belief state in the sense that it is done in Shai et al., nor do we establish that TFM models follow a Bayesian updating rule a la Xie et al.. One finding that we would like to highlight is that although we were able to shift outputs by intervening on the representations we identify, we did not observe consistent improvements in prediction quality (beyond training on the long context predictions alone) when using them during the learning process. We will investigate this further through additional experiments.
We plan to make these connections more explicit in future renditions of the project. In particular, our goal is to connect our work more directly to Xie et al., as we believe it directly connects to the setup we employ. Namely, if we take the interpretation that TFMs learn a prior over the (largely or wholly synthetic) data generating processes present in their pre-training data, and that during inference the in-context data is used to perform a belief update that facilitates the generation of posterior predictive distributions aligned with the context, then we believe that the key question we seek to address is essentially: “to what extent can we update the prior learned by the model by fine-tuning against internal representations?” Our project builds toward that question, but we are missing some connections that we will add moving forward.
Context-length comparison. Yes, for Figure 3 and the context-length sweep in Appendix B.4, the task basis is calibrated against the longest context condition (eight independent 4,096-row contexts) and then held fixed when evaluating shorter contexts. We did test calibration against a shorter (512 row) context in an adjacent experiment as well (Figure 4 in the appendix), and we saw similar results there, but we need to strengthen this analysis and clarify this scope in the manuscript.
Recalibration. We agree that the fixed-basis design we implement does not fully distinguish the reductions in variability from the changes in representational geometry relative to the long-context basis. We will add at least one test where we recalibrate at each context length as you suggest.
Two-dimensional projections. In Figure 2, we separately fit a visualization projection rather than displaying the first two dimensions of the intervention basis. That said, your recommendation is well taken and we agree that using the PCs of the calibration centroids could make the relationship between the visualization and the experiments clearer. Thank you for the suggestion.
Thank you again for your careful reading and detailed guidance.