- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This paper presents the concept of an argmax partition as a way to understand geometry in a transformer model's output embedding space. This partition is theoretically defined with a single cell for each token in the vocabulary, the edges of which are described by the output embeddings that would result in that token being the argmax token after unembedding. The main hypothesis in this paper is that this argmax partition accurately represents relationships between tokens, such that next-token generation would smoothly "walk" between neighboring cells. They define a metric called "transition affinity" that measures the relatedness of two tokens in the model's output space, and show that it is related to bigram frequency.
The experiments in the paper can be divided into two sections:
- Experiments that demonstrate the validity of the argmax partition. These include visualizations of the argmax partitions for several toy models that are trained on synthetic data (Markov chains), and which demonstrate next-token geometry according to token transition probabilities.
- Experiments that compare a model's "transition affinity" to the bigram frequencies of two tokens and demonstrate that these two metrics are correlated.
Critical strengths
- The concept of using the argmax partition to describe a model's output distribution is interesting and seems to be differentiated from the application of the argmax partition in the cited Meyer et al. paper
- The toy examples suggest that synthetic next-token relationships are represented in argmax partition geometry.
- The results demonstrate a real relationship between embedding geometry, token probability, and bigram frequency.
Critical weaknesses
- The relationship between the transition affinity and the argmax partition is overstated, and the experiments do not demonstrate that the metric holds information about the "length of the border" between two token cells. As I understand it, the transition affinity metric shows whether two tokens frequently have the highest generation probabilities for a given embedding, but it does not generate an empirical argmax partition and compare the position or shape of token cells. While this property might indicate something about the shape of the argmax cells, the authors do not provide direct evidence to support it, nor do they acknowledge that the proposed metric is a proxy for their hypothesized geometric property.
- The authors do not provide details on how the argmax partition figures are generated in Section 3, but the clusters of points in Figure 3 would lead me to believe that the cell borders and labels may not come directly from the data.
- The finding that transition affinity can identify models that are distilled or fine-tuned from another model is not particularly surprising, and I would guess that a simple embedding similarity test would provide similar results. There is good, if old, evidence that fine tuning only has minor effects on a model's embeddings geometry (Merchant et al., 2020; Zhou and Srikumar, 2022), and since many distillation methods specifically train the student model to reproduce the teacher model's embeddings, we would expect similar structure there as well.
Questions for authors
I have one request for clarification:
- How are the cell boundaries generated for your partition plane figures in Section 3?
And some notes for future revisions:
- You have some major issues with your numbering throughout the paper. It took me several read-throughs before I realized that all of the definitions/properties/etc. are numbered in a single sequence, and I spent an unfortunate amount of time looking through your appendices for the missing information (e.g. Properties 1 and 2).
- Some of your section titles don't obviously match the content of those sections--it seems like maybe the relevant content was moved to an appendix. I would go back through and make sure that these are all consistent with what you actually present in the main paper.
5 comments
Thank you to the authors for their response to my concerns.
In Figure 3, having a yellow point in the green zone means that for this instance, the last token was yellow (c), and the next predicted token was green (d).
This is a very helpful explanation for interpreting Figure 3 and I recommend that you add a similar specific example to your manuscript for clarity.
We revised the paper to make it more centered around the interpretability contribution and softened the claims around distillation detection.
I think that this is the right framing and I support the changes that you have made to this discussion.
We revised the paper and now define transition affinity exclusively through the term contact region.
While I agree that contact region is a better term for what you describe, my main concern here remains, which is that your definition of transition affinity is only loosely related to the argmax partition. If I understand your methods correctly, you randomly sample vectors from within the model's functional embedding space (to avoid out-of-distribution sampling) and compare bigram frequency to the frequency with which terms have the two highest generation probabilities for a given vector.
I can see how this could be related to the argmax partition, but I think you are missing evidence to support that connection. The strongest conclusion that I think your results can support is that generation probability distributions show evidence of bigram frequency statistics, but I do not see clear evidence that your transition affinity metric approximates a relationship between argmax partitions or other embedding geometry.
My recommendation for a Weak Rejection remains. I would recommend that the authors consider additional experiments that would more clearly support the relationship between their metric and the geometry that they describe. Perhaps a demonstration of how the transition affinity interacts with your toy examples would be a good place to start.
We thank the reviewer for their comments which push us to strongly enhance the clarity and rigour of our work. We have integrated all changes, including the added explanation for Figure 3.
If I understand your methods correctly, you compare bigram frequency to the frequency with which terms have the two highest generation probabilities for a given vector.
Exactly! We say that the bigram frequencies correlate to the size of the contact regions (which are exactly the vectors for which terms have the two highest generation probabilities).
The intuition behind this statement stems directly from the argmax partition. We see a transformer as doing small steps in the argmax partition (as shown in Figure 1). It hops from cell to cell, where every successive cell corresponds to the most likely predicted token. Intuitively, if the hops are small, the transformer should only hop to neighbour cells. If we push this idea even further, on average the hops should be to the closest cell. That means that which cell is closest should correspond to the next token on average.
We can view this geometrically In Figure 2, we can further decompose each cell depending on which is the second closest (which corresponds directly to the contact regions). As such, contact regions can be seen as some kind of second order argmax partition. We show this on the second column of this Figure.
I do not see clear evidence that your transition affinity metric approximates a relationship between argmax partitions or other embedding geometry.
As such, we agree that our transition affinity metric does not directly approximate a relationship regarding the argmax partition. It is a correlation between the contact regions size of one model and either those of another model or the bigram statistics of a corpus.
However, those contact regions directly appear on the argmax partition, and can be seen as the second order argmax partition.
I would recommend that the authors consider additional experiments that would more clearly support the relationship between their metric and the geometry that they describe.
We added Appendix B, which computes the contact regions of the toy models of Figure 2 and compares them with the Markov chains.
First, the sizes of the contact regions recover the Markov chain. Over ten seeds, 87%, 100% and 86% of the total size of the contact regions falls on pairs of tokens that follow one another in the chain. If all contact regions had the same size, these shares would be 43%, 67% and 32%, and the same models at initialisation give 45%, 72% and 32%. This is transition affinity in its simplest setting, where the corpus geometry is the Markov graph itself.
Second, the new Figure 7 shows the contact regions on the argmax partition of these models. Its middle row is the plane cut of Figure 2 with the contact regions drawn inside each cell, and its bottom row is the graph read from their sizes, one line per pair of tokens, wider when the contact regions are larger. The thick lines reproduce the edges of the Markov chain, as shown in Figure.
We thank the reviewer for their thoughtful review, and adress their concerns below.
We apologize for the confusion. We realize our use of the term "border" is misleading, and attempt to clarify it here.
Let us call border the usual definition (that is a length in 2d, a surface in 3d, etc), and rename what we called in the paper a border as a contact region. What we mean by contact region between cell i and cell j is the proportion of cell i that is closest to cell j. So a large contact region necessarily entails the existence of a border in the usual sense. And intuitively, the larger the shared border between both cells, the more of cell i should be close to it, and hence the larger the contact region. However, it is true that this is not necessarily the case.
We revised the paper and now define transition affinity exclusively through the term contact region.
The argmax partition can be generated exactly. If the last token at the last layer of the transformer has an embedding vector h, the next token is sampled following the softmax of (ℓi(h)=⟨ei,h⟩)i. Here, the ei are simply the rows of the unembedding matrix. Whatever ℓi(h) is greater corresponds to the most likely next token. We can compute those linear convex cells exactly to generate the argmax partition figures.
In Figure 3, having a yellow point in the green zone means that for this instance, the last token was yellow (c), and the next predicted token was green (d).
We thank the reviewer for informing us of this issue.
First, we note that as transition affinity is a quite straightforward metric on the unembedding matrix, we expect other types of metrics, such as cosine similarity between embeddings, to lead to similar results (if two vectors share a high cosine similarity, then they should also share a large contact region). Our main purpose in this paper is to provide an intuitive and interpretable way of analyzing the embedding space, and to show how it might lead to useful applications down the line. We acknowledge that transition affinity as defined in the paper is still a rough tool, which needs further refinement before beating state of the art techniques on specific sub-areas of machine learning. As such, we intended the distillation experiments more as a validation of our hypothesis.
We revised the paper to make it more centered around the interpretability contribution and softened the claims around distillation detection.
We also added experiments to make sure the distillation does not involve the teacher model's embeddings directly. We use MiniLLM/teacher-gpt2-1.5B (GPT-2 XL fine-tuned on Dolly) as the reference model and compare it with four GPT-2 (120M) models that share the same architecture and initialization (Section 5, Table 1). The two distilled students are trained only on the teacher's outputs, never on its embeddings.
We also confirmed that transition affinity is lower for four independent models sharing the exact same architecture as the ones above, and differing only in their training (and hence their weights).
We sincerely apologize for this, and have modified it. We once again thank the reviewer for their careful feedback.