acceptodds
Under review as a conference paper at ICLR 2027

Transition Affinity: A Unified Notion of Geometry for LLMs and Text Corpora

Abstract

Transformer next-token predictions induce an argmax partition of the embedding space, where each cell consists of representations for which a given vocabulary token has maximal logit. We first argue that this representation is a useful way to interpret a transformer. We then show that the geometry of this partition is governed by a phenomenon we call transition affinity: the size of the shared region between two cells is highly correlated to the probability of the two corresponding tokens following one another. This yields a common geometric representation, allowing us to compute correlations between two LLMs, two corpora, or an LLM and a corpus. We conclude by showing how transition affinity can be used to detect that a model has been distilled.

Primary area:
interpretability and explainable AI
Keywords:
Argmax partition, transformer, transition affinity, embedding space, voronoi, markov chain

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FRRxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper presents the concept of an argmax partition as a way to understand geometry in a transformer model's output embedding space. This partition is theoretically defined with a single cell for each token in the vocabulary, the edges of which are described by the output embeddings that would result in that token being the argmax token after unembedding. The main hypothesis in this paper is that this argmax partition accurately represents relationships between tokens, such that next-token generation would smoothly "walk" between neighboring cells. They define a metric called "transition affinity" that measures the relatedness of two tokens in the model's output space, and show that it is related to bigram frequency.

The experiments in the paper can be divided into two sections:

  1. Experiments that demonstrate the validity of the argmax partition. These include visualizations of the argmax partitions for several toy models that are trained on synthetic data (Markov chains), and which demonstrate next-token geometry according to token transition probabilities.
  2. Experiments that compare a model's "transition affinity" to the bigram frequencies of two tokens and demonstrate that these two metrics are correlated.
Critical strengths
  • The concept of using the argmax partition to describe a model's output distribution is interesting and seems to be differentiated from the application of the argmax partition in the cited Meyer et al. paper
  • The toy examples suggest that synthetic next-token relationships are represented in argmax partition geometry.
  • The results demonstrate a real relationship between embedding geometry, token probability, and bigram frequency.
Critical weaknesses
  • The relationship between the transition affinity and the argmax partition is overstated, and the experiments do not demonstrate that the metric holds information about the "length of the border" between two token cells. As I understand it, the transition affinity metric shows whether two tokens frequently have the highest generation probabilities for a given embedding, but it does not generate an empirical argmax partition and compare the position or shape of token cells. While this property might indicate something about the shape of the argmax cells, the authors do not provide direct evidence to support it, nor do they acknowledge that the proposed metric is a proxy for their hypothesized geometric property.
  • The authors do not provide details on how the argmax partition figures are generated in Section 3, but the clusters of points in Figure 3 would lead me to believe that the cell borders and labels may not come directly from the data.
  • The finding that transition affinity can identify models that are distilled or fine-tuned from another model is not particularly surprising, and I would guess that a simple embedding similarity test would provide similar results. There is good, if old, evidence that fine tuning only has minor effects on a model's embeddings geometry (Merchant et al., 2020; Zhou and Srikumar, 2022), and since many distillation methods specifically train the student model to reproduce the teacher model's embeddings, we would expect similar structure there as well.
Questions for authors

I have one request for clarification:

  • How are the cell boundaries generated for your partition plane figures in Section 3?

And some notes for future revisions:

  • You have some major issues with your numbering throughout the paper. It took me several read-throughs before I realized that all of the definitions/properties/etc. are numbered in a single sequence, and I spent an unfortunate amount of time looking through your appendices for the missing information (e.g. Properties 1 and 2).
  • Some of your section titles don't obviously match the content of those sections--it seems like maybe the relevant content was moved to an appendix. I would go back through and make sure that these are all consistent with what you actually present in the main paper.
5 comments

We thank the reviewer for their thoughtful review, and adress their concerns below.

The relationship between the transition affinity and the argmax partition is overstated, and the experiments do not demonstrate that the metric holds information about the "length of the border" between two token cells.

We apologize for the confusion. We realize our use of the term "border" is misleading, and attempt to clarify it here.

Let us call border the usual definition (that is a length in 2d, a surface in 3d, etc), and rename what we called in the paper a border as a contact region. What we mean by contact region between cell ii and cell jj is the proportion of cell ii that is closest to cell jj. So a large contact region necessarily entails the existence of a border in the usual sense. And intuitively, the larger the shared border between both cells, the more of cell ii should be close to it, and hence the larger the contact region. However, it is true that this is not necessarily the case.

We revised the paper and now define transition affinity exclusively through the term contact region.

The authors do not provide details on how the argmax partition figures are generated in Section 3, but the clusters of points in Figure 3 would lead me to believe that the cell borders and labels may not come directly from the data.

The argmax partition can be generated exactly. If the last token at the last layer of the transformer has an embedding vector h\mathbf h, the next token is sampled following the softmax of (ℓi(h)=⟨ei,h⟩)i(\ell_i(\mathbf h)=\langle \mathbf e_i,\mathbf h\rangle)_i. Here, the ei\mathbf e_i are simply the rows of the unembedding matrix. Whatever ℓi(h)\ell_i(\mathbf h) is greater corresponds to the most likely next token. We can compute those linear convex cells exactly to generate the argmax partition figures.

In Figure 3, having a yellow point in the green zone means that for this instance, the last token was yellow (c), and the next predicted token was green (d).

The finding that transition affinity can identify models that are distilled or fine-tuned from another model is not particularly surprising, and I would guess that a simple embedding similarity test would provide similar results.

We thank the reviewer for informing us of this issue.

First, we note that as transition affinity is a quite straightforward metric on the unembedding matrix, we expect other types of metrics, such as cosine similarity between embeddings, to lead to similar results (if two vectors share a high cosine similarity, then they should also share a large contact region). Our main purpose in this paper is to provide an intuitive and interpretable way of analyzing the embedding space, and to show how it might lead to useful applications down the line. We acknowledge that transition affinity as defined in the paper is still a rough tool, which needs further refinement before beating state of the art techniques on specific sub-areas of machine learning. As such, we intended the distillation experiments more as a validation of our hypothesis.

We revised the paper to make it more centered around the interpretability contribution and softened the claims around distillation detection.

We also added experiments to make sure the distillation does not involve the teacher model's embeddings directly. We use MiniLLM/teacher-gpt2-1.5B (GPT-2 XL fine-tuned on Dolly) as the reference model and compare it with four GPT-2 (120M) models that share the same architecture and initialization (Section 5, Table 1). The two distilled students are trained only on the teacher's outputs, never on its embeddings.

ModelRelation to the reference modelAll tokensExcl. top-20
gpt2Same pretraining data0.520.49
init-gpt2-120Mgpt2 + further fine-tuning on the same data0.550.53
SeqKD-gpt2-120Mgpt2 + further fine-tuning with teacher token output0.580.56
KD-gpt2-120Mgpt2 + further fine-tuning with teacher logit output0.630.61

We also confirmed that transition affinity is lower for four independent models sharing the exact same architecture as the ones above, and differing only in their training (and hence their weights).

ModelAll tokensExcl. top-20
DarwinAnim8or/gpt2-124M-tinystories0.280.23
UDACA/gpt2-117M-2.2B-PubMedAbs0.430.41
UDACA/gpt2-117M-2.2B-USPTO0.470.44
UDACA/gpt2-117M-2.2B-USPTOAndPubMedAbs0.460.44

On the notes for future revisions.

We sincerely apologize for this, and have modified it. We once again thank the reviewer for their careful feedback.

Thank you to the authors for their response to my concerns.

In Figure 3, having a yellow point in the green zone means that for this instance, the last token was yellow (c), and the next predicted token was green (d).

This is a very helpful explanation for interpreting Figure 3 and I recommend that you add a similar specific example to your manuscript for clarity.

We revised the paper to make it more centered around the interpretability contribution and softened the claims around distillation detection.

I think that this is the right framing and I support the changes that you have made to this discussion.

We revised the paper and now define transition affinity exclusively through the term contact region.

While I agree that contact region is a better term for what you describe, my main concern here remains, which is that your definition of transition affinity is only loosely related to the argmax partition. If I understand your methods correctly, you randomly sample vectors from within the model's functional embedding space (to avoid out-of-distribution sampling) and compare bigram frequency to the frequency with which terms have the two highest generation probabilities for a given vector.

I can see how this could be related to the argmax partition, but I think you are missing evidence to support that connection. The strongest conclusion that I think your results can support is that generation probability distributions show evidence of bigram frequency statistics, but I do not see clear evidence that your transition affinity metric approximates a relationship between argmax partitions or other embedding geometry.

My recommendation for a Weak Rejection remains. I would recommend that the authors consider additional experiments that would more clearly support the relationship between their metric and the geometry that they describe. Perhaps a demonstration of how the transition affinity interacts with your toy examples would be a good place to start.

We thank the reviewer for their comments which push us to strongly enhance the clarity and rigour of our work. We have integrated all changes, including the added explanation for Figure 3.

If I understand your methods correctly, you compare bigram frequency to the frequency with which terms have the two highest generation probabilities for a given vector.

Exactly! We say that the bigram frequencies correlate to the size of the contact regions (which are exactly the vectors for which terms have the two highest generation probabilities).

The intuition behind this statement stems directly from the argmax partition. We see a transformer as doing small steps in the argmax partition (as shown in Figure 1). It hops from cell to cell, where every successive cell corresponds to the most likely predicted token. Intuitively, if the hops are small, the transformer should only hop to neighbour cells. If we push this idea even further, on average the hops should be to the closest cell. That means that which cell is closest should correspond to the next token on average.

We can view this geometrically In Figure 2, we can further decompose each cell depending on which is the second closest (which corresponds directly to the contact regions). As such, contact regions can be seen as some kind of second order argmax partition. We show this on the second column of this Figure.

I do not see clear evidence that your transition affinity metric approximates a relationship between argmax partitions or other embedding geometry.

As such, we agree that our transition affinity metric does not directly approximate a relationship regarding the argmax partition. It is a correlation between the contact regions size of one model and either those of another model or the bigram statistics of a corpus.

However, those contact regions directly appear on the argmax partition, and can be seen as the second order argmax partition.

I would recommend that the authors consider additional experiments that would more clearly support the relationship between their metric and the geometry that they describe.

We added Appendix B, which computes the contact regions of the toy models of Figure 2 and compares them with the Markov chains.

First, the sizes of the contact regions recover the Markov chain. Over ten seeds, 87%, 100% and 86% of the total size of the contact regions falls on pairs of tokens that follow one another in the chain. If all contact regions had the same size, these shares would be 43%, 67% and 32%, and the same models at initialisation give 45%, 72% and 32%. This is transition affinity in its simplest setting, where the corpus geometry is the Markov graph itself.

Second, the new Figure 7 shows the contact regions on the argmax partition of these models. Its middle row is the plane cut of Figure 2 with the contact regions drawn inside each cell, and its bottom row is the graph read from their sizes, one line per pair of tokens, wider when the contact regions are larger. The thick lines reproduce the edges of the Markov chain, as shown in Figure.

Reviewer_fAbVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies the "argmax partition" of LM output space: for each token, the cone of hidden states where this token has the top logit. Authors propose transition affinity, the Spearman correlation over token pairs between (a) how often token j is runner-up inside cell of token i, estimated by uniform sampling in a box, and (b) corpus bigram probability π(j|i).

The experiments have 4 parts:

  1. Tiny transformers trained on Markov chains, showing 2D plane cuts of the partition match the Markov graph.

  2. Correlations of 0.62-0.70 between 3 pretrained models and English Wikipedia, beating all 199 random relabelings.

  3. 4 OLMo-2-1B models trained from same initialization on 4 domains, where each model correlates best with its own training corpus.

  4. Llama-3.1-8B correlates higher with 4 derived models (Llama-3.2-3B, Swallow, Tulu 3, Hermes 3) than 4 unrelated models, especially after removing top-20 frequent tokens. The authors propose this as a distillation detection method.

Critical strengths
  1. Cheap and weight-only probe. The metric only needs unembedding matrix without any input prompts or forward passes, so it can be easily applied to any open-weight model. Using argmax partition to link output geometry and data statistics is also a clean perspective. If the signal holds under strict controls, it will be useful for model interpretability and provenance.

  2. Clean experimental design. Training the same OLMo-2-1B architecture from same initialization on 4 different corpora cleanly isolates the effect of training data. Also, the random-label control was fixed beforehand, and the paper reports the full null range instead of just the mean.

Critical weaknesses
  1. Main claim is not separated from simpler explanations. This is my main concern. The paper claims runner-up structure of token i shows which tokens follow i. But the statistic is a single Spearman correlation pooled over all (i, j) pairs, and both τ and π are normalized inside each row i. So row-level properties (frequency of i, cell size, number of successors) can give high pooled correlation even when identity of j has no information. Frequency of j also affects both sides. Random relabeling breaks these alignments together with the bigram one, so beating it does not show pair-level signal. Paper's own numbers also show this: 1 random relabeling reaches 0.532 for Llama, and the pair set shrinks from 3.12M to 54.3k after relabeling, so observed and null are computed on very different sets. Also, paper does not discuss a simple mechanism. With tied embeddings, the residual stream carries e_i to last layer, so learning to predict successor j directly pulls e_j toward e_i. All toy models are tied, and in Fig. 4 the 2 tied models have the highest scores while untied Claire-Mistral has the lowest.
  2. The cross-family ranking may just come from tokenizer overlap instead of model lineage. I checked the tokenizers: OLMo-2 and Hunyuan-7B use the exact same token IDs as Llama-3 for first 100,256 entries (all 3 come from cl100k), and they get high scores of 0.78 and 0.74. Meanwhile, Phi-4-mini and stable-code-3B use different ID spaces and only get 0.17 and 0.37. The paper does not explain how tokens are matched across vocabularies. Since distillation detection is the main application claimed in abstract and conclusion, this issue directly affects the paper's contribution.
Questions for authors

Clarification requests:

  1. What are the bounds of sampling box B? Is the final RMSNorm gain folded into unembeddings? How sensitive are results to this choice? In GPT-2, I find uniform box gives the largest cells to under-trained "glitch" tokens, which model almost never reaches on real text.
  2. How many pairs are used for Phi-4-mini and stable-code-3B? What fraction of pairs in P+ have n_ij = 1 at 10^9 samples?

Questions whose answers could materially affect my assessment:

  1. For models in Fig. 4: mean within-row Spearman correlation, pooled correlation after shuffling τ inside each row, and a frequency-preserving permutation null, together with unigram p(j) baseline on the same pairs.
  2. For Sec. 3: exact cell adjacency computed by Property 4(ii) (1 LP per pair), report precision and recall against Markov edges and against random embeddings, for both tied and untied models.
  3. For Sec. 5: a student distilled from scratch on teacher logits, compare with independently trained models that use same tokenizer, and report ROC/AUC.
4 comments

We thank the reviewer for a very careful review, and for checking the tokenizers and GPT-2 themselves. We are glad they find the argmax partition a clean perspective, and the OLMo experiment well designed. We marked the corresponding changes in orange in the PDF, to distinguish them from the updates in our earlier revision.

Main claim is not separated from simpler explanations.

We agree this is the central question, so we tested it directly. In short:

  • Between tokens of matched frequency, the geometry and the corpus rank pairs the same way in 55% of comparisons, and in 61 to 63% when both are measured clearly, against 50% for a null that keeps frequencies.
  • This gets even stronger when frequencies are matched within 2%, so frequency does not explain it.
  • It does not need tied embeddings.
  • It is causal: changing only which token follows which in the training data moves the argmax partition (p=0.011p=0.011, with a test fixed before training).

Sampling. Following Q1, we now sample the hidden states from N(μ,Σ)\mathcal N(\mu,\Sigma), the Gaussian with the mean and covariance of the model's own final hidden states on held-out Wikipedia, instead of the uniform box. Both give the same conclusions, and the Gaussian gives a stronger signal. The Q3 controls below are on the pairs and box of Figure 4, as requested; the other results use the Gaussian. All use 10910^9 samples, for Llama-3.2-1B (tied) / Qwen2.5-1.5B (tied) / Claire-Mistral-7B (untied).

The controls of Q3, on the pairs and box of Figure 4:

Figure 4 pairs, box of the paperLlamaQwenClaire
Pooled correlation (Figure 4)0.7040.6770.623
Pooled, τ\tau shuffled within each row (200 permutations)0.6210.5900.509
Pooled, frequency-preserving null (100 replicates)0.709±0.0070.709\pm0.0070.678±0.0090.678\pm0.0090.601±0.0060.601\pm0.006
Mean within-row Spearman (same null)0.323 (0.313)0.261 (0.251)0.319 (0.286)
Unigram frequency p(j)p(j) alone, pooled0.6850.5950.592

Shuffling τ\tau inside each row lowers the correlation in every model, so the order within a row matters. And τ\tau predicts the bigrams better than the frequency of jj alone. The frequency-preserving null only swaps tokens within blocks of 16 consecutive frequency ranks, so frequency keeps its effect on both geometries: a frequent token has a large cell that touches many others, and it also has many successors. To isolate the pairs, we added a test in which frequency cancels exactly.

Pair-level test. Take two source tokens i,i′i,i' of similar frequency and two candidates j,j′j,j' of similar frequency, and compute Dx=xij−xij′−xi′j+xi′j′D_x=x_{ij}-x_{ij'}-x_{i'j}+x_{i'j'}, with x=log⁡τx=\log\tau or x=log⁡πx=\log\pi. Anything that depends only on ii or only on jj and multiplies the score (frequency, cell size, number of successors, row normalization) becomes a term aia_i or bjb_j on the log scale, and cancels exactly: ai−ai−ai′+ai′=0a_i-a_i-a_{i'}+a_{i'}=0. We then count how often DτD_\tau and DπD_\pi have the same sign. If the corpus carried no information about pairs, this would be 50%.

Tetrad concordance (frequency-preserving null)LlamaQwenClaire
Under N(μ,Σ)\mathcal N(\mu,\Sigma)0.548 (0.500)0.550 (0.500)0.551 (0.500)
Under the box of the paper0.520 (0.500)0.527 (0.500)0.519 (0.500)

None of the 200 null replicates goes above 0.502 (p≤0.005p\le0.005, 2M tetrads each). The tetrad removes the effect of each token on its own, but not an effect that would depend on both frequencies at once, so we checked this too. When the two sources, and the two candidates, differ in frequency by less than 2%, the concordance goes up: 0.563 / 0.562 / 0.554, and 0.69 / 0.70 / 0.64 when both contrasts are measured clearly. Removing any joint function of the two frequencies from both geometries leaves it unchanged. A standard statistic, which does not rely on our tetrad construction, agrees: the within-row partial correlation of τ\tau and π\pi given the frequency of jj is 0.167 / 0.164 / 0.150, against 0.064 / 0.056 / 0.039 for the null.

How strong is the agreement? A concordance of 0.55 may look small, but many tetrads rest on very small counts, which pull it towards 0.5. So we grouped tetrads by how clearly each side measures its contrast (∣D∣|D| divided by its Poisson standard error ∑1/n\sqrt{\sum 1/n}). The agreement grows with the evidence, and reaches 0.61 to 0.63 when both sides are clear, in all five models. The null stays at 0.50 in every group:

Smaller of the two z-scoresbelow 11 to 22 to 33 to 5above 5
Llama-3.2-1B0.5170.5610.5980.6200.614
Claire-Mistral-7B0.5190.5600.5960.6070.619
Qwen3-1.7B0.5170.5580.5920.6130.618
Qwen3-8B0.5170.5570.5930.6150.617
Null (largest of 100 replicates)0.5020.5030.5050.5070.507

This is now the right panel of Figure 4, with the table in Appendix C. Two simpler checks, easier to read than the tetrad, agree. Neighbouring cells are bigrams of the corpus 27 to 32% more often than under the frequency-preserving null. And for one source and two candidates of matched frequency, the candidate with the larger τ\tau is the more frequent successor in 58.4 / 58.8 / 58.5% of cases, while picking the more frequent candidate is right in only 51.3 / 50.2 / 51.0% of cases.

The relabeling at 0.532. In this relabeling, the corpus rows of "." and "(" land on frequent rows of the geometry. Only 97k pairs remain, these two rows hold 14% of them, and inside them both τ\tau and π\pi simply follow the frequency of jj. This is why the new tests use the frequency-preserving null, which keeps a pair set close in size to the observed one (2.48M against 3.12M pairs).

Tied embeddings. This mechanism would pull eje_j towards eie_i, which the cosine between output vectors measures. So we compared the cosine with τ\tau on the same tetrads. The cosine reaches only 0.518 and 0.527 for Qwen and Claire, against 0.550 and 0.551 for τ\tau, and in all three models τ\tau carries pair information that the cosine does not (partial correlation 0.087 / 0.138 / 0.125). Tying is also not needed. Qwen3-1.7B (tied) and Qwen3-8B (untied) share one tokenizer, so we compared them on the same corpus counts [3]: they give the same concordance (0.544 and 0.543), and in both the cosine adds nothing to τ\tau. The same holds in Figure 4: untied Claire has the lowest pooled correlation, but its pair-level concordance (0.551) is as high as that of the tied Llama and Qwen (0.548 and 0.550). We added this to Section 4.1 and Appendix C.

Causal test. A correlation alone cannot show that the geometry follows the training data, so we also changed the data. We continued pre-training Llama-3.2-1B on two versions of the same Wikipedia text that differ only in which token follows which. For random quadruples (A,B,C,D)(A,B,C,D) with frequent bigrams A→BA\to B and C→DC\to D, the treated text swaps the successors (A→DA\to D, C→BC\to B), so every token keeps exactly the same frequency. The models learn the swap: in the second experiment, the log-probability of the new successor goes up on every treated quadruple (by 4.9 nats on average), and by 0.05 nats on the others.

We measure the change through rijr_{ij}, the correlation of the logits ℓi\ell_i and ℓj\ell_j under the base model's N(μ,Σ)\mathcal N(\mu,\Sigma), kept fixed for all runs. It equals cos⁡(Σ1/2ei,Σ1/2ej)\cos(\Sigma^{1/2}e_i,\Sigma^{1/2}e_j), and is computed exactly from the weights, with no sampling noise. We use [rAD−rAB]+[rCB−rCD][r_{AD}-r_{AB}]+[r_{CB}-r_{CD}], in which the effect of each single token cancels, and compare each treated run with a control run trained on the original text. The effect is the change on swapped quadruples minus the change on the others, tested by shuffling which quadruples are swapped.

The swapped tokens move towards their new partners, and the others do not: in the first experiment, the mean change is 0.0190.019 and 0.0250.025 for swapped quadruples, against −0.0003-0.0003 and 0.0030.003 for the others (−0.0001-0.0001 in the second experiment). Continued training on the original text alone does not move them either. In a first experiment (133 quadruples, tied and untied output layer), z=6.3z=6.3 and 5.95.9 (p<10−4p<10^{-4}). In a second, independent one (145 new quadruples, 3 epochs, two training seeds), the effect is positive in all four runs (p=0.004p=0.004 to 0.0490.049), and 71 to 74% of swapped quadruples move as predicted, against 38 to 45% of the others. Relative to the starting gap between old and new partners, the effect has the same size in both experiments (5 to 6%).

In the second experiment we also measured τ\tau itself, with a test fixed before training and the same 4×1094\times10^9 samples for every run. The contrast log⁡(τAD/τAB)+log⁡(τCB/τCD)\log(\tau_{AD}/\tau_{AB})+\log(\tau_{CB}/\tau_{CD}) moves towards the new partners: +0.49+0.49, z=2.25z=2.25, p=0.011p=0.011, positive for both tied and untied layers, with 65% of swapped quadruples moving as predicted against 40% of the others. So the argmax partition itself follows the swap.

Robustness. Since Σ\Sigma is computed on Wikipedia, like the bigram counts, we checked that this is not what drives the result: the concordance stays well above its null when Σ\Sigma is computed on FineWeb [2] instead (0.541 / 0.543 / 0.542, and 0.59 to 0.61 when both sides are clear).

What are the bounds of sampling box B? Is the final RMSNorm gain folded into unembeddings? How sensitive are results to this choice? In GPT-2, I find uniform box gives the largest cells to under-trained "glitch" tokens, which model almost never reaches on real text.

The bounds Lk,UkL_k,U_k are the coordinate-wise minimum and maximum of the final hidden states, after the final norm. For Qwen and Claire, they come from 786k tokens of held-out Wikipedia (262k each in English, French and Chinese), separate from the text used for the bigram counts. For Llama, we used the box of [1], built from random and natural sentences until it stops growing. In both cases the box is on post-gain states and is used with the raw unembedding, so the RMSNorm gain is included; this is the same as a pre-gain box with WU diag(γ)W_U\,\mathrm{diag}(\gamma).

We ran the reviewer's GPT-2 check on our models. The box gives large cells to rarely used tokens for Qwen (13% of samples land in cells of tokens seen fewer than 1,000 times in our corpus), much less for Llama (1.4%) and Claire (0.3%). Under N(μ,Σ)\mathcal N(\mu,\Sigma), with μ\mu and Σ\Sigma computed on 4M tokens of held-out Wikipedia, fewer than 1% of samples land in such cells. This law also does not depend on how the hidden space is parametrized: if h↦A−1hh\mapsto A^{-1}h and W↦WAW\mapsto WA, the logits keep the same law, while a box would change. The pooled correlation hardly moves (0.696 / 0.718 / 0.610, against 0.704 / 0.677 / 0.623), and every pair-level statistic gets stronger. The pair-level results also do not depend on the box: rebuilding Llama's box with the Qwen/Claire recipe (about 1.7 times wider) leaves them unchanged (concordance 0.523 against 0.520, within-row partial correlation 0.107 against 0.100). Section 4.1 now uses the Gaussian, and the box is in Appendix C.

Like the box, the Gaussian needs one forward pass over generic text, here to get μ\mu and Σ\Sigma. It needs no task-specific prompts and no corpus counts, so the probe stays cheap and works for any open-weight model.

How many pairs are used for Phi-4-mini and stable-code-3B? What fraction of pairs in P+\mathcal P^+ have nij=1n_{ij}=1 at 10910^9 samples?

Pairs for Phi-4-mini and stable-code-3B.

We used 2946178 pairs (2758506 if we exclude the 20 most common tokens) for Phi-4-mini, and 6718829 (resp 6261952) for stable-code-3B.

Pairs seen once. Under N(μ,Σ)\mathcal N(\mu,\Sigma), 41.7 / 40.6 / 44.1% of the pairs in P+\mathcal P^+ have nij=1n_{ij}=1 (45.5 / 45.6 / 49.2% under the box). The results do not depend on them:

  • On pairs seen at least twice, the concordance is the same: 0.547 / 0.550 / 0.552.
  • On pairs seen in two independent samples, the within-row reliability of log⁡τ\log\tau is 0.93 / 0.93 / 0.94.
  • Neighbouring pairs never seen carry about 0.6 / 0.7 / 0.1% of the runner-up mass (Good–Turing estimate [4]).

For models in Fig. 4: mean within-row Spearman correlation, pooled correlation after shuffling τ\tau inside each row, and a frequency-preserving permutation null, together with unigram p(j) baseline on the same pairs.

All four are in the first table above, on the pairs and box of Figure 4. They show that the order within a row carries information beyond frequency, and led us to the pair-level tests, which are well above their frequency-preserving null in every model.

For Sec. 3: exact cell adjacency computed by Property 4(ii) (1 LP per pair), report precision and recall against Markov edges and against random embeddings, for both tied and untied models.

We thank the reviewer for this suggestion. Adjacency over the whole embedding space has to be restricted first. Indeed, in higher dimensions nearly every pair of cells shares a border whatever the embeddings are. What is relevant is instead the part of the space the model actually navigates. We sample 512 sequences of length 16, take the final-layer state at every position, join consecutive states by a segment, and keep the borders that segments cross. Note that this makes no difference in two dimensions this makes no difference.

We trained 10 seeds of each of the four chains of Section 3, in the setting of the paper (same dimensions, steps and hyperparameters), tied and untied, computed the exact adjacency with one linear program per pair, and restricted it as above. The baseline is the same model at initialisation under the identical procedure. Precision and recall are those of the navigated borders against the Markov edge set, averaged over 10 seeds.

chain (nn, edges/pairs)ddtied P / Runtied P / Rat init P / R
6-cycle, ε=0\varepsilon=0 (6, 6/15)21.00 / 1.000.48 / 0.450.47 / 0.40
cycle + hub (6, 10/15)40.82 / 0.910.70 / 0.890.67 / 0.86
cycle + one-way detour (8, 9/28)30.79 / 0.800.52 / 0.520.34 / 0.54
cycle + four detours (8, 12/28)71.00 / 0.670.78 / 0.750.46 / 0.91

The tied six-cycle is exact in every seed, and the three noiseless chains sit well above their initialisation. For the four-detour chain the navigated graph is, in all 10 seeds, the Markov graph with the four direct edges a→ba\to b, b→cb\to c, c→dc\to d, d→ad\to a removed. As mentioned in the paper, transition affinity is accompanied by a second constraint, confusability. Coming from aa, the tokens bb and ff are confusable, since both follow aa with probability 1/21/2, so the state after aa lies on the border between the cells of bb and ff. The path from the cell of aa to that border therefore crosses only one of the two borders aa–bb and aa–ff, in our runs always aa–ff, which lowers the recall to 8/128/12 while nothing outside the chain is crossed. Taking confusability into account is one way to refine transition affinity in further work.

While we discussed before why transition affinity is also relevant for untied embeddings, our visual analysis of Section 3 is not as relevant. We modified the manuscript to make this clear.

For Sec. 5: a student distilled from scratch on teacher logits, compare with independently trained models that use same tokenizer, and report ROC/AUC.

We study eight models with the exact same architecture. There are four GPT-2 models initialized randomly trained independently of the teacher. Their affinity is 0.28–0.47 (0.23–0.44 excluding the top 20 tokens). The four models in the teacher’s lineage (gpt2, init, SeqKD, KD) all score higher: 0.52–0.63 (0.49–0.61).

Lineage models vs independent models: AUC = 1.0 on both metrics, with a minimum margin of 0.05. Perfect separation by chance has probability 1/70.

ransition affinity separates the teacher's lineage from independently trained GPT-2 models (AUC = 1.0). Spearman correlation of next-token predictions with MiniLLM/teacher-gpt2-1.5B. The four independent models share GPT-2's architecture and tokenizer but were trained without the teacher. The four lineage models increase in relatedness: gpt2 shares only the training data, init-gpt2-120M adds the same fine-tuning data, SeqKD-gpt2-120M trains on the teacher's generated tokens, and KD-gpt2-120M trains on its full next-token distributions. Every lineage model scores higher than every independent model, with or without the 20 most frequent tokens (minimum margin 0.05; chance probability of perfect separation 1/70). Initialization, Epochs and Target describe fine-tuning on Dolly.

GroupModelInitializationEpochsTargetAll tokensExcl. top-20
IndependentDarwinAnim8or/gpt2-124M-tinystories–––0.280.23
IndependentUDACA/gpt2-117M-2.2B-PubMedAbs–––0.430.41
IndependentUDACA/gpt2-117M-2.2B-USPTO–––0.470.44
IndependentUDACA/gpt2-117M-2.2B-USPTO…–––0.460.44
Lineagegpt2–––0.520.49
LineageMiniLLM/init-gpt2-120Mgpt23Human tokens0.550.53
LineageMiniLLM/SeqKD-gpt2-120Mgpt220Teacher tokens0.580.56
LineageMiniLLM/KD-gpt2-120Mgpt220Teacher logits0.630.61

[1] Meyer et al., How Many Different Outputs Can a Transformer Generate? ICML'26.

[2] Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS'24 Datasets and Benchmarks.

[3] Yang et al., Qwen3 Technical Report. arXiv:2505.09388

[4] Good, The Population Frequencies of Species and the Estimation of Population Parameters. Biometrika, 1953.

Reviewer_8FN5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper studies how the geometry of the unembedding vectors in language models relates to the statistics of the training corpora. The authors consider the argmax partition of the embedding space and introduce transition affinity, a measure connecting the neighboring cells in the argmax partition to the bigram statistics of the data. Through experiments on Markov chains and pretrained language models, they show that the unembedding geometry correlates with corpus statistics, as measured by transition affinity. They further demonstrate how this measure can be used to detect distillation in language models.

Critical strengths

The paper is generally well-written (apart from a few issues mentioned below). The methodology and discussions are interesting, and the paper does a good job of introducing the relevant concepts.

The empirical study covers a range of settings, from synthetic experiments to evaluations on different pretrained models and datasets.

I also appreciate that the authors explicitly discuss the limitations of their approach.

Critical weaknesses
  1. Literature review: The paper overlooks relevant prior work on several topics. There is a substantial body of literature studying the training dynamics of transformers on Markov chains, including works specifically related to the paper's observations about models learning bigram statistics [1,2]. There is also extensive literature on the connection between embedding geometry and co-occurrence statistics, ranging from classical embedding models (e.g., [3]) to modern language models (e.g., [4,5]). Moreover, the paper does not discuss prior work on detecting distillation or model lineage (e.g., [6,7]). These references are only a few representative examples, and I believe the paper should engage more thoroughly with the relevant literature in each of these areas.

  2. Experimental setting of Section 5: Section 5 proposes transition affinity as a method for detecting distillation, but the experimental setup has two limitations. First, I believe the models compared with Llama 8B as adapted or distilled models share the same architecture. Thus, the high similarity between their embedding spaces could arise either from distillation/adaptation, as the paper claims, or from shared architectural biases. The paper uses this similarity as evidence of distillation without ruling out the latter explanation. Second, the paper does not compare the proposed method with existing approaches, either in terms of performance or methodology.

  3. Analysis of transition affinity during training: The study of transition affinity over training in Section 4.2 is interesting, but the main text focuses only on the final checkpoint. I believe the discussion of Figure 6 in the appendix, which shows how the metric evolves throughout training, should be included in the main body. Moreover, related to my first concern, there are prior works studying training dynamics and how models learn increasingly complex statistics (e.g., [1,8]) that seem relevant to the observation in this figure, particularly the fact that the correlation with bigram statistics peaks during training before decreasing.

  4. Writing and presentation: There are a few issues with notation and presentation:

    • Line 140: Should nn be ∣V∣|V|? Also, what do ee and tt each refer to and how are they different?
    • Around line 283: Earlier in the paper, the subscript of tt denotes the position in a sequence (line 95), but here tit_i and tjt_j seem to refer to two consecutive tokens.
    • Section 4.1: The term "transition affinity" is used to refer to both τ\tau (line 282) and the Spearman correlation coefficient (line 319).
    • Table 1: The caption states that "A dash indicates a result not reported," but there are no dashes in the table.

References:

[1] Edelman et al., The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

[2] Bietti et al., Birth of a Transformer: A Memory Viewpoint

[3] Levy and Goldberg, Neural Word Embedding as Implicit Matrix Factorization

[4] Yang et al., Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

[5] Zhao et al., Implicit Geometry of Next-Token Prediction: From Language Sparsity Patterns to Model Representations

[6] Zhu et al., Independence Tests for Language Models

[7] Rawat et al., Reference-Based Distillation Detection in LLMs

[8] Belrose et al., Neural Networks Learn Statistics of Increasing Complexity

Questions for authors

The following questions do not affect my overall assessment of the paper directly, but I believe they would benefit from clarification.

  1. In the visualizations of the argmax partition (Figures 1–3), wouldn't the choice of the plane used for the visualization affect which tokens appear to have neighboring cells, and consequently the interpretation of the embedding geometry based on these visualizations?

  2. In Section 4.1, how are the boundaries of the box, LkL_k and UkU_k, chosen? Wouldn't this choice affect the estimates, as well as whether all pairs of neighboring cells are observed in the estimation?

  3. In Figure 5, do the authors have an explanation for why the model trained on CN exhibits similar correlations across all datasets?

3 comments

We are especially grateful to the reviewer for their detailed review and careful reading of our paper. We are encouraged by them highlighting the clartity of the exposition, and believe the paper has been significantly improved thanks to their remarks.

Literature review

  • Markov chains: We added a discussion to Remark 1.
  • Connection between embedding geometry and co-occurrence statistics: We added a discussion on the second page. The main advantage of our analysis is that it is more interpretable than previous work.
  • Detecting distillation or model lineage: As explained in the following answer, we see distillation more as a way to validate transition affinity, and we do not claim to beat state of the art techniques. However, it was clearly a mistake not to mention existing methods, and we added it. We did test the method of the paper cited by the reviewer [1], which does identify the distilled and fine-tuned model. A limitation of the second paper [2] is that it requires an earlier-generation checkpoint from the same lineage, and we were unable to test it as a result.

Experimental setting of Section 5

We ran an additional set of experiments on GPT-2 models that share the same architecture and differ only in their training, which removes the architectural component. We give a detailed answer to reviewer FRRx and have updated Section 5 of the paper accordingly.

We stress that the main purpose in this paper is to provide an intuitive and interpretable way of analyzing the embedding space, and to show how it might lead to useful applications down the line. We acknowledge that transition affinity as defined in the paper is still a rough tool, which needs further refinement before beating state of the art techniques on specific sub-areas of machine learning. We expect other distillation methods to work on the tested models as well, and confirmed it for [1]. As such, we intended the distillation experiments more as a validation of our hypothesis.

Analysis of transition affinity during training

We thank the reviewer for these pointers. Indeed, prior work shows that language models first match unigram and then bigram statistics before moving away from them [3-5]. Our Figure is consistent with these findings, and we moved the discussion to the main text.

Writing and presentation

  • Line 140: Indeed, nn should be ∣V∣|\mathcal V|. t\mathbf t denotes the token in natural language, while e\mathbf e denotes the associated vector in the embedding space.

  • Around line 283: We fixed the notations. Now, ti1,…,tin\mathbf t_{i_1},\ldots,\mathbf t_{i_n} denote a sequence of tokens, with V={t1…,t∣V∣}\mathcal V=\{\mathbf t_1\ldots,\mathbf t_{|\mathcal V|}\} the vocabulary.

  • Section 4.1: We thank the reviewer for noticing this, and have corrected it in the revised version.

On the visualizations of the argmax partition

That is totally correct! Since the agmax partition is in dd dimensions, we need to choose a plane cut to represent, which affects the interpretation. We tried to mitigate this issue by following an objective criterion for the choice of the plane: we pick the plane that maximizes the size of the smallest cell. We expect such a plane to be the most representative, as all cells have close to equal surfaces.

This objective criterion departs from previous work which specifically choose the plane that match their prediction the best [6].

On the boundaries of the box

Following [7], we sample points in the embedding space using both random and natural language sentences. We update the boundaries as the smallest rectangle countaining all embeddings. We stop once it stabilizes.

This way, the space we consider encompasses the vast majority of all possible embeddings. However, it is not exact as it contains part of the embedding space that are not used (no embedding ever lies there). But estimating the precise manifold being used is extremely hard, and that is why this rectangle estimation had been used by previous work.

In Figure 5, do the authors have an explanation for why the model trained on CN exhibits similar correlations across all datasets?

The explanation is that since the set of tokens used by both models have an almost empty intersection, transition affinity doesn't work very well. Tokens such as punctuation marks dominate the metric and make it lose precision. We mitigated this in Section 5 by removing the top 20 most frequent tokens.

[1] Zhu et al., Independence Tests for Language Models.

[2] Rawat et al., Reference-Based Distillation Detection in LLMs.

[3] Chang et al., Word acquisition in neural language models.

[4] Chang et al., Characterizing learning curves during language model pre-training: Learning, forgetting, and stability.

[5] Belrose et al., Neural Networks Learn Statistics of Increasing Complexity.

[6] Shai et al., Transformers represent belief state geometry in their residual stream.

[7] Meyer et al., How Many Different Outputs Can a Transformer Generate?

Thanks for your additional analyses and responses.

I find the current framing of the paper more convincing, particularly with the addition of the related literature and the discussions. I also appreciate the clarity of the presentation, which I often find lacking in other papers I've reviewed recently. So, I'll slightly increase my score to weak accept.

At the same time, I agree with the weaknesses raised by reviewer FRRx, and addressing them could definitely make the paper more solid.

We thank the reviewer for recognizing the clarity of our presentation and raising their score as a consequence.

The remaining weakness raised by reviewer FRRx is that the link between transition affinity and the argmax partition is not clear enough. We hope to have adressed this issue through our most recent answer. In short, we added Appendix B to explicitly show the role of the contact regions in the synthetic setting of Section 3.

Regardless of the final decision, we are very grateful to the reviewer for their valuable comments, which helped us significantly improve our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.