- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+2)
Read review
Paper summary
This paper presents a method for understanding LLM output token probability distributions from a geometric perspective. The authors decompose a model's next-token probability vector and remove the context-independent component to isolate the context-specific variation in token probability over a dataset of 1M tokens. They then use an eigendecomposition of this context-specific distribution to find that 99.9% of variation in that distribution can be explained by ~10-24% of the available dimensions. They further show that this low-rank distribution does not substantially change during model fine-tuning.
Critical strengths
- The paper is very well-written and easy to follow. The theoretical proofs cleanly support the empirical experiments and the analysis was similarly well organized.
- The mathematical theory behind their spectral framework is well-founded and thorough. The proofs in the main text provide enough explanation to follow their methods, and the appendix provides substantial support for those interested.
- The authors were careful in outlining the separation of their methods from logit rank, and in demonstrating that their results cannot be directly attributed to architectural components.
Critical weaknesses
- The authors never acknowledge a simple explanation for their results, which is the output token frequency distribution. Although I recognize that the methods in the paper are looking at the entire output probability distribution (not limited to the argmax tokens), the simplest explanation for their findings is that it is a direct representation of the frequency distribution in their dataset.
- Given the near-Zipfian distribution of language, a sample of 1M tokens is too small to reliably capture the probability distribution over a 100K+ token vocabulary. In this size dataset, we would only expect to see a few thousand tokens more than a handful of times, and a huge number of rare tokens would never appear at all. Even with a large enough sample, we would expect to capture 99.9% of the token frequency distribution in just 35-60% of a 100-200K token vocabulary. A small dataset that cuts off the long tail would skew that proportion, making the ~10% concentration shown in this paper a possible side effect of the dataset alone.
- The fine-tuning results could similarly be explained by term frequency--if the fine tuning dataset frequency distribution is similar to that of the pre-training dataset, we would not expect to see a difference in the output distribution after fine tuning. The separation of those results on the ELI5 task actually seems to support this further, since ELI5 likely has a more limited vocabulary than GSM8K and would, thus, cause a model to produce a lower number of tokens (and a "tighter" probability distribution).
- I'm not sure that the random-model ablation demonstrates that the low-dimensional geometry is learned. It is difficult to see the random line on the plots, but I am reading this as evidence that context-dependence is learned. If the low-dimensional geometry were learned, wouldn't we expect to see the model using more of the probability distribution before training? Instead, we see that a few directions capture all of the context-dependent information before training, and that signal is spread out across more directions after training.
Questions for authors
Clarifications that could change my vote:
- I want to see some evidence or clear explanation that would distinguish your results from word frequency statistics. Even if you find that your spectral framework is capturing the same information as word frequency, that needs to be acknowledged and addressed.
- Similarly, I would like to see an explicit acknowledgment of the issues that your dataset size introduces. I recognize that compute is not always accessible, so the small dataset size is not an issue as long as you include some discussion of how that changes your ability to draw conclusions.
Very minor suggestions for a revision:
- It seems like you are using "canonical" in the sense of a canonical correlation analysis (CCA). It would be helpful if you state this (or otherwise provide a definition) early on, as there are several meanings for "canonical" and you use it throughout the paper.
- The dashed line in Figure 2 and Appendix B is basically not visible--think about how to fix these so that they are clearer.
4 comments
Official Comment by Authors (2/2)
3. The ELI5 result is not explained by a smaller effective predictive vocabulary.
The reviewer additionally suggests that the ELI5 behavior may arise because ELI5 uses a smaller vocabulary. Here it is important to distinguish the realized tokens in an ELI5 corpus from the model's predictive distribution. At every ELI5 context, our estimator evaluates all 152064 next-token probabilities; the analysis is therefore not restricted to the set of tokens actually appearing in ELI5 outputs.
Moreover, the predictive marginal itself is not uniformly narrower on ELI5. For the pretrained model, 99.9% of marginal probability mass requires 32503 tokens on ELI5, compared with only 11544 on GSM8K (Appendix E, Figure 11). Thus, ELI5 does not simply correspond to probing the model over a smaller predictive vocabulary.
Finally, we thank the reviewer for the presentation suggestions. We have clarified the meaning of “canonical,” made the dependence on the probing distribution explicit, clarified the interpretation of random initialization, and added the predictive-marginal analysis in Appendix E, Figure 11.
Rebuttal reminder
Hi,
Just a gentle reminder as the discussion deadline is approaching. We hope our response and the additional clarifications/experiments addressed your concerns, and we would be grateful if you could reconsider your assessment in light of our revised response. We would also greatly appreciate any further feedback you may have before the deadline. Thank you again for your time and consideration.
Thank you to the authors for your response to my concerns.
Appendix E Thank you for the additional analysis here. This work addresses most of my concerns about the relationship between token frequency (or your predictive marginal distribution) and your canonical spectrum. In particular, your demonstration of a disconnect between and on the fine-tuning and random initialization experiments provide sufficient evidence that the two distributions are not causally linked. I also appreciate the acknowledgment that they are still related.
A few minor concerns remain:
As described in Section 2.2, every one of our (10^6) contexts contributes the complete next-token probability vector
I understand that this is the case, but we would expect little variation in the probability estimates for rare tokens, especially in a relatively limited dataset. I still think that it is worth acknowledging that some of your low-rank findings may be directly related to the frequency distribution of language more generally.
We have revised the discussion around Figure 2 to make this interpretation explicit.
I am not seeing a change in your revised manuscript here. I think this does deserve one or two additional sentences to make the distinction clear.
In light of the authors' response and revisions, I would like to update my recommendation to Weak Acceptance.
Official Comment by Authors (1/2)
We thank the reviewer for raising the possibility that the observed spectral concentration could be explained by token-frequency statistics. We believe this interpretation mixes two distinct issues: finite-sample token counts and the model's predictive marginal qθ. Neither accounts for the canonical spectral behavior we observe.
The canonical geometry is defined jointly by the model pθ and probing distribution μ, so we do not claim a universal, dataset-independent rank. The question is instead whether the concentration of the canonical spectrum is simply inherited from the concentration of qθ. Our additional analysis in Appendix E, Figure 11 shows that it is not.
1. The 106-context estimate is not a 106-token frequency histogram.
A central part of the reviewer's concern appears to rely on the analogy with a corpus containing 106 sampled tokens, where rare vocabulary items may occur only a few times or not at all. This is not how our estimator is constructed. As described in Section 2.2, every one of our 106 contexts contributes the complete next-token probability vector
pθ(⋅∣hi)∈RV.
The predictive marginal is therefore
= \frac{1}{N} \sum_{i=1}^{N}p_\theta(w\mid h_i),$$ which averages model probabilities rather than observed next-token counts. Consequently, every vocabulary coordinate contributes $$p_\theta(w\mid h_1),\ldots,p_\theta(w\mid h_{10^6})$$ to the analysis, including tokens that are never generated. For Qwen, **each of the $152064$ vocabulary coordinates receives approximately $10^6$ probability evaluations**. The observed spectral concentration is therefore not caused by rare vocabulary coordinates being absent from a $10^6$-sample token histogram. **2. The concentration of $q_\theta$ does not determine the concentration of the canonical spectrum.** We directly test the proposed explanation by plotting the ranked predictive marginal $q_\theta(w)$ over the complete vocabulary in **Appendix E, Figure 11**. Its behavior tells a qualitatively different story from the canonical spectrum. Two comparisons make this especially clear: - **Random initialization $\rightarrow$ pretraining:** the random model has a much broader predictive marginal---approximately $151K}$ of $152064$ tokens are required to capture $99.9\%$ of its marginal mass. Pretraining makes $q_\theta$ dramatically more concentrated. Yet the canonical spectrum changes in the opposite direction: the random model has a **nearly degenerate spectrum**, by which we mean that almost all eigenvalues are negligible and essentially all spectral energy is confined to a very small number of leading modes. After pretraining, a much broader set of nontrivial canonical modes becomes active (**Figure 2 and Appendix B, Figure 6**). - **Pretraining $\rightarrow$ ELI5 fine-tuning:** on ELI5 contexts, fine-tuning again makes $q_\theta$ more concentrated. In this case, however, the canonical spectrum also becomes slightly more concentrated, exhibiting an earlier decay. Thus, **the same change in the marginal---increased concentration of $q_\theta$---is associated with opposite changes in spectral concentration in the two settings**. From random initialization to pretraining, a more concentrated marginal accompanies a *broader* canonical spectrum; from pretraining to ELI5 fine-tuning, a more concentrated marginal accompanies a *more concentrated* spectrum. Therefore, the concentration of $q_\theta$ alone cannot explain the canonical spectral behavior. This is also clear from the definitions. The marginal $$q_\theta(w)=\mathbb{E}_{h\sim\mu}[p_\theta(w\mid h)]$$ describes where probability mass lies on average, whereas $$C_\theta = \mathbb{E}_{h\sim\mu} \left[ z_\theta(h)z_\theta(h)^\top \right]$$ describes how that probability distribution changes as context varies. For an arbitrary, highly non-uniform $q_\theta$, if $p_\theta(\cdot\mid h)=q_\theta$ for every context, then $C_\theta=0$. Hence $q_\theta$ is part of the canonical construction, but does not determine its eigenspectrum. This also clarifies the random-initialization result. The nearly degenerate spectrum of the random model should not be interpreted as a learned low-dimensional representation. It reflects the near absence of structured context-dependent prediction. Pretraining creates substantial context-dependent predictive variation across many modes; the nontrivial observation is that this learned variation nevertheless remains confined to a restricted fraction of the available token space. We have revised the discussion around **Figure 2** to make this interpretation explicit.