- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper, the authors derive a vector that encodes the unigram distribution of a language models training data after being multiplied with the models unembedding matrix. The authors name this vector the “direction of ignorance” and interpret it as a Bayesian prior of the models output when given little information. They decompose the final embedding of an LM generation into a linear combination of this vector, weighted by a factor , and an orthogonal residual they interpret as the information content of the given input. They argue that this lambda decreases when providing more information to the LM, and show that it can be used to steer generation and increase LM performance on tasks such as GSM8k.
Critical strengths
-
The approach is clever, well defined and argued for
-
Interesting approach to studying the unigram bias in LLMs and potential for applying it to steering of generation
-
Well written and good figures
-
Clear research question
Critical weaknesses
I emphasize that some of these weaknesses aren't "critical", but nonetheless points of contention.
-
On multiple occasions the authors argue based on “how much information a context carries”, but this measure is never quantified. It would be interesting to quantify what information is in this context.
-
No comparable baseline is computed. It would be interesting to the compared to just the entropy of the predictive distribution.
-
The application to steering is very limited. It would be very interesting to see how this prior affects other types of tasks and whether it could be relevant as an uncertainty signal.
-
Using the 70% most frequent tokens of a training corpus is perhaps not an arbitrary decision and it isn't argued for. This could have significant effects on the results.
Questions for authors
-
You define the decomposition of the final prediction vector such that the prior and the context vector are always orthogonal. I get that this makes the math possible, but what is the conceptual argument for this?
-
Have you performed any experiments to see the impact of discarding the 1-c least frequent tokens?
-
Since we don’t know the training data of Llama/Gemma/Qwen, your unigram distribution estimates for those are based on other models training data. Could this be problematic?
-
Would it make sense to compare to baseline measures related to uncertainty such as entropy?
4 comments
rebuttal continued 1
W3. Other tasks and applications
We thank the reviewer for this very good suggestion to investigate further tasks and applications. The added forking-path analysis shows how reducing prior loading can promote a context-relevant continuation and correct a reasoning path; the Gemma-3-12B result extends the steering evidence to a larger model. We agree that it would be interesting to investigate how this prior affects other types of tasks and whether it could serve as an uncertainty signal. We do think that our phenomenological observation has application potential.
However, we regard the goal of the current manuscript primarily as drawing attention to an interesting phenomenon: a prominent geometrical feature, the unigram embedding, and its function inside LLMs, namely prior regulation. As such, our steering experiment is intended to support this functional interpretation of the direction of ignorance, with a more limited claim about its general applicability.
To strengthen this functional interpretation, we have added a forking-path analysis in Section 5 (Figure 5), with additional examples in Appendix J.3 (Figure 16). This analysis illustrates how reducing prior loading can promote a context-relevant continuation already present in the model's predictions, leading to a corrected reasoning path. The larger-model steering result is reported in Appendix I (Figure 14 and Table 3). We believe these additions provide a useful connection between the geometric interpretation and the observed behavioral effect, while broader task evaluation remains an important direction for further investigation. We thank the reviewer for pointing out these promising directions.
W4 / Q2. Vocabulary cutoff
We thank the reviewer for bringing up this subtle point about the vocabulary cutoff and the estimation of rare-token frequencies. The decrease in prior loading with model size is preserved at every tested cutoff reported here: 40%, 50%, 60% and 70% vocabulary coverage. Llama-3.2-3B remains below 1B, and Qwen2.5-3B remains below 1.5B, with all eight paired 95% intervals above zero.
| Vocabulary coverage | Llama 3.2: Δλ [95% CI] | Qwen2.5: Δλ [95% CI] |
|---|---|---|
| 40% | 0.558 [0.549, 0.567] | 0.307 [0.288, 0.327] |
| 50% | 0.522 [0.513, 0.530] | 0.272 [0.255, 0.289] |
| 60% | 0.485 [0.478, 0.493] | 0.205 [0.190, 0.219] |
| 70% | 0.416 [0.409, 0.423] | 0.137 [0.127, 0.147] |
Δλ is the smaller-minus-larger difference in mean loading over positions 250–299, pairing the same 200 WikiText passages: Llama 1B versus 3B and Qwen 1.5B versus 3B. Fits use fixed Pile counts. Brackets show pointwise 95% paired passage-bootstrap intervals (2,000 replicates). All eight intervals are positive, showing preservation of the model-size ordering across vocabulary cutoffs.
Vocabulary coverage is the fraction of vocabulary IDs retained after ranking tokens by their observed corpus counts. At 70% coverage, we retain the most frequent 70% of IDs. The boundary count is the observed count of the least frequent retained token. Low boundary counts indicate that tokens near the cutoff have few observations, so their estimated frequencies are sensitive to which documents are sampled: even a few additional occurrences can produce a large relative change. Under a simple independent-token counting approximation, the relative sampling error scales as 1/√n for a token with count n. Rare-token log-frequency estimates, which enter the prior fit, are therefore less stable. The 70% cutoff balances vocabulary coverage against this estimation uncertainty.
For the Llama tokenizer, the boundary count falls from 25 at 70% coverage to eight at 80% in a sample of approximately 211 million tokens. For Qwen, it falls from nine to one in approximately 497 million tokens. At these empirical frequencies, matching the expected counts at the 70% boundary would require approximately 3.1× as much streamed text for Llama and 9× for Qwen at the 80% boundary. This steep sampling cost motivates our choice of 70% vocabulary coverage for the main analysis and the range summarized above.
The robustness claim concerns the qualitative comparison: fitted angles and absolute λ values do change with the cutoff.
rebuttal continued 2
Q1. Why an orthogonal decomposition?
We thank the reviewer for inviting us to explain this conceptual basis. The decomposition follows from the unigram prior being encodable along a single direction: that direction defines a one-dimensional subspace, and linear algebra gives the unique split into components along it and orthogonal to it. Softmax then gives the loading its tempered-prior interpretation.
The conceptual starting point is sufficiency. Unigram statistics specify a single probability distribution. Through the softmax readout, this distribution can be represented by a single logit vector: for positive probabilities , choosing logits recovers after softmax. Our fitting procedure identifies a prediction-space vector whose image under the unembedding approximates these log probabilities, up to an additive constant. Figure 2 establishes the quality of this encoding. Thus a single direction suffices to encode the unigram prior; varying its coefficient changes the strength of that prior.
Once this one-dimensional subspace is identified, the orthogonal decomposition follows from standard linear algebra. With the usual inner product, the prediction space decomposes uniquely into and its orthogonal complement. For unit , any prediction state therefore has the decomposition
The linear unembedding maps these two components to additive logit contributions, and softmax turns them into multiplicative probability factors. With the encoded prior and ,
The decomposition therefore rests on two ingredients: the unigram distribution being encodable along a one-dimensional subspace of the prediction space, and the softmax readout that gives its loading a tempered-prior interpretation. Orthogonality supplies the natural, unique separation of the state into its component along that prior direction and the remaining contextual component. Section 3.5 presents this factorization, with the fitting and decomposition details in Appendix B.
Q3. Unknown training distributions
We thank the reviewer for highlighting this subtle point about the unknown pretraining distribution. Our additional results show that the decrease in prior loading with model size is not an artifact of choosing Pile as the reference corpus: the same ordering holds with FineWeb and C4 in both tested model families. All six paired differences have positive 95% confidence intervals. The table below reports the direct comparison:
| Reference corpus | Llama 3.2: Δλ [95% CI] | Qwen2.5: Δλ [95% CI] |
|---|---|---|
| Pile | 0.396 [0.388, 0.403] | 0.223 [0.206, 0.242] |
| FineWeb | 0.302 [0.295, 0.308] | 0.109 [0.093, 0.123] |
| C4 | 0.333 [0.327, 0.339] | 0.108 [0.092, 0.123] |
The paired difference and bootstrap procedure are as defined in C4 above; corpus comparisons use matched-size samples and a common supported vocabulary within each tokenizer. All six intervals are positive, showing preservation of the model-size ordering across reference corpora. This establishes robustness of the observed comparison without requiring knowledge of a model's exact pretraining distribution.
Thank you for the rather extensive response to the few concerns and questions that I had. I've raised my score to reflect my view of the paper, I think this is good and very interesting work.
On reading your response I started wondering whether one might see a difference between "non-reasoning" models such as Llama 3.2 and "reasoning" models such as Qwen 2.5. There seems to be a difference in how large is for Llama 3.2 and Qwen 2.5, and one might imagine that in post-trained models such as the Qwen family, the prior loading factor would generally be lower. Just an additional thought that could be interesting for future work.
We thank the reviewer for appreciating the novelty and cleverness of our work, as well as the clarity of its presentation. Please find below our point-by-point response to the weaknesses and questions raised.
W1. Quantifying contextual information
This is a very insightful remark. In Figure 1, we loosely describe the amount of information through the specificity of a geographical location: Paris is more specific than Europe and therefore restricts the set of plausible continuations more strongly. This figure is intended to provide an intuitive introduction.
Later, in Figure 3, we make this notion more concrete by varying the number of context-relevant tokens and averaging over many different passages. This was our attempt to operationalize the amount of contextual evidence. We regard the context-switching experiment as a useful sanity check, since it changes the relevance of the preceding context at a fixed sequence position. The additional Spanish experiment reproduces this response in Llama-3.2-3B at both switch positions (Appendix E.1, Figure 9).
We thank the reviewer for encouraging us to clarify this point. Context length serves here as an operational proxy for available evidence, rather than a formal measure of semantic information; the shuffling and context-switch controls in Section 4 test the role of that evidence beyond token count alone.
W2 / Q4. Comparison with predictive entropy
We thank the reviewer for proposing this informative comparison, which sharpens the distinction between predictive uncertainty and reliance on the unigram prior. Predictive entropy and prior loading are not interchangeable: they frequently rank the same passages in opposite orders in Llama, Qwen and Gemma, in both English and Spanish.
We compared the two quantities on 200 English WikiText passages and 200 Spanish Wikipedia passages per model, averaging each over positions 250–299 to hold context length fixed. A pair has opposite ordering when the passage with higher predictive entropy has lower prior loading. The table below reports the fraction of such pairs; Appendix E.2 (Figure 10) shows the passage-level scatterplots.
Each comparison includes all 19,900 passage pairs. Brackets show pointwise 95% intervals from 2,000 whole-passage bootstrap replicates, preserving the dependence among pairs that share a passage. The measures can be positively associated while still assigning different orderings to many passages; the model-specific correlations are reported in the figure caption.
We have expanded Section 7 to explain this distinction: λ acts conceptually as a confidence-control knob by regulating reliance on the unigram prior, whereas entropy measures the dispersion of the full predictive distribution. A passage's entropy does not uniquely determine its prior loading. The two therefore capture distinct notions of uncertainty. Appendix E.2 provides the passage-level evidence; a detailed investigation of the types of data that elicit high entropy versus high prior loading remains a direction for future work.