acceptodds
Under review as a conference paper at ICLR 2027

The Geometry of Ignorance: How LLMs Encode and Adjust a Bayesian Prior

Abstract

What does a language model predict when it has few clues? The answer lies in its unembedding geometry: a single direction of the unembedding matrix encodes the unigram distribution of the training corpus, which serves as the Bayesian prior the model falls back on when uncertain. This structure, termed the *direction of ignorance*, appears across all model families examined (Llama, Qwen, Gemma, and Pythia), ranging from 0.4B to 405B parameters. Projecting the final prediction state onto this direction yields a per-token *prior loading factor* , which declines steadily as the context becomes more informative. Formally, the same projection decomposes the prediction state into two orthogonal vectors interpretable as the two components of a tempered Bayesian update: a unigram prior raised to the exponent and a context-driven likelihood. This geometric-probabilistic connection makes meaningfully comparable across model sizes and families. Larger models generally exhibit lower prior reliance, and reasoning models exhibit lower reliance than their base counterparts. Finally, we show that serves as a causal, inference-time control: reducing it can promote context-driven continuations and improve Llama-3.2-3B-Instruct on GSM8K, whereas amplifying it or applying matched perturbations along random directions degrades performance.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretation, Bayesian inference, natural language processing, Inference-time control, post-training

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_zAWvOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors derive a vector that encodes the unigram distribution of a language models training data after being multiplied with the models unembedding matrix. The authors name this vector the “direction of ignorance” and interpret it as a Bayesian prior of the models output when given little information. They decompose the final embedding of an LM generation into a linear combination of this vector, weighted by a factor λ\lambda, and an orthogonal residual they interpret as the information content of the given input. They argue that this lambda decreases when providing more information to the LM, and show that it can be used to steer generation and increase LM performance on tasks such as GSM8k.

Critical strengths
  1. The approach is clever, well defined and argued for

  2. Interesting approach to studying the unigram bias in LLMs and potential for applying it to steering of generation

  3. Well written and good figures

  4. Clear research question

Critical weaknesses

I emphasize that some of these weaknesses aren't "critical", but nonetheless points of contention.

  1. On multiple occasions the authors argue based on “how much information a context carries”, but this measure is never quantified. It would be interesting to quantify what information is in this context.

  2. No comparable baseline is computed. It would be interesting to the λ\lambda compared to just the entropy of the predictive distribution.

  3. The application to steering is very limited. It would be very interesting to see how this prior affects other types of tasks and whether it could be relevant as an uncertainty signal.

  4. Using the 70% most frequent tokens of a training corpus is perhaps not an arbitrary decision and it isn't argued for. This could have significant effects on the results.

Questions for authors
  1. You define the decomposition of the final prediction vector such that the prior and the context vector are always orthogonal. I get that this makes the math possible, but what is the conceptual argument for this?

  2. Have you performed any experiments to see the impact of discarding the 1-c least frequent tokens?

  3. Since we don’t know the training data of Llama/Gemma/Qwen, your unigram distribution estimates for those are based on other models training data. Could this be problematic?

  4. Would it make sense to compare λ\lambda to baseline measures related to uncertainty such as entropy?

4 comments

We thank the reviewer for appreciating the novelty and cleverness of our work, as well as the clarity of its presentation. Please find below our point-by-point response to the weaknesses and questions raised.

W1. Quantifying contextual information

This is a very insightful remark. In Figure 1, we loosely describe the amount of information through the specificity of a geographical location: Paris is more specific than Europe and therefore restricts the set of plausible continuations more strongly. This figure is intended to provide an intuitive introduction.

Later, in Figure 3, we make this notion more concrete by varying the number of context-relevant tokens and averaging over many different passages. This was our attempt to operationalize the amount of contextual evidence. We regard the context-switching experiment as a useful sanity check, since it changes the relevance of the preceding context at a fixed sequence position. The additional Spanish experiment reproduces this response in Llama-3.2-3B at both switch positions (Appendix E.1, Figure 9).

We thank the reviewer for encouraging us to clarify this point. Context length serves here as an operational proxy for available evidence, rather than a formal measure of semantic information; the shuffling and context-switch controls in Section 4 test the role of that evidence beyond token count alone.

W2 / Q4. Comparison with predictive entropy

We thank the reviewer for proposing this informative comparison, which sharpens the distinction between predictive uncertainty and reliance on the unigram prior. Predictive entropy and prior loading are not interchangeable: they frequently rank the same passages in opposite orders in Llama, Qwen and Gemma, in both English and Spanish.

We compared the two quantities on 200 English WikiText passages and 200 Spanish Wikipedia passages per model, averaging each over positions 250–299 to hold context length fixed. A pair has opposite ordering when the passage with higher predictive entropy has lower prior loading. The table below reports the fraction of such pairs; Appendix E.2 (Figure 10) shows the passage-level scatterplots.

ModelEnglish: opposite ordering [95% CI]Spanish: opposite ordering [95% CI]
Llama-3.2-1B49.0% [44.4, 53.6]44.9% [40.3, 49.5]
Qwen2.5-1.5B31.8% [28.1, 35.7]29.8% [25.5, 34.2]
Gemma-3-1B-PT39.9% [35.7, 44.3]49.2% [43.8, 54.3]

Each comparison includes all 19,900 passage pairs. Brackets show pointwise 95% intervals from 2,000 whole-passage bootstrap replicates, preserving the dependence among pairs that share a passage. The measures can be positively associated while still assigning different orderings to many passages; the model-specific correlations are reported in the figure caption.

We have expanded Section 7 to explain this distinction: λ acts conceptually as a confidence-control knob by regulating reliance on the unigram prior, whereas entropy measures the dispersion of the full predictive distribution. A passage's entropy does not uniquely determine its prior loading. The two therefore capture distinct notions of uncertainty. Appendix E.2 provides the passage-level evidence; a detailed investigation of the types of data that elicit high entropy versus high prior loading remains a direction for future work.

AuthorsOpenReview ↗

rebuttal continued 1

W3. Other tasks and applications

We thank the reviewer for this very good suggestion to investigate further tasks and applications. The added forking-path analysis shows how reducing prior loading can promote a context-relevant continuation and correct a reasoning path; the Gemma-3-12B result extends the steering evidence to a larger model. We agree that it would be interesting to investigate how this prior affects other types of tasks and whether it could serve as an uncertainty signal. We do think that our phenomenological observation has application potential.

However, we regard the goal of the current manuscript primarily as drawing attention to an interesting phenomenon: a prominent geometrical feature, the unigram embedding, and its function inside LLMs, namely prior regulation. As such, our steering experiment is intended to support this functional interpretation of the direction of ignorance, with a more limited claim about its general applicability.

To strengthen this functional interpretation, we have added a forking-path analysis in Section 5 (Figure 5), with additional examples in Appendix J.3 (Figure 16). This analysis illustrates how reducing prior loading can promote a context-relevant continuation already present in the model's predictions, leading to a corrected reasoning path. The larger-model steering result is reported in Appendix I (Figure 14 and Table 3). We believe these additions provide a useful connection between the geometric interpretation and the observed behavioral effect, while broader task evaluation remains an important direction for further investigation. We thank the reviewer for pointing out these promising directions.

W4 / Q2. Vocabulary cutoff

We thank the reviewer for bringing up this subtle point about the vocabulary cutoff and the estimation of rare-token frequencies. The decrease in prior loading with model size is preserved at every tested cutoff reported here: 40%, 50%, 60% and 70% vocabulary coverage. Llama-3.2-3B remains below 1B, and Qwen2.5-3B remains below 1.5B, with all eight paired 95% intervals above zero.

Vocabulary coverageLlama 3.2: Δλ [95% CI]Qwen2.5: Δλ [95% CI]
40%0.558 [0.549, 0.567]0.307 [0.288, 0.327]
50%0.522 [0.513, 0.530]0.272 [0.255, 0.289]
60%0.485 [0.478, 0.493]0.205 [0.190, 0.219]
70%0.416 [0.409, 0.423]0.137 [0.127, 0.147]

Δλ is the smaller-minus-larger difference in mean loading over positions 250–299, pairing the same 200 WikiText passages: Llama 1B versus 3B and Qwen 1.5B versus 3B. Fits use fixed Pile counts. Brackets show pointwise 95% paired passage-bootstrap intervals (2,000 replicates). All eight intervals are positive, showing preservation of the model-size ordering across vocabulary cutoffs.

Vocabulary coverage is the fraction of vocabulary IDs retained after ranking tokens by their observed corpus counts. At 70% coverage, we retain the most frequent 70% of IDs. The boundary count is the observed count of the least frequent retained token. Low boundary counts indicate that tokens near the cutoff have few observations, so their estimated frequencies are sensitive to which documents are sampled: even a few additional occurrences can produce a large relative change. Under a simple independent-token counting approximation, the relative sampling error scales as 1/√n for a token with count n. Rare-token log-frequency estimates, which enter the prior fit, are therefore less stable. The 70% cutoff balances vocabulary coverage against this estimation uncertainty.

For the Llama tokenizer, the boundary count falls from 25 at 70% coverage to eight at 80% in a sample of approximately 211 million tokens. For Qwen, it falls from nine to one in approximately 497 million tokens. At these empirical frequencies, matching the expected counts at the 70% boundary would require approximately 3.1× as much streamed text for Llama and 9× for Qwen at the 80% boundary. This steep sampling cost motivates our choice of 70% vocabulary coverage for the main analysis and the range summarized above.

The robustness claim concerns the qualitative comparison: fitted angles and absolute λ values do change with the cutoff.

AuthorsOpenReview ↗

rebuttal continued 2

Q1. Why an orthogonal decomposition?

We thank the reviewer for inviting us to explain this conceptual basis. The decomposition follows from the unigram prior being encodable along a single direction: that direction defines a one-dimensional subspace, and linear algebra gives the unique split into components along it and orthogonal to it. Softmax then gives the loading its tempered-prior interpretation.

The conceptual starting point is sufficiency. Unigram statistics specify a single probability distribution. Through the softmax readout, this distribution can be represented by a single logit vector: for positive probabilities p(w)p(w), choosing logits log⁡p(w)\log p(w) recovers pp after softmax. Our fitting procedure identifies a prediction-space vector βd\beta d whose image under the unembedding approximates these log probabilities, up to an additive constant. Figure 2 establishes the quality of this encoding. Thus a single direction suffices to encode the unigram prior; varying its coefficient changes the strength of that prior.

Once this one-dimensional subspace is identified, the orthogonal decomposition follows from standard linear algebra. With the usual inner product, the prediction space decomposes uniquely into span⁡(d)\operatorname{span}(d) and its orthogonal complement. For unit dd, any prediction state therefore has the decomposition

y=(y⋅d)d+ycontext,ycontext⋅d=0.y=(y\cdot d)d+y_{\mathrm{context}},\qquad y_{\mathrm{context}}\cdot d=0.

The linear unembedding maps these two components to additive logit contributions, and softmax turns them into multiplicative probability factors. With the encoded prior q(w)∝exp⁡(βWw⋅d)q(w)\propto\exp(\beta W_w\cdot d) and λ=(y⋅d)/β\lambda=(y\cdot d)/\beta,

p(w∣context)∝q(w)λexp⁡(Ww⋅ycontext).p(w\mid\mathrm{context})\propto q(w)^\lambda\exp(W_w\cdot y_{\mathrm{context}}).

The decomposition therefore rests on two ingredients: the unigram distribution being encodable along a one-dimensional subspace of the prediction space, and the softmax readout that gives its loading a tempered-prior interpretation. Orthogonality supplies the natural, unique separation of the state into its component along that prior direction and the remaining contextual component. Section 3.5 presents this factorization, with the fitting and decomposition details in Appendix B.

Q3. Unknown training distributions

We thank the reviewer for highlighting this subtle point about the unknown pretraining distribution. Our additional results show that the decrease in prior loading with model size is not an artifact of choosing Pile as the reference corpus: the same ordering holds with FineWeb and C4 in both tested model families. All six paired differences have positive 95% confidence intervals. The table below reports the direct comparison:

Reference corpusLlama 3.2: Δλ [95% CI]Qwen2.5: Δλ [95% CI]
Pile0.396 [0.388, 0.403]0.223 [0.206, 0.242]
FineWeb0.302 [0.295, 0.308]0.109 [0.093, 0.123]
C40.333 [0.327, 0.339]0.108 [0.092, 0.123]

The paired difference and bootstrap procedure are as defined in C4 above; corpus comparisons use matched-size samples and a common supported vocabulary within each tokenizer. All six intervals are positive, showing preservation of the model-size ordering across reference corpora. This establishes robustness of the observed comparison without requiring knowledge of a model's exact pretraining distribution.

Thank you for the rather extensive response to the few concerns and questions that I had. I've raised my score to reflect my view of the paper, I think this is good and very interesting work.

On reading your response I started wondering whether one might see a difference between "non-reasoning" models such as Llama 3.2 and "reasoning" models such as Qwen 2.5. There seems to be a difference in how large Δλ\Delta \lambda is for Llama 3.2 and Qwen 2.5, and one might imagine that in post-trained models such as the Qwen family, the prior loading factor would generally be lower. Just an additional thought that could be interesting for future work.

Reviewer_JmVyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper describes a single vector which encodes unigram statistics in the final hidden states encoded by an LLM. The vector is defined as that which when multiplied by the unembedding matrix produces an observed unigram distribution. It can be regressed using least squares. It decomposes into a direction d_prior and loading factor λ, which allows decoding geometry to be framed as Bayesian inference where prior strength is controlled by λ. Empirical results show interesting features of λ: λ declines as mode context increases, indicating that models rely less on the prior; λ is lower for larger models and for reasoning-trained models, indicating these models use context better; controlling λ on GSM-8K impacts results, with lower settings (increased context reliance) improving performance and higher settings degrading. Future directions of research on this and other potential encoded statistics are proposed.

Critical strengths

Framing decoding geometry as tempered Bayesian inference

Clear analysis of λ versus model size, training regime.

The analysis of Pythia training steps lends credence that this prior is encoded through training

Critical weaknesses

It’s not clear that gathering unigram statistics and computing OLS over unembedding matrices is less hassle than running forward passes of the model, contrary to the implication of contribution 1.

The quality of p_unigram as an approximation to the actual unigram statistics of a model’s training data is unknown.

Questions for authors

Unigram probability is set independently of the model, meaning that d_prior does not encode the actual unigram distribution of the text shown to a model at training but an approximation. Can the decline in λ for stronger models be interpreted as a measure of how far p_prior deviates from the unigram distribution of a model’s training data, rather than that these models have less reliance on unigram priors?

1 comment

We thank the reviewer for recognizing the insight and rigor of our analysis and for appreciating the clarity of its presentation. Please find below our point-by-point response to the weaknesses and questions raised.

W1. Practical advantage of contribution 1

We thank the reviewer for prompting us to clarify the purpose and practical value of this contribution. Contribution 1 identifies how the unigram prior is embedded in the model and provides a direction whose loading can be measured and causally manipulated. Our main goal is mechanistic interpretation: to identify this prominent geometric feature and explain its function as context-dependent prior regulation. A pure forward run tells us the model's final prediction, but it does not by itself reveal this internal representation.

The OLS fit is a one-time, model-level cost: it uses the output embedding and corpus unigram statistics to identify this direction (Section 3.3; Appendix B). Once fitted, the direction can be reused for every subsequent input. In particular, the steering in Section 5 (Figure 4) uses the already fitted direction directly during the ordinary forward pass; it requires no additional model forward passes, activation collection, or per-input optimization. This makes the result useful both as an interpretable diagnostic of when a model relies on its prior and as a lightweight control coordinate for testing or adjusting that reliance at inference time.

W2. Quality of the unigram reference

We thank the reviewer for emphasizing the quality of the unigram reference. Figure 2 establishes that the unigram reference is well represented in the model's unembedding geometry, with per-model fit statistics in Appendix C (Table 1). Our additional experiments preserve the model-size ordering across Pile, FineWeb and C4. These comparisons use identical supported vocabulary within each tokenizer. The paired results are included in the table below, together with the cutoff analysis addressing the related question.

Both tested within-family size comparisons survive all three reference choices. Furthermore, across Pile-versus-web fits, per-passage late-context loadings remain strongly correlated, with Pearson correlations of 0.863–0.974. These results support the stability of the functional comparisons across alternative unigram references. They do not require the three corpora to have identical token frequencies.

Question. Could lower loading in larger models reflect proxy mismatch?

We thank the reviewer for bringing up this subtle alternative explanation. The observed decrease in prior loading with model size survives every reference-corpus and vocabulary-cutoff comparison reported below, arguing against a peculiarity of the Pile reference as its explanation. All 14 paired differences have positive 95% passage-bootstrap intervals. The analysis holds the reference distribution and fitting support fixed across sizes within each family, then repeats the comparison under Pile, FineWeb and C4 and, separately, across four vocabulary cutoffs (40–70%). Llama-3.2-3B has lower late-context λ than 1B, and Qwen2.5-3B has lower λ than 1.5B, in every case.

These results give us confidence that the observed comparisons are robust to the choice of unigram reference and vocabulary cutoff. We therefore do not think that a peculiarity of the original Pile estimate is a likely explanation for these comparisons. We appreciate the reviewer drawing attention to this question and provide the paired evidence directly here. This argument does not require assuming that all checkpoints within a named family were trained on exactly the same data.

Paired analysis across reference corpora and vocabulary coverage

TestSettingLlama 3.2: Δλ [95% CI]Qwen2.5: Δλ [95% CI]
CorpusPile0.396 [0.388, 0.403]0.223 [0.206, 0.242]
CorpusFineWeb0.302 [0.295, 0.308]0.109 [0.093, 0.123]
CorpusC40.333 [0.327, 0.339]0.108 [0.092, 0.123]
Coverage40%0.558 [0.549, 0.567]0.307 [0.288, 0.327]
Coverage50%0.522 [0.513, 0.530]0.272 [0.255, 0.289]
Coverage60%0.485 [0.478, 0.493]0.205 [0.190, 0.219]
Coverage70%0.416 [0.409, 0.423]0.137 [0.127, 0.147]

Paired analysis: Δλ is the smaller-minus-larger difference in passage-level mean λ over positions 250–299, pairing the same 200 WikiText passages across models. Comparisons are Llama-3.2-1B versus 3B and Qwen2.5-1.5B versus 3B. Brackets show pointwise 95% paired passage-bootstrap intervals (2,000 replicates). All 14 intervals lie above zero, showing that the ordering is preserved across the tested reference corpora and vocabulary cutoffs.

Corpus and coverage checks are separate experiments: corpus comparisons use a common supported vocabulary; coverage comparisons use fixed Pile counts.

Reviewer_XZYMOpenReview ↗
Rating
4(+1)
Confidence
4(+2)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This works shows that they can decompose the unembedding matrix into two components one resembling the prior over the dataset and another the likelihood. They show that the prior component exists and is universal across the examined models. There perform several experiments on this component showing that reducing improves performance on mathematics and that the component expressed more strongly is more uncertain contexts such as the start of a text or under text manipulations (fig. 3)

Critical strengths
  • they show that prior component exists universally (across examined models), is unique and can be extracted efficiently.
  • They show that this component has practical application by showing improved performance for some models on GSM8k (the effect is notably strong for llama 3.2 3B)
Critical weaknesses
  • figure 4 and figure 1 I would consider misleading. In both cases they show a very clear picture which upon further inspection is not as clear as presented. I would in both cases replace two present with multiple figures which reasonably present the diversity.
  • Scope is somewhat narrow for the claims of universality: E.g. focuses only uses one corpus to extract the prior but does not explore the effect of varying that corpus, similar experiments are performed only on english corpora and generalization is not shown across languages. Similarly the effect of reducing the loading is also only shown on a single downstream dataset.While these weaknesses are already mentioned in the limitations, I think they are important to consider given the claims or adjusting the claims appropriately.
Questions for authors
  • Seems like the experiments are not consistently applied to all the examined models, e.g. why was GSM8k sweeps not performed to the larger llama models? I would suspect that these to be particularly interesting given its reliance on the prior loading factor.
4 comments

Thank you for recognizing the universality of our proposed phenomenon of interest and appreciating the practical ramifications of our work. Please find below our point-by-point response to the weaknesses and questions raised.

Figures 1 and 4

Thanks for pointing out this limitation. We believe Figure 1 is quite representative of the prior embeddings found across the examined models: a spatial gradation from rare to common tokens. However, certain features, such as the bifurcated, three-beam-shaped distributions found in larger models, do appear geometrically different. We thank the reviewer again for this suggestion. Appendix D (Figure 7) presents the corresponding PCA visualizations across all examined models; Appendix F (Figures 11–12) examines the bifurcated structure in larger Llama models.

Similarly, Appendix I (Figure 14) provides the model-by-model prior sweeps corresponding to Figure 4. If space permits, while accommodating the other review suggestions, we would be happy to expand the main-text illustrations with examples from different model sizes and families in the camera-ready draft. The current draft is already close to the ten-page limit after adding the forking-path analysis (Section 5, Figure 5), which connects prior regulation to changes in the reasoning trajectory. Therefore, in the current manuscript, we direct the reviewer to these appendix comparisons.

Scope and robustness

We thank the reviewer for encouraging us to examine the scope of the finding more broadly. Our additional results preserve the model-size ordering across Pile, FineWeb and C4, and reproduce the contextual effects on Spanish Wikipedia.

Reference-corpus robustness. The decrease in prior loading with model size persists across all three reference corpora, showing that the observed ordering is not an artifact of choosing Pile. For every reference, Llama-3.2-3B exhibits lower late-context loading than Llama-3.2-1B, and Qwen2.5-3B exhibits lower loading than Qwen2.5-1.5B. All six paired differences below have positive 95% passage-bootstrap intervals. These comparisons use matched-size corpus samples and identical fitting support within each tokenizer.

Reference corpusLlama 3.2: Δλ [95% CI]Qwen2.5: Δλ [95% CI]
Pile0.396 [0.388, 0.403]0.223 [0.206, 0.242]
FineWeb0.302 [0.295, 0.308]0.109 [0.093, 0.123]
C40.333 [0.327, 0.339]0.108 [0.092, 0.123]

Δλ is the smaller-minus-larger difference in mean loading over positions 250–299, pairing the same 200 WikiText passages: Llama 1B versus 3B and Qwen 1.5B versus 3B. Brackets show pointwise 95% paired passage-bootstrap intervals (2,000 replicates). All six intervals are positive, showing preservation of the model-size ordering across reference corpora.

Extension beyond English. To test generality beyond English, we replicated the contextual controls on 200 Spanish Wikipedia passages, matching the English sample size and using the original fitted directions without language-specific refitting. Appendix E.1 (Figure 9), now introduced in Section 4, follows the main-text layout, comparing coherent passages in Llama 1B, 3B and 8B with a shuffled Llama 1B control. Prior loading decreases with model size on coherent Spanish passages (late-context means: 0.432, 0.146 and 0.078 for 1B, 3B and 8B). Token shuffling increases loading relative to coherent text, and switching to an unrelated article produces a transient upward response in Llama 3B. This supplies additional evidence that the contextual effect extends beyond the original English evaluation.

Larger-model sweeps

We thank the reviewer for this insightful suggestion to examine larger models. The added Gemma-3-12B experiment shows a 17.8% reduction in remaining errors at α=0.5, extending the steering evidence to a larger model. We have tried our best to accommodate this request within our computational resources and research budget.

For each model, a full sweep of 16 nonbaseline settings over 1,536 problems requires 24,576 additional reasoning trajectories, in addition to the baseline. Performing this sweep on the 70B and 405B models therefore exceeds our available computational capacity and budget.

To address the reviewer's suggestion as far as possible, we have included Gemma-3-12B-Instruct in the prior-sweep experiments (Appendix I, Figure 14 and Table 3). At α=0.5, accuracy increases from 93.0% to 94.3%, with 29 baseline errors corrected and 10 correct answers broken—a 17.8% reduction in remaining errors. The nominal exact McNemar p-value is 0.0034; as stated in Table 3, this is unadjusted for the post-hoc choice of α.

We would like to thank the reviewer again for this suggestion. Although we cannot fulfill the request exactly because of our budget constraints, we hope that this additional 12B result helps address the reviewer's interest. Extending these sweeps to still larger models remains an important direction for further investigation.

I have read the answer and it indeed improved some of the concerns.

In addition to those I also did a after my initial reading of the paper I followed it up with some personal experiments.

Here is some findings that I found:

  1. Fitting in on a 1b model trained two language show that their prior loading were nearly orthogonal (cos(d_lang1, d_lang2)=-0.02). This does not discount the fact that it is there, but at least that it is the same across languages. This was on a bilingual model, so I suspect that it might encode stuff differently, but I would find it hard that the direction is universal across languages. This experiment of course differs from your own in that you do not do a language specific refit

  2. I also wanted to test if steering across the dimension would improve performance (does less reliance on the prior) improve downstream performance across languages and here I generally found improves performance across 3/4 languages (though the threshold differs) and the effect were largest dan, nob and smaller for eng. It was not there for deu.

Here i used the multilingual GSM templates from: danish-foundation-models/multilingual-gsm-symbolic

  1. Lastly wanted to see if this only applied to mathematics or if the effects were more general so I applied to EuroEval. Here I saw quite large improvements for Zebra (logic puzzles), and smaller effect for summarization but negative effect for reading comprehension, though for many of the task the effects were mixed.

Overall my impression is that to a large extent the conclusions in the paper hold, but that the performance gains for steering a more muddy and I am afraid they only apply to smaller models and specific contexts.

That being said I would be happy to see this paper published and have changed by score accordingly.

“Fitting in on a 1b model trained two language show that their prior loading were nearly orthogonal (cos(d_lang1, d_lang2)=-0.02).”

This is a really striking observation, which we think may be related to Figure 11 in our current appendix: even in predominantly English-trained models such as LLaMA, higher-order priors appear to exist. For example, the beams representing full words and subwords become increasingly separated and nearly orthogonal in larger models. The bilingual prior separation you observed could be another example of this structure.

These bifurcated priors could be useful when the model is confident about the rough category of the next token—full word versus subword, or, in your case, language—but unsure of the specific token within that category. We speculate about such higher-order priors around line 486 under “LLMs as hierarchies of priors.” We are excited and grateful that you found potential evidence of them in bilingual models. This also helps clarify why the prior directions need not be universal across languages.

“I generally found improves performance across 3/4 languages (though the threshold differs) and the effect were largest dan, nob and smaller for eng. It was not there for deu.”

Could you clarify whether you measured improvement using answer accuracy or test-time perplexity? If it is perplexity, we believe this is a new type of benefit that we have not seen in our experiments with LLMs trained primarily on English corpora.

“Here I saw quite large improvements for Zebra (logic puzzles), and smaller effect for summarization but negative effect for reading comprehension, though for many of the task the effects were mixed.”

Fascinating. We did suspect prior tuning would be most effective on tasks that require reasoning and logical deduction, and we thank you for finding evidence consistent with this. As argued alongside Figure 5 of the revised manuscript, we think that “Reducing the prior therefore does not miraculously add reasoning capacity; it promotes context-driven deductions already manifest in the model’s predictions.”

The mixed effects on other tasks are also important, and we agree that prior tuning should not be presented as a universal performance boost.

“I am afraid they only apply to smaller models and specific contexts.”

In Appendix I of our updated manuscript, we added experiments on Gemma 3 12B Instruct, which show improved reasoning performance at α=0.5\alpha=0.5. This extends our evidence to a larger model, although we agree that the benefits may depend on the context. As explained earlier, we could not conduct similar sweeps on still larger models due to limited compute.

We would like to thank you again for conducting such in-depth experiments. If you are willing to share the model identifier and fitting and steering configurations, these would help us connect your additional experiments to our setup.

We also respect your intellectual contribution and would ask for your permission before incorporating any of these observations into the revised discussion, clearly identifying them as findings you shared with us.

“That being said I would be happy to see this paper published and have changed by score accordingly.”

We sincerely appreciate your encouraging words and tremendous effort in helping make the paper more rigorous and thorough. We noticed that your overall rating still appears as 3, while the confidence has increased, and wondered whether you intended to adjust the overall score. We fully respect your current rating; we just wanted to make sure the interface reflects your intended update.

Could you clarify whether you measured improvement using answer accuracy or test-time perplexity?

Accuracy

As explained earlier, we could not conduct similar sweeps on still larger models due to limited compute.

Totally understandable.

If you are willing to share the model identifier and fitting and steering configurations, these would help us connect your additional experiments to our setup.

I used the recent mimir 1.5. Released as a part of this paper: MIMIR: A SUB-2B HIERARCHICAL REASONING MODEL TRAINED ON PERMISSIBLE DATA PUNCHES ABOVE ITS WEIGHT CLASS

So quite an unusual model (Looped transformer), but wanted something that was somewhat out of distribution (I also wanted to do some experiments on applying it to the different staged in processing and the looped architecture made that output represenation more consistent). Applying it to earlier layers didn't really work though.

For the steering configuration I obtained overall positive results with a prior multiplier of -0.5 and -0.25 with -0.5 for the math problems. The results were similar when applied to euroeval, -0.5 improved reasoning tasks more, but also more negatively other tasks.

... we fully respect your current rating; we just wanted to make sure the interface reflects your intended update.

Thanks and it was indeed intending to change both. Should be fixed now

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.