acceptodds
Under review as a conference paper at ICLR 2027

How Do Language Models Manage Uncertainty?: A Case Study of the Number Game

Abstract

Language models must navigate ambiguity to successfully complete sentences or perform tasks; however, we have little mechanistic understanding of how they represent or handle such uncertainty. To understand this better, we study how Llama-3.1 (8B) plays the Number Game, a concept learning task in which participants view a set of numbers that conform to an unknown concept, and must determine which other numbers conform. Llama's behavior, like humans', appears Bayesian; we search for the mechanisms underlying this seemingly normative behavior. We first find that Llama's representational geometry constrains the concepts that it considers as hypotheses, allowing it to consider only a specific range of possible hypotheses. We next find that Llama has internal circuitry that identifies common concepts in its input, and downweights poorly supported ones. Finally, we find that Llama's increased confidence upon seeing more inputs stems largely from an attention mechanism that simply tracks the number of examples seen. We use these insights to design a new model that better predicts Llama's behavior, including on new concept types. In sum, we offer a detailed account of how Llama manages its uncertainty and what mechanisms underlie this capacity.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, number game, uncertainty, language models, bayesian reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QksPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates how the internal representations of the Llama family of models allow such models to perform a concept learning task (the Number Game) and how this Llama behavior matches and differs from Bayesian and human learning. The main paper is in the form of a case study of a particular Llama model in this context. The results provide an interesting direction for research but the generality of the result is debatable given that only a single model family and a single type of learning task is studied in the paper.

Critical strengths
  • the general problem, of understanding how language models come up with internal hypotheses for concept classes, is important and interesting

  • the paper is generally clearly written

  • the Appendices provide extensive details about the methods and experiments in the paper

Critical weaknesses
  • Focusing the case study only on Llama models and on a very specific task (the Numbers Game) significantly limits the potential impact of the paper and the generality of the conclusions drawn. I am aware for example that prior related work also focused largely on Llama models: but this still seems like quite a limiting factor (even if other studies made the same choice). It would be quite a bit more interesting and useful to the AI research community if a study of this nature could analyze results across multiple SOA open-weight models and assess if they all have similar characteristics or not. It would also be helpful to validate these results on a different type of number-related learning problem. I would encourage the authors to pursue these extensions in future work on this topic. As a sidenote, the results on the larger 70B model in Appendix ) seem to suggest that although there are similarities with the model in the main paper, there are also some differences, providing further motivation for analyzing these behaviors acros a broader range of model sizes and families.

  • Bayesian modeling of LM Behavior (lines 130 to 146): its interesting that the proposed Bayesian model fits well, but I would like to see here some discussion of alternative potential models, e.g., is this particular model giving significantly better predictions than other candidate models we might consider, or are there multiple plausible models for Llamas's outputs. This seems like a crucial question.

  • Similar comments apply to the model in Section 6 and Appendix N (even more so, given that this is one of the main proposed contributions of the paper). The authors propose a more complex Bayesian model (than the original) that they argue more closely reflects the mechanisms underlying how Llama reasons about the Numbers Game. But its not clear what this is really telling us. Are there other simpler (or equivalent complexity) models that would also achieve such Kl scores? How sensitive are the results to the choices for the priors or the specific choices for the 6 families? (Appendix N).

Questions for authors

No specific questions beyond Weaknesses above

Reviewer_cuizOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how Llama-3.1-8B represents and updates uncertainty in the Number Game. The authors identify Fourier-based belief representations, attention heads that aggregate numerical features, and circuits related to input consistency and evidence accumulation. Based on these findings, they propose a revised Bayesian model to predict Llama's behavior.

Critical strengths

The paper studies an interesting question about how language models represent and update uncertainty. The paper provides detailed mechanistic analyses, including probing, circuit identification, and causal interventions.

Critical weaknesses
  1. The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors has already studied belief representations and causal interventions in numerical prediction tasks. Arithmetic in the Wild: Llama Uses Base-10 Addition to Reason About Cyclic Concepts has identified Fourier features in Llama. Although this paper provides additional circuit-level analysis, it is unclear what fundamentally new understanding of uncertainty emerges beyond these existing findings.
  2. Section 4 identifies context-consistency circuits, but Appendix J explicitly acknowledges that these circuits have not been shown to directly cause the nonlinear increase in logits as more inputs match a hypothesis. Similarly, Section 5 shows that manipulating attention patterns recovers 71% of the performance difference between shorter and longer inputs, leaving part of the evidence accumulation mechanism unexplained.
  3. In Section 6, the new model reduces KL divergence from 0.074 to 0.068, despite using more parameters (71 vs. 67). Moreover, the new model explicitly includes hypotheses and compositional structures that the original Bayesian baseline cannot represent. Therefore, the improvements on new concept types do not clearly demonstrate a better model of Llama's inference process. A comparison with stronger Bayesian baselines using the same hypothesis space would be more convincing.
  4. The main circuit analysis focuses on Llama-3.1-8B and the Number Game. Although Appendix O extends the representation analysis to Llama-3.1-70B, the full circuit mechanisms are not similarly validated. Furthermore, Appendix M shows that the proposed BOS-attention mechanism is much less effective on some other mathematical tasks, including addition and subtraction. This limits the generality of the proposed explanation.
Questions for authors
  1. What specific mechanistic insights does this paper provide beyond The Shape of Beliefs and Arithmetic in the Wild?
  2. Can the authors directly demonstrate that the identified context-consistency circuits cause the nonlinear confidence changes observed in Section 4?
  3. Can the authors compare the proposed Bayesian model against a baseline with the same hypothesis space and similar flexibility?
  4. Does the proposed mechanism generalize to other models and tasks beyond the Number Game?
Reviewer_JNxYOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how Llama-3.1 (8B) represents and manages uncertainty on Tenenbaum's Number Game, a concept-learning task in which the model sees numbers drawn from an unknown rule and must predict the next number. Behaviorally, a fitted Bayesian model (KL 0.074) predicts Llama's outputs reasonably well. Mechanistically: (i) using DAS, the authors localize a ~32-dim belief subspace (learnable from layer 15, fully effective by layer 23) and show hypotheses are encoded via Fourier features — period-2/5/10 planes for multiples-of-n/ends-in-n, long-period first-five harmonics for intervals — so representational geometry both supports certain hypotheses (multiples of 6 as the conjunction of "multiple of 2" and "multiple of 3") and precludes others (8B never learns multiples of 7/11/13/14/17/19); (ii) averaging attention heads construct belief manifolds as weighted means of per-number Fourier features (verified via LEACE and pullback probes), while "context-consistency" circuits (e.g., A5.H22 aggregating parity; M5.N10478 computing parity-match/XNOR) downweight inconsistent evidence, producing near-exponential logit growth in the fraction f of matching examples with a threshold near f = 0.4; (iii) growing confidence with more examples stems largely from attention shifting off the BOS token onto content tokens, yielding hyperbolic (rather than power-law) evidence accumulation (median R² 0.995 vs 0.981). The authors then build a mechanism-grounded "product of Bayesian experts" model (71 params; noise-tolerant, hyperbolic evidence scaling, non-normalized cross-manifold conjunction) that beats the fitted Bayesian baseline (67 params) on pooled KL (0.068) and on three novel concept types (x ≡ 1 mod 5, conjunctions, disjunctions). Appendix O replicates the manifold and steering results on Llama-3.1 70B.

Critical strengths
  1. From behavioral fit → geometry (DAS + probes) → components (LEACE + circuits) → causal verification (analytic-manifold steering, recovery near 1.0 at α=2) → a mechanism-grounded model that beats the purely behavioral Bayesian baseline on novel concepts. Most interpretability papers stop at localizing components; this one predicts behavior from mechanism, which is rare and valuable.
  2. Attributing concrete behavioral asymmetries (learns multiples of 6 via conjunction, fails multiples of 7 for lack of a manifold) to specific representational structure, and confirming it causally via steering, goes well beyond generic "model bias" claims and offers a paradigm for geometry–capability research.
Critical weaknesses
  1. The evidence does not support the title. All core claims come from Llama-3.1 8B on the Number Game (single-token integers 1–999), a setting unusually favorable to periodic representations. The general claim in the title/abstract has no corresponding breadth of evidence: the Appendix M transfer results are themselves mixed (the BOS-only edit fails on addition), and the manifold story has no cross-model or cross-task validation.
  2. The final model's quantitative contribution is weak, and its validation is circular by construction. The KL gain is tiny (0.068 vs 0.074) and bought with +4 parameters (no AIC/BIC reported); the model is designed to implement the discovered mechanisms, so its advantage on mechanism-predicted behaviors is architecturally assured; "generalizes to new settings" does not go beyond the same task and model. This makes "mechanistic model beats Bayesian model" more decorative than decisive.
Questions for authors

See Above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.