acceptodds
Under review as a conference paper at ICLR 2027

Transformer Provably Hallucinates on Hierachical Classification

Abstract

Large language models (LLMs) can generate fluent and seemingly credible responses that nevertheless contain false statements or unsupported claims, a phenomenon commonly referred to as hallucination. In this paper, we theoretically investigate one particular form of hallucination through a hierarchical sequence classification problem, where a common signal token encodes a coarse label, and a rarer nuance token encodes a fine-grained label. We characterize hallucination as a model having high predicted probability mass on the correct coarse class but near-chance probability on the correct fine-grained label within that class. We prove that a shallow transformer trained by gradient flow exhibits this behavior when nuance tokens are sufficiently rare, and token noise is small. Our results identify a mechanism through which the interaction between data frequency and learned attention can make hallucination a prolonged intermediate phase of training: The model first learns the common signal, but attention weights to the signal tokens further suppress learning from the already rare nuance, producing a long training plateau where the model hallucinates on input sequences containing the fine-grained cue from nuance tokens. Experiments on an LLM-generated synthetic BirdText dataset exhibit analogous coarse-to-fine learning behavior.

Primary area:
learning theory
Keywords:
Transformer, Hallucination, Gradient Flow

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_EXQwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies hallucination in Transformers from a theoretical perspective through a hierarchical classification task. The authors construct a data distribution containing frequent coarse-grained signal tokens and rare fine-grained nuance tokens. Hallucination is defined as a gap in the model's average predictive probabilities: the correct coarse class receives high probability, whereas the probability assigned to the correct full label remains near chance even when the fine-grained cue is present in the input. For a simplified single-head Transformer, the authors exploit symmetry to reduce its gradient flow exactly to a system of four scalar ODEs (Theorem 1). They prove that the model undergoes a prolonged hallucination plateau when fine-grained information is sufficiently rare and token noise is sufficiently small (Theorem 2). They further analyze the effect of noise on learning dynamics (Theorem 3) and provide experiments on synthetic classification tasks and BirdText, using Transformers, a classifier built on frozen BERT features, and GPT-2.

Critical strengths

S1. Technical depth of the theoretical analysis.

The authors exploit symmetries in the data distribution and model parameters to reduce the high-dimensional Transformer gradient flow exactly to a dynamical system with four scalar variables. They also explicitly characterize the time scales of different learning phases. In particular, Theorem 2 quantifies the relationship between the hallucination plateau and the frequency of fine-grained information, going beyond a qualitative description of the learning behavior. This gives the work technical value.

S2. The paper examines the interaction between attention and the data distribution to explain delayed fine-grained learning.

Beyond considering the difficulty of learning rare information, the paper analyzes how attention learned early in training concentrates on common signal tokens and subsequently suppresses learning from nuance tokens. This provides a concrete mechanism for understanding why a model can fail to use relevant information even when that information is available in its input.

Critical weaknesses

W1. Limited substantive novelty over prior work on stage-wise learning and hallucination mechanisms.

The tendency of neural networks to learn easier or more frequent features before more complex or rare ones is a familiar pattern in learning dynamics. In more directly related work, Nichani et al. (2025) have analyzed stage-wise learning and intermediate hallucination in Transformer factual recall; Gong et al. (2025) have provided a theoretical analysis of two-stage feature learning in Transformers; and Zucchet et al. (2025) have studied plateaus in factual learning, the effects of the data distribution, and attention-related mechanisms.

The main addition here is to introduce a probability governing the presence of fine-grained cues and characterize the time scale of the hallucination plateau under specific model and data assumptions. While this result has technical value, the paper does not make sufficiently clear what more fundamental mechanistic understanding this quantitative result adds beyond prior theory. In particular, it needs to distinguish more clearly between general stage-wise learning, delays caused by rare information, and the additional suppression attributed to attention. Thus, although the theoretical results apply to a new setting, I find the advance over prior work and its significance insufficiently compelling.

W2. The independent effect of learned attention is not sufficiently validated empirically.

The paper attributes the prolonged hallucination plateau to the joint effects of rare fine-grained information and learned attention. Theorem 2 establishes the corresponding time scales, and Figure 2 shows attention concentrating on signal tokens. However, Appendix B.2.2 also reports similar coarse-to-fine learning in MLPs and LSTMs on the same data, showing that this phenomenon itself is not unique to Transformers or attention.

The authors further conduct cue-deletion and cue-swapping experiments on BirdText. These experiments primarily test whether the model uses the relevant input features at different training stages; they do not directly isolate how attention affects learning speed during training.

The theoretical analysis therefore offers an explanation in which attention further suppresses the learning of rare features, but the experiments still lack direct controls that quantify the additional delay caused by this mechanism beyond data rarity alone. This limits the evidence for the paper's core mechanistic explanation in more general models.

Overall, the reduction to a four-dimensional dynamical system and the characterization of learning time scales have technical value. However, the substantive advance over prior work and the evidence for the independent role of learned attention are not sufficiently compelling. My current rating is 2. If the authors clarify the substantive advance over prior theory and provide more convincing evidence for the independent role of attention, I would consider raising my rating to 3.

Questions for authors

Q1. Distinction from prior theoretical work.

What is the most important new theoretical insight relative to existing analyses of stage-wise learning, such as Nichani et al. (2025) and Gong et al. (2025)? In particular, beyond quantifying an established stage-wise learning phenomenon, does the dependence of the hallucination plateau on the fine-grained cue probability pp in Theorem 2 reveal a fundamental mechanism that previous theories cannot explain? Please clarify the differences from the closest prior work in both theoretical conclusions and research significance.

Q2. Independent contribution of attention.

Could the authors use fixed or uniform attention, interventions on the evolution of attention weights, or other suitable theoretical or experimental controls to distinguish the delay caused by data rarity itself from the additional delay caused by learned attention? In particular, can the independent effect of attention on the duration of the hallucination plateau be quantified to provide more direct support for the proposed core mechanism?

Q3. Robustness to the theoretical assumptions.

The four-dimensional ODE reduction and the main results rely on strong assumptions, including a uniform label distribution, orthonormal token patterns, and zero initialization. Could the authors discuss or provide experiments showing whether the hallucination plateau and attention-induced suppression persist when these assumptions are moderately relaxed, for example with non-orthogonal features or a non-uniform label distribution? Which assumptions are essential to the core conclusions, and which mainly simplify the analysis?

3 comments

We thank the reviewer for the review. Please see below for our responses to your concerns:

Comparison with prior works: We respectfully disagree with the reviewer's view that our manuscript has limited novelty compared to prior work: Gong et al. 2025 studied stage-wise learning, but the problem is for in-context learning, and their settings do not have a properly defined hallucination phase; Zucchet et al. 2025 focus mostly on empirical studies; The closest to ours is Nichani et al. 2025, and we provided a detailed comparison with Nichani et al. 2025 in the original submission. The first key difference is the cause of hallucination: We study the case when the nuance information might not be available in a new type of general sequences, whereas Nichani et al. 2025 concerns the case when the number of fine-grained classes is much greater than the coarse ones, i.e., L ≫ K (stated in our terminology); therefore, we study two different factors that may cause the model to hallucinate. The second key difference is that we have shown an explicit dependence of the length of the hallucination plateau on the nuance probability p, while Nichani et al. 2025 have only shown the existence of a time epoch when hallucination occurs and do not provide bounds for the plateau length. In our view, all of them are new understandings of the fundamental mechanism behind this particular hallucination behavior. Moreover, based on our new theoretical understandings, we have designed natural language domain tasks (albeit synthetic), and have probed this mechanism in more practical settings, showcasing the practical relevance of our results.

Independent contribution of the attention We thank the reviewer for the suggestion. In the newly added Section B.3 in our revised manuscript, we conduct attention intervention experiments following a very similar idea to the one in your review. If the attention concentration towards coarse tokens prolongs the plateau, then amplifying (attenuating) the attention should extend (reduce) the plateau length, which can be controlled by introducing an inverse temperature to the softmax attention in the model. Indeed, we observed in the new experiments that perturbing the inverse attention β\beta (default β=1\beta=1) has the same effect on the plateau length as we conjectured, for a simplified model and a one-head, one-layer full transformer. However, such an effect almost diminishes for a multi-layer transformer; we conjecture that this is because the network depth also has an effect on this coarse-to-fine learning, and the attention intervention would not be effective unless the depth effect is addressed as well, which could be investigated in future work.

Robustness to the theoretical assumptions: We thank the reviewer for the suggestions. We believe the hallucination plateau we studied in this manuscript can be robustly observed even when our data assumptions are violated, most of which are imposed to make the analysis tractable. However, we also feel that testing such robustness requires careful designs and extensive experiments, which can not be done during the rebuttal phase. Nonetheless, we conducted some preliminary experiments along the lines of what the reviewer suggested, in the newly added sections B.4.1 and B.4.2. In particular, when we introduce token correlations (B.4.1), the correlation slightly delays both the coarse and fine learning, but a similar hallucination phase persists. When we consider a non-uniform prior (B.4.2), the rarity of the fine class now varies, and the rarer fine classes are learned later, which is somewhat expected but a very interesting observation to study in future work.

Thanks for the clarifications and additional experiments. My main concerns have been adequately addressed, so I will raise my rating to 3.

Thank you very much. We believe the additional experiments you suggested have made the empirical results in this manuscript more comprehensive, and we will incorporate the clarifications as we revise the main paper.

Reviewer_YhC7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a stylized hierarchical classification problem on sequences where every input contains a coarse-label cue and a fine-label cue. A shallow softmax-attention model with a trainable prediction head is optimized by zero-initialized population gradient flow. Under uniform labels and orthonormal token patterns, the paper provides theoretical results for learning dynamics. The paper interprets the intermediate probability regime as a form of hallucination.

Experiments on synthetic sequences and BirdText descriptions examine qualitative coarse-to-fine behavior beyond the analyzed setting. They include a classifier using frozen BERT features, LoRA-adapted GPT-2, and cue interventions. Empirical argmax and greedy-generation outcomes are reported separately from the predictive-probability quantities analyzed in the theorems.

Critical strengths
  1. The paper provides concrete theoretical results for softmax attention and prediction. Theorem 1 gives an exact four-scalar ODE reduction of learning dynamics. Theorem 2 gives a predictive-probability plateau with high correct coarse-class mass and near-chance correct full-label mass. Theorem 3 gives a complementary intermediate-noise bound on fine-label improvement. These constitute a concrete technical contribution beyond observing staged learning.

  2. Empirical results of controlled experiment support delayed use of available fine cues. Appendix C.1 matches BirdText's attribute pairs on information content, hides species names during generation, and screens duplicates across splits. Figure 11 reports seven-seed BirdCLS trajectories across 17 cue fractions. Figures 13-14 edit raw text before BERT re-encoding and compare cue deletion/swapping with matched noncue deletion or same-value rewriting. Figure 17 reports unrestricted GPT-2 greedy generation while retaining unmapped answers in the denominator. These controls substantiate a behavioral analogue of delayed fine-cue use. However, they do not establish the broader explanatory claim (see weakness point below)

Critical weaknesses
  1. The failure analyzed in this paper does not connect well to the real-world hallucination.

    • While there are empirical studies, the connection is still limited. BirdText asks models to classify a fixed set of species from explicitly supplied attributes, with every species represented in each split. The theoretical rarity parameter controls the presence of a fine input cue under uniform labels, however, it does not model rarity of factual identities or associations.
    • Figure 17 does establish decoded errors on this controlled task. However, it does not show that the same mechanism explains the errors in factual generation outside that specific task. Figures 13-14 establish cue sensitivity, and Appendix C.2.2 explicitly disclaims attention mediation. Contextualized BERT features and separately trained classification/attention studies further limit that inference.
    • While a theory of all hallucinations is unnecessary, a missing element here is a connection between the proved mechanism and the factual-generation failure that the paper intend to explain. Without that connection, the supported result will be limited a narrower field of hierarchical cue learning, and should not be framed as explaining hallucinations.
  2. There are prior works with similar results, which limits the significance of results of this paper. In particular, Nichani et al. already analyze gradient flow in a simplified attention model. They prove an intermediate relation-only predictor that ignores an available informative subject, followed by eventual convergence. Their appendix also bounds subject-weight growth in terms of subject frequency.

  • The remaining novelty will lie on the exact four-scalar reduction, matching pp-dependent time bounds with an explicit logarithmic factor, and the noise analysis. However, the experiments do not directly test the theorem-matched hitting times, the logarithmic factor, or γ\gamma's dependence on KK and MM. Table 1's principal simplified-model trajectories use moderate pp values. The text experiments demonstrate the already-established qualitative behavior.
  • Also, the paper does not demonstrate its practical explanatory gain over the prior works. This would limit significance even further if the scope of the claims are narrowed as in point 1 above.

Reference

Eshaan Nichani, Jason D. Lee, Alberto Bietti, Understanding Factual Recall in Transformers via Associative Memories, ICLR 2025

Questions for authors

On the closest sequential-learning analyses: what explanatory conclusion depends on the logarithmic time factor or noise result? What does that conclusion add beyond the existence of coarse-to-fine learning? Is there existing quantitative or controlled evidence validating that distinctive gain, rather than only the qualitative examples?

1 comment

Scope of this paper: We have scoped our contribution to understanding specifically hallucination modes in hierarchical classification problems, and not to fully explain real-world hallucinations. We believe that we have made the distinction between practical factual hallucination and the hallucination modes studied in this manuscript clear. In the introduction section, the first two paragraphs provide a broader introduction to hallucination and quickly narrow down to factual hallucinations in LLMs. Then we immediately introduce the hierarchical classification afterwards, motivated by the empirical findings, and from this point, the word "hallucination" refers to the prediction failure for this hierarchical classification problem, and we have a precise mathematical definition of this hallucination. We have discussed some potential gaps between the hallucination studied in this paper and practical hallucinations in several places, for example, when we compare with Nichani et al. 2025 and when we discuss the limitations of our work. Nonetheless, we believe our practical experiments on BirdText already showcase a better practical connections to real-world hallucinations, compared to other theoretical work whose experiments mainly focus on validating theorems.

Comparison with prior works: We respectfully disagree with the reviewer's view that our manuscript has limited significance compared to prior work: The closest to ours is Nichani et al. 2025, and we provided a detailed comparison with Nichani et al. 2025 in the original submission. The first key difference is the cause of hallucination: We study the case when the nuance information might not be available in a new type of general sequences, whereas Nichani et al. 2025 concerns the case when the number of fine-grained classes is much greater than the coarse ones, i.e., L ≫ K (stated in our terminology); therefore, we study two different factors that may cause the model to hallucinate. The second key difference is that we have shown an explicit dependence of the length of the hallucination plateau on the nuance probability p, while Nichani et al. 2025 have only shown the existence of a time epoch when hallucination occurs and do not provide bounds for the plateau length. In our view, all of them are distinctive gains from previous understandings of this particular hallucination behavior.

Practical gain: Moreover, based on our new theoretical understandings, we have designed natural language domain tasks (albeit synthetic) and have probed these hallucination modes in more practical settings, showcasing the practical relevance of our results; We believe the fact that we can design practical experiments based on our theoretical model and that we can do controlled experiments such as the cue deletion and swapping (in C.2.2) as suggested by our theorem already show the practical gain over prior work (for example, Nichani et al. 2025 only provided numerical validation in their theoretical setting).

Test the plateau length scaling: We have conducted additional experiments to test the plateau length for much smaller pp and for different MM, and we verified that the theoretical prediction is asymptotically accurate as pp decreases. Please see the new paragraph "Population gradient flow (B.1.3)" in Section B.1 of our revised manuscript.

Reviewer_HxQsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies hallucination in a minimal, analytically tractable setting: a two-level hierarchical classification task in which a common signal token determines a coarse label and a rarer nuance token determines a fine-grained label. Hallucination is formalized as high probability on the correct coarse class together with near-chance probability on the correct fine-grained label (Definition 1). Under a uniform label prior, orthonormal token patterns, and zero initialization, the population gradient flow of a simplified single-head transformer is reduced to four scalar ODEs (Theorem 1), and for sufficiently rare nuance tokens and sufficiently small token noise the training trajectory is shown to pass through a prolonged hallucination plateau (Theorem 2), with an intermediate-noise regime analyzed in Theorem 3. Experiments on a simplified transformer and on an LLM-generated synthetic BirdText dataset show qualitatively similar coarse-to-fine learning behaviour.

Critical strengths

Strength 1. The paper reduces "hallucination" to a minimal, provable setting. Hallucination is operationalized as "correct coarse class, near-chance fine-grained label" (Definition 1), and the whole paper is organized around two controllable quantities, the data frequency p and the token noise σ, so that the chain from frequency to attention to hallucination can be disentangled step by step. Compared with the broad narrative of "why LLMs make things up", this is a minimal model that can actually carry a theorem, and the problem is well chosen.

Strength 2. The reduction in Theorem 1 and the explicit scaling in Theorem 2 are the strongest part of the paper. Under a uniform prior, orthonormal patterns, and zero initialization, the gradient flow is exactly reduced to four scalar ODEs for any σ ≥ 0, and the plateau length is given as Θ((log 1/p)^γ/p) with γ = 2M/(K+M), followed by a Θ(1/p) transition. High-dimensional attention dynamics are compressed into an analytically tractable scalar system, and the result yields a testable scaling in p rather than a merely qualitative claim that a plateau exists.

Strength 3. The intermediate-noise regime (Theorem 3) and the cue-intervention experiments go beyond what a purely theoretical paper usually delivers. Theorem 3 does not simply say "more noise is worse"; it characterizes the window C_ε p ≤ σ²/d ≤ C_ε p_ε in which the coarse class is predicted with high confidence while the fine-grained label stays at chance, thereby accounting directly for the characteristic form of hallucination. On the empirical side, beyond learning curves the authors run cue deletion and cue swapping (C.2.2; at p = 0.1, deletion loss 0.599 and swap transfer 1.515), which comes closer to causal evidence than accuracy curves alone.

Critical weaknesses

Weakness 1. The link between theory and experiment currently stops at qualitative similarity; a quantitative connection is missing. The theorems yield a scaling law for the plateau length in p, including the testable exponent γ = 2M/(K+M), yet Figs. 2, 11, and 16 present curve shapes without any fit of τ against p or test of the predicted exponent. Consequently, "experiments exhibit analogous behavior" and "provably" remain one step apart: the reader cannot tell whether the empirical phenomenon arises from the same mechanism. We suggest fitting the plateau length against p in at least one setting (the simplified transformer is the natural choice) and reporting whether the fitted exponent agrees with the prediction for γ. The authors state clearly in Sec. 5 that the theory concerns gradient flow with a population loss on a simplified transformer, so this is a suggestion for closing the theory–experiment loop, not a claim that the theoretical setting is at fault.

Weakness 2. The constants in the theorems are opaque, so readers cannot locate the experiments within the proven regime. Theorems 2 and 3 state only that p_ε, τ_ε, c_ε, and C_ε depend on (K, L, M, T, ε), without giving their magnitude or explicit form; moreover, Theorem 2 requires σ²/d ≤ p/(log 1/p)^γ while Theorem 3 requires C_ε p ≤ σ²/d ≤ C_ε p_ε, and the relative placement of the two intervals can only be inferred by guessing at the constants. As a result, for the experimental setting (K, L, M, T) = (5, 3, 20, 10), p = 0.1, σ = 0.1 (σ²/d ≈ 10⁻⁴), one cannot determine whether it lies inside the regime covered by the theorems or outside it, where the behaviour would hold only empirically. We suggest providing numerical estimates of the constants at small scale, or a worked numerical instance of a verifiable sufficient condition.

Weakness 3. The title and conclusions claim more than what is proved, and external validity is not discussed. What is proved is a fine-grained label error under conditions of no background knowledge, a shallow/simplified architecture, a two-level hierarchy, a synthetic vocabulary, and no retrieval, whereas the title states provably hallucinates, which invites the reading that the result applies to factual hallucination in general. The gap between the two — multi-level hierarchies, real long-tailed labels, and the role of retrieval or in-context evidence — is not addressed head-on in the main text. In addition, all empirical evidence comes from BirdText (CUB-200 rendered into synthetic passages by an LLM), and results there do not automatically transfer to real hierarchical tasks. Since the authors acknowledge in the Limitations that multi-level hierarchies "require new techniques", this is not an omission but a suggestion to temper the title and conclusions so that their strength matches the scope of the proof, and to add a dedicated paragraph on how hallucination in this setting relates to factual hallucination. Finally, the boundary of the contribution relative to Nichani et al. (2025) — in particular whether the role of noise, beyond the hierarchical setting, is the main novelty — would benefit from being drawn more explicitly.

Questions for authors

Q1. Can the testable scaling of the theorems be used as a prediction? Specifically, can the authors extract the plateau length as a function of p from Fig. 2 (and Figs. 11 and 16), test τ ∝ (log 1/p)^γ/p, and report the deviation of the fitted exponent from γ = 2M/(K+M)? If (K, L, M) are varied — for instance sweeping M at fixed p — does the prediction for γ still hold? This is arguably the single most informative experiment for establishing that theory and experiment share a mechanism.

Q2. What is the explicit order of dependence of p_ε, τ_ε, c_ε, and C_ε on (K, L, M, T, ε) in Theorem 2? With the experimental values (5, 3, 20, 10) and ε = 0.1, roughly what range does p_ε cover, and is p = 0.1 inside the covered regime? If it is not, it would also help to state explicitly that the experiments serve as a mechanism demonstration rather than a verification of the theorem.

Q3. The assumptions of a uniform label prior and orthonormal patterns clearly do not hold for BirdCLS (frozen BERT features) or GPT-2 (pretrained embeddings). Do the authors expect such deviations to change only the plateau length rather than its existence? Could an intermediate case with synthetic but non-orthogonal patterns (e.g., correlated Gaussian features, or the empirical feature covariance) be used to check how far the qualitative predictions of the ODE are preserved?

Q4. The key quantity in the mechanism is the nuance-direction weight w_ν being suppressed by signal learning. Fig. 12 provides a snapshot at p = 0.1 with signal attention 0.8 and nuance attention 0.4, but not the temporal trajectory. Can the authors provide curves of nuance attention (or an equivalent quantity, such as a loss decomposition on specified samples) over training, overlaid with the trajectories of w_ν and v_ν obtained from the ODE? This would demonstrate "attention suppression causes the plateau" far better than an endpoint snapshot.

Q5. Is the intermediate-noise window of Theorem 3 observable with realistic model components? Fig. 3 already provides non-monotonic evidence in the simplified model; if noise strength is swept in BirdCLS or GPT-2 using dropout, token perturbation, or feature noise as proxies, does the regime reappear in which the coarse class is near 1 while the fine-grained label stays at chance for a long time? If not, does the role of noise in realistic models need to be reformulated?

1 comment

We thank the reviewer for the review. Please see below for our responses to your concerns:

Scope of this paper: We believe that we have made the distinction between practical factual hallucination and the hallucination modes studied in this manuscript clear. In the introduction section, the first two paragraphs provide a broader introduction to hallucination and quickly narrow down to factual hallucinations in LLMs. Then we immediately introduce the hierarchical classification afterwards, motivated by the empirical findings, and from this point, the word "hallucination" refers to the prediction failure for this hierarchical classification problem, and we have a precise mathematical definition of this hallucination. We have discussed some potential gaps between the hallucination studied in this paper and practical hallucinations in several places, for example, when we compare with Nichani et al. 2025 and when we discuss the limitations of our work. Therefore, we believe our use of the word hallucination is accurate, even in the title, because we have made it clear that this is for the hierarchical classification problem; and we have scoped our contribution specifically to these hallucination modes.

Test the plateau length scaling: We have conducted additional experiments to test the plateau length for much smaller pp and for different MM, and we verified that the theoretical prediction is asymptotically accurate as pp decreases. Please see the new paragraph "Population gradient flow (B.1.3)" in Section B.1 in our revised manuscript.

Order of the constants: The current analysis does support an explicit dependence of the constants on ϵ\epsilon, and the experiments shown in the main paper for simplified transformers do not fall into the regime where the theorem becomes quantitatively informative, as we can see from the new experiments (B.1.3) that tested the plateau length scaling. We will revise our limitations section to make this point clear.

What happens if some data assumptions no longer hold: We thank the reviewer for the suggestions. We believe the hallucination plateau we studied in this manuscript can be robustly observed even when our data assumptions are violated, most of which are imposed to make the analysis tractable. However, we also feel that testing such robustness requires careful designs and extensive experiments, which can not be done during the rebuttal phase. Nonetheless, we conducted some preliminary experiments along the lines of what the reviewer suggested, in the newly added sections B.4.1 and B.4.2. In particular, when we introduce token correlations (B.4.1), the correlation slightly delays both the coarse and fine learning, but a similar hallucination phase persists. When we consider a non-uniform prior (B.4.2), the rarity of the fine class now varies, and the rarer fine classes are learned later, which is somewhat expected but a very interesting observation to study in future work.

Q4: nuance attention trajectory: The figure the reviewer mentioned already has the full trajectory of the nuance attention.

Intermediate-noise regime: We thank the reviewer for the suggestion. However, we feel that adding noise directly to BERT features seems very artificial and does not reflect what can actually happen in practice. A more careful experimental design is needed to provide practical evidence for Theorem 3. Nonetheless, we can share what we tried when we worked on this manuscript: We had multiple BERT models with different parameter counts (building on the intuition that a smaller model is less capable of extracting information, thus making the sequence "noisier"), and we observed that with features from smaller models, both the coarse and fine learning are delayed, but not to the point that the fine information is not learned at all. We think this is a more practical setting and potentially a good starting point for exploring Theorem 3.

Reviewer_T7e7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why a model learns a coarse label early but uses a rare fine-grained cue much later, even when the cue is present in the input. Each sequence contains a signal token for the coarse class, and only a fraction p of the sequences also contains a nuance token for the fine label. For a one-layer softmax-attention model trained with gradient flow on the population loss, the authors reduce the training dynamics to four scalar ODEs. Theorem 2 shows that for small p there is a long plateau, in which the coarse label is learned but the probability of the correct fine label stays near chance inside the coarse class. The authors call this phase hallucination, and they explain its extra length by the attention, which focuses on the signal token. Theorem 3 shows that intermediate noise can prevent the learning of the fine label. Experiments on the simplified model, a full transformer, MLPs and LSTMs, and a new text dataset (BirdText) with BERT and GPT-2 show a similar coarse-to-fine behavior.

Critical strengths
  1. The studied question is interesting and relevant. A model learns the coarse answer early, but it uses a rare fine-grained cue much later, even when the cue is in the input. The paper asks why this happens.
  2. The reduction of the training dynamics to four scalar ODEs is clean, and it can be useful also for other works on the training of attention.
  3. The theory gives explicit time scales for the plateau and for the transition. At least in theory, it also separates the delay caused by rarity from the additional delay caused by attention. The closest previous work (Nichani et al., 2025) only shows that such a phase exists. Theorem 3 gives a complementary result, where intermediate noise prevents the learning of the fine label.
  4. In the simplified model, the experiments agree qualitatively with the theory. The coarse label is learned early, independently of p, and the plateau becomes longer when p decreases (Fig. 2).
  5. The experiments are reported carefully, with several seeds and well-designed controls in the cue-deletion and cue-swapping experiments. In GPT-2, the level of the plateau is close to the best accuracy that is possible from the coarse attributes only, which is a good sanity check. The limitations are also discussed in an honest way.
Critical weaknesses
  1. In the LLM literature, hallucination usually means that a trained model gives a wrong answer, often with high confidence. In Theorem 2, the model is only uncertain inside the correct coarse class, and part (ii) shows that this phase ends with longer training. The abstract calls it "one particular form of hallucination", and the authors themselves describe it as "an intermediate phase of coarse-to-fine learning" (Fig. 1), in which attention "delays fine-grained learning". In my opinion, this description fits the result better, and I suggest to use the word hallucination more carefully, especially in the title.
  2. Remark 1 says that the probability-based definition may not agree with argmax, but I think that the gap is bigger. As far as I understand the theory, in the noiseless setting of Theorem 2 the correct fine label always keeps a small positive margin over the other fine labels in its class. This means that the argmax prediction is correct during the whole plateau. Therefore, the claim that "hallucination persists" under argmax comes only from the experiments, and it is not explained by the theory. I do not expect a new theorem, but it would be good if the authors explain which differences from the theory (e.g., SGD, finite data, noise) produce the argmax plateau in the experiments.
  3. The support for the attention mechanism outside the simplified model is limited. For the full transformer, the main text says that it "also exhibits the same hallucination phase" and that the attention is more concentrated on the signal tokens in early training. In BirdCLS, the authors also show that the attention prefers the signal tokens early in training. In both cases, this is a correlation, and it does not show that attention causes the delay. The authors themselves note that the cue interventions "do not establish mediation by attention". However, the introduction says that the results "identify a mechanism by which data frequency and learned attention jointly lead to hallucination". I think that this claim should be limited to the simplified model, or it should be supported by an intervention on the attention. Also, in BirdText the plateau is clearly visible only for p = 0.1, so the evidence from natural language is limited.
  4. The predicted (log 1/p)^gamma / p scaling of the plateau length is not tested. Fig. 2 uses only four values of p (0.1 to 0.7), and it is not clear if these values are in the small-p regime of the theorem. Also, the plateau length itself (e.g., the time when the fine-label probability passes a fixed threshold) is not reported with uncertainty across seeds. Since the reduced system has only four variables, it can be solved numerically without noise for much smaller p, and the cost of this is small. This would directly test the main theoretical prediction.
  5. There is no experiment with frozen attention, which would separate the effect of attention from the effect of rarity. The authors write that "MLPs, and LSTMs also exhibit coarse-to-fine learning", and they use this to show that the behavior is "not unique to the analyzed transformer architecture". However, a similar delay appears also without attention, so rarity alone may explain part of the delay, and the paper does not measure how much additional delay comes from the learned attention. A run of the simplified model with the query fixed at zero, which gives uniform attention, would help here.
  6. The larger models are trained with AdamW, and the GPT-2 run uses also warmup and cosine decay. This is quite different from gradient flow. The authors mention the discrete optimizer as a limitation. All values of p use the same training setup, so the optimizer alone does not explain the dependence on p, but it can still change the length of the plateau. One run of the full transformer with plain SGD would help to separate the effect of the data from the effect of the optimizer.
Questions for authors
  1. How is the prolonged intermediate phase related to grokking (e.g., Murty et al., 2023, "Grokking of Hierarchical Structure in Vanilla Transformers", ACL)? In grokking, the training data is already fitted, but here the loss on the specified sequences stays high during the plateau.
  2. Theorem 2(ii) says that once the fine label is learned, it stays learned with longer training. Is it possible to detect the end of the plateau in practice, e.g., from the loss on the specified sequences? And how is this related to Gekhman et al. (2024), where fitting new facts during fine-tuning increases hallucinations?
  3. The theory describes the weights with only four numbers. In Fig. 2, these four numbers are computed by projecting the trained weights, but a projection always gives some value, even if a large part of the weights is not of this form. Could the authors report how much of the trained weights is explained by the four-scalar form?
3 comments

We thank the reviewer for the review. Please see below for our responses to your concerns:

The word hallucination can be a broad term: We agree that hallucination refers to a broader phenomenon in the LLM literature, compared to what we consider hallucination in our manuscript. We believe that we have made that distinction clear. In the introduction section, the first two paragraphs provide a broader introduction to hallucination and quickly narrow down to factual hallucinations in LLMs. Then we immediately introduce the hierarchical classification afterwards, motivated by the empirical findings, and from this point, the word "hallucination" refers to the prediction failure for this hierarchical classification problem, and we have a precise mathematical definition of this hallucination. We have discussed some potential gaps between the hallucination studied in this paper and practical hallucinations in several places, for example, when we compare with Nichani et al. 2025 and when we discuss the limitations of our work. Therefore, we believe our use of the word hallucination is accurate, even in the title, because we have made it clear that this is for the hierarchical classification problem. Similarly, our claim "identify a mechanism by which...jointly lead to hallucination" is also for the hierarchical classification. Nonetheless, we appreciate the reviewer's suggestion of using the terminology hallucination more carefully; if the reviewer feels the distinction should be made clearer in certain places, please let us know.

Argmax v.s. probability mass First of all, we would like to point out that the only case argmax does not plateau is the noiseless, population ODE flow, precisely because the correct class gets a tiny positive margin quickly. However, as soon as noise or finite-sample (full GD or mini-batch SGD) is introduced, the argmax accuracy starts to plateau as well. In the noiseless case, the finite-sample effect makes each weight's update deviate from its ideal one-dimensional subspace, removing the tiny margin on the correct class; Then with noise, the positive margin gets dominated by the noise.

Test the plateau length scaling: We have conducted additional experiments to test the plateau length for much smaller pp, and it is verified that the theoretical prediction is asymptotically accurate as pp decreases. Please see the new paragraph "Population gradient flow (B.1.3)" in Section B.1 in our revised manuscript.

Separate the effect of attention We thank you for your suggestion. In the newly added Section B.3 in our revised manuscript, we conduct attention intervention experiments following a very similar idea to the one in your review. If the attention concentration towards coarse tokens prolongs the plateau, then amplifying (attenuating) the attention should extend (reduce) the plateau length, which can be controlled by introducing an inverse temperature to the softmax attention in the model. Indeed, we observed in the new experiments that perturbing the inverse attention β\beta (default β=1\beta=1) has the same effect on the plateau length as we conjectured, for a simplified model and a one-head, one-layer full transformer. However, such an effect almost diminishes for a multi-layer transformer; we conjecture that this is because the network depth also has an effect on this coarse-to-fine learning, and the attention intervention would not be effective unless the depth effect is addressed as well, which could be investigated in future work.

Larger model with SGD We add full transformer SGD runs in experiments B.2.2 in the revised manuscript. The same hallucination plateau happens.

How much does the four-scalar form explain the trained weights On average, 98% of the weights lie within their respective ideal one-dimensional subspace introduced in Section 3.1, for all our experiments on the simplified model in B.1.1.

Relation to Murty et al., 2023 and Gekhman et al. 2024 We thank the reviewer for the reference. We think the key difference is the problem setting: roughly speaking, grokking, e.g., Murty et al., 2023, concerns cases where different rules are available to predict based on the same cues in the context, and the model takes a long intermediate phase to learn those rules that generalize. We study the case of learning one rule, but some cues are not available for a fraction of the data, which also leads to some plateau. I think they describe different model behaviors, and we believe both can contribute to the intermediate learning phase in practical training. As for Gekhman et al. 2024, we think it presents an empirical observation that is very interesting but hard to explain with our current theoretical setting.

Thank you for the answer and the new experiments. The SGD runs and the check of the plateau length answer my questions on these two points. The explanation of argmax is also clear to me now (however, please consider adding it to Remark 1). I have one more observation regarding this: in the simplified model the argmax accuracy first rises to a high value and drops only when attention moves to the signal token. The noise is there from the start, so to me it looks like the attention shift removes a cue that the model already used. Maybe this is worth a comment in the paper.

I am less convinced by the attention experiment. In the one-layer transformer the curves for all temperatures overlap during the plateau and separate only after it, so the temperature does not change the plateau length there. In your attention figures for the full transformer, the attention on the nuance token also does not fall during the plateau. MLPs and LSTMs show a similar delay. So, as far as I can see, the attention mechanism of the theory is present only in the simplified model, and the claim in the introduction should be limited to this setting.

Thank you very much for the response and the observations. We agree with the points (The early high argmax of the simplified model, the weaker effect of temperature control in the full transformer) you made, and will revise the main paper to make the main claims clearer w.r.t. those observations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.