- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies hallucination in Transformers from a theoretical perspective through a hierarchical classification task. The authors construct a data distribution containing frequent coarse-grained signal tokens and rare fine-grained nuance tokens. Hallucination is defined as a gap in the model's average predictive probabilities: the correct coarse class receives high probability, whereas the probability assigned to the correct full label remains near chance even when the fine-grained cue is present in the input. For a simplified single-head Transformer, the authors exploit symmetry to reduce its gradient flow exactly to a system of four scalar ODEs (Theorem 1). They prove that the model undergoes a prolonged hallucination plateau when fine-grained information is sufficiently rare and token noise is sufficiently small (Theorem 2). They further analyze the effect of noise on learning dynamics (Theorem 3) and provide experiments on synthetic classification tasks and BirdText, using Transformers, a classifier built on frozen BERT features, and GPT-2.
Critical strengths
S1. Technical depth of the theoretical analysis.
The authors exploit symmetries in the data distribution and model parameters to reduce the high-dimensional Transformer gradient flow exactly to a dynamical system with four scalar variables. They also explicitly characterize the time scales of different learning phases. In particular, Theorem 2 quantifies the relationship between the hallucination plateau and the frequency of fine-grained information, going beyond a qualitative description of the learning behavior. This gives the work technical value.
S2. The paper examines the interaction between attention and the data distribution to explain delayed fine-grained learning.
Beyond considering the difficulty of learning rare information, the paper analyzes how attention learned early in training concentrates on common signal tokens and subsequently suppresses learning from nuance tokens. This provides a concrete mechanism for understanding why a model can fail to use relevant information even when that information is available in its input.
Critical weaknesses
W1. Limited substantive novelty over prior work on stage-wise learning and hallucination mechanisms.
The tendency of neural networks to learn easier or more frequent features before more complex or rare ones is a familiar pattern in learning dynamics. In more directly related work, Nichani et al. (2025) have analyzed stage-wise learning and intermediate hallucination in Transformer factual recall; Gong et al. (2025) have provided a theoretical analysis of two-stage feature learning in Transformers; and Zucchet et al. (2025) have studied plateaus in factual learning, the effects of the data distribution, and attention-related mechanisms.
The main addition here is to introduce a probability governing the presence of fine-grained cues and characterize the time scale of the hallucination plateau under specific model and data assumptions. While this result has technical value, the paper does not make sufficiently clear what more fundamental mechanistic understanding this quantitative result adds beyond prior theory. In particular, it needs to distinguish more clearly between general stage-wise learning, delays caused by rare information, and the additional suppression attributed to attention. Thus, although the theoretical results apply to a new setting, I find the advance over prior work and its significance insufficiently compelling.
W2. The independent effect of learned attention is not sufficiently validated empirically.
The paper attributes the prolonged hallucination plateau to the joint effects of rare fine-grained information and learned attention. Theorem 2 establishes the corresponding time scales, and Figure 2 shows attention concentrating on signal tokens. However, Appendix B.2.2 also reports similar coarse-to-fine learning in MLPs and LSTMs on the same data, showing that this phenomenon itself is not unique to Transformers or attention.
The authors further conduct cue-deletion and cue-swapping experiments on BirdText. These experiments primarily test whether the model uses the relevant input features at different training stages; they do not directly isolate how attention affects learning speed during training.
The theoretical analysis therefore offers an explanation in which attention further suppresses the learning of rare features, but the experiments still lack direct controls that quantify the additional delay caused by this mechanism beyond data rarity alone. This limits the evidence for the paper's core mechanistic explanation in more general models.
Overall, the reduction to a four-dimensional dynamical system and the characterization of learning time scales have technical value. However, the substantive advance over prior work and the evidence for the independent role of learned attention are not sufficiently compelling. My current rating is 2. If the authors clarify the substantive advance over prior theory and provide more convincing evidence for the independent role of attention, I would consider raising my rating to 3.
Questions for authors
Q1. Distinction from prior theoretical work.
What is the most important new theoretical insight relative to existing analyses of stage-wise learning, such as Nichani et al. (2025) and Gong et al. (2025)? In particular, beyond quantifying an established stage-wise learning phenomenon, does the dependence of the hallucination plateau on the fine-grained cue probability in Theorem 2 reveal a fundamental mechanism that previous theories cannot explain? Please clarify the differences from the closest prior work in both theoretical conclusions and research significance.
Q2. Independent contribution of attention.
Could the authors use fixed or uniform attention, interventions on the evolution of attention weights, or other suitable theoretical or experimental controls to distinguish the delay caused by data rarity itself from the additional delay caused by learned attention? In particular, can the independent effect of attention on the duration of the hallucination plateau be quantified to provide more direct support for the proposed core mechanism?
Q3. Robustness to the theoretical assumptions.
The four-dimensional ODE reduction and the main results rely on strong assumptions, including a uniform label distribution, orthonormal token patterns, and zero initialization. Could the authors discuss or provide experiments showing whether the hallucination plateau and attention-induced suppression persist when these assumptions are moderately relaxed, for example with non-orthogonal features or a non-uniform label distribution? Which assumptions are essential to the core conclusions, and which mainly simplify the analysis?
3 comments
Thanks for the clarifications and additional experiments. My main concerns have been adequately addressed, so I will raise my rating to 3.
Thank you very much. We believe the additional experiments you suggested have made the empirical results in this manuscript more comprehensive, and we will incorporate the clarifications as we revise the main paper.
We thank the reviewer for the review. Please see below for our responses to your concerns:
Comparison with prior works: We respectfully disagree with the reviewer's view that our manuscript has limited novelty compared to prior work: Gong et al. 2025 studied stage-wise learning, but the problem is for in-context learning, and their settings do not have a properly defined hallucination phase; Zucchet et al. 2025 focus mostly on empirical studies; The closest to ours is Nichani et al. 2025, and we provided a detailed comparison with Nichani et al. 2025 in the original submission. The first key difference is the cause of hallucination: We study the case when the nuance information might not be available in a new type of general sequences, whereas Nichani et al. 2025 concerns the case when the number of fine-grained classes is much greater than the coarse ones, i.e., L ≫ K (stated in our terminology); therefore, we study two different factors that may cause the model to hallucinate. The second key difference is that we have shown an explicit dependence of the length of the hallucination plateau on the nuance probability p, while Nichani et al. 2025 have only shown the existence of a time epoch when hallucination occurs and do not provide bounds for the plateau length. In our view, all of them are new understandings of the fundamental mechanism behind this particular hallucination behavior. Moreover, based on our new theoretical understandings, we have designed natural language domain tasks (albeit synthetic), and have probed this mechanism in more practical settings, showcasing the practical relevance of our results.
Independent contribution of the attention We thank the reviewer for the suggestion. In the newly added Section B.3 in our revised manuscript, we conduct attention intervention experiments following a very similar idea to the one in your review. If the attention concentration towards coarse tokens prolongs the plateau, then amplifying (attenuating) the attention should extend (reduce) the plateau length, which can be controlled by introducing an inverse temperature to the softmax attention in the model. Indeed, we observed in the new experiments that perturbing the inverse attention β (default β=1) has the same effect on the plateau length as we conjectured, for a simplified model and a one-head, one-layer full transformer. However, such an effect almost diminishes for a multi-layer transformer; we conjecture that this is because the network depth also has an effect on this coarse-to-fine learning, and the attention intervention would not be effective unless the depth effect is addressed as well, which could be investigated in future work.
Robustness to the theoretical assumptions: We thank the reviewer for the suggestions. We believe the hallucination plateau we studied in this manuscript can be robustly observed even when our data assumptions are violated, most of which are imposed to make the analysis tractable. However, we also feel that testing such robustness requires careful designs and extensive experiments, which can not be done during the rebuttal phase. Nonetheless, we conducted some preliminary experiments along the lines of what the reviewer suggested, in the newly added sections B.4.1 and B.4.2. In particular, when we introduce token correlations (B.4.1), the correlation slightly delays both the coarse and fine learning, but a similar hallucination phase persists. When we consider a non-uniform prior (B.4.2), the rarity of the fine class now varies, and the rarer fine classes are learned later, which is somewhat expected but a very interesting observation to study in future work.