Dynamical Anatomy of Emergent Abilities in Large Language Models

In large language models (LLMs), the phenomenon of emergent abilities, characterized by abrupt performance improvements at specific scales or training stages, has received considerable attention. Notably, the learning process is observed to transition through three distinct phases: an initial non-linguistic output phase, a formally and syntactically consistent but semantically hollow phase, and finally, a meaningfully valid output phase. However, the underlying properties of internal information processing within these three phases remain unknown. In particular, it remains to be clarified what qualitative shifts in the model's internal processes underpin the acquisition of linguistic proficiency and higher-order intelligence. In this study, we focus on the high-dimensional internal dynamics of the LLM and analyze its dynamical properties across the three identified phases. We employ power spectral density (PSD) to investigate the long-range temporal correlations within the model's internal dynamics during text processing. Our analysis reveals that the transition from the first to the second phase is characterized by the emergence of a power-law spectrum with an exponent of $\beta \approx 0.5$, which reflects the characteristic strength of long-range correlations typically observed in natural language corpora. In the third phase, measuring $\beta$ across individual hidden dimensions shows a heavy-tailed broadening of the exponent distribution while maintaining a constant layer-mean, marked by the emergence of distinct outlier units. Further inspection of these outlier units, specifically those exhibiting weakened temporal correlations, reveals that they selectively respond to functional vocabulary like punctuations. Together, these observations suggest that analyzing processing dynamics along the temporal axis offers a valuable new lens for understanding how language capabilities emerge in LLMs.