- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the underlying mechanisms of "emergent abilities" in LLMs from a dynamical systems and multivariate time-series perspective. The study characterizes the long-range temporal correlations within the network activations via the spectral exponent (). The primary empirical contributions are: Self-organization to natural language scaling and Dynamic diversification and heavy-tailed divergence.
Critical strengths
Anchoring the macroscopic progression of pre-training to cumulative compute (FLOPs, estimated via ) and tracking internal temporal dynamics offers a fresh, quantitative angle on the emergence phenomenon. This provides an intrinsic lens that does not rely exclusively on discrete downstream benchmark metrics.
The observed dynamical transition points correspond remarkably well with the phase shifts and performance plateaus documented in prior literature.
Critical weaknesses
The paper defines the boundaries between Phase A, B, and C using specific FLOPs cutoffs ( and FLOPs) calibrated against MMLU 5-shot accuracy. However, in practice, the emergence trajectory is profoundly modulated by external data variables, including task diversity, sample complexity/difficulty, token distribution, and vocabulary granularity. A model trained on syntactically repetitive or code-heavy corpora may cross the grammatical threshold (Phase A B) or reasoning threshold (Phase B C) at significantly different compute budgets. Because FLOPs solely accounts for non-embedding parameter count and total tokens, treating these numerical FLOPs boundaries as universal thresholds oversimplifies the role of data composition.
While the paper establishes a compelling correlation between the kurtosis spike (appearance of outlier units) and benchmark leaps, the mechanistic link remains largely observational.
Questions for authors
Recent studies have explored emergent capabilities by quantifying internal representational structures, information propagation, and circuit formation (for example [1]). How do the authors reconcile their macroscopic, compute-oriented spectral dynamics with these fine-grained internal mechanistic formulations of semantic emergence?
I strongly encourage the authors to expand their literature review to include this body of work on internal/semantic emergence and discuss how spectral dynamics can be integrated with these mechanistic approaches.
[1]. Quantifying Semantic Emergence in Language Models
Thank you for the thorough response. The DFA and extended PSD analyses with AR(1) and shuffled controls address my main methodological concern, the FLOPs labels are corrected, and the narrowed framing of the kurtosis and outlier claims (a necessary rather than sufficient marker, and a temporal perspective on known outlier dimensions) now matches the evidence. I am therefore raising my rating from 2 to 3, and I encourage the authors to bring the promised main-text updates, confidence intervals and outlier-literature discussion into the final version. No further response is needed.