- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether monitoring changes in the geometry of internal representations during training (the Gram matrix and its support subspace) can detect key events in model learning in advance, or locate them automatically.
Critical strengths
The paper provides a clear theoretical explanation for why internal learning can precede behavioral expression. I find this distinction particularly interesting and useful for understanding apparently sudden learning transitions. (1) The problem is meaningful. A geometric method for monitoring training dynamics that does not rely on labels for the target capability is valuable for understanding emergent abilities and for the interpretability of the training process. (2) The experiments build on each other step by step, moving from "known learning target" to "known capability, unknown mechanism," and then to "both capability and mechanism unknown." The logic is clear.
Critical weaknesses
Points that may need further clarification: (1) What is the detector's false positive rate? For example, how many events does it trigger when the checkpoint order is shuffled, or when random or unstructured probe data is used? (2) On Pythia, do the geometric events reliably come before the corresponding changes in behavior or metrics? (3) How is the detection threshold chosen? I suggest adding a sensitivity analysis of the detection threshold to show that the conclusions do not depend on one particular set of settings.
Questions for authors
Show the four motion components separately for the different observation views and layers. Testing alternative probe distributions would further clarify which findings reflect a general learning progression.
Response to Reviewer iWr2
Thank you for the careful review. We address your points below.
False positives, timing, and thresholds
False positives. We use random-token prefixes for induction detection and find no alarms in the 12 control runs, while detecting all 28 measured transitions in advance. The controls include training without repetition and slower-learning runs. We use these controls because shuffling checkpoints would itself introduce abrupt changes into the trajectory.
Timing relative to behavior. Some Pythia alarms precede substantial changes in the measured behavior, including the late 1B event shown in the paper. Other events overlap the measured changes or have no identified behavioral counterpart. We therefore find advance warnings in specific cases and can locate learning changes more generally. We will distinguish these outcomes more clearly in the text.
Threshold choice. We first compare each geometric departure with the size of ordinary fluctuations in its recent history. We then require the resulting score to exceed a fixed cutoff at two consecutive observations. We chose the cutoffs empirically during development: 4.5 for induction and 3 for Pythia. We kept the Pythia cutoff when moving from the smaller model to larger models. Raising the induction cutoff from 4.5 to 6 preserves all 28 pre-transition detections and zero alarms in the 12 controls. The Pythia 1B catalogue is stable over a narrower local range. We will clarify the selection procedure and report these sensitivity checks in the text.
Motion by layer and observation view
In response Figure 2, we show the four motion components across all stored layers and observation views in Pythia 1B. We find that internal rotation has the largest median share of later motion throughout the transformer layers, while growth remains dominant at the embeddings. The zero-transport panels reflect full resolved support in the hidden-state views, not missing data or an absence of learning; attention also shows transport across the spectral cutoff. We also checked different subsets of the existing probes, but not a new probe distribution.
The two response figures and their captions are in the anonymous figure supplement.