- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper shows that unbiased mini-batch gradients can still change finite-step training stability. In one-layer linear self-attention for in-context linear regression, mini-batch training reduces exactly to a random two-factor map, where batch variation in curvature and target correlation creates moving stability boundaries. As a result, the full-batch boundary coordinate drifts strictly outward even though parameter updates are unbiased. The paper derives an exact one-step crossing criterion, exponential batch-size crossing bounds, and a finite-token softmax extension, with qualitative support from four small trained Transformers.
Critical strengths
- Rigorously shows that parameter-level unbiasedness does not preserve stability, by compressing all batch variation into a single centered scalar whose variance drives outward boundary drift.
- Covers several time scales within one coherent model, from exact one-step crossing intervals to finite-horizon bounds, almost-sure escape, and a finite-token softmax extension.
- Matches experiments quantitatively, with reduced and full models agreeing to about and crossing probabilities matching predictions with no fitted parameters.
Critical weaknesses
- The real-Transformer evidence is limited. The experiments use one small architecture (1.4M parameters, WikiText-2, plain SGD) and do not test the quantitative predictions, so it remains unclear whether the mechanism applies to practical Transformer training.
- The curvature result may not isolate the proposed mechanism. The larger curvature increase for smaller batches may simply reflect larger displacement from the reference point. A control with norm-matched random perturbations would help determine whether the effect is specific to mini-batch noise.
- Positioning relative to prior work is brief. The paper cites work on batch-dependent sharpness and the stochastic edge of stability, but does not explain how the present results differ from or sharpen those findings.
Questions for authors
-
The populations in Figures 1–2 appear chosen to make crossings visible, with high-type curvature the mean. How large are the fluctuations of when prompts are drawn naturally rather than designed as two extreme types?
-
All comparisons fix while varying . How do the crossing bounds and the outward drift behave under the common practice of scaling with ?
-
In Figure 3, the normal propagator remains mildly amplifying (median maximum around –) over a 25-step horizon. Does the shared-mode approximation remain accurate over longer horizons, or does off-plane leakage eventually accumulate and invalidate the reduced dynamics?