acceptodds
Withdrawn from ICLR 2027

Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unbiased stochastic gradients can match the full gradient in expectation while changing finite-step stability. We study this effect in one-layer linear self-attention for in-context linear regression, where shared-mode mini-batch training reduces exactly to a random two-factor map. The full-batch map preserves an elliptic region, whereas sampled curvature and target correlation create batch-dependent stability boundaries. Their combined fluctuation is centered at the parameter-update level but induces a strictly outward drift of the full-batch boundary coordinate. We derive an exact one-step crossing criterion, adaptive concentration bounds, and conditional drift identities, and prove almost-sure escape in a balanced switching model. For independent long prompts, mode leakage decays at the inverse square root of prompt length. For finite-token softmax attention, we derive an explicit boundary-crossing law governed by an effective context length that shrinks as attention scores concentrate. Experiments in ambient attention objectives verify the reduced predictions, and four trained causal softmax Transformers show larger mean curvature shifts for smaller first-step batches, consistent with the predicted nonlinear outward bias.

Primary area:
optimization
Keywords:
Mini-batches, Large step-size, stability, linear self-attention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_b2KzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that unbiased mini-batch gradients can still change finite-step training stability. In one-layer linear self-attention for in-context linear regression, mini-batch training reduces exactly to a random two-factor map, where batch variation in curvature and target correlation creates moving stability boundaries. As a result, the full-batch boundary coordinate drifts strictly outward even though parameter updates are unbiased. The paper derives an exact one-step crossing criterion, exponential batch-size crossing bounds, and a finite-token softmax extension, with qualitative support from four small trained Transformers.

Critical strengths
  • Rigorously shows that parameter-level unbiasedness does not preserve stability, by compressing all batch variation into a single centered scalar ρ\rho whose variance drives outward boundary drift.
  • Covers several time scales within one coherent model, from exact one-step crossing intervals to finite-horizon bounds, almost-sure escape, and a finite-token softmax extension.
  • Matches experiments quantitatively, with reduced and full models agreeing to about 10−1510^{-15} and crossing probabilities matching predictions with no fitted parameters.
Critical weaknesses
  • The real-Transformer evidence is limited. The experiments use one small architecture (1.4M parameters, WikiText-2, plain SGD) and do not test the quantitative predictions, so it remains unclear whether the mechanism applies to practical Transformer training.
  • The curvature result may not isolate the proposed mechanism. The larger curvature increase for smaller batches may simply reflect larger displacement from the reference point. A control with norm-matched random perturbations would help determine whether the effect is specific to mini-batch noise.
  • Positioning relative to prior work is brief. The paper cites work on batch-dependent sharpness and the stochastic edge of stability, but does not explain how the present results differ from or sharpen those findings.
Questions for authors
  • The populations in Figures 1–2 appear chosen to make crossings visible, with high-type curvature 2.5×2.5\times the mean. How large are the fluctuations of (αB,νB)(\alpha_B, \nu_B) when prompts are drawn naturally rather than designed as two extreme types?

  • All comparisons fix η\eta while varying mm. How do the crossing bounds and the outward drift behave under the common practice of scaling η\eta with mm?

  • In Figure 3, the normal propagator remains mildly amplifying (median maximum around 1.11.1–1.51.5) over a 25-step horizon. Does the shared-mode approximation remain accurate over longer horizons, or does off-plane leakage eventually accumulate and invalidate the reduced dynamics?

Reviewer_oRMAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes finite-step mini-batch training in a one-layer linear self-attention model for in-context linear regression. Under shared signed spectral modes, training reduces exactly to a random two-factor map. Relative to the known full-batch invariant ellipse, the authors derive a state-dependent crossing criterion, concentration bounds, and a conditional outward drift identity.

Critical strengths
  1. An exact nonlinear characterization of one-step crossings. The results combine batch curvature and target variation into one state-dependent scalar and give a necessary-and-sufficient safe interval. This provides a concrete account of crossing a full-batch invariant boundary away from a local linearization around a stationary point.

  2. Explicit softmax calculations with targeted validation. The trained-Transformer experiments use useful controls, including matched future data and a disjoint Hessian batch, and the authors appropriately describe these experiments as a qualitative bridge.

Critical weaknesses
  1. The significance of the increment over existing stochastic-stability theory is not sufficiently established. Instability under unbiased mini-batch updates, and its dependence on learning rate, batch size, and sample heterogeneity, are already studied by Wu et al. (2018), Lee and Jang (2023), and Agarwala and Pennington (2024). The new exact crossing geometry is a useful refinement, but several highlighted consequences have limited additional conceptual content once that geometry is available: outward drift follows from applying centered noise to the convex quadratic boundary coordinate, and the exponential batch-size bound follows from standard concentration. I would also like the paper to identify a consequential prediction or insight enabled by its exact analysis that existing batch-aware stability criteria do not provide.

  2. the evidence that the exact characterization informs more general attention training remains limited. Exact sharing requires aligned query directions and a common spectral plane, and the long-prompt approximation retains the common query direction. The softmax calculation uses a Gaussian rank-one sector with fixed score and dimension. The real-model study in Appendix A.5 measures a fixed-batch Hessian proxy and short matched-continuation loss, which do not directly establish crossing of a stability boundary or identify the proposed mechanism. These limitations matter jointly: the applicability of the exact theory is narrow, and the empirical bridge does not yet show that its added geometric precision gives useful predictions beyond general gradient-noise sensitivity. A theory paper need not propose an improved optimizer, but the value of this particular refinement needs stronger support.

Questions for authors
  1. Which result in the submission provides the clearest prediction that existing batch-aware local stability analyses cannot capture? Could you identify a concrete regime in the submitted theory or experiments where the exact boundary criterion changes the prediction, and explain why this difference is consequential?

  2. Is Corollary 24 intended only for projected dynamics? For repeated full ambient updates, is the uniform map-discrepancy assumption verified after the trajectory leaves the sector? It would be great to provide the missing estimate or state precisely which ambient-transfer claim is supported.

  3. Appendix A.5 mentions preregistered secondary threshold events and individual proxy-outcome correlations. Could you report the four-checkpoint results or point to their location in the supplement? In particular, does the curvature proxy predict matched-continuation outcomes beyond gradient-deviation magnitude, and is the result robust to the finite reference gradient and Hessian batch? Existing results or a small check within the current setup would help assess the empirical bridge.

Reviewer_HbuoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is studying GD/mini-batch GD dynamics of a simplified linear attention model for icl of linear regression, where under certain assumptions on data and initialization they reduce the dynamics to that of two scalar parameters. For these reduced dynamics, they seem to identify regions where the parameters remain under full-batch GD, and when mini batch can cause the iterates to leave these regions.

Critical strengths

Studying the effect of batch size under tractable models is an important and relevant direction.

Critical weaknesses

The paper is very hard to read and follow. I don’t think the paper does a good enough job of conveying their contributions in the introduction, and introduce a lot of jargon without giving a broader idea first of their results.

Writing is a problem throughout the paper. For instance, Prop 1, doesn’t introduce what Φ\Phi is, what are an and b? Going to Prop 2, there’s some quantities c+,c- which are not introduced, and reading at the end you finally realize they are saying that the loss is a function of only a and b, which I’m assuming means the original quadratic form reduces to this, and looking at this PhiPhi in Prop 1 was probably update equations. This should be clearly written, instead of the reader joining bits and pieces together.

Prop. 3 is again the same, reads very dense, and there’s so many terms. I would suggest (and this is a suggestion throughout the paper) to write in informal form, with only important terms. There’s also terminology like G introduced without saying what they mean.

Additionally, and this is a problem throughout, please write in words what your propositions/theorems imply, what is the reader supposed to get out of it. For example, here I think what you are saying is if you relax the assumption of Prop. 2 (shared eigen vecs of G), you can bound gradients that lie outside this space assuming collinear queries, and this decays with number of examples N. I have several questions here: this setting seems very restrictive, you’re initializing within a subspace of these two EVs of H, why is this good/relevant? There is no discussion/intuition here. Even this relaxation of shared EVs to collienar query vectors seems rather strong, why is this better? Further, even if one gradient contribution outside the subspace is small, how does that affect the trajectory, does is accumulate?

Going to section 2.2: Again reads very dense (some many quantities to track of without enough text to guide you through), you derive some invariant ellipse for full batch GD, where does this come from? What does it tell us? Does it mean that if you initialize a,b on/in the ellipse they don’t leave it? How does it connect to the original model you started with? Similarly, you derive for mini-batch GD, I think there needs to be some discussion after you present a result. It is incredibly hard to read like this

Even with numerical results, figure 1: there isn’t enough discussion of what they demonstrate, what the reader needs to take from them.

At this point, I stopped reading because the paper was simply too difficult to go through and make sense of. I think it needs a major rewrite, with much more intuition, and discussion of what the setting/results actually mean.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.