acceptodds
Under review as a conference paper at ICLR 2027

Gaussian Limits for SGD Without Stationary Moments

Abstract

Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein–Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.

Primary area:
learning theory
Keywords:
stochastic gradient descent, dependent data, stationary distributions, Gaussian approximation, total variation, heavy tails, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5nxBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies stationary last-iterate distributions of constant-step least-squares SGD under temporally dependent observations. Its central contribution is an explicit separation between distributional approximation and moment stability: a persistent-refresh design with standard Gaussian marginals produces a stationary error with every positive moment infinite, although the normalized stationary error converges to a Gaussian as the stepsize decreases.

Critical strengths

1.The one-coefficient anti-concentration argument in Appendix G convincingly explains how sufficiently long runs of an extreme covariate overwhelm their probability of occurrence. The coexistence with uniform geometric ergodicity makes the example particularly useful. 2. Lemma A.2 handles noncommuting positive-semidefinite matrices, and the block argument produces a common bound over sufficiently small stepsizes without assuming integrability of the exponential prefactor. 3. Weak convergence, conditional TV convergence, unconditional first-order approximation, and feasible coverage have separate assumptions. The authors correctly emphasize that the limiting Gaussian covariance need not be the limit of stationary covariance matrices.

Critical weaknesses
  1. The constants and admissible stepsize threshold depend on the design law, while the corrected approximation has an unspecified o(a)o(a) remainder. The schedule in Corollary 5.6 is asymptotically sufficient for each fixed admissible law, but does not provide a usable finite-stepsize guarantee. This limitation is visible in Table 7: for the log-Weibull model with c=1c=1 and a=0.000625a=0.000625, baseline coverage is 94.9772%94.9772\%, whereas corrected coverage is 95.0545%95.0545\%. The correction remains less accurate despite the small stepsize.

  2. Corollary 5.4 uses population σ2\sigma^2 and JJ. The feasible variance-estimation argument establishes asymptotic coverage, but consistency alone does not preserve an o(a)o(a) coverage expansion. The manuscript does not provide an implementable estimation scheme with conditions ensuring that calibration error is negligible at the required scale.

  3. The general projective-score theory is broad, but the sharp TV expansion relies on an exact Gaussian mixture representation. For non-Gaussian errors, the established result is a weak limit.

  4. Minor issues:

  • Theorem 4.1 should explicitly reference the standing causal Bernoulli-shift framework. A finite design second moment and H≻0H\succ0 alone would not imply the result for an arbitrary stationary design; the block-ergodicity argument is essential.
  • A compact assumptions table would help readers distinguish pathwise existence, the projective-score CLT, the quenched CLT, conditional TV convergence, and unconditional first-order TV approximation.
  • The distinction between a correction to the approximating Gaussian law and an expansion of stationary covariance is crucial and should remain prominent wherever “covariance correction” is used.
  • Appendix N.5 should explicitly identify the design used for the non-Gaussian-error and two-time OU experiments. Specifying ρ=.7\rho=.7 does not uniquely distinguish the refresh and AR constructions.
  • The small legends and uncertainty bars in the main figures make some comparisons difficult to inspect. Enlarged panels would improve readability.
Questions for authors
  1. Can you provide an explicit finite-stepsize remainder bound or a practical diagnostic for when the correction improves approximation, at least for the Gaussian-refresh family? How does the onset depend on persistence ρ\rho?

  2. What conditions on estimators of σ2\sigma^2 and JJ, and on their calibration sample sizes, preserve the o(a)o(a) coverage improvement? Can this be established when calibration observations overlap the SGD history?

  3. Under independent Gaussian errors, when should a practitioner use the deterministic first-order correction instead of the exact conditional procedure in Appendix K? Please clarify the computational and precision tradeoffs.

  4. Can you add an asymmetric non-Gaussian error distribution and a multidimensional independent-Gaussian-error experiment that directly tests the matrix correction in Theorem 5.2?

Reviewer_4JKuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies constant-stepsize SGD for linear least squares with temporally dependent observations. It constructs designs, including a geometrically ergodic process with standard Gaussian marginals, for which every positive stationary error moment is infinite at each sufficiently small fixed stepsize, while the scaled stationary law still converges to a Gaussian as the stepsize vanishes.

The theoretical contributions include pathwise contraction and stationary Gaussian and Ornstein--Uhlenbeck limits without requiring stationary iterate moments. For independent Gaussian regression errors, an exact conditional Gaussian representation yields total-variation convergence. Under stronger design moment and dependence conditions, the paper establishes a sharp O(a)O(a) baseline approximation error and a deterministic Gaussian correction with o(a)o(a) error. It also derives coverage expansions, finite-history transfer bounds, and inference procedures.

Synthetic scalar and vector experiments examine distributional error, coverage, initialization, and dependent-score calibration. They support the asymptotic predictions while documenting settings where the first-order correction worsens coverage at the tested stepsizes.

Critical strengths

A concrete distinction between stationary moments and Gaussian approximation. The Gaussian-refresh example holds the feature marginal fixed while showing that temporal persistence can destroy every positive stationary error moment without preventing a Gaussian small-stepsize limit. This identifies a meaningful limitation of reasoning from marginal regularity or geometric mixing alone and clarifies what a stationary Gaussian approximation can describe.

A quantitative approximation beyond stationary moment control. Under independent Gaussian errors and stronger design conditions, the paper establishes a sharp first-order total-variation expansion and an o(a)o(a) corrected approximation. The localization argument justifies the unconditional result even when stationary covariance does not exist. This gives the contribution substance beyond the counterexamples by quantifying central probability and coverage errors in that regime.

Critical weaknesses

Limited finite-stepsize guidance for the coverage correction. The first-order expansion is pointwise for each fixed data-generating law and does not characterize when correction improves coverage. In the log-Weibull stress test at a=0.000625a=0.000625, the corrected coverage error is approximately 0.05450.0545 percentage points, compared with 0.02280.0228 for the baseline. The paper acknowledges this behavior, but provides no quantitative remainder bound or justified criterion for deciding when to apply the correction. This limits the practical usefulness of the coverage refinement, although it does not undermine the stated asymptotic results.

Questions for authors

Clarification on the scope of the qualitative limit theorem.

Could the authors provide an assumption-level comparison between Theorem 4.2 and classical constant-gain Gaussian/OU limit results, particularly Kushner--Huang (1981)? Which specific hypotheses of the closest classical result fail for the Gaussian-refresh example, and how do the present pathwise contraction and score-projective conditions replace them? This would help assess the additional generality of the qualitative convergence theorem, separately from the all-moment-failure construction and sharp TV expansion.

Reviewer_pbkMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors show that for constant step least-squares problem, temporal dependence can make all stationary moments infinite while the small-step Gaussian limit remains, so moment-based stationary analysis is not representative. Via pathwise analysis, they prove Gaussian limits. They also provide experiments on coverage and calibration.

Critical strengths

1)Technical analysis seems nontrivial and rewarding.

Critical weaknesses

1)The number of required steps is step size dependent (Corollary 5,6).

2)The text is struggling to follow.

Questions for authors

1)Can the dependence on the number of steps be eliminated? What does numerical experiments on toy examples tell us, if we stop earlier, than prescribed by theory?

2)What happens with changing step size? For example, linearly decaying, or cosine annealing?

3)Does your bound recover the standard results for the considered problem with ρ=0\rho = 0?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.