acceptodds
Under review as a conference paper at ICLR 2027

Initialization Shapes the Computational Regime of Adaptive Optimization

Abstract

Optimization is central to modern machine learning, and its efficiency directly shapes the cost of model training. However, initialization is usually viewed as affecting only the starting point and transient trajectory, rather than the asymptotic convergence rate. We show that this separation can fail for adaptive quasi-Newton optimization. Parameter initialization alone can select distinct asymptotic computational regimes with the learning objective, optimizer family, and line-search hyperparameters held fixed. On a multidimensional regularized pinball-loss family, we identify an infinite hierarchy of such regimes for the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method and its limited-memory variant (L-BFGS), and develop a trace-to-complexity analysis that links recurring line-search decisions to asymptotic progress, evaluation cost, and stability. The analysis reveals an exact tradeoff between contraction and sensitivity to initialization, and we further show that each exact regime persists over any fixed finite horizon within an open full-dimensional neighborhood. Extensive experiments, including multidimensional and real-world learning settings, show that the predicted structure is observable in finite precision and persists beyond the exact analytic construction.

Primary area:
optimization
Keywords:
Adaptive optimization, Initialization, Quasi-Newton methods, BFGS, line search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Hje8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how initialization affects BFGS and L-BFGS with a fixed objective and optimizer configuration. For a structured convex pinball-loss covex objective, It constructs infinitely many recurring line-search patterns with distinct convergence rates per trial, relates contraction to initialization sensitivity, and proves finite-horizon persistence under perturbations. The theory is interesting but effectively one-dimensional. Experiments investigate these patterns and initialization-dependent optimization costs in some toy examples.

Critical strengths

Theoretical contribution - explicit infinite hierarchy of convergence regimes. The paper strengthens the known initialization-dependent behavior by constructing infinitely many recurring line-search patterns with exact per-trial convergence rates and a contraction–sensitivity relation. The construction and phenomenon appear interesting, but its scope and experiments seem limited.

Critical weaknesses
  1. The scope of the contribution is narrower than the framing. “Initialization shapes the computational regime of adaptive optimization” suggests a broad mechanism. The theorem establishes specially chosen trajectories of one nonsmooth scalar model under a specified line-search rule. In addition, there is an limited connection between theory and experiments. The exact regimes are effectively one-dimensional, and robustness holds only over finite horizons. The real-data experiments show cost variation without demonstrating the recurring regimes or asymptotic rate differences established theoretically. This leaves the broader practical significance insufficiently supported.

  2. Unclear presentation and scope. I found the setup and presentation of the results difficult to follow. The main text should more clearly distinguish the scalar construction, multidimensional embedding, and finite-horizon guarantees, and summarize the key proof ideas.

Questions for authors
  1. Which theoretical conclusions extend beyond the one-dimensional invariant subspace to genuinely coupled high-dimensional objectives or other losses?

  2. Do the existing experiments demonstrate recurring regimes or persistent convergence-rate differences beyond the analytic construction, or only differences in finite-time optimization cost?

Reviewer_b6CwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how initialization affects the computational behavior of adaptive quasi-Newton methods. On a specifically constructed objectives, BFGS and L-BFGS are shown to exhibit asymptotic convergence rates that depend on the initializations. The paper also provides numerical experiments on synthetic and forecasting problems.

Critical strengths

A detailed characterization of the line-search dynamics on the constructed objective is provided.

Critical weaknesses

The theoretical analysis is essentially on a special low-dimensional toy model. On this objective, the considered dynamics reduce to (essentially) an one-dimensional dynamics and hence are amenable to theoretical analysis. The trajectories with different asymptotic rates also need not to be stable under small perturbations (Theorem 3.8 only shows stability under finite horizon). It is therefore unclear whether the phenomenon is a consequence of the carefully engineered objectives and initializations.

Overall, the broader message about initialization-dependent convergence is not particularly surprising, and the theory does not establish a sufficiently meaningful explanation (e.g., the mechanism appears on more general objectives) or takeaway (e.g., how to select initializations).

Questions for authors

See weakness.

Reviewer_q25POpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Summary

This paper investigates how initialization of an algorithm affects its convergence behavior. The authors use (L-)BFGS and a toy function example to demonstrate that the algorithm can converge at very different rates depending on the initialization.

Critical strengths

Strength

The problem of initialization is interesting.

Critical weaknesses

Weaknesses

I have several concerns.

  1. Presentation issues

    The paper is very hard to read. In particular, the proof keeps defining new quantities and uses weird terminologies such as "regime hierarchy", "protocol", "regime cell". I don't think the paper is very readable in its current form.

  2. Importance of the result

    I don't think the conclusion that "initialization can affect convergence rate" needs such a complicated example. In particular, I don't think analyzing a quasi-Newton type method on a nonsmooth objective will provide many reproducible insights.

    Actually I think the same conclusion can be drawn even with gradient descent on simple qudratics 12⟨x,Λx⟩\frac{1}{2} \langle x, \Lambda x\rangle, where Λ\Lambda is a diagonal quadratic function: suppose x0=0x^0 = 0 on some dimensions, then GD with line-search will converge with effective condition number determined by the subspace where x0x^0 is nonzero.

I don't think the paper is suitable for publication at ICLR in its current form.

Questions for authors

Questions

  1. Is there any motivation for analyzing BFGS on a nonsmooth objective?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.