- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper provides an apparently novel way to incorporate curvature adaptivity inside the distance generating function of Bregman-divergence based methods, by encoding the kernel as a squared gauge of a sequence of sets. The paper provides convergence theory for gauges that are strongly convex, but propose a "practical" method for polytopal gauges that are not strongly convex, without accompanying theory.
Critical strengths
- The idea appears truly unusual. Something we don't see a lot in ML, where all papers are "LLM LLM LLM Transformer Transformer Attention", or "yet another gradient step-size rule".
- The rotated ellipsoid method also appears unusual and new. I am not familiar with something like this in the literatrure.
- Experiments provide some evidence that the idea may be useful in practice. They are parameter-tuned, in contrast to what we see a lot in the literature, so at least for this small dataset and model the empirical performance appears be sound.
Critical weaknesses
- The claim that the literature dominantly uses ellipsoids is not true in general. Even the classical mirror descent does not use ellipsoids - it uses entropy-based Bregman distance. In general, Bregmal proximal methods are not ellipsoid based. More generally, the parameter-free methods use the so-called linearithmic regularizer, which is also not an ellipsoid. See [3,4].
- The use of Minkowski-functionals is not new for distances. See, for example, [1]. And the more general idea of using non-negative positively-homogeneous functions is not new in optimization. The classical (deterministic) optimization community came up with such methods too, see [2] for example.
- Since there is no theory for the polytopal method, I would expect it to be tested more extensively experimentally. Alternatively, bring the rotated ellipsoid method to the front, for which you do have theory. In either case, the paper as written requires a major revision.
Overall, appears the paper needs serious work before it is ready, and the authors need to familiarize themselves with the existing literature in both stochastic, online, and deterministic optimization before revising.
[1]: Faster Rates for Convex-Concave Games. Jacob Abernethy, Kevin A. Lai, Kfir Y. Levy, Jun-Kun Wang. COLT 2018. [2]: Generalized Bundle Methods. Antonio Frangion. SIAM Journal on Optimization 2018. [3]: Parameter-free Stochastic Optimization of Variationally Coherent Functions. Francesco Orabona, David Pal. Arxiv (2018) [4]: Online Learning Without Prior Information. Ashok Cutkosky, Kwabena Boahen. COLT 2017.
Questions for authors
NA