- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when mirror descent and gradient descent in transformed coordinates give the same dynamics. It considers continuous time, finite steps, and the relation between the corresponding coordinate maps. It also looks at changing the objective or regularizer. The experiments include continuous maps and first-order corrections in 2D. Direct learning of exact finite-step maps is tested in 1D.
Critical strengths
-
Proposition 1 states its assumptions precisely, includes the converse, and has a short proof that is easy to follow.
-
The local existence proof is detailed, with a power-series argument and explicit contraction constants. I also find the correction construction useful for understanding finite-step equivalence
Critical weaknesses
-
The scope of the finite-step results needs to be clearer. Proposition 1 fixes a positive step size but assumes equality for every sufficiently small gradient vector. Since the update depends on the product of the step size and gradient, the hypothesis concerns small increments. It is an exact result, but it does not study large-update behavior. I would replace "gradient directions" with "gradient vectors in a neighborhood of zero." Theorem 2 also needs an interior stationary point with specific curvature assumptions. The example in Figure 4 has no stationary point in its domain, so the theorem does not cover it. This is not a contradiction, but I am not sure what selects the reference map in that case.
-
I am not yet convinced by the case for learning these maps. The paper motivates learning through expensive repeated solves, but I could not find solver, inference, or training timings. Given the low-dimensional problems, this comparison seems necessary. Some data-generation details are also missing, and I could not find code in the supplied material. Section 4 promises a numerical study of the conditioning of the connecting map, but I could not find the conditioning results in Appendix C.3. There are some domain examples, but the promised analysis seems incomplete.
Presentation issues: Figure 1 takes more than half a page for just four boxes, the figures generally look unpolished, and the standalone ('floating') numerical values around lines 195 to 204 are surprising on a first read. Together with the unused space across the paper and appendix, these choices make the work feel unfinished. Polishing the presentation would greatly help this paper.
Overall there are useful ideas and interesting theory in this paper. However, the presentation and experimental support still leave too many questions open. I would recommend rejection for the current version. I would like to see a clearer statement of the new results, better justification of the numerical references, and the missing computational comparisons.
Questions for authors
-
Which results are new relative to the cited work? In particular, how should Proposition 1 be understood beyond the earlier observations about discretization? Section 5 should also discuss the related result with varying mirror potentials in Even et al. (2023).
-
What selects the scalar reference map when Theorem 2 does not apply? Which benchmark objectives satisfy its assumptions?
-
What are the solver, inference, and training costs, including reference-data generation? Is there a regime where learning saves computation?
-
Can the authors provide the conditioning results promised in Section 4, or clarify what was actually evaluated?
3 comments
Thank you for your response and for making significant revisions. These changes address many of my earlier questions.
After reviewing the new version, I see that the paper has two clear parts: a strong theoretical section and a learning section that still needs substantial work. I think the paper should be restructured around this difference, so I am keeping my score. Below, I explain my reasoning and suggest a possible way forward.
Main concerns
- The learning component, which is highlighted in the title and introduction, is not supported by the paper's own evidence. In two dimensions, the new comparisons show that classical solvers are at least as fast as the learned models and more accurate (Appendix C.5, Table 3). Only the classical correction improves the observed order of agreement with mirror descent in ; the learned correction does not (Section 3.4). In one dimension, a low-degree classical fit is more accurate than any learned model (Table 14).
- There is a gap between the theoretical and learning parts. Theorem 2 needs a minimizer, but neither the scalar learning family nor the two-dimensional problems have one in the domains studied (Section 3.1, Appendix B.1.3). The existence proof in Appendix B.1.2 only applies to the Figure 4 family, not the whole scalar learning family. So, the exact existence results do not cover the problems used for operator learning. On the other hand, the experiments that Theorem 2 does cover use maps fitted to a single problem, not operator learning (Appendix C.2.4).
- The introduction should clearly state what is new compared to Amid and Warmuth (2020), who posed the discrete case as an open problem, and Ghai et al. (2022), who answered it with an approximate equivalence, bounding the one-step gap between the two updates. Right now, Ghai et al. are only cited for their regret bounds (Section 3.1). The main new contribution seems to be the analysis of exact finite-step equivalence.
Possible directions
- One way forward is to organize the paper around the theoretical results, use classical solvers as the main numerical method, and show learning only briefly or leave it out.
- Alternatively, the paper could keep its main focus on learning, but it would need to show a situation where learning clearly helps. For example, this could mean using higher-dimensional settings or using learning as a warm start for classical solvers, with clear comparisons of accuracy and computational cost for different step sizes.
What works well
- Proposition 1, the local existence results, and the new existence proof together give a clear account of exact finite-step equivalence. The revision has made this part much stronger.
- The complete code package and the clearer scope of the theorems make the work much easier to review and build on.
- The detailed comparisons with classical solvers are valuable and should stay in any future version.
Minor presentation suggestions (these did not affect my score)
- Please number the key displayed equations, such as the finite-step equation in Section 3, so they can be referenced.
- Several figures have wide white margins or sparse panels. Using tighter layouts or combining related panels would help each figure show more information.
- Extra care with typesetting and figures would go a long way toward improving how the paper is received.
In its current form, I cannot recommend a higher score, as the case for operator learning, which is central to the paper, is not supported by the paper's own experiments. This assessment does not reflect on the theoretical contributions, which I find interesting and notably improved in the revision. I encourage the authors to pursue the suggested restructuring; the theoretical core merits a paper focused on it.
Finite-step contributions and construction costs
Thank you for the careful reassessment and constructive suggestions. We have revised the introduction and conclusion to emphasize finite-step coordinate relations and their numerical construction, with operator learning presented as an approach to approximating these constructions across problem families.
The introduction now distinguishes our results from continuous-time reparameterization (Amid and Warmuth, 2020) and bounds on the gap between discrete updates (Ghai et al., 2022). We study conditions and constructions for exact finite-step agreement.
The mathematical guarantees are stated separately. Theorem 2 gives local exact maps near a nondegenerate minimum under its stated assumptions, and Appendix B.1.2 treats the Figure 4 family. Theorem 3 gives a local analytic construction of the two-dimensional correction under the transversality condition. The broader scalar experiments use numerical reference maps assessed through equation residuals and refinement.
Table 18 adds construction times for the existing implementations. On 32 inputs, FNO takes 230 ms per input, compared with 450 ms for the fastest tested fixed-coordinate solver, on a grid. Their mean relative correction errors are 7.38% and %, respectively. These times include input preparation and coordinate values and derivatives; the earlier 21 ms covers only the correction solve after coefficient preparation. Appendix C.5 specifies the computations on each grid, including separate continuous-map construction on the classical solver's grid, and reports training costs separately.
The updated anonymous code package includes the scripts, inputs, saved results, and reproduction instructions for the construction-cost comparison in Table 18.
Thank you for helping us make the contribution and numerical evidence clearer.
Response to Reviewer CYSD
Thank you for the helpful questions about the contribution, reference constructions and implementation. We have clarified these points in the revised manuscript.
Contribution and prior work (Q1). We use operator learning to study coordinate relations between optimization algorithms across families of objectives and regularizers, with particular emphasis on discrete algorithms used in practice. Four closely-related questions lead from continuous equivalence to finite-step equivalence, then to connecting the coordinates and modifying the problem. Proposition 1 makes the requirement at a fixed positive step size precise: agreement for every gradient vector near zero at each point forces an affine map (p. 5; proof p. 15). Gunasekar et al. (2021) discuss the effect of nonlinear coordinates on discretization. Section 5 also compares our fixed modified regularizer with Even et al.'s iteration-dependent potential (p. 9).
Reference maps (Q2). For analytic data, Theorem 2 gives a local analytic map near a nondegenerate minimum with proportional Hessians and 0<ηc<1. The quadratic and Welsch tests illustrate this setting. Figure 4 uses a separate construction. The boundary limit xp′(x)→Cη selects its positive derivative, and p(0.3025)=1.1 fixes the additive constant. Appendix B.1.2 (pp. 25–27) proves existence and uniqueness and describes the independent numerical refinements.
Computational costs (Q3). A numerical solver constructs a coordinate correction separately for each objective–regularizer pair. The learned operator is trained across a family of these pairs and then predicts corrections for new inputs with the same parameters. Its training cost is shared across subsequent predictions. For the two-dimensional FNO, median training takes 336 s per run and each evaluation takes 33.9 ms, including decoding and error metrics. Appendix C.5 (p. 46) reports solver costs and independent reference construction separately.
Conditioning and implementation (Q4). We evaluate the reference map Φ=id+η(aR,f∘qR). Across 197 problems, the largest sampled Jacobian condition number is at most 1.0166 at η=0.1. This ratio compares stretching across directions; the factor η makes the map approach the identity as the step size decreases. The model predicts the correction aR,f before this scaling. The equation-trained FNO has mean relative L2/H1 errors of 2.77%/8.32% on the in-distribution group and 3.78%/10.20% on the geometry-scale group (Appendix C.2.2, p. 38).
The condition aR,f(0,x2)=0 makes DaR,f singular on that line, so its condition number for inversion is infinite. Invertibility concerns pR+ηaR,f. We also report the derivative before scaling, K=DaR,f(DpR)−1: its 197 per-problem maximum norms on the 65×65 grids have median 0.0650 and largest value 0.1554 (rounded). Appendix C.3.3 (pp. 44–45) gives these measurements and the scalar slope ranges.
Appendix C.6 (pp. 47–49) gives the objective formulas and input distributions. We have also reduced Figure 1.
We have uploaded an anonymized implementation package as supplementary material, including code for reference construction, model evaluation and figure reproduction.
Thank you again for your suggestions. We would be happy to discuss the construction and experiments further.