acceptodds
Under review as a conference paper at ICLR 2027

Operator Learning for Algorithmic Equivalence: From Mirror Flows to Finite Steps

Abstract

Mirror descent and Euclidean gradient methods use different geometries, but in continuous time they can sometimes describe the same dynamics after a nonlinear change of coordinates. We ask how far this equivalence extends when both methods take finite steps of the same size. For a mirror regularizer , we first construct the continuous coordinate map and study operator learning of the map across regularizer families. At finite step size, requiring one objective-independent nonlinear coordinate map to match the two updates for a neighborhood of gradient directions forces the map to be affine and the mirror geometry to be quadratic. We therefore allow the finite-step coordinate map to depend on the objective. For fixed , , and step size , we solve for ; across problem families, we study operator learning of . We then study how the continuous and finite-step coordinate maps are related as varies. Finally, we ask whether the objective or regularizer can be modified so that the continuous and finite-step formulations produce the same coordinate map. Our numerical experiments include one-dimensional examples with exact solutions and genuinely coupled two-dimensional function families.

Primary area:
optimization
Keywords:
Mirror descent, algorithmic equivalence, operator learning, neural operators, gradient methods, finite-step equivalence, optimization geometry.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CYSDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when mirror descent and gradient descent in transformed coordinates give the same dynamics. It considers continuous time, finite steps, and the relation between the corresponding coordinate maps. It also looks at changing the objective or regularizer. The experiments include continuous maps and first-order corrections in 2D. Direct learning of exact finite-step maps is tested in 1D.

Critical strengths
  • Proposition 1 states its assumptions precisely, includes the converse, and has a short proof that is easy to follow.

  • The local existence proof is detailed, with a power-series argument and explicit contraction constants. I also find the correction construction useful for understanding finite-step equivalence

Critical weaknesses
  1. The scope of the finite-step results needs to be clearer. Proposition 1 fixes a positive step size but assumes equality for every sufficiently small gradient vector. Since the update depends on the product of the step size and gradient, the hypothesis concerns small increments. It is an exact result, but it does not study large-update behavior. I would replace "gradient directions" with "gradient vectors in a neighborhood of zero." Theorem 2 also needs an interior stationary point with specific curvature assumptions. The example in Figure 4 has no stationary point in its domain, so the theorem does not cover it. This is not a contradiction, but I am not sure what selects the reference map in that case.

  2. I am not yet convinced by the case for learning these maps. The paper motivates learning through expensive repeated solves, but I could not find solver, inference, or training timings. Given the low-dimensional problems, this comparison seems necessary. Some data-generation details are also missing, and I could not find code in the supplied material. Section 4 promises a numerical study of the conditioning of the connecting map, but I could not find the conditioning results in Appendix C.3. There are some domain examples, but the promised analysis seems incomplete.

Presentation issues: Figure 1 takes more than half a page for just four boxes, the figures generally look unpolished, and the standalone ('floating') numerical values around lines 195 to 204 are surprising on a first read. Together with the unused space across the paper and appendix, these choices make the work feel unfinished. Polishing the presentation would greatly help this paper.

Overall there are useful ideas and interesting theory in this paper. However, the presentation and experimental support still leave too many questions open. I would recommend rejection for the current version. I would like to see a clearer statement of the new results, better justification of the numerical references, and the missing computational comparisons.

Questions for authors
  1. Which results are new relative to the cited work? In particular, how should Proposition 1 be understood beyond the earlier observations about discretization? Section 5 should also discuss the related result with varying mirror potentials in Even et al. (2023).

  2. What selects the scalar reference map when Theorem 2 does not apply? Which benchmark objectives satisfy its assumptions?

  3. What are the solver, inference, and training costs, including reference-data generation? Is there a regime where learning saves computation?

  4. Can the authors provide the conditioning results promised in Section 4, or clarify what was actually evaluated?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CYSD

Thank you for the helpful questions about the contribution, reference constructions and implementation. We have clarified these points in the revised manuscript.

Contribution and prior work (Q1). We use operator learning to study coordinate relations between optimization algorithms across families of objectives and regularizers, with particular emphasis on discrete algorithms used in practice. Four closely-related questions lead from continuous equivalence to finite-step equivalence, then to connecting the coordinates and modifying the problem. Proposition 1 makes the requirement at a fixed positive step size precise: agreement for every gradient vector near zero at each point forces an affine map (p. 5; proof p. 15). Gunasekar et al. (2021) discuss the effect of nonlinear coordinates on discretization. Section 5 also compares our fixed modified regularizer with Even et al.'s iteration-dependent potential (p. 9).

Reference maps (Q2). For analytic data, Theorem 2 gives a local analytic map near a nondegenerate minimum with proportional Hessians and 0<ηc<10<\eta c<1. The quadratic and Welsch tests illustrate this setting. Figure 4 uses a separate construction. The boundary limit xp′(x)→Cη\sqrt{x}p'(x)\to C_\eta selects its positive derivative, and p(0.3025)=1.1p(0.3025)=1.1 fixes the additive constant. Appendix B.1.2 (pp. 25–27) proves existence and uniqueness and describes the independent numerical refinements.

Computational costs (Q3). A numerical solver constructs a coordinate correction separately for each objective–regularizer pair. The learned operator is trained across a family of these pairs and then predicts corrections for new inputs with the same parameters. Its training cost is shared across subsequent predictions. For the two-dimensional FNO, median training takes 336 s per run and each evaluation takes 33.9 ms, including decoding and error metrics. Appendix C.5 (p. 46) reports solver costs and independent reference construction separately.

Conditioning and implementation (Q4). We evaluate the reference map Φ^=id+η(aR,f∘qR)\widehat\Phi=\mathrm{id}+\eta(a_{R,f}\circ q_R). Across 197 problems, the largest sampled Jacobian condition number is at most 1.0166 at η=0.1\eta=0.1. This ratio compares stretching across directions; the factor η\eta makes the map approach the identity as the step size decreases. The model predicts the correction aR,fa_{R,f} before this scaling. The equation-trained FNO has mean relative L2/H1L^2/H^1 errors of 2.77%/8.32% on the in-distribution group and 3.78%/10.20% on the geometry-scale group (Appendix C.2.2, p. 38).

The condition aR,f(0,x2)=0a_{R,f}(0,x_2)=0 makes DaR,fDa_{R,f} singular on that line, so its condition number for inversion is infinite. Invertibility concerns pR+ηaR,fp_R+\eta a_{R,f}. We also report the derivative before scaling, K=DaR,f(DpR)−1K=Da_{R,f}(Dp_R)^{-1}: its 197 per-problem maximum norms on the 65×6565\times65 grids have median 0.0650 and largest value 0.1554 (rounded). Appendix C.3.3 (pp. 44–45) gives these measurements and the scalar slope ranges.

Appendix C.6 (pp. 47–49) gives the objective formulas and input distributions. We have also reduced Figure 1.

We have uploaded an anonymized implementation package as supplementary material, including code for reference construction, model evaluation and figure reproduction.

Thank you again for your suggestions. We would be happy to discuss the construction and experiments further.

Thank you for your response and for making significant revisions. These changes address many of my earlier questions.

After reviewing the new version, I see that the paper has two clear parts: a strong theoretical section and a learning section that still needs substantial work. I think the paper should be restructured around this difference, so I am keeping my score. Below, I explain my reasoning and suggest a possible way forward.

Main concerns

  • The learning component, which is highlighted in the title and introduction, is not supported by the paper's own evidence. In two dimensions, the new comparisons show that classical solvers are at least as fast as the learned models and more accurate (Appendix C.5, Table 3). Only the classical correction improves the observed order of agreement with mirror descent in η\eta; the learned correction does not (Section 3.4). In one dimension, a low-degree classical fit is more accurate than any learned model (Table 14).
  • There is a gap between the theoretical and learning parts. Theorem 2 needs a minimizer, but neither the scalar learning family nor the two-dimensional problems have one in the domains studied (Section 3.1, Appendix B.1.3). The existence proof in Appendix B.1.2 only applies to the Figure 4 family, not the whole scalar learning family. So, the exact existence results do not cover the problems used for operator learning. On the other hand, the experiments that Theorem 2 does cover use maps fitted to a single problem, not operator learning (Appendix C.2.4).
  • The introduction should clearly state what is new compared to Amid and Warmuth (2020), who posed the discrete case as an open problem, and Ghai et al. (2022), who answered it with an approximate equivalence, bounding the one-step gap between the two updates. Right now, Ghai et al. are only cited for their regret bounds (Section 3.1). The main new contribution seems to be the analysis of exact finite-step equivalence.

Possible directions

  • One way forward is to organize the paper around the theoretical results, use classical solvers as the main numerical method, and show learning only briefly or leave it out.
  • Alternatively, the paper could keep its main focus on learning, but it would need to show a situation where learning clearly helps. For example, this could mean using higher-dimensional settings or using learning as a warm start for classical solvers, with clear comparisons of accuracy and computational cost for different step sizes.

What works well

  • Proposition 1, the local existence results, and the new existence proof together give a clear account of exact finite-step equivalence. The revision has made this part much stronger.
  • The complete code package and the clearer scope of the theorems make the work much easier to review and build on.
  • The detailed comparisons with classical solvers are valuable and should stay in any future version.

Minor presentation suggestions (these did not affect my score)

  • Please number the key displayed equations, such as the finite-step equation in Section 3, so they can be referenced.
  • Several figures have wide white margins or sparse panels. Using tighter layouts or combining related panels would help each figure show more information.
  • Extra care with typesetting and figures would go a long way toward improving how the paper is received.

In its current form, I cannot recommend a higher score, as the case for operator learning, which is central to the paper, is not supported by the paper's own experiments. This assessment does not reflect on the theoretical contributions, which I find interesting and notably improved in the revision. I encourage the authors to pursue the suggested restructuring; the theoretical core merits a paper focused on it.

AuthorsOpenReview ↗

Finite-step contributions and construction costs

Thank you for the careful reassessment and constructive suggestions. We have revised the introduction and conclusion to emphasize finite-step coordinate relations and their numerical construction, with operator learning presented as an approach to approximating these constructions across problem families.

The introduction now distinguishes our results from continuous-time reparameterization (Amid and Warmuth, 2020) and bounds on the gap between discrete updates (Ghai et al., 2022). We study conditions and constructions for exact finite-step agreement.

The mathematical guarantees are stated separately. Theorem 2 gives local exact maps near a nondegenerate minimum under its stated assumptions, and Appendix B.1.2 treats the Figure 4 family. Theorem 3 gives a local analytic construction of the two-dimensional correction aR,fa_{R,f} under the transversality condition. The broader scalar experiments use numerical reference maps assessed through equation residuals and refinement.

Table 18 adds construction times for the existing implementations. On 32 inputs, FNO takes 230 ms per input, compared with 450 ms for the fastest tested fixed-coordinate solver, on a 33×4933\times49 grid. Their mean relative H1H^1 correction errors are 7.38% and 7.89×10−67.89\times10^{-6}%, respectively. These times include input preparation and coordinate values and derivatives; the earlier 21 ms covers only the 65×8165\times81 correction solve after coefficient preparation. Appendix C.5 specifies the computations on each grid, including separate continuous-map construction on the classical solver's grid, and reports training costs separately.

The updated anonymous code package includes the scripts, inputs, saved results, and reproduction instructions for the construction-cost comparison in Table 18.

Thank you for helping us make the contribution and numerical evidence clearer.

Reviewer_xPduOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies coordinate transformations relating mirror descent to Euclidean gradient descent on a transformed objective. In continuous time, such a transformation flattens the prescribed mirror metric. At finite step size, the paper proves that an objective-independent transformation valid for all nearby gradient directions must be affine, with quadratic mirror geometry. Allowing objective dependence, it establishes local analytic existence under a proportional-Hessian assumption and constructs small-step corrections to arbitrary order. Neural models approximate these transformations across restricted one- and two-dimensional problem families. The results concern algorithmic equivalence; they do not establish improved conditioning or faster optimization.

Critical strengths

Learning the coordinate transformation for a family of problems between mirror descent and gradient descent is interesting and new.

Critical weaknesses
  1. The problem formulation and terminology are insufficiently precise. Several central concepts are undefined, including the numerical model, the deformation associated with mirror descent, and the decoder parameterization. The relationship between the MLP baseline and the operator-learning task is also unclear.

  2. The experiments are difficult to assess or reproduce. The paper does not clearly specify the learned maps or ground-truth generation procedure; even objective formulas sometimes appear only in figures. The one-dimensional quadratic example provides limited evidence for performance on more general, particularly nonconvex, problems.

  3. The optimization benefit is unclear. The method assumes a prescribed mirror geometry and learns a coordinate transformation reproducing its dynamics. From an optimization perspective, the more consequential problem is selecting or learning a geometry that improves convergence. The paper does not establish when recovering coordinates for a given geometry is preferable to applying mirror descent directly, or whether the resulting computational savings justify learning the transformation. Please clarify the intended benefit and distinguish it from learning an effective preconditioner or mirror map.

  4. The practical benefit remains unclear. The experiments are limited to low-dimensional examples, and the learned method underperforms the numerical baseline in Table 6. Related work by Tan et al. (2022) learns a convex mirror potential to improve optimization and includes high-dimensional experiments. Although the objectives differ, can the authors demonstrate when learning coordinates for a prescribed geometry is preferable to learning the geometry itself, particularly on a challenging high-dimensional problem?

Questions for authors
  1. What maps are learned in the one- and two-dimensional examples, and how are the ground-truth operators generated? For the MLP baseline, what are the inputs, outputs, and “free decoder coefficients”? What is the corresponding infinite-dimensional operator?

  2. Why is exact finite-step equivalence the appropriate target? For optimization, a transformation that improves conditioning and convergence may be sufficient, even if it does not reproduce mirror descent exactly. Please explain what exact matching provides beyond studying algorithmic equivalence, and whether this constraint helps or restricts the search for useful transformations.

  3. The overview diagram occupies disproportionate space relative to its information content. Please reduce its size.

  4. What do “ID,” “Higher-frequency,” and “Geometry-scale” denote on the horizontal axes? Why is the H1H^1 error appropriate, and what physical or optimization-relevant quantity does it measure?

  5. What is the “numerical model” in line 177, and which nonlinear deformation is associated with mirror descent?

  6. Please define “even-Gaussian components,” “density floor,” and “degree-four Chebyshev dependence” on η∈[0,1.4]\eta\in[0,1.4].

  7. Does the method improve convergence to a minimum or stationary point? Can the authors demonstrate this beyond the quadratic objective α2x2+βx\frac{\alpha}{2}x^2+\beta x, for example on a nonconvex problem such as Rosenbrock?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xPdu

Thank you for the detailed questions about the learned maps and experiments. We have clarified the constructions, input functions and numerical comparisons.

Purpose (Q2). We use operator learning to study how optimization algorithms are related by changes of coordinates across families of objectives and regularizers. The four questions progress from continuous-time equivalence to finite-step equivalence, then to connecting the coordinates and modifying the problem. Exact matching gives a concrete target: corresponding initial points produce the same physical iterates and objective values. Tan et al. (2023) learn a mirror potential to improve optimization; our models learn the coordinate relation for a given potential.

Learned maps and inputs (Q1, Q5). The operators are M↦pRM\mapsto p_R, (R′′,f′)↦pR,f(R'',f')\mapsto p_{R,f} and (M,∇f)↦aR,f(M,\nabla f)\mapsto a_{R,f}. Here p=q−1p=q^{-1} maps physical points to Euclidean coordinates, and aa is the first finite-step correction to pRp_R. The fixed-grid MLP, FNO and DeepONet read sampled input functions and predict coefficients defining the output map (Tables 4–5, p. 31). Appendix B.1 (pp. 25–28) gives the independent numerical reference constructions. For both coupled tasks, each trained FNO is evaluated on different input grids with its parameters fixed (Appendix C.1.2, pp. 36–37).

Error measures and test groups (Q4–Q5). We describe the coordinate deformation by the displacement pR(x)−xp_R(x)-x. The H1H^1 error measures map values and first derivatives because the Jacobian determines the metric. The in-distribution group uses new functions from the training distribution. The higher-frequency group increases their oscillation, and the geometry-scale group increases the geometry amplitude (pp. 3–4; Appendix C.6, pp. 47–49).

Scalar representation (Q6). The scalar model predicts a positive derivative, which gives an increasing coordinate map. Gaussian pairs at opposite centers define an even function of x\sqrt{x}. Its normalized factor equals one at zero and stays above 0.9. Degree-four Chebyshev expansions describe how the weights vary with step size; softplus makes them positive. Appendix C.6 (p. 49) gives the formula.

Nonconvex example (Q7). The Welsch experiment follows nine initial states, including three with negative curvature, for 400 iterations at six step sizes. At η=0.5\eta=0.5, the learned maps reduce peak disagreement with mirror descent by factors of 1.61–2.23 relative to continuous coordinates. The endpoints at all tested step sizes lie within 3.84×10−103.84\times10^{-10} of the common minimum (Appendix C.2.4, pp. 41–43).

Higher dimensions and presentation. Efficient higher-dimensional construction is future work. On a full grid, nn points per state coordinate give ndn^d locations per input function, increasing data storage, reference construction and training costs (Section 6, p. 9). Table 12 (p. 46) and Appendix C.5 (pp. 46–48) report reconstruction accuracy and computational cost. Figure 1 has been reduced (p. 2).

We have uploaded an anonymized implementation package as supplementary material, including code for reference construction, model evaluation and figure reproduction.

Thank you again for the detailed feedback. We would be happy to clarify any of these points further.

Thank you for clarifying the learned operators, reference constructions and error measures. The distinction between learning coordinates for a prescribed geometry and learning the geometry itself is now clearer.

The Welsch experiment extends the evidence beyond quadratic objectives, but the reported improvement concerns agreement with mirror descent rather than optimization efficiency. Exact finite-step equivalence is a coherent theoretical target; however, the response does not establish when learning these coordinates is computationally preferable to applying mirror descent directly. Higher-dimensional validation also remains open.

The clarifications address several presentation concerns, but the practical justification for operator learning remains insufficient. I therefore retain my score of 2.

AuthorsOpenReview ↗

Clarification on the Intended Contribution

Thank you for revisiting the paper and for acknowledging the clarifications and additional experiment. We appreciate your thoughtful assessment.

Our main goal is to understand the coordinate relations between prescribed optimization algorithms, particularly at finite step sizes. Running mirror descent produces iterates for a given objective, whereas a coordinate map explains how these iterates relate to those of Euclidean gradient descent and how this relation changes after discretization. The entropy example in Section 1 illustrates why this distinction matters.

Operator learning allows us to study these relations across families of objectives and regularizers. In our coupled experiments, the trained FNO predicts corrections for unseen problems, works at different input resolutions with fixed parameters, and reduces trajectory disagreement over repeated updates.

We appreciate your perspective on computational efficiency and higher-dimensional validation as valuable directions for further research.

AuthorsOpenReview ↗

Additional construction-cost comparison

Thank you again for your comments on the computational motivation. We have added a construction-time and accuracy comparison in Table 18 (p. 49).

For 32 coupled two-dimensional test inputs, the trained FNO takes 230 ms per input, compared with 450 ms for the fastest tested fixed-coordinate implementation on a 33×4933\times49 grid. Their mean relative H1H^1 correction errors are 7.38% and 7.89×10−67.89\times10^{-6}%, respectively. These timings include input preparation, continuous-coordinate construction, and correction values and derivatives. Appendix C.5 describes the computations for each implementation and reports training costs separately.

These measurements quantify the cost of constructing coordinates. The comparison with direct mirror descent concerns the total cost of optimization. We have revised the introduction and conclusion to emphasize finite-step coordinate relations and their numerical construction.

The updated anonymous code package includes the scripts, inputs, saved results, and reproduction instructions for the construction-cost comparison in Table 18.

Reviewer_9CHkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the equivalence between mirror descent and Euclidean gradient descent under coordinate transformations, focusing on the distinction between continuous-time and finite-step dynamics. It establishes theoretical results on the existence and limitations of exact finite-step equivalence, develops small-step corrections, and uses neural operators to learn the corresponding coordinate maps across problem families. Numerical experiments are conducted on one-dimensional and coupled two-dimensional synthetic problems.

Critical strengths
  • The theoretical results appear mathematically sound, particularly the local existence and uniqueness result for exact finite-step coordinate maps. The proofs are technically careful and provide useful insights into the conditions required for algorithmic equivalence.
  • The construction of a genuinely coupled two-dimensional family of flat Hessian metrics is interesting. It provides a meaningful test setting beyond separable or purely one-dimensional examples.
  • The authors enforce invertibility of the learned maps and train the models using equation-based losses. The experiments also include several neural operator architectures and independently computed reference solutions.
  • The appendix provides additional experimental results, including less favorable outcomes and comparisons with classical numerical methods. This helps readers assess the performance and limitations of the proposed approach.
Critical weaknesses
  • The main object of interest is an objective-dependent coordinate map, but its practical usefulness remains unclear. In applications such as online learning and implicit-bias analysis, objective-independent transformations are often essential, and the paper does not demonstrate what new capabilities the learned maps enable.
  • Some theoretical results are relatively straightforward extensions of existing ideas, while the main local existence theorem requires restrictive assumptions that do not directly cover the experimental benchmarks. The remaining theoretical investigations are largely based on standard perturbative arguments or special cases, limiting the overall theoretical contribution.
  • The main-text accuracy metrics can be misleading because the learned coordinate maps are close to the identity, while errors on the actual nonlinear displacement are substantially larger. Moreover, the finite-step agreement experiments consider only very short trajectories and a single step size, making it difficult to assess the effectiveness and generality of the proposed corrections.
Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9CHk

Thank you for the helpful suggestions on error normalization and longer trajectories. They helped us make the experimental evidence clearer.

Contribution. We provide a new perspective on optimization algorithms by learning the coordinate maps that relate them across families of objectives and regularizers, especially for the discrete algorithms used in practice. Four questions organize the study: continuous equivalence, finite-step equivalence, connecting their coordinates, and modifying the objective or regularizer to obtain a shared map. The models predict these maps for new problems in the chosen families. The FNO models for the coupled continuous maps and finite-step corrections also operate at different input resolutions with their parameters fixed (Appendix C.1.2, pp. 36–37). Section 3.1 (p. 5) identifies the theorem and construction used for each experimental setting.

Error normalization. Table 1 (p. 4) reports errors relative to both the full map and its displacement from the identity. Predicting the identity gives 100% displacement-relative error whenever the reference displacement is nonzero. Across the three test groups, the FNO's full-map errors are 0.10%, 0.12% and 0.27%; the displacement-relative errors are 8.51%, 10.31% and 14.44%.

Longer trajectories. The models learn a coordinate correction from its defining equation. For each unseen problem, we keep the predicted correction fixed and repeatedly apply gradient descent in the resulting coordinates. We evaluate six step sizes from 0.0025 to 0.08 and up to four time horizons, giving trajectories of up to 64 steps within the test domain. Table 3 (p. 7) and Appendix C.2.3 (pp. 39–41) compare these transformed gradient iterates with direct mirror descent.

We have uploaded an anonymized implementation package as supplementary material, including code for reference construction, model evaluation and figure reproduction.

Thank you again for these suggestions. We would be happy to discuss the revised comparisons further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.