acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates

Abstract

Multiple-gradient descent is typically formulated in Euclidean geometry, whereas matrix-aware optimizers use directions induced by other norms. We analyze a single-loop multi-objective method combining a norm-constrained linear minimization oracle with entropic updates of task weights. For any finite-dimensional norm pair, we establish an ergodic stationarity rate in the deterministic setting, deriving the required trajectory gradient bound from smoothness and lower-boundedness in the unregularized case. With stochastic gradients and a bounded-gradient assumption, one persistent blended-gradient momentum buffer yields an rate whose leading term has no explicit dependence on the number of tasks. We also establish static regret for the inner weighting objective on arbitrary bounded gradient sequences and, with entropy regularization and a weight floor, iteration-wise inner-gap control after burn-in while retaining ergodic stationarity, conditional on a trajectory gradient bound. The optimal blended gradient can be nonunique in spectral and sign geometries, motivating a selection-independent inner gap. Our analysis extends the spectral construction studied by concurrent MOON to general norms. Experiments compare the method with existing multi-task optimizers and examine its oracle, weighting, and momentum choices.

Primary area:
optimization
Keywords:
Multi-Objective Optimization, Mirror Descent, Norm-constrained LMOs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bQ2BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adapts the multiple gradient descent algorithm (MGDA) to optimizers that takes steps through a LMO (linear minimization oracle) in an arbitrary norm: normalized gradient descent, signup and Muon. Each iteration: 1) blends the K task gradients with the current weight w_t to get G_t; 2) makes one oracle call, D_t = lmo(G_t); 3) score each task by g_{t,i} = <D_t, grad of f_i>; 4) updates the weights by Hedge. The paper provides the convergence of the proposed algorithm for both deterministic and stochastic settings. Experiments show its effectiveness over some benchmark datasets.

Critical strengths
  • The analysis use only the oracle identity <g, lmo(g)> = -R |g|_*. One proof covers normalized GD, signup, muon and layer-wise norms.
  • the gradient bound is derived in the deterministic case. It is more challenging for the stochastic case, and its design is careful: 1) calling the oracle on the previous buffer makes the noise cross term exactly zero in expectation (I found this technique interesting). 2) take advantages of the proof for the deterministic case and bound the noise term more conveniently.
Critical weaknesses
  • The components of the proposed methods are well-known in optimization: LMO descent (Pethick et al., 2025), single loop weight update for MGDA, and exponentiated gradient weighting (mirror descent). The convergence analysis for deterministic case is simply the combination of results of these components. I particularly like the design and theoretical analysis of stochastic case, which call the oracle on the previous buffer, making the noise cross term exactly zero in expectation. Is this technique used by the authors for the first time?

  • Each step of the proposed method needs K backward passes and storages of K full gradients, which costs K x d memory. The analysis of compute and memory costs should be discussed.

  • Step sizes depend on L, Delta, which requires knowing every f*_i, M and T. That's acceptable for theory, but practical claims should rely on tuned hyperparameters. It would be better if a sensitivity study for the hyperparameterers is provided.

Questions for authors

See the weaknesses.

3 comments

Thank you for the careful reading and for the positive assessment of the stochastic analysis.

W1: The components are known. We agree: LMO steps (Pethick et al., 2025), a single-loop weight update and exponentiated gradient are all known. The contribution is their analysis in an arbitrary norm pair, with three steps that do not follow from existing results. Lemma 4.3 derives a gradient bound along the deterministic trajectory, so the deterministic analysis needs no bounded-gradient assumption, which MOON and MSMD make. Lemma 4.7 bounds the expected weight increment by a weighted average of the noisy task scores rather than their maximum, which keeps the leading stochastic term free of KK; together with the log⁡K\log K regret of the entropic update, the dependence on the number of objectives KK is polylogarithmic, whereas MOON's constants are polynomial in it. Theorem 5.1 gives iteration-wise tracking of the inner optimum with a regularized, clipped weight update.

Is the previous-buffer evaluation new? Not as a device. Analyses of AdaGrad-type step sizes compute the step size from past gradients only, so that it is independent of the current stochastic gradient (Li & Orabona, 2019). Algorithm 2 applies it to the direction: evaluating the oracle at the previous momentum buffer Mt−1M_{t-1} makes DtD_t independent of the current minibatch, so the noise cross term has zero conditional mean. The device matters more here than in single-objective LMO methods (Pethick et al., 2025), where noise enters only through the momentum tracking error: our weight update also uses the raw task scores, whose noise momentum does not average. With the oracle at the current buffer, DtD_t would depend on the current minibatch, which makes the expected scores, and hence the cross term, difficult to control. The multi-objective baselines we cite do not use the device, and we are not aware of its use in other multi-objective methods. We will add these references.

W2: KK backward passes and K×dK\times d memory. These costs come from our implementation, not from the method: when we wrote the paper, we did not realize that the task scores can be computed by a Jacobian-vector product, and using one reduces the cost. Write F=(f1,…,fK)F=(f_1,\dots,f_K) and JF(xt)∈RK×dJ_F(x_t)\in\mathbb R^{K\times d} for its Jacobian. The task scores form the Jacobian-vector product gt=JF(xt)Dtg_t=J_F(x_t)D_t, which one forward-mode pass computes for all KK tasks at once, and the blended gradient JF(xt)⊤wtJ_F(x_t)^\top w_t is one backward pass of ∑iwt,ifi\sum_i w_{t,i}f_i. Each iteration therefore needs one backward pass and one forward-mode pass, of comparable cost, and stores no per-task gradient. The shared network is then traversed a constant number of times instead of once per task, and the memory for gradients of the shared parameters no longer grows with KK. The task heads are still evaluated once per task, as in any method, so computation drops when the shared network dominates the cost and K≥3K\ge3; for K=2K=2 it does not drop. Since Algorithm 2 fixes Dt=D_t= lmo$(M_{t-1})$ before drawing the minibatch, the forward-mode pass can be fused with the forward pass of the weighted loss. Section 4 will include this discussion.

W3: The step sizes depend on problem constants. We agree, and all reported experiments use tuned hyperparameters: Appendix I lists the search grids, and on MultiMNIST the selection uses a validation split. Tables 5 and 6 already report the sensitivity to the weight step η\eta and to the clipping parameter κ\kappa: MultiMNIST test accuracy stays between 89.5% and 91.1% across both grids. We will try our best to add a sensitivity study for the model step size γ\gamma. In the theory, the problem constants enter the step sizes only through a multiplicative factor: given a gradient bound along the trajectory, the choices γR=c/T\gamma R=c/\sqrt T and ηR=c′/T\eta R=c'/\sqrt T give the O(T−1/2)O(T^{-1/2}) rate of Theorem 4.4 for any c,c′>0c,c'>0, with a constant that depends on cc and c′c'. In particular, as for single-objective LMO methods (Pethick et al., 2025, Insight 5.1), the rate does not require knowing LL.

Update after the revision. The revised version (changes in blue) contains the following, in reference to our response:

  • Previous-buffer evaluation: Section 4.2 relates it to AdaGrad-type step sizes computed from past gradients only (Li & Orabona, 2019).
  • Compute and memory (W2): Section 4 states the cost per iteration, one backward pass and one forward-mode pass with no per-task gradient stored; the comments of Algorithm 1 give the pass counts, and Appendix I gives the cost model, including when this is cheaper than KK backward passes.
  • Hyperparameter sensitivity (W3): Table 7 adds the model step size γ\gamma: MultiMNIST test accuracy is 73.15%, 80.92%, 89.47%, 90.54% and 88.68% for γ\gamma equal to 5×10−45\times10^{-4}, 10−310^{-3}, 5×10−35\times10^{-3}, 10−210^{-2} (selected on the validation split) and 2×10−22\times10^{-2}. Together with Tables 5 and 6 (η\eta and κ\kappa), this shows that the step size γ\gamma matters most.
Reviewer_bQ2BOpenReview ↗

Thank you for the rebuttal.

I'd like to thank the authors for the clarification. I'd like to keep my score. Good luck!

Reviewer_s3YfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends MGDA beyond Euclidean geometry by combining norm-constrained update directions with entropic task-weight updates. The framework accommodates normalized-gradient, sign, and Muon-style spectral updates. The authors establish deterministic and stochastic convergence guarantees, derive a trajectory gradient bound for the deterministic unregularized method, and analyze stochastic updates using one blended-gradient momentum buffer. Experiments on Cityscapes, NYU-v2, and MultiMNIST evaluate the method.

Critical strengths
  1. The general-norm framework provides a useful extension of Euclidean multi-objective optimization and connects it with matrix-aware optimization methods.

  2. The theoretical analysis is substantial. In particular, the CA distance is also studied.

  3. The discussion of non-unique optimal blended gradients is insightful. The paper carefully distinguishes Pareto stationarity, static regret, and iteration-wise tracking, while clearly acknowledging the limitations of its guarantees.

Critical weaknesses
  1. Although the convergence exponents hold across norm pairs, the smoothness and noise-conversion constants depend on the geometry. Could the authors provide a concrete comparison showing when the general-norm formulation yields a meaningful advantage over Euclidean MGDA?

  2. The baselines are somewhat outdated. Could the authors add more recent MOO methods? Besides, could the authors report the multi-seed results?

  3. Table 2 shows that LMO-MGDA is worse than MOON, but the other tables show extraordinarily better performance than MOON and other baselines. How do the authors interpret such results?

Questions for authors
  1. Although the convergence exponents hold across norm pairs, the smoothness and noise-conversion constants depend on the geometry. Could the authors provide a concrete comparison showing when the general-norm formulation yields a meaningful advantage over Euclidean MGDA?

  2. The baselines are somewhat outdated. Could the authors add more recent MOO methods? Besides, could the authors report the multi-seed results?

  3. Table 2 shows that LMO-MGDA is worse than MOON, but the other tables show extraordinarily better performance than MOON and other baselines. How do the authors interpret such results?

3 comments

Thank you for the positive assessment and the precise questions.

Q1: When does the general-norm formulation help over Euclidean MGDA? When the losses have comparable smoothness constants in the Euclidean and spectral geometries, the spectral bound gives a stronger guarantee in nuclear-norm stationarity, by up to the square root of the matrix rank. The rate exponents are the same in every norm pair, so the comparison runs through the constants. For the square loss of a linear predictor W∈Rdout×dinW\in\mathbb R^{d_{out}\times d_{in}} with inputs normalized to ∥x∥2=din\Vert x\Vert_2=\sqrt{d_{in}}, Bernstein & Newhouse (2024, Proposition 6) show smoothness in the spectral norm with constant din/doutd_{in}/d_{out}; the same constant holds in the Frobenius norm, since ∥ΔWx∥2≤∥ΔW∥F∥x∥2\Vert\Delta Wx\Vert_2\le\Vert\Delta W\Vert_F\Vert x\Vert_2. Theorem 4.4 then bounds the average nuclear norm of the blended gradient under spectral steps by the same smoothness term as its average Frobenius norm under Euclidean steps. With r=min⁡(din,dout)r=\min(d_{in},d_{out}), the inequality ∥G∥∗≤r∥G∥F\Vert G\Vert_{\ast}\le\sqrt{r} \Vert G\Vert_F shows that the Euclidean bound gives nuclear-norm stationarity only after a factor r\sqrt r. More generally, writing LspecL_{spec} and LEuclL_{Eucl} for the smoothness constants in the two pairs, the spectral bound is no worse whenever Lspec≤rLEuclL_{spec}\le r L_{Eucl}. These statements compare upper bounds. In practice the norm is chosen from the architecture, as in single-objective LMO methods (Pethick et al., 2025); our analysis makes this choice available for multi-objective training. Section 3 will include this remark.

Q2: Recent baselines and multiple seeds. We used the same baselines as the multi-task optimization papers we are aware of on these benchmarks. Could the reviewer specify which recent methods they would like us to include? Running them on Cityscapes and NYU-v2 before the end of the discussion period will be difficult, since each run takes more than 20 hours on an H100 GPU, but we can compare directly with the results these methods report under the same protocol. Our dense-prediction results are single runs; multi-seed runs of our method are in progress, and we will post them during the discussion period if they finish in time. The MultiMNIST results are already means over three seeds.

Q3: Why is LMO-MGDA worse than MOON on Cityscapes but much better elsewhere? On MultiMNIST the margin comes from the optimizer configuration, not from the adaptive weighting. As κ→1/K\kappa\to1/K our method reduces to linear scalarization with the same optimizer, and Table 6 shows that it still reaches 90.96%, against 90.54% for the main configuration. The configuration differs from MOON's released implementation in three ways: unit-radius spectral blocks on all matrix-valued tensors with a single Euclidean block for vector parameters (MOON uses AdamW for these), momentum 0.1 without Nesterov, and the oracle evaluated at the previous momentum buffer. Section 6.1 will state this.

On the dense tasks, MOON is nearly a special case of our algorithm: with the spectral oracle and regularization τ>0\tau>0, our weight update matches MOON's logit decay to first order in the product of the step size and the regularization strength, and the remaining difference is that MOON evaluates the oracle at the current momentum buffer. In all our experiments we kept τ=0\tau=0 without tuning it, whereas MOON uses its logit decay; this regularization may explain MOON's advantage on Cityscapes. That advantage comes entirely from relative depth error: MOON reaches 31.39 against 36.29 for our method, which accounts for 3.21 points of Δm\Delta m%, while our method is better on the other three Cityscapes metrics (mIoU, pixel accuracy and absolute depth error), so that MOON's overall margin is 2.04 points (−5.42-5.42% against −3.38-3.38%). Our method is better on NYU-v2. The dense-task results are single runs (multi-seed runs are in progress), and our step sizes were tuned on the reported split while the baselines use their reference hyperparameters (Section 6, Appendix I.1); with these caveats, the results show competitive performance rather than a uniform advantage.

Update after the revision. The revised version (changes in blue) contains the following, in reference to our response:

  • Q1: Section 3 includes the paragraph "Comparing geometries" with the linear-predictor example and the comparison of the two smoothness constants.
  • Q3: Section 6.1 and Tables 7 and 8 add step-size sweeps for our method and for MOON. Table 4 reports MOON at its default learning rate 10−310^{-3}; with its learning rate swept over the same grid as ours, MOON reaches at most 89.71%, at 10−210^{-2}, where our method reaches 90.54%, and its logit decay and task-weight step change its accuracy by less than 0.5 points. Section 6 states that MOON is close to the variant of our stochastic algorithm with entropic regularization that evaluates the oracle at the current momentum buffer, and that all our runs use τ=0\tau=0.
  • Q2: the multi-seed runs of our method on Cityscapes and NYU-v2 are still in progress. We would also appreciate knowing which recent baselines the reviewer has in mind.

Update: multi-seed results. The three-seed runs of our method on the dense-prediction tasks have finished. Mean and standard deviation over the three seeds:

CityscapesmIoUPix accAbs errRel errΔm%
Ours79.11 ± 0.1894.48 ± 0.060.0136 ± 0.000137.44 ± 2.34-2.59 ± 1.41
NYU-v2mIoUPix accAbs errRel errMeanMedian<11.25<22.5<30Δm%
Ours42.66 ± 1.0467.94 ± 0.340.5135 ± 0.00530.2137 ± 0.007724.55 ± 0.6919.37 ± 0.2829.95 ± 0.5256.93 ± 0.5969.24 ± 0.43-5.90 ± 1.30

The single runs reported in Tables 2 and 3 lie within about one standard deviation of these means. The segmentation metrics vary little across seeds; on Cityscapes most of the variation in Δm% comes from the relative depth error. The conclusions of Section 6 are unchanged: on NYU-v2 our mean Δm% remains below MOON's (-3.72), and on Cityscapes MOON remains better in Δm%, as reported. The baselines, including MOON, are single runs, so these comparisons are between our mean and single baseline runs.

Reviewer_fbtBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a geometry-aware multi-objective optimization framework that combines norm-constrained LMO updates with entropic mirror descent for task weighting. The method extends MGDA beyond the standard Euclidean setting to more general geometries, including those corresponding to SGD, Signum, and Muon. The authors provide convergence analysis for both deterministic and stochastic settings, along with additional guarantees for the task-weighting process. Experiments demonstrate competitive performance compared with existing multi-task optimization methods.

Critical strengths
  1. The paper provides a unified framework for geometry-aware multi-objective optimization beyond the standard Euclidean setting. Extending the analysis to different norm geometries, including L2, L-infinity, and spectral norms, is a meaningful contribution, as it connects multi-objective optimization with recent geometry-aware optimizers such as Signum and Muon.

  2. The theoretical analysis is relatively comprehensive. In particular, the authors establish convergence guarantees for both deterministic and stochastic settings. The use of a shared momentum buffer to handle stochastic updates, as well as the analysis of inner-gap tracking under non-unique optimal solutions, provides additional technical value.

Critical weaknesses

The novelty over existing work is not sufficiently clear. The proposed framework combines several existing ideas, including entropic mirror descent for task weighting and norm-constrained LMO updates. In particular, the spectral-norm setting has substantial overlap with MOON, which already studies geometry-aware multi-objective optimization and establishes similar convergence rate exponents. Although the general-norm analysis and additional theoretical results are useful, the paper does not clearly demonstrate what fundamentally new optimization capability is enabled by the proposed framework.

Questions for authors

The proposed framework supports different norm geometries, such as L2, L-infinity, and spectral norms. However, the choice of geometry appears to be predefined. Can the authors extend the framework to automatically select or adapt the norm during training based on the task gradients or optimization dynamics, rather than requiring manual selection? What would be the main theoretical challenges in doing so?

3 comments

Thank you for the review. We address the novelty relative to MOON, the capability question and adaptive norms.

Novelty relative to MOON. The main difference is the dependence on the number of objectives: MOON's guarantees are polynomial in it, ours are polylogarithmic. MOON is concurrent work: it appeared on arXiv in August 2026, around the time this work was completed, and to our knowledge it is not published at a peer-reviewed venue, which makes it contemporaneous under the ICLR 2027 guidelines. MOON is nearly a special case of our algorithm: with the spectral oracle and regularization τ>0\tau>0, our weight update matches MOON's logit decay to first order in the product of the step size and the regularization strength, and the remaining algorithmic difference is that MOON evaluates the oracle at the current momentum buffer. The rate exponents coincide, as Section 2 states. Our work differs from MOON in five respects.

  1. Polylogarithmic dependence on the number of objectives. MOON's weight step is β=1/(mHT)\beta=1/(mHT), where mm is the number of objectives, HH a bound on the gradients and TT the horizon, and its constants carry mm and m\sqrt m. With KK objectives, our bounds depend on KK only polylogarithmically:

    • Deterministic. Theorem 4.4 with the gradient bound derived in Lemma 4.3 reaches an ϵ\epsilon-stationary point in O(LΔlog⁡2K/ϵ2)O(L\Delta\log^2K/\epsilon^2) iterations, and the weights have static regret M2log⁡K/TM\sqrt{2\log K/T} (Theorem 4.5).
    • Stochastic. The leading T−1/4T^{-1/4} term of Theorem 4.9 has no explicit KK. Under bounded variance the lower-order terms are linear in KK; under sub-Gaussian noise every constant depends on KK only through log⁡K\log K (Corollary D.4).

    The key step is Lemma 4.7, which bounds the expected weight increment by a weighted average of the noisy task scores rather than their maximum; bounding it through the maximum, as a direct pathwise argument does, costs a factor K\sqrt K.

  2. Generality. One analysis covers every norm pair: Euclidean, sign (ℓ∞\ell_\infty), spectral and layer-wise product norms. MOON treats the spectral case.

  3. Previous momentum buffer. Algorithm 2 evaluates the oracle at Mt−1M_{t-1}, so the direction is fixed before the minibatch is drawn, as in AdaGrad-type step sizes computed from past gradients only (Li & Orabona, 2019). The noise in the task scores is then independent of the direction, and the corresponding cross term has zero conditional mean. MOON evaluates the oracle at the current buffer.

  4. Clipped weights and conflict avoidance. With a regularized, clipped weight update, Theorem 5.1 shows that the inner gap is small at every iterate after a burn-in, which yields an iteration-wise bound on the CA distance when the dual norm is uniformly convex (Corollary 5.2). MOON gives no guarantee on the weights.

  5. Assumptions. Lemma 4.3 derives the gradient bound along the deterministic trajectory; MOON assumes bounded gradients and bounded losses.

What new optimization capability is enabled? Beyond the guarantees above, the formulation has a practical consequence for cost. The task scores are directional derivatives along one direction, so each iteration needs one backward pass and one forward-mode pass: the shared network is traversed a constant number of times instead of once per task, and no per-task gradient is stored (see our response to Reviewer bQ2B). With the previous-buffer evaluation of Algorithm 2, the forward-mode pass can be fused with the forward pass.

Can the norm be adapted during training? This direction is interesting and outside the scope of the paper. The analysis already allows the norm at each iteration to be chosen from a fixed finite family using past information, with LL and MM replaced by their maxima over the family; a selection rule with a provable benefit is future work.

Update after the revision. The revised version (changes in blue) contains the following, in reference to our response:

  • Relation to MOON: Section 6 states that MOON is close to the variant of our stochastic algorithm with entropic regularization that evaluates the oracle at the current momentum buffer.
  • MultiMNIST comparison: Table 8 sweeps MOON's model learning rate, logit decay and task-weight step. With both step sizes swept over the same grid, our method reaches 90.54% and MOON 89.71%, both at a step size of 10−210^{-2} (Section 6.1).
  • Cost: Section 4 and Appendix I state the cost per iteration: one backward pass and one forward-mode pass, with no per-task gradient stored.

I appreciate the authors' detailed clarification regarding the theoretical differences from MOON. I will improve my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General answer

We thank the reviewers for their careful reading. Three revisions address points raised by more than one reviewer.

  1. Cost per iteration (bQ2B, fbtB). The task scores gt,i=⟨Dt,∇fi(xt)⟩g_{t,i}=\langle D_t,\nabla f_i(x_t)\rangle are directional derivatives along the single direction DtD_t, so all KK of them form one Jacobian-vector product, and the blended gradient is one backward pass of the weighted loss. Each iteration therefore traverses the shared network a constant number of times instead of once per task, and stores no per-task gradient. Section 4 will state this.
  2. Relation to MOON (bQ2B, fbtB). MOON is concurrent work. Our analysis improves the dependence on the number of objectives KK from polynomial to polylogarithmic: O(LΔlog⁡2K/ϵ2)O(L\Delta\log^2K/\epsilon^2) iterations in the deterministic case, and a leading stochastic term with no explicit KK. It also covers every norm pair, evaluates the oracle at the previous momentum buffer, and clips the weights to obtain iteration-wise conflict-avoidance guarantees.
  3. MultiMNIST margin (s3Yf). Table 6 shows that the margin over MOON comes from the optimizer configuration, not from the adaptive weighting. Section 6.1 will state this.

Multi-seed runs of our method on the dense-prediction tasks are in progress (more than 20 hours per run on an H100 GPU). We will try our best to add the other experiments the reviewers ask for.

AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers for their careful reviews and constructive discussion, and the AC for handling the paper. After the discussion, all three reviewers rate the paper 3 (weak accept): Reviewer fbtB raised their score from 2 after our clarification of the theoretical differences from MOON, Reviewer bQ2B, whose review highlighted the stochastic analysis and the previous-buffer evaluation as strengths, thanked us for the clarifications and kept their score, and Reviewer s3Yf indicated that our responses resolved most of their concerns; we then answered their remaining request, a comparison with FairGrad. We have responded to every concern raised in the reviews and the discussion, and summarize the main points below.

  1. Relation to MOON (fbtB, bQ2B). MOON is concurrent work. Our analysis improves the dependence on the number of objectives from polynomial to polylogarithmic, covers every norm pair rather than the spectral case, evaluates the oracle at the previous momentum buffer so that the score noise has zero conditional mean, and uses clipped weights to obtain iteration-wise guarantees on the inner gap; the deterministic analysis also derives the gradient bound instead of assuming it.
  2. Cost and step sizes (bQ2B). The task scores form one Jacobian-vector product, so an iteration needs one backward pass and one forward-mode pass and stores no per-task gradient (Section 4, Appendix I). The rates do not require knowing the smoothness constant, and Tables 5 to 7 report the sensitivity to the weight step, the clipping parameter and the model step size.
  3. Choice of geometry (s3Yf). A new paragraph in Section 3 compares the constants of the spectral and Euclidean geometries for the square loss of a linear predictor.
  4. Experiments (s3Yf). We posted three-seed results for our method on Cityscapes and NYU-v2. Table 8 sweeps MOON's hyperparameters: with both step sizes swept over the same grid, our method reaches 90.54% on MultiMNIST and MOON 89.71%. Using FairGrad's published results, which follow the same protocol, our method is better on all nine NYU-v2 metrics; on Cityscapes it is better on segmentation, while FairGrad has the lower aggregate, through relative depth error. FairGrad's guarantee is an asymptotic, deterministic subsequence result, whereas our analysis gives non-asymptotic deterministic and stochastic rates.

The revised version, with changes in blue, includes these additions (Sections 3, 4, 4.2, 6 and 6.1, Tables 7 and 8, Appendix I). We thank everyone again for their time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.