acceptodds
Under review as a conference paper at ICLR 2027

AdaMult: Adaptive Dual Optimization for Lagrangian Constrained Deep Learning

Abstract

We introduce AdaMult, a dual optimizer for Lagrangian constrained deep learning that produces feasible, well-performing solutions out of the box, at its one default hyperparameter configuration, across a wide range of tasks. This removes the costly trial-and-error tuning process of dual hyperparameters that constrained deep learning otherwise requires. AdaMult normalizes each constraint by a running estimate of its scale, so the multipliers are driven by a dimensionless signal; updates the multipliers multiplicatively, so they traverse orders of magnitude quickly; and extrapolates each multiplier along its current violation, which, under suitable step sizes and damping, makes the primal–dual dynamics locally stable at every regular local constrained minimizer. On six tasks, AdaMult at its default is feasible on every seed of every task and matches the performance of baselines that were tuned separately for each task. Held to a single configuration across tasks, every existing dual optimizer we test is infeasible on at least one of them, or feasible at a loss in performance. To our knowledge, AdaMult is the first dual optimizer that transfers reliably across tasks without tuning.

Primary area:
optimization
Keywords:
constrained optimization, Lagrangian methods, dual optimizer, Lagrange multipliers, adaptive optimizer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fnexOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes AdaMult, an optimization algorithm for constrained learning whose main feature is that it normalizes the constraint violation in the dual update step in order to become more robust to scaling issues.

Critical strengths

The paper aims to reduce the algorithm's sensitivity to the choice hyper-parameter, and therefore the amount of hyper tuning that is typically required to achieved good performance. That is a meaningful goal and achieving it would be relevant and valuable to the community.

Critical weaknesses

The paper's main weakness/obstacle is its (lack of) clarity. I have a very hard time following the narrative and methodology, and the paper is full of phrases / sentences that feel over-hyped or over-confident, such as:

  • "ADAMULT’S ingredients all earn their place"
  • "ADAMULT does for the multipliers what Adam did for the model: it works out of the box"

Several claims are made regarding the lack of tuning needed for the proposed method, which are only demonstrated on a small set of tasks. Given the breadth of the machine learning and optimization literature, six tasks is not enough to claim that no tuning is necessary; the paper only demonstrates that a particular choice of hyperparameters performed well on the particular set of tasks reported in the paper.

I would highly recommend that the authors

  1. revise the tone of the paper to only make claims that are substantiated by evidence and leave reasonable doubt elsewhere
  2. demonstrate the behavior and performance of the proposed algorithm on a few small nonlinear optimization instances. This would help the reader under how the algorithm performs in practice
Questions for authors
  • The "related work" section is one massive blob of text which makes it hard to read and understand
  • The first half of section 3 only mentions multipliers associated to inequality constraints
  • The algorithm tracks the log-magnitude of the multipliers, which may cause numerical issues if the log-state becomes too large. The algorithm appears to include several clipping steps to prevent such growth, which may however prevent convergence to the true optimum.
1 comment
AuthorsOpenReview ↗

Response to Reviewer fnex

We thank the reviewer for their feedback and for recognizing the value of reducing hyperparameter tuning in constrained learning.

W1. "A very hard time following the narrative and methodology". The paper is built around one line of argument, in four steps. It states two requirements for a dual optimizer to serve as a default (Section 2), presents a method built to meet them (Section 3), proves one proposition for each requirement (Propositions 2 and 4), and tests its two empirical claims in one table each (Tables 1 and 2). The review points to two short passages whose presentation it questions, the related-work paragraph and the first half of Section 3, which we address next.

Q1. The related work is "one massive blob of text". We will split it into three paragraphs (the origins of AdaMult's three ingredients, parameter-free optimizers, and the methods closest to AdaMult).

Q2. "The first half of section 3 only mentions multipliers associated to inequality constraints". We will say at the start of Section 3 that equality multipliers follow the same update, with sinh in place of the exponential (currently stated later, at line 185).

W1, W3. "Only make claims that are substantiated by evidence". We share this standard, and the paper already states its results with their scope where it reports them, for example in the abstract (lines 19-21), the introduction (lines 93-94), Section 4.1 (lines 450 and 463) and the conclusion (line 478). In six places, a summary of these results leaves that scope implicit, and we will make it explicit in each, in most cases by adding the qualifier "on the six tasks tested". Two are the sentences the reviewer quotes, and the other four state that AdaMult makes tuning unnecessary (lines 13-15, 23-24 and 52, and the title of Section 4.1). We therefore respectfully disagree that the paper is "full of" such sentences.

W2. "Six tasks is not enough to claim that no tuning is necessary". We agree that no finite set of tasks can show this, and the paper claims it only for the tasks it tests. In particular, it acknowledges that "no default pp is guaranteed to suffice in general" (line 364).

What the paper does show has two parts. Proposition 4 proves, for every problem and under mild conditions, that rescaling the constraints leaves AdaMult's primal iterates unchanged, which removes one reason to retune. The rest is, in the paper's words, "an empirical question" (line 369), and the six tasks are a small but demanding test of it. They range from one constraint to 50,000, cover inequalities and equalities, and include supervised learning, a 7B language model and reinforcement learning. One default is feasible and competitive on all six, where no baseline held to one configuration is (Table 1).

The newly added experiments on 808 synthetic problems (next answer) also show that, indeed, many problems are not solved by AdaMult at its default. However, they support our broader claim that AdaMult can transfer better than the baselines.

W4. "Demonstrate the behavior and performance of the proposed algorithm on a few small nonlinear optimization instances". We ran AdaMult and the baselines on 808 synthetic problems, which are quadratic and exponential programs with 10 or 100 variables and inequality or equality constraints. AdaMult's default, which was selected on this family of problems, solves 45% of them, against 32% for the best baseline at its best single configuration, thus supporting our thesis that one default can transfer across problems. The full table is in our response to all reviewers, and we will add the experiments to the paper.

Q3. The clipping steps "may however prevent convergence to the true optimum". A bound that is active at the solution would indeed prevent convergence to it. We will remove the upper bound: it was only aimed at guarding against overflow, and no AdaMult run on our tasks ever came near it. However, we will keep the lower bound: it holds the normalized multiplier of an inactive constraint at e−10e^{-10}, which is practically zero. Under this bound, Proposition 2 still proves that a point near the true solution is locally stable, and that this point converges to the true solution as zmin⁡→−∞z_{\min} \to -\infty. In practice, no bound will then prevent the method from converging to essentially the true solution.


We hope this addresses the reviewer's concerns, and we are happy to clarify further.

Reviewer_x7waOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AdaMult updates dual variables in constrained neural network training. It combines constraint residual normalization, an exponential parameterization of nonnegative multipliers, and a proportional–integral correction; a signed parameterization is used for equality constraints. The goal is to reduce dual optimization sensitivity to constraint scaling and hyperparameter selection. The authors provide a local stability analysis and experiments on six tasks, including classification, sparsification, preference optimization, and reinforcement learning. The paper addresses an important practical problem and proposes a meaningful combination of stabilization mechanisms. However, the theory applies to simplified dynamics, and the experiments do not isolate the proposed method’s contribution from established components or fully substantiate the robustness of a single hyperparameter configuration.

Critical strengths
  1. Practically relevant problem formulation. Lagrange multiplier sensitivity to constraint scaling and dual learning rate makes constrained deep learning hard to apply. AdaMult addresses this.
  2. Diversity of experimental tasks. The evaluation covers equality and inequality constraints, and problems with different numbers of constraints. This covers more ground than a single model or constraint type.
  3. An attempt to explain the stabilization mechanism theoretically. The local stability analysis clarifies the role of the proportional correction. It is useful, though narrower in scope than the implemented algorithm.
Critical weaknesses
  1. Novelty relative to existing approaches is not isolated. Exponential multiplier updates, adaptive scaling, and PI control have precedents. The contribution lies in their combination and implementation. Significance requires controlled comparisons against combinations of the same components: normalized dual ascent, normalized PI updates, and exponential updates without the proportional correction. The results do not show which component drives the improvement.
  2. The stability theorem does not cover the full practical algorithm. The analysis assumes fixed normalization scales, vanilla gradient descent on primal variables, and no upper clipping. It also requires local regularity conditions, a sufficiently strong proportional correction, and sufficiently small step sizes. Thus, it does not establish stability of the full stochastic procedure with adaptive RMS normalization and default hyperparameters. The main claims should reflect this limitation.
  3. Upper clipping can introduce additional infeasible fixed points, even in a simple convex problem. In an independent numerical check based on the paper’s pseudocode, I considered \min_x \tfrac12(x-1000)^2,\qquad x\leq0, initialized at x_0=1000 with zero internal AdaMult states. With default AdaMult hyperparameters and a primal gradient descent step size of 0.01, the iterate stabilized near x=977.47 after 20,000 iterations, substantially violating the constraint. Increasing the upper bound on the internal log-state from 10 to 20 resolved the issue in this example. The feasible solution is representable even with the original bound, so the issue concerns optimization dynamics and an additional infeasible equilibrium. This does not contradict the theoretical analysis, which excludes active upper clipping, but it warrants a sensitivity analysis and discussion in the paper.
  4. Experimental breadth does not replace systematic evaluation of claimed robustness. The claim of reduced tuning requirements calls for systematic variation of objective and constraint scales, constraint tightness, and number of constraints. Comparisons under equal hyperparameter tuning budgets are also needed. Without these, it is hard to distinguish an algorithm advantage from favorable settings for the selected tasks.
Questions for authors
  1. Does AdaMult retain its advantage over normalized PI updates and normalized exponential updates under an equal hyperparameter tuning budget? Which component accounts for most of the improvement?
  2. How should clipping bounds be selected? Can one detect when they prevent convergence to a feasible solution, and how does the method behave on the convex example above?
  3. Which conclusions of the theoretical analysis remain valid with adaptive normalization, stochastic gradients, and Adam for primal updates? Do the default settings satisfy the theorem’s conditions in any of the experimental regimes studied?
  4. How robust is the single hyperparameter configuration to changes in objective scale, constraint tightness, and number of constraints? Does the advantage persist when methods are compared at matched levels of constraint violation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer x7wa

We thank the reviewer for their careful feedback and for recognizing AdaMult as "a meaningful combination of stabilization mechanisms".

W1, Q1. "The results do not show which component drives the improvement". The component that drives most of the improvement is normalization, which is directly tied to robustness: it is what makes AdaMult invariant to constraint rescaling (Proposition 4).

The paper already runs an ablation of AdaMult's ingredients, including the reviewer's three suggested controls under other names:

  • "normalized dual ascent" is the RMSProp baseline of Table 1;
  • "normalized PI updates" is Table 2's row without the multiplicative update;
  • "exponential updates without the proportional correction" is Table 2's row without extrapolation.

Even tuned to its best single configuration across tasks, each of these controls still falls short of AdaMult at its default, as Table 2 and Table 12 of Appendix C.2 show. Without normalization, a single configuration does not transfer across tasks. Without extrapolation, runs can fail to reach feasibility even when tuned per task. Without the multiplicative update, one default works well across all tasks, but task performance is lower, though only by 3% on average.

The importance of each ingredient also shows on the 808 synthetic problems presented in our response to all reviewers. At a single configuration, using one ingredient alone solves at most 29% of the problems and using two at most 35%, against 45% for AdaMult, which combines all three.

IngredientsWhichSingle configurationTuned per problem
NoneGA25%39%
OneNormalization (additive AdaMult without extrapolation)29%42%
OneMultiplicative20%29%
OneExtrapolation (PI control)28%50%
TwoNormalization + Multiplicative (the reviewer's "normalized exponential updates")35%48%
TwoNormalization + Extrapolation (the reviewer's "normalized PI updates")34%55%
TwoMultiplicative + Extrapolation20%37%
All threeAdaMult45%68%

The table gives the fraction of problems solved in two settings: at the best single configuration for all problems (for AdaMult at its default), and tuned separately for each problem. The rows are grouped by how many of AdaMult's three ingredients are used.

We will add this comparison to the paper and label the rows of Table 2 more clearly.

W2, Q3. "The stability theorem does not cover the full practical algorithm". The reviewer describes Proposition 2 accurately, and as the paper already states it: a local convergence guarantee under a deterministic objective and constraints, and plain primal gradient descent. Although "narrower in scope than the implemented algorithm", this is the standard guarantee that first-order, single-step ALM [1] and dual PI control [2] have, and one that gradient ascent and RMSProp lack.

Two theoretical results do hold with adaptive normalization and an Adam primal: AdaMult's fixed points correspond to the KKT points of the problem (Proposition 1), and its primal iterates are invariant to constraint rescaling (Proposition 4), the latter under stochastic gradients as well.

Q3. "Do the default settings satisfy the theorem's conditions in any of the experimental regimes studied?" No default can be guaranteed to meet the conditions of Proposition 2, including the one we chose. In particular, while a large enough damping coefficient exists for every regular solution, how large depends on that solution's curvature, so no value can suffice for every problem (as already noted in line 364).

Nor did we check the conditions on our six tasks, since evaluating that curvature requires potentially costly second-order computations after a run has already converged; and a run that converges has already demonstrated the stability these conditions are meant to guarantee. In our experiments, AdaMult satisfactorily achieves this on all six tasks, including under an Adam primal and stochastic gradients, and thus demonstrates that stability there.

W3, Q2. "Upper clipping can introduce additional infeasible fixed points". The reviewer is right, and we thank them for the example. We will remove the upper bound, so there will be no such bound left to select or monitor. It was aimed at guarding against overflow, and no AdaMult run on our tasks ever came near the original bound. We also reproduced the reviewer's example, and with the upper bound removed, AdaMult converges to the solution, x=0x = 0.

W4, Q4. "Systematic variation of objective and constraint scales, constraint tightness, and number of constraints". In our response to all reviewers, we test AdaMult and the baselines on 808 synthetic problems that vary the number of constraints, the spread of their scales and of their optimal multipliers (and thus how tightly each constraint binds), the conditioning of the problem and the noise on the constraint values. At its default, AdaMult solves 45% of the 808 problems, against 32% for the best baseline at its best single configuration.

For constraint scales, Proposition 4 goes further than these experiments can: it proves that rescaling a constraint by any positive factor leaves AdaMult's primal iterates unchanged.

For the objective's scale, our comment "Additional Theoretical Result" proves that, under a specific initialization of the multipliers, AdaMult is invariant to rescaling the objective, the constraints, or both.

W4, Q1. "Comparisons under equal hyperparameter tuning budgets". While the tuning budgets in Table 1 of the paper are unequal, they favor the baselines. Each baseline in the table's top block is tuned on every task (13 to 46 configurations each), yet AdaMult, with no tuning at all, performs on par with them. Each baseline in the middle block is held to its best single configuration shared across the six tasks, yet AdaMult at its default outperforms them all. In either case, equalizing the budgets would mean tuning AdaMult as well, which could only work in its favor. We deliberately did not, to show that its default holds up even at a disadvantage.

For a comparison at equal tuning budgets, we point again to the synthetic problems above. There, AdaMult is tuned as well: its default was selected on this family of problems, using a grid of 125 configurations, the same size as the one considered for PI control. Tuned in this way, AdaMult solves 45% of the problems, against 28% for PI control and 32% for ALMA. Thus, AdaMult leads when the tuning budgets are equal, and holds up when they favor the baselines.

Q4. Whether the advantage persists "at matched levels of constraint violation". Yes. In our experiments, every method is held to the same feasibility tolerance, so none can gain performance from a violation beyond it. From its whole tuning grid, each baseline is then represented by its best-performing feasible configuration. While this could still produce a reported result that over-satisfies a constraint at a cost in performance, this only happens when no other feasible configuration in the baseline's grid performs better. In other words, no baseline is held back by satisfying a constraint more than it had to.

Moreover, this pathology can be checked in Tables 10 and 11 of Appendix C.2, which report each method's signed constraint violation next to its performance. For example, on some tasks where AdaMult performs best, its violation is also the lowest among the feasible methods (0.00% in Sparsity) or matches those of RMSProp and Adam (−20.6%-20.6\% against −20.7%-20.7\% and −20.4%-20.4\% in Safe DPO).

In any case, the paper does not claim that AdaMult produces models that outperform those of baselines tuned per task. Its claim is that AdaMult can match them at one default on our six machine learning tasks, without tuning (lines 450-452).


We hope this addresses the reviewer's concerns, and we are happy to clarify further.

References

  • [1] Bertsekas. Nonlinear Programming. Athena Scientific, 2016.
  • [2] Ramirez and Lacoste-Julien. Dual Optimistic Ascent (PI Control) is the Augmented Lagrangian Method in Disguise. ICLR, 2026.
Reviewer_cCZMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a primal-dual Lagrangian method for constrained deep learning. The method could be viewed as a special case of the augmented-Lagrangian paradigm with additional heuristics targeting better stability of the primal-dual dynamics and robustness with respect to hyper-parameters. It integrates three technical ingredients: (1) normalizing each constraint by a running estimate of its scale, (2) extrapolating each multiplier along its current violation and (3) updating the multipliers through their log-states. The authors report evaluations and comparisons over six tasks, showing the method match the performance of baselines that were tuned separately for each task.

Critical strengths
  • Constrained optimization for deep learning is a wide-interest problem.

  • In deep learning, the stability of the dual updates is a real issue (at least in my own experience) and the search for good heuristics that work is a highly relevant problem in practice.

  • Making constrained optimizers robust to the optimization hyper-parameters across tasks is also an interesting problem.

Critical weaknesses
  • Questionable claims: The main claim is that the proposed method provides task-adaptive updates of the dual ascent parameters (such as step size and damping coefficient), enabling a single hype-parameter configuration across tasks, unlike the existing constrained optimization methods. For instance, the authors specifically mention in line 158 that the Augmented Langrangian Method (ALM) fails in transferability, requiring both the step size and the damping coefficient to be tuned per task. Well, this is not true. In fact, in ALM methods, the two parameters of a penalty-Lagrangian function (the penalty multiplier and an extra parameter) are not tuned but updated in an adaptive way (See references [1] and [2] below, for instance). Specifically, the penalty multiplier is updated to the derivative of the penalty function, and the extra parameter is increased based on the value of the constraint function at the current outer iteration, compared to the previous one. BTW, I did not check in details but it is possible that the log-state dual updates proposed in this submission correspond to an ALM method with a specific choice of penalty-Lagrangian function – See Section 3.1 Univariate Functions in [1] for a wide choice of Lagrangian-penalty functions. This makes the novelty claim in regard to the multiplicative form of updates of the dual parameters also questionable (as this could stems from the specific choice of a Lagrangian-penalty function). The updates in Eq. (4) might be just a special case the augmented Lagrangian paradigm except that the penalty function is different from the standard quadratic one.

  • Limited novelty: The use of extrapolation of the Lagrange multipliers to stabilize their dynamics in the context of deep learning is not totally new. For instance, the work in [2] used the exponential moving average to smooth the evolution of the lagrangian-penalty multiplier (see Eq. 9 in [2]) in the context of adversarial attacks. While the mathematical form of the extrapolation is different here, the concept is quite the same.

  • Missing comparisons: Following on the two points above, I think authors should at the very least compare with augmented Lagrangian methods done properly, i.e., with adaptive hyper-parameter settings [1, 2] and extrapolation [2], as discussed above.

  • The many technical results in the paper (5 Propositions and 1 Lemma) do not really provide insights into the empirical performance gains: For example, Proposition 4 appears largely to formalize a natural scale-equivariance property. As rescaling a constraint is absorbed into an inverse rescaling of the associated multiplier, the optimization problem itself remains unchanged. Am I missing something here? As currently presented, I found it difficult to assess how much insights this result provides into the empirical performance gains. Also, Lemma 1 seems to be a text-book result for very general constrained optimization problems of the form in Eq. (1). I am not sure how relevant is including this result here when the authors are introducing specific Lagrangian-method updates and some heuristics for stability. Adding such not-so-relevant technicality just affects the flow of ideas in the presentation and makes reading the manuscript difficult.

[1] Numerical Comparison of Augmented Lagrangian Algorithms for Nonconvex Problems, Computational Optimization and Applications, 2005

[2] Augmented Lagrangian Adversarial Attacks, ICCV 2021

Questions for authors

I do not have further questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer cCZM

We thank the reviewer for their feedback and for acknowledging the significance of developing stable and robust dual updates for deep learning.

W1. ALM parameters are "not tuned but updated in an adaptive way". Our statement that the augmented Lagrangian method requires per-task tuning concerns the form most often used in deep learning practice: the first-order, single-step update of Eq. (ALM/PI) with a fixed step size and damping. We agree that ALM methods can adapt these parameters, and we will clarify this in the manuscript.

That said, adaptive rules are not guaranteed to solve the tuning problem, as they move the burden to their own hyperparameters. In [1, 2], these include the initial penalty coefficient, a growth factor γ\gamma, and a threshold τ\tau that decides when the penalty grows. A poor choice of these hyperparameters is known to lead to ill-conditioning, and [1, Sec. 4.4] itself notes that performance is sensitive to them and tunes them by grid search over 162 combinations. These adaptive rules are not guaranteed to perform well beyond the problem they were tuned on, whereas AdaMult has one such robustness guarantee in its invariance to constraint rescaling (Proposition 4).

W3. "Missing comparisons" with adaptive ALM. The submitted paper already compares against one adaptive ALM: RMS-ALM [3], which gives each constraint its own penalty parameter and adapts it at every step. Table 1 of the paper reports it tuned per task, tuned across tasks, and at its published default. At that default, it is feasible on 53% of the runs across the six tasks, against 100% for AdaMult at its own default.

We have nonetheless now also run the comparison with one of the methods the reviewer cites, ALMA [2], on the 808 synthetic problems of our response to all reviewers, tuned like every other baseline. Even at its best single configuration, ALMA solves 32% of the problems, against 45% for AdaMult at its default. Even tuned separately for every problem, it solves 42%, still fewer than AdaMult at its default. ALMA also diverges easily, in line with the ill-conditioning noted in our previous answer.

MethodConfigurationSolved (tuned across problems)Diverged (tuned across problems)Solved on held-out problems (tuned across problems)Solved (tuned per problem)
ALMA [2]Best of 14032%27%33%42%
AdaMultDefault45%8%44%68%

This table is an excerpt of the one in our response to all reviewers, where the held-out problems are described. ALMA's grid of 140 varies its initial penalty parameter ρ∈{10−4,10−3,10−2,10−1,1,10,100}\rho \in \{10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 1, 10, 100\}, its growth factor γ∈{1.001,1.01,1.05,1.2,2}\gamma \in \{1.001, 1.01, 1.05, 1.2, 2\} and its averaging weight α∈{0,0.5,0.9,0.99}\alpha \in \{0, 0.5, 0.9, 0.99\}. Since ALMA is defined for inequality constraints only [2], we give it each equality constraint as two inequalities.

ALMA's runs on our six machine learning tasks are still in progress, and we will add their results to a future revision of the paper.

We do not compare against [1] directly, because its methods minimize the augmented Lagrangian approximately between multiplier updates, whereas our scope is a single primal step per dual update. ALMA [2] essentially adapts the methods of [1] to that single-step setting, which is why we choose to compare against it.

W1. AdaMult's updates "might be just a special case" of the augmented Lagrangian paradigm. The reviewer is right that AdaMult's extrapolation and multiplicative update correspond to an ALM with the exponential penalty [4], as the paper already notes (lines 95-101, 148-149). AdaMult's constraint normalization can also be absorbed into the dual step sizes and penalty coefficients, and thus viewed as a strategy for setting them. Taken together, these connections indeed mean that one could equally formulate AdaMult as an ALM, instead of as a dual optimizer applied to the plain Lagrangian. We will add an appendix that states AdaMult in ALM form, for readers more familiar with that framework.

Under the single-step, first-order regime we consider, the two forms would produce exactly the same primal iterates, by an argument analogous to the equivalence result of [5]. The choice between the two is thus a matter of presentation, and since the paper compares dual optimizers, we wrote AdaMult as one. In any case, this choice does not affect the invariance guarantee of Proposition 4, which, to our knowledge, all other ALM variants lack.

W1, W2. "Limited novelty": the multiplicative update and extrapolation. While none of AdaMult's ingredients is new on its own, as the paper credits (lines 95-107), our algorithm is new: it combines all of them and, by reversing the normalization at the output (step ④), obtains the invariance guarantee of Proposition 4. Our contribution is thus this algorithm, together with the theory that it is stable and robust (Propositions 2 and 4) and the evidence that it performs robustly in deep learning practice.

W2. AdaMult's extrapolation is "quite the same" in concept as the EMA of [2]. An EMA and extrapolation both change how the multiplier responds to the violation, but in opposite directions. An EMA averages the multiplier with its past values, so the multiplier responds to changes in the constraint violation more slowly. Extrapolation moves the multiplier ahead along the current violation, so it responds to them sooner. AdaMult extrapolates its multipliers but does not average them.

W4. The propositions "do not really provide insights into the empirical performance gains". While Lemma 1 is indeed fundamental and says that rescaling a constraint leaves the problem's solution unchanged, Proposition 4 goes further and says that it leaves AdaMult's primal iterates unchanged.

Most dual optimizers lack this property. When a constraint is multiplied by a positive constant, gradient ascent, RMSProp, and the ALM variants (PI control and RMS-ALM) follow a different path and may need retuning (Section 3.2, Proposition 5). With AdaMult, a configuration that works at one constraint scale is guaranteed to work at every scale. This provably removes one reason to retune a default across tasks.

Without normalization, AdaMult loses this invariance; without extrapolation, it loses the stability of Proposition 2. Both losses matter in practice. Without normalization, no single hyperparameter configuration is feasible on every task (Table 2 and Table 12 of Appendix C.2); without extrapolation, no feasible solution can be found on the Sparsity task, even when tuning specifically for it.

In any case, to improve the flow, we will move Lemma 1 next to Proposition 4, as context for the proposition.


We hope this addresses the reviewer's concerns, and we are happy to clarify further.

References

  • [1] Birgin et al. Numerical Comparison of Augmented Lagrangian Algorithms for Nonconvex Problems. Computational Optimization and Applications, 2005.
  • [2] Rony et al. Augmented Lagrangian Adversarial Attacks. ICCV, 2021.
  • [3] Basir and Senocak. An Adaptive Augmented Lagrangian Method for Training Physics and Equality Constrained Artificial Neural Networks. arXiv:2306.04904, 2023.
  • [4] Kort and Bertsekas. Combined Primal-Dual and Penalty Methods for Convex Programming. SIAM Journal on Control and Optimization, 1976.
  • [5] Ramirez and Lacoste-Julien. Dual Optimistic Ascent (PI Control) is the Augmented Lagrangian Method in Disguise. ICLR, 2026.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to all Reviewers

We thank the reviewers for their feedback.

New experiments: 808 synthetic problems with known solutions. To address the concerns of Reviewers x7wa (W4, Q4) and fnex (W2, W4) about evidence beyond our six machine learning tasks, we ran AdaMult and the baselines on 808 synthetic problems.

At its default, AdaMult solves 45% of the problems, against 32% for the best baseline at its best single configuration. On 1,000 held-out problems, with every configuration left unchanged, it solves 44%, against 33%. This one default also solves more problems than GA, Adam, RMSProp, RMS-ALM or ALMA do when each is tuned separately for every problem; only PI control tuned per problem solves more (50%).

MethodConfigurationSolved (tuned across problems)Diverged (tuned across problems)Solved on held-out problems (tuned across problems)Solved (tuned per problem)
GABest of 2525%26%25%39%
PI controlBest of 12528%28%29%50%
AdamBest of 2522%9%21%28%
RMSPropBest of 2526%7%24%40%
RMS-ALMBest of 2521%13%20%33%
ALMA†Best of 14032%27%33%42%
AdaMultDefault*45%8%44%68%

*AdaMult's default was selected on this family of problems, using a grid of 125 configurations, the same size as the one considered for PI control.

†ALMA is defined for inequality constraints only; for equalities, we give it each constraint as two inequalities.

Standard test collections such as CUTEst [1] are fixed sets of problems and do not offer the systematic variation that Reviewer x7wa asks for. We therefore generated our own problems by a classical construction [2] that fixes the solution and its multipliers first, so each problem has a known solution that we compare the final iterate of every run against. They are quadratic programs and programs with an exponential objective and exponential constraints, either all inequalities or all equalities. Across problems, we vary:

  • the size: 10 or 100 variables, with half as many constraints;
  • the conditioning of the objective and of the constraints, including near-duplicate constraints;
  • the spread of the constraint scales within a problem, up to four orders of magnitude;
  • the spread of the optimal multipliers within a problem, up to three orders of magnitude;
  • the noise on the observed constraint values;
  • the primal optimizer, Adam or gradient descent;
  • whether the Lagrangian is convex at the solution.

Every run lasts 10,000 iterations, with one primal step and one dual update per iteration. On a given problem, all methods share the same primal optimizer and step size, both set independently of the dual method. A run solves its problem when its last iterate is within 1% of the known solution and of satisfying every constraint; a noisy problem must be solved on all three of its seeds. A run diverges when its iterates or multipliers blow up.

As in Table 1 of the paper, each baseline is tuned on these same problems over a grid of its hyperparameters in two ways: to the single configuration that solves the most problems, and separately for each problem. To test generalization, the held-out column takes 1,000 further problems, generated in the same way as these 808, and runs every method at the configuration selected on the 808, unchanged.


Changes to the paper. We list here the main changes we will make to the PDF after the discussion period.

  • New experiments. The synthetic problems above, including an ablation of AdaMult's ingredients on them (Reviewer x7wa, W1).
  • New baseline. ALMA, one of the methods Reviewer cCZM cites (W3), on the synthetic problems above and on our six machine learning tasks.
  • Algorithm. Three of the five clips and floors in AdaMult will be removed, including the upper bound on the log-state (Reviewers x7wa, W3, and fnex, Q3).
  • Scope of the claims. Six summaries of the results will state their scope explicitly, "on the six tasks tested" (Reviewer fnex, W1 to W3).

References

  • [1] Gould et al. CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization. Computational Optimization and Applications, 2015.
  • [2] Rosen and Suzuki. Construction of nonlinear programming test problems. Communications of the ACM, 1965.
AuthorsOpenReview ↗

Additional Theoretical Result: AdaMult's Invariance to Objective and Constraint Scales

Since posting our responses, we have proven an additional result, which establishes AdaMult's invariance to rescalings of both the objective and the constraints. This extends our initial result in Proposition 4, which covers constraint rescalings only. This invariance allows one default hyperparameter configuration to transfer more easily, since it removes the need to retune when the objective, the constraints, or both are rescaled.

The crucial ingredient for this new invariance result is the multiplicative update. Each of AdaMult's three components thus supports one guarantee: extrapolation gives stability (Proposition 2), RMS normalization gives invariance to constraint rescaling (Proposition 4 and now this result), and the multiplicative update gives invariance to objective rescaling (this result).

Result. Consider a problem with inequality constraints only, and rescale the objective to f′=ϱff' = \varrho f and each constraint to gi′=αigig'_i = \alpha_i g_i, with ϱ,αi>0\varrho, \alpha_i > 0. From the same primal initialization, a primal-dual run with AdaMult as the dual optimizer produces the same primal iterates on both problems, θt′=θt\theta'_t = \theta_t, with rescaled multipliers λi,t′=(ϱ/αi),λi,t\lambda'_{i,t} = (\varrho/\alpha_i)\\,\lambda_{i,t}, provided that:

  1. the primal optimizer is deterministic and invariant to rescaling the gradients it receives: its iterates do not change when all gradients are multiplied by the same positive constant (e.g., Adam or RMSProp with ϵ=0\epsilon = 0, or sign descent);
  2. each multiplier is initialized at λi,0=c,∣∇f(θ0)∣/∣∇gi(θ0)∣\lambda_{i,0} = c\\,\\|\nabla f(\theta_0)\\| / \\|\nabla g_i(\theta_0)\\|, for any constant c>0c > 0;
  3. the initial gradients ∇f(θ0)\nabla f(\theta_0) and ∇gi(θ0)\nabla g_i(\theta_0) are nonzero, so that the previous initialization is well defined and positive;
  4. the initial constraint values gi(θ0)g_i(\theta_0) are nonzero (as in Proposition 4);
  5. ϵ=0\epsilon = 0, so that no absolute floor is applied to the RMS scale (as in Proposition 4);
  6. zmin⁡=−∞z_{\min} = -\infty, so that no lower bound is applied to the log-states.

For simplicity, the result is stated for inequality constraints. An analogous result holds for equality constraints under a corresponding initialization; we will also include it in the revised paper. Note that the initialization of (2) differs from the one used in the paper and in our experiments. It is also more expensive, since it requires computing primal gradients at initialization.

AuthorsOpenReview ↗

Proof of the Additional Theoretical Result

Proof. We show by induction on tt that, for all t≥0t \ge 0 and all ii,

θt′=θt,λi,t′=ϱαi,λi,t,zi,t′=zi,t+log⁡ϱ.\theta'_t = \theta_t, \qquad \lambda'_{i,t} = \frac{\varrho}{\alpha_i}\\,\lambda_{i,t}, \qquad z'_{i,t} = z_{i,t} + \log\varrho.

Base case (t=0t = 0). Both runs start from the same primal initialization, so θ0′=θ0\theta'_0 = \theta_0. By assumptions 2 and 3,

λi,0′=c,∣∇f′(θ0)∣∣∇gi′(θ0)∣=c,ϱ,∣∇f(θ0)∣αi,∣∇gi(θ0)∣=ϱαi,λi,0.\lambda'_{i,0} = c\\,\frac{\\|\nabla f'(\theta_0)\\|}{\\|\nabla g'_i(\theta_0)\\|} = c\\,\frac{\varrho\\,\\|\nabla f(\theta_0)\\|}{\alpha_i\\,\\|\nabla g_i(\theta_0)\\|} = \frac{\varrho}{\alpha_i}\\,\lambda_{i,0}.

The initial log-state is defined by λi,0=ezi,0/ai,0\lambda_{i,0} = e^{z_{i,0}} / a_{i,0}, where ai,0=∣gi(θ0)∣a_{i,0} = |g_i(\theta_0)| is the initial RMS scale of gig_i, nonzero by assumption 4. Since θ0′=θ0\theta'_0 = \theta_0,

ai,0′=∣gi′(θ0)∣=αi,∣gi(θ0)∣=αi,ai,0,a'_{i,0} = |g'_i(\theta_0)| = \alpha_i\\,|g_i(\theta_0)| = \alpha_i\\,a_{i,0},

and therefore

zi,0′=log⁡(λi,0′,ai,0′)=log⁡(ϱαi,λi,0⋅αi,ai,0)=log⁡(λi,0,ai,0)+log⁡ϱ=zi,0+log⁡ϱ.z'_{i,0} = \log\bigl(\lambda'_{i,0}\\,a'_{i,0}\bigr) = \log\Bigl(\frac{\varrho}{\alpha_i}\\,\lambda_{i,0} \cdot \alpha_i\\,a_{i,0}\Bigr) = \log\bigl(\lambda_{i,0}\\,a_{i,0}\bigr) + \log\varrho = z_{i,0} + \log\varrho.

Inductive step. Assume that the three relations hold up to step tt.

Dual step. At θt\theta_t, AdaMult computes

gˉi,t=gi(θt)ai,t,zi,t+1=zi,t+r,gˉi,t,λi,t+1=ezi,t+1+p,gˉi,tai,t,\bar g_{i,t} = \frac{g_i(\theta_t)}{a_{i,t}}, \qquad z_{i,t+1} = z_{i,t} + r\\,\bar g_{i,t}, \qquad \lambda_{i,t+1} = \frac{e^{z_{i,t+1} + p\\,\bar g_{i,t}}}{a_{i,t}},

where ai,ta_{i,t} is the running RMS scale of the constraint values observed so far, gi(θ0),…,gi(θt)g_i(\theta_0), \dots, g_i(\theta_t). By assumption 6, no lower bound is applied.

  • Scale. The primal iterates coincide, so the rescaled run has observed gi′(θs)=αi,gi(θs)g'_i(\theta_s) = \alpha_i\\,g_i(\theta_s) for all s≤ts \le t. Since ϵ=0\epsilon = 0 (assumption 5), therefore

    ai,t′=αi,ai,t.a'_{i,t} = \alpha_i\\,a_{i,t}.

  • Normalized constraint.

    gˉi,t′=gi′(θt)ai,t′=αi,gi(θt)αi,ai,t=gˉi,t.\bar g'_{i,t} = \frac{g'_i(\theta_t)}{a'_{i,t}} = \frac{\alpha_i\\,g_i(\theta_t)}{\alpha_i\\,a_{i,t}} = \bar g_{i,t}.

  • Log-state.

    zi,t+1′=zi,t′+r,gˉi,t′=(zi,t+log⁡ϱ)+r,gˉi,t=zi,t+1+log⁡ϱ.z'_{i,t+1} = z'_{i,t} + r\\,\bar g'_{i,t} = \bigl(z_{i,t} + \log\varrho\bigr) + r\\,\bar g_{i,t} = z_{i,t+1} + \log\varrho.

  • Multiplier.

    λi,t+1′=ezi,t+1′+p,gˉi,t′ai,t′=elog⁡ϱ,ezi,t+1+p,gˉi,tαi,ai,t=ϱαi,λi,t+1.\lambda'_{i,t+1} = \frac{e^{z'_{i,t+1} + p\\,\bar g'_{i,t}}}{a'_{i,t}} = \frac{e^{\log\varrho}\\,e^{z_{i,t+1} + p\\,\bar g_{i,t}}}{\alpha_i\\,a_{i,t}} = \frac{\varrho}{\alpha_i}\\,\lambda_{i,t+1}.

Primal step. The primal optimizer of the rescaled run thus receives the gradient

∇θL′(θt,λt+1′)=∇f′(θt)+∑iλi,t+1′,∇gi′(θt)=ϱ,∇f(θt)+∑iϱαi,λi,t+1,αi,∇gi(θt)=ϱ,∇θL(θt,λt+1),\nabla_\theta \mathcal{L}'(\theta_t, \lambda'_{t+1}) = \nabla f'(\theta_t) + \sum_i \lambda'_{i,t+1}\\,\nabla g'_i(\theta_t) = \varrho\\,\nabla f(\theta_t) + \sum_i \frac{\varrho}{\alpha_i}\\,\lambda_{i,t+1}\\,\alpha_i\\,\nabla g_i(\theta_t) = \varrho\\,\nabla_\theta \mathcal{L}(\theta_t, \lambda_{t+1}),

which is ϱ\varrho times the gradient of the original run, as at every previous step. By assumption 1, θt+1′=θt+1\theta'_{t+1} = \theta_{t+1}.

The three relations therefore hold at step t+1t + 1. ■\blacksquare

We will include this result in the revised paper.

AuthorsOpenReview ↗

Closing Remarks for the AC

Dear Area Chair,

Thank you for your time assessing our submission. All three reviews agree that developing robust constrained methods for deep learning is important to the community.

During the discussion period, we addressed all reviewer concerns, including these main ones:

  • Reviewer x7wa argued that our experiments neither test AdaMult's robustness systematically nor isolate its contribution from its known ingredients. To address this, we added a systematic evaluation on a suite of 808 synthetic problems with known solutions. It supports our claims on both points, as do the paper's six machine learning tasks.
    • Robustness. On the suite, AdaMult at a shared default solves more problems than any baseline held to one shared configuration (45% vs 32%). On the six tasks (Table 1), AdaMult at a shared default is feasible on every task and competitive in performance, unlike any baseline held to one shared configuration.
    • Ingredients. On the suite, removing any one of AdaMult's three ingredients makes it solve fewer problems (45% vs at most 35%). On the six tasks (Table 2), removing any one of them costs feasibility or performance. The advantage therefore comes from combining all three.
  • Reviewer cCZM questioned the novelty of AdaMult's ingredients, and argued that adaptive augmented Lagrangian methods (ALMs) such as ALMA already avoid per-task tuning.
    • Novelty. We agree that no ingredient is new on its own, as the paper credits, but together they form a new algorithm. As the Ingredients results under Reviewer x7wa show, removing any one of them costs feasibility or performance.
    • Adaptive ALMs. In the comparison the reviewer asked for, AdaMult at its shared default solves more of the suite's problems than ALMA does even when tuned for every problem. We also highlighted that AdaMult is provably robust in the sense that it is invariant to constraint rescaling; to our knowledge, neither ALMA nor any other ALM has such a guarantee.
  • Reviewer fnex found the paper hard to follow and its claims broader than its experiments support. We will fix the two presentation issues the review names and state the scope of our claims explicitly in the summaries that leave it implicit. The suite above also provides the small nonlinear problems that the review recommended.
  • Reviewers x7wa and fnex pointed out that AdaMult's upper clipping can prevent convergence. We agreed and have removed it in our updated algorithm. The clip was never reached on our six tasks, so their results are unchanged. The new synthetic experiments were run without it, and our stability analysis does not rely on it.
  • Reviewer x7wa also asked how robust AdaMult's single configuration is to changes in the objective's scale. Since posting our responses, we have proven that, under a specific initialization of the multipliers, AdaMult is invariant to rescalings of both the objective and the constraints (see our comment "Additional Theoretical Result").

None of the reviewers has engaged in the discussion or reacted to our responses, so their scores are still those of the initial reviews.

The six machine learning tasks, the new suite of synthetic problems, and our new invariance result all support our broader claim that AdaMult transfers better than the baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.