acceptodds
Withdrawn from ICLR 2027

When Full-Space and Reduced-Space Methods Diverge in Equality-Constrained Optimization: A Conditioning Perspective

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Equality-constrained optimization is central to modern physical AI, where desired feasible solutions are typically computed via Newton-type methods due to their rapid convergence. However, when selecting between the two primary formulations, reduced-space vs. full-space, practitioners routinely rely on domain-specific heuristics rather than a principled framework. In this work, we show that the practical boundary between these two approaches is fundamentally governed by the spectral profile of the state constraint Jacobian. While both approaches succeed in well-conditioned settings, their numerical behaviors diverge under ill-conditioning. Our theoretical analysis shows that reduced-space methods excel when ill-conditioning is driven by excessively large singular values, whereas full-space methods prove superior when driven by vanishing singular values. Empirical evaluations across four diverse benchmarks validate our theoretical findings.

Primary area:
optimization
Keywords:
constrained optimization, condition number, Newton method

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_hwDVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies when to use reduced-space (RS) or full-space (FS) methods for equality-constrained optimization. It argues that the singular values of the state constraint Jacobian JxJ_x explain their different behavior: small singular values hurt RS, while very large ones hurt FS. The paper presents three theorems, a solver-selection rule, and four optimization examples.

Critical strengths
  1. The choice between RS and FS is a useful practical question. Separating the effects of very small and very large singular values is a clear way to study it.
  2. The experiments cover different problem types and report both solver outcomes and conditioning measurements.
Critical weaknesses
  1. Theorem 4.1 assumes that every singular vector has a nonzero projection onto every design direction. With two or more design variables, one can choose a nonzero direction whose projection is zero, so this assumption cannot hold as stated. The bound in Theorem 4.2 also appears false under its written assumptions. For example,
1&10&0\\\\ 10&1&0\\\\ 0&0&1 \end{bmatrix}, \qquad J_z=\begin{bmatrix} 1&0&0\\\\ 0&0.01&1 \end{bmatrix}.$$ This example satisfies the stated conditions, but the KKT matrix has a condition number of about 1105, while Equation 8 gives an upper bound of about 123. These issues weaken the paper’s main explanation of RS and FS behavior. 2. Multiplying an equality constraint by a constant changes its Jacobian’s singular values without changing the optimization problem. The paper does not explain how to set its absolute thresholds or show whether its recommendation remains reliable after ordinary constraint scaling.
Questions for authors
  1. How can the projection assumption hold for every design direction when there is more than one design variable? Can the RS conclusion be proved under an assumption that applies to your benchmarks?

  2. Does the counterexample above violate an assumption that is missing from the theorem? If so, please state it. Otherwise, how should the bound in Equation 8 be corrected?

  3. If the same constraints are multiplied by a constant or normalized, does the recommendation change? How should users choose τlowτ_low and τhighτ_high in practice?

Reviewer_G6ZvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies equality constrained optimization. The authors compare theoretically and practically two possible formulations: reduced-space and full-space. They propose the practical criteria (Algorithm 1) based on singular values of constraint function Jacobian w.r.t. state variables.

Critical strengths

The main question of the paper, when to prefer RS vs FS formulations, is relevant and practical.

Critical weaknesses
  1. Resulting decision rule is not scale invariant. As far as I understand, Algorithm 1 is an answer to the main question of the paper: when to use reduced-space vs full-space methods. However, rewriting the constraint g(x,u)=0→cg(x,u)=0g(x, u) = 0 \to c g(x, u) = 0 for c>0c > 0 changes the output of Algorithm 1. This scaling does not affect the condition number, but scales largest and smallest singular values, so the comparison with the thresholds in Algorithm 1 can flip. Thus, Algorithm 1 can return different recommendations for the same optimization problem.

  2. Issues with theoretical results.

  • The assumption ∣wiTJuv∣≥c∥Juv∥|w_i^T J_u v| \geq c \|J_u v\| for all ii and vv in Theorem 4.1 is too strong. It does not hold for n≥2n \geq 2. For all wiw_i there exists v≠0v \neq 0 s.t. vv orthogonal to JuTwiJ_u^T w_i. Since JuJ_u has full column rank (by Assumption 3), ∥Juv∥>0\|J_u v\| > 0 for this vv, and the assumption fails for every c>0c>0.
  • Moreover, even if the assumption holds, the bound in Theorem 4.1 implies that κ(Jx−1Juv)\legκ(Ju)/c\kappa (J_x^{-1} J_u v) \leg \kappa(J_u)/c which does not depend explicitly on spectrum of JxJ_x, This is opposite is what is discussed after the theorem.
  • Theorem 4.2 does not hold as stated. The proof uses the false inequality that JzvJ_z v is bounded below by σmin(Jz\sigma_min(J_z), even though JzJ_z has a nontrivial kernel.
  • The comparison between FS and RS compares different things: for FS, KKT Newton iterates are studied, while RS section never studies Newton steps. Instead it considers the total gradient, which can be used with any gradient optimization scheme.
  • Equations in lines 183 and 207 missing the minus sign
  1. Experiments:
  • It is unclear how the results support the proposed criterion. RS performs poorly on neural PDE with σmin⁡(Jx)≈10−2)–(10−3\sigma_{\min}(J_x)\approx 10^{-2})–(10^{-3}, but succeeds on the beam problem with σmin⁡(Jx)=3.2⋅10−3\sigma_{\min}(J_x)=3.2\cdot10^{-3}. This raises questions about the proposed universal criteria. Moreover, Algorithm 1 is never evaluated on benchmarks.
  • In structural design and thermal topology optimization, the objective functions are linear, so ∇2f=0\nabla^2 f = 0, which contradicts Assumption 3.
  • There is no controlled experiment studying how the conditioning of JxJ_x affects RS and FS methods. I suggest using the same optimization problem and varying σmin⁡(Jx)\sigma_{\min}(J_x) and σmax⁡(Jx)\sigma_{\max}(J_x) to directly validate the theoretical claims.
Questions for authors
  1. What do you mean exactly by "well-conditioned" in Assumption 3?
  2. Lines 150-152. Could you please provide a citation or a proof for the claim "a highly inaccurate search direction forces αk→0\alpha_k \to 0, leading to premature termination or numerical stagnation". Which line search strategy is meant here?
  3. Figure 1. Which algorithm and which optimization problem are shown, and what are the starting point and hyperparameters? Could you provide a condition number plot? Why objective is getting stuck while gradient is still nonzero? Why gradient oscillates? Why the third plot only shows 6k-7k interval? Doesn't the result of this plot contradicts the statement about αk→0\alpha_k \to 0?
  4. Line 229. what does "BB decouples state and design variables" mean, and why does well-conditioning of BB follow from it?
  5. Line 261. Could you explain why vanishing minimum singular value of JxJ_x causes RS Hessian to explode? I do not see where RS Hessian is studied in the paper.
Reviewer_4M8oOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper demonstrates theoretical understanding of the behavior of Newton-type method for solving equality-constrained optimization problem through machine learning. Specifically focusing on reduced-space (RS) and full-space (FS) methods, the paper explains the boundary on which the preference of one method to the other changes. The analysis is based on the condition of Jacobian ∂g/∂x\partial g / \partial x of constraint g(x,u)=0g(x, u) = 0 over the state variable xx; both approaches struggle when this Jacobian is ill-conditioned. The distinction between preference comes up with how the smallest and largest singular value are dispersed: if the Jacobian has extremely small nonzero singular value, FS dominates and RS struggles with converging, and vice versa. These arguments are supported by numerical experiments with benchmarks on neural PDE optimization (PDEBench), trajectory optimization, structural design optimization, and thermal topology optimization.

Critical strengths

This paper answers an important research question which encompasses a variety of applications in optimization. The distinction of two methods (RS and FS) and the treatment on when such distinction is made is clearly an interest of not only optimization community but also machine learning researchers. Adding on to the theoretical analysis which cleanly explains the boundary where the preference of algorithms change, the paper provides numerous experiments with various setups to back up the argument. Finally, the paper is clearly written and well organized, and its proposed selection criterion (Algorithm 1) gives simple but intuitive interpretation.

Critical weaknesses

Although the paper is clearly written, I believe that assumptions for the main result (Theorem 4.1) need more thorough treatment. For example, the assumption stated at L190-194 is not list in preliminary section, which explicitly presents a set of assumptions required for the analysis of this paper. The paper lacks justification and verification of this assumption over numerical experiment:

Assume further that v↦Juvv \mapsto J_u v is agnostic to Jx−1J_x^{-1}, so that JuvJ_u v has balanced projections across all right singular vectors of Jx−1J_x^{-1}.

The Jacobian of gg over uu, denoted as Ju=∂g/∂uJ_u = \partial g / \partial u, need not be satisfying the condition above, but it is indeed an important assumption to impose. The analysis drops the dependence of analysis on JuJ_u from Jx−1JuJ_x^{-1} J_u, which was the true quantity to be analyzed, using this assumption.

Furthermore, I feel like the cases treated in the paper do not cover every possible cases happening in practice. I detail those in the questions below.

Questions for authors

Q1 (Validity and Justification of the assumption on JuJ_u)

Assume further that v↦Juvv \mapsto J_u v is agnostic to Jx−1J_x^{-1}, so that JuvJ_u v has balanced projections across all right singular vectors of Jx−1J_x^{-1}.

This is an important assumption which enables the use of the spectrum of Jx−1J_x^{-1} as an analysis metric rather than Jx−1JuJ_x^{-1} J_u. Such argument is valid when both well-conditioned-ness and alignment of singular vectors of JuJ_u with regards to those of Jx−1J_x^{-1} holds true. Otherwise, this requires more complex and overall sophisticated analysis. Can you explain why this assumption is generic enough for this paper's setup and whether this assumption is true for the presented numerical experiments?

Q2 (Is JxJ_x a valid metric?) I wonder if JxJ_x on one state xx can be used as absolute metric/indicator of Algorithm 1. It is possible that the optimization landscape is unstable; JxJ_x can vary significantly across the landscape. Then, with respect to the choice of state xx, the preferred method among FS and RS may differ along the problem solving process.

Q3 (Algorithm 1) I wonder if Algorithm 1 is a conceptual algorithm which hasn't been implemented yet. Considering practical usefulness of Algorithm 1, I would state more on computational and complexity aspect of Algorithm 1. How tractable it is to evaluate the condition number of JxJ_x and JuJ_u? It will be helpful to state those at least for the numerical experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Withdrawal Message

We thank the reviewers for their comments. Upon carefully considering the reviews, we realized that the technical presentation of our paper may be overly demanding, to the extent that readers might need to rely on LLMs to fully understand some of the content. We provide several examples below to illustrate this point:

What do you mean exactly by "well-conditioned" in Assumption 3?

This is a fundamental concept in linear algebra. As stated in the Wikipedia entry on matrix conditioning: "A matrix with a low condition number is said to be well-conditioned."

There is no controlled experiment studying how the conditioning of affects RS and FS methods. I suggest using the same optimization problem and varying σmin⁡\sigma_{\min} and σmax⁡\sigma_{\max} to directly validate the theoretical claims.

This is precisely why we evaluate RS and FS with different numbers of optimization variables. We would like to clarify that, for a given optimization problem, the singular values of the equality-constraint Jacobian are intrinsic quantities and cannot be arbitrarily adjusted. Rather, they can only be influenced indirectly through changes to the problem formulation, such as varying the number of optimization variables.

We recognize that this distinction may not be immediately apparent, particularly when relying on LLM-based interpretations, which may fail to distinguish between intrinsic numerical properties of an optimization problem and independently controllable variables. This may explain the concern raised in this question.

The bound in Theorem 4.2 also appears false under its written assumptions.

Under Assumptions 1 and 3 (lines 121–128), the Hessian matrix BB is required to be positive definite. The matrix BBB in your example does not satisfy this essential assumption.

This distinction is critical. As defined in lines 857–858, μmin⁡\mu_{\min} is indeed min⁡∣v∣=1vTBv\min_{|v| = 1} v^T B v, which coincides with σmin(B)\sigma_{min}(B) when BB is positive-definite. However, in your example, we have μmin⁡=−9\mu_{\min} = -9 which is negative. Consequently, dividing both sides of the inequality by μmin⁡\mu_{\min} reverses the inequality direction. The resulting inequality becomes trivial: it merely states that the condition number of the KKT matrix is greater than some negative number, which is automatically true.

We understand that connecting a theorem to its underlying assumptions requires careful consideration of the preceding technical details. The counterexample in your question appears to overlook these assumptions, which may explain the issue.

Multiplying an equality constraint by a constant changes its Jacobian’s singular values without changing the optimization problem. The paper does not explain how to set its absolute thresholds or show whether its recommendation remains reliable after ordinary constraint scaling.

Resulting decision rule is not scale invariant.

We agree that scaling equality constraints can sometimes be beneficial for full-space methods, and that numerical solvers such as IPOPT and SNOPT provide built-in options for this purpose. However, constraint scaling does not offer the same benefit for reduced-space methods. In particular, the reduced-space Jacobian Jx−1JuJ_x^{-1} J_u remains unchanged when both JxJ_x and JuJ_u are multiplied by the same nonzero constant.

We would also like to reiterate the central message of this paper: to demonstrate the fundamentally different behaviors of reduced-space and full-space methods as the equality-constraint Jacobian JxJ_x becomes increasingly ill-conditioned. Algorithm 1 is intended primarily as a conceptual framework for guiding the selection between the two formulations, rather than as a directly implementable algorithm. This is also why we have placed it in the Appendix, rather than presenting it as a primary algorithmic contribution.

If the same constraints are multiplied by a constant or normalized, does the recommendation change? How should users choose τlow\tau_{low} and τhigh\tau_{high} in practice?

We recognize that capturing the overall scope and central message of a technically involved paper can be challenging, particularly when the assessment focuses disproportionately on individual details without considering their broader context. Nevertheless, we remain willing to clarify any aspects of the paper that the reviewer would like to understand further.

We recognize that further responses to the questions raised may not meaningfully change the outcome of the review. Therefore, we have decided to withdraw the paper, also in the interest of reducing the AC's workload. In retrospect, we may have overestimated the level of technical familiarity of the intended audience. We will instead submit the work to another venue where we hope its technical contributions can be evaluated more directly and independently, based on their merits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.