acceptodds
Under review as a conference paper at ICLR 2027

dOPT: Differentiating Conic Optimization via Geometric Reduction

Abstract

Optimization layers enable the incorporation of structured constraints and decision problems into learning systems. Training such systems requires differentiating through the embedded optimization problem, which can be challenging for general conic programs. We introduce , a solver-agnostic framework that, rather than differentiating the full conic formulation, reduces it at a computed primal-dual solution to an equality-constrained quadratic program that preserves the reference solution and its first-order sensitivity. The reduction captures the local first- and second-order conic geometry relevant to differentiation and remains well defined at singular configurations. Computing solution derivatives then requires a single symmetric linear solve, independently of the forward solver. We derive explicit reductions for convex NLPs, QPs, SOCPs, and SDPs. Numerical experiments validate the computed gradients and show favorable backward-pass scalability, with substantial speedups over existing differentiable conic optimization methods as problem size increases.

Primary area:
optimization
Keywords:
optimization, differentiable optimization, conic program

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bim3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a second-order approximation of convex conic problems to accelerate their differentiation in differentiable optimization frameworks.

There is use of vague language in the intro that could be improved. "the structure needed to solve...", what structure? An algorithm? A representation of the problem?

Several sections are similarly obfuscated, 3.2 uses vague terms and "sells" some mathematical concepts, e.g "this restricts attention to the perturbation direction", what attention? Of the reader? Restricting the feasible set? What is a "cone-spaced object"?

Critical strengths

Computationally, the proposed method appears to accelerate differentiable optimization layers of SDPs, thus showing the effectiveness of the implementation.

Critical weaknesses

The proposed method solves (sparse) linear systems, exactly like other methods from the literature. It is unclear why this would be faster than other methods. Replacing matrix inversions by linear system solves for backward differentiation is already mentioned in the literature. The experiments should provide a finer analysis of the amount of work performed by iteration (size of the linear systems, sparsity) compared to the baselines.

Questions for authors

Can the authors explain the difference in performance? If fundamentally, the alternative baselines are also solving sparse linear systems, what is the main effect making dopt faster

A minor thing: although the github repository respects anonymity, it is not a good practice to submit temporary repos on github using a temporary account, this is unsustainable for the platform. Use the ZIP file submission of openreview and publish a github repo along public versions of the paper.

1 comment

We thank the reviewer for the constructive feedback and suggestions for improving the presentation and computational analysis.

Computational performance. A main advantage of dOPT is that it reduces the size and changes the structure of the backward system. Using linear solves in place of matrix inversions is already standard practice in backward differentiation.

dOPT exploits the local cone geometry to retain only perturbation directions relevant to first-order sensitivity, incorporating cone curvature into the reduced Hessian. Inactive conic constraints are eliminated, while SDP constraints are reduced to conditions on the zero eigenspace. The resulting backward problem is a smaller, symmetric KKT system that can be solved directly.

The baselines use different backward formulations. FFOLayer constructs an active-set Lagrangian surrogate and approximates hypergradients through finite differences involving an additional perturbed optimization solve of the same problem class as the forward solve (e.g., SOCP or SDP). CVXPYLayers differentiates the homogeneous self-dual embedding of the full conic program, retaining both active and inactive components and auxiliary variables. This yields a larger, generally nonsymmetric system, for which DiffCP uses iterative LSQR by default, with cost depending on the number of operator evaluations and the convergence rate.

Two examples illustrate the difference in backward system dimension. In the synthetic SOCP benchmark (n=500n=500), dOPT's reduced KKT system has dimension 570 versus 42,263 for CVXPYLayers. In the switched linear system SDP task (50×5050\times50 matrix, 1,276 variables), dOPT's system has an average dimension of 2,050 over the first 10 training steps versus 5,103 for CVXPYLayers.

Runtime also depends on sparsity, conditioning, solver choice, and implementation details, so system dimension alone does not fully explain the observed speedups. We find it encouraging that substantial speedups were achieved with a prototype that has not been heavily optimized.

Presentation. We appreciate the reviewer identifying imprecise language in Sections 1 and 3.2 and will revise these passages. Specifically, "the structure needed to solve" refers to the full conic constraints of the forward problem; "restricts attention" means imposing the critical-cone restriction dy∗∈CK(y∗)dy^\ast\in\mathcal{C}_{K}(y^\ast) on admissible first-order perturbations; and "cone-space objects" refers to quantities y=G(z)∈Ky=G(z)\in K, as distinct from the decision variables zz.

Reviewer_q55eOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an efficient approach to differentiating through optimization problems. By incorporating critical-cone restrictions and curvature corrections, the proposed method constructs an equality-constrained quadratic program that preserves both the reference solution and its first-order sensitivity, thereby enabling differentiation via a symmetric linear system. The authors develop specialized formulations for convex NLPs, QPs, SOCPs, and SDPs, and demonstrate substantial backward-pass speedups on synthetic benchmarks.

Critical strengths
  1. The proposed geometric reduction establishes a clear connection between classical sensitivity analysis and differentiable optimization while explicitly accounting for the curvature of nonpolyhedral cones.
  2. The SOCP and SDP formulations accommodate challenging geometric configurations, including SOC apexes and PSD matrices with multiple zero eigenvalues, under the stated regularity assumptions.
  3. The experimental results demonstrate substantial speedups in backward-pass computation compared with CVXPYLayers and FFOLayer, along with meaningful reductions in training time on the switched-system learning benchmark.
Critical weaknesses
  1. The guarantees rely on the regularity assumptions in Section 3.3, yet gradient sensitivity to solver accuracy and near-degenerate conditions is not systematically evaluated. This is particularly relevant for the SDP formulation, where nullspace identification and pseudoinverses raise concerns about spectral separation and numerical rank thresholds.

  2. The envelope-theorem checks validate optimal-value gradients but not general solution sensitivities. For the non-value loss 1⊤z∗(θ)\mathbf{1}^{\top}z^*(\theta), Table E.1 reports a mean relative discrepancy of 0.16 between dOPT and CVXPYLayers across ten SOCP instances n=500, without establishing which is more accurate. Independent directional finite-difference checks would strengthen the validation.

  3. Targeted tests of curvature effects and singular configurations would strengthen the empirical evidence, including reporting potential singularity when H∗H^* is removed. Runtime comparisons also use different forward solvers (MOSEK vs. SCS), motivating benchmarks at matched forward accuracy. This is distinct from Appendix E's gradient comparisons, which use the same forward solution.

Questions for authors
  1. How are numerical rank and pseudoinverse thresholds selected, and how sensitive are gradients to these choices, forward-solver tolerances, and small positive SDP eigenvalues?

  2. Could the authors validate solution-map JVPs or randomly weighted VJPs using finite differences of accurately re-solved perturbed problems, including a step-size sweep for the (n=500) SOCP setting? How is the shared forward solution in Appendix E implemented?

  3. Could the authors provide a controlled curvature diagnostic, reporting any degeneracy when H∗H^* is removed, alongside separate numerical tests of SOC apexes and PSD solutions with prescribed nullspace dimensions under the stated assumptions?

3 comments

We thank the reviewer for the thoughtful feedback, particularly for highlighting the connection between classical sensitivity analysis and differentiable optimization as a strength of our work.

A1: Numerical rank identification and robustness. In practice, dOPT uses a threshold εactive\varepsilon_{\mathrm{active}} to distinguish zero from positive SDP eigenvalues when constructing the nullspace U0∗U_0^\ast and pseudoinverse Z∗†Z^{\ast\dagger}. We choose the forward-solver tolerance εsol\varepsilon_{\mathrm{sol}} smaller than εactive\varepsilon_{\mathrm{active}} to reliably identify numerical zero eigenvalues, provided sufficient spectral separation.

Our synthetic experiments use εactive=10−5\varepsilon_{\mathrm{active}}=10^{-5} and εsol=10−8\varepsilon_{\mathrm{sol}}=10^{-8}. To assess sensitivity, complementing the SCS-based forward-solver robustness comparison in our response to Reviewer b5VK (A1), we varied both tolerances on the 50×5050\times50 synthetic SDP benchmark with MOSEK as the forward solver, using gradients computed with the original settings as references. The mean relative differences are:

εsol\varepsilon_{\mathrm{sol}}εactive=10−6\varepsilon_{\mathrm{active}}=10^{-6}εactive=10−5\varepsilon_{\mathrm{active}}=10^{-5}εactive=10−4\varepsilon_{\mathrm{active}}=10^{-4}εactive=10−3\varepsilon_{\mathrm{active}}=10^{-3}
10−810^{-8}6.58×10−46.58\times10^{-4}000
10−710^{-7}1.28×10−21.28\times10^{-2}7.77×10−47.77\times10^{-4}6.44×10−46.44\times10^{-4}6.44×10−46.44\times10^{-4}
10−610^{-6}8.91×10−28.91\times10^{-2}1.02×10−21.02\times10^{-2}3.45×10−33.45\times10^{-3}3.45×10−33.45\times10^{-3}

For εactive≥100εsol\varepsilon_{\mathrm{active}}\geq100\varepsilon_{\mathrm{sol}}, relative differences remain below 0.35%. Even when the thresholds become comparable, discrepancies remain below 9%, without numerical instability. These results indicate robustness across the tested settings, although accurate rank identification remains dependent on spectral separation, particularly when positive eigenvalues approach zero.

A2: Independent gradient validation. We agree that envelope-theorem checks validate optimal-value gradients rather than general solution sensitivities. Following the reviewer's suggestions, we performed independent central finite-difference (FD) checks on the n=500n=500 SOCP benchmark to investigate the discrepancy reported in Table E.1. All forward and perturbed problems are solved with SCS, so dOPT and CVXPYLayers differentiate at the same forward solution and are compared against the same FD reference. We averaged relative errors over three random linear losses and three random parameter directions per instance, using step sizes h∈{10−2,10−3,10−4,10−5,10−6}h\in\lbrace10^{-2},10^{-3},10^{-4},10^{-5},10^{-6}\rbrace.

Across this sweep, the discrepancies between dOPT and the FD estimate are 6.71×10−86.71\times10^{-8}, 4.41×10−74.41\times10^{-7}, 4.65×10−64.65\times10^{-6}, 4.45×10−54.45\times10^{-5}, and 4.19×10−44.19\times10^{-4}, respectively. The increase at smaller step sizes is consistent with solver-noise amplification of order O(εsol/h)O(\varepsilon_{\mathrm{sol}}/h). By comparison, CVXPYLayers exhibits approximately 0.210.21 relative error throughout the sweep. These independent checks provide strong evidence for the accuracy of dOPT's gradients and indicate that the discrepancy in Table E.1 arises primarily from the CVXPYLayers gradients in this experiment. We will include these results in Appendix E.

Regarding the shared forward solution, Appendix E evaluates dOPT's backward pass using the solution produced by CVXPYLayers through SCS, eliminating differences due to independently computed forward solutions.

A3: Curvature, singularities, and solver comparisons. The curvature correction H∗H^\ast is essential for first-order equivalence in general. For linear-objective SOCPs and SDPs, removing H∗H^\ast yields Qθ0=0Q_{\theta_0}=0 in the reduced KKT system. When the number of independent equality constraints is smaller than the decision dimension, this system becomes singular. Thus, omitting curvature can cause structural degeneracy, not merely inaccurate gradients.

At an SOC apex, a scalar inequality representation is nonsmooth, while dropping the constraint breaks equivalence. We evaluated dOPT on the illustrative problem

\quad\text{s.t.}\quad \lVert x-u \rVert_2\leq t-v,\qquad a> \lVert b \rVert_2,$$ whose unique solution is $(x^\ast,t^\ast)=(u,v)$, at the SOC apex. dOPT reproduces the corresponding analytical solution sensitivities. Solutions with multiple zero eigenvalues occur throughout our SDP experiments. For example, $50\times50$ synthetic instances in Section 6.1 have solutions with more than 20 zero eigenvalues. Section 6 compares full pipelines, with dOPT and FFOLayer using MOSEK through their solver-agnostic formulations and CVXPYLayers using SCS. We report forward accuracy alongside runtime to make these differences transparent. Appendix E has a different goal of evaluating gradient accuracy, so both methods differentiate at the same SCS forward solution.

Thank you for your response. The additional empirical results alleviate most of my concerns.

However, i still think the numerical robustness is not fully addressed. The experiments demonstrate robustness on the tested instances rather than general numberical stability, particularly in nearly degenerate settings. Nevertheless, given the encouraging empirical evidence, I am willing to raise my score.

Reviewer_b5VKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an efficient differentiable optimization layer for general conic programs leveraging active set reduction. The solution mapping derivatives can be computed by solving a smaller linear system. Experiments demonstrate substantial speedups over existing differentiable conic optimization methods such as CVXPYLayers and FFOLayers.

Critical strengths
  1. The paper extends active set reduction from convex functional constraints to conic constraints, which is the main technical contribution.
  2. dOPT does not fail in high-dimensional problem, while the two baselines fail.
Critical weaknesses
  1. Even with the inset explaining the intuition, it is hard to understand Theorem 4.1 because C∗C^* and H∗H^* are defined abstractly, and it is quite different from polyhedral constraints. A proof sketch for Theorem 4.1 may help with the exposition.
Questions for authors
  1. Why does dOPT rely on high-quality solution? Is it more sensitive to error in the solver's solution than baselines? Is there any mathematical proof for this?
  2. Why didn't prior works (e.g. CVXPYLayers, FFOLayer) consider active set reduction for conic programs? Is there anything inherent in their methods that prevent them from doing the active set reduction?
  3. The canonicalization procedure of CVXPYLayers is slow. Does the speedup of dOPT over CVXPYLayers comes from removing a large amount of inactive constraints or not doing canonicalization?
4 comments

We thank the reviewer for the insightful feedback and for recognizing the extension of active-set reduction to conic constraints and the scalability of dOPT to high-dimensional problems.

W1: We will clarify the definitions of C∗C^\ast and H∗H^\ast in Theorem 4.1, add a brief proof sketch, and strengthen the connection to the explicit constructions in Theorem 5.1.

The general characterization is intentionally implicit, since explicit matrix constructions depend on the local cone geometry, and presenting them in Theorem 4.1 would obscure the central geometric reduction.

In particular, C∗C^\ast can be any full-row-rank matrix satisfying ker⁡(C∗)=CK(y∗)\ker(C^\ast)=\mathcal C_K(y^\ast), whose rows therefore form a basis for CK(y∗)⊥\mathcal C_K(y^\ast)^\perp. Appendix B establishes this characterization and derives a symmetric Hessian formula for H∗H^\ast. Under our regularity assumptions, this formula yields v⊤H∗v=sup⁡w∈TK2(y∗;v)⟨−μ∗,w⟩v^\top H^\ast v=\sup_{w\in\mathcal T_K^2(y^\ast;v)}\langle-\mu^\ast,w\rangle for all v∈CK(y∗)v\in\mathcal C_K(y^\ast), where TK2\mathcal T_K^2 denotes the second-order tangent set (Liu et al., 2019, Lemma 2.4). At a smooth point of ∂K\partial K with μ∗≠0\mu^\ast\ne0, the critical subspace is the tangent space, and v⊤H∗v=∥μ∗∥⟨v,Sy∗v⟩v^\top H^\ast v=\Vert\mu^\ast\Vert\langle v,S_{y^\ast}v\rangle for all v∈CK(y∗)v\in\mathcal C_K(y^\ast), where Sy∗=−Dn(y∗)S_{y^\ast}=-Dn(y^\ast) is the shape operator associated with the outward unit normal nn.

We will incorporate these geometric interpretations into the discussion following Theorem 4.1. For specific cones, they yield the explicit matrix formulas in Theorem 5.1, with the NLP construction in Section 5.1 and detailed SOCP and SDP derivations in Appendices C.1 and C.2. The non-apex SOC calculation in Appendix C.1 illustrates the smooth-boundary interpretation.

Q1. Why does dOPT rely on high-quality solutions? Is it more sensitive to errors in the solver's solution than the baselines? Is there a mathematical proof?

A1: Thank you for pointing this out. We agree that our reference to "reliance on high-quality solutions" in the conclusion could have been clearer, and we will revise it accordingly. Sensitivity to forward-solve errors is an inherent concern for all methods that differentiate through optimality conditions. For dOPT, our regularity assumptions support local stability once cone geometry is identified, while global error bounds and comparative guarantees remain beyond this work's scope. Our experiments, including an additional forward-tolerance ablation, provide strong empirical evidence of robustness. We elaborate below.

The impact of forward-solve errors depends on both the type of error, such as constraint violation or objective suboptimality, and the differentiation mechanism. CVXPYLayers differentiates a conic residual map involving projections evaluated at the computed solution. FFOLayer uses the computed solution to construct a linearized surrogate and estimates gradients through perturbed solves. dOPT uses the computed solution to identify the local cone geometry and construct the reduced KKT system.

Our theory establishes exact first-order equivalence at the optimal solution. In a local neighborhood of the solution, strict complementarity separates the active and inactive primal-dual cone geometry, while nondegeneracy and second-order sufficiency ensure nonsingularity of the reduced KKT system. Once the correct geometry is identified, small perturbations can be analyzed using standard linear-system perturbation bounds, with sensitivity governed by conditioning.

We do not establish global bounds relating forward-solution errors to gradient accuracy or comparative guarantees across methods. Results on inexact hypergradients (Bertrand et al., 2022) and conic error bounds (Ding and Udell, 2023) offer potential starting points for further analysis, including conditions for correctly identifying the active geometry. Developing such guarantees is beyond this work's scope, and a general comparison would require substantially different analyses of methods' backward formulations.

Our experiments throughout the paper, including the gradient analysis in Appendix E, provide empirical support for robustness. To examine sensitivity to forward-solve errors directly, we conducted an additional ablation study varying the forward solver's tolerance on synthetic SDP instances with n=50n=50. At each tolerance, all three methods use the same SCS primal-dual solution, with dOPT's default backward configuration held fixed. We report the mean relative gradient deviation from each method's own reference at sol=10−8\mathrm{sol}=10^{-8}.

Forward solver tolerancedOPTCVXPYLayersFFOLayer
10−810^{-8}000
10−710^{-7}6.19e-065.09e-051.35e-04
10−610^{-6}9.91e-051.25e-046.78e-04
10−510^{-5}4.16e-043.92e-044.25e-03
10−410^{-4}8.47e-038.01e-034.03e-02

Across the tested tolerances, dOPT exhibits stability comparable to CVXPYLayers and consistently lower deviation than FFOLayer. Even at 10−410^{-4}, its mean deviation remains below 1%, providing further evidence of competitive robustness to forward-solve errors on these instances.

Q2. Why didn't prior works (e.g., CVXPYLayers, FFOLayer) consider active-set reduction for conic programs? Is there anything inherent in their methods that prevents such a reduction?

A2: Active-constraint information has long been used in sensitivity analysis (Fiacco, 1983; Bonnans and Shapiro, 2000) and differentiable optimization, including BPQP (Pan et al., 2024) and dQP (Magoon et al., 2025). FFOLayer itself exploits active-constraint geometry for scalar constraints. However, extending active-set reduction to more general cones introduces additional challenges: linearization alone does not capture the curvature of nonpolyhedral cone boundaries, while singular configurations, such as SOC apexes and multiple zero SDP eigenvalues, further complicate the analysis. There is no inherent obstruction to incorporating such a reduction into other methods, including CVXPYLayers and FFOLayer, but the construction is non-canonical and method-dependent. Incorporating it requires a dedicated derivation and corresponding changes to the backward formulation. dOPT builds on classical sensitivity theory to systematically incorporate critical-cone restrictions and directional curvature into a solver-agnostic, first-order-equivalent reduced formulation, with explicit constructions for NLPs, QPs, SOCPs, and SDPs.

Q3. The canonicalization procedure of CVXPYLayers is slow. Does the speedup of dOPT over CVXPYLayers come from removing a large number of inactive constraints or avoiding canonicalization?

A3: CVXPYLayers' canonicalization time is excluded from our runtime measurements and therefore does not contribute to the reported speedups. We will emphasize this in the revision.

dOPT exploits local cone geometry to eliminate irrelevant constraints and perturbation directions, reducing the backward pass to a single symmetric linear solve. In contrast, CVXPYLayers differentiates the full conic program, retaining active and inactive components and auxiliary variables. This yields a larger, generally nonsymmetric system, solved iteratively by DiffCP using LSQR by default.

The observed speedups depend on several factors, including system size, conditioning, and implementation details, whose individual contributions we have not isolated. Nevertheless, we find it encouraging that substantial speedups were achieved with a prototype implementation, without extensive performance optimization.

We thank the author for clarification. While most of the derivation is inherited from classical sensitivity analysis literature, I believe this work has sufficient algorithmic contribution for acceptance due to its effectiveness compared to prior works and the applicability of its first-order reduction to general conic programs. I am willing to maintain my positive score.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank you for your valuable feedback. We have revised the paper accordingly (with changes highlighted in blue), and we are happy to discuss if you have any further questions or comments.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.