- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The main strength of this paper is that the backward pass is largely solver-independent. Unlike PiNet to unroll/differentiate solver iterates, ASPL doesn’t depend on specific solver like DR, Dystra, but just differentiate through the optimality condition of the projection geometry. This makes the method more general and easy to apply.
Critical strengths
The main strength of this paper is that the backward pass is largely solver-independent. Unlike PiNet to unroll/differentiate solver iterates, ASPL doesn’t depend on specific solver like DR, Dystra, but just differentiate through the optimality condition of the projection geometry. This makes the method more general and easy to apply.
Critical weaknesses
-
My main concern is about the contribution of this paper. Identifying the active set in the forward pass to reduce the backward pass cost is not a new idea in (differentiable) constrained optimization (like FFOLayer, dQP, and BPQP), and some of these methods are solver-agnostic as well. Furthermore, the constraint block decomposition idea comes from PiNet. The paper extends it to curved active manifolds and gives a reduced IFT formulation, but I am not fully convinced how large the conceptual gap is from existing active-set sensitivity methods, and the paper doesn’t present any experimental results comparing with existing methods. Although the paper compares with PiNet and JAXOpt, a comparison with dQP or BPQP on the QP task would help show the gain over existing active-set methods.
-
Another issue of this paper is the computation cost. Although Eq. (5) provides an analytical expression for the projection Jacobian, computing it still requires solving a reduced linear system. The efficiency depends heavily on the dimension , which may not be small when many constraints are active or the active manifold is curved. Also, if using CG, it depends on the conditional number as well. Could the author give more discussion on when the reduced system is actually cheaper than existing implicit differentiation methods, especially for more general constraints?
-
The method seems to rely strongly on correctly identifying a stable active manifold. The theory uses strict complementarity and local regularity assumptions. If the solution is close to a degenerate point, or the active set changes frequently during training, it is less clear how robust the reduced backward is in practice. Could the authors discuss how the implementation handles active-set changes or failed identification during training?
-
How to decompose the constraint for general constraints (like having both SOC and SD constraints) into C1 and C2 blocks is not introduced. The paper mentions that more constraint blocks can be converted to two blocks using a product-space construction, but there is no very concrete recipe for complicated cases, for example when affine, SOC, and semidefinite constraints all appear together. A practical example would make the generality claim more convincing.
-
Also Eq. 5 depends on t, but how to compute it when C2 is not convex?
Minor: This paper has submission format issue of missing the “Under review as conference paper at ICLR 2027” in the page header.
Questions for authors
See weaknesses.
6 comments
Response to Reviewer 5AUj - Part II
Revisions made to the paper.
Section 1 (lines 82–90), revised text:
Dropping inactive constraints from the backward pass is known for polyhedral or partly smooth problems with known support (Niculae et al., 2018; Bertrand et al., 2022), and dQP (Magoon et al., 2025) applies it to any QP solver; BPQP (Pan et al., 2024), FFOLayer (Zhao et al., 2026) and deep declarative networks (Gould et al., 2022) also restrict the backward pass to the active constraints (Appendix A1). For a polyhedral projection, ASPL returns the same gradient as these active-set KKT methods from a smaller per-sample matrix whose condition number is at most theirs (Proposition 2). New here are the block lower-triangular form of the implicit-differentiation system of a projection onto two simple sets, which also covers curved and prox-regular sets (dQP is QP-only), and active-set identification by the layer's own solver before it converges.
Appendix A4.1, Table 5 added (backward passes on the lifted QP set; backward time in ms at four active-set ratios):
| method | matrix factored (size at ratio 0.45) | cond | rel. err. | ms at 0.05 | ms at 0.25 | ms at 0.45 | ms at 0.75 |
|---|---|---|---|---|---|---|---|
| ASPL | $\vert \mathcal{A}\vert $ (900) | 11 | 2.5e-14 | 1.8 | 5.8 | 14 | 34 |
| Schur | $n_{\mathrm{eq}}+\vert \mathcal{A}\vert $ (1000) | 19 | 2.3e-14 | 2.2 | 6.5 | 15.7 | 38 |
| dQP | $d+n_{\mathrm{eq}}+\vert \mathcal{A}\vert $ (3000) | 95 | 2.3e-14 | 389 | 608 | 943 | 1520 |
| Dense IFT | 4000 | 16 | 2.5e-14 | 1790 | 1790 | 1790 | 1790 |
| BPQP | 3000 | 95 | 2.8e-6 | 1.3e5 | 2.1e5 | 2.8e5 | 5.7e5 |
Appendix A4.1, Table 9 added (total training time in seconds at three active-set ratios; medians over seeds):
| method | d=100 0.05 | d=100 0.25 | d=100 0.45 | d=200 0.05 | d=200 0.25 | d=200 0.45 |
|---|---|---|---|---|---|---|
| ASPL | 71 | 72 | 77 | 93 | 92 | 101 |
| ASPL-B | 66 | 63 | 66 | 140 | 138 | 170 |
| dQP | 68 | 81 | 98 | 207 | 287 | 384 |
| BPQP | 596 | 657 | 839 | 2,301 | 2,897 | 3,164 |
| FFOLayer | 654 | 650 | 757 | 1,579 | 1,558 | 1,598 |
Also changed: Table 2 (Appendix A1.1); Proposition 2 and its proof (Appendix A2.5); Figure 3 (Section 4.1).
Weakness 2 (when is the reduced system cheaper): Another issue of this paper is the computation cost. Although … than existing implicit differentiation methods, especially for more general constraints?
Response: We thank the reviewer for this question. ASPL reduces the cost of training and inference in three ways.
-
A smaller backward system. The reduced system is significantly smaller than the systems of full-space layers such as OptNet, cvxpylayers and Πnet. On polyhedra it is also smaller than those of active-set methods: dQP, for example, solves a system of size , while ASPL solves one of size (Table 2).
-
Our switching approach: staying on the active face. Each sample's forward pass stays on its stored active set and switches to the full solver only when the check fails (Algorithm 1). Algorithm 2 states this switch as a rule with convergence guarantees (Theorem 4). After the first epoch almost no sample needs the full solver (newly added Table 8), so training and inference are faster (Table 1 and newly added Table 9). This is often the larger saving.
-
A pooled warm start. One warm-start state, shared across the dataset, starts each forward pass near its solution, so solver steps per iteration suffice (Theorems 3 and 4).
When the reduced system is the smaller one depends on the constraint class:
(this answer continues in the next part)
Response to Reviewer 5AUj - Part III
On polyhedra, ASPL's system is smaller than dQP's however many constraints are active. dQP solves for all variables, the equality multipliers and the active multipliers, a system of size . ASPL eliminates the variables and the equality multipliers and solves only for the active multipliers. This gives the same gradient with a well-conditioned matrix, whose condition number is never larger than that of dQP's matrix (newly added Proposition 2). The elimination needs one factorization of the equality block. This block does not depend on the active set, so it is not refactored when our switching approach adds or removes active constraints. Only the small matrix depends on the active set: ASPL factors it once for a reference active set per batch, handles each sample's differences from that set (a few added or removed constraints) by a low-rank update instead of a new factorization, and reuses the factor of the accepted face in the backward pass (Appendix A4.1). This significantly reduces the computation when the active set changes from step to step (newly added Table 7). On the QP benchmark, ASPL's backward pass is – faster than dQP's, even when three quarters of the inequalities are active (newly added Table 5).
On curved sets, its size depends on the cones. On second-order cones it is the smaller system as soon as the variables outside the cones outnumber the curved directions, as in large layers, where ASPL's backward pass is much faster than KKT differentiation on the active cones (newly added Table 12). With few such variables, as in the SOCP benchmark of the paper, the two are comparable. On semidefinite cones its size is set by the rank of the solution and falls as the rank grows (newly added Table 13).
CG is an option, not our default. Cholesky, our default, does not depend on the condition number. CG is an option for large reduced systems on flat faces. Its iteration count depends on the condition number of the reduced matrix, and on flat faces it is set by the angle between the two active manifolds. In the QP training runs this condition number stayed between and , and CG matched Cholesky (Appendix A4.1).
Revisions made to the paper.
Appendix A2.5, after Proposition 2, revised text:
The gradient is therefore the same, and only the matrix factored per sample differs. On the QP benchmark, ASPL's backward pass is the fastest at equal accuracy (Table 5). dQP covers QP layers with any positive definite Hessian and returns derivatives with respect to all QP data, while ASPL is a projection layer, so neither contains the other. On curved sets, the formulas of declarative networks (Gould et al., 2022), BPQP (Pan et al., 2024) and Vaiter et al. (2017) solve a system of size about , one row per variable and per active constraint, while Theorem 1 factors directions; on second-order cones of dimension , is the smaller exactly when the variables outside the cones exceed per boundary cone (Appendix A4.2).
Appendix A4.2, Table 12 added (reduced backward against KKT differentiation on the active cones, time per sample in ms, by the number of variables outside the cones):
| cones × dim | n | boundary cones | ASPL rows | KKT rows | ASPL | active-cone KKT | dense adjoint | full KKT |
|---|---|---|---|---|---|---|---|---|
| 50×10 | 25 | 15 | 135 | 40 | 0.017 | 0.012 | 0.15 | 0.79 |
| 50×10 | 2000 | 15 | 135 | 2015 | 0.024 | 4.7 | 0.15 | 22.8 |
| 50×10 | 4000 | 15 | 135 | 4015 | 0.087 | 20.7 | 0.55 | 28.1 |
| 200×4 | 2000 | 60 | 180 | 2060 | 0.035 | 5.1 | 0.41 | 38.6 |
| 200×10 | 2000 | 60 | 540 | 2060 | 0.095 | 5.2 | 4.2 | 37.8 |
| 200×10 | 4000 | 108 | 972 | 4108 | 0.50 | 23.7 | 14.5 | 65.0 |
| 800×4 | 4000 | 240 | 720 | 4240 | 0.47 | 23.5 | 14.9 | 106 |
| 800×10 | 2000 | 240 | 2160 | 2240 | 1.7 | 11.0 | 66.7 | 346 |
Section 4.1 (lines 362–420), revised text:
A CG solve of the reduced system is as accurate as Cholesky and slower (ASPL-B (CG) in Table 1), so ASPL uses Cholesky.
Appendix A4.1 (robustness of identification), revised text:
Conditioning. In the runs of Table 1 the reduced system has condition number at ratio and at ratio , and warm-started CG solves it to relative tolerance in to iterations on average.
(continued in Part IV)
Response to Reviewer 5AUj - Part I
We thank the reviewer for the concrete questions. They sit inside the weakness paragraphs, so we answer each weakness paragraph in turn. Blue text in the revised PDF marks every change. Under each answer we quote the text that we added and show the added tables; figures are cited by their number.
Response: We thank the reviewer for this assessment. ASPL differs from these methods in three ways, and we added the comparison the reviewer asks for.
First, we show that, independent of the overall optimization method (active-set or not), the backward system of implicit projection layers that split the feasible set into two or more sets is block lower-triangular (Theorem 1). In the triangular form the free directions form a scalar block that needs no solve, and only the dN normal and curved directions need one small solve.
Against implicit layers: Πnet runs truncated BiCGSTAB on the full system in Rd; we show that its system is exactly this triangular one, and ASPL solves it exactly in the reduced dimension, a significantly smaller system when dN is smaller than d (Table 2).
Against active-set and KKT-sensitivity methods: dQP and BPQP keep every primal variable and equality multiplier, so they solve a system of size d+neq+∣J∣; on polyhedra ASPL factors ∣J∣×∣J∣ and returns the same gradient as dQP (newly added Proposition 2). FFOLayer also uses only the active constraints, but it assumes that the active set is correct and returns an approximate gradient. For curved sets, dQP does not apply, and BPQP needs the Lagrangian Hessian with the linear independence constraint qualification (LICQ), which fails at the SOC apex and at low-rank PSD points; ASPL works with any forward solver and uses only the block projections.
Second, unlike existing active-set extensions, which need a nearly accurate active set to build their KKT systems, ASPL switches adaptively between the ambient space and the active manifold, with theoretical guarantees (Theorems 2–4). Methods such as dQP, BPQP, DDN and FFOLayer obtain the active set by thresholding the solver output or assume it is correct, and guarantee neither that it is correct nor that training converges for (PCO) (Table 2). ASPL's training (Algorithm 2, Appendix A2.6) introduces a new switching rule: the forward pass stays on the active manifold while the share of the projection displacement in the candidate normal cone stays above a threshold, and returns to the ambient space when it drops below. This rule is what makes active-set identification adaptive during training while keeping its guarantees. To the best of our knowledge, this is the first theoretical guarantee for stochastic active-set optimization in (PCO).
Third, we added the experimental comparison the reviewer asks for. With the same forward pass, active set and code, ASPL and dQP return the same gradient, and ASPL's backward pass is more than an order of magnitude faster (newly added Table 5). We also trained dQP, BPQP and FFOLayer end to end with their released code on the QP task, at sizes that these baselines can run (newly added Table 9). ASPL is faster than BPQP and FFOLayer at every size and faster than dQP from d=200 on, and its advantage grows with the problem size.
(this answer continues in the next part)