acceptodds
Under review as a conference paper at ICLR 2027

SphericalDEQ: An Iteration-Free Deep-Equilibrium Layer

Abstract

Deep equilibrium models (DEQs) define neural layers implicitly as fixed points, enabling infinite-depth computation with constant activation memory. Their practical use, however, is hindered by the need to solve a nonlinear fixed-point problem at every forward pass and an implicit system during backpropagation. We introduce SphericalDEQ, an equilibrium layer that eliminates these iterative computations. Its key ingredient is a homogeneous spherical normalization that reduces the nonlinear fixed-point problem to a single linear solve followed by normalization. We establish conditions for the existence and uniqueness of this equilibrium and derive closed-form parameter derivatives, enabling exact backpropagation without iterative implicit differentiation, series approximations, or phantom gradients. This yields an equilibrium layer whose inference cost is comparable to that of an explicit layer once the inverse operator is cached. Experiments across image classification and neural PDE simulation show that SphericalDEQ substantially reduces the computational and memory overhead of conventional DEQs while matching or improving their predictive performance and, scales to ImageNet-1K classification and large-scale autoregressive physical simulation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
DEQ, deep equilibrium

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NeCaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel deep equilibrium model layer, SphericalDEQ, which does not require iterative solution of nonlinear fixed points but instead directly obtains equilibrium points via linear solution and row normalization.

Critical strengths

1.This paper bypasses the iterative solving method. 2.Theoretical analysis is comprehensive.

Critical weaknesses

The core theorem of SphericalDEQ relies on a key assumption: 1 cannot be the eigenvalue of W. The paper did not discuss how the condition number of the linear solution changes as the eigenvalue of W approaches 1, nor did it provide a numerical processing scheme for this pathological situation.

Questions for authors

1.Can SphericalDEQ be extended to sequence modeling tasks?

1 comment

We thank the reviewer for the comments. We respond in order:

Weaknessess

Thanks for raising this point. The assumption that 1 is not in the spectrum of W ensures uniqueness of the linear solve defining the layer, but indeed does not by itself guarantee good conditioning. When 1 is an eigenvalue of W, the linear system can have no solution or infinitely many solutions. In the latter case, one would need a procedure to select which one to use, which can be done in different ways. One option is the minimum Euclidean norm solution given by the Moore–Penrose pseudoinverse, provided its row normalizations are nonzero. We added this observation in Appendix B.3.

As the reviewer points out, invertibility alone does not exclude sensitivity to perturbations. The relevant quantity is the 2-norm condition number of the linear system is κ(I−W)=∥I−W∥∥(I−W)−1∥=σmax⁡(I−W)σmin⁡(I−W)\kappa(I-W) = \| I-W\| \|(I-W)^{-1} \| = \frac{\sigma_{\max}(I-W)}{\sigma_{\min}(I-W)}, where σmax⁡,σmin⁡\sigma_{\max}, \sigma_{\min} indicate the maximum and minimum singular values, which can be arbitrarily large even when I−WI-W is invertible. In particular, it diverges when the smallest singular value approaches zero while the largest stays bounded away from zero. While we never observed a blowup of (I−W)−1(I-W)^{-1}, in particularly ill-conditioned situations, it is possible to cache the inverse of I−WI-W in truncated SVD format to stably solve the linear system. We added a comment on this option in Appendix B.3, paragraph "Nearly singular systems".

Questions

As discussed in section 4.4, SphericalDEQ can be used as an alternative to the residual MLP after the self-attention layer in a transformer mechanism, making it therefore a viable option also for sequence modeling tasks.

We thank again the reviewer for the comments. Please let us know if there is anything else we can clarify to improve your opinion.

Reviewer_9khqOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SphericalDEQ, an implicit equilibrium layer based on homogeneous spherical normalization. The key idea is to exploit the structure of the normalized equilibrium map to characterize its fixed point explicitly. In particular, Theorem 4.6 shows that, under a structured linear operator WW, the equilibrium can be obtained by solving a linear system and then applying a row-wise normalization, thereby eliminating nonlinear fixed-point iterations. The paper further derives closed-form parameter derivatives, enabling exact backpropagation without iterative implicit differentiation, Neumann approximations, or phantom gradients. The paper also establishes existence and uniqueness of the equilibrium under the condition 1∉Eigenvalues⁡(W)1 \notin \operatorname{Eigenvalues}(W), and evaluates the proposed method on image classification and neural PDE simulation, including ImageNet-1K and large-scale autoregressive physical simulation. The experiments report competitive predictive performance together with substantially reduced computational and memory costs compared with conventional DEQ baselines.

Critical strengths
  1. A clear and interesting theoretical characterization of the equilibrium. Theorem 4.6 is a strong contribution: the nonlinear equilibrium can be computed by a linear solve followed by normalization, so no nonlinear fixed-point iteration is required. This is conceptually clean and directly addresses one of the main computational bottlenecks of conventional DEQs. The corresponding closed-form gradients in Theorem 4.7 further make the forward and backward computations analytically tractable.

  2. The method is demonstrated at nontrivial scale. The experiments cover both image classification and neural PDE simulation, including ImageNet-1K and large-scale autoregressive physical simulation. The reported inference and memory improvements over conventional DEQs suggest that the proposed formulation is not merely a theoretical construction but can have practical computational benefits.

Critical weaknesses
  1. The conceptual distinction from an explicit linear-solve-plus-normalization layer is not sufficiently clear. Theorem 4.6 shows that the proposed equilibrium has the explicit representation \begin{align*} z^* = (I-W)^{-1}b \oslash \varphi \big((I-W)^{-1}b\big). \end{align*} Thus, once WW and bb are given, the proposed equilibrium can be evaluated directly using standard linear algebra followed by normalization. This raises an important question about what is gained by viewing the transformation specifically as a DEQ rather than simply as an explicit layer implementing the same computation. The paper would be substantially more convincing if it clarified what essential modeling or representational benefit the equilibrium formulation provides beyond the computational advantage of avoiding nonlinear fixed-point iterations.

  2. The existence/uniqueness result does not by itself provide a quantitative numerical stability guarantee. Theorem 4.5 only requires 1∉Eigenvalues⁡(W)1 \notin \operatorname{Eigenvalues}(W). However, I−WI-W can still be arbitrarily ill-conditioned when an eigenvalue of WW approaches 11. In that regime, both the forward solution (I−W)−1b(I-W)^{-1}b and the exact gradients in Theorem 4.7 can become highly sensitive to perturbations. Thus, almost-everywhere existence and uniqueness does not necessarily imply numerical stability during training. A quantitative analysis or empirical investigation of the conditioning of I−WI-W would help establish whether the claimed robustness extends beyond exact invertibility.

Questions for authors
  1. Role of α∗\alpha^*. Theorem 4.5 introduces an entrywise positive scaling α∗\alpha*, while the proof of Theorem 4.6 constructs it explicitly as \begin{align*} \alpha^* = \mathbf{1}\oslash \varphi(y), \qquad y=(I-W)^{-1}b. \end{align*} Thus, α∗\alpha* appears to be determined by WW and bb, rather than being an independently learned parameter. Could the authors clarify how α∗\alpha^* should be interpreted as part of the SphericalDEQ layer, and how it is handled during training and inference?

  2. DEQ formulation versus an explicit layer. Given the closed-form expression for z∗z^*, what specific modeling advantage does the DEQ formulation provide over an explicit layer consisting of the same linear solve and normalization? In particular, is there an aspect of the infinite-depth/equilibrium interpretation that leads to a representational or optimization benefit that would not be obtained from the corresponding explicit transformation?

  3. Generality of the practical parameterization. The theoretical results allow a general Kronecker-rank-rr operator, whereas the implementations typically use the much more restrictive rank-one form \begin{align*} W=A^\top\otimes I_n. \end{align*} How does this restriction affect the expressivity of SphericalDEQ compared with conventional DEQs? Are the reported empirical benefits dependent critically on this rank-one structure?

  4. Conditioning and training stability. Since the forward and backward computations involve (I−W)−1(I-W)^{-1} and (I−W)−⊤(I-W)^{-\top}, have the authors monitored the condition number or smallest singular value of I−WI-W during training? In particular, can the authors provide evidence that the learned models do not approach nearly singular I−WI-W, where both the equilibrium and its gradients could become numerically unstable?

4 comments

We thank the reviewer for the comments. We respond in order:

Weaknesses

  1. Thank you for raising this interesting point. We agree that the role of the equilibrium formulation was not clear enough in our original submission and have made several changes to clarify it (please also see responses to S4u9).

    At the same time, we do not see the availability of an explicit representation as a weakness of the equilibrium formulation. An equilibrium interpretation and an explicit input–output representation are compatible. For example, under standard nondegeneracy assumptions, the implicit function theorem guarantees a locally defined solution function for any general equilibrium layer; however, that does not necessarily provide an explicit formula or a method for evaluating it without solving the implicit problem. Our results characterize homogeneous normalizations and structured operators for which the selected equilibrium and its derivatives can instead be computed directly by linear algebra.

    We also note that, although an unrestricted (invertible) linear layer and the resulting B=(I−A)−1B=(I-A)^{-1} describe the same function class, learning AA changes the training dynamics through this nonlinear parameterization. For a differentiable loss depending on AA through BB, with the other model parameters held fixed, the chain rule gives ∇AL=B⊤(∇BL)B⊤.\nabla_A L=B^\top(\nabla_B L)B^\top. Thus, training in the equilibrium coordinates follows different optimization dynamics from learning BB directly. To investigate this effect, we added the direct-BB baseline in the revision (Appendix D). The inverse parameterization improves classification accuracy by 1.001.00 percentage point on SVHN and 3.153.15 on CIFAR-10, and achieves lower rollout error on most PDE benchmarks. We have also added a clarification about this distinction in Section 4.3.

  2. An exact linear stability analysis can be performed in our setting, but we want to remark that stability and ill-conditioning is present everywhere in equilibrium layers, and stability is typically hidden behind assumptions on the regularity of the fixed point map (uniformly across all parameters WW). Nevertheless, in our experiments we never observed instability issues caused by I−WI-W becoming singular during training.

Questions

  1. Thank you for raising this point. As highlighted by the reviewer, α∗\alpha^* is determined by the network parameters and input through α∗=1⊘φ(y)\alpha^\ast=\mathbf{1}\oslash\varphi(y), where y=(I−W)−1by=(I-W)^{-1}b and the normalization is entrywise positive. It is not an independently learned parameter. During both training and inference, its effect is already included in the normalized output z∗=y⊘φ(y)z^\ast=y\oslash\varphi(y), so no separate optimization or evaluation of α∗\alpha^\ast is needed.

    We have also clarified the scope of the uniqueness statement in Section 4.2. Theorems 4.5–4.6 characterize the unique pair (z∗,α∗)(z^\ast,\alpha^\ast) satisfying Wz∗+α∗⊙b=z∗Wz^\ast+\alpha^\ast\odot b=z^\ast and φ(z∗)=1\varphi(z^\ast)=\mathbf{1}. The original normalized map can have additional fixed points that do not satisfy this constrained linear equation. The construction therefore provides a uniquely specified equilibrium for the layer, with the same forward computation and parameter derivatives used in the experiments.

  2. We refer to the answer to the first point in "Weaknessess".

  3. We remark that WW is a linear operator mapping matrices to matrices, and the rank has to be intended in the sense of Kronecker. Therefore, the choice W=A⊤⊗IW = A^\top \otimes I only means that as a matrix linear operator, WW acts only on the right through AA, which in the case of transformers means pointwise on each token. In this sense, this choice is not restrictive as it is the typical kind of operation performed in classical transformers after the mixing layer.

  4. We have monitored this, and we never observed the problem becoming highly ill-conditioned. We believe this is also due to the initialization that we chose, which is in fact WW close to zero (and therefore (I−W)−1(I-W)^{-1} close to the identity).

We thank the reviewer again for the comments. Please let us know if there is anything else we can clarify to change your opinion.

Reviewer_9khqOpenReview ↗

Numerical Stability and Conditioning of the Proposed Parameterization

Thank you for the detailed responses and the substantial revisions. The clarification of the role of the equilibrium parameterization, the uniqueness statement, and the additional comparison with directly learned explicit weights have addressed most of my concerns.

I still have some concerns regarding the numerical stability of the proposed method. As discussed in Appendix B.3, even when I−WI-W is invertible, computing (I−W)−1b(I-W)^{-1}b can be highly sensitive to perturbations if I−WI-W is ill-conditioned. In particular, invertibility and numerical stability are distinct properties: the condition 1∉spec(W)1\notin\mathrm{spec}(W) guarantees the existence of the inverse, but does not ensure that the inverse can be computed accurately or that the resulting solution is insensitive to perturbations. Moreover, the conditioning of I−WI-W can affect not only the forward computation but also the backward pass. For example, since \begin{align*} y=(I-W)^{-1}b, \end{align*} the gradient with respect to bb is given by \begin{align*} \nabla_b L=(I-W)^{-\top}\nabla_y L. \end{align*} Thus, a large condition number of I−WI-W may amplify errors or perturbations in both the forward and backward computations, potentially affecting the stability of the training process itself. I believe this issue deserves particular attention, especially because numerical stability is one of the main motivations for the proposed approach. The authors mention that no instability caused by the singularity of I−WI-W was observed in their experiments. While this is encouraging, the absence of observed singularity alone does not establish that the linear systems remain well-conditioned throughout training. I would therefore appreciate additional quantitative evidence addressing the following points:

  1. Conditioning during training: Report the evolution of the condition number κ2(I−W)\kappa_2(I-W), or alternatively the minimum singular value σmin⁡(I−W)\sigma_{\min}(I-W), throughout training. This would help clarify whether the linear systems remain well-conditioned in practice, rather than merely invertible.
  2. Worst-case conditioning: Report the largest condition number (or smallest singular value) observed during training, possibly across different layers and training runs. Average behavior alone may not reveal occasional but potentially problematic near-singular cases.
  3. Sensitivity to initialization: Investigate whether the observed numerical stability depends strongly on the initialization of WW. In particular, since the authors attribute the favorable behavior partly to initializing WW close to zero, it would be useful to understand how robust the method is to alternative initializations.
  4. Sensitivity to optimization hyperparameters: Examine whether changes in the learning rate and weight decay affect the conditioning of I−WI-W during training. This would help determine whether the observed stability is an intrinsic property of the proposed parameterization or depends substantially on the particular optimization settings.

I believe that even a relatively small-scale diagnostic study addressing these points would strengthen the paper. In particular, since Section 2 illustrates how instability in conventional DEQs can lead to solver failure and performance degradation, a quantitative characterization of the numerical stability of SphericalDEQ would make the comparison more convincing. More importantly, it would help clarify whether the proposed parameterization itself provides a robust mechanism for numerical stability, rather than simply avoiding instability under the particular experimental settings considered in the paper.

AuthorsOpenReview ↗

Answer to questions 1-3

We thank the reviewer for this question; to address it directly, we tracked κ2(I−W)\kappa_2(I-W) in fp64 for every layer at every optimizer step across 44 runs on MNIST, SVHN, CIFAR-10 and the AM, GS and VI PDE tasks, and measured the actual fp32 error of outputs and gradients against an fp64 reference, as summarized in the four tables below, one per question.

Conditioning during training. Paper recipe; κ2(I−W)\kappa_2(I-W) computed in fp64, maximum over layers, at fixed points of training (ranges across 3 seeds).

Each cell reports κ2(I−W)\kappa_2(I-W) (fp64, worst layer) at the given percentage of total training iterations. For MNIST, CIFAR-10 and SVHN it is the min-max range over 3 seeds, while the PDE tasks (AM, GS, VI) use a single run. The last column gives the smallest singular value of I−WI-W (on the worst conditioned layer) at the start and at the end of training, showing that I−WI-W stays far from singular.

taskstart10%25%50%75%endσmin⁡\sigma_{\min} start →\rightarrow end
MNIST (3 seeds)5.75.7--6.76.75.45.4--6.86.84.34.3--6.26.24.34.3--6.06.04.44.4--6.16.14.44.4--6.16.10.270.27--0.320.32 →\rightarrow 0.320.32--0.440.44
CIFAR-10 (3 seeds)6.36.3--6.86.86.26.2--8.88.84.84.8--5.55.54.94.9--5.75.74.04.0--4.34.33.93.9--4.14.10.270.27--0.290.29 →\rightarrow 0.390.39--0.410.41
SVHN (3 seeds)5.75.7--6.76.76.06.0--580∗580^{*}8.68.6--14148.28.2--11118.08.0--8.88.88.18.1--8.98.90.270.27--0.320.32 →\rightarrow 0.230.23--0.240.24
AM6.46.46.36.37.37.37.77.78.38.38.58.50.290.29 →\rightarrow 0.220.22
GS6.66.66.56.57.17.11515212119190.280.28 →\rightarrow 0.0970.097
VI6.16.17.27.27.57.59.29.29.59.510100.300.30 →\rightarrow 0.180.18

∗^{*}At 10%, two SVHN seeds are still in the initial chance-level plateau, which also occurs in our explicit baselines without an inverse.

Worst-case conditioning. Worst value over every optimizer step, every layer and every seed; errors are fp32 vs an fp64 reference on real inputs and gradients.

tasklayers ×\times runssteps per runworst κ\kappa, whole trainingworst κ\kappa, second halfsmallest σmin⁡\sigma_{\min}worst error, output / ∇W\nabla Wworst fixed-point residual
MNIST8×38 \times 323,45023,45024247.67.60.0760.0762.5×10−72.5 \times 10^{-7} / 5.1×10−75.1 \times 10^{-7}1.1×10−61.1 \times 10^{-6}
CIFAR-1010×310 \times 358,80058,80012126.66.60.1540.1543.3×10−73.3 \times 10^{-7} / 5.1×10−75.1 \times 10^{-7}9.1×10−79.1 \times 10^{-7}
SVHN8×38 \times 322,92022,9203.7×1063.7 \times 10^6 (initial plateau only)12125.0×10−75.0 \times 10^{-7}3.7×10−73.7 \times 10^{-7} / 1.9×10−51.9 \times 10^{-5}1.2×10−41.2 \times 10^{-4}
AM2×12 \times 12,0802,0808.58.58.58.50.2200.2202.6×10−72.6 \times 10^{-7} / 4.4×10−74.4 \times 10^{-7}7.3×10−77.3 \times 10^{-7}
GS4×14 \times 13,7173,717232323230.0820.0825.5×10−75.5 \times 10^{-7} / 1.1×10−51.1 \times 10^{-5}6.1×10−76.1 \times 10^{-7}
VI2×12 \times 11,9001,900101010100.1760.1764.8×10−74.8 \times 10^{-7} / 6.8×10−76.8 \times 10^{-7}5.5×10−75.5 \times 10^{-7}

Sensitivity to initialization. κ2\kappa_2 at start →\rightarrow κ2\kappa_2 at end (worst κ\kappa during training). SVHN: AdamW, 40 epochs; CIFAR-10: SGD, 50 epochs; AM: AdamW, 5 epochs. One change at a time with respect to the reference.

initialization of WWSVHNCIFAR-10AMworst error, output / ∇W\nabla W
paper init6.7→8.16.7 \rightarrow 8.1 (3,3463,346)6.8→4.36.8 \rightarrow 4.3 (1212)6.4→7.76.4 \rightarrow 7.7 (7.77.7)3.2×10−73.2 \times 10^{-7} / 8.3×10−78.3 \times 10^{-7}
W=0W=01.0→6.81.0 \rightarrow 6.8 (498498)1.0→3.81.0 \rightarrow 3.8 (4.34.3)1.0→2.41.0 \rightarrow 2.4 (2.42.4)2.9×10−72.9 \times 10^{-7} / 9.1×10−79.1 \times 10^{-7}
Gaussian, ρ(W0)=0.9\rho(W_0)=0.930→2230 \rightarrow 22 (2.4×1052.4 \times 10^5)39→6.139 \rightarrow 6.1 (2.5×1062.5 \times 10^6)14→1314 \rightarrow 13 (2222)1.5×10−61.5 \times 10^{-6} / 1.2×10−51.2 \times 10^{-5}
Gaussian, ρ(W0)=2\rho(W_0)=2449→16449 \rightarrow 16 (7.7×1047.7 \times 10^4)1,035→4961,035 \rightarrow 496 (3.2×1063.2 \times 10^6)164→28164 \rightarrow 28 (164164)2.1×10−52.1 \times 10^{-5} / 2×10−42 \times 10^{-4}
near-singular, κ0=103\kappa_0=10^3103→9.510^3 \rightarrow 9.5 (3.7×1053.7 \times 10^5)103→73810^3 \rightarrow 738 (1.9×1061.9 \times 10^6)103→9010^3 \rightarrow 90 (2.4×1042.4 \times 10^4)1.1×10−51.1 \times 10^{-5} / 5.9×10−45.9 \times 10^{-4}
near-singular, κ0=106\kappa_0=10^6106→8.910^6 \rightarrow 8.9 (3.1×1063.1 \times 10^6)106→80410^6 \rightarrow 804 (6.8×1076.8 \times 10^7)106→13010^6 \rightarrow 130 (3.0×1063.0 \times 10^6)3.5×10−33.5 \times 10^{-3} / 5.8×10−35.8 \times 10^{-3}
AuthorsOpenReview ↗

Answer to question 4

Sensitivity to learning rate and weight decay. κ\kappa at end (worst κ\kappa during training); all runs start from the paper init (κ≈6.5\kappa \approx 6.5). Base values: SVHN lr 10−310^{-3}, wd 10−610^{-6}; CIFAR-10 lr 5×10−25 \times 10^{-2}, weight decay 10−410^{-4}; AM lr 5×10−45 \times 10^{-4}, weight decay 10−410^{-4}.

changeSVHNCIFAR-10AMworst error, output / ∇W\nabla W
reference8.18.1 (3,3463,346)4.34.3 (1212)7.77.7 (7.77.7)3.2×10−73.2 \times 10^{-7} / 8.3×10−78.3 \times 10^{-7}
lr ×0.1\times 0.17.37.3 (1616)6.66.6 (6.96.9)6.46.4 (6.76.7)3.2×10−73.2 \times 10^{-7} / 5.1×10−75.1 \times 10^{-7}
lr ×10\times 10474474 (1.1×1071.1 \times 10^7)5.75.7 (5.4×1055.4 \times 10^5)4.44.4 (488488)1.6×10−61.6 \times 10^{-6} / 3.4×10−43.4 \times 10^{-4}
weight decay =0=08.18.1 (3,3463,346)7.47.4 (1818)7.87.8 (7.87.8)3.2×10−73.2 \times 10^{-7} / 8.3×10−78.3 \times 10^{-7}
wd high (SVHN 10−210^{-2}, CIFAR-10 10−310^{-3}, AM 10−210^{-2})1212 (5.5×1055.5 \times 10^5)2.42.4 (1111)7.87.8 (7.87.8)4.5×10−74.5 \times 10^{-7} / 3.3×10−53.3 \times 10^{-5}

We hope that these results, showing that I−WI-W remains well-conditioned throughout training and that the solve is accurate to near fp32 round-off, satisfactorily address the reviewer's concerns.

Reviewer_S4u9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SphericalDEQ, an equilibrium layer whose nonlinearity is a row-wise homogeneous normalization (RMSNorm in all experiments). With a Kronecker-structured weight operator, the fixed point equation reduces to one linear solve followed by normalization. Experiments on MNIST, SVHN, CIFAR-10, ImageNet-1K and PDE datasets shows higher accuracy and much lower time and memory than DEQ and Neural ODE baselines.

Critical strengths
  • The closed-form forward and backward passes remove the fixed-point solver and the implicit differentiation, the two costs that limit DEQs in practice.
  • The evaluation is broad, from small image benchmarks to ImageNet-1K and autoregressive PDE rollouts, with noise-robustness and gradient ablations.
Critical weaknesses
  1. SphericalDEQ is an explicit layer, so the comparison with DEQs does not seem fair. All experiments use W=A⊤⊗InW = A^\top \otimes I_n (L235-239), and Algorithm 1 computes y = (x + b) @ inverse(I - W) followed by z = y / rms(y). This is z∗=RMSNorm((x+b)B)z^* = \mathrm{RMSNorm}((x+b)B) with B=(I−A)−1B = (I-A)^{-1}, and any invertible BB is reachable by setting A=I−B−1A = I - B^{-1}. So the layer is a linear layer followed by RMSNorm, with the weight written as an inverse. The restriction that makes the solve explicit also removes what makes the layer implicit.
    • The speed and memory gains in Tables 1-3 come from comparing an explicit network against iterative ones, not from a better way to compute equilibria.
    • The accuracy gains could come from the backbone rather than the formulation. A missing baseline (that should be added) is the same architecture with a directly parameterized linear layer plus RMSNorm.
  2. The design is not motivated.
    • The paper does not explain why a homogeneous normalization, or why the sum-of-Kronecker-product form of WW, beyond the fact that they make the solve explicit. Section 4 reads as a construction that comes out of nowhere.
Questions for authors
  1. If you are writing a paper with a lot of equations, please use numbered equation.
  2. There are many works in designing a constrained form of dynamics for deep implicit layers in the adjacent literature of Neural ODEs [1, 2], which should be discussed in the related work.
  3. I think that there are newer Neural ODEs work [1, 2], which merits comparison, at least for Section 5.2.
  4. Can you add the baseline from Weakness 1, the same architecture with SphericalDEQ replaced by a directly parameterized linear layer plus RMSNorm, to Tables 1-3?
  5. Under what conditions on AA is z⋆z^\star an attracting fixed point of Fα⋆F_{\alpha^\star}? If it is not, in what sense is the layer the limit of an infinite-depth network?

[1] Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò. On Second Order Behaviour in Augmented Neural ODEs. NeurIPS 2020. arXiv:2006.07220.

[2] Ho Huu Nghia Nguyen, Tan Nguyen, Huyen Vo, Stanley Osher, and Thieu Vo. Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method. NeurIPS 2022.

4 comments
AuthorsOpenReview ↗

Weaknesses

We thank the reviewer for the comments. We respond in order:

Weaknesses

  1. Thank you for raising this interesting point. We agree that this aspect was not clear enough in our original submission and have made several changes to attempt to clarify it. Below we summarize our points.

    We want to remark that our layer is implicit by definition: its output is defined as an equilibrium similar to the system in Eq. 2 of [1]. Our aim is to identify a class of nonlinear equilibrium layers whose selected equilibria and derivatives admit an explicit characterization, eliminating nonlinear fixed-point iterations and iterative implicit differentiation. The equivalence identified by the reviewer is a consequence of this construction. For the practical choice W=A⊤⊗InW=A^\top\otimes I_n and RMS normalization, the resulting transformation can indeed be implemented as a linear layer followed by RMSNorm, with effective weight B=(I−A)−1B=(I-A)^{-1}.

    At the same time, we do not see this as a weakness and, in principle, this is not different in spirit from any equilibrium model. An equilibrium interpretation and an explicit input–output representation are compatible. For example, under standard nondegeneracy assumptions, the implicit function theorem guarantees a locally defined solution function for a general equilibrium layer; the difference is that one does not necessarily have access to an explicit formula or a method for evaluating it without solving the implicit problem. Our results characterize homogeneous normalizations and structured operators for which the selected equilibrium and its derivatives can instead be computed directly by linear algebra. Further to your point, this also provides an equilibrium interpretation of familiar normalized linear transformations.

    We also note that, although unrestricted AA and invertible BB describe the same function class, learning AA changes the training dynamics through the nonlinear parameterization B=(I−A)−1B=(I-A)^{-1}. We added the requested direct-BB baseline, retaining the same surrounding architecture and normalization (Appendix D). The inverse parameterization improves classification accuracy by 1.001.00 percentage point on SVHN and 3.153.15 on CIFAR-10, and achieves lower rollout error on most PDE benchmarks. We have clarified this distinction in Section 4.3 and in our answer to question 4.

  2. Thanks for raising this point. Our choice of homogeneous normalization is motivated by the possibility of separating each row's magnitude from its direction. This allows us to solve the linear equation first and then determine the row scalings that place its solution on the normalization sphere. The resulting construction characterizes a uniquely specified equilibrium and makes its computation and differentiation tractable.

    We also notice that the Kronecker structure is not restrictive: a general linear operator on matrices can be written as a sum of Kronecker products. The structural assumption is the diagonal factors in Theorem 4.5. Their role is to make WW commute with row-wise scaling, so that the normalization can be imposed on the linear solution while preserving the equilibrium equation. This keeps the equilibrium dynamics independent across rows. We have added a brief clarification about this in Section 4.2 and Section 4.3.

[1] Bai at al., Deep Equilibrium Models, NeurIPS 2019.

Questions

  1. Thanks for this suggestion. We appreciate that equation numbers make it easier to refer to specific expressions during the review. Our choice was to number only equations referenced in the text, to keep the presentation uncluttered.

  2. As suggested by the reviewer, we included works [1,2] in the literature discussion.

  3. As suggested, we also included these families of Neural ODEs in the experiments in Section 5.2. The results show that second-order dynamics substantially improve over first-order Neural ODEs (e.g., ~90% vs. 60–73% on CIFAR-10) and are competitive on MNIST. However, with the same backbone, parameter budget and training recipe, they remain below S$^d$-DEQ on SVHN (−1.1 pts) and CIFAR-10 (−3.1 pts). They are also 33–50× slower at inference, due to the many function evaluations required by the ODE solver.

  4. Thank you for raising this point. We included the proposed baseline in the experimental evaluation (same architecture, data and optimization, with the pointwise weight BB learned directly). See Appendix D in the revised manuscript. As we can observe, the results indicate that the equilibrium parameterization improves accuracy on SVHN (+1.0 pts) and CIFAR-10 (+3.15 pts) and is comparable on MNIST. Experiments on ImageNet are still running. On the PDE benchmarks, it attains the lower rollout error in 11 of 12 dataset/horizon combinations. The only exception is the first 15% of the Gray–Scott rollout, where the explicit model is marginally better (0.4408 vs. 0.4475). From there on, our model is better at every horizon: on Gray–Scott its error stays roughly constant, while that of the explicit model grows to 0.4962 (12.5% higher over the full horizon).

  5. Thanks for raising this interesting point. For RMS normalization and the implemented operator W=A⊤⊗InW=A^\top\otimes I_n, the sufficient condition ∥A∥2<1/2\lVert A\rVert_2<1/2 ensures that every forward iterate of Fα∗\mathcal F_{\alpha^{\ast}} converges geometrically to z∗z^{\ast}. Invertibility alone does not ensure attraction of this particular map, but the same output can be obtained as the infinite-depth limit of a different weight-tied network.

    Specifically, for each row, write M=I−AM=I-A, y∗=bM−1y^\ast=bM^{-1}, and choose 0<η<2/∥M∥220<\eta<2/\lVert M\rVert_2^2. Consider the normalized residual iteration sk=⟨ykM,b⟩∥b∥22,uk+1=yk+η(skb−ykM)M⊤,yk+1=uk+1∥uk+1∥2.s_k=\frac{\langle y_kM,b\rangle}{\lVert b\rVert_2^2},\quad u_{k+1}=y_k+\eta(s_kb-y_kM)M^\top,\quad y_{k+1}=\frac{u_{k+1}}{\lVert u_{k+1}\rVert_2}. With the explicit initialization y0=bM⊤y_0=bM^\top, the hidden state converges geometrically to y∗/∥y∗∥2y^\ast/\lVert y^\ast\rVert_2. Applying the layer's row normalization to this state therefore converges to z∗z^\ast. More generally, from almost every initialization, the eventually defined readout yk/sky_k/s_k converges geometrically to y∗y^\ast and its row normalization converges to z∗z^\ast. Thus the layer has an infinite-depth realization without requiring attraction under the original map. We clarified this in Section 4.2 and added the proofs in Appendix B.5, "Attraction and infinite-depth interpretation".

We thank again the reviewer for the comments. Please let us know if there is anything else we can clarify to change your opinion.

Thank you for the response, and for adding the Neural ODEs comparisons.

My main concern is still that SphericalDEQ is an explicit layer.

  1. Being implicit by definition does not make the layer a non-trivial equilibrium model. Any explicit layer z=f(x)z = f(x) is trivially the fixed point of F(z,x)=f(x)F(z, x) = f(x). The question is what the equilibrium formulation adds, and you agree that the layer is Linear + RMSNorm with B=(I−A)−1B = (I - A)^{-1}.
    • So the speed and memory gains over DEQs still come from comparing an explicit layer against iterative ones.
  2. The new motivation for the design (commuting with row scaling) explains why the solve is explicit, not why this is a good model. What is left is the claim that the parameterization B=(I−A)−1B = (I - A)^{-1} trains better than learning BB directly. That could be interesting, but it is a paper about reparameterizing a linear layer, not about DEQs. I think the paper should be framed and evaluated that way, with Linear + RMSNorm as the main baseline.

I have increased my score by 1.

We thank the reviewer for the careful follow-up and for raising the score.

  1. We agree with the core observation (which we now state in Section 4.3): even though the broader contribution is the characterization of a family of normalized equilibrium layers whose selected equilibria and parameter derivatives can be computed directly, the specific choice W=A⊤⊗InW = A^\top \otimes I_n, implies that the SphericalDEQ layer at inference is a Linear + RMSNorm layer with B=(I−A)−1B = (I − A)^{-1}. We also agree that learning BB directly is an essential baseline for assessing the practical benefit of this parameterization (see next bullet).

    At the same time, our contribution provides significantly more than the identity F(z,x)=f(x)F(z,x)=f(x). In the example F(z,x)=f(x)F(z,x)=f(x), evaluating the update requires evaluating the full output f(x)f(x) and so does the corresponding "naive" infinite-depth formulation z←λz+(1−λ)f(x)z \gets \lambda z + (1-\lambda)f(x) with λ∈(0,1)\lambda \in (0,1). In Section 4.2 and Appendix B.5 we give a weight-tied normalized residual iteration that uses AA and bb without evaluating the closed-form output. This provides a constructive infinite-depth realization of the layer, while the closed form allows us to implement it efficiently with direct evaluation and differentiation.

  2. We already compare against the explicit layer on every benchmark, and we will move all these comparisons to the main paper. We agree that Linear + RMSNorm is the essential baseline for assessing the benefit of the parameterization. We have already evaluated this baseline in Appendix D on all six classification and PDE benchmarks (MNIST, SVHN, CIFAR-10, Active Matter, Gray–Scott and Viscoelastic Instability). With the same function class and the same inference cost (BB is cached), the equilibrium parameterization improves accuracy by 1.0 point on SVHN and 3.15 points on CIFAR-10, is comparable on MNIST (99.61% vs 99.64%), and lowers PDE rollout error in 11 of 12 settings, with 12.5% lower full-horizon error on Gray–Scott. The gain depends on differentiating exactly through the equilibrium: with the usual Jacobian-free DEQ approximation, CIFAR-10 accuracy drops from 93.18% to 89.48% (Table 4).

    These comparisons are currently in the Appendix; following the reviewer's suggestion, we will move them into the main paper's results.

    We also appreciate the suggested interpretation as a way to parameterize a normalized linear layer. We regard this as a useful complementary perspective and a choice of emphasis between two mathematically compatible descriptions (the proposed equilibrium framing and the explicit parameterization perspective). We will add a brief discussion of this complementary interpretation in Section 4.3, after the paragraph "Relation to an explicit layer".

We thank the reviewer again for the constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.