- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a novel deep equilibrium model layer, SphericalDEQ, which does not require iterative solution of nonlinear fixed points but instead directly obtains equilibrium points via linear solution and row normalization.
Critical strengths
1.This paper bypasses the iterative solving method. 2.Theoretical analysis is comprehensive.
Critical weaknesses
The core theorem of SphericalDEQ relies on a key assumption: 1 cannot be the eigenvalue of W. The paper did not discuss how the condition number of the linear solution changes as the eigenvalue of W approaches 1, nor did it provide a numerical processing scheme for this pathological situation.
Questions for authors
1.Can SphericalDEQ be extended to sequence modeling tasks?
We thank the reviewer for the comments. We respond in order:
Weaknessess
Thanks for raising this point. The assumption that 1 is not in the spectrum of W ensures uniqueness of the linear solve defining the layer, but indeed does not by itself guarantee good conditioning. When 1 is an eigenvalue of W, the linear system can have no solution or infinitely many solutions. In the latter case, one would need a procedure to select which one to use, which can be done in different ways. One option is the minimum Euclidean norm solution given by the Moore–Penrose pseudoinverse, provided its row normalizations are nonzero. We added this observation in Appendix B.3.
As the reviewer points out, invertibility alone does not exclude sensitivity to perturbations. The relevant quantity is the 2-norm condition number of the linear system is κ(I−W)=∥I−W∥∥(I−W)−1∥=σmin(I−W)σmax(I−W), where σmax,σmin indicate the maximum and minimum singular values, which can be arbitrarily large even when I−W is invertible. In particular, it diverges when the smallest singular value approaches zero while the largest stays bounded away from zero. While we never observed a blowup of (I−W)−1, in particularly ill-conditioned situations, it is possible to cache the inverse of I−W in truncated SVD format to stably solve the linear system. We added a comment on this option in Appendix B.3, paragraph "Nearly singular systems".
Questions
As discussed in section 4.4, SphericalDEQ can be used as an alternative to the residual MLP after the self-attention layer in a transformer mechanism, making it therefore a viable option also for sequence modeling tasks.
We thank again the reviewer for the comments. Please let us know if there is anything else we can clarify to improve your opinion.