- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies covariance of temporally pooled features of the following SDE: where is defined to be the infinite-size limit of a complex Ginibre (Gaussian) matrix with entry varince , and is a Brownian motion. The covariance of interest is defined as follows: where is a chosen pooling window function.
For this covariance , the authors characterize the limiting one-point spectral density, i.e., the limits of for all (Theorem 1). The limit considered in this work is: first , followed by with and , where ("intermediate window" in paper's languge). This spectral distribution is further characterized via a product of the Marchenko-Pastur (MP) variable and a random variable determined by the temporal pooling function (Corollary 1).
The rest of the paper discusses the consequences of this limiting distribution, in particular the resulting "participation dimension" and the risk of some linear models, such as Ridge regression. The authors seem to emphasize that different pooling kernels behave differently even when their first two spectral moments are matched.
Critical strengths
1. New spectral result: The paper includes a result on one-point spectral density for the covariance in the "intermediate window" scaling regime (Theorem 1). To the best of my knowledge, this result is novel. I have not checked the proofs in detail, but they appear sound, and likely contain some technical novelty to take care of the pooling window effects (coefficients in Eq. (28)), while the noncrossing pairing counting part of the proof seems standard.
Critical weaknesses
1. Poor writing, lack of coherrent narrative: The paper's writing and structure were very difficult for me to follow. The introduction does not clearly explain the setting and the contributions, the notation and the terminology are often used before definining them (e.g., in the introduction it is not clear what are , ; is never defined).
The paper's overall message is hard to extract from the text. The discussion after the main spectral theorem is a collection of various facts related to the covariance spectrum, which do not seem to be connected by any coherrent narrative. The takeaway seems to be just "different pooling kernels behave differently".
2. No clear relation to RNNs (despite the title): After reading the paper, I do not see what question it answers about RNNs' behavior.
On one hand, it is unclear what real RNN model the studied SDE is intended to represent. The continous model matrix in the paper has the form with complex Gaussian . The discretized analogue with step would be . This recurrent matrix structure does not correspond to any linear RNN/SSM model known to me, and the authors do not explain this choice. In addition, the choice to take the infinite-width limit of is itself unclear.
The model of RNN inputs is also questionable. The discretized version of the studied SDE has i.i.d. Gaussian inputs with certain -dependent covariance. The paper does not discuss why this should be a reasonable model for sequential inputs encountered by RNNs.
On the other hand, the paper does not pose any clear question about RNN behavior. The main results are about the covariance spectrum in a very specific random linear system. They do not seem to concern any concrete aspects of RNNs.
Participation dimension (minor): Another setting issue is that the theoretical participation dimension is defined as a ratio of expectations rather than expectation of ratio (see Eq. (8)). This may not be an issue if the involved traces concentrate, but this doesn't seem to be shown/discussed. The experiments use the correct definition.
Questions for authors
-
Which RNN model(s) is the studied setting intended to represent?
-
Is there a justification to define participation ratio as in Eq. (8)?