acceptodds
Under review as a conference paper at ICLR 2027

Temporal Pooling Shapes Covariance Geometry and Readout Risk in Random Recurrent Networks

Abstract

Temporal pooling determines the covariance seen by a downstream learner, yet this covariance is often reduced to total variance or participation dimension. We study a stationary linear random recurrent system with stability gap . In the ordered limit where width tends to infinity first, and define an intermediate observation window. For a broad class of normalized pooling kernels, every fixed moment of the variance-carrying part of the spectrum converges, and its covariance-size-biased limit is . Here has the aspect-one Marchenko–Pastur law, while the independent filter-gain factor depends on the kernel. Pooling rules can therefore match trace and participation dimension while retaining different higher-order geometry. To test whether this distinction matters for a common prediction task, we study 56 independent real Ginibre networks at widths 128 and 256. For each network, a calibration based only on feature covariances matches the trace and second spectral moment of boxcar and causal-exponential pooling to numerical precision. Both pooled views then predict the same future recurrent state. At width 256 and , causal-exponential pooling reduced population normalized mean-squared error by 5.86–7.14 percentage points across three stability gaps. Aligning the kernels' mean amplitude-weighted sample ages reduced these differences to 1.71–2.43 points but did not eliminate them. Validation-selected finite-sample ridge differences were 1.93–3.91 points in the same direction, although two width-128 fits reached the edge of the penalty grid, making that comparison descriptive. A separate Gaussian-design analysis shows why no kernel ranks universally: higher-order geometry raises ridge bias but can lower variance enough to reverse the ordering under minimum-norm regression. These finite experiments measure task behavior at fixed parameters; they do not validate the ordered asymptotic law.

Primary area:
learning on time series and dynamical systems
Keywords:
random recurrent networks, temporal pooling, covariance spectrum, random matrix theory, ridge regression, representation geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZoacOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies covariance of temporally pooled features of the following SDE: dxt=−Axtdt+dBt,A=1−gc,dx_t = -Ax_tdt + dB_t, \quad A = 1 - gc, where cc is defined to be the infinite-size limit of a complex Ginibre (Gaussian) matrix C∈CN×NC\in \mathbb C^{N\times N} with entry varince 1/N1/N, and BtB_t is a Brownian motion. The covariance of interest is defined as follows: CT,h:=Cov(zT,h),zT,h:=∫T−1/2h(s/T)xsds,C_{T,h} := Cov(z_{T,h}), \quad z_{T,h} := \int T^{-1/2} h(s/T) x_s ds, where hh is a chosen pooling window function.

For this covariance CT,hC_{T,h}, the authors characterize the limiting one-point spectral density, i.e., the limits of N−1∑iλip(CT,h)N^{-1}\sum_i \lambda_i^p(C_{T,h}) for all pp (Theorem 1). The limit considered in this work is: first N→∞N\to\infty, followed by T→∞T\to\infty with Tϵ→∞T\sqrt{\epsilon} \to \infty and Tϵ→0T\epsilon\to 0, where ϵ:=1−g2\epsilon:=1-g^2 ("intermediate window" in paper's languge). This spectral distribution is further characterized via a product of the Marchenko-Pastur (MP) variable and a random variable determined by the temporal pooling function (Corollary 1).

The rest of the paper discusses the consequences of this limiting distribution, in particular the resulting "participation dimension" and the risk of some linear models, such as Ridge regression. The authors seem to emphasize that different pooling kernels behave differently even when their first two spectral moments are matched.

Critical strengths

1. New spectral result: The paper includes a result on one-point spectral density for the covariance CT,hC_{T,h} in the "intermediate window" scaling regime (Theorem 1). To the best of my knowledge, this result is novel. I have not checked the proofs in detail, but they appear sound, and likely contain some technical novelty to take care of the pooling window effects (coefficients bjkb_{jk} in Eq. (28)), while the noncrossing pairing counting part of the proof seems standard.

Critical weaknesses

1. Poor writing, lack of coherrent narrative: The paper's writing and structure were very difficult for me to follow. The introduction does not clearly explain the setting and the contributions, the notation and the terminology are often used before definining them (e.g., in the introduction it is not clear what are gg, TT; yy is never defined).

The paper's overall message is hard to extract from the text. The discussion after the main spectral theorem is a collection of various facts related to the covariance spectrum, which do not seem to be connected by any coherrent narrative. The takeaway seems to be just "different pooling kernels behave differently".

2. No clear relation to RNNs (despite the title): After reading the paper, I do not see what question it answers about RNNs' behavior.

On one hand, it is unclear what real RNN model the studied SDE is intended to represent. The continous model matrix in the paper has the form A=−I+gCA = -I + gC with complex Gaussian CC. The discretized analogue with step Δ\Delta would be Ad=eΔ(−I+gC)=e−ΔeΔgCA_d = e^{\Delta(-I + gC)} = e^{-\Delta}e^{\Delta gC}. This recurrent matrix structure does not correspond to any linear RNN/SSM model known to me, and the authors do not explain this choice. In addition, the choice to take the infinite-width limit of CC is itself unclear.

The model of RNN inputs is also questionable. The discretized version of the studied SDE has i.i.d. Gaussian inputs with certain AA-dependent covariance. The paper does not discuss why this should be a reasonable model for sequential inputs encountered by RNNs.

On the other hand, the paper does not pose any clear question about RNN behavior. The main results are about the covariance spectrum in a very specific random linear system. They do not seem to concern any concrete aspects of RNNs.

Participation dimension (minor): Another setting issue is that the theoretical participation dimension is defined as a ratio of expectations rather than expectation of ratio (see Eq. (8)). This may not be an issue if the involved traces concentrate, but this doesn't seem to be shown/discussed. The experiments use the correct definition.

Questions for authors
  1. Which RNN model(s) is the studied setting intended to represent?

  2. Is there a justification to define participation ratio as in Eq. (8)?

Reviewer_UDYPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how temporal pooling affects the covariance structure of representations obtained from stationary linear random recurrent networks. The authors consider recurrent matrices drawn from the Ginibre ensemble and analytically demonstrate that different temporal pooling kernels can produce representations with the same total variance and participation dimension but different higher-order covariance spectra.

The authors then investigate how these differences affect downstream learning, including a task in which a linear predictor uses the temporally pooled representation to predict the future recurrent state. They find differences in prediction performance between pooling kernels. The authors report that aligning the kernels' mean sample ages reduces the observed prediction advantage by 66–71%, which they interpret as evidence that temporal recency accounts for much of the original difference.

The overall conclusion is that participation dimension alone is insufficient to characterize a representation's usefulness for downstream learning, as task alignment and the choice of estimator also matter.

Critical strengths
  • The paper provides a mathematical characterization of how temporal pooling affects covariance spectra in random recurrent networks, including a general asymptotic result for higher-order spectral moments.

  • The authors investigate consequences for downstream learning and include controls to distinguish temporal-recency effects from covariance differences.

Critical weaknesses
  • I found the paper unnecessarily difficult to understand. The abstract introduces several technical concepts without sufficiently motivating the underlying problem, and the introduction does little to make the main question accessible to a broader ICLR audience. Fundamental concepts such as "covariance geometry" and the role of temporal pooling in downstream learning should be explained more intuitively before introducing the mathematical machinery. In its current form, it is difficult to appreciate the contributions without substantial effort.

  • I am not convinced that the problem being studied is sufficiently compelling for the ICLR audience. It is already clear that matching total variance and participation dimension does not determine the entire covariance spectrum. While the analytical characterization may be mathematically interesting, the authors do not sufficiently explain what new insight it provides for representation learning or why this particular problem deserves such an extensive theoretical treatment.

  • The primary experiments do not convincingly establish that higher-order covariance geometry is responsible for the observed prediction differences. Much of the advantage of exponential pooling disappears after accounting for temporal recency. This weakens the motivation for the proposed theoretical analysis from a downstream learning perspective.

Questions for authors
  • Can the authors provide a concrete example explaining why characterizing higher-order covariance moments is important for downstream learning? What practical question does this analysis answer that participation dimension alone cannot?

  • Why is the analysis specifically focused on networks near the stability boundary? Is the motivation the emergence of slow timescales, and if so, what role do these timescales play in the results?

  • Could the authors clearly define what they mean by covariance geometry early in the paper? How should a reader interpret the higher-order spectral moments, and what additional information do they provide?

  • Given that prediction performance also depends on feature–target alignment and the choice of estimator, how useful is the derived covariance spectral law for predicting performance on general downstream learning tasks?

Reviewer_oZF6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies temporal pooling in a linear Ginibre/circular recurrent model, taking width to infinity before a critical intermediate-window limit. Theorem 1 identifies the leading asymptotics of every fixed-order covariance moment. Corollary 1 gives a scalar product representation for the covariance-size-biased limiting distribution. The paper also derives a kernel-class tail classification, population readout curves, an ordered Gaussian sample-Gram law, and a finite-width stability constraint. Its finite-network studies examine fixed-parameter covariance geometry and a common future-state prediction task.

Critical strengths

A substantive central theorem. The frequency-resolved covariance matrices generally do not commute, so a general pooling law cannot be obtained by averaging their individual spectra. The signed-kernel localization and its connection to the pairing asymptotics appear technically substantial. General consequences beyond selected kernel comparisons. The product law separates a kernel-independent Marchenko–Pastur factor from a kernel-dependent gain distribution within the circular model. Corollaries 2, 3, and 5 apply to the admissible kernel class. Proposition 1 distinguishes spectral-tail classes, and Appendix B establishes a positive limiting Gram lower edge at every finite sampling ratio. These are meaningful structural results. Careful experimental scope. The manuscript separates fixed-parameter circular references, Gaussian-design calculations, and shared-target experiments. Matrixwise matching removes differences in the first two spectral moments, while the delay sensitivity exposes the role of temporal alignment. The reported checks in Appendix F provide quantitative comparisons with fixed-parameter references.

Critical weaknesses

The significance judgment rests on the reach of the proved setting I credit the general pooling law and its analysis as the central technical contribution. My assessment of its novelty is based on the manuscript's comparison with the cited covariance results, rather than an exhaustive independent priority search. The paper already establishes more than the generic observation that two moments do not determine risk. In particular, Proposition 1 combined with Corollaries 2 and 3 implies population risk asymptotics with and without logarithmic factors for different kernel classes. Appendix A.4 already states the large-argument boxcar transform asymptotic. Foregrounding the corresponding readout implications would improve the presentation; it would not require inventing an additional qualitative result that is absent from the existing theory. My remaining reservation is about the breadth of the learning significance demonstrated by this contribution. The direct predictions concern a particular linear random ensemble and an ordered critical regime; the clean readout results concern representation denoising, isotropic population teachers, or independent Gaussian feature sampling. The shared future-state task supplies an additional relevance check, but the current theory does not predict that task's error from the covariance spectrum alone. These are clearly stated scope choices. I nevertheless place substantial weight on a concrete account of why the critical classification is consequential for the learning questions motivating the paper. On that criterion, I remain marginally below the acceptance threshold. This is a conservative significance judgment, not a finding that the main theorem is redundant, incorrect, or lacking general consequences. The empirical evidence supports complementary, more limited claims Appendix F provides useful reported checks against exact fixed-parameter circular references and compares covariance-dependent population readout curves. Figure 4 also shows that the limiting kernel calibration differs from the calibration required at accessible parameters. This evidence should receive credit on its own terms. For context, the central shared-task settings have two-sided separation approximately 1.33–1.54. Equation (21), evaluated in the infinite-width circular reference at those same parameters, gives ετ(CT,box)≈0.447\varepsilon\tau(C_{T,\mathrm{box}})\approx0.447–0.4890.489, compared with the critical-limit value 1. This quantifies a correction to the raw critical trace normalization. It is not an empirical estimate from the finite real networks, nor a bound on the error of the moment-matched spectral shape or normalized prediction risk. In particular, feature rescaling can change the trace while preserving prediction risk when the regression penalty is rescaled accordingly. I therefore use this calculation only to describe the regime, not as an additional experimental failure. For the common future-state task, Eq. (15) depends on both feature covariance ChC_h and feature–target cross-covariance KhK_h. Matching two spectral moments and the amplitude centroid does not determine KhK_h, so the remaining performance contrast cannot identify a spectrum-only contribution. The manuscript explicitly recognizes this. The experiment demonstrates a joint pooling effect after the specified controls; it leaves open the quantitative attribution of that effect to the critical spectral law. Accordingly, I give the experiments credit for fixed-parameter geometry and shared-task relevance, while treating the importance of the central asymptotic theorem as the main basis for my recommendation.

Questions for authors

Could the paper foreground the readout asymptotics already implied by Proposition 1 and Corollaries 2–3, and make the order of limits explicit? In particular, weak regularization and long training refer here to the limiting functions and their rescaled arguments. Given the calibration contrast already shown in Figure 4, which predictions of the critical law remain quantitatively informative after the fixed-parameter calibration? A discussion using appropriately normalized moments or readout curves would be more informative for this question than the raw trace correction alone. For Appendix G, please specify the empirical ridge objective and loss normalization, the exact penalty grid and scaling, and the joint Gaussian sampling construction, including the cross-covariance between the two feature views. The settings in Appendix H describe another experiment. These are requests for implementation transparency; the stationary model and kernels already determine the joint covariance in principle. Minor correction In Appendix B.2, the flat surrogate has a Marchenko–Pastur aspect ratio γ/3\gamma/3. Its continuous spectral bulk reaches zero at γ=3\gamma=3, where there is no atom at zero. A positive zero atom appears only for γ>3\gamma>3, with mass 1−3/γ1-3/\gamma. The statement using γ≥3\gamma\ge3 for the appearance of the atom should be corrected. This endpoint issue does not affect the main pooling theorem or drive the recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.