- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies warm-started streaming PCA when historical observations have been discarded and only their leading empirical eigenvector u, sample count nu, and two spectral scalars (lambda_hat_1, Delta_hat) are retained. New observations X_1,...,X_n arrive one at a time. In a fixed-dimensional, stationary, rank-one isotropic Gaussian spike with balanced sample sizes (nu/N -> rho in (0,1)), the main theorem (Theorem 3.1) shows that normalized Oja's algorithm started at u with the count-shifted harmonic step c_k = 1/(Delta(nu+k)) tracks, on the same realization and uniformly over all prefixes k <= n, both projector-anchored PCA (the TL-PCA estimator with penalty nu*Delta/k on uu^T) and pooled PCA refitted on the discarded data plus the prefix, with expected maximal squared projector distance o(N^-1). All three estimators share the same first-order influence path, attain risk (d-1)(sigma^2+Delta)sigma^2/(Delta^2(nu+k)) + o(N^-1) uniformly in k, and at the endpoint attain the Gaussian local asymptotic minimax constant; an LAN argument shows the O(d) retained summary carries the same tangent information as the O(d^2) historical covariance for the leading projector. Supporting results: a clipped plug-in schedule using the retained scalars preserves these conclusions (Cor. 3.2); nu is the unique first-order-optimal pseudocount in the harmonic schedule; the projector comparison extends to growing dimension d + log N = o(N) at scale o(d/N) (Cor. 3.3); empirical Riemannian flows restarted from u on each prefix reproduce the Oja path at calibrated times t_k = Delta^-1 log(1+k/nu) (Cor. 5.1); on a fixed dataset the flow follows an anchored-PCA regularization path and has a pooled-efficient stopping time (Cor. 5.2); and a linearized obstruction (Prop. F.1) shows no scalar harmonic schedule achieves pooled equivalence across unequal eigengaps, delimiting the positive result to the equal-gap model.
Validation is a Monte Carlo study at d = 8, Sigma = I + 2vv^T, nu = n in {100, 200, 400, 800}, 1000 trials, reporting normalized endpoint risk (target 1) for oracle Oja, plug-in Oja, plug-in anchored PCA, pooled PCA, and half/double-pseudocount Oja (target 10/9). A separate 2000-trial endpoint audit is described in text as showing N^2 E||P_{w_n} - P_{q~_n}||_F^2 approximately constant over nu = 100..1600, but no values are tabulated; the prefix maximum is not checked, and the flow corollaries receive no numerical evaluation. The proofs are entirely discrete (no SDE), rest on a projective-chart linearization with predictable localization and martingale maximal inequalities, and are given in full in Appendices A-F. The paper discloses that its main results were developed in collaboration with generative AI agents.
Critical strengths
-
The paper is correct and verifiable in every claim I could test. I re-derived the chart recursion (4.1) and its telescoped form (3.3), checked symbolically every displayed identity in (3.14), (5.3)-(5.6), (E.1)-(E.2), Cor. 5.1 (pilot and score weights), Appendix D and Prop. F.1, recomputed the Fisher information to confirm the LAM constant in (3.11), reproduced all 24 entries of Table 2 with independent seeds, and resolved all eleven version-qualified arXiv citations in Appendix G. Nothing failed. This matters to my assessment because it rules out the most common grounds for rejection and means the recommendation turns entirely on contribution and fit, where the authors have a fair chance to argue in rebuttal.
-
The scoping and attribution are unusually honest. Section 7 and Appendix G identify theorem numbers and arXiv versions of the closest prior work, state explicitly which ingredients are not new, and Section 8 lists each assumption the positive theorem depends on. Section 6 discloses what the experiment does not test. This is consequential because the intersection the paper carves out (arrival-indexed prefix path, compressed history with non-negligible pilot fluctuation, unconditional expected prefix maximum, two empirical comparators) is correctly described and does appear to be unoccupied; the question for the committee is only whether that intersection is wide enough to merit an ICLR paper, not whether the authors have overstated it.
Critical weaknesses
-
The statistical content reduces to one elementary weighting identity, and the technique that carries the rest is conceded to be prior work. Every estimator in the paper (oracle Oja, pseudocount-a Oja, anchored PCA with penalty mu, the restarted flow at time t, the Appendix D K-step scheme) has first-order chart alpha * g_v(u) + beta * sum xi_i with alpha + Deltabetak = 1; I verified this symbolically. The normalization is forced, because the two terms are the first-order charts of the history-only and new-data-only PCA estimates, so any such scheme is a convex combination of two independent efficient estimators. Its risk is (Gamma/Delta^2)(alpha^2/nu + (1-alpha)^2/n), minimized at alpha = nu/N. Theorem 3.1's shared path, the "common clock" of Cor. 5.1, the pseudocount uniqueness of Cor. 3.2, the stopping time of Cor. 5.2, the sharp constant, and the Appendix D offset are that minimization in different parametrizations. The genuine work is remainder control for the nonlinear normalized recursion, and Section 7 and Appendix G.1 state that the chart decomposition is Liang (2023) and that harmonic warm starts, localization and discrete moment analysis are standard. What remains is an incremental extension of a known technique to a single model point (rank one, isotropic, Gaussian, balanced, fixed d, Delta known or estimated under exactly that model), with Prop. F.1 showing the equal-gap structure is the mechanism rather than a convenience. This determines my recommendation: a correct paper whose novel content is this narrow is not an ICLR contribution.
-
The evidence offered does not test the claim, and the claim is stated weaker than the evidence supports. The paper's contribution is the same-realization coupling of three paths below the risk scale, uniformly over prefixes. Table 2 reports marginal risks, which three mutually independent efficient estimators would reproduce exactly while having N E||P_a - P_b||_F^2 = 2A = 10.5; the actual coupled values (my replication: 0.33, 0.14, 0.06, 0.03 at the endpoint, 0.49, 0.27, 0.11, 0.05 for the prefix maximum, nu = n = 100..800) appear nowhere in the paper as numbers, and the prefix maximum and the flow corollaries are not checked at all. At the same time, Theorem 3.1 states (3.5)-(3.8) at o(N^-1) while Corollary 3.3 and the Lemma A.2 mechanism give O(log N / N^2) at fixed d, and the replication is consistent with the sharper rate; no obstruction is named. This is consequential because it means the headline quantity has neither a rate in the theorem nor a measurement in the experiment, so the reader cannot tell from the submission how strong the result actually is. It is fixable in a revision, but as submitted the paper does not substantiate its own main claim.
Questions for authors
Clarification requests (answers could change my assessment)
-
Corollary 3.3 bounds the same pooled/anchored prefix comparison by C d(d+log N)/N^2, which at fixed d is O(log N / N^2), and the Lemma A.2 remainder is a product of two factors each O(sqrt((d+log N)/(nu+k))) after the nu*Delta/k anchoring, with the cubic pilot term controlled by E||g_0||^6 = O(nu^-3). Which step of Appendix A.3 prevents (3.5)-(3.8) from being stated at O(log N / N^2)? If there is none, will you restate Theorem 3.1 with the explicit rate?
-
Every scheme in the paper has first-order chart alpha * g_v(u) + beta * sum xi_i with alpha + Deltabetak = 1 and risk (Gamma/Delta^2)(alpha^2/nu + (1-alpha)^2/n). Do you agree that Cor. 3.2's pseudocount uniqueness, Cor. 5.1's clock, Cor. 5.2's stopping time, and the Appendix D offset are the single minimization alpha* = nu/N in four parametrizations? If not, please identify which of them contains information not implied by that minimization.
-
Please report the separate 2000-trial audit as a table: N^2 E||P_{w_n} - P_{q~n}||F^2 and, if available, N^2 E max_k ||P{w_k} - P{q~_k}||_F^2, over the nu grid with Monte Carlo standard errors, alongside the uncoupled reference 2A = 10.5. Table 2 as printed cannot distinguish same-realization coupling from three independent efficient estimators.
-
The half- and double-count columns exceed 10/9 by roughly 3 s.e. at nu = 100 and have not converged at nu = 800 (I reproduce this). Is this the expected sign and order of the second-order correction to (3.14), and does it affect the "unique first-order pseudocount" claim at practical sample sizes?
-
Under rank-two or anisotropic noise, Delta_hat from (2.3) is inconsistent and the Prop. F.1 mechanism gives contraction exponent rho^{Delta/Delta_hat} rather than rho. What is the realized first-order risk of the plug-in schedule under such misspecification, and is the loss first- or second-order in the misspecification? Computing the exact discrete first-order endpoint risk at nu = n = 800, a factor-of-two error in Delta_hat costs 17 to 29 percent; is this acknowledged anywhere?
-
The anchored comparator (2.6) retains only (u, nu, Delta_hat) = O(d) from history; Table 1's O(d^2) is its running new-data covariance. If retained-historical and running state are separated, does Contribution 2 remain a statement about compression rather than about per-observation work?
-
Is there any numerical check of Corollary 5.1 or 5.2? The "stopped full-gradient" column mentioned in Section 6 appears to be the Appendix D scheme, not the flow.
Suggestions for future work (not expected in rebuttal)
-
The preconditioned update mentioned at the end of Appendix F, with a matrix step (nu+i)^-1 G_hat^-1 for an estimated gap operator, would turn the calibration identity into a method that survives unequal gaps; a result along those lines would substantially change the significance of the paper.
-
One experiment in the regime that motivates the paper, d in the thousands on a real stream against History PCA and TL-PCA, would make the O(d) versus O(d^2) argument concrete.
8 comments
Question 4
Related common response: C4 (numerical context). The finite-sample correction is addressed below.
The first-order theorem does not determine the sign of the finite-sample correction to the leading risk or an exact finite-sample optimum. In the full-precision records underlying Table 3, the half-count mean at is , below ; the double-count mean is , about 2.77 standard errors above it. At 800, the means and are only 0.11 and 0.18 standard errors above . Thus the 800 entries are within 0.2 standard errors of the limit, while the half-count entry at 100 is below it. These values do not establish nonconvergence or a universal sign for the finite-sample correction. We have not derived an analytic second-order expansion that predicts its sign and order. The first-order uniqueness statement therefore does not identify the exact best pseudocount at a finite sample size. Appendix H explains this distinction; Question 3 above separates the historical controls from the new run.
Question 5
Common response: C2 (calibration sensitivity) and C4 (experiments).
Proposition B.1 addresses calibration error for the linear error recursion. Let be the assumed eigengap divided by the true eigengap, , and the limiting historical sample fraction. Under the stated assumptions on the historical starting error, the limiting mean squared error, multiplied by and divided by the pooled risk constant, is
Thus is a risk ratio for the linear recursion, with value one at correct calibration. At , doubling the assumed gap gives and halving it gives . For the ideal linear recursion initialized at the historical first-order term, the exact finite-sample risk ratios at are and . Near correct calibration, . A fixed error changes the leading risk coefficient, while a small calibration error has a quadratic relative cost.
For unequal gaps , consider scalar effective steps with fixed . If the historical starting error in the linear recursion has the corresponding score expansion, the linear recursion's mean squared error, multiplied by , converges to . Here and , where is the population eigenvector with eigenvalue and the sum runs over . This describes the linear recursion; it does not establish the risk of nonlinear Oja under a misspecified model. Under general anisotropy, the retained gap estimate converges to the leading eigenvalue minus the average trailing eigenvalue, which generally differs from the individual .
The controlled gap-factor experiment stays within the original sampling model. At , half and twice the true oracle gap give normalized risks and , versus with correct calibration. This does not test an estimated gap under arbitrary anisotropic sampling.
Question 6
Common response: C5 (historical information and streaming cost).
Section 2 and revised Table 1 make this distinction explicit. Projector-regularized PCA retains historical information but stores an new covariance. Oja also has running state and per-arrival work. The compression claim concerns historical information needed for the leading projector; it is not an implementation claim for the refitted comparator.
Question 7
Common response: C3 (flow) and C4 (flow experiments).
We evaluate the empirical flow using the exact matrix exponential. The older "stopped full-gradient" column uses finitely many gradient steps from Appendix D; it does not evaluate the exact continuous flow. For balanced , the new -scaled means of the maximum squared flow/Oja projector discrepancy over arrivals are , , and .
For the time grid evaluated on each final dataset, , the normalized mean risks at are . The empirical mean is smallest at on the tested grid. Paired increases at and are and . Appendix H.2, Figure 1 and Table 5 give these checks. The tested times use the true eigengap; these experiments did not evaluate times computed from an estimated eigengap. The observed grid minimum does not establish an exact finite-sample optimum or an off-grid minimum.
The review identifies items 8-9 as future-work suggestions, not requirements for the rebuttal.
Question 8
Common response: C2 (extensions beyond equal gaps).
Section 5 describes the three alternatives. Proposition F.3 uses the inverse eigengaps in the update and gives a PCA objective with a matrix penalty having the same leading error expansion. It assumes a known noise covariance satisfying and uses the true leading eigenvalue for calibration. Proposition F.2 treats linear recursions with nearly equal eigengaps; F.4 gives an averaging identity that includes the historical estimate with weight . Uniform statistical guarantees for the nonlinear algorithms remain open.
Question 9
Common response: C4 (experiments and remaining validation).
The present revision adds the trajectory, calibration and exact empirical-flow checks. A large-scale experiment on a real data stream can potentially help enhance the validation and demonstrate the strength of Oja. However, current numerical checks together with rigorous theoretical derivation are sufficient in confirming the correctness of our results. And the strength of Oja can be directly justified from Table 1. Thus, we leave the proposed large experiment only as useful future validation.
Contribution and significance
Common response: C5 (significance), C3 (flow), and C2 (scope).
Your review reports successful checks of selected identities, the efficiency constant, numerical entries and citations. We appreciate those checks and address the contribution and scope concerns below.
The contribution is first-order agreement with the pooled estimates computed from the same observations, uniformly as data arrive. The expected maximal squared projector discrepancy is smaller than the statistical risk. We agree that the calibrations share one first-order weighting mechanism. Controlling the nonlinear error in the normalized Oja recursion is needed to establish that the actual algorithm has this first-order behavior.
The weighting identity identifies the candidate calibration, but it does not establish that the actual normalized stochastic algorithm follows the same empirical trajectory uniformly. The contribution is that statistical conclusion; it need not introduce a new general proof technique. The closest results compared in Section 7 and Appendix G address different estimator comparisons, data access or notions of a trajectory.
We agree that the assumptions restrict scope. A structured model can nevertheless reveal a worthwhile statistical phenomenon: here it permits a sharp characterization of when compressed historical PCA supports the same first-order evolution as full-information refitting. Greater model generality is a further research question; it does not replace the value of this characterization. Section 5 gives additional first-order results beyond equal gaps, with their nonlinear statistical extensions left open.
Question 1
Common response: C2 (explicit rates).
Equations (3.4)-(3.5) now state the explicit oracle rates for fixed dimension. Equation (3.6) concerns a different quantity: risk relative to the population direction. Expanding that squared error introduces a cross term between the leading error and its remainder, as well as a correction involving fourth powers when coordinates are converted to projector loss. The sharper comparison between estimators therefore does not by itself give the same risk remainder. The plug-in comparison remains o(N−1); the endpoint central limit theorem states convergence in distribution without a convergence rate.
Question 2
Common response: C3 (calibration) and C5 (contribution).
Yes. At first order, the pseudocount, flow time, stopping time and Appendix D step size determine the same variance-minimizing historical weight, ν/(ν+n). Beyond the common minimization, the proof establishes this leading error term for normalized Oja and the empirical flows and controls their remaining errors uniformly. This connects the actual estimators to the pooled estimates on the same data. Minimizing a two-term variance expression alone does not establish those comparisons.
Question 3
Common response: C4 (numerical evidence).
The values from the new trajectory experiment are Monte Carlo means plus or minus one standard error. At ν=n=800, the N2-scaled oracle endpoint discrepancy is 47.6723±2.6019, while the maximum over arrivals is 87.7539±4.0283; the plug-in endpoint discrepancy is 30.0114±1.3697. These are distinct from the older 2,000-trial endpoint audit, whose preserved rounded values are:
Its standard errors and maxima over arrivals were not retained. We do not reconstruct them or attach the new run's errors to these historical means.
For the independent-estimator reference, let A denote the leading risk constant, so NEL→A for one efficient estimator. In this experiment A=5.25, and L(Pa,Pb)=21∥Pa−Pb∥F2. For two independent efficient fits whose leading errors have mean zero, each based on N observations, NE∥Pa−Pb∥F2→4A=21, whereas NE[21∥Pa−Pb∥F2]→2A=10.5. All N2-scaled discrepancies quoted above and our pairwise columns use full squared Frobenius distance, without the factor 1/2. These are endpoint references, not references for a maximum over arrivals or for fits sharing their historical estimate. On an N2-scaled full-distance plot the independent endpoint reference would grow like 21N.