acceptodds
Under review as a conference paper at ICLR 2027

Warm-Started Oja Matches Pooled PCA to First Order

Abstract

Suppose historical observations are no longer available, but their leading empirical principal component, sample count, and spectral calibration have been retained. Can one-pass continuation track the PCA estimates that would be obtained by pooling the old data with every incoming prefix? In a fixed-dimensional, stationary rank-one isotropic Gaussian spike with iid historical and streaming observations and comparable sample sizes, we answer yes to first order. We prove that the actual normalized Oja recursion with a sample-count-shifted harmonic effective step is, uniformly over every prefix and on the same realization, within expected maximal squared projector distance of both projector-regularized PCA and pooled empirical PCA. All three share the same influence path and, uniformly in the prefix, attain risk . At the final prefix this is the Gaussian local asymptotic minimax constant. Clipped old-sample plug-in calibration preserves the conclusion, and the historical sample size is the unique first-order pseudocount. The projector-risk comparison extends to at the natural scale. The Oja proof is entirely discrete. Empirical Riemannian flows restarted on the same prefixes track the Oja path at calibrated times; a separate fixed-data time profile identifies pooled-efficient early stopping.

Primary area:
optimization
Keywords:
Streaming PCA, Oja's algorithm, trajectory equivalence, Riemannian gradient flow, early stopping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EGPVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies warm-started streaming PCA when historical observations have been discarded and only their leading empirical eigenvector u, sample count nu, and two spectral scalars (lambda_hat_1, Delta_hat) are retained. New observations X_1,...,X_n arrive one at a time. In a fixed-dimensional, stationary, rank-one isotropic Gaussian spike with balanced sample sizes (nu/N -> rho in (0,1)), the main theorem (Theorem 3.1) shows that normalized Oja's algorithm started at u with the count-shifted harmonic step c_k = 1/(Delta(nu+k)) tracks, on the same realization and uniformly over all prefixes k <= n, both projector-anchored PCA (the TL-PCA estimator with penalty nu*Delta/k on uu^T) and pooled PCA refitted on the discarded data plus the prefix, with expected maximal squared projector distance o(N^-1). All three estimators share the same first-order influence path, attain risk (d-1)(sigma^2+Delta)sigma^2/(Delta^2(nu+k)) + o(N^-1) uniformly in k, and at the endpoint attain the Gaussian local asymptotic minimax constant; an LAN argument shows the O(d) retained summary carries the same tangent information as the O(d^2) historical covariance for the leading projector. Supporting results: a clipped plug-in schedule using the retained scalars preserves these conclusions (Cor. 3.2); nu is the unique first-order-optimal pseudocount in the harmonic schedule; the projector comparison extends to growing dimension d + log N = o(N) at scale o(d/N) (Cor. 3.3); empirical Riemannian flows restarted from u on each prefix reproduce the Oja path at calibrated times t_k = Delta^-1 log(1+k/nu) (Cor. 5.1); on a fixed dataset the flow follows an anchored-PCA regularization path and has a pooled-efficient stopping time (Cor. 5.2); and a linearized obstruction (Prop. F.1) shows no scalar harmonic schedule achieves pooled equivalence across unequal eigengaps, delimiting the positive result to the equal-gap model.

Validation is a Monte Carlo study at d = 8, Sigma = I + 2vv^T, nu = n in {100, 200, 400, 800}, 1000 trials, reporting normalized endpoint risk (target 1) for oracle Oja, plug-in Oja, plug-in anchored PCA, pooled PCA, and half/double-pseudocount Oja (target 10/9). A separate 2000-trial endpoint audit is described in text as showing N^2 E||P_{w_n} - P_{q~_n}||_F^2 approximately constant over nu = 100..1600, but no values are tabulated; the prefix maximum is not checked, and the flow corollaries receive no numerical evaluation. The proofs are entirely discrete (no SDE), rest on a projective-chart linearization with predictable localization and martingale maximal inequalities, and are given in full in Appendices A-F. The paper discloses that its main results were developed in collaboration with generative AI agents.

Critical strengths
  1. The paper is correct and verifiable in every claim I could test. I re-derived the chart recursion (4.1) and its telescoped form (3.3), checked symbolically every displayed identity in (3.14), (5.3)-(5.6), (E.1)-(E.2), Cor. 5.1 (pilot and score weights), Appendix D and Prop. F.1, recomputed the Fisher information to confirm the LAM constant in (3.11), reproduced all 24 entries of Table 2 with independent seeds, and resolved all eleven version-qualified arXiv citations in Appendix G. Nothing failed. This matters to my assessment because it rules out the most common grounds for rejection and means the recommendation turns entirely on contribution and fit, where the authors have a fair chance to argue in rebuttal.

  2. The scoping and attribution are unusually honest. Section 7 and Appendix G identify theorem numbers and arXiv versions of the closest prior work, state explicitly which ingredients are not new, and Section 8 lists each assumption the positive theorem depends on. Section 6 discloses what the experiment does not test. This is consequential because the intersection the paper carves out (arrival-indexed prefix path, compressed history with non-negligible pilot fluctuation, unconditional expected prefix maximum, two empirical comparators) is correctly described and does appear to be unoccupied; the question for the committee is only whether that intersection is wide enough to merit an ICLR paper, not whether the authors have overstated it.

Critical weaknesses
  1. The statistical content reduces to one elementary weighting identity, and the technique that carries the rest is conceded to be prior work. Every estimator in the paper (oracle Oja, pseudocount-a Oja, anchored PCA with penalty mu, the restarted flow at time t, the Appendix D K-step scheme) has first-order chart alpha * g_v(u) + beta * sum xi_i with alpha + Deltabetak = 1; I verified this symbolically. The normalization is forced, because the two terms are the first-order charts of the history-only and new-data-only PCA estimates, so any such scheme is a convex combination of two independent efficient estimators. Its risk is (Gamma/Delta^2)(alpha^2/nu + (1-alpha)^2/n), minimized at alpha = nu/N. Theorem 3.1's shared path, the "common clock" of Cor. 5.1, the pseudocount uniqueness of Cor. 3.2, the stopping time of Cor. 5.2, the sharp constant, and the Appendix D offset are that minimization in different parametrizations. The genuine work is remainder control for the nonlinear normalized recursion, and Section 7 and Appendix G.1 state that the chart decomposition is Liang (2023) and that harmonic warm starts, localization and discrete moment analysis are standard. What remains is an incremental extension of a known technique to a single model point (rank one, isotropic, Gaussian, balanced, fixed d, Delta known or estimated under exactly that model), with Prop. F.1 showing the equal-gap structure is the mechanism rather than a convenience. This determines my recommendation: a correct paper whose novel content is this narrow is not an ICLR contribution.

  2. The evidence offered does not test the claim, and the claim is stated weaker than the evidence supports. The paper's contribution is the same-realization coupling of three paths below the risk scale, uniformly over prefixes. Table 2 reports marginal risks, which three mutually independent efficient estimators would reproduce exactly while having N E||P_a - P_b||_F^2 = 2A = 10.5; the actual coupled values (my replication: 0.33, 0.14, 0.06, 0.03 at the endpoint, 0.49, 0.27, 0.11, 0.05 for the prefix maximum, nu = n = 100..800) appear nowhere in the paper as numbers, and the prefix maximum and the flow corollaries are not checked at all. At the same time, Theorem 3.1 states (3.5)-(3.8) at o(N^-1) while Corollary 3.3 and the Lemma A.2 mechanism give O(log N / N^2) at fixed d, and the replication is consistent with the sharper rate; no obstruction is named. This is consequential because it means the headline quantity has neither a rate in the theorem nor a measurement in the experiment, so the reader cannot tell from the submission how strong the result actually is. It is fixable in a revision, but as submitted the paper does not substantiate its own main claim.

Questions for authors

Clarification requests (answers could change my assessment)

  1. Corollary 3.3 bounds the same pooled/anchored prefix comparison by C d(d+log N)/N^2, which at fixed d is O(log N / N^2), and the Lemma A.2 remainder is a product of two factors each O(sqrt((d+log N)/(nu+k))) after the nu*Delta/k anchoring, with the cubic pilot term controlled by E||g_0||^6 = O(nu^-3). Which step of Appendix A.3 prevents (3.5)-(3.8) from being stated at O(log N / N^2)? If there is none, will you restate Theorem 3.1 with the explicit rate?

  2. Every scheme in the paper has first-order chart alpha * g_v(u) + beta * sum xi_i with alpha + Deltabetak = 1 and risk (Gamma/Delta^2)(alpha^2/nu + (1-alpha)^2/n). Do you agree that Cor. 3.2's pseudocount uniqueness, Cor. 5.1's clock, Cor. 5.2's stopping time, and the Appendix D offset are the single minimization alpha* = nu/N in four parametrizations? If not, please identify which of them contains information not implied by that minimization.

  3. Please report the separate 2000-trial audit as a table: N^2 E||P_{w_n} - P_{q~n}||F^2 and, if available, N^2 E max_k ||P{w_k} - P{q~_k}||_F^2, over the nu grid with Monte Carlo standard errors, alongside the uncoupled reference 2A = 10.5. Table 2 as printed cannot distinguish same-realization coupling from three independent efficient estimators.

  4. The half- and double-count columns exceed 10/9 by roughly 3 s.e. at nu = 100 and have not converged at nu = 800 (I reproduce this). Is this the expected sign and order of the second-order correction to (3.14), and does it affect the "unique first-order pseudocount" claim at practical sample sizes?

  5. Under rank-two or anisotropic noise, Delta_hat from (2.3) is inconsistent and the Prop. F.1 mechanism gives contraction exponent rho^{Delta/Delta_hat} rather than rho. What is the realized first-order risk of the plug-in schedule under such misspecification, and is the loss first- or second-order in the misspecification? Computing the exact discrete first-order endpoint risk at nu = n = 800, a factor-of-two error in Delta_hat costs 17 to 29 percent; is this acknowledged anywhere?

  6. The anchored comparator (2.6) retains only (u, nu, Delta_hat) = O(d) from history; Table 1's O(d^2) is its running new-data covariance. If retained-historical and running state are separated, does Contribution 2 remain a statement about compression rather than about per-observation work?

  7. Is there any numerical check of Corollary 5.1 or 5.2? The "stopped full-gradient" column mentioned in Section 6 appears to be the Appendix D scheme, not the flow.

Suggestions for future work (not expected in rebuttal)

  1. The preconditioned update mentioned at the end of Appendix F, with a matrix step (nu+i)^-1 G_hat^-1 for an estimated gap operator, would turn the calibration identity into a method that survives unequal gaps; a result along those lines would substantially change the significance of the paper.

  2. One experiment in the regime that motivates the paper, d in the thousands on a real stream against History PCA and TL-PCA, would make the O(d) versus O(d^2) argument concrete.

8 comments

Contribution and significance

Common response: C5 (significance), C3 (flow), and C2 (scope).

Your review reports successful checks of selected identities, the efficiency constant, numerical entries and citations. We appreciate those checks and address the contribution and scope concerns below.

The contribution is first-order agreement with the pooled estimates computed from the same observations, uniformly as data arrive. The expected maximal squared projector discrepancy is smaller than the statistical risk. We agree that the calibrations share one first-order weighting mechanism. Controlling the nonlinear error in the normalized Oja recursion is needed to establish that the actual algorithm has this first-order behavior.

The weighting identity identifies the candidate calibration, but it does not establish that the actual normalized stochastic algorithm follows the same empirical trajectory uniformly. The contribution is that statistical conclusion; it need not introduce a new general proof technique. The closest results compared in Section 7 and Appendix G address different estimator comparisons, data access or notions of a trajectory.

We agree that the assumptions restrict scope. A structured model can nevertheless reveal a worthwhile statistical phenomenon: here it permits a sharp characterization of when compressed historical PCA supports the same first-order evolution as full-information refitting. Greater model generality is a further research question; it does not replace the value of this characterization. Section 5 gives additional first-order results beyond equal gaps, with their nonlinear statistical extensions left open.

Question 1

Common response: C2 (explicit rates).

Equations (3.4)-(3.5) now state the explicit oracle rates for fixed dimension. Equation (3.6) concerns a different quantity: risk relative to the population direction. Expanding that squared error introduces a cross term between the leading error and its remainder, as well as a correction involving fourth powers when coordinates are converted to projector loss. The sharper comparison between estimators therefore does not by itself give the same risk remainder. The plug-in comparison remains o(N−1)o(N^{-1}); the endpoint central limit theorem states convergence in distribution without a convergence rate.

Question 2

Common response: C3 (calibration) and C5 (contribution).

Yes. At first order, the pseudocount, flow time, stopping time and Appendix D step size determine the same variance-minimizing historical weight, ν/(ν+n)\nu/(\nu+n). Beyond the common minimization, the proof establishes this leading error term for normalized Oja and the empirical flows and controls their remaining errors uniformly. This connects the actual estimators to the pooled estimates on the same data. Minimizing a two-term variance expression alone does not establish those comparisons.

Question 3

Common response: C4 (numerical evidence).

The values from the new trajectory experiment are Monte Carlo means plus or minus one standard error. At ν=n=800\nu=n=800, the N2N^2-scaled oracle endpoint discrepancy is 47.6723±2.601947.6723\pm2.6019, while the maximum over arrivals is 87.7539±4.028387.7539\pm4.0283; the plug-in endpoint discrepancy is 30.0114±1.369730.0114\pm1.3697. These are distinct from the older 2,000-trial endpoint audit, whose preserved rounded values are:

ν=n\nu=nN2E∥Pwn−Pq~n∥F2N^2\mathbb E\|P_{w_n}-P_{\widetilde q_n}\|_F^2
10060.57
20052.32
40047.02
80049.23
160048.85

Its standard errors and maxima over arrivals were not retained. We do not reconstruct them or attach the new run's errors to these historical means.

For the independent-estimator reference, let AA denote the leading risk constant, so NEL→AN\mathbb E L\to A for one efficient estimator. In this experiment A=5.25A=5.25, and L(Pa,Pb)=12∥Pa−Pb∥F2L(P_a,P_b)=\tfrac12\|P_a-P_b\|_F^2. For two independent efficient fits whose leading errors have mean zero, each based on NN observations, NE∥Pa−Pb∥F2→4A=21N\mathbb E\|P_a-P_b\|_F^2\to4A=21, whereas NE[12∥Pa−Pb∥F2]→2A=10.5N\mathbb E[\tfrac12\|P_a-P_b\|_F^2]\to2A=10.5. All N2N^2-scaled discrepancies quoted above and our pairwise columns use full squared Frobenius distance, without the factor 1/21/2. These are endpoint references, not references for a maximum over arrivals or for fits sharing their historical estimate. On an N2N^2-scaled full-distance plot the independent endpoint reference would grow like 21N21N.

Question 4

Related common response: C4 (numerical context). The finite-sample correction is addressed below.

The first-order theorem does not determine the sign of the finite-sample correction to the leading risk or an exact finite-sample optimum. In the full-precision records underlying Table 3, the half-count mean at ν=n=100\nu=n=100 is 1.099202±0.0192611.099202\pm0.019261, below 10/910/9; the double-count mean is 1.175642±0.0233191.175642\pm0.023319, about 2.77 standard errors above it. At 800, the means 1.113179±0.0196421.113179\pm0.019642 and 1.114613±0.0196141.114613\pm0.019614 are only 0.11 and 0.18 standard errors above 10/910/9. Thus the 800 entries are within 0.2 standard errors of the limit, while the half-count entry at 100 is below it. These values do not establish nonconvergence or a universal sign for the finite-sample correction. We have not derived an analytic second-order expansion that predicts its sign and order. The first-order uniqueness statement therefore does not identify the exact best pseudocount at a finite sample size. Appendix H explains this distinction; Question 3 above separates the historical controls from the new run.

Question 5

Common response: C2 (calibration sensitivity) and C4 (experiments).

Proposition B.1 addresses calibration error for the linear error recursion. Let κ\kappa be the assumed eigengap divided by the true eigengap, s=1/κs=1/\kappa, and ρ=lim⁡ν/N∈(0,1)\rho=\lim\nu/N\in(0,1) the limiting historical sample fraction. Under the stated assumptions on the historical starting error, the limiting mean squared error, multiplied by NN and divided by the pooled risk constant, is

Rs(ρ)=1+(s−1)2∫ρ1t2s−2 dt.R_s(\rho)=1+(s-1)^2\int_\rho^1 t^{2s-2}\,dt.

Thus RsR_s is a risk ratio for the linear recursion, with value one at correct calibration. At ρ=1/2\rho=1/2, doubling the assumed gap gives 1.17331.1733 and halving it gives 1.29171.2917. For the ideal linear recursion initialized at the historical first-order term, the exact finite-sample risk ratios at ν=n=800\nu=n=800 are 1.173381.17338 and 1.292031.29203. Near correct calibration, R1/κ(ρ)=1+(1−ρ)(κ−1)2+O(∣κ−1∣3)R_{1/\kappa}(\rho)=1+(1-\rho)(\kappa-1)^2+O(|\kappa-1|^3). A fixed error changes the leading 1/N1/N risk coefficient, while a small calibration error has a quadratic relative cost.

For unequal gaps δj=λ1−λj\delta_j=\lambda_1-\lambda_j, consider scalar effective steps ci=γ/(ν+i)c_i=\gamma/(\nu+i) with fixed γ>0\gamma>0. If the historical starting error in the linear recursion has the corresponding score expansion, the linear recursion's mean squared error, multiplied by NN, converges to ∑jAjRγδj(ρ)\sum_j A_jR_{\gamma\delta_j}(\rho). Here Aj=Var⁡(ξj,i)/δj2A_j=\operatorname{Var}(\xi_{j,i})/\delta_j^2 and ξj,i=(v⊤Xi)(vj⊤Xi)\xi_{j,i}=(v^\top X_i)(v_j^\top X_i), where vjv_j is the population eigenvector with eigenvalue λj\lambda_j and the sum runs over j≥2j\ge2. This describes the linear recursion; it does not establish the risk of nonlinear Oja under a misspecified model. Under general anisotropy, the retained gap estimate converges to the leading eigenvalue minus the average trailing eigenvalue, which generally differs from the individual δj\delta_j.

The controlled gap-factor experiment stays within the original sampling model. At ν=n=800\nu=n=800, half and twice the true oracle gap give normalized risks 1.3375±0.03101.3375\pm0.0310 and 1.1926±0.02671.1926\pm0.0267, versus 1.0308±0.02341.0308\pm0.0234 with correct calibration. This does not test an estimated gap under arbitrary anisotropic sampling.

Question 6

Common response: C5 (historical information and streaming cost).

Section 2 and revised Table 1 make this distinction explicit. Projector-regularized PCA retains O(d)O(d) historical information but stores an O(d2)O(d^2) new covariance. Oja also has O(d)O(d) running state and per-arrival work. The compression claim concerns historical information needed for the leading projector; it is not an O(d)O(d) implementation claim for the refitted comparator.

Question 7

Common response: C3 (flow) and C4 (flow experiments).

We evaluate the empirical flow using the exact matrix exponential. The older "stopped full-gradient" column uses finitely many gradient steps from Appendix D; it does not evaluate the exact continuous flow. For balanced ν=n=100,200,400,800\nu=n=100,200,400,800, the new N2N^2-scaled means of the maximum squared flow/Oja projector discrepancy over arrivals are 12.1375±0.401612.1375\pm0.4016, 12.7253±0.423312.7253\pm0.4233, 12.4721±0.378012.4721\pm0.3780 and 11.9345±0.362611.9345\pm0.3626.

For the time grid evaluated on each final dataset, t/t∗∈{1/2,3/4,1,5/4,3/2}t/t_*\in\{1/2,3/4,1,5/4,3/2\}, the normalized mean risks at ν=n=800\nu=n=800 are 1.1898,1.0610,1.0316,1.0626,1.12821.1898,1.0610,1.0316,1.0626,1.1282. The empirical mean is smallest at t∗t_* on the tested grid. Paired increases at 3t∗/43t_*/4 and 5t∗/45t_*/4 are 0.02945±0.006220.02945\pm0.00622 and 0.03106±0.005290.03106\pm0.00529. Appendix H.2, Figure 1 and Table 5 give these checks. The tested times use the true eigengap; these experiments did not evaluate times computed from an estimated eigengap. The observed grid minimum does not establish an exact finite-sample optimum or an off-grid minimum.

The review identifies items 8-9 as future-work suggestions, not requirements for the rebuttal.

Question 8

Common response: C2 (extensions beyond equal gaps).

Section 5 describes the three alternatives. Proposition F.3 uses the inverse eigengaps in the update and gives a PCA objective with a matrix penalty having the same leading error expansion. It assumes a known noise covariance BB satisfying Bv=bvBv=bv and uses the true leading eigenvalue λ1\lambda_1 for calibration. Proposition F.2 treats linear recursions with nearly equal eigengaps; F.4 gives an averaging identity that includes the historical estimate with weight ν\nu. Uniform statistical guarantees for the nonlinear algorithms remain open.

Question 9

Common response: C4 (experiments and remaining validation).

The present revision adds the trajectory, calibration and exact empirical-flow checks. A large-scale experiment on a real data stream can potentially help enhance the validation and demonstrate the strength of Oja. However, current numerical checks together with rigorous theoretical derivation are sufficient in confirming the correctness of our results. And the strength of Oja can be directly justified from Table 1. Thus, we leave the proposed large experiment only as useful future validation.

Reviewer_FHKHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies warm-started normalized Oja for incremental estimation of the leading principal component when historical observations and their covariance are no longer available. The retained information consists of the historical empirical PC1, its sample count, and spectral calibration scalars. In a stationary rank-one isotropic Gaussian model, the main claimed result is first-order agreement, uniformly over incoming-data prefixes, among Oja, PCA regularized toward the historical direction, and PCA fitted to the pooled historical and incoming data. The paper also presents a restarted empirical-flow interpretation.

Critical strengths

The authors provide a detailed theoretical comparison of three estimators with different access to historical information. This is useful for understanding how a retained principal component can support estimation as new observations arrive.

Critical weaknesses

The current version of the paper looks more like a draft than a finished manuscript, which makes it difficult to recommend it for publication.

  1. The geometric presentation obscures the central contribution. I found the explanation of the distinct roles and connections of Oja iterates, anchored and pooled PCA, spherical gradient flows, and local projective coordinates insufficient. Although the corresponding formulas and results are present, their conceptual relationship is difficult to follow. In particular, the explanation needs to connect the restarted flow to the advertised Riemannian interpretation and make the transition from localized chart analysis to unconditional projector comparisons transparent. These connections are essential to interpreting the trajectory agreement. The current presentation leaves substantial room for confusion and makes the claims difficult to assess even after careful reading. The manuscript needs substantial reorganization and explanation before the contribution is communicated convincingly.

  2. Terminology, notation, and organization increase the burden on the reader. Names such as “pooled prefix PCA” and “anchored prefix PCA” are used throughout, but their practical meaning and provenance should be made immediately clear when introduced. The fuller related-work discussion should appear earlier so readers can identify the novelty before encountering the technical development. The notation should also be easier to navigate: distinguish the Gaussian distribution symbol N\mathcal{N} from the total sample count NN, use consistent vector typography, and consider mm and nn for the historical and incoming sample counts. Consistent use of “i.i.d.” would improve readability. These editorial changes should accompany a clearer conceptual narrative.

  3. The restricted model and empirical evaluation leave practical relevance insufficiently demonstrated. The positive result concerns a stationary rank-one signal with isotropic noise. Section 6 specifies the synthetic Gaussian sampling distribution, but the evaluation remains confined to a highly controlled setting. It does not show a concrete application in which both historical observations and their covariance are unavailable. The manuscript should explain why this access constraint arises in a realistic setting and demonstrate what the result contributes there. A real-data illustration would strengthen that motivation. The simulation protocol should also be consolidated into a clear description of sampling, historical fitting, calibration, and clipping.

  4. The AI-use statement is not sufficiently specific at the task level. The statement discloses substantial assistance, including proof development, auditing, implementation, and drafting. Its broad categories leave the scope of assistance for some required tasks unclear. Please provide a more precise account of use, nonuse, or inapplicability for the remaining required-task categories.

Questions for authors
  1. Estimator terminology and provenance. Please explain the meaning of “anchored prefix PCA” (why use this name? Is this name due to a previous work (eventually in the Related works sections)? ) when Equation (2.6) is introduced and identify the connection to TL-PCA at that point. Please also make clear which objects are computational procedures and which serve as theoretical comparators.

  2. Riemannian-flow interpretation. Please explain directly how the spherical projected-gradient equation in Equation (5.1) implements the Riemannian flow advertised in the abstract and contribution statements, and how Corollary 5.1 establishes the stated alignment. The distinction between restarting a flow at each prefix and continuing the Oja recursion needs a clear conceptual explanation, including the different roles of optimization time and sample-arrival index.

  3. Chart qualifications and geometry. Please explain the geometric space represented by the local chart in Equation (3.1) and its relationship to the spherical flow and principal-component projectors. In Theorem 3.1, what precisely changes when moving from the localized chart statements to conclusions “without chart qualifications”? The exposition should make clear which restrictions concern the proof's coordinates and which are assumptions of the statistical result.

  4. Scope beyond rank one. Please explain how much of the main insight depends on the rank-one isotropic structure. Section 8 mentions an equal-gap top-kk extension; which components of the argument could carry over to a rank-kk signal and top-kk subspace estimation, and what prevents the existing theorem from covering that setting?

  5. Experimental support. Please provide a consolidated simulation protocol, including the construction of the historical PC1 and spectral calibration, and the clipping configuration. Please also describe a realistic application that requires the stated historical-data access model and explain how the reported synthetic evaluation informs that application.

  6. AI-use disclosure. Please clarify the scope of assistance for the remaining required-task categories, including synthetic-data generation, formulation of mathematical claims, research or experimental design, and interpretation of results, with explicit use, nonuse, or inapplicability where relevant, consistent with the author AI policy.

13 comments

Critical Weaknesses

Common response: C1 (presentation), C3 (flow interpretation), C4 (empirical evidence and motivation).

Specific issues:

We agree that the relationships among the objects needed a clearer explanation. All three estimators target the same population principal component. Oja updates its current unit vector with each observation; the two PCA comparators refit eigenvectors at each arrival, using different historical information. The empirical flow is a batch comparison that helps interpret the calibration. Local coordinates are used only to analyze their errors.

The connection is a shared first-order term in those errors, computed from the same historical and incoming observations. The proof controls the remaining nonlinear errors, then obtains a bound that includes all sampled datasets by using bounded projector distance. Questions 1-3 below explain the estimator definitions, the flow comparison and this coordinate argument.

For notation, we distinguish the Gaussian symbol N\mathcal N from sample count NN, define ν,n,N\nu,n,N together, state norm conventions, and use "i.i.d." consistently. We retain the established count notation in the proofs.

Question 1: Estimator terminology and provenance

Common response: C1 (estimator roles and presentation).

We now use the term projector-regularized PCA. Equation (2.6) identifies the rank-one TL-PCA-P estimator of Hendy and Dar, Section 4.2, Equations (7)-(8), applied to observations with known mean zero and our penalty νΔ/k\nu\Delta/k. It fits the leading eigenvector of the new covariance plus (νΔ/k)uu⊤(\nu\Delta/k)uu^\top, where uu is the historical empirical principal component. Oja is the continuing one-pass procedure. Pooled PCA is the full-information benchmark, refitted on the historical sample and the first kk new observations. In the earlier estimator names, "prefix" referred only to the first kk new observations. It is no longer part of either name.

Question 2: Riemannian-flow interpretation

Common response: C3 (flow interpretation).

For a fixed kk, the objective is 12w⊤Skw\tfrac12 w^\top S_kw on the unit sphere. Its Euclidean gradient is SkwS_kw. Projecting that vector onto the sphere's tangent space gives (I−ww⊤)Skw(I-ww^\top)S_kw, the Riemannian gradient in Equation (4.5).

The two indices have different meanings. The count kk determines which observations enter SkS_k; optimization time tt varies with those data held fixed. For each kk, the flow Fk(t)F_k(t) starts from the same historical component uu. Corollary 4.1 compares Fk(tk)F_k(t_k) at its calibrated time with the actual kkth Oja iterate, uniformly over kk. Oja itself continues from its previous iterate. Thus the result compares selected evaluations of separate batch flows with a single streaming recursion.

Question 3: Chart qualifications and geometry

Common response: C1 (coordinates and unconditional bounds).

We assume the word chart here means a system of local coordinates used in the proof. A unit eigenvector aa and its negative represent the same principal component because both give the projector Pa=aa⊤P_a=aa^\top. The statistical parameter is therefore a line through the origin; this collection of lines is real projective space RPd−1\mathbb{RP}^{d-1}. The spherical flow uses a unit vector to represent that line.

Equation (3.1) assigns coordinates gv(a)=V⊥⊤a/(v⊤a)g_v(a)=V_\perp^\top a/(v^\top a), where the columns of V⊥V_\perp form an orthonormal basis perpendicular to the population direction vv. These coordinates divide the perpendicular component by the component along vv. They are unchanged when aa is replaced by −a-a. The unknown vv is used only in the proof; a zero denominator would invalidate this coordinate formula, not the projector or the implemented algorithm.

On a high-probability event, the denominators stay bounded away from zero for all three estimator paths. The coordinate expansions can then be compared uniformly. On the remaining datasets, the squared projector distance is still at most two. Together with the small probability of that event's complement, this gives Equation (3.5), whose expectation includes all sampled datasets. This is what the earlier phrase "without chart qualifications" meant. No observations are removed and the algorithm is unchanged. Unconditional second moments of the raw coordinate ratios are not required or claimed.

Question 4: Scope beyond rank one

Common response: C2 (model restrictions and extensions).

In an equal-gap model for the leading rr-dimensional subspace, its linearized error has r(d−r)r(d-r) coordinates with a common contraction factor. This suggests the same sample-count calibration. A full extension would need matrix-valued coordinates, control of the matrix inverses in their updates, and uniform bounds for matrix-valued fluctuations and nonlinear errors. The current proof does not supply these steps; section 8 states them as open work.

Section 5 also explains the three explicit first-order results beyond equal gaps. Their complete statements are in Appendix F; they do not provide the missing nonlinear statistical extension.

Question 5: Experimental support

Common response: C4 (experiments) and C5 (historical information).

A device may send a model checkpoint containing only a leading component, count and calibration scalars to reduce storage or transfer cost. The recipient then lacks both the historical observations and covariance. Retaining the covariance and count would instead suffice to compute pooled PCA. The synthetic study tests what the compressed state preserves under stationarity; real-data deployment and covariance drift remain untested.

Appendix H specifies how the historical empirical component and spectral estimates are fitted, the clipping interval [0.05,20][0.05,20], the leading-eigenvalue cap λ^1≤12Δ^(ν+1)\widehat\lambda_1\le\tfrac12\widehat\Delta(\nu+1), shared observations and Monte Carlo standard errors. Section 6 gives the setups and conclusions.

Question 6: AI-use disclosure

Common response: C1 (AI-use disclosure).

In the manuscript, we expanded the AI disclosure specifying assistance with formulation, proofs, synthetic-data implementation, experimental design, and drafting from task level.

Reviewer_FHKHOpenReview ↗

Follow-up: where is the rebuttal for the Weaknesses ?

Thank you for the replies and the revised manuscript. However, the author(s) didn't addresses the Critical Weaknesses in my review. I would like a direct response to the two weaknesses that still drive my assessment. The first is the writing of the manuscript. The second is how convincingly the paper demonstrates the significance of the problem and of the model used to study it, i.e., missing of real data experiments.

The revised paper still reads to me like a working manuscript that needs substantial polishing before it is ready for publication. Currently, it is more like written for AI, but not ready for human reading. When I pause at a formula and try to check it myself, I often have to backtrack through the manuscript to tell which symbols are vectors, which are matrices, and which are scalar parameters. On p. 4, Eq. (3.2) uses aia_i for a scalar projection, while Eq. (3.4) uses aka_k for an estimated direction. Consistent vector typography, distinct symbols for different types of objects, and definitions close to the equations would make the paper easier to read. I am retaining my original Presentation assessment.

The new experiments measure the discrepancy over the entire trajectory, which is an improvement. They still use the same rank-one isotropic Gaussian model. A real-data example would help establish why the proposed information constraint matters in practice and whether the method remains useful when the data depart from that model. In particular, could you demonstrate plug-in Oja on a real data stream where the update has access only to the retained checkpoint, with the full historical data reserved for an offline pooled PCA benchmark? I also understand that the current proof leaves the top-rr case open. Have you tested the corresponding update for r>1r>1, on real data or in a controlled simulation? Even an equal-gap simulation could show how closely it tracks the pooled top-rr projector over arrivals.

Reviewer_d5S4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies warm-started streaming PCA when the historical observations are unavailable but their empirical leading eigenvector, sample count, and spectral calibration have been retained. For a stationary Gaussian rank-one isotropic spike with comparable historical and incoming sample sizes, it proves that normalized Oja, projector-regularized PCA, and pooled empirical PCA agree to first order uniformly over incoming prefixes. The estimators share the sharp first-order PCA risk. Supporting results address plug-in calibration, pseudocount selection, growing dimension, and connections to empirical Riemannian flows.

Critical strengths

a) The main result compares the estimators on the same observations and controls the expected maximum discrepancy over all incoming prefixes. This is stronger than showing that streaming and pooled PCA have the same convergence rate or endpoint limiting distribution, which has been more commonly attempted in the literature.

b) The authors recognize that projective coordinates can have infinite unconditional moments, preserve the martingale structure through predictable localization, and use bounded projector loss to obtain unconditional conclusions.

Critical weaknesses

I will pose as questions:

b) Section 6 evaluates endpoints, while Theorem 3.1 concerns the maximum discrepancy over prefixes. Could the authors report this quantity, including for plug-in Oja, across sample sizes and old/new sample ratios?

c) Appendix B suggests that ηi=1/{Δ^(ν+i)}\eta_i=1/\{\widehat\Delta(\nu+i)\} already preserves the first-order guarantees. Does the additional λ^1\widehat\lambda_1 correction improve statistical performance, or mainly simplify the analysis?

Questions for authors

see above.

4 comments

Thank you for recognizing both the strength of the same-observation trajectory comparison and the use of predictable localization with bounded projector loss. We address your questions below.

Question b

Common response: C4 (numerical evidence).

We now measure the requested maximum over all arrivals, including plug-in Oja and the three ratios of incoming to historical sample sizes. Appendix H.1 and Table 4 give the full grid and the projector-regularized comparisons. The error bars are calculated across trial-level maxima, not substituted from endpoint errors.

Question c

Common response: C4 (step comparisons).

The correction makes the oracle effective step exactly harmonic: proportional to the reciprocal of the total observation count. Here ηi\eta_i is the step used in the normalized Oja update, while cic_i is the coefficient that appears when the normalized update is rewritten in local coordinates. They satisfy ci=ηi/(1+λ1ηi)c_i=\eta_i/(1+\lambda_1\eta_i). Thus the corrected oracle step 1/{Δ(ν+i)−λ1}1/\{\Delta(\nu+i)-\lambda_1\} gives ci=1/{Δ(ν+i)}c_i=1/\{\Delta(\nu+i)\}. The simpler plug-in raw step 1/{Δ^(ν+i)}1/\{\widehat\Delta(\nu+i)\} instead gives ci=1/{Δ^(ν+i)+λ1}c_i=1/\{\widehat\Delta(\nu+i)+\lambda_1\}. The true λ1\lambda_1 appears in this last formula because it describes the effective coefficient in the analysis; the algorithm uses the estimated gap. Both schedules satisfy the first-order guarantee for the clipped historical estimates in Corollary 3.2.

To assess statistical performance, Table 5 compares the two schedules on identical observations. For ν=n=100,200,400,800\nu=n=100,200,400,800, respectively, corrected minus simple normalized endpoint risks are 0.000035±0.0002740.000035\pm0.000274, 0.000112±0.0001310.000112\pm0.000131, −0.000059±0.000068-0.000059\pm0.000068, and 0.000040±0.0000310.000040\pm0.000031. Here normalized risk is NEL/AN\mathbb E L/A, with L(Pa,Pb)=12∥Pa−Pb∥F2L(P_a,P_b)=\tfrac12\|P_a-P_b\|_F^2 and A=5.25A=5.25 for this experiment. Each uncertainty is one Monte Carlo standard error, computed from the within-trial difference between the schedules.

Across all twelve cells the absolute mean difference is at most 0.00104. The ν=100,n=50\nu=100,n=50 difference is small but detectable (−0.001034±0.000234-0.001034\pm0.000234). The correction therefore provides exact oracle calibration; the experiments do not show a consistent finite-sample risk improvement.

Thank you again for your review. We hope our responses and the revised manuscript address your questions about the trajectory experiments and step-size correction. We would appreciate your feedback and reconsideration of your assessment.

The central result shows what compressed historical information preserves during continued learning: one-pass Oja follows pooled PCA’s first-order trajectory throughout the incoming stream, with sharp risk and linear storage and work per arrival. The added first-order results beyond equal gaps further broaden this picture. We believe this combination offers a substantive contribution to understanding streaming learning algorithms.

I thank the authors for the response. It addresses my main questions. I have accordingly raised my score.

Thank you for your careful review and updated assessment. We appreciate your confirmation that our response addressed your main questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response (Part I)

Thank you for the careful reading and specific questions. We address the shared concerns in C1-C5, then respond to each reviewer with the relevant common point and any additional detail. In the marked manuscript, technical additions and new discussion are red; explanatory rewrites are black. References below use the revised manuscript's numbering.

C1. Clarity and AI-use

Reviews have mixed comments on clarity of the paper. Reviewer d5S4 found the paper easy to follow while reviewer FHKH identified unclear estimator roles, difficult conceptual connections, and insufficient task-level AI disclosure. To address this issue, we manually rewrote almost all paragraphs that were previously generated by AI in the initial submission, and made terminology and notation consistent throughout the paper. We hope this modification can overcome the reservation pointed out by reviewer FHKH.

In specific, we have rewritten the introduction to explain the historical-data restriction and the three methods before introducing the trajectory comparison. Related work now precedes the contribution list. Oja is the streaming procedure; projector-regularized PCA and pooled PCA are refitted comparators. These terminologies are fixed throughout the paper. The proof sketch now explains the shared first-order expression for estimation error and how the coordinate calculations give projector bounds averaged over all sampled datasets. Experimental protocols and numerical detail are consolidated in Appendix H.

To address the Question 6 of reviewer FHKH, we expanded the AI disclosure specifying assistance with formulation, proofs, synthetic-data implementation, experimental design, and drafting from task level.

C2. Rates, calibration and model restrictions

The reviews ask for explicit comparison rates, sensitivity to calibration, and a clearer account of what can extend beyond equal gaps. We address these with three changes:

  • Explicit rates: For fixed dimension, Equations (3.4)-(3.5) state the rates already supported by the proof: O(N−2)O(N^{-2}) for Oja's localized remainder and O(log⁡N/N2)O(\log N/N^2) for both PCA remainders and the expected maximal pairwise projector discrepancy. The logarithm comes from the simultaneous covariance bound used in the proof, not a demonstrated separation between estimators. Comparisons using clipped historical estimates and the risk expansion retain o(N−1)o(N^{-1}) remainders.
  • Beyond equal gaps: Propositions F.2-F.4 give nearly equal-gap linear recursions, preconditioning by the inverse eigengaps with a corresponding first-order comparison to PCA under a matrix penalty, and averaging that includes the historical estimate. Preconditioning assumes a known noise covariance BB satisfying Bv=bvBv=bv and the true leading eigenvalue λ1\lambda_1 for calibration. These are explicit first-order results; uniform nonlinear statistical extensions remain open. Under its stated starting-error assumptions, Proposition F.1 shows why a scalar harmonic schedule cannot generally match pooled PCA when the eigengaps differ.
  • Calibration sensitivity: Proposition B.1 quantifies how gap errors affect the linear error recursion's leading risk. The relative cost is quadratic near correct calibration. This is not a nonlinear misspecified-model theorem.

The main theorem retains stationary rank-one isotropic Gaussian sampling and comparable old/new sample sizes. The growing-dimensional extension covers projector risk and discrepancy; distributional and local asymptotic minimax claims remain fixed-dimensional.

AuthorsOpenReview ↗

Common Response (Part II)

C3. Why the empirical flow is relevant to Oja

The flow makes the common first-order weighting explicit. At tk=Δ−1log⁡(1+k/ν)t_k=\Delta^{-1}\log(1+k/\nu), its weights on the historical error and each new observation's first-order contribution equal those of Oja and the PCA fits. Corollary 4.1 bounds the expectted maximal squared projector discrepancy between the flow evaluations and oracle Oja by O(N−2)O(N^{-2}). Weight matching alone would not establish this; the nonlinear remainder bounds are also needed.

Each flow holds the first kk new observations fixed and restarts from the historical empirical component uu. Its calibrated evaluation is compared with Oja's continuing recursion. Corollary 4.2 fixes the final dataset, compares this same flow with the regularized PCA fits over fixed bounded time intervals, and identifies a stopping time attaining pooled PCA's first-order risk. These batch flows interpret the calibration; the proof for actual Oja remains discrete.

C4. Evidence for the trajectory claim

The earlier endpoint tables did not test the theorem's maximum over arrivals. The new experiment computes that maximum within each trial before averaging, with 500 trials for each ν∈{100,200,400,800}\nu\in\{100,200,400,800\} and n/ν∈{1/2,1,2}n/\nu\in\{1/2,1,2\}. All methods see identical observations within a trial. As ν=n\nu=n grows from 100 to 800, NN times the mean maximal full squared Frobenius projector discrepancy from pooled PCA decreases from 0.6439±0.06520.6439\pm0.0652 to 0.05485±0.002520.05485\pm0.00252 for oracle Oja, and from 0.3701±0.02990.3701\pm0.0299 to 0.03098±0.001110.03098\pm0.00111 for plug-in Oja. Uncertainties are one Monte Carlo standard error.

Comparisons of step schedules on identical data, exact empirical-flow evaluations and controlled gap errors provide additional evidence. The step correction has small, mixed-sign risk effects on this grid; flow times use the true eigengap. Appendix H, Tables 4-5 and Figure 1 give the full results and uncertainty. Code and saved per-trial records accompany the revision. These simulations illustrate the claimed quantities; they do not establish sharper rates, anisotropic robustness or real-stream performance.

C5. Contribution and significance

Our trajectory theorem establishes more than equal accuracy at an endpoint: actual normalized Oja, projector-regularized PCA and pooled PCA agree to first order on the same observations throughout continuation. The expected largest squared projector discrepancy is below the statistical risk scale, so the theorem identifies agreement in the evolving estimates themselves. All three reviewers recognize strengths in this comparison: d5S4 calls the guarantee "stronger than showing that streaming and pooled PCA have the same convergence rate or endpoint limiting distribution", and EGPV notes that the precisely scoped comparison "does appear to be unoccupied".

The statistical significance is that historical covariance storage is unnecessary for first-order continuation of the leading projector in the stated fixed-dimensional, stationary rank-one isotropic Gaussian model. The retained empirical component, count and spectral calibration preserve the old sample's leading contribution even when historical and incoming samples are comparable in size. Oja carries that particular historical fluctuation forward with the correct weight. The compact summary therefore preserves what continued leading-projector estimation needs at first order, despite not generally determining the historical covariance.

The computational payoff is equally concrete: Oja delivers pooled PCA's sharp first-order risk and follows its realized trajectory using O(d)O(d) retained historical state, running state and work per arrival; the pooled-covariance benchmark retains an O(d2)O(d^2) historical summary. This gives a quantitative guarantee that one-pass continuation can replace repeated full-information refitting at the leading statistical scale. The paper thus establishes when a standard streaming algorithm can retain the evolving batch estimator's statistical accuracy after the historical covariance has been discarded.

Area_Chair_gpraOpenReview ↗

Reviewer–Author Discussion Period Ending Soon

Dear Reviewers,

With the discussion period coming to a close, I encourage you to revisit the authors’ responses and let us know if you have any further comments.

Thank you for your careful reviews and engagement in the discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.