acceptodds
Withdrawn from ICLR 2027

Doubly Robust Principal Component Analysis through Robust Latent Subspace with Bounded Noise

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Principal component analysis (PCA) is a fundamental technique for dimensionality reduction and the recovery of underlying data structure, with broad applicability across real-world scenarios. To enhance its robustness against noise and outliers, numerous robust PCA (RPCA) methods have been proposed. However, most existing approaches focus predominantly on the input space, overlooking the potential presence of noise in the latent space. Moreover, their optimization typically relies on alternating updates across multiple matrix variables, such as the alternating direction method of multipliers (ADMM), which is prone to premature convergence and suboptimal solutions. To overcome these limitations, we introduce a novel doubly robust PCA approach, termed DRPCA, which jointly addresses robustness in both the input and latent spaces. By deriving upper bounds on the DRPCA objective, we further establish a theoretical connection between model robustness and sparsity-inducing regularization. To efficiently solve the resulting models, we propose a hybrid alternating optimization algorithm grounded in parameter-free exact solvers, which decomposes the original DRPCA problem into tractable subproblems, integrating rapid matrix-wise updates with element-wise updates that admit closed-form, globally optimal solutions. A prescreening-based acceleration strategy is additionally incorporated to expedite the optimization process without compromising solution quality. Extensive experiments on benchmark datasets of varying scales demonstrate the superior performance of DRPCA over state-of-the-art RPCA methods. Notably, the proposed hybrid optimization algorithm consistently surpasses its matrix-wise-only or element-wise-only counterparts, yielding better solutions and higher-quality reconstruction.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unsupervised Learning, Robust Principal Component Analysis, Structured Sparse Learning, Data Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LYvrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new optimization for robust PCA and a strategy to solve it. In doing so, they introduce two RPCA variants. In experiments, they find their method is robust and produces low reconstruction error.

Critical strengths

The paper contributes to the well-studied problem of robust PCA by contributing a new optimization problem and an algorithm to approximate it's solution. Overall, the message is simple and self-contained, contributing to an active field of research.

Critical weaknesses
  1. There are some mathematical details that need clarification.

  2. Some work including dual principal component pursuit and robust subspace analysis were missed as related works and potential baselines.

  3. The work is missing rigorous ablation studies and assessment of the method compared to baselines under various metrics. This is especially important to see that kind/ quantity of noise and outliers break this method.

Questions for authors

Mathematics

  • L99 Maybe better to write V = U^\top X
  • L102 Cite or explain your statement about Laplacian noise
  • L136 We switch from a data matrix X to a data matrix (X,b) what is b?
  • Lemma 3 is an upper bound for ||\Delta U V|| but in the proof of theorem 1, we reference lemma 3 for an upper bound of ||U \Delta V^\top||_1. Where is the upper bound of ||U \Delta V^\top||_1 proven?
  • L237 Maybe say the Frobenius norm of difference is 0, rather than restrict the difference to 0 since this is what is actually used to formulate the Lagrangian.
  • L237 Shouldn’t there be orthogonality constraints on U for it to be PCA?

Related works and citations

  • Inline
    • L39 Hotelling, Analysis of a complex of statistical variables into principal components
    • L247 Has this strategy been used before? If so, please cite.
  • Introduction and/ or potential baselines to compare to
    • Gilad et al, Fast, robust and non-convex subspace recovery
    • Gilad et al, A novel m-estimator for robust pca
    • Gilad et al, An overview of robust subspace recovery
    • Gilad et al, A well-tempered landscape for non-convex robust subspace recovery
    • Vidal et al, Robust principal component analysis
    • Vidal et al, Generalized principal component analysis (gpca): an algebraic geometric approach to subspace clustering and motion segmentation
    • Vidal et al, Generalized principal component analysis
    • Mankovich et al, Fun with Flags: Robust Principal Directions via Flag Manifolds

Experiments

  • Tab 1 Maybe use relative reconstruction error rather than raw reconstruction error?
  • Tab 1 & Fig 1 Was only one noise value checked? An appropriate evaluation of this method would be a noise ablation study as well as an outlier ablation study.
  • Fig 1 The text in these plots is too small and is unreadable in a printed version of the text. The legend, axis ticks and axis labels should be increased for readability.

Smaller points:

  • some citations appear as links and others do not
  • there is not AI nor repreocibility statement
  • Do you have any intuition for why this method is more robust than other methods?
Reviewer_EYn9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper starts from ℓ1\ell_1-PCA and allows bounded perturbations on the columns of both factor matrices UU and VV. It then minimizes the worst-case reconstruction error over those perturbations (Eq. 2). Because the inner maximization is hard, the authors bound it with the triangle inequality (Theorems 1–2). After dropping a constant, this gives an ℓ1\ell_1 loss with ℓ1\ell_1 penalties on UU and VV (D1RPCA) or ℓ2,1\ell_{2,1} penalties (D21RPCA).

The surrogate is optimized in two stages. ADMM gives a starting point. Each entry of UU and VV is then updated in turn by an exact one-dimensional solver: a weighted median for ℓ1\ell_1, and bisection for ℓ2,1\ell_{2,1}. A residual-based screening rule limits which entries of VV are refined, which saves time.

Experiments use eight face datasets with synthetic occlusions and grayscale 32×3232\times 32 subsets of CIFAR-10 and ImageNet, against 13 PCA/RPCA baselines. A DRPCA variant has the lowest ℓ1\ell_1 test error in all reported settings.

Critical strengths

S1. The results are consistent, and the comparison is broad. One of the two DRPCA variants has the lowest ℓ1\ell_1 test error in every setting of Table 1 and Table 4, against 13 baselines that include recent methods such as eRPCA, DW-RPCA and DCA. On the face data the margin is clear: for example, 11,410 against 13,262 for L1-PCA (ALM) on CMU with single-type noise. For me this shows that the model, together with the refinement step, works well for removing occlusions in this setting.

S2. The refinement step is practical. Each one-entry subproblem is solved exactly and needs no step size; the weighted-median version costs only O(qlog⁡q)O(q \log q). I also liked that Fig. 2 compares ADMM alone, the exact solver alone and the two together, which separates the effect of the optimizer from that of the model. With acceleration, the hybrid D1RPCA is even faster than L1-PCA (ALM) on ImageNet-5000 (about 220 s against 282 s), so the method can scale reasonably.

Critical weaknesses

W1. The new part of the method is what happens after ADMM, and that part is close to existing work, with weaker guarantees than the paper claims.

The ADMM stage (Eq. 7) is the usual splitting, and the paper itself relates it to earlier ADMM work (L219–221). I have no problem with using ADMM, but since ADMM has already been applied to very similar ℓ1\ell_1 factorization problems, the contribution has to come from the model and the element-wise refinement. Both have close relatives. The model (an ℓ1\ell_1 loss with sparse factors) is close to robust sparse PCA (Croux, Filzmoser & Fritz, 2013). The weighted-median updates are close to the cyclic weighted median method (Meng, Xu, Zhang & Zhao, AAAI 2013). Neither is cited.

Theorems 3–4 give the exact optimum of one entry when all other entries are fixed, but nothing is shown about the joint optimality of (U,V)(U, V), or even about stationarity. Still, the abstract and the introduction (L69–71) speak of "the global optimum for each element" and less premature convergence, and the paper has no limitations section where this is acknowledged.

I also could not find support for the claim that prescreening keeps solution quality. There is no comparison with and without it, and the appendix itself says its effect on quality would "require empirical evaluation" (L1665–1666).

Finally, L196–197 ("guaranteed to decrease the original loss") and L207 ("mathematically equivalent") claim more than an upper bound can give. Known exact links between robustness and regularization (Xu, Caramanis & Mannor; Bertsimas & Copenhaver, 2018) are not cited.

Taken together, the two main stated contributions, a theoretical link and a new optimizer, are either not fully established or not clearly new.

W2. The experiments do not test the motivation, and the evaluation protocol is unclear.

All corruptions are occlusions in pixel space, and no experiment perturbs the latent factors. The gains could therefore come simply from sparsity regularization.

I am also unsure how λ\lambda was chosen. Sec. 5.1 says λ\lambda "follows Section 5.3", where the best λ\lambda is the one with the lowest error against the clean data X0X_0 (Figs. 3, 12, 13). If the main tables use this choice while the baselines keep their default settings, the comparison favours DRPCA.

No variance over corruption seeds is reported, and some wins are very small (375,506 against 375,687 on ImageNet-10000). On ImageNet-5000 (Table 5), DRPCA is best on ℓ2\ell_2 error but not on SSIM, PSNR or LPIPS, and its SSIM is even below plain L2-PCA.

For these reasons I cannot tell how much of the reported gain comes from "latent robustness".

Questions for authors

Q1. This first question is the most important one for me, and the answer could change my soundness score. How were λ\lambda and kk chosen for Tables 1, 4 and 5? Was the error against the clean X0X_0 used for DRPCA, for the baselines, or for both?

Q2. This question is about novelty. How does your element-wise solver differ from the cyclic weighted median method (Meng et al., AAAI 2013)? And how does DRPCA differ from sparse ℓ1\ell_1-PCA without the robust framing?

Q3. I would like to see evidence for the prescreening claim. Can you report the reconstruction error with and without prescreening for a few values of γ\gamma?

Q4. Can anything be said about stationarity or joint optimality of (U,V)(U, V)? If not, could you state this as a limitation and revise L69–71, L196–197 and L207?

Q5. To make the paper easier to follow, please define single-type and mixed-type noise in Section 5.1, and give the dataset, noise, rank and λ\lambda used for Figure 1.

Q6. This last point is a suggestion rather than a question. An experiment with noise injected into UU or VV, with mean ±\pm std over several corruption seeds, would directly support the latent-robustness claim.

Minor: there are grammar slips at L71, L161, L175, L196–197 and L304, and the ordering claim in App. F.3 does not match Table 5. In some PDF viewers the math labels in Figure 1 show as "?" boxes, so please embed the fonts or convert the text to outlines.

Reviewer_bDLmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of robust PCA. The main objective is to optimize a minmax error, in the space of matrix factors. To ease the computation, authors consider minimizing relaxed upper bounds, and they considered two types. Alternating minimization is then used to obtain progressively improved solutions.

Critical strengths

I think the paper is easy to follow. Some experiments are conducted to show the performance of the algorithm.

Critical weaknesses

The problem itself, robust PCA, has been well established. The technical contribution is very limited. Indeed, I cannot extract certain novelty from this submission. For example, the notion of doubly robustness is not new, the upper bound relaxation is known, and alternating minimization has been broadly understood.

On the empirical side, it is unclear what modern applications will benefit from the proposed algorithm.

Questions for authors

See weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.