- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of recovering linear relation using noisy measurements with unknown noise covariance which may be non-diagonal and/or singular. It proposes a method GenEVD, which covers IterativePCA and OLS as special cases, where the key steps are alternating between a solving a generalized eigenvector problem, and updating the estimate of . Theoretical results on finite-sample error bounds are provided. Experiment results are shown on datasets with synthetic noise injections.
Critical strengths
- The proposed GenEVD method provides a promising improvement over the widely used IPCA, with statistical guarantees.
- The theoretical analysis is solid and of interest to the community. In particular, Theorem 8 provides an end-to-end recovery bound for the proposed algorithm and greatly enhances the rigor of the proposed method. The proof is correct as far as I could check.
Critical weaknesses
- All numerical simulation results shown are on datasets with synthetic Gaussian noise. The real data experiment does not directly use the raw dataset, but first preprocess it by projection into low-rank. While this serves its purpose of demonstrating theoretical results, it's still valuable and important to understand how the proposed method performs on real data, e.g. how well it can handle sparse noise and data that are not perfectly low rank.
Questions for authors
- Can you provide simulation results on real-world datasets, compared with other baseline methods? At the minimum you can create synthetic experiments to test the sensitivity of performance to data suboptimality patterns (e.g. sparse noise, non-perfect low rank), but real data experiments are still valuable.
- Minor: GenEVD has not been defined before Lemma 1, which already references it. Can you reorganize the flow or reword Lemma 1 to make this clearer?
4 comments
Rebuttal 2/n
Table R10: Our proposed method outperforms others under outliers (synthetic data).
| OLS | Std PCA | RPCA | GoDec | HeteroPCA | Deflated-HeteroPCA | GenEVD (Ours) | |
|---|---|---|---|---|---|---|---|
| 500 | |||||||
| 2000 | |||||||
| 5000 |
Table R11: Our proposed method outperforms others when the signal covariance is approximately low rank (synthetic data).
| OLS | Std PCA | RPCA | GoDec | HeteroPCA | Deflated-HeteroPCA | GenEVD (Ours) | |
|---|---|---|---|---|---|---|---|
| 500 | |||||||
| 2000 | |||||||
| 5000 |
Inference.
-
Non-Gaussian noise distributions (Table R9): the error of GenEVD is for every distribution, the same as for Gaussian noise (Table 2, , ), and it remains lower than that of the other methods.
-
Outliers with (Table R10): GenEVD performs better than the other methods at every .
-
Approximately low-rank signal covariance with (Table R11): we make a similar observation at every .
Our method performs better on real-world datasets for non-gaussian noise or outliers or approximate low rank
Can you provide simulation results on real-world datasets, compared with other baseline methods? At the minimum you can create synthetic experiments to test the sensitivity of performance to data suboptimality patterns (e.g. sparse noise, non-perfect low rank), but real data experiments are still valuable.
A:
Real-world datasets. We repeat these experiments on the seven real-world datasets of Table 3, with the same rank-two projection, noise covariance, support and samples, and change one aspect at a time. We report the largest of each dataset.
-
Non-Gaussian noise distributions (Tables R12–R15: uniform, Rademacher, Laplace and Student-) and outliers with (Table R16) are generated as above.
-
Approximate low rank (Table R17): the signal is , where is the rank-two projection of the real data and is the real component removed by it. We choose so that gives .
Table R12: Our proposed method outperforms others for uniform noise on real-world datasets.
| Dataset | OLS | Std PCA | RPCA | GoDec | HeteroPCA | Deflated-HeteroPCA | GenEVD (Ours) | |
|---|---|---|---|---|---|---|---|---|
| Energy | 768 | |||||||
| Air Quality | 8991 | |||||||
| Appliances | 19735 | |||||||
| Gas Drift | 13910 | |||||||
| MHEALTH | 10000 | |||||||
| Power Plant | 9568 | |||||||
| Household | 20000 |
Rebuttal 3/n
Table R13: Our proposed method outperforms others for Rademacher noise on real-world datasets.
| Dataset | OLS | Std PCA | RPCA | GoDec | HeteroPCA | Deflated-HeteroPCA | GenEVD (Ours) | |
|---|---|---|---|---|---|---|---|---|
| Energy | 768 | |||||||
| Air Quality | 8991 | |||||||
| Appliances | 19735 | |||||||
| Gas Drift | 13910 | |||||||
| MHEALTH | 10000 | |||||||
| Power Plant | 9568 | |||||||
| Household | 20000 |
Table R14: Our proposed method outperforms others for Laplace noise on real-world datasets.
| Dataset | OLS | Std PCA | RPCA | GoDec | HeteroPCA | Deflated-HeteroPCA | GenEVD (Ours) | |
|---|---|---|---|---|---|---|---|---|
| Energy | 768 | |||||||
| Air Quality | 8991 | |||||||
| Appliances | 19735 | |||||||
| Gas Drift | 13910 | |||||||
| MHEALTH | 10000 | |||||||
| Power Plant | 9568 | |||||||
| Household | 20000 |
Table R15: Our proposed method outperforms others for Student- noise on real-world datasets.
| Dataset | OLS | Std PCA | RPCA | GoDec | HeteroPCA | Deflated-HeteroPCA | GenEVD (Ours) | |
|---|---|---|---|---|---|---|---|---|
| Energy | 768 | |||||||
| Air Quality | 8991 | |||||||
| Appliances | 19735 | |||||||
| Gas Drift | 13910 | |||||||
| MHEALTH | 10000 | |||||||
| Power Plant | 9568 | |||||||
| Household | 20000 |
Rebuttal 4/n
Table R16: Our proposed method outperforms others under outliers on real-world datasets.
| Dataset | OLS | Std PCA | RPCA | GoDec | HeteroPCA | Deflated-HeteroPCA | GenEVD (Ours) | |
|---|---|---|---|---|---|---|---|---|
| Energy | 768 | |||||||
| Air Quality | 8991 | |||||||
| Appliances | 19735 | |||||||
| Gas Drift | 13910 | |||||||
| MHEALTH | 10000 | |||||||
| Power Plant | 9568 | |||||||
| Household | 20000 |
Table R17: Our proposed method outperforms others when the data are not perfectly low rank (real-world datasets).
| Dataset | OLS | Std PCA | RPCA | GoDec | HeteroPCA | Deflated-HeteroPCA | GenEVD (Ours) | |
|---|---|---|---|---|---|---|---|---|
| Energy | 768 | |||||||
| Air Quality | 8991 | |||||||
| Appliances | 19735 | |||||||
| Gas Drift | 13910 | |||||||
| MHEALTH | 10000 | |||||||
| Power Plant | 9568 | |||||||
| Household | 20000 |
Inference on the real-world datasets.
-
Non-Gaussian noise distributions (Tables R12–R15): we make the similar observation of GenEVD performing better than the other methods on every dataset.
-
Outliers with (Table R16): GenEVD again compares favorably on every dataset.
-
Approximately low-rank signal covariance with (Table R17): the same holds on every dataset.
We use semi-synthetic data because raw data do not provide the true constraints and noise covariance, which are needed to measure recovery. We can include these new experiments in the main manuscript.
Minor: GenEVD has not been defined before Lemma 1, which already references it. Can you reorganize the flow or reword Lemma 1 to make this clearer?
A: We thank the reviewer for this suggestion. We will define GenEVD before Lemma 1: given a noise-covariance estimate , GenEVD estimates by the span of the generalized eigenvectors of whose finite eigenvalues are closest to one, computed by the QZ algorithm, and Algorithm 1 alternates this step with an update of .
Rebuttal 1/n
We thank the reviewer for checking the proof and for the constructive comments. We have conducted new experiments with non-Gaussian noise distributions, outliers, and approximately low-rank data, as requested by the reviewer.
We also summarize the new experiments conducted for Reviewer 6waE; the details are in our response to that reviewer.
Initialization (Tables R1–R3). From six starts, including random ones, GenEVD reached the same estimate in every run.
Number of relations (Table R4). Compared with thresholding the spectrum of Sy, our method never returned a wrong m, including Example A, where thresholding was wrong in every trial.
OLS given the noise-free variables (Table R5). GenEVD coincides with OLS when only the response is noisy, and recovers the coefficients when a predictor is also noisy, where OLS is biased.
Larger dimensions (Table R6). At d=50 and d=100, GenEVD achieves a lower error than the other methods.
We kindly request the reviewer to let us know if they have any further questions.
A: We thank the reviewer for the summary. In continuation, the method also estimates the number of relations m by counting the generalized eigenvalues equal to one (Algorithm 2), with a guarantee (Theorem 6).
A: We appreciate the reviewer’s recognition. GenEVD coincides with IPCA when Σϵ≻0 (Lemma 1), and it still applies when some variables are noise-free, where IPCA is undefined.
A: We sincerely thank the reviewer for checking the proof.
Our method performs better on synthetic datasets for non-gaussian noise or outliers or approximate low rank
A: We thank the reviewer for this constructive comment. We ran experiments with non-Gaussian noise distributions, outliers, and approximately low-rank data, and observed that GenEVD remains accurate. The setup, results, and inference are presented below.
Setup. Case Study 1 (d=5, m=3) with s=1 noise-free variable, 50 trials, all baselines of Table 2. We change one aspect of the data at a time:
Noise distribution (Table R9): the entries of the standardized noise follow the uniform, Rademacher, Laplace or Student-t5 distribution, with the same covariance Σϵ.
Outliers (Table R10): ±10 noise standard deviations on a fraction p=0.1 of the entries of the noisy variables, on top of the dense noise. This is the sparse noise mentioned by the reviewer.
Approximate low rank (Table R11): Σx=Σx′+ξPC with ξ=0.01,σC2, so the relations hold only up to a residual variance ξ.
Metric. θmax (degrees) between the estimated and the true constraint subspaces, as in Table 2; lower is better.
Table R9: Our proposed method outperforms others for non-Gaussian noise (synthetic data).