acceptodds
Desk rejected at ICLR 2027

Contrastive Principal Component Regression

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Positive pairs reveal reproducible variation, but the most reproducible directions need not be the most predictive. We introduce contrastive principal component regression (C-PCR), which combines response association and pair alignment to learn a low-dimensional representation, then fits ridge regression for prediction from a single measurement. Under a paired-factor model, we characterize when this joint selection preserves a predictive direction that selection based only on reproducibility would discard. We derive a computable prediction-risk limit as feature dimension and sample size grow proportionally. With one predictive factor and one retained component, the scalar-response formula accounts for using the same responses in component selection and coefficient fitting. The formula separates predictive-direction error from accumulated error elsewhere. Simulations support it and demonstrate finite-sample prediction gains from combining pairing with supervision. Across five naturally paired spectroscopy datasets, our method achieves the best predictive performance or remains competitive with the best alternative.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Prinicipal Component Regression, positive pairs, random matrix theory, contrastive learning, prediction risk

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SJxAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces C-PCR, which extends PCA++ to incorporate response information into component selection. Under a linear paired-factor model with shared latent factors across two views of each unit and outcomes depending linearly on those factors, the authors characterize when supervised selection retains a predictive direction that pair-only selection discards. They derive an asymptotic prediction-risk formula for one retained component, extend the analysis to multiple responses and components, and evaluate the method on five paired spectroscopy datasets. The method is potentially useful, and the theoretical analysis provides insight into its prediction risk. However, the methodological contribution appears to be a relatively direct supervised extension of PCA++. The substantial tuning burden associated with four hyperparameters, limited breadth of the experimental evaluation, and modest real-data gains over the competing methods limit the significance of the contribution.

Critical strengths

• A useful supervised extension of PCA++ for naturally paired measurements. • A multi-response version (Appendix D) that could serve as an alternative to PLS. • Theoretical investigation and asymptotic prediction-risk characterization.

Critical weaknesses

• Limited methodological novelty. The estimator adds a response-association term to the PCA++ criterion; it seems to be a direct extension of PCA++. • Table 1 does not clearly demonstrate an advantage. C-PCR is best on 3 of 5 datasets, by ≤ 0.005 relative MSE on Raman and Beef. Mango is the only dataset with a clear gain over PCA++, but the performance is still very close to Ridge. • Tuning burden and practicality. The method has four tuning parameters (q, λ, τ, ρ), and preprocessing is additionally selected by validation on four of the five datasets. Table 7 reports more than 100,000 candidate evaluations per fitting split for C-PCR on four datasets, versus far fewer evaluations for PLS and PCR. It is mentioned that fits are reused, but no runtime is reported, and I doubt the practicality for larger datasets. • Simulations are narrow relative to the real data. All simulated settings have p/(2n) ≤ 1 (maximum γ = 1 in Fig. 2b; γ = 1/3, 0.2 and 0.5 elsewhere), whereas Raman, Corn and Tablets have γ > 1.

Questions for authors

• As stated in the weaknesses, the authors should expand the simulation studies to investigate the performance of the method in diverse scenarios. • The computational burden of the tuning parameter selection should be clearly reported. • When I looked at Table 6, the fitting dataset size is quite small compared with the validation and test sets (e.g., Raman: 50, 150, 1088). If more samples were used for training, so that n increases and γ decreases, I wonder whether the relative performance among the competing methods would change.

6 comments
AuthorsOpenReview ↗

What C-PCR adds beyond PCA++ and consistency ridge, with 150 random splits, 18 datasets, runtimes and simulations up to γ = 4

We thank the reviewer for the careful review. The concerns come down to two questions: is C-PCR more than PCA++ with a response term, and are its gains real and practical? We answer the first below. For the second, we ran the larger experiments the reviewer asked for: 150 random splits per dataset instead of the original fixed splits, 13 more datasets, simulations up to γ=4\gamma=4, and timings up to p=4000p=4000.


Limited methodological novelty. The estimator adds a response-association term to the PCA++ criterion; it seems to be a direct extension of PCA++.

C-PCR does add a response term to the PCA++ criterion, but the term changes where the response enters: at truncation, not after it.

  • PCA++-PCR keeps qq directions without the response and only then regresses, so a direction dropped at truncation cannot be recovered, and a weakly shared but predictive direction is exactly what unsupervised truncation drops.

  • Retaining more directions avoids this only with a wider representation: in Appendix E.5, PCA++-PCR needs q0≥32q_0\geq32 pair directions before supervised compression to come close to C-PCR, which still has lower risk (Table 7).

  • C-PCR, in contrast, uses the response in the truncation itself, so it keeps the predictive direction with a single component.

Both criteria are generalized eigenproblems in the same metric, max⁡U⊤Σ^τU=Iq⟨Σ^ct,UU⊤⟩⏟PCA++vsmax⁡U⊤Σ^τU=Iq⟨b^b^⊤+λΣ^ct,UU⊤⟩⏟C-PCR,\underbrace{\max_{U^\top\widehat\Sigma_\tau U=I_q}\bigl\langle\widehat\Sigma_{\rm ct},UU^\top\bigr\rangle}_{\text{PCA++}} \qquad\text{vs}\qquad \underbrace{\max_{U^\top\widehat\Sigma_\tau U=I_q}\bigl\langle\widehat b\widehat b^\top+\lambda\widehat\Sigma_{\rm ct},UU^\top\bigr\rangle}_{\text{C-PCR}}, and the added term has three consequences.

First, the added term changes which direction survives truncation. If the predictive direction u1∗u_1^* is weaker in the pair term than a shared distractor u2∗u_2^* (η2>η1\eta_2\gt \eta_1), Proposition 3.1 gives PCA++ selects u2∗for every τ⟹R=Var⁡(y)for every τ,ρ,λ<λc⟹C-PCR selects u1∗,\text{PCA++ selects }u_2^*\\ \text{for every }\tau\\ \Longrightarrow\\ \mathcal R=\operatorname{Var}(y)\\ \text{for every }\tau,\rho, \qquad \lambda\lt \lambda_c\\ \Longrightarrow\\ \text{C-PCR selects }u_1^*, so PCA++ misses the predictive direction under every tuning, while C-PCR finds it; with each method at its own optimum, C-PCR’s risk is lower by a1ϑ2/(1+a1)a_1\vartheta^2/(1+a_1).

Second, it changes the selected direction itself. For one component, C-PCR selects the direction of a ridge fit that also penalizes disagreement between the two views, with penalties set by the selected eigenvalue (Proposition B.8, Appendix B.6). The connection explains why C-PCR works: the pair term acts as a penalty on disagreement between the two measurements of a unit. Because the penalties are set by each component’s eigenvalue, directions with a strong response signal are penalized less and weak ones are pushed harder toward agreement, and, whereas consistency-regularized reduced-rank regression fixes only LL directions and leaves the rest arbitrary, C-PCR ranks the components beyond the response rank by pair alignment (Result B.9); these components carry most of C-PCR’s gain on Raman (Table 14).

Third, it has its own risk theory: Theorem 4.2 and Proposition D.4 give the exact limiting risk of the full procedure, in which the same responses both select the direction and fit its coefficient. PCA++ theory does not account for this.


Table 1 does not clearly demonstrate an advantage. C-PCR is best on 3 of 5 datasets, by ≤0.005\le 0.005 relative MSE on Raman and Beef. Mango is the only dataset with a clear gain over PCA++, but the performance is still very close to Ridge.

With 150 random splits per dataset, C-PCR is significantly better than both no-pair and ridge on four of the five datasets, and across all 18 datasets it is better than every comparator on most datasets. We agree that the original fixed splits could not separate margins of this size. Median rMSE reduction of C-PCR against the two strongest baselines (%):

AgainstRamanBeefTabletsCornMango
No-pair+5.9+1.1+12.3−0.1-0.1+4.7
Ridge+6.8+0.9+14.3−2.1-2.1+5.5

Continued: Against both, the 95% intervals over splits lie above zero on four of the five datasets (Table 13), including the small margins on Raman and Beef; on Corn, both baselines are better (ridge by a median of 2.1%, no-pair by 0.1%). On Corn, C-PCR thus falls back almost to no-pair, as designed: because λ=0\lambda=0 lies in its grid, validation can return the supervised fit, so the method should match the best supervised method when the pairs carry no extra information and improve on it when they do. These intervals reflect variability over random splits, not sampling of the dataset itself. Across all 18 datasets (Tables 12 and 13), C-PCR is better than each of the nine comparators below on most datasets (one-sided Wilcoxon tests over datasets, p≤0.03p\leq0.03 each):

C-PCR againstBetter / worsepp
PCR18 / 03.8×10−63.8\times10^{-6}
PLS15 / 39.5×10−59.5\times10^{-5}
Ridge15 / 35.2×10−35.2\times10^{-3}
Elastic net14 / 40.01
No-pair14 / 44.5×10−34.5\times10^{-3}
Ji et al. (linear)17 / 01.5×10−41.5\times10^{-4}
PCA++-PCR15 / 03.3×10−43.3\times10^{-4}
PCA++, then supervised screening16 / 27.2×10−57.2\times10^{-5}
PCA++ with no-pair directions12 / 60.03

Appendix F.6 documents the protocol of the original fixed splits, including how the Raman preprocessing and the Tablets fit were selected.


Tuning burden and practicality. The method has four tuning parameters (qq, λ\lambda, τ\tau, ρ\rho), and preprocessing is additionally selected by validation on four of the five datasets. Table 7 reports more than 100,000 candidate evaluations per fitting split for C-PCR on four datasets, versus far fewer evaluations for PLS and PCR. It is mentioned that fits are reused, but no runtime is reported, and I doubt the practicality for larger datasets. The computational burden of the tuning parameter selection should be clearly reported.

The candidate count overstates the cost, because almost all candidates reuse earlier work. The search reuses work at every level (Result B.5, Algorithm 2; r=min⁡(p,2n)r=\min(p,2n)):

StepHow oftenCost each
SVD and projectionsonceO(n+nval),p,rO\\{(n+n_{\rm val})\\,p\\,r\\}
Eigenproblemonce per (τ,λ)(\tau,\lambda) pairO(r3)O(r^3)
Refit and validationonce per candidateO(q3+nval,q)O(q^3+n_{\rm val}\\,q)

The refit cost does not depend on pp and grows only linearly with the validation size nvaln_{\rm val}, so for large data, where r≫qr\gg q, T=O\bigl\\{(n+n_{\rm val})\\,p\\,r+N_{\rm eig}\\,r^3\bigr\\},\qquad N_{\rm eig}=\text{number of }(\tau,\lambda)\text{ pairs}. Thus pp enters only through the SVD, linearly, and the large candidate count multiplies only q×qq\times q solves.

Preprocessing is chosen on validation in the same way for every method, from three options (Appendix F.1). Each option repeats the same search, so it triples the cost; this is why four of the five datasets exceed 100,000 candidates. The runtimes below include it.

On one core, the median full search takes 0.9 to 14.4 s per split across the 18 datasets, and at most 16.6 s on the four largest (up to 2,866 training pairs). On larger simulated data, the times per split in seconds are, including the reduced C-PCR (rC-PCR), which fixes λ=sY\lambda=s_Y and ρ=τ\rho=\tau (261 candidates):

(n,p)(n,p)C-PCRrC-PCRNo-pairPLSRidge
(4000,1000)(4000,1000)8.01.61.53.31.0
(1000,4000)(1000,4000)39.85.75.73.82.6

rC-PCR runs about as fast as no-pair, and on the five datasets of Table 1 it still beats no-pair on four, keeping the full gain on Beef and Mango but less of it on Raman and Tablets (Appendix F.1). In the other direction, with no-pair given as many candidates as C-PCR, C-PCR still wins on Raman, Tablets and Mango; on Beef the difference is no longer significant, and on Corn no-pair is better (Appendix F.1). Beyond these sizes, the NeigN_{\rm eig} eigenproblems, each O(r3)O(r^3), dominate once rr reaches several thousand. rC-PCR needs only one per value of τ\tau. The cost formula is now in Appendix B.3, and the runtimes are in Table 11 and Appendix F.1.

Simulations are narrow relative to the real data. All simulated settings have p/(2n)≤1p/(2n)\le 1 (maximum γ=1\gamma = 1 in Fig. 2b; γ=1/3\gamma = 1/3, 0.2 and 0.5 elsewhere), whereas Raman, Corn and Tablets have γ>1\gamma \gt 1. As stated in the weaknesses, the authors should expand the simulation studies to investigate the performance of the method in diverse scenarios.

We extended the simulations to γ=4\gamma=4 in three scenarios: complementary signals (S1), a shared factor plus variation in only one view (S2), and the model of Theorem 4.2 (S3). The natural datasets now span γ\gamma from 0.005 to 13.5. All methods are tuned on validation, and the lower rows break the assumptions at γ=1\gamma=1. Mean rMSE reduction over no-pair (%):

SettingS1S2S3
γ=1/3\gamma=1/3+9.3+3.7+0.5
γ=1\gamma=1+2.5+2.7+0.9
γ=2\gamma=2+0.6+1.4+1.3
γ=4\gamma=4+0.3+0.8+1.2
t3t_3 noise+2.3+2.4+1.7
AR(1) noise+9.7+4.0+8.2
Unequal view scales+0.6+1.6+1.8
50% mismatched pairs+0.2+0.4+0.5

These simulations are Appendix E.4 (Tables 5 and 6):

  • C-PCR wins in at least 131 of 150 simulated datasets per γ\gamma, with the 95% interval of the mean difference above zero in every setting;

  • the gain shrinks as γ\gamma grows in S1 and S2, to +0.3% and +0.8% at γ=4\gamma=4, but stays significant, whereas in S3 it stays between +0.5% and +1.3% at every γ\gamma;

  • it has the lowest risk of all six methods in 11 of 12 settings, tying the linear objective of Ji et al. in the last;

  • when all pairs are broken, it falls back to no-pair in S1 and S2 and is 1.9% worse in S3.

Separately, the risk formula stays within 0.21% of simulation up to γ=4\gamma=4 (Appendix E.2).


When I looked at Table 6, the fitting dataset size is quite small compared with the validation and test sets (e.g., Raman: 50, 150, 1088). If more samples were used for training, so that nn increases and γ\gamma decreases, I wonder whether the relative performance among the competing methods would change.

In theory the absolute gain vanishes with enough training data, but on two of the three datasets we could test, it persists at every available size. Pairing helps only by reducing estimation error, which shrinks as nn grows, and Result C.2 makes this precise: as γ=p/(2n)\gamma=p/(2n) goes to zero, the limiting risk tends to the same population risk for every λ\lambda, γ→0⟹R∞(λ,τ,ρ;γ)→Rpop(τ,ρ)for every λ,\gamma\to0\\ \Longrightarrow\\ \mathcal R_\infty(\lambda,\tau,\rho;\gamma)\to\mathcal R_{\rm pop}(\tau,\rho)\\ \text{for every }\lambda, so with pp fixed, no-pair eventually catches up in risk, although the gain relative to the shrinking excess risk can remain large (Appendix E.6). We enlarged the training set on the three datasets with the largest pools, keeping the validation size fixed and testing on the remaining units (now Appendix F.7):

Datasetnnγ\gammaGain in mean rMSE vs no-pair (%)Splits won
Tablets4500.72+11.660/60
Mango3000.46+6.647/60
Raman2001.76−0.8-0.820/60

On Raman, no-pair and ridge already slightly overtake C-PCR at n=200n=200, by 0.7% to 0.8%, whereas on Tablets and Mango the gain persists at the largest training sets we can form.

Reviewer_6AHvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes contrastive principal component regression (C-PCR) for data where each training unit is measured twice and has a shared response, and where prediction is made from a single measurement. Components are chosen by a generalized eigenproblem. Its objective adds the feature-response association to a pair-alignment term weighted by lambda, under a ridge-regularized total-covariance constraint (Eq. 5). The response is then refitted by ridge on the selected scores (Eq. 6). Reduced-rank ridge, PCA++-PCR and one-component PLS are special or limiting cases, and for one component the selected direction coincides with that of a consistency-regularized ridge with matched penalties.

Critical strengths

The main strength is the risk analysis. Theorem 4.2 handles the fact that the same responses are used to choose the direction and to fit its coefficient, which is the hard part of analysing supervised component methods. The decomposition in Eq. (13) is easy to interpret, since it separates the error along the predictive direction from the error spread over all other directions. Section 4.1 reduces the limit to three scalar steps. I also find the calibration convincing, since the limit matches the simulations closely in Figures 2 and 5, also for non-Gaussian innovations. A re-implementation of the limit, run for this review, reproduces the curves in Figure 2. I did not check the random-matrix proofs line by line. I think this is the strongest part of the paper.

Critical weaknesses

One weakness is that the theory does not show that pairing improves on a well-tuned supervised method. Both terms have precedents. The supervised term is linear-kernel supervised PCA (Appendix B.1), and the pair-only criterion amounts to a ridge-regularized CCA with one shared projection. Ji et al. (JMLR 2023), which is not cited, also combine a linear contrastive loss with a supervised HSIC term and then regress on the representation, though in a different setting with augmentation pairs and rate bounds. As the authors note, Proposition 3.1 mainly shows the value of supervision, since lambda = 0 also keeps the predictive direction, so the theoretical case for pairing rests on Result C.2 and Figure 2(a), both at fixed tau and rho. Numerical checks of the limit, run for this review in the Figure 2 setting and a range of other settings, suggest that once tau and rho are also tuned, pairing lowers the limiting risk by at most a few percent of the excess risk. A small simulation at tau = infinity, outside the theorem, also gives only a tiny gain in total risk. Finally, the theory requires q <= L, while on Raman validation selects 7-20 components. Since pairing is what C-PCR adds to existing methods, this is the most important weakness.

Questions for authors
  1. Can the limit in Theorem 4.2 be extended beyond lambda_ex, which shrinks as tau grows? If so, is there a regime where the best lambda > 0 has a clearly lower limiting risk than the best lambda = 0 when tau and rho are tuned for both?
  2. Can you repeat the real-data comparison in Table 1 on all five datasets over many random splits and report the mean difference to no-pair and to ridge with its uncertainty? Do the gains hold when all methods are tuned over grids of similar size?
  3. Ji et al. (2023, Section 4.2) also combine a contrastive term with a supervised cross-covariance term before regressing. How does C-PCR relate to their objective, and what do you see as the main difference?
6 comments
AuthorsOpenReview ↗

Extended limit shows pairing beats tuned no-pair in all 96 settings, and the theory now covers the components beyond the response rank

We thank the reviewer for the careful review and for checking our limit numerically. The central question is whether pairing beats a well-tuned supervised method. Inside λex\lambda_{\rm ex}, the gain is indeed small. The current submission extends Theorem 4.2 beyond λex\lambda_{\rm ex}, where the gain is clear in most settings. On natural data the gain also holds, although the scalar theory does not explain its size; we discuss this gap below.


One weakness is that the theory does not show that pairing improves on a well-tuned supervised method.

The extended Theorem 4.2, evaluated numerically, now shows it: in its range, pairing lowers the risk of tuned no-pair in all 96 settings, by a median of 8.0% of the excess risk (Appendix E.6). Only inside λex\lambda_{\rm ex} is the gain small. The new Corollary 4.3 reduces the comparison to one quantity. At the closed-form optimal coefficient ridge ρ∗(λ,τ)\rho^*(\lambda,\tau), R∞(λ,τ,ρ∗)−R∗_lin=a,ϑ2,(1−c)(1+a)(1+a,c),c(λ,τ)=α\*2χ\*∈[0,1],\mathcal R_\infty(\lambda,\tau,\rho^*)-\mathcal R^*\_{\rm lin}=\frac{a\\,\vartheta^2\\,(1-c)}{(1+a)(1+a\\,c)}, \qquad c(\lambda,\tau)=\frac{\alpha_\*^2}{\chi_\*}\in[0,1], so a tuned method’s risk depends only on the alignment cc of its direction with u1∗u_1^*. A well-tuned supervised method is the endpoint λ=0\lambda=0, so pairing improves on it exactly when max⁡λ>0,,τc(λ,τ)>max⁡τc(0,τ).\max_{\lambda\gt 0,\\,\tau}\\ c(\lambda,\tau)\\ \gt \\ \max_{\tau}\\ c(0,\tau). The answers below measure this with τ\tau tuned as well.


As the authors note, Proposition 3.1 mainly shows the value of supervision, since lambda = 0 also keeps the predictive direction, so the theoretical case for pairing rests on Result C.2 and Figure 2(a), both at fixed tau and rho.

We agree, and the current submission now makes the case for pairing with τ\tau and ρ\rho tuned. Proposition 3.1 is a population result, in which λ=0\lambda=0 already selects u1∗u_1^*; there, pairing can help only through estimation, which a population result cannot show. In the original submission, the finite-sample case therefore rested on Result C.2 and Figure 2(a) (now Figure 3(a)), both at fixed τ\tau and ρ\rho.

The current submission closes this gap in three steps. The mechanism: Section 3.2 explains how, in finite samples, the pair term filters chance correlations that the two views do not share. Tuned ρ\rho: Corollary 4.3 gives the risk at the optimal ρ\rho in closed form (previous answer). Tuned τ\tau: the next answer tunes τ\tau as well, and shows that pairing still lowers the risk of tuned no-pair.


Numerical checks of the limit, run for this review in the Figure 2 setting and a range of other settings, suggest that once tau and rho are also tuned, pairing lowers the limiting risk by at most a few percent of the excess risk. A small simulation at tau = infinity, outside the theorem, also gives only a tiny gain in total risk.

Our checks confirm this inside λex\lambda_{\rm ex}, but beyond it pairing reduces the excess risk by more than 1% in 82 of 96 settings (Appendix E.6). We compared the limits with τ\tau and ρ\rho tuned for both methods over 96 settings with γ\gamma from 0.1 to 4, measuring how much pairing reduces the excess risk of tuned no-pair (it helps if the reduction exceeds 0.001%):

\hline \text{Pair weights searched} & \text{Pairing helps in} & \text{Above 1\\% in} & \text{Median reduction} & \text{Largest reduction}\\\\ \hline \lambda\lt \lambda_{\rm ex}\text{ (original)} & \text{32 of 96} & \text{11 of 96} & 0.0\\% & 4.7\\%\\\\ z_\*\gt \lambda\upsilon\text{ (extended)} & \text{96 of 96} & \text{82 of 96} & 8.0\\% & 77.2\\%\\\\ \hline \end{array}$$ **Inside the threshold, the first row matches the reviewer’s checks**, because the threshold is small in these settings. In the population limit it shrinks with the selection ridge, $$\lambda_{\rm ex}\\ \longrightarrow\\ \frac{a\\,\vartheta^2}{1+\tau},$$ and tuning makes this ridge very large for no-pair (effectively $\tau=\infty$, the PLS-like endpoint), because the views differ by unit noise in every coordinate while the signal is moderate ($a\leq10$). _**Pairing therefore has little room inside the threshold.**_ **Beyond the threshold, the gains are large, and the extended Theorem 4.2 is what captures them** (second row). In 95 of the 96 settings, the best $\lambda$ (median 1.6) lies beyond $\lambda_{\rm ex}$, where _**only the extended theorem gives the risk**_. The answer on extending the limit, below, breaks this gain down by $\gamma$ and by the response noise.

Continued: The same holds for the reviewer’s simulation at τ=∞\tau=\infty, and for the same reason: pairing can reduce only the excess risk, ΔRR=ΔRR−R∗_lin⏟excess-risk reduction×R−R∗_linR⏟share of excess risk,\frac{\Delta\mathcal R}{\mathcal R} =\underbrace{\frac{\Delta\mathcal R}{\mathcal R-\mathcal R^*\_{\rm lin}}}_{\text{excess-risk reduction}} \times\underbrace{\frac{\mathcal R-\mathcal R^*\_{\rm lin}}{\mathcal R}}_{\text{share of excess risk}}, and with σ=0.8\sigma=0.8 the best linear risk is most of the total. Measured on the excess risk, with λ\lambda, τ∈(0,∞]\tau\in(0,\infty] and ρ\rho all tuned, C-PCR is significantly better than tuned no-pair in all 16 simulated settings of Appendix E.6, by up to 23.5% (Appendix E.6), and the total-risk gain reaches 2 to 3% for σ≥1\sigma\geq1.


Finally, the theory requires q <= L, while on Raman validation selects 7-20 components. Since pairing is what C-PCR adds to existing methods, this is the most important weakness.

The current submission extends the theory to q>Lq\gt L (new Proposition D.4), and the components beyond the response rank turn out to be where pairing matters most.

Why the original theory needed q≤Lq\leq L. Each C-PCR component is an eigenvector of the selection problem, with eigenvalue zjz_j (z1≥z2≥⋯z_1\geq z_2\geq\cdots), and Theorem D.2 assumes zj>λz_j\gt \lambda. With LL responses, the response term b^,b^⊤\widehat b\\,\widehat b^\top has rank LL, so it can lift at most LL eigenvalues above the pair-only values λμj<λ\lambda\mu_j\lt \lambda, and component L+1L+1 always has z<λz\lt \lambda (Lemma B.7).

How the current submission removes this limit. The proof of Theorem D.2 uses z>λz\gt \lambda only to keep the noise part of the selection problem positive definite. Proposition D.4 shows that this holds for every selected eigenvalue above a lower margin, z>λυ,υ=s+2τ+s+<1,s+=(1+2γ)2,z\gt \lambda\upsilon,\qquad \upsilon=\frac{s_+}{2\tau+s_+}\lt 1,\qquad s_+=(1+\sqrt{2\gamma})^2, and proves the risk limit of Theorem D.2 there for any qq, when the feature-noise coordinates are identically distributed.

Why these components matter for pairing. The components beyond the response rank are pair-only directions shifted by the response, so C-PCR ranks them by pair alignment, whereas PCR ranks them by variance; at λ=0\lambda=0 they are not determined at all (Appendix B.4). The beyond-rank result (Proposition E.4 and Remark E.5) shows this: beyond λex\lambda_{\rm ex}, a single component loses the signal, whereas a second component beyond the response rank recovers it.

Raman. Raman is the dataset where these components count: restricting C-PCR to q≤Lq\leq L removes most of its gain there, but keeps it on Tablets, Mango and Beef (Table 14, Appendix F.4; median reduction relative to no-pair, %, with 95% intervals over splits):

DatasetC-PCR with q≤Lq\leq LC-PCR
Tablets+12.1[10.4,,13.1]+12.1\\ [10.4,\\,13.1]+12.3[11.3,,13.4]+12.3\\ [11.3,\\,13.4]
Mango+5.3[3.6,,6.9]+5.3\\ [3.6,\\,6.9]+4.7[3.2,,6.3]+4.7\\ [3.2,\\,6.3]
Beef+0.8[0.5,,1.3]+0.8\\ [0.5,\\,1.3]+1.1[0.7,,1.7]+1.1\\ [0.7,\\,1.7]
Raman+0.8[0.0,,1.3]+0.8\\ [0.0,\\,1.3]+5.9[4.9,,7.5]+5.9\\ [4.9,\\,7.5]

On Raman, most of the gain thus comes from components beyond the response rank. These Raman fits now fall within the new theory of the current submission, Proposition D.4 and Appendix D.5, in 126 of the 136 fits with finite τ\tau and λ>0\lambda\gt 0 (Appendix F.4). Measured in units of the noise variance, the τ\tau selected on Raman is either moderate or near zero, and each case is covered by a different result:

  • Moderate τ\tau (78 fits): the risk formula applies. In 70 of these fits, every selected eigenvalue exceeds λυ\lambda\upsilon, so Proposition D.4 gives the limiting risk of the whole fit, components beyond the response rank included.

  • τ→0\tau\to0 (58 fits): the extra components are signal. They come from the pair-only directions, which become the leading principal components of the directions on which both measurements of every training unit agree (new Proposition D.7). In 56 of these fits, every selected component stands out from the noise (Proposition D.8), so the extra components are real structure, mostly shared by the two views.

These results take the model’s assumptions as given (isotropic feature noise, fixed tuning parameters and a fixed number of components), which validation-tuned fits on real spectra meet only approximately (Section 6); for τ→0\tau\to0, they also describe which components are selected rather than the risk. Within these limits, the theory now describes the components beyond the response rank that carry the Raman gain.

Can the limit in Theorem 4.2 be extended beyond lambda_ex, which shrinks as tau grows? If so, is there a regime where the best lambda > 0 has a clearly lower limiting risk than the best lambda = 0 when tau and rho are tuned for both?

Extending the limit: yes, and Theorem 4.2 now states it. As explained in our answer on q≤Lq\leq L, Proposition D.4 secures the proof’s key condition down to λυ\lambda\upsilon. Assumption 4.1(iii) now asks only that the selected eigenvalue stay above the noise levels (λυ\lambda\upsilon and the distractors’ pair-only outliers, all below λ\lambda), which holds automatically for λ<λex\lambda\lt \lambda_{\rm ex}, so the risk formula of Theorem 4.2 holds beyond λex\lambda_{\rm ex}.

The new Theorem D.6 extends the limit further, down to the edge E(λ)E(\lambda) of the noise spectrum, the lowest possible threshold (Appendix D.5). It holds for Gaussian noise, and simulation gives the same edge for non-Gaussian noise. The extension matters most when p/np/n is small: in the setting of Figure 3(a), the threshold falls from 0.66λ0.66\lambda to 0.47λ0.47\lambda.

A regime with a clear gain: yes. In the proved range z\*>λυz_\*\gt \lambda\upsilon, with τ\tau and ρ\rho tuned for both methods, we measured the reduction of the excess risk of tuned no-pair in 16 settings per dimension ratio (a∈0.5,1,3,10a\in\\{0.5,1,3,10\\}, ϑ∈0.5,1,1.2,2\vartheta\in\\{0.5,1,1.2,2\\}), taking the better of a search inside λex\lambda_{\rm ex} and a grid search over the whole range:

γ\gammaMedian reduction (%)Range (%)Above 1%
0.119.31.6 to 77.216 of 16
0.2516.61.1 to 76.516 of 16
0.511.60.6 to 75.715 of 16
16.10.3 to 74.714 of 16
24.50.1 to 72.811 of 16
42.40.04 to 69.010 of 16

The gain relative to the excess risk is largest when γ\gamma is small, that is, with many pairs per feature, even though the excess risk itself, and with it the absolute gain, vanishes as γ→0\gamma\to0 (Result C.2); pairing helps in all 96 settings, and the gain stays below 1% only for a weak factor (a≤1a\leq1) with γ≥0.5\gamma\geq0.5. The gain also grows with the response noise, because the pairs do not depend on the labels (a=3a=3, γ=1\gamma=1):

Response noise σ\sigma0.5123
Reduction of excess risk (%)3.629.565.776.5
Reduction of total risk (%)0.52.33.22.7

Can you repeat the real-data comparison in Table 1 on all five datasets over many random splits and report the mean difference to no-pair and to ridge with its uncertainty? Do the gains hold when all methods are tuned over grids of similar size?

Yes. Each of the five datasets now has 150 random splits. No-pair (C-PCR with λ=0\lambda=0) and ridge are tuned on the same validation data as C-PCR. The table gives the mean rMSE of each baseline minus that of C-PCR (×10−3\times10^{-3}, with 95% intervals over splits; positive values favor C-PCR):

RamanBeefTabletsCornMango
No-pair+21.2±4.7+21.2\pm4.7+3.9±2.8+3.9\pm2.8+55.4±4.5+55.4\pm4.5−3.1±2.0-3.1\pm2.0+4.7±1.2+4.7\pm1.2
Ridge+24.9±6.2+24.9\pm6.2+4.4±3.2+4.4\pm3.2+67.0±4.6+67.0\pm4.6−5.6±2.2-5.6\pm2.2+5.2±1.2+5.2\pm1.2

C-PCR is better than both baselines on Raman, Beef, Tablets and Mango, with every interval above zero; on Corn, both baselines are better. These intervals reflect variability over random splits, not sampling of the dataset itself.

On grid size, the gains mostly hold when the budgets are matched. C-PCR searches far more candidates than no-pair (39,789 against 589 on Raman), so we matched the budgets in both directions (Appendix F.1):

  • More candidates for no-pair. A matched no-pair searches as many candidates as C-PCR, on finer grids of τ\tau and ρ\rho.

  • Fewer candidates for C-PCR. A reduced C-PCR (rC-PCR) fixes two of its four tuning parameters, the pair weight at the response scale (λ=sY\lambda=s_Y) and the coefficient ridge at the selection ridge (ρ=τ\rho=\tau), so it searches fewer candidates than no-pair on four of the five datasets.

Mean rMSE of no-pair minus that of each C-PCR variant (×10−3\times10^{-3}, 95% intervals over splits; positive values favor the C-PCR variant):

ComparisonRamanBeefTabletsCornMango
C-PCR vs no-pair+21.2±4.7+21.2\pm4.7+3.9±2.8+3.9\pm2.8+55.4±4.5+55.4\pm4.5−3.1±2.0-3.1\pm2.0+4.7±1.2+4.7\pm1.2
C-PCR vs matched no-pair+16.0±4.2+16.0\pm4.2−2.4±3.3-2.4\pm3.3+47.3±4.0+47.3\pm4.0−5.3±1.9-5.3\pm1.9+4.9±1.1+4.9\pm1.1
rC-PCR vs no-pair+3.6±3.1+3.6\pm3.1+6.5±3.1+6.5\pm3.1+22.7±4.4+22.7\pm4.4−4.9±2.1-4.9\pm2.1+5.5±0.9+5.5\pm0.9
Reviewer_Xk8zOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes contrastive principal component regression, which is a combination between principal component regression and PCA++-PCR. The setting is the supervised learning setting, where the input xix_i has paired feature xi+x_i^+ that differ by some nuisance factor hih_i. C-PCR consists of two steps to leverage such structural information in the data generation process. In the first step, it uses a constrained optimization step to extract the principal components of a fused covariance matrix. In the second step, it performs the standard PCR, with a different ridge parameter ρ\rho. The paper then identifies some toy examples where C-PCR identifies the predictive principal components more accurately than standard PCA and PCA++, and also calculates the excessive risk in the random matrix regime.

Critical strengths

The proposed algorithm is interesting and presents an innovative way to combine partial linear regression, PCA, and PCA++. The excessive risk matches the numerical experiments.

Critical weaknesses
  1. Figure 2 suggests that increasing λ\lambda reduces the excess risk, whereas the theoretical results require λ\lambda to remain below a specified threshold. How does the method behave when λ\lambda exceeds this threshold? Please clarify which values of λ\lambda are used in the numerical experiments and whether they satisfy the theoretical condition. This would help readers assess how well the theory explains the empirical results.

  2. The writing and presentation of notation could be improved. Definitions are scattered throughout the manuscript, making the technical arguments difficult to follow. For example, equation (1) denotes the diagonal matrices by Γ\Gamma and Δ\Delta, but their diagonal entries by aja_j and bjb_j, respectively. The correspondence is not immediately apparent. Explicitly stating these relationships and introducing related notation together would improve readability.

Questions for authors
  1. The example in Figure 1 is confusing. Are u2u_2, u1u_1, and v1v_1 estimated by the algorithms, or are they generated by AI for illustration? The figure does not clearly explain why C-PCR outperforms the two baselines. More generally, the intuition behind how an appropriate choice of λ\lambda emphasizes the signal ww remains unclear. Although the table near line 207 presents intermediate results, these results do not directly clarify the advantage of C-PCR over the baselines. A clearer explanation connecting the choice of λ\lambda, the resulting representations, and the performance improvement would be helpful.

  2. How are λ\lambda, ρ\rho, and τ\tau specified in the theoretical analysis and selected in practice? How are the corresponding parameters chosen for the baselines? In particular, the optimal value of ρ\rho may differ between PCA++-PCR and C-PCR, which raises concerns about the comparison in Proposition 3.1. Please clarify whether the comparison allows each method to use its own optimal parameter values and, if not, how the shared parameter choices affect the interpretation of the result.

6 comments
AuthorsOpenReview ↗

Theory extended beyond the threshold, pair weights reported against it, and Figures 1 and 2 redrawn from data

We thank the reviewer for the careful reading. The main changes are these: Theorem 4.2 now covers pair weights beyond the threshold, and the simulation matches it there; the calibration experiments and the natural data report their pair weights against the threshold (Appendix E.2); Figure 1 is redrawn from the data, and the new Figure 2 shows how λ\lambda improves prediction; and Corollary 4.3 gives the theory’s choice of ρ\rho.


Figure 2 suggests that increasing λ\lambda reduces the excess risk, whereas the theoretical results require λ\lambda to remain below a specified threshold. How does the method behave when λ\lambda exceeds this threshold?

Beyond the threshold, the risk rises again, toward that of pair-only PCA++-PCR. The revised Figure 3(a), formerly Figure 2, extends the pair weight to λ=∞\lambda=\infty and compares the theory with simulation:

\hline \lambda & 0 & 2 & 3.6 & 9.6 & \infty\text{ (pair-only)}\\\\ \hline \text{Excess risk, theory} & 0.098 & 0.073 & 0.076 & 0.085 & 0.098^{\dagger}\\\\ \text{Excess risk, simulation} & 0.099 & 0.075 & 0.077 & 0.087 & 0.102\\\\ \hline \end{array}$$ $\lambda=2$ is the best weight and $\lambda=3.6$ lies just past $\lambda_{\rm ex}=3.18$. $^{\dagger}$Theory at $\lambda=100$; simulation at $\lambda=\infty$. At the fixed ridges $\tau=\rho=0.7$ of Figure 3(a), increasing $\lambda$ helps up to about $\lambda=2$, inside the threshold; with $\tau$ and $\rho$ also tuned, the gain inside the threshold becomes small, and the larger gains lie beyond it (Appendix E.6). **Theory and simulation agree within 0.21% of the risk** across panels (a) and (b); the excess risk is a small part of the risk, so the excess risks above differ by a few thousandths. The curve beyond the threshold *is now covered by Theorem 4.2 itself*: its condition (iii) asks only that the selected eigenvalue stay above a noise margin below $\lambda$ (proof in Appendix D.4). The new Figure 2 (Section 3.2) shows why: beyond $\lambda_{\rm ex}$, the selection relies more on pair alignment than on the response, so the selected feature drifts toward the shared distractor. --- > *Please clarify which values of $\lambda$ are used in the numerical experiments and whether they satisfy the theoretical condition. This would help readers assess how well the theory explains the empirical results.* **In Figure 3 all of them satisfy it: every weight lies inside $\lambda_{\rm ex}$ except the eight in panel (a) that deliberately go beyond it, and these still meet the extended condition of Theorem 4.2; on natural data, 84% of the selected fits lie inside $\lambda_{\rm ex}$ (Appendix E.2).** The threshold $\lambda_{\rm ex}$, the pair weight at which the selected eigenvalue reaches $\lambda$ (Appendix C.2), depends on $\gamma=p/(2n)$, $a$, $\vartheta$, $\sigma^2$ and $\tau$. On natural data, where these are unknown, we use the sample threshold $\widehat\lambda_{\rm ex}$, the weight at which the selected eigenvalue $\widehat z$ meets $\lambda$; it needs only the training data. $$\begin{array}{lccc} \hline \text{Experiment} & \lambda\text{ used} & \lambda_{\rm ex} & \lambda\lt \lambda_{\rm ex}\\\\ \hline \text{Figure 3(a), }\gamma=0.2 & \text{0 to }\infty\text{ (21 weights)} & 3.18 & \text{13 of 21}\\\\ \text{Figure 3(b), }\gamma=0.05\text{ to 1} & 0.4 & \text{2.69 to 6.14} & \text{all}\\\\ \text{Figure 3(c), }n=200\text{ to 1600} & 0.4 & 3.18 & \text{all}\\\\ \text{Natural data, validation-selected} & \text{tuned} & \widehat\lambda_{\rm ex}\text{ per split} & \text{84\\% of 1,800 (incl. }\lambda=0\text{)}\\\\ \hline \end{array}$$ On natural data, **the selected $\lambda$ usually lies far inside the threshold** (median $\lambda/\widehat\lambda_{\rm ex}=0.002$ over 1,800 fits, from 18 datasets and 3 seeds): $$\begin{array}{lcc} \hline \text{Selected }\lambda & \text{Fits} & \text{Share}\\\\ \hline 0\lt \lambda\lt \widehat\lambda_{\rm ex}\text{ (inside)} & 1{,}457 & 80.9\\%\\\\ \lambda=0\text{ (no-pair endpoint)} & 61 & 3.4\\%\\\\ \widehat\lambda_{\rm ex}\le\lambda\lt \infty\text{ (beyond)} & 49 & 2.7\\%\\\\ \lambda=\infty\text{ (pair-only endpoint)} & 200 & 11.1\\%\\\\ \tau=\infty\text{ (no selection ridge)} & 31 & 1.7\\%\\\\ \text{mean predictor} & 2 & 0.1\\%\\\\ \hline \end{array}$$ The 49 fits with finite $\lambda$ beyond the threshold come mostly from four datasets on which the views agree little, so $\widehat\lambda_{\rm ex}$ is small; even there, going beyond does not help on test data: with a finer $\lambda$ grid, it changes the test rMSE by a median of $-0.04\\%$ (Appendix E.2).

Continued: In simulation, the theory predicts how much pairing helps; on natural data, it explains why pairing helps, but not how much. In simulation from the model, it predicts the risk within 0.21% (Figure 3), and with τ\tau and ρ\rho tuned for both methods it shows that pairing lowers the risk of tuned no-pair in all 96 settings, mostly beyond the threshold (Appendix E.6). On natural data, by contrast, the selected λ\lambda lies mostly inside the threshold, yet the gains are larger, up to 12.3% on Tablets (Table 13).

On natural data, the theory’s explanation of why pairing helps still applies: the pair term filters chance correlations that the two views do not share. What the theory cannot predict there is the size of the gain, because real spectra carry structure that the scalar model leaves out and that favors pairing further, such as components beyond the response rank (Table 14) and correlated feature noise (Appendix E.4).


The writing and presentation of notation could be improved. Definitions are scattered throughout the manuscript, making the technical arguments difficult to follow. For example, equation (1) denotes the diagonal matrices by Γ\Gamma and Δ\Delta, but their diagonal entries by aja_j and bjb_j, respectively. The correspondence is not immediately apparent. Explicitly stating these relationships and introducing related notation together would improve readability.

We renamed the two matrices so that each shares a letter with its entries, Da=diag⁡(a1,…,ak),Db=diag⁡(b1,…,bkˉ),D_a=\operatorname{diag}(a_1,\dots,a_k),\qquad D_b=\operatorname{diag}(b_1,\dots,b_{\bar k}), so that Equation (1) and its moments read xi=U∗Da1/2wi+W∗Db1/2hi+ϵi,Σtot=Ip+U∗DaU∗⊤+W∗DbW∗⊤,Σct=U∗DaU∗⊤.x_i=U^*D_a^{1/2}w_i+W^*D_b^{1/2}h_i+\epsilon_i,\qquad \Sigma_{\rm tot}=I_p+U^*D_a U^{*\top}+W^*D_b W^{*\top},\qquad \Sigma_{\rm ct}=U^*D_a U^{*\top}. Both are defined with their entries directly below (1), and the change is applied throughout, including the multiple-response model. We also brought related notation together:

  • The introduction states the conventions once: a star for population truths, a hat for sample estimates, and the roles of λ\lambda, τ\tau, ρ\rho and qq.

  • Section 2 lists the four population moments next to their estimates.

  • Section 3.2 defines the single-predictor setting and the pair-only score ηj\eta_j where they are first used.

  • Section 3 opens with a roadmap, the supplement with a guide to the proofs, and every symbol is defined where it first appears, and the shared definitions are in Appendix A.


The example in Figure 1 is confusing. Are u2u_2, u1u_1, and v1v_1 estimated by the algorithms, or are they generated by AI for illustration?

We agree that the original figure was confusing, and we have redrawn it. The three directions are neither estimates nor AI illustrations: they are the true directions of the model in Equation (1), predictive (u1∗u_1^*), distractor (u2∗u_2^*) and background (v1∗v_1^*). The new Figure 1 is drawn from data: each point is one of 150 units drawn from the model and scored on the direction that each method’s population criterion selects (Section 3.2). Estimated from data over 300 training sets (Appendix C.1), each method aligns most with that direction:

\hline \text{Method} & \cos^2(\widehat u,u_1^*) & \cos^2(\widehat u,u_2^*) & \cos^2(\widehat u,v_1^*)\\\\ \hline \text{PCR} & 0.00 & 0.01 & 0.93\\\\ \text{PCA++-PCR} & 0.01 & 0.72 & 0.00\\\\ \text{C-PCR} & 0.58 & 0.00 & 0.00\\\\ \hline \end{array}$$ The values stay below one because the rest of $\widehat u$ is spread over noise, the estimation error that Theorem 4.2 measures. --- > *The figure does not clearly explain why C-PCR outperforms the two baselines.* **C-PCR is the only method that retains the predictive direction $u_1^*$.** The revised figure shows this in two rows: the top row plots each unit’s two view scores and its response, and the bottom row shows the 1-D score used for prediction. $$\begin{array}{lllll} \hline \text{Panel} & \text{Retains} & \text{Views} & \text{Corr. with }y & \text{1-D score}\\\\ \hline \text{(a) PCR} & \text{background }v_1^* & \text{disagree} & +0.12 & \text{mixed}\\\\ \text{(b) PCA++-PCR} & \text{distractor }u_2^* & \text{agree} & +0.09 & \text{mixed}\\\\ \text{(c) C-PCR} & \text{predictive }u_1^* & \text{agree} & +0.90 & \text{sorted}\\\\ \hline \end{array}$$

Continued: Each baseline fails for a different reason. PCR ranks by variance and retains the background v1∗v_1^*, which appears in one view only; the pair term discards it. PCA++-PCR ranks by agreement and retains the distractor u2∗u_2^*, which both views share but which does not predict yy; the response term breaks this tie. C-PCR scores the three directions as (u⊤breg)2+λ,u⊤Σctuu⊤Στu:(ϑ2+λ),η1⏟u1∗,λ,η2⏟u2∗,0⏟v1∗,ηj=aj1+aj+τ,\frac{(u^\top b_{\rm reg})^2+\lambda\\,u^\top\Sigma_{\rm ct}u}{u^\top\Sigma_\tau u}:\qquad \underbrace{(\vartheta^2+\lambda)\\,\eta_1}_{u_1^*},\qquad \underbrace{\lambda\\,\eta_2}_{u_2^*},\qquad \underbrace{0}_{v_1^*},\qquad \eta_j=\frac{a_j}{1+a_j+\tau}, so it retains u1∗u_1^* whenever λ<λc=ϑ2η1/(η2−η1)\lambda\lt \lambda_c=\vartheta^2\eta_1/(\eta_2-\eta_1). Scores along v1∗v_1^* and u2∗u_2^* carry no information about yy, so both baselines do no better than the mean predictor: with n=120n=120 and p=200p=200, over 300 simulated training sets, their excess risk is 2.68, against 0.62 for C-PCR, which is lower in all 300. With two components each, C-PCR is still better (answer on ρ\rho below).


More generally, the intuition behind how an appropriate choice of λ\lambda emphasizes the signal ww remains unclear.

An appropriate λ\lambda trusts a direction as far as it repeats across the two measurements of a unit, but not so far that repetition outweighs the response. Section 3.2 now explains this after Proposition 3.1.

  • At λ=0\lambda=0, selection partly follows noise: with finite data, the estimated link between features and response mixes the signal ww with chance correlations.

  • A positive λ\lambda emphasizes ww: ww appears in both measurements of a unit, whereas chance correlations come from noise that differs between them; the pair term rewards this agreement, so it favors directions that carry ww and discounts chance correlations.

  • A large λ\lambda overrides the response: agreement alone then decides, and a shared factor that does not predict the response can displace ww.

Figure 2 shows both effects; our answer on connecting λ\lambda to performance, below, gives the numbers.


Although the table near line 207 presents intermediate results, these results do not directly clarify the advantage of C-PCR over the baselines.

Section 3.2 now states what that table shows and, as the reviewer notes, what it does not: it ranks directions in the population, where every λ<λc\lambda\lt \lambda_c selects u1∗u_1^*, so it shows which direction is selected but not how large the gain is. The gain has two sources, which two complementary figures isolate:

\hline \text{Gain over} & \text{Source} & \text{Evidence}\\\\ \hline \text{PCR and PCA++-PCR} & \text{the response term} & \text{Proposition 3.1; Figure 1 (0.62 vs 2.68)}\\\\ \text{No-pair (}\lambda=0\text{)} & \text{the pair term, in finite samples} & \text{Theorem 4.2; Figures 2 and 3}\\\\ \hline \end{array}$$ Figure 2 isolates the second source: in the setting of Appendix E.3, its view-specific background spans many directions; as $\lambda$ goes $0\to2$, the pair term filters them out and the excess risk falls by 19%. It thus complements Figure 1, whose background is a single direction, so the pair term has nothing to filter there ($\lambda=1$ and $\lambda=0$ give excess risks of 0.623 and 0.625; Appendix C.1). --- > *A clearer explanation connecting the choice of $\lambda$, the resulting representations, and the performance improvement would be helpful.* **The new Figure 2 makes this link in three panels:** (a) the risk decomposition of Theorem 4.2 along $\lambda$, (b) where the variance of the selected feature comes from, and (c) its excess risk. Panels (b) and (c) trace the estimated feature along the $\lambda$ path (setting in Appendix E.3); the last column is the share of fits whose selected eigenvalue falls below $\lambda$, that is, beyond the threshold: $$\begin{array}{cccccc} \hline \lambda & \text{Predictive} & \text{View-specific} & \text{Distractor} & \text{Excess risk vs }\lambda=0 & \widehat z\le\lambda\\\\ \hline 0 & 0.64 & 0.095 & 0.003 & \text{—} & 0\\%\\\\ 0.3 & 0.65 & 0.085 & 0.004 & -9.1\\% & 0\\%\\\\ 1 & 0.66 & 0.068 & 0.003 & -17.6\\% & 0\\%\\\\ 2 & 0.66 & 0.065 & 0.007 & -19.0\\% & 0\\%\\\\ 3 & 0.65 & 0.066 & 0.018 & -11.0\\% & 53\\%\\\\ 5 & 0.63 & 0.069 & 0.015 & +6.3\\% & 28\\%\\\\ 30 & 0.02 & 0.041 & 0.78 & +538\\% & 100\\%\\\\ \infty\text{ (PCA++-PCR)} & 0.00 & 0.040 & 0.80 & +550\\% & {}\\\\ \hline \end{array}$$

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.