acceptodds
Desk rejected at ICLR 2027

Adaptive Invariant PCA from Augmentation Portfolios

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning from an augmentation collection requires knowing which background variation it exposes and how strongly to suppress it. We study signal-preserving linear views with correlated backgrounds and shared noise. Complementary channels can identify the background space even when every pair is insufficient; we characterize their required number. We propose Adaptive Invariance PCA (AIPCA), a matrix-power path connecting pooled PCA, normalized alignment, and a limit that suppresses variation across views. Under complete coverage, its population limit eliminates background while retaining all signal coordinates when signal and background spaces are disjoint. Proportional-dimensional risk theory explains the tradeoff between background suppression and estimation error: intermediate powers can outperform the reference methods, whereas sufficiently strong signals favor pooled PCA. Source-wise validation selects the power using covariance profiles and paired comparisons with a reconstruction reference. Synthetic and controlled real-image experiments examine coverage, risk predictions, and adaptation to the available views.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Invariant representation learning, data augmentation, multi-view learning, subspace identifiability, spectral estimation.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QvubOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

The paper studies linear representation learning under data augmentation: each image is passed through several augmentation channels to produce several views, in which the signal is unchanged while the background varies with the channel. It asks two questions: when a set of augmentations suffices to identify the background subspace, and how strongly cross-view variation should be suppressed with finite samples. The proposed method, AIPCA, uses a single matrix power α to connect pooled PCA, normalized alignment and difference PCA along one path, and selects α adaptively on validation data; on the theoretical side it gives a coverage condition and the asymptotic risk under a simplified model. The experiments comprise synthetic data and 8×8 Optdigits with artificially injected lighting, and adaptive power selection outperforms a fixed α = 1 only when the background is highly shared across views.

Critical strengths

The question of whether an augmentation portfolio covers all nuisance directions is formalized as positive definiteness of the coverage matrix H, and a sign-flip construction shows that every fixed pair of channels is unidentifiable while the full portfolio is identifiable; the problem statement is clear.

Critical weaknesses
  1. The adaptive selector never acts in practice, and a simpler baseline outperforms it

The paper's practical contribution is adaptive power selection, yet the authors themselves state that, in every real-data setting, the screening and profile steps never change the selection, so AIPCA reduces to choosing α by reconstruction error. Table 6 shows that Fixed Q-PCA, which simply performs PCA inside the estimated background complement, outperforms AIPCA at all five values of ρ (for example 87.64 vs. 86.04 at ρ = 0.99 and 82.20 vs. 77.30 at ρ = 0.999), and this counterexample appears only in the appendix; Table 1 in the main text does not include Q-PCA. The practical core of the method therefore does not hold.

  1. The experiments are far from the target setting

The only real data are 8×8 Optdigits with artificially injected linear lighting, which is far from the crops, color changes and blur described in the introduction. The ρ = 0.99 column of Table 1 is the most favourable point of the sweep: Table 6 shows that AIPCA equals α = 1 at ρ = 0.9 and falls below it at ρ = 0.9999.

Questions for authors
  1. Since Q-PCA is no worse than AIPCA in every row of Table 6, where does the advantage of AIPCA lie? Can you give a setting in which Q-PCA fails and AIPCA succeeds?
  2. Why is the pairwise screen retained? Table 3 shows that profile-only selection is better in most settings, and the screen never acts in any real-data setting.
  3. All experiments supply the true background rank m (m = 3 on Optdigits, where the three lighting components are injected by the authors), and both the selector's background estimate and the Q-PCA baseline depend on it; in a real augmentation portfolio m is unknown. How do the selector and the final loss behave when m must be estimated or is misspecified?
3 comments
AuthorsOpenReview ↗

Official response

Thank you for asking when adaptive power selection improves on simpler methods. We added matched comparisons, tested retrieval across physical lighting changes, and examined how the selector behaves under weak coverage and rank misspecification.


 

1 and Q1. When does our method improve on Q-PCA? With 745 fitting scenes in Multi-Illumination, our method selects α=4 and reaches 54.06% retrieval accuracy. Q-PCA reaches 34.48%. The gain over Q-PCA is 19.58 percentage points (95% paired bootstrap interval: 14.27 to 24.90). With 22 fitting scenes per model, our method reaches 46.97% and Q-PCA reaches 35.16%. See our response to Reviewer 1 (s9yS), Point 1 for the full Multi-Illumination comparison. On Optdigits, the two methods have similar accuracy under matched CV, where differences between views already reveal the illumination directions and thus Q-PCA achieves strong performance.

In Figure 3, with strong signal and two views, mean normalized projector loss is 0.9082 for Q-PCA and 0.0711 for our method. Cross-moment PCA and mean PCA reach 0.0635 and 0.0688. See the full Figure 3 comparison in our response to Reviewer 1 (s9yS), Point 1.

Why Q-PCA can lose signal. In the population example in Appendix G.7, the signal direction has a component along the background direction. Q-PCA removes that component when it projects out the background, giving loss 0.5. At fixed power 0.1, our method retains part of the component and gives loss about 0.00010, compared with 0.00248 for pooled PCA.

Overall, our method achieves the best or close to the best performance across these matched comparisons. It reduces loss relative to Q-PCA in the strong-signal simulation with two views, improves cross-light retrieval, and achieves similar accuracy on Optdigits. We added the matched comparisons to Section 6.1 (Figure 3 and Table 1) and Section 6.2 (Tables 2–3), and the population example to Appendix G.7.


 

2. Experimental scope. We added a cross-light retrieval experiment on Multi-Illumination [1]. The official training split contains 985 indoor scenes, each photographed from a fixed viewpoint under different physical lights. The fixed viewpoint keeps the scene aligned across views while the lighting changes. Across the four fitting lights, the leading four directions of the within-scene contrasts explain 88.6% of their variance.

Each model uses four lighting views per fitting scene. For each query, we retrieve the closest of eight gallery photos under a different light using cosine similarity. With 22 fitting scenes per model across 32 disjoint groups, our method reaches 46.97% accuracy versus 35.16% for Q-PCA. With one fit on 745 scenes and 30 disjoint test galleries, it reaches 54.06% versus 34.48%. See our response to Reviewer 1 (s9yS), Point 1, for the full comparison.

The revised manuscript tests retrieval across physical lighting changes in repeated captures of the same scene. Section 6.2 (Table 2) reports the results, and Appendix G.5 gives the protocol and variance diagnostics.


 

Q2. What replaces the pairwise screen? Please refer to our response to Reviewer 1 (s9yS), Point 2 for the full comparison. We agree that the pairwise screen can be too conservative because it can discard useful candidate powers before likelihood ranking. We replaced it with CV based on the matrix profile negative log-likelihood and a separate coverage check that can trigger a fallback. In the 745-scene Multi-Illumination fit, the old rule selects α=1 and reaches 46.56% retrieval accuracy, while the revised selector selects α=4 and reaches 54.06%. In the incomplete-coverage trials, the fallback reduces severe failures from 50 to 7. In the complete-coverage stress trials, the check never triggers and the old screen has fewer severe failures (10 versus 50). Section 5 and Appendix E.3 describe the revised selector. Section 6.3 reports the fallback results.


 

Q3. Unknown or misspecified background rank. Please refer to our response to Reviewer 1 (s9yS), Point 3 for the rank estimator and the full comparison. We estimate the unknown background rank mm using an elbow rule that selects the index with the largest ratio of consecutive eigenvalues of the training contrastive covariance, ordered from largest to smallest. The estimator recovers the true rank in all 50 complete-coverage block-action trials and all five Optdigits splits. In the 216 complete-coverage stress trials, estimating the rank changes the selected power in 25 trials, while mean loss rounds to 0.367 with either rank. On Optdigits, supplying a rank one below the truth lowers accuracy from 85.98% to 72.02%. Appendix G.6 (Table 8) reports these results.

References

[1] Lukas Murmann, Michaël Gharbi, Miika Aittala, and Frédo Durand. A Dataset of Multi-Illumination Images in the Wild. ICCV, 2019.

Dear Reviewer Qvub,

Thank you again for your review. We have posted our point-by-point responses and updated the manuscript with additional comparisons and clarifications addressing your questions. We would appreciate any further feedback you may have, particularly on whether these revisions address your concerns. We are happy to clarify any remaining points.

Best, authors

Thank you for the response. The new comparisons show that no single fixed method works well in all settings, while your method stays close to the best, which supports adaptive selection. The real-image evidence is still small in scale, but my main concerns are addressed. I raise my rating to 3.

Reviewer_q72YOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work is working on PCA, a classic and well-known problem. The author is focusing on a special sub-problem about PCA: learning the representation from an augmentation collection, which requires knowing and suppressing the background. They proposed Adaptive Invariance PCA (AIPCA) to learn from different views of data that the data augmentation created. They assume the data $ x= Az + Bc +\epsilon $ as signal, plus linear decomposed background information, plus noise, so is an unsupervised matrix factorization problem with three structural terms. Compared to PCA, maybe it’s closer to linear representation learning from multiple augmented view. Experiments use synthetic and semi-synthetic data to validate their claim.

Critical strengths
  1. The idea of using invariance with finite data to learn the representation is clear and stratforward.

  2. The theorem covers detailed discussion including power path, population recovery, and its proportional-dimensional risk. The separation of directional distortion and sampling error is making sense.

Critical weaknesses
  1. The theory assumptions of their augmentation related method are not natural and flexible. And the reviewer is unclear where the assumptions are held in real PCA scenario. It relies on AzAz canceled within source differences. This is essentially assumed the signal was completely unchanged, and the background must additively live in a fixed low-rank subspace shared for all sources. Meanwhile, the background must be independent from the signal, which is not always true. Meanwhile, the problem is, many operations like rotation, translation, blur, (just name a few) are not additive operations. The multi-view / augmentation at different angles will contain signal directions.

1 (b). Following above theory side weakness, the corresponding experiments are missing as the results were almost synthetic and tailored to their theorem. The reviewer is not objecting to synthetic experiments, but there have to be a use case with more reasonable setting to use their PCA method. For example, maybe multi-view sensor problem as they are measuring the same object and can do it repeatedly. It make the estimation of nuisance subspace more natural. More theoretical analysis and experiments were needed.

  1. The practical performance and significance were unclear. The Q-PCA, as shown in table 6, looks equal or better in almost all settings. And such comparison and discussion seem missing in the main paper. Besides, please correct me if I were wrong, the PCA++ was proposed to normalize the covariance of one view and truncates it to rank s, while the baseline is untruncated versions. These should also need to be discussed and compared in experiments.

  2. The identification coming from second moments and changes across channels. This work looks closer to ICA like auxiliary-variable based ICA, rather than PCA. It would be helpful if the author could clarify the novelty against existing methods.

Questions for authors
  1. Could the author report Q-PCA and truncated PCA++?

  2. What does the error of identification when signal are (slightly) different across the views?

  3. Could the author find a real experiments that the assumptions approximately holds, and the AIPCA was better than classical baslines like hard projection, etc?

4 comments
AuthorsOpenReview ↗

Official response

Thank you for your inspiring comments which give such chance to further clarify our contributions and connections/differences with prior work. We clarify what paired views can identify, test how changes in the signal affect recovery, and add a scene retrieval experiment under natural lighting changes.


 

Q1. Q-PCA and truncated PCA++. With 2 views and α=1, our method solves the same uniformity-constrained alignment problem as PCA++ using the pooled covariance of the 2 views. PCA++ uses the covariance of the first view. Before truncation the 2 objectives coincide when the normalization covariances agree, as they do at the population level in the PCA++ model. Appendix C.1 of our manuscript derives this connection. Our method uses all available views and selects α to vary the strength of suppression. PCA++ truncation serves a separate purpose. It restricts the whitening step to the leading covariance directions while keeping the output rank fixed (see Section 4 and Appendix C of [1]).

We fix the same pair of Optdigits views and cross-moment matrix, then vary the normalization covariance and retain 16, 32, or all 64 of its directions before extracting 8 components. The table reports mean test accuracy (%) over the same 5 splits. In the matched Optdigits CV comparison, Q-PCA reaches 84.94% under the original illumination and 88.80% at ρ=0.99.

Optdigits settingFirst view, 16First view, 32First view, 64Pooled, 16Pooled, 32Pooled, 64
Original illumination72.9172.0171.3372.6972.5472.73
Shared illumination, ρ=0.9969.7469.8267.5269.6069.8070.26

On this pair, truncation improves the first-view PCA++ fit in both settings (Appendix G.3, Table 6). With pooled normalization it has little effect. The pooled 64-direction fit corresponds to our method with α=1 on the same pair. Under the same Optdigits holdout protocol, our method with adaptive power selection and all 4 views reaches 84.77% and 85.98% in the 2 settings. This last comparison combines the use of additional views with power selection.

We now report Q-PCA in the main Optdigits comparison (Section 6.2, Table 3) and compare truncated and untruncated PCA++ on the same two views (Appendix G.3, Table 6). Truncation helps with first-view normalization on this pair. Appendix C.1 derives when the untruncated two-view objectives coincide.


 

Q2. When signal changes between views. The shared part of the signal still cancels in within-source differences, even if signal and background are dependent (Appendix G.7). The part that changes across views can enter the contrasts and tilt the estimated background space. This effect is larger when the weakest background direction is hard to distinguish from noise.

We simulate the within-source contrasts directly (400 contrasts per repetition, 500 repetitions). To represent signal changes across views, we add an independent component to each contrast along a direction 45° from the background. As the variance of this component increases from 0 to 0.2, mean projector error rises from 0.187 to 0.335 at a background eigengap of 0.25 and from 0.022 to 0.031 at an eigengap of 4.

Write a within-source contrast as B+H+εB+H+\varepsilon, where BB is the background variation, HH is the signal variation, and ε\varepsilon is isotropic noise uncorrelated with both. Let G=Cov⁡(B)G=\operatorname{Cov}(B), L=Cov⁡(H)L=\operatorname{Cov}(H), and C=Cov⁡(B,H)C=\operatorname{Cov}(B,H). Adding HH changes the contrast covariance by Δ=L+C+C⊤\Delta=L+C+C^\top.

For unit vectors u,vu,v, Cauchy–Schwarz gives

\le \sqrt{(u^\top G u)(v^\top L v)} \le \sqrt{\|G\|\,\|L\|}.$$ Taking the supremum over $u,v$ gives $\|C\|\le\sqrt{\|G\|\,\|L\|}$. The triangle inequality then gives $\|\Delta\|\le\eta=\|L\|+2\sqrt{\|G\|\,\|L\|}$, where $\|\cdot\|$ is the operator norm. In the simulation, $B$ and $H$ are independent, so $C=0$ and $\Delta=L$. With full coverage, $G$ has rank $m<d$. Before adding the changing signal, the population contrast covariance is $G+\sigma^2 I$, where $\sigma^2$ is the isotropic noise variance in the contrast. Its $m$th eigenvalue is $\sigma^2+\lambda_m(G)$ and its $(m+1)$st is $\sigma^2$, so the gap is $\lambda_m(G)$. If $\|\Delta\|<\lambda_m(G)/2$, the Davis–Kahan bound gives an operator-norm change in the population background projector of at most $2\|\Delta\|/\lambda_m(G)$. A shared signal cancels even when it depends on the background. **Changes in the signal can rotate the estimated background space, with larger errors when its eigengap is small.** We added a projector perturbation bound and a simulation that varies the eigengap in Appendix G.7.
AuthorsOpenReview ↗

Official response 2


 

Q3. Scope of the view model and evidence. The view model applies when repeated measurements share approximately stable content in a common coordinate system. Multi-Illumination [2] provides one example: each indoor scene is photographed from a fixed viewpoint under different physical lights. Across the four fitting lights, the leading four contrast directions explain 88.6% of the within-scene contrast variance, and the total variance of the scene means is about five times the total contrast variance. Shadows and reflections can still change scene features.

Cross-light scene retrieval on Multi-Illumination. See our response to Reviewer 1 (s9yS), Point 1, for the retrieval protocol. We report mean retrieval accuracy (%) for both fitting sizes below. Section 6.2 (Table 2) and Appendix G.5 give the full comparison and variance diagnostics.

ProtocolMean PCACross-moment PCAQ-PCAOld screening based ruleOurs
Matched size, 32 blocks33.5033.1135.1646.0946.97
Larger fit, 30 galleries32.7132.7134.4846.5654.06

Our method improves on Q-PCA at both fitting sizes. With 745 fitting scenes, it reaches 54.06% accuracy, compared with 34.48% for Q-PCA.

Controlled illumination on Optdigits. We use the same digit classification experiment as in our response to Reviewer 1, Point 1. The table reports mean classification accuracy (%) over five splits per setting (Section 6.2, Table 3, and Appendix G.3).

SettingMean PCACross-moment PCAQ-PCAOurs
Original67.8868.8784.9485.06
ρ=071.5071.0488.8088.56
ρ=0.972.4072.4288.8488.58
ρ=0.9972.4272.4888.8088.54

Q-PCA and our method achieve similar accuracy under controlled illumination, where the view contrasts reveal the shared illumination directions. Together, the two datasets show that our method adapts across lighting conditions. It remains competitive when Q-PCA works well and improves retrieval under physical lighting changes.

Figure 3 simulations. See our response to Reviewer 1, Point 1, for the full simulation comparison. With strong signal and only 2 views, our method reduces mean projector loss from 0.908 for Q-PCA to 0.071.

Overall, our method achieves the best or close to the best performance in all studies, showing its adaptivity. We have added these comparisons to Sections 6.1–6.2 (Figure 3 and Tables 1–3).


 

4. Relation to auxiliary-variable ICA (Point 3). Our identification result shows when changes across paired views jointly identify the entire background subspace. In the sign-flip example (Example 2.1), each channel has the same marginal distribution, so a classifier cannot distinguish observation–channel pairs from pairs with shuffled channel labels at the population level. Yet differences between views of the same source cancel the shared signal and reveal a different background direction for each flip. Every fixed pair misses at least one direction, while the full set identifies the background span. Auxiliary-variable ICA [3] uses changes in the conditional latent distributions to recover components under its variability assumptions. Their result does not give the joint coverage condition for this paired design.

Pairing supplies information even when all channels have identical marginal distributions. We added the comparison with auxiliary-variable ICA in Appendix A and illustrate the distinction with Example 2.1 in Section 2.2.

References

[1] Mingqi Wu, Qiang Sun, and Yi Yang. PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning. arXiv:2511.12278, 2025.

[2] Lukas Murmann, Michaël Gharbi, Miika Aittala, and Frédo Durand. A Dataset of Multi-Illumination Images in the Wild. ICCV, 2019.

[3] Aapo Hyvärinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning. AISTATS, 2019.

Reviewer_q72YOpenReview ↗

Response to the author

The reviewer thanks the author for the response. Most conceren were addressed or partially addressed.

I have minor concern for the theorectical gurantee for selector, and the emperical advantages in real world. But I think it's fine to leave it in fufure.

I want to encourage this line of work, and I think it's enough for me to raise my score to 3.

Thank you for your follow-up and positive assessment of our responses and revised manuscript. Your feedback helped us clarify the method, strengthen the comparisons, and improve the overall presentation. We also want to echo your earlier point that the work may be best framed as linear representation learning from multiple augmented views and will make this clearer in the final version.

Reviewer_s9ySOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AIPCA presented in this manuscript combines augmentation-coverage identifiability, a matrix-power PCA family, asymptotic risk analysis, and a validation-based selector, evaluated on synthetic data and controlled Optdigits augmentations.

Critical strengths

Coverage theory clarifies the value of augmentation diversity. The risk analysis explains when intermediate powers outperform endpoints.

Critical weaknesses
  1. The benefit over simpler alternatives is still unclear. My main concern is that mean-PCA and cross-moment PCA are missing, while Q-PCA already achieves higher mean accuracy across Table 6. So, what do we gain by restricting adaptation to the power family? I’d like to see matched comparisons with these methods, including simple CV selection among them, on a few representative existing settings. A meaningful gain under the same data splits, supported by uncertainty estimates, would substantially strengthen the practical contribution.

  2. I’m not yet convinced that screening justifies its complexity. In the reported q>0q>0 stress trials and image experiments, it retains only the reconstruction reference, and the calibration also undercovers. The main-text description makes it rather difficult to see what each stage contributes. Could you explain where screening actually helps on top of Table 3? If its purpose is to prevent bad selections, I’d particularly like to see whether it reduces severe failures, together with an explanation of how the undercoverage affects that protection. A concise account of the stages would also help.

  3. The reliance on knowing the background rank is a further concern and needs additional clarification. In section 4.2, you acknowledge that an unknown mm must be estimated, but the experiments supply it. Well, in practice, that estimate may be slightly wrong and how much would this change the result?

For me, the first two points are more critical. Convincing evidence of an advantage over the simple baselines and a useful role for screening, supported by reasonable rank robustness, would give me a strong reason to raise my score.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Official response

Thank you for raising questions about the baselines, the role of screening, and the background rank. We added matched comparisons, simplified power selection, and tested how rank estimation and misspecification affect the results.


 

1. Added comparisons. Thank you for suggesting these comparisons. We add mean PCA, cross-moment PCA, and Q-PCA as baselines. We also add reconstruction CV as a separate baseline for power selection. For each candidate power, reconstruction CV compares two projections of each held-out view: one is onto the fitted subspace at that power, and the other is onto the orthogonal complement of the estimated background subspace, both estimated using the training data. Reconstruction CV selects the power with the smallest mean squared distance between the two projections. We use profile likelihood CV and fall back to reconstruction CV when the coverage check rejects (Ours). See Section 5. Our method is competitive across the Figure 3 simulations and substantially reduces loss relative to Q-PCA and reconstruction CV in the strong-signal setting with 2 views. In cross-light retrieval, our method outperforms Q-PCA at both fitting sizes and improves on reconstruction CV by 7.50 percentage points with 745 fitting scenes, while all 3 methods achieve similar accuracy on Optdigits.

Figure 3 simulations. The table reports mean normalized projector loss over 24 paired trials per setting from Section 6.1 and Figure 3 of the revised manuscript (reconstruction CV comparison in Table 1). Lower is better. Check the Figure 3 simulation comparison at this anonymous link or the following table:

SettingViewsMean PCACross-moment PCAQ-PCAReconstruction CVOurs
Changing background20.95710.75070.20150.20150.1901
Changing background80.10370.07570.01730.01730.0151
Changing background160.04480.03790.00790.00790.0068
Persistent background20.99860.99830.90250.90290.9658
Persistent background80.99920.99910.11110.11110.1111
Persistent background160.99850.99850.04850.04850.0485
Strong signal20.06880.06350.90820.90790.0711
Strong signal80.01610.01550.11870.11870.0151
Strong signal160.00760.00750.04940.04940.0069

Across all nine settings, our method achieves the lowest mean loss or is competitive with the best of the five methods. With strong signal and two views, Q-PCA and reconstruction CV both have losses near 0.91, compared with 0.071 for our method and 0.064 for cross-moment PCA. Here Q-PCA commits to the single weakest empirical contrast direction, which can poorly estimate the signal direction with only two views. Under persistent background, mean PCA and cross-moment PCA have losses near one at every view count because both retain the shared background. Averaged across all nine settings, our method and the old screen have similar losses.

AuthorsOpenReview ↗

Official response 2

Cross-light scene retrieval. We evaluate our method on the task of matching photos of the same scene under different lighting conditions (Section 6.2, Table 2, and Appendix G.5). We use the official training split of the Multi-Illumination dataset [1], which contains 985 indoor scenes. Each scene is photographed from a fixed viewpoint under 25 lighting conditions, for a total of 24,625 photographs in the training split.

Each method learns a 4-dimensional representation of 4 × 4 grayscale images, using photos from 4 lighting conditions per fitting scene. A test gallery contains 1 photo from each of 8 scenes, all taken under the same lighting condition. We use 4 photos per test scene, each taken under a different lighting condition from the gallery photo, as separate queries. For each query, we retrieve the gallery photo with the highest cosine similarity in the learned representation. Retrieval is correct if the two photos show the same scene. Fitting and testing use separate lighting conditions.

We consider two settings.

  • Setting 1: 22 fitting scenes per model. We divide 960 scenes into 32 disjoint groups of 30. In each group, we fit a separate model on 22 scenes and use the other 8 for its test gallery.
  • Setting 2: 745 fitting scenes for one model. We fit 1 model on 745 scenes and divide the remaining 240 into 30 test galleries of 8 scenes each.

All methods use the same features and splits. Reconstruction CV, the old screening rule, and our method use the same 5 folds for power selection within each fitting set, keeping all views of each scene in the same fold. The coverage check did not trigger a fallback in any of the 33 fits.

ProtocolMean PCACross-moment PCAQ-PCAReconstruction CVOld screening based ruleOurs
Matched size, 32 blocks33.5033.1135.1646.0946.0946.97
Larger fit, 30 galleries32.7132.7134.4846.5646.5654.06

Our method improves on Q-PCA at both fitting sizes. In the expanded comparison in Section 6.2, Table 2, difference PCA is slightly ahead at 22 fitting scenes (47.07% versus 46.97%). Our method leads at 745 fitting scenes (54.06% versus 52.71% for difference PCA). The old screening based rule selects the same powers as reconstruction CV in both experiments. With 22 fitting scenes, the 3 selectors perform similarly. With 745, our method selects α=4, while the old rule and reconstruction CV select α=1. This choice improves retrieval accuracy by 7.50 percentage points over both on the held-out galleries.

The paired gain over Q-PCA was 11.82 percentage points with 22 fitting scenes (95% bootstrap interval: 7.81 to 15.92, resampling the 32 groups) and 19.58 points with 745 fitting scenes (14.27 to 24.90, resampling the 30 galleries with the fit held fixed). In the larger fit the gain over both reconstruction CV and the old rule was 7.50 points (95% bootstrap interval: 2.50 to 12.71).

Controlled illumination on Optdigits. We repeat the controlled illumination experiment from Table 6 of the submitted manuscript. Section 6.2, Table 3, of the revised manuscript reports the matched CV comparison, with mean accuracy (%) over five splits per setting. Appendix G.3 describes both protocols. We apply the coverage check for Ours on every split. It did not trigger in any of the 20 splits shown.

SettingMean PCACross-moment PCAQ-PCAReconstruction CVOurs
Original67.8868.8784.9485.0685.06
ρ=071.5071.0488.8088.5688.56
ρ=0.972.4072.4288.8488.5888.58
ρ=0.9972.4272.4888.8088.5488.54

In this application, Q-PCA, reconstruction CV, and our method all achieve nearly the best accuracy across all four settings. The added illumination can brighten or darken the whole image, or vary gradually from left to right or from top to bottom. In the shared-illumination settings, the persistent background (illumination that stays the same across views of a digit) and the changing background (illumination that varies from view to view) use the same brightness patterns. Differences between views of the same digit cancel the digit and the persistent illumination, leaving changes in illumination and noise. Q-PCA estimates the lighting directions from these differences and fits PCA in their orthogonal complement, suppressing both kinds of illumination (because they share the same patterns). When the background space is poorly estimated, hard projection (Q-PCA) can discard signal, as in the strong-signal simulation with 2 views above.

Overall, our method achieves the best or close to the best performance in all studies, showing its adaptivity. We added these comparisons to Sections 6.1–6.2 (Figure 3 and Tables 1–3).


References

[1] Lukas Murmann, Michaël Gharbi, Miika Aittala, and Frédo Durand. A Dataset of Multi-Illumination Images in the Wild. ICCV, 2019.

AuthorsOpenReview ↗

Official response 3


 

2. Screening and severe failures. We agree that the old pairwise screen adds complexity because it calibrates bounds separately and checks each candidate against the reconstruction choice. In the revised manuscript, we replaced pairwise screening with matrix profile likelihood and a coverage fallback (Ours). For each candidate power, we hold the signal and background spaces fitted on each training fold fixed and minimize the validation negative log likelihood over the covariance parameters. We then select the power with the lowest average profiled negative log-likelihood across folds (Section 5, Eqs. (12)–(14)). A separate coverage check switches to reconstruction when it finds evidence of weak contrast variance relative to noise in a background direction (Appendix E.3).

The table counts trials with normalized projector loss above 0.9, using the same trials for all 3 rules. The first row uses the 50 incomplete-coverage trials with 2 or 3 views from Appendix G.4 (Table 7). The second row uses the 432 complete-coverage stress trials from Appendix G.2 (Table 5). Section 6.3 reports the counts for matrix profile with and without fallback.

Simulation settingOld screening based ruleMatrix profile aloneOurs w/ coverage fallback
Structurally deficient coverage6/5050/507/50
Stress trials10/43250/43250/432

Under deficient coverage, fallback switched to reconstruction in 49/50 trials and reduced severe failures from 50 to 7. In the stress trials, it never switched and the old screen prevented more failures. Persistent background with 2 views accounts for 49 of the 50 severe failures of the revised rule. In the image experiments, the revised selector ties the old rule in the reported Optdigits holdouts, and our new method improves retrieval accuracy from 46.56% to 54.06% with 745 fitting scenes in the Multi-Illumination dataset. The old screening rule can be overly restrictive and discard useful powers before likelihood ranking. In this experiment, it selected α=1, the same power as reconstruction CV. Our method selected α=4.

We replaced pairwise screening with profile likelihood CV and a coverage check (Section 5 and Appendix E.3). The fallback reduced severe failures from 50 to 7 under deficient coverage. The old screen can be overly conservative and discard candidate powers that could improve performance before they are ranked by likelihood. Section 6.3 and Appendix G.4 describe the coverage designs and the fallback results.


 

3. Background rank (Appendix G.6, Table 8). We assess rank sensitivity in 50 complete-coverage block-action trials with four or five views, 216 complete-coverage stress trials with true background rank 12, and five Optdigits holdout splits at ρ=0.99. Within each comparison, we keep the representation dimension kk fixed. To estimate the background rank mm, we order the eigenvalues of the training contrastive covariance from largest to smallest and choose the index with the largest ratio to the next eigenvalue. We search indices 1 through min⁡(20,d−k)\min(20,d-k), where dd is the ambient dimension. The fit and the coverage check each use their own training data.

The simulation rows report mean normalized signal projector loss. A dash means that the supplied rank would exceed d−kd-k.

SettingTrue rankEstimated rankRank − 1Rank + 1
Optdigits, ρ=0.99 (accuracy)85.98%85.98%72.02%85.98%
Block-action, complete coverage (4–5 views)0.0010.0010.999—
Stress trials, complete coverage (true rank 12)0.3670.3670.3580.367

The estimator recovers the true rank in all 50 complete block-action trials and all five Optdigits splits. In the stress trials it changes the selected power in 25 of 216 trials, while mean loss rounds to 0.367 with either rank. Supplying one rank too few can be costly. It raises loss in the block-action trials and reduces Optdigits accuracy from 85.98% to 72.02%. Section 6.3 and Appendix G.6 (Table 8) report the rank estimation and sensitivity results.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.