- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies linear representation learning under data augmentation: each image is passed through several augmentation channels to produce several views, in which the signal is unchanged while the background varies with the channel. It asks two questions: when a set of augmentations suffices to identify the background subspace, and how strongly cross-view variation should be suppressed with finite samples. The proposed method, AIPCA, uses a single matrix power α to connect pooled PCA, normalized alignment and difference PCA along one path, and selects α adaptively on validation data; on the theoretical side it gives a coverage condition and the asymptotic risk under a simplified model. The experiments comprise synthetic data and 8×8 Optdigits with artificially injected lighting, and adaptive power selection outperforms a fixed α = 1 only when the background is highly shared across views.
Critical strengths
The question of whether an augmentation portfolio covers all nuisance directions is formalized as positive definiteness of the coverage matrix H, and a sign-flip construction shows that every fixed pair of channels is unidentifiable while the full portfolio is identifiable; the problem statement is clear.
Critical weaknesses
- The adaptive selector never acts in practice, and a simpler baseline outperforms it
The paper's practical contribution is adaptive power selection, yet the authors themselves state that, in every real-data setting, the screening and profile steps never change the selection, so AIPCA reduces to choosing α by reconstruction error. Table 6 shows that Fixed Q-PCA, which simply performs PCA inside the estimated background complement, outperforms AIPCA at all five values of ρ (for example 87.64 vs. 86.04 at ρ = 0.99 and 82.20 vs. 77.30 at ρ = 0.999), and this counterexample appears only in the appendix; Table 1 in the main text does not include Q-PCA. The practical core of the method therefore does not hold.
- The experiments are far from the target setting
The only real data are 8×8 Optdigits with artificially injected linear lighting, which is far from the crops, color changes and blur described in the introduction. The ρ = 0.99 column of Table 1 is the most favourable point of the sweep: Table 6 shows that AIPCA equals α = 1 at ρ = 0.9 and falls below it at ρ = 0.9999.
Questions for authors
- Since Q-PCA is no worse than AIPCA in every row of Table 6, where does the advantage of AIPCA lie? Can you give a setting in which Q-PCA fails and AIPCA succeeds?
- Why is the pairwise screen retained? Table 3 shows that profile-only selection is better in most settings, and the screen never acts in any real-data setting.
- All experiments supply the true background rank m (m = 3 on Optdigits, where the three lighting components are injected by the authors), and both the selector's background estimate and the Q-PCA baseline depend on it; in a real augmentation portfolio m is unknown. How do the selector and the final loss behave when m must be estimated or is misspecified?
3 comments
Follow up
Dear Reviewer Qvub,
Thank you again for your review. We have posted our point-by-point responses and updated the manuscript with additional comparisons and clarifications addressing your questions. We would appreciate any further feedback you may have, particularly on whether these revisions address your concerns. We are happy to clarify any remaining points.
Best, authors
Thank you for the response. The new comparisons show that no single fixed method works well in all settings, while your method stays close to the best, which supports adaptive selection. The real-image evidence is still small in scale, but my main concerns are addressed. I raise my rating to 3.
Official response
Thank you for asking when adaptive power selection improves on simpler methods. We added matched comparisons, tested retrieval across physical lighting changes, and examined how the selector behaves under weak coverage and rank misspecification.
1 and Q1. When does our method improve on Q-PCA? With 745 fitting scenes in Multi-Illumination, our method selects
α=4and reaches 54.06% retrieval accuracy. Q-PCA reaches 34.48%. The gain over Q-PCA is 19.58 percentage points (95% paired bootstrap interval: 14.27 to 24.90). With 22 fitting scenes per model, our method reaches 46.97% and Q-PCA reaches 35.16%. See our response to Reviewer 1 (s9yS), Point 1 for the full Multi-Illumination comparison. On Optdigits, the two methods have similar accuracy under matched CV, where differences between views already reveal the illumination directions and thus Q-PCA achieves strong performance.In Figure 3, with strong signal and two views, mean normalized projector loss is 0.9082 for Q-PCA and 0.0711 for our method. Cross-moment PCA and mean PCA reach 0.0635 and 0.0688. See the full Figure 3 comparison in our response to Reviewer 1 (s9yS), Point 1.
Why Q-PCA can lose signal. In the population example in Appendix G.7, the signal direction has a component along the background direction. Q-PCA removes that component when it projects out the background, giving loss 0.5. At fixed power 0.1, our method retains part of the component and gives loss about 0.00010, compared with 0.00248 for pooled PCA.
Overall, our method achieves the best or close to the best performance across these matched comparisons. It reduces loss relative to Q-PCA in the strong-signal simulation with two views, improves cross-light retrieval, and achieves similar accuracy on Optdigits. We added the matched comparisons to Section 6.1 (Figure 3 and Table 1) and Section 6.2 (Tables 2–3), and the population example to Appendix G.7.
2. Experimental scope. We added a cross-light retrieval experiment on Multi-Illumination [1]. The official training split contains 985 indoor scenes, each photographed from a fixed viewpoint under different physical lights. The fixed viewpoint keeps the scene aligned across views while the lighting changes. Across the four fitting lights, the leading four directions of the within-scene contrasts explain 88.6% of their variance.
Each model uses four lighting views per fitting scene. For each query, we retrieve the closest of eight gallery photos under a different light using cosine similarity. With 22 fitting scenes per model across 32 disjoint groups, our method reaches 46.97% accuracy versus 35.16% for Q-PCA. With one fit on 745 scenes and 30 disjoint test galleries, it reaches 54.06% versus 34.48%. See our response to Reviewer 1 (s9yS), Point 1, for the full comparison.
The revised manuscript tests retrieval across physical lighting changes in repeated captures of the same scene. Section 6.2 (Table 2) reports the results, and Appendix G.5 gives the protocol and variance diagnostics.
Q2. What replaces the pairwise screen? Please refer to our response to Reviewer 1 (s9yS), Point 2 for the full comparison. We agree that the pairwise screen can be too conservative because it can discard useful candidate powers before likelihood ranking. We replaced it with CV based on the matrix profile negative log-likelihood and a separate coverage check that can trigger a fallback. In the 745-scene Multi-Illumination fit, the old rule selects α=1 and reaches 46.56% retrieval accuracy, while the revised selector selects α=4 and reaches 54.06%. In the incomplete-coverage trials, the fallback reduces severe failures from 50 to 7. In the complete-coverage stress trials, the check never triggers and the old screen has fewer severe failures (10 versus 50). Section 5 and Appendix E.3 describe the revised selector. Section 6.3 reports the fallback results.
Q3. Unknown or misspecified background rank. Please refer to our response to Reviewer 1 (s9yS), Point 3 for the rank estimator and the full comparison. We estimate the unknown background rank m using an elbow rule that selects the index with the largest ratio of consecutive eigenvalues of the training contrastive covariance, ordered from largest to smallest. The estimator recovers the true rank in all 50 complete-coverage block-action trials and all five Optdigits splits. In the 216 complete-coverage stress trials, estimating the rank changes the selected power in 25 trials, while mean loss rounds to 0.367 with either rank. On Optdigits, supplying a rank one below the truth lowers accuracy from 85.98% to 72.02%. Appendix G.6 (Table 8) reports these results.
References
[1] Lukas Murmann, Michaël Gharbi, Miika Aittala, and Frédo Durand. A Dataset of Multi-Illumination Images in the Wild. ICCV, 2019.