- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes GRIT, which estimates an unstable subspace from invariant-pair differences and projects it out before fitting a predictor on fixed pretrained features. The paper also analyzes how pair count, noise, coverage, and removal rank affect target risk, provides matching constructions for leading dependencies, and evaluates the method against several objectives and prediction consistency.
Critical strengths
The paper gives a theoretical analysis that links finite, noisy pair information to subspace estimation and to the target risk of the fitted predictor. Its bounds separate the roles of coverage, noise, and rank selection, and the lower-bound constructions match the main dependence on the number of pairs and on noise. In this way, it gives a fairly concrete statistical account of how the projection method behaves and what its limitations are.
Critical weaknesses
- It seems the methodological novelty is insufficient. With matched pairs, weights, removal rank, GRIT’s core projection coincides with classical NAP (Nuisance Attribute Projection), which is uncited, undiscussed. Finite/imperfect invariance supervision already covered in consistency and augmentation theory. Needs sharper comparison: what substantive advance beyond these precedents?
- Theoretical analysis has limited practical implications. Guarantees assume clean pair differences span the true unstable subspace; noise independent and isotropic, not established for real-feature experiments. They concern population-risk minimizers, retain unknown shift and predictor-dependent quantities. No operational test of pair sufficiency; no certification of trained models. Systematic analysis, but added guidance for practical method design is unclear.
Questions for authors
-
How does GRIT’s projection differ from classical NAP? What methodological contribution remains beyond this precedent?
-
Which theoretical conclusions are substantively new? What insight beyond combining subspace perturbation and risk-comparison bounds?
-
Evidence for coverage and isotropic-noise assumptions on pretrained-feature benchmarks? Without it, how strongly can the theory explain observed improvements?
-
Does GRIT’s advantage over consistency persist when regularization grid extends beyond reported boundary optima, and consistency uses the same estimated rank-r projector? These controls separate spectral truncation from hard enforcement.
3 comments
Practical implications of the theory (W2, Q3)
"Theoretical analysis has limited practical implications. ... No operational test of pair sufficiency; no certification of trained models."
The bounds explain how the number, noise and coverage of pairs determine robustness. Certifying the test risk of a trained model would require estimates of the unknown shift and unstable subspace. We see this as natural future work.
"... added guidance for practical method design is unclear."
The theory tells us how many pairs to collect, what they should vary, which rank to use and what noise GRIT tolerates.
- Pair error falls as at a fixed noise-to-signal ratio. Thm. 2 gives this rate, and Prop. 3 shows that it is tight. Empirically, few pairs suffice, and GRIT beats PC's best mean with a fraction of the pairs on both datasets (App. F.5, Table 6).
- Pairs should vary every unstable direction, because more pairs of one type cannot recover a direction they never vary. Recovery also requires the weakest varied direction to stand out from the pair noise. On CueMNIST, recovery and accuracy follow both requirements (Sec. 6.2). On UrbanCars, combining background and object pairs raises worst-group accuracy to 71.9%, from 56.7% and 43.8% with either type alone (Sec. 6.3).
- The rank should match the number of unstable directions, and we select it on validation (Sec. 3). Removing too few directions leaves unstable sensitivity (Prop. 4), and removing too many discards stable information (Thm. 2).
- The noise does not need to be isotropic. Prop. 2 only requires that the noise does not mix stable and unstable directions or push a stable direction above an unstable one.
"Evidence for coverage and isotropic-noise assumptions on pretrained-feature benchmarks?"
On pretrained features the unstable subspace is unknown, so we test what the assumptions predict. We added a controlled study of anisotropic pair noise (new App. H.3). At , anisotropy does not hurt GRIT, but with condition number 100 and its accuracy falls to 2.71% (PC: 12.08%). On UrbanCars with frozen CLIP features, combining background- and object-changing pairs at a fixed budget of 256 pairs raises worst-group accuracy to 71.9%, versus 56.7% and 43.8% with either type alone (Sec. 6.3). This is consistent with the predicted benefit of broader coverage, although it does not directly verify latent-subspace coverage or noise isotropy.
Wider grids and the same rank- projector (Q4)
"Does GRIT's advantage over consistency persist when regularization grid extends beyond reported boundary optima ...?"
GRIT's advantage persists on the extended grids. After seeing the test results, we extended the grids at their boundaries and kept validation-only selection. Waterbirds-INV worst-group accuracy (%), submitted → extended, bold best and italic second (new Table 3, App. F.3):
| Objective | Vanilla | +PC | +LAM | +IPG | +GRIT |
|---|---|---|---|---|---|
| ERM | 52.69 → 52.67 | 66.84 | 51.98 → 50.38 | 42.25 → 43.28 | 70.09 |
| SD | 52.06 → 50.63 | 66.82 | 51.98 → 51.03 | 54.31 → 48.62 | 70.09 |
| RDM | 52.80 → 53.60 | 78.59 → 79.36 | 59.35 → 66.66 | 67.42 | 76.29 → 80.84 |
| IRMv1 | 47.78 → 51.92 | 77.86 | 47.68 → 49.79 | 64.60 → 60.31 | 78.78 |
| V-REx | 48.27 → 49.24 | 78.09 | 49.63 → 50.03 | 64.07 → 58.40 | 80.99 |
| Fishr | 47.16 → 49.48 | 76.72 → 77.88 | 66.30 → 76.16 | 75.01 → 74.08 | 80.84 |
| GroupDRO | 79.83 | 81.66 | 81.01 → 82.16 | 81.34 → 81.64 | 81.91 → 82.59 |
| LISA | 79.65 → 82.08 | 80.77 | 83.67 | 77.87 → 80.63 | 82.30 |
With the extended grids, GRIT has the highest mean in 15 of 16 rows, up from 14 of 16 in Table 1.
"... and consistency uses the same estimated rank-r projector?"
Role of the top directions (Thm. 3; new App. J). At the same rank, PC restricted to these directions nearly matches GRIT on UrbanCars (72.9% vs. 73.1%) but trails it on ColoredMNIST-INV (57.08% vs. 68.44%, post-test; Tables 14–15). As Thm. 2 permits, a soft penalty can do better with very few noisy pairs ( simulation).
We would be glad to clarify any of these points further.
Follow-up on our response
Dear Reviewer phR1,
As the discussion period ends tomorrow (Wed, Nov 18), we wanted to follow up on our response, including the NAP discussion and the extended-grid and rank-r projector controls you suggested.
Is there anything that remains unclear, or that we could clarify further?
Thank you again for your time and feedback.
We thank the reviewer for describing our analysis as "a fairly concrete statistical account of how the projection method behaves and what its limitations are." Section, table and appendix numbers refer to the revised PDF.
Comparison to NAP (W1, Q1)
Under matched conditions, GRIT and NAP compute the same projection. NAP removes the leading eigenvectors of a weighted second moment of pair differences [1, 2]. With equal weights, the Euclidean metric and the same rank, this is GRIT's projection. The introduction and Secs. 3 and 5 now discuss NAP, and the new App. B.3 derives the equivalence.
Our contribution is the analysis of this projection when it is estimated from finite, noisy pairs, together with its evaluation for OOD robustness. The NAP papers derive the projection algebraically and choose its rank by experiment for speaker verification, with no statistical analysis of pair count or pair noise. Our results relate pair count, noise, coverage and rank to the target risk of the fitted predictor.
Prior theory on imperfect pairs (W1)
To the best of our knowledge, no prior work relates pair count, pair noise and rank to OOD target risk. The closest analyses assume perfect pairs, rely on asymptotics, or study a single distribution.
Unlike prior theory, our analysis explains how the number of pairs, their noise and the chosen rank together determine target risk. Empirically, a few hundred pairs suffice when the rank is selected on validation.
What the theory adds (Q2)
The new conclusions are about the predictor fitted after projection. We build on standard perturbation and risk-comparison tools and obtain four results:
References for this comment:
[1] Solomonoff, Quillen & Campbell. Channel Compensation for SVM Speaker Recognition, Odyssey 2004.
[2] Campbell. Weighted Nuisance Attribute Projection, Odyssey 2010.
[3] Mahajan, Tople & Sharma. Domain Generalization using Causal Matching, ICML 2021.
[4] Heinze-Deml & Meinshausen. Conditional Variance Penalties and Domain Shift Robustness, Machine Learning 2021.
[5] Yang et al. Sample Efficiency of Data Augmentation Consistency Regularization, AISTATS 2023.