acceptodds
Under review as a conference paper at ICLR 2027

GRIT: Out-of-Distribution Robustness from Finite and Noisy Invariant Pairs

Abstract

Invariant pairs provide information about variation that should not affect prediction, but a finite, noisy collection can leave relevant directions unseen or suppress predictive information. We study out-of-distribution (OOD) robustness on fixed pretrained features, assuming that the label distribution given a linear projection of these features is invariant across environments. We propose Geometric Robustness Invariant Training (GRIT), which estimates a linear subspace from invariant pair differences and projects it out before predictor fitting, allowing the same constraint to accompany ERM or compatible robustness objectives. Our theoretical analysis bounds target risk through source prediction risk, a term related to covariate shift, and residual sensitivity along the unstable subspace. For GRIT, the bound depends on pair count, noise relative to the weakest clean signal, and selected rank. A parallel guarantee uses the fitted optimum to characterize quadratic prediction consistency under the same noise and shift assumptions. On ColoredMNIST-INV and Waterbirds-INV, reported means improve over each of eight evaluated objectives and exceed prediction consistency in 15 of 16 settings. On UrbanCars, combining pairs with background or object changes improves robustness to two spurious features.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Out-of-Distribution Robustness, Spurious Correlations, Invariant Pairs, Invariant Learning, Consistency Regularization, Domain Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_phR1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GRIT, which estimates an unstable subspace from invariant-pair differences and projects it out before fitting a predictor on fixed pretrained features. The paper also analyzes how pair count, noise, coverage, and removal rank affect target risk, provides matching constructions for leading dependencies, and evaluates the method against several objectives and prediction consistency.

Critical strengths

The paper gives a theoretical analysis that links finite, noisy pair information to subspace estimation and to the target risk of the fitted predictor. Its bounds separate the roles of coverage, noise, and rank selection, and the lower-bound constructions match the main dependence on the number of pairs and on noise. In this way, it gives a fairly concrete statistical account of how the projection method behaves and what its limitations are.

Critical weaknesses
  1. It seems the methodological novelty is insufficient. With matched pairs, weights, removal rank, GRIT’s core projection coincides with classical NAP (Nuisance Attribute Projection), which is uncited, undiscussed. Finite/imperfect invariance supervision already covered in consistency and augmentation theory. Needs sharper comparison: what substantive advance beyond these precedents?
  2. Theoretical analysis has limited practical implications. Guarantees assume clean pair differences span the true unstable subspace; noise independent and isotropic, not established for real-feature experiments. They concern population-risk minimizers, retain unknown shift and predictor-dependent quantities. No operational test of pair sufficiency; no certification of trained models. Systematic analysis, but added guidance for practical method design is unclear.
Questions for authors
  1. How does GRIT’s projection differ from classical NAP? What methodological contribution remains beyond this precedent?

  2. Which theoretical conclusions are substantively new? What insight beyond combining subspace perturbation and risk-comparison bounds?

  3. Evidence for coverage and isotropic-noise assumptions on pretrained-feature benchmarks? Without it, how strongly can the theory explain observed improvements?

  4. Does GRIT’s advantage over consistency persist when regularization grid extends beyond reported boundary optima, and consistency uses the same estimated rank-r projector? These controls separate spectral truncation from hard enforcement.

3 comments

We thank the reviewer for describing our analysis as "a fairly concrete statistical account of how the projection method behaves and what its limitations are." Section, table and appendix numbers refer to the revised PDF.

Comparison to NAP (W1, Q1)

"With matched pairs, weights, removal rank, GRIT's core projection coincides with classical NAP ..., which is uncited, undiscussed."

Under matched conditions, GRIT and NAP compute the same projection. NAP removes the leading eigenvectors of a weighted second moment of pair differences [1, 2]. With equal weights, the Euclidean metric and the same rank, this is GRIT's projection. The introduction and Secs. 3 and 5 now discuss NAP, and the new App. B.3 derives the equivalence.

"What methodological contribution remains beyond this precedent?"

Our contribution is the analysis of this projection when it is estimated from finite, noisy pairs, together with its evaluation for OOD robustness. The NAP papers derive the projection algebraically and choose its rank by experiment for speaker verification, with no statistical analysis of pair count or pair noise. Our results relate pair count, noise, coverage and rank to the target risk of the fitted predictor.

Prior theory on imperfect pairs (W1)

"... invariance supervision already covered in consistency and augmentation theory."

To the best of our knowledge, no prior work relates pair count, pair noise and rank to OOD target risk. The closest analyses assume perfect pairs, rely on asymptotics, or study a single distribution.

  • MatchDG's guarantee assumes perfect pairs. Its Thm. 1 [3] also requires infinitely many examples per domain.
  • CoRe's guarantee is asymptotic and removes every direction in which grouped examples differ. Its Thm. 1 [4] takes the penalty to infinity. Its model places those differences exactly in the style directions, so the pairs are perfect. With noisy pairs, the differences also span stable directions, which an infinite penalty would remove. Thm. 3 quantifies this cost for a finite penalty, and GRIT limits it by removing only rr directions.
  • Yang et al. analyze noisy augmentations only in distribution, and their OOD example assumes perfect pairs. Their domain-adaptation example [5, App. D] requires augmentations that leave the invariant features exactly unchanged. It also forces the predictor to be orthogonal to every pair difference, which would remove stable directions if the pairs were noisy. Their analysis augments every labeled example at least once, so it has at least as many pairs as labeled examples. We use 256 pairs for 50,000 training digits and 240 for 4,795 training images.

Unlike prior theory, our analysis explains how the number of pairs, their noise and the chosen rank together determine target risk. Empirically, a few hundred pairs suffice when the rank is selected on validation.

What the theory adds (Q2)

"Which theoretical conclusions are substantively new? What insight beyond combining subspace perturbation and risk-comparison bounds?"

The new conclusions are about the predictor fitted after projection. We build on standard perturbation and risk-comparison tools and obtain four results:

  1. Thm. 1 bounds target risk by source risk, shift in the stable features, and sensitivity to unstable features. Invariance controls only the third.
  2. Thm. 2 carries finite-pair error into target risk, and together with Prop. 4 it shows that a rank that is too small or too large can increase target risk.
  3. Prop. 3 shows that this finite-pair term is tight, even when source risk is minimized exactly.
  4. Thm. 3 shows that with noisy pairs, quadratic consistency also penalizes stable directions.

References for this comment:

[1] Solomonoff, Quillen & Campbell. Channel Compensation for SVM Speaker Recognition, Odyssey 2004.

[2] Campbell. Weighted Nuisance Attribute Projection, Odyssey 2010.

[3] Mahajan, Tople & Sharma. Domain Generalization using Causal Matching, ICML 2021.

[4] Heinze-Deml & Meinshausen. Conditional Variance Penalties and Domain Shift Robustness, Machine Learning 2021.

[5] Yang et al. Sample Efficiency of Data Augmentation Consistency Regularization, AISTATS 2023.

Practical implications of the theory (W2, Q3)

"Theoretical analysis has limited practical implications. ... No operational test of pair sufficiency; no certification of trained models."

The bounds explain how the number, noise and coverage of pairs determine robustness. Certifying the test risk of a trained model would require estimates of the unknown shift and unstable subspace. We see this as natural future work.

"... added guidance for practical method design is unclear."

The theory tells us how many pairs to collect, what they should vary, which rank to use and what noise GRIT tolerates.

  1. Pair error falls as 1/k1/\sqrt{k} at a fixed noise-to-signal ratio. Thm. 2 gives this rate, and Prop. 3 shows that it is tight. Empirically, few pairs suffice, and GRIT beats PC's best mean with a fraction of the pairs on both datasets (App. F.5, Table 6).
  2. Pairs should vary every unstable direction, because more pairs of one type cannot recover a direction they never vary. Recovery also requires the weakest varied direction to stand out from the pair noise. On CueMNIST, recovery and accuracy follow both requirements (Sec. 6.2). On UrbanCars, combining background and object pairs raises worst-group accuracy to 71.9%, from 56.7% and 43.8% with either type alone (Sec. 6.3).
  3. The rank should match the number of unstable directions, and we select it on validation (Sec. 3). Removing too few directions leaves unstable sensitivity (Prop. 4), and removing too many discards stable information (Thm. 2).
  4. The noise does not need to be isotropic. Prop. 2 only requires that the noise does not mix stable and unstable directions or push a stable direction above an unstable one.

"Evidence for coverage and isotropic-noise assumptions on pretrained-feature benchmarks?"

On pretrained features the unstable subspace is unknown, so we test what the assumptions predict. We added a controlled study of anisotropic pair noise (new App. H.3). At σ=0.1\sigma=0.1, anisotropy does not hurt GRIT, but with condition number 100 and σ=0.15\sigma=0.15 its accuracy falls to 2.71% (PC: 12.08%). On UrbanCars with frozen CLIP features, combining background- and object-changing pairs at a fixed budget of 256 pairs raises worst-group accuracy to 71.9%, versus 56.7% and 43.8% with either type alone (Sec. 6.3). This is consistent with the predicted benefit of broader coverage, although it does not directly verify latent-subspace coverage or noise isotropy.

Wider grids and the same rank-rr projector (Q4)

"Does GRIT's advantage over consistency persist when regularization grid extends beyond reported boundary optima ...?"

GRIT's advantage persists on the extended grids. After seeing the test results, we extended the grids at their boundaries and kept validation-only selection. Waterbirds-INV worst-group accuracy (%), submitted → extended, bold best and italic second (new Table 3, App. F.3):

ObjectiveVanilla+PC+LAM+IPG+GRIT
ERM52.69 → 52.6766.8451.98 → 50.3842.25 → 43.2870.09
SD52.06 → 50.6366.8251.98 → 51.0354.31 → 48.6270.09
RDM52.80 → 53.6078.59 → 79.3659.35 → 66.6667.4276.29 → 80.84
IRMv147.78 → 51.9277.8647.68 → 49.7964.60 → 60.3178.78
V-REx48.27 → 49.2478.0949.63 → 50.0364.07 → 58.4080.99
Fishr47.16 → 49.4876.72 → 77.8866.30 → 76.1675.01 → 74.0880.84
GroupDRO79.8381.6681.01 → 82.1681.34 → 81.6481.91 → 82.59
LISA79.65 → 82.0880.7783.6777.87 → 80.6382.30

With the extended grids, GRIT has the highest mean in 15 of 16 rows, up from 14 of 16 in Table 1.

"... and consistency uses the same estimated rank-r projector?"

Role of the top rr directions (Thm. 3; new App. J). At the same rank, PC restricted to these directions nearly matches GRIT on UrbanCars (72.9% vs. 73.1%) but trails it on ColoredMNIST-INV (57.08% vs. 68.44%, post-test; Tables 14–15). As Thm. 2 permits, a soft penalty can do better with very few noisy pairs (k=16k=16 simulation).

We would be glad to clarify any of these points further.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer phR1,

As the discussion period ends tomorrow (Wed, Nov 18), we wanted to follow up on our response, including the NAP discussion and the extended-grid and rank-r projector controls you suggested.

Is there anything that remains unclear, or that we could clarify further?

Thank you again for your time and feedback.

Reviewer_ufPTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GRIT, which leverages a finite set of noisy invariant pairs to improve out-of-distribution generalization on top of frozen pretrained features. GRIT estimates the principal subspace of feature differences between paired samples and trains a predictor on features with those directions removed. Because invariance estimation is decoupled from predictor training, GRIT can be combined with various objectives, including ERM.

Theoretically, the paper decomposes the target risk into the source prediction risk, the distribution shift of stable features, and the residual sensitivity along unstable directions. It analyzes how the number of pairs, pair noise, the weakest signal direction, and the number of removed dimensions affect the performance guarantee. It also provides a guarantee for quadratic prediction consistency under the same setting.

On ColoredMNIST-INV and Waterbirds-INV, GRIT achieves higher mean OOD performance than the base method across all eight objectives, measured by test accuracy and worst-group accuracy, respectively.

Critical strengths
  1. The method is simple yet compatible with a variety of training methods. Incorporating pair information through a projection is a simple design that applies uniformly across objectives, which makes it practical. In Table 1, GRIT achieves higher average performance than the base method for all eight objectives on both datasets. In particular, on Waterbirds-INV, worst-group accuracy improves from 52.69% to 70.09% for ERM and from 79.83% to 81.91% for GroupDRO. Using identical data and features for each objective, and comparing against prediction consistency with the same pair of information, strengthens the credibility of the results.

  2. The paper explains the relationship between pair quality and performance through both theory and experiments. Theorem 2 links subspace estimation error to the target risk and makes explicit both the difficulty of estimating weak signal directions and the cost of removing too many dimensions. Proposition 3 is also meaningful, as it presents a case in which even the GRIT predictor that minimizes the source risk depends on the number of pairs and the noise level. The CueMNIST and UrbanCars experiments complement this analysis by showing that which variations the collected pairs contain matters, not only how many pairs are collected.

Critical weaknesses
  1. Additional validation beyond the noise assumption would be helpful. The theory assumes that pair noise is not biased toward any particular direction. In real images, however, changing the background or other attributes may also alter information needed for classification, and GRIT may then remove part of that useful information. It is commendable that the authors acknowledge this limitation in Appendices A.3 and I.2. However, the current experiments alone make it difficult to judge whether the improvements persist in such settings, so additional validation is needed to establish the scope of applicability.

  2. Direct comparisons with closely related methods would be helpful. The experiments combining GRIT with multiple objectives demonstrate its versatility. However, comparisons centered on the base methods and prediction consistency make it difficult to assess the additional benefit over methods that use invariant pairs differently or that mitigate bias in frozen features. A comparison with IPG [1], which the manuscript discusses, would highlight differences in how the same pair of information is exploited. Comparisons with NeuronTune [2] or PRISM-mini [3] would highlight differences in how specific feature dimensions or subspaces are suppressed. Comparing against representative methods while accounting for their information requirements and training settings would make GRIT’s contribution clearer.

Questions for authors
  1. How sensitive is GRIT to anisotropic or systematically biased pair noise, or to changes in information needed for classification? Could you clarify the scope of applicability when the assumptions are not fully satisfied, through relevant analysis or experiments?

  2. Could you add comparisons with representative methods among IPG [1], NeuronTune [2], and PRISM/PRISM-mini [3] that are applicable to the frozen-feature setting? If the conditions can be matched, GIC [4] or PDE [5] would also be useful candidates. To the extent possible, it would be helpful to match the feature extractor as well as the data and validation budgets. Please also describe whether group labels, auxiliary data, or external models are used, along with any modifications made to the original methods.


[1] Surner, Martin, et al. “Invariance Pair Guidance: Robustness to Spurious Correlations via Corrective Gradients.” Machine Learning, vol. 115, 2026, article 188.

[2] Zheng, Guangtao, et al. “NeuronTune: Towards Self-Guided Spurious Bias Mitigation.” Proceedings of the 42nd International Conference on Machine Learning, vol. 267, PMLR, 2025, pp. 78454–78477.

[3] Molahasani, Mahdiyar, et al. “PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection.” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 688–697.

[4] Han, Yujin, and Difan Zou. “Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference.” Proceedings of the 41st International Conference on Machine Learning, vol. 235, PMLR, 2024, pp. 17480–17504.

[5] Deng, Yihe, et al. “Robust Learning with Progressive Data Expansion Against Spurious Correlation.” Advances in Neural Information Processing Systems, vol. 36, 2023.

4 comments

We thank the reviewer for the thoughtful review, and for recognizing both the practicality of GRIT and our theoretical and empirical analysis of how pair quality affects performance. Section, table and appendix numbers refer to the revised PDF.

Pair noise that is anisotropic, biased, or changes useful information (W1, Q1)

"The theory assumes that pair noise is not biased toward any particular direction."

Isotropic noise is sufficient for recovering the unstable subspace, but not necessary. Proposition 2 requires only that the average covariance of pair differences does not mix stable and unstable directions, and that its leading directions are the unstable ones (Sec. 4). We use isotropic noise (Assumption 2) to obtain the finite-pair rate in Theorem 2.

"How sensitive is GRIT to anisotropic ... pair noise, or to changes in information needed for classification?"

On CueMNIST, GRIT tolerates anisotropic noise aimed at a predictive direction unless the noise is both strong and highly anisotropic (new App. H.3). We add noise to 256 CueMNIST pair differences. Its dominant direction mixes a nuisance cue with the class-mean direction, so the pairs also change information needed for classification. We vary the noise's condition number κ while keeping its total energy fixed. Target accuracy (%, mean of ten replicates):

Noise level σMethodκ = 1 (isotropic)κ = 10κ = 100
0.1GRIT75.8678.5884.26
0.1PC71.5855.1524.40
0.15GRIT69.9873.732.71
0.15PC66.0831.9612.08

Fig. 8 has the full sweep.

"How sensitive is GRIT to ... systematically biased pair noise?"

If every pair also shifts a stable feature by more than it varies the nuisance, GRIT removes the stable direction no matter the number of pairs (new App. H.3). In a balanced two-dimensional example, the average pair covariance is the same for every pair count, so the error comes from how the pairs were built, not from estimating the subspace.

"... the current experiments alone make it difficult to judge whether the improvements persist in such settings ..."

Our Waterbirds-INV pairs are real image edits that can also change useful information, and GRIT still improves all eight objectives (Table 1). Each pair places the same bird on a different background, and Waterbirds backgrounds can contain other birds [6], so a swap may add label-relevant content. We have not measured how often this happens in our pairs. This shows GRIT can remain useful with imperfect real pairs, while App. H.3 shows that changes overlapping predictive directions can cause failure (Sec. 7). Natural next steps are to extend the finite-pair analysis beyond isotropic noise and to test edits that deliberately change label-relevant content.

[6] Asgari Taghanaki et al. MaskTune: Mitigating Spurious Correlations by Forcing to Explore, NeurIPS 2022.

Comparisons with methods that use pairs or other information (W2, Q2)

"Could you add comparisons with ... IPG [1], NeuronTune [2], and PRISM/PRISM-mini [3] ... GIC [4] or PDE [5] would also be useful candidates."

We added all five methods, using the same frozen CLIP features, data splits, seeds and validation selection as GRIT (Table 1, App. F.3 and F.7). The methods use different information:

MethodInvariant pairsTraining group labelsValidation labels for fittingText prompts
GRIT✓
IPG✓
NeuronTune✓
PRISM-mini✓
GIC✓
PDE✓

Every method selects hyperparameters on the same validation set. None uses an LLM, extra images or test data, and PRISM-mini encodes its prompts with the same CLIP model. Mean accuracy (%) with a linear head: test accuracy on ColoredMNIST-INV and worst-group accuracy on Waterbirds-INV (95% intervals in Tables 4 and 8):

Linear headColoredMNIST-INVWaterbirds-INV
ERM17.0252.69
ERM + GRIT67.8970.09
IPG29.1842.25
NeuronTune24.2367.72
PRISM-mini + probe16.9852.65
GIC71.2158.12
GIC + GRIT72.3578.60
PDE39.9879.16
PDE + GRIT66.8482.28

"A comparison with IPG [1] ... would highlight differences in how the same pair of information is exploited."

On a linear head, GRIT uses the same pairs more effectively than IPG. GRIT has the highest mean in 14 of 16 rows of Table 1 (15 of 16 with the wider grids in App. F.3), and IPG in none. Because IPG was designed to train a representation, we also compare the two with a trainable MLP head, where each leads on one dataset (App. F.6).

To fit frozen features, we apply IPG's corrective update to the linear head, or to the MLP's hidden layer.

"Comparisons with NeuronTune [2] or PRISM-mini [3] would highlight differences in how specific feature dimensions or subspaces are suppressed."

GRIT outperforms both NeuronTune and PRISM-mini on both datasets. NeuronTune masks feature dimensions flagged by validation errors. It improves on its unmasked control, but stays below GRIT on Waterbirds-INV (67.72 vs. 70.09). PRISM-mini projects out the CLIP text embeddings of two words naming the nuisance, and performs about the same as ERM on both datasets.

We made two changes to fit frozen CLIP features. NeuronTune scores dimensions by their absolute value, because CLIP features can be negative. PRISM-mini uses fixed words instead of LLM-guided discovery (the released "water" and "land", and our choice of "red" and "green"), followed by a linear probe.

"If the conditions can be matched, GIC [4] or PDE [5] would also be useful candidates."

Adding GRIT improves both GIC and PDE. On Waterbirds-INV, GRIT raises GIC by 20.49 worst-group points and PDE by 3.12. On ColoredMNIST-INV, it raises GIC by 1.14 points and PDE by 26.86. Each method and its GRIT version share the same pipeline, so the gains measure what GRIT adds. GIC alone also beats ERM + GRIT on ColoredMNIST-INV.

With the released settings, GIC's first stage collapsed or diverged on our features, so we standardize its inputs and clip its gradients. PDE keeps its released schedule, but on a linear head its selected checkpoints all come from the group-balanced warm-up (App. F.7).

We hope these answers address the reviewer's concerns, and we would be glad to clarify anything further.

[1] Surner et al. Invariance Pair Guidance: Robustness to Spurious Correlations via Corrective Gradients, Machine Learning 2026.

[2] Zheng et al. NeuronTune: Towards Self-Guided Spurious Bias Mitigation, ICML 2025.

[3] Molahasani et al. PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection, ICCV 2025.

[4] Han & Zou. Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference, ICML 2024.

[5] Deng et al. Robust Learning with Progressive Data Expansion Against Spurious Correlation, NeurIPS 2023.

Thank you for taking the time to write such a detailed response. Your clarifications and additional results have addressed my concerns. I believe the paper makes a valuable contribution, and the revisions will strengthen it further.

I will maintain my current score.

Thank you for your follow-up and for your support of the paper. Your suggestions improved the paper.

Reviewer_8ZnmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method, namely geometric robustness invariant training (GRIT), to improve out-of-distribution robustness from finite, noisy invariant pairs. Given a set of invariant pairs, GRIT estimates a subspace such that projecting a feature vector onto it removes the unstable component; then the training fits the predictor with the projected features. The paper presents theoretical analyses showing when unstable directions can be extracted, logistic risk bounds under a noise assumption, and an additional analysis of existing quadratic prediction consistency. Experiments compare GRIT’s performance with the vanilla predictor and a competitor representing other existing approaches. Additional experiments show how signal direction and noise level affect performance, and the benefit of using two different invariance types.

Critical strengths
  1. The paper provides thorough theoretical analyses of the proposed idea's desired properties. Under these assumptions, the analyses prove properties that guarantee the proposed idea’s success when they hold.
  2. While there are some concerns about the experiments’ quality, some results demonstrate promising improvements with some additional practical findings. Please read Questions For Authors for more details.
Critical weaknesses
  1. The subspace existence assumption (Assumption 1) may be too strong. While the paper provides detailed theoretical results, they all depend on this assumption, which is unlikely to hold when the extracted features show complicated geometry. The authors should justify this assumption and discuss why it should be realizable in practice. This might not be as unlikely as the reviewer believes, so the authors should discuss the assumption during the review process.
  2. There are some concerns about the quality of the experiments. Rather than directly comparing OOD robustness with the other approaches mentioned in the related work, such as CoRe, MatchDG, and IPG, the authors defined another prediction consistency (PC) method that penalizes only paired logit differences. On the Waterbirds-INV dataset, the improvement seems marginal compared to PC.
Questions for authors
  1. Please justify how much Assumption 1 reflects real-world datasets. This could be justifying that Assumption 1 is not too strong, or thoroughly demonstrating that a small violation of this assumption would not undermine the performance in practice.
    • I may have misunderstood the statement, and Assumption 1 might hold even for data with complicated underlying geometry. If so, consider adding some experiment that demonstrates this. For example, we can try verifying Assumption 1 on some toy data (in the feature space) with some controllable geometric structure. This might help the reader understand Assumption 1.
  2. Regarding the experiments,
    • Why does the paper use prediction consistency (PC) rather than comparing the performance directly with existing works such as CoRe, MatchDG, and SelfReg? If there is no clear reason, please add the comparison to those competitors. Otherwise, justify the reason why comparison with PC would suffice.
    • Justify the reason why GRIT marginally improves the performance on the Waterbird-INV dataset. This might be some violation of assumptions, which reflects the hardness of Assumption 1.
    • It would be worth emphasizing the experimental results on the performance over different pair counts. Not only is the final target accuracy much higher than the competitor, GRIT also performs better early on. For example, at 256 pairs, GRIT already started outperforming the competitor. An additional experiment could demonstrate this sample efficiency, i.e., that fewer pairs are required to reach a target accuracy.
    • In Section 1, the authors pointed out that GRIT can serve as a preprocessing step. Because the improvement was marginal on some datasets, practitioners would also care about the preprocessing's efficiency. Consider providing some measurement of GRIT’s efficiency. Also, how does GRIT scale with the feature’s dimensionality or the required rank?
    • One interesting idea is visualizing the stable/unstable directions. This may require a feature extractor that can visualize the extracted features, so I’m unsure if this is always possible.
  3. For a better paper presentation, please outline the flow of the theoretical analysis at the beginning of Section 4. It would also help to briefly summarize each subsection's key takeaways in the outline.
6 comments

We thank the reviewer for the careful reading and for noting the "thorough theoretical analyses". Section, table, figure and appendix numbers refer to the revised PDF.

Assumption 1 (W1, Q1)

"Assumption 1 may be too strong ... unlikely to hold when the extracted features show complicated geometry."

Assumption 1 constrains only the label conditional after projection, so it allows complicated feature geometry. It requires Pe(y∣PSz)=Pinv(y∣PSz)P_e(y\mid P_Sz)=P_{\rm inv}(y\mid P_Sz) in every environment ee. In the new Fig. 3 (App. E.4), the distribution of the stable coordinate changes between two environments and the class-conditional densities are curved, yet the probability of the label given the stable coordinate is the same in both, so Assumption 1 holds. App. E.4 also gives an example where this shared probability is a nonlinear function of the stable coordinate. Assumption 1 still holds there, although a linear probe cannot represent it exactly.

"Please justify how much Assumption 1 reflects real-world datasets."

Assumption 1 builds on prior work on robust prediction from learned features. Last-layer retraining [1] and invariant-feature subspace recovery [2] show that robust predictors can often be read out linearly from learned features, which is the setting GRIT builds on. GRIT also needs the retained subspace to keep the label information, and the pairs to vary the directions it removes. Rigorously testing Assumption 1 directly on pretrained features is a challenging hypothesis test given the number of possible linear subspaces and finite labeled samples, though it is an interesting scientific question that could be explored in future work.

"... demonstrating that a small violation of this assumption would not undermine the performance ... verifying Assumption 1 on some toy data ... with some controllable geometric structure."

In a new controlled study, GRIT handles small bends that violate Assumption 1 but degrades as expected under large ones (App. E.4, Fig. 4). We bend the stable coordinate ss along the nuisance coordinate tt, zα=(s+α(1+t2−1),t)z_\alpha=(s+\alpha(\sqrt{1+t^2}-1),t). Assumption 1 holds at α=0\alpha=0. For α>0\alpha>0 it fails on test distribution B, whose label conditional given the retained coordinate changes. With 20 seeds and rank-one GRIT:

Bend α\alpha00.10.43.2
Test B accuracy (%)84.1182.6677.7025.19

App. E.4 gives the full sweep, intervals and two control test sets.

References for this comment:

[1] Kirichenko et al. DFR, ICLR 2023.

[2] Wang et al. ISR, ICML 2022.

Comparison with existing pair methods (W2, Q2)

"Rather than directly comparing ... with ... CoRe, MatchDG, and IPG, the authors defined another prediction consistency (PC) method ..."

With supplied pairs, PC equals MatchDG's matching penalty and is four times CoRe's. MatchDG's second phase penalizes squared distances between the representations of matched inputs, and its reported architecture takes that representation to be the logit vector [3, Sec. 4, App. C.1]. With our supplied pairs and equal weights, its squared-distance penalty is exactly PC: k−1∑j∥ℓj−ℓj′∥2k^{-1}\sum_j\Vert\ell_j-\ell'_j\Vert^2 for paired logits ℓj,ℓj′\ell_j,\ell'_j. CoRe's logit-variance penalty on two-member groups (exponent one) is one quarter of this quantity [4, Secs. 3.2–3.3]. Our comparison uses these penalties on a linear head over frozen features, with supplied pairs replacing MatchDG's learned matches (App. F.2).

We added LAM [5] and IPG [6] to Table 1, and GRIT has the highest mean in 14 of 16 rows. All columns use the same frozen CLIP features, pairs, seeds and validation-only selection, with LAM's penalty and IPG's update rule applied to the linear head (App. F.3). The two exceptions are on Waterbirds-INV: PC with RDM, and LAM with LISA. A post-test check with wider grids puts GRIT first in 15 of 16 rows (App. F.3, Table 3). The ERM row:

Dataset / metric (%)VanillaPCLAMIPGGRIT
ColoredMNIST-INV / test accuracy17.0257.1616.9929.1867.89
Waterbirds-INV / worst-group52.6966.8451.9842.2570.09

"... CoRe, MatchDG, and SelfReg?"

SelfReg does not use supplied pairs, and every variant we ran lost to GRIT (App. F.7). SelfReg uses random same-class examples as positives and trains a feature map with mixup [7]. Our variants reach at most 40.39% on ColoredMNIST-INV and 50.77% on Waterbirds-INV, against 67.89% and 70.09% for linear ERM + GRIT.

We added experiments with a two-layer MLP head (App. F.6). GRIT preprocesses the frozen features before the MLP and outperforms PC, LAM and IPG on Waterbirds-INV, while IPG leads marginally on ColoredMNIST-INV. Extending the linear-probing theory and adapting GRIT to nonlinear heads are directions for future work.

Smaller margins on Waterbirds-INV (W2, Q2)

"Justify the reason why GRIT marginally improves the performance on the Waterbird-INV dataset. This might be some violation of assumptions ..."

GRIT has higher mean worst-group accuracy than PC in seven of eight Waterbirds-INV objectives, LAM in seven, and IPG in all eight (Table 1). Seed-paired 95% intervals favor GRIT over PC and LAM in six objectives each, and over IPG in seven (App. F.4, Table 5). Against PC, the margin averages 1.7 points, compared with 7.9 on ColoredMNIST-INV.

Smaller margins alone do not necessarily imply a violation of Assumption 1. With unweighted cross-entropy, ERM, PC and GRIT reach 52.69%, 66.84% and 70.09%, while linear ERM trained on all 4,555 training birds on both backgrounds (about 19 times GRIT's counterfactual data) reaches 79.38% (App. F.4). GRIT closes about two thirds of the gap from ERM to this reference, and PC about half. MaskTune reports that some Waterbirds images contain both a landbird and a waterbird, because the background already held a bird before another was pasted onto it, and that some species labels are wrong [8, Sec. 4.1]. Swapping such a background could change label-relevant content and make a pair less faithful.

References for this comment:

[3] Mahajan et al. MatchDG, ICML 2021.

[4] Heinze-Deml & Meinshausen. CoRe, Machine Learning 2021.

[5] Gao et al. LAM, UAI 2024.

[6] Surner et al. IPG, Machine Learning 2026.

[7] Kim et al. SelfReg, ICCV 2021.

[8] Asgari Taghanaki et al. MaskTune, NeurIPS 2022.

Pair efficiency (Q2)

"An additional experiment could demonstrate this sample efficiency, i.e., that fewer pairs are required to reach a target accuracy."

GRIT passes PC's best result with about a fifth of the pairs on ColoredMNIST-INV and about half on Waterbirds-INV (App. F.5, Table 6). With ERM, GRIT reaches 62.01% with 48 of 256 ColoredMNIST-INV pairs and 68.10% with 128 of 240 Waterbirds-INV pairs. PC's best mean at any tested budget is 57.70% and 66.84%, respectively. PC leads at small budgets: 16–32 pairs on ColoredMNIST-INV and 32 on Waterbirds-INV. Table 6 reports every budget with seed-paired 95% intervals over ten seeds.

Cost and scaling (Q2)

"Consider providing some measurement of GRIT's efficiency. Also, how does GRIT scale with the feature's dimensionality or the required rank?"

The added cost is small (App. F.8). GRIT computes one thin SVD of the k×dk\times d matrix of pair differences, costing O(kdmin⁡{k,d})O(kd\min\lbrace k,d\rbrace) for kk pairs in dd dimensions. Projection costs O(dr)O(dr) per feature vector at rank rr. Truncated or randomized SVD, which computes only the top directions, is practical for much larger matrices [9]. On one CPU thread, SVD takes 18–28 ms and projecting all stored features takes 20–343 ms across the two datasets. The current minibatch implementation adds about 1.8 s per Waterbirds-INV training run. App. F.8 gives hardware details and scaling with dimension and rank (Table 10).

Outline of the theory (Q3)

"Please outline the flow of the theoretical analysis at the beginning of Section 4 ... summarize each subsection's key takeaways."

Section 4 now opens with this outline. Section 4.1 splits target risk into source risk, the shift in stable features, and sensitivity to unstable directions, and shows that invariance controls only the last. Section 4.2 gives the condition under which pairs reveal the unstable subspace. Section 4.3 carries finite-pair estimation error and the rank choice into the target risk of the fitted predictor. Section 4.4 analyzes PC in the same setting.

We would be glad to clarify any of these points further.

References for this comment:

[9] Halko, Martinsson & Tropp. Finding Structure with Randomness, SIAM Review 2011.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank the reviewers and the AC. After the discussion, Reviewer ufPT wrote that our response "addressed my concerns" and that the paper "makes a valuable contribution." Reviewer 8Znm raised their score and wrote that they are "more convinced of the main assumption's validity."

The revised PDF shows substantive changes in blue. Our response in brief:

  • Clarified that our contribution is the analysis of finite, noisy pairs (phR1). We now cite NAP, whose projection matches GRIT's under matched conditions (new App. B.3); to the best of our knowledge, no prior work relates pair count, pair noise and rank to OOD target risk, as MatchDG and CoRe assume perfect pairs.
  • Explained the practical guidance from the theory (phR1, 8Znm). The theory shows how pair count, coverage and rank determine robustness under noisy pairs, and experiments demonstrate these effects (Secs. 6.2–6.3).
  • Clarified Assumption 1 and tested anisotropic pair noise (8Znm, ufPT, phR1). Assumption 1 constrains only the label conditional after projection, so feature distributions can still change across environments, and GRIT can handle small violations (new App. E.4, Figs. 3–4). New pair-noise analyses show that GRIT can fail when strong anisotropic noise overlaps a predictive direction or when pairs are systematically biased (new App. H.3).
  • Added baselines (8Znm, ufPT). We show that in our paired setting, the MatchDG and CoRe penalties reduce to PC up to a constant factor (App. F.2); with LAM and IPG included, GRIT has the highest mean in 14 of 16 settings (Table 1), and adding GRIT improves GIC and PDE on both datasets (App. F.7).
  • Extended the hyperparameter grids (phR1). In a post-test check, GRIT has the highest mean in 15 of 16 settings (new Table 3).
  • Added an MLP head, timings and a theory outline (8Znm). The head has two layers (App. F.6), the timings cover SVD and projection (App. F.8), and the outline opens Sec. 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.