acceptodds
Under review as a conference paper at ICLR 2027

How Rare Counterexamples Change What Models Learn from the Majority

Abstract

Models often rely on shortcuts that predict most training examples correctly but fail on rare counterexamples. We study why the same counterexamples can correct shortcut reliance at different rates depending on the remaining training data. In a two-feature model trained with logistic loss, we vary how robust and shortcut feature values are paired within majority examples while holding the counterexamples, feature marginals, and covariance fixed. We identify an indirect mechanism: counterexample updates change majority-example margins, reactivating gradients that both promote robust-feature learning and reinforce the shortcut. The balance between these effects evolves as different majority examples come to dominate learning. Our analysis links these shifting gradient contributions to feature growth and the time to first correct classification, showing that matched marginals and covariance do not determine correction times and that approximations based on the initially hardest examples can miss the subsequent dynamics. A controlled intervention that removes only majority-example updates to the robust feature reduces the ratio of mean correction times between two pairing conditions from 2.17 to 1.02, supporting the role of this indirect pathway. Experiments with small tanh networks on handwritten digits with synthetic backgrounds further show that feature pairing affects worst-group accuracy even when parameter-update norms are matched. Together, these results show how counterexamples reshape subsequent learning from majority examples, revealing a mechanism for shortcut correction that feature marginals and covariance alone do not capture.

Primary area:
learning theory
Keywords:
shortcut learning, learning theory, learning dynamics, spurious correlations, feature learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nsgqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how rare counterexamples change subsequent learning from majority samples through their effects on feature margins and gradients. It derives correction time and counterexample count bounds in a model with two fixed features, tests the mechanism through controlled interventions and small neural networks, and introduces Shortcut to Knowledge (SK) to predict learning progress using fixed representations.

Critical strengths
  1. Theorem 4.1 and Appendix A establish that matching feature marginals and covariance does not determine correction times. The analysis also considers endpoint error and the persistence of correction, which strengthens the contribution beyond the initial crossing criterion.
  2. The intervention in Figure 4(a) provides direct evidence that majority robust feature updates contribute to the correction time gap in the proposed model. The appendices also provide detailed numerical checks and additional controls.
Critical weaknesses
  1. The theoretical model has limited generality, and the neural network validation is not yet sufficiently convincing. The main theory assumes predefined robust and shortcut features, a perfectly label-aligned robust signal, and balanced diagonal factorization. The correction time analysis depends on the resulting multiplicative weight dynamics, with no corresponding guarantee for jointly learned representations. Moreover, the additional ordinary CNN control reports a mean matched-norm, time-averaged worst-group accuracy gap of only 0.413 percentage points across four settings, including one negative result (Appendix B.3). Although these experiments differ in setup and evaluation from Figure 4(c), they raise concerns about the consistency and generality of the effect in neural networks. Given existing analyses of gradient competition and spurious feature learning [1, 2], the new feature pairing mechanism would benefit from stronger validation beyond this restricted setting.
  2. The SK evaluation provides limited evidence for its broader practical significance. SK predicts a local quadratic approximation to the training of two readout coefficients with fixed representations. Prediction and validation appropriately use the same initialization, context candidates, training budget, and scoring rule (Appendix C.1). The reported dataset-level Spearman correlation of 0.991 therefore supports this specific forecasting task, but does not directly validate the proposed majority learning mechanism or robust representation learning. Appendix C.3 also reports positive SK improvement for GOOD-HIV despite zero endpoint all-context accuracy, highlighting that improved loss does not necessarily translate into robust classification. Finally, no counterexample selection procedure is evaluated against existing methods. The paper identifies an interesting mechanism, but its broader practical consequences remain insufficiently demonstrated.

References

[1] Gradient Starvation: A Learning Proclivity in Neural Networks

[2] Complexity Matters: Feature Learning in the Presence of Spurious Correlations

Questions for authors
  1. What evidence establishes that the proposed indirect majority learning mechanism improves learned representations, beyond increasing gradients on existing features? Appendix B.2 acknowledges that pretraining and update norm matching prevent the channel interventions from isolating cross-step feedback alone. Could additional controls distinguish representation improvement from changes in readout weights or update magnitudes?
  2. How do you explain the small and inconsistent ordinary CNN results in Appendix B.3 compared with the larger effects in Figure 4(c), accounting for differences in experimental settings and evaluation metrics? Would the conclusions remain consistent across additional initializations in networks that learn robust and shortcut features jointly?
  3. Does SK outperform a simpler first-order predictor, and does its predictive accuracy extend beyond the specified two-coefficient readout? Results connecting SK to endpoint robustness would be especially important.
1 comment

Thank you for reading the appendices so carefully and for recognizing the intervention in Fig. 4(a). Your observations about the CNN results and GOOD-HIV brought out limits we should have discussed in the main text. We ran the following controls to address your questions.

Q1. Does feedback from ordinary examples improve the learned representation?

A1. We agree that a stronger gradient alone leaves this question open. Our new control found no consistent gain in how well a fresh classifier could read digit identity from the hidden features. In a shared tanh network trained from scratch on 3s and 8s, an ordinary-image update increased held-out digit margins by 0.00293 on average, or 0.00765 after a counterexample step (step size 0.05). To test the features separately, we froze the output layer and took a hidden-layer counterexample step. From that same state, two copies took equal-length updates on ordinary images, using the gradient computed before or after the counterexample step.

We fit fresh linear probes with balanced synthetic backgrounds and five-fold cross-validation on 77 held-out digit identities. Across six checkpoints for each of ten seeds, mean squared probe error was 0.239138 with the original gradient and 0.239115 with the recomputed gradient. Only 5/10 seeds favored recomputation on average. In a 200-cycle control, the two probes had identical worst-group accuracy within every seed (95.10% mean in both arms). We will narrow our claim to how counterexamples change later updates and explain how this relates to the work you cited on gradient competition and feature complexity. Pezeshki et al. and Qiu et al..

Q2. Why is the ordinary CNN gap smaller than in Fig. 4(c), and does it persist across initializations?

A2. We should have discussed the small average CNN gap and the negative result alongside Fig. 4(c). That figure reports the digit-network gap at step 500 (3.52–19.07 percentage points across three seeds). Appendix B.3 reports a 6,000-step average on Fashion-MNIST (+0.413 points across four settings, one negative). We measured both on the same digit trajectories, matching update norms. The table gives mean worst-group accuracy gaps (Q minus P, percentage points) over eight new seeds per architecture.

Shared networkStep 500Average over steps 0–6000Step 6000
tanh MLP+11.37−0.02−2.04
CNN+4.54+0.07−1.50

For all eight seeds in each architecture, Q was ahead at step 500 and behind at step 6000. This shows how an early advantage can coexist with a near-zero time average. The original experiments also changed data and architecture, so their individual contributions remain open. We will report both time scales together.

Q3. Does SK improve on a first-order forecast? Does it extend to wider readouts or endpoint accuracy?

A3. The first-order baseline you suggest was missing from our evaluation. We tested c(t)=c(0)−t∇L(c(0))c(t)=c(0)-t\nabla L(c(0)) on the original 142 tasks and 426 folds with identical starting logits, contexts, budgets and scoring. At B=8B=8, mean absolute error against direct training across the 20 dataset scores was 3.603 points for first order and 0.784 for SK (Spearman 0.962 and 0.991). We repeated the comparison with dense affine heads (10–105 parameters) on fixed logits from Waterbirds, PACS, UCI HAR and GOOD-HIV. The errors were 5.54 and 2.62. The largest quadratic error was 6.07 points on GOOD-HIV.

Your GOOD-HIV example illustrates why both outcomes matter. The fitted loss-progress score was +5.300, yet no query was classified correctly under every candidate context. Across 20 datasets, the SK score had Spearman correlation 0.649 with endpoint all-context accuracy. Computing accuracy directly from the predicted final logits gave a mean absolute error of 0.59 percentage points. We will report loss and accuracy together and state explicitly that these forecasts concern a classifier on fixed features.

Your question also prompted a selection test. On digits, we selected 8 of 72 counterexamples by how much they improved the next ordinary-image step on a separate calibration set. Over ten paired runs, held-out all-context accuracy was 15.7%, versus 23.0% for uniform sampling, 11.0% for highest training loss, and 19.1% for immediate calibration improvement. Given this result, we will present counterexample selection as an open question.

Reviewer_QCj2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how the robust and shortcut features in majority samples affects shortcut correction. Using a two-feature factorized logistic model, the authors identify an indirect mechanism through which counterexample updates alter the gradient contributions of majority samples, thereby influencing robust feature learning. The theoretical analysis characterizes the learning dynamics and derives bounds on correction times and counterexample requirements under different feature pairings. The authors further validate the proposed mechanism through controlled gradient interventions and experiments with small neural networks on synthetic-background image datasets.

Critical strengths
  1. The paper provides a quantitative theoretical analysis of shortcut correction dynamics.
  2. The paper includes controlled experiments that support the proposed mechanism.
Critical weaknesses
  1. The theoretical analysis relies on a highly specialized data distribution, a two-dimensional model, and specific initialization schemes. It is unclear whether such a simplified setting can capture the essential dynamics of high-dimensional machine learning models trained from random initialization.
  2. The experiments are largely restricted to settings where robust and shortcut features are explicitly separated, leaving it unclear whether the proposed dynamics generalize to end-to-end representation learning, where these features are learned jointly and may be entangled.
Questions for authors
  1. To what extent do the theoretical results depend on the factorized parameterization and the specific initialization scheme? In particular, would the observed correction-time differences persist under standard random initialization or direct parameterization?
  2. Is there evidence that the proposed majority-mediated gradient mechanism also holds in end-to-end neural network training, where robust and shortcut features are learned jointly rather than explicitly separated?
1 comment

Thank you for the care you took to distinguish what our controlled experiments show from what remains open in a network that learns its own features. We should have made that distinction clearer in the submission. Your questions prompted two additional experiments.

Q1. Do correction-time differences persist with random initialization or direct parameterization?

A1. In the example from Fig. 3(a), the difference persists under both random initialization schemes for directly trained weights. We agree that this was an open question in the submission. Theorem 4.1 assumes equal positive factors at initialization and a very weak initial weight on the reliable cue. Under these assumptions, each weight's effective learning rate is proportional to the weight itself (Eq. 6). Direct training removes that scaling, so we tested the correction times explicitly.

P and Q have the same rare examples (1/16 of the data), feature marginals and covariance. Only the pairing of cue strengths in ordinary examples changes. For each row, P and Q shared the same 128 random starts. We report mean gradient-flow time to first correction for starts that misclassified the rare example. All reached correction. The remaining starts had time zero under both pairings.

Parameters and initializationInitially wrongMean time PMean time QQ slower
Direct weights, standard normal67/1282647.44203.267/67
Direct weights, uniform on [−1/2,1/2][-1/\sqrt{2},1/\sqrt{2}]69/1282649.44204.269/69
Independent standard-normal factors59/128209.0222.746/59

For directly trained weights, Q took about 59% longer on average under both initialization schemes. In the direct Gaussian case, Q was also slower in all 55 pairs where both models already classified every ordinary example correctly but missed the rare one. With independent random factors, the mean gap shrank to 6.5%, and P was slower in 13 of 59 pairs. These reversals illustrate why your question about initialization matters. In the revision we will report them alongside the averages and clarify that all three comparisons use the same controlled two-feature distribution.

Q2. Does the majority-mediated mechanism occur when digit and background features are learned jointly?

A2. Your concern about Fig. 4(b) is well founded. A separate background branch allowed us to isolate one update, but it left open whether the same effect would arise when all features are learned together. To test this, we trained a single 64–32–1 tanh network from random initialization to distinguish handwritten 3s from 8s using only the mixed pixels. A checkerboard was added to the digit pixels. Its sign agreed with the label on 266 training images and was reversed on 14. All layers learned jointly.

At six checkpoints for each of 10 random seeds, we made two copies of the network. One took an update on the 14 counterexamples, while the other stayed at the checkpoint. Both then took one step on the same 266 ordinary images. We measured the gain from that last step alone in each copy. Our score was the mean signed margin on 77 held-out digits, testing each digit on both backgrounds and weighting the two classes equally.

Step sizeMean gain from ordinary images aloneMean gain after a counterexample stepLarger gain after counterexamples
0.050.002930.0076560/60 checkpoints
0.0250.001540.0023059/60 checkpoints

To understand why the ordinary images had more influence, we repeated their update on the copy that had seen the counterexamples. This time we held each image's logistic loss weight at its value before the counterexample update. At step size 0.05, this reduced the mean gain from 0.00765 to 0.00469. The gain dropped at 60/60 checkpoints at 0.05 and 49/60 at 0.025. This control shows that the change in how the loss weights individual ordinary images accounts for part of the observed gain, even with a shared network.

This gives us direct evidence that a counterexample update can change what ordinary images contribute on the next step in a network that learns both cues jointly. We share your caution about extending this finding beyond one small network with synthetic backgrounds. In the revision we will add these controls and state explicitly that predicting long-run accuracy or correction times in larger networks remains open.

Reviewer_uzf1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper examines the impact of rare counterexamples on the training dynamics of a 2-feature logistic classifier. It proves a number of theoretical results relating to this scenario. For example, they demonstrate that feature distribution characteristics beyond the first two moments impact the time it takes during training for counterexamples to correct the model's predictions. The authors then verify the theory on small scale experiments, and empirically observe similar behaviour on small neural networks. Finally they introduce Shortcut-to-Knowledge, which is a way to estimate the improvement of a simple classifier built on top of pretrained features.

Critical strengths
  1. The theoretical contributions seem to be correct, and involve a decent amount of technical work.
Critical weaknesses
  1. The theoretical and empirical results have narrow applicability. It appears that the assumptions in the problem setup are so restrictive that I struggle to see how they might be relevant to practical deep learning. The focus is on a very specific set of scenarios that constrain the input feature distribution and force the logistic classifier weights ww to be a product of two equal parameters uvuv where u=vu=v. Moreover, they assume for some of the results that one of the weights is much smaller than the other and then drive it asymptotically to zero at initialisation. The following experiments either demonstrate the theory works sticking with the narrow limiting assumptions, or show qualitative behaviour on neural network models (that break most of the assumptions) that roughly matches the qualitative outcomes suggested by the theory without establishing the theory concretely generalises to NNs. Finally, the practical value of SK is unclear to me -- it predicts the outcome of an inexpensive optimisation that could just be done directly, moreover, I struggle to see the usefulness of said optimisation in the first place. Furthermore, SK doesn't really follow from the rest of the paper and feels like a random separate contribution.
  2. The presentation is difficult to follow. This paper was extremely difficult to read. Figures are weirdly aesthetic upon first inspection, but barely helped me understand the conceptual content. The language is dense, and quickly loses the reader in jargon and notation, leaving me scratching my head. Some simple, intuitive language to guide understanding and motivate the central problem would massively improve the readability of the paper -- it is remarkably difficult to extract the high-level idea that the paper is interested in the scenario, during learning with spurious features, where rare examples that demonstrate feature spuriousness can correct model behaviour to in turn disregard said spurious features.
Questions for authors

Note that weakness 2 may have caused me to misinterpret or misunderstand the paper. If I have done so, I encourage the authors to clarify any errors I've made.

1 comment

Thank you for the careful reading and for explaining where the argument became difficult to follow. Your comments helped us clarify what the theory predicts and what the experiments test.

Q1. Does the result require factorized weights and a very small initial weight on the reliable cue?

A1. The formula in Theorem 4.1 uses both assumptions. Each weight is a product of two factors that start equal, and we derive the formula as the reliable cue's initial weight tends to zero. To test whether pairing still matters without these restrictions, we repeated Fig. 3(a) under four settings.

The two training sets contain the same rare examples. Among the ordinary examples, each cue has the same distribution in both sets, and the covariance is also the same. What changes is which strengths of the two cues occur together in the same example. The table gives the time until the reliable cue first offsets the misleading shortcut on the rare example.

Training ruleInitial weights (reliable, shortcut)Time on PTime on Q
Two factors per weight(e−8,2)(e^{-8}, 2)324.3298.2
Two factors per weight(2,2)(2, 2)150.6184.7
Train weights directly(e−8,2)(e^{-8}, 2)2663.04212.4
Train weights directly(2,2)(2, 2)2563.04149.4

The first row reproduces Table A.2. The other three are new numerical results. With both restrictions removed (last row), Q takes 62% longer than P. In the original setting, Q was faster. Thus pairing affects correction time in this controlled example even under ordinary logistic regression, while which pairing is faster depends on the training setup.

Q2. What did the neural-network experiments test directly?

A2. We agree that similar accuracy curves alone would leave the mechanism unclear. In Fig. 4(b), we copied a Fashion-MNIST network at each checkpoint and used the counterexamples to update only its background branch. We then measured the update that the same ordinary clothing images would produce. After the background change, those images pushed more strongly to increase the weight on the clothing image itself. This held at all 60 checkpoints along each of 12 trajectories across two architectures (App. B.2).

This intervention tests the immediate learning signal. In ordinary CNN controls, the accuracy differences between pairings were small and changed sign across settings (App. B.3). We agree that predicting correction times in general deep networks remains open.

Q3. What does SK add when direct fitting is inexpensive?

A3. Direct fitting is indeed cheap. The question SK shares with the theory is how much progress a model can make within a limited budget when its usual context is misleading. For a fixed representation and a two-weight classifier, SK uses the initial slope and curvature of the loss to estimate the average relative loss reduction over that budget under the least favorable candidate context.

At B=8B=8, SK and direct training gave closely matching rankings across 20 datasets (Spearman ρ=0.991\rho=0.991, Fig. 4(e)). This validates the estimate for this classifier. Its value for choosing training data remains untested. We will shorten the main-text discussion and move the detailed comparison to Appendix C.

Q4. How will the revision make the main idea easier to follow?

A4. We appreciate your explaining what you took away from the paper. We should have asked more plainly what the model learns when it returns to familiar images after seeing a rare example. We will introduce this example before the equations.

Suppose most handwritten 3s appear on blue backgrounds and most 8s on orange. A 3 on orange can teach the model to trust color less. Now show it a familiar blue 3 with faint handwriting. Color used to make this example easy. If the model is now less sure, the same image has more influence on the next update. Its pen strokes can reinforce shape, while its blue background still rewards the old color rule. Which cue gains more depends on how handwriting and background strengths are paired in the ordinary images.

Fig. 2(c) will follow that same blue 3 before and after the update. We will also explain that first correction occurs when the digit first supplies enough evidence to offset the misleading background on the rare example. The model can still use color at that point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.