acceptodds
Under review as a conference paper at ICLR 2027

The Role of Data-Reuse on Shortcut Learning with Linear Networks

Abstract

Many dynamical theories of shortcut learning use fresh samples or population gradients, whereas most practical training repeatedly uses a finite dataset. We study full-batch gradient descent with square loss in a synthetic linear model of spurious correlations. Our analysis uses a non-rigorous dynamical mean-field calculation from statistical physics. The resulting equations separate shortcut and core learning from fitting directions specific to the training dataset. A perturbative analysis predicts a reuse crossover on a timescale proportional to , where is dataset size and is input dimension. Without regularisation, the interpolation peak at can degrade test performance without impairing core or shortcut acquisition. For , overlap between the features can amplify the shortcut coefficient. Ridge suppresses this peak but can increase the ratio of shortcut to core coefficients. Simulations validate the theory, while image benchmarks provide qualitative evidence beyond the solvable setting.

Primary area:
learning theory
Keywords:
Shortcut Learning, Offline Learning, Dynamical Mean Field Theory, Statistical Physics of Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zb7vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the dynamics of shortcut learning when a finite dataset is reused, in contrast to prior analyses that use fresh samples or population gradients. The setting is a linear predictor trained by full-batch gradient descent with square loss. The data come from a teacher–mixture model with a shortcut direction and a core direction. Using a non-rigorous dynamical mean-field theory (DMFT), the authors track the shortcut and core coefficients and the norm outside their span, which measures sample-specific fitting. The theory predicts when data reuse begins to matter (after about $\sqrt{P/D} steps) and shows that reuse and weight decay can increase shortcut reliance, with simulations matching the theory and image benchmarks giving qualitative support.

Critical strengths
  • Comparing the reuse timescale with the shortcut and core learning timescales is interesting and, as far as I know, new in the shortcut-learning literature. In particular, Result 1 and the regimes in Fig. 3 show that the train–test gap and the shortcut-conflicting risk behave very differently depending on how these timescales are ordered.
  • The message of each result is stated clearly, and the figures illustrate the different theoretical scenarios well.
  • The decomposition into c_sc, c_core and Q_out is a useful tool. It separates "relying on the wrong feature" from "memorizing the training samples", and it shows that the interpolation peak comes entirely from Q_out while the signal coefficients stay finite.
Critical weaknesses
  1. Data model. Labels are generated from the full input (Eq. 4), so the shortcut enters the labeling rule itself. This seems unnatural for shortcut learning: in the standard setting (e.g., Sagawa et al., 2020b), the correlation arises from P(s|y), the labeling rule P(y|core) is invariant, and hence a core-only predictor is correct across environments. How does this difference affect the results, and why is the present model a natural choice for studying shortcut learning?

  2. Interpretation of Result 2. Since the shortcut and core directions are not orthogonal, the coefficient c_sc is not the same as the component of q along v (e.g., the overlap m), which is what the prediction actually depends on through the shortcut sign (Eq. 9). An increase in c_sc therefore does not necessarily mean that the model relies more on the shortcut. Is the coefficient, rather than the actual magnitude along v, the right measure of shortcut reliance here? Reporting m, or the risk on shortcut-conflicting examples, alongside c_sc would clarify whether the title "can amplify the shortcut" is justified.

  3. The experiments do not test the main theoretical results. Fig. 6(a,b) shows that the shortcut coefficient increases when the shortcut agrees with the label more often. This seems trivial, and the paper notes it already happens without data reuse (App. E.3.1). Could you add experiments that test the theory directly? For example, how does the shortcut coefficient change with the weight-decay strength (Result 3)?

  4. The role of the DMFT is unclear. For this shallow linear model, the training trajectory already has an exact closed-form solution (Eq. 155), and Sec. 3.2 says Results 2–4 follow "independently of the formal DMFT derivation". So the DMFT seems to be needed only for Result 1, which is an approximate estimate. Could you clarify what the DMFT analysis adds beyond the exact solution in this setting?

Questions for authors
  • What is the practical takeaway? Applying DMFT to shortcut learning is novel and interesting, but what problem does it newly solve? My concern is that applying this theory may be the main goal of the paper, rather than addressing an existing problem.
  • Result 3: the message that larger weight decay can increase reliance on the shortcut feature is interesting. However, the assumption μ=0\mu = 0 makes this interpretation difficult, because it means the shortcut direction has larger variance than the core direction. Is this a reasonable assumption in practice?
  • Do the predictions hold with pretrained features (e.g., ImageNet-pretrained ResNet embeddings) on Waterbirds, which is the standard setup?
3 comments

We thank the Reviewer for highlighting the comparison between the reuse and feature-learning timescales. In “Invariant labels and shortcut flips”, we tested whether the risk reversal survives when the core alone fixes the label (W1) and whether a larger shortcut coefficient makes predictions more sensitive to a flip (W2). At the Reviewer’s suggestion, we also used “Pretrained Waterbirds features” to test the reversal on ImageNet embeddings (Q3) and the effect of ridge on the fitted coefficients (W3). Both experiments are described in the general reply.

W1. Does the risk reversal survive when changing the shortcut cannot change the label?

Yes. The original model allowed the core and shortcut directions to overlap so we could study coefficient amplification (Result 2). For the rebuttal, we fixed the label from the core, y=sign⁡(z)y=\operatorname{sign}(z), before drawing the shortcut. For 90%90\% versus 50%50\% training agreement, the new calculation predicts higher average test loss after one update but lower loss at convergence for 1<P/D<1.4211<P/D<1.421 (App. D.5.2, Eq. 199 and following). The mechanism is the same as in Result 4: stronger agreement leaves less training residual to fit in noise directions. At P/D=1.25P/D=1.25 and D=512D=512, across 256 paired training datasets, stronger agreement lowered average squared test loss at convergence by 0.2550.255 but raised it by 0.3540.354 on shortcut-conflicting examples (Fig. 10).

W2. “Is the coefficient, rather than the actual magnitude along vv, the right measure of shortcut reliance here?”

For sensitivity to a shortcut flip, ∣m∣|m| is the relevant quantity. Sec. 2.1 distinguishes the overlap mm from the coefficient cscc_{\mathrm{sc}}. The statement of Result 2 is about cscc_{\mathrm{sc}}, but its original title did not make that clear, so we modified it. With overlapping directions, changing the shortcut from −1-1 to +1+1 at fixed Gaussian input changes the score by 2m2m, where m=νcsc+κccorem = \nu c_{\mathrm{sc}} + \kappa c_{\mathrm{core}}. The additional calculation in App. D.4 shows that, for μ=0\mu = 0 and P<DP < D, the fit can increase cscc_{\mathrm{sc}} while reducing ∣m∣|m|. We tested that prediction by measuring both quantities and the loss on conflicting examples on the same training sets. At D=1024D = 1024 and P/D=0.5P/D = 0.5 in a moderate-salience setting, the minimum-norm fit increased cscc_{\mathrm{sc}} by 0.0440.044 relative to the population fit, yet ∣m∣|m| fell by 0.1540.154 and conflicting-group squared loss rose by 0.6040.604. We report the flip sensitivity and conflicting-group loss in App. D.4.

W3. “Could you add experiments that test the theory directly?” In particular, how does weight decay change the shortcut coefficient?

Fig. 6(a,b) of the submission varied training agreement on random features, not weight decay. Following the Reviewer’s suggestion, we added a ridge sweep: we held the pretrained Waterbirds features and the 20 training subsets fixed and varied only ridge. At 95%95\% training agreement near interpolation, we could not resolve a change in the shortcut coefficient when we raised the ridge multiplier from 00 to 0.10.1. At P/D≈1.1P/D \approx 1.1 its mean moved from 0.0720.072 to 0.0660.066 (paired difference −0.006-0.006; 95%95\% interval [−0.015,0.004][-0.015, 0.004]). The interval includes zero. Nor could we resolve an increase in the shortcut-to-core ratio, the quantity in Result 3 (App. E.3.2, Fig. 19C reports its magnitude). What ridge did change was the test loss. Without ridge, the readouts trained at 95%95\% agreement had lower balanced test loss than those trained at 80%80\% because their predictions varied less across training sets. Ridge narrowed that difference in variability enough that the 95%95\% readouts no longer had lower loss (Fig. 19B).

W4. “Could you clarify what the DMFT analysis adds beyond the exact solution in this setting?”

Eq. 155 gives the gradient descent trajectory for one realised training set. DMFT predicts the typical trajectory across draws in the proportional limit, both when the same examples are reused and when each update uses fresh ones. For Result 1, comparing the fixed-dataset and online dynamics identifies an early excess in out-of-plane fitting proportional to t(t−1)/(P/D)t(t-1)/(P/D). Fig. 3 compares its onset with core recovery: the same feature geometry admits both early and late reuse. Results 2–4 have separate least-squares arguments; they do not require the formal DMFT derivation.

Q1. “What is the practical takeaway?”

The practical question is whether the core can be learned before reuse adds appreciably to sample-specific fitting. Fig. 3(b,c) shows this trade-off: with feature geometry held fixed, appreciable reuse precedes core recovery at P/D=2P/D = 2 but follows it at P/D=44P/D = 44. In the latter case, online learning describes core recovery well; in the former, stopping before reuse would also stop core learning too early. There is also a prediction for a change in the test population. In the fixed-label example at P/D=1.25P/D = 1.25, we kept both fitted predictors fixed and varied only the proportion of shortcut-conflicting test examples. The theory predicts that stronger training agreement loses its average-loss advantage at 69.3%69.3\% conflicting examples; the measured crossings were 71.0%71.0\% at D=512D = 512 and 69.0%69.0\% at D=1024D = 1024 (Fig. 10d).

Q2. Is it reasonable to assume higher variance along the shortcut than along the core?

Yes. Giving the shortcut a larger scale than the core is an established modelling choice. For instance: Puli et al. (2023) scale the shortcut directly; Jain et al. (2024) obtain higher variance along the shortcut by separating the group means; and Hermann et al. (2024) vary the relative feature scales independently of their predictivity.

In Result 3 (μ=0\mu=0), ridge shrinks the higher-variance shortcut direction less, so the shortcut-to-core coefficient ratio increases giving a sufficient condition. App. D.3 shows that adding variance along the core μ>0\mu>0 can reverse the effect.

Q3. Do the predictions hold on pretrained Waterbirds features?

Yes. We can show this in the risk reversal tasks. While the submission used random features on Waterbirds (Fig. 6a), in the revision, we trained a linear readout on frozen ImageNet-pretrained ResNet-50 features and tested the same risk ordering as in Result 4 (revised Fig. 16B–D). At P/D≈1.1P/D \approx 1.1, increasing bird–background agreement from 80%80\% to 95%95\% reduced balanced half-squared test loss by 0.3130.313 at convergence, even though worst-group accuracy fell by 7.97.9 percentage points. Stronger agreement made the average readout worse, but it reduced variation in predictions across training sets enough to lower the overall squared loss. At P/D=4P/D = 4 the loss advantage disappeared. These measurements support the risk-reversal mechanism on pretrained features.

Thank you for the detailed responses and the additional experiments. The new calculation with labels fixed by the core and the experiment on the pretrained features, as well as your answer to Q2, addressed my concerns about the data and experimental settings. The reply on the role of DMFT is also clear. The different scenarios in Figure 3 are interesting.

However, some of my concerns remain. For Result 2, the new title is more accurate, but the new analysis shows that a larger shortcut coefficient can come with a smaller response to flipping the shortcut, so it is still unclear what the amplification means for shortcut reliance. Which one should we take as the measure of what hinders generalization? The population risk in Section 2 seems to depend on m rather than c_{sc}, while Results 2 and 3 are stated in terms of the coefficients. Additionally, for Result 3, the ridge sweep on pretrained features didn't show a change in the shortcut-core ratio, so the experiments still don't directly support it. I also still think that the practical takeaway is limited. I therefore keep my score.

Reviewer_jLAwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies how reusing a finite training dataset affects shortcut learning and generalization. The authors consider a synthetic high-dimensional model with core and shortcut features and analyze full-batch gradient descent on a linear predictor trained with square loss. They employ dynamical mean-field theory (DMFT) to derive effective evolution equations that separate the acquisition of core and shortcut features from sample-specific fitting.

The paper reports four main findings: (i) a characteristic timescale at which data-reuse effects become significant, depending on the dataset-to-dimension ratio; (ii) amplification of shortcut coefficients under certain conditions when the dataset is smaller than the input dimension; (iii) suppression of sample-specific fitting by ridge regularization, potentially accompanied by an increased shortcut-to-core coefficient ratio; and (iv) conditions under which stronger shortcut correlations can reduce decorrelated test risk by reducing sample-specific fitting. The theoretical predictions are compared against simulations in the synthetic setting and experiments on Waterbirds, Spawrious, and Colored MNIST.

The problem is relevant to understanding the dynamics of shortcut learning. However, the analysis relies on a formal, explicitly non-rigorous DMFT calculation, and the extent to which the findings provide actionable insights beyond the specific setting considered remains unclear. More importantly, in my opinion, the paper is cryptic and often vaguely written.

I should also note that I am not familiar with DMFT, so it is possible that I have overlooked some key aspects of the analysis.

Critical strengths

AI is used to polish my writings.

  1. Relevant research question. Understanding how repeated use of a finite dataset affects shortcut acquisition, overfitting, and generalization is an interesting problem. The paper attempts to connect these phenomena within a unified dynamical framework.

  2. Separation of different learning effects. Tracking the core coefficient, shortcut coefficient, and the norm outside their span provides a useful decomposition. In particular, it helps distinguish changes in feature reliance from sample-specific fitting, which may affect test risk differently.

  3. Several potentially interesting observations. The results concerning shortcut amplification under feature overlap, the interaction between ridge regularization and relative feature reliance, and the reversal of decorrelated-risk ordering are interesting hypotheses that merit further investigation.

  4. Multiple levels of empirical validation. The authors validate their theoretical predictions through synthetic simulations and examine related phenomena on two random-feature image benchmarks and a CNN trained on Colored MNIST. This provides some evidence that the observations are not restricted entirely to the synthetic Gaussian model, although the strength of this evidence varies across experiments.

Critical weaknesses

AI is used to polish my writings.

  1. Lack of mathematical rigor. The central analysis relies on a formal, non-rigorous DMFT derivation. Although this approach can be valuable for generating predictions, the paper does not establish rigorous guarantees for its principal theoretical claims. The reliance on additional assumptions and numerical validation makes it difficult to assess the generality and reliability of the conclusions. The authors should more clearly distinguish results derived exactly from the model from predictions contingent on the DMFT approximation.

  2. Narrow problem setting and limited practical relevance. The main analysis concerns a highly specific Gaussian model with prescribed core and shortcut directions, a shallow linear predictor, square loss, and full-batch gradient descent. It is unclear how much the findings tell us about contemporary neural-network training. In particular, the motivation for focusing on full-batch gradient descent deserves further discussion, given the prevalence of mini-batch SGD and the fact that stochastic sampling can itself alter the effects of data reuse. While the paper discusses extensions to persistent SGD and other architectures, these do not establish the practical relevance of the main conclusions for standard training pipelines.

  3. Insufficiently compelling theoretical insights. Several findings appear to characterize particular interactions between sample size, feature overlap, regularization, and sample-specific fitting, rather than establish broadly applicable principles. For example, the observation that ridge regularization can increase the shortcut-to-core coefficient ratio is interesting, but its practical implications and the extent to which it changes actual out-of-distribution performance remain insufficiently clear. The paper would benefit from a sharper explanation of what fundamentally new understanding its results provide beyond existing analyses of high-dimensional regression and spurious correlations.

  4. Limited empirical support for broader claims. The image experiments provide qualitative evidence rather than a strong validation of the theoretical framework. In Waterbirds and Spawrious, the predicted direction of change in the core coefficient is less accurate. In Colored MNIST, the reported power-law exponent for the dataset-size dependence is explicitly descriptive, and the authors acknowledge that the observed departure times do not uniquely identify data reuse as the cause. Consequently, these experiments offer limited support for the broader applicability of the proposed theory.

  5. Serious presentation and exposition problems. The paper is difficult to follow, even for a technically sophisticated reader. The abstract and introduction do not communicate the contributions clearly, the problem setup and notation are inadequately introduced, and several claims depend on notation or assumptions that are explained only later. The presentation of the main results is also unnecessarily cryptic. These issues substantially impede the assessment of the technical contributions and would require major revision.

Overall, although the problem is relevant and some of the observations are potentially interesting, I am not convinced that the combination of a non-rigorous theoretical analysis, a narrow modeling framework, and limited empirical validation constitutes a sufficiently strong contribution for ICLR.

Questions for authors
  1. Relevance to standard training procedures: What is the main motivation for focusing on full-batch gradient descent? Which of the principal findings persist under mini-batch SGD, and can the authors provide evidence that the proposed reuse timescale and the reported shortcut-amplification effects remain meaningful in that setting?

  2. Validity of the DMFT predictions: Which assumptions are essential to obtaining the main results, and how sensitive are the predictions to violations of those assumptions? Can the authors provide quantitative error estimates or a clearer characterization of the regimes in which the DMFT approximation is expected to be reliable?

  3. Novelty and practical implications: How do the main findings differ conceptually from existing results on high-dimensional least squares, interpolation, and spurious correlations? In particular, what new insight does the reuse crossover provide that cannot be explained by the relaxation times of the empirical covariance and the associated sample-specific fitting effects?

  4. Evidence beyond the synthetic model: Given the limitations acknowledged in the image experiments, what evidence supports the claim that the proposed framework explains the observed behavior in realistic neural networks? Can the authors distinguish the effects of data reuse from those of changing sample composition, BatchNorm statistics, and learned representations in the Colored MNIST experiments?

4 comments

We thank the Reviewer for the detailed comments and for highlighting the distinction between learning a feature and fitting individual examples. Three new tests address the theory: “Mini-batch SGD” tests whether reuse can precede core recovery under ordinary SGD (Q1), “DMFT accuracy” checks the approximation (Q2), and “Covariance spectrum” asks what the eigenvalues alone can explain (Q3). On images, we varied ridge on “Pretrained Waterbirds features” (W3) and controlled which examples the CNN saw again in “Colored MNIST” (W4, Q4). Experimental details are in the general reply.

W1. “Lack of mathematical rigor.”

Sec. 1.1 discusses rigorous DMFT results for related settings, while progress has been made in the recent years (e.g. Celentano et al., 2021; Gerbelot et al., 2024), a general rigorous DMFT proof is still missing and beyond the scope of this work. Our derivation is for this mixture, and it is formal but not rigorous, for this reason we extensively verify the results with numerical simulations (see errorbars in all figures). To address further the Reviewer’s concern, we added a new Fig. 12 testing the full equations and the short-time approximation against the same simulations. The full equations track the reuse correction in all four regimes. When the crossover occurs late, it is the short-time estimate that predicts it too early (Q2). Results 2–4 have independent least-squares arguments. They characterise the converged fit and, for Result 4, compare it with the first update; they do not determine whether reuse precedes core recovery.

W2. “Narrow problem setting and limited practical relevance.”

We respectfully disagree: our setup covers both the isotropic, shared-teacher shortcut model studied by Jain et al. (2024) and mixtures with extra variance along the core direction. Jain et al. explain when a shortcut is learned before the core if each example is new. That leaves open what happens when the learner sees the same examples again: has it learned the core by the time fitting individual examples affects its predictions? Full-batch training lets us isolate repetition. To address the Reviewer’s comment, we tested ordinary mini-batch SGD and found that reuse can precede or follow core recovery (Q1). In the new matched CNN experiment, repeating the same coloured digits also increased sensitivity to colour late in training (Q4). We also asked whether the risk reversal requires the shortcut to affect the label. In the additional calculation, changing the shortcut leaves the label fixed, as in Sagawa et al. (2020). We derived a range of dataset sizes where stronger training agreement lowers average squared test loss but raises it on conflicting examples, and verified both predictions in simulations (App. D.5.2).

W3. What does the ridge result imply for out-of-distribution performance?

A larger shortcut-to-core ratio alone does not determine whether shifted loss rises: ridge can also reduce sample-specific fitting (see Result 3 of the submission). To expand on this point, we varied ridge on pretrained Waterbirds features (new Fig. 19). Near interpolation, raising training agreement from 8080\\% to 9595\\% lowered balanced half-squared test loss by 0.3130.313 without ridge but raised it by 0.0540.054 at a ridge multiplier of 0.10.1. Ridge removed most of the difference in prediction variance across training sets; the loss difference between the two average predictors barely changed. We did not resolve an increase in the shortcut-to-core ratio on these features.

W4. “Limited empirical support for broader claims.”

To address the Reviewer’s comment we run additional to go beyond the qualitative trends reported in Fig. 6. We test why stronger training agreement can lower test loss on Waterbirds and how much delaying reuse changes CNN scores on Colored MNIST. On pretrained Waterbirds features, stronger agreement made the average predictor worse, yet lowered balanced test loss near interpolation because predictions varied less across training sets, as Result 4 predicts (Fig. 16). 

We agree with the Reviewer that dataset-size sweep leads other confounding effect and does not isolate reuse alone. Therefore we compared CNNs trained with the same class and colour-agreement counts: one saw the same coloured digits again, while the other saw freshly drawn digits. Reuse increased sensitivity to recolouring in all six settings (Q4). We also tested whether the theory predicts the size of the early reuse effect. At fixed dataset size, delaying the first repeat by reshuffling should leave 4040\\% of the excess squared change in the red–blue score contrast seen with independent batch draws; we measured 3737\\% (9595\\% interval: 28–4628\text{–}46\\%).

W5. “The abstract and introduction do not communicate the contributions clearly”

We made three changes:

  1. The revised abstract says what the square-root timescale measures: when reuse adds appreciably to sample-specific fitting.

  2. The introduction defines that crossover at first mention, rather than waiting until Result 1.

  3. We also changed the title of Result 2 to name the quantity that increases: the shortcut coefficient.

If these changes miss the concern, could the Reviewer point us to the passage in the abstract or introduction and the result it leaves unclear? We would be happy to revise it. 

Q1. Why use full-batch gradient descent, and which results persist under mini-batch SGD?

We used full-batch descent to isolate reuse from the additional mini-batch sampling noise. The original submission also tested persistent SGD (App. E.2.1); for the rebuttal we analysed ordinary SGD with independently drawn batches. Two batches of size MM from PP examples share M2/PM^2/P examples on average. Averaging each gradient over its MM examples cancels the M2M^2, so the leading small-step reuse correction still scales as 1/P1/P. After four updates at step size 0.00050.0005, the measured correction was 9797\\% of the predicted value for batches of 8 and 9999\\% for batches of 32. In the new simulations with batches of 32, excess out-of-plane fitting reached our 22\\%-of-online-risk threshold after 10 updates at P/D=2P/D = 2 and 70 at P/D=32P/D = 32; online core recovery stayed near update 39. The ordering can thus switch under SGD, though the crossing depends on the threshold (37 updates at 11\\% for P/D=32P/D = 32). For P<DP < D and full row rank, stable ridgeless SGD from zero converges to the same minimum-norm interpolant as full-batch descent. Result 2 therefore also applies to the final shortcut coefficient under SGD.

Q2. Which assumptions matter for the DMFT predictions, and how accurate are they?

The full equations assume Gaussian-mixture inputs with PP examples and DD dimensions large at fixed P/DP/D. App. E.2.1 of the submission reported the accuracy across three linear network depths: the root-mean-square error against simulated population risk was 0.0020–0.00230.0020\text{–}0.0023 for full-batch training over the interval tested. The square-root estimate further requires residuals to stay correlated across updates, before earlier fluctuations have decayed appreciably. In the new Fig. 12 the linearised approximation retains residual correlations and damping. It predicts the 22\\% crossing time to within about 55\\% of full DMFT in three regimes and about 2020\\% early in the fourth. The simpler square-root estimate crosses too early when reuse occurs late. These are measured errors, not bounds.

Q3. What does the reuse crossover tell us beyond the relaxation times of the empirical covariance?

For a fixed training set, the eigenvalues give the relaxation rates, and the eigenvectors show how each mode aligns with the core and shortcut. Given the training inputs and labels, Eq. 155 in the submission gives the exact trajectory. In a new test, we kept the inputs fixed and changed only the teacher. After four updates at P/D=4P/D=4, the two teachers differed in conflicting-group squared loss by 1.2941.294 (spectral prediction: 1.2891.289). Switching from fresh to reused examples changed that difference by only −0.019-0.019: almost all of the loss difference was present without reuse.

The reuse crossover instead measures when training on the same examples produces appreciably more fitting outside the core-shortcut plane than training on fresh examples, which is different from what relaxation times can give us. DMFT predicts this excess from the data distribution and lets us compare its onset with core recovery. In Fig. 3b,c, the feature geometry is the same, yet stopping before reuse cuts core learning short at P/D=2P/D=2 and leaves enough time to learn the core at P/D=44P/D=44.

Q4. Can the Colored MNIST experiment separate reuse from changes in sample composition, BatchNorm and learned features?

For the rebuttal, we ran a new experiment that separates reuse from changes in class and colour-agreement counts and rules out BatchNorm as an explanation. In the comparison added to App. E.3.2, both CNNs start from the same weights, take the same first update and use GroupNorm instead of BatchNorm. One then sees the same coloured digits again; the other sees freshly drawn digits and colours, with the same class and colour-agreement counts at every update. Across all six settings, the CNN trained on repeated examples had higher final balanced cross-entropy and was more sensitive to recolouring held-out digits. This is direct evidence that reuse can increase shortcut reliance. The original dataset-size sweep in Fig. 6(c,d) did not isolate reuse, so its fitted exponent remains descriptive. This shows that reuse can increase colour sensitivity. Whether that increase comes from a change in the features learned by the CNN remains open since our theory does not describe how those features evolve during training.

I would like to thank the authors for their reply.

Several of my concerns have been addressed through the authors' additional experiments. Nevertheless, the paper remains poorly written and requires more comprehensive rewriting. While I understand the authors' justification, the analysis is still non-rigorous, which remains a weakness. The concern regarding mini-batch SGD has been addressed only experimentally. Most importantly, I still find the main novelty and significance of the work unclear.

I therefore maintain my original score but lower my confidence, giving more weight to reviewers with greater expertise in this area.

Reviewer_6umPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how reusing a finite training dataset affects shortcut and core feature learning. The authors consider a simple linear regression model and use dynamical mean-field theory (DMFT) to analyze its training dynamics. They estimate when data reuse becomes important, derive conditions under which finite-dataset training can amplify the shortcut coefficient relative to population regression, show that ridge regularization can increase the shortcut-to-core coefficient ratio, and demonstrate that stronger shortcut correlations can sometimes lead to lower risk in a decorrelated test environment. The theoretical results are supported by numerical simulations, with additional experiments on Waterbirds, Spawrious, and Colored MNIST to investigate whether the qualitative conclusions extend beyond the theoretical setting.

Critical strengths

None

Critical weaknesses

1. Contribution and novelty concerns.

My main concern is the novelty and scientific significance of the results. Although the authors derive several potentially original analytical statements, the underlying phenomena have already been studied in the literature, and I find it difficult to identify what substantial new understanding of shortcut learning the paper provides.

In particular:

Result 1. Finite-dataset corrections to online learning have already been analyzed by Bordelon et al. (2024), A Dynamical Model of Neural Scaling Laws, including closely related early-time scaling with training time and dataset size. Although the authors cite this work and acknowledge previous studies of finite-dataset effects, they do not directly compare their reuse-crossover estimate with these earlier results. Consequently, it remains unclear what substantial new insight Result 1 provides beyond applying a related mechanism to their specific model.

Result 2. Sagawa et al. (2020), An Investigation of Why Overparameterization Exacerbates Spurious Correlations, already investigate how overparameterized models can favor spurious features over core features, including through a theoretical analysis of memorization and minimum-norm solutions. Although the authors acknowledge this work, they do not clearly explain what additional scientific insight follows from their precise condition for shortcut coefficient amplification relative to population regression. The mathematical statement is distinct, but its significance beyond the previously studied phenomenon remains unclear.

Result 3. Bombari and Mondelli (2025), Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization, already establish theoretically that ridge regularization can increase learned spurious correlations. The authors acknowledge this work and describe their result as complementary, since it establishes monotonicity of a different quantity, the shortcut-to-core coefficient ratio. However, it remains unclear what substantial new insight this additional characterization provides beyond the previously studied phenomenon.

Result 4. A related counterintuitive phenomenon, where stronger shortcut correlations during training can improve generalization on shortcut-conflicting examples, has been observed by Chou and Hoang (2026), When Data Imbalance Helps: Robust Generalization Through Shortcut Saturation (arXiv preprint). Although this work considers a different setting and mechanism, it suggests that the broader phenomenon is not entirely new. Result 4 provides a distinct analytical explanation based on reduced sample-specific fitting, but I am not convinced that this characterization offers a sufficiently substantial new insight beyond the specific linear-regression setting.

I do not claim that these previous works establish the same mathematical statements. However, I believe the authors should explain much more clearly what genuinely new knowledge follows from their results, beyond exact characterizations of already known phenomena in a specific model.

Another concern is the limited generality of the theory. Despite the title, the main results concern essentially standard linear regression, while the extension to deep linear networks is placed in the appendix. Although the authors include experiments beyond this setting, they do not establish clear conditions under which their theoretical predictions should remain valid more generally. In particular, it remains unclear which assumptions of the model are essential and which can be relaxed.

Overall, I find the scientific contribution insufficiently significant relative to existing literature.

2. Presentation quality and clarity of the theoretical results.

Although the paper is generally well organized, I found it very difficult to follow despite careful reading. The exposition is highly technical and relies on extensive model-specific notation and terminology, while the conceptual meaning and significance of the results are often insufficiently explained.

For example, the Introduction introduces quantities such as residual correlations, reuse corrections, and the reuse crossover before their meaning is properly explained. Similarly, Section 3.1 introduces several characteristic timescales without clearly explaining which dynamical quantities they describe. The discussion of Fig. 3 largely describes the observed trajectories, without sufficiently establishing what general conclusions follow from the proposed classification. Fig. 1 is also somewhat misleading. While it correctly illustrates the special case μ=0\mu=0, Eq. (1) generally expresses the input xx as a sum of three components: shortcut, orthogonal Gaussian noise, and a core-aligned Gaussian component. For μ=0\mu=0, the last two can be combined into isotropic Gaussian noise, as shown in the figure. However, this simplification is no longer valid for μ>0\mu>0, despite the caption stating that introducing μ>0\mu>0 does not change the graph. I believe this distinction should be clarified or the general construction illustrated explicitly.

I also find it difficult to assess the mathematical status of the numbered results from the main text. The authors acknowledge that their DMFT derivation is formal and non-rigorous, but provide little high-level explanation of the method itself, its main assumptions, and the approximations involved. For readers who are not familiar with DMFT, this makes it particularly difficult to understand how the theoretical conclusions are obtained and how much mathematical justification they have. In particular, Result 1 appears to rely on a more restrictive approximation than Results 2–4, but this distinction is not sufficiently clear from the main text. Although the technical derivations are provided in the appendix, I believe the main text would benefit from a brief explanation of the core idea behind DMFT, the key assumptions and limitations of the analysis, and a clearer distinction between formal predictions and results supported by separate analytical arguments.

More generally, the paper focuses heavily on technical results specific to its theoretical model, without sufficiently explaining their broader implications. Even after understanding individual statements, it remains difficult to extract meaningful, generalizable insights about shortcut learning. In my opinion, this substantially reduces the accessibility and scientific value of the work.

Questions for authors
  1. Could the authors clarify what genuinely new scientific understanding follows from each of Results 1–4, beyond providing more precise mathematical characterizations in their particular setting? In particular, how do these contributions differ conceptually from the related phenomena studied by Bordelon et al. (2024), Sagawa et al. (2020), Bombari and Mondelli (2025), and Chou and Hoang (2026), respectively? I would be especially interested in understanding the significance of these results, rather than only the differences between their mathematical formulations.
  2. Which of the main theoretical predictions do the authors expect to remain valid beyond shallow linear regression, and under what conditions? In particular, which assumptions of the model are essential for the proposed mechanisms, and which can be relaxed? Although the appendix considers deep linear networks and Section 4 includes experiments on image benchmarks, it remains unclear which specific predictions these extensions support and where the authors expect their theory to cease to apply. Could the authors clarify the boundaries of applicability of their results?
3 comments

We thank the Reviewer for the careful reading and the four comparisons with earlier work, which we address individually in Q1. We ran two new tests in response to the review: “DMFT accuracy” checks when the early-time estimate departs from the full equations (W2), and “Nonlinear Waterbirds readout” tests whether the predicted early effect of reuse survives when both layers are trained (Q2). The general reply describes both tests.

W1. What do we learn about shortcut learning from data reuse that earlier work did not already explain?

  1. The training budget matters because learning the core and fitting individual examples take different amounts of time. In Fig. 3b,c, keeping the feature geometry unchanged and changing the dataset size, at two examples per input dimension, fitting individual examples becomes appreciable before core recovery; while at 44, it starts to matter only afterwards. Stopping before reuse therefore leaves time to learn the core in one case but not the other.

  2. The strongest deviation from population regression occurs near interpolation. The peak in test loss comes from fitting directions outside the core–shortcut plane, while the two feature coefficients stay finite. Therefore a large peak is not evidence that the fit has put a large weight on the shortcut.

  3. This characterisation also explains why stronger training correlations can lower test loss when the shortcut and label are uncorrelated at test time. The shortcut fits more of the training labels, leaving less residual to fit in sample-specific directions. Result 4 gives the range of dataset sizes where this reduction lowers test loss. If stronger training agreement raises the population predictor’s test loss by pp but reduces its training residual variance by dd, the converged ridgeless fit has test-loss difference p−d/(P/D−1)p - d/(P/D - 1) in the proportional limit with P/D>1P/D > 1. For p,d>0p, d > 0, stronger agreement hurts after the first update but helps at convergence when 1<P/D<1+d/p1 < P/D < 1 + d/p. Increasing dataset size past that boundary removes the advantage. 

W2. What do the timescales in Fig. 3 measure, and which results rely on a DMFT approximation rather than a separate analytical argument?

In Sec. 3.1, the core and shortcut times describe feature learning with fresh examples; the slowest timescale is given by the bulk time associated to the relaxation of the lower nonzero bulk edge of the training covariance spectrum; finally, the reuse crossover is associated to when the extra sample-specific loss first exceeds a chosen fraction of the online test loss.

DMFT tracks how earlier updates fitted each repeated example. The calculation is formal and assumes Gaussian-mixture inputs with PP examples and DD dimensions large at fixed P/DP/D. If residuals at different updates remain strongly correlated and earlier gradients have not yet decayed appreciably, the reuse correction grows as t(t−1)/(P/D)t(t-1)/(P/D). A fixed relative threshold is then reached at a time proportional to P/D\sqrt{P/D}. In the new Fig. 12, the full equations track the simulations even when the short-time estimate predicts reuse too early. Results 2–4 instead follow from separate least-squares arguments; Result 4 also compares the first update with convergence.

Finally, we redrew Fig. 1 to show the two Gaussian components in Eq. 1 as suggested by the Reviewer. Both have variance Δ\Delta when μ=0\mu = 0. For μ>0\mu > 0 only the variance along the core increases, to Δ+μ\Delta + \mu; the orthogonal variance stays at Δ\Delta.

Q1. How do Results 1–4 differ from the findings of Bordelon et al., Sagawa et al., Bombari and Mondelli, and Chou and Hoang, respectively?

  • Bordelon et al. (2024, Eq. 18) showed how repeated use of PP examples opens a train–test gap, with an early correction of order 1/P1/P. We find the same inverse dependence on dataset size for a different quantity: the excess fitting outside the core and shortcut directions. A train–test gap alone does not say which feature has been learned. Comparing this excess with core recovery is what gives the two different stopping outcomes in Fig. 3b,c.

  • Sagawa et al. (2020) explain why an overparameterised classifier can favour a shortcut: it can memorise the relatively few examples that contradict it. Result 2 instead explains whether fitting a finite dataset gives the shortcut a larger coefficient than population regression on the same distribution. When there are fewer examples than dimensions (P<DP < D), the minimum-norm fit retains different fractions of the signal along different directions. We derive the condition for amplification: it is impossible when the two features are orthogonal. When they overlap, a larger shortcut coefficient can even accompany a smaller response to flipping the shortcut, as our new test shows (App. D.4).

  • Bombari and Mondelli (2025) show that their measure of spurious correlation can increase as ridge approaches the strength that minimises in-distribution loss. Result 3 explains how ridge can suppress sample-specific fitting yet tilt the remaining signal towards the shortcut. Ridge shrinks low-variance directions more strongly. When the shortcut has the higher variance and the core has no added variance (μ=0\mu = 0), the shortcut-to-core coefficient ratio increases even if the absolute shortcut coefficient falls. Less overfitting therefore does not imply greater relative weight on the core.

  • Chou and Hoang (2026) find that stronger training agreement makes a two-layer Transformer more likely to learn a parity rule that classifies shortcut-conflicting examples correctly. Result 4 explains how average test loss can fall even when prediction on conflicting examples gets worse: near interpolation, stronger training agreement leaves less residual to fit in sample-specific directions. In particular, our new test on pretrained Waterbirds features separates the two: stronger bird–background agreement lowered balanced squared loss near interpolation, yet worst-group accuracy fell by 7.97.9 percentage points. The loss gain disappeared when ridge suppressed variation across training sets (Figs. 16 and 19). Their preprint appeared in July, when we were already finalising our paper; we did not include it before submission but cite and discuss it in the related work now.

Q2. “Which of the main theoretical predictions do the authors expect to remain valid beyond shallow linear regression, and under what conditions?”

The submitted paper already tests the dynamics beyond a shallow predictor. In App. E.2.1 the deep-linear DMFT tracks the core and shortcut coefficients and the population risk for networks with one and three hidden layers, under both full-batch training and persistent SGD. Sec. 4 of the submission tested qualitative predictions on images. In the new Colored MNIST experiment, paired CNNs shared the first update and saw the same class and colour-agreement counts thereafter. Reusing the same coloured digits, rather than redrawing them, made the final predictions more sensitive to recolouring in all six settings (App. E.3.2). Thus reuse can increase shortcut reliance even when the features are learned; this comparison does not estimate a CNN scaling exponent.

For the rebuttal we asked which predictions survive when the learner is nonlinear. For a smooth predictor taking a few small steps, the initial gradients predict how much repeating examples changes held-out scores. With both layers of a tanh⁡\tanh readout trained on pretrained Waterbirds features, the measured excess squared change in logits was 0.94–1.060.94\text{–}1.06 times the prediction after four updates. At a 100-fold larger step size the calculation overestimated the effect. This is a quantitative prediction for early training, not an extrapolation of the linear solution to convergence.

The assumptions at convergence depend on the result. The coefficient amplification in Result 2 also holds for stable, ridgeless mini-batch SGD from zero when P<DP < D and the training inputs have full row rank: SGD selects the same minimum-norm interpolant. For Result 3, μ=0\mu = 0 guarantees an increasing shortcut-to-core ratio under the stated signal assumptions; with added variance along the core, ridge can favour the core instead. For Result 4, our new calculation fixes the binary label from the core before drawing the shortcut. The reversal persists because stronger agreement reduces the residual that a linear score must fit in independent Gaussian noise directions. If the core score already fits the labels exactly, there is no such saving (App. D.5.2).

I appreciate the authors' clarifications and additional theoretical and empirical investigations. They address several of my specific questions and strengthen the paper.

However, my main concerns remain. Although the authors have clarified how their results differ from previous work, I am still not convinced that they provide sufficiently substantial and generalizable new scientific insights. The theory remains largely restricted to linear regression, while the additional experiments provide only limited evidence for the broader applicability of the main theoretical results.

I also remain concerned about presentation. Despite some useful clarifications, the main theoretical exposition remains largely unchanged and difficult to follow. In particular, the conceptual significance of the results and their broader implications are still insufficiently explained.

Therefore, I maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and new experiments

Thank you for the feedback.

Common questions

  1. What does reuse explain about shortcut learning beyond known finite-dataset effects?
  2. What does DMFT add to exact least squares, and which results depend on its approximations?
  3. Which predictions survive changes in the label rule, optimiser or architecture?

Common responses 

  1. The submission identifies when stopping before sample-specific fitting also means stopping before core recovery. Fig. 3b,c shows that, at fixed feature geometry, sample-specific fitting becomes appreciable before core recovery at P/D=2P/D = 2 and after it at P/D=44P/D = 44. Separating core and shortcut learning from sample-specific fitting also explains Result 4: stronger shortcut–label agreement leaves less residual to fit outside the signal plane. Near interpolation, this can lower squared loss on a test set with no shortcut–label correlation even as the shortcut coefficient grows.
  2. Eq. 155 gives the trajectory on one labelled dataset. DMFT predicts the typical core, shortcut and sample-specific trajectories from the data distribution. Result 1 uses an early-time approximation to the formal DMFT equations for the square-root timescale; Results 2–4 have separate least-squares arguments. DMFT thus tells us whether the core is learned before reuse matters.
  3. In a new calculation (App. D.5.2, Fig. 10), we identify a range of dataset sizes where stronger training agreement lowers average test loss but raises it on conflicting examples, even when the core alone sets the label. For square loss on a fixed dataset, independent uniform mini-batches give exactly the full-batch core and shortcut coefficients on average. The risk can still differ: shared examples correlate the gradient fluctuations across updates and give the leading reuse correction. For a smooth nonlinear predictor taking small steps, the same overlap calculation predicts the excess squared change in held-out scores.

New experiments we ran

  • DMFT accuracy. For Reviewers 6umP (W2) and jLAw (Q2), we tested full DMFT and its short-time approximation against simulations in the four regimes of Fig. 3. Full DMFT tracked excess sample-specific fitting in all four. When reuse came late, the approximation crossed the threshold too early (Fig. 12).
  • Covariance spectrum. For Reviewer jLAw (Q3), we changed the teacher but kept the inputs fixed. Squared loss on shortcut-conflicting examples changed despite identical covariance eigenvalues. Most of that difference persisted with fresh batches, so the loss contrast alone does not identify reuse.
  • Invariant labels and shortcut flips. For Reviewer zb7v (W1), we tested Result 4 with labels fixed by the core. At P/D=1.25P/D=1.25, stronger training agreement lowered average squared loss at convergence but raised conflicting-group loss (App. D.5.2, Fig. 10). For W2, we tested Result 2 with overlapping features: compared with the population fit, the shortcut coefficient rose, but flipping the shortcut changed the score less (App. D.4).
  • Mini-batch SGD. For Reviewer jLAw (Q1), we extended the submitted persistent-SGD check to ordinary mini-batch SGD. With batches of 32, core recovery took about 39 updates with fresh examples. The extra test loss from fitting sample-specific directions reached 2%2\% of online test loss after 10 updates at P/D=2P/D=2 and after 70 at P/D=32P/D=32. Thus the reuse crossover preceded core recovery on the smaller dataset and followed it on the larger one, as in Fig. 3.
  • Pretrained Waterbirds features. We extended the submitted random-feature experiment to pretrained features (Reviewer zb7v, Q3; Fig. 16) and varied ridge on the same training sets (Reviewers zb7v and jLAw, W3; Fig. 19). Near interpolation, stronger training agreement lowered balanced squared loss but cost 7.97.9 percentage points of worst-group accuracy. As in Result 4, predictions varied less across training sets at stronger agreement. Once ridge narrowed that difference, the loss advantage disappeared.
  • Nonlinear Waterbirds readout. For Reviewer 6umP (Q2), we trained both layers of a tanh⁡\tanh readout to test the prediction from the initial gradients. After four small updates, the excess squared change in held-out logits was 0.94–1.060.94\text{–}1.06 times the prediction across four initialisations and two batch sizes. At a 100-fold larger step size, the prediction was too high.
  • Colored MNIST. For Reviewer jLAw (Q4; App. E.3.2), we compared reuse with redrawing digits, matching class and colour-agreement counts and replacing BatchNorm with GroupNorm. Reuse increased final colour sensitivity and balanced cross-entropy in all six CNN settings. For W4, we changed the sampling order. With reshuffling, fewer pairs of updates can share a digit by the first repeat. Counting those pairs predicts 40%40\% of the excess squared change in red–blue logit contrast seen with independent batch draws; we measured 37%37\% (95%95\% interval: 28–46%28\text{–}46\%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.