acceptodds
Under review as a conference paper at ICLR 2027

How Models Memorize vs. Generalize: Intervening via Features and Labels

Abstract

Although deep learning is widely adopted for its capability to fit training data effectively, it often memorizes outliers and/or mislabeled instances, a phenomenon known as label memorization. Despite this, there has been no clear distinction between memorization and generalization at the feature level, and how they interact with label memorization. Hence, in this paper, first, we precisely distinguish feature memorization from feature generalization and define the conditions under which they occur. Second, we investigate the interactions among feature memorization, feature generalization, and label memorization, revealing that label memorization suppresses feature memorization while causing feature generalization. These findings offer a systematic understanding of memorization and generalization in deep neural networks.

Primary area:
interpretability and explainable AI
Keywords:
Generalization, Memorization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3qArOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the difference between feature memorization and feature generalization, trying to establish a distinction between memorization at the feature level, generalization of a feature and the relationship with memorization of labels. In order to do that they perform an ample set of experiments along different modalities and architectures to show the claimed effect on different situations. The main structure of the experiments consists of planting a marker on images or text and check if the network ends up memorizing marked examples or generalize the marker, i.e. use the marker as a cue for classification. They ask two questions about this process: Is the network sensitive to the marker? Is the network generalizing the marker?

They do these experiments under three regimes, one with the marker keeping the label of the image/text where it is planted, two, with the marker assigned to a random label different from the original one and three, with the marker assigned to a fixed label not related to the original one for the image.

The authors make three claims: Concrete distinction between feature memorization and feature generalization, When more proportion of examples has the planted feature, the model generalizes and when is low the model memorizes. Label memorization suppresses feature memorization and causes feature generalization.

Critical strengths

The work seems to be clearly structured and the set of experiments is broad. Using a planted feature is a clever way to control the conditions to study memorization. Measuring each example with and without the marker add credibility to the set up. Experiments run across several domains and architectures with different types of tracked features. This makes it possible to check which effects are setting-specific and which ones hold along settings. The paper also makes a great effort in providing complementary metrics to support the observations, like Grad-CAM, and Sensitivity readings.

The use of 5 seeds gives a sense of run-to-run variability, and they provide the raw numbers in the appendix.

Critical weaknesses

Overall, this work aims to present a new designation of shortcut learning, spurious correlations, random label fitting and backdoor attacks.

When the number of images with Z_u are low but label agrees with the original class, the feature Z_u acts as spurious correlation. When more training images carry Z_u the network can use it as a shortcut for those images (Geirhos et al., 2020), creating a cheaper representation, simplicity bias (Shah et al., 2020) and gradient starvation (Pezeshki et al., 2021). The closest prior work is Yang et al. (2022), where also a feature or cue is planted in training examples. When the images with Z_u have a random label different from the original class assigned to them section 5.2 is then random-label fitting (Zhang et al., 2017/2021; Arpit et al., 2017): the feature provides valid information about the label, (Not C_i) and the network uses it to avoid classifying these examples as the original class. S_Sense reflects that going negative. When the Z_u is paired with a single fixed label different from the original class, the information provided by Z_u is informative, gradient flows to this feature, since it is the only consistent marker with the new class in those examples. This is the dirty-label backdoor setting (Gu et al., 2017; Chen et al., 2017; see Li et al., 2022 for a survey), a planted cue with a different label where the network relay on this added Z_u feature to output the assigned label, since that is the only thing in common for that set of images and label.

I would ask the authors to state which observations go beyond these works. As it stands, the FM, FG and LM vocabulary relabels known effects rather than adding a new mechanism.

The FG definition seems too permissive: under condition (ii), which uses ‘or’, a model that performs worse under the tracked feature condition would have a high Avg-TestAcc-Diff without the need to generalize to the tracked feature. Zone C starts in the apparent sharp increase in slope of MP in figure 1, however it is interesting to see that the Test Accuracy of Z_u examples are not affected and many more examples needs to be added to really notice a decrease. What happened in this interval between MP increases but Test Accuracy does not change? Is this FM, FG or in between? Would changing the definition of FG to “and” create a fourth zone, Zone B’.?

In Figure 1, Grad-CAM shows that the child and woman images are already misclassified as class 3 in zone A, with the same hotspot active over all regimes. They appear to be baseline errors and say nothing about Z_u. In zone B (0.45%), there is a faint activation of the marker “ID” for two of the three examples presented, the figure seems to not support lines 298 and 299 (one of them is not the same as in zone A and zone C. How were the examples selected? Would be interesting to see a heatmap of the Grad-CAM activation over all set of images.

Plots are hard to read with up to four vertical axes in the same plot. In the plots (e.g Figure 4) the P-values are shown with confidence interval, since P-value averaging is not a valid way to combine them, how was the aggregation computed? With hundreds of images, even a tiny S_sens reaches p < 0.05, and there is no minimum effect size.

Another concern is the broad use of the word memorization; it seems to have different definitions depending on what is applied to. Is feature memorization only for Z_u or does it apply to any feature available? Is label memorization about one feature or many features or the whole image? The label has to be associated to some feature in the image to be able to correctly classify it.

In the experiments of section 5.1 with Z_u and random labels if the training acc is 100% the images with Z_u are “memorized”. Those examples have to be remember based on some features on the images, even if the feature is not known or different for every image. If they are using a feature to be memorized then FM is not suppressed, just not measured because the feature metrics only track Z_u. In section 5.2 with a fixed label, the claim that LM removes FM and pushes FG, can also be stated as that a marker paired with a consistent label is learned as a rule, which is the backdoor setting.

Minor points:

The main text does not specify what set of classes from the UTK face dataset is used. Appendix state it is ethnicity. This should be stated in the main text. I would suggest to reduce the tone of affirmations like “These findings clearly verify that…” Regenerate table 6, last three rows read inconsistent values for Avg-TestAcc-Diff . Since the Avg-Test Accuracy in those rows reads 0, it should be Avg-TestAcc-Diff of 69.17.

Errata:

Line 336, 1452: “fixed” is repeated? figure 12, 14: No FM or FL, shouldn’t it be No FM or FG? What is FL?

Overall assessment:

The experiments are extensive and carefully run, however in their current form the three claims either look like a restatement of known phenomena (shortcut learning, random-label fitting and backdoor learning) or the metrics do not fully support the claims. Clarifying the different definitions of memorization used implicitly in the text, would help to clarify if some of the claims are a rewording of existing phenomena or genuinely new observations. Especially for claim 3, it is not clear whether FM is completely suppressed or present but not observed. Improving the readability of the plots and how the information is presented would help make the results easier to follow.

Questions for authors

Do the zone boundaries (the proportions where zones B and C begin) depend on the marker’s complexity or salience? Hermann, K. L., Mobahi, H., Fel, T., & Mozer, M. C. (2024) studied this for shortcut learning.

How to address the memorization of features when the network uses other features different from z_u?

How was the P-value aggregated over the runs? And how does having hundreds to thousands of examples affect the calculated P-value? with n in the hundreds to thousands, p < 0.05 is reached by very small effects.

3 comments
AuthorsOpenReview ↗

Response (I/III)

We thank the reviewer for putting in time to review our paper. We address their questions below in detail. Please note that the new edits are marked in blue in the updated PDF.


Weakness 1: Thank you for raising this point. We clarify that our work is not merely relabeling of prior work and articulate our distinction and novelty.

  1. Our work shows the emergence of memorization and generalization through the interactive dynamics of features and labels for the entire spectrum of feature proportion. Shortcut learning/simplicity bias/spurious correlations/(dirty-label) backdoor attacks [1,2,3,4,5,6] are specific applications or examples that arise from establishing undesirable correlations between a certain feature and a behavior/outcome. Through our work, we can learn that those phenomena fall into the FG regime, as the feature is (undesirably) generalized. However, these works do not study the conditions when/where such phenomena can/cannot be established and the interplays with supervision through the labels, which is precisely the novelty of our study.
  2. [7,8] only studied memorization of random labels but never articulated how it interacts with feature memorization or generalization, which makes the contribution of our paper distinctive from theirs.
  3. We would like to clarify that the dirty-label backdoor setting is one application of feature generalization, and such work does not investigate the interplays between label memorization and feature memorization as we do. In our work, we identify, for the first time, that label memorization suppresses feature memorization and causes feature generalization.

In summary, there was no prior work that investigated the grand entire picture via a mechanistic and systematic framework, and we are the first to make this contribution.


Weakness 2: Thank you for your careful observation. No, this is not a new zone, and it still lies in the FG regime. To validate this, we analyze these intermediate points in Appendix H for the UTK-Face, ResNet18 setup where MP2,3MP_{2,3} sharply increases but test accuracy minimally drops (see Fig 8). As seen in Table 9 in Appendix H, the misclassification % of class 2 xux_u test samples as class 3 does rise up (3.02% 7.51% 9.20%) as it shifts from the FM zone (0.45%) to the intermediate FG zone (0.82%, 1.14%) . However, the total misclassification % of class 2 xux_u test samples does not increase significantly in the intermediate FG zones because the misclassification % of class 2 xux_u test samples as other classes other than class 3 also decrease (19.94% 14.91% 15.55%). This causes the test accuracy to remain minimally impacted in the intermediate FG zones. Now coming to the question of why these intermediate points are still categorized under the FG zone, we report the mean gradcam activations of class 2 xux_u test samples misclassified as class 3 in Fig 44. We can clearly see that even in these intermediate points, the model is relying upon zuz_u to misclassify the images as class 3 during test time which is also reflected by the increase in MP2,3MP_{2,3} as seen in Fig. 8. Therefore, to classify the feature to be memorized, the “AND” criteria is required. We hope this resolves the reviewer’s concern.


Weakness 3: We respectfully state that there might be a misunderstanding of what the GradCam plots convey, and we clarify it as follows. GradCam aims to provide support to our FM-FG regimes, where we want to check which features the model dominantly latches on to misclassify xux_u test samples belonging to class CjC_j as CiC_i. If it is largely using zuz_u as the prediction anchor, then it is in Zone C (FG regime); otherwise in either Zone A (neither FM nor FG) or Zone B (FM regime). We put Zone A Gradcam plots as a baseline comparison so that we can assess that Zone B should also not largely use zuz_u as the prediction anchor like in Zone A. To further show that this does not reflect a random subset of our choice, we provide mean GradCam activation analysis across all sets of xux_u test samples belonging to class CjC_j that are misclassified as CiC_i in Appendix E.1.1. It is clearly observed that the mean gradcam activations are alike where the model is largely utilizing other features (not zuz_u) to misclassify CjC_j as CiC_i in Zone A (neither FM nor FG) and Zone B (FM). However, in Zone C (FG), the model relies on zuz_u to misclassify them. We thank the reviewer for suggesting this analysis and we hope it resolves their concern.

AuthorsOpenReview ↗

Response (II/III)

Weakness 4: Thanks a lot for raising this question. We provide further clarification as follows:

  1. We agree the plots are complex but our goal was to simultaneously make everything legible to analyze how different metrics behave as the proportion of zuz_u increases. However, if the reviewer has any better suggestion, we are happy to update it.
  2. As discussed in the paper, we compute SsensS_{sens} and p-value for each seed. To show the mean trend we average both metrics. The p-value curve in the figure is the across-seed mean and standard deviation, showing a descriptive summary of how significance evolves as the proportion of zuz_u increases. However, we would like to clarify that Recognition is decided per seed (p < 0.05 and SsensS_{sens} > 0). To make this explicit, we have added a graph in Appendix I, reporting the percentage of seeds (out of 5) satisfying the recognition criterion at each proportion.
  3. We agree that with hundreds to thousands of samples, even a small SsensS_{sens} can reach p < 0.05, and that our original criterion (SsensS_{sens} > 0 and p < 0.05) did not specify a minimum effect size. Hence, to alleviate this issue, we make the following changes:
  • First, our sweeps across increasing xux_u proportions already show how SsensS_{sens} and the p-value behave when no (0%) xux_u training samples are present. At 0% of xux_u training samples (the leftmost point in Figure 4), the models are trained without zuz_u, under the same settings as every other proportion. Across all model-dataset settings, SsensS_{sens} at 0% stays close to zero and p-value ≥ 0.05 whereas in the recognition regions (zones B and C) SsensS_{sens} is clearly higher and p-value < 0.05. So, at our sample sizes, the baseline of 0% of zuz_u in training does not reach significance, and the significance we observe with zuz_u proportions in Zones B and C cannot not be explained by larger sample size, because if that was the case then Zone A could have also had p-value < 0.05.
  • Second, to handle the minimum effect size issue, we now define recognition as Ssens>TS_{sens} > T and p-value < 0.05, where T is the one-sided 95% upper prediction bound of SsensS_{sens} when the proportion of xux_u train samples is 0%. Recognition here is a binary decision about whether zuz_u was recognized or not. The threshold imposes an explicit minimum effect on the recognition criteria. We have made this update in Section 3.3.1 and added the upper prediction bound results in Appendix I, showing that SsensS_{sens} in the FM and FG zones remains above T (minimum effect size threshold).

Weakness 5: Thank you for your question. To measure when and how a feature is memorized, we propose feature memorization (analogous to how humans process information as presented in Section 3.1), in which the model memorizes a feature without generalizing it to the test set, by tracking zuz_u. On the other hand, label memorization [9] is about the (image, label) pair (see Eq 2), and not at the ‘feature level’. If you have further questions regarding these, we will be happy to answer those.


Weakness 6: We believe there is a misunderstanding in how the reviewer is perceiving our claim. We clarify it as follows:

  1. The aim of Section 5 is to understand how the FM-FG regime of Section 4 interacts with LM. In Section 5.1, we insert zuz_u together with random labels and observe that the FM-FG zones disappear under the definitions discussed in Section 3.4. We then hypothesize in Section 5.1.2 that this is because the model memorizes the noisy labels while latching onto zuz_u, and not onto unrelated features of the image and later verify the hypothesis in Section 5.2 by fixing the noisy label. To add further support, we now also add the mean Grad-CAM plots for the (xux_u, yLy_L) samples in Appendix J (see Fig 46), where we can see that while undergoing random label memorization (yLy_L), the model is fixating onto zuz_u. This further indicates that the memorization is largely organized around zuz_u, and not around some other features under LM. Regarding why FM-FG regions disappear when random labels are added, since the labels are random, the model does not learn a fixed correlation and hence the FM-FG zones are not observed under the definitions discussed in Section 3.4.

  2. For the second part of the question, we would like to clarify that the backdoor setting is one application of feature generalization, and such work does not investigate the interplays between label memorization and feature memorization as we do. In our work, we identify, for the first time, that label memorization suppresses feature memorization and causes feature generalization by showing the full spectrum of memorization/generalization through features and labels.


Minor Points: Thanks to your comment we have added the set of classes considered in the UTK-Face dataset in the main text in line 229. We have also made the edits in the table - thanks for pointing them out.

AuthorsOpenReview ↗

Response (III/III)

Errata: Thank you for your careful reading. Yes, “fixed” is accidentally repeated - we have removed it. For the second question, you are right, we actually meant “FG” and not FL (a typo). We have updated it now.


Question 1: This is a valid question. To answer to it, we carried out additional analysis where to vary the saliency of the feature, we vary its relative size w.r.t original image size. We do this for CIFAR10, ResNet9 with zuz_u as the English letter ``A’’. Relative size of zuz_u is varied in the range of [(1:10), (1:8), (1:6), (1:5), (1:4)] for both height and width, where 1:10 relative ratio corresponds to the smallest feature size (least salient) and 1:4 corresponds to the largest feature size (most salient). We report the results in Appendix G. From the results, we observe that as the feature size increases from 1:10 relative ratio (least salient) to 1:4 relative ratio (most salient), the FM regime (Zone B) reduces and FG starts happening at earlier proportions. On the other hand, as the feature size decreases from 1:4 relative ratio (most salient) to 1:10 relative ratio (least salient), the FM region expands. We thank the reviewer for suggesting this experiment which has helped us to further strengthen our paper.


Question 2: We are sorry, but could you please try to rephrase what you asked in this question for us? All the authors tried to understand this correctly, but we could not. We would appreciate it if you could help us understand it.


Question 3: Please kindly refer to the response to Weakness 4.


[1] Last layer re-training is sufficient for robustness to spurious correlations, ICLR 2023

[2] The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learning, TMLR 2026

[3] Freeze then train: Towards provable representation learning under spurious correlations and feature noise, AISTATS 2023

[4] Shortcut Learning in Deep Neural Networks, NMI 2020

[5] The pitfalls of simplicity bias in neural networks, NeurIPS 2020

[6] Simplicity bias in 1-hidden layer neural networks, NeurIPS 2023

[7] Understanding deep learning requires rethinking generalization, ICLR 2017

[8] A Closer Look at Memorization in Deep Networks, ICML 2017

[9] Feldman, Vitaly. “Does Learning Require Memorization? A Short Tale about a Long Tail”, STOC 2020

We hope our response resolves your concerns, and helps you consider raising your score. We would be happy to answer any more questions that you may have. Thank you.

Reviewer_H3YBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is asking a simple but important question about feature memorization and feature generalization. The paper setups up metrics that use the counterfactual setup, i.e. expected difference in score when a feature is included vs excluded from the input (unlike label memorization this is not multiple trained models with and without). Additionally the paper also uses s single sided t-test. Together these metrics, S_sens and p-value on t-test, along with model performance (misclassification, accuracy and their delta), is used to measure if a feature is memorized or not. The paper then setups up experiments in vision and text modes. The paper introduces a synthetic feature zuz_u which is an overlay for images and tokens such as "egg" or "ship" in text. Using the metrics the paper measures if this synthetic features is memorized. First the paper setup up experiments to test what proportion of the samples having the feature causes memorization and concludes zone (B) $\approx$1% is when feature memorization is highest. Next experiments on its effect over epochs is presented and finally its interaction with label memorization is presented.

Critical strengths
  • The research question about feature memorization is surprisingly a gap in current literature, and the paper makes a meaningful contribution to our understanding. However it does miss some more recent literature [1,2], but still this is distinct from sample-level literature [2, 3]

  • The setup of S_sens and p_value in a counterfactual setup makes a lot of sense and is shown across both vision and language. And some of the results are corroborated in sample-level literature. For example, under clean label setting the association between zuz_u and CiC_i forms early. Under a fixed noisy label, later in training, this is similar to results in [2]

[1] Bayat, Reza, et al. "The pitfalls of memorization: When memorization hurts generalization. ICLR 2025 (spurious correlation)
[2] Ravikumar, Deepak, et al. "Memorization Through the Lens of Sample Gradients." ICLR 2026 (dynamics of memorization over epochs)
[3] Feldman, Vitaly, and Chiyuan Zhang. "What neural networks memorize and why: Discovering the long tail via influence estimation." (seminal sample level paper)

Critical weaknesses
  • The paper does not clarify how zuz_u is constructed, does the overlay on images always stay in the same position? The GradCam maps show a fixed bottom-right position, but the the paper never states position, font, size, or where tokens are inserted in text.
  • There is a confounder here the overlay can act as a backdoor since they perfectly correlate with class and anti-backdoor learning [1] found poisoned samples are learned early. Would be good to clarify this.
  • The other concern is that the label memorization here is different from the counterfactual label memorization from Feldman and Zhang [2] and would be good to clarify that as well

Minor

  • There are several inconsistencies between text, tables and figures
    • The Table 1 (CIFAR-10) discussion cites Fig. 1c instead of Fig. 10c.
    • Table 6's Avg-TestAcc-Diff column
  • Update arXiv references to their published versions:
    • Frankle et al.: ICLR 2020.
    • Garg et al.: ICML 2024.
    • Ravikumar et al.: ICML 2024.
    • Maini et al.: ICML 2023.
    • Toneva et al.: ICLR 2019.
    • Valle-Pérez et al.: ICLR 2019.
    • Yiding Jiang et al.: ICLR 2019.
    • Ziheng Jiang et al.: ICML 2021.
    • Kirichenko et al.: ICLR 2023.
    • MobileViT: ICLR 2022.
    • VGG: ICLR 2015.
    • Schwarzschild et al.: NeurIPS 2024.
    • Ye et al.: TMLR, retitled "The Clever Hans Mirage".
    • BadNets: IEEE Access 2019.
  • MobileViT and Schwarzschild et al. are missing years.

[1] Li, Yige, et al. "Anti-backdoor learning: Training clean models on poisoned data." NeurIPS 2021.
[2] Feldman, Vitaly, and Chiyuan Zhang. "What neural networks memorize and why: Discovering the long tail via influence estimation." (seminal sample level paper)

Questions for authors
  • In the fixed-noisy-label setup, is the LM-score computed by removing each noisy sample individually (leave-one-out) or all noisy samples together?
  • How is zuz_u constructed?
  • For S_sens, I assume its calculated on the training set? Could the authors clarify how exactly this was measured, train/val/test set?
3 comments

We thank the reviewer for reviewing our paper and providing thoughtful comments. Below we address the question in detail. Please note that the new edits are marked in blue in the updated PDF.


Weakness 1: Thank you for the comment. To address your concern, we have articulated the details in Appendix C.1.


Weakness 2: We thank the reviewer for raising this point. We would like to clarify that zuz_u may act analogously to a backdoor trigger [1], but only in the Feature Generalization (FG) regime, as the correlation generalizes (although undesirable) to the test set, resulting in misclassification. As opposed to FG, in the Feature Memorization (FM) regime, zuz_u is recognized, but it does not generalize to the test set. Hence, it does not behave as a backdoor trigger in the FM regime. This is precisely the point of our study: our aim is to examine the full spectrum of how a feature is perceived, from FM to FG, by tracking zuz_u. We articulate the distinction between FM and FG and the conditions under which each occurs. Although our work is a fundamental investigation of feature memorization and generalization, backdoor is one interesting (malicious) application of feature generalization (FG). Hence, we will be happy to cite [1] and discuss it further. Thank you for your comment.


Weakness 3: Thank you for your comment. We believe there is some misunderstanding here. We do indeed follow the counterfactual label memorization definition from Feldman & Zhang [2] as written in Eq.7.


Minor Points: Thank you for your comment. We have reflected your comment in the updated manuscript. Thank you.


Question 1: We have done this for each noisy sample individually and then averaged the LM-score obtained for each of them as mentioned in line 196.


Question 2: Thanks to your comment, we have articulated the details in Appendix C.1.


Question 3: Thank you for raising this question. At first, please let us restate that the role of SsensS_{sens} is only to determine whether the model has recognized zuz_u during training or not before even commenting whether it is memorized or generalized. To compute SsensS_{sens} we use a held-out calibration set, which consists of samples from class CiC_i, the class in which zuz_u was inserted during training. For each sample we keep both the actual version (xcx_c) and the version with zuz_u inserted (xux_u), and compute SsensS_{sens} from these pairs according to Eq. (1). We use the same calibration set across all zuz_u train set proportions, so the recognition check is comparable across them. For example, a model trained with zuz_u in 1% of the data yields Ssens=S1S_{sens}=S_1, and a model trained with 10% yields Ssens=S2S_{sens}=S_2, both measured on the identical set of (xux_u, xcx_c) pairs. This ensures that the differences between S1S_1 and S2S_2 reflect the effect of the zuz_u proportion in training, rather than differences in the evaluation data. Thanks to your comment, we have added further details in Appendix C.1.


[1] Li, Yige, et al. "Anti-backdoor learning: Training clean models on poisoned data." NeurIPS 2021.

[2] Feldman, Vitaly, and Chiyuan Zhang. "What neural networks memorize and why: Discovering the long tail via influence estimation." NeurIPS 2020

We sincerely thank the reviewer for their valuable comments and suggestions, and hope that our responses resolve their concerns and help the reviewer consider increasing the rating. We would be happy to answer any further questions that they may have.

Thank you for the responses they answer my questions

We are glad to know that our responses resolved all your questions. We thank you for the time and effort you have put into reviewing our paper.

Reviewer_UjugOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper studies the memorization and generalization from a features point of view. they provide an interesting new notion of feature memorization/generalization rather than the conventional label memorization people generally talk about. they study these quantities over a variety of datasets and models and provide interesting analysis of training dynamics and their prevalence across different settings, including a study of their relationship with label memorization.

Critical strengths
  • the distinction between memorizing and generalizing a feature, as opposed to a label, is a useful framing and the definitions are simple to measure.

  • the setup is controlled and the same qualitative three-zone picture appears across vision and language, with gradcam and shap analysis supporting the transitions.

  • the training-dynamics plots (figs. 2, 7) add a temporal view of when each regime appears, which is more informative than the final checkpoint alone.

Critical weaknesses
  • the fm/fg boundary rests on two fixed thresholds (σ1 = 3.5%, σ2 = 3%) chosen by hand. the authors acknowledge this is subjective (sec. 3.4), but it means “memorization” here is operationally “recognized but below a cutoff,” and the zone boundaries move with the cutoff. some zone b entries in appendix f sit right at the threshold (e.g. table 5, nico++).

  • all models are at least seven years old and the datasets are small classification benchmarks. it is unclear whether the three-zone picture or the label-memorization interaction holds for modern pretrained models or generative settings.

  • eq. 1 appears to have a notation error: the expectation is over x ∈ C_i but x_u and x_c are the two variants of each sample.

Questions for authors
  • in sec. 5.2, z_u is injected together with a fixed wrong label for every injected sample. this makes z_u a perfectly predictive spurious feature for that label, so the model picking it up looks like standard spurious correlation learning. why attribute it to label memorization rather than to the correlation itself?

  • how sensitive are the zone boundaries to σ1 and σ2? a plot of boundary location versus threshold would help.

  • do the results hold when z_u is a natural feature already present in the data rather than an inserted token?

1 comment

We thank the reviewer for their time and efforts invested to review our paper. Below we address their questions. Please note that the new edits are marked in blue in the updated PDF.


Weakness 1: To reflect the reviewer’s comment, we provide a sensitivity analysis of these thresholds in Appendix F.9.


Weakness 2: Thanks to your comment, we have run additional experiments for newer model families - Qwen2.5-0.5B and OPT-1.3B models using the Emotions dataset, and report the new graphs in Figs. 18 and 19. We can again observe the three distinctive zones - Zone (A) - Neither FM nor FG, Zone (B) - FM, and Zone (C) - FG, further corroborating our main findings.


Weakness 3: Thank you for pointing it out. We have added additional explanation to it in Section 3.3.1.


Question 1: Thanks for your question. We attribute the reasoning to label memorization because in Section 4 when we didn’t inject any incorrect labels and just zuz_u, we observed the 3 distinctive zones, separating feature memorization and generalization. However, in Section 5.2, we inject zuz_u with the fixed wrong label together, and from prior works [1,2], we know that mislabeled samples are memorized. Hence, in the setup, the model memorizes the mislabeled samples by latching onto zuz_u even for low proportions of zuz_u in the train set. That is FM disappears while causing FG in the presence of LM.


Question 2: To address the reviewer’s question, we provide a sensitivity analysis of these thresholds in Appendix F.9.


Question 3: Thank you for this question. As discussed in the paper, the distinction between feature memorization and generalization depends on the proportion of the tracked feature in the training set. Testing this requires a feature whose proportion and presence can mechanistically be controlled and varied in each sample. To the best of our knowledge, no existing datasets provide naturally occurring features with controllable proportions (also discussed in Appendix D). We believe constructing such datasets that allow such control would be an interesting and useful future work.


[1] Feldman, Vitaly, and Chiyuan Zhang. "What neural networks memorize and why: Discovering the long tail via influence estimation", NeurIPS 2020

[2] Feldman, Vitaly. “Does Learning Require Memorization? A Short Tale about a Long Tail”, STOC 2020

We hope our response resolves your concerns and questions, and helps you consider raising your score. If there is anything that we can further clarify, please let us know. We appreciate your time and effort again.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up Response (I/II)

We thank the reviewer for their continued discussion on our paper. We address the remaining concerns below.


Distinction from Yang et al. (2022): First, let us explain why a held-out set had to be used to measure SsensS_{sens}. If we had only used train set samples containing zuz_u to measure SsensS_{sens} then we would not have been able to establish the minimum effect threshold TT which is derived when proportion of zuz_u samples was 0% in training. This is because in the 0%-proportion case, we will not have any zuz_u samples in training to measure SsensS_{sens}. Hence, a held-out set to measure SsensS_{sens} is needed, and since the minimum effect threshold T will be calculated on the held-out set, SsensS_{sens} will also have to be computed on the same held-out set when the proportion of zuz_u train samples > 0% for a “fair” comparison.

Regarding the question that FM is weak generalization. Yes, we are not against the perspective. We believe it is a matter of wording. If you suggest, we are happy to use weak/poor generalization and memorization in parallel, since it is the nature of memorization which is poor generalization, i.e., high overfitting gap (high relative train-heldout performance gap). To support this, we carry out a further experiment using CIFAR10-ResNet9, where we measure how inserting zuz_u during training changes the log-odds of predicting CiC_i for the train set and held-out set samples containing zuz_u, compared to when no zuz_u was inserted in training, followed by computing the relative train-heldout log-odds change gap. Intuitively, if the relative train-heldout log-odds change gap decreases as the proportion of xux_u train samples increases, that suggests a shift from memorizing the feature to generalizing it. We do this analysis for 8 different proportions across FM-FG zones - (1) FM Zone: 0.2%, 0.25%, 0.3%, 0.35%, and (2) FG Zone: 5%, 7.5%, 10%, 12.5% and report them in Appendix K, Table 11. It is clearly observed that the relative train-heldout log-odds change gap in the FM zone is significantly higher (≈\approx 10x) than the FG zone. It provides further evidence that the feature is memorized (high train-heldout log-odds change gap) at low proportions and generalized (low train-heldout log-odds change gap) at high proportions. This notion of distinction has not been studied previously in Yang et al. (2022) because it was not delineated what happens when a feature is memorized vs. generalized, apart from not doing the recognition check.

The very precise aim of the saliency analysis experiment (as suggested by the reviewer) is to show what happens to FM and FG if the feature is made less/more salient. Shrunk FM when the feature saliency is high (relative size 1:4, Fig 43e), and widened FM at low saliency (relative size 1:10, Fig 43a) show that FM-FG regimes get impacted as the feature saliency changes. Particularly at high saliency FG is dominated which indeed is also observed in Yang et al. (2022). However, otherwise (Figs 43a-d), FM zone exists which has not been investigated in Yang et al. (2022).

We hope this resolves your concern. Please correct us if we overlooked anything. We will discuss the distinctive contributions of Yang et al. (2022) and ours clearly in the revised manuscript, so that future readers can fairly understand the value of each.


Distinction from Turner et al. (2019): We agree that Turner et al. (2019) does provide an intuition or clue that while fitting (memorizing) the noisy labels the model can use zuz_u. However, we genuinely do not believe our contribution could have been trivially and directly inferred from the Turner et al. (2019)’s insight, since they did not investigate whether LM suppresses FM or not. We, on the other hand, investigate what happens when a feature (that was originally memorized in Zone B) is put together with noisy labels at the same proportion level. We will cite Turner et al. (2019) as further support for our Claim 3, and clarify their own contribution.

AuthorsOpenReview ↗

Follow-up Response (II/II)

Claim 3: Looks like we had misunderstood your question earlier. We thought you were asking us to do a new setup where we remove zuz_u and train a fresh new model on only noisy labels samples, i.e., (xcx_c, yLy_L) where xcx_c = xux_u - zuz_u, because of which we said model can memorize yLy_L using other features when zuz_u was absent during training. Having now understood your actual intended question, we now report what happens when we get rid of zuz_u from each of the (xux_u, yLy_L) pairs during inference in the experiment in Appendix J, Table 10. From the results, we can clearly observe that the model can no longer predict yLy_L in the absence of zuz_u, which reflects the fact that the model was indeed predicting yLy_L by latching onto zuz_u but not the other features.

Regarding the question on Ssens≈−0.7S_{sens} \approx -0.7, we believe there is a misunderstanding in how the reviewer perceived our response’s wording. It indeed tells us that the model is not correlating zuz_u with CiC_i and we hypothesized in Section 5.1.2 that this is happening because it is correlating zuz_u with yL≠Ciy_L \neq C_i. Hence, to verify this, we fixed the noisy label to yLfixedy_{L_\text{fixed}} in Section 5.2. Moreover, in the newly added Table 10 results (Appendix J), based on your further clarified intended question, we again affirmed that when we remove zuz_u from the original (xux_u, yLy_L) samples (during inference), then the model can no longer predict yLy_L, further confirming that the model is indeed associating zuz_u with yLy_L. We hope this resolves your concern.


Question regarding Appendix J: GradCams are computed w.r.t class yLy_L for each of the (xux_u, yLy_L) samples in the train set. In the previous image, we accidentally put 0.5% GradCam plots instead of 2.5%. We have now also added GradCam plots corresponding to 2.5% (see Figs 46-47). We used 100 examples for 2.5% proportion (i.e., 2.5% of 4,000).


We hope that our responses resolve your remaining concerns. If you have any more questions, then please let us know, we will be happy to answer them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.