- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies the difference between feature memorization and feature generalization, trying to establish a distinction between memorization at the feature level, generalization of a feature and the relationship with memorization of labels. In order to do that they perform an ample set of experiments along different modalities and architectures to show the claimed effect on different situations. The main structure of the experiments consists of planting a marker on images or text and check if the network ends up memorizing marked examples or generalize the marker, i.e. use the marker as a cue for classification. They ask two questions about this process: Is the network sensitive to the marker? Is the network generalizing the marker?
They do these experiments under three regimes, one with the marker keeping the label of the image/text where it is planted, two, with the marker assigned to a random label different from the original one and three, with the marker assigned to a fixed label not related to the original one for the image.
The authors make three claims: Concrete distinction between feature memorization and feature generalization, When more proportion of examples has the planted feature, the model generalizes and when is low the model memorizes. Label memorization suppresses feature memorization and causes feature generalization.
Critical strengths
The work seems to be clearly structured and the set of experiments is broad. Using a planted feature is a clever way to control the conditions to study memorization. Measuring each example with and without the marker add credibility to the set up. Experiments run across several domains and architectures with different types of tracked features. This makes it possible to check which effects are setting-specific and which ones hold along settings. The paper also makes a great effort in providing complementary metrics to support the observations, like Grad-CAM, and Sensitivity readings.
The use of 5 seeds gives a sense of run-to-run variability, and they provide the raw numbers in the appendix.
Critical weaknesses
Overall, this work aims to present a new designation of shortcut learning, spurious correlations, random label fitting and backdoor attacks.
When the number of images with Z_u are low but label agrees with the original class, the feature Z_u acts as spurious correlation. When more training images carry Z_u the network can use it as a shortcut for those images (Geirhos et al., 2020), creating a cheaper representation, simplicity bias (Shah et al., 2020) and gradient starvation (Pezeshki et al., 2021). The closest prior work is Yang et al. (2022), where also a feature or cue is planted in training examples. When the images with Z_u have a random label different from the original class assigned to them section 5.2 is then random-label fitting (Zhang et al., 2017/2021; Arpit et al., 2017): the feature provides valid information about the label, (Not C_i) and the network uses it to avoid classifying these examples as the original class. S_Sense reflects that going negative. When the Z_u is paired with a single fixed label different from the original class, the information provided by Z_u is informative, gradient flows to this feature, since it is the only consistent marker with the new class in those examples. This is the dirty-label backdoor setting (Gu et al., 2017; Chen et al., 2017; see Li et al., 2022 for a survey), a planted cue with a different label where the network relay on this added Z_u feature to output the assigned label, since that is the only thing in common for that set of images and label.
I would ask the authors to state which observations go beyond these works. As it stands, the FM, FG and LM vocabulary relabels known effects rather than adding a new mechanism.
The FG definition seems too permissive: under condition (ii), which uses ‘or’, a model that performs worse under the tracked feature condition would have a high Avg-TestAcc-Diff without the need to generalize to the tracked feature. Zone C starts in the apparent sharp increase in slope of MP in figure 1, however it is interesting to see that the Test Accuracy of Z_u examples are not affected and many more examples needs to be added to really notice a decrease. What happened in this interval between MP increases but Test Accuracy does not change? Is this FM, FG or in between? Would changing the definition of FG to “and” create a fourth zone, Zone B’.?
In Figure 1, Grad-CAM shows that the child and woman images are already misclassified as class 3 in zone A, with the same hotspot active over all regimes. They appear to be baseline errors and say nothing about Z_u. In zone B (0.45%), there is a faint activation of the marker “ID” for two of the three examples presented, the figure seems to not support lines 298 and 299 (one of them is not the same as in zone A and zone C. How were the examples selected? Would be interesting to see a heatmap of the Grad-CAM activation over all set of images.
Plots are hard to read with up to four vertical axes in the same plot. In the plots (e.g Figure 4) the P-values are shown with confidence interval, since P-value averaging is not a valid way to combine them, how was the aggregation computed? With hundreds of images, even a tiny S_sens reaches p < 0.05, and there is no minimum effect size.
Another concern is the broad use of the word memorization; it seems to have different definitions depending on what is applied to. Is feature memorization only for Z_u or does it apply to any feature available? Is label memorization about one feature or many features or the whole image? The label has to be associated to some feature in the image to be able to correctly classify it.
In the experiments of section 5.1 with Z_u and random labels if the training acc is 100% the images with Z_u are “memorized”. Those examples have to be remember based on some features on the images, even if the feature is not known or different for every image. If they are using a feature to be memorized then FM is not suppressed, just not measured because the feature metrics only track Z_u. In section 5.2 with a fixed label, the claim that LM removes FM and pushes FG, can also be stated as that a marker paired with a consistent label is learned as a rule, which is the backdoor setting.
Minor points:
The main text does not specify what set of classes from the UTK face dataset is used. Appendix state it is ethnicity. This should be stated in the main text. I would suggest to reduce the tone of affirmations like “These findings clearly verify that…” Regenerate table 6, last three rows read inconsistent values for Avg-TestAcc-Diff . Since the Avg-Test Accuracy in those rows reads 0, it should be Avg-TestAcc-Diff of 69.17.
Errata:
Line 336, 1452: “fixed” is repeated? figure 12, 14: No FM or FL, shouldn’t it be No FM or FG? What is FL?
Overall assessment:
The experiments are extensive and carefully run, however in their current form the three claims either look like a restatement of known phenomena (shortcut learning, random-label fitting and backdoor learning) or the metrics do not fully support the claims. Clarifying the different definitions of memorization used implicitly in the text, would help to clarify if some of the claims are a rewording of existing phenomena or genuinely new observations. Especially for claim 3, it is not clear whether FM is completely suppressed or present but not observed. Improving the readability of the plots and how the information is presented would help make the results easier to follow.
Questions for authors
Do the zone boundaries (the proportions where zones B and C begin) depend on the marker’s complexity or salience? Hermann, K. L., Mobahi, H., Fel, T., & Mozer, M. C. (2024) studied this for shortcut learning.
How to address the memorization of features when the network uses other features different from z_u?
How was the P-value aggregated over the runs? And how does having hundreds to thousands of examples affect the calculated P-value? with n in the hundreds to thousands, p < 0.05 is reached by very small effects.
3 comments
Response (II/III)
Weakness 4: Thanks a lot for raising this question. We provide further clarification as follows:
- We agree the plots are complex but our goal was to simultaneously make everything legible to analyze how different metrics behave as the proportion of increases. However, if the reviewer has any better suggestion, we are happy to update it.
- As discussed in the paper, we compute and p-value for each seed. To show the mean trend we average both metrics. The p-value curve in the figure is the across-seed mean and standard deviation, showing a descriptive summary of how significance evolves as the proportion of increases. However, we would like to clarify that Recognition is decided per seed (p < 0.05 and > 0). To make this explicit, we have added a graph in Appendix I, reporting the percentage of seeds (out of 5) satisfying the recognition criterion at each proportion.
- We agree that with hundreds to thousands of samples, even a small can reach p < 0.05, and that our original criterion ( > 0 and p < 0.05) did not specify a minimum effect size. Hence, to alleviate this issue, we make the following changes:
- First, our sweeps across increasing proportions already show how and the p-value behave when no (0%) training samples are present. At 0% of training samples (the leftmost point in Figure 4), the models are trained without , under the same settings as every other proportion. Across all model-dataset settings, at 0% stays close to zero and p-value ≥ 0.05 whereas in the recognition regions (zones B and C) is clearly higher and p-value < 0.05. So, at our sample sizes, the baseline of 0% of in training does not reach significance, and the significance we observe with proportions in Zones B and C cannot not be explained by larger sample size, because if that was the case then Zone A could have also had p-value < 0.05.
- Second, to handle the minimum effect size issue, we now define recognition as and p-value < 0.05, where T is the one-sided 95% upper prediction bound of when the proportion of train samples is 0%. Recognition here is a binary decision about whether was recognized or not. The threshold imposes an explicit minimum effect on the recognition criteria. We have made this update in Section 3.3.1 and added the upper prediction bound results in Appendix I, showing that in the FM and FG zones remains above T (minimum effect size threshold).
Weakness 5: Thank you for your question. To measure when and how a feature is memorized, we propose feature memorization (analogous to how humans process information as presented in Section 3.1), in which the model memorizes a feature without generalizing it to the test set, by tracking . On the other hand, label memorization [9] is about the (image, label) pair (see Eq 2), and not at the ‘feature level’. If you have further questions regarding these, we will be happy to answer those.
Weakness 6: We believe there is a misunderstanding in how the reviewer is perceiving our claim. We clarify it as follows:
-
The aim of Section 5 is to understand how the FM-FG regime of Section 4 interacts with LM. In Section 5.1, we insert together with random labels and observe that the FM-FG zones disappear under the definitions discussed in Section 3.4. We then hypothesize in Section 5.1.2 that this is because the model memorizes the noisy labels while latching onto , and not onto unrelated features of the image and later verify the hypothesis in Section 5.2 by fixing the noisy label. To add further support, we now also add the mean Grad-CAM plots for the (, ) samples in Appendix J (see Fig 46), where we can see that while undergoing random label memorization (), the model is fixating onto . This further indicates that the memorization is largely organized around , and not around some other features under LM. Regarding why FM-FG regions disappear when random labels are added, since the labels are random, the model does not learn a fixed correlation and hence the FM-FG zones are not observed under the definitions discussed in Section 3.4.
-
For the second part of the question, we would like to clarify that the backdoor setting is one application of feature generalization, and such work does not investigate the interplays between label memorization and feature memorization as we do. In our work, we identify, for the first time, that label memorization suppresses feature memorization and causes feature generalization by showing the full spectrum of memorization/generalization through features and labels.
Minor Points: Thanks to your comment we have added the set of classes considered in the UTK-Face dataset in the main text in line 229. We have also made the edits in the table - thanks for pointing them out.
Response (III/III)
Errata: Thank you for your careful reading. Yes, “fixed” is accidentally repeated - we have removed it. For the second question, you are right, we actually meant “FG” and not FL (a typo). We have updated it now.
Question 1: This is a valid question. To answer to it, we carried out additional analysis where to vary the saliency of the feature, we vary its relative size w.r.t original image size. We do this for CIFAR10, ResNet9 with as the English letter ``A’’. Relative size of is varied in the range of [(1:10), (1:8), (1:6), (1:5), (1:4)] for both height and width, where 1:10 relative ratio corresponds to the smallest feature size (least salient) and 1:4 corresponds to the largest feature size (most salient). We report the results in Appendix G. From the results, we observe that as the feature size increases from 1:10 relative ratio (least salient) to 1:4 relative ratio (most salient), the FM regime (Zone B) reduces and FG starts happening at earlier proportions. On the other hand, as the feature size decreases from 1:4 relative ratio (most salient) to 1:10 relative ratio (least salient), the FM region expands. We thank the reviewer for suggesting this experiment which has helped us to further strengthen our paper.
Question 2: We are sorry, but could you please try to rephrase what you asked in this question for us? All the authors tried to understand this correctly, but we could not. We would appreciate it if you could help us understand it.
Question 3: Please kindly refer to the response to Weakness 4.
[1] Last layer re-training is sufficient for robustness to spurious correlations, ICLR 2023
[2] The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learning, TMLR 2026
[3] Freeze then train: Towards provable representation learning under spurious correlations and feature noise, AISTATS 2023
[4] Shortcut Learning in Deep Neural Networks, NMI 2020
[5] The pitfalls of simplicity bias in neural networks, NeurIPS 2020
[6] Simplicity bias in 1-hidden layer neural networks, NeurIPS 2023
[7] Understanding deep learning requires rethinking generalization, ICLR 2017
[8] A Closer Look at Memorization in Deep Networks, ICML 2017
[9] Feldman, Vitaly. “Does Learning Require Memorization? A Short Tale about a Long Tail”, STOC 2020
We hope our response resolves your concerns, and helps you consider raising your score. We would be happy to answer any more questions that you may have. Thank you.
Response (I/III)
We thank the reviewer for putting in time to review our paper. We address their questions below in detail. Please note that the new edits are marked in blue in the updated PDF.
Weakness 1: Thank you for raising this point. We clarify that our work is not merely relabeling of prior work and articulate our distinction and novelty.
In summary, there was no prior work that investigated the grand entire picture via a mechanistic and systematic framework, and we are the first to make this contribution.
Weakness 2: Thank you for your careful observation. No, this is not a new zone, and it still lies in the FG regime. To validate this, we analyze these intermediate points in Appendix H for the UTK-Face, ResNet18 setup where MP2,3 sharply increases but test accuracy minimally drops (see Fig 8). As seen in Table 9 in Appendix H, the misclassification % of class 2 xu test samples as class 3 does rise up (3.02% 7.51% 9.20%) as it shifts from the FM zone (0.45%) to the intermediate FG zone (0.82%, 1.14%) . However, the total misclassification % of class 2 xu test samples does not increase significantly in the intermediate FG zones because the misclassification % of class 2 xu test samples as other classes other than class 3 also decrease (19.94% 14.91% 15.55%). This causes the test accuracy to remain minimally impacted in the intermediate FG zones. Now coming to the question of why these intermediate points are still categorized under the FG zone, we report the mean gradcam activations of class 2 xu test samples misclassified as class 3 in Fig 44. We can clearly see that even in these intermediate points, the model is relying upon zu to misclassify the images as class 3 during test time which is also reflected by the increase in MP2,3 as seen in Fig. 8. Therefore, to classify the feature to be memorized, the “AND” criteria is required. We hope this resolves the reviewer’s concern.
Weakness 3: We respectfully state that there might be a misunderstanding of what the GradCam plots convey, and we clarify it as follows. GradCam aims to provide support to our FM-FG regimes, where we want to check which features the model dominantly latches on to misclassify xu test samples belonging to class Cj as Ci. If it is largely using zu as the prediction anchor, then it is in Zone C (FG regime); otherwise in either Zone A (neither FM nor FG) or Zone B (FM regime). We put Zone A Gradcam plots as a baseline comparison so that we can assess that Zone B should also not largely use zu as the prediction anchor like in Zone A. To further show that this does not reflect a random subset of our choice, we provide mean GradCam activation analysis across all sets of xu test samples belonging to class Cj that are misclassified as Ci in Appendix E.1.1. It is clearly observed that the mean gradcam activations are alike where the model is largely utilizing other features (not zu) to misclassify Cj as Ci in Zone A (neither FM nor FG) and Zone B (FM). However, in Zone C (FG), the model relies on zu to misclassify them. We thank the reviewer for suggesting this analysis and we hope it resolves their concern.