acceptodds
Under review as a conference paper at ICLR 2027

What Can the Representation Space Reveal About Memorization?

Abstract

Memorization plays an important role in how neural networks fit rare and atypical training examples, but measuring per-example memorization is computationally expensive: estimating the Feldman–Zhang memorization score requires training many models. Existing practical proxies primarily rely on losses, gradients, or loss-surface geometry. We investigate whether memorization is directly reflected in the geometry of learned representations. We propose a single-run proxy that tracks, throughout training, how strongly an example's position relative to class centroids in representation space disagrees with its observed label. Under explicit assumptions, we relate this trajectory-averaged disagreement to Feldman–Zhang memorization through a sandwich bound, linking our score to memorization weighted by the fraction of training before the sample’s representation becomes consistently aligned with its assigned class. Empirically, our score achieves the strongest agreement with Feldman–Zhang memorization among the evaluated proxies on ImageNet, while remaining competitive on CIFAR-100. The same signal is highly effective for mislabel detection and true-label recovery. For mislabel detection on CIFAR-100 under within-superclass label flips (asymmetric noise), our method achieves the highest AUROC across nearly all evaluated noise rates. Its AUROC decreases by less than one percentage point as the noise rate increases from 5% to 30%, compared with drops of up to approximately twelve percentage points for competing methods. These results suggest that memorization leaves a persistent and measurable geometric signature in learned representations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Memorization, Representation geometry, Training dynamics

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_xfGLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to define a new proxy metric called RGD for the F&Z memorization scores in classification settings. They base the RGD metric on representation level analysis at the penultimate layer (before classification head). They show that their new metric has a higher correlation with the F&Z memorization scores than prior proposed metrics for ImageNet while achieving comparable values for CIFAR100. They also utilize the new metric to detect mislabeled samples (symmetric and asymmetric label noise) and recovering their true labels.

Critical strengths
  1. Understanding memorization in the representation space is novel and to my prior knowledge it has not been previously explored to estimate the highly expensive F&Z memorization scores. The paper is easy to follow as well with a clear theme, which I really appreciate, especially given how recent papers in LLM era are portrayed to be overly complicated.

  2. The theory associated with the newly proposed RGD metric also is interesting and adds value to the proposed metric to relate with the F&Z memorization score. The assumptions made in the theory also seem reasonable and intuitive.

Critical weaknesses

I have thoroughly reviewed this paper and below are the major, critical weaknesses that I see in the current form:

  1. There are some key memorization literature papers (for classification settings - as this paper is about that in the current form) missing in Related Work and I believe they should be cited:
  • Learning and Memorization, ICML 2018
  • Deep Learning Through the Lens of Example Difficulty, NeurIPS 2021
  • Can Neural Network Memorization Be Localized?, ICML 2023
  • Impact of Layer Norm on Memorization and Generalization in Transformers, NeurIPS 2025
  1. I understand that the F&Z scores were only made available for the classification tasks. However, in this era of LLMs it is not clear whether this new metric will even estimate the "counterfactual memorization" definition proposed in [1]. The counterfactual memorization definition is essentially the leave-one-out (LOO) variant for generative tasks. I believe it is important (critical) to show that the metric can estimate LOO for generative tasks as well. Authors could look into long-tail generative language tasks to simulate the long-tail settings. I will be okay if they could provide even a single dataset-model setting.

  2. No timing analysis is provided to compare with existing available proxy metrics. I believe it is important that the authors provide a comparison between them.

  3. I checked Appendix A and A.1 of the paper and the training hyper-parameters differ from the ones used in F&Z [2] original paper (see their Appendix B). So, how can the authors compare their metric values with the original F&Z scores if the hyper-parameters are different. Further clarification is required regarding this with a controlled same setup.

  4. The authors said they only did experiments for ResNets, however their mislabeled sample detection setup can easily be shown for vision and language transformers easily since they do not need to estimate the F&Z scores for the same. Hence, it is important that they show how the proposed metric performs for transformers atleast for mislabeled samples detection and recovery experiments. Also, why are CIFAR-10 asymmetric label noise experiments not provided in the current version (only CIFAR100 is provided for both types of noise and CIFAR-10 with symmetric noise in Tables 2, 3, 4)?

  5. The authors say that they use only the penultimate layers representation analysis for developing their metric. However, they do not consider doing the representation analysis at other intermediate layers. An experiment to check this is going to be very beneficial to understand which layers are the most important for efficiently approximating the F&Z scores, and further strengthen this paper.

Questions for authors
  1. [3] did an adversarial distance analysis showing F&Z M-scores and CSL (their proposed metric) aligned with poor adversarial robustness, i.e., M-score/CSL-score is inversely proportional to Adversarial Distance. It will be informative to know if RGD also follows the same pattern or not, further confirming that memorization means failure to learn robust, generalizable patterns.

  2. [3] also did a duplicate detection experiment showing CSL can identify duplicates quickly than other methods. Does RGD outperform CSL for duplicates detection as well? An analysis regarding this would be nice.

  3. Please address my Weaknesses as well. All the points listed in Weaknesses and Questions are important from my opinion.

[1] Counterfactual memorization in neural language models, NeurIPS 2023

[2] What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation, NeurIPS 2020

[3] Towards Memorization Estimation: Fast, Formal and Free, ICML 2025

I am willing to increase the score if the authors can successfully answer the weaknesses and questions raised by me. (PS: I have not used any AI tool to review your paper and am genuinely curious about your paper).

6 comments

The author-reviewer end of discussion deadline is close by. If the authors wish to respond to my concerns, then please do so at the earliest. It would be easier to do the discussion if the rebuttal is posted early.

We thank the reviewer for their time, suggestions, and patience. We have been conducting additional experiments and preparing detailed responses to address your concerns. We appreciate your feedback and welcome further discussion.

Weakness 1 - Related Work

Thank you for the suggestion. All four papers are now cited in the Related Work section (Section 2).

Weakness 2 - Counterfactual Memorization in Language Modeling

We have added a counterfactual memorization experiment for autoregressive language modeling in Section 4.4.1 and Appendix I of the revised manuscript.

Following Wu and Papyan (2024) [1], we extend RGD to language modeling by treating next-token identities as classes: prediction positions sharing the same target token belong to the same class, and the hidden representations of their preceding contexts are used to construct class centroids. Averaging the resulting representation–label disagreement over token positions and training epochs gives a document-level score, computed from a single training run.

We evaluate this extension on WikiBio [2] using a small decoder-only Transformer trained from scratch. Following the counterfactual memorization protocol of Zhang et al. [3], we train 128 models on different 4,000-document subsets drawn from a pool of 8,000 biographies. To investigate the long-tail setting suggested by the reviewer, we evaluate 128 biographies: 64 sampled from the highest decile of lexical rarity and 64 length-matched biographies from the lowest decile. Each biography is included in 64 training subsets and excluded from the other 64. Counterfactual memorization is estimated as the difference in mean teacher-forced next-token prediction accuracy between the two groups.

RGD achieves a Pearson correlation of 0.710 ± 0.012 with counterfactual memorization, compared with 0.597 ± 0.008 for trajectory loss and 0.542 ± 0.011 for final loss (mean ± std over three reference runs). The positive association also persists within each group separately: 0.529 ± 0.026 for rare and 0.673 ± 0.013 for common biographies. Cosine similarities are also reported in Table 6, but are uniformly high (0.96–0.97). We therefore emphasize Pearson correlation in this setting.

These results provide empirical evidence that representation-space disagreement captures counterfactual memorization signals in an autoregressive generative setting. Broader evaluation across language models remains future work.

References

[1] Wu and Papyan, "Linguistic Collapse: Neural Collapse in (Large) Language Models," NeurIPS 2024. https://arxiv.org/abs/2405.17767

[2] Lebret et al., "Neural Text Generation from Structured Data with Application to the Biography Domain," EMNLP 2016. https://doi.org/10.18653/v1/D16-1128

[3] Zhang et al., "Counterfactual Memorization in Neural Language Models," NeurIPS 2023. https://arxiv.org/abs/2112.12938

Weakness 3 - Computational Cost

We thank the reviewer for highlighting the importance of computational efficiency.

We have added a computational-cost comparison with existing memorization proxies in Appendix L, following the analytical, hardware-independent cost framework of Ravikumar et al. (2025, 2026). The comparison includes both full-training and early-stopping settings, with costs normalized to one standard full-length training run. These are operation-count estimates rather than measured wall-clock times, which depend on hardware.

RGD requires one additional forward-only pass per recorded epoch, without additional backward passes or retraining. This corresponds to approximately 1.33× the cost of standard training, compared with approximately 1× for loss-based proxies accumulated during training and 14× for curvature-based estimation under the specified assumptions.

Furthermore, our shorter-trajectory experiments show that RGD retains substantial agreement with FZ memorization using only the first 10% of training, achieving cosine similarities of 0.73 on ImageNet and 0.83 on CIFAR-100. If training is terminated at this point, the estimated total computational cost is approximately 0.13× that of one standard full-length training run, within the 0.1–0.3× analytical cost range reported for CSG and SGAL with early stopping.

Weakness 4 - Training Configuration and FZ Scores

We agree that FZ memorization is formally defined with respect to a learning algorithm and its training configuration. However, there are several reasons why comparing our scores with the published FZ estimates remains meaningful.

First, Feldman and Zhang [1] themselves investigated the consistency of memorization estimates across different architectures (Section 1.1 and Appendix C, Figure C.2 of the NeurIPS version). On CIFAR-100, they observed substantial consistency across ResNet-50, ResNet-18, Inception, and DenseNet-100, with ResNet-18 showing the closest agreement with ResNet-50 among the alternative architectures evaluated. Notably, their DenseNet experiments also used a different learning rate and batch size, both halved relative to ResNet-50 (Appendix B). These findings provide empirical evidence that memorization estimates can remain consistent across architectures and training configurations.

Second, using the publicly released FZ scores as a common reference under different training configurations is an established practice in memorization-proxy research. Curvature [2], CSL [3], and CSG [4] all evaluate their proxies against the published FZ scores despite using different training hyperparameters from the original FZ experiments. Our training configurations closely follow those adopted in these studies.

Importantly, we use the published FZ scores as a common reference for evaluating agreement across all methods, rather than claiming that they are the exact memorization scores of our specific training algorithm.

References

[1] Feldman, V., & Zhang, C. (2020). What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. NeurIPS.

[2] Garg, I., Ravikumar, D., & Roy, K. (2024). Memorization Through the Lens of Curvature of Loss Function Around Samples. ICML.

[3] Ravikumar, D., Soufleri, E., Hashemi, A., & Roy, K. (2025). Towards Memorization Estimation: Fast, Formal and Free. ICML.

[4] Ravikumar, D., Soufleri, E., Hashemi, A., & Roy, K. (2026). Memorization Through the Lens of Sample Gradients. ICLR.

Reviewer_fh6wOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The work proposed Representation Geometry Disagreement (RGD), a single-training-run proxy for Feldman–Zhang (FZ) memorization. At every epoch, each training example's penultimate feature is compared with the class centroids by cosine distance. RGD has the "best" FZ agreement on ImageNet (CS 0.84 / Pearson 0.74 vs. 0.79 / 0.64 for CSL). On CIFAR-100 it trails CSL and SGAL. Under asymmetric noise it is the most stable detector. Appendices cover truncated trajectory windows, alternative geometric statistics, and a temperature sweep.

Critical strengths

RGD needs only penultimate features from one training run and has essentially one hyperparameter. Structured noise is the realistic case for mislabel detection, and that robustness is what most supports acceptance

Scalar memorization scores cannot propose an alternative label. RGD's per-class geometry can, and it beats In-Confidence and Confident Learning on true-label recovery in all three settings

Critical weaknesses

Assumption 2 identifies the centroid softmax with the vMF posterior only when τ = 1/κ_t. Remark 1 then fixes τ for all epochs, so the identity s_i^(t) = 1 − Pr[Y = y_i | h^(t)] used in the proof no longer holds as stated.

Assumption 3 state that with an unbounded per-example error ε_i, the epoch-t centroid posterior approximates the leave-one-out prediction probability. This is circular reasoning. With ε_i unconstrained, the sandwich bound is vacuous; it becomes informative only under the heuristic in Remark 2, but this is never proved in paper.

The ImageNet baselines are copied (to two decimals, without variance) from Ravikumar et al. (2026), while RGD is re-run over three seeds. The headline ImageNet win therefore compares numbers from different training runs.

On CIFAR-100, where all baselines were re-run, RGD is below CSL and SGAL for FZ agreement and below four baselines for symmetric mislabel detection, which is not aligned with other results and not discuss the reason.

There are no representation-space baselines, even though the core claim is that the memorization signal "lives in" representation space. Missing are the kNN/density proxies studied in Jiang et al. (2021, C-score), deep-kNN label-noise filtering, and FINE / SimiFeat-style feature-based noisy-label detectors. Without them, the novelty of using representation geometry, and the claim of being "the first" evaluation under asymmetric noise, are hard to assess.

Questions for authors

Can you test Assumption 3 directly by comparing Pr[Y = y_i | h^(t)(x_i)] at unfit epochs with the held-out probabilities in FZ's released ImageNet/CIFAR-100 estimates, and report the empirical distribution of ε_i?

Can you re-run CSL, SGAL, and CSG on ImageNet under your own pipeline and seeds, so that Table 1 is like-for-like?

Please add representation-space baselines (kNN label disagreement, class-conditional density / Mahalanobis, FINE). This would establish whether the centroid-softmax formulation matters beyond "use the features"; Appendix C suggests a centroid-distance ratio is already competitive.

3 comments

We thank the reviewer for their time and suggestions.

Q1, weakness 1 and weakness 2 –

We understand that the error term εi\varepsilon_i mentioned in the review refers to δi\delta_i in our original formulation.

We agree that the original formulation did not sufficiently justify the magnitude of the estimation discrepancy. Moreover, using a fixed temperature is an approximation, and the equality between the centroid-induced softmax and the vMF posterior is not guaranteed when τ≠1/κt\tau\neq 1/\kappa_t. To address these issues, we have revised Assumption 3, Remark 1, and Proposition 1 to explicitly account for epoch-dependent estimation errors.

Specifically, we define the epoch-dependent estimation discrepancy as

δi(t)=Pr⁡f←A(S∖i)[f(xi)=yi]−Pr⁡[Y=yi∣h(t)(xi)]\delta_i^{(t)}=\Pr_{f\leftarrow A(S\setminus i)}[f(x_i)=y_i]-\Pr[Y=y_i\mid h^{(t)}(x_i)].

To account for the fixed-temperature approximation, we introduce

ei(t)=Pr⁡[Y=yi∣h(t)(xi)]−pown(t)(xi)e_i^{(t)}=\Pr[Y=y_i\mid h^{(t)}(x_i)]-p_{\mathrm{own}}^{(t)}(x_i).

We define the combined discrepancy as

δ^i(t)=δi(t)+ei(t)\widehat{\delta}_i^{(t)}=\delta_i^{(t)}+e_i^{(t)}.

The revised Proposition 1 establishes the conditional sandwich bound

ρi(mem(A,S,i)−Ai)≤RGDi≤ρi(mem(A,S,i)+Ai)+(1−ρi)ε\rho_i(\mathrm{mem}(A,S,i)-A_i)\leq \mathrm{RGD}_i\leq\rho_i(\mathrm{mem}(A,S,i)+A_i)+(1-\rho_i)\varepsilon,

where

Ai=∣1Ti−1∑t<Tiδ^i(t)∣A_i=\left|\frac{1}{T_i-1}\sum_{t<T_i}\widehat{\delta}_i^{(t)}\right|.

Unlike the original formulation, the revised bound uses the absolute value of the mean epoch-dependent combined discrepancy rather than a single per-example error bound. We have also removed the previous heuristic argument regarding the smallness of the discrepancy. The revised proposition establishes a conditional relationship between RGD and FZ memorization, with the bound magnitude examined empirically in Appendix E.

Empirical evaluation. Following the reviewer's suggestion, we evaluated the combined discrepancy on CIFAR-100 (ResNet-18, seed 0) using the published FZ memorization scores across five fitting thresholds, ε∈0.01,0.02,0.05,0.1,0.2\varepsilon\in\\{0.01,0.02,0.05,0.1,0.2\\}.

Figure 5 (Appendix E.2) shows the empirical distribution of δ^i(t)\widehat{\delta}_i^{(t)} over representation-unfit epochs, with a pronounced peak near zero and tails in both directions. Figure 6 reports the distribution of AiA_i, with median values ranging from 0.114 to 0.177.

Table 20 reports median values of 0.061–0.067 for ρiAi\rho_i A_i and 0.070–0.196 for the bound term Bi=ρiAi+(1−ρi)εB_i=\rho_i A_i+(1-\rho_i)\varepsilon. Furthermore, the Pearson and Spearman correlations between RGD and ρi,mem(A,S,i)\rho_i\\,\mathrm{mem}(A,S,i) are 0.804–0.823 and 0.861–0.863, respectively.

These results empirically characterize the discrepancy and its contribution to the revised bound. The revised theoretical formulation is presented in Section 3.2, with the empirical analysis in Appendix E.

Q2, weakness 3 –

We agree that evaluating the baselines under the same experimental pipeline provides a more controlled comparison.

We have now rerun the three requested baselines, CSL, CSG, and SGAL, on ImageNet using our training pipeline and the same three seeds as RGD. The results (mean ± std) are:

MethodCosine SimilarityPearson
CSL0.7561 ± 0.00080.5893 ± 0.0004
CSG0.7248 ± 0.00270.5257 ± 0.0027
SGAL0.7952 ± 0.00170.6386 ± 0.0033
RGD (Ours)0.8394 ± 0.00100.7361 ± 0.0011

RGD maintains the highest agreement with Feldman–Zhang memorization among these methods, outperforming the strongest rerun baseline, SGAL, in both metrics. We have added the three-seed results for CSL, CSG, and SGAL on ImageNet to Appendix H of the revised manuscript. We will also evaluate the remaining baselines across three seeds and include these results in the next revision.

Q3, weakness5 -

Our “first” claim specifically concerns evaluating memorization proxies under structured asymmetric label noise, not the first use of representations for noisy-label detection. To avoid this ambiguity, we have clarified this distinction in the revised manuscript in Section1. The suggested methods primarily target noisy-label detection or atypicality, whereas Jiang et al.'s C-score measures held-out predictive consistency rather than FZ memorization.

We evaluated additional representation-based measures on CIFAR-100 (three seeds), using the same trajectory averaging as RGD. Our method achieves CS/Pearson of 0.8577 ± 0.0010 / 0.7584 ± 0.0018, outperforming kNN distance-based atypicality [1] (0.7110 ± 0.0002 / 0.6135 ± 0.0011) and class-conditional Mahalanobis atypicality [2] (0.6026 ± 0.0008 / 0.0426 ± 0.0023), while remaining competitive with kNN label disagreement (majority-vote adaptation [3], 0.8510 ± 0.0006 / 0.7630 ± 0.0010). These results further support our finding that memorization-related information is present in representation geometry. Moreover, Appendix C shows that RGD achieves stronger FZ agreement than the simpler centroid-distance ratio.

On CIFAR-100 with 20% within-superclass asymmetric noise (three seeds), RGD achieves AUROC of 0.9337 ± 0.0004, compared with 0.7767 ± 0.0021 for FINE [4] and 0.6280 ± 0.0015 for Mahalanobis filtering [4,5]. The latter uses a continuous distance score derived from the filtering procedure. Implementation details and adaptations are provided in Appendix G. We will extend the mislabel comparison to the remaining noise rates in the next revision.

References

[1] Ramaswamy, Rastogi, & Shim (2000). Efficient Algorithms for Mining Outliers from Large Data Sets. SIGMOD.

[2] Lee et al. (2018). A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS.

[3] Bahri et al. (2020). Deep k-NN for Noisy Labels. ICML.

[4] Kim, Ko, Cho, Choi, & Yun (2021). FINE Samples for Learning with Noisy Labels. NeurIPS.

[5] Lee et al. (2019). Robust Inference via Generative Classifiers for Handling Noisy Labels. ICML.

weakness4 -

Regarding our results, we do not claim that RGD outperforms all baselines in every setting. Our results show the strongest FZ agreement among evaluated proxies on ImageNet and competitive performance on CIFAR-100, although CSL and SGAL perform better on the latter.

Regarding symmetric CIFAR-100 mislabel detection, although four baselines outperform RGD, its AUROC remains above 0.97, with a maximum gap of only 0.0232 from the best baseline.

The differences in mislabel-detection rankings are accompanied by substantial changes in baseline performance between symmetric and asymmetric corruption, whereas RGD remains comparatively stable. RGD achieves AUROC ≥ 0.9281 across all evaluated symmetric and asymmetric noise settings. For example, on CIFAR-100 with 30% noise, CSG drops from 0.9936 under symmetric corruption to 0.7192 under asymmetric corruption. Similar decreases are observed for Curvature (0.9932 to 0.7159), CSL (0.9903 to 0.8533), and SGAL (0.9861 to 0.8693). In contrast, RGD decreases only from 0.9704 to 0.9281.

One possible explanation is that symmetric corruption often creates pronounced label inconsistencies that loss- and gradient-based methods may more readily detect, whereas asymmetric corruption involves semantically related classes, making these inconsistencies more subtle. Since RGD measures relative alignment with class centroids, this geometric information may help preserve its discriminative ability under structured corruption. Our results are consistent with this interpretation.

We clarified these differences in Section 4.2 of the revised manuscript.

We hope that our responses and additional experimental results address your concerns, and we would appreciate it if you could reconsider your assessment in light of these clarifications.

Thanks for your responses. Authors have resolved my concerns. I have raised my score to 2: Weak Reject.

Reviewer_bWFUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds a new way to spot examples a model had to memorize. Instead of looking at loss or gradients, it watches where each image sits compared to other images from the same class during training. Weird or mislabeled images stay far from their group for a long time. From that idea the paper derives a simple score, show with math why it should track memorization.

The paper claims that this score matches what they consider as the gold-standard memorization measure best on ImageNet, while being significantly less costly. It finds wrong labels reliably, even when many labels are wrong, and it can often guess what the correct label should have been.

Critical strengths

Having a fast technique to mesure memorization is certainly interesting.

The paper contains multiple ablations, although they are mostly carried on CIFAR, which is a bit toyish as a dataset.

Critical weaknesses

I am not an expert on membership inference, but I know a few works, and i wonder why this paper hasn't discussed the Bayes-optimal work of Sablayrolles et al. namely "White-box vs Black-box: Bayes Optimal Strategies for Membership Inference". This paper claims to derive the optimal strategy for membership inference attack, and proposes some approximation of it.

While I understand that the target Feldman-Zhang's metric is a different target, Sablayrolles' paper shows that the optimal strategy depends only on the loss, and since the method in the submitted paper is empiracally motivated and provide limited guarantees (in the form of the sandwich bounds), I wonder whether one would rely on such a score for memorization evaluation.

Memorization is here considered for a traditional supervised training setting for image classification with labels, while recent models are mostly trained with self-supervised objective and unannotated datasets. This sounds like an inherent limitation of this class of method.

Questions for authors

Please address my concern above.

In addition, all datasets considered for memorization here are balanced (i.e., fixed number of images per class). Can we expect the method to work when the number of image per class is very low?

4 comments

We thank the reviewer for their time and suggestions.

Q1 and Weakness:

  • Relation to Sablayrolles et al.: Their Bayes-optimal result concerns membership inference, i.e., determining whether a sample belongs to the training set. In contrast, our objective is to estimate the continuous, counterfactual Feldman–Zhang (FZ) memorization score, which measures the change in prediction probability when a training example is removed. Optimality for membership inference therefore does not establish that loss is an optimal estimator of FZ memorization. Moreover, their loss-based optimality result relies on specific assumptions about the distribution of model parameters, in particular a Gibbs-form posterior. Suri et al. [2] subsequently showed that the sufficiency of black-box information does not generally extend to SGD, where optimal membership inference requires white-box information.

  • Reliability of the score: We do not propose RGD as a replacement for the FZ estimator; it is a single-run proxy. We establish a theoretical relationship between RGD and the FZ score under explicit assumptions, as stated in Proposition 1. Our primary evidence is empirical agreement with FZ scores on ImageNet and CIFAR-100 over three seeds, following the evaluation protocol of prior proxies. As discussed in Section 4.1, representation–label disagreement retains discriminative information even after instantaneous training loss has largely collapsed. We do not claim that representation-based scores universally outperform loss-based ones; rather, our contribution is to demonstrate that memorization is also reflected in representation-space dynamics.

  • Self-supervised learning: We acknowledge that our formulation relies on labels and class centroids. This is consistent with the FZ definition considered here, which is label-based, and the supervised setting of the proxies evaluated in our experiments. Extending RGD to self-supervised settings would require a notion of representation alignment that does not use ground-truth labels. We agree this is an important direction for future work, and we have now stated this limitation explicitly in the conclusion.

References

[1] A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou, “White-box vs Black-box: Bayes Optimal Strategies for Membership Inference,” in Proc. 36th International Conference on Machine Learning (ICML), PMLR, vol. 97, pp. 5558–5567, 2019. https://proceedings.mlr.press/v97/sablayrolles19a.html

[2] A. Suri, X. Zhang, and D. Evans, “Do Parameters Reveal More than Loss for Membership Inference?” Transactions on Machine Learning Research (TMLR), 2024. https://openreview.net/forum?id=fmKJfbGKFC

Q2:

We first note that ImageNet is not perfectly balanced, with training-class sizes ranging from 732 to 1,300 images [1], although this represents relatively mild imbalance.

To investigate more severe class imbalance, we conducted an additional experiment on CIFAR-100. We could not identify a severely imbalanced dataset with publicly available Feldman–Zhang (FZ) memorization estimates. Moreover, the existing FZ scores cannot be directly reused after modifying the training set, and obtaining new estimates requires training hundreds to thousands of models, making this computationally prohibitive.

We therefore consider mislabel detection as a related experimental setting. Mislabeled examples are typically atypical relative to their assigned classes, and fitting their incorrect labels often requires memorization. Feldman and Zhang [2] report that mislabeled examples are among examples with high memorization scores. This allows us to investigate whether RGD retains its representation-space signal under class imbalance using known label-corruption indicators.

Specifically, for each seed, we randomly select 8 superclasses (40 fine classes) from CIFAR-100 and retain 100, 50, or 20 training examples per minority class, out of the original 500, corresponding to 80%, 90%, and 96% reductions in the number of training examples per class, respectively. The remaining 60 classes retain all 500 examples. We use the original asymmetric label-noise protocol (nominally 20%) and the same training configuration as in our paper. The mislabel-detection results (AUROC, mean ± std over three seeds) are:

Samples/class (reduction)OverallMinorityMajority
500 (Balanced, 0%)0.9337 ± 0.0004——
100 (80%)0.9149 ± 0.00790.8945 ± 0.01480.9193 ± 0.0107
50 (90%)0.9132 ± 0.00640.8761 ± 0.01810.9181 ± 0.0078
20 (96%)0.9140 ± 0.01000.8474 ± 0.04230.9177 ± 0.0110

The balanced result corresponds to our original experiment, where minority and majority groups are not defined.

Although minority-class performance decreases as the number of training examples becomes smaller, RGD retains substantial discriminative ability even with only 20 examples per class (96% reduction), achieving an AUROC of 0.8474, while majority-class AUROC remains approximately 0.918 across all settings. One possible contributing factor is that class centroids estimated from very few samples may be less reliable. These results provide indirect evidence, through mislabel detection, that the representation-space signal exploited by RGD remains informative under severe class imbalance.

References

[1] O. Russakovsky et al., "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision (IJCV), 2015.

[2] V. Feldman and C. Zhang, "What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation," NeurIPS, 2020.

We hope that our responses and additional experimental results address your concerns, and we would appreciate it if you could reconsider your assessment in light of these clarifications.

Reviewer_bWFUOpenReview ↗

Imbalance

I thank the authors for the answer to my question.

Thank for the clarification regarding membership inference criterion, it may be worth discussing it in the paper.

I am satisfied by the answer provided by the authors to the data imbalance, although I feel like considering an imbalance dataset would be even more convincing.

I am still on the edge for this paper, since the importance of of the criterion and its potential impact seems a bit limited considering the massive adoption of self-supervised training.

Thank you for following up and for your suggestions.

MIA: We have clarified the distinction by adding brief definitions in the revised manuscript.

Self-supervised learning: We agree that extending memorization estimation beyond supervised image classification is important. To address this concern, we have added experiments in Section 4.4 covering both self-supervised language modeling and self-supervised pretrained visual representations.

First, we extend RGD to autoregressive language modeling, a self-supervised task requiring no manually annotated labels. We train a decoder-only Transformer from scratch on WikiBio texts using next-token prediction, treating target-token identities as classes for constructing representation centroids. RGD achieves a Pearson correlation of 0.710 ± 0.012 with subset-based counterfactual memorization estimates, outperforming trajectory loss (0.597 ± 0.008) and final loss (0.542 ± 0.011). This demonstrates that RGD can capture counterfactual memorization beyond supervised classification, using prediction targets derived directly from unannotated data. Importantly, RGD requires discrete prediction targets for constructing class centroids, not human-annotated labels; dependence on manual annotation is therefore not inherent to the approach.

Second, we evaluate RGD using DINOv2-S/14, a self-supervised pretrained Vision Transformer, fine-tuned for downstream classification. RGD achieves strong mislabel detection and label recovery performance, including 0.9851 AUROC and 0.8814 recovery accuracy on CIFAR-100 with 20% asymmetric label noise. These results demonstrate the applicability of RGD to models initialized through self-supervised pretraining, although the downstream evaluation uses labeled data. The new results broaden the demonstrated scope of RGD beyond conventional supervised image classification to self-supervised language modeling and modern pretrained vision architectures.

We hope that our responses and additional experimental results address your concerns, and we would appreciate it if you could reconsider your assessment in light of these clarifications.

Reviewer_MK9pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary: This paper proposes Representation Geometry Disagreement (RGD), a metric for sample-level memorization in classifiers without needing counterfactual training. RGD draws from the penultimate layer geometry of the classifier implied by Neural Collapse and is shown to be substantially correlated with the Feldman-Zhang counterfactual memorization score for CIFAR-10, CIFAR-100, and ImageNet. Additionally, it is shown that RGD is capable of detecting mislabeled samples by quantifying the atypicality of the mislabeled samples from their (incorrectly) assigned class in the representation space, and is also capable of identifying the correct class once a sample is mislabeled.

Contributions:

  1. The RGD metric and the methodology that utilizes neural collapse to derive the RGD.

  2. Empirical results comparing RGD with other metrics from prior works for (i) serving as surrogates for the FZ memorization score (ii) detecting mislabeling, and (iii) identifying the correct labels of mislabeled samples. Here RGD is shown to be competitive and even occasionally surpass the metrics chosen for comparison across these three tasks.

Validation:

  1. The central empirical claims of the paper relating RGD to FZ are reasonably validated by comparison with two measures of correlation. The paper also provides a somewhat light theoretical account for why RGD measures the FZ scores.

  2. The paper also validates its claims relating to mislabel detection and label recovery through extensive experimental evidence--however this would be additionally clear if more context for the measurements and details were provided (See below).

Critical strengths
  1. The insight for using the rigidity of penultimate layer geometry for measuring the typicality of an example is a novel and interesting conceptual advance. It provides a valuable tool for a large community of researchers studying the intersection of privacy, deep learning, and theory.

  2. The clearest evidence for using RGD is empirical, and demonstrated via Table 1, where, on ImageNet RGD seems to outperform the other metrics chosen for comparison.

  3. The application to mislabel ID and label identification is interesting and the empirical validation is substantive.

Critical weaknesses
  1. The theoretical account given in Remark 2 for why RGD approximates memorization is rather perfunctory. In particular, the justification hinges on arguing why ρiδi\rho_i \delta_i is small. While the authors make a somewhat plausible justification in the regime where ρi\rho_i is small, the reasoning for the regime where ρi\rho_i is large is not convincing (see Lines 273-274). I would suggest the authors give a more rigorous justification (along with any additional relevant assumptions required) for this line of argument, or seek a different way to substantiate why their bound is tight. For instance, δi\delta_i can in fact be computed from the FZ memorization scores--in particular, by adding and subtracting the trained model's classification probabilities Pf\textleftarrowA(S)[f(xi)=yi]P_{f \textleftarrow A(S)}[f(x_i) = y_i] we get:

δi=P[Y=yi∣h(t)(xi)]−Pf\textleftarrowA(S∖i)[f(xi)=yi]=P[Y=yi∣h(t)(xi)]−Pf\textleftarrowA(S)[f(xi)=yi]+mem(A,S,i).\delta_i = P[Y = y_i \mid h^{(t)}(x_i)] - P_{f \textleftarrow A(S \setminus i)}[f(x_i) = y_i] = P[Y = y_i \mid h^{(t)}(x_i)] - P_{f \textleftarrow A(S)} [f(x_i) = y_i] + mem(A, S, i).

Thus it seems that δi\delta_i can be measured using the experimental data considered in the paper, and thus showing empirically that ρiδi\rho_i \delta_i is small (or roughly bounded) would really tighten up this plausibility argument offered in Remark 2. There are additional minor issues with the theoretical substantiation:

(a) In line 235 the authors claim to use pown(t)(xi)p_{own}^{(t)}(x_i) as a proxy for the LOO prediction probabilities, but there are no assumptions on the size of δi\delta_i that make pown(t)(xi)p^{(t)}_{\text{own}}(x_i) a good approximator. In this sense δi\delta_i is just good notation and not a structural assumption, and the authors should make this clear. If the authors did intend to put additional assumptions on δi\delta_i they should clarify this.

(b) In line 207 it is written that the threshold ϵ\epsilon for defining TiT_i is fixed. However, based on Remark 2 line 270, it is said that a choice of κ\kappa implies a choice of ϵ\epsilon, which seems inaccurate because κ\kappa and ϵ\epsilon are free hyperparameters set before training since κ\kappa only controls how quickly the si(t)s^{(t)}_i converges to zero, not the threshold below which it drops, which is ϵ\epsilon. This is true unless the user also prescribes a target value of TiT_i (which I don't believe was the authors' intention).

  1. The main construction of the paper involves the probabilities constructed in Eqn. (4) based on the penultimate layer class means. However, when considering neural collapse, due to (NC3, self duality), the weight vectors of the terminal layer themselves collapse to the penultimate layer class means μc\mu_c, and due to (NC2) the penultimate class means collapse to the vertices of an ETF (or a spherical code in the overcomplete case [1]). Applying NC4, the bias vector collapses to WhGWh_G (see Fig 6, in [2]), where hGh_G is the average of the penultimate features and due to (NC2), this is zero so the bias vector is γ1d\gamma 1_d for some γ≥0\gamma \geq 0. This seems to imply that:

pown(t)(xi)→Pf\textleftarrowA(S)[f(xi)=yi]p_{\text{own}}^{(t)}(x_i) \to P_{f \textleftarrow A(S)}[f(x_i) = y_i]

Of course, one may argue that this is just an asymptotic result and only analytically possible for idealized models or recentered μc\mu_c's. Nonetheless it does beg the question for why it is necessary (and not just sufficient) to measure memorization by defining RGD using pown(t)(xi)p^{(t)}_{\text{own}}(x_i) instead of model's own softmax probabilities at time t given by:

P(t)[f(xi)=yi]:=softmax(W(t)h(t)+b(t))[i]P^{(t)}[f(x_i) = y_i] := softmax \left( W^{(t)}h^{(t)} + b^{(t)} \right)[i]

Currently, I believe the paper makes a conceptual advance and provides empirical validation, but I believe that addressing the points (1) and (2) above would introduce a grounding and necessity to the main object of the paper, meriting acceptance.

References:

[1] Jiang, J., Zhou, J., Wang, P., Qu, Q., Mixon, D.G., You, C., Zhu, Z. (2024). Generalized Neural Collapse for a Large Number of Classes. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:22010-22041, 2024.

[2] A Geometric Analysis of Neural Collapse with Unconstrained Features. Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, Qing Qu. Advances in Neural Information Processing Systems 34 (NeurIPS 2021).

Questions for authors
  1. Could the authors further clarify their account for the size of ρiδi\rho_i \delta_i in Remark 2 either empirically or theoretically? For example, they may estimate the size of δi\delta_i using the FZ scores and their own values for ρi,P[Y=yi∣h(t)(xi)],Pf\textleftarrowA(S)[f(xi)=yi]\rho_i, P[Y = y_i \mid h^{(t)}(x_i)], P_{f \textleftarrow A(S)} [f(x_i) = y_i].

  2. On line 206, the authors introduce this interesting quantity TiT_i that indicates onset of alignment with class centroids, but make little note of it in the empirical results. How does TiT_i scale with ϵ\epsilon? How does TiT_i differ over samples?

  3. Could the authors clarify why it it is necessary to measure memorization with the penultimate layer probabilities instead of the model's own softmax logits (see Item 2 in the "Weaknesses" section)? For instance, suppose RGD is defined instead by averaging out the model softmax probability for training sample xix_i on the class yiy_i (i.e the class probability for yiy_i assigned by the model to xix_i) over all epochs. Does this quantity also correlate heavily with the FZ scores and/or depart from the RGD proposed in the paper?

5 comments

We thank the reviewer for their time and suggestions.

Q1 - Theoretical justification and related weaknesses

We agree that the original discussion in Remark 2 did not adequately justify the magnitude of the estimation term, particularly when ρi\rho_i is large.

We have revised Assumption 3 and Proposition 1 to account for epoch-dependent estimation errors. Specifically, we define

δi(t)=Pr⁡f←A(S∖i)[f(xi)=yi]−Pr⁡[Y=yi∣h(t)(xi)].\delta_i^{(t)} = \Pr_{f\leftarrow A(S\setminus i)}[f(x_i)=y_i] - \Pr[Y=y_i\mid h^{(t)}(x_i)].

We also account for the fixed-temperature approximation. Under Assumption 2, the vMF posterior corresponds to an epoch-dependent temperature τt=1/κt\tau_t=1/\kappa_t, whereas our method uses a fixed temperature. We define the resulting approximation error as

ei(t)=Pr⁡[Y=yi∣h(t)(xi)]−pown(t)(xi).e_i^{(t)} = \Pr[Y=y_i\mid h^{(t)}(x_i)] - p_{\mathrm{own}}^{(t)}(x_i).

The combined estimation discrepancy is therefore

δ^i(t)=δi(t)+ei(t),\widehat{\delta}_i^{(t)} = \delta_i^{(t)}+e_i^{(t)},

and the absolute value of its mean over representation-unfit epochs is

Ai=∣1Ti−1∑t<Tiδ^i(t)∣.A_i = \left| \frac{1}{T_i-1} \sum_{t\lt T_i}\widehat{\delta}_i^{(t)} \right|.

The revised Proposition 1 establishes the sandwich bound

ρi(mem(A,S,i)−Ai)≤RGDi≤ρi(mem(A,S,i)+Ai)+(1−ρi)ε.\rho_i\big(\mathrm{mem}(A,S,i)-A_i\big) \leq \mathrm{RGD}_i \leq \rho_i\big(\mathrm{mem}(A,S,i)+A_i\big) +(1-\rho_i)\varepsilon.

Unlike the original formulation, the revised bound uses the absolute value of the mean epoch-dependent combined estimation error, rather than a single per-example error bound applying to all unfit epochs. As the reviewer correctly points out in weakness (a), defining the estimation error alone does not guarantee its smallness. We have clarified the qualitative premise of Assumption 3 without imposing a bound on the size of the estimation error. We have also removed the previous argument that a large ρi\rho_i necessarily implies a small estimation error. The revised proposition establishes a conditional relationship between RGD and FZ memorization, with the combined estimation discrepancy explicitly accounted for in the bound.

Following the reviewer's suggestion, we empirically examine the magnitude of this combined discrepancy using the published FZ memorization scores. The empirical distributions of δ^i(t)\widehat{\delta}_i^{(t)} and AiA_i are presented in Appendix E.

On CIFAR-100 (ResNet-18, seed 0), across ε∈0.01,0.02,0.05,0.1,0.2\varepsilon\in\\{0.01,0.02,0.05,0.1,0.2\\}, the median ρiAi\rho_i A_i ranges from 0.061 to 0.067, while the median resulting bound ranges from 0.070 to 0.196. Accordingly, the revised text does not rely on this term being negligible. Furthermore, RGD exhibits a strong empirical association with ρi,mem(A,S,i)\rho_i\\,\mathrm{mem}(A,S,i), with Pearson correlations of 0.804–0.823 and Spearman correlations of 0.861–0.863.

Regarding weakness (b), we agree that the temperature and the fitting threshold ε\varepsilon are independently chosen parameters. Our previous statement was intended to convey that a smaller temperature sharpens the centroid-induced softmax and makes si(t)s_i^{(t)} closer to zero when a sample is well aligned with its assigned-class centroid, allowing a smaller fitting threshold ε\varepsilon to be chosen, rather than determining its value. To avoid this ambiguity, we have removed the statement from Remark 1.

The revised theoretical formulation is presented in Section 3.2, with the empirical analysis in Appendix E.

Q2 - Representation-fitting epoch

We have added an empirical analysis in Appendix E.1, examining the dependence of TiT_i on ε\varepsilon and its relationship with memorization.

We evaluate ε∈0.01,0.02,0.05,0.1,0.2\varepsilon\in\\{0.01,0.02,0.05,0.1,0.2\\} on CIFAR-100 (ResNet-18, seed 0). By definition, TiT_i is non-increasing with ε\varepsilon, as a larger threshold allows examples to satisfy the representation-fitting criterion earlier.

Importantly, memorized examples consistently exhibit larger fitting times. At ε=0.1\varepsilon=0.1, the mean TiT_i is 134.4 for memorized examples (FZ > 0.25), compared with 89.3 for non-memorized examples. The full distributions of TiT_i across different thresholds and memorization levels are provided in Appendix E.1.

Q3 - Representation-space vs. classifier-softmax disagreement

Our motivation for using representation-space probabilities is to capture class-relative geometric information independently of the classifier head, rather than to claim that classifier-softmax probabilities cannot serve as memorization proxies.

Why representation-space disagreement? Let qyiq_{y_i} denote the model's softmax probability for the assigned label and ℓi=−log⁡qyi\ell_i=-\log q_{y_i} its per-example cross-entropy loss. The corresponding classifier-softmax disagreement is therefore 1−qyi=1−e−ℓi1-q_{y_i}=1-e^{-\ell_i}, a direct transformation of the loss. As the model confidently fits training examples, this instantaneous signal approaches zero. In contrast, RGD measures class-relative representation geometry independently of the classifier head, allowing memorization-related differences to remain observable after the loss has largely collapsed. This is precisely the motivation discussed in Section 4.1 and illustrated in Figure 1.

Regarding Neural Collapse, NC3 relates classifier weights to centered class means, while NC4 establishes an asymptotic connection between classifier and nearest-class-center decisions. However, equivalence of classification decisions does not imply equality of softmax probabilities. Classifier logits depend on learned weights, biases, and feature magnitudes, whereas RGD uses cosine-normalized similarities and a fixed temperature. Moreover, Neural Collapse does not establish equivalence throughout training.

We evaluated the proposed trajectory-averaged classifier-softmax disagreement on CIFAR-100 (200 epochs, seed 0). It achieves cosine similarity (CS)/Pearson correlation of 0.8743/0.8182 with FZ memorization, compared with 0.8588/0.7604 for RGD, confirming that trajectory-averaged classifier-softmax disagreement is also a strong memorization proxy.

Additional experimental evidence. Our per-epoch analysis (Appendix F, Figures 7 and 8) further illustrates this distinction: the correlation between classifier-softmax disagreement and FZ deteriorates substantially as training progresses, whereas RGD retains stronger agreement during later epochs. This supports our motivation for investigating representation-space dynamics, even though the trajectory-averaged softmax baseline achieves higher overall agreement in this CIFAR-100 experiment.

Thus, while classifier-softmax disagreement, like other loss-derived proxies, can effectively estimate memorization when aggregated over training, RGD captures a distinct geometric signal that remains more informative about FZ memorization during late training.

We hope that our responses and additional experimental results address your concerns, and we would appreciate it if you could reconsider your assessment in light of these clarifications.

Thanks for addressing these items raised in my comment above. I want to draw attention to the following findings from your work:

"We evaluated the proposed trajectory-averaged classifier-softmax disagreement on CIFAR-100 (200 epochs, seed 0). It achieves cosine similarity (CS)/Pearson correlation of 0.8743/0.8182 with FZ memorization, compared with 0.8588/0.7604 for RGD, confirming that trajectory-averaged classifier-softmax disagreement is also a strong memorization proxy."

I have two comments/questions regarding this:

(a) Could you also report the trajectory-averaged classifier-softmax disagreement also be computed on imagenet, and if not, please specify the reasons why.

(b) Since apparently the trajectory-averaged classifier-softmax disagreement baseline outpefroms the RGD, i think this is point worth highlighting in the main text instead of a small line in the appendix. What you have shown is that the classifier's own logits are in fact a better proxy for the FZ memorization, as long as you average over the trajectory. The per-epoch analysis is nonetheless informative as it serves to motivate the other results from the paper. Thus, given the results from Appendix E, I would suggest that you (i) also introduce the trajectory-averaged classifier-softmax disagreement in the main text (or introduce it in the appendix and point to it from the main text), and (ii) highlight the fact that this is the one baseline that performs better than your proposed metric (at least for CIFAR-100) for measuring the FZ scores, but that you choose to focus on the representational version because of its other benefits in understanding the mislabeling experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.