- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares two categories of memorization measures: verbatim memorization and counterfactual memorization. Verbatim memorization captures how easily a sample can be generated as it is by the model, while counterfactual memorization captures how much a sample's inclusion in the training data changes its generation by the model. While both measures target memorization in LLM, they are fundamentally different. If we were to suppress memorization, we could for sure reduce verbatim memorization. But this can inadvertantly hurt generalization.
The paper therefore focuses on relating memorization with generalization in a theoretical setup, which is later verified using experiments. The authors find that two memorization measures capture fundamentally different properties of the training data distribution: rare or long-tail samples that are dissimilar to most training samples tend to have higher counterfactual memorization but lower verbatim memorization, whereas samples surrounded by many similar training samples exhibit the opposite behavior. Their analysis therefore suggests that all forms of memorization in LLM are not necessarily harmful: counterfactual memorization is good for generalization, while verbatim memorization is closely related to training data leakage. This further motivates a training data selection strategy based on memorization score.
Critical strengths
The critical strength is the theoretical analysis to compare two different categories of memorization and a furhter demonstration on training data selection.
Critical weaknesses
The paper rediscovers concepts already known to the community. Counterfactual memorization is originally proposed to focus on rare strings, which are rarely caught when applied to verbatim memorization. The authors carefully disregard this central piece of information and rediscover the relation between the two measures. While I appreciate the theoretical connection, the contribution of the paper is overly stated.
If we agree that the paper is rediscovering old concepts, all the empirical results look non-surprising. For example, it is well anticipated that the memorization scores of the two measures won't be correlated. Also, counterfactual memorization is more impactful if a sample is rare, and rare samples are good for generalization—therefore, training data selection based on such a score would definitely help. While I do not see any mistakes in the theory of the paper, such a theory only confirms what is expected and adds nothing new.
I would request the authors to compare with recent works (https://arxiv.org/pdf/2507.14777) exactly focusing on the topic.
Questions for authors
What is the rationale behind considering in-context learning theory to study memorization, which involves training? In this setup, the input outputs are real numbers. How suitable are they in LLMs where the outputs are usually discretized tokens?
6 comments
Rebuttal (Part 2/2)
W3: Compare with the work at arXiv:2507.14777.
Response: We thank the reviewer for pointing out Ghosh et al. [r2]. Their work primarily investigates the distinction between memorization and learning in LLMs, introducing contextual memorization as a refined counterfactual-based measure. Their main finding, that some memorization is unavoidable for optimal learning, extends the classical insights of Feldman et al. [r6] to the LLM setting.
In contrast, our work focuses on a different question: the relationship between CF-Mem and VB-Mem. We theoretically characterize their inverse relationship and leverage it to mitigate verbatim memorization while preserving model utility through data selection.
Q1: Why use ICL theory to study memorization during training, and how does it relate to discrete-token LLMs?
Response: Thanks for raising this question. We use ICL theory because it enables tractable explanations of LLM behavior. Previous studies [r3], [r4], [r5] derive predictions from simplified ICL frameworks and support them with experiments on real-world LLMs. Following this approach, our framework explicitly trains model parameters (Eq. (4)) and evaluates CF-Mem by comparing models trained with and without a given task.
ICL also provides an abstraction of token-based language modeling. As discussed in Section 4.2 of Fu et al. [r4], tokens can be represented as feature vectors through one-hot encoding, while next-token targets play the role of prediction labels. This motivates an analogy between token sequences and ICL input–label pairs. Our framework uses real-valued input vectors and discrete binary labels .
While this abstraction does not capture the full complexity of autoregressive language modeling, it enables tractable theoretical analysis. We further observe the predicted negative CF-Mem–VB-Mem relationship in real-world LLMs (Figure 1).
References:
[r1] Zhang et al. "Counterfactual Memorization in Neural Language Models." NeurIPS 2023.
[r2] Ghosh et al. "Rethinking Memorization Measures and their Implications in Large Language Models." NeurIPS (ED Track) 2026.
[r3] Shi et al. "Why Larger Language Models Do In-context Learning Differently?" ICML 2024.
[r4] Fu et al. "Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence." NeurIPS 2025.
[r5] Fu et al. "Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theory." ICLR 2026.
[r6] Feldman et al. "Does Learning Require Memorization? A Short Tale about a Long Tail." arXiv 2019.
Thanks for the rebuttal. I have some questions related to your discussion.
-
How can I intuitively understand the negative relation between counterfactual and verbatim memorization? Is it a universal phenomenon? Why did [r1] miss that? What additional assumptions did you make? [r2] empirically show that during the training dynamics, the order of memorization of different strings can be negatively correlated when applying these two measures -- it may give you empirical support.
-
You often talk about negative correlation. Are there any empirical results? Figure 1 shows a negative correlation, but not exactly -1.
-
How much of your analysis depends on the ICL framework? Does any other data give you a similar result?
Rebuttal (Part 1/2)
Thank you for your thorough review and constructive comments. All your concerns have been carefully responded below. The manuscript is carefully revised accordingly. We sincerely hope our responses fully address your questions.
W1: About the novelty of the CF-Mem–VB-Mem relationship.
Response: Thanks for the comment. We acknowledge that Zhang et al. [r1] observed that CF-Mem favors rare samples, whereas generation-time memorization is sensitive to duplication. However, Appendix A of [r1] explicitly notes that these two notions are not necessarily opposites. Thus, their different responses to duplication do not by themselves establish a direct inverse relationship between the two scores.
Our contribution goes beyond these observations by (1) deriving an explicit negative relationship under our linear-attention ICL framework (Theorems 4.2–4.3 and Corollary 4.4), (2) connecting memorization to generalization through a risk bound (Theorem 4.5), and (3) empirically demonstrating that CF-Mem-guided data selection reduces verbatim leakage while maintaining utility comparable to Random selection.
We have acknowledged the prior observations of [r1] in the Related Work section in our revised paper.
W2.1: It is well anticipated that the memorization scores of the two measures won't be correlated.
Response: We respectfully argue that our contribution is to establish a negative correlation between CF-Mem and VB-Mem, rather than simply report that the two scores are uncorrelated. Theorems 4.2–4.3 and Corollary 4.4 derive an explicit negative relationship under our ICL framework, and Figure 1 reports negative Spearman correlations across all three evaluated LLM settings. The observed direction—higher CF-Mem tending to accompany lower VB-Mem—also motivates our CF-Mem-guided selection of training data with lower verbatim leakage.
W2.2: Rare examples are useful, so CF-Mem selection would definitely help.
Response: We clarify that our objective is to reduce verbatim leakage while preserving model performance. The usefulness of rare samples for generalization alone does not establish that CF-Mem can achieve both objectives. Our experiments show that, at the same 40% token budget, CF-Mem maintains average performance comparable to Random (58.53% vs 58.00% on OpenThoughts and 53.07% vs 53.21% on Mixture-of-Thoughts), while lowering the top-0.5% VB-Mem thresholds from 46.77% to 25.83% and from 5.64% to 2.90%, respectively (See Tables 1-2 in our paper).