acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual and Verbatim Memorization Disagree: From In-context Learning Theory to LLM Data Selection

Abstract

Large language models (LLMs) can memorize their training data, but whether such memorization is desirable remains an open question. Some forms of memorization are associated with verbatim training-data leakage, while others capture the contribution of individual training samples to model generalization. This creates an apparent dilemma: suppressing memorization could reduce verbatim training-data leakage, but may also hurt model utility. This paper shows that the dilemma is largely metric-dependent. We study two notions of LLM memorization: counterfactual memorization (CF-Mem), which characterizes how model behavior changes when a sample is included in or excluded from training, and verbatim memorization (VB-Mem), which measures how easily training content can be reproduced verbatim from an LLM. Through an in-context learning (ICL)-theoretic analysis, we prove that these two notions are driven by distinct mechanisms and can be inversely related: samples with higher CF-Mem tend to exhibit lower VB-Mem, while CF-Mem is positively associated with LLM generalization. This disagreement suggests that not all forms of memorization are harmful. Motivated by these findings, we propose a CF-Mem-guided LLM training data selection method. Our method efficiently estimates CF-Mem using a small proxy model and allocates more token budget to samples with higher CF-Mem scores. This yields a smaller training set that preserves samples important for generalization while reducing those associated with high VB-Mem. Across reasoning benchmarks, our method substantially reduces verbatim extraction risk and training tokens while preserving most of the full-data fine-tuning performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Memorization, Data selection, Training efficiency, Copyright

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cdofOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares two categories of memorization measures: verbatim memorization and counterfactual memorization. Verbatim memorization captures how easily a sample can be generated as it is by the model, while counterfactual memorization captures how much a sample's inclusion in the training data changes its generation by the model. While both measures target memorization in LLM, they are fundamentally different. If we were to suppress memorization, we could for sure reduce verbatim memorization. But this can inadvertantly hurt generalization.

The paper therefore focuses on relating memorization with generalization in a theoretical setup, which is later verified using experiments. The authors find that two memorization measures capture fundamentally different properties of the training data distribution: rare or long-tail samples that are dissimilar to most training samples tend to have higher counterfactual memorization but lower verbatim memorization, whereas samples surrounded by many similar training samples exhibit the opposite behavior. Their analysis therefore suggests that all forms of memorization in LLM are not necessarily harmful: counterfactual memorization is good for generalization, while verbatim memorization is closely related to training data leakage. This further motivates a training data selection strategy based on memorization score.

Critical strengths

The critical strength is the theoretical analysis to compare two different categories of memorization and a furhter demonstration on training data selection.

Critical weaknesses

The paper rediscovers concepts already known to the community. Counterfactual memorization is originally proposed to focus on rare strings, which are rarely caught when applied to verbatim memorization. The authors carefully disregard this central piece of information and rediscover the relation between the two measures. While I appreciate the theoretical connection, the contribution of the paper is overly stated.

If we agree that the paper is rediscovering old concepts, all the empirical results look non-surprising. For example, it is well anticipated that the memorization scores of the two measures won't be correlated. Also, counterfactual memorization is more impactful if a sample is rare, and rare samples are good for generalization—therefore, training data selection based on such a score would definitely help. While I do not see any mistakes in the theory of the paper, such a theory only confirms what is expected and adds nothing new.

I would request the authors to compare with recent works (https://arxiv.org/pdf/2507.14777) exactly focusing on the topic.

Questions for authors

What is the rationale behind considering in-context learning theory to study memorization, which involves training? In this setup, the input outputs are real numbers. How suitable are they in LLMs where the outputs are usually discretized tokens?

6 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

Thank you for your thorough review and constructive comments. All your concerns have been carefully responded below. The manuscript is carefully revised accordingly. We sincerely hope our responses fully address your questions.

W1: About the novelty of the CF-Mem–VB-Mem relationship.

Response: Thanks for the comment. We acknowledge that Zhang et al. [r1] observed that CF-Mem favors rare samples, whereas generation-time memorization is sensitive to duplication. However, Appendix A of [r1] explicitly notes that these two notions are not necessarily opposites. Thus, their different responses to duplication do not by themselves establish a direct inverse relationship between the two scores.

Our contribution goes beyond these observations by (1) deriving an explicit negative relationship under our linear-attention ICL framework (Theorems 4.2–4.3 and Corollary 4.4), (2) connecting memorization to generalization through a risk bound (Theorem 4.5), and (3) empirically demonstrating that CF-Mem-guided data selection reduces verbatim leakage while maintaining utility comparable to Random selection.

We have acknowledged the prior observations of [r1] in the Related Work section in our revised paper.

W2.1: It is well anticipated that the memorization scores of the two measures won't be correlated.

Response: We respectfully argue that our contribution is to establish a negative correlation between CF-Mem and VB-Mem, rather than simply report that the two scores are uncorrelated. Theorems 4.2–4.3 and Corollary 4.4 derive an explicit negative relationship under our ICL framework, and Figure 1 reports negative Spearman correlations across all three evaluated LLM settings. The observed direction—higher CF-Mem tending to accompany lower VB-Mem—also motivates our CF-Mem-guided selection of training data with lower verbatim leakage.

W2.2: Rare examples are useful, so CF-Mem selection would definitely help.

Response: We clarify that our objective is to reduce verbatim leakage while preserving model performance. The usefulness of rare samples for generalization alone does not establish that CF-Mem can achieve both objectives. Our experiments show that, at the same 40% token budget, CF-Mem maintains average performance comparable to Random (58.53% vs 58.00% on OpenThoughts and 53.07% vs 53.21% on Mixture-of-Thoughts), while lowering the top-0.5% VB-Mem thresholds from 46.77% to 25.83% and from 5.64% to 2.90%, respectively (See Tables 1-2 in our paper).

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

W3: Compare with the work at arXiv:2507.14777.

Response: We thank the reviewer for pointing out Ghosh et al. [r2]. Their work primarily investigates the distinction between memorization and learning in LLMs, introducing contextual memorization as a refined counterfactual-based measure. Their main finding, that some memorization is unavoidable for optimal learning, extends the classical insights of Feldman et al. [r6] to the LLM setting.

In contrast, our work focuses on a different question: the relationship between CF-Mem and VB-Mem. We theoretically characterize their inverse relationship and leverage it to mitigate verbatim memorization while preserving model utility through data selection.

Q1: Why use ICL theory to study memorization during training, and how does it relate to discrete-token LLMs?

Response: Thanks for raising this question. We use ICL theory because it enables tractable explanations of LLM behavior. Previous studies [r3], [r4], [r5] derive predictions from simplified ICL frameworks and support them with experiments on real-world LLMs. Following this approach, our framework explicitly trains model parameters (Eq. (4)) and evaluates CF-Mem by comparing models trained with and without a given task.

ICL also provides an abstraction of token-based language modeling. As discussed in Section 4.2 of Fu et al. [r4], tokens can be represented as feature vectors through one-hot encoding, while next-token targets play the role of prediction labels. This motivates an analogy between token sequences and ICL input–label pairs. Our framework uses real-valued input vectors and discrete binary labels y∈{−1,+1}y\in\{-1,+1\}.

While this abstraction does not capture the full complexity of autoregressive language modeling, it enables tractable theoretical analysis. We further observe the predicted negative CF-Mem–VB-Mem relationship in real-world LLMs (Figure 1).

References:

[r1] Zhang et al. "Counterfactual Memorization in Neural Language Models." NeurIPS 2023.

[r2] Ghosh et al. "Rethinking Memorization Measures and their Implications in Large Language Models." NeurIPS (ED Track) 2026.

[r3] Shi et al. "Why Larger Language Models Do In-context Learning Differently?" ICML 2024.

[r4] Fu et al. "Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence." NeurIPS 2025.

[r5] Fu et al. "Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theory." ICLR 2026.

[r6] Feldman et al. "Does Learning Require Memorization? A Short Tale about a Long Tail." arXiv 2019.

Thanks for the rebuttal. I have some questions related to your discussion.

  • How can I intuitively understand the negative relation between counterfactual and verbatim memorization? Is it a universal phenomenon? Why did [r1] miss that? What additional assumptions did you make? [r2] empirically show that during the training dynamics, the order of memorization of different strings can be negatively correlated when applying these two measures -- it may give you empirical support.

  • You often talk about negative correlation. Are there any empirical results? Figure 1 shows a negative correlation, but not exactly -1.

  • How much of your analysis depends on the ICL framework? Does any other data give you a similar result?

Reviewer_46deOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper characterizes LLM memorization as counterfactual memorization and verbatim memorization. The authors provide a theoretical analysis showing that they capture different properties of the training-data distribution. Based on the theoretical insights, the authors further propose a CF-Mem-guided training-data selection strategy to reduce verbatim extraction risks while preserving model utility.

Critical strengths
  • The problem (verbatim memorization risks) that the authors try to address is real and significant. The authors analyze the problem and propose an interesting solution.
  • The authors offer an analysis of the two types of memorization in LLMs and give an interesting insight into using memorization for data selection.
Critical weaknesses
  • The authors make a strong assumption about uniformly distributed test-task directions. Higher average CF-Mem reflects greater diversity within the training set, but does not necessarily imply better alignment with the test distribution. For example, if test tasks are concentrated along one direction, selecting high-CF-Mem samples may reduce test-relevant coverage.
  • The empirical results seem a bit weak. The reported Spearman correlation of -0.073, with the rest around -0.2, is not a strong signal of a correlation. Meanwhile, the performance of the proposed method is quite close to that of Random across most of the datasets (except for a few, like AIME25). In addition, the authors only compare their method with Random and Longest. At minimum, comparisons with other recent data selection methods should be included.
  • The authors did not provide an empirical comparison regarding the efficiency of the proposed method and the baseline methods.
Questions for authors
  • From my understanding, a more concentrated training direction would produce a lower CF-Mem. By choosing a higher average CF-Mem, does it mean that these concentrated task directions are less likely to be selected?
  • Can the authors enhance the current empirical results, such as by considering other baseline methods?
  • Can the authors further test the verbatim memorization risks under adversarial settings, such as when extraction attacks are applied?
5 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

Thank you for your thorough review and constructive comments. All your concerns have been carefully responded below. The manuscript is carefully revised accordingly. We sincerely hope our responses fully address your questions.

W1&Q1: About the test distribution.

Response: Our analysis focuses on generalization across a broad range of tasks, consistent with the general capabilities expected of LLMs, including diverse tasks within a target domain. The uniform test task distribution is a deliberate idealization of this objective: it does not privilege a particular task direction and allows us to study how training data coverage affects generalization to unseen tasks (Section 4.3). In this setting, prioritizing high-CF-Mem examples favors less redundant task directions and encourages broad task coverage. A test distribution strongly dominated by a few task directions represents a different generalization objective, and we leave its analysis for future studies.

W2.1: About the empirical correlation.

Response: The purpose of Figure 1 is to examine whether the negative relationship between CF-Mem and VB-Mem suggested by our theoretical analysis also appears empirically in LLMs. Although the Spearman correlations are relatively modest, they are consistently negative across all three settings, providing directional evidence for this relationship.

Importantly, our data selection experiments provide additional empirical evidence consistent with this negative relationship. As shown in Section 6.2, prioritizing high-CF-Mem samples consistently reduces VB-Mem compared with Random selection under the same token budget. For example, the top-0.5% VB-Mem thresholds decrease from 46.77% to 25.83% on OpenThoughts and from 5.64% to 2.90% on Mixture-of-Thoughts, while maintaining comparable model utility. These results further support the practical relevance of the theoretically predicted relationship, beyond the sample-wise correlations in Figure 1.

W2.2: About performance compared with Random.

Response: Our goal is to maintain performance comparable to Random while substantially reducing verbatim leakage risk. This is the intended benefit of CF-Mem-guided selection: under the same 40% token budget, average performance remains close (58.53% vs 58.00% on OpenThoughts and 53.07% vs 53.21% on Mixture-of-Thoughts), while the top-0.5% VB-Mem thresholds decrease from 46.77% to 25.83% and from 5.64% to 2.90%, respectively (see Tables 1-2 in our paper).

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

W2.3&Q2: Additional data-selection baselines.

Response: Thanks for the suggestion. We adopted a new baseline selection method, global average log probability (GALP) [r1], at a 40% token budget on both Qwen2.5-7B datasets. The results are shown in the following Table r1 and Table r2:

Training setSelectionGSM8KMATH-500AMC23AIME25 (Pass@8)AIME26 (Pass@8)Average
OpenThoughts-114k (50k)CF-Mem (40%)89.6274.551.884036.6758.53
OpenThoughts-114k (50k)GALP (40%)90.2776.953.7536.6736.6758.85
Mixture-of-Thoughts (50k)CF-Mem (40%)88.7772.650.6226.6726.6753.07
Mixture-of-Thoughts (50k)GALP (40%)89.77552.81303055.50

Table r1: Performance (%) on Qwen2.5-7B under different data selection methods (40% budget).

Training setSelectionTop 0.1%Top 0.2%Top 0.5%Top 1%Top 2%
OpenThoughts-114k (50k)CF-Mem (40%)94.2476.5625.834.600.99
OpenThoughts-114k (50k)GALP (40%)99.4198.1483.1942.977.26
Mixture-of-Thoughts (50k)CF-Mem (40%)60.6823.002.900.480.06
Mixture-of-Thoughts (50k)GALP (40%)98.2181.5625.464.890.58

Table r2: VB-Mem auditing (%) on Qwen2.5-7B under different data selection methods (40% budget).

CF-Mem achieves close average performance to GALP while yielding substantially lower VB-Mem thresholds at all five quantiles on both datasets. For example, the top-0.5% threshold is 25.83% for CF-Mem versus 83.19% for GALP on OpenThoughts, and 2.90% versus 25.46% on Mixture-of-Thoughts.

We have added these results in our revised paper. Please see Section 6.2 for more details.

W3: No empirical efficiency comparison.

Response: Thanks for the suggestion. We report the training cost of Qwen2.5-7B on both datasets, measured in hours from our training logs, as shown in the following Table r3.

Training setFullRandom (40%)Longest (40%)GALP (40%)CF-Mem (40%)
OpenThoughts-114k (50k)9.052.523.693.571.85
Mixture-of-Thoughts (50k)5.473.042.972.991.7

Table r3: Training time (hours) on Qwen2.5-7B under different data selection methods.

CF-Mem has the lowest reported training cost among these methods on both datasets. Compared with Full, training time is reduced by roughly 79% on OpenThoughts and 68% on Mixture-of-Thoughts; compared with Random, the reductions are 26% and 44%.

We have added these results in our revised paper. Please see Section 6.2 for more details.

Q3: Evaluate verbatim memorization under adversarial extraction attacks.

Response: Thanks for the suggestion. We use probabilistic extraction (PE) [r2] for verbatim leakage auditing mainly because its effectiveness has been empirically validated and can audit efficiently at scale. Adversarial extraction involves additional prompt search and repeated generation, making it more computationally costly. We will add an adversarial extraction analysis in the final version, comparing CF-Mem and Random under matched query budgets.

References:

[r1] Zhang et al. "The Best Instruction-Tuning Data are Those That Fit." NeurIPS 2025.

[r2] Hayes et al. "Measuring memorization in language models via probabilistic extraction." NAACL 2025.

Dear Reviewer 46de,

Thank you again for your valuable comments! We have carefully considered your comments and have provided our responses.

Please let us know if our replies have satisfactorily addressed your concerns. Please do not hesitate to let us know if you have any further questions or if you require any additional clarification.

Thank you very much!

Reviewer_PSUvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates the relationship between counterfactual and verbatim memorization. They start with a theoretical model derived from ICL theory, and also propose a training algorithm that relies on removing examples with lower counterfactual memorization, in order to reduce verbatim memorization while maintaining performance.

Critical strengths

I think the theoretical model is interesting. I haven't seen many good theoretical models that capture the behavior we see in memorization.

Studying CF Mem in SFT is novel - most of the prior work looks at pretraining.

Critical weaknesses

Unfortunately, I think the biggest weakness of the paper is the limited novel empirical contribution. The cited literature already contains empirical evidence of the following claims which the authors consider contributions: 1) verbatim and counterfactual memorization measure different things (Zhang et al shows this), 2) removing examples with high verbatim memorization doesn't hurt learning (this is a highly correlated conclusion to Lee et al - deduplicating training data makes LMs better), 3) examples with high CF mem are useful for generalization (Feldman and Zhang and Zhang et al have several "counterfactual influence" experiments that show a version of this, but I concede that this is a less strongly substantiated claim relative to 1 + 2).

The empirical results are not very convincing and make a variety of seemingly arbitrary choices. This overall makes me quite suspicious of the paper's findings. Here are things that seem pretty arbitrary/weakly justified to me: 1) 40% token budget, 2) the CF mem sampling "schedule" phi, 3) the use of the proxy model. Also, the only strong finding overall is that verbatim memorization decreases, and accuracy even decreases somewhat in a way that doesn't seem supported by the theory.

Questions for authors

I'd appreciate the authors' comments on any of the "arbitrary decisions". The strongest way to respond would be an experiment with one or two other reasonable choices of these parameters, showing essentially no difference in the findings.

If your goal is to keep performance similar but reduce verbatim memorization, it seems another reasonable baseline to try would be removing high verbatim memorized documents?

5 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

Thank you for your thorough review and constructive comments. All your concerns have been carefully responded below. The manuscript is carefully revised accordingly. We sincerely hope our responses fully address your questions.

W1: Limited novel empirical contribution.

Response: Our empirical novelty is demonstrating that CF-Mem itself can guide the selection of training data that yield lower verbatim leakage after fine-tuning. Motivated by the inverse relationship derived in our theoretical analysis, we prioritize examples with higher CF-Mem and test this strategy in reasoning SFT. At the reported 40% token budget, CF-Mem selection lowers the top-0.5% VB-Mem threshold from 46.77% to 25.83% on OpenThoughts and from 5.64% to 2.90% on Mixture-of-Thoughts, compared with the Random selection baseline (Qwen2.5-7B).

W2.1: About 40% token budget.

Response: Thanks for the suggestion. We added 20% and 60% selection budgets alongside 40% and comparing Random and CF-Mem on Qwen2.5-7B and OpenThoughts-114k (50k), as shown in the following Table r1 and Table r2:

Data selectionGSM8KMATH-500AMC23AIME25 (Pass@8)AIME26 (Pass@8)Average
Full91.0679.2855.943036.6758.59
Random (20%)89.2272.645.9423.3326.6751.55
Random (40%)90.1975.4254.3733.3336.6758.00
Random (60%)90.0878.1858.123043.3359.94
CF-Mem (20%)89.4973.0849.6933.333055.12
CF-Mem (40%)89.6274.551.884036.6758.53
CF-Mem (60%)89.8778.0850.6236.6743.3359.71

Table r1: Performance (%) on Qwen2.5-7B and OpenThoughts-114k (50k) under different data selection budget.

Data selectionTop 0.1%Top 0.2%Top 0.5%Top 1%Top 2%
Random (20%)96.5891.4439.054.680.66
Random (40%)98.3592.4046.776.791.21
Random (60%)98.6494.2347.2210.231.88
CF-Mem (20%)92.3170.4015.593.550.54
CF-Mem (40%)94.2476.5625.834.600.99
CF-Mem (60%)97.9287.9332.066.541.30

Table r2: VB-Mem auditing (%) on Qwen2.5-7B and OpenThoughts-114k (50k) under different data selection budget.

We find that CF-Mem achieves closed average performance with Random at the same selection budget, while yielding lower VB-Mem thresholds across all five audited quantiles. For example, at 20% budget, the top-0.5% threshold decreases from 39.05% to 15.59%. Even CF-Mem (60%) has a lower top-0.5% threshold than Random (20%) (32.06% vs 39.05%), despite its larger selection budget.

We have added these results in our revised paper. Please see Appendix C.4 for more details.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

W2.2: About the selection function.

Response: Thanks. We additionally evaluated a linear selection function alongside the original cosine function at the same 40% token budget on Qwen2.5-7B using OpenThoughts-114k (50k) and Mixture-of-Thoughts (50k). The preliminary results are shown in the following Table r3 and Table r4.

Training setSelection (40%)GSM8KMATH-500AMC23AIME25 (Pass@8)AIME26 (Pass@8)Average
OpenThoughts-114k (50k)CF-Mem (cosine)89.6274.551.884036.6758.53
OpenThoughts-114k (50k)CF-Mem (linear)89.9473.8351.5636.6726.6755.73
Mixture-of-Thoughts (50k)CF-Mem (cosine)88.7772.650.6226.6726.6753.07
Mixture-of-Thoughts (50k)CF-Mem (linear)88.1872.8547.1926.6733.3353.64

Table r3: Performance (%) on Qwen2.5-7B under different selection functions (40% budget).

Training setSelection (40%)Top 0.1%Top 0.2%Top 0.5%Top 1%Top 2%
OpenThoughts-114k (50k)CF-Mem (cosine)94.2476.5625.834.600.99
OpenThoughts-114k (50k)CF-Mem (linear)95.5178.5822.755.291.03
Mixture-of-Thoughts (50k)CF-Mem (cosine)60.6823.002.900.480.06
Mixture-of-Thoughts (50k)CF-Mem (linear)66.0028.133.270.480.07

Table r4: VB-Mem auditing (%) of Cf-Mem method on Qwen2.5-7B under different selection functions (40% budget).

The two selection functions show broadly similar overall performance and VB-Mem auditing patterns: both retain useful task performance while reducing verbatim leakage, suggesting that this observed benefit is not specific to the cosine function. We leave a more systematic investigation of different selection functions for future study.

These results have been added in our revised paper. Please see Appendix C.2 for details.

W2.3: About the proxy model.

Response: Thanks for the comment. Estimating CF-Mem requires comparing prediction performance between models trained with and without a given example. We use Qwen2.5-0.5B as a lightweight proxy to make these repeated training runs practical. Specifically, we train 50 proxy models, each for one epoch with LoRA on a randomly sampled 10% of the training data, and estimate each example's CF-Mem from the difference in average prediction performance between the models trained with and without it (Section 5 and Appendix B.1). A single 0.5B training run takes about 1.6 hours, while our preliminary estimate for a 1.5B proxy is about 2.6 hours per run. These independent training runs can be distributed across multiple GPUs to accelerate CF-Mem estimation, but our current computing resources limit the parallelism we can exploit. We will add the larger-proxy comparison in the final version.

Q1: Compare against removing highly verbatim-memorized documents.

Response: Thank you for the suggestion! We will add a comparison with directly removing highly verbatim-memorized training examples in the final version.

Dear Reviewer PSUv,

Thank you again for your valuable comments! We have carefully considered your comments and have provided our responses.

Please let us know if our replies have satisfactorily addressed your concerns. Please do not hesitate to let us know if you have any further questions or if you require any additional clarification.

Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.