acceptodds
Withdrawn from ICLR 2027

Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In this work, we take a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters. Across the Pythia family, we study memorization of duplicated training sequences (recitation) and rare ones (recollection). We find that memorization in both cases is characterized by sequence-level gradient alignment, though recitation suffers from misalignment with other training influences which causes forgetting, explaining the necessity for higher duplication of these examples. We further show that the lower model layers are the most involved in memorization and forgetting. Predicting memorization, our decomposition improves over a cross-entropy baseline, especially in larger models and early in training. Intervening on a small set of highly influential parameters we are able to ablate memorization in the final model. Together, these findings advance our understanding of how memorization develops during training and offer insights for predicting and intervening on it.

Primary area:
interpretability and explainable AI
Keywords:
memorization, interpretability, training dynamics, model component attribution, data attribution, memorization prediction

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_jz6XOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on studying memorization with a fine-grained look at training dynamics. In particular, the authors ask which learning contributions along the dimensions of parameters, data, and training process characterize memorization, and whether such understanding is actionable for prediction and intervention.

To this end, they resort to a recently proposed decomposition framework that maps a model's loss trajectory over sequences onto its influences along training, parameters, and data. Such decomposition derives an intuitive mathematical interpretation of how memorization is the result of these three aspects. Particularly, they focus on memorization of two categories of sequences: highly frequent training sequences, called recitation, and rare training sequences, called recollection. Prior work shows that recitation is the dominant mode of memorization, but it is still unclear why their memorization requires deduplication, while recollected sequences do not.

They find that a large fraction of memorization happens in very early pretraining before basic generalization has manifested. The decomposition suggests two aspects of memorization: a sequence's alignment with parameter updates from its own occurrences in training and its interaction with remaining training influences. Highly duplicated recited sequences exhibit stronger negative alignment with both other training data and weight-decay updates. Recollected sequences, in contrast, exhibit less negative alignment.

Critical strengths

The most interesting aspect of the paper is the decomposition framework (proposed by prior work) on studying memorization. The paper also has sufficient experiments.

Critical weaknesses

Although the paper looks at training dynamics to study memorization, the findings seem non-surprising. What are the implications of these results? The authors have shown that recited and recollected strings, when decomposed, yield different results—isn't it expected and could be derived without any decomposition?

Also, the central premise of using an extractable notion of memorization is questionable. Given a prefix of a string, its suffix can be extracted to some extent without having the same string in the training data (see counterfactual or contextual memorization). How do you expand your analysis to these measures where learning and not direct training can still leads to extraction?

References:

  1. https://arxiv.org/abs/2112.12938
  2. https://arxiv.org/abs/2507.14777
Questions for authors

What would be the implications of your results? Can we design training where memorization is reduced yet generalization is improved?

The paper is strong in the technical aspect. Where I feel it lacks is on motivation? It is not immediately clear why looking at training dynamics for memorization is needed. What results or misconceptions of the prior works does the new decomposition reveal? Is decomposition the only way to answer?

4 comments

Thank you for the feedback.

Although the paper looks at training dynamics to study memorization, the findings seem non-surprising.

We would appreciate more details on this. Which of our results did you not find surprising? What previous work has reported them or due to which rationale would one assume them?

The authors have shown that recited and recollected strings, when decomposed, yield different results—isn't it expected and could be derived without any decomposition?

How would one show that their decomposition is different without performing said decomposition? Again, we would appreciate which concrete results this refers to and which concrete arguments would derive them without our decomposition.

Where I feel it lacks is on motivation? It is not immediately clear why looking at training dynamics for memorization is needed. Also, the central premise of using an extractable notion of memorization is questionable.

The problem we look at is well motivated by previous work such as Biderman et al. (2023) and Prashanth et al. (2025), published at NeurIPS 2023 and ICLR 2025 respectively. Our contribution is explaining its dynamics, rather than simply finding differences between frequency groups (line 133–135: “how the rest of the training influences work towards or against memorization solutions.”). Our analysis uncovers several aspects of verbatim memorization that are not reported in previous work and which, as we show, indicate actionable routes for prediction and intervention, both, a priori and post hoc.

How do you expand your analysis to these measures where learning and not direct training can still leads to extraction?

We agree that extractability does not establish counterfactual or contextual memorization and will discuss this more explicitly. Our definition of verbatim memorization is explicit in lines 158–160 and follows Prashanth et al. (2025)’s taxonomy. They also include a third type they coin “reconstruction” which is defined to include repetitive or template cases, i.e. some common cases of contextual memorization. As we describe in lines 160-161, we did our best to exclude these examples.

What would be the implications of your results? Can we design training where memorization is reduced yet generalization is improved?

We agree that the understanding we gained through this work should be used to improve models in practice. Prediction and memorization ablation, while we mainly use it to validate our insights, have practical value (see, e.g., Biderman et al. (2026)). Given that most verbatim memories are subject to forgetting, which we show to be a causal result in our new training experiments (described in the comment to reviewer cNrf), we show that deduplication is one of the avenues to avoid memorization of undesirable sequences. We will extend our discussion with these points.

We would appreciate your feedback on whether your concerns have been addressed. We are happy to provide any further clarifications. Thank you.

My impression is the rebuttal is performed in an incomplete manner. There was plenty of time to address raised concerns (e.g., implications of the work, additional related work), and I find no action taken by the authors in updating the paper. I therefore decrease my score.

  • The motivation of a paper cannot be "many prior works have established the memorization problem, and hence we make an epsilon step towards it."

  • Verbatim memorization is highly criticized for exaggerating the threat of memorization. Many works show that LLMs can generate a text verbatim without having seen them during training. So, the choice of memorization measure should be well justified as the field becomes more mature in considering what definition of memorization makes sense or not.

Liu, Ken Ziyu, et al. "Language models may verbatim complete text they were not explicitly trained on." arXiv preprint arXiv:2503.17514 (2025).

I agree with Reviewer jz6X. I look forward to author's detailed response with full experimental results and justifications.

My impression is the rebuttal is performed in an incomplete manner.

We are sorry to hear that our rebuttal did not meet the level of detail you were expecting. We tried our best to stay brief as is encouraged by ICLR.

and I find no action taken by the authors in updating the paper. I therefore decrease my score.

We are sorry to hear this as well. Given the extensive amount of edits required to incorporate all the feedback, we decided against updating the paper and instead discussed them here.

The motivation of a paper cannot be "many prior works have established the memorization problem, and hence we make an epsilon step towards it."

This is a misrepresentation of our motivation and contribution.

Verbatim memorization is highly criticized for exaggerating the threat of memorization.

We would like to clarify that our motivation is not “memorization is a threat”.

Many works show that LLMs can generate a text verbatim without having seen them during training. So, the choice of memorization measure should be well justified [...].

Which we do as we explain above, building on a recently proposed taxonomy that arguably takes into account the phenomena you are describing. We are sorry the result of this careful deliberation did not convince you.

as the field becomes more mature in considering what definition of memorization makes sense or not.

We strongly agree that a discussion about the definition of memorization is necessary and we also discuss this in the paper. But we would like to raise that no such consensus on a definition for memorization has been reached in the community. We, therefore, argue this is part of a larger discourse on finding such consensus to which we think our work contributes

Reviewer_TnrXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the training dynamics of verbatim memorization (32-extractability) in the deduplicated Pythia models. Building on the ExPLAIND loss decompositio, it factors each training influence on a sequence’s loss into test sensitivity, update magnitude, and the cosine alignment between the sequence’s loss gradient and a parameter update component, and further splits this alignment across layers. Following Prashanth et al. (2025), sequences are grouped by extractability in the final model and by the occurrence count of their continuation: recited (extractable, >5 occurrences), recollected (extractable, ≤5), and two non-extractable controls with the same count split (32 sequences each). The auhtors discuss a batch of findings based on this framwork.

Critical strengths
  • The paper conducts an in-depth study on training dynamics.
  • The paper studies an important question.
Critical weaknesses

1. Signifance of the research

Connection with preivous research ExPLAIND

  • One thing that is not clear to the reviewer: does the proposed split add anything over raw ExPLAIND influence scores? The prediction experiment (Sec. 4.2) uses only the factored features. It should include the raw influences as a baseline feature set. Please justify / provide experimental results.
  • Besides, The causal ablation (Sec. 4.4) picks parameters by raw self-influence Iˢˣ(x;θ), which is plain ExPLAIND. So this application of this paper stem directly from ExPLAIND instead of this paper.
  • There is no contribution statement. The introduction summarizes findings but does not state which parts of the methodology are new and which are taken from prior work. Together with the two points above, this makes the significance of the work difficult to judge. I would ask the authors to add an explicit list of contributions that separates methodological novelty from empirical findings.

Memorization prediction

  • I appreciate the authors' detailed analysis in Section 4.2, but the motivation for the memorization prediction task is underdeveloped. The task is to predict which training sequences will become extractable. The introduction motivates prediction in a single sentence by reference to privacy (lines 31–33). Sec. 4.2 frames it as a validation of the analysis, and the limitations describe it as a proof of concept whose features are infeasible to compute over the full training set (llines 478–480). What is the intended use of such a predictor in a real world? How does it relate to the membership inference and training-data extraction literature? For example, could these features inform MIA or extraction methods, or data curation before training? Please clarify.

2. Evaluation

  • the authors limit their evalation only to Pythia model family. The scope of experiment might hurt the generalizability of the claims — for instance, ``memorization learning is largely located in lower layers'' might be an artifact of the specific pythia architecture. The authors shall justify other model choices like Olmo.
  • I understand the authors follow previous paper to use recitation (high frequency one) and recollection (rare one). However, the reviewer believes it is worthy to explore a fine-grained catagory of memorization, e.g., explore the effects of duplication by varying nn in duplicating a document nn times. In this way, we can extend target variable from binary (high or low frequency) to integers, enabling further regression analysis.

3. Presentation

The paper is hard follow.

  • The methodology section (section 3) is half-baked, and very confusing to readers. 1) The readers need a background (sub)section to learn what is established in ExPLAIND, and what is new in this paper. 2) Notation is introduced after it is used, or never. The superscripts ρ ∈ {z, fm, reg} appear in Eqs. 1–4. They are only explained in a passing sentence afterwards (l.113–115), and z only implicitly through “z ∈ Bₛ”. 3) The two quantities the paper relies on most are never defined in Sec. 3: The self update uₛˣ (and Iₛˣ). x is a test sequence, not an element of Bₛ, so uₛˣ only makes sense after the injection step described later (l.196). 4) Interpretation is mixed into definitions. For example, the claim that a flat loss landscape makes “forgetting the instance … more difficult” (l.123–124) is stated as part of defining test sensitivity, with no support.
  • Taking introduction section as an example, the notation of “model parameter update”, “the magnitude of the trainign updates”, “the senstivitiy of a predction” is neither well estabilished term, nor explained in details.
  • The reviewer suggests move “Experimental settings” (line 151) to a dedicate subsection / section to sepearate from “method” section.
Questions for authors
  1. What does the factorization of influence into test sensitivity, update magnitude, and alignment (Eqs. 5 and 7) add over the raw ExPLAIND influence scores? Could you report the Sec. 4.2 prediction accuracy using the raw influences as features, both alone and combined with cross-entropy?

  2. Which parts of Sec. 3 and App. A are new relative to ExPLAIND, and which are restated from it?

  3. What is the intended practical use of the memorization predictor, given that its features are infeasible to compute over the full training set? Could the features inform membership inference, extraction attacks, or data curation before training?

  4. Do the main findings, in particular the localization of memorization in lower layers (l.375), hold for another model family with public checkpoints and training data, such as OLMo?

  5. How do the alignment features vary with duplication when the occurrence count is treated as a continuous variable rather than split at five occurrences?

  6. How many copies of each sequence are injected into the simulated batch at each checkpoint (line196), and is this the same for recited and recollected sequences?

  7. How faithful is the adapted decomposition (first moment dropped, second moment estimated, substitute batch from the tail of the Pile) to Pythia's actual training trajectory? For example, do the summed simulated loss changes track the actual loss change between consecutive checkpoints, or can the approximation be checked against exact ExPLAIND on a small model?

  8. Sec. 3 reports that 10–20% of the sampled memorized sequences could not be verified as 32-extractable (l.181). But it seems that no data in figures support this claim

I will raise my rating if my concern got solved.

4 comments

Thank you for the feedback and detailed suggestions.

What is new relative to ExPLAIND: Our work is an interpretability that uses and extends the ExPLAIND framework to study the training dynamics of memorization in Pythia. Since ExPLAIND does not provide a measure of gradient alignment in terms of a cosine, we extend the aggregated case of their decomposition to a factorization of more intuitive quantities, i.e., test sensitivity, update magnitude, and alignment. We further provide methodological extensions and implementations needed to study Pythia without released optimizer states. We will distinguish the existing method from our extensions more clearly.

The prediction experiment [...] should include the raw influences [...].

We have added this as another experiment. Note again that we use the prediction setting mainly as a validation of our interpretability study and to show that the decomposition we perform helps inform early memorization prediction. The results indicate gains over the CE baseline, though CE + raw influences performs worse than the CE + our extension, especially in early training where our results are the most interesting in our opinion. We added the respective plots to the Appendix and report this in the main paper.

The causal ablation (Sec. 4.4) picks parameters by raw self-influence Iˢˣ(x;θ), which is plain ExPLAIND.

Yes, and we do not claim anything else. Since the raw influences reproduce the observed behavior changes exactly, they are the most suitable for a ranking of influence, as is also argued in Eichin et al. (2026). We did our best to improve where we might have been ambiguous about this.

Just to be clear, citing from the introduction, we claim the following main contributions:

  1. “We extend the [...] ExPLAIND decomposition”

  2. “We study the training dynamics of memorization in Pythia and show that a large fraction of memorization happens in very early pretraining”

  3. “[...] two aspects of memorization: a sequence's alignment with parameter updates from its own occurrences in training as well as its interaction with the remaining training influences. [more decomposition insights]”

  4. “To validate our understanding, we test these explanations through early prediction and post-hoc intervention. [results of both experiments]”

We do not see any benefit in providing these as an additional, redundant list. We do agree that the first point might be unclear on exactly our extensions to ExPLAIND entail. We extend this part of the introduction to mirror the more precise description in lines 93-95. We also add more precise statements about what is contributed by the ExPLAIND paper and what contributions are ours in the new background section.

Motivation: The prediction experiment tests whether the quantities identified by our analysis distinguish memorized examples. Lines 280–281 state: “To validate our explanations of memorization above, we ask to what extent the memorized examples are really separated by the phenomena we describe above.” Lines 478–479 likewise call it “a validation of our explanations and a proof-of-concept.” We do not present it as a deployable membership inference or extraction method, but we agree with you and other previous work, that reliable prediction in practice would be highly desirable for various settings. We extend our discussion on this.

Presentation: We extend and clarify the problems you raise in your review. In particular, we restructure into dedicated background and experiment settings sections, introduce relevant notation and self-updates earlier, and remove intuition parts that overextend the interpretation of our decomposition.

How many copies of each sequence are injected into the simulated batch at each checkpoint (line196) [...]?

Each selected sequence is injected exactly once into the shared batch, equally for recited and recollected sequences, as stated in lines 196–199. We will add “once” to make this unambiguous.

Sec. 3 reports that 10–20% of the sampled memorized sequences could not be verified as 32-extractable (l.181). But it seems that no data in figures support this claim

Figure 1(c–d) shows both sequence and token extractability; lines 182–184 state: “To investigate this discrepancy, we plot the average fraction of the correct continuation token being the maximum logit prediction across all sequences in Fig. 1d.”, the reported result is a direct interpretation of the last step on these plots.

Do the main findings [...] hold for [...] OLMo?

Our Limitations state that “[...]generalizability to larger models, other architectures [...] remains open” (lines 484, 485). Note that one of our contributions is the high engineering effort and cost required to extend and run ExPLAIND on other model families. While we agree an extension to OLMo would have been very interesting, but due to this cost and effort we decided against extending to this new model family.

How do the alignment features vary with duplication [...]?

The examples we study are clustered: Around 2-4 for the low-duplication examples, and at very high duplication counts. Therefore, they are too skewed to consider duplication as a numeric feature instead of our two-bin setting. We will discuss this in the Limitations and plot mean features as a function of duplication in the revision's Appendix.

How faithful is the adapted decomposition [...] to Pythia's actual training trajectory?

We test this on the training runs of our new experiments. Our results indicate that in terms of L_\inf, L1L1, and L2L2 norm and, the reconstructed moments are close to each other, and the same holds for the parameter updates that they introduce. We report on this in more detail in the revision’s Appendix. We will also mention this in the updated Limitations section.

We would appreciate your feedback on whether your concerns have been addressed. Thank you.

Thanks for your response!

At this stage, the response is too high-level -- many claims remain unsupported by actual experiment data / descriptions. For instance, The authors didn't provide details on what's established in ExPALIND and what's new in this work for the reviewer to assess whether the technical novelty.

Besides, some concerns remain unsolved: the reviewer disagree that the (1) contribution statement is a "redundant list". The author's proposed contribution statement does not genuinely distinguishes what's new to the (memorization) community. For instance, Point 2 "study the training dynamics of memorization in Pythia" has been explored in various amounts of previous works: predictability across checkpoints [1], the recitation/recollection taxonomy this paper adopts [2], and injecting sequences while continuing training from Pythia checkpoints [3]. [3] is not cited, and it reports that later checkpoints memorize more readily, which should be discussed alongside the “early memorization” claim. (2) The authors fails to justify why OLMo is out-of-scope. The "early memorization" claim could be a artifact of training curriculum / architecture defects without testing on other model families.

I will keep my rating as it is at this moment.

[1] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and Predictable Memorization in Large Language Models. NeurIPS 2023.

[2] USVSN Sai Prashanth, Alvin Deng, Kyle O’Brien, Jyothir S V, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A. Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, Katherine Lee, and Naomi Saphra. Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon. ICLR 2025. https://openreview.net/forum?id=3E8YNv1HjU

[3] Jing Huang, Diyi Yang, and Christopher Potts. Demystifying Verbatim Memorization in Large Language Models. EMNLP 2024. https://aclanthology.org/2024.emnlp-main.598/

At this stage, the response is too high-level -- many claims remain unsupported by actual experiment data / descriptions.

We tried our best to find a good trade off between brevity, which ICLR encourages, and addressing all the points raised by the reviewers. We are sorry you did not find it to be on the level of detail you required.

The authors didn't provide details on what's established in ExPALIND and what's new in this work for the reviewer to assess whether the technical novelty.

We have provided said details in the rebuttal answer (Starting with “What is new relative to ExPLAIND:”) , highlighted some of the places where we describe this in the paper, and proposed how we would be clearer about this in the paper.

For instance, Point 2 "study the training dynamics of memorization in Pythia" has been explored in various amounts of previous works: predictability across checkpoints [1], the recitation/recollection taxonomy this paper adopts [2],

We cite both studies extensively in the paper and explain the insights our study adds compared to this previous work.

and injecting sequences while continuing training from Pythia checkpoints [3]

We agree that we should have cited and discussed Huang et al (2024) as it is very relevant related work. Our work complements theirs: Through a precise gradient alignment decomposition of the actual training, we explain how memorization develops and why continued repetition is necessary for memorization. We further combine this with an exact account of memorization localization in parameter space, where we validate Huang et al.’s hypothesis that memorization is distributed across the model, though sparse interventions are possible to an extent. Their key finding of how recall interacts with general language modeling is consistent with our results: Gradient self alignment implementing memorization, which starts after basic linguistic capability is acquired, is strong further evidence for their hypothesis. Further, our work paves the way for new actionable directions addressing the practical problems of predicting and intervening on memorization.

(2) The authors fails to justify why OLMo is out-of-scope.

We justified it in the answer above and reiterate that adding another model family would mean a significant increase in computing and engineering resources. We would also like to highlight that none of the three papers the reviewer cited studied models beyond the Pythia family. Ours even includes larger checkpoints than considered by Huang et al. (2024). Therefore, we maintain that while definitely interesting, it is reasonable to argue it out of scope for this already extensive study. We have updated the above reply to be clearer on this.

The "early memorization" claim could be a artifact of training curriculum / architecture defects without testing on other model families.

Which is a limitation we discuss transparently in the Limitations and could be argued for all the papers the reviewer cites as well.

Reviewer_cNrfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends ExPLAIND to study recitation and recollection through simulated updates at multiple Pythia pretraining checkpoints. The authors find that recited examples conflict more strongly with other training updates and use this observation to explain their dependence on repeated exposure. They also use decomposition features to predict final memorization and select parameters from early influence scores to intervene on verbatim reproduction in the final model.

Critical strengths
  • Rather than only counting which texts can be reproduced, the paper separates learning from an example itself from forgetting caused by other updates. This is a useful way to study why both rare and frequent examples can be memorized.
  • The paper tests its attributions through prediction and parameter interventions rather than only presenting decomposition curves. In particular, the observation that early influence scores can guide sparse interventions on the final model deserves further validation.
  • The analysis covers several Pythia sizes and examines early checkpoints in considerable detail. The cross-model results in the appendix help assess whether the observed trends are specific to a single model.
Critical weaknesses
  • The methodological advance is limited. Section 3 largely builds on ExPLAIND and further separates its influence scores into sensitivity, update magnitude, and directional alignment. The memorization taxonomy also comes from prior work. The contribution therefore depends on reliable new findings, but the explanation for why repeated exposure is necessary still rests mainly on correlations between groups.
  • The mechanistic claims go beyond the evidence. Section 3 uses simulated updates with injected examples, omits the true first moment, and estimates the second moment without sufficiently validating agreement with actual updates. The zero ablations in Section 4.4 show that selected parameters affect reproduction of the target texts, but do not distinguish damage to stored content from disruption of general retrieval ability. Cross-entropy on 64 control examples is insufficient to rule out the latter. The paper does not directly control repetition count and spacing to test whether recited examples actually require more repetition to resist forgetting.
  • The training phases and evaluation measures need stronger robustness checks. Section 5 emphasizes transitions around 5k and 20k steps without testing their dependence on effective global batch size, learning-rate schedule, and example exposure. Agreement across Pythia sizes does not replace validation across training settings. Page 4 also reports that roughly 10%–20% of the original memorization labels could not be confirmed in the current implementation, while the ablation conclusions still rely substantially on strict 32-token extractability. These issues limit the robustness of the conclusions.
Questions for authors
  • How much additional predictive value does the full decomposition provide over cross-entropy combined with simple gradient norms or gradient alignment without path integration? This comparison would help establish whether the added method is necessary.
  • Could a small-scale experiment with saved AdamW states validate the simulated influences, followed by controlled repetition counts and spacing for the same sequences to test whether the measured interference predicts subsequent forgetting?
  • Do the three phases persist after changing effective batch size or the learning-rate schedule, when compared by cumulative tokens or example exposure? For the unconfirmed memorization labels, do the main results hold with verified labels or continuous token-level
2 comments

Thank you for recognizing the value and use of our work and your feedback.

Methodological novelty: Our main contribution is an interpretability study of the training dynamics of memorization, for which we use the framework by Eichin et al. (2026). Since ExPLAIND does not provide a measure of gradient alignment in terms of a cosine, we extend the aggregated case of their decomposition to a factorization of more intuitive quantities, i.e., test sensitivity, update magnitude, and alignment. We further provide methodological extensions and implementations needed to study Pythia without released optimizer states.

Reliability and causal evidence: The paper explicitly describes how we “decompose a simulated update per checkpoint” (lines 189–191). It also states: “Our methodology omits the influences of the first moment term in AdamW and estimates the second moment. Both choices could bias our results” (lines 476–478). We disagree that these limitations, which we discuss transparently, are a “critical weakness” of our paper.

Robustness of our results. Our limitations state that “our findings concern verbatim memorization in the Pythia family up to 6.9B parameters. Their generalizability to larger models, other architectures, and other forms of memorization [...] remains open” (lines 484, 485) and we will add "batch size, learning-rate schedule, and example exposure” as further confounders we could not account for due to the infeasibility of training our own models and the high engineering effort required to extend ExPLAIND to other model families. We wish to stress that we indeed never claimed such generality and will rework the draft where we might have been ambiguous.

Additional experiments & results:

but the explanation for why repeated exposure is necessary still rests mainly on correlations between groups. The paper does not directly control repetition count and spacing to test whether recited examples actually require more repetition to resist forgetting.

We added the following experiment that strengthens our causal hypothesis regarding forgetting. From the final checkpoint with memorization, we keep training the model on data without any occurrence of the memorized sequences with the same hyperparameters as the respective Pythia model. As predicted by our hypothesis, we find that recollected examples tend to be more robust to further training. The experiment also shows that scaling up duplication avoids these forgetting effects, and even further decreasing cross entropy on recollected instances.

The zero ablations in Section 4.4 show that selected parameters affect reproduction of the target texts, but do not distinguish damage to stored content from disruption of general retrieval ability. Cross-entropy on 64 control examples is insufficient to rule out the latter

We address this by evaluating on MMLU, Hellaswag, and Winogrande, which are all tasks that require some form of knowledge retrieval. We do this on the sparsest intervention that achieves memorization ablation while leaving cross entropy intact. Our results confirm that like CE, benchmark performance is only affected slightly, indicating that most if not all retrieval ability stays intact. We are open to add further control evaluations to the ablation experiments. Do you have a concrete recommendation on how you would test for “retrieval ability”?

while the ablation conclusions still rely substantially on strict 32-token extractability.

We agree that we should have adapted a more consistent and robust definition of memorization in the light of our findings. We relax it to demanding only 90% of all sequence tokens to be extractable to call a sequence “memorized” for ablation and other findings. We further also now report token extractability in the ablation results. Our results in the memorization ablation remain the same in the light of this change.

How much additional predictive value does the full decomposition provide over cross-entropy combined with simple gradient norms or gradient alignment without path integration? This comparison would help establish whether the added method is necessary.

Since the goal of our study is to understand model behavior, which we validate by the memorization prediction experiments, we don't think that this is within scope for this paper. We agree that this is a promising direction for future work however, which we have highlighted in the Limitations section: "we think that gradient based proxies derived from parameter updates of the train loop are a promising possibility".

Thanks again for your feedback. We would appreciate a response on whether we could address your concerns and are happy to provide further clarifications.

The rebuttal adds some useful controls, but the key concerns remain, particularly on methodological novelty and the evidence for the forgetting explanation. I therefore keep my rating.

Comments on the paper

1 comment

We thank the reviewers for their time and feedback. The reviewers raised critiques regarding presentation and soundness requiring additional clarifications in the paper and additional experiment results, which we addressed as follows:

  • We restructured the background and experimental settings, clarified notation, and distinguished our methodological extensions from ExPLAIND.
  • We added comparisons of reconstructed optimizer moments and parameter updates against those from our new training runs
  • We added continued-training experiments supporting our hypothesis that recollected examples resist forgetting more strongly and that duplication helps sustain memorization as predicted by our hypothesis
  • We added raw ExPLAIND influences as prediction features, showing that our factorized features provide greater predictive value when combined with cross-entropy, particularly early in training
  • We evaluated our memorization ablation interventions on MMLU, HellaSwag, and WinoGrande, observing only small performance changes thus confirming that the model, in particular retrieval, stays intact.
  • We added token-level extractability results and confirmed that the main ablation conclusions hold under a relaxed memorization threshold
  • We clarified the motivation, contribution, and expanded the discussion of limitations regarding model training and alternative notions of memorization

We do think the work has become much stronger through this and are grateful for this.

Ultimately, some concerns regarding motivation, framing of our contributions, and scope remained in an unresolvable disagreement. We do not expect further discussion, clarifications, or details to have a sufficient impact on the reviewers’ assessment to change the final decision. We therefore withdraw our submission. We thank everybody involved for their efforts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.