acceptodds
Under review as a conference paper at ICLR 2027

Memorisation, convergence, and generalisation in generative models for text

Abstract

How to define generalisation for a generative model? According to an influential recent proposal, an image diffusion model is said to generalise if two models trained independently on disjoint subsets of a dataset generate nearly identical images given the same latent noise. Extending this idea to autoregressive transformers is challenging: unlike diffusion models, transformers lack a shared latent variable that naturally couples generation, and discrete tokens lack an obvious notion of overlap. Here, we introduce the survival probability – the probability that two transformers trained on disjoint subsets of the training data generate the same sequence under maximal coupling – and use it to empirically demonstrate a transition from verbatim memorisation to generalisation in transformers as training set size increases. We further show that independently trained transformers develop a shared predictive subspace, defined via the leading principal components of next-token probabilities, when trained on data sets that are substantially smaller than those required for convergence. We then introduce a simple auto-regressive generative model, the spiked Markov chain, and develop an asymptotically sharp characterisation of convergence and subspace recovery for this model. Our framework provides new tools for assessing generalisation in auto-regressive generative models and motivate investigating how transformer inductive biases shape both the emergence of shared predictive structure and the agreement between generated sequences.

Primary area:
learning theory
Keywords:
memorisation, generalisation, transformers, theory, markov chains, random matrix theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FPCnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper considers how we might define a meaningful metric for generalization for autoregressive language models, based on recent research that provides such measures for diffusion models. Inspired by the approach taken with diffusion models, the authors consider a metric that is based on coupling -- a property that can be analyzed for Markov chains and applies naturally in the autoregressive setting. The paper presents theoretical results using this metric that are validated in experiments with simple transformers, and that suggest a phase transition from memorization to generalization similar to that seen with diffusion models.

Critical strengths

This paper provides a creative analysis that is grounded out in both a detailed theoretical treatment using a simplified model as well as empirical results with small transformers. I appreciated the motivation from the case of diffusion models and I think the approach did a nice job of translating those ideas into the autoregressive setting.

Critical weaknesses

The key ideas were ultimately explained but the accessibility of those ideas could be increased. Use of an illustrative figure early in the paper that explains the analogy between the diffusion model case and the current approach or illustrates key ideas such as coupling (and maximal coupling) would be valuable.

Translating the results from the theoretical and small-scale settings to more realistic language models would significantly increase the potential impact of this work.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Official Reply to Reviewer FPCn

We thank the Reviewer for their insightful questions that will indeed improve the quality of the manuscript. We will proceed now to answer to their comments here below.

W1: We agree that an introductory figure would make the connection to diffusion models and the idea of maximal coupling clearer. Creating a figure that conveys both ideas well has proved difficult, and we were not satisfied with our initial attempts. Given the short rebuttal period, we focused on addressing the reviewers' points on the contents; however, we take this suggestion seriously and will draft the figure with care after the rebuttal.

W2: We thank the reviewer for this suggestion. Unfortunately, our shared cluster was overwhelmed with requests during this rebuttal period. We managed to run additional experiments to ablate the survival length on transformers trained on the same data set (see our reply to reviewer 7cbD), but we did not manage to run additional experiments on larger instances as well. We will add them to the revised version of the manuscript.

Reviewer_7cbDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a way to estimate memorisation and generalization in autoregressive transformers using survival probability under maximal coupling. The authors empirically demonstrate a transition from memorisation to convergence as training-set size increases across three common text datasets, while these independently trained transformers recover shared predictive subspaces at substantially smaller training set sizes than those required for convergence. The paper then studies Markov chain models, deriving analytical results for convergence in a homogeneous setting and heuristic characterisation of the subspace recovery transition in a spiked Markov chain.

Critical strengths

Understanding convergence (in a way analogous to the platonic representation hypothesis) is a rather interesting problem, and this paper provides a nice theoretically grounded approach to this problem using survival probabilities, resulting in an extension of convergence based analyses previously developed for diffusion models. This empirical separation is also backed by Markov chain theory, which results in the right qualitative picture.

There is also an interesting observation that disjointly trained transformers aligned predictive directions at smaller training sizes than those required by convergence. The reported separation appears across three text datasets (Figure 2,1,f.1), suggesting that learning reproducible predictive structure and learning similar full next-token distributions may occur at different data scales. The spiked Markov-chain model provides a useful theoretical setting for investigating this phenomenon.

Critical weaknesses

Unfortunately, the empirical results fall short of showing that proposed metrics actually capture generalisation. The paper defines generalisation, following Kadkhodaie et al., as independence of the learned model from the realisation of its training data. However, disagreement between models trained on disjoint splits mixes two sources: dependence on the training data and stochasticity of initialization/optimisation. The authors compare the two sources only at the largest nn, where the same-data, different-seed baseline matches the disjoint-data result. At small nn, low LconL_{con} is attributed to memorisation, but if models trained on the same data also disagree there, the disagreement reflects training stochasticity rather than data dependence. Similarly, when a training prefix uniquely identifies a sequence in the empirical training distribution, the memorisation survival probability reduces to the model's probability of reproducing the corresponding training suffix, making it closely related to training negative log-likelihood. The paper does not sufficiently establish what additional information this measure provides over existing memorisation metrics.

The theoretical part of the paper does not analyse the quantities measured in the experiments. The convergence result in Theorem 4.1 concerns a homogeneous Markov-chain Poisson surrogate with fixed row denominators, whereas Claim 4.1 provides a heuristic characterisation of subspace recovery in a spiked Markov chain, relying on additional approximations. The transformer experiments measure overlap between frequency rescaled principal directions of next-token probability covariance matrices, whereas theory studies eigenvectors of non-Hermitian transition matrices. These need not have the same recovery thresholds.

There are also some inconsistencies. In Claim 4.1, the definition of the spectral radius ρα\rho_\alpha is missing a square root compared to Appendix E, equation 16 uses an incorrect normalisation. Equation 7 does not specify whether directions are orthonormalised to be bounded and basis invariant.

Questions for authors

Could the authors report the same-data, different-seed survival length (the dashed baseline in Fig 1c) across all training-set sizes and for all three datasets, rather than only at the largest nn? I would also encourage a different optimizer comparison, e.g. Adam vs Muon, as this is likely to provide even more insights which may not be obvious if other hyperparameters are kept fixed.

Could you clarify if survival length captures any aspects of model behaviour beyond standard losses, e.g. do you have examples where the two disagree?

Could the authors confirm whether ρα\rho_\alpha in Claim 4.1 is correct, and likewise equation 16.

Under maximal coupling the distance function becomes TV, so it inherits properties of it. I.e. it is dominated by differences on high-probability tokens and largely insensitive to relative differences in the tail. Beyond maximising agreement, is there a reason to prefer this over other discrepancies between next-token distributions, such as KL or JS? Are the location and sharpness of the transition robust to this choice?

5 comments
AuthorsOpenReview ↗

Official Reply to Reviewer 7cbD (part I)

We thank the reviewer for their careful reading of the manuscript and for their valuable comments, which will be incorporated to improve the manuscript. We now proceed to address all their questions in detail.

W1: We thank the Reviewer for suggesting an ablation where we train transformers independently on the same data set. We reported the same-data, different-initialization survival length at every training-set size for all three corpora in Fig. B.3 of the Appendix. At the smallest sizes, same-data models agree much longer than models trained on disjoint data. The disagreement of small models therefore stems to a large extent from variations of the training data. Note also that memorisation persists for longer than agreement of transformers trained on the same data set, since we test memorisation by prompting with the first token of a training sequence, while survival is measured using the test set.

We also agree that memorisation survival is related to training loss, but we would like to stress that it goes beyond a single average training loss in at least two ways. First, sampling coupled continuations gives a survival curve across the full sequence, showing how the probability of verbatim reproduction changes with length; averaging loss over tokens and examples removes this information. Second, when a prefix admits multiple training continuations, survival depends on the overlap between the model's next-token distribution and the empirical distribution: models with the same average loss can have different survival probabilities (see the concrete example in the reply to Q2).

We also agree that memorisation survival is related to training loss, but we would like to stress that it goes beyond a single average training loss in at least two ways. First, sampling coupled continuations gives a survival curve across the full sequence, showing how the probability of verbatim reproduction changes with length; averaging loss over tokens and examples removes this information. Second, when a prefix admits multiple training continuations, survival depends on the overlap between the model's next-token distribution and the empirical distribution: models with the same average loss can have different survival probabilities (see the concrete example in the reply to Q2).

W2: We thank the Reviewer for advising us to clarify the actual goal of our analysis, which is, ultimately, building a simple and tractable Markov model that captures the qualitative behaviour of convergence and subspace recovery in transformers. Specifically, our goal is to qualitatively reproduce the transformer’s learning scales in terms of data set size, without predicting exact thresholds.

For convergence, both settings - i.e. transformer and Markov models - measure survival under maximal coupling. Theorem 4.1 analyses a Poisson surrogate for the Markov chain, whose fixed denominators are justified by uniform concentration of row totals in its regime, and which is found to be in excellent agreement with finite-size simulations of the Markov chain (see Figure 3).

For subspace recovery, the sparse regime requires retaining random denominators, which hardens the analysis. To tackle these difficulties, we define a surrogate model, which uses Poisson visit counts and visit-weighted destinations, followed by a Gaussian approximation matching conditional means and covariances. We find that these approximations yield predictions that agree very well with Markov-chain simulations (see Figure 4): establishing rigorously that Claim 4.1 holds for the original Markov chain is an interesting and technical question of random matrix theory, which we leave as a conjecture for future work. Appendix C.2 provides an argument showing that in the spiked population model, the next-token covariance and transition matrix share the planted signal direction, providing a structural connection to the transformer observable. However, we emphasize that their empirical principal directions and eigenvectors need not have identical recovery thresholds, and we do not make any quantitative claim connecting the thresholds and overlaps between the Markov chain models and transformer experiments.

We have used the additional space for the reply to expand on these points throughout Section 4, for example by expanding the introduction to this section. We also highlight the difference between recovering the singular vectors and the eigenvectors.

Q1: We now report the same-data, different-initialization survival length at every training-set size for all three corpora, see Fig. B.3 of the appendix; as we discuss above, this data supports our memorisation interpretation of low survival length at small nn. This ablation required us to rerun our entire experimental suite; unfortunately, this meant that we were not able to provide a further ablation on the choice of the optimiser. We will look into this after the rebuttal period.

AuthorsOpenReview ↗

Official Reply to Reviewer 7cbD (part II)

Q2: We agree that, in the memorization setting, when there is a single training suffix, the quantity Smem(t)S_{\mathrm{mem}}(t) matches the negative log-likelihood. This holds because we are effectively coupling the training dataset itself to the learned model. If there are several possible suffixes, we instead study the empirical conditional distribution over suffixes given the prefix. This reasoning does not carry over to convergence, since in that case we are coupling the two separately trained transformers, rather than coupling a transformer to the dataset. Convergence survival can, in fact, reveal phenomena that ordinary validation losses fail to capture. Take a vocabulary of size 33 with token frequencies r=(0.6,0.2,0.2)r=(0.6,0.2,0.2) after prefix hh, and two trained models AA and BB that estimate the conditional next-token distribution given a prefix hh. Their distributions are rhA=(0.6,0.3,0.1)r_h^A=(0.6,0.3,0.1) and rhB=(0.6,0.1,0.3)r_h^B=(0.6,0.1,0.3), respectively. The mean one step negative log-likelihood is given by LhM‾=−∑i=13rilog⁡(rh,iM)\overline{L^M_h} = -\sum_{i=1}^{3} r_{i} \log\left(r_{h,i}^{M}\right) with M∈{A,B}M \in \{A, B \}, which amounts to averaging the loss according to the empirical token frequencies. Because the two tokens whose probabilities are exchanged occur with the same frequency, the validation loss ends up identical. Here, however, we consider S(1)=∑i=13min⁡{rh,iA,rh,iB}S(1)=\sum_{i=1}^3 \min\{r_{h,i}^{A}, r_{h,i}^{B}\}, yielding a one-step survival probability of 0.80.8.

Q3, W3: In Claim 4.1, the correct bulk radius is ρα=(1−e−α)/α\rho_\alpha=\sqrt{(1-e^{-\alpha})/\alpha}, in agreement with Appendix E. In addition, the overlap in Eq. 16 should be expressed as the squared cosine $ |\langle u^\star,u\rangle|^2/(|u^\star|^2|u|^2)$; under the given normalisation, the denominator becomes V2V^2. For what concerns the details around Eq. (7), we did clarify it in the text, and we direct the Reviewer to the reply to Q1 from Reviewer sqcF, for an extensive definition of the way we process the principal directions.

Q4: Our maximum-coupling construction builds on the total variation distance because it yields the largest possible probability that two sampled tokens agree during joint sampling. Using different distances, like Jensen-Shannon divergence also compare predictive distributions, but do not themselves define a coupling or a survival probability. Following your suggestion, we did however compute the Jensen-Shannon divergence, essentially a symmetric version of the Kullback-Leibler divergence, between the next-token probability distributions of independently trained transformers on validation sequences, see Fig B.2 of the revised manuscript. Agreement between models increases sharply at the same sample complexity as the survival length. However, this measure evaluates predictions on held-out data, rather than generation, which is the ultimate goal of a generative model. We discuss this point now at the end of p.4, and Fig B.2.

We hope that our replies and our revised analysis have resolved your concerns. If so, we would appreciate if if you could increase your score; if not, we look forward to clarifying any remaining issues during the discussion period.

Thank you for the detailed response, additional experiments and technical corrections. A number of concerns remain:

Memorisation and convergence. Figure B.4 supports the memorisation-based interpretation only at the smallest dataset sizes. At intermediate sizes, both same-data and disjoint-data pairs exhibit near-zero survival lengths, which increase at similar scales. It is unclear whether the observed transition primarily reflects decreasing dependence on the training data or increasing reproducibility across training runs. The interpretation at the end of the Discussion on page 4 remains stronger than the evidence supports.

Value of survival metrics? For unique training continuations, log memorisation survival is just the negative cumulative training NLL of the suffix. Moreover, the example cited in the response to W1 yields identical memorisation survival probabilities (0.9 for both models) when each model is coupled to the empirical distribution rr. While survival metrics capture information not contained in average NLL, their additional empirical value remains unclear, particularly given that JSD identifies the same convergence transition.

Theory -- experiment? The revised manuscript better acknowledges the limitations of the theoretical analysis, but the convergence asymptotics remain restricted to a homogeneous Poisson surrogate, despite the broader claims in the abstract. Furthermore, the revised Section 3 estimator discards the leading principal direction, which in the spiked Markov model is precisely the planted signal. The claimed correspondence in Section 4.3 therefore does not hold as written.

Reviewer_sqcFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memorisation, convergence and generalisation in generative models for text. The authors use step-wise maximal coupling to define survival probability and survival length, respectively measuring the model's verbatim memorisation of the training sequence and the generation consistency between models trained independently on disjoint subsets. The paper shows that as the amount of training data increases, verbatim memorisation weakens and the consistency between models improves; at the same time, the shared predictive subspace extracted from next-token probabilities begins to appear at smaller data sizes. The authors further analyze convergence through the Poisson surrogate model and give a heuristic justification for the structure recovery threshold.

Critical strengths
  1. The comparison method is intuitive and has a clear probabilistic interpretation. Maximal coupling directly links the matching probability of two next-token distributions to the total variation distance. Using the same framework to compare the model and the empirical distribution of the training set, as well as two independently trained models, provides a unified perspective for research on verbatim memorisation and convergence.
  2. It is instructive to distinguish different levels of learning phenomena. This paper shows that a shared predictive structure may appear earlier than the consensus of the generated sequences. It suggests that learning primary structures consistent with a wider distribution may require different data sizes.
Critical weaknesses
  1. The definition of subspace overlap may have a gap with empirical results, which affects the interpretation of Figure 2. Equation (7) first calculates the principal directions, then divides the coordinates by the empirical unigram frequency, and finally sums up the squares of the pairwise inner products of the two sets of directions. This operation usually changes the length of the directions and the mutual perpendicular relationship, so the written formula cannot automatically become a subspace overlap with a value in ([0,1]). The result in Figure 2 is at ([0,1]), indicating that the actual implementation may contain unreported steps. The authors need to clarify the post-recalibration process and review the conclusions of the shared predictive subspace with a complete definition. Relevant locations: Equation (7) on page 5 and Figure 2. For example, under uniform word frequency, if the original unit length direction is divided by (1/V) strictly according to the text, even if the two sets of directions are exactly the same, equation (7) will result in (Q=V^4). Even if each direction is additionally restored to unit length, the different directions are not guaranteed to be perpendicular to each other, so (Q>1) is still possible.
  2. The explanation from convergence to generalisation is still strong. Survival probability directly measures the generative consistency between models. Two models that are equally unable to accurately depict text may be highly consistent. Therefore, it is possible to adopt convergence as an operational definition of generalisation, but additional evidence is needed between this definition and the quality of the generative models. Existing experiments mainly show that this indicator rises as the amount of training data increases, and is close to synchronization with the training loss and validation loss. The paper doesn't yet fully demonstrate whether it can distinguish between "learning the data structure" and "the model making the same error". It is recommended to add corresponding comparisons and to distinguish more clearly between phenomena directly measured by indicators and further inferences.
  3. The main text formula of survival probability is inconsistent with the appendix. Equation (3) uses the expectation under the maximum coupling process (\mathbb E_{\max}), while Appendix Equation (D.6) explicitly uses the expectation under the matched-path process (\mathbb E_K). The two are generally not interchangeable. The correct calculation method is given in the appendix, but the author needs to express it uniformly and explain which calculation process is used in the experiment.
Questions for authors
  1. After frequency-rescaled in equation (7), are the directions reprocessed to be perpendicular to each other and have a length of 1? Please provide complete calculation steps and confirm whether the corrected subspace overlap retains the main conclusions of Figure 2.
  2. Does the survival probability in Figure 1 directly count co-survival events, or does it calculate the average of products under a matched-path process? Can you explain how the implementation corresponds to Equation (D.6) and modify Equation (3)?
2 comments
AuthorsOpenReview ↗

Official Reply to Reviewer sqcF

We appreciate very much the reviewer’s careful review of our manuscript and their helpful feedback. Their comments highlighted points that we had not fully considered. We now proceed to answer to their comments and questions.

W1: We thank the reviewer for their careful reading which revealed this issue. We divided the eigenvectors by the unigram distribution because that step yields an unbiased estimator of group structure in the Markov chain. Our idea was thus to apply the same principled estimator to transformers, too. However, the reviewer is correct that the normalisation breaks exact orthonormality of the eigenvectors. We have therefore simplified the subspace overlap: we are now simply computing the overlap between eigenvectors of the correlation matrix of next-token probability distributions evaluated over the test set. In this way, the subspace overlap is guaranteed to be between 0 and 1. We still find a clear transition that precedes convergence on all three data sets. We have updated the corresponding plots and descriptions in the main and in the appendix (see details in our answer to your Q1; full details in sec. 3, appendix B of the revised manuscript).

W2: We agree with the Reviewer that convergence does not necessarily correspond to generalization in the sense of learning the correct data structure and not committing a systematic error. A goal of our analysis paper is precisely to show, following recent work on diffusion models [1, 2, 3], that generalization in generative models is multi-faceted, and that convergence and subspace recovery require a different number of training samples -- they capture different aspects of generalisation, and neither one by itself fully accounts for it. We have revised the manuscript in order to stress the difference between these two concepts; cf. in particular the abstract; second paragraph of Introduction; contribution statement; renaming of the title Section 3, and our Discussion.

W3: Thank you for spotting this issue - the correct equations do indeed appear in the Appendix, and Equation D.6 is the correct one. We have revised Equation (3) and updated the manuscript by redefining measures EK=EmatchE_K = E_{\text{match}} to clarify this distinction. Equation D.6 also describes accurately how how we computed the survival probability in the transformer experiments.

Q1: As we explained above, we originally defined the subspace overlap in terms of unbiased estimators of the group structure defined for the Markov chain. We have revised this definition: for each transformer b∈{0,1}b\in\{0,1\}, we now compute next-token probabilities pj(b)(ha)p_j^{(b)}(h_a) over the test set. We then centre and standardize each token's predicted probability across contexts: \begin{equation} Y_{aj}^{(b)}\coloneqq \frac{p_j^{(b)}(h_a)-\bar p_j^{(b)}}{s_j^{(b)}}, \end{equation} where pˉj(b)\bar p_j^{(b)} and sj(b)s_j^{(b)} are the mean and standard deviation of next-token predictions over test sequences, respectively. We then compare the leading eigenvectors of the correlation matrix \begin{equation} R^{(b)}\coloneqq\frac{1}{M}\bigl(Y^{(b)}\bigr)^\top Y^{(b)}. \end{equation} We define the subspace overlap as the overlap between the subspaces spanned by the leading kk eigenvectors vr(b)v_r^{(b)}. Full details in appendix B.

Q2: We thank the Reviewer for the observation and we accordingly fixed Eq. (3). Concerning the interpretation of Figure 1, the survival length measures the expected number of matched occurrences between the two paths, averaged over the initializations of the paths. The ordering of such matched co-survival events is not taken into account. We made this point more clear in the manuscript, cf. caption in figure 1 and Section 2.

We hope that our replies and our revised analysis have resolved your concerns. If so, we would appreciate if if you could increase your score; if not, we look forward to clarifying any remaining issues during the discussion period.

References:

[1] Maillard & Goldt, Memorisation, convergence and generalisation in generative models, (2026).

[2] Garnier-Brun et al., Biased Generalization in Diffusion Models, (2026).

[3] Marion & Wu, Understanding diffusion models requires rethinking (again) generalization, (2026).

Thank you for the detailed rebuttal and revision. I appreciate the revised claims regarding generalization. And I find that the newly added Figure B.3 (in response to Reviewer 7cbD) is also a good ablation study that addresses my concern in W2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.