acceptodds
Under review as a conference paper at ICLR 2027

Memorization Dynamics in Masked Diffusion Language Models

Abstract

Recent work on continuous diffusion models suggests that generalization and memorization can emerge on distinct training timescales, with population-level structure learned before individual training examples are memorized. In parallel, masked diffusion language models (DLMs) have become competitive language models while replacing autoregressive generation with iterative denoising. This raises a natural question: *does population learning also precede memorization in masked DLMs, or can the ordering change with the training regime?* We study this question using controlled Markov and hierarchical-grammar populations, trained canaries, never-trained controls, and Bayes-referenced population denoising. Contrary to a universal generalization-before-memorization picture, the relative timing of population learning and training-record extraction depends on the training regime. At low repetition, population learning can precede training-record extraction by thousands of optimization steps, whereas increasing repetition reverses the ordering and causes extraction to emerge first; in a contrasting Markov regime, extraction appears without an observed population-learning phase over the tested horizon. Across joint dataset-size and repetition interventions, normalized record frequency strongly organizes these dynamics, but controlled decoupling experiments show that it is not sufficient alone: both the sampling frequency of an individual repeated record and the aggregate training mass assigned to repeated records can alter the ordering. Mask-rate distribution shape further changes extraction onset even when its first two moments are matched, while pretrained LLaDA and MDLM reproduce the directional effect of repetition. Together, our results show that population learning and train-specific extraction are distinct phenomena whose relative timing is regime-dependent rather than universally ordered.

Primary area:
generative models
Keywords:
Diffusion Language Models, Memorization, Training Data Extraction, Population Learning, Canary Extraction, Data Repetition, Masking Schedules, Generalization Dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uiGDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper contains experiments showcasing how canary repetition changes the order of population learning and memorization in masked diffusion models. Despite good experiments, training-mixture effects and the limited explanatory power of the theoretical results lead to a weak rejection assessment.

Critical strengths

The paper establishes experimental results using masked diffusion language models trained on synthetic distributions as well as pretrained models. Using their measures for memorization and population learning they provide examples of one happening before the other in training and vice versa, varying canary repetition RR. The authors highlight that this differs from previous results obtained for continuous diffusion models (Bonnaire et al., 2025) where population learning happens before memorization. This comparison could benefit from the use of a more similar measure of memorization.

Further, they find a promising driver for this regime change, namely the sampling probability of a canary. They investigate this effect further by varying the sampling probability of all canaries fcanf_{\text{can}} or the sampling probability of a canary pcanp_{\text{can}} while holding the other one fixed. Both factors lead to a regime change.

Thorough robustness checks of the experiments are performed in appendix D. They may benefit from additional seeds.

Critical weaknesses

The proofs of the theoretical results look correct. But the theoretical results posed in the main part are rather elementary. The results from appendix B and C are more involved but rely on assumptions as mentioned by the authors, they consider them as motivation for separated optimization time scales. Maybe these assumptions could be resolved by future work or verified empirically.

The main weakness, together with the fact that the main theorems are elementary, is that they exhibit convincing empirical evidence showing that a memorization-before-population-learning mechanism exists, but the underlying mechanism is not really clear or studied in theory.

To be more precise: they hand-pick several thresholds, like 0.5, 0.1 etc in the definitions in Sections 3, and the regime shift they see I think it depend on those choices: probably the general mechanism would be true but the shift would appear for different levels of R. The general relation though is still completely unclear.

Some relevant terms could be defined more explicitly: geometry, normalized edit distance, Lchance\mathcal L_{\text{chance}}, LBayes\mathcal L_{\text{Bayes}}, $\mathcal L_{\text{val}}(s) $, Enc. In appendix B a notation table could be helpful.

In the way memorization and population learning are defined in this paper it seems reasonable that at some point τmem<τpop\tau_{\text{mem}}<\tau_{\text{pop}} when the canary repetition R→∞R\to\infty since the canaries will then dominate the empirical distribution (especially for K=1K=1). This effect seems to get acknowledged by the authors in Section 7: "The decoupling experiments therefore characterize competition induced by the training mixture rather than an intrinsic architecture-specific memorization clock."

Questions for authors

How is Enc\text{Enc} in Appendix B defined?

4 comments

We thank the reviewer for the constructive feedback. We conducted additional experiments supporting precise explanation: training-mixture allocation predicts much of the onset ordering, while masking distributions influence extraction timing beyond record-presentation counts.

  1. Mechanism underlying the ordering reversal

We agree that increasing canary repetition changes the training mixture and can induce extraction-first behavior without an intrinsic architecture-specific memorization clock.

For nn background records and KK canaries repeated RR times, individual-canary sampling probability and aggregate canary mass are $ p_{\mathrm{can}}=\frac{R}{n+KR}, \qquad f_{\mathrm{can}}=\frac{KR}{n+KR}. $

Under fixed exposure thresholds CmemC_{\mathrm{mem}} and CpopC_{\mathrm{pop}}, with BB record presentations per update, the two-clock model predicts

$ \frac{\widehat{\tau}{\mathrm{mem}}} {\widehat{\tau}{\mathrm{pop}}}

\frac{C_{\mathrm{mem}}}{C_{\mathrm{pop}}} \frac{n}{R}. $

Thus, increasing R can produce extraction-first ordering through exposure allocation alone, including in the large-RR limit. We fitted this model on controlled dataset-size/repetition experiments and evaluated it without refitting.

EvaluationCorrect orderingExtraction errorPopulation error
30 held-out mixtures26/3019.7%24.6%
In-distribution, R=1R=13/3154.6%56.9%
In-distribution, R=16R=163/311.4%56.6%

Table R1. Frozen exposure-model predictions. Errors are mean absolute relative onset errors under the original first-crossing criterion.

To test whether presentation counts fully explain extraction timing, we compared two masking schedules with identical realized per-record presentation histories and equal training budgets (40.96 million tokens), but different corruption distributions.

SeedSchedule ASchedule BDifference
10124028040
20220028080
30320028080
Mean213.3280.066.7

Table R2. Extraction onsets (optimizer updates) under matched record-presentation histories.

The count-only model predicts 295.6 updates for both schedules, yet schedule A extracts earlier in all three paired seeds. Thus, record-presentation counts alone do not determine extraction timing across these schedules. Since the masking distributions differ in their tails and higher-order properties, the comparison establishes a schedule-associated effect without isolating a unique mechanism. Population onsets remain censored beyond 20,000 updates in these conditions.

2. Robustness to onset thresholds

We agree that operational thresholds influence measured onset times. The original criteria use Bayes-referenced population gain G≥0.50G\geq0.50 for three consecutive evaluations and canary-minus-control extraction advantage ≥0.10\geq0.10 at first crossing.

The original low-/high-repetition ordering persists across 80 tested threshold, persistence, and evaluation-grid configurations, although two near-boundary configurations elsewhere change classification.

We additionally evaluated independent Q32 reconstruction queries requiring three consecutive extraction-threshold crossings.

SeedRRPopulation onsetQ8 extractionQ32 sustained extraction
10114207803240
20213601000Censored at 4000
30314401320Censored at 4000
10116820280280
20216800240220
30316680280280

Table R3. In-distribution grammar onsets (optimizer updates, native 20-step grid). Q8 uses first crossing; Q32 uses an independent fixed-query panel and three consecutive crossings.

The ordering persists across all paired seeds: population learning precedes sustained extraction at R=1R=1, while extraction occurs first at R=16R=16. However, precise low-repetition extraction onsets are sensitive to query and persistence criteria. Thus, the qualitative contrast is robust under tested conditions, whereas numerical onset times and transition boundaries are operationally dependent.

  1. Record-specific extraction

We additionally evaluated canaries and controls sampled from the same hierarchical grammar as the background distribution. To control for record-specific reconstruction difficulty, paired inclusion/exclusion models evaluated identical candidate records using fixed queries.

RRMean inclusion effect at step 4000Seed SDSustained ordering
10.28130.1507Population first (3/3)
160.93750.1083Extraction first (3/3)

Table R4. Paired inclusion-associated exact-recovery differences.

Recovery depends on whether the specific record was included in training, with substantially larger effects at high repetition. Since inclusion changes record exposure, this does not establish an exposure-independent mechanism or a calibrated membership-inference attack.

4. Interpretation of the theoretical results

We agree that the theoretical results have distinct roles. Theorem 1 and Proposition 1 characterize population and empirical denoising conditionals through standard cross-entropy minimization. The context-uniqueness result characterizes when revealed tokens identify a training record, without predicting neural extraction onset.

Appendices B and C derive optimization timescales for a squared-loss random-feature surrogate under explicit spectral and corruption assumptions. These assumptions are not verified for the trained Transformers; consequently, the surrogate provides conditional analytical motivation rather than a proof of the observed neural ordering.

The central contribution is therefore the controlled empirical characterization, supported by quantitative exposure-model predictions, matched-exposure masking experiments, and record-inclusion controls.

5. Notation and definition of Enc\mathrm{Enc}

We clarify the requested notation:

SymbolDefinition
Conditioning geometryArrangement of revealed and masked positions; controlled evaluations use 32/64 scattered revealed tokens
Normalized edit distanceToken-level edit distance divided by sequence length
Lchance\mathcal{L}_{\mathrm{chance}}Chance-reference denoising loss
LBayes\mathcal{L}_{\mathrm{Bayes}}Bayes-optimal population denoising loss
Lval(s)\mathcal{L}_{\mathrm{val}}(s)Held-out validation denoising loss at update ss
pcan,fcanp_{\mathrm{can}},f_{\mathrm{can}}Individual-canary sampling probability and aggregate canary mass

The population gain is

\mathrm{Bayes}}}.$$ In Appendix B, \$\mathrm{Enc} \$ denotes the abstract representation map producing centered clean and corrupted context vectors: $$z^\nu=\mathrm{Enc}(x^\nu)\in\mathbb{R}^d, \qquad z_t^\nu=\mathrm{Enc}(x_t^\nu)\in\mathbb{R}^d.$$ It is distinct from the random-feature transformation $$\phi_t(z_t)=\sigma\left(\frac{Wz_t}{\sqrt d}\right).$$ The analysis does not prescribe a unique token-to-vector implementation of $\mathrm{Enc}$ or identify it with the trained Transformer encoder; its conclusions remain conditional on the surrogate assumptions. ### 6. Conclusion **The additional results provide a quantitative explanation of the reversal through exposure allocation, demonstrate masking-dependent extraction timing under matched record presentations, and establish record-specific inclusion effects for in-distribution targets. The principal ordering remains robust under the tested evaluation criteria, while exact onset times depend on the operational definitions.**

Thank you for the additional experiments. The new experiments are convincing and strengthen the empirical contribution, particularly the additional robustness checks.

My main concern remains the theoretical part. While the exposure-based model helps explain the observed reversal, it relies on assumed thresholds. Also, the random-feature analysis still depends on assumptions that are not verified for the models considered.

So, I would be willing to maintain my score for now.

Thank you for recognizing that the additional experiments strengthen the empirical contribution. To further address your theoretical concerns, we have made a focused extension to the existing conditional analysis. This derives learning times from stated categorical dynamics rather than assuming exposure thresholds, without relying on random-feature spectral assumptions. The results are in Theorems 1–2 (pp. 4–5, lines 197–238), and the proofs are in Appendix A.4 (pp. 15–17, lines 761–892). All additions are blue.

The analysis separates two claims. The exposure model fitted to the experiments remains a quantitative baseline. Its threshold constants are estimated from data. Theorem 2 solves gradient flow for the expected cross-entropy objective with separate categorical logits for each context and no explicit mask-rate input. It starts from uniform predictions and assumes a context with a deterministic target token.

Consider a positive-weight context that identifies one record, with sampling probability precordp_{\mathrm{record}}. Let SS be its revealed positions, excluding the scored masked token, and let LL be sequence length. Under the inverse-rate objective, Eq. (3), the time to reach confidence uu, with 1/V<u<11/V < u < 1, is

T(u)=CV(u)precord,κπ(S),κπ(S)=Et∼π[tL−∣S∣−1(1−t)∣S∣],T(u)=\frac{C_V(u)}{p_{\mathrm{record}}\\,\kappa_\pi(S)},\qquad \kappa_\pi(S)=\mathbb E_{t\sim\pi}[t^{L-|S|-1}(1-t)^{|S|}],

Here VV is the vocabulary size. Specifically,

CV(u)=V−1V,[F(u)−F(1/V)],F(q)=log⁡q1−q+11−q.C_V(u)=\frac{V-1}{V}\\,[F(u)-F(1/V)],\qquad F(q)=\log\frac{q}{1-q}+\frac{1}{1-q}.

The function CV(u)C_V(u) follows from the differential equation (p. 16, lines 841–863). It is not an assumed exposure requirement. No random-feature spectrum enters the proof. Appendix A.4 also derives sufficient conditions for ordering across an interval of different criteria (p. 17, lines 864–885).

This makes threshold dependence explicit. In the theorem's illustrative record-specific and pooled-background cells, equal mask geometry and a common confidence criterion give a hitting-time ratio of n/Rn/R at every interior confidence level. Different criteria can move the crossover; the appendix gives conditions for an ordering to hold across an interval of criteria. Our empirical population and extraction measures use different criteria. Their observed low-/high-repetition contrast is supported separately by threshold and sustained-extraction sensitivity experiments.

The accompanying objective decomposition explains the masking factor κπ(S)\kappa_\pi(S). Even when two runs receive the same records, they can assign different weights to the prediction problems defined by their masks. Equality of the complete weighted mask law requires equality of moments through degree L−1L-1. Thus, matching only the first two moments does not generally match these weights. This statement concerns the inverse-rate objective and uses no assumed Transformer spectrum. Appendix A.4 treats the normalized minibatch loss separately (p. 16, lines 820–840).

The scope is explicit. Separate context logits do not capture parameter sharing in a Transformer, and token confidence is different from the paper's canary-minus-control exact-recovery advantage. The theorem gives an exact instance of joint exposure and masking dependence. The matched-presentation experiments establish the corresponding timing difference in the trained neural models. Predicting their numerical onset or crossover repetition remains open.

We ground this extension in the conditional view of Ou et al. (2025). We also distinguish our change of mask-rate distribution from the compensated time reparameterization studied by Sahoo et al. (2024). Together, the updated theory and experiments clarify that record allocation explains much of the observed ordering, while mask-conditioned supervision provides an additional factor not captured by record counts alone.

References

  1. Ou, J., et al. (2025). Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. ICLR.
  2. Sahoo, S. S., et al. (2024). Simple and Effective Masked Diffusion Language Models. NeurIPS.
Reviewer_hQYAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies, in masked diffusion language models, the relative timing of population learning and the extraction of inserted random canaries. It uses synthetic grammar and Markov data, matched never-trained controls, and a Bayes-referenced population gain. It reports that increasing canary repetition reverses the ordering from population-first to extraction-first, and that this ordering is organized by the per-canary sampling probability and the total canary mass. It also reports that the shape of the masking schedule affects extraction onset, and that the repetition effect carries over directionally to LLaDA and MDLM.

Critical strengths

The question is relevant and the methodology is careful: matched controls, censoring of unobserved onsets, held-out confirmation, paired seeds, and prespecified criteria.

The decoupling of p_can and f_can is a nice controlled design.

The paper is honest about the scope of every claim.

Critical weaknesses
  1. What is memorized. The canaries are random strings that do not follow the grammar. Extracting them is memorization of noise injected into the mixture. It is not memorization of the background training samples, which is the phenomenon studied in the continuous-diffusion literature the paper positions itself against. The natural experiment is missing: measure whether and when the n background sequences become extractable relative to population learning. Without it, the comparison with the “generalization before memorization” picture is not like-for-like.

  2. The main reversal is expected. A random string sampled with probability p_can is fitted after roughly 1/p_can presentations. Out-of-distribution canaries carrying mass f_can take training signal away from the background. Extraction-first at high repetition and later population learning at high f_can follow directly. The fitted exponents (τ ∝ n^0.94, R^−0.47) should be discussed in this light, for instance by expressing them in terms of p_can.

  3. The Markov regime is under-sampled. With vocabulary 256, the transition kernel has about 65K free parameters, but n = 128 sequences of length 64 provide only about 8K transitions. The population cannot be learned at all, so “extraction without population learning” is unsurprising. The vocabulary-64 run, which drops slightly below chance, is consistent with this. A study varying the complexity of the rule at fixed n, and n at fixed complexity, would turn this into an informative phase diagram.

  4. The theory is weak. Theorem 1 and Proposition 1 are standard cross-entropy facts. The uniqueness formula is elementary, and it has no predictive value empirically. The random-feature appendices rest on unverified assumptions (e.g. the entire timescale separation comes from Assumption 2) and are not directly connected to the measured onsets.

Questions for authors

none, see weaknesses above.

4 comments

We thank the reviewer for the constructive assessment. The review raises four important concerns: (i) memorization of ordinary training records versus injected canaries, (ii) the role of exposure allocation in the ordering reversal, (iii) Markov population learnability, and (iv) the explanatory scope of the theoretical results. We conducted additional controlled experiments addressing the empirical concerns and clarified the interpretation of the theory.

1. Memorization of original training records versus injected canaries

We agree that extracting OOD canaries differs from memorizing ordinary training records. We therefore conducted complementary experiments evaluating original background records and in-distribution grammar-generated targets.

(a) Original background-record extraction.

We evaluated 128 original training records against 128 disjoint held-out records across six grammar runs (three paired seeds, R∈{1,16}R\in\{1,16\}). The stochastic reconstruction protocol uses 50% scattered revelation, eight denoising steps, temperature 0.7, and 32 queries per record. Extraction requires a train-minus-held-out exact-recovery advantage ≥0.10\geq0.10 for three consecutive evaluations, measured every 200 updates.

RRSeedPopulation onsetBackground extraction onset
1101420Censored at 4000
1202360Censored at 4000
1303440Censored at 4000
16101820Censored at 4000
16202800Censored at 4000
16303680Censored at 4000

Table R1. Population-learning and background-record extraction onsets (optimizer updates). Censoring indicates no sustained extraction crossing through step 4000.

Population learning occurs in all six runs, whereas sustained background-record extraction is not observed within the evaluated horizon. This directly addresses the requested comparison with ordinary training records without implying that such records cannot be memorized later.

(b) In-distribution extraction and record-inclusion controls.

We additionally sampled canaries and controls from the same hierarchical grammar as the background, without likelihood filtering. Across three paired seeds, independent Q32 sustained evaluation confirms population-first ordering at R=1R=1 and extraction-first ordering at R=16R=16.

SeedR=1R=1: Population / ExtractionR=16R=16: Population / Extraction
101420 / 3240820 / 280
202360 / Censored at 4000800 / 220
303440 / Censored at 4000680 / 280
OrderingPopulation first (3/3)Extraction first (3/3)

Table R2. In-distribution onsets (optimizer updates, native 20-step grid). Extraction requires three consecutive Q32 advantage crossings; unobserved onsets are right-censored.

To control for record-level reconstruction difficulty, we also compared the same candidate records in paired inclusion/exclusion models using identical fixed queries.

RRMean inclusion effect at step 4000Seed SDSustained ordering
10.28130.1507Population first (3/3)
160.93750.1083Extraction first (3/3)

Table R3. Mean inclusion-associated exact-recovery differences across three paired seeds. This endpoint differs from the canary–control advantage in Table R2.

These results establish that recovery depends on training inclusion of the specific record, with substantially larger effects at high repetition. Since inclusion also changes record exposure, the comparison does not establish an exposure-independent mechanism or calibrated membership leakage.

Together, these experiments distinguish the extraction dynamics of ordinary background records from those of deliberately repeated in-distribution records, providing a more directly comparable assessment of population learning and memorization than the original OOD-canary experiments alone.

  1. Interpretation of the theoretical results

We agree that the theoretical results have distinct explanatory roles. Theorem 1 and Proposition 1 follow from standard conditional cross-entropy minimization and characterize the population and empirical denoising targets. The uniqueness result provides a probabilistic characterization of when a revealed context identifies a training record, without predicting neural extraction onset. The random-feature appendices derive optimization timescales for a squared-loss surrogate under explicit spectral and corruption assumptions. These assumptions are not verified for the trained Transformers, so the resulting separation is conditional and does not establish their empirical neural onset ordering.

2nd, 3rd questions continued in next part.

2. Exposure allocation and empirical scaling laws

We agree that the reversal should be evaluated against exposure allocation. For nn background records, KK canaries repeated RR times, and BB record presentations per update, the per-canary and background sampling probabilities are

p_{\mathrm{bg}}=\frac{n}{n+KR}.$$ Assuming fixed exposure requirements $C_{\mathrm{mem}}$ and $C_{\mathrm{pop}}$, the predicted onsets satisfy $$\hat{\tau}_{\mathrm{mem}}=\frac{C_{\mathrm{mem}}(n+KR)}{BR}, \qquad \hat{\tau}_{\mathrm{pop}}=\frac{C_{\mathrm{pop}}(n+KR)}{Bn},$$ giving $$\frac{\hat{\tau}_{\mathrm{mem}}}{\hat{\tau}_{\mathrm{pop}}} =\frac{C_{\mathrm{mem}}}{C_{\mathrm{pop}}}\frac{n}{R}.$$ Thus, sufficiently large repetition can produce extraction-first ordering through allocation alone. We fitted this model on controlled dataset-size/repetition experiments and evaluated it without refitting. | Evaluation | Correct ordering | Extraction error | Population error | |---|---:|---:|---:| | 30 held-out mixture configurations | 26/30 | 19.7% | 24.6% | | In-distribution, $R=1$ | 3/3 | 154.6% | 56.9% | | In-distribution, $R=16$ | 3/3 | 11.4% | 56.6% | *Table R4. Frozen exposure-model predictions. Errors are mean absolute relative onset errors using the compatible first-crossing endpoint.* The model predicts ordering well, although absolute timing transfers imperfectly. The fitted scaling relationships, approximately $\tau\propto n^{0.94}$ and $\tau\propto R^{-0.47}$, should therefore be interpreted as finite-range empirical fits. Indeed, $$\hat{\tau}_{\mathrm{mem}} =\frac{C_{\mathrm{mem}}}{B}\left(\frac{n}{R}+K\right),$$ implies local logarithmic sensitivities of $n/(n+KR)$ to dataset size and $-n/(n+KR)$ to repetition, providing an exposure-based explanation for sublinear exponents without assuming universal power laws. To test whether record counts fully determine extraction timing, we compared two masking schedules with identical realized per-record presentation histories and equal training budgets (40.96 million tokens). | Seed | Schedule A onset | Schedule B onset | Difference | |---|---:|---:|---:| | 101 | 240 | 280 | 40 | | 202 | 200 | 280 | 80 | | 303 | 200 | 280 | 80 | | Mean | 213.3 | 280.0 | 66.7 | *Table R5. Extraction onsets (optimizer updates) under matched record-presentation histories.* The count-only model predicts 295.6 updates for both schedules, yet schedule A extracts earlier in all three paired seeds. Thus, record-presentation counts alone do not determine extraction timing. Since the masking distributions differ in tail mass and higher-order properties, this establishes a schedule-associated effect without isolating a unique mechanism. 3. Markov sample complexity and neural population learning We agree that the original Markov setting ($V=256,n=128$, sequence length 64) is under-sampled relative to its transition complexity. We therefore conducted additional neural experiments varying vocabulary size at fixed $n=128$, first-order structure, $\alpha=0.30$, and $R=16$, using three seeds per setting. | Vocabulary | Q32 extraction onset | Population $G\geq0.50$ onset | Population outcome | |---|---|---|---| | $V=16$ | 240 / 280 / 280 | 1320 / 1680 / 1980 | Transient crossing; final $G=-1.1316/-0.9489/-1.0655$ | | $V=64$ | 280 / 280 / 240 | Censored beyond 12000 (3/3) | Maximum $G=0.0107/-0.0012/-0.0013$ | | $V=256$ | Historical setting | No meaningful observed population-learning regime | Original under-sampled setting | *Table R6. Neural Markov results. The historical $V=256$ setting is not a new Q32 evaluation.* At $V=16$, population gain crosses the threshold after extraction but subsequently deteriorates, indicating transient rather than stable generalization. At $V=64$, population learning remains unobserved through 12,000 updates. A structured first-order Dirichlet estimator additionally achieves held-out gain $G=0.698$–$0.994$ in six simpler configurations, compared with $0.157$ in the historical $V=256,n=128$ setting. This demonstrates statistical learnability under the correct structural prior, not corresponding neural convergence. These findings support the importance of population complexity and data coverage. The neural experiments vary vocabulary at fixed dataset size; they do not constitute a complete dataset-size–complexity phase diagram. Accordingly, the original Markov result is interpreted within its limited-data regime rather than as a universal absence of population learning. Together, these results provide a controlled empirical characterization of how training-mixture allocation, record inclusion, masking-dependent supervision, and data complexity shape observable memorization and population-learning dynamics in masked diffusion language models.

I appreciate the new numerical experiments that indeed bring additional value to the work. However, I still find the theory part weak. For this reason, I won't raise my recommendation.

Thank you for recognizing the value of the additional experiments. To further address your theoretical concerns, we have made a focused extension to the existing conditional analysis. This clarifies how record repetition and masking jointly affect the training objective and derives the corresponding learning time in a simplified categorical model, without relying on assumed exposure thresholds or random-feature spectral assumptions. The results appear in Theorems 1–2 (pp. 4–5, lines 197–238) and Appendix A.4 (pp. 15–17, lines 761–892), with additions marked in blue.

For a predictor without explicit mask-rate input, Theorem 1 decomposes the inverse-rate masked cross-entropy objective, Eq. (3), into conditional prediction problems. It separates the weight of a record from the weight of its revealed context. Let SS be the revealed positions, excluding the scored masked token, in a sequence of length LL. The context weight is

κπ(S)=Et∼π[tL−∣S∣−1(1−t)∣S∣].\kappa_\pi(S)=\mathbb E_{t\sim\pi}\big[t^{L-|S|-1}(1-t)^{|S|}\big].

This gives an exact reason that matching record presentations need not match the supervised problems seen during training. The weighted probabilities of all mask subsets agree if and only if the mask-rate moments agree through degree L−1L-1. Matching the mean and variance alone therefore does not determine these weights when L≥4L \geq 4. This result concerns the inverse-rate objective. Appendix A.4 derives separate coefficients for the normalized minibatch loss (p. 16, lines 820–840). For predictors that also use the mask-rate label, the full decomposition retains that label explicitly.

Using the exact moment-matching weights corresponding to the displayed rounded schedule coefficients, their third moments also agree. Their fourth moments differ by 133/16000133/16000 (pp. 15–16, lines 807–819). This proves that their ideal conditional-supervision weights differ. It does not single out the fourth moment as the cause of earlier extraction.

We also solve gradient flow for the expected cross-entropy objective in a restricted model with separate logits for each context and uniform initialization. Consider a context that identifies one record. Its confidence-crossing time is an explicit function of the chosen confidence, divided by the record weight and κπ(S)\kappa_\pi(S). The exposure requirement follows from the stated dynamics. It is not a fitted constant or an assumed spectral gap. This calculation shows how record frequency and masking jointly determine learning speed in that model.

These results extend the existing conditional analysis. We credit the standard log-score argument to Gneiting and Raftery (2007) and the clean-data conditional view to Ou et al. (2025). Our contribution is the explicit separation of record and context weights, together with the controlled experiments that test whether record counts suffice.

The connection to the experiments has a precise scope. The frozen exposure baseline predicts 26/30 held-out orderings. Under identical sample histories, the masking intervention changes extraction onset in all three paired seeds. The theoretical analysis explains why these findings are compatible. It does not predict the observed update differences or the Transformer extraction threshold. Theorems 1–2 do not depend on the random-feature assumptions in Appendices B–C. Those assumptions remain unverified for the trained Transformers, so the random-feature analysis provides conditional motivation.

The updated account also preserves your distinction between ordinary and repeated records. Sustained background-record extraction remains unobserved through step 4000 in all six grammar runs, while repeated in-distribution targets show the reversal. Predicting these neural trajectories from first principles remains future work. The updated manuscript combines the controlled empirical study with exact statements about the objective factors its interventions change. We ask that these results be considered alongside the additional experiments you found valuable.

References

  1. Gneiting, T., and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. JASA, 102(477), 359–378.
  2. Ou, J., et al. (2025). Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. ICLR.
Reviewer_cYjqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when population learning and train-specific extraction emerge during masked diffusion language model training. Using controlled Markov and hierarchical-grammar populations, trained canaries, and never-trained matched controls, it demonstrates that increasing repetition can reverse the ordering between population-denoising improvement and canary extraction. Dataset-size and repetition interventions examine the roles of individual-record frequency and aggregate canary mass. Additional experiments show that mask-rate distributions with matched first two moments can produce different extraction onsets, while pretrained LLaDA and MDLM reproduce the directional repetition effect. The theoretical analysis characterizes optimal conditionals and context uniqueness, while explicitly separating these results from finite-model extraction dynamics.

Critical strengths

A clear and reproducible-within-study reversal of the measured learning–extraction ordering. In the hierarchical setting, population learning precedes extraction at low repetition, whereas extraction appears first at higher repetition. The direction is consistent across the three seeds and remains unchanged under the lower population-learning threshold. This provides concrete evidence against treating the ordering as invariant across the tested training regimes. Useful controls and appropriately qualified conclusions. Never-trained matched controls help distinguish train-specific recovery from reconstruction based on population structure. The interventions separating individual-record frequency from aggregate canary mass, together with the matched-moment masking comparison, go beyond a single repetition sweep. The paper also transparently reports the limitations of pretrained-model transfer and the lack of incremental predictive value from context uniqueness.

Critical weaknesses

The contribution beyond exposure allocation and training-mixture competition remains insufficiently established. The hierarchical canaries are out of distribution relative to the grammar population. Increasing their repetition both increases individual-canary exposure and reduces the fraction of background examples processed per optimization step. A simple baseline in which extraction requires a roughly fixed number of canary presentations and population learning requires a roughly fixed number of background presentations already predicts an onset ratio proportional to 1−fcanpcan=nR\frac{1-f_{\mathrm{can}}}{p_{\mathrm{can}}}=\frac{n}{R}. It therefore predicts both organization by normalized repetition and a possible ordering reversal without invoking a diffusion-specific mechanism. The decoupling interventions establish that neither sampling frequency nor aggregate mass alone is sufficient, but do not quantify how much of the observed ordering is explained by a joint exposure-allocation baseline with shared exposure thresholds. Although the paper acknowledges mixture competition, quantifying what remains beyond this explanation is important for assessing its significance. The pretrained experiments primarily reproduce the repetition–extraction trend rather than the central ordering reversal.

The primary extraction onset needs a more fully specified and better-calibrated measurement protocol. The PDF does not fully specify the conditioning geometry, reveal fraction, decoding procedure, query budget, and scored positions for the main controlled experiments. These choices directly determine when extraction first becomes observable. Moreover, population onset requires three consecutive successful evaluations, whereas extraction onset uses a first crossing. Appendix F.2 shows that changing the evaluation/query sequence can alter some seed-level schedule comparisons, making query stability particularly relevant for smaller onset differences and regime boundaries. Fixed-query or repeated-query sensitivity checks, together with sustained-crossing results, would make the central timing comparisons easier to assess. This concern does not by itself negate the large separation in Table 1, but limits confidence in precise onset estimates and finer comparisons.

Questions for authors

Exposure-allocation baseline: Using the recorded presentation counts, how well does a baseline based on individual-canary exposure and aggregate background exposure predict the onset times across the joint dataset-size/repetition grid and the decoupling experiments? Which systematic effects remain unexplained by this baseline? In-distribution canaries: Does the ordering reversal persist when canaries and matched controls are sampled from the same hierarchical population as the background, rather than being random out-of-distribution strings? A comparison of representative low- and high-repetition conditions would help clarify the scope and significance of the result. Extraction measurement: Please specify the reveal fraction, conditioning geometry, decoding steps, queries per canary, scored positions, and whether query masks are reused across checkpoints in the primary controlled experiments. Do the key orderings and finer onset comparisons remain stable under repeated or fixed queries and a sustained-crossing criterion aligned with the population-onset definition?

3 comments

We thank the reviewer for the constructive feedback. We addressed all the questions through a held-out quantitative exposure baseline, exposure-matched masking comparisons, in-distribution grammar experiments, paired record-inclusion controls, and independent-query evaluations. Together, the results support a two-factor empirical account: mixture allocation strongly predicts onset ordering, while the training mask distribution affects extraction timing beyond record-presentation counts.

  1. Quantifying the exposure-allocation explanation

Let nn be the number of background records, KK the number of distinct canaries, RR their repetition factor, and BB the number of record presentations per optimizer update. An individual canary is sampled with probability pcan=R/(n+KR)p_{\mathrm{can}}=R/(n+KR), while the probability of sampling any background record is pbg=n/(n+KR)p_{\mathrm{bg}}=n/(n+KR). Under the reviewer's fixed-exposure-threshold hypothesis, if extraction requires CmemC_{\mathrm{mem}} presentations of one canary and population learning requires CpopC_{\mathrm{pop}} aggregate background presentations, the predicted onset times are

\widehat{\tau}_{\mathrm{pop}}=\frac{C_{\mathrm{pop}}(n+KR)}{Bn}.$$ Consequently, $$\frac{\widehat{\tau}_{\mathrm{mem}}}{\widehat{\tau}_{\mathrm{pop}}}=\frac{C_{\mathrm{mem}}}{C_{\mathrm{pop}}}\frac{n}{R}.$$ This derivation shows why increasing repetition can reverse the onset ordering **without** assuming an intrinsic diffusion-specific memorization clock. We fitted the two-clock model to the controlled dataset-size/repetition configurations and evaluated it **without refitting** on 30 additional held-out mixture configurations and six in-distribution grammar runs. | Evaluation set | Correct ordering | Extraction onset error | Population onset error | |:--|--:|--:|--:| | Held-out dataset-mixture configurations | **26/30** | 19.7% | 24.6% | | In-distribution grammar, $R=1$ | **3/3** | 154.6% | 56.9% | | In-distribution grammar, $R=16$ | **3/3** | 11.4% | 56.6% | Table 1. Frozen-model transfer; errors are mean absolute relative onset errors. Only the original eight-query, first-crossing extraction endpoint, which matches the model calibration, is used. Thus, allocation successfully predicts the qualitative reversal, but absolute timing transfers imperfectly, especially for low-repetition in-distribution extraction. We do not interpret prediction residuals alone as evidence of a new mechanism. *Does masking matter when record exposure is matched?* We additionally compared two training masking schedules with identical realized per-record presentation histories at every common update, an equal 40.96-million-token training budget, and different mask distributions. | Paired seed | Schedule A onset | Schedule B onset | Difference (B − A) | |:--|--:|--:|--:| | 101 | 240 | 280 | 40 | | 202 | 200 | 280 | 80 | | 303 | 200 | 280 | 80 | | **Mean** | **213.3** | **280.0** | **66.7** | Table 2. Extraction first-crossing times in optimizer updates for exposure-matched masking schedules. The count-only model predicts the same onset (295.6 updates) for both schedules, whereas A extracts earlier in each paired seed. Matching record presentations therefore does not suffice to match extraction timing across these masking distributions. The schedules differ in tail mass and higher-order properties, so this contrast establishes a **mask-schedule-associated** effect, not the unique contribution of a particular mask statistic.

Continued..

  1. In-distribution reversal and record-specific recovery

To address the OOD-canary concern, we trained six models using canaries and controls independently sampled from the same hierarchical grammar as the background data (n=128n=128, K=8K=8, R∈{1,16}R\in\{1,16\}; three paired seeds). No likelihood-based filtering was applied, and identity-disjointness was verified.

SeedRRPopulation onsetExtraction (Q8)Extraction (Q32 sustained)
10114207803240
20213601000Censored at 4000
30314401320Censored at 4000
10116820280280
20216800240220
30316680280280

Table 3. Onsets in optimizer updates on the native 20-step evaluation grid. Q8 uses first crossing; independent Q32 uses three consecutive crossings. Censored events are not assigned finite onsets. These are three paired training seeds.

The reversal persists across all three seeds: at R=1R=1, population learning precedes sustained extraction (late or censored), whereas at R=16R=16, extraction consistently occurs first. Thus, the effect extends beyond OOD canaries and remains under the more conservative evaluation.

To control for inherent reconstruction difficulty, we also trained paired record-inclusion/exclusion models and evaluated the same candidate records under identical fixed queries.

RepetitionMean inclusion effect at step 4000Seed SDQ32 sustained ordering
R=1R=10.28130.1507Population first (3/3)
R=16R=160.93750.1083Extraction first (3/3)

Table 4. Inclusion-associated exact-recovery differences across three paired seeds per repetition level.

The results demonstrate record-specific recovery effects, substantially stronger at high repetition. Since inclusion also changes record exposure, this is not an exposure-independent effect or a calibrated membership-inference result. As a complementary diagnostic, the R=16R=16 Bayes canary–control completion differences (+0.0234,−0.0234,−0.0469+0.0234,-0.0234,-0.0469) are smaller in magnitude than the measured trained advantages (+0.2852,+0.1250,+0.3086+0.2852,+0.1250,+0.3086). These quantities use different sampling procedures and are not treated as a calibrated causal correction.

3. Extraction protocol and onset robustness

The primary experiments use 64-token sequences with 32 randomly scattered revealed positions (50%), eight progressive denoising steps, temperature 0.7, and eight stochastic queries per canary. Exact recovery requires matching all 64 tokens; since revealed positions are copied, success depends on reconstructing the 32 masked positions. Extraction onset is the first canary-minus-control exact-recovery advantage crossing of 0.10. Query masks are fixed across checkpoints. Population onset requires three consecutive Bayes-referenced validation evaluations with G≥0.50G\geq0.50.

For robustness, we evaluated three independently generated scattered-mask panels with 8/16/32 nested query budgets, keeping masks and decoding streams fixed across checkpoints. Q32 extraction additionally requires three consecutive threshold crossings. Full panel sensitivity was evaluated for representative runs; the primary independent Q32 panel covers all six in-distribution runs.

The principal ordering is stable, but finer onset estimates are protocol-sensitive. For example, at R=1R=1, seed 101, Q8 extraction occurs at step 780, whereas Q32 sustained onsets across three panels are 3240, 3740, and censored at 4000. At R=16R=16, the corresponding onsets are 280, 280, and 260. Hence, the low-/high-repetition ordering remains consistent, although precise low-repetition extraction times vary.

The original OOD low-/high-repetition ordering also persists across 80 tested threshold, persistence, and evaluation-grid configurations; two near-boundary configurations change classification. Source-fidelity replays reproduce five historical metrics at the checked checkpoints. We distinguish the native 20-step grid from the common 40-step comparison grid and correct the Table 7 seed-202 transcription error (260 rather than 280 in the relevant setting).

Thanks for the extra experiments. The exposure baseline does explain most of the ordering. What changed my mind is that even when two runs see the exact same samples at every step, just changing the masking distribution still changes how fast they hit the extraction threshold. That's a neat control others can reuse to compare different training setups. The reversal also holds up on in-distribution data and under the stricter extraction check. We still don't know if this masking effect holds on natural text when held-out denoising is matched, but overall, I think there's enough here to raise my score from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.