- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper contains experiments showcasing how canary repetition changes the order of population learning and memorization in masked diffusion models. Despite good experiments, training-mixture effects and the limited explanatory power of the theoretical results lead to a weak rejection assessment.
Critical strengths
The paper establishes experimental results using masked diffusion language models trained on synthetic distributions as well as pretrained models. Using their measures for memorization and population learning they provide examples of one happening before the other in training and vice versa, varying canary repetition . The authors highlight that this differs from previous results obtained for continuous diffusion models (Bonnaire et al., 2025) where population learning happens before memorization. This comparison could benefit from the use of a more similar measure of memorization.
Further, they find a promising driver for this regime change, namely the sampling probability of a canary. They investigate this effect further by varying the sampling probability of all canaries or the sampling probability of a canary while holding the other one fixed. Both factors lead to a regime change.
Thorough robustness checks of the experiments are performed in appendix D. They may benefit from additional seeds.
Critical weaknesses
The proofs of the theoretical results look correct. But the theoretical results posed in the main part are rather elementary. The results from appendix B and C are more involved but rely on assumptions as mentioned by the authors, they consider them as motivation for separated optimization time scales. Maybe these assumptions could be resolved by future work or verified empirically.
The main weakness, together with the fact that the main theorems are elementary, is that they exhibit convincing empirical evidence showing that a memorization-before-population-learning mechanism exists, but the underlying mechanism is not really clear or studied in theory.
To be more precise: they hand-pick several thresholds, like 0.5, 0.1 etc in the definitions in Sections 3, and the regime shift they see I think it depend on those choices: probably the general mechanism would be true but the shift would appear for different levels of R. The general relation though is still completely unclear.
Some relevant terms could be defined more explicitly: geometry, normalized edit distance, , , $\mathcal L_{\text{val}}(s) $, Enc. In appendix B a notation table could be helpful.
In the way memorization and population learning are defined in this paper it seems reasonable that at some point when the canary repetition since the canaries will then dominate the empirical distribution (especially for ). This effect seems to get acknowledged by the authors in Section 7: "The decoupling experiments therefore characterize competition induced by the training mixture rather than an intrinsic architecture-specific memorization clock."
Questions for authors
How is in Appendix B defined?
4 comments
4. Interpretation of the theoretical results
We agree that the theoretical results have distinct roles. Theorem 1 and Proposition 1 characterize population and empirical denoising conditionals through standard cross-entropy minimization. The context-uniqueness result characterizes when revealed tokens identify a training record, without predicting neural extraction onset.
Appendices B and C derive optimization timescales for a squared-loss random-feature surrogate under explicit spectral and corruption assumptions. These assumptions are not verified for the trained Transformers; consequently, the surrogate provides conditional analytical motivation rather than a proof of the observed neural ordering.
The central contribution is therefore the controlled empirical characterization, supported by quantitative exposure-model predictions, matched-exposure masking experiments, and record-inclusion controls.
5. Notation and definition of
We clarify the requested notation:
| Symbol | Definition |
|---|---|
| Conditioning geometry | Arrangement of revealed and masked positions; controlled evaluations use 32/64 scattered revealed tokens |
| Normalized edit distance | Token-level edit distance divided by sequence length |
| Chance-reference denoising loss | |
| Bayes-optimal population denoising loss | |
| Held-out validation denoising loss at update | |
| Individual-canary sampling probability and aggregate canary mass |
The population gain is
\mathrm{Bayes}}}.$$ In Appendix B, \$\mathrm{Enc} \$ denotes the abstract representation map producing centered clean and corrupted context vectors: $$z^\nu=\mathrm{Enc}(x^\nu)\in\mathbb{R}^d, \qquad z_t^\nu=\mathrm{Enc}(x_t^\nu)\in\mathbb{R}^d.$$ It is distinct from the random-feature transformation $$\phi_t(z_t)=\sigma\left(\frac{Wz_t}{\sqrt d}\right).$$ The analysis does not prescribe a unique token-to-vector implementation of $\mathrm{Enc}$ or identify it with the trained Transformer encoder; its conclusions remain conditional on the surrogate assumptions. ### 6. Conclusion **The additional results provide a quantitative explanation of the reversal through exposure allocation, demonstrate masking-dependent extraction timing under matched record presentations, and establish record-specific inclusion effects for in-distribution targets. The principal ordering remains robust under the tested evaluation criteria, while exact onset times depend on the operational definitions.**Thank you for the additional experiments. The new experiments are convincing and strengthen the empirical contribution, particularly the additional robustness checks.
My main concern remains the theoretical part. While the exposure-based model helps explain the observed reversal, it relies on assumed thresholds. Also, the random-feature analysis still depends on assumptions that are not verified for the models considered.
So, I would be willing to maintain my score for now.
Thank you for recognizing that the additional experiments strengthen the empirical contribution. To further address your theoretical concerns, we have made a focused extension to the existing conditional analysis. This derives learning times from stated categorical dynamics rather than assuming exposure thresholds, without relying on random-feature spectral assumptions. The results are in Theorems 1–2 (pp. 4–5, lines 197–238), and the proofs are in Appendix A.4 (pp. 15–17, lines 761–892). All additions are blue.
The analysis separates two claims. The exposure model fitted to the experiments remains a quantitative baseline. Its threshold constants are estimated from data. Theorem 2 solves gradient flow for the expected cross-entropy objective with separate categorical logits for each context and no explicit mask-rate input. It starts from uniform predictions and assumes a context with a deterministic target token.
Consider a positive-weight context that identifies one record, with sampling probability . Let be its revealed positions, excluding the scored masked token, and let be sequence length. Under the inverse-rate objective, Eq. (3), the time to reach confidence , with , is
Here is the vocabulary size. Specifically,
The function follows from the differential equation (p. 16, lines 841–863). It is not an assumed exposure requirement. No random-feature spectrum enters the proof. Appendix A.4 also derives sufficient conditions for ordering across an interval of different criteria (p. 17, lines 864–885).
This makes threshold dependence explicit. In the theorem's illustrative record-specific and pooled-background cells, equal mask geometry and a common confidence criterion give a hitting-time ratio of at every interior confidence level. Different criteria can move the crossover; the appendix gives conditions for an ordering to hold across an interval of criteria. Our empirical population and extraction measures use different criteria. Their observed low-/high-repetition contrast is supported separately by threshold and sustained-extraction sensitivity experiments.
The accompanying objective decomposition explains the masking factor . Even when two runs receive the same records, they can assign different weights to the prediction problems defined by their masks. Equality of the complete weighted mask law requires equality of moments through degree . Thus, matching only the first two moments does not generally match these weights. This statement concerns the inverse-rate objective and uses no assumed Transformer spectrum. Appendix A.4 treats the normalized minibatch loss separately (p. 16, lines 820–840).
The scope is explicit. Separate context logits do not capture parameter sharing in a Transformer, and token confidence is different from the paper's canary-minus-control exact-recovery advantage. The theorem gives an exact instance of joint exposure and masking dependence. The matched-presentation experiments establish the corresponding timing difference in the trained neural models. Predicting their numerical onset or crossover repetition remains open.
We ground this extension in the conditional view of Ou et al. (2025). We also distinguish our change of mask-rate distribution from the compensated time reparameterization studied by Sahoo et al. (2024). Together, the updated theory and experiments clarify that record allocation explains much of the observed ordering, while mask-conditioned supervision provides an additional factor not captured by record counts alone.
References
- Ou, J., et al. (2025). Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. ICLR.
- Sahoo, S. S., et al. (2024). Simple and Effective Masked Diffusion Language Models. NeurIPS.
We thank the reviewer for the constructive feedback. We conducted additional experiments supporting precise explanation: training-mixture allocation predicts much of the onset ordering, while masking distributions influence extraction timing beyond record-presentation counts.
We agree that increasing canary repetition changes the training mixture and can induce extraction-first behavior without an intrinsic architecture-specific memorization clock.
For n background records and K canaries repeated R times, individual-canary sampling probability and aggregate canary mass are $ p_{\mathrm{can}}=\frac{R}{n+KR}, \qquad f_{\mathrm{can}}=\frac{KR}{n+KR}. $
Under fixed exposure thresholds Cmem and Cpop, with B record presentations per update, the two-clock model predicts
$ \frac{\widehat{\tau}{\mathrm{mem}}} {\widehat{\tau}{\mathrm{pop}}}
\frac{C_{\mathrm{mem}}}{C_{\mathrm{pop}}} \frac{n}{R}. $
Thus, increasing R can produce extraction-first ordering through exposure allocation alone, including in the large-R limit. We fitted this model on controlled dataset-size/repetition experiments and evaluated it without refitting.
Table R1. Frozen exposure-model predictions. Errors are mean absolute relative onset errors under the original first-crossing criterion.
To test whether presentation counts fully explain extraction timing, we compared two masking schedules with identical realized per-record presentation histories and equal training budgets (40.96 million tokens), but different corruption distributions.
Table R2. Extraction onsets (optimizer updates) under matched record-presentation histories.
The count-only model predicts 295.6 updates for both schedules, yet schedule A extracts earlier in all three paired seeds. Thus, record-presentation counts alone do not determine extraction timing across these schedules. Since the masking distributions differ in their tails and higher-order properties, the comparison establishes a schedule-associated effect without isolating a unique mechanism. Population onsets remain censored beyond 20,000 updates in these conditions.
2. Robustness to onset thresholds
We agree that operational thresholds influence measured onset times. The original criteria use Bayes-referenced population gain G≥0.50 for three consecutive evaluations and canary-minus-control extraction advantage ≥0.10 at first crossing.
The original low-/high-repetition ordering persists across 80 tested threshold, persistence, and evaluation-grid configurations, although two near-boundary configurations elsewhere change classification.
We additionally evaluated independent Q32 reconstruction queries requiring three consecutive extraction-threshold crossings.
Table R3. In-distribution grammar onsets (optimizer updates, native 20-step grid). Q8 uses first crossing; Q32 uses an independent fixed-query panel and three consecutive crossings.
The ordering persists across all paired seeds: population learning precedes sustained extraction at R=1, while extraction occurs first at R=16. However, precise low-repetition extraction onsets are sensitive to query and persistence criteria. Thus, the qualitative contrast is robust under tested conditions, whereas numerical onset times and transition boundaries are operationally dependent.
We additionally evaluated canaries and controls sampled from the same hierarchical grammar as the background distribution. To control for record-specific reconstruction difficulty, paired inclusion/exclusion models evaluated identical candidate records using fixed queries.
Table R4. Paired inclusion-associated exact-recovery differences.
Recovery depends on whether the specific record was included in training, with substantially larger effects at high repetition. Since inclusion changes record exposure, this does not establish an exposure-independent mechanism or a calibrated membership-inference attack.