acceptodds
Under review as a conference paper at ICLR 2027

Generalization in Continuous Flows for Categorical Data

Abstract

Flow matching regresses a velocity field onto a finite training set, and the minimizer of that objective transports noise exactly onto it. Trained models nonetheless generalize, and the cause has been attributed in turn to the stochastic training target, to the sampler, and to the model. These explanations were developed for continuous data such as images, whose distributions are known to concentrate near low-dimensional manifolds. Flow matching has recently been extended to categorical data such as text and protein sequences, whose support is a finite set of points with no such structure, and it is unclear whether and why it generalizes there. We study this question on discrete supports such as permutations and bracket languages. On these supports we derive the optimal denoiser in closed form and show that the flow map it induces is a lookup table from noise to training points. A one-step network regressed onto it is trained on an exact target and never integrates an ODE, yet it still produces valid samples absent from the training set. We measure memorization across training-set size, model depth, training steps and sampling steps, and find that at matched architecture and compute, flow matching generalizes at less data than masked diffusion and less than autoregression.

Primary area:
generative models
Keywords:
flow matching, memorization, discrete generative models, continuous diffusion, flow-maps

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_2cv2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides the analysis on the generalization mechanics of flow matching on discrete/categorical data by systematically removing the two candidate explanations. The authors construct the optimal denoiser and the optimal flow map with rigorous derivation and then design the related experiments. Through solid experiments with matched architecture, parameters and training budget, the authors conclude that the reason for the generalization of flow matching is neither the stochastic target nor the sampler.

Critical strengths
  1. The paper is well-structured and easy to follow. The authors constructed the closed-form optimal denoiser and also the optimal flow map with rigorous derivation.

  2. Based on these formulations, the authors conducted solid experiments to isolate the reasons of the generalization.

Critical weaknesses
  1. The authors claimed in Sec 4.3 that FLM generalize at training-set size ∣D∣=104|\mathcal{D}|=10^4 on every distribution. However, as shown in Table 6, FLM under ∣D∣=104|\mathcal{D}|=10^4 and RHM distribution achieves 0% memorization score but only get 23% validation. The generalization should be evaludated based both on the validity and memorization. If most generated samples from FLM are invalid, to claim the generalization property of FLM is unreasonable.

  2. The ablation study on the impact of different rollout integration steps on model performance is insufficient. The authors chose the 8 integration steps for all the experiments based on an empirical result that 8 integration steps can achieve over 97% matching rate. However, it remains unclear whether the potential mismatched training points would affect the training procedure and the generalization. While the experiments of larger integration steps with ∣D∣=106|\mathcal{D}|=10^6 are indeed prohibitive, this concern can be addressed by conducting ablations on varying integration steps (both fewer or larger, e.g., 4 or 16 steps) at smaller training-set size (e.g., ∣D∣≤104|\mathcal{D}| \le 10^4).

Questions for authors
  1. I understand that it requires two assumptions (the limit exists and the unique maximizer) to prove Proposition 3.2, but would these assumptions holds in actual experiments?

  2. The inference compute differs among the three models. In Appendix C, for example, FLM takes 128 Euler steps, while MDLM uses L=16L = 16 steps. Could the authors clarify this point?

5 comments

We thank the reviewer for the careful and positive review, and for finding our experiments well designed.


W1: Evaluating generalization.

We agree that generalization should be evaluated on both validity and memorization. To incorporate this, we report the fraction of samples that are both valid and novel, validity × (1 − memorization). To further strengthen the evaluation, we have also added a fourth distribution, palindromes of length L=16L = 16 over V=16V = 16 tokens, trained with the recipe of Fig. 6.

Validity / memorization (valid and novel) (%), ∣D∣=104|D| = 10^4.

PermutationDyckRHMPalindrome
AR100.0 / 100.0 (0.0)100.0 / 99.8 (0.2)99.6 / 100.0 (0.0)100.0 / 99.9 (0.1)
MDLM96.4 / 0.8 (95.6)71.0 / 10.6 (63.5)22.5 / 46.5 (12.0)100.0 / 26.3 (73.7)
FLM100.0 / 0.0 (100.0)97.3 / 0.0 (97.3)23.0 / 0.0 (23.0)100.0 / 0.0 (100.0)

Under this combined metric, FLM produces the most valid and novel samples on every distribution, including RHM. We therefore believe the claim that FLM generalizes is supported, and we will qualify it in Sec. 4.3 accordingly.

The low validity of FLM on RHM is due to undertraining. With longer training (1M steps instead of 200k), FLM reaches 90.1% validity on RHM at larger training sets, while staying at 0% memorization. We hypothesize that FLM learns RHM more slowly because it noises every position, which weakens the training signal for the long-range dependency of the grammar. We will report the longer RHM runs and the palindromes in the paper.


W2: Integration steps for FLM-X*.

One-step generalization is stable as target accuracy increases. We first note that a less accurate target is still a valid sample, since the target is the training point with the largest posterior weight at the end of integration. Therefore, integration error only changes which training point a noise sample is assigned to. As suggested, we ran the ablation on Dyck at ∣D∣=105|D| = 10^5, where target error is largest (Tab. 8), training FLM-X* with targets from 4, 8 and 16 Euler steps.

Dyck (L=10L = 10), ∣D∣=105|D| = 10^5, 16,384 samples (%).

StepsTarget agreement with 1,024 stepsValidityMemorization
491.737.42.0
895.837.81.9
1697.537.72.2

Validity and memorization stay the same within sampling noise as target agreement increases, so the mismatched targets do not affect generalization. We note that while the validity of FLM-X* is low, its valid samples cover the distribution well, with an effective support of 98% on permutation (Reviewer ZwbK, W1). With more samples, the memorization of FLM-X* here is 1.9% rather than the 0.0% in Tab. 3.


Q1: Assumptions of Proposition 3.2.

We thank the reviewer for this question. Both assumptions hold in all our experiments. We checked them numerically on the 1,024 fixed noise samples of Fig. 5, integrating with 1,024 Euler steps up to t=0.999t = 0.999.

Smallest gap between the two largest inner products ⟨xt,x(i)⟩\langle x_t, x^{(i)} \rangle at t=0.999t = 0.999.

∥D∥\|D\|PermutationDyck
10210^23.993.99
10310^32.001.99
10410^41.991.99
10510^51.991.99

The gap stays well above zero for every sample, so the maximizer is unique (Assumption 2). Moreover, every trajectory converges to a training point, so the limit exists (Assumption 1). We note that one can construct examples that violate Assumption 2, e.g. noise on the diagonal between (1,0)(1, 0) and (0,1)(0, 1), but such instances have probability zero. We will include this discussion after Prop. 3.2.


Q2: Inference compute across paradigms.

We thank the reviewer for raising this point. MDLM's step count is set by its sampler. We use a uniform-order sampler that unmasks one position per step, so it takes exactly L=16L = 16 steps and avoids the factorization error of unmasking several tokens at once. To check that a larger budget does not change our conclusions, we also sampled the same checkpoints with the standard ancestral sampler at up to 256 steps.

MDLM on permutation (L=16L = 16), validity / memorization (%).

∥D∥\|D\|Uniform, S=16S = 16Ancestral, S=16S = 16Ancestral, S=128S = 128Ancestral, S=256S = 256
10210^299.9 / 100.047.3 / 99.890.2 / 100.095.2 / 100.0
10310^382.6 / 99.940.3 / 97.877.7 / 99.582.3 / 99.8
10410^496.5 / 0.249.3 / 0.487.1 / 0.291.1 / 0.3
10510^5100.0 / 0.030.3 / 0.085.4 / 0.093.4 / 0.0
10610^6100.0 / 0.031.9 / 0.086.5 / 0.091.8 / 0.0

The memorization threshold remains the same under every sampler. More ancestral steps raise validity, but it stays below the uniform sampler at 16 steps. We will state the step counts of each paradigm in App. C and add this comparison.


We hope these responses address the reviewer's concerns, and would be grateful if they could be considered in the assessment.

Thanks for the detailed rebuttal. I have carefully read the other reviews and the authors’ responses and my concerns have been addressed.

However, I'm confused about the argument "under a one-hot embedding, every point between two training sequences lies off the support", as stated in the Rebuttal to Reviewer 6rN7. While interpolation between distinct one-hot sequences does leave the discrete support, the interpolation between continuous data is also not guaranteed to remain on the manifold, please see [R1]. Therefore, this observation seems insufficient to claim the difference between one-hot and continuous data. Could the authors clarify this?

Also, I could not locate these additions, discussions and rebuttals with other reviewers in the currently available revised manuscript and I would suggest adding them to the revision.

Overall, I think the rebuttal improves the paper's clarity and I will maintain my current score.

[R1] Kacper Kapusniak et al, "Metric Flow Matching for Smooth Interpolations on the Data Manifold", NeurIPS 2024.

We thank the reviewer for the positive assessment and for engaging in further discussion. We will cite the shared reference in our paper. We address the concern below.

  1. [1] shows that straight paths can lie outside the data manifold. However, we believe it in fact supports our claim. [1] builds on the manifold hypothesis, under which data such as images, while embedded in a high-dimensional pixel space, lie on a curved low-dimensional manifold.
  2. [2] measures this directly, finding that natural image datasets have a low intrinsic dimension, and that such low-dimensional data is easier for networks to learn. [2] further notes that this manifold is locally connected, i.e. each natural image is surrounded by other highly similar images. This is why [1] can construct paths between data points that stay on the manifold.
  3. Under a one-hot embedding, the support is a finite set of isolated points, with no locally connected structure, and no continuous path between two distinct sequences stays on it. This matters because prior explanations of why diffusion models generalize rely on this local structure, e.g. by stitching together local patches of training images [3].

We hope this addresses the reviewer's question. We are finalizing the revised manuscript with the new discussions and results, and will upload it shortly.


[1] Kapuśniak et al. Metric flow matching for smooth interpolations on the data manifold, '24

[2] Pope et al. The intrinsic dimension of images and its impact on learning, '21

[3] Kamb & Ganguli. An analytic theory of creativity in convolutional diffusion models, '24

Reviewer_ZwbKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies memorization and novel valid generation in continuous flow models on synthetic categorical distributions: permutations, Dyck words, and a random hierarchy model. It compares standard flow matching with models trained against the empirical optimal denoiser and a numerically approximated optimal flow map. The one-step flow-map model produces valid unseen samples despite being trained on targets drawn exclusively from the training set. The paper also compares memorization thresholds across flow matching, masked diffusion, and autoregressive models under a shared training recipe, reporting lower dataset-size thresholds for flow matching.

Critical strengths
  • The controlled setting enables precise measurements of validity and memorization. Both constraint satisfaction and training-set membership can be checked exactly on the chosen supports. This avoids ambiguities associated with perceptual similarity thresholds and provides a useful setting for investigating memorization.

  • The one-step experiment provides an informative intervention. Novel valid generation without inference-time ODE integration narrows the possible explanations for generalization. The random-lookup control further indicates that the structure of the noise-to-training-example assignment matters. This is the paper’s most interesting contribution, although its interpretation requires additional validation.

Critical weaknesses
  • The evaluation establishes novel valid generation more directly than distributional generalization. The target distributions are uniform over known supports, but high validity and low memorization do not establish coverage or uniformity. A generator concentrated on a small subset of valid unseen sequences could perform well on both metrics. Figure 4 checks nearest-training-example distances, but matching this histogram does not establish distributional fidelity. Additional diagnostics, such as permutation position marginals, Dyck shape and depth statistics, and RHM production-rule frequencies, would strengthen the central claims.

  • The benchmarks are insufficient to establish the distinctiveness and breadth of the reported advantage. The paper includes AR and MDLM baselines, but FLM-X* is not compared with related one-step generators or alternative structured noise-to-example targets under comparable conditions. Such comparisons would clarify whether its behavior is specific to the empirical flow-map construction or occurs more broadly in one-step learning. This omission does not invalidate the experiment as an intervention, but limits what can be concluded about the construction itself.

  • The headline 10×/100× data-efficiency claim also requires qualification. Matching architecture and training steps gives a useful comparison under a shared recipe, but does not establish a robust paradigm-level ordering. The AR learning-rate sweep and token-dropout experiments address this concern only partially. More importantly, the comparison does not consistently demonstrate comparable generation quality. The paper should distinguish memorization-threshold differences from data efficiency at comparable validity and distributional coverage.

  • The strongest interpretation of the one-step experiment requires a target-accuracy ablation. FLM-X* removes ODE integration at inference, but its training labels are generated using eight numerical integration steps

Questions for authors
  • Do the generators approximate the uniform target distributions beyond the reported metrics?
  • Do the observed advantages persist against relevant baselines at comparable generation quality?
  • Is one-step generalization stable as target accuracy increases?
5 comments

We thank the reviewer for the careful review and for finding our one-step experiment insightful.


W1 and Q1: Distributional coverage.

We agree that validity and memorization alone do not establish coverage. We therefore generated 10510^5 valid samples from each model at ∣D∣=105|D| = 10^5, the setting of Tab. 3, and compared them with 10510^5 samples drawn uniformly from all valid sequences. A model that concentrates on a subset of valid sequences produces more duplicate pairs, which lowers its effective support, n2/(2×duplicate pairs)n^2 / (2\times \text{duplicate pairs}).

Permutation, ∣S∣|S| = 3.63M.

Duplicate pairsEffective supportMemorization (%)
Uniform1,372 ± 413.65M ± 0.11M0.0
FLM1,3013.84M0.0
FLM-D*1,3583.68M0.1
FLM-X*1,4063.56M0.1

Dyck, ∣S∣|S| = 1.38M.

Duplicate pairsEffective supportMemorization (%)
Uniform3,615 ± 671.38M ± 0.03M0.0
FLM3,5451.41M0.1
FLM-D*3,7121.35M0.1
FLM-X*4,1761.20M1.9

We note that FLM and FLM-D* closely match uniform sampling on both distributions. The statistics suggested by the reviewer, such as position marginals and bracket depth, are also close to uniform (total variation of at most 0.012). FLM-X* reaches an effective support of 98% of ∣S∣|S| on permutation, and is mildly concentrated on Dyck, at 87% of ∣S∣|S|.


W2: Other structured noise-to-data mappings.

To understand which properties of the optimal flow map allow FLM-X* to generalize, we trained the same one-step network on two more structured mappings from noise to data. (1) The shuffled mapping randomly permutes the training points assigned by the optimal flow map. (2) The nearest-neighbour mapping sends each noise sample to its nearest training point.

Dyck (L=10L = 10), ∣D∣=105|D| = 10^5, 1,024 samples (%).

MappingValidityMemorization
Hashed lookup (Sec. 4.2)0.0–
Shuffled flow map0.0–
Nearest neighbour36.40.5
Flow map (FLM-X*)39.30.0

Thus, the flow map generalizes while its shuffled version does not, even though both use the same training points. We hypothesize that generalization requires locality, i.e. that nearby noise maps to similar sequences, a property the nearest-neighbour mapping also shares. This suggests that one-step generalization is not unique to the flow map. We will add these baselines to Sec. 4.2.


W3 and Q2: Comparison at comparable generation quality.

We agree that memorization thresholds are best compared at comparable validity. In our revision, we will describe the 10× and 100× gaps as differences in memorization thresholds rather than in data efficiency. The main case of low validity in our comparison is FLM on RHM. With longer training (1M steps instead of 200k), FLM reaches 90.1% validity on RHM at both ∣D∣=105|D| = 10^5 and 10610^6, while staying at 0% memorization. Its low validity in Tab. 6 was therefore due to undertraining. We hypothesize that FLM learns RHM more slowly because it noises every position, which weakens the training signal for the long-range dependency of the grammar.

To further test the ordering at equal validity, we have added a fourth distribution, palindromes of length L=16L = 16 over V=16V = 16 tokens, trained with the recipe of Fig. 6.

Validity / memorization (%), ∣D∣=104|D| = 10^4.

PermutationDyckRHMPalindrome
AR100.0 / 100.0100.0 / 99.899.6 / 100.0100.0 / 99.9
MDLM96.4 / 0.871.0 / 10.622.5 / 46.5100.0 / 26.3
FLM100.0 / 0.097.3 / 0.023.0 / 0.0100.0 / 0.0

Overall, on multiple distributions, including palindromes, all three paradigms reach comparable validity. The ordering is consistent between them, with AR memorizing the most, followed by MDLM, while FLM does not memorize.


W4 and Q3: Target accuracy of FLM-X*.

One-step generalization is stable as target accuracy increases. We first note that a less accurate target is still a training point, since the target is the training point with the largest posterior weight at the end of integration. Therefore, integration error only changes which training point a noise sample is assigned to. We then ran the requested ablation on Dyck at ∣D∣=105|D| = 10^5, where target error is largest (Tab. 8), training FLM-X* with targets from 4, 8 and 16 Euler steps.

Dyck (L=10L = 10), ∣D∣=105|D| = 10^5, 16,384 samples (%).

StepsTarget agreement with 1,024 stepsValidityMemorization
491.737.42.0
895.837.81.9
1697.537.72.2

Validity and memorization stay the same within sampling noise as target agreement increases. With more samples, the memorization of FLM-X* here is 1.9% rather than the 0.0% in Tab. 3, and we will update Tab. 3 accordingly.


We hope these responses address the reviewer's concerns, and would be grateful if they could be considered in the assessment.

Reviewer_ZwbKOpenReview ↗

Response to Author

The coverage diagnostics and target-accuracy ablation substantially address several of my concerns. However, coverage is evaluated conditional on validity, while FLM-X* retains low overall validity. The alternative mappings also show that one-step generalization is not unique to the flow-map construction, with locality remaining a hypothesis. Overall, thank you for the detailed response and additional experiments, but does not sufficiently resolve my remaining concerns. I therefore maintain my original rating

We thank the reviewer for finding the new diagnostics useful, and for further engaging in discussion. We would like to briefly clarify the two remaining points.


Validity of FLM-X*.

While FLM-X* has low validity, we believe it serves its intended purpose as an intervention rather than as a generator. With FLM-X*, we remove the sampler completely, and the model still generalizes. This is important because several prior works attribute generalization to the sampling dynamics. [1] finds that generalization arises at specific times during integration, and [2] argues that sampling trajectories largely avoid the regions where the model has memorized.

As a generator, FLM-X* is limited, since one-step generation is typically much harder than multi-step generation, which has more inference compute. Lower validity of one-step generators on discrete data also appears in prior work. FMLM [3], on which our work builds, reaches 9.4% validity on Sudoku in one step against 94.9% with 1,024 steps. Moreover, 70% of the invalid permutations of FLM-X* are a single edit away from a valid one (Tab. 10).


Alternative mappings.

We believe the alternative mappings strengthen our results rather than weaken them. The nearest-neighbour result shows that our central claim holds more broadly, i.e. one-step mappings can generalize without any sampling dynamics.

However, while the nearest-neighbour mapping generalizes, it does not match the data distribution. We illustrate this with a minimal example. We take three training points on a line, D={−1,0.8,1}D = \{-1, 0.8, 1\}, each with empirical probability 1/3, and noise x0∼N(0,1)x_0 \sim \mathcal{N}(0, 1). We integrate the exact flow ODE, and compute the probability of each point as the Gaussian mass of the noise mapped to it.

PointEmpiricalFlow mapNearest neighbour
−1.0-1.00.3330.3330.460
0.80.80.3330.3330.356
1.01.00.3330.3330.184

The flow map assigns each training point equal probability, matching the empirical distribution. The nearest-neighbour mapping instead splits the noise by distance between points, so isolated points receive more probability and nearby points less. Thus, while the nearest-neighbour mapping supports our claim, it is not a suitable generator.


We hope this clarifies the remaining points, and are happy to discuss further.


[1] Bertrand et al. On the closed-form of flow matching: Generalization does not arise from target stochasticity, '26

[2] Kaiser & Kollmann. Diffusion models memorize in training and generalize in inference, '26

[3] Lee et al. Flow map language models, '26

Reviewer_6rN7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Flow matching (FM) has become a popular paradigm for generative modeling in continuous domains. Interestingly, while the empirical optimal model of the FM objective will lead to memorization of training data, practical learned models often exhibit strong generalization, and there has been much recent work aiming to understand what factors might be at play for continuous data such as images. This paper studies the generalization behavior for the case when continuous FM is applied to categorical data, by first embedding categorical data into a continuous space. In particular, similar to previous work such as [Lee et al 2026], one-hot encoding is used. In this setting, the optimal denoiser admits a simpler closed form (which intuitively is a simplified special case of the general continuous-data formula), and will cause memorization of training data. The main contributions are a set of well-designed experiments on three categorical distributions that show (1) the memorization/generalization behavior across training dataset sizes and other factors, (2) the persistence of generalization after potential sources such as target stochasticity or the sampler are "removed" (by using a denoiser directly regressed on the optimal denoiser (FLM-D*), or by a direct one-step model directly regressed on the optimal flow map (FLM-X*)), and (3) a comparison with other generative models for categorical data (both augoregressive based, and masked-diffusion based) showing that on the three synthetic distributions considered, the continuous FM learns a generalized model with substantially smaller training-set sizes.

Critical strengths

++ The experiments are rather comprehensive and well designed. I also like the use of these test distributions where one has a clear view of what count as memorization.

++ I think the observed behaviors are valuable contributions to the field, especially that those regressed models still exhibit similar levels of memorization.

Critical weaknesses

-- The biggest weakness in my view is that, while the experiments are interesting, there is not much insights offered. The direct regression type of replacement strategy is also not entirely new (e.g, [Bertrand et al 2026]). I feel that the paper could have gone much further beyond the empirical observations, for example, by developing a deeper understanding of what, if anything, makes categorical data + continuous embedding fundamental different (or maybe not different) from continuous data.

For example, is it clear that this one-hot categorical setting is really fundamentally different from the continuous setting from the perspective of generalization mechanism? Although the target distribution after one-hot embedding has discrete support, the flow itself evolves in a continuous space. Essentially we can view that the samples here are really sparse / well-separated. In light of the type of analysis and perspectives from [ref 1; see below], such well-separateness might even give more room for a learned model to "extrapolate" for most of the noise levels considered during the training, making it more reliant on the extrapolation ability of denoiser architecture. On the other hand, if discrete support indeed changes the memorization/generalization behavior relative to finite training samples drawn from an underlying continuous support, then it would have been nice to study and provide insights on what could have contributed to the generalization mechanisms that makes the learned model generalizes (while still maintaining good validity).

Without deeper analysis or mechasnistic insights, the work feels somewhat incremental.

[ref 1]: "Selective Underfitting in Diffusion Models", Song, Kim, Chen, Du, Kakade, Sitzmann.

Questions for authors

See weakness

2 comments

We thank the reviewer for the careful and constructive review, and for finding our observations valuable. We agree with the premise of the question, and would like to clarify our contributions below.


W1: Relation to Bertrand et al.

While we agree that FLM-D* is a discrete analogue of [1], we consider FLM-X* to be our main contribution. With FLM-X*, we remove the sampler entirely, and generalization still holds. This matters because several prior works attribute generalization to the sampling dynamics. [1] finds that generalization occurs at specific times during integration, [2] argues that sampling trajectories largely avoid the regions where the model has memorized, and [3] argues that the model is queried at inference in regions it was not supervised on during training. FLM-X* is trained and sampled on the same noise distribution, without integration. Its generalization therefore shows that these mechanisms are not necessary.


W2: One-hot versus continuous data.

We agree that while the data is discrete, the flow operates in a continuous space, as it does for images. However, many existing explanations of why flow matching generalizes rely on properties specific to continuous data, such as images.

  1. [4, 5] find that models generalize by stitching together local patches of training images. On permutations, combining patches of valid permutations generally does not give a valid permutation, yet FLM generalizes.
  2. [6] attributes generalization to an inductive bias towards geometry-adaptive harmonic bases, which match the smooth contours of natural images. Categorical sequences have no such geometry.
  3. These explanations assume data near a low-dimensional manifold, where points between training examples remain plausible. Under a one-hot embedding, every point between two training sequences lies off the support.

Since continuous flow matching has only recently been shown to be effective for discrete data [7], our setting lets us test whether these explanations are necessary. Moreover, on our distributions, memorization and validity can be measured exactly, while on images both rely on distances between pretrained embeddings and on scores such as FID. This exact measurement reveals details that are hard to see on images. For example, at an intermediate ∣D∣|D|, validity dips while the model moves from generalization to memorization during training (Fig. 2). We therefore believe that studying flow matching on discrete data is valuable, even where it behaves similarly to the continuous case.


W3: Selective underfitting.

We thank the reviewer for this reference, and will cite it. As discussed in W1, FLM-X* is trained and sampled on the same noise distribution, so extrapolation cannot be the sole source of its generalization. To understand what else contributes, we trained the same one-step network on two other mappings from noise to the same training points. (1) The shuffled mapping randomly permutes the training points assigned by the optimal flow map. (2) The nearest-neighbour mapping sends each noise sample to its nearest training point.

Dyck (L=10L = 10), ∣D∣=105|D| = 10^5, 1,024 samples (%).

MappingValidityMemorization
Hashed lookup (Sec. 4.2)0.0–
Shuffled flow map0.0–
Nearest neighbour36.40.5
Flow map (FLM-X*)39.30.0

All mappings use the same training points, yet only the flow map and the nearest-neighbour mapping generalize. Hence, the separation of the data alone does not explain generalization. We hypothesize that generalization instead requires locality, i.e. that nearby noise maps to similar sequences. Moreover, we note that while the validity of FLM-X* is low, its valid samples cover the distribution well, with an effective support of 98% on permutation (Reviewer ZwbK, W1). We will add this discussion and the new controls to the paper.


We hope these responses address the reviewer's concerns, and would be grateful if they could be considered in the assessment.


[1] Bertrand et al. On the closed-form of flow matching: Generalization does not arise from target stochasticity, '26

[2] Kaiser & Kollmann. Diffusion models memorize in training and generalize in inference, '26

[3] Song et al. Selective underfitting in diffusion models, '25

[4] Kamb & Ganguli. An analytic theory of creativity in convolutional diffusion models, '24

[5] Niedoba et al. Towards a mechanistic explanation of diffusion model generalization, '24

[6] Kadkhodaie et al. Generalization in diffusion models arises from geometry-adaptive harmonic representations, '24

[7] Lee et al. Flow map language models, '26

We thank the reviewer again for the constructive feedback. We have now revised our submission, with all changes highlighted in blue, and summarize them below.

  • We added a discussion of sampler-based explanations of generalization to the background, including the selective underfitting view of Song et al. (Sec. 2). (W1, W3)
  • We added two more noise-to-data mappings to further investigate the mechanism behind one-step generalization (Sec. 4.2, Tab. 11). (W3)
  • We added coverage diagnostics, showing that the valid samples of our models cover the support (Sec. 4.2, Tab. 14). (W3)

We believe that these updates strengthen this work, and thank the reviewer for prompting them. We would be grateful if the reviewer would consider them in their assessment.

Reviewer_iyxyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

It is a well known fact that flow matching or diffusion, if perfectly trained on the data empirical distribution, reproduce the samples from the training dataset. In the empirical (finite) train dataset setting, the optimal velocity field even has a closed form. For continuous data (eg, images), several works have already discussed possible reasons why a trained model tends to generalize instead of memorizing. The main hypothesis are 1) stochasticity of the target, 2) the sampling error, 3) the model's capacity / the number of training samples.

The contribution of the paper is to clarify whether any of these hypothesis can explain generalization of flow matching models in the case of discrete data.

The conclusion is that 1) and 2) are not relevant for generalization, and that 3) is, similarly as in the continuous case. The paper derives clear and rigorous experiments to prove their claim numerically. The mecanisms behind why a model generalizes when we increase the number of training samples is not analyzed.

In my opinion, the most interesting result is the one showing that Flow Language Models memorize less that masked diffusion or that autoregressive models, which is a separate result from the rest. It would have been interesting to explore why this is the case.

Critical strengths
  • Very easy to follow, well written, good references
  • Experiments are serious and well designed. The conclusions of the experiments are clear: they eliminate progressively the non relevant stochasticity and sampler hypothesis.
Critical weaknesses
  • Does anyone still believe that generalization comes from stochasticity of the target? This assumption have been perfectly debunked in the continuous case (Bertrand et al. 2025) with the same experiment. I do find the general experiment convincing, but I don't think that doing it in the discrete case brings anything. This stochasticity assumption is outdated.
  • The paper lacks of new insights on the mecanisms behind memorization / generalization, and gives no evidence that these ones would be different in the discrete case.
Questions for authors

Questions

  1. Is it a new finding that the size of training influences the memorization on discrete data? (Figure 2)
  2. The observation for the memorization threshold for autoregressive and masked diffusion is very interesting. Could you provide an explanation for this?
  3. In terms of memorization, MDLM on Permutation and Dyck-K doesn't seem so far from FLM. Can you repeat this experiment on another dataset?

Minor remarks

  • Proposition 3.1 is quite obvious given the closed form expression for the velocity (see Bertrand et al. 2025 for example).
  • Table 3: Why is the validity score so bad with FLM-X$^*$?
4 comments

We thank the reviewer for the careful review, and for finding our experiments well designed.


W1: Relation to Bertrand et al.

We agree with the reviewer that FLM-D* is a discrete analogue of [1]. We included it in our paper because continuous flow matching has only recently been shown to be effective for discrete data [2]. Whether flow matching generalizes in this domain, and whether it does so for the same reasons, remains unexplored. The discrete domain differs from continuous domains because its support is finite and disjoint, and therefore existing explanations, e.g. that models generalize by stitching together patches of training images [3], do not hold. Specifically, [1] finds that generalization occurs at specific times during the integration of the flow ODE. FLM-D*, and consequently FLM-X*, allow us to show that while stochasticity is indeed not needed, we can further remove the sampling and the generalization still holds.


W2: Mechanism behind generalization.

We believe our most fundamental contribution is FLM-X*, which shows that existing hypotheses based on the sampler [1, 4] are not necessary for generalization. To understand which properties of the optimal flow map allow it to generalize, we trained the network on two more structured mappings from noise to data: (1) shuffled mapping randomly permutes the training points assigned by the optimal flow map. (2) nearest-neighbour mapping sends each noise sample to its nearest training point.

Dyck (L=10L = 10), ∣D∣=105|D| = 10^5 (%)

MappingValidityMemorization
Hashed lookup (Section 4.2)0.0–
Shuffled flow map0.0–
Nearest neighbour36.40.5
Flow map (FLM-X*)39.30.0

Thus, the flow map generalizes while its shuffled version does not. We hypothesize that generalization requires locality, i.e. that nearby noise maps to similar sequences, which the nearest-neighbour mapping also has.


Q1: Training-set size and memorization.

We agree that the link between memorization and training-set size has been studied before [5]. We believe our setting adds two things. First, on our distributions, memorization and validity can be measured exactly. On images, both rely on distances between pretrained embeddings and on scores such as FID. Second, this exact measurement reveals a detail that is hard to see on images: at an intermediate ∣D∣|D|, validity dips while the model moves from generalization to memorization during training (Fig. 2). To the best of our knowledge, this has not been reported before. More broadly, our intention in Sec. 4.1 is to study flow matching on discrete data along several axes and to compare it with known results on images.


Q2: AR and MDLM thresholds.

We thank the reviewer for finding this result interesting. Following prior work [6, 7], we attribute the gap to an implicit data augmentation effect. We support this hypothesis with two experiments. First, masking a fraction of AR's input tokens during training lowers its memorization at ∣D∣=105|D| = 10^5 from 100% to 69% (Tab. 4). Second, we train and sample FLM with only K fixed noise vectors per sequence.

Permutation, ∣D∣=6000|D| = 6000 (%)

KValidityMemorization
1100.0100.0
16100.0100.0
∞\infty (standard)91.749.2

With a small set of noise vectors, FLM also learns to reproduce the training set exactly, just like AR. Training on fresh noise at every epoch helps to prevent this overfitting. We hypothesize that FLM's continuous noise creates more varied views than MDLM's binary masking, which is consistent with the results in Fig. 6.


Q3: Additional dataset.

As requested, we have added a fourth distribution, palindromes, and trained all three paradigms with the recipe of Fig. 6. The trend across paradigms remains consistent with the other distributions.

Palindromes, L=16L = 16, V=16V = 16, ∣D∣=104|D| = 10^4 (%)

ValidityMemorization
AR100.099.9
MDLM100.026.3
FLM100.00.0

M1: Proposition 3.1.

We agree it follows from the closed form in [1], and will credit them explicitly.


M2: Validity of FLM-X*.

FLM-X* generates samples in a single forward pass, which is typically harder than generating over many steps. The same gap appears in prior work. On Sudoku, FMLM [2] reaches 9.4% validity in one step against 94.9% with 1,024 steps. Nevertheless, 70% of the invalid permutations of FLM-X* are one edit away from a valid sequence (Tab. 10). We furthermore find that its valid samples cover the data distribution well, with an effective support of 98% on permutation (Reviewer ZwbK, W1).


We hope these responses address the reviewer's concerns, and would be grateful if they could be considered in the assessment.


[1] Bertrand et al. '26

[2] Lee et al. FMLM, '26

[3] Kamb & Ganguli. '24

[4] Kaiser & Kollmann. '26

[5] Yoon et al. '23

[6] Prabhudesai et al. '26

[7] Gao et al. '25

We thank the reviewer again for the constructive feedback. We have now revised our submission, with all changes highlighted in blue, and summarize them below.

  • We credited Bertrand et al. for the optimal denoiser and Prop. 3.1 (Sec. 3.2). (M1)
  • We added a discussion of sampler-based explanations of generalization to the background (Sec. 2). (W1)
  • We added two more noise-to-data mappings to further investigate the mechanism behind one-step generalization (Sec. 4.2, Tab. 11). (W2)
  • We added the fixed-noise experiment supporting the data augmentation hypothesis (Sec. 4.3, Tab. 12). (Q2)
  • We added palindromes as a fourth distribution in the paradigm comparison (Sec. 4.3, Tab. 13, App. B). (Q3)

We believe that these updates strengthen this work, and thank the reviewer for prompting them. We would be grateful if the reviewer would consider them in their assessment.

Thank you for your answer.

W2: Mechanism behind generalization. Thanks for the additional experiment. I am not sure I fully understand what conclusion you intend to draw from them. In particular, the 0% validity of the shuffled mapping seems to indicate that the network simply fails to learn this more complex mapping, rather than providing insight into memorization versus generalization. I am also not sure how locality explains the transition between memorization and generalization. For instance, the optimal flow map retains the same type of local structure when the training set is small, yet the network memorizes in that regime.

Could you clarify what these experiments establish about the mechanism of generalization, beyond the fact that some mappings are easier to learn than others?

Q2 & Q3 Thanks, I found these experiments insightful.

Overall assessment.

My main reservation remains the novelty of the findings regarding FLM itself, as several observations appear to extend results already established in the continuous FM setting. More fundamentally, I am not entirely convinced that the discrete setting should lead to substantially different generalization mechanisms. After all, the discrete data are embedded in a continuous space, and the flow-matching framework remains largely unchanged apart from adaptations to the output parameterization and training loss. I would therefore appreciate a clearer explanation of what makes the discrete setting fundamentally different in this context.

On the other hand, I find the comparison between AR, MDLM, and FLM valuable, and the additional experiments strengthen this aspect of the paper.

I will take these points into account when reassessing my overall evaluation.

We thank the reviewer for the follow-up, and are glad that the reviewer found the paradigm comparison valuable. We address the concerns below.


Mechanism behind generalization.

We agree that the additional mappings do not explain the transition from memorization to generalization. As the training set grows, the network can no longer fit the lookup exactly with its limited capacity, and starts to underfit it. Whether this underfitting produces new valid samples (generalization) or invalid ones (collapse) depends on the map it is trying to learn. Our experiment in W2 investigates which property of the map leads to the former. The optimal flow map has a structure we call "locality", where nearby noise samples map to similar sequences. The nearest-neighbour mapping shares this structure and also generalizes, while the shuffled mapping does not, and fails to produce valid samples.


Discrete versus continuous setting.

We agree that while the data is discrete, the flow operates in a continuous space, as it does for images. However, many existing explanations of why flow matching generalizes rely on properties specific to continuous data, such as images.

  1. [1, 2] find that models generalize by stitching together local patches of training images. On permutations, combining patches of valid permutations generally does not give a valid permutation, yet FLM generalizes.
  2. [3] attributes generalization to an inductive bias towards geometry-adaptive harmonic bases, which match the smooth contours of natural images. Categorical sequences have no such geometry.
  3. These explanations assume a locally connected manifold, where each image is surrounded by similar images [4]. Under a one-hot embedding, the support is a finite set of isolated points.

Thus, discrete data lets us test whether these explanations are necessary, while providing ground truth for memorization and validity (Q1). We therefore believe studying flow matching on discrete data is valuable.


We hope this clarifies the remaining points, and are happy to discuss further.


[1] Kamb & Ganguli. An analytic theory of creativity in convolutional diffusion models, '24

[2] Niedoba et al. Towards a mechanistic explanation of diffusion model generalization, '24

[3] Kadkhodaie et al. Generalization in diffusion models arises from geometry-adaptive harmonic representations, '24

[4] Pope et al. The intrinsic dimension of images and its impact on learning, '21

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.