- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work introduces a hybrid flow-map model for few-step text generation, combining ideas from Flow-Map Language Models (FMLM) and Masked Diffusion Models (MDM). A subset of the tokens is noised using a stochastic interpolant, and the rest is kept as clean tokens conditioning for a two-time denoiser (TTD) parameterizing the distilled transport. This enables a Posterior Refinement sampling procedure, which iteratively selects and locks most confident tokens according to a certain rule, based on a flow-map step from to 1. The method is evaluated on language modeling and Sudoku experiments.
Critical strengths
- Making conditioning on clean tokens native to the generative model is a sensible idea. At the boundary, the cross entropy loss targets the same denoising posterior as MDM, justifying initialization from a pretrained MDM checkpoint. This yields good experimental results, even though masked positions are represented by pure Gaussian noise rather than mask tokens as input.
- Good Sudoku results.
Critical weaknesses
-
The TTD and its self distillation objective are largely inherited from FMLM and closely related Categorical Flow Maps. The main contributions are the arbitrary clean tokens conditioning and the posterior refinement heuristic, which are reasonable, but I find the novelty limited.
-
In the proposed sampler, committed tokens cannot be modified, limiting the claimed self-correction capabilities.
-
The comparison with block diffusion inference is a bit overstated, the former involves an appropriate attention architecture setup for block autoregressive generation. The method rather resembles conditional/arbitrary order sampling as already available in MDLM or SEDD.
-
The strongest results for FMLM+ (e.g., on GSMK8k), rely on initilazing from a pretrained MDM checkpoint followed by additional training. Comparisons with the original MDLM should account for the additional training compute.
-
I find the discussion of a-priori versus a-posteriori confidence insufficiently precise and unconvincing. What probabilistic quantity does the proposed confidence actually represent? The interpretation of the final flow map jump is mostly hand-wavy in the current state. The provided theoretical results do not establish anything on high confidence tokens / the interpretation of posterior confidence, as they consist in a straightforward error analysis. Moreover, distinguishing uncertainty of the denoising posterior and of the distilled transport is intuitive, but the proposed interpretation in terms of aleatoric and epistemic uncertainty seems questionable. I encourage the authors to clarify or substantially simplify this discussion.
-
I have serious concerns about the fairness of the experiments. On OWT, the scores reported for the FMLM baseline (GenPPL 62.23, entropy 5.33 at 1024 NFE) correspond to FLM, the undistilled model, in Lee et al., rather than the actual distilled FMLM. Although the authors clarify that their baseline uses only the diagonal objective, labeling it FMLM and comparing against it is misleading in a paper focused on few-step generation. The actual FMLM achieves GenPPL 45.09 at only 32 NFE, compared with 66.60 at 128 NFE for FMLM+, while maintaining slightly higher entropy. This directly challenges the claimed improvement in the speed–quality tradeoff. Similar concerns apply to the choice of S-FLM baseline from Deschenaux et al.; in this case, the correct scores for S-FLM are reported, but these are the scores of the worst-performing method, while stronger architectural variants are available. These choices substantially weaken the experimental takeaways which is a major reason for my negative assessment.
-
The baselines presented are not the best-performing baselines currently available. Several strong recent baselines are missing, including RePlaid and LangFlow for CDLMs, as well as IDLM and Discrete-MMD for discrete diffusion distillation. Given that the main claim concerns the speed–quality tradeoff, comparisons with strong existing distillation methods are essential. The current results do not establish the claim that FMLM+ improves over available approaches.
-
There is no generative frontier drawn for the unconditional text generation experiments (gen ppl / entropy curves). As explained in [1], GenPPL values reported at different entropies cannot be compared, as the metric can be gamed by generating text in lower entropy regimes. This echoes the fairness concerns raised previously.
Overall, I find the methodological novelty limited, and the experimental weaknesses / fairness issues are too substantial to support the main claims in the current version of the paper.
[1] Pynadath et al., Generative Frontiers: Why Evaluation Matters for Diffusion Language Models, 2026
Questions for authors
-
How did you produce the baselines results? Did you report them from the original papers, evaluated released checkpoints, trained from scratch from released code or re-implementation?
-
Why are generative quality–diversity frontiers not reported? Could the authors provide GenPPL/entropy curves by sweeping the sampling temperature for all major baselines?
-
Why is the undistilled FLM reported as FMLM with the "diagonal objective"? Was it intentional, and if so why? The original FMLM results appear substantially stronger than those of FMLM+ on OWT, at much lower NFE. Could the authors provide a direct comparison?
-
For S-FLM, why didn't you report the results of the best performing baseline (S-arch)? In the original paper, simple S-FLM seems like the worst performing method, which does not seem appropriate to take as a baseline.
-
Can the authors substantially clarify the probabilistic interpretation of their a-posterior confidence and explain why endpoint confidence should measure token correctness?
6 comments
Rebuttal-2/2
W6.1 and Q3: FLM and distilled FMLM.
We agree that the FMLM baseline in Tab. 2, trained with only the diagonal objective, coincides with FLM [1], and we will relabel it accordingly. This choice was intentional, as all baselines in Tab. 2 are evaluated in their many-step regime, but we agree the label is confusing. We add a distilled FMLM with the same architecture and training budget as FMLM+. We note that the Gen-PPL of 45.09 at 32 NFE reported in [1] uses the -sampler, whereas all methods here use standard decoding. On OpenWebText, FMLM+ uses an improved PR schedule (8 rounds × 16 flow steps), which we will also update in Tab. 2.
| Dataset | NFE (FLM, FMLM / FMLM+) | FLM | FMLM | FMLM+ (PR) |
|---|---|---|---|---|
| Sudoku Easy / Med. / Hard (%, ↑) | 180 / 4 | 94.2 / 82.7 / 44.5 | 94.6 / 86.2 / 50.0 | 97.9 / 92.0 / 71.2 |
| GSM8K (%, ↑) | 1024 / 32 | 0.3 | 0.1 | 19.0 |
| TinyStories Gen-PPL ↓ (entropy) | 128 / 32 | 57.50 (4.17) | 62.81 (4.09) | 17.53 (3.96) |
| OpenWebText Gen-PPL ↓ (entropy) | 1024 / 128 | 62.23 (5.33) | 45.05 (5.11) | 53.00 (5.26) |
We note that FMLM+ (PR) significantly outperforms both FLM and FMLM on the two conditional generation tasks, while using 32–45× fewer NFEs. On TinyStories, it also achieves a significantly lower Gen-PPL with only a moderate drop in entropy. On OpenWebText, all three methods lie in a similar range with different entropy tradeoffs, while FMLM+ uses 8× fewer NFEs.
W6.2 and Q4: S-FLM variant.
We would like to clarify that we report the best-performing S-FLM variant on each task from [6]: S-FLM + α*(0.1) + adaptive on Sudoku (94.8 / 85.2 / 45.0 on Easy / Medium / Hard), and S-FLM with S-arch and top-k = 1 decoding on GSM8K (18.0%). For comparison, the base S-FLM in the same tables reaches 81.5 / 50.6 / 14.0 and 1.2%. We will name these variants explicitly in Tab. 2.
W7: Additional baselines.
We thank the reviewer for these pointers, and add RePlaid, IDLM and the concurrent DBTM [9] on GSM8K. LangFlow and Discrete-MMD do not report GSM8K results. Since RePlaid reports GSM8K only at 1024 steps and has not released code or checkpoints, we use its numbers from [7] without self-conditioning or temperature tuning, matching the setting of FMLM+. Methods using an MDLM teacher are compared with FMLM+ using the same teacher (Tab. 4).
GSM8K accuracy (%, ↑).
| Method | Teacher | NFE | Acc. |
|---|---|---|---|
| RePlaid (DDPM) [7] | – | 1024 | 17.1 |
| RePlaid (DDIM) [7] | – | 1024 | 13.0 |
| DBTM [9] | – | 32 | 16.8 |
| FMLM+ (PR) | – | 32 | 19.0 |
| IDLM [8] | MDLM | 32 | 12.8 |
| IDLM [8] | Duo | 32 | 15.4 |
| FMLM+ (Distill, PR) | MDLM | 32 | 21.6 |
| FMLM+ (Init, PR) | MDLM | 32 | 31.8 |
FMLM+ outperforms these methods on GSM8K at equal or fewer NFEs. Self-conditioning and temperature tuning, which improve RePlaid substantially [7], are orthogonal to our method, and we leave their combination with FMLM+ to future work.
W8 and Q2: Gen-PPL / entropy frontiers.
We agree that Gen-PPL should be compared at matched entropy [10]. Following [6], we sweep the sampling parameters of FMLM+ at 1024 NFE on OpenWebText and compare Gen-PPL with the baselines (taken from the temperature-swept frontiers in [6]) at matched entropy.
OpenWebText, Gen-PPL ↓ at matched entropy, NFE = 1024.
| Entropy | Duo | MDLM | S-FLM | FLM | FMLM+ (PR) |
|---|---|---|---|---|---|
| 5.00 | 23.7 | – | 34.3 | 43.4 | 25.1 |
| 5.05 | 25.7 | 26.0 | 37.0 | 45.0 | 27.8 |
| 5.16 | 30.3 | 32.6 | 43.3 | 49.4 | 32.7 |
| 5.18 | 31.5 | 34.0 | 44.7 | 50.8 | 34.5 |
| 5.23 | 34.4 | 37.2 | 49.4 | 54.4 | 40.4 |
At matched entropy, FMLM+ outperforms the continuous baselines while remaining close to the discrete baselines in this many-step regime. We thank the reviewer for this suggestion, and will report these frontiers in the revised paper.
Q1: Baseline results.
For Sudoku and GSM8K, we follow the exact protocol of [6] and report all baseline numbers (AR, MDLM, Duo, CANDI, FLM, S-FLM) from their paper. Following [6], all Sudoku baselines use 180 NFEs, and Tab. 2 incorrectly lists 128, which we will correct. For TinyStories, we train and evaluate all baselines from scratch using their released code, with the same architecture and training budget. For OpenWebText, we evaluate the released checkpoints of each baseline under the same protocol.
We hope these responses address your concerns, and would be grateful if you would consider them in your assessment.
[1] Lee et al. FMLM, '26
[2] Roos et al. Categorical Flow Maps, '26
[3] Nie et al. LLaDA, '25
[4] Arriola et al. Block Diffusion, '25
[5] Kim et al. Train for the Worst, Plan for the Best, '25
[6] Deschenaux et al. Hyperspherical Flows, '26
[7] Yang et al. RePlaid, '26
[8] Li et al. IDLM, '26
[9] Tang and Wang. DBTM, '26
[10] Pynadath et al. Generative Frontiers, '26
We thank the reviewer again for the constructive feedback. We have now revised our submission, with all changes highlighted in blue, and summarize them below.
- We relabeled the FMLM baseline in Tab. 2 as FLM, and added a distilled FMLM baseline (Tab. 2, App. C.5). (W6.1, Q3)
- We added RePlaid, IDLM and DBTM as additional baselines on GSM8K (App. C.6). (W7)
- We added a comparison of Gen-PPL at matched entropy (App. C.4). (W8, Q2)
- We added FMLM+ trained from scratch for 500k steps, matching the total compute of FMLM+ (Init) (Tab. 4). (W4)
- We replaced "self-correct" with "refine" throughout the paper, removed the aleatoric/epistemic terminology (Sec. 3.3, Fig. 4), and rephrased block diffusion as block-wise decoding order (Sec. 3.2). (W2, W3, W5, Q5)
- We described the source of each baseline and the S-FLM variants in App. B.2. (Q1, W6.2, Q4)
We believe that these updates strengthen this work, and thank the reviewer for prompting them. We would be grateful if the reviewer would consider them in their assessment.
Rebuttal-1/2
We thank the reviewer for the careful review, and for finding our Sudoku results strong. We address the reviewer's concerns below.
W1: Novelty.
We agree that the two-time denoiser and its self-distillation objective are inherited from FMLM [1] and Categorical Flow Maps [2]. However, these works do not use the per-token outputs of the flow map beyond rounding them to tokens. To the best of our knowledge, we are the first to show that these outputs provide an improved confidence signal for masking-style iterative generation. MDMs cannot use this signal unless combined with a flow map formulation, which is our methodological contribution. The effect is substantial: on unconditional Sudoku, the same FMLM+ checkpoint reaches 0% at 9 NFEs with a-priori confidence, and 96.0% with PR (Tab. 1).
Our second contribution is the connection between MDMs and FMLM+. The two objectives share their minimizer at s=t=0 (Prop. A.1), which allows FMLM+ to be initialized from pretrained MDMs. This significantly improves GSM8K accuracy (Tab. 4). We believe this is a meaningful contribution, as it offers a path to scale continuous language models using the growing pool of large pretrained MDMs (e.g., LLaDA [3]).
W2: Self-correction Claim.
We agree, and we state this in the Limitations section. By self-correct, we meant that PR renoises low-confidence tokens so that they get a chance to be remapped to a different data mode. In practice, confidence is strongly correlated with correctness: on Sudoku, tokens with confidence ≥0.9 are correct 99.3%, 97.4% and 91.9% of the time on Easy, Medium and Hard (Tab. 3). To avoid confusion, we will revise our text to better reflect this.
W3: Block-wise decoding.
We mention block diffusion only once (L316–317), as an example of an inference schedule that FMLM+ supports: choosing the committed set U as the next contiguous block of tokens commits tokens in block order. Such block-wise decoding is also how MDMs are commonly used in practice, e.g., in LLaDA [3] (Sec. 2.4). We do not claim that FMLM+ matches the block-causal attention or caching of block diffusion [4], and we will rephrase this as "block-wise decoding order" to make this explicit.
W4: MDM initialization and training compute.
We clarify that our main results do not rely on MDM initialization: every FMLM+ result in Tab. 2 is trained from scratch, and already matches or outperforms MDLM (e.g., 19.0% vs. 18.0% on GSM8K) with 32× fewer NFEs. The Init and Distill variants in Sec. 4.2 are an additional result, showing that pretrained MDMs can further improve FMLM+. To account for the additional training compute, we also train FMLM+ from scratch for 500k steps, matching the total compute of FMLM+ (Init). It reaches 23.1% on GSM8K at 32 NFE (vs. 19.0% at 250k steps), compared with 31.8% for FMLM+ (Init), so the benefit of initialization is not explained by compute alone.
W5 and Q5: Interpretation of a-posteriori confidence.
We agree that PR confidence is a heuristic, as are the confidence scores used in MDM decoding [5]. Since these methods matter precisely when the model makes errors, tight guarantees are generally challenging to establish. A-priori confidence is computed from δ0,0, which equals the MDM posterior given only the context (Prop. A.1), while a-posteriori confidence is computed from the endpoint δ0,1, which is one-hot for the exact flow. Our analysis shows that low confidence in δ0,1 certifies an error (Prop. A.3). Empirically, confidence is strongly correlated with correctness: on Sudoku, tokens with confidence ≥0.9 are correct 99.3%, 97.4% and 91.9% of the time on Easy, Medium and Hard (Tab. 3). Following the reviewer's suggestion, we will simplify this discussion and remove the aleatoric/epistemic interpretation.