acceptodds
Under review as a conference paper at ICLR 2027

Posterior Refinement: Few-Step Language Generation via Any-Order Flow Maps

Abstract

Non-autoregressive generation offers a powerful paradigm for iterative refinement, allowing models to recursively critique, erase and regenerate arbitrary subsets of tokens. However, existing non-autoregressive models fail to realize this potential. Masked Diffusion Models (MDMs) suffer from factorization error, causing sample quality to collapse when generating multiple tokens simultaneously. Flow Map Language Models (FMLMs) circumvent this bottleneck via joint sequence transport for excellent few-step generation, but sacrifice the inference-time flexibility of MDMs. We introduce FMLM+, a framework that bridges this gap by equipping FMLM with masking-style noise schedules. While generating the full sequence in a single step, FMLM+ simultaneously scores the global consistency of each token a posteriori. We leverage this to introduce Posterior Refinement, a novel inference-time refinement strategy that enables the model to adaptively self-correct its outputs, matching the performance of discrete baselines with 32× fewer NFEs. Across diverse benchmarks, we demonstrate that FMLM+ with Posterior Refinement improves the speed–quality tradeoff over both MDM and FMLM families, providing a scalable foundation for high-fidelity language modeling.

Primary area:
generative models
Keywords:
non-autoregressive generation, iterative refinement, few-step generation, continuous diffusion, flow-maps

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_8GroOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces a hybrid flow-map model for few-step text generation, combining ideas from Flow-Map Language Models (FMLM) and Masked Diffusion Models (MDM). A subset of the tokens is noised using a stochastic interpolant, and the rest is kept as clean tokens conditioning for a two-time denoiser (TTD) parameterizing the distilled transport. This enables a Posterior Refinement sampling procedure, which iteratively selects and locks most confident tokens according to a certain rule, based on a flow-map step from t=0t=0 to 1. The method is evaluated on language modeling and Sudoku experiments.

Critical strengths
  • Making conditioning on clean tokens native to the generative model is a sensible idea. At the t=s=0t=s=0 boundary, the cross entropy loss targets the same denoising posterior as MDM, justifying initialization from a pretrained MDM checkpoint. This yields good experimental results, even though masked positions are represented by pure Gaussian noise rather than mask tokens as input.
  • Good Sudoku results.
Critical weaknesses
  • The TTD and its self distillation objective are largely inherited from FMLM and closely related Categorical Flow Maps. The main contributions are the arbitrary clean tokens conditioning and the posterior refinement heuristic, which are reasonable, but I find the novelty limited.

  • In the proposed sampler, committed tokens cannot be modified, limiting the claimed self-correction capabilities.

  • The comparison with block diffusion inference is a bit overstated, the former involves an appropriate attention architecture setup for block autoregressive generation. The method rather resembles conditional/arbitrary order sampling as already available in MDLM or SEDD.

  • The strongest results for FMLM+ (e.g., on GSMK8k), rely on initilazing from a pretrained MDM checkpoint followed by additional training. Comparisons with the original MDLM should account for the additional training compute.

  • I find the discussion of a-priori versus a-posteriori confidence insufficiently precise and unconvincing. What probabilistic quantity does the proposed confidence actually represent? The interpretation of the final flow map jump is mostly hand-wavy in the current state. The provided theoretical results do not establish anything on high confidence tokens / the interpretation of posterior confidence, as they consist in a straightforward error analysis. Moreover, distinguishing uncertainty of the denoising posterior and of the distilled transport is intuitive, but the proposed interpretation in terms of aleatoric and epistemic uncertainty seems questionable. I encourage the authors to clarify or substantially simplify this discussion.

  • I have serious concerns about the fairness of the experiments. On OWT, the scores reported for the FMLM baseline (GenPPL 62.23, entropy 5.33 at 1024 NFE) correspond to FLM, the undistilled model, in Lee et al., rather than the actual distilled FMLM. Although the authors clarify that their baseline uses only the diagonal objective, labeling it FMLM and comparing against it is misleading in a paper focused on few-step generation. The actual FMLM achieves GenPPL 45.09 at only 32 NFE, compared with 66.60 at 128 NFE for FMLM+, while maintaining slightly higher entropy. This directly challenges the claimed improvement in the speed–quality tradeoff. Similar concerns apply to the choice of S-FLM baseline from Deschenaux et al.; in this case, the correct scores for S-FLM are reported, but these are the scores of the worst-performing method, while stronger architectural variants are available. These choices substantially weaken the experimental takeaways which is a major reason for my negative assessment.

  • The baselines presented are not the best-performing baselines currently available. Several strong recent baselines are missing, including RePlaid and LangFlow for CDLMs, as well as IDLM and Discrete-MMD for discrete diffusion distillation. Given that the main claim concerns the speed–quality tradeoff, comparisons with strong existing distillation methods are essential. The current results do not establish the claim that FMLM+ improves over available approaches.

  • There is no generative frontier drawn for the unconditional text generation experiments (gen ppl / entropy curves). As explained in [1], GenPPL values reported at different entropies cannot be compared, as the metric can be gamed by generating text in lower entropy regimes. This echoes the fairness concerns raised previously.

Overall, I find the methodological novelty limited, and the experimental weaknesses / fairness issues are too substantial to support the main claims in the current version of the paper.

[1] Pynadath et al., Generative Frontiers: Why Evaluation Matters for Diffusion Language Models, 2026

Questions for authors
  • How did you produce the baselines results? Did you report them from the original papers, evaluated released checkpoints, trained from scratch from released code or re-implementation?

  • Why are generative quality–diversity frontiers not reported? Could the authors provide GenPPL/entropy curves by sweeping the sampling temperature for all major baselines?

  • Why is the undistilled FLM reported as FMLM with the "diagonal objective"? Was it intentional, and if so why? The original FMLM results appear substantially stronger than those of FMLM+ on OWT, at much lower NFE. Could the authors provide a direct comparison?

  • For S-FLM, why didn't you report the results of the best performing baseline (S-arch)? In the original paper, simple S-FLM seems like the worst performing method, which does not seem appropriate to take as a baseline.

  • Can the authors substantially clarify the probabilistic interpretation of their a-posterior confidence and explain why endpoint confidence should measure token correctness?

6 comments
AuthorsOpenReview ↗

Rebuttal-1/2

We thank the reviewer for the careful review, and for finding our Sudoku results strong. We address the reviewer's concerns below.


W1: Novelty.

We agree that the two-time denoiser and its self-distillation objective are inherited from FMLM [1] and Categorical Flow Maps [2]. However, these works do not use the per-token outputs of the flow map beyond rounding them to tokens. To the best of our knowledge, we are the first to show that these outputs provide an improved confidence signal for masking-style iterative generation. MDMs cannot use this signal unless combined with a flow map formulation, which is our methodological contribution. The effect is substantial: on unconditional Sudoku, the same FMLM+ checkpoint reaches 0% at 9 NFEs with a-priori confidence, and 96.0% with PR (Tab. 1).

Our second contribution is the connection between MDMs and FMLM+. The two objectives share their minimizer at s=t=0s = t = 0 (Prop. A.1), which allows FMLM+ to be initialized from pretrained MDMs. This significantly improves GSM8K accuracy (Tab. 4). We believe this is a meaningful contribution, as it offers a path to scale continuous language models using the growing pool of large pretrained MDMs (e.g., LLaDA [3]).


W2: Self-correction Claim.

We agree, and we state this in the Limitations section. By self-correct, we meant that PR renoises low-confidence tokens so that they get a chance to be remapped to a different data mode. In practice, confidence is strongly correlated with correctness: on Sudoku, tokens with confidence ≥0.9\geq 0.9 are correct 99.3%, 97.4% and 91.9% of the time on Easy, Medium and Hard (Tab. 3). To avoid confusion, we will revise our text to better reflect this.


W3: Block-wise decoding.

We mention block diffusion only once (L316–317), as an example of an inference schedule that FMLM+ supports: choosing the committed set UU as the next contiguous block of tokens commits tokens in block order. Such block-wise decoding is also how MDMs are commonly used in practice, e.g., in LLaDA [3] (Sec. 2.4). We do not claim that FMLM+ matches the block-causal attention or caching of block diffusion [4], and we will rephrase this as "block-wise decoding order" to make this explicit.


W4: MDM initialization and training compute.

We clarify that our main results do not rely on MDM initialization: every FMLM+ result in Tab. 2 is trained from scratch, and already matches or outperforms MDLM (e.g., 19.0% vs. 18.0% on GSM8K) with 32× fewer NFEs. The Init and Distill variants in Sec. 4.2 are an additional result, showing that pretrained MDMs can further improve FMLM+. To account for the additional training compute, we also train FMLM+ from scratch for 500k steps, matching the total compute of FMLM+ (Init). It reaches 23.1% on GSM8K at 32 NFE (vs. 19.0% at 250k steps), compared with 31.8% for FMLM+ (Init), so the benefit of initialization is not explained by compute alone.


W5 and Q5: Interpretation of a-posteriori confidence.

We agree that PR confidence is a heuristic, as are the confidence scores used in MDM decoding [5]. Since these methods matter precisely when the model makes errors, tight guarantees are generally challenging to establish. A-priori confidence is computed from δ0,0\delta_{0,0}, which equals the MDM posterior given only the context (Prop. A.1), while a-posteriori confidence is computed from the endpoint δ0,1\delta_{0,1}, which is one-hot for the exact flow. Our analysis shows that low confidence in δ0,1\delta_{0,1} certifies an error (Prop. A.3). Empirically, confidence is strongly correlated with correctness: on Sudoku, tokens with confidence ≥0.9\geq 0.9 are correct 99.3%, 97.4% and 91.9% of the time on Easy, Medium and Hard (Tab. 3). Following the reviewer's suggestion, we will simplify this discussion and remove the aleatoric/epistemic interpretation.

AuthorsOpenReview ↗

Rebuttal-2/2

W6.1 and Q3: FLM and distilled FMLM.

We agree that the FMLM baseline in Tab. 2, trained with only the diagonal objective, coincides with FLM [1], and we will relabel it accordingly. This choice was intentional, as all baselines in Tab. 2 are evaluated in their many-step regime, but we agree the label is confusing. We add a distilled FMLM with the same architecture and training budget as FMLM+. We note that the Gen-PPL of 45.09 at 32 NFE reported in [1] uses the γ\gamma-sampler, whereas all methods here use standard decoding. On OpenWebText, FMLM+ uses an improved PR schedule (8 rounds × 16 flow steps), which we will also update in Tab. 2.

DatasetNFE (FLM, FMLM / FMLM+)FLMFMLMFMLM+ (PR)
Sudoku Easy / Med. / Hard (%, ↑)180 / 494.2 / 82.7 / 44.594.6 / 86.2 / 50.097.9 / 92.0 / 71.2
GSM8K (%, ↑)1024 / 320.30.119.0
TinyStories Gen-PPL ↓ (entropy)128 / 3257.50 (4.17)62.81 (4.09)17.53 (3.96)
OpenWebText Gen-PPL ↓ (entropy)1024 / 12862.23 (5.33)45.05 (5.11)53.00 (5.26)

We note that FMLM+ (PR) significantly outperforms both FLM and FMLM on the two conditional generation tasks, while using 32–45× fewer NFEs. On TinyStories, it also achieves a significantly lower Gen-PPL with only a moderate drop in entropy. On OpenWebText, all three methods lie in a similar range with different entropy tradeoffs, while FMLM+ uses 8× fewer NFEs.


W6.2 and Q4: S-FLM variant.

We would like to clarify that we report the best-performing S-FLM variant on each task from [6]: S-FLM + α*(0.1) + adaptive on Sudoku (94.8 / 85.2 / 45.0 on Easy / Medium / Hard), and S-FLM with S-arch and top-k = 1 decoding on GSM8K (18.0%). For comparison, the base S-FLM in the same tables reaches 81.5 / 50.6 / 14.0 and 1.2%. We will name these variants explicitly in Tab. 2.


W7: Additional baselines.

We thank the reviewer for these pointers, and add RePlaid, IDLM and the concurrent DBTM [9] on GSM8K. LangFlow and Discrete-MMD do not report GSM8K results. Since RePlaid reports GSM8K only at 1024 steps and has not released code or checkpoints, we use its numbers from [7] without self-conditioning or temperature tuning, matching the setting of FMLM+. Methods using an MDLM teacher are compared with FMLM+ using the same teacher (Tab. 4).

GSM8K accuracy (%, ↑).

MethodTeacherNFEAcc.
RePlaid (DDPM) [7]–102417.1
RePlaid (DDIM) [7]–102413.0
DBTM [9]–3216.8
FMLM+ (PR)–3219.0
IDLM [8]MDLM3212.8
IDLM [8]Duo3215.4
FMLM+ (Distill, PR)MDLM3221.6
FMLM+ (Init, PR)MDLM3231.8

FMLM+ outperforms these methods on GSM8K at equal or fewer NFEs. Self-conditioning and temperature tuning, which improve RePlaid substantially [7], are orthogonal to our method, and we leave their combination with FMLM+ to future work.


W8 and Q2: Gen-PPL / entropy frontiers.

We agree that Gen-PPL should be compared at matched entropy [10]. Following [6], we sweep the sampling parameters of FMLM+ at 1024 NFE on OpenWebText and compare Gen-PPL with the baselines (taken from the temperature-swept frontiers in [6]) at matched entropy.

OpenWebText, Gen-PPL ↓ at matched entropy, NFE = 1024.

EntropyDuoMDLMS-FLMFLMFMLM+ (PR)
5.0023.7–34.343.425.1
5.0525.726.037.045.027.8
5.1630.332.643.349.432.7
5.1831.534.044.750.834.5
5.2334.437.249.454.440.4

At matched entropy, FMLM+ outperforms the continuous baselines while remaining close to the discrete baselines in this many-step regime. We thank the reviewer for this suggestion, and will report these frontiers in the revised paper.


Q1: Baseline results.

For Sudoku and GSM8K, we follow the exact protocol of [6] and report all baseline numbers (AR, MDLM, Duo, CANDI, FLM, S-FLM) from their paper. Following [6], all Sudoku baselines use 180 NFEs, and Tab. 2 incorrectly lists 128, which we will correct. For TinyStories, we train and evaluate all baselines from scratch using their released code, with the same architecture and training budget. For OpenWebText, we evaluate the released checkpoints of each baseline under the same protocol.


We hope these responses address your concerns, and would be grateful if you would consider them in your assessment.


[1] Lee et al. FMLM, '26

[2] Roos et al. Categorical Flow Maps, '26

[3] Nie et al. LLaDA, '25

[4] Arriola et al. Block Diffusion, '25

[5] Kim et al. Train for the Worst, Plan for the Best, '25

[6] Deschenaux et al. Hyperspherical Flows, '26

[7] Yang et al. RePlaid, '26

[8] Li et al. IDLM, '26

[9] Tang and Wang. DBTM, '26

[10] Pynadath et al. Generative Frontiers, '26

We thank the reviewer again for the constructive feedback. We have now revised our submission, with all changes highlighted in blue, and summarize them below.

  • We relabeled the FMLM baseline in Tab. 2 as FLM, and added a distilled FMLM baseline (Tab. 2, App. C.5). (W6.1, Q3)
  • We added RePlaid, IDLM and DBTM as additional baselines on GSM8K (App. C.6). (W7)
  • We added a comparison of Gen-PPL at matched entropy (App. C.4). (W8, Q2)
  • We added FMLM+ trained from scratch for 500k steps, matching the total compute of FMLM+ (Init) (Tab. 4). (W4)
  • We replaced "self-correct" with "refine" throughout the paper, removed the aleatoric/epistemic terminology (Sec. 3.3, Fig. 4), and rephrased block diffusion as block-wise decoding order (Sec. 3.2). (W2, W3, W5, Q5)
  • We described the source of each baseline and the S-FLM variants in App. B.2. (Q1, W6.2, Q4)

We believe that these updates strengthen this work, and thank the reviewer for prompting them. We would be grateful if the reviewer would consider them in their assessment.

Reviewer_fQ7YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces FMLM+ and Posterior Refinement (PR) to bridge the gap between Masked Diffusion Models (MDMs) and Flow Map Language Models (FMLMs). The authors extend FMLM (Lee et al. [35]) with a masking-style noise schedule and replace the shared scalar time conditioning in the Diffusion Transformer (DiT) backbone with per-position time pairs for the two-time denoiser. Observing that the diagonal term of the loss at s=t=0s = t = 0 coincides with the MDM objective allows the authors to warm-start FMLM+ from a pretrained MDLM checkpoint or distill from the logits of an MDLM teacher. Conditioning on arbitrary clean token subsets also enables Posterior Refinement (PR): at each round, the model drafts all remaining tokens jointly with a one- or few-step flow map, commits high-confidence tokens based on endpoint probability sharpness, and re-noises the rest for subsequent rounds. Empirical results show that FMLM+ with PR matches or outperforms discrete and continuous baselines with 8–32$\times$ fewer NFEs, and warm-starting from a pretrained MDLM checkpoint further boosts accuracy.

Critical strengths
  • Principled Unification of Continuous Flow Maps and Masked Diffusion with Direct Checkpoint Transfer: Proving that the diagonal flow objective shares the posterior minimizer with MDMs, and showing that the first adaLN block maps pure Gaussian noise at t=0t = 0 onto representations aligned with the MDM teacher's [MASK] token, is highly consequential because it provides a direct way to warm-start continuous flow maps from pretrained discrete MDMs.

  • Clean Empirical Diagnosis of Confidence-Based Decoding in MDMs: The paper provides a well-isolated empirical finding on where confidence-based decoding in MDMs fails (multimodal target distributions before symmetry breaking) and where confidence is usable. The controlled studies make a convincing case that the sharpness of the one-step flow-map endpoint δ0,1(x0)\delta_{0,1}(x_0) detects token errors, yielding a reliable a posteriori confidence signal for iterative refinement.

Critical weaknesses
  • The Efficiency Claim Is Not Tested Against the Right Controls: In Table 2 and Table 6, the baseline labeled "FMLM" is not trained with the full Flow Map Language Model objective of Lee et al. [35]. As noted in Section 4.1, it is trained only with the diagonal velocity objective (which Lee et al. designate as FLM). Although the paper's framing is that "FMLM+ beats both MDM and FMLM families at 8–32$\times$ fewer NFEs":

    • (a) the "FMLM" baseline is actually FLM evaluated at 128–1024 steps,
    • (b) no true FMLM (Ldiag+Loff\mathcal{L}_{\text{diag}} + \mathcal{L}_{\text{off}}) is evaluated at 1–32 NFEs, and
    • (c) there is no fixed-NFE ablation of refinement rounds RR versus flow steps NN. Specifically, the paper does not report an ablation comparing FMLM+ with Posterior Refinement (RR rounds ×\times NN flow steps) against a full FMLM or FMLM+ run with R=1R = 1 and R×NR \times N standard flow-map steps at the exact same NFE budget (e.g., 4, 32, or 128 NFEs). This makes it difficult to isolate how much of the improvement over "FMLM" in Table 2 is attributable to Posterior Refinement and masking schedules versus simply turning on the flow-map self-distillation objective Loff\mathcal{L}_{\text{off}}.
  • Omission of Stronger Confidence Heuristics: On line 161 (page 3) and in Eq. (14), the paper only considers maximum token probability max⁡vpθ(yl=v∣yt)\max_v p_\theta(y^l = v \mid y_t) as a confidence heuristic. However, the discrete diffusion literature has studied several stronger confidence criteria, including negative predictive entropy ∑vpvlog⁡pv\sum_v p_v \log p_v, top-2 probability margin p(1)−p(2)p_{(1)} - p_{(2)}, and sampled-token confidence, which are neither discussed nor compared against for MDLM or PR.

  • Training Overhead of EC\mathbb{E}_{\mathcal{C}} in Eq. (11) and Compute Parity: Introducing the expectation over random context sets EC\mathbb{E}_{\mathcal{C}} in Eq. (11) with mask mm masks out about 5050 % of tokens per batch from gradient supervision while requiring the model to generalize jointly over 2L2^L masking patterns and time triples (s,u,t)(s, u, t). This extra sample complexity likely explains why from-scratch FMLM+ lags far behind FMLM+ (Init) on GSM8K (19.0% vs. 31.8%). In addition, in Figure 3 and Table 4, FMLM+ (Init) is initialized from an MDLM teacher trained for 250k steps and then trained for an additional 250k steps (500k total steps), whereas FMLM+ from scratch is trained for 250k steps; without a 500k-step from-scratch baseline, the benefit of MDM initialization is partially confounded with 2×2\times total training compute. Finally, in Figure 3 (Right), the training loss of FMLM+ (Init) drops steeply until 20 - 25k steps and then monotonically increases throughout the remaining 225k steps, a counterintuitive optimization dynamic that is not discussed.

  • Inverse Perplexity Scaling with NFE on OpenWebText and Overstated "Self-Correction" Framing:

    • Perplexity–Entropy Confounding: On OpenWebText (Table 2), FMLM+ at 128 NFEs yields worse generative perplexity (66.60) than the diagonal FMLM baseline (62.23) alongside lower unigram entropy (5.21 vs. 5.52–5.71 for other diffusion baselines and 5.58 for AR). More importantly, Table 9 (Appendix C.3) shows that as PR rounds/NFEs increase on OpenWebText from 16→32→64→12816 \to 32 \to 64 \to 128, generative perplexity monotonically worsens from 30.4→52.7→59.9→66.630.4 \to 52.7 \to 59.9 \to 66.6 while entropy rises from 4.70→5.214.70 \to 5.21 (and similarly worsens on TinyStories from 14.8 at 16 NFEs to 17.5 at 32 NFEs). This indicates that low-NFE PR reduces sample diversity/entropy and that additional PR rounds do not monotonically improve sample quality on open-ended language modeling.
    • "Self-Correction" Framing: The Abstract and Introduction state that FMLM+ allows models to "recursively critique, erase and regenerate arbitrary subsets of tokens" and "scores the global consistency of each token a posteriori conditioned on the full draft." However, in Algorithm 1, committed tokens are monotonically frozen (commit = commit | C(tokens, x_1)) and never erased or revised once committed. When N=1N = 1 flow step per round is used (the setting for both GSM8K and Sudoku in Table 5), the model never takes the generated draft x^1\hat{x}_1 as input to critique it; instead, x^1=δ0,1(x0)\hat{x}_1 = \delta_{0,1}(x_0) and pmax⁡lp_{\max}^l are produced in the exact same single forward pass evaluated on Gaussian noise x0x_0.
Questions for authors
  • Ablation Against Standard Few-Step Flow Maps (R=1R=1) at Matched NFE: How does FMLM+ with Posterior Refinement (RR rounds ×\times NN flow steps) compare against a full FMLM trained with Ldiag+Loff\mathcal{L}_{\text{diag}} + \mathcal{L}_{\text{off}} (or the exact same FMLM+ checkpoint evaluated with R=1R = 1 round and Ntotal=R×NN_{\text{total}} = R \times N standard flow-map steps via Eq. 9) at matched NFE budgets across TinyStories (32 NFE), OpenWebText (128 NFE), Sudoku (4 NFE), and GSM8K (32 NFE)? Additionally, why does the 1024-step diagonal FMLM baseline collapse to 0.3% accuracy on GSM8K in Table 2?

  • Compute-Matched From-Scratch Baseline and Rising Loss in Section 4.2 (Figure 3): Since FMLM+ (Init) and FMLM+ (Distill) rely on an MDLM teacher trained for 250k steps followed by 250k steps of FMLM+ training (500k steps total), what accuracy and training loss does FMLM+ from scratch achieve when trained for 500k steps on TinyGSM? Also, can you explain why the training loss of FMLM+ (Init) in Figure 3 (Right) reaches a minimum around 25k steps and then steadily increase through 250k steps while test accuracy continues to improve?

  • Why Does Generative Perplexity Worsen With More NFEs on OpenWebText (Table 9)?: Why does FMLM+ generative perplexity on OpenWebText degrade from 30.4→52.7→59.9→66.630.4 \to 52.7 \to 59.9 \to 66.6 as NFEs increase from 16→12816 \to 128 (and from 14.8→17.514.8 \to 17.5 between 16 and 32 NFEs on TinyStories), and how does FMLM+ compare to baselines when sampling temperature or threshold is tuned to match unigram entropy (~5.55–5.60)?

  • Collapse of 1-Step Generation (N=1,R=1N=1, R=1) and Alternative Confidence Heuristics: Prior FMLM work (Lee et al. [35]) highlights competitive 1-NFE generation, yet in Table 1 (Unconditional Sudoku), Table 4 (GSM8K), and Table 10, FMLM+ achieves 0.0% accuracy at 1 NFE on GSM8K and unconditional Sudoku (and 4.1% on Hard Sudoku). Does conditioning on arbitrary masks C\mathcal{C} during training degrade the 1-step (C=∅\mathcal{C} = \emptyset) transport compared to a standard unconditional FMLM? Finally, please explain if the entropy- or margin-based confidence criteria can be applied to FMLM+ and PR?

(Minor presentation notes: The first row of numbers in Table 1 and Table 4 need to be explicitly labeled as NFEs / refinement rounds, and Lee et al. [35] should be cited explicitly at Equation (8) for the simplex KL/cross-entropy objective.)

7 comments
AuthorsOpenReview ↗

Rebuttal-1/2

We thank the reviewer for the careful and constructive review, and for finding our empirical analysis convincing. We address the reviewer's concerns below.


W1.1: Distilled FMLM baseline.

We agree that the FMLM baseline in Tab. 2 is trained with only the diagonal objective, which coincides with FLM [1], and we will relabel it accordingly. To clarify the source of the gains, we add a distilled FMLM with the same architecture and training budget as FMLM+. We compare FMLM+ (PR) with both FLM and FMLM using many-step sampling, as in Tab. 2. On OpenWebText, FMLM+ uses an improved PR schedule (8 rounds × 16 flow steps), which we will also update in Tab. 2.

DatasetNFE (FLM, FMLM / FMLM+)FLMFMLMFMLM+ (PR)
Sudoku Easy / Med. / Hard (%, ↑)180 / 494.2 / 82.7 / 44.594.6 / 86.2 / 50.097.9 / 92.0 / 71.2
GSM8K (%, ↑)1024 / 320.30.119.0
TinyStories Gen-PPL ↓ (entropy)128 / 3257.50 (4.17)62.81 (4.09)17.53 (3.96)
OpenWebText Gen-PPL ↓ (entropy)1024 / 12862.23 (5.33)45.05 (5.11)53.00 (5.26)

We note that FMLM+ (PR) significantly outperforms both FLM and FMLM on the two conditional generation tasks, while using 32–45× fewer NFEs. On TinyStories, it also achieves a significantly lower Gen-PPL with only a moderate drop in entropy. On OpenWebText, all three methods lie in a similar range with different entropy tradeoffs, while FMLM+ uses 8× fewer NFEs.


W1.2: Few-step Comparison.

At the same few-step NFE, FMLM+ (PR) outperforms the distilled FMLM on all benchmarks, e.g., 71.2% vs. 42.3% on Sudoku Hard and 19.0% vs. 0.0% on GSM8K.

DatasetNFEFMLMFMLM+ (PR)
Sudoku Easy / Med. / Hard (%, ↑)491.9 / 81.1 / 42.397.9 / 92.0 / 71.2
GSM8K (%, ↑)320.019.0
TinyStories Gen-PPL ↓ (entropy)3271.60 (4.11)17.53 (3.96)
OpenWebText Gen-PPL ↓ (entropy)12862.48 (5.28)53.00 (5.26)

W1.3: Fixed-NFE ablation.

We fix the budget at 32 NFE on GSM8K and vary the number of PR rounds RR and flow steps per round NN, using the same FMLM+ checkpoint. R=1R = 1 recovers standard flow-map decoding, and N=1N = 1 is our default setting on GSM8K.

Rounds RR × flow steps NNGSM8K (%, ↑)
1 × 32 (flow map)5.2
2 × 165.5
4 × 85.5
8 × 47.6
16 × 214.9
32 × 1 (ours)19.0

Since all rows use the same checkpoint, the gain from 5.2% to 19.0% comes from PR rather than from the self-distillation objective. Interestingly, we note that at a fixed budget, spending compute on refinement rounds is more effective than spending it on flow steps.


W2 and Q4.2: Alternative confidence heuristics.

We thank the reviewer for this suggestion. We can directly use these confidence scores with PR, since each is computed from the same endpoint distribution δ0,1\delta_{0,1}. We compare them on GSM8K at 32 NFE, first with a fixed number of tokens committed per round (top-kk), and then with our threshold rule (pmax⁡≥0.999p_{\max} \geq 0.999 with top-kk fallback).

Confidence scoreCommit ruleGSM8K (%, ↑)
Max probability pmax⁡p_{\max}top-kk6.9
Top-2 margintop-kk6.8
Negative entropytop-kk6.6
Sampled-token confidencetop-kk2.4
Max probability pmax⁡p_{\max} (ours)threshold + top-kk19.0

Max probability, margin and entropy perform similarly, since the endpoint δ0,1\delta_{0,1} is usually sharply peaked and all three then rank positions almost identically. We therefore keep pmax⁡p_{\max} as the simplest choice. The threshold rule further raises accuracy from 6.9% to 19.0%, consistent with Tab. 11.


W3.1 and Q2.1: Training compute and MDM initialization.

We agree that the comparison in Tab. 4 should account for the total training compute. We therefore train FMLM+ from scratch for 500k steps, matching the total compute of FMLM+ (Init), which uses an MDLM teacher trained for 250k steps followed by 250k steps of FMLM+ training.

GSM8K accuracy (%, ↑) by NFE.

MethodTotal stepsNFE = 1248163264
FMLM+250k0.00.12.98.713.419.019.1
FMLM+500k0.30.53.811.219.323.123.0
FMLM+ (Init)250k + 250k0.30.75.115.126.131.833.6

Training FMLM+ from scratch for 500k steps improves accuracy (23.1% vs. 19.0% at 32 NFE), but it remains well below FMLM+ (Init) at the same total compute (31.8%). The benefit of MDM initialization is therefore not explained by additional compute. We agree with the reviewer that learning over arbitrary context sets adds sample complexity. MDM initialization alleviates this, since the MDM has already learned the conditionals p(yl∣yC)p(y^l \mid y^{\mathcal{C}}) that FMLM+ targets at s=t=0s = t = 0.

AuthorsOpenReview ↗

Rebuttal-2/2

W3.2 and Q2.2: Training loss in Fig. 3.

We thank the reviewer for spotting this. Upon inspection, we found that the rise comes from how the plotted loss is normalized: it divides the summed loss by the number of tokens with non-zero loss. As training progresses, more tokens are predicted with near-certainty and drop out of this count (from 45% to 30% of tokens). With a fixed per-token normalization, both the diagonal and off-diagonal terms decrease monotonically for FMLM+ (Init), and it ends below FMLM+ trained from scratch (0.029 vs. 0.032), so the conclusion of Fig. 3 still holds. We will replot Fig. 3 with the fixed normalization.


W4.1 and Q3: Gen-PPL at matched entropy.

We agree that Gen-PPL should be compared at matched entropy [3]. Following [2], we sweep the sampling parameters of FMLM+ at 1024 NFE on OpenWebText and compare Gen-PPL with the baselines (taken from the temperature-swept frontiers in [2]) at matched entropy.

OpenWebText, Gen-PPL ↓ at matched entropy, NFE = 1024.

EntropyDuoMDLMS-FLMFLMFMLM+ (PR)
5.0023.7–34.343.425.1
5.0525.726.037.045.027.8
5.1630.332.643.349.432.7
5.1831.534.044.750.834.5
5.2334.437.249.454.440.4

At matched entropy, FMLM+ outperforms the continuous baselines while remaining close to the discrete baselines in this many-step regime. The increase in Gen-PPL with NFE in Tab. 9 follows from the same tradeoff: more rounds of PR raise sample entropy, and Gen-PPL rises with entropy along the frontier. We thank the reviewer for this suggestion, and will report these frontiers in the revised paper.


W4.2: Self-correction framing.

We agree that committed tokens are never revised, and we state this in the Limitations section. By self-correct, we meant that PR renoises low-confidence tokens so that they get a chance to be remapped to a different data mode. To avoid confusion, we will revise our text to better reflect this.

We also agree that with N=1N = 1, the model does not take the draft as input, since the draft and its confidence pmax⁡lp^l_{\max} come from the same forward pass. By "a posteriori", we mean that the confidence is computed at the endpoint of the joint transport, which determines all tokens together, rather than from token-wise marginals given only the context. We will clarify this in the abstract and introduction.


Q1.1: PR vs. standard flow-map decoding.

We thank the reviewer for this suggestion. We compare PR against standard flow-map decoding using the same FMLM+ checkpoint and the same number of forward passes, so training (including distillation) is held fixed.

DatasetNFEStandard flow mapPR
Sudoku Easy / Med. / Hard (%, ↑)488.3 / 67.5 / 29.697.9 / 92.0 / 71.2
GSM8K (%, ↑)325.219.0
TinyStories Gen-PPL ↓ (entropy)3250.70 (4.14)17.53 (3.96)
OpenWebText Gen-PPL ↓ (entropy)128198.88 (5.59)53.00 (5.26)

We note that PR outperforms standard flow-map decoding on every benchmark. It improves accuracy by 2.4× on Sudoku Hard and 3.7× on GSM8K, and it reduces Gen-PPL by 3–4× on both TinyStories and OpenWebText with only a moderate drop in entropy. Together with the random-selection ablation in Tab. 8, this shows that the gains of PR come from a-posteriori confidence, not from spending more forward passes.


Q1.2: FLM on GSM8K.

This result is not specific to our implementation: [2] reports the same accuracy for FLM on GSM8K (Tab. 2 of [2]), and the distilled FMLM shows the same behavior (W1.1). We hypothesize that this is because FLM denoises all positions in parallel without committing intermediate tokens, so an early error is hard to correct. This matters little for open-ended text generation, but on GSM8K a single wrong token can break the Python solution.


Q4.1: One-step generation.

Few-step generation on these tasks is considerably harder than the open-ended text generation studied in [1] (Q1.2). Indeed, the distilled FMLM also fails at few steps (W1.2). We agree that FMLM+ may be somewhat weaker than FMLM at exactly one step, as it shares its capacity across all context sets C\mathcal{C} rather than only C=∅\mathcal{C} = \emptyset. In practice, FMLM+ spends its budget on refinement rounds rather than a single jump, which is where it is most effective (W1.3).


Minor presentation notes.

We thank the reviewer for these suggestions. We will label the first row of Tables 1 and 4 as NFEs, and cite [1] explicitly for Eq. (8).


We hope these responses address your concerns, and would be grateful if you would consider them in your assessment.


[1] Lee et al. FMLM, '26

[2] Deschenaux et al. Hyperspherical Flows, '26

[3] Pynadath et al. Generative Frontiers, '26

We thank the reviewer again for the constructive feedback. We have now revised our submission, with all changes highlighted in blue, and summarize them below.

  • We relabeled the FMLM baseline in Tab. 2 as FLM, added a distilled FMLM baseline, and added comparisons at matched few-step NFE together with the fixed-NFE ablation over rounds and flow steps (Tab. 2, App. C.5). (W1.1, W1.2, W1.3, Q1.1)
  • We added a comparison of alternative confidence scores for PR (App. C.8). (W2, Q4.2)
  • We added FMLM+ trained from scratch for 500k steps, matching the total compute of FMLM+ (Init) (Tab. 4), and replotted the training loss in Fig. 3. (W3.1, W3.2, Q2.1, Q2.2)
  • We added a comparison of Gen-PPL at matched entropy, and explained the increase in Gen-PPL with NFE on OpenWebText (App. C.3, C.4). (W4.1, Q3)
  • We replaced "self-correct" with "refine" throughout the paper. (W4.2)
  • We labeled the NFE rows in Tab. 1 and Tab. 4, and cited Lee et al. at Eq. (8). (Minor)

We believe that these updates strengthen this work, and thank the reviewer for prompting them. We would be grateful if the reviewer would consider them in their assessment.

Reviewer_Cd5JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FMLM+, an extension of Flow Map Language Models that supports generation conditioned on arbitrary subsets of clean tokens through masking-style noise schedules. Building on this formulation, the authors propose Posterior Refinement (PR): the model jointly generates candidate tokens, retains confident predictions, and regenerates the remaining positions from fresh noise. The paper also establishes a shared optimal predictor between the FMLM+ boundary objective and masked diffusion, motivating initialization and distillation from pretrained MDMs. Experiments results demonstrate substantial reductions in function evaluations relative to the evaluated non-autoregressive baselines, supported by wall-clock measurements and several inference ablations.

Critical strengths
  1. An interesting extension of flow-map language modeling. I find flow-map language modeling a promising direction for generating text in fewer steps. This paper builds on that idea by allowing the model to keep selected tokens and regenerate the rest. Combining this flexibility with confidence-guided refinement is a meaningful contribution that could be useful beyond the tasks studied here.

  2. A clear motivation and a sensible design. To refine a draft efficiently, a model needs to generate several tokens coherently while preserving useful parts of the existing text. FMLM+ brings these two capabilities together. Using confidence from the predicted endpoint to decide which tokens to keep is a sensible idea, and the paper clearly explains how this differs from using confidence based only on the currently available context.

  3. Strong results on structured generation, with useful mechanism-level evidence. The Sudoku results are particularly compelling. On the Hard split, PR achieves 71.2% accuracy with four NFEs, compared with 58.4% for Duo using 128 NFEs. The equal-compute random-selection comparison and the analysis of repairs versus regressions provide evidence that confidence-guided selection contributes substantially beyond repeated sampling alone.

Critical weaknesses

The comparisons leave the source of the gains unclear. The FMLM baseline in Table 2 is trained without the distillation objective used for few-step generation, while FMLM+ includes this objective. This makes it difficult to tell how much of the improvement comes from distillation and how much comes from the proposed method. Comparing against a distilled FMLM under matched budgets would help. Using the same FMLM+ checkpoint to compare standard flow-map decoding with PR at equal inference cost would also clarify what PR itself adds.

Questions for authors

The FMLM baseline in Table 2 is trained without the distillation objective used for few-step generation. How would FMLM+ compare with a distilled FMLM given the same training and inference budgets? It would also be helpful to compare standard flow-map decoding and PR using the same FMLM+ checkpoint and the same number of forward passes. Table 8 shows that confidence-based selection works better than random selection, but would PR also outperform spending those forward passes on standard flow-map decoding?

3 comments

We thank the reviewer for the constructive comments, and for finding our design well motivated and our experiments compelling. We address the concerns below.


W1.1 and Q1.1: Comparison with distilled FMLM.

We agree that the FMLM baseline in Tab. 2 is trained with only the diagonal objective, which coincides with FLM [1], and we will relabel it accordingly. To clarify the source of the gains, we add a distilled FMLM with the same architecture and training budget as FMLM+. We first compare FMLM+ (PR) with both FLM and FMLM using many-step sampling, as in Tab. 2, and then with FMLM at the same few-step NFE. On OpenWebText, FMLM+ uses an improved PR schedule (8 rounds × 16 flow steps), which we will also update in Tab. 2.

DatasetNFE (FLM, FMLM / FMLM+)FLMFMLMFMLM+ (PR)
Sudoku Easy / Med. / Hard (%, ↑)180 / 494.2 / 82.7 / 44.594.6 / 86.2 / 50.097.9 / 92.0 / 71.2
GSM8K (%, ↑)1024 / 320.30.119.0
TinyStories Gen-PPL ↓ (entropy)128 / 3257.50 (4.17)62.81 (4.09)17.53 (3.96)
OpenWebText Gen-PPL ↓ (entropy)1024 / 12862.23 (5.33)45.05 (5.11)53.00 (5.26)

We note that FMLM+ (PR) significantly outperforms both FLM and FMLM on the two conditional generation tasks, while using 32–45× fewer NFEs. On TinyStories, it also achieves a significantly lower Gen-PPL with only a moderate drop in entropy. On OpenWebText, all three methods lie in a similar range with different entropy tradeoffs, while FMLM+ uses 8× fewer NFEs.

DatasetNFEFMLMFMLM+ (PR)
Sudoku Easy / Med. / Hard (%, ↑)491.9 / 81.1 / 42.397.9 / 92.0 / 71.2
GSM8K (%, ↑)320.019.0
TinyStories Gen-PPL ↓ (entropy)3271.60 (4.11)17.53 (3.96)
OpenWebText Gen-PPL ↓ (entropy)12862.48 (5.28)53.00 (5.26)

At the same few-step NFE, FMLM+ (PR) outperforms FMLM on every benchmark. FMLM drops to 42.3% on Sudoku Hard and 0.0% on GSM8K, while FMLM+ (PR) achieves 71.2% and 19.0%. We conclude from these results that distillation alone does not explain the gains of FMLM+.


W1.2 and Q1.2: PR vs. standard flow-map decoding.

We thank the reviewer for this suggestion. We compare PR against standard flow-map decoding using the same FMLM+ checkpoint and the same number of forward passes, so training (including distillation) is held fixed.

DatasetNFEStandard flow mapPR
Sudoku Easy / Med. / Hard (%, ↑)488.3 / 67.5 / 29.697.9 / 92.0 / 71.2
GSM8K (%, ↑)325.219.0
TinyStories Gen-PPL ↓ (entropy)3250.70 (4.14)17.53 (3.96)
OpenWebText Gen-PPL ↓ (entropy)128198.88 (5.59)53.00 (5.26)

We note that PR outperforms standard flow-map decoding on every benchmark. It improves accuracy by 2.4× on Sudoku Hard and 3.7× on GSM8K, and it reduces Gen-PPL by 3–4× on both TinyStories and OpenWebText with only a moderate drop in entropy. Together with the random-selection ablation in Tab. 8, this shows that the gains of PR come from a-posteriori confidence, not from spending more forward passes.

Furthermore, we fix the budget at 32 NFE on GSM8K and vary the number of PR rounds RR and flow steps per round NN. R=1R = 1 recovers standard flow-map decoding, and N=1N = 1 is our default setting on GSM8K.

Rounds RR × flow steps NNGSM8K (%, ↑)
1 × 32 (flow map)5.2
2 × 165.5
4 × 85.5
8 × 47.6
16 × 214.9
32 × 1 (ours)19.0

Interestingly, we note that at a fixed budget, spending compute on refinement rounds is more effective than spending it on flow steps.


We hope these responses address your concerns, and would be grateful if you would consider them in your assessment.


[1] Lee et al. FMLM, '26

We thank the reviewer again for the constructive feedback. We have now revised our submission, with all changes highlighted in blue, and summarize them below.

  • We relabeled the FMLM baseline in Tab. 2 as FLM, and added a distilled FMLM baseline. On OpenWebText, FMLM+ now uses an improved PR schedule (Tab. 2, App. B.3). (W1.1, Q1.1)
  • We added comparisons against the distilled FMLM and standard flow-map decoding at matched few-step NFE, as well as the fixed-NFE ablation over rounds and flow steps (App. C.5). (W1.1, W1.2, Q1.1, Q1.2)

We believe that these updates strengthen this work, and thank the reviewer for prompting them. We would be grateful if the reviewer would consider them in their assessment.

We thank the reviewer for the response, and are glad that the additional experiments address the reviewer's main concerns.

Reviewer_QFBoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces FMLM+, which extends Flow Map Language Models to masking-style noise schedules. Clean context tokens are pinned at t=1 through per-position time conditioning, and the remaining tokens follow the usual Gaussian-to-one-hot flow. This lets one model do both joint few-step transport (like FMLM) and conditioning on arbitrary subsets (like MDMs). On top of this, the authors propose Posterior Refinement (PR). PR generates the full sequence with a one-step flow-map jump, commits the tokens whose endpoint confidence p_max is above a threshold, re-noises the rest, and repeats. The authors also show that the MDM objective is the s=t=0 boundary case of the FMLM+ objective, and use this to distill from or initialize with a pretrained MDM. Experiments on TinyStories, OpenWebText, GSM8K (trained on TinyGSM) and Sudoku report matching or better quality than MDM and continuous baselines with 8–32× fewer NFEs.

Critical strengths

I liked quite a few things here. The core idea is simple and clean. Per-position time conditioning, with t=1 on the clean context tokens, lets one model handle both FMLM-style and MDM-style inference, with no other architectural change. I also liked the two-step design of draft-then-refine. FMLM+ drafts the whole sequence in one shot, and Posterior Refinement then decides which parts of that draft to keep.

The training objective is neat too. The diagonal term (L_diag) trains the denoiser δ_{s,s} to reconstruct the clean tokens at each noise level. The off-diagonal term (L_off) uses the semigroup property to distill a large jump s→t from two smaller jumps, s→u and u→t, with a stop-gradient teacher. So the model learns to take few-step jumps through self-distillation in one stage, rather than the two-stage training of Lee et al. Dropping that extra stage is a nice simplification.

The authors also notice a useful connection. At the boundary s=t=0, the off-diagonal term vanishes, and FMLM+ is just predicting the original tokens from clean context plus pure Gaussian noise. That’s essentially the MDM task: predict missing tokens from the unmasked context. The inputs differ (noise vs. [MASK]), but Prop. A.1 shows both objectives share the same minimizer, p(y^l | y^C). I appreciated that this isn’t just a theoretical remark, since it gives a direct route to distilling from or initializing with a pretrained MDM. The selection rule is also refreshingly simple. Commit the positions whose top token probability exceeds a threshold. If none clears it, fall back to committing the top-k most confident positions, so the process doesn’t stall.

The MDM↔FMLM+ correspondence (Prop. A.1) is neat and sound as far as I checked. FMLM+ (Init) reaches 31.8% vs. 19.0% from scratch on GSM8K (Tab. 4). The adaLN analysis (Fig. 5) is a nice mechanistic explanation for why warm-starting from [MASK]-trained weights works with Gaussian inputs. The controls are good: random selection at equal compute (Tab. 8), the repairs vs. regressions breakdown (Tab. 7), and confidence-binned accuracy (Tab. 3). These are exactly the experiments I’d want for a refinement method. The wall-clock measurements (Tab. 6) are Limitations are honest, which i like.

Critical weaknesses

1

Structurally, the PR loop (generate in parallel, keep high-confidence tokens, re-noise the rest, repeat) is Mask-Predict. Iterative refinement for non-AR text goes back isn't brand new . Remasking and corrector schemes for discrete diffusion already revisit tokens conditioned on a full draft. So the claim that PR is “native to neither the MDM nor the FMLM paradigms” feels too strong to me. Same goes for per-token noise levels as conditioning . As far as I can tell, what’s new is using the flow-map endpoint confidence as the signal, plus the MDM warm-start. Both are useful, but I see this as a careful adaptation rather than a new class of method.

2

I feel the paper can benfit from using precise language and several headline claims exceed the evidence.

  1. “State of the art across all benchmarks” feels a bit overstated. On OWT, FMLM has better Gen-PPL than FMLM+ (62.23 vs. 66.60). FMLM+’s low entropy may partly explain its low-NFE gains; as NFEs increase, OWT perplexity worsens while entropy rises.

  2. I feel the GSM8K advantage is not clearly significant. FMLM+ scores 19.0% vs. 18.0% for the baseline—about a one-point gap, with no seeds or error bars I feel “Matches” is safer to use than “outperforms.”

  3. Self-correct overstates what PR does. Once tokens are committed, they are never revisited, so committed errors persist. Please state this better.

  4. Confidence is a heuristic, not a guarantee. The theory propositions establish distance identities, not error detection. Empirical support is limited, and the epistemic/aleatoric explanation is a hypothesis based partly on a toy example.

  5. The scalability claim is premature. Experiments use models up to 170M parameters; OWT samples are not high-fidelity, and the L×VL\times V state may be costly at large vocabularies.

Especially witrh regards to points 2, 4, and 5 while this paper is timely in its research question, but not yet decisive in its evidence. I see this paper as a timely early contribution.

3

Writing can be improved at certian places: 1) when factoriztaion error is discussed on page 1, please elaborate a bit more. for ex. a schematic could help. (this helps the flow) 2) In eq. 4 I˙t is never defined. The dot notation needs to be defined else that can make equations harder to read.

Questions for authors

Q1 FMLM+ drafts several tokens at once, and Posterior Refinement chooses which parts of that draft to commit and which to regenerate. But confidence is a selection signal, not a guarantee that a committed token is correct, and the current method never revises a token once it is committed. How often do committed tokens turn out to be wrong (e.g., on Sudoku, where correctness can be checked exactly)? Have you tried any way to recover from these errors, such as letting PR un-commit tokens? And do you expect the confidence signal to stay this reliable beyond the small-scale settings tested here?

Q2 Can you report Gen-PPL at matched entropy, or MAUVE, and explain why OWT Gen-PPL gets worse as NFE grows in Tab. 9?

5 comments

We thank the reviewer for the careful and positive review, and for finding our experiments well designed. We address the reviewer's concerns below.


W1: Relation to Mask-Predict.

We agree that PR and MDM-based decoding such as Mask-Predict [1] share the same structure. We call PR a new class of method because our choice of confidence makes a substantial difference in practice, and its use is only enabled by flow map training, which is fundamentally unavailable for MDMs. While we believe this wording is adequate, we would be happy to tone it down if the reviewer still finds it necessary.

The difference stems from factorization error. MDMs sample each token independently from its marginal, which leads to inconsistencies when generating several tokens in parallel. FMLM+ instead generates all tokens jointly, and PR scores each token after the full draft is generated. This is clearly visible on unconditional Sudoku: decoded with a-priori confidence (δ0,0\delta_{0,0}), FMLM+ fails like MDLM, while PR succeeds in a few steps.

Unconditional Sudoku, accuracy (%) (Tab. 1).

MethodNFE = 3NFE = 9NFE = 27NFE = 81
MDLM (confidence)0.00.00.097.2
FMLM+ (δ0,0\delta_{0,0}, a-priori)0.00.00.098.8
FMLM+ (δ0,1\delta_{0,1}, PR)49.396.098.399.1

We note that while remasking samplers such as ReMDM [2] revisit tokens conditioned on the full draft, they still resample them independently from MDM marginals, inheriting the same factorization error.


W2.1 and W2.2: Claims on OWT and GSM8K.

We agree and will revise our text to better reflect our results. We note that while the GSM8K accuracies in Tab. 2 are close, FMLM+ with PR uses 32× fewer NFEs than the baselines. At the same budget of 32 NFEs, MDLM reaches 9.0% and FMLM 0.0%, compared with 19.0% for FMLM+ with PR.


W2.3: Self-correction Claim.

By self-correct, we meant that PR renoises low-confidence tokens so that they get a chance to be remapped to a different data mode. To avoid confusion, we will revise our text to better reflect this.


W2.4: Confidence Heuristic.

We agree that PR confidence is a heuristic, as are the confidence scores used in MDM decoding [3]. Since these methods matter precisely when the model makes errors, tight guarantees are generally challenging to establish. Our analysis shows that low confidence certifies an error (Prop. A.3). Empirically, confidence is strongly correlated with correctness. On Sudoku, tokens with confidence ≥0.9\geq 0.9 are correct 99.3%, 97.4% and 91.9% of the time on Easy, Medium and Hard (Tab. 3).


W2.5: Scalability Claim.

We agree with the reviewer's comment and would like to note that our limitations section has made similar remarks. Given resource limits, we leave the scaling up of our methods to future work. We would also like to add that maintaining an L x V state can be optimized for memory consumption through techniques such as tiling. In our experiments, the L x V state increases peak memory by only ~1.2x compared to MDLM.


W3: Writing Notation.

We thank the reviewer for these suggestions. In the revision, we will expand the discussion of factorization error and define the dot notation I˙t=ddtIt\dot{I}_t = \frac{d}{dt} I_t in Eq. (4).


Q1: Correctness of committed tokens.

We point the reviewer to Tab. 3 and Tab. 7, where we conduct this analysis on Sudoku. Renoising low-confidence tokens repairs incorrect tokens far more often than it breaks correct ones (Tab. 7). Allowing PR to uncommit tokens is an exciting direction that we leave to future work. Given the strong correlation between confidence and correctness (Tab. 3), we are optimistic that PR-like confidence will remain useful at larger scales.


Q2: Gen-PPL at matched entropy.

Following [4], we sweep the sampling parameters of FMLM+ at 1024 NFE on OpenWebText and compare Gen-PPL with the baselines (taken from [4]) at matched entropy.

OpenWebText, Gen-PPL ↓ at matched entropy, NFE = 1024.

EntropyDuoMDLMS-FLMFLMFMLM+ (PR)
5.0023.7–34.343.425.1
5.0525.726.037.045.027.8
5.1630.332.643.349.432.7
5.1831.534.044.750.834.5
5.2334.437.249.454.440.4

At matched entropy, FMLM+ outperforms the continuous baselines while remaining close to the discrete baselines in this regime. The increase in Gen-PPL with NFE in Tab. 9 reflects the same tradeoff, as more PR rounds raise entropy [5]. We will report these frontiers in the revised paper.


We hope these responses address your concerns, and would be grateful if you would consider them in your assessment.


[1] Ghazvininejad et al. Mask-Predict, '19

[2] Wang et al. ReMDM, '25

[3] Kim et al. Train for the Worst, Plan for the Best, '25

[4] Deschenaux et al. Hyperspherical Flows, '26

[5] Pynadath et al. Generative Frontiers, '26

We thank the reviewer again for the constructive feedback. We have now revised our submission, with all changes highlighted in blue, and summarize them below.

  • We revised our claims on OWT and GSM8K in the introduction and Sec. 4.1, and added a comparison of Gen-PPL at matched entropy in App. C.4. (W2.1, W2.2, Q2)
  • We replaced "self-correct" with "refine" throughout the paper, to better reflect that PR renoises low-confidence tokens before committing them. (W2.3)
  • We cited Mask-Predict and ReMDM alongside MDM confidence-based sampling in Sec. 3.2. (W1)
  • We added an example of factorization error in Sec. 1, and defined the dot notation after Eq. (4). (W3)

We believe that these updates strengthen this work, and thank the reviewer for prompting them. We would be grateful if the reviewer would consider them in their assessment.

We thank the reviewer again for their time and for the detailed review. As the discussion period closes on November 18, we would greatly appreciate knowing whether our rebuttal and revisions have addressed the reviewer's concerns. If any concern remains, we would be happy to provide further clarification.

Comments on the paper

1 comment

We thank all reviewers for the constructive feedback. We summarize below for the AC and reviewers how our submission was revised in response, with all changes highlighted in blue.

  • We relabeled the FMLM baseline in Tab. 2 as FLM, and added a distilled FMLM baseline. FMLM+ outperforms it on every benchmark at matched few-step NFE (Tab. 2, App. C.5). (Cd5J, fQ7Y, 8Gro)
  • We added new ablations for PR: standard flow-map decoding, a fixed-NFE sweep over rounds and flow steps, and alternative confidence scores (App. C.5, C.8). (Cd5J, fQ7Y)
  • We added a comparison of Gen-PPL at matched entropy on OpenWebText, where FMLM+ outperforms all continuous baselines (App. C.4). (QFBo, fQ7Y, 8Gro)
  • We added FMLM+ trained from scratch for 500k steps, confirming that the benefit of MDM initialization is not explained by compute alone (Tab. 4). (fQ7Y, 8Gro)
  • We added RePlaid, IDLM and DBTM as additional baselines on GSM8K (App. C.6). (8Gro)
  • We revised our claims and terminology to better reflect our results, including replacing "self-correct" with "refine" throughout the paper. (QFBo, fQ7Y, 8Gro)

We believe that these updates address the main concerns raised by the reviewers and strengthen this work, and thank the reviewers for prompting them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.