acceptodds
Under review as a conference paper at ICLR 2027

Context-weighted Discrete Flow Matching

Abstract

Discrete flow matching provides a flexible framework for generative modeling on discrete structures. However, the standard factorized training objective exposes the model to targets of varying difficulty, mixing well-conditioned, predictable tokens with ambiguous, high-entropy ones. We empirically demonstrate that the uncertainty over the value of each token is closely related to the density of available context in its neighborhood. Motivated by this observation, we propose a simple modification to the underlying Continuous-Time Markov Chain (CTMC) that incorporates local context information. Our context-weighted sampler improves generation quality with negligible computational overhead, while our scaled cross-entropy loss function reweights the training signal from different tokens and reduces generative perplexity by up to 63% on OpenWebText. Moreover, our approach matches a strong semi-autoregressive block diffusion baseline in quality while retaining the ability for any-order generation. These results highlight the role of local context as an important factor in discrete generative modeling and show that simple context-aware modifications can significantly improve both sampling and training efficiency.

Primary area:
generative models
Keywords:
flow matching, language modelling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BxKGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the relationship between local context and token prediction difficulty in discrete flow matching. It introduces context-dependent coordinate transition weights based on neighboring revealed or endpoint-agreeing positions. For a fixed-endpoint conditional CTMC, a normalization condition preserves the distribution of completed transitions and the source and target endpoints; coordinate marginals are also preserved under additional symmetry assumptions.

The paper develops Neighbor-weighted and Entropy-weighted inference procedures, a Scaled Cross-Entropy (SCE) objective that retains the standard training path, and a token-wise sampler for the modified training path. Experiments primarily cover OpenWebText and QM9, with additional transfer results for larger text models and image generation. The main empirical improvements come from uniform-source SCE and inference-only context-weighted sampling.

Critical strengths
  1. The improvements over the directly matched baseline are practically meaningful

On uniform-source OWT, SCE increases MAUVE from 0.628 to 0.777 while reducing generative perplexity from 82.17 to 30.20 (Tables 1–2). The direct DFM baseline uses the same architecture, data, and evaluation setup (Appendix B.1). The accompanying MAUVE improvement makes the result more compelling than a reduction in external-language-model perplexity alone, and the comparisons with MDLM, SEDD, and BD3LM provide useful context. These results support a potentially valuable practical contribution.

  1. The method combines a simple structural construction with useful transfer evidence

The normalization that preserves transition-count distributions for fixed endpoints, together with exact token-wise path sampling for time-independent weights, is a useful design. The inference-only NW modification also requires no additional training. Transferring the small-model hyperparameters to a public 1.3B model decreases generative perplexity from 49.75 to 43.19 and increases MAUVE from 0.1923 to 0.3175, while token entropy also increases (Table 6). This provides initial evidence of practical value beyond a single small model.

Critical weaknesses
  1. The training contribution is not sufficiently distinguished from closely related planner-aware approaches

PAPL (Planner Aware Path Learning in Diffusion Language Models Training, ICLR 2026) already uses confidence-weighted cross-entropy while retaining standard random masking (Section 3.4, Eq. 7 of that paper). The submission does not discuss this work, and Table 2 compares only CE, NELBO, and Bregman losses. The neighborhood proxy, count-preserving conditional path, and applicability to uniform-source dynamics may provide meaningful distinctions, but their independent value is not sufficiently established.

The locality correlations in Figure 2 and the final metric improvements do not isolate the contribution of locality itself. In particular, the headline uniform-source OWT result is obtained with the original path; switching to the modified path does not improve the corresponding results in Table 3. Consequently, the evidence does not yet establish how much of the main training improvement is specifically attributable to the proposed local-context design.

  1. The connection between the conditional-path theory and the practical uniform-source procedures needs a more precise definition

Theorem 1 concerns a fixed-endpoint process. Under a uniform source, neighborhood agreement and its normalization depend on the endpoint, which is replaced by newly sampled proposals at inference (Section 3.2). Marginalizing the resulting weighted rates generally involves joint endpoint dependencies.

The paper should clarify how independent token proposals approximate this process, as well as the exact SCE weighting, normalization set, and treatment of already agreeing coordinates. The principal SCE experiments also retain the original path and use ordinary Euler sampling (Appendix B.2), so the rate-divergence homogeneity identity in Appendix A.4 does not by itself establish consistency of the practical combination.

I am not claiming that Theorem 1 is incorrect; my concern is that the relationship between conditional guarantees and empirical design choices is insufficiently specified. The claim about preserving the stepwise update-count distribution at lines 265–266 also needs to be distinguished from the continuous-time guarantee.

Questions for authors

Q1 — Clarification and targeted validation.

What distinguishes SCE from PAPL? Could the authors provide a matched-backbone, matched-budget comparison in one existing masked-source setting where both directly apply, or other evidence that distinguishes the two weighting mechanisms? This does not require developing a new uniform-source extension of PAPL.

Q2 — Method clarification.

Please provide the complete SCE weighting and normalization formulas. For a uniform source, do training weights use ground-truth endpoint agreement, the corruption process’s reveal indicator, or model proposals, and how are already agreeing coordinates treated? Please also separate fixed-endpoint CTMC guarantees from properties of the implemented proposal-redrawing sampler, including the precise scope of the “stepwise distribution” claim at lines 265–266.

Q3 — Evaluation clarification and targeted validation.

Table 11 reports worse test-perplexity upper-bound values for uniform-source SCE on all three datasets despite improved MAUVE and generative perplexity. How are these bounds estimated, and what evidence distinguishes improved local learning from a changed quality–diversity operating point? A matched-diversity sampling comparison using existing checkpoints, or a control that disrupts locality while preserving the weight distribution, would be informative. I do not interpret the increased upper-bound values as proof that the true likelihood deteriorates.

Q4 — Comparison conditions.

What NFE, hyperparameter-selection protocol, and number of training runs underlie Tables 1–3 and Table 6? Does Figure 5 include a predictor–corrector baseline without NW under the same corrector configuration? What NFE does EB actually use at γ = 0.001 in Table 7? These details materially affect the assessment of the quality–compute trade-off.

Reviewer_Ut6NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a context-aware approach to Discrete Flow Matching. The authors find that tokens with more unmasked neighbors tend to have lower predictive entropy and negative log-likelihood. They use this observation to design a context-weighted sampler and a scaled cross-entropy loss (SCE). The sampler adjusts token update rates and works with pretrained models without fine-tuning, while SCE changes the weights of token-level training losses. Experiments on OpenWebText and QM9 show improvements in text and molecular generation. SCE reduces generative perplexity by approximately 63% on OpenWebText under uniform-source noise.

Critical strengths
  1. The method is simple and practical. The sampler requires no fine-tuning, and SCE does not require changes to the model architecture. Both modifications are easy to incorporate into existing models.

  2. The experiments cover both text and molecular generation. The ablation studies are reasonably thorough and help clarify the effects of the main design choices.

Critical weaknesses
  1. The observation that more local context generally makes prediction easier is intuitive. The conceptual novelty is therefore somewhat limited, although its practical use is valuable.

  2. In Section 3.1, the authors argue that better-determined targets provide more informative supervision, motivating higher training weights for tokens with more local context. However, Figure 2 only examines the baseline model, while the SCE experiments mainly report aggregate results. These results support the method’s effectiveness, but do not directly show how SCE changes token-level NLL across different context densities.

Questions for authors

Could you compare CE and SCE token-level NLL across context densities, following Figure 2? This would help me understand which positions benefit from SCE and whether predictions at positions with less context become worse.

Reviewer_QneiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces context-weighted discrete flow matching, prioritizing token updates based on local context density. It introduces an inference-time sampling strategy and a scaled cross-entropy training objective. Experiments on text and molecular generation show improvements in generation quality, particularly for uniform-source DFM.

Critical strengths

1.⁠ The proposed sampling approach is straightforward and can be applied to pretrained DFM models without retraining.

Critical weaknesses
  1. Insufficient motivation and theoretical justification. The paper shows that local context density correlates with lower prediction uncertainty but does not establish why easier tokens can serve as a better training signal. The entropy-KL decomposition does not directly justify the proposed loss weighting.

  2. Limited experimental validation and generalization. The comparisons with recent, adaptive sampling methods are insufficient. The reported 63% generative perplexity improvement is specific to uniform-source DFM, while Appendix Table 11 reports substantially worse test-perplexity performance across three external datasets. Moreover, the evaluation does not cover known benchmarks and metrics other than perplexity.

Questions for authors

1.⁠ ⁠Why should low-entropy tokens provide more useful training gradients than "vanilla" tokens? 2.⁠ ⁠Can the method be compared against recent adaptive sampling baselines under matched computational budgets, and can the authors better define successful text generation with additional evidence of coherence and quality other than perplexity? 3.⁠ ⁠Do the theoretical guarantees extend to inference with model-predicted endpoints? 4.⁠ ⁠How do the authors explain the discrepancy between improved generative perplexity and worse test-perplexity bounds in Appendix Table 11? 5.⁠ ⁠Why does Neighbor-weighted sampling underperform Euler at low NFE in Figure 3?

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

As the discussion period draws to a close, we thank you for your time and feedback. Below, we summarize our responses to the main concerns. Numerical results and experimental details are provided in the individual replies.

  • Training motivation. [Qnei, Ut6N] We derived the conditional logit-gradient variance, explaining how better-conditioned targets can provide lower-noise supervision.
  • Novelty and relationship to PAPL. [BxKG, Ut6N] We clarified the differences between confidence-based and structural locality weights, and CW-DFM’s broader source-distribution scope within the convex-mixture family.
  • Token-level effects and isolation of locality. [BxKG, Ut6N] We provided CE versus SCE NLL comparisons across context densities and a shuffled-weight control. The latter supports the importance of weight assignment based on locality principle.
  • Contribution of the modified path. [BxKG] We explained that locality already enters through SCE under the original path. Table 3 evaluates the modified-path configuration, whose benefits depend on the source and dataset.
  • Weighting formulas and theoretical guarantees. [BxKG, Qnei] We specified SCE’s normalization and treatment of agreeing tokens, distinguished fixed-endpoint CTMC guarantees from proposal-redrawing inference, and clarified the samplers’ update-count properties.
  • Evaluation breadth and adaptive baselines. [Qnei] We pointed to the existing entropy, MAUVE, chemical validity and novelty metrics, and adaptive EB sampler comparison.
  • Held-out bounds and quality–diversity trade-offs. [BxKG, Qnei] We explained the standard ELBO evaluation and its upper-bound interpretation, and provided an entropy-matched generation comparison.
  • NW at low NFE. [Qnei] We explained the hypothesis that coarse parallel updates leave fewer opportunities to use newly revealed context.
  • Comparison conditions. [BxKG] We clarified sampling budgets, documented hyperparameter sweeps, and the training protocol. We also supplied the best-to-best predictor–corrector comparison with and without NW and EB’s measured model-call count.

We have incorporated these clarifications and additional results into the revised manuscript and would be happy to address any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.