- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the relationship between local context and token prediction difficulty in discrete flow matching. It introduces context-dependent coordinate transition weights based on neighboring revealed or endpoint-agreeing positions. For a fixed-endpoint conditional CTMC, a normalization condition preserves the distribution of completed transitions and the source and target endpoints; coordinate marginals are also preserved under additional symmetry assumptions.
The paper develops Neighbor-weighted and Entropy-weighted inference procedures, a Scaled Cross-Entropy (SCE) objective that retains the standard training path, and a token-wise sampler for the modified training path. Experiments primarily cover OpenWebText and QM9, with additional transfer results for larger text models and image generation. The main empirical improvements come from uniform-source SCE and inference-only context-weighted sampling.
Critical strengths
- The improvements over the directly matched baseline are practically meaningful
On uniform-source OWT, SCE increases MAUVE from 0.628 to 0.777 while reducing generative perplexity from 82.17 to 30.20 (Tables 1–2). The direct DFM baseline uses the same architecture, data, and evaluation setup (Appendix B.1). The accompanying MAUVE improvement makes the result more compelling than a reduction in external-language-model perplexity alone, and the comparisons with MDLM, SEDD, and BD3LM provide useful context. These results support a potentially valuable practical contribution.
- The method combines a simple structural construction with useful transfer evidence
The normalization that preserves transition-count distributions for fixed endpoints, together with exact token-wise path sampling for time-independent weights, is a useful design. The inference-only NW modification also requires no additional training. Transferring the small-model hyperparameters to a public 1.3B model decreases generative perplexity from 49.75 to 43.19 and increases MAUVE from 0.1923 to 0.3175, while token entropy also increases (Table 6). This provides initial evidence of practical value beyond a single small model.
Critical weaknesses
- The training contribution is not sufficiently distinguished from closely related planner-aware approaches
PAPL (Planner Aware Path Learning in Diffusion Language Models Training, ICLR 2026) already uses confidence-weighted cross-entropy while retaining standard random masking (Section 3.4, Eq. 7 of that paper). The submission does not discuss this work, and Table 2 compares only CE, NELBO, and Bregman losses. The neighborhood proxy, count-preserving conditional path, and applicability to uniform-source dynamics may provide meaningful distinctions, but their independent value is not sufficiently established.
The locality correlations in Figure 2 and the final metric improvements do not isolate the contribution of locality itself. In particular, the headline uniform-source OWT result is obtained with the original path; switching to the modified path does not improve the corresponding results in Table 3. Consequently, the evidence does not yet establish how much of the main training improvement is specifically attributable to the proposed local-context design.
- The connection between the conditional-path theory and the practical uniform-source procedures needs a more precise definition
Theorem 1 concerns a fixed-endpoint process. Under a uniform source, neighborhood agreement and its normalization depend on the endpoint, which is replaced by newly sampled proposals at inference (Section 3.2). Marginalizing the resulting weighted rates generally involves joint endpoint dependencies.
The paper should clarify how independent token proposals approximate this process, as well as the exact SCE weighting, normalization set, and treatment of already agreeing coordinates. The principal SCE experiments also retain the original path and use ordinary Euler sampling (Appendix B.2), so the rate-divergence homogeneity identity in Appendix A.4 does not by itself establish consistency of the practical combination.
I am not claiming that Theorem 1 is incorrect; my concern is that the relationship between conditional guarantees and empirical design choices is insufficiently specified. The claim about preserving the stepwise update-count distribution at lines 265–266 also needs to be distinguished from the continuous-time guarantee.
Questions for authors
Q1 — Clarification and targeted validation.
What distinguishes SCE from PAPL? Could the authors provide a matched-backbone, matched-budget comparison in one existing masked-source setting where both directly apply, or other evidence that distinguishes the two weighting mechanisms? This does not require developing a new uniform-source extension of PAPL.
Q2 — Method clarification.
Please provide the complete SCE weighting and normalization formulas. For a uniform source, do training weights use ground-truth endpoint agreement, the corruption process’s reveal indicator, or model proposals, and how are already agreeing coordinates treated? Please also separate fixed-endpoint CTMC guarantees from properties of the implemented proposal-redrawing sampler, including the precise scope of the “stepwise distribution” claim at lines 265–266.
Q3 — Evaluation clarification and targeted validation.
Table 11 reports worse test-perplexity upper-bound values for uniform-source SCE on all three datasets despite improved MAUVE and generative perplexity. How are these bounds estimated, and what evidence distinguishes improved local learning from a changed quality–diversity operating point? A matched-diversity sampling comparison using existing checkpoints, or a control that disrupts locality while preserving the weight distribution, would be informative. I do not interpret the increased upper-bound values as proof that the true likelihood deteriorates.
Q4 — Comparison conditions.
What NFE, hyperparameter-selection protocol, and number of training runs underlie Tables 1–3 and Table 6? Does Figure 5 include a predictor–corrector baseline without NW under the same corrector configuration? What NFE does EB actually use at γ = 0.001 in Table 7? These details materially affect the assessment of the quality–compute trade-off.
Dear Reviewers and Area Chair,
As the discussion period draws to a close, we thank you for your time and feedback. Below, we summarize our responses to the main concerns. Numerical results and experimental details are provided in the individual replies.
We have incorporated these clarifications and additional results into the revised manuscript and would be happy to address any remaining questions.