acceptodds
Withdrawn from ICLR 2027

Geometric Consistency of Concrete Scores in Discrete Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In score-based formulations of discrete diffusion models, the reverse process is parameterized by probability ratios between neighboring states, commonly referred to as the concrete score. We characterize this score through discrete differential geometry on the graph induced by the transition structure. The logarithmic concrete score of any positive probability distribution forms an exact -cochain: the discrete differential of log probability on vertices. Therefore, it must be antisymmetric under edge reversal and have zero circulation along every cycle. Existing discrete diffusion models do not explicitly enforce these conditions, so their learned ratios need not correspond to any single distribution. To investigate this gap, we use the weighted Hodge decomposition to separate log concrete score error into three mutually orthogonal components: antisymmetry violations, cycle inconsistency, and data-score mismatch. We investigate their contributions to squared field error and how interventions on them relate to generation. Closed-form geometric projections and a trained data-informed corrector provide complementary interventions on the same frozen backbone. Projection targets geometric inconsistency; the corrector uses data-space feedback and can change all three components. We study each operator and their composition, including projection amortization. Experiments on uniform-noise language models examine geometric residuals, data-space likelihood, diversity, and inference cost. This investigation connects component-wise geometry and generation while distinguishing field realizability from alignment with the target distribution.

Primary area:
generative models
Keywords:
Discrete Diffusion Models, Concrete Score, Hodge Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rtznOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the geometric consistency of concrete scores in discrete diffusion models. It characterizes true log probability ratios as exact 1-cochains and uses a weighted Hodge decomposition to separate estimation error into antisymmetry violations, cycle inconsistency, and data-score mismatch. The authors derive component-wise error identities and investigate local geometric projections, a learned corrector trained using terminal teacher likelihood on frozen backbones, and surrogate models that amortize projection. Validation includes DUO, UDLM, and GIDD language models with ancestral and Ψ samplers, measuring geometric residuals, generative perplexity, unigram entropy, and runtime. Finite-state toy studies and Potts-model experiments further examine target alignment and training effects. The results show model-dependent benefits and illustrate that reducing geometric inconsistency does not necessarily improve the terminal distribution.

Critical strengths
  1. A useful framework for diagnosing score inconsistency. The paper separates antisymmetry violations and cycle inconsistency from data-score mismatch through a mathematically sound orthogonal decomposition. The geometric residuals can be measured without access to the true noisy density. This provides useful diagnostics and clarifies which errors projection can remove, giving the analysis value beyond the particular correction method.

  2. Informative controls on the relationship between consistency and generation. The exact-target experiments demonstrate that geometric projection can worsen terminal fit and that optimizing teacher likelihood can reduce entropy while increasing true distributional error. Together with experiments across several frozen language models and samplers, these controls substantiate the distinction between field consistency and generation quality and help delimit the benefits of the proposed interventions.

Critical weaknesses
  1. The connection between the global theory and the deployed intervention remains incomplete. The analysis concerns fields on the full sequence-state graph, whereas practical projection operates on candidate slices at one token position. These slices miss cycles involving multiple positions, and the subsequent clean-coordinate correction does not realize the ideal noisy-field projection. Although acknowledged, these limitations leave the global error identities insufficient to explain the deployed method’s behavior. Quantifying the omitted inconsistencies would substantially strengthen the contribution.

  2. The evidence for meaningful additional benefits from geometric projection is limited. In Table 1, adding projection to DUO’s Ψ+C improves PPL from 59.799 to 59.684, approximately 0.19%; the significance marker tests against Baseline rather than Ψ+C. Benefits vary across backbones, and none of Table 2’s projection contrasts passes the stated multiple-testing corrections. Without direct paired evidence for the incremental benefit and comparisons at matched inference cost, the practical value of projection beyond the learned corrector and sampler remains insufficiently established.

Questions for authors
  1. Clarification of existing results: What are the paired effect estimates and confidence intervals for C+P versus C and Ψ+C+P versus Ψ+C, accounting for paired seed bundles and the learned heads? The significance markers against Baseline do not establish this incremental benefit. Are results at matched inference cost available that would support the practical value of adding projection?

  2. Suggested additional validation: Could you measure circulation around cycles involving two token positions, at common fixed states, before and after the deployed correction? Single-position slice residuals cannot detect these cycles, and the clean-coordinate correction does not inherit the ideal projection guarantees. This targeted test would materially affect my assessment of how well the practical intervention addresses the global inconsistency motivating the paper.

Reviewer_KTsROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the geometric consistency of concrete scores in discrete diffusion models. The authors observe that the logarithm of the true concrete score forms an exact discrete gradient field, satisfying antisymmetry and zero circulation along graph cycles. Using weighted Hodge decomposition, they separate the learned score error into three orthogonal components: antisymmetry violations, cycle inconsistency, and data-score mismatch. Based on this decomposition, they investigate geometric projection operators and a data-informed corrector trained using terminal generation feedback.

Critical strengths

The paper provides a useful interpretation of concrete scores through discrete differential geometry. Separating geometric inconsistency from data-score mismatch offers a potentially informative framework for diagnosing errors in discrete diffusion models. Moreover, this work recognizes that an exact score field need not correspond to the desired target distribution. The decomposition helps clarify the respective roles of geometric projection and data-informed correction.

Critical weaknesses
  1. The main results Proposition 2 and Theorem 3 primarily follow from standard Hodge decomposition and the orthogonal projection identities. While Theorem 3 guarantees that removing geometric residuals reduces the weighted score-field error, the paper does not establish how these residuals affect the reverse-process or the sampling error. Consequently, the theoretical results provide limited insight into when geometric consistency actually improves generation.

  2. As shown in Table 1, the additional gains from geometric projection are modest and inconsistent across models. In several configurations, particularly for UDLM-HF and GIDD, the data-informed corrector alone outperforms its combination with projection, leaving the incremental practical benefit of geometric correction unclear.

  3. The theoretical guarantees concern exact geometric projection, whereas the practical method operates on restricted candidate sets and modifies denoiser predictions. Consequently, the theoretical guarantees do not directly apply to the implemented algorithm, and the extent to which it achieves the intended geometric correction remains unclear.

Questions for authors
  1. Beyond the standard Hodge decomposition, can the authors clarify what new theoretical insights the proposed framework provides for discrete diffusion models? In particular, can geometric inconsistency be quantitatively related to sampling accuracy?

  2. Table 1 shows that geometric projection does not consistently improve performance when combined with the data-informed corrector. Can the authors clarify under what conditions projection provides additional benefits and whether these improvements are statistically significant relative to correction alone?

Reviewer_VM67OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a decomposition of the error of discrete generative models in three quantities, two coming from geometric inconsistencies (antisymmetry violations, cycle inconsistency) and one coming from not exactly fitting the data. The authors then provide a corrector for each of these errors, and validate them on discrete diffusion based language models.

Critical strengths
  • The authors propose a sound decomposition of the error in discrete diffusion models, which is novel as far as I know. While the decomposition is not new and uses existing mathematical tools, it seems to be a novel analysis in the context of discrete diffusion models.

  • The authors propose correctors for geometric inconsistencies in discrete diffusion models, and for the data error term.

  • A validation of the algorithm on discrete diffusion language models

Critical weaknesses
  • While the motivation of the geometric corrector is clear to me, the motivation of the data driven corrector term is less clear. The main question that I have is: why do we need this data correction term at all? Why is it better than a higher capacity training of the model with denoising loss.

  • Is theory working for both uniform noise and mask noise settings? It seems theory is derived for uniform noise, which is not the setting of real-world sota models. Moreover, experiments are mostly made on uniform-noise discrete diffusion language models (except for GIDD which is hybrid). As far as I know, sota discrete diffusion language models are rather using masked-based discrete diffusion (absorbing states). Can you clarify this point, both on the theoretical part and on the experimental part?

  • The analytical geometric projectors are very expensive at inference, so the authors propose surrogate of geometric projectors. However this is only presented in the Appendix. A presentation of the surrogate projectors should be done in the main paper since it is a critical component of the final model used in practice. What is the additional training cost of the surrogates?

Experimental validation of the proposed method is not clear to me; and doesn't show a very clear and neat advantage of the proposed method. To me, a few things are lacking to get a full picture of the proposed method, and to validate experimentally a superiority:

  • Could you provide iso-runtime experiments?

  • How does performance evolve with number of denoising steps for both methods?

  • From table 2, it seems that most of the improvement comes from the data-correction term.

  • How can you explain that geometric correctors have limited impact on performance?

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.