acceptodds
Under review as a conference paper at ICLR 2027

DiffMath: Symbol- and Graph-Aware Latent Diffusion Transformer for Handwritten Mathematical Expression Generation

Abstract

Handwritten Mathematical Expression Generation (HMEG) remains challenging because it requires accurate modeling of complex hierarchical structures and two-dimensional layouts. Existing methods typically rely on explicit spatial supervision to model expression layouts, resulting in high annotation costs and limited scalability. To address this, we propose , a structure-aware latent diffusion framework that leverages structural priors derived from LaTeX, thereby eliminating the need for positional supervision. First, we design a ational bstract yntax ree ( ), a generation-oriented structural representation that distills MathML trees into compact triplet sequences encoding symbol identities, spatial relations, and nesting depths. Building upon RelAST, we introduce , which learns a structure-preserving latent space through symbol-aware and relation-aware perceptual regularization, jointly encoding symbol semantics and structural topology. Finally, performs conditional denoising in the learned latent space conditioned on RelAST, further guided by a global symbol-count prior via Adaptive Layer Normalization (AdaLN) to improve structural coherence. Experiments demonstrate that DiffMath generates structurally consistent handwritten mathematical expressions, consistently outperforms existing HMEG methods, and effectively improves downstream OCR performance through synthetic data augmentation. The code will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Handwritten Mathematical Expression Generation, Structural Modeling, Latent Diffusion Transformer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gYTrOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses handwritten mathematical expression generation (HMEG), with the goal of synthesizing an online pen trajectory from a target LaTeX expression. The central contribution is RelAST, a compact structural representation obtained by converting LaTeX to Presentation MathML and serializing the resulting structure as symbol-relation-depth triplets. RelAST is intended to expose two-dimensional layout and hierarchy without requiring symbol bounding boxes, point-level alignment, or other explicit positional annotations. Based on RelAST, the authors propose DiffMath, which contains two main components. MathVAE encodes handwriting trajectories into a latent sequence and reconstructs them using a Gaussian-mixture trajectory decoder; an auxiliary CTC-based recognition objective encourages the latent space to preserve symbol and relation information. MathDiT then performs conditional diffusion in this latent space using RelAST tokens, depth embeddings, and a global symbol-count prior. On MathWriting, DiffMath reports 70.70% expression recognition accuracy and an FID of 5.43, outperforming the baselines included in the main table. Component ablations support the usefulness of symbol, relation, depth, and count conditioning. The generated samples also improve two handwriting-recognition models when used for data augmentation on MathWriting and CROHME.

Critical strengths
  1. A useful structural representation that avoids fine-grained positional supervision. RelAST makes symbol identity, spatial relation, and hierarchy explicit while remaining derivable from LaTeX alone. This directly addresses the central difficulty of HMEG and could be useful beyond the exact architecture proposed here.
  2. Consistent validation across main results, ablations, and a downstream task. DiffMath achieves strong reported recognition, perceptual, and user-preference results; the ablations show consistent gains from the proposed structural components; and generated data improve two OCR systems. This triangulation makes the empirical contribution more persuasive than a single benchmark result and suggests that the outputs preserve task-relevant mathematical structure.
Critical weaknesses
  1. The state-of-the-art claim is not established against the closest prior work or under a clearly controlled comparison.The paper cites the CVPR 2024 symbol-graph HMEG method and the ICDAR 2025 SFRD diffusion method, but neither is included in the main results. Meanwhile, the reported baselines mix online trajectory generation with raster-image generation and use different training regimes, whereas DiffMath is trained for roughly 220k steps on eight H800 GPUs. This is consequential because the claimed performance advantage is central to the paper, yet it is difficult to separate architectural benefit from differences in modality, data exposure, compute, and tuning.
  2. Several central technical and evaluation details are too ambiguous for reliable reproduction or interpretation. The separate symbol and relation losses in the ablation are not reconciled with the single interleaved CTC objective in the method; the alignment between variable-length RelAST and latent trajectory sequences is unspecified; and the VAE posterior and KL setup are incomplete. The synthetic-data source, deduplication protocol, and statistical variation are also missing. These omissions affect both the validity of the ablation conclusions and confidence in the downstream augmentation results.
Questions for authors
  1. Closest comparisons and fairness: Can the authors provide results for the symbol-graph HMEG method of Chen et al. (CVPR 2024) and SFRD (ICDAR 2025), or explain precisely why direct comparison is infeasible? For all main baselines, what are the effective numbers of training examples, optimization steps, parameter counts, compute budgets, and tuning procedures? How are raster-image outputs made comparable to online-trajectory outputs?
  2. Method definition: How are the separate LsymL_{sym} and LrelL_{rel} settings in Table 3 implemented when Equation 4 defines a single CTC loss on an interleaved symbol-relation target? How are RelAST features aligned with the latent trajectory sequence before channel-wise concatenation? Please also specify the VAE posterior heads, reparameterization operation, KL expression, and all associated loss weights.
  3. Augmentation validity and statistical reliability: What is the source of the LaTeX expressions used to generate the additional MathWriting and CROHME samples, and how are exact or structurally equivalent overlaps with validation and test expressions excluded? Are the number of added samples and optimization updates matched against conventional augmentation? Please provide variability across training seeds or other uncertainty estimates for the principal results.
Reviewer_ebQVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Considering existing methods relying on explicit spatial supervision to model expression layouts and resulting in high annotation costs and limited scalability, the paper investigates handwritten mathematical expression generation from LaTeX without additional symbol-level positional annotations. The paper proposes DiffMath, a latent diffusion framework with three main components. (1) RelAST converts MathML trees into symbol–relation–depth triplets to provide explicit structural conditions. (2) MathVAE learns a trajectory representation using reconstruction and auxiliary symbol-and-relation recognition objectives. (3) MathDiT generates trajectories in this latent space, conditioned on RelAST and a global symbol-count signal. Experiments on MathWriting report generation improvements in recognition-based generation metrics, FID, and human preference. The paper also demonstrates the effectiveness of synthetic-data augmentation for Handwritten Mathematical Expression Recognition (HMER) on MathWriting and CROHME.

Critical strengths
  • The proposed approach is well motivated. RelAST, MathVAE and MathDiT form a reasonable framework that learns from paired LaTeX and trajectory data without requiring additional layout annotations or symbol-level positional supervision.
  • The experiments provide meaningful evidence of effectiveness. The results in Table 3 support the importance of explicit structure in DiffMath. Table 4 further reports consistent recognition improvements across two models (Qwen3-VL 8B-Instruct and Uni-MuMER) and two benchmark settings (MathWriting and CROHME), demonstrating the practical value of augmenting training data for HMER.
Critical weaknesses
  • It seems that the scope of RelAST’s syntax coverage needs clarification. Table 1 claims full syntax coverage, but Algorithm 1 does not explain how some MathML structures retain their spatial relations. It explicitly handles superscripts, subscripts, fractions, and root, while unlisted nodes are processed by connecting their children with “right”. However, the paper does not clearly define the range of LaTeX or MathML structures covered by this procedure. The paper should define the scope of full syntax coverage.
Questions for authors
  • What does “Full” syntax coverage in Table 1 refer to? Does it mean the normalized syntax present in MathWriting, or a broader set of LaTeX structures? Could you clarify the supported scope, any preprocessing or normalization used before Algorithm 1, and whether any expressions were excluded because they could not be converted? These details would help establish the applicability of RelAST.
  • Could the authors clarify the implementation details connecting RelAST, MathVAE, and MathDiT? Section 3.5 states that the noisy latent (xtx_t) is concatenated with the RelAST condition (ZZ) along the channel dimension. How are their sequence lengths matched, including padding or other processing, and how is the length of the initial Gaussian noise (xTx_T) determined during inference? Please also specify the preprocessing applied to the raw handwriting trajectories. What reconstruction fidelity does MathVAE achieve before training MathDiT?
Reviewer_fTugOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose DiffMath, a structure-aware latent diffusion framework that takes structural prior information from LaTeX into account without requiring layout annotations or positional guidance signals for symbolic units. DiffMath is capable of generating handwritten mathematical expressions with high structural consistency and visual fidelity.

Critical strengths
  • By using data in the RelAST format—converted from the MathML AST format—and training the VAE to reconstruct this information, the method can incorporate structural and relational information.
  • Compared to recent text-to-image generation methods, it demonstrated the best performance in handwritten equation generation in both automated evaluation and the user study.
  • They also demonstrated that performance improves in the handwritten equation recognition task by using the handwritten equation generation from the proposed method for data synthesis.
Critical weaknesses
  • I could not understand the method of the user study in Table 2 without reading the Appendix, so I think it would be better to include at least a brief description in the main text.
  • In the user study, it states that one best system is selected from multiple systems, but how would you handle a situation where there are multiple systems with equivalent performance? Since there are multiple handwriting styles, I believe there could be multiple correct answers as long as the accuracy is correct.
  • In Table 2, they compare their method with recent text-to-image methods; I think it would be clearer if they also compared it with recent methods specialized in equation generation. For example, it is better to compare their method with SFRD (ICDAR 2025), which is also mentioned in the related work section.
Questions for authors
  • In the data augmentation experiments for handwritten mathematical expression recognition, they mentioned that the performance of their method was better than that of existing data augmentation techniques. However, it appears that recent handwritten mathematical expression generation methods were not used in the existing augmentation techniques. For example, if the authors were to use recent image generation methods such as DiffInk or Qwen-Image—which were compared in Table 2—or the mathematical expression generation method SFRD as data augmentation techniques, would the proposed method still remain effective?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.