acceptodds
Under review as a conference paper at ICLR 2027

FontBench: A Multidimensional Benchmark for Style-Consistent and Cross-Lingual Font Generation

Abstract

Font generation requires models to jointly preserve character identity, glyph structure, typographic style, and visual quality. However, existing evaluations are often limited to specific scripts, narrow font categories, or reconstruction-oriented metrics, making it difficult to assess the true capabilities of current generative models. We introduce FontBench, a comprehensive benchmark for systematically evaluating font generation across diverse styles, character complexities, and writing systems. FontBench contains 920 fonts organized into three font groups and eight fine-grained subcategories spanning structurally regular to highly stylized fonts. It supports two benchmark tasks, style-conditioned and cross-lingual font generation, together with a multidimensional evaluation protocol covering character readability, structural fidelity, style consistency, and visual quality. Extensive experiments on five specialized font generation models and seven general-purpose image-to-image models reveal complementary strengths and limitations: general models produce highly readable glyphs but often lack precise structural control, whereas specialized font models better preserve font-specific styles yet struggle with complex artistic fonts. Cross-lingual evaluation further reveals directional asymmetry, with English-to-Chinese limited mainly by structural adaptation and Chinese-to-English by style preservation. These results demonstrate the necessity of multidimensional evaluation and establish FontBench as a standardized benchmark for analyzing and advancing font generation systems.

Primary area:
datasets and benchmarks
Keywords:
Font Generation, Font Generation Benchmark, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sa6uOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces FontBench, a multidimensional evaluation benchmark for font generation tasks, designed to address the limitations of existing datasets regarding font types, character complexity, and cross-lingual evaluation. The benchmark comprises 920 fonts, spanning three major categories and eight subcategories, and defines two specific tasks: style-conditioned font generation and cross-lingual font generation. The authors establish an evaluation framework based on three dimensions—structural correctness of characters, style consistency, and visual quality—and conduct a systematic assessment of various specialized font generation models as well as general-purpose image generation models, analyzing the performance gaps and limitations of current approaches.

Critical strengths
  1. FontBench addresses the limitations of existing datasets for font generation, particularly regarding font diversity, complex artistic fonts, and cross-lingual evaluation. Provided that copyright issues concerning font data can be effectively resolved, this benchmark has the potential to become a valuable research resource in the field of font generation.
  2. The authors categorize fonts into three main classes and eight subcategories, evaluating them across three dimensions—character structure, stylistic consistency, and visual quality—which effectively cover the core requirements of font generation tasks. In addition, this paper also covers the task of cross-lingual font generation.
  3. The paper evaluates a variety of both specialized font-generation models and general-purpose image-generation models, analyzing performance variations across different font categories, levels of character complexity, and cross-lingual generation scenarios.
Critical weaknesses
  1. Currently, the primary contributions of FontBench are limited to providing training/testing data and a unified evaluation protocol. While the paper demonstrates performance disparities among various models, it lacks an in-depth analysis of the reasons behind model failures or the key challenges inherent in font generation; consequently, it offers little actionable guidance for the design of future models.
  2. The paper employs feature similarity—based on OpenCLIP ViT-B/32—to evaluate style consistency. However, a general-purpose CLIP feature space may not be well-suited for distinguishing fine-grained stylistic differences between fonts. As shown in Table 2, the style scores for different models are generally high with minimal variance, which contradicts the clear stylistic differences observed in the qualitative results. It is recommended that the authors consider approaches such as HWD (BMVC2023)—which utilizes large-scale font data to train a dedicated style-evaluation model—to enhance the discriminative power and reliability of the metrics. I believe this is crucial for strengthening the core contribution of this paper, as well as for the development of the community. If the author can address this point appropriately, I will consider raising my score.
  3. Although the authors have collected a substantial volume of font resources, the licensing terms and permitted scope of use—specifically regarding their application for model training and public distribution—remain unclear. This ambiguity could potentially affect the reproducibility and long-term utility of the benchmark.
  4. This paper primarily employs an evaluation setup involving unseen font styles, yet it fails to explicitly distinguish between the SFSC, SFUC (Seen Font Unseen Chars.), UFSC, and UFUC settings—standard classifications in the field of font generation. In particular, it remains unclear whether there is any overlap between the characters used for testing and those used for training. It is recommended that the authors clarify their character-level splitting strategy and provide additional evaluation results across various generalization settings, thereby enabling a more comprehensive analysis of the model’s ability to generalize to both unseen fonts and unseen characters.
Questions for authors

see weaknesses

6 comments

We thank the reviewer for the constructive feedback. We have expanded the analyses, incorporated HWD, and clarified dataset provenance and generalization settings, as detailed below.

A1. Failure Diagnosis and Research Implications

FontBench aims not only to provide standardized evaluation protocols but also to identify key challenges in font generation. Our category-wise results (Appendix E.1) and qualitative analyses (Appendix F) reveal three recurring failure modes.

First, transformable fonts expose difficulties in balancing structural correctness and stylistic deformation: some models produce malformed components, while others suppress intended transformations. Second, decorative, handwritten, and serif fonts reveal losses in ornamental details, stroke variations, and subtle typographic features. Third, some models generate recognizable but overly regularized glyphs, whereas others preserve stronger styles at the expense of structural fidelity.

To provide actionable insights, we have added Section 5.5, Failure Diagnosis and Research Implications, and Fig. 6, explicitly connecting these observations to potential research directions: topology-aware generation, stroke-aware and multi-scale style representations, and joint content–style modeling. These additions strengthen FontBench's diagnostic value beyond aggregate performance comparisons.

A2. Improving Style Evaluation with HWD

We agree that general-purpose CLIP embeddings may lack sensitivity to fine-grained font characteristics and may encode structural information alongside style.

Following the reviewer's suggestion, we have incorporated Handwriting Distance (HWD, BMVC 2023) as a complementary domain-specific metric, extending our overall, category-wise, and character-complexity evaluations.

The additional results reveal distinctions that are less apparent from CLIP-based scores. On the shared 45-character subset, IF-Font and wan2.6-image achieve nearly identical Style-Con scores (0.8940 vs. 0.8945), but substantially different HWD scores (0.8001 vs. 2.0900; lower is better). Similarly, MX-Font achieves an HWD of 0.7930, compared with 2.2800 for Doubao-Seedream-4.5.

We have updated the evaluation protocol and quantitative tables(Tbale2, Tbale 3, Tbale 4) to report HWD alongside CLIP-I and Style-Con. These complementary metrics capture different aspects of visual similarity, improving the discrimination of model differences.

A3. Dataset Licensing and Public Availability

We agree that transparent data provenance and licensing information are essential for reproducibility and responsible reuse.

FontBench is primarily curated from FTransGAN, LSG-FCST, and TE141K through quality screening, redundancy reduction, and font-style categorization, resulting in 920 fonts and 609,432 glyph images.

We have revised Appendix A.5, Font Licensing and Public Availability, to clarify the dataset sources and release plan. Upon acceptance, we plan to release our curated annotations, font categories, dataset splits, and evaluation protocols, together with source attribution and licensing information.

A4. Generalization Settings and Character-level Splits

We agree that distinguishing font-level and character-level generalization is essential for correctly interpreting benchmark results.

We have clarified in Section 3.2 and Appendix A that FontBench adopts the Unseen Font, Seen Character (UFSC) setting for specialized models trained on our dataset. Specifically, the 524 training fonts and 396 test fonts are disjoint, while all 225 test characters are included in the 993-character training vocabulary. Thus, our evaluation measures generalization to unseen font identities rather than unseen character identities.

We have additionally evaluated all five specialized models on the shared 45-character subset, while retaining their full 225-character evaluation. Character-complexity analyses further examine performance across different glyph structures.These experiments strengthen evaluation coverage within UFSC.

Dear Reviewers,

As the discussion deadline approaches, we would appreciate your feedback on our rebuttal and revised manuscript, including the additional experiments and clarifications addressing your comments.

If our responses and new evidence have resolved your concerns, we would be grateful if you could consider updating your assessment and score accordingly. If any concerns remain, please let us know so that we can address them while the discussion is still open.

Thank you for your time and consideration.

Best regards, The Authors

Thank you to the authors for their response and the additional experiments. My original suggestion was for the authors to develop a style evaluation metric tailored to the font domain—drawing inspiration from the approach used for HWD—rather than simply employing HWD to evaluate the results, which reflects a lack of independent insight from the authors themselves.

Furthermore, as a benchmark for fonts, the current selection appears to cover a limited number of characters, which constrains the statistical significance of the metrics. Could the authors explain why it is not feasible to scale up the number of characters?

Reviewer_BMrgOpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FontBench, a multidimensional benchmark for evaluating style-conditioned and cross-lingual font generation. The benchmark contains 920 fonts organized into three major categories and eight fine-grained subcategories, covering diverse font styles and character complexities. It establishes two evaluation tasks and integrates multiple existing metrics to assess structural fidelity, style consistency, and visual quality. Extensive experiments on five specialized font generation models and seven general-purpose image generation models reveal different strengths and limitations across font categories, character complexities, and cross-lingual generation settings.

Critical strengths
  1. The paper addresses a meaningful problem in font generation evaluation by considering structural fidelity, style consistency, and visual quality within a unified benchmark. The hierarchical font taxonomy provides a useful framework for analyzing model performance across different font styles and character complexities.
  2. The experimental evaluation covers a diverse set of specialized font generation models and general-purpose image generation models. The category-wise analyses, qualitative comparisons, and human evaluations provide useful insights into the capabilities and limitations of existing approaches.
Critical weaknesses
  1. The methodological novelty of FontBench appears relatively limited, as the benchmark primarily builds upon existing font resources, established font categories, and conventional evaluation metrics. Although the experiments demonstrate the usefulness of the proposed taxonomy and multidimensional evaluation protocol, their distinctive advantages over existing font generation benchmarks and evaluation schemes remain insufficiently established.
  2. The validity of the proposed multidimensional evaluation protocol requires further justification. In particular, the style consistency evaluation relies on pretrained CLIP features, which may capture character content and structural similarity in addition to font style. Although human evaluations are provided, the reported correlations are moderate, leaving room for further validation of the evaluation metrics.
  3. The experimental comparisons have certain limitations. Different model groups are evaluated using different target character sets and sample sizes, while the cross-lingual experiments cover a relatively small number of target words. Although the authors distinguish the evaluation tracks, these differences may limit the strength of the general conclusions regarding model capabilities and cross-lingual generation difficulty.
Questions for authors
  1. Although FontBench demonstrates the usefulness of fine-grained font categorization and multidimensional evaluation, its distinctive advantages over existing font generation benchmarks remain unclear. Could the authors provide a more direct comparison with existing evaluation protocols to demonstrate whether the proposed taxonomy and evaluation dimensions reveal model limitations that conventional evaluation methods cannot adequately capture?
  2. The proposed style consistency evaluation relies on CLIP-based similarity measures. However, CLIP representations may encode character content and structural information in addition to font style. How do the authors ensure that CLIP-I and Style-Con accurately reflect font style consistency rather than content similarity? Further controlled evaluations or comparisons with specialized font-style metrics would help strengthen the validity of the proposed evaluation protocol.
  3. Different model groups are evaluated using different target character sets and sample sizes. Although the paper distinguishes these evaluation tracks, could the authors further clarify how these differences affect the interpretation of the comparative results? Additional experiments on matched evaluation subsets would provide stronger evidence for the conclusions regarding the relative capabilities of specialized and general-purpose models.
  4. The paper identifies directional asymmetry in cross-lingual font generation, suggesting that English-to-Chinese generation is mainly limited by structural adaptation, whereas Chinese-to-English generation is primarily constrained by style preservation. However, the two directions involve different model groups and relatively limited evaluation samples. Could the authors further validate this conclusion using more extensive cross-lingual evaluations under controlled settings?
  5. The benchmark includes comparisons with several existing font generation methods, but some recent specialized and cross-lingual font generation approaches are not evaluated. Could the authors clarify the baseline selection criteria and discuss whether including more recent or task-specific methods would provide a more representative assessment of current font generation capabilities?
3 comments

We thank the reviewer for the constructive questions. We have expanded the experiments and revised the manuscript to clarify the benchmark's diagnostic value, evaluation validity, and comparison scope.

A1. Advantages over Conventional Evaluation

FontBench complements conventional aggregate evaluation by revealing metric-level trade-offs and category-specific failures.

First, our multidimensional evaluation captures differences that reconstruction-oriented metrics alone cannot fully characterize. For example, on the shared 45-character subset, IF-Font outperforms FontDiffuser in SSIM (0.4188 vs. 0.3450) and LPIPS (0.1711 vs. 0.1811), whereas FontDiffuser achieves higher Style-Con (0.9067 vs. 0.8940). This demonstrates that reconstruction fidelity and reference-style alignment provide complementary assessments of model performance.

Second, our fine-grained taxonomy enables category-specific diagnosis beyond overall scores. The category-wise results (Appendix E.1) and qualitative analyses (Appendix F) reveal distinct challenges, including structural corruption in transformable fonts, ornament loss in decorative fonts, and weakened stroke characteristics in handwriting styles.

Furthermore, category-wise results (Appendix E.1) and qualitative examples (Appendix F) expose deformation-related structural errors, lost decorative details, and weakened handwriting strokes that overall averages can obscure. We added Section 5.5, Failure Diagnosis and Research Implications, and Fig. 6 to connect these observations to topology-aware generation, stroke-aware/multi-scale representations, and joint content–style modeling.

A2. Content–Style Entanglement in CLIP Metrics

We agree that CLIP embeddings are not pure style representations. CLIP-I compares each generated glyph with its same-character ground truth and measures overall visual alignment; Style-Con compares it with a prototype averaging CLIP embeddings of three different-character real references in the target font, reducing direct content matching without eliminating structural influences.

Following the reviewer's suggestion, we added HWD (BMVC 2023) as a complementary domain-specific metric and updated the quantitative tables and evaluation protocol. On the matched 45-character subset, IF-Font and wan2.6-image have nearly identical Style-Con (0.8940 vs. 0.8945) but markedly different HWD (0.8001 vs. 2.0900; lower is better). This illustrates complementary sensitivity.

We have updated the evaluation protocol and quantitative tables to report HWD alongside CLIP-I and Style-Con.

A3. Matched Subsets and Interpretation

To address this concern, we have conducted additional experiments evaluating all five specialized font generation models on the same 45-character subset used by the four general-purpose Chinese models. This provides a matched evaluation setting for comparing the two model groups while controlling for differences in target character selection.

The new results support our main observations under shared test content. Specialized models generally achieve stronger style-related performance. In contrast, general-purpose models demonstrate competitive readability. .

We now separate the shared 45-character Chinese comparison (Table 2), full 225-character specialized evaluation (Table 9), and six-character English evaluation (Table 3), with best/second-best highlights confined to each comparable setting. We also report 225-versus-45 score differences to expose sensitivity to character-set composition.

A4. Controlled Bidirectional Transfer

To remove the model-group confound in the original comparison, we evaluated Qwen-Image-Edit-2511 and FireRed-Image-Edit-1.1 in both EN→ZH and ZH→EN directions. The additional results reveal consistent direction-dependent performance patterns. These findings suggest that generating structurally complex Chinese characters poses greater readability challenges, while transferring Chinese font styles to English may involve greater difficulty in maintaining visual style alignment.

We have updated Table 4 and the corresponding qualitative comparisons(Fig.5).

A5. Baseline Selection and Coverage

We select baselines for architectural diversity, task relevance, and reproducibility. Specifically, the five specialized baseline models (MX-Font, FUNIT, LF-Font, FontDiffuser, and IF-Font) represent distinct generative architectures and style modeling strategies, encompassing various methods such as GANs, local style modeling, diffusion-based generation, and autoregressive generation.; general-purpose image-editing systems provide complementary coverage.

To expand recent-model representation, we added GPT-Image-2.6 to the English style-conditioned generation evaluation(Table 3). We also extended Qwen and FireRed to bidirectional cross-lingual evaluation.

Dear Reviewers,

As the discussion deadline approaches, we would appreciate your feedback on our rebuttal and revised manuscript, including the additional experiments and clarifications addressing your comments.

If our responses and new evidence have resolved your concerns, we would be grateful if you could consider updating your assessment and score accordingly. If any concerns remain, please let us know so that we can address them while the discussion is still open.

Thank you for your time and consideration.

Best regards, The Authors

Thank you for your response. While I consider this a solid piece of work, the novelty does not particularly impress me. Therefore, I have decided to maintain my current score while increasing my confidence rating.

Reviewer_AuLcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FontBench, which evaluates structural fidelity, style consistency, and visual quality through 920 fonts, a hierarchical style taxonomy, and same-language and cross-lingual generation tasks. A systematic evaluation is practically valuable because font generation requires both character correctness and style preservation. However, the paper provides insufficient support for the reliability of its data splits, the additional diagnostic value of its taxonomy, and its main experimental conclusions.

Critical strengths

The paper addresses a meaningful problem. Generating recognizable characters does not necessarily imply faithfully reproducing the reference font. Including handwritten and complex artistic fonts, distinguishing character complexity, and comparing specialized and general-purpose models help move font-generation evaluation beyond a single reconstruction score.

Critical weaknesses
  1. The paper establishes that font files and identities are separated between splits, but does not adequately describe font sources, relationships among fonts, or near-duplicate screening. Different files or names may correspond to closely related variants, so the split does not fully establish style independence.

  2. The paper claims that its three groups and eight subcategories correspond to different capability requirements and enable systematic diagnosis of model limitations. However, the experiments do not sufficiently demonstrate which problems this taxonomy reveals that existing evaluations would miss, or how these findings could guide subsequent research.

  3. Section 4.1 describes Style-Con as measuring consistency among generated glyphs, whereas Appendix B.2 actually computes similarity between generated glyphs and a prototype derived from real reference glyphs. It therefore remains unclear whether the evaluation dimensions can reliably distinguish key failure modes, such as correct character identity but incorrect style.

  4. The specialized Chinese models, general-purpose Chinese models, and English models are evaluated on 225, 45, and six test characters, respectively. Although the authors acknowledge that different tracks are not directly comparable, Table 2 still highlights the best results across tracks. Without comparisons on a shared subset, the influence of differences in test content cannot be ruled out.

  5. The paper also claims directional asymmetry in cross-lingual generation. However, Table 4 evaluates English-to-Chinese generation using Qwen and FireRed, and Chinese-to-English generation using GPT, Grok, and Nano Banana. Because both the models and transfer direction change, these results cannot establish that the observed differences are attributable to transfer direction.

Questions for authors

Please see the above Weaknesses.

6 comments

We thank the reviewer for the constructive feedback. We have revised the manuscript and conducted additional experiments to address the five concerns, as detailed below.

A1. Font Provenance and Style Independence

FontBench integrates resources from FTransGAN, LSG-FCST, and TE141K, followed by quality screening and font-style annotation. To reduce redundancy, we retain only one representative weight variant (e.g., Regular, Medium, or Bold) per typeface.Therefore, different weight variants of the same typeface are not independently included in FontBench and cannot be assigned to different training and test splits. Three independent annotators categorize fonts based on visual glyph characteristics rather than filenames or metadata, achieving a Fleiss' κ\kappa of 0.78.

We have clarified these procedures in Appendix A.1.

A2. Diagnostic Value of the Taxonomy

Our category-wise results (Appendix E.1) and qualitative analyses (Appendix F) reveal limitations obscured by aggregate scores. Transformable fonts expose structural corruption versus suppressed geometric deformation, while decorative and handwritten fonts reveal losses in ornamental and stroke-level details. Moreover, some models generate recognizable but overly regularized glyphs.

To make these findings actionable, we have added Section 5.5, Failure Diagnosis and Research Implications, and Fig. 6, explicitly connecting observed failures to potential research directions: topology-aware generation, stroke-aware and multi-scale style modeling, and joint content–style modeling. These analyses demonstrate the taxonomy's diagnostic value beyond overall performance rankings.

A3. Style-Con Definition and Metric Validity

We acknowledge the inconsistency and have corrected Section 4.1 to match Appendix B.2. Specifically, Style-Con measures similarity between each generated glyph and a prototype formed by averaging the CLIP embeddings of three real reference glyphs, rather than consistency among generated glyphs.

To strengthen style evaluation, we have incorporated HWD (BMVC 2023) as a complementary domain-specific metric. On the shared 45-character subset, IF-Font and wan2.6-image achieve nearly identical Style-Con scores (0.8940 vs. 0.8945), but substantially different HWD scores (0.8001 vs. 2.0900; lower is better). This reveals differences less apparent in CLIP-based scores.

Furthermore, our human evaluation (Section 5.4) reports a Spearman correlation of 0.67 between the overall style-consistency dimension and human judgments.

A4. Matched Evaluation and Fair Comparison

We agree that comparing models across different character sets may introduce content-related biases. To address this, we additionally evaluated all five specialized models on the same 45-character subset used by the four general-purpose Chinese models (revised Table 2).

Under matched target content, wan2.6-image achieves the highest OCR-Acc (0.9976), MX-Font achieves the highest CLIP-I (0.9517) and lowest HWD (0.7930), while IF-Font obtains the best SSIM (0.4188) and LPIPS (0.1711). These results confirm distinct model strengths under shared test content.

We have also reorganized the quantitative results into separate evaluation settings: all nine Chinese models on 45 characters(Table 2), specialized models on the full 225-character set(Table 9), and English models on six characters (Table 3). Best and second-best scores are now highlighted only within comparable settings, eliminating misleading cross-track rankings.

A5. Bidirectional Cross-lingual Evaluation

We agree that the original cross-lingual comparison confounded transfer direction with model identity. We have therefore evaluated both Qwen-Image-Edit-2511 and FireRed-Image-Edit-1.1 in both EN→ZH and ZH→EN directions, enabling within-model comparisons.

For Qwen and FireRed, OCR-Acc increases from 0.5904 and 0.6452 (EN→ZH) to 0.7214 and 0.8500 (ZH→EN), respectively. Conversely, Style-Con decreases from 0.7761 to 0.7071 (Qwen) and from 0.7581 to 0.6808 (FireRed), with CLIP-I exhibiting the same directional trend.These findings suggest a direction-dependent trade-off between character readability and visual style alignment.

We have updated Table 5 and the corresponding qualitative comparisons(Fig.6) to include these results.

Dear Reviewers,

As the discussion deadline approaches, we would appreciate your feedback on our rebuttal and revised manuscript, including the additional experiments and clarifications addressing your comments.

If our responses and new evidence have resolved your concerns, we would be grateful if you could consider updating your assessment and score accordingly. If any concerns remain, please let us know so that we can address them while the discussion is still open.

Thank you for your time and consideration.

Best regards, The Authors

Dear Reviewers,

As the discussion deadline approaches, we hope our clarifications and additional experiments in the revised draft have sufficiently addressed your concerns.

If you are satisfied, we kindly ask you to consider updating your score.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.