- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces FontBench, a multidimensional evaluation benchmark for font generation tasks, designed to address the limitations of existing datasets regarding font types, character complexity, and cross-lingual evaluation. The benchmark comprises 920 fonts, spanning three major categories and eight subcategories, and defines two specific tasks: style-conditioned font generation and cross-lingual font generation. The authors establish an evaluation framework based on three dimensions—structural correctness of characters, style consistency, and visual quality—and conduct a systematic assessment of various specialized font generation models as well as general-purpose image generation models, analyzing the performance gaps and limitations of current approaches.
Critical strengths
- FontBench addresses the limitations of existing datasets for font generation, particularly regarding font diversity, complex artistic fonts, and cross-lingual evaluation. Provided that copyright issues concerning font data can be effectively resolved, this benchmark has the potential to become a valuable research resource in the field of font generation.
- The authors categorize fonts into three main classes and eight subcategories, evaluating them across three dimensions—character structure, stylistic consistency, and visual quality—which effectively cover the core requirements of font generation tasks. In addition, this paper also covers the task of cross-lingual font generation.
- The paper evaluates a variety of both specialized font-generation models and general-purpose image-generation models, analyzing performance variations across different font categories, levels of character complexity, and cross-lingual generation scenarios.
Critical weaknesses
- Currently, the primary contributions of FontBench are limited to providing training/testing data and a unified evaluation protocol. While the paper demonstrates performance disparities among various models, it lacks an in-depth analysis of the reasons behind model failures or the key challenges inherent in font generation; consequently, it offers little actionable guidance for the design of future models.
- The paper employs feature similarity—based on OpenCLIP ViT-B/32—to evaluate style consistency. However, a general-purpose CLIP feature space may not be well-suited for distinguishing fine-grained stylistic differences between fonts. As shown in Table 2, the style scores for different models are generally high with minimal variance, which contradicts the clear stylistic differences observed in the qualitative results. It is recommended that the authors consider approaches such as HWD (BMVC2023)—which utilizes large-scale font data to train a dedicated style-evaluation model—to enhance the discriminative power and reliability of the metrics. I believe this is crucial for strengthening the core contribution of this paper, as well as for the development of the community. If the author can address this point appropriately, I will consider raising my score.
- Although the authors have collected a substantial volume of font resources, the licensing terms and permitted scope of use—specifically regarding their application for model training and public distribution—remain unclear. This ambiguity could potentially affect the reproducibility and long-term utility of the benchmark.
- This paper primarily employs an evaluation setup involving unseen font styles, yet it fails to explicitly distinguish between the SFSC, SFUC (Seen Font Unseen Chars.), UFSC, and UFUC settings—standard classifications in the field of font generation. In particular, it remains unclear whether there is any overlap between the characters used for testing and those used for training. It is recommended that the authors clarify their character-level splitting strategy and provide additional evaluation results across various generalization settings, thereby enabling a more comprehensive analysis of the model’s ability to generalize to both unseen fonts and unseen characters.
Questions for authors
see weaknesses
6 comments
Dear Reviewers,
As the discussion deadline approaches, we would appreciate your feedback on our rebuttal and revised manuscript, including the additional experiments and clarifications addressing your comments.
If our responses and new evidence have resolved your concerns, we would be grateful if you could consider updating your assessment and score accordingly. If any concerns remain, please let us know so that we can address them while the discussion is still open.
Thank you for your time and consideration.
Best regards, The Authors
Thank you to the authors for their response and the additional experiments. My original suggestion was for the authors to develop a style evaluation metric tailored to the font domain—drawing inspiration from the approach used for HWD—rather than simply employing HWD to evaluate the results, which reflects a lack of independent insight from the authors themselves.
Furthermore, as a benchmark for fonts, the current selection appears to cover a limited number of characters, which constrains the statistical significance of the metrics. Could the authors explain why it is not feasible to scale up the number of characters?
We thank the reviewer for the constructive feedback. We have expanded the analyses, incorporated HWD, and clarified dataset provenance and generalization settings, as detailed below.
A1. Failure Diagnosis and Research Implications
FontBench aims not only to provide standardized evaluation protocols but also to identify key challenges in font generation. Our category-wise results (Appendix E.1) and qualitative analyses (Appendix F) reveal three recurring failure modes.
First, transformable fonts expose difficulties in balancing structural correctness and stylistic deformation: some models produce malformed components, while others suppress intended transformations. Second, decorative, handwritten, and serif fonts reveal losses in ornamental details, stroke variations, and subtle typographic features. Third, some models generate recognizable but overly regularized glyphs, whereas others preserve stronger styles at the expense of structural fidelity.
To provide actionable insights, we have added Section 5.5, Failure Diagnosis and Research Implications, and Fig. 6, explicitly connecting these observations to potential research directions: topology-aware generation, stroke-aware and multi-scale style representations, and joint content–style modeling. These additions strengthen FontBench's diagnostic value beyond aggregate performance comparisons.
A2. Improving Style Evaluation with HWD
We agree that general-purpose CLIP embeddings may lack sensitivity to fine-grained font characteristics and may encode structural information alongside style.
Following the reviewer's suggestion, we have incorporated Handwriting Distance (HWD, BMVC 2023) as a complementary domain-specific metric, extending our overall, category-wise, and character-complexity evaluations.
The additional results reveal distinctions that are less apparent from CLIP-based scores. On the shared 45-character subset, IF-Font and wan2.6-image achieve nearly identical Style-Con scores (0.8940 vs. 0.8945), but substantially different HWD scores (0.8001 vs. 2.0900; lower is better). Similarly, MX-Font achieves an HWD of 0.7930, compared with 2.2800 for Doubao-Seedream-4.5.
We have updated the evaluation protocol and quantitative tables(Tbale2, Tbale 3, Tbale 4) to report HWD alongside CLIP-I and Style-Con. These complementary metrics capture different aspects of visual similarity, improving the discrimination of model differences.
A3. Dataset Licensing and Public Availability
We agree that transparent data provenance and licensing information are essential for reproducibility and responsible reuse.
FontBench is primarily curated from FTransGAN, LSG-FCST, and TE141K through quality screening, redundancy reduction, and font-style categorization, resulting in 920 fonts and 609,432 glyph images.
We have revised Appendix A.5, Font Licensing and Public Availability, to clarify the dataset sources and release plan. Upon acceptance, we plan to release our curated annotations, font categories, dataset splits, and evaluation protocols, together with source attribution and licensing information.
A4. Generalization Settings and Character-level Splits
We agree that distinguishing font-level and character-level generalization is essential for correctly interpreting benchmark results.
We have clarified in Section 3.2 and Appendix A that FontBench adopts the Unseen Font, Seen Character (UFSC) setting for specialized models trained on our dataset. Specifically, the 524 training fonts and 396 test fonts are disjoint, while all 225 test characters are included in the 993-character training vocabulary. Thus, our evaluation measures generalization to unseen font identities rather than unseen character identities.
We have additionally evaluated all five specialized models on the shared 45-character subset, while retaining their full 225-character evaluation. Character-complexity analyses further examine performance across different glyph structures.These experiments strengthen evaluation coverage within UFSC.