acceptodds
Under review as a conference paper at ICLR 2027

AnimeText: A Large-Scale Dataset and Benchmark for Text Detection in Anime-Style Images

Abstract

Text in anime-style images includes dialogue, titles, sound effects, signatures, and lettering drawn into the artwork. We introduce AnimeText, a dataset of 735,060 images and 4,360,393 text boxes spanning comics, illustrations, posters, and edited images. AnimeText includes four kinds of annotation: fine boxes for compact text regions, coarse boxes for larger groups, polygons for irregular boundaries, and rejected candidates for text-like patterns and poorly localized proposals. We build these annotations with detector proposals and a fine-tuned candidate classifier, and annotators correct the boxes and add text the detector missed. Trained on 5,965 AnimeText images, YOLO11m reaches 0.850 F1 on the full test split of 73,725 images, compared with 0.545 when trained on the same number of Manga109s images. In a labeled sample of test images, non-dialogue text such as sound effects, titles, and signatures accounts for most of the newly detected regions, and its recall rises from 18.3% to 69.8%; dialogue recall also rises from 63.4% to 91.3%. Training on 513,159 AnimeText images raises F1 from 0.850 to 0.896 and AP50:95 from 0.781 to 0.882. Very small text and crowded pages remain the hardest cases.

Primary area:
datasets and benchmarks
Keywords:
Text Detection, Dataset, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8nbyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AnimeText, a dataset of 735,060 images and 4,360,393 text boxes spanning comics, illustrations, posters, and edited images, which includes four kinds of annotation. Experiments demonstrate improvements over training on an equally sized dataset, particularly for non-dialogue text, and show further performance gains as the training scales.

Critical strengths
  1. AnimeText surpasses most existing manga and scene-text datasets considered in this study in both size and diversity.
  2. The study examines different training-set scales and detection architectures, with detailed evaluations across diverse text attributes.
Critical weaknesses
  1. Insufficient dataset transparency and quality assurance. This paper lacks essential annotation details, including annotator qualifications, agreement measures, quality audits. These issues weaken confidence in the dataset’s usability and reliability.
  2. Confounded comparisons and unclear data separation. Evaluating models on AnimeText favors training data that matches its domain, text categories, and annotation format, making it difficult to isolate the dataset’s actual advantage.
Questions for authors

1.What are the origins and licensing terms of the 735,060 images, and are the authors authorized to publicly distribute them? 2. How are the 985 duplicated images distributed among the training, validation, and test sets, and were they excluded from every reported experiment, particularly those in Tables 5, 6, 16, 19, and 20? 3. Please also explain the approach used to identify exact duplicates and visually similar copies. 4. Could the authors conduct an independent assessment of the quality of each annotation type? 5. Please report the human-validated accuracy of the labels generated by Qwen and provide the specific model version, classification prompt, and inference parameters. 6. Could the authors further disentangle the effects of domain diversity, annotation format, and dataset scale through size-matched training, standardized bounding-box definitions, and testing on an external anime or comic dataset?

1 comment

We thank the reviewer for the detailed questions.

Q1. Origins and licensing. The images come from Danbooru, and we have obtained permission for their use. The dataset will be released on Hugging Face.

Q2. The 985 images. They are distinct training images with an exact duplicate in another split, forming 1,065 cross-split groups: 643 training–validation, 333 training–test, 87 validation–test, and 2 shared by all three. The matched-budget and scaling experiments train on the 513,159-image pool, which excludes all 985 and shares no exact duplicate with the test split. On the test split, removing all 4,572 duplicate-linked images leaves the matched-budget advantage at 19.3 points or more in the worst case.

Tables 5, 16, 19, and 20 use the validation split (protocols in Table 6). Re-evaluating our full-data YOLO11m checkpoint on all 147,191 validation images:

ImagesExclusionF1AP50:95
147,191None0.83300.7847
146,546Exact duplicates0.83300.7848
138,389Exact and near duplicates0.83360.7846

F1 changes by at most 0.06 points.

Q3. Duplicate identification. Exact duplicates: SHA256 of the original image bytes. Near duplicates: 64-bit DCT perceptual hash, Hamming distance ≤ 4, aspect-ratio factor ≤ 1.1, comparing every test image with every training and validation image. Two human reviewers confirmed 97.6% (164/168) and 99.5% (199/200) of sampled candidate pairs as genuine duplicates and agreed on 98.2% of shared pairs.

Q4. Assessment of each annotation type. Five reviewers took part, two of them external to dataset construction.

AnnotationAssessmentResult
Fine boxesIndependent re-annotation, 30 random test images, source hidden86.8% of text regions annotated; precision 0.908; median IoU 0.846; benchmark gap unchanged (23.9 → 24.0 points)
Text regions1,423 annotated regions98.8% confirmed as text
Accepted candidates150 random, class hidden93.8% (internal), 88.0% / 88.3% (external)
Rejected candidates150 random, class hiddenConcentrated on watermarks, English captions, display lettering, symbol marks (v4Md W1)
Coarse boxesGranularity-tolerant matchingRecall gap of 37.2 points remains

The two external reviewers agree on 88.7% of candidate decisions.

Q5. Qwen. Qwen/Qwen3.8-27B (snapshot 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0), temperature 0, 1,536-token limit, thinking disabled, strict JSON schema. Each request supplied the full image with numbered boxes and matching crops for up to six regions. The sample holds all 1,423 fine regions of 256 random test images. The prompt defines 10 function, 5 lettering, and 5 layout classes and will appear verbatim in the appendix. An external reviewer labeled all 1,423 regions:

AxisAgreementκ
Dialogue / non-dialogue91.1%0.84
Function (10 classes)83.5%0.77
Layout86.8%0.78
Lettering77.0%0.64

Under human labels, non-dialogue text is 61.5% of the 581 gained regions (95% CI 55.6–67.4%), versus 63.0% under Qwen labels.

Q6. Domain, format, and scale.

  • Scale. At 5,965 images with identical architecture and recipe, AnimeText raises F1 from 0.545 to 0.850 with fewer annotated regions (32,759 vs. 78,704); 35,000 and 513,159 images raise it to 0.879 and 0.896.
  • Format. Accepting coarse boxes as alternative matches standardizes box definitions; the recall gap remains 37.2 points (95% CI 33.0–41.8).
  • External data. On the Manga109s test set (862 pages, no exact or near duplicates with AnimeText training), the AnimeText detector reaches 0.772 F1 without Manga109s training, while the matched Manga109s-trained detector reaches 0.545 F1 on AnimeText.
  • Reference annotation. Against an independent human reference, the advantage is unchanged (General Response §2).
Reviewer_v4MdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AnimeText, a large-scale dataset for text detection in anime-style images, containing 735,060 images and 4,360,393 text boxes across comics, illustrations, posters, and edited images. The dataset provides fine boxes, coarse boxes, polygons, and rejected candidates, constructed through detector proposals, a fine-tuned candidate classifier, and human correction. The authors train YOLO11m on AnimeText subsets and report strong gains over training on Manga109s, including improved recall for non-dialogue text and dialogue, and further gains when scaling training images from 5,965 to 513,159.

Critical strengths
  1. The dataset is large and covers diverse anime-style text types, including dialogue, sound effects, titles, and signatures, which are underrepresented in existing benchmarks.

  2. The four annotation types are thoughtfully designed to handle compact text, grouped text, irregular boundaries, and text-like negatives.

  3. The construction pipeline combines detector proposals, a candidate classifier, and human correction, which is a practical way to scale annotation.

  4. The experiments show clear empirical benefits from AnimeText, including stratified recall improvements and scaling behavior, making the dataset’s utility concrete.

Critical weaknesses
  1. Dataset documentation and reproducibility are insufficient. The paper lacks clear information on image sources, copyright/licensing, redistribution policy, annotation guidelines, quality control, and inter-annotator agreement, which are essential for a large anime-style dataset.

  2. The benchmark protocol and baseline comparison need clarification. The 5,965-training / 73,725-test split is highly imbalanced, and the Manga109s comparison may not be apples-to-apples due to domain, annotation taxonomy, and budget differences. Splits, duplicate removal, artist/series-level leakage control, and evaluation matching for fine/coarse/polygon boxes should be specified.

  3. Claims about non-dialogue text and “newly detected regions” need stronger evidence. Detection F1/AP do not establish recognition or readability, and newly detected regions may include text-like false positives. The definitions of dialogue/non-dialogue, stratified recall, and false-positive analysis need to be formalized.

  4. The evaluated methods include only YOLO11m, DBNet, and LRANet. More baseline methods and an extension to the full spotting benchmark would benefit this work.

Questions for authors

Please refer to the weaknesses.

2 comments

We thank the reviewer for recognizing the design of the four annotation types and the construction pipeline.

W1. Documentation, quality control, and agreement. The images come from Danbooru, and we have obtained permission for their use. The dataset will be released on Hugging Face with the annotation guidelines and audit records.

Guidelines. A fine box (B⁰) tightly covers a complete line, column, or compact character group. A coarse box (B¹) groups related fine boxes, and a polygon covers the text region including inter-character spacing. Signatures, watermarks, and artistic lettering are text; interface icons and decorative patterns are not. Rejected candidates are proposals that annotators excluded from the positive target set. Compared with 150 accepted candidates, 150 rejected candidates contain more watermarks and signatures (17 vs. 3), overlaid English captions (73 vs. 22), large display lettering (30 vs. 13), and symbol-only marks (10 vs. 1), which are the regions detectors most often confuse. Using them in training improves detection (Table 5).

Quality and agreement. General Response §1 gives the human audit: 86.8% of independently drawn text regions are annotated, released boxes reach 0.908 precision, and accepted candidates are confirmed at 93.8%. Reviewers agree on 88.7% of candidate decisions, 98.2% of duplicate decisions, and 82.2% of dialogue/non-dialogue labels.

W2. Protocol and the Manga109s comparison. The released splits contain 514,144 training, 147,191 validation, and 73,725 test images. The 5,965-image set is a size-matched subset for the controlled comparison; the scaling study trains on up to 513,159 images. Both matched models share the YOLO11m architecture, COCO initialization, 50-epoch recipe, augmentation, and resolution, and both use thresholds selected on the same 5,000-image validation subset. With fewer training regions (32,759 vs. 78,704), AnimeText raises test F1 from 0.545 to 0.850, and the advantage holds at a common threshold of 0.25 (0.511 vs. 0.856).

Format and domain do not explain the gap. Accepting coarse boxes as alternative matches leaves a 37.2-point recall gap (95% CI 33.0–41.8), and the AnimeText detector reaches 0.772 F1 on Manga109s without Manga109s training data. Duplicate handling is described in our answer to ssP2 Q1: removing all 4,572 duplicate-linked test images leaves an advantage of at least 19.3 points.

The main benchmark evaluates the 422,417 test fine boxes with one-to-one matching at IoU 0.5 and reports P/R/F1, AP50, and AP50:95. The additional YOLO11m and YOLO12 experiments evaluate all text boxes on the validation split (Table 6).

W3. Non-dialogue text and false positives. A newly detected region is a ground-truth region matched (IoU ≥ 0.5) by the AnimeText-trained detector and missed by the Manga109s-trained detector, so every such region is annotated text. False positives enter precision and F1, where AnimeText training also improves. Stratified recall is the fraction of reference regions in a category that a detector matches. Under human labels for all 1,423 regions (91.1% agreement with Qwen on dialogue/non-dialogue), non-dialogue text accounts for 61.5% of the 581 gained regions. Recall by human-labeled function:

FunctionManga109sAnimeText
Non-dialogue9.6%72.3%
Onomatopoeia19.2%80.8%
Title / display4.8%61.9%
Dialogue73.6%92.3%

W4. Methods and spotting. The experiments cover four detector designs: YOLO11m and YOLO12 (one-stage), DBNet (segmentation-based), and LRANet (contour-based). AnimeText training improves every one; relative to ICDAR15 training, DBNet rises from 0.008 to 0.743 F1 and LRANet from 0.134 to 0.855. Detection is the first stage of any spotting pipeline, and the polygon annotations support curved and irregular text for spotting systems.

Dear Reviewer v4Md,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

Reviewer_ssP2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AnimeText, a dataset and benchmark for text detection in anime-style images.

Critical strengths

(1) AnimeText covers dialogue, titles, sound effects, signatures, andlettering drawn into the artwork. It is a useful resource for studying text that existing manga datasets represent poorly.

(2) With the same architecture, initialization, training schedule, and 5,965 training images, AnimeText raises test F1 from 0.545 to 0.850 despite providing fewer annotated regions. Analysis of a 256-image sample shows that non-dialogue text accounts for most additional detections.

Critical weaknesses

(1) The paper describes human correction but reports no quantitative audit of missed text, box accuracy, or annotation consistency. The analysis attributing gains to non-dialogue text also uses Qwen-generated labels for 256 images without reported human validation. For a paper whose main contribution is a dataset, these omissions leave benchmark quality and the interpretation of category-specific gains insufficiently established.

(2) Data provenance and split integrity need clarification.

(3) This dataset is beneficial for advancing research on text image analysis and understanding communities, which is also one of the biggest contributions of this article. The author should release the dataset.

Questions for authors

(1) Which image sources contribute to each split, and how were exact and near duplicates across sources identified? This would help assess the claim that the test set measures generalization beyond training sources.

(2) What quantitative quality checks were performed after human correction to measure missed text, localization errors, and consistency between accepted and rejected candidates? Were any evaluation images independently reannotated to check completeness?

(3) Were Qwen’s text-function labels on the 256-image sample checked by humans? If so, what disagreement rate was observed, and did corrections change the conclusion that non-dialogue text accounts for most additional detections?

3 comments

We thank the reviewer for the positive assessment and for the precise questions.

Q1. Image sources, and identification of exact and near duplicates. All images come from Danbooru. Exact duplicates are identified by SHA256 of the original encoded image bytes. Near duplicates are screened with a 64-bit DCT perceptual hash of the EXIF-corrected 32×32 grayscale image, at Hamming distance ≤ 4 and aspect-ratio factor ≤ 1.1. Human review confirms the screen: two reviewers judged 97.6% (164/168) and 99.5% (199/200) of sampled candidate pairs to be genuine duplicates (exact copies, transformed copies, or partial overlaps), and agreed on 98.2% of the pairs both reviewed.

The main training pool removes the 985 training images that have an exact duplicate in validation or test (513,159 images remain), and it shares no exact duplicate with the test split. To bound the effect of near duplicates, we removed all 4,572 test images linked to the original training or validation pool by either screen and computed the worst case at fixed predictions. The AnimeText model's F1 stays at or above 0.842 and the Manga109s model's F1 at or below 0.649, so the matched-budget advantage remains at least 19.3 points. The test split therefore measures generalization beyond duplicated training content. The transfer result in the General Response (0.772 F1 on Manga109s with no Manga109s training data, after a screen that found no overlap with any of the 514,144 training images) supports the same conclusion on an external dataset.

Q2. Quantitative quality checks, consistency of candidates, and independent re-annotation. Yes, we independently re-annotated evaluation images. On 30 randomly sampled test images, an annotator drew every fine box with the source annotation hidden, producing 212 text regions. 184 of them (86.8%) are already annotated in the release; the remaining 28 (13.2%) are missed text. The released boxes reach 0.908 precision against this reference, with a median matched IoU of 0.846.

The re-scoring in General Response §2 shows that the benchmark conclusions do not depend on the missed regions: the F1 gap between the AnimeText and Manga109s detectors is 23.9 points against the release and 24.0 points against the independent reference.

For candidate consistency, we sampled 150 accepted candidates uniformly from the test split and reviewed them without showing the source class. The internal review confirms 93.8% (95% CI 88.7–96.7%) as text with adequate extent, and two external reviewers confirm 88.0% and 88.3%. The two external reviewers agree on 88.7% of accept/reject decisions. Across the 1,423 annotated regions of the content sample, a reviewer confirmed 98.8% as text.

Q3. Human check of Qwen's labels. An external reviewer labeled all 1,423 regions. Agreement with Qwen is 91.1% for dialogue vs. non-dialogue (κ = 0.84), 83.5% for the 10 function classes (κ = 0.77), and 86.8% for layout (κ = 0.78). With human labels, non-dialogue text accounts for 61.5% of the 581 gained regions (95% CI 55.6–67.4%), compared with 63.0% under Qwen labels. The human correction leaves the conclusion unchanged.

Release. The dataset will be released on Hugging Face with all four annotation forms, the annotation guidelines, and the audit records. We have obtained permission for the Danbooru images.

Dear Reviewer ssP2,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

Reviewer_ssP2OpenReview ↗

I keep my positive score unchanged

Thank you very much to the authors for their detailed responses to my three questions and for answering my previous confusion. I believe that the dataset established in this article will be helpful for future research in the document image analysis and recognition community. We once again hope that the authors will release the dataset as soon as possible. Finally, regarding the second question, the authors only randomly selected 30 samples for analysis. I think the scale is too small and may not be suitable for statistical analysis of the results. I will keep my positive score unchanged.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. All three reviewers recognize AnimeText's coverage of dialogue, sound effects, titles, signatures, and lettering drawn into artwork, which existing manga and scene-text datasets represent poorly. The reviews raise four shared questions: annotation quality, split integrity, human validation of the content analysis, and whether the gains reflect the training resource. We ran a dedicated human audit and additional experiments to answer each of them.

1. Annotation quality, measured by humans (ssP2, v4Md, 8nby). Five reviewers took part: three from our team and two external reviewers who did not take part in dataset construction. Reviewers judged each item from the image and the target region alone, without access to the source class or to other reviewers' answers.

AuditSampleResult
Independent re-annotation30 random test images, source annotations hidden; reviewer draws every fine box86.8% of independently drawn text regions already annotated in the release; precision 0.908 [0.828, 0.970]; matched-box IoU median 0.846 (IQR 0.792–0.891)
Accepted text regions1,423 annotated regions in 256 random test images98.8% (1,406/1,423) confirmed as text
Accepted candidates150 sampled uniformly from the test split93.8% confirmed as text with adequate extent (internal), 88.0% and 88.3% (two external reviewers)
Inter-reviewer agreement, accept/reject decisions271–296 shared candidates88.7% between the two external reviewers; 87.8% and 88.6% between each external reviewer and the internal review
Inter-reviewer agreement, duplicate decisions168 shared pairs98.2%

2. The benchmark conclusion holds under the independent human reference. We re-scored our released detectors and the publicly released Manga109s YOLO11m detector on the 30 independently annotated images, once against the released annotation and once against the new human annotation.

DetectorF1 (released ref.)F1 (independent ref.)AP50 (released ref.)AP50 (independent ref.)
AnimeText YOLO11m0.7560.7170.7880.720
AnimeText YOLO12l0.8260.7850.8730.791
Manga109s YOLO11m0.5180.4760.4330.387

The F1 gap between the AnimeText and Manga109s YOLO11m detectors is 23.9 points with the released reference and 24.0 points with the independent reference. For YOLO12l the gap is 30.8 points under both. Against the independent reference, the AnimeText detector's precision rises from 0.750 to 0.790: 18 of its detections that count as false positives under the release match text drawn by the independent annotator. The model therefore detects real text beyond the annotated set, and the ranking and size of the advantage do not depend on the reference annotation.

3. Human validation of all Qwen labels (ssP2, 8nby). An external reviewer labeled all 1,423 regions of the content-analysis sample with the same label scheme. Agreement with Qwen is 91.1% for dialogue vs. non-dialogue (κ = 0.84) and 83.5% over the 10 function classes (κ = 0.77). Human labels reproduce the paper's conclusion: non-dialogue text accounts for 61.5% (357/581, 95% CI 55.6–67.4%) of the regions gained by AnimeText training, compared with 63.0% under Qwen labels.

4. Split integrity, verified by humans (ssP2, v4Md, 8nby). Human reviewers confirmed 97.6% (164/168) and 99.5% (199/200) of sampled perceptual-hash candidates as genuine duplicates, so the screen identifies duplicates precisely. The main training pool excludes every training image with an exact duplicate elsewhere and has no exact overlap with the test split. Removing all 4,572 test images linked to training or validation by exact or near-duplicate screening leaves the matched-budget F1 advantage at 19.3 points or more under any assignment of the original errors. On the validation split used by the remaining experiments, excluding all 8,802 duplicate-linked images changes F1 by at most 0.06 points.

5. The gain comes from the resource (8nby, v4Md). The AnimeText detector transfers to the Manga109s test set at 0.772 F1 without any Manga109s training data, while Manga109s-trained detectors reach 0.52–0.55 F1 on AnimeText. When coarse boxes are accepted as alternative matches to remove annotation-granularity differences, the matched-budget recall gap remains 37.2 points (95% CI 33.0–41.8).

6. Release. The images come from Danbooru, and we have obtained permission for their use. AnimeText, including all four annotation forms, the annotation guidelines, and the audit records, will be released on Hugging Face.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.