acceptodds
Under review as a conference paper at ICLR 2027

Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characters

Abstract

Vision Large Language Models (VLLMs) have achieved remarkable success in modern text-rich visual understanding. However, their perceptual robustness in the face of the continuous morphological evolution of historical writing systems remains largely unexplored. Existing ancient text datasets typically focus on isolated historical periods, failing to capture the systematic visual distribution shifts spanning thousands of years. To bridge this gap and empower Digital Humanities, we introduce Chronicles-OCR, the first comprehensive benchmark specifically designed to evaluate the cross-temporal visual perception capabilities of VLLMs across the complete evolutionary trajectory of Chinese characters, known as the “Seven Chinese Scripts”. Curated in collaboration with top-tier institutional domain experts, the dataset comprises 2,800 strictly balanced images encompassing highly diverse physical media, ranging from tortoise shells to paper-based calligraphy. To accommodate the drastic morphological and topological variations across different historical stages, we propose a novel Stage-Adaptive Annotation Paradigm. Based on this, Chronicles-OCR formulates four rigorous quantitative tasks: cross-period character spotting, fine-grained archaic character recognition via visual referring, ancient text parsing, and script classification. By isolating visual perception from semantic reasoning, Chronicles-OCR provides an authoritative platform to expose the limitations of current VLLMs, paving the way for robust, evolution-aware historical text perception. The benchmark and evaluation code will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
VLLMs, Benchmark, Chinese Ancient Text

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_BbrYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces Chronicles-OCR, a comprehensive benchmark designed to evaluate the cross-temporal visual perception capabilities of VLLMs, especially on the evolutionary trajectory of Chinese characters. 2,800 images from tortoise shells to paper-based calligraphy are curated, together with a stage-adaptive annotation paradigm. Chronicles-OCR formulates four quantitative tasks, i.e., cross-period character spotting, fine-grained archaic character recognition via visual referring, ancient text parsing, and script classification.

Critical strengths
  1. A comprehensive OCR benchmark, Chronicles-OCR, covering seven Chinese scripts from unstandardized archaic symbols to mature pre-modern scripts is proposed.
  2. Four tasks tailored to each historical stage, namely spanning character spotting, fine-grained recognition for archaic scripts, ancient text parsing, and script classification, are designed.
  3. Experiments establish performance baselines, quantify perceptual bottlenecks across eras, and provide optimization directions.
Critical weaknesses
  1. All evaluations rely on a single zero-shot prompt template per task (Sec. B.1), omitting few-shot exemplars or prompt sensitivity analyses, which leaves open whether low archaic performance stems from visual decipherment limits or output formatting misalignment.
  2. The primary benchmark results in Tables 2 and 3 provide only single point estimates without standard errors or significance testing across repeated samplings, reducing statistical rigor for stochastic model evaluations.
  3. The dataset relies predominantly on data obtained through closed-source collaborations with museums and similar institutions. While such data are undoubtedly valuable, this dependence poses a barrier for future follow-up work seeking to build upon or improve the proposed approach. Did any open-source data/datasets are used?
  4. For each font type, the data size is only 400, and there are a total of 2800 images. This is smaller than other datasets, as compared in Tab. 1. Besides, is it feasible to construct the proposed time-evolving text OCR dataset by simply sampling from each of the compared datasets, given that font types and task types can be freely combined? This raises the question of whether the data presented here duplicates that of the aforementioned datasets. Further clarification is needed, and i recommend introducing quantitative measures (e.g., image attribute comparisons) to substantiate the distinction.
  5. In Section A.2 and Section C.3, the text reports normalized mutual information values (I(medium;script)/H(script)=100.0%I(\text{medium}; \text{script})/H(\text{script}) = 100.0\% for archaic scripts and 16.8%16.8\% for mature scripts) without providing the underlying formula or defining the joint and marginal probability mass functions.
  6. As shown in Table A.2 (Sec. A.2), Oracle Bone and Bronze scripts each appear in only a single medium (bone rubbings and vessel rubbings, respectively), meaning script labels are completely confounded with background textures, which undermines the validity of archaic script classification. In other words, is it possible to obtain the classification result (i.e., identify which character it is) by recognizing only the textual carrier/medium? This point requires clarification.
  7. Table A.1 and Table A.3 show severe length disparities across categories (e.g., 99.5% of Seal script samples have ≤10\le 10 characters, while Running script has a median of 72 characters), confounding parsing difficulty across scripts with sequence-length degradation and weakening experimental rigor. It is suggested to provide length-controlled evaluation slices for the Ancient Text Parsing task in Table 3 (e.g., comparing scripts across fixed character-length bins such as 1–10 and 11–30 characters) to separate sequence-length challenges from script decipherment performance.
  8. Format error in Sec. 4.3 “The Gap in Fine-grained Feature Mining.”. There is an inconsistent line break that is not in line with the context.
  9. Page 8 line 394: It is suggested to add the corresponding bbox-based prompts to Fig. 4 for better readability.
  10. The answers and prediction results in Figure 6 need to be accompanied by English translations to take into account readers who do not speak Chinese.

I will change my rating if the author addresses the above questions I raised.

Questions for authors
  1. How was the IoU threshold of 0.75 determined in the character spotting task?
  2. Page 8 line 421-424: It is suggested to conduct experiments on some open-source models and add attention map visualizations to illustrate the differences in layout perception.
  3. The claims about model size and reasoning mode should be supported by direct quantitative comparisons (e.g., figures and tables). Specifically, I recommend including a scatter plot of performance versus the number of parameters, and the model pairs compared in the reasoning analysis at Tab. C.1 should be moved to the main text. The layout of the article can also be improved.
9 comments

We thank the reviewer for the careful and detailed feedback. We greatly appreciate these suggestions, which have helped us improve Chronicles-OCR. We have addressed each comment with additional experiments, analyses, or clarifications, and incorporated the new experimental results into the main text and supplementary material. In the responses below, we identify the relevant sections, tables, and figures in the revised manuscript and explain the corresponding experiments and analyses.

Weakness 1 & 2: Prompt sensitivity and statistical reliability

Revision locations: Section 4.1; Appendices B.1 and E.1; Table E.1.

Response: Thank you for the suggestions on prompting and statistical reliability. We conducted additional experiments with multiple prompts and repeated generations to examine both the effects of prompt wording and the variability of responses under a fixed prompt.

We evaluated Qwen3.5-35B-A3B and Qwen3.5-397B-A17B on the same 700 images, with 100 images per script. Character spotting and fine-grained archaic character recognition used the 300 Oracle Bone, Bronze, and Seal images; text parsing and script classification used all 700 images. We report parsing results separately for the archaic scripts (Oracle Bone, Bronze, and Seal) and the mature scripts (Clerical, Regular, Running, and Cursive).

Each task used three prompts. P0 is the original prompt from Appendix B.1. P1 rephrases and reorganizes the instructions while retaining the task requirements and output format. P2 retains the task requirements and clarifies format constraints, including field names, answer prefixes, and category names.

All runs used zero-shot, single-turn, non-thinking inference. Input images and decoding settings were held fixed across prompts, with temperature set to 0.6 and top-p to 0.95. We generated five responses with different random seeds for each applicable image–task–prompt combination, yielding 60,000 responses across the two models. Before scoring, we applied the same output-normalization rules to all conditions: numeric script labels were mapped to the corresponding category names, and bbox and bbox_2d were accepted as equivalent bounding-box fields. Responses that could not be extracted remained in the evaluation denominator.

The table reports mean ± sample standard deviation over five repetitions, with all metrics on a 0–100 scale. Within each repetition, we first averaged image scores for the relevant evaluation group, then calculated the mean and standard deviation of these five averages. Parsing is reported as 100×NED100\times\mathrm{NED}, where NED is the score defined in Eq. (1): one minus the normalized Levenshtein edit distance, with higher values indicating better performance.

ModelEvaluationImagesP0P1P2
Qwen3.5-A3BCharacter spotting (F1)3007.13 ± 0.597.40 ± 0.277.21 ± 0.38
Qwen3.5-A3BFine-grained recognition (Acc.)30019.00 ± 0.2419.53 ± 1.0219.33 ± 0.75
Qwen3.5-A3BText parsing: archaic scripts30019.16 ± 0.5919.81 ± 0.2719.38 ± 0.61
Qwen3.5-A3BText parsing: mature scripts40071.79 ± 0.6572.74 ± 0.6472.66 ± 0.41
Qwen3.5-A3BScript classification (Acc.)70071.89 ± 1.0873.37 ± 1.1770.86 ± 0.59
Qwen3.5-A17BCharacter spotting (F1)30011.48 ± 0.2811.17 ± 0.0911.07 ± 0.26
Qwen3.5-A17BFine-grained recognition (Acc.)30024.60 ± 1.9124.07 ± 1.4824.53 ± 1.30
Qwen3.5-A17BText parsing: archaic scripts30021.78 ± 0.6122.37 ± 0.4922.02 ± 0.75
Qwen3.5-A17BText parsing: mature scripts40074.95 ± 0.2975.52 ± 0.4577.56 ± 0.50
Qwen3.5-A17BScript classification (Acc.)70076.46 ± 0.6776.11 ± 0.3478.57 ± 1.14

Prompt effects are modest under this common scoring protocol. The difference between the highest and lowest prompt means ranges from 0.28 to 2.61 points across the reported model–evaluation settings. The relative ranking of the two models is unchanged. For archaic-script parsing, the range is below 0.7 points for both models; for mature-script parsing, it is 0.95 points for A3B and 2.61 points for A17B.

For both A3B and A17B, parsing scores are substantially higher for mature scripts than for archaic scripts under each of the three prompts, P0, P1, and P2.

Recoverable output-format errors had little effect on the scores. To check this directly, we conducted an additional answer-extraction audit using the original inference outputs. The evaluation code already accommodates code fences and several answer-prefix variants. We additionally recovered unambiguous predictions from nonstandard outputs, retaining the emitted characters and the original coordinate-processing rules. This check required no new inference, and reference answers were not used during extraction.

Weaknesses 1–2 (continued)

For character spotting, 0.053% of the audited responses received higher scores after recovery. Averaging the score changes within each model and then equally across models gave an increase of approximately 0.003 points on a 0–100 scale. The mean increase was below 0.001 points for both archaic- and mature-script parsing, with no score change for fine-grained recognition or script classification. Together with the prompt comparisons, these results indicate that the low archaic-script scores cannot be explained primarily by recoverable formatting or answer-extraction errors.

Under fixed prompts, the five-repetition standard deviations range from 0.09 to 1.91 points, with standard errors of the mean ranging from 0.04 to 0.85 points, calculated as SD/5\sqrt{5}. These statistics describe generation variability on the fixed image set.

We also used the same generated results to estimate uncertainty in prompt differences. For each image and prompt, we averaged the five scores, then performed 10,000 paired bootstrap resamples of images within each script. Each resample used the same images for all prompts, and the image remained the sampling unit rather than its individual generated responses. We report 95% percentile confidence intervals for the score differences.

For example, in mature-script parsing, P2 improves over P0 by 0.87 points for A3B, with a 95% interval of [0.10, 1.67], and by 2.61 points for A17B, with a 95% interval of [1.67, 3.58]. These results indicate modest but measurable prompt effects. We report the generation standard deviations and image-bootstrap intervals separately because they describe different sources of uncertainty.

In these experiments, output-format issues and prompt changes have only a small effect on performance, indicating that the main difficulty lies in the models' limited ability to perceive and read archaic characters. Repeated generations introduce some variation, but this variation remains small under each of the three prompts. We have added the full P1/P2 prompts, extraction rules, audit results, and statistical analysis to Appendix E.1, with the prompt and repeated-generation results in Table E.1. P0 remains in Appendix B.1, and Section 4.1 now points readers to the additional analyses.

Weakness 3: Data sources, accessibility, and support for further research

Revision locations: Sections 3.1–3.2; Appendices A.2–A.3 and A.5; Ethics and Reproducibility Statements.

Response: Thank you for raising this concern. The revised manuscript describes the data sources and expert annotation in Sections 3.1–3.2, gives the provenance and handling of undeciphered characters in Appendix A.2–A.3, and states the benchmark release commitment in the Reproducibility Statement. We provide further clarification of the collection process and future data releases below.

Chronicles-OCR was assembled in collaboration with museums and research teams. Some samples came from museum materials and associated expert annotations that were not publicly available at the time of collection, as well as rubbings recently annotated by research institutions. Others came from publicly accessible historical artifacts and their digitized images, including steles, cliff inscriptions, plaques, antique objects, and calligraphic works. We curated and annotated these materials independently and did not draw images directly from existing public benchmarks. Similar rubbings, calligraphy, and artifact images are also available through open museum collections and other public resources in China and abroad, providing materials for further research.

Annotations were produced by teams with expertise in the relevant scripts. Quality checks included cross-checking within each team, independent expert review of every sample, and revisions following feedback. These checks covered character readings, transcription completeness, and reading order. Characters without a reliable decipherment were marked [UNK] to avoid treating conjectures as established readings. Details of the experts' backgrounds and verification procedures are now provided in Appendix A.5 (Expert Annotation and Validation), as well as in our response to Q2 from Reviewer 46j3.

We will publicly release all images, annotations, and evaluation code for Chronicles-OCR upon acceptance, allowing researchers to use the materials, reproduce the evaluation, and develop improved methods. Beyond the benchmark release described in the manuscript, we also plan to release additional data from the same source collections as the previously unpublished materials, to support research on ancient-script recognition, cultural heritage digitization, and the digital humanities.

Weakness 4: Dataset size, scope, and comparison with existing resources

Revision locations: Sections 2.3 and 3.1–3.3; Table 1; Appendices A.1–A.2 and B.1–B.3; Tables A.1–A.2.

Response: Thank you for this suggestion. The revised manuscript documents the benchmark's script and task coverage in Table 1, per-script composition in Table A.1, and physical-media distribution in Table A.2. We explain below how these properties differ from those of existing resources and provide additional image-attribute comparisons and overlap checks for this response.

Chronicles-OCR is designed to evaluate historical-script understanding from native visual inputs across diverse physical media. The main evaluation preserves the layout, background texture, and carrier context of each input image, without further text-region cropping or decomposition into isolated characters. The data include digitized oracle-bone rubbings, vessel-inscription rubbings, seal impressions, and calligraphy on paper or silk, as well as images of steles, cliff inscriptions, plaques, and portable antique objects. These materials allow us to evaluate localization, recognition, and continuous transcription across different media and visual contexts.

The benchmark contains 2,800 evaluation samples, with 400 for each of seven scripts. Equal sample counts facilitate comparisons across scripts. Character spotting and fine-grained archaic character recognition are evaluated on Oracle Bone, Bronze, and Seal scripts; text parsing and script classification cover all seven. This design supports evaluation across scripts and media.

We provide the following comparisons to make these differences more concrete.

For Oracle Bone, Chronicles-OCR contains more characters per image at a smaller relative scale. Since OBIMD provides oracle-bone rubbings and facsimiles, we compared the rubbing images using a common protocol based on the existing character boxes in both datasets. The median number of visible characters per image is 28 for Chronicles-OCR versus 7 for OBIMD. Median character height, normalized by image height, is 4.89% versus 10.92%, respectively. These differences provide distinct input conditions for evaluating localization, spatial relationships, and transcription of multiple characters within an image.

For Bronze and Seal, we preserve native artifact images, including their text layout and carrier context. Chronicles-OCR contains 400 images per script, covering vessel-inscription rubbings, seal impressions, and calligraphy on paper, without further cropping into isolated characters. EVOBC and GEVO provide character-level images that preserve individual glyphs but not the original multi-character layout or broader carrier context. Chronicles-OCR therefore supports character localization and reading within the original visual context. Because the original layouts and carrier context cannot be recovered from isolated-character crops, this evaluation setting cannot be reproduced simply by resampling or combining existing character-level datasets.

For Clerical, Regular, Running, and Cursive, Chronicles-OCR covers diverse artifact carriers beyond historical books and documents. M5HisDoc provides a historical-document recognition benchmark, while HisDoc1B provides a large historical-document corpus; both center on books and documents. Our evaluation covers calligraphy, steles, cliff inscriptions, plaques, and portable antique objects, preserving their native layouts and carrier context. Of the 1,600 images in these four scripts, 511 belong to the Monument category and 476 to the Antique category, totaling 987 images (61.7%) (Table A.2). Each script is represented across calligraphy, monuments, and antiques. The resulting variation in physical carriers and visual context cannot be reproduced simply by resampling the script subsets of these document-focused resources.

Retaining this native visual information also supports evaluation at different levels of granularity. For Oracle Bone, Bronze, and Seal, the character-box annotations can be used to extract individual characters for recognition experiments. Subsets can also be selected by script or medium to study particular domains. The same source images can therefore support both evaluation in context and evaluation on finer-grained crops. By comparison, spatial relationships and carrier information removed during character cropping are difficult to reconstruct by combining samples. Resampling historical documents likewise does not add coverage of steles, plaques, or antique objects. Preserving native visual context across diverse media is an important contribution of Chronicles-OCR alongside these resources, and one that is difficult to obtain through sampling and concatenation alone.

Reviewer_46j3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Chronicles-OCR, a benchmark for evaluating how well Vision-Language Large Models (VLLMs) perceive text across seven historical Chinese scripts. It contains 2,800 images, 400 per script, curated in collaboration with domain experts. The authors propose a stage-adaptive annotation paradigm and four evaluation tasks: cross-period character spotting, fine-grained archaic character recognition, ancient text parsing, and script classification. Experiments on a broad set of open-source and proprietary VLLMs reveal clear opportunities for improvement, especially in recognizing and localizing archaic Chinese characters.

I find this a timely and valuable contribution to historical document understanding and Digital Humanities. I have a few suggestions about dataset documentation, the interpretation of some results, and the framing of the cross-temporal claim. I believe addressing them would make the benchmark and its conclusions even more compelling.

Critical strengths
  1. The paper addresses an interesting and underexplored problem by collecting historical images from artifacts, museum collections, and domain experts. Its unified coverage of seven Chinese scripts provides a useful resource for historical document understanding and digital humanities research.
  2. The participation of domain experts, paleography researchers, and museum specialists is crucial in constructing the dataset, especially due to the specialized knowledge needed to interpret archaic Chinese characters.
  3. The stage-adaptive annotation paradigm accommodates differences between archaic and mature scripts. The four complementary tasks, particularly the distinction between character spotting and fine-grained recognition with a provided location, help investigate different aspects of historical character understanding. The broad coverage of vision-language models and specialized OCR systems also provides useful empirical comparisons.
Critical weaknesses
  1. The authors acknowledge in Appendix C.3 that the exceptionally high script-classification accuracy on archaic texts is strongly influenced by physical media. This raises concerns about whether the classification task primarily evaluates material or texture recognition rather than actual script comprehension. Although the authors investigate this limitation, a more controlled evaluation would strengthen the interpretation of the reported results. Furthermore, while the benchmark includes mappings from archaic characters to modern Chinese characters, the experiments primarily report performance by script category rather than directly comparing matched characters across historical stages. A clearer definition of the cross-temporal scope would help align the claims with the evaluation.
  2. Although the dataset is curated and cross-verified by domain experts, the paper provides limited quantitative evidence of annotation reliability and validation procedures. Furthermore, the selection of 400 images per script is not sufficiently justified in terms of representativeness across historical periods, writing styles, and physical conditions. The absence of human expert baselines and character-vocabulary statistics also makes it difficult to contextualize the benchmark's difficulty and understand the diversity of characters being evaluated.
Questions for authors
  1. Given the physical-medium confounding identified in Appendix C.3, could the authors provide a controlled script-classification evaluation that reduces material or background cues to better assess actual script recognition?
  2. Could the authors clarify the expert-validation process, including the number of annotators, validation coverage, disagreement-resolution procedures, and inter-annotator agreement where applicable?
  3. How were the 400 images per script selected, and how representative are they of different historical periods, writing styles, and physical conditions? Could the authors also provide character-vocabulary statistics, including the number of unique characters and their frequency distributions across scripts?
  4. Could the authors provide a human expert baseline, at least on a representative subset, to help contextualize model performance and distinguish model limitations from the inherent difficulty of historical character recognition?
  5. Could the authors clarify how the current tasks specifically evaluate cross-temporal perception? For example, an analysis of matched characters appearing across multiple historical scripts, or a more precise statement of the benchmark's scope, would help support this claim.
7 comments

We thank the reviewer for the positive assessment of our work, careful evaluation, and valuable suggestions. During the rebuttal period, we conducted additional experiments and analyses in response to these comments and incorporated the resulting findings and clarifications into the main text and supplementary material. Below, we address the reviewer's five questions in order, specifying the corresponding revision locations followed by our response to each question.

Q1: Background and physical-carrier cues in script classification

Revision locations: Section 4.2 (Script Classification); Appendix D.1, Physical Medium as a Shortcut in Script Classification; Table D.1; Figure D.1.

Response: Thank you for this suggestion. We added a background-reduction control experiment comparing Original images with two granularities of background removal, Crop and Mask, to assess whether script classification is sensitive to carrier and background cues.

We started from a pool of 1,200 images, with 400 images each of Oracle Bone, Bronze, and Seal scripts. Paired comparisons use the same subset with available character-box annotations across all three conditions.

  • Crop (coarse background removal): We retain the padded bounding rectangle enclosing all annotated character boxes, removing the background outside the overall text region.
  • Mask (fine background removal): We retain the union of individually padded character boxes and replace all remaining pixels with uniform gray, preserving the original canvas size and character positions.

Figure D.1 in the supplementary material presents the same images under Original, Crop, and Mask, illustrating the extent of background removal at each granularity.

We evaluated Qwen3.5-A3B (BF16) and Qwen3.5-A17B (official FP8 weights) on these three script subsets, retaining the benchmark's original seven-way classification task and prompt. Within each model, decoding and image preprocessing settings were fixed across conditions, with thinking disabled and temperature set to zero. Original was rerun under the same settings for this control experiment.

The results below reproduce Table D.1. Values are classification accuracy (%), with the decrease relative to the rerun Original baseline, in percentage points (pp), shown in parentheses.

ModelScriptOriginal (rerun)Crop: coarse removalMask: fine removal
Qwen3.5-A3BOracle Bone99.5090.25 (↓9.25)82.00 (↓17.50)
Qwen3.5-A3BBronze28.1016.71 (↓11.39)11.14 (↓16.96)
Qwen3.5-A3BSeal99.7599.00 (↓0.75)99.25 (↓0.50)
Qwen3.5-A17BOracle Bone99.2585.75 (↓13.50)74.00 (↓25.25)
Qwen3.5-A17BBronze73.6766.08 (↓7.59)62.53 (↓11.14)
Qwen3.5-A17BSeal98.7596.50 (↓2.25)96.00 (↓2.75)

Both models lose accuracy on Oracle Bone and Bronze under Crop and Mask, with larger decreases under Mask. For both scripts and both models, 95% confidence intervals from 10,000 paired image-level bootstrap resamples exclude zero for both Crop–Original and Mask–Original differences.

Seal classification is less affected: Mask accuracy remains at 99.25% for A3B and 96.00% for A17B. Seal Script has distinctive character forms and is less closely tied to specific physical carriers than Oracle Bone and Bronze scripts, which may help explain its smaller accuracy drop under background removal.

These results provide experimental support for material shortcuts in Oracle Bone and Bronze classification: reducing carrier and background context consistently lowers accuracy in both tested models. One plausible explanation is that current multimodal models have been exposed to complete rubbing images during training, learning associations between script categories and carrier or background cues alongside character morphology. The revised Section 4.2 summarizes this finding, and Appendix D.1 provides the experimental details, quantitative results (Table D.1), and input visualizations (Figure D.1).

Q2: Annotator expertise and the annotation and verification process

Revision locations: Appendix A.5, Expert Annotation and Validation.

Response: Thank you for this suggestion. We agree that clearly describing annotators' expertise, responsibilities, and verification procedures is important for understanding the reliability of the dataset. Chronicles-OCR was annotated collaboratively by domain teams whose expertise matches the requirements of each script, with the final annotations developed through cross-checking within teams, item-by-item review and acceptance by third-party experts, and revisions following feedback.

We organized specialist teams according to the knowledge required for each script:

  • Oracle Bone: Annotation and verification were carried out by a specialist laboratory dedicated to Oracle Bone research. The participating personnel have backgrounds in Oracle Bone studies and were responsible for identifying character forms, annotating character locations, mapping them to modern Chinese characters, and recovering the original reading order.
  • Bronze and Seal: Annotation was undertaken by professional researchers specializing in Bronze and Seal scripts, with relevant academic qualifications and research training at the master's or doctoral level. Their research directly concerns character-form analysis, decipherment, and correspondences between ancient and modern characters in these scripts, providing expertise for handling morphological variation, variant forms, and difficult characters.
  • Mature scripts, including Clerical, Regular, Running, and Cursive: A specialist annotation team comprising museum experts participated in curation, annotation, and verification. Drawing on expertise in reading historical calligraphy, paintings, and inscriptions on cultural artifacts, they transcribed texts in different writing styles, reconstructed reading order, and checked the annotated content.

The annotation procedure was adapted to the characteristics of each script. For Oracle Bone, Bronze, and Seal scripts, annotations include individual character locations, corresponding modern Chinese characters, and full-text transcriptions following the original reading order. For Clerical, Regular, Running, and Cursive scripts, annotations primarily consist of line- and paragraph-level transcriptions, preserving the content and reading order of continuous text.

After initial annotation, each annotation team conducted rigorous internal cross-checks, with team members checking one another's script labels, character readings, transcription completeness, and reading order. For the three archaic scripts with character boxes, these checks also covered character localization and the correspondence between boxes and transcription labels. Issues identified during these checks were discussed and corrected within the teams.

Following delivery, third-party expert reviewers with knowledge of the relevant scripts, who had not participated in the initial annotation, reviewed every delivered sample individually for acceptance. They consulted the original images, dictionaries relevant to the corresponding scripts, and other reference materials to assess annotation accuracy, completeness, and the evidence supporting character readings. Errors and uncertain cases identified during review were returned to the responsible annotation teams for re-examination, correction, and further confirmation.

When readings were disputed or characters were difficult to identify, multiple specialists with expertise in the corresponding script discussed the character-form evidence and reference materials and revised the annotations accordingly. Characters for which paleographic research has not established a reliable reading were consistently labeled [UNK], avoiding the use of speculative readings as definitive reference answers.

We have added these annotation and validation details to Appendix A.5, Expert Annotation and Validation. We will explicitly disclose the participating experts' institutional affiliations in the final version of the paper.

Q3: Sample selection, coverage, and character-vocabulary statistics

Revision locations: Appendix A.1, Per-Script Composition; Appendix A.6, Character Coverage and Frequency Distribution; Tables A.4–A.5.

Response: Thank you for this suggestion. We provide further clarification on image sources and sample coverage, together with character-vocabulary sizes and frequency distributions for each script.

Chronicles-OCR was developed in collaboration with museums and relevant research teams. Some samples were drawn from museum artifacts and associated expert annotations that were not publicly available at the time of collection, as well as rubbings recently annotated by research institutions. These artifacts are generally well preserved, and the materials span a broad historical range and diverse writing styles. We also digitized accessible historical artifacts, including stone inscriptions, plaques, antique objects, and calligraphy works, to incorporate text images from different physical media and forms of presentation.

Together, these sources constitute the evaluation materials covering all seven scripts. We use 400 images per script to maintain a balanced evaluation scale across scripts and prevent differences in sample counts from dominating the overall results. The samples cover different historical sources, physical carriers, and writing styles, enabling the benchmark to assess model perception across varied character forms and images of actual cultural artifacts.

To further quantify character coverage, we computed statistics separately for each script from the full-text transcription annotations of all 2,800 samples. We count Han characters as represented in the annotations, preserving simplified and traditional forms and differently encoded variants without merging them. Punctuation, whitespace, and other non-Han characters are excluded from the vocabulary. The undeciphered-character marker [UNK] is handled separately and is not counted as a known character. The results are shown below:

ScriptTotal Han-character occurrencesDistinct Han characters
Oracle Bone12,213529
Bronze4,377898
Seal1,8961,009
Clerical9,9712,364
Regular13,9582,676
Running31,5913,864
Cursive15,2262,807

Across the seven scripts, there are 89,232 Han-character occurrences and 5,724 distinct Han characters after deduplicating character types across scripts.

We further grouped characters by their occurrence frequency within each script. Each cell below reports the number of distinct character types in the corresponding frequency interval; each row sums to that script's vocabulary size.

ScriptOnce2–5 times6–10 times11–50 times51–100 timesMore than 100 times
Oracle Bone12418564933330
Bronze470284706356
Seal65330740900
Clerical1,08284821321371
Regular1,089998295270195
Running1,2941,3665036037028
Cursive1,1311,041314290256

The statistics reveal different character-frequency patterns across scripts. Oracle Bone has a relatively concentrated frequency distribution, with its 100 most frequent characters accounting for 83.52% of all known-character occurrences. Bronze includes both frequent characters and many infrequent ones. Although Seal images typically contain little text, the subset includes 1,009 distinct Han characters, 64.72% of which occur only once, reflecting character diversity within short texts. Clerical, Regular, Running, and Cursive have larger vocabularies, ranging from 2,364 to 3,864 distinct Han characters, and include substantial numbers of infrequent characters. Together, these statistics describe the benchmark's character coverage and frequency structure, providing additional context for interpreting model performance across scripts.

The revised Appendix A.1 clarifies the per-script composition and diversity of historical-text settings. Appendix A.6 now reports the character-counting protocol, vocabulary sizes (Table A.4), and frequency distributions (Table A.5). All images and annotations will be publicly released, together with complete character vocabularies and frequency information for each script, providing a reproducible evaluation platform for ancient-script perception.

Reviewer_ijozOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new benchmark, Chronicles-OCR, for evaluating the VLLMs' visual perception ability of ancient Chinese text. The proposed benchmark comprehensively encompasses the complete evolution trajectory of Chinese characters across thousands of years, seven Chinese scripts, and diverse physical media. The data is meticulously annotated with multiple tasks. Extensive experiments and analysis offer a solid foundation for future research.

Critical strengths
  1. For the first time, the proposed Chronicles-OCR extensively covers Chinese characters spanning thousands of years, diverse physical media, and seven Chinese scripts. Moreover, the authors design a novel Stage-Adaptive Annotation Paradigm and collaborate with domain experts to ensure the high-quality annotation of Chronicles-OCR.
  2. Chronicles-OCR supports evaluation of four OCR-related tasks including cross-period character spotting, fine-grained archaic character recognition, ancient text parsing, and script classification. Comprehensive experiments and analysis are conducted mainly based on both open-source and proprietary VLMs.
Critical weaknesses

None

Questions for authors

In Table C.3, nearly all OCR models exhibit very poor performance on the parsing task of Chronicles-OCR. However, these OCR models are not trained for ancient Chinese text but for modern document parsing. Could you provide the performance of some recent OCR models that support ancient Chinese text parsing?

3 comments

We thank the reviewer for the positive assessment of our work and the constructive feedback. In response, we have conducted additional experiments and incorporated the results into the revised manuscript. Below, we provide our response and indicate the corresponding revision locations.

Q1: OCR models for historical Chinese documents

Revision locations: Appendix D.2, Parsing with Specialized OCR Models; Table D.2.

Response: We thank the reviewer for the positive assessment of our work and for suggesting additional evaluation of OCR models for historical Chinese texts.

Following this suggestion, we evaluated five additional OCR systems on the full Chronicles-OCR test set, covering all seven scripts. Gulian OCR, TongGuOCR, and KanDianGuJi v2 focus on ancient books and historical documents; HunyuanOCR and Xiaomi-OCR-0 are recent models that support ancient Chinese text parsing.

We accessed Gulian OCR through its official web platform and KanDianGuJi v2 through its official OCR API, selecting the v2 recognition engine. TongGuOCR, HunyuanOCR, and Xiaomi-OCR-0 were deployed on our GPU servers using their publicly released model weights. For TongGuOCR, we used the complete official pipeline for layout detection, text-block segmentation, cropping, and recognition. For HunyuanOCR and Xiaomi-OCR-0, we used the official inference implementations with the text-extraction and document-parsing prompts, respectively.

The table below reports the five additional systems included in Table D.2. We use the NED score defined in Eq. (1) on a 0–1 scale, with higher scores indicating better performance. The archaic-script average covers Oracle Bone, Bronze, and Seal, while the mature-script average covers Clerical, Regular, Running, and Cursive.

ModelAvg. (Archaic)Avg. (Mature)Oracle BoneBronzeSealClericalRegularRunningCursive
Gulian OCR0.040.490.000.020.090.400.350.600.60
TongGuOCR0.050.420.010.020.110.450.430.500.28
KanDianGuJi v20.040.300.000.010.110.310.260.390.25
HunyuanOCR0.110.700.010.060.260.770.820.700.52
Xiaomi-OCR-00.160.730.010.110.370.740.740.760.66

The additional results show a consistent pattern: these systems perform substantially better on mature scripts than on archaic scripts, with Oracle Bone and Bronze remaining particularly challenging. This gap suggests that extending OCR capabilities from ancient books and historical documents to earlier forms of Chinese writing remains an open challenge. It also supports the value of evaluating recognition across the historical scripts covered by Chronicles-OCR.

We have included these results in Appendix D.2 (Parsing with Specialized OCR Models) and Table D.2 of the revised manuscript. Our search may have overlooked other relevant OCR models for historical Chinese texts. We would be grateful for any recommendations from the reviewer and would be happy to extend the evaluation to include these models in the final version.

Thank you for your additional experimental results of specialized OCR models. I would like to maintain my original rating.

We sincerely appreciate your response and continued support. Thank you for your time, constructive feedback, and positive recognition of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.