- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces Chronicles-OCR, a comprehensive benchmark designed to evaluate the cross-temporal visual perception capabilities of VLLMs, especially on the evolutionary trajectory of Chinese characters. 2,800 images from tortoise shells to paper-based calligraphy are curated, together with a stage-adaptive annotation paradigm. Chronicles-OCR formulates four quantitative tasks, i.e., cross-period character spotting, fine-grained archaic character recognition via visual referring, ancient text parsing, and script classification.
Critical strengths
- A comprehensive OCR benchmark, Chronicles-OCR, covering seven Chinese scripts from unstandardized archaic symbols to mature pre-modern scripts is proposed.
- Four tasks tailored to each historical stage, namely spanning character spotting, fine-grained recognition for archaic scripts, ancient text parsing, and script classification, are designed.
- Experiments establish performance baselines, quantify perceptual bottlenecks across eras, and provide optimization directions.
Critical weaknesses
- All evaluations rely on a single zero-shot prompt template per task (Sec. B.1), omitting few-shot exemplars or prompt sensitivity analyses, which leaves open whether low archaic performance stems from visual decipherment limits or output formatting misalignment.
- The primary benchmark results in Tables 2 and 3 provide only single point estimates without standard errors or significance testing across repeated samplings, reducing statistical rigor for stochastic model evaluations.
- The dataset relies predominantly on data obtained through closed-source collaborations with museums and similar institutions. While such data are undoubtedly valuable, this dependence poses a barrier for future follow-up work seeking to build upon or improve the proposed approach. Did any open-source data/datasets are used?
- For each font type, the data size is only 400, and there are a total of 2800 images. This is smaller than other datasets, as compared in Tab. 1. Besides, is it feasible to construct the proposed time-evolving text OCR dataset by simply sampling from each of the compared datasets, given that font types and task types can be freely combined? This raises the question of whether the data presented here duplicates that of the aforementioned datasets. Further clarification is needed, and i recommend introducing quantitative measures (e.g., image attribute comparisons) to substantiate the distinction.
- In Section A.2 and Section C.3, the text reports normalized mutual information values ( for archaic scripts and for mature scripts) without providing the underlying formula or defining the joint and marginal probability mass functions.
- As shown in Table A.2 (Sec. A.2), Oracle Bone and Bronze scripts each appear in only a single medium (bone rubbings and vessel rubbings, respectively), meaning script labels are completely confounded with background textures, which undermines the validity of archaic script classification. In other words, is it possible to obtain the classification result (i.e., identify which character it is) by recognizing only the textual carrier/medium? This point requires clarification.
- Table A.1 and Table A.3 show severe length disparities across categories (e.g., 99.5% of Seal script samples have characters, while Running script has a median of 72 characters), confounding parsing difficulty across scripts with sequence-length degradation and weakening experimental rigor. It is suggested to provide length-controlled evaluation slices for the Ancient Text Parsing task in Table 3 (e.g., comparing scripts across fixed character-length bins such as 1–10 and 11–30 characters) to separate sequence-length challenges from script decipherment performance.
- Format error in Sec. 4.3 “The Gap in Fine-grained Feature Mining.”. There is an inconsistent line break that is not in line with the context.
- Page 8 line 394: It is suggested to add the corresponding bbox-based prompts to Fig. 4 for better readability.
- The answers and prediction results in Figure 6 need to be accompanied by English translations to take into account readers who do not speak Chinese.
I will change my rating if the author addresses the above questions I raised.
Questions for authors
- How was the IoU threshold of 0.75 determined in the character spotting task?
- Page 8 line 421-424: It is suggested to conduct experiments on some open-source models and add attention map visualizations to illustrate the differences in layout perception.
- The claims about model size and reasoning mode should be supported by direct quantitative comparisons (e.g., figures and tables). Specifically, I recommend including a scatter plot of performance versus the number of parameters, and the model pairs compared in the reasoning analysis at Tab. C.1 should be moved to the main text. The layout of the article can also be improved.
9 comments
Weaknesses 1–2 (continued)
For character spotting, 0.053% of the audited responses received higher scores after recovery. Averaging the score changes within each model and then equally across models gave an increase of approximately 0.003 points on a 0–100 scale. The mean increase was below 0.001 points for both archaic- and mature-script parsing, with no score change for fine-grained recognition or script classification. Together with the prompt comparisons, these results indicate that the low archaic-script scores cannot be explained primarily by recoverable formatting or answer-extraction errors.
Under fixed prompts, the five-repetition standard deviations range from 0.09 to 1.91 points, with standard errors of the mean ranging from 0.04 to 0.85 points, calculated as SD/. These statistics describe generation variability on the fixed image set.
We also used the same generated results to estimate uncertainty in prompt differences. For each image and prompt, we averaged the five scores, then performed 10,000 paired bootstrap resamples of images within each script. Each resample used the same images for all prompts, and the image remained the sampling unit rather than its individual generated responses. We report 95% percentile confidence intervals for the score differences.
For example, in mature-script parsing, P2 improves over P0 by 0.87 points for A3B, with a 95% interval of [0.10, 1.67], and by 2.61 points for A17B, with a 95% interval of [1.67, 3.58]. These results indicate modest but measurable prompt effects. We report the generation standard deviations and image-bootstrap intervals separately because they describe different sources of uncertainty.
In these experiments, output-format issues and prompt changes have only a small effect on performance, indicating that the main difficulty lies in the models' limited ability to perceive and read archaic characters. Repeated generations introduce some variation, but this variation remains small under each of the three prompts. We have added the full P1/P2 prompts, extraction rules, audit results, and statistical analysis to Appendix E.1, with the prompt and repeated-generation results in Table E.1. P0 remains in Appendix B.1, and Section 4.1 now points readers to the additional analyses.
Weakness 3: Data sources, accessibility, and support for further research
Revision locations: Sections 3.1–3.2; Appendices A.2–A.3 and A.5; Ethics and Reproducibility Statements.
Response: Thank you for raising this concern. The revised manuscript describes the data sources and expert annotation in Sections 3.1–3.2, gives the provenance and handling of undeciphered characters in Appendix A.2–A.3, and states the benchmark release commitment in the Reproducibility Statement. We provide further clarification of the collection process and future data releases below.
Chronicles-OCR was assembled in collaboration with museums and research teams. Some samples came from museum materials and associated expert annotations that were not publicly available at the time of collection, as well as rubbings recently annotated by research institutions. Others came from publicly accessible historical artifacts and their digitized images, including steles, cliff inscriptions, plaques, antique objects, and calligraphic works. We curated and annotated these materials independently and did not draw images directly from existing public benchmarks. Similar rubbings, calligraphy, and artifact images are also available through open museum collections and other public resources in China and abroad, providing materials for further research.
Annotations were produced by teams with expertise in the relevant scripts. Quality checks included cross-checking within each team, independent expert review of every sample, and revisions following feedback. These checks covered character readings, transcription completeness, and reading order. Characters without a reliable decipherment were marked [UNK] to avoid treating conjectures as established readings. Details of the experts' backgrounds and verification procedures are now provided in Appendix A.5 (Expert Annotation and Validation), as well as in our response to Q2 from Reviewer 46j3.
We will publicly release all images, annotations, and evaluation code for Chronicles-OCR upon acceptance, allowing researchers to use the materials, reproduce the evaluation, and develop improved methods. Beyond the benchmark release described in the manuscript, we also plan to release additional data from the same source collections as the previously unpublished materials, to support research on ancient-script recognition, cultural heritage digitization, and the digital humanities.
Weakness 4: Dataset size, scope, and comparison with existing resources
Revision locations: Sections 2.3 and 3.1–3.3; Table 1; Appendices A.1–A.2 and B.1–B.3; Tables A.1–A.2.
Response: Thank you for this suggestion. The revised manuscript documents the benchmark's script and task coverage in Table 1, per-script composition in Table A.1, and physical-media distribution in Table A.2. We explain below how these properties differ from those of existing resources and provide additional image-attribute comparisons and overlap checks for this response.
Chronicles-OCR is designed to evaluate historical-script understanding from native visual inputs across diverse physical media. The main evaluation preserves the layout, background texture, and carrier context of each input image, without further text-region cropping or decomposition into isolated characters. The data include digitized oracle-bone rubbings, vessel-inscription rubbings, seal impressions, and calligraphy on paper or silk, as well as images of steles, cliff inscriptions, plaques, and portable antique objects. These materials allow us to evaluate localization, recognition, and continuous transcription across different media and visual contexts.
The benchmark contains 2,800 evaluation samples, with 400 for each of seven scripts. Equal sample counts facilitate comparisons across scripts. Character spotting and fine-grained archaic character recognition are evaluated on Oracle Bone, Bronze, and Seal scripts; text parsing and script classification cover all seven. This design supports evaluation across scripts and media.
We provide the following comparisons to make these differences more concrete.
For Oracle Bone, Chronicles-OCR contains more characters per image at a smaller relative scale. Since OBIMD provides oracle-bone rubbings and facsimiles, we compared the rubbing images using a common protocol based on the existing character boxes in both datasets. The median number of visible characters per image is 28 for Chronicles-OCR versus 7 for OBIMD. Median character height, normalized by image height, is 4.89% versus 10.92%, respectively. These differences provide distinct input conditions for evaluating localization, spatial relationships, and transcription of multiple characters within an image.
For Bronze and Seal, we preserve native artifact images, including their text layout and carrier context. Chronicles-OCR contains 400 images per script, covering vessel-inscription rubbings, seal impressions, and calligraphy on paper, without further cropping into isolated characters. EVOBC and GEVO provide character-level images that preserve individual glyphs but not the original multi-character layout or broader carrier context. Chronicles-OCR therefore supports character localization and reading within the original visual context. Because the original layouts and carrier context cannot be recovered from isolated-character crops, this evaluation setting cannot be reproduced simply by resampling or combining existing character-level datasets.
For Clerical, Regular, Running, and Cursive, Chronicles-OCR covers diverse artifact carriers beyond historical books and documents. M5HisDoc provides a historical-document recognition benchmark, while HisDoc1B provides a large historical-document corpus; both center on books and documents. Our evaluation covers calligraphy, steles, cliff inscriptions, plaques, and portable antique objects, preserving their native layouts and carrier context. Of the 1,600 images in these four scripts, 511 belong to the Monument category and 476 to the Antique category, totaling 987 images (61.7%) (Table A.2). Each script is represented across calligraphy, monuments, and antiques. The resulting variation in physical carriers and visual context cannot be reproduced simply by resampling the script subsets of these document-focused resources.
Retaining this native visual information also supports evaluation at different levels of granularity. For Oracle Bone, Bronze, and Seal, the character-box annotations can be used to extract individual characters for recognition experiments. Subsets can also be selected by script or medium to study particular domains. The same source images can therefore support both evaluation in context and evaluation on finer-grained crops. By comparison, spatial relationships and carrier information removed during character cropping are difficult to reconstruct by combining samples. Resampling historical documents likewise does not add coverage of steles, plaques, or antique objects. Preserving native visual context across diverse media is an important contribution of Chronicles-OCR alongside these resources, and one that is difficult to obtain through sampling and concatenation alone.
We thank the reviewer for the careful and detailed feedback. We greatly appreciate these suggestions, which have helped us improve Chronicles-OCR. We have addressed each comment with additional experiments, analyses, or clarifications, and incorporated the new experimental results into the main text and supplementary material. In the responses below, we identify the relevant sections, tables, and figures in the revised manuscript and explain the corresponding experiments and analyses.
Weakness 1 & 2: Prompt sensitivity and statistical reliability
Revision locations: Section 4.1; Appendices B.1 and E.1; Table E.1.
Response: Thank you for the suggestions on prompting and statistical reliability. We conducted additional experiments with multiple prompts and repeated generations to examine both the effects of prompt wording and the variability of responses under a fixed prompt.
We evaluated Qwen3.5-35B-A3B and Qwen3.5-397B-A17B on the same 700 images, with 100 images per script. Character spotting and fine-grained archaic character recognition used the 300 Oracle Bone, Bronze, and Seal images; text parsing and script classification used all 700 images. We report parsing results separately for the archaic scripts (Oracle Bone, Bronze, and Seal) and the mature scripts (Clerical, Regular, Running, and Cursive).
Each task used three prompts. P0 is the original prompt from Appendix B.1. P1 rephrases and reorganizes the instructions while retaining the task requirements and output format. P2 retains the task requirements and clarifies format constraints, including field names, answer prefixes, and category names.
All runs used zero-shot, single-turn, non-thinking inference. Input images and decoding settings were held fixed across prompts, with temperature set to 0.6 and top-p to 0.95. We generated five responses with different random seeds for each applicable image–task–prompt combination, yielding 60,000 responses across the two models. Before scoring, we applied the same output-normalization rules to all conditions: numeric script labels were mapped to the corresponding category names, and
bboxandbbox_2dwere accepted as equivalent bounding-box fields. Responses that could not be extracted remained in the evaluation denominator.The table reports mean ± sample standard deviation over five repetitions, with all metrics on a 0–100 scale. Within each repetition, we first averaged image scores for the relevant evaluation group, then calculated the mean and standard deviation of these five averages. Parsing is reported as 100×NED, where NED is the score defined in Eq. (1): one minus the normalized Levenshtein edit distance, with higher values indicating better performance.
Prompt effects are modest under this common scoring protocol. The difference between the highest and lowest prompt means ranges from 0.28 to 2.61 points across the reported model–evaluation settings. The relative ranking of the two models is unchanged. For archaic-script parsing, the range is below 0.7 points for both models; for mature-script parsing, it is 0.95 points for A3B and 2.61 points for A17B.
For both A3B and A17B, parsing scores are substantially higher for mature scripts than for archaic scripts under each of the three prompts, P0, P1, and P2.
Recoverable output-format errors had little effect on the scores. To check this directly, we conducted an additional answer-extraction audit using the original inference outputs. The evaluation code already accommodates code fences and several answer-prefix variants. We additionally recovered unambiguous predictions from nonstandard outputs, retaining the emitted characters and the original coordinate-processing rules. This check required no new inference, and reference answers were not used during extraction.